Most of the existing proxy design is to proxy the entire IP range of all the datagram, but can not achieve the specified IP+Port proxy and support custom protocol header, today we talk on macOS how to achieve a complete application layer Port proxy design to do a discussion, welcome coin, like, forward. No, no, no bricks are not welcome

The status quo

Existing VPN solutions usually use virtual network cards, such as openVPN and the next generation VPN Wireguard, to directly implement traffic hijacking and proxy forwarding at the application layer. Because the virtual NIC works on the network layer network protocol stack, the specific port cannot be filtered, so traffic hijacking and proxy of the specified port can be implemented. The network kernel driver acts on the kernel state to realize more flexible traffic hijacking, such as socket filter, IP filter and so on. However, its application in kernel mode can not meet the general trend that Apple has decided to abandon the network kernel driven extension. The new system network extension scheme provided by Apple is the main choice to implement proxy now and in the future.

Transparent proxy scheme

In order to meet the zero-trust minimum permission control principle, traffic hijacking in a specified port range is required to forward traffic through application layer schemes, such as HTTP proxy and SOCK5 proxy. Apple should have considered this requirement and provided an effective solution to implement transparent proxy by extending the related classes NEAppProxyProvider and NEDNSProxyProvider. Its realization principle is shown in the figure below:

You can specify NENetworkRule to forward TCP/UDP packets to the system extension process for processing. Forwarding rules can specify IP+Port or domain name and inbound/outbound direction. Therefore, traffic can be forwarded to a specified Port or Port range to control the minimum Port permission. However, the above scheme is not perfect, it can not fully implement DNS traffic hijacking, such as command line tools. But Apple has done it all for you by providing an extended DNS proxy, which you can see below:

It can intercept all TCP/UDP DNS query packets and forward them to the DNS system extension process at the application layer, thus realizing DNS proxy.

Transparent proxy solution issues and solutions

The transparent proxy scheme mentioned above is “perfect”, which meets the REQUIREMENTS of TCP/UDP proxy and DNS proxy, and the rules are of small granularity, unlike the virtual network card, which needs to specify network layer IP rules for forwarding. However, not all schemes are perfect. Packets hijacked by transparent proxy schemes at the application layer are TCP/UDP payload packets that do not contain the transport layer protocol header. Therefore, it is difficult to modify or customize the protocol header. Apple’s point of view will not provide you with a “unified world” system extension solution, so Apple provides a packet filtering system extension NEPacketTunnelProvider to help you solve the above problems. Its realization principle is shown in the figure below:

It is actually a proxy solution in the form of a virtual nic. However, the virtual NIC is implemented by the system for you and provides an application-layer encapsulation interface to configure the virtual NIC (such as IP address and MTU) and routing rules. Therefore, in the user mode, the packet filtering system extends the process to intercept the traffic of the specified IP address, and obtains the IP datagram (including the IP data header), so that the network layer packet modification, encapsulation and forwarding can be realized. Now that we have access to the complete IP datagram, we can make changes based on it, such as implementing TCP header changes and parsing TCP/UDP protocol headers to implement port filtering and proxy forwarding.

Port proxy implementation scheme

Since the system has provided the IP datagram filtering mechanism, that is, it can realize the proxy scheme of virtual network card, which is also the implementation scheme of most VPN clients at present. However, macOS also provides a user-mode interface to create and configure virtual network cards, rather than being extended through a system-encapsulated packet filtering system. However, the packet filtering system extension scheme encapsulated by the system has more advantages than the customized virtual network card. It can be seen from the implementation framework of the whole system extension, as shown in the figure below:

It provides complete system extension daemon, rule management and session management, and is easier to implement and maintain than a custom implementation. Therefore, the subsequent virtual network card is implemented based on the packet filtering system extension scheme.

The virtual nic is faulty

However, filtering rules are configured for virtual nics based on the network layer, that is, only IP addresses can be filtered, but ports cannot be filtered in the transparent proxy solution. Therefore, if the port range proxy is implemented, or the TCP protocol header is modified and forwarded on the basis of the proxy, further optimization and improvement should be made on the basis of again.

Tap Device Forwarding

Because the TAP device that simulates the data link layer can implement the packet routing function, it can implement unnecessary IP packet forwarding based on this function. However, Apple does not open corresponding interfaces to create TAP devices at the application layer, so this scheme has drawbacks in the system to implement port range proxy at the application layer.

Top forward

Since you can get all the complete IP datagrams of the specified IP rule, you can complete the proxy forwarding, and the proxy server will implement the complete port filtering proxy. However, this scheme violates the principle of minimum port permissions and causes proxy server performance burden.

A local forward

To implement the principle of minimum port permission, all intercepted IP packets are parses locally and unwanted port packets are forwarded to the physical network adapter in the form of raw sockets. The physical network adapter of the local device is responsible for the traffic egress, that is, the virtual network adapter is restored to intercept traffic. The specific scheme framework is shown as follows:

Scheme in this paper, the

The key link of the scheme is to filter unwanted IP packets and forward them to the physical network adapter instead of the proxy server, such as port filtering and forwarding, DNS filtering and forwarding, user-defined TCP protocol header and IP packet filtering. The key mechanism is Packet Filter, a “firewall” technology based on macOS system.

PF firewall (full name: Packet Filter) is a mechanism for filtering TCP/IP traffic and translating network addresses on Unix. It also provides shaping and control of TCP/IP traffic, bandwidth control, and Packet priority set control. The underlying system of macOS inherits the firewall mechanism as well as freeBSD. In the user mode, it provides the PFCTL command for filtering configuration. For example, the redirection configuration is as follows:

RDR pass on EN1 proto TCP from 110.242.68.3 port 80 to EN1 :network -> (utun3)Copy the code

The preceding rule changes the IP address of the destination TCP packet (IP+Port:110.242.68.3:80) sent to EN1 to the virtual IP address of UTun3. The default packet is filtered without filtering rules.

The redirection function of the PF filter is used to prevent external traffic from flowing into the internal network through the NAT device. The PF filter is used to forward the external traffic to the socket of the virtual nic, and then the local application receives the traffic.

It can also be used for packet filtering. For example, the configuration of IP packet filtering is as follows:

Block in proto TCP from 110.242.68.4 port 80 to anyCopy the code

In this way, the external traffic can be blocked by TCP 110.242.68.4.80 or the internal traffic can be blocked by block out XXX to implement packet filtering.

Ok, so much for today’s chat, have questions or discussion partners can private message, message