Press "Enter" to skip to content

Routing back to the Internet via non-default gateway [on hold]

        My ISP with my plan, while not including a static IPv4 address on my line (PPPoE on GPON) it does include a VPS and like most VPS instances it has a static IPv4 address.
I've been trying to set it up as some sort of tunneled remote gateway/proxy, inbound; instead of public-IP-space-to-public-IP-space routing I'm using tunneling because that way the on-prem side would dial non-stop to a fixed location to bring up the VPN whereas if if tried routing it'd be pointless having an extra hop to a still moving target, the local IP address. I got the tunneling sorted out but I can't get servers to respond over the foreign gateway other than their default. I watched the firewall logs on each side of each point and ran remote Wireshark and I can see the incoming traffic does make it to the destination server(s) but out the way back there are lots TCP-Out-Of-Order or TCP Retransmissions errors. That makes me think it's asymmetric routing because of the whole non-default gateway thing. I still learning about routing so I'm just wild guessing, really. Wireshark is as helpful as your understanding of the protocols reaches. I have tried this three ways now, first I tried a site-to-site tunnel, and basically duplicated the inbound NAT from the local firewall on the remote one: enter image description here Then took another approach instead of port forwarding directly to the servers, I created a new interface on the local firewall to handle the "cloud" traffic. I just duplicated the main interface's ruleset onto that one. On the cloud side I forwarded everything to that single IP address. enter image description here Both had similar results: enter image description here Then since the tunnel was (still is) up on an actual physical box on layer 2, I figured I would just cut the middle man: I bridged the interfaces on the VPN endpoint, gave the cloud firewall an address within the servers' subnet and did the first ruleset from the beginning because I didn't think ahead. enter image description here ...and: enter image description here AGAIN! Should the traffic have no problem getting back regardless of the gateway? If the firewall is doing NAT, shouldn't the traffic/server be none the wiser and just reply to the firewall's network address translated location and be it? I'm very confused and a little more frustrated than I want to admit that I can't make it work. I thought about setting the remote instance as the default gateway but unlike the local connection that one has a transfer quota. It is a multi-terabyte thing, but that's flies by on fiber. I'll be super grateful on whatever you can school me on. Thanks!


I didn't think it mattered other than the topology of the network, but I was ask to detail devices; except for the on-prem tunnel endpoint, all of the firewalls are virtualized, all of them are either pfSense or OPNsense, basically the same thing. I'm not using VLANs except for the communication between the physical VPN endpoint and the main firewall. I'm avoiding any complexity until I manage to reach a server from the remote firewall. Ping does work from any point of the network to any other point of the network as long as it's not coming from the remote, non-default gateway. I don't think there's anything else relevant because the hardware is abstracted by the hypervisors. Thanks!

Be First to Comment

Leave a Reply

%d bloggers like this: