Isn’t that what the logs on the proxy are for?

Short answer no, but you can add the source IP as part of the http header https://www.nginx.com/resources/wiki/start/topics/examples/forwarded/ then you have to log that bit of the header at the app level.

There can be ways of your are using ipv6, basically turning your cloud host into a router, but but ipv4 you would have to have a 1:1 mapping and setup the routing carefully to make it work.

Short answer: Don’t bother, it’s too complex to setup (unless your app is HTTP or supports the PROXY protocol). You better read your proxy logs instead.

Long answer: What you want is called “IP transparency” and require your proxy to “spoof” the IP address of the client when forwarding packets to the remote server. Some proxies do it (Nginx plus, Avi Vantage, Fortinet) but are paid services. I don’t know for free solutions as I only ever implemented it with those listed above.

This require a fairly complex setup though:

0. IP address spoofing

The proxy must rewrite all downstream request to spoof the client IP address, making it look like the traffic originates from the client at the TCP layer.

1. Backend server routing

As the packet will most likely originate from random IP on the internet, your backend server must have a way to route back the traffic to the proxy, instead of it’s default gateway. Otherwise you’d implement what is called "Direct Server Return*, which won’t work in your case (packet will be dropped by the client as originating from your backend server directly, and not from the proxy).

You have two solutions here:

• set your default gateway to the proxy over its VPN interface (don’t do that unless you truly understand all the implications of such a setup)
• use packet tagging and VRF on the backend server to route back all traffic coming from the VPN, back to the VPN interface (I’m not even sure this would work with an IPsec VPN though because of ACL…)

3. Intercept and route back return traffic

The proxy must be aware that it must intercept this traffic targeted at the destination IP of the client as part of a proxied request. This require a proxy that can bind on an IP that is not configured on the system.

So yeah, don’t do that unless you NEED to do that (trust me as I had to do it, and hated setting it up).

Edit: apparently haproxy supports this feature, which they call transparent mode

I realised I forgot to update this. Thank you to everyone that contributed, I appreciate it. This was a weird use case and barely anyone online has documented it, only a handful of places. Nevertheless, I figured it out.

So basically, you run HAProxy with the send-proxy-v2 protocol. Let’s say I’m forwarding SSH from VPS to home, I’d have the VPS running HAProxy listening on port 22. Then I’d have it forward to home on port 220. Then, on the home server, you run this amazing piece of software called go-mmproxy. Configure that to listen on port 220 and forward to localhost 22. And there you have it.

HAProxy passes the real source IP to go-mmproxy with the proxy protocol, go-mmproxy takes the proxy header and strips it from the request, spoofs the source IP address from localhost to the real source IP contained in the proxy header then makes the request to localhost. And then you also have to configure traffic to go back through localhost so go-mmproxy can pick it up and add the proxy header back to the request, to be sent back to the source.

You’re best off using the PROXY protocol assuming your application(s) support it.

The way I would solve this is by putting nginx or other reverse proxy directly on your instance in the cloud. You can use this to set one of the well known proxy headers and proxies as others have mentioned and have this then proxy to your backend instances over the VPN (even if it’s pointing to an internal nginx instance). Then the access logs on your cloud instance will also contain the actual IP address of the client, setting headers will obviously only work for HTTP traffic, there really isn’t a similar mechanism for TCP/UDP traffic as those are layer 3 and HTTP is layer 4. If you are concerned about it you can always ship the logs to somewhere on prem as well.

Is there any solution (program/Docker image) that will take a port, forward it to another host (or maybe another program listening on the host) that then modifies the traffic to contain the real source IP. The whole idea is that in the server logs I want to see people’s real IP addresses, not the server in the cloud private VPN IP.

Not that I’m aware of. Most methods require some kind of out-of-band way to send the client’s real IP to the server. e.g. X-Forwarded-For headers, Proxy Protocol, etc.

If your backend app supports proxy protocol, you may be able to use HAProxy in front on the VPS and use proxy protocol from there to the backend. Nginx may also support this for streams (I don’t recall if it does or not since I mainly use HAProxy for that).

Barring that, there is one more way, but it’s less clean.

You can use iptables on the VPS to do a prerouting DNAT port forward. The only catch to this is that the VPN endpoint that hosts the service must have its default gateway set to the VPN IP of the VPS, and you have to have a MASQUERADE rule so traffic from the VPN can route out of the VPS. I run two services in this configuration, and it works well.

iptables -t nat -A PREROUTING -d {VPS_PUBLIC_IP}/32 -p tcp -m tcp --dport {PORT} -j DNAT --to-destination {VPN_CLIENT_ADDRESS}
iptables -t nat -A POSTROUTING -s {VPN_SUBNET}/24 -o eth0 -j MASQUERADE

Where eth0 is the internet-facing interface of your VPS.

Edit: One more catch to the port forward method. This forward happens before the traffic hits your firewall chain on the VPS, so you’d need to implement any firewalls on the backend server.