Proxmox NAT & CSF

AlwaysSkint · November 2019

I have the above setup, at long last nearing the point of using it, rather than idling. I can connect to a VM from the 'net via redirected ports and the VM is able to connect to the world.
When I ssh in to the VM, it is being logged as coming from the host server, not my home IP, so I'm obviously missing something in the setup. Any pointers for me?
Ta.

InceptionHosting · November 2019

The NAT les servers just use simple SNAT per external IP e.g.
iptables -t nat -A POSTROUTING -s 192.168.0.0/24 -o br0 -j SNAT --to 104.105.106.107
Then port forwarding with DNAT e.g for ssh access
iptables -t nat -A PREROUTING -p tcp -d 104.105.106.107 --dport 321 -j DNAT --to 192.168.20.3:22
and then a port range through DNAT for the rest:
iptables -t nat -A PREROUTING -p tcp -d 104.105.106.107 --dport 301:320 -j DNAT --to-destination 192.168.20.3

But that assumes a few things, its been an age since I used proxmox properly but I remember it had a number of different options on how to pass traffic through when adding an interface type for the VM's the above would only work in a bridged network, I do seem to remember (going back maybe 7 - 8 years) that the default NAT setup in proxmox was to masquerade which may be the cause of your problem, I think a bond would do the same so you need bridged or a seperate VLAN for guest networks using a manually configured bridge would be my guess.

Could be wrong, just guessing without seeing.

AlwaysSkint · November 2019

I'm using a bridged network on an OVH host. I have the following if that makes things clearer.
Part of my iptables-set.sh/csfpost.sh

echo 1 >/proc/sys/net/ipv4/ip_forward
echo 1 > /proc/sys/net/ipv4/conf/vmbr1/proxy_arp
/sbin/iptables -t nat -A PREROUTING -i vmbr0 -p tcp --dport 8000 -j DNAT --to 10.0.0.100:80
...
iptables -t nat -A POSTROUTING -s '10.0.0.0/24' -o vmbr0 -j MASQUERADE
/sbin/iptables -F FORWARD
/sbin/iptables -P FORWARD ACCEPT

In /etc/network/interfaces

post-up iptables -t nat -A POSTROUTING -s '10.0.0.0/24' -o vmbr0 -j MASQUERADE
post-up iptables -t nat -A PREROUTING -i vmbr0 -p tcp --dport 8000 -j DNAT --to 10.0.0.100:80

There appears to be a duplication above but it's taken me ages to get packets traversing the host. Seems like I'm missing SNAT entries .. I'm sure I spotted them someplace. :-S

InceptionHosting · November 2019

I think the issue is the MASQUERADE, so it is well, masquerading if i were you I would strip it all back and follow the examples above, its actually a lot more simple than it seems to achieve what you want.

AlwaysSkint · November 2019

I was following a mashup of 'instructions' on the web and got pretty confused along the way. I also have nginx reverse proxy and webserver VMs on this host, so I'll take my time over the reconfiguration. Thanks!

mikho · November 2019

I was about to write something useful, @AnthonySmith beat me to it.
SNAT is the way to go unless you either want to MASQUERADE the original IP or in some cases uses a dynamic IP.

AlwaysSkint · November 2019

Latest configuration:
/etc/network/interfaces (snippet)

post-up echo 1 > /proc/sys/net/ipv4/ip_forward
post-up iptables -t nat -A POSTROUTING -s '10.0.0.0/24' -o vmbr0 -j SNAT --to 178.32.xxx.xxx
post-down iptables -t nat -D POSTROUTING -s '10.0.0.0/24' -o vmbr0 -j SNAT --to 178.32.xxx.xxx
post-up iptables -t nat -A PREROUTING -d 178.32.xxx.xxx -p tcp --dport 8000 -j DNAT --to 10.0.0.100:80
post-down iptables -t nat -D PREROUTING -d 178.32.xxx.xxx -p tcp --dport 8000 -j DNAT --to 10.0.0.100:80

I couldn't get outbound (eg. apt update) from the VM when the host had CSF active, until I placed this in csfpost.sh

#!/bin/bash
echo 1 >/proc/sys/net/ipv4/ip_forward
echo 1 > /proc/sys/net/ipv4/conf/vmbr1/proxy_arp
/sbin/iptables -F FORWARD
/sbin/iptables -P FORWARD ACCEPT
/sbin/iptables -t nat -A POSTROUTING -s '10.0.0.0/24' -o vmbr0 -j SNAT --to 178.32.xxx.xxx

I dunno the consequences of those iptables FORWARD directives, however.. will need to do a search.

EDIT: VM still shows host IP for my laptop doing an ssh to it. In other words the originating IP is being stripped during the forwarding.

kmmm · November 2019

I routed all traffic to the bare metal nginx instance before passing traffic to corresponding virtual network interfaces (created and assigned by Proxmox, ofc). This is dead simple, and I do not need to mess with iptables since I am also using block IPs based on ipset. My setup is for personal use, so no idea how it really effects the performance of Proxmox host under load.

vimalware · November 2019

What I do is simpler
Access to the NAT vm's is via a http(s) load balancer KVM(dedicated IPv4)

The individual KVMs serial terminals can be 'accessed' by ' qm terminal 101' in a tmux ssh/mosh session to the proxmox host.

My vm templates don't have openssh-server. I install it when required

AlwaysSkint · November 2019

A proxmox console is one way around it but doesn't address other services. I used ssh as an example because it is relatively simple to monitor activity to it. You also assume that an additional IP is available (I know I didn't cover that aspect) but that's not an option on Kimsufi servers, unfortunately. I have another proxmox instance elsewhere with /29, so that this subject is never considered, except for IPv6 trials.

TLDR; Originating IP is replaced by host IP at the VM

terrorgen · November 2019

Use IPv6

AlwaysSkint · March 2020

Sorry to drag up this old thread again.. I'm back trying to get NAT working, though I am a bit further forward than previously.
On a different server, again with only one public IP, with Proxmox handling VMs; it's nginx that I've been tearing my hair out with. I'm trying to use it on the main server to redirect to the appropriate VMs.
I can access VMs from the outside world by using the server defined ports but I can't access using the VMs internal ports. There's also the issue of how to handle SSL certs.
Example:
http://tx200.vmdomain.com:80 <-- 502 Bad Gateway
http://tx200.vmdomain.com:42080 <-- Testing 123 (nginx default page)
https://tx200.vmdoamin.com:443 <-- PR_END_OF_FILE_ERROR
https://tx200.vmdomain.com:42443 <-- (Firefox:) Unable to connect
https://tx200.vmdomain.com:10000 <-- Error code: SSL_ERROR_RX_RECORD_TOO_LONG
https://tx200.vmdomain.com:42000 <-- webmin loads, with self-certified, correct FQDN SSL warning

This is what I've cobbled together for the nginx tx200.vmdomain.com.conf

server {
server_name tx200.vmdomain.com www.tx200.vmdomain.com;
listen 80;
location / {
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_pass http://tx200.vmdomain.com:42080;
proxy_read_timeout 90;
proxy_redirect http://tx200.vmdomain.com:42080 http://tx200.vmdomain.com:80;
}
}
server {
server_name tx200.vmdomain.com www.tx200.vmdomain.com;
listen 443;
location / {
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_pass https://tx200.vmdomain.com:42443;
proxy_read_timeout 90;
proxy_redirect https://tx200.vmdomain.com:42443 https://tx200.vmdomain.com:443;
}
}
server {
server_name tx200.vmdomain.com www.tx200.vmdomain.com;
listen 10000;
location / {
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_pass https://tx200.vmdomain.com:42000;
proxy_read_timeout 90;
proxy_redirect https://tx200.vmdomain.com:42000 https://tx200.vmdomain.com:10000;
}
}

I've tried without proxy_redirect but made no odds.
Obviously I have the IP addresses defined in the server hosts file and the firewall has a block of ports opened.
VMs can access the outside world and I can ssh in using the appropriate redirected port.
As well as the KVM above, I have a LXC that unsurprisingly displays similar symptoms.
Next thing I'm about to try is copy the self-cert SSL from webmin over to the default Apache location to see if it resolves the 443 issue.
I feel that I'm close but no cigar.

Neoon · March 2020

HAProxy takes care of TLS/Plain inspection with SNI and forwards it to the right machine:
https://gist.github.com/Ne00n/b52e112afa5d587178920e0164691083
Maybe helpful.
https://github.com/Ne00n/NanoKVM-Tools

AlwaysSkint · March 2020

@Neoon Thanks! Will take a look.

Edit: I'm definitely missing something there.. The 'tools' appear to use DHCP which I specifically don't want use - will only have about 4 VMs max.
The HAProxy SNI TLS + HTTP looks promising, assuming I can extend to other services. Goes off to read about yet another prog.

Neoon · March 2020

@AlwaysSkint said:
@Neoon Thanks! Will take a look.

Edit: I'm definitely missing something there.. The 'tools' appear to use DHCP which I specifically don't want use - will only have about 4 VMs max.
The HAProxy SNI TLS + HTTP looks promising, assuming I can extend to other services. Goes off to read about yet another prog.

The HAProxy config is just copy & paste, also DHCP is just optional for every network, its not mandatory.
Same goes for radvd for IPv6.

AlwaysSkint · March 2020

Think I'm too stupid for this game. :-(
Same errors as above, apart from port 10000 which now has PR_END_OF_FILE_ERROR.
(that Testing 123 page is Apache on the VM, not nginx.)

frontend http
bind *:80
mode http
#Define hosts
acl host_tx100.domain.com hdr(host) -i tx100.domain.com
use_backend tx100.domain.com if host_tx100.domain.com
acl host_tx200.domain.com hdr(host) -i tx200.domain.com
use_backend tx200.domain.com if host_tx200.domain.com
backend tx100.domain.com
server tx100.domain.com 10.0.0.100:80
mode http
backend tx200.domain.com
server tx200.domain.com 10.0.0.120:80
mode http
frontend https
bind *:443
mode tcp
acl tls req.ssl_hello_type 1
tcp-request inspect-delay 5s
tcp-request content accept if tls
#Define hosts
acl host_tx100.domain.com_https req.ssl_sni -i tx100.domain.com
use_backend tx100.domain.com_https if host_tx100.domain.com_https
acl host_tx200.domain.com_https req.ssl_sni -i tx200.domain.com
use_backend tx200.domain.com_https if host_tx200.domain.com_https
backend tx100.domain.com_https
server tx100.domain.com 10.0.0.100:443
mode tcp
backend tx200.domain.com_https
server tx200.domain.com 10.0.0.120:443
mode tcp
frontend webmin
bind *:10000
mode tcp
acl tls req.ssl_hello_type 1
tcp-request inspect-delay 5s
tcp-request content accept if tls
#Define hosts
acl host_tx100.domain.com_webmin req.ssl_sni -i tx100.domain.com
use_backend tx100.domain.com_webmin if host_tx100.domain.com_webmin
acl host_tx200.domain.com_webmin req.ssl_sni -i tx200.domain.com
use_backend tx200.domain.com_webmin if host_tx200.domain.com_webmin
backend tx100.domain.com_webmin
server tx200.domain.com 10.0.0.100:12321
mode tcp
backend tx200.domain.com_webmin
server tx200.domain.com 10.0.0.120:10000
mode tcp

Am I right in thinking there's no need to have a httpd service running now, as haproxy takes over?

Neoon · March 2020

HAProxy does forward HTTP/HTTPS to a specific IP:PORT, you need to have a webserver running at that destination.
If you have no webserver running, then you get errors.

Try plain, if plain works, setup LetsEncrypt, then https will work also.
You can setup https like on any normal webserver since HAProxy forward encrypted https traffic.

Use pastebin if you post any configs, I wont read that otherwise.
The formating is painful.

AlwaysSkint · March 2020

Yes, of course the VMs have a httpd running, I meant at the server.
Yep the code flag on the forum goes mental - I'll try a pastebin.
https://pastebin.com/gDhXQZ7W

Neoon · March 2020

I see no error does plain work?

AlwaysSkint · March 2020

@Neoon
Thanks for bearing with me. Turnkey NextCloud/LAMP confused issues a bit due to redirections at the VM.
Current situation: port 80 requests work fine. 433 works only after I access webmin via the NAT 41000/42000 ports and request LE SSL certificates, plus add them manually to the webmin defined virtual hosts. A very convoluted way of doing things but at least I'm getting there.

Neoon · March 2020

Yes, SSL won't work until you setup valid certificates.
You just need a for example and nginx with LetsEncrypt and HAProxy forwards the requests fine.

I do not know your exact setup, you need to debug and check what's wrong. eg log files.
As far as I know it should work as expected.

Proxmox NAT & CSF

Comments