Hetzner and Proxmox: pfSense as Gateway

As many of my readers know, I use some Hetzner servers in combination with the virtualization solution Proxmox.

To make the setup a little more secure and not rely on the Hetzner firewall, I recently took care of shielding my virtual machines behind a pfSense VM. The network configuration was not quite trivial and therefore I would like to give my readers a little insight.

Prerequisites

  • dedicated Hetzner server (here: EX41-SSD)
  • running Proxmox system
  • an additional IP for the pfSense VM with MAC address (Hetzner-Robot)
  • Subnet, routed to the pfSense VM

As soon as these requirements are met, we can start configuring the system.

Proxmox Configuration

My Proxmox configuration looks like this (/etc/network/interfaces):

auto lo
iface lo inet loopback

iface eth0 inet manual

auto vmbr0
iface vmbr0 inet static
        address  <SERVER-HAUPT-IP>
        netmask  255.255.255.255
        gateway  <SERVER-HAUPT-GATEWAY>
        pointopoint <SERVER-HAUPT-GATEWAY>
        bridge_ports eth0
        bridge_stp off
        bridge_fd 0

        up ip route add 192.168.0.0/16 via 138.201.203.59 dev vmbr0
        up ip route add 172.16.0.0/12 via 138.201.203.59 dev vmbr0
        up ip route add 10.0.0.0/8 via 138.201.203.59 dev vmbr0

        up sysctl -w net.ipv4.ip_forward=1
        up sysctl -w net.ipv4.conf.eth0.send_redirects=0

# Virtual switch for DMZ
# (connect your firewall/router KVM instance and private DMZ hosts here)
auto vmbr1
iface vmbr1 inet manual
        bridge_ports none
        bridge_stp off
        bridge_fd 0

You can read the main server IP and the main server gateway from the Hetzner Robot – there are actually not many sources of error here.

pfSense Configuration

WAN/LAN configuration

The pfSense configuration is not really more complicated. In the Proxmox interface we create the VM with two network devices, one bound to vmbr0 – our WAN interface – and one to vmbr1 – the LAN interface.

After the pfSense installation, we assign the interfaces in pfSense accordingly and configure the WAN interface:

pfSense WAN Configuration

The configuration of the LAN interface should be self-explanatory – private subnet, no gateway.

Virtual IP Address

To be able to use our subnet behind pfSense, we now have to enter each IP to be used under „Firewall -> Virtual IPs“. Here it is possible to enter all IP addresses of the subnet, we do not need a broadcast or host address.

Internal IP addresses

We are now able to distribute their IP addresses to our clients in the LAN. Either you use the DHCP server of pfSense and assign static reservations in the simplest case or you configure the clients manually.

I decided to use the DHCP variant, but I don’t want to go into the configuration here, because it’s quite self-explanatory.

1:1 NAT

In order to be able to reach our internal clients now also from the Internet, we still have to configure the 1:1 NAT.

1:1 NAT configuration

Firewall configuration

Finally, it is up to the administrator to configure the firewall correctly. There are so many solutions here, I don’t want to go into them any further. which ports are opened where, the operator of the firewall has to decide for himself.

ICMP ping rule

I find it quite pleasant to be able to check the accessibility of a host from the outside with a ping. I always create this rule in the pfSense firewall first:

pfSense ICMP Ping Rule

The internal IP or the internal subnet must of course be specified as the destination.

Bottom line

The configuration of a pfSense VM at Hetzner is not quite trivial, but it can be easily done. A little Trial & Error is included, but that’s always the case with firewalls. As a little tip I would like to give you not to block the access to the Proxmox web interface too early, so that in case of doubt you still have access to the pfSense console when you configure yourself.

I will be happy to answer any questions or to receive information about my setup. It’s certainly not perfect, I know that, but it works reliably for me, which is why I’m actually quite happy about it. If somebody has a good tip for you: please feel free to add it to your comments!

IPv6 configuration

The IPv6 configuration I have a own article.

Hetzner & Proxmox: Network Configuration

proxmox_logo_standard_hex_400px

As already known, I use a Proxmox setup on servers of the German host Hetzner for myself and my customers. The wiki article about the Proxmox configuration is unfortunately not really up-to-date or easy to understand…

Often my readers asked me to pour the basic network configuration into an article: said, done – here’s the article!

Preparation

In principle, you don’t have to prepare much to use a simple Proxmox setup.

Of course we need a dedicated server with installed Proxmox image and an additional IP from the Hetzner robot. We assume the following IP addresses:

  • dedicated main IP
    • IP: 138.201.203.16
    • Gateway: 138.201.203.1
  • Additional IPs (ordered via Robot – without MAC address!)
    • 138.201.203.49
    • 138.201.203.52
    • 138.201.203.56
    • 138.201.203.57

We also enable IP forwarding by executing the following command:

sysctl -w net.ipv4.ip_forward=1

We permanently enable forwarding by inserting the corresponding line in /etc/sysctl.d/99-hetzner.conf or removing the comment character.

More preparation is not necessary in this case.

Configuration Proxmox Host

On the Proxmox host, the /etc/network/interfaces file looks like this:

auto eth0
iface eth0 inet static
        address  138.201.203.16
        netmask  255.255.255.255
        gateway  138.201.203.1
	pointopoint 138.201.203.1
		
auto vmbr0
iface vmbr0 inet static
	address  138.201.203.16
	netmask  255.255.255.255
	bridge_ports none
	bridge_stp off
	bridge_fd 0
	bridge_maxwait 0
		up ip route add 138.201.203.49/32 dev vmbr0
		up ip route add 138.201.203.52/32 dev vmbr0
		up ip route add 138.201.203.56/32 dev vmbr0
		up ip route add 138.201.203.57/32 dev vmbr0

It is important here to enter the main IP (here: 138.201.203.16) for the eth0 and vmbr0 devices as a quasi „double“ entry. The gateway address must also be entered at gateway and pointopoint – Attention: pointopoint has only one t in the middle, this is not a typo!

A reboot later should display two network devices – eth0 and vmbr0.

Virtual Machine Configuration (Linux)

After the installation I change the file /etc/network/interfaces as follows:

auto eth0
iface eth0 inet static
        address 138.201.203.49
        netmask 255.255.255.255
        pointopoint 138.201.203.16
        gateway 138.201.203.16

After a restart the VM should be reachable via ping as well as via SSH.

Virtual Machine Configuration (Windows)

Sometimes you also need a Windows machine, which is why I would like to show the configuration here simply by screenshot:

he DNS servers can of course be customized, but the rest should look the same – with your own IP addresses, of course.

The option „Check settings on exit“ must not be activated, since Windows – just like Debian – checks the configuration and would report an error – despite these error messages of the The virtual machines are running completely stable.

Bottom line

The configuration is actually quite simple and very clear. By adding a single line in the network configuration, additional IP addresses can be added and superfluous ones removed. The configuration runs very stable, you don’t have to bother with DHCP servers and the performance is also satisfactory.

I hope I have been able to help some readers – and of course Google visitors – with my instructions and of course I am happy to support you in setting up your own setup – just contact me for a non-binding offer!

Questions, criticism and hints to the article I accept naturally gladly in the comments – please have however also understanding that I can offer free support only in certain framework.

Debian Buster: SSH „Connection refused“

Nach einer Neuinstallation oder dem Upgrade auf Debian Buster konnte ich mich nicht mehr per SSH an meinem Server anmelden. Mich begrüßte ständig nur die Meldung Connection refused.

Interessanterweise konnte ich das Problem immer „lösen“, indem ich mich per Konsole direkt an der Maschine angemeldet hatte (geht dank Proxmox ja ganz einfach). Nach nur einem Login funktionierte die SSH-Verbindung wieder problemlos.

Nach einiger Recherche bin ich dann auf diesen Bug gestoßen. Um es kurz zu machen benötigt die Generierung von zufälliger Entropie mehr Zeit als zuvor unter Debian Stretch – der Start des OpenSSH-Servers benötigt allerdings ebendiese Entropie und dauert daher sehr viel länger als zuvor.

Eine einfache Lösung des Problems ist die Installation des Pakets haveged. Direkt nach der Installation dieses Pakets und einem erneuten Neustart ist das Problem behoben.

Was ist haveged? Kurz umrissen haben es die Kollegen des Linux Magazin:

Das Verfahren macht sich den Umstand zunutze, dass moderne Prozessoren Elemente zur Verzweigungsvorhersage (Branch Prediction), Caches, Pipelines und vieles mehr besitzen. Das normale Benutzen der CPU löst ein Trommelfeuer an Statusänderungen bei Tausenden dieser Elemente aus, und genau daraus produziert das Havege-Verfahren viel und hochwertigen Zufall.

Linux Magazin

Das erklärt auch, wieso der Start nun sehr viel schneller von statten geht. Vielleicht hilft es ja dem ein oder anderen verzweifelten Buster-User…

Proxmox: ungenutzte Kernel entfernen

Heute nur ein Kurztipp zum Sonntag!

Jeder Admin kennt das: man nutzt seinen Proxmox-Server schon ein paar Monate und es sammeln sich immer mehr ungenutzte Kernelversionen an… möchte man diese nun entfernen, kann man das natürlich händisch mit dpkg, apt oder aptitude machen – einfacher ist es aber mit pvekclean!

pvekclean ist ein kleines Skript von Jordan Hillis, das man gratis bei Github bekommt. Die Installation ist sehr simpel:

git clone https://github.com/jordanhillis/pvekclean.git
cd pvekclean
chmod +x pvekclean.sh

Fertig!

Beim ersten Start fragt das Skript, ob es sich zum einfacheren Aufruf in den Ordner /usr/local/sbin kopieren darf; kann man halten wie man möchte, ich finde es ganz angenehm. Hat man die Nachfrage bestätigt, kann man das Script ganz einfach via

pvekclean

aufrufen. Die Ausgabe bei mir sah so aus:

█▀▀█ ▀█ █▀ █▀▀   █ █ █▀▀ █▀▀█ █▀▀▄ █▀▀ █
█  █  █▄█  █▀▀   █▀▄ █▀▀ █▄▄▀ █  █ █▀▀ █
█▀▀▀   ▀   ▀▀▀   ▀ ▀ ▀▀▀ ▀ ▀▀ ▀  ▀ ▀▀▀ ▀▀▀

█▀▀ █   █▀▀ █▀▀█ █▀▀▄ █▀▀ █▀▀█
█   █   █▀▀ █▄▄█ █  █ █▀▀ █▄▄▀   ⎦˚◡˚⎣ v1.2
▀▀▀ ▀▀▀ ▀▀▀ ▀  ▀ ▀  ▀ ▀▀▀ ▀ ▀▀
By Jordan Hillis [contact@nulljordanhillis.com]
___________________________________________

Usage: pvekclean [OPTION]

  -f,   --force         Remove all old PVE kernels without confirm prompts
  -s    --scheduler     Have old PVE kernels removed on a scheduled basis
  -v,   --version       Shows current version of pvekclean
  -r    --remove        Uninstall pvekclean from the system
___________________________________________

OS: Debian GNU/Linux 9 (stretch)
Boot Disk: 74% full [342M/488M used, 121M free]
Current Kernel: pve-kernel-4.15.18-16-pve
___________________________________________

[*] Boot disk space used is healthy at 74% capacity (121M free)
[-] Searching for old PVE kernels on your system...
[*] "pve-kernel-4.15.18-10-pve" has been added to the kernel remove list
[*] "pve-kernel-4.15.18-11-pve" has been added to the kernel remove list
[*] "pve-kernel-4.15.18-12-pve" has been added to the kernel remove list
[*] "pve-kernel-4.15.18-13-pve" has been added to the kernel remove list
[*] "pve-kernel-4.15.18-14-pve" has been added to the kernel remove list
[*] "pve-kernel-4.15.18-15-pve" has been added to the kernel remove list
[-] PVE kernel search complete!
[!] Would you like to remove the 6 selected PVE kernels listed above? [y/N]: Y
[*] Removing 6 old PVE kernels...DONE!
[*] Updating GRUB...DONE!
[-] Have a nice day ⎦˚◡˚⎣

Natürlich sollte man, bevor man die Abfrage zum endgültigen Löschen bestätigt, sicher sein, dass der aktuelle Kernel nicht entfernt wird. Führt man das Skript nun in regelmäßigen Abständen aus, bleibt die Kernelliste auf dem Server immer schön sauber!

Zabbix: Fehler nach Upgrade auf 4.2

Fehler im Zabbix-Frontend

„The frontend does not match Zabbix database. Current database version (mandatory/optional): 4010005/4010005. Required mandatory version: 40200000. Contact your system administrator.“

Diese nette Fehlermeldung hat mich heute nach dem Upgrade meines Zabbix Server auf Version 4.2 begrüßt.

Im Log des Servers habe ich dann den entscheidenden Hinweis gefunden, wieso das automatische Upgrade der Datenbank nicht funktioniert hatte:

[Z3005] query failed: [1071] Specified key was too long; max key length is 767 bytes [create unique index lld_macro_path_1 on lld_macro_path (itemid,lld_macro)]

Letzten Endes hatte ich offenbar beim damaligen Erstellen der Datenbank für Zabbix nicht gut genug aufgepasst und das Charset/die Kollation falsch gesetzt. Sollte euch das auch passiert sein: keine Panik!

Ihr könnt die Datenbank mit zwei einfachen Befehlen auf die empfohlenen Optionen utf8/utf8_bin umstellen, vorher sollte natürlich unbedingt ein Backup der Datenbank stattfinden!

# Herausfinden, ob das falsche Charset gewählt ist
SELECT default_character_set_name FROM information_schema.SCHEMATA S WHERE schema_name = "zabbix";

# Alle Tabellen der Datenbank "zabbix" umstellen
mysql --database=zabbix -B -N -e "SHOW TABLES" | awk '{print "SET foreign_key_checks = 0; ALTER TABLE", $1, "CONVERT TO CHARACTER SET utf8 COLLATE utf8_bin; SET foreign_key_checks = 1; "}' | mysql --database=zabbix

# Die eigentliche Datenbank "zabbix" umstellen
ALTER DATABASE zabbix CHARACTER SET utf8 COLLATE utf8_bin;

Wurden die Befehle korrekt ausgeführt, könnt ihr den Zabbix Server neugestartet werden und das Log sollte nun das korrekt gelaufene Datenbank-Upgrade ausspucken – und das Frontend wieder erreichbar sein.