Tricks to Running HAProxy on pfSense Embedded

HAProxy is available as an addon module for pfSense 1.2.3. This makes it really easy to have pfSense control the gateway and load balancing. There are a couple of tricks to getting it all up and running.

Although everything looked good in the webgui HAProxy just wouldn't start. After logging in it seemed that there were 2 problems, firstly as mentioned in the forums the IP addresses must be an interface or CARP addresses not Virtual IPs for HAProxy to work and secondly the file descriptor limits have to be increased. To increase the file descriptor limits run the following commands from a shell on pfSense.

mount -o rw /dev/ufs/pfsense1  /
echo >> /etc/sysctl.conf
echo '# File descriptor limits for HAProxy' >> /etc/sysctl.conf
kern.maxfiles=2000011 >> /etc/sysctl.conf
kern.maxfilesperproc=2000011 >> /etc/sysctl.conf
sysctl kern.maxfiles=2000011
sysctl kern.maxfilesperproc=2000011
mount -o ro /dev/ufs/pfsense1  /

The mount commands are only needed if running on embedded pfSense to make the CF card writeable while we make the changes then make it read only again once we are done. The echo commands add the new limits to /etc/sysctl.conf so the settings persist and the sysctl commands make them apply now.

I haven't tested to see if the file descriptor issue effects the non embedded version of pfSense, feel free to let me (and others know) via the comments.

pfSense and Routed Subnets

I have a few clients running IPCop firewall appliance boxes, but for more complex setups (such as multiple WAN connections) I use pfSense. pfSense is a FreeBSD based firewall appliance. pfSense comes in 2 flavours, one of which is designed to run on low spec embedded hardware, such as that sold by Yawarra Information Appliances. I know that I could just use a bash scripts or Shorewallbut not all my clients are command line ninjas, and I have better things to do with my time.

Until recently in Austalia, "residential grade" ADSL connections used PPPoA/PPPoE (aka Layer 3), while "business grade" services were almost exclusively RFC 1483 bridged connections (aka Layer 2). Earlier this year, Telstra Wholesale have stopped offering Layer 2 connections, and are they are now in the process of migrating all resellers' customers to Layer 3 services. For customers with a single usable static IP address this is unlikely to mean any real change. For customers with larger IP allocations (say /29s or larger) they will switch from an IP block being available from the modem to PPPoE with additional IPs being available via a routed subnet.

After some discussion and playing, I found out there are 2 ways to get a routed subnet working with a pfSense box.

Option A - Firewall handles PPPoE and subnet used on DMZ

This is the solution I went for recently for a new connection setup for a client.

  • Configure ADSL modem/router to run in fully bridged mode
  • Configure pfSense's WAN interface to use PPPoE and fill in the appropriate information.
  • Configure the DMZ to use the routed subnet
  • Assign the first usable IP address to the DMZ interface (usually OPT1) on the pfSense box
  • Allocate the remaining IPs to the boxes in the DMZ
  • Setup your rules appropriately

Option B - Modem handles the PPPoE and subnet used on WAN

This method seems to make more sense for people moving from Layer 2 to Layer 3 connections. Please be aware that I haven't tested this, but I am told it should work.

  • Configure ADSL modem/router to work as router connecting via PPPoE
  • Configure the Ethernet port on the modem/router to use the first usable IP address from the routed subnet range
  • Configure pfSense's WAN interface to use a "static" connection and fill in the appropriate information, with the second usable IP address being assigned to the interface.
  • Assign any left over IP addresses as "Proxy ARP" addresses under Virtual IPs
  • Setup your rules and NATing appropriately

I hope someone find this useful.