steven n fettig's Jitterin' Thoughts

Written by Steven N. Fettig
Rev. 1.1 - 8 May 2004
(See Creative Commons Copyright in right pane for copy restrictions)
Materials:
IBM Small Form Factor Workstation (Overkill)
Integrated Intel 10/100 NIC
Sangoma Dual CSU/DSU T1 PCI Card

Preface: As part of the Darien-WiFi project, I spent the past two weeks trying to install a router for our two T1's from Global Crossing. I had originally intended to use FreeBSD as the base system. What I found was that there was no (documentation showing a) way to do native load balancing across both T1's. The Sangoma Wanpipe drivers for Linux were able to do load balancing using the TEQL module (used in firewall QoS). Although an open supporter of what works - which can often be Linux - there was no way that I was going to use Linux on my router considering that I have had much, much more experience with FreeBSD and am familiar with locking it down for security uses. With quite a bit of Google digging, I found that OpenBSD'spf was not only capable of doing load balancing, but a whole array of items that I had spent much time learning how to do with ipfw on FreeBSD. After working until 10:30 pm on a Wednesday, I decided not to try OpenBSD, but instead opted for the pf port for FreeBSD. At about 1 am the next morning, I found out that not only would pf not load on a pre-5.0 machine (I know, I know - it says so in the docs, but I'm not enough of a kernel hacker to understand that it really wouldn't work), but I also couldn't get the Wanpipe drivers to load on the FreeBSD 5.2.1 system I just installed. By this time, I had been in contact with the Wanpipe BSD developer, Alex Feldman, and found that he hadn't thoroughly tested the driver install packages on 5.1 + and he asked me to wait for him to do his work. (He was extremely helpful - if you continue reading, you will find that I found more than one helpful soul through this whole process.) Instead of waiting, though - I had already spent enough wasted time on this project (which might have been completely avoided by using a Cisco Router with dual CSU/DSU's)* - I decided to dive into OpenBSD for the first time. I thought, with all of the structural similarities, it shouldn't be that difficult to acclimate myself to OpenBSD's peculiarities. So, here are basic instructions as to how I was able to get load balancing to work using the Sangoma card and pf on OpenBSD.

Instructions:
It happens that the University of Wisconsin - Madison runs an OpenBSD mirror. I downloaded the 3.4 boot iso from http://mirror.cs.wisc.edu/pub/mirrors/OpenBSD/3.4/i386/ on my OS X workstation:

Using DiskTools, I burned the bootable iso image to CD. There are a slew of tools out there to burn ISO's to CD. I assume that if you are reading how to configure a custom workstation/server to be your router, that you know how to burn a CD. I put the CD in my future router and booted the machine.
I was careful to follow the install instructions at:

Some people have complained that FreeBSD's install procedure was difficult... well, they obviously have not tried to install OpenBSD. This is not to say that I found it all too difficult - I'm neither afraid of a command line, nor am I worried if I hose the system - I can rebuild it, right? I learn best through trial-and-error anyway. This was one time, however, that I was able to get the base system installed without a hitch. Well, that is not entirely true - I tried to do a base install on a system with a Broadcom Gigabit NIC and could never get the NIC to activate during the install. Perhaps had I read the supported hardware information on the main site, I would have found that support for the NIC needs to be added via a custom kernel. I had another machine waiting idly by to get used - with an onboard Intel 10/100 NIC (supported by the generic kernel) - and went through the same procedure again.
Because I had not yet received the CD's I purchased (c'mon - support the open source projects you use!), I installed from the UW-Madison http server.
After choosing the install source, I selected the following filesets (in FreeBSD lingo, I suppose this is base packages):

There are a few more items that need to be configured after the appropriate distributions have been downloaded. Again, follow the install FAQ for assistance. I would take more time to outline how to install your OpenBSD system, but the FAQ provides an amazingly detailed explanation of the installation procedure. It would be wrong of me to provide another explanation when it is laid out so well in the OpenBSD documentation.
Since I had already physically installed the Sangoma card, I rebooted the system and started to install the drivers and configure the T1's (according to the settings that Global Crossing is using). I downloaded the Wanpipe drivers and software:

I took care to follow the installation instructions given to us by Sangoma and simply ran:

I had the install script do all the defaults - install drivers, add support to kernel, install startup scripts, etc. Everything went off without a hitch. When I had originally tried the installation on the pre 5.x FreeBSD system, the compilation of the drivers failed. I have yet to test the Wanpipe installation on OpenBSD 3.5, so I can't say whether it works or not. Due to the stability of OpenBSD, I imagine the packages will install without any problems as they did on my 3.4 system.
Again, following instructions that come with the Sangoma card, I configured the T1's.

This part requires that you have pertinent protocol and IP information from your provider.
Sidenote: I must say that Global Crossing is amazing. Their Circuit Turn-up staff was amazingly helpful. Through this whole ordeal, a gentleman named John Brox helped me out and was patient with my insistence on doing a custom install - I say patient because I initially was not able to even bring up the circuits with a box I had intended to use (a Eden 500 VIA mini-itx machine), as the ethernet interface caused the circuits to cycle up and down at weird intervals. I don't know if this was a problem with the motherboard itself or with the VIA ethernet interface. I have yet to do the same install on another VIA mini-itx based system, so I don't know. He was more than willing to help me make sure the circuit was up for good before turning it over to the monitoring group.
I haven't worked on routers in years and considering that, not only does Sangoma make the circuit configuration easy, but Global Crossing made the circuit turn-up a breeze.
On to the actual configuration of the firewall and load balancing. The Wanpipe drivers give you two interfaces to work with (once the T1's have been enabled). The last line in wanrouter.rc (in /etc/wanpipe) should read:

so that both lines are enabled when the machine is rebooted. If you have not rebooted the machine, you can start the wanpipes by (assuming you configured wanpipe1 and wanpipe2):

After starting the circuits, running

gives you:

at the end.
You also need to make sure that your machine is set up to act as a gateway. Straight from the FAQ: http://www.openbsd.org/faq/faq6.html

My internal ethernet interface (fxp0) was set as 24.192.154.1 - the beginning of my address block available for use. At first - just to try the load balancing - I modified the rules at http://openbsd.org/faq/pf/pools.html#outgoing for my interface:

so that the whole pf.conf looked like:

At this point, I was able to ping out of both interfaces, ping beyond the gateway, ping internally, etc. So, I set up a client behind the firewall and ran speed tests. Lo and behold, I was getting load balancing on the inbound (vis a vis Global Crossing's router configuration - it equally passes packets down both pipes), but it did not look like it was working on the outbound. It was about 12 am by this time and I was simply happy to see the OpenBSD machine working with the T1's and everything being pingable. Plus, John was gone by this time. I decided to call it quits and start again in the morning.
I woke up still puzzled as to why I couldn't get load balancing on the outbound route to work - theoretically, I had everything configured correctly! I called John and asked him if he could take a look at packet counts while I was running the bandwidth tests. After running a few tests, we found that the outbound were indeed being passed across both interfaces, but for some reason the bandwidth tests were not showing that. So, I decided to try to run more than one bandwidth test at once to see if I could fill up the pipes with more data. After getting my timing right (I kept starting one test well before the other), we were able to show that both pipes were indeed being load balanced, but for whatever reason, the speed tests would "sense" a limit at the 1 T1 barrier. Running multiple speed tests showed that just as much bandwidth was available down as it was up - i.e. just under 3 mbit/s. John was able to confirm that data was being sent down both pipes - one being favored by about 2% over the other. We chalked this up to "imperfect" round-robin'ing and called it a day. To this day, I still have not been able to figure out why the speed tests show outbound limited by one T1, but I don't really care as long as my available outbound is really the two T1's. I'm sure a ttcp test would be a better measure of throughput, but I haven't had the time to run such tests and am content that things are working as advertised.
After closing the firewall by enabling the default deny rules, I started working on allowing inbound connections. I struggled quite a bit with ssh. So much so that I finally emailed the OpenBSD misc list. I never did get a response to my question - I never really expected to seeing as I doubted many people were trying to do what I was and I further figured that they would pass my query off as being that of someone who failed to read the manual. They were wrong... I read the manual over and over but was never able to wrap my head around setting up the rules such that I could pass ssh through after default denying everything (I did confirm that ssh worked without the firewall...) So, I set out to test this through trial-and-error. Finally, after 5 hrs of testing, I came up with:

This was the culmination of so many combinations of pass etc. I feign to write them all down out of sheer embarrassment. How simple and elegant it ended up being...
There are a number of other ports I ended up opening. There are also a few clients behind the T1 who want their IP's completely open. I was able to accomplish all of this with the following ruleset:

I realize that these rules can be redundant and must be rewritten in a much more efficient manner. That is what I am in the middle of working on. I hope, though, that these will help guide you to set the firewall and routing rules for your needs.
Remaining Problems:
One other item that needs to be cleaned up is having the rules loaded after the T1 interfaces are activated at boot time. This process usually is finished 45-60 seconds after the machine is fully booted. This means that if there is a power failure or problem with the machine and it reboots, one has to physically re-set the default gateway and load the rules. I don't understand enough of OpenBSD to make this happen, but when I do, I'll tell you how it is done.

HTH,
Steve

* In between all of this, I had given up on using pf for load balancing because of what I had seen as problems with pf - I was mistaken at the time, but I was convinced the problem was with pf and not with what I was seeing. I unwrapped a brand new Cisco 1721 and two T1 interfaces from a box - my just in case box - and spent eight hours trying to get it working. Fortunately, I was never able to bring up the circuits with the Cisco. I say fortunately because I ended up back with the BSD box and got it working - what I wanted in the beginning.