May 2004 Archives

!!!WARNING!!!
This HOWTO - which should be renamed the "qmail quick, uninformed install guide" is horribly outdated. Please see lifewithqmail.org for better and nore up-to-date instructions or help!
SF 2005-11-30

(with support for vqadmin, qmailadmin and sqwebmail)

Steven N. Fettig
See Creative Commons Copyright for copyright details.
Rev 1: 14 May 2004

This howto is very similar to the one I started translating a few years back from Oliver Lehman and is intended for people running FreeBSD 4.9 and up (it may work on lower versions, but I have not tested it and make no implied guarantee that it will work either way). I have found that a number of items in Oliver's howto didn't fit my needs, so I altered the HOWTO accordingly. Firstly, I don't totally understand the ssl certs, so I have not taken the time to make sure they are compiled/created correctly. The qmail and patchset I install includes the tls version of qmail and the courier-imap version I install will handle connections over ssl. I have installed and reinstalled various versions of qmail, vpopmail and courier-imap on the test server and have hosed certain portions of the installation. I have things working well, however, for new domains and intend to do a rebuild with the ssl support.

Assumptions:
- You have a fair grasp of FreeBSD and how to move around the filesystem.
- You can install services/programs via the ports.
- You understand that incorrectly configured email systems can be used as spam gateways - if you don't understand how to test and make sure you have NOT set up an open relay, STOP. Do us all a favor and learn/test/learn before placing a server on the net that is open to spammers.
- I don't know what I am doing half of the time.
- You may need to modify parts of these instructions to get this service combination to work on your machine.
- The smtp-auth patch will only work with vpopmail 5.4 and above.
- I use /usr/ports/distfiles/qmail/ as my SRC directory for compiling and installing software not installed via the ports.

Chapter 1 - Installation of Services

1.1 qmail

Install netqmail-1.05-tls with smtpauth patch (installation instructions are right in the text of the beginning of the patch):

Set sendmail_enable="YES" to sendmail_sendmail="NONE" in /etc/rc.conf

1.2 ucspi-tcp

ucspi-tcp is used so that use of inetd can be avoided in setting up tcp port connections. There are plenty of resources showing why people don't like inetd and I suggest you look for them on google.com.
There is a diff patch applied to rblsmtpd.c from Alan Curry so that "rblsmtpd works with A records." Same as Oliver's instructions:

1.3 daemontools

Daemontools is used to start and log the services we are going to set up. In our case, daemontools only watches qmail, pop3 services and the smtp services. Courier-imap is started via sh scripts. I, personally, like daemontools, but people have made good arguments for and against its use. DJB is a person that people seem to love to hate or love to love, so some of the comments pro or contra are juvenile. So, decide for yourself whether you want to employ daemontools for other items.

1.4 vpopmail

vpopmail administers virtual domains. This allows you to set up multiple domains on one host and has many tools to go along with it (also from vpopmail's makers, inter7) that allow you to add and remove users/domains/etc. with very little effort.

1.5 courier-imap

courier-imap is the service used to allow for imap access to email accounts. It is also an inter7 invention.

If you want to make the ssl certificates you need to follow Oliver's instructions:

Like I said in the beginning - I don't understand enough about ssl to know whether this is the *right thing* or whether it will give you a false sense of security if connecting to imap and pop3 via ssl.

1.6 ezmlm-idx

If you want to enable your server for mailing lists, ezmlm-idx makes this very, very easy.

1.7 qmail-conf

From Tetsu Ushijima's website:

We will end up modifying the vanilla smtp run script so that we can enable smtp-auth. BUT, these -conf scripts have made the creation of run scripts for qmail extremely easy.

Chapter 2 - Configuration of start scripts, vpopmail, smtp-auth and courier-imap

2.1 vpopmail crontab script

vpopmail is automatically sets up a selective relay when a user authenticates via pop3. We need to make sure that the tcp.smtp under /usr/local/vpopmail/etc is "cleaned" every 40 minutes.
Add the following line to your favorite root, vpopmail OR system crontab:

if adding to the system crontab:

2.2 Make the courier-imap startup scripts readable

The courier-imap port sets up install scripts in /usr/local/etc/rc.d that aren't yet readable at boot time because they are appended by the name .sample. We are simply going to copy the default the same name while removing the .sample ending.

You should take a quick look at the authdaemonrc file and take note of the configuration changes you can make.

To increase the number of concurrent sessions per IP, change the default value of imapd under /usr/local/etc/courier-imap

Look through the file for MAXPERIP and change to a value you think will be reasonable for your needs. I need to be able to access 10 plus accounts at any given time, so I set mine to a high number of 20. You may not need to change this value, but it might become important if you find your clients timing out all the time.

2.3 Create the qmail control files and start scripts

We will start by creating the control files and dot postmaster files:

We also need to make sure that the rcpthosts file is created so that the smtp server does not act as an open relay:

The following commands are almost identical to Oliver's, but I do not want to go though the hack to set up the selective relaying through vpopmail because I want to use auth-smtp. Also, we are going to edit the qmail-smtp run file afterwards. I still go through this process because I am not a good script writer and I like to see what someone else thinks is going to be a good run script.

Now, go into /var/qmail/service/smtpd and we will change the run file:

Copy the following into run:

and replace VCHKPWUID and VCHKPWGID with the appropriate UID and GID for vchkpw. Also replace mail.yourdomain.tld with your domain or fdqn of the server.

Now, create symlinks to the /var/service/ directory for qmail, pop3d and smtpd

There are a number of ways you can proceed. Either start each of the start scripts you have installed under /usr/local/etc/rc.d manually or reboot the system. The next chapter covers the basics of adding and removing domains/users and mailing lists.

3. Administration of vpopmail and ezmlm-idx

3.1 Adding/Removing Domains/Users

vpopmail offers a very easy interface with which you can add and remove both domains and users on the fly. The commands to do this are found under /usr/local/vpopmail/bin. If you run the commands directly with no options, the command will list the available switches and options that must be used with that command. Inter7's documentation is also a good source for information.

Add a domain:

Add a user to that domain:

Create an alias domain:

Change a user's password:

Delete a user:

Delete domain:

There are also ways to specify directories other than the default (/usr/local/vpopmail/domains) to store the virtual domains and user files. I have found, however, that this isn't necessarily a better way to manage security because the domain has to still have vpopmail:vchkpw read/write priviledges, so until I can find a way to give the user AND vpopmail access to the domain information/users and not others, I see no reason to use this option.

3.2 ezmlm-idx mailing lists

These set of commands are taken directly from Oliver's howto - I have yet to use mailing lists although I have always had ezmlm-idx installed.
To set up a moderated list (TEST) whose moderator can be reached at user1@domain1.tld, do the following:

3.3 Using SMTP-AUTH

The whole purpose for rewriting this howto was that I wanted to include smtp-auth into the mix. I have constantly had troubles with the courier-imap hack that allowed for selective relaying, so I decided a better way to open up the smtp server for relaying for my clients was to set up smtp-auth. In order to use smtp-auth (especially for those using imap folders and cannot open the smtp relay via the old hack), you simply need to set your email client to use smtp-auth with one of the usernames and passwords of one of the domains you have installed. Remember, a valid username is user@domain.tld and not simply user. I use Mozilla Thunderbird as my main email client these days and have not had any problems using smtp-auth. The only issue I have is that it slows the sending of emails because the authentication adds a delay to being able to send out the email. I would rather have the security, however - and perhaps it is an issue that I will find some solution to at a later time.

4. Web Administration

This is the section that I have been long waiting to work on. Now that I have a need for it - i.e. my own startup ISP that needs some remote-management interfaces - I have worked to get vqadmin, qmailadmin and sqwebmail working (all from inter7, of course).

I installed apache13-modssl from the ports and did do anything special to get it running other than making sure the ssl service started so that I could test running all of the above services via ssl.

4.1 Install apache13-modssl

To start apache13 with ssl activated:

I have had problems getting apache and ssl to start with the installed start script (/usr/local/etc/rc.d/apache.sh) and have yet to take the time to look further into the fact that neither apache nor the ssl portion of apache will start automatically once the server has been rebooted.

4.2 Install vqadmin

Make sure the destination directory for the cgi's is a place where you want them. Otherwise, modify your options or the Makefile accordingly. I set everything to install to /usr/local/www/cgi-bin-dist. While I know this may not be the proper way to do things, I wanted to have all of the web administration tools installed to the same directory so that I could first make sure they work. I will eventually modify this to fit my needs. Then make sure you follow the instructions on setting up .htaccess password protection to the directory where you will access vqadmin (it will not work without password restriction). You also need to modify the vqadmin.acl file to set up access permissions for your users. In my case, I only added myself to the passwd file (which .htaccess points to) and gave myself full permissions under vqadmin.acl.

4.3 Install qmailadmin

4.4 Install sqwebmail

I modified this installation a little so that it always runs over the ssl server. Although email is inherently insecure and open in nature, at least the web clients cannot be directly monitored.

Add the following to your root or system crontab: Again, if you are adding it to the system crontab:

You need to use the run scripts installed in /usr/local/etc/rc.d to test sqwebmail or reboot the machine.

There you have it... You should have a fully functioning email server.

Written by Steven N. Fettig
Rev. 1.1 - 8 May 2004
(See Creative Commons Copyright in right pane for copy restrictions)
Materials:
IBM Small Form Factor Workstation (Overkill)
Integrated Intel 10/100 NIC
Sangoma Dual CSU/DSU T1 PCI Card

Preface: As part of the Darien-WiFi project, I spent the past two weeks trying to install a router for our two T1's from Global Crossing. I had originally intended to use FreeBSD as the base system. What I found was that there was no (documentation showing a) way to do native load balancing across both T1's. The Sangoma Wanpipe drivers for Linux were able to do load balancing using the TEQL module (used in firewall QoS). Although an open supporter of what works - which can often be Linux - there was no way that I was going to use Linux on my router considering that I have had much, much more experience with FreeBSD and am familiar with locking it down for security uses. With quite a bit of Google digging, I found that OpenBSD'spf was not only capable of doing load balancing, but a whole array of items that I had spent much time learning how to do with ipfw on FreeBSD. After working until 10:30 pm on a Wednesday, I decided not to try OpenBSD, but instead opted for the pf port for FreeBSD. At about 1 am the next morning, I found out that not only would pf not load on a pre-5.0 machine (I know, I know - it says so in the docs, but I'm not enough of a kernel hacker to understand that it really wouldn't work), but I also couldn't get the Wanpipe drivers to load on the FreeBSD 5.2.1 system I just installed. By this time, I had been in contact with the Wanpipe BSD developer, Alex Feldman, and found that he hadn't thoroughly tested the driver install packages on 5.1 + and he asked me to wait for him to do his work. (He was extremely helpful - if you continue reading, you will find that I found more than one helpful soul through this whole process.) Instead of waiting, though - I had already spent enough wasted time on this project (which might have been completely avoided by using a Cisco Router with dual CSU/DSU's)* - I decided to dive into OpenBSD for the first time. I thought, with all of the structural similarities, it shouldn't be that difficult to acclimate myself to OpenBSD's peculiarities. So, here are basic instructions as to how I was able to get load balancing to work using the Sangoma card and pf on OpenBSD.

Instructions:
It happens that the University of Wisconsin - Madison runs an OpenBSD mirror. I downloaded the 3.4 boot iso from http://mirror.cs.wisc.edu/pub/mirrors/OpenBSD/3.4/i386/ on my OS X workstation:

Using DiskTools, I burned the bootable iso image to CD. There are a slew of tools out there to burn ISO's to CD. I assume that if you are reading how to configure a custom workstation/server to be your router, that you know how to burn a CD. I put the CD in my future router and booted the machine.
I was careful to follow the install instructions at:

Some people have complained that FreeBSD's install procedure was difficult... well, they obviously have not tried to install OpenBSD. This is not to say that I found it all too difficult - I'm neither afraid of a command line, nor am I worried if I hose the system - I can rebuild it, right? I learn best through trial-and-error anyway. This was one time, however, that I was able to get the base system installed without a hitch. Well, that is not entirely true - I tried to do a base install on a system with a Broadcom Gigabit NIC and could never get the NIC to activate during the install. Perhaps had I read the supported hardware information on the main site, I would have found that support for the NIC needs to be added via a custom kernel. I had another machine waiting idly by to get used - with an onboard Intel 10/100 NIC (supported by the generic kernel) - and went through the same procedure again.
Because I had not yet received the CD's I purchased (c'mon - support the open source projects you use!), I installed from the UW-Madison http server.
After choosing the install source, I selected the following filesets (in FreeBSD lingo, I suppose this is base packages):

There are a few more items that need to be configured after the appropriate distributions have been downloaded. Again, follow the install FAQ for assistance. I would take more time to outline how to install your OpenBSD system, but the FAQ provides an amazingly detailed explanation of the installation procedure. It would be wrong of me to provide another explanation when it is laid out so well in the OpenBSD documentation.
Since I had already physically installed the Sangoma card, I rebooted the system and started to install the drivers and configure the T1's (according to the settings that Global Crossing is using). I downloaded the Wanpipe drivers and software:

I took care to follow the installation instructions given to us by Sangoma and simply ran:

I had the install script do all the defaults - install drivers, add support to kernel, install startup scripts, etc. Everything went off without a hitch. When I had originally tried the installation on the pre 5.x FreeBSD system, the compilation of the drivers failed. I have yet to test the Wanpipe installation on OpenBSD 3.5, so I can't say whether it works or not. Due to the stability of OpenBSD, I imagine the packages will install without any problems as they did on my 3.4 system.
Again, following instructions that come with the Sangoma card, I configured the T1's.

This part requires that you have pertinent protocol and IP information from your provider.
Sidenote: I must say that Global Crossing is amazing. Their Circuit Turn-up staff was amazingly helpful. Through this whole ordeal, a gentleman named John Brox helped me out and was patient with my insistence on doing a custom install - I say patient because I initially was not able to even bring up the circuits with a box I had intended to use (a Eden 500 VIA mini-itx machine), as the ethernet interface caused the circuits to cycle up and down at weird intervals. I don't know if this was a problem with the motherboard itself or with the VIA ethernet interface. I have yet to do the same install on another VIA mini-itx based system, so I don't know. He was more than willing to help me make sure the circuit was up for good before turning it over to the monitoring group.
I haven't worked on routers in years and considering that, not only does Sangoma make the circuit configuration easy, but Global Crossing made the circuit turn-up a breeze.
On to the actual configuration of the firewall and load balancing. The Wanpipe drivers give you two interfaces to work with (once the T1's have been enabled). The last line in wanrouter.rc (in /etc/wanpipe) should read:

so that both lines are enabled when the machine is rebooted. If you have not rebooted the machine, you can start the wanpipes by (assuming you configured wanpipe1 and wanpipe2):

After starting the circuits, running

gives you:

at the end.
You also need to make sure that your machine is set up to act as a gateway. Straight from the FAQ: http://www.openbsd.org/faq/faq6.html

My internal ethernet interface (fxp0) was set as 24.192.154.1 - the beginning of my address block available for use. At first - just to try the load balancing - I modified the rules at http://openbsd.org/faq/pf/pools.html#outgoing for my interface:

so that the whole pf.conf looked like:

At this point, I was able to ping out of both interfaces, ping beyond the gateway, ping internally, etc. So, I set up a client behind the firewall and ran speed tests. Lo and behold, I was getting load balancing on the inbound (vis a vis Global Crossing's router configuration - it equally passes packets down both pipes), but it did not look like it was working on the outbound. It was about 12 am by this time and I was simply happy to see the OpenBSD machine working with the T1's and everything being pingable. Plus, John was gone by this time. I decided to call it quits and start again in the morning.
I woke up still puzzled as to why I couldn't get load balancing on the outbound route to work - theoretically, I had everything configured correctly! I called John and asked him if he could take a look at packet counts while I was running the bandwidth tests. After running a few tests, we found that the outbound were indeed being passed across both interfaces, but for some reason the bandwidth tests were not showing that. So, I decided to try to run more than one bandwidth test at once to see if I could fill up the pipes with more data. After getting my timing right (I kept starting one test well before the other), we were able to show that both pipes were indeed being load balanced, but for whatever reason, the speed tests would "sense" a limit at the 1 T1 barrier. Running multiple speed tests showed that just as much bandwidth was available down as it was up - i.e. just under 3 mbit/s. John was able to confirm that data was being sent down both pipes - one being favored by about 2% over the other. We chalked this up to "imperfect" round-robin'ing and called it a day. To this day, I still have not been able to figure out why the speed tests show outbound limited by one T1, but I don't really care as long as my available outbound is really the two T1's. I'm sure a ttcp test would be a better measure of throughput, but I haven't had the time to run such tests and am content that things are working as advertised.
After closing the firewall by enabling the default deny rules, I started working on allowing inbound connections. I struggled quite a bit with ssh. So much so that I finally emailed the OpenBSD misc list. I never did get a response to my question - I never really expected to seeing as I doubted many people were trying to do what I was and I further figured that they would pass my query off as being that of someone who failed to read the manual. They were wrong... I read the manual over and over but was never able to wrap my head around setting up the rules such that I could pass ssh through after default denying everything (I did confirm that ssh worked without the firewall...) So, I set out to test this through trial-and-error. Finally, after 5 hrs of testing, I came up with:

This was the culmination of so many combinations of pass etc. I feign to write them all down out of sheer embarrassment. How simple and elegant it ended up being...
There are a number of other ports I ended up opening. There are also a few clients behind the T1 who want their IP's completely open. I was able to accomplish all of this with the following ruleset:

I realize that these rules can be redundant and must be rewritten in a much more efficient manner. That is what I am in the middle of working on. I hope, though, that these will help guide you to set the firewall and routing rules for your needs.
Remaining Problems:
One other item that needs to be cleaned up is having the rules loaded after the T1 interfaces are activated at boot time. This process usually is finished 45-60 seconds after the machine is fully booted. This means that if there is a power failure or problem with the machine and it reboots, one has to physically re-set the default gateway and load the rules. I don't understand enough of OpenBSD to make this happen, but when I do, I'll tell you how it is done.

HTH,
Steve

* In between all of this, I had given up on using pf for load balancing because of what I had seen as problems with pf - I was mistaken at the time, but I was convinced the problem was with pf and not with what I was seeing. I unwrapped a brand new Cisco 1721 and two T1 interfaces from a box - my just in case box - and spent eight hours trying to get it working. Fortunately, I was never able to bring up the circuits with the Cisco. I say fortunately because I ended up back with the BSD box and got it working - what I wanted in the beginning.