Setting up your own email server with hMailServerSubmitted by Dominic Ryan on Wed, 2007-04-11 15:55. Categories:

There are a many email servers for Windows out there that range in price depending on

functionality and the software vendor. However, it is not often you find an email server for

Windows that is open source (free to use and modify via the GPL license), feature rich, stable

and very well supported. There are some notable free offerings such as MailEnable Standard

and Windows 2003 Server even comes with a basic POP3 email server, but it seems that

they are always just crippled enough to make you want something a bit more. This is where

hMailServer comes in as it offers just about everything you could want in an email server.

SMTP with advanced routing abilities, POP3, IMAP4, distribution lists, powerful anti-spam

tools, real time anti virus protection, PHP/COM based web administration, log file analysis,

configurable server messages and rules, advanced performance options. The list goes on,

and all this in a RFC compliant package with an easy to use GUI interface.

In this guide we'll cover the basics of what you need to do to configure your email

environment properly using hMailServer 4.3.1 as your email server. This includes;

← Configuring you DNS records

← A (Address) record

← PTR (Pointer Record) record

← MX (Mail eXchange) record

← Considerations for hosting from home

← Installing hMailServer

← Select installation path

← Select installation type

← Select built-in or external database

← Select program group for start menu access

← Configure hMailServer

← Create domain

← Create postmaster account

← Create abuse alias for postmaster account

← Set SMTP host name

← Configure RFC compliance

← Configure SMTP relay options to prevent open relay

← Test your email environment

← Test against dnsreport

← Test for open relay

1

Configuring your DNS records

Before we install your mail server it is a good idea to ensure you DNS records are correctly

setup for email. DNS is vital to the operation of the Internet, and in a nutshell what it does is

create a link between a human friendly domain name (e.g. example.com) and an IP

addresses. When a user enters a web address into their browser, that domain is then

resolved to an IP address which is then used to communicate. When hosting your own email

server there are three very important types of DNS records to consider, and these are A, PTR

and MX records.

An A record, or Address record is one of the most common types of DNS records you'll come

across. Its purpose is to simply create a link between an IP address and a domain name. For

example for the domain name of example.com you would have an A record that contained the

IP address of the server that was hosting example.com. With A records the domain must be

unique, but the IP address does not. This means you can have as many domain names

pointing to the same IP address as you want, and is used extensively these days for both

Email and Web hosting. In regards to your Email environment, it is best to create a separate

A record that identifies your email server. This can be something like mail.example.com or

similar. You'll need to create an A record for each server you plan on using as a mail server.

A PTR, or Pointer Record is unique in DNS in that you can only have one PTR address per IP

address. This is because the function of a PTR record is to resolve a human friendly name

from an IP address, instead of the other way around. Because you are resolving a domain

name from an IP address, there can only be one authoritative record. If possible it is best to

set your PTR record of the IP your email will be hosted from to be the same as your A record

(e.g. mail.example.com), and you may need to contact your hosting company to do this for

you.

Finally we have the MX, or Mail eXchange record which is used by other mail servers to direct

email to the right place. Unlike an A or PTR record an MX record is not associated with an IP

address in anyway, but instead contains the human friendly name of the A record you wish to

use for your mail server. This might sound a little redundant as all it is doing is pointing from

one record to another, but MX records also have one other important function. This is to

establish the pecking order of your email servers by using a preference field in which you can

enter a numerical value (the lower the number, the more important that server is) to define in

what order other email servers should contact your email servers. If the email server with the

highest preference (lowest number in preference field) is not contactable, then incoming email

servers will simply use the server identified by the MX record with the next highest

preference.

2

Considerations for hosting from home

Before we go on it is important to note that if you are wanting to host your own email server

from home over a standard ADSL or Cable connection, then there are a few things that may

make your environment a little more complex. These are;

← You'll most likely have a dynamically allocated IP address, meaning that the A record

for your mail server will need to be updated everytime your IP address changes.

← You most likely will not be able to set a PTR record for your IP address, especially if

you are using a dynamically allocated IP address.

← A lot of email servers block email originating from dynamic IP addresses.

← A lot of ISP's block the standard TCP/IP port numbers used by email servers on

home connections.

← You ISP may well have prohibited hosting your own email (or web) server over your

home broadband connection in their terms of service or acceptable usage policy.

With exception to the last few points, these issues can usually be pretty easily be overcome.

To update your mail server A record every time your IP changes you'll need to use a Dynamic

DNS service. These services are all over the web, and will require you to install a Dynamic

DNS client on your server which updates your DNS records everytime your local IP address

changes. You'll probably won't be so lucky in regards to the PTR record as ISP's usually have

their own and will almost never change these for you. A lot of email servers will check for the

existence of a PTR record for an IP before accepting any email from it, so as long as you do

have a PTR you should be ok. If you are hosting your email server on a dynamic IP address

then you'll need to look at using an SMTP relay server (sometime called smart relay) which is

usually your ISP SMTP server. With this cause all outgoing email from your server is sent out

through your relay server which should be set up properly and will allow you to send email to

servers that don't allow incoming email from dynamic IP addresses.

Among the trickier problems to solve is that a lot of the time ISP's will block commonly

attacked ports on home broadband services to try and stop (or slow at least) the spread of

viruses and prevent the saturation of their network. Some ISP's will let you remove these

blocks if you ask, or via a web interface. If your ISP has specified in the TOS or AUP that it

does not allow the hosting of services over your connection, then your are bang out of luck.

My suggestion is to look elsewhere as there are plenty of ISP's that will give you unrestricted

use of your connection (as it should be).

3

Installing hMailServer

That is most of the heavy stuff out of the way. From here on in it is all installing and

configuring hMail, which is all done via a GUI interface meaning I can use screen shots to do

the talking and is hopefully easier to follow. Click any of the images to get a pop-up window

displaying a full sized image.

After downloading the latest stable hMail server installer package (currently 4.3.1) from the

hmailserver website, double click on it to initiate the install process as shown below in figure

1.

Figure 1

 

Select the installation path for hMail as shown below in figure 2.

4

Figure 2

 

Select a full or custom install of hmail (full recommended) as shown below in figure 3.

Figure 3

 

5

Select wether to use the MySQL server built into the hMail distribution, or use an external

database. If this is a dedicated email server then it is recommended to use the built in

database server as shown below in figure 4. However, if you already have MySQL installed

(or are planning to) or would like to use Microsoft SQL server then select the external

database server option.

Figure 4

 

Set the start menu program group for hMail as shown below in figure 5.

6

Figure 5

 

Confirm your settings as shown below in figure 6.

Figure 6

 

7

Click install and hMailServer will be installed as shown below in figure 7.

Figure 7

 

Once installation is complete, make sure the "Run hMailServer administrator" option is

checked as shown below in figure 8 and click finish.

Figure 8

8

Configuring hMailServer

With the installtion of hMailServer successfully completed, the next step is configuration. The

configuration steps below show you how to add a domain, add an account, create an alias,

setting the server host name, configure RFC settings and configure SMTP relay options to

prevent open relay. Start by clicking the Add domain button as shown below in figure 9.

Figure 9

 

Enter the domain name as shown below in figure 10, and then set the catch-all address. If a

mail is sent to an address on your domain that does not have a POP account or alias, then it

is redirected to the catch-all address. In this example we have set the catch-all address to

postmaster@example.com. From here you can also set the global maximum mailbox size as

well as the maximum message size for your domain. Once the domain is created you are also

able to access several other tabs to set global settings, but they are not covered in this guide.

9

Figure 10

 

With the domain setup, it is now time to create accounts. To be RFC compliant all domains

should accept email to a postmaster account, and as we have set postmaster to be the catch-

all we will now set up an account for it. Place the name of the account in the Account

address feild, and then enter a password as shown below in figure 11. You can also set

individual mailbox and message size, as well as many other options not covered in this guide

such as Active Directory intergration, auto-reply, forwarding, signature and fetching of email

from external accounts.

Figure 11

Sometimes it is not practical or desirable to setup an account for every email address you

want, and in this case it can be handy to uses an email alias that points to an existing

account. In this example we will create an alias of abuse@example.com that points to the

10

postmaster@example.com. Simply enter the alias you'd like to use (in this case abuse) in the

Redirect from feild and enter the account and domain ( in this case postmaster and

example.com repsectively) in the To feilds. Like the postmaster address, domains are also

required to accept email to the abuse email address to be RFC compliant.

Figure 12

Now that we have a domain, account and alias setup lets look at selecting what email

services we want to use. Using the navigation window in the left, select and expand the

Settings item and then select the Protocols option. Make sure you have at least the SMTP

and POP3 servers ticked, as otherwise you will not be able to send or recieve mail. You may

not wish to use the IMAP server, but you will need it if you wish to provide webmail

functionality to your users.

11

Figure 13

Next step is to set the server host name. In the navigation window on the left, expand the

Protocols item and select SMTP. This can be very important as some email servers will not

accept email or mark it as spam if the host name does not match the hostname specified in

the MX record we set earlier. In the Host Name field enter the full host name you specified in

your MX records. In this example we'll use mail.example.com. Note: If you are looking to

host your own email server over your home broadband connection, then you will want

to enter the name of your ISP's SMTP server in the Relayer field. If your ISP requires

authentication, then you'll also need to provide those details in the fields below.

Figure 14

With the host name set we will now set some extra RFC compliance settings. From the page

you are on, simply click on the RFC Compliance tabe at the top. It is important that your

email server be RFC compliant as otherwise it is likely that many domains will mark your

email as spam, and that is if they accept it at all. One of the RFC requirements for email

servers is that they accpet a null sender address. You can enable this ticking the Allow

empty sender address option. It is also a good idea to enable the Allow incorrectly

formatted line endings option as several popular email server packages out their vary

slightly in the way they terminate email messages, and without this option set you email

server may not be able to recieve emails from them

12

Figure 15

The final step in the basic configuration of your email server is to enure it is not an open relay.

An open relay is when a server enables mails to be sent through it to other domains on behalf

of domains that do not exist on the local server. Being an open relay is a very quick way in

which to get yourself blacklisted, and it can be near impossible to get off these blacklists once

you're on it. Luckily hMailServer makes in very easy to prevent this, and in fact by default you

should not have to change a thing. Just to be sure though it is always best to check the

settings. In the navigation window to the left, select the select and expand the Advanced

menu item from under the Settings tree. From here select the IP Ranges option and then

select the Internet option. All you have to do here is ensure that the Local to local, Local to

external, and External to local options are ticked from under the Allow deliveries from

section. Finally also ensure that the To remote accounts option is ticked from under the

Require authentication for deliveries section. All these options are shown below in figure

16.

Figure 16

13

Testing the configuration

Almost done now, the only thing left to do is test your email environment to make sure

everything is configured correctly and ensure it is not an open relay. First stop is to plug your

domain in at DNSReport.com which will give you a good overview of how well your email

system is setup. If DNSReport finds any issues of concern it will notify you and offer advice on

what needs to be fixed.

There are several tests available for free on the internet for testing your email server for open

relay, and the ones I'd suggest using are the tests at abuse.net and aupads.org. Once you

have passed these tests then you should be all clear to go ahead and start creating other

account and finally sending and recieving email from your very own email server.

14