How to build a Mail Server on an iSCSI SAN with HA?

Low cost SAN with cluster to achieve Failover

I have recently been window shopping for some networking stuff on Amazon. Amazingly, 10GbE NICs and switches of brands and Cat7 cables are getting cheaper than before. In my head, I started to wonder if I could build a mail server on an iSCSI SAN and cluster with two-step HA mechanism? Since my concept is not totally automated, perhaps I should call it MA (medium availability) rather than HA (high availability)? LOL…

Name Brand Solution

For the past two decades or so, mail server solution on cluster and HA have definitely been available from big names, such as Microsoft Exchange Server, IBM Lotus, Oracle Veritas Cluster Server/Messaging Server and etc. But, the cost of implemnting such solutions is rocket-high, like 10 grand, 20 grand or more. They all come with top specs, server license, cluster license, cluster node license, mail server connection license, directory service connection license, tech support tickets, man labor and blah blah.

Amazon Solution

In 2019, price tags on SSD and 10GbE parts are lowering and they are getting more accessible than ever. DIY iSCSI SAN and HA cluster implemntation are not only for enterprise anymore. So, I think it is totally doable to equip mail server with more advanced but less expensive NAS gadgets as well as lower cost PCs for cluster purpose.

Theory

Why doable? According to theoretical value of transmitting speed of 10GbE, it is roughly 1250MB/s, which is 1.25GB/s. The theoretical value of SATA-3 is 6Gb/s and that turns out to be 750MB/s.

In the real world, there will definitely be bandwidth loss in network transmission due to many factors. Thus, let’s cut 1.25GB/s in half and it is 625MB/s. For SATA-3, let it optimally remain 70% of theoretical value and make it 525MB/s. The 625MB/s is gonna cover 525MB/s during data transmission no matter what and it is unprecedented in the past.

Next, we will discuss about software and hardware components that compose the iSCSI SAN & HA Cluster Mail Server infrastructure.

Network Speed

In the past, 1GbE NIC used to be a great enhancement from 100MbE NIC. At that time, perhaps only data center could affor 10GbE equipments. Nowadays, these gadgets have become so affordable by regular users. To achieve 10GbE transmission speed, it requires SFP+ (copper or fiber) or RJ-45 (Cat6 or Cat7 cable) to run between Cluster Server and NAS.

SSD storage

Just about one year ago, a 1TB SSD would cost over USD$400. Recently, 1TB SATA-3 SSD is running about USD$150 or even less. For my personal preference, I use either Micron or Intel:

Micron MX500 2TB: USD$289 / 1TB: USD$134
Intel 545S 1TB: USD$222

Synology NAS

Due gradual price drop on SSD drives over time, building a RAID-5 or -6 based on 4 Micron 2TB or 1TB SSD is a great news for IT staff because users will less likely complain about poor disk performance. When it comes to Synology NAS, I think DS1517+ is not a bad choice. Also, to achieve my iSCSI NAS plus HA Cluster Mail Server goal, its hot-swappable HDD capability and expandability to 10GbE E10G18-T2 (dual-port high-speed 10GBASET add-in card) are must-have. By the way, upgrade memory module to 16GB for the best performance.

So far, issues of mail data backup and contingency plan are resolved on Synology unit. For example, on DS1517+, we can store mail data on a RAID-5 on 4 SSDs. Then, install a 4TB conventional HDD on the 5th drive bay for storing ancient mail archive as well as virtua disk images of Windows and Windows Backup data. Or, another option is use the 5th drive by for SSD cache and it is not a bad idea either.

Cluster PC

Hardware: Buy two computers with specs of Intel i7, 16GB RAM, and one 120GB SSD. Equip each PC with 10GbE NIC for connections with NAS. The built-in 1GbE NIC is used to connect to switch or router. Then, do we still need additional NIC for cluster heartbeat? We will discuss about it later in this article.
Software: Install Windows 10 and EVO Mail Server on each of PC described above. Then, do we need to deploy VMWare Workstation to virtualize the mail services? Also, we will discuss about it later in this article.

Cluster Heartbeat Myth

Generally, a separate NIC will be installed on each member PC of a cluster group for heartbeat purpose. Windows Server 2008, 2012 and 2016 and Linux Server have this built-in feature of hearbeat. But, there is no such feature on Windows 10. I also googled it but found no answer. If you happen to know any tool software available for Windows 10, please let me know and thanks in advance.

The purpose of hearbeat feature works along with quorum disk to automatically determine whether the backup server should take over the primary server or not. I think such automated decision makings should be used on mission-critical servers, such as web servers or database servers

For mail server, by looking at it from another pint of view, I do not think mail server operation is not that mission-critical. With a bit higher fault tolerance, IT staff can take a a few minutes to diagnose what went wrong and then judge if backup server should take over the primary server. Factors on decision-making:

Resolving issues on primary server and continue?
Immediate action of taking over primary server?

In general, issues are likely to be one of the following occasion:

Hardware: mainboard, memory module, harddisk, power, NIC, network cable – If parts replacement is not readily available, then switch to backup server asap.
Software: Bugs on OS or mail server software – If opearting system is breaking down and cannot boot up, then switch to backup server asap. If a bug can be fixed within 10 minutes or so, such as a version update or parameter adjustment, then there is no need to switch.

There may be some delay on restoring mail service if IT staff diagoses issues before switching. However, it is always not a bad idea if IT staffs keep the whole situation under control.

Necessary to deploy VMWare?

Whether or not deploying VMWare is a debatable issue. I think if there is sufficient fund, then it is always better to prepare additional contingency plan other than a cluster backup server by installing VMWare Workstation on the primary server.

This is how I plan:

On the primary server, install a virtual Windows 10 under VMWare Workstation. Let’s call it VM A.
Once VM A runs mail service properly, shut it down.
Install another 120GB SSD on primary server.
Copy VM A image file to the new SSD and make it VM B.

Let’s assume that the primary server hardware is working. Under the circumstance that VM A image is broken, we can immediately start VM B to continue mail services.

Backup server taking over primary server

In the case of EVO Mail Server, since all data of mail, certificate and configuration and etc are all located on NAS, the role of cluster PCs is solely for running mail server services, accessing NAS data, and communicating with LAN and WAN. Each IT staff has his or her own preference on switching from primary server to backup server. I list some of the ways below for you to make decision on which one fits your need:

If servers are inside LAN:
- Log in router and change the LAN IP under Port Forwarding page (some router calls it Virtual Server) to backup server’s internal IP
- If you always bring primary server offline first before throwing backup server online, then we can disconnect backup server from the network by turning off its NIC during standby mode . When the switch-over is about to take place, we turn backup server’s NIC back on. In this scenario, backup server’s LAN IP can be configured as the same as primary server’s. The downside of it is that we cannot perform the swtich-over remotely and have to do it by actually logging in the backup server.
- For the above method, there is a workaround. While backup server is in standby mode, its NIC is enabled and assigned with an distinctive internal IP other than the primary server’s IP to avoid conflict. Before switching-over, disconnect primary server from the LAN first. Then, log in the backup server and change its LAN IP to primary server’s IP.
- If the primary server is equipped with VMWare and if server’s hardware are working normally but VM A image is corrupted, run VM B image to take over mail services. (Note: VM A and VM B have the same public IP)
If servers are in DMZ or directly connect to Internet:
- Log in DNS hosting service or your own DNS server first. Then, modify IP of mail server hostname’s (literally MX record) A record to backup server’s public IP. Such DNS record modification will usually take mintues to a few hours to kick in.
- If the primary server is equipped with VMWare and if server’s hardware are working normally but VM A image is corrupted, run VM B image to take over mail services. (Note: VM A and VM B have the same public IP)

Note:

Before switching-over, if the primary server is still accessible, simply log in the server and de-activate EVO Mail Server license key first.
Before switching-over, if the primary server already inaccessible, use other email account to contact EVO staff and let them manually de-activate license key. Then, activate the license key again on backup server.
To make switch-over quicker, purchase two EVO Mail Server licenses, one for primary and one for backup, and activate them separately. Once backup server is active, copy its license.dat file to the license key folder location on NAS and overwrite the existing one. Then, start mail service.
If VM images solution is deployed, there is no need to de-activate and re-activate since both VM A and VM B are actually identical. However, make sure not to boot up both images at the same time. Otherwise, both Windows license and mail server license will conflict each other.
Here is a shortcut and it works on both network environments mentioned above. Install identicaal hot-swappable drive bays on both servers. If any hardware parts fails but the 120GB OS SSD. Then, simply take out drawer containing the OS SSD. Stuff it into backup server’s drive bay and boot up immediately. However, it is recommended to prepare a spare ready-to-boot 120GB OS SSD aside.

Build NAS, iSCSI SAN and Cluster PC together

Let’s assume EVO Mail Server is operating normally. For installing, configuring and testing EVO Mail Server, please refer to my previous article.

#1. On Synology NAS, create iSCSI storage:

If NAS storage volume has not been created, follow instruction here. If already created, go to next step.

Now, build TARGET and build LUN. If they have been configured, then skip this part or make slight modification if necessary.

Open iSCSI Manager.

Let’s start from scratch.

Click on Target to the left and click on Create.

Enter name and IQN as well as making decision to enable CHAP or not.

The next step is mapping LUN. Here, we will click on Create a new iSCSI LUN.

Define capacity of iSCSI LUN.

Confirm settings and click on Apply to save settings.

Click on Target to the left again to double-check settings.

Click on LUN to the left again to confirm if target mapping is connected.

Click on Settings to the left to increase I/O Queue Depth value. Setting a greater value will improve 10GbE network connectivity and SSD performance.

Finally, click on Overview to confirm whether iSCSI has already been properly configured.

#2. Enable iSCSI initiator in Windows 10.

There are a few articles that you can read through on what can be done for iSCSI optimzation on Windows end.

Next, search iSCSI initiator and open it in Windows 10.

In iSCSI initiator window, under Targets tab, enter NAS IP and click on Quick Connect… button to initiate a quick search and connection.

In Quick Connect window, available target is listed and the state is connected.

Upon closing Quick Connect window, Discovered targets will list the connected NAS as well.

Search Computer Management and open it in Windows 10.

Click on Storage/Disk Management and the newly created disk is already listed.

In the screenshot above, mouse right-click on Disk 2 area and click on Convert to GPT Disk. Why GPT? GPT enables creating partition beyond 2.2TB.

Next, mouse right-click on area of newly created partition to continue creating volume and formatting it.

Check the formatted disk.

#3. Move mail server related data to iSCSI disk.

Open EVO Mail Server GUI. Click on Status page and scroll down to Server setting status.

In the screenshot above, click on arrow loop symbol to the right of Mail Data Folder. There will open a window browsing folder tree.

Once a folder is selected, click on OK. There will be a Reminder popping up and inform actions being taking place.

Once data migration is complete, another Reminder will pop up and inform actions taken place.

At the moment, Mail Data Folder is displaying new folder location.

#4. Finally, perform testing routines described in the setting up mail server article I mentioned previously.

Price breakdown

Nowadays, running a company with 100 to 200 employees is never an easy task. IT infrastructure investment ought to be carefully assessed. Here is a rough price quote for what it costs to build this solution:

Intel i7, 16GB RAM, 120GB SSD, 10GbE NIC (options are fiber and RJ-45), 450W power supply, Windows 10 computer x 2, USD$2000
Synology NAS DS1517+ x 1, USD$1000
Synology DDR3 Memory Module, USD$300
Synology E10G18-T2 or E10G17-F2 NIC x 2, USD$270 or USD$257
Micron MX500 2TB SSD x 4, USD$1200
WD 4TB HD (Mail Archive HD) External (if 5th drive bay is used for SSD cache) or Internal (if using 5th drive bay) x 1, USD$99 or USD$169
VMWare Workstation 15 x 1, USD$249 (optional if VMWare VM A and VM B solution is deployed)

All prices above are just random search from Amazon. The total price may be cheaper than USD$15000 if looking around more. But, the bottomline is that 10GbE NIC and TB-grade SSDs must stay. Here are comments on particularly these two parts:

NIC: If possible, use fiber NIC. You will be rewarded with much better performance.
NAS: Other than DS1517+, DS1817+ (8GB RAM version, USD$1400) is a good choice. It has two 10GbE ports built-in. Yet, two drive bays can be used for building SSD cache while there are still six drive bays for constructing RAID-5 or -6. Any idea on sizes for SSD cache drives? Consult your SSD dealer or check out this article. Though DS1817+ is USD$200 more expensive than DS1517+, RAM modules and 10Gbe NIC are taken care of. The overall expenses on NAS is USD$200 less at least. Nevertheless, hardware spec on NAS varies from one to another. It is up to you to decide.
NAS SSD: Why I strongly recommend on getting 2TB SSDs? Based on technical aspect, the bigger capacity the faster speed it is. Hence, if I were you, I would get 2TB SSDs once for all and it will definitely last longer.

Conclusion

It is a long article and it took me a long time to wrap it up. I just happened to write this article because I thought that we can benefit from huge price drop on both 10GbE NIC and TB-grade SSD. There are definitely other ways of implementation giving even faster performance, such as stuffing several PCI-e and NVME SSDs directly into mail server machine itself.

But, my main focus this time is providing a low-budget solution that offer fault tolerance for mail server. For exqmple, NAS running on RAID-5 or -6 provide fault-tolerance for storage while cluster deployment provides fault-tolerance for hardware failure.

At last, this article is for your reference when you happen to build a fault-tolerance mail server. However, each company has different IT environment. Perhaps you can survey on all left-over IT assets that are sitting around first. Then, purchase what is missing. Eventually, you will save even more for yourself or for your boss. Hope your server keep running and lasting. Yet, you can have sweet dreams every night with no emergency call from IT colleagues.