TeraServer 2

I need an upgraded server. I want something that can handle replication of a SaaS database with on-disc encryption. Keeping with my habit of building my own machines, it would require more space than my current TeraServer could provide, and frankly the disk access times are a little slow now on the old one (I suspect the RAID backplane I sourced wasn’t capable of delivering the SATA-2 speeds the drives could). It’s being used for far more applications than I had planned too.

5 years is a long time in hardware and I’m amazed to see how storage costs have dropped further. This time I could get two disks that would provide me with full drive RAID redundancy at a capacity of 3TB at SATA-6 speeds. That means I could go for the 1U rack mount case at mini-itx.com (C2-RACK-V3). This case also supports a PCI-E card – so I can add a video capture card for some security monitoring using ZoneMinder too.

TerraServer2_Ext

The drives are fast and surprisingly quiet too (Seagate 3.5″ 3TB Barracuda 7200RPM). To ramp up the system performance of this one even further I used a Crucial M4 SSD for the Linux system partition. That was a new experience! The thing boots in seconds.

With the spinning drives mounted using rubber washers to reduce vibration and noise, the three drives fitted in comfortably, with room to spare.

TerraServer2_Int_2

For the mainboard, I wanted something with a bit more processor power (it would need to handle Linux software RAID and video capture analysis for ZoneMinder). It also needed to fit the case and I didn’t want any cooling fan that would add to the height and noise. Eventually, I found an Asus C60-M1 at a very good price from an on-line store in Germany. It was a compromise – the processor is the same clock speed as the previous server board, but anything higher wouldn’t come fanless. I settled for the same clock speed (but twice the cores helps!) It fitted ok in height, but there was a bit too much of a gap between the board and the chassis back-plate. I don’t think any of the provided blanking plates would cover the entire back-plate either. That wasn’t important for me anyway.

4GB of 1333MHz DDR3 RAM would be plenty to handle the extra Linux apps I have been (and will be) running on the server. With that amount of RAM, I could even run the Linux tmpfs in a ramdisk to make it even faster.

TerraServer2_Int_1

I installed Centos 6, configured the drives, and I’m amazed at how fast this little mini-itx powered beast flies compared to my previous creation. 5 years is a long time in hardware.

SELinux Policies for Mono (.NET) Web Applications

I’m curious to see how production ready Mono on Linux is. What better way to find out than to take an existing .NET web app and run it on Mono? I tried – the web app wouldn’t even run – reporting an internal server error. After a little delving it turns out that SELinux was blocking Apache mod-mono from spawning the mono process. From /var/log/messages:

May 14 15:18:29 slave kernel: type=1400 audit(1374675509.274:34550): avc:  denied  { search } for  pid=5744 comm="mono" name="/" dev=dm-2 ino=2 scontext=unconfined_u:system_r:httpd_sys_script_t:s0 tcontext=system_u:object_r:file_t:s0 tclass=dir
May 14 15:18:29 slave kernel: type=1400 audit(1374675509.325:34551): avc:  denied  { search } for  pid=5744 comm="mono" name="/" dev=dm-2 ino=2 scontext=unconfined_u:system_r:httpd_sys_script_t:s0 tcontext=system_u:object_r:file_t:s0 tclass=dir
May 14 15:18:29 slave kernel: type=1400 audit(1374675509.325:34552): avc:  denied  { search } for  pid=5744 comm="mono" name="/" dev=dm-2 ino=2 scontext=unconfined_u:system_r:httpd_sys_script_t:s0 tcontext=system_u:object_r:file_t:s0 tclass=dir
May 14 15:18:29 slave kernel: type=1400 audit(1374675509.610:34553): avc:  denied  { search } for  pid=5744 comm="mono" name="/" dev=dm-2 ino=2 scontext=unconfined_u:system_r:httpd_sys_script_t:s0 tcontext=system_u:object_r:file_t:s0 tclass=dir
May 14 15:18:29 slave kernel: type=1400 audit(1374675509.610:34554): avc:  denied  { search } for  pid=5744 comm="mono" name="/" dev=dm-2 ino=2 scontext=unconfined_u:system_r:httpd_sys_script_t:s0 tcontext=system_u:object_r:file_t:s0 tclass=dir
May 14 15:18:29 slave kernel: type=1400 audit(1374675509.611:34555): avc:  denied  { search } for  pid=5744 comm="mono" name="/" dev=dm-2 ino=2 scontext=unconfined_u:system_r:httpd_sys_script_t:s0 tcontext=system_u:object_r:file_t:s0 tclass=dir
May 14 15:18:29 slave kernel: type=1400 audit(1374675509.657:34556): avc:  denied  { execmem } for  pid=5744 comm="mono" scontext=unconfined_u:system_r:httpd_sys_script_t:s0 tcontext=unconfined_u:system_r:httpd_sys_script_t:s0 tclass=process
May 14 15:18:29 slave kernel: type=1400 audit(1374675509.658:34557): avc:  denied  { execmem } for  pid=5744 comm="mono" scontext=unconfined_u:system_r:httpd_sys_script_t:s0 tcontext=unconfined_u:system_r:httpd_sys_script_t:s0 tclass=process
May 14 15:18:29 slave kernel: type=1400 audit(1374675509.863:34558): avc:  denied  { ptrace } for  pid=29877 comm="gdb" scontext=unconfined_u:system_r:httpd_sys_script_t:s0 tcontext=unconfined_u:system_r:httpd_sys_script_t:s0 tclass=process

I was later to find out that this was only the first thing SELinux would prevent it from doing!

The mono guys recommend that you switch SELinux off for mono apps, but that won’t really cut it for a production system. I wanted to create an SELinux policy that could be applied to allow the mono web app to work. SELinux is very fine-grained in the permissions that can be applied, so I can see why the mono guys won’t provide a standard mono SELinux policy to install. The policy depends very much on what the individual mono app needs to be able to do. If you tried to create an overall policy for mono apps it would have to allow so many permissions that you might as well switch SELinux off.

So how to derive a custom SELinux policy for a mono app?

To start with, try to run the app. Any SELinux enforcement messages are logged to the system log. The SELinux audit2allow command is helpful in deriving policy rules to resolve the enforcement errors.

cat /var/log/messages | audit2allow

The output will give you a list of rules to apply, e.g.:

[Bren@slave devel]$ sudo cat /var/log/messages | audit2allow

#============= httpd_sys_script_t ==============
allow httpd_sys_script_t file_t:dir search;
allow httpd_sys_script_t self:process execmem;
allow httpd_sys_script_t self:process ptrace;
allow httpd_sys_script_t tmpfs_t:file unlink;

To create a policy module start by creating a type enforcement (.te) file and enter the audit2allow discovered permissions into that file.

vi /usr/share/selinux/devel/mod_mono.te

After making any alteration to the .te file the selinux policy module needs to be re-compiled.

make /usr/share/selinux/devel/mod_mono.pp

Install the new policy module as follows (you’ll need root permissions to do these things).

sudo semodule -i /usr/share/selinux/devel/mod_mono.pp

When you do all that and run the app again, it should get further, but will only encounter new enforcement errors. Unfortunately I found no easy of discovering all the permissions for an app but to iteratively test it, run audit2allow, add new rules to the .te file, re-compile, and try the app again.

Some rules will be common to all mono web apps, and some as unique as the web app itself. It is a process that will be determined by the individual app’s requirements. For example, does it use pipes or TCP ports to connect to a database server? If new features are added to the app in later releases, it could require addition policy rule changes to work.

In the end, here is the full policy I needed to get my web app functionality to work as expected. (Note that the samba_share_t permissions are needed because my web app views are locate on a development samba share.)

policy_module(mod_mono,1.0.0)

gen_require(`
   type lib_t;
   type tmp_t;
   type file_t;
   type tmpfs_t;
   type mono_exec_t;
   type samba_share_t;
   type inotifyfs_t;
   type httpd_t;
   type httpd_sys_script_t;
   type httpd_sys_rw_content_t;
   type postgresql_port_t;
   type port_t;
   class process { execmem ptrace };
   class capability { sys_admin ipc_owner };
   class sock_file { write create unlink };
   class sem create;
   class dir { open search getattr read write add_name };
   class filesystem getattr;
   class file { open read getattr execute_no_trans };
   class tcp_socket { name_connect };
')

#============ httpd_sys_script_t =============
allow httpd_sys_script_t self:process execmem;
allow httpd_sys_script_t file_t:dir search;
allow httpd_sys_script_t inotifyfs_t:dir { search read };
allow httpd_sys_script_t samba_share_t:dir { open search getattr read };
allow httpd_sys_script_t samba_share_t:file { open read getattr };
allow httpd_sys_script_t tmpfs_t:dir { search read write open add_name remove_name };
allow httpd_sys_script_t tmpfs_t:filesystem getattr;
allow httpd_sys_script_t tmpfs_t:file { read write open create unlink } ;
allow httpd_sys_script_t postgresql_port_t:tcp_socket name_connect;
allow httpd_sys_script_t port_t:tcp_socket name_connect;

allow httpd_t lib_t:file execute_no_trans;
allow httpd_t mono_exec_t:file { read execute_no_trans };
allow httpd_t self:process ptrace;
allow httpd_t tmp_t:sock_file { write create };
allow httpd_t httpd_sys_rw_content_t:sock_file unlink;
allow httpd_t self:capability { sys_admin ipc_owner };
allow httpd_t file_t:dir search;
allow httpd_t samba_share_t:dir search;
allow httpd_t samba_share_t:file { read open getattr };

Configuring Linux Solid State Drive (SSD) Performance

SSDs are really fast (especially for reads) compared with spinning platter drives, but are relatively expensive and so generally have smaller capacity. This makes an SSD a good choice for the system partition of a Linux installation where all the frequently read system files are located – programs will boot up faster.

SSD drives also have a potentially shorter lifespan, especially if used in a write-delete-write fashion. At the time of writing, it’s estimated that system drives used this way will start to show degradation after about 1-2 years. To improve the performance and longevity of the SSD here are some of the best practices I’ve discovered for configuring Linux with an SSD.

  1. Check the SSD’s page size and erase block size (4K and 512K respectively in my case)
  2. Install the latest firmware available for the SSD
  3. Create the partition so as to align its start to the SSD erase block size (or multiple of it). This will ensure your SSD won’t have to do extra write operations.
    fdisk -S<sectors> -H<heads> -cu /dev/sda
    

    You do this by specifying a number of heads and sectors. Ultimately, you need to look this up because it depends on the erase block size of your drive. There’s some good advice here.

  4. Only format the partition with the ext4 filesystem – it has TRIM support
  5. Align the filesystem with the block size on the SSD when formatting the partition. If you aligned the partition (as above) then this further ensures minimum writes by choosing an ext4 block size that matches the page size on the drive.
    mkfs.ext4 -b <block size in bytes> /dev/sda2
    

    For me, I didn’t need to specify the block size because the default ext4 block size is 4K which already matches my drive’s 4K page size.

  6. Once Linux is install, edit fstab to enable TRIM on the SSD filesystem. This is done by adding the discard flag to the partition.
  7. A further tweak that can be done in fstab to minimize unnecessary writes on the SSD is to switch off updates to the file access times – remove atime and relatime flags on ssd partitions and replace with noatime and nodiratime
    UUID=XXXX    /   ext4  defaults,noatime,nodiratime,discard  1 1
    
  8. If you have a second, platter drive installed, move the swap partition to that device.
  9. You can minimize swap disk accesses by disabling swap which will keep swap space in memory (provided you have enough memory and it is not full). In /etc/rc.local file add the line:
    echo 0 > /proc/sys/vm/swappiness
    
  10. Use the deadline scheduler for writes. In /etc/rc.local file add the line:
    echo deadline > /sys/block/sda/queue/scheduler
    
  11. Move small temporary file directories away from the SSD storage and into ramdisk, by adding the following to fstab. (Note that this would could mean data loss if server crashes, but it’s an acceptable tradeoff for me).
    tmpfs   /tmp       tmpfs   defaults,noatime,mode=1777   0  0
    tmpfs   /var/spool tmpfs   defaults,noatime,mode=1777   0  0
    tmpfs   /var/tmp   tmpfs   defaults,noatime,mode=1777   0  0
    
  12. On a per application basis, repoint any temporary file write directories to platter partitions. The best example of such an application is a web browser, which maintains writes a large cache of temporary files. (In the end, to handle all these temporary application file scenarios, I decided to redirect the entire /var directory onto a second platter drive – this also covered Linux system /var/log files.)

TeraServer Project – Part 2

Choosing RAID configuration means deciding on priorities – which is more important: available disk space or failure recovery? In a production system I would choose differently, but for a home server, having bought four disks, I wanted to maximize the storage potential by using the facilities of RAID to combine the disks into one volume (it is a TeraServer after all!). I also wanted to get some of the benefit of RAID failure checking and recovery – it would be storing all my important files (yes, I have backups too). In the end I decided to allow for 1 drive failure to maximize volume, and settled on a RAID-3 configuration. In the end, after building the RAID array, mounting it as a Linux volume, and allowing space for system partitions and Linux files, I was left with about 825GB for storage.

So with the arrival of the SCSI disks and Adaptec SCSI RAID card, I could begin.

Getting the RAID caddy into the case was fairly easy. I just cut out some portions of the metal front plate, and the caddy slotted into the chassis four-bay drive frame perfectly. Good choice.

One thing I was keen to try on this project was to add a cool front status display. There was just enough room on the case below the drive bay to fix a 4-row backlight LCD display. Next to it, a recessed power button. Finally, I added a new plastic front-plate, cut to fit the custom parts and sprayed the whole thing black and the case externals were done.

TerraServer_4_CaseModsCompleted

The internal space was – tight! There was plenty of room at the back, behind the drive bay above the mainboard, but the mainboard runs some of the way underneath the drive bay. This left some contact between the SCSI card and the RAID caddy. Some precautionary insulation of the bottom of the RAID caddy with electrical tape solved that problem. I also added some sound baffling and insulation around the top of the RAID caddy.

Another issue was the LPT connector on the board. This was how the LCD display module needed to be connected to the mainboard. Clearance was tight for the LPT connector, but the bigger issue was making the internal! I bought a standard shielded parallel cable, cut it, and armed with some veroboard, a soldering iron, LPT pinouts and the HP LCD module tech sheet I had a custom parallel cable made!

TerraServer_5_CompletedInternal

TerraServer_6_CompletedInternal2

With all the hardware done, it was time to install the OS. I wanted a stable server distro, so opted for CentOS Linux. The install process picked up the correct drivers (I just had to provide the Adaptec storage driver). I installed Samba to share out the drives, VNC server to connect to administer it (local subnet access only), and I had my TeraServer! The trickiest part was getting the LCD display to work. Parallel ports are rarely used anymore. With a bit of research I found an open-source lpt driver to install. Finally, configuring and installing LCDproc for my display module gave me a server that really looked like it means business.

TerraServer_7_@home

$1000 less than buying one, and all the fun of the build too!

TeraServer Project – Part 1

I need a server. It’s been on my mind for a while that I want a low-power, low-noise, compact server for storing all my work and personal files. I also wanted it to act as an always-on media server for playback of music and old digitized VHS tapes. That all needs quite a bit of storage capacity. Disk storage costs have come down now to a point where I can consider a home TeraServer. Home NAS devices are available but they are still pretty expensive (that’s before you even price in the hard disks.)

Whenever possible, I try to build my own home computers (I do draw the line at building custom laptops however), so building a server would be a fun new challenge. I would install Linux and use that to provide the file serving capabilities. Most NAS devices use Linux anyway, but a Linux install that I could customize would allow complete flexiblility in what the server could be used for later on.

On the hardware side, the TERA-ITX project inspired me. I could use a mini-itx motherboard and a small form-factor case. The storage could be done with four 250GB disks, configured in a RAID array. I decided early on not to consider using software RAID because it would just be another drain on the limited mini-itx processor. So I would need a hardware RAID card and a caddy for the disks. When pricing all that in, I worked out I could build it for a least $1000 less than buying a NAS and filling it with the same disks. A no-brainer!

And so to begin..

First, the motherboard. There were a lots of different specification mini-itx boards out there. The priority for me was to keep power consumption low – the case PSU would need to drive four SCSI disks, so I didn’t want an added drain from a powerful processor. A lower power processor would also mean less cooling, reducing power and noise further. The processor didn’t need to powerful just to server up files via the network anyway. Network speed was important, so a GigaBit on-board LAN was vital. In the end, I went for the EPIA-CN 1000

Next: the case. The case would need to support four disk RAID slots. There aren’t any such mini-itx cases like that at the time of writing, so I would just get a suitable cube form-factor case from mini-itx.com.

TerraServer_1_StartCase

The front plate configuration didn’t matter because I would be cutting it out to install a front-loading RAID caddy.

Therein lay the first rub.

I thought it would relatively easy to get a RAID caddy that would fit the case. In fact, I found out it was very difficult to find a bare-bones SCSI RAID caddy. Most of them come embedded as part of a case.

With a lot of internet searching I found that some IBM servers have a removable RAID caddy that fits a four 3.5″ drive bay slot. Perhaps I could find one of those IBM RAID caddy spare parts. The server line was discontinued though so finding the part proved difficult, but serendipity had a part to play.

TerraServer_3_RaidCage

With a lot more internet searching I eventually found just one available – from a small, independent computer part supplier just twenty minutes drive away from me! I had never known about the company until now and when I went to collect the caddy found out it’s owned by an old childhood neighbour from down the road. Dublin can be a small place!

Goto Part 2 ->