College of Technology - Staff & Faculty File Server Project

Mark StanislavCollege of Technology

Eastern Michigan Universitymstanisl@emich.edu

July 2007

Abstract

This document represents all relevant information regarding the College of Tech-nology’s Staff & Faculty file server as well as the back-up/monitoring server. This isnot an exhaustive documentation of every last aspect of the servers involved in thisproject, but it is comprehensive.

The College of Technology in late 2006 under the guidance of Markus Shane Davisfound the need to replace a mixture of under-maintained and mostly insecure Windowsservers with a single UNIX-like system. The project was to examine the problems withthe current implementation and greatly improve the technology currently being used.Notably, this project actually utilizes two servers. One server is the actual file serverthat is being accessed day-to-day by the employees. The other server is tasked withproviding back-up storage as well as system monitoring for the production server usingsoftware such as Nagios & Cacti. You can certainly run only the file server and stillhave a functioning situation, but it is not recommended as part of the strength of thisconfiguration is file remote redundancy and ensured uptime.

Portions of the document may need to be updated in regards to distribution ver-sioning, package installation methods, or network addressing. Such updates should bedone as soon as possible, with a log to show changes with attribution to the personmaking the amendment.

1

Contents

2

1 General Server Configurations

File Server Backup ServerMachine Dell PowerEdge 6650 Dell DimensionOperating System FreeBSD 6.2-Stable FreeBSD 6.2-StableProcessors 2 x 2.0GHz Intel Xeon 1 x 3.0GHz P4Memory 3.5GB 2.0GBHard Drive Capacity 340GB 300GBRAID Configuration Hardware None ConfiguredNetworking 2xGbit 1x100MbitHostname cotdrive.emich.edu cotbackup.emich.eduIP Address 164.76.127.39 164.76.127.48

1.1 File Server Details

This machine was previously a Windows-based file server for the College of Technologythat we re-commissioned for the purpose of this project. This machine works well as afile server because of its hardware RAID array accepting up to eight drives, each with acapacity of 73.4GB. Currently, seven of these drive bays are in use. The additional baycould be used in the future for a hot-swap drive that sits offline until needed. From this,we have generated two separate RAID arrays. This was done for logical simplicity, as wellas filesystem management and integrity.

RAID Array #1

• Uses two drives, totaling approximately 70GB

• Configured as RAID-1

• Contains OS installation and a 30GB slice for on-server backups (/backup)

RAID Array #2

• Uses five drives, totaling approximately 270GB

• Configured as RAID-5

• Contains entire space for user file storage (/storage)

The extra Gbit ethernet card in the server has been left available for future expansion andnetworking. No IP address has been allocated for it, so be sure to acquire one from theproper channels if you plan to use a non-internal IP address.

3

1.2 Backup Server Details

This machine is a classical tower-style machine that overly suffices the needs of a back-upand monitoring server. Nothing extremely unique as far as the configuration, other thanthat there are two individual hard drives creating the overall filesystem structure. Thiscould potentially lead to an issue as there is no symmetry in the drives and no RAIDis configured. However, for the purpose of what this box does, we are confident in theimplementation as is. A hardware failure will have to be addressed at that time.

2 Software Overview

The following is a quick overview of software installed on each server with its respectivepurpose mentioned. This list is in no way exhaustive, but does give a fair showing of whatwas originally installed. The major version number may be shown, only to reference whattree is being followed at the time of writing. Please be advised that any future tree changes(such as with Apache) needs to be carefully tested in a similar configuration as it may affectsome of the core services.

2.1 File Server

The major portion of this software relates to the web-based file manager. Gollem is aproject built in the Horde Framework, and officially released from the same developers. Ithas been customized for simplicity and design. In order for this software to function (whichis mandatory), we must have Apache installed with PHP 5. Pure-FTPD and OpenLDAPalso need to be installed.

Software PurposeApache Web Server (2.0.x) Allows for our web-based file manager to runMegarc A utility to interact with the RAID array on the serverTripwire File integrity monitoring and reportingGollem Web-based file manager for data accesssPure-FTPD Our method to authenticate GollemOpenLDAP Support for LDAP authenticationrSSH Provides SCP and SFTP abilities to users, without shell accessSSHD Remote system accessCVSup Synchronizes system software and ports treeOpenVPN Securely connect between the file server and backup serverSNMPD Allows our backup server to monitor the file server’s metricsRSync File synching for backups

4

2.2 Backup Server

The backup server is very similar to the actual file server in that it needs Apache to functionappropriately. The main services on this server are Nagios and Cacti. Both are web basedprojects, so there is reliance on Apache’s uptime. Also however, Cacti uses MySQL for abackend and must be present and functioning.

Software PurposeApache Web Server (2.0.x) Allows for Cacti and Nagios to runMySQL (5.0.x) Provides the database backend for CactiCacti Network and system metric graphing for monitoringNagios Network service and system uptime monitoringSSHD Remote system accessCVSup Synchronizes system software and ports treeOpenVPN Securely connect between the file server and backup serverRSync File synching for backups

3 Information Security

3.1 Overview

With faculty and staff data requiring much privacy and protection from tampering, it isessential that the file server is as secure as possible without interfering in the operation ofthe needed software. Many security features have been implemented through the FreeBSDkernel, system software, and also the network. As mentioned previously, the backup serveralso acts as a monitoring server for the file server.

Each subsection of this section could potentially lead to problems if re-configured improp-erly. Anyone attempting to rulesets of any security feature should produce a testing boxof a similar configuration, or better yet, an image of this machine. It becomes easy to lockyourself or others out of this machine by being too hasty with changes.

Changes listed will be outlined in greater detail later on in this document.

3.2 Basic Security Enhancements

The following portions will be changes in the configuration of very basic functionality.While most of these changes may first appear picky, they actually have a lot of power tochange the available exploitations of a server like this.

5

3.2.1 /var/tmp & /tmp Modifications

On a standard FreeBSD installation, /var/tmp and /tmp exist. Since we have fine-grainedcontrol over mounting options of our separate filesystem partition /tmp, we will symlink/var/tmp to that and create one large temporary file repository. File-system mountingoptions will be discussed later in this section.

A key part of our /tmp security is that we enabled the ’t’ bit on our /tmp folder. Thisallows us to prevent users from deleting other users files. By default, in a world-readablesituation, anyone could delete anyone else’s files.

3.2.2 Login Class Modifications - /etc/login.conf

One quick note is that we have changed the default password type from MD5 to Blowfishfor user accounts. This change is also reflected in /etc/auth.conf. Be sure to update bothif you change one or the other.

We have created two new login classes: restricted and admin. All LDAP authenticatedusers (which is our entire user-base sans administrators) should be part of this restrictedclass. Any student or staff administrators of this machine should be part of the adminlogin class.

The restricted class receives a lower priority on their processes than any administrator orsystem account by defining the ’priority’ parameter equal to 2. Also, the users in thisclass will be allowed to be idle on the system for no more than 30 minutes, with an overallsession time of 60 minutes.

Finally, users have a umask of 0017. This makes files readable and writable by the ownerand members of the same group. Of course, users are capable within the web-based filemanager to change file permissions to make certain files private from even co-workers intheir department.

3.2.3 RSSH for Restricted-Class Users

At the time of writing, users in the restricted class should have no access to an actualcommand-line interface. However, we do want to offer them access to SCP and SFTPservices. To do this easily, we use the RSSH as their login shell. This will prevent them fromlogging into an actual shell account, but grants them access to use SCP and SFTP.

At a later time, an administrator could grant person-by-person access to the server throughchanging simply their shell to bash or otherwise. This is a great way to provide fine-grainedsystem access without implementing overly complicated software.

6

3.2.4 Mounted Filesystems

One genuinely easy way to protect the system from unauthorized attacks is to limit the abil-ities of files on a mounted filesystem. Because our installation has been created on multiplepartitions, we are allowed to configured major portions of our installation differently.

On our installation, we have limited the file system in places that should not normallyinterfere with any functionality. The partitions for /backup, /home, /tmp, /var, and/storage are all configured mostly the same way. They each do not allow for executablesto be ran, or set-uid files to exist on their respective partitions.

In addition to those primary mounting options, the partitions of /backup, /home, and/storage also have the ability to utilize multi-label Mandatory Access Controls. At thetime of writing, no labeling is currently really being implemented, but the ability shouldthe need arise is already enabled.

Editing /etc/fstab is the appropriate way to make edits to these configurations. Anychanges you make will be reflected on a reboot. It is not advised to try and unmountfilesystems while the system is production-live; better to reboot than corrupt data.

3.2.5 Encrypted Swap Space

One rarely implemented, but easily configured feature of FreeBSD is encrypted swap space.Once physical memory is exhausted, FreeBSD will start using part of our hard drive tostore information. When this happens, sensitive contents can be kept on the hard driveand vulnerable to stealing. We’d rather not have our passwords readably cached. Onceenabled, the swap partition will be encrypted with AES-256-CBC.

3.2.6 Simplistic File Security - /usr/local/sbin/lockdown.sh

This very simple shell script executes a long list of changes to files and directories thatare mostly associated with the base installation. Primary objectives of this script are toprevent global reading access and snooping of the filesystem, as well as removal of set-uidbits that are not needed.

While the former task could be considered security through obscurity, this layer does a greatjob of frustrating a would-be attacker from spending their time trying to find interestingfiles. While the script is in no way exhaustive, it does do a great job. This script shouldexecute regularly within crontab.

7

3.2.7 User dot-file Security

The .bashrc, .bash profile, and .bash logout files all have the system immutable flag en-abled. This means that the user cannot do anything at all to these files. A very notablechange is that we set the user’s .bash history file to append only. This allows them to notwipe their bash history file to hide any possible wrong doing or otherwise. It does howeverallow the file to function as normal.

Note that this does not prevent them from simply switching shells, but few people com-mitting such acts are very aware of such rare system changes until it is too late.

3.3 Kernel Security Enhancements

Changes with the kernel have either been compiled directly into the kernel, or have beenaltered in some way using sysctl values. Viewing /etc/sysctl.conf will shows relevant boot-time changes.

3.3.1 Basic Networking Features

• We drop all packets on closed TCP and UDP ports without sending anything backto our sender. This saves on CPU time and can help aid in the making DDoS attacksless effective. (net.inet.tcp.blackhole=2 and net.inet.udp.blackhole=1)

• By allowing a large amount of sockets to be allowed open, we can prevent sim-ple DDoS attacks to our system. SYN attacks will have a harder time succeeding.(kern.ipc.somaxconn=32768)

• We will drop and log ICMP redirects to help prevent and monitor possible attacks.(net.inet.icmp.drop redirect=1 and net.inet.icmp.log redirect=1)

• Protect ourselves from SMURF attacks and other broadcast attacks.(net.inet.icmp.bmcastecho=0 and net.inet.icmp.maskrepl=0)

• Prevent information gathering attacks by someone tracking our IP IDs leaving arouter. (net.inet.ip.random id=1)

• We utilize IPFW2 for our firewall needs. In addition, we also enable Dummynet forbandwidth shaping, as it is part of the IPFW facilities. We will discuss the firewallconfiguration later in this document. No Dummynet ruleset is used, but the featureis enabled if needed. (options IPFIREWALL and options DUMMYNET)

8

3.3.2 Miscellenous System Features

• Our Process IDs (PID) are randomly generated with a defined modulus. This is an ob-scurity type feature, but that doesn’t hurt anything. (kern.randompid=modulus value)

• We don’t allow for core dumps as they could be used for an attacks on the system towork on exploiting software easier. (kern.coredump=0)

• Prevent users from seeing other users processes, sockets, and the alike. Great to limitusers from snooping on other people’s activities.(security.bsd.see other uids=0 and security.bsd.see other gids=0)

• We do not allow unprivileged processes from reading the system console messagebuffer or to invoke system debugging primitives.(security.bsd.unprivileged read msgbuf=0 and security.bsd.unprivileged proc debug=0)

• Quota support is compiled into the kernel, but not currently implemented. (optionsQUOTA)

3.3.3 Port ACLs

With the integration of part of the TrustedBSD framework, FreeBSD has the ability tolimit to whom a port can be bound by. In example, we limit the MySQL user (uid 88) toonly bind port 3306 (MySQL’s default port).

By configuring our Port ACL sysctl values appropriately, we actually limit the abilityof every single port on the machine to be bound by anyone except those authoritativelyspecified. By issuing security.mac.portacl.port high=65535 we force the administrator toalter the Port ACL ruleset to allow for any new ports that should be getting bound. All nec-essary ports have already been issued appropriately in the ruleset, found in /etc/sysctl.conf(security.mac.portacl.rules).

3.3.4 Security Event Audit Daemon

This is an implementation of Sun’s BSM API and file format for a security audit system.This system provides very invasive monitoring of users. The control file can allow forextremely fine-grained control of what events are logged, and for which users.

Files for editing how the system works are stored in /etc/security, while the actual auditlogs are stored in /var/audit. You can use the praudit utility on a logfile to view itscontents.

9

3.3.5 File System Firewall

Another part of the TrustedBSD framework integration is the bsdextended module. Thismodule provides for a file system firewall of sorts. The syntax is very similar to that ofIPFW, with a very different purpose. The advantage to this is that since is part of theMandatory Access Control side of FreeBSD, users aren’t capable of editing these permis-sions. The positive behind this is that even if they were to make all of their files worldreadable, it wouldn’t actually matter.

The default ruleset that we use can be found in /etc/rc.bsdextended. Please note that thisfile should be edited very carefully. As is, a script does most of the work, but you can addadditional rules for refined security if needed.

3.3.6 Process Accounting

Through the process accounting facility, we are able to easily monitor who is issuing whatcommands to our system, and what resources those processes took up. The process ac-counting database is saved as /var/accounting/acct and can be viewed using lastcomm andsa. It is advisable to read the man pages for both commands to get the best out of thisdata.

Process accounting is a great way to track a user’s activity or to help find where an attackmay have originated. The databases are automatically rotated, so space shouldn’t be anissue. It would be wise to occasionally monitor what commands a local user is executingif they are not a known administrator.

3.3.7 MAC Paritioning, MLS, and Biba Modules

For a full Mandatory Access Control configuration, FreeBSD provides three modules tohandle almost anyone’s paranoid needs. These are all available to the system but aredisabled due to the apparent overkill factor for this server. While we value the privacy ofevery person’s files, the privacy of each file doesn’t warrant such overbearing restrictionsfrom what we already have. If our system had more of a logical hierarchy to the staffing,with obvious lines in power for multiple people or groups, the usefulness of these would begreatly increased.

Be advised if you plan on implementing these that you can easily lock yourself out of thesystem very easily. Also, make sure you have a real purpose to configure all of the settings,as it can get quite involved depending on your userbase.

10

3.4 Security/Monitoring Software

3.4.1 Tripwire File Integrity Monitoring

Tripwire is a software package that allows you to create a list of files that you want tomonitor for changes. This is helpful to make sure critical system files aren’t being alteredaccidently or by another person. Reports can be generated and the output is easily readableto see what is occurring on the system.

By default, a report is generated and e-mail nightly to the person listed in the file/usr/local/etc/tripwire/twcfg.txt under the parameter ’GLOBALEMAIL’. Anytime majorchanges occur to this file and the report shows a lot of changes, the policy database shouldbe updated so that you can easily parse through future changes.

3.4.2 Cacti

Cacti provides us with constant monitoring of our file server for a number of metrics. Eachmetric is graphed multiple times an hour to provide an accurate picture of what is happen-ing with it. As of now, we are currently monitoring each hard drive partition individually,load average, logged in users, ping latency, processes, and network traffic.

Obviously each one of these metrics serves a different usefulness for monitoring the securityof our server. For example, if all of the sudden we see three users logged in when there areonly supposed to be two allowed accounts, we may want to see if there is a dupe login ora rogue user. Network traffic is always good to measure, especially for a file server.

3.4.3 Nagios

This services provides us another web interface to help monitor the availability of our fileserver. Nagios will allow us to get alerts via e-mail or otherwise if there is an unexpecteddowntime to our file server. More so, we can monitor individual services to make sure theyare available at all times. This is especially needed for our web server as it is the crux ofthe file server for end users.

3.4.4 OpenVPN

To establish a cryptographically secure connection between our file server and backupmachines, we will use OpenVPN. This allows us to transmit anything between the serversand have guaranteed security while doing so. This point-to-point link is established as alocal private network as far as IP addressing is concerned.

11

4 File Server Re-Installation Guide

The following is a very comprehensive set of steps to give an idea of how the server wasinitially configured. This isn’t a 100% step-by-step guide, but it is pretty close. If atanytime a re-installation is needed, follow the guide as closely as possible, fixing errorswhere they arise and document them. As FreeBSD and its package set changes, so will thetypes of problems that can occur during what used to be a quick installation.

IMPORTANT: You will note in a number of steps that files are copied from /files/. Asof this writing, there should be an attached tar file with this documentation, most likely ona CD-ROM. If you do not have these files, the installation will be much less smooth. If youdo re-install, please backup the files, if not more. Less configuration can be the differencebetween a 2-hour setup and a 20-hour setup.

4.1 Installation of FreeBSD

Obviously if you are following this document, you have some idea how to install FreeBSDon the computer already. This part of the documentation will merely list a number ofnotes for the initial installation to make sure things are in decent order on first boot.These may not necessarily be in the correct order, so read them over before beginning yourinstallation.

4.1.1 Initial Setup Questions

• Do not install a boot loader since we only are booting one OS on this server

• Install a full set of sources to ensure less downloading time to sync for re-buildingworld, use your best discretion which set to take

• Select ”YES” when prompted if we want the ports system installed, but do notactually install any ports until after we reboot

• Enable the SSH daemon, do NOT enable the INET deamon

• Do NOT act as a network gateway

• Select ”YES” when prompted if we want Linux Compatability

• Create the ’admins’ group

• Create a user (not your my.emich account) and add it into the admins and wheelgroups

12

4.1.2 File System Partitioning

That should get you through almost everything except partitioning of the file system.This is a very important part since it relates directly to our system security as well asoverall functionality. Sizes are approximate, so when creating your last partition (shouldbe /storage), just give it all of the rest of the space available.

Take time to make sure you edit the proper of the two arrays for each step. I *highly*recommend you keep the array structure the same or similar to what was established earlierin the document. Of course, any changes to the logical RAID arrays would have had to ofbeen done prior to installing FreeBSD.

Partition Size Reason/ 5GB Most files will be inside of other partitions, not much is neededSWAP 2GB Most likely overkill for our physical RAM, but worth having/backup 30GB Available if the need arises for local backups just-in-case/home 3GB Majority of accounts will be in /storage, only for local admin accounts/tmp 2GB Overkill again most likely, but we have the space/usr 15GB Bulk of our installations of software will be here, extra space is good/var 10GB A lot of logging and e-mail storage is needed/storage 245GB Our file server’s primary data storage space

4.1.3 Network Configuration

Choose to setup the network information manually, as you need to configure a static IPaddress. Use the assigned IP address for the server at the time you are configuring it. Besure to setup the fully-qualified domain name (FQDN) to ensure smooth setups of networkservices. Also be sure to use a current DNS server when configuring the name server inthe initial setup.

13

4.2 User Shells: Installation and Setup

First we need to install both Bash and RSSH as they are the two choices for accounts inthis system. Remember, all admins should run bash, all file server users should use rsshunless there is a specific need for shell access.

# cd /usr/ports/shells/bash && make install clean# cd /usr/ports/shells/rssh && make install clean

# cp -f /files/rssh.conf /usr/local/etc/

Next we will copy over a bunch of bash & vim dot-files.# cp -f /files/dot.bash history /root/.bash history# cp -f /files/dot.bashrc /root/.bashrc# cp -f /files/dot.bash profile /root/.bash profile# cp -f /files/dot.bash logout /root/.bash logout# cp -f /files/dot.vimrc /root/.vimrc

# cp -f /files/dot.bash history /home/myUser/.bash history# cp -f /files/dot.bashrc /home/myUser/.bashrc# cp -f /files/dot.bash profile /home/myUser/.bash profile# cp -f /files/dot.bash logout /home/myUser/.bash logout# cp -f /files/dot.vimrc /home/myUser/.vimrc

After the files are copied, we will set appropriate permissions to insecure their security.# chown myUser:admins /home/myUser/.bash*# chown root:wheel /root/.bash*

# chflags sappnd /root/.bash history /home/myUser/.bash history# chflags schg /root/.bash profile /home/myUser/.bash profile# chflags schg /root/.bashrc /home/myUser/.bashrc# chflags schg /root/.bash logout /home/myUser/.bash logout

Lastly, we will change our root and myUser accounts to use the bash shell. Once we havedone this, we will reboot. Rebooting in this process is helpful because it’s a good way tomake sure you don’t get too far with critical mistakes.

# pw usermod myUser -s /usr/local/bin/bash# pw usermod root -s /usr/local/bin/bash

# reboot

14

4.3 Quick Housecleaning

Below we will clean out some undesired files, create a useful symlink for updating ourfile location database and delete the toor user. Of importance also, we are going tosymlink /var/tmp to /tmp and add our /tmp file security feature that was mentionedpreviously.

NOTE: Be sure to alter these commands to fit your current situation. Change ’myUser’to the unprivileged user account that you created initially.

# cd /root && rm -f .cshrc .login* .mail* .profile .rhosts .k5login# cd /home/myUser && rm -f .cshrc .login* .mail* .profile .rhosts

# rm -f /.cshrc /.profile

# ln -s /usr/libexec/locate.updatedb /usr/sbin/updatedb# /usr/sbin/updatedb

# pw userdel toor

# mv /var/tmp/* /tmp/# rm -rf /var/tmp# ln -s /tmp /var/tmp# chmod +t /tmp

4.4 Adduser Configuration

For our addition of users to go more smoothly, we pre-configure a few portions of oursystem. We will add the ’restricted’ user group into /etc/group, copy our adapted ad-duser.conf to /etc/, as well as priming /etc/skel with user files and permissions.

# echo “restricted:*:1002:” >>/etc/group

# cp -f /files/adduser.conf /etc/

# cp -f /files/dot.* /etc/skel/

# chflags sappnd /etc/skel/dot.bash history# chflags schg /etc/skel/dot.bash profile# chflags schg /etc/skel/dot.bashrc# chflags schg /etc/skel/dot.bash logout

15

4.5 Logging Configuration

We have some custom logging demands for our server, as well as files that need to becreated to do so. Also, there are some log files that are not needed and we will delete themin the interest of directory cleanliness.

# cp -f /files/newsyslog.conf /etc/# cp -f /files/syslog.conf /etc/

# touch /var/log/ipfw /var/log/console.log

# rm -f /var/log/pflog /var/log/amd.log /var/log/all.log# rm -f /var/log/lpd-errs /var/log/slip.log /var/log/ppp.log# rm -f /var/log/kerberos.log /var/log/pf.today

4.6 Configuring /etc/fstab

Since your /etc/fstab may have different partitions then what we originally setup, it isbetter to explain what changes should be made to your /etc/fstab file and why. Adaptthese changes to your partitions and be sure to type them properly so that you don’t lockyourself out. These changes are primarily for security, but a couple are just functionalityrelated (like quota and proc support).

Mountpoint Options/ rw/backup rw,noexec,nosuid/home rw,noexec,nosuid/tmp rw,noexec,nosuid/usr rw/var rw,noexec,nosuid/storage rw,noatime,noexec,nosuid,userquota,groupquota

Also though, we want to add two lines for proc and swap. For the encrypted swap partition,at the end of the device name, add ’.eli’ (e.g. /dev/amrd0s1b.eli). For the proc filesystemadd a line like:

proc /proc procfs rw 0 0

16

4.7 User Classes & Password Security

One of the important portions of our user configuration is to separate each user into adifferent login class. Also, we configure users to have passwords with the Blowfish cipher.Regarding the user base we will mostly have, they never have local passwords stored, sothis is only for administrators. We have to reset our local account passwords in order forthis change to be accepted.

# cp -f /files/auth.conf /etc/

# cp -f /files/login.conf /etc/login.conf# cap mkdb /etc/login.conf

# passwd myUser# passwd root

4.8 Local Mail Configuration

For our cron job e-mails and system reports to be sent to an appropriate e-mail address,we are going to forward all e-mail bound for the root user to an external e-mail address.Also, we want to make sure mail is properly handled for our host.

Edit /etc/mail/aliases and where you find “root: ”, set it to an e-mail address (e.g. root:myUser@emich.edu). After doing that, you must rebuild the e-mail alias database.

# cd /etc/mail && make

Lastly, edit /etc/mail/local-host-names (the file doesn’t exist by default) and type eachhostname that the server should handle e-mail for on a new line.

4.9 IPFW Configuration

IPFW is an important configuration step because it grants us a lot of ability to filter outunwanted traffic to and from our server). With a properly configured firewall you canminimize a lot of basic security threats and add another layer of security to the overallmodel. The included IPFW ruleset should be perfect for what the file server is designedto do, but you will of course want to verify IPs and rules before launching it live.

# cp -f /files/ipfw.rules /etc/

17

4.10 Quota Support

In addition to our kernel and /etc/fstab changes to support user and group quotas, we alsowant to run a few commands. To administrate quotas easier, we will install the setquotapackage from ports.

# cd /usr/ports/sysutils/setquota/ && make install clean

# quotacheck -a# quotaon -a

4.11 Miscellaneous Configuration Files

Since not every configuration file needs its own step, we will lump a lot of them together.Please go ahead and read over each one if you have any questions as to why. Also, check forconsistency in things like IP addresses and hostnames. Also, we want to quickly removesome default sample templates for applications. You may want to run this second stepagain after all of your software installations.

# cp -f /files/rc.conf /etc/# cp -f /files/make.conf /etc/# cp -f /files/sysctl.conf /etc/# cp -f /files/motd /etc/# cp -f /files/banner /etc/# cp -f /files/ttys /etc/# cp -f /files/crontab /etc/# cp -f /files/rc.bsdextended /etc/# cp -f /files/hosts /etc/# cp -f /files/shells /etc/

# cp -f /files/loader.conf /boot/

# rm -f /usr/local/etc/*.sample# rm -f /usr/local/etc/*-dist# rm -f /usr/local/etc/*-recommended# rm -f /usr/local/etc/*.default

18

4.12 Software Installations

Obviously we have already done some software installations, and we will do others later.However, we will try and organize things some by separating out the majority of oursoftware installations, with some basic notes.

4.12.1 OpenSSL & OpenSSH Portable

The portable versions of OpenSSL and OpenSSH are easier to maintain than the built-inversions that ship with FreeBSD. The installation for these is really quick.

# cd /usr/ports/security/openssh-portable# make -DOPENSSH OVERWRITE BASE install clean

# cp -f /files/sshd config /etc/ssh/sshd config# /etc/rc.d/sshd restart

# cd /usr/ports/security/openssl# make -DOPENSSL OVERWRITE BASE install clean

4.12.2 OpenVPN

We are using OpenVPN for our secure communications across the unsecured network. Theconfiguration must account for both our servers in order to have a purpose. The backupserver will act as the server for the VPN, and the file server will act as the client.

# cd /usr/ports/security/openvpn && make install

# mkdir /usr/local/etc/openvpn

4.12.3 Pure-FTPD

For our web file manager Gollem, we are going to authenticate users through Pure-FTPD.Not only can Pure-FTPD give us LDAP authentication as a mechanism, but it also givesus some great security features. It is possible to run Gollem via LDAP directly, but aftertesting, I would recommend using Pure-FTPD.

# cd /usr/ports/ftp/pure-ftpd/ && make install clean# cp -f /files/pure-ftpd.conf /usr/local/etc/

19

4.12.4 LDAP Authentication

LDAP is the authentication method of choice for all staff and faculty accessing the server.We are authenticating them via their my.emich username and password. Each accountutilizing this type of authentication must have a basic shell template already established,otherwise they can’t login.

# cd /usr/ports/net/openldap23-client/ && make install clean# cd /usr/ports/net/nss ldap/ && make install clean# cd /usr/ports/security/pam ldap/ && make install clean

# cp -f /files/pam.d/* /etc/pam.d/

# cp -R /files/openldap /usr/local/etc/# cp -f /files/ldap.conf /usr/local/etc/# cp -f /files/nss ldap.conf /usr/local/etc/

4.12.5 CVSup

CVSup is an easy way to synchronize your system and port sources very easily. We willinstall the CVSup port, create a directory for the application to use, and copy our configfiles into that directory.

# cd /usr/ports/net/cvsup-without-gui && make install clean# mkdir /var/db/sup

# cp -f /files/cvs-supfile /var/db/sup/# cp -f /files/refuse /var/db/sup/

4.12.6 SNMP

For our backup server to monitor information about our server, we want to run SNMPto provide a standardized way for that collection to occur. SNMP will allow for manymetrics to pass their information on request to our backup server to use as desired by ourconfiguration settings.

# cd /usr/ports/net-mgmt/net-snmp && make install clean

# cp -f /files/snmpd.conf /usr/local/share/snmp/

20

4.12.7 Apache 2.0.x & SSL

As of this writing, there have been issues running Horde & Gollem on anything but Apache2.0.x. Be sure to test on another server if you plan to upgrade Apache, or even Gollem forthat matter.

# cd /usr/ports/www/apache20/ && make install clean

# cp -f /files/httpd.conf /usr/local/etc/apache/# cp -f /files/ssl.conf /usr/local/etc/apache/

# cp -f /files/cotdrive-cert.pem /etc/ssl/crt/# cp -f /files/cotdrive-key.pem /etc/ssl/key/

# rm -f /usr/local/www/data/index.*# rm -f /usr/local/www/data/apache pb*

4.12.8 PHP 5 & Horde/Gollem

Horde is a PHP framework, and PHP5 is the appropriate choice as of this writing. Thesesteps will get PHP installed, copy over our php.ini into the correct directory, installPEAR::DB, and give you a selection of PHP 5 extensions to install. Use some judge-ment in what you will need. Things like Session support, MySQL support, SSL support,and the alike are definitely needed.

# cd /usr/ports/lang/php5/ && make install clean

# cp -f /usr/local/etc/php.ini-recommended /usr/local/etc/php.ini# cp -f /files/php.ini /usr/local/etc/php.ini

# cd /usr/ports/databases/pear-DB && make install clean# pear channel-update pear.php.net# pear upgrade-all

# cd /usr/ports/lang/php5-extensions && make install clean

# cp -Rf /files/horde /usr/local/www/

21

4.12.9 Tripwire

Tripwire will monitor system files that should not be changed, or that we want to knownwhen changes have been made. It is a great way to monitor things daily on a server forchanges.

# cd /usr/ports/security/tripwire/ && make install clean

# cp -f /files/twcfg.txt /usr/local/etc/tripwire/# cp -f /files/twpol.txt /usr/local/etc/tripwire/# cp -f /files/site.key /usr/local/etc/tripwire/

# twadmin –create-cfgfile –site-keyfile site.key twcfg.txt# twadmin –create-polfile twpol.txt

# tripwire –init# tripwire –update-policy twpol.txt

4.12.10 Nagios

In regards to monitoring services and system availability, we use Nagios. The setup to getthe process going is quick, but configuration can be time-consuming. This is again onlymeant for the backup server to run.

Once installation is done, create htaccess and htpasswd files for authentication. Also verifyNagios is enabled inside of /etc/rc.conf.

# cd /usr/ports/net-mgmt/nagios && make install clean

# cp /files/nagios/* /usr/local/etc/nagios/# chmod 644 /usr/local/etc/nagios/*

# cp /files/httpd-nagios.conf /usr/local/etc/apache2/Includes/# chmod 644 /usr/local/etc/apache2/Includes/*

22

4.12.11 Cacti

On our backup server we need to setup Cacti for our graphing of the file server’s systemmetrics like memory usage, network usage, and system load. This requires MySQL to beused, so make sure that is setup on the backup server first.

After the installation is done, the username and password is admin/admin. Login, changethe password and set a font path.

# cd /usr/ports/net/rrdtool && make install clean# cd /usr/ports/net/cacti && make install clean# cd /usr/ports/net/cactid && make install clean

# mysqladmin –user=root -p create cacti# echo “GRANT ALL PRIVILEGES ON cacti.* TO cacti@localhost IDEN-TIFIED BY ’a password’;” — mysql -uroot -p# mysql -ucacti -p cacti ¡ /usr/local/share/cacti/cacti.sql

# cp /files/db-settings.php /usr/local/share/cacti/include/# cp /files/httpd-cacti.conf /usr/local/etc/apache2/Includes/

# chmod -R 655 /usr/local/share/cacti/

We need a cronjob task added for Cacti to properly process data. Add to your /etc/crontab:*/5 * * * * cacti php /usr/local/share/cacti/poller.php >/dev/null 2>&1

4.13 Archiving Utilities

These are a list of ports to install for archiving/unarchiving utilities that are good to havehandy.

# cd /usr/ports/archivers/bzip2/ && make install clean# cd /usr/ports/archivers/rar && make install clean# cd /usr/ports/archivers/unrar && make install clean# cd /usr/ports/archivers/unzip && make install clean# cd /usr/ports/archivers/zip && make install clean# cd /usr/ports/archivers/untar && make install clean

23

4.14 Miscellaneous Software

These are a list of ports to install for archiving/unarchiving utilities that are good to havehandy.

# cd /usr/ports/editors/vim-lite && make install clean# cd /usr/ports/editors/nano && make install clean# cd /usr/ports/www/lynx-ssl && make install clean# cd /usr/ports/ftp/wget && make install clean

# cd /usr/ports/sysutils/megarc && make install clean# cd /usr/ports/sysutils/lsof && make install clean# cd /usr/ports/sysutils/screen && make install clean# cd /usr/ports/devel/strace && make install clean

# cd /usr/ports/net/netcat && make install clean# cd /usr/ports/net/rsync && make install clean

24

5 How to Maintain the Server

After you have configured your system appropriately, before setting it for production use,you will want to recompile the base system as well as upgrading all port software. Moreso, you will want to execute these tasks on an interval of your choosing.

It would be ideal if you could schedule downtime every three months to keep the sys-tem adequately safe (under general circumstances) from vulnerability. Of course, theremay be a time when a security issue presents its self and an immediate upgrade may beappropriate.

It is worth mentioning that you could synchronize your sources daily if you really wanted.That would minimize the amount that you would have to download in case of an emergencyrecompile of the system. Just because you download the sources does not mean you needto compile the system.

5.1 CVS Synchronization

The initial step in keeping your system software up to date is to synchronize all of yoursources. When using CVSup you will gather the source updates for the system tree that youare following. Look over /var/db/sup/cvs-supfile for the options you have in configuringthis task.

NOTE: If you receive a rejection note from the CVS server, try editing the cvs-supfilewith a different number in the server name.

# cvsup -g -L 2 /var/db/sup/cvs-supfile

5.2 Building World

In FreeBSD, the task of recompiling all of your base system software is called ’BuildingWorld’. Once your sources have been updated, this task will run through all of your sources,compiling where appropriate and preparing the contents of your updated system. At thispoint, nothing is being overwritten.

NOTE: If you receive any errors during this process, do not just ignore them. Buildingworld should be an error-free process.

# cd /usr/src# make buildworld

25

5.3 Compiling a New Kernel

Your kernel is the cornerstone of the entire Operating System. If your kernel is improperlyconfigured, your system will most likely fail on the next reboot. The included configurationfile for the kernel will be copied over to the appropriate spot in the first step. Dependingon what version of FreeBSD is out when you are compiling, you may want to read up onany changes between FreeBSD 6.2 and today’s version.

# cp -f /files/cotdrive.emich.edu /usr/src/sys/i386/conf/

# cd /usr/src# make buildkernel KERNCONF=HOSTNAME# make installkernel KERNCONF=HOSTNAME

5.4 Install World

Pending your kernel compiled and installed properly, the next step before you boot into yourfreshly compiled system is to install all of the software we previously compiled. NOTE:When you install world, you must make sure that /tmp is configured without noexec & no-suid mounting options. If you have to, change the options for the mount point in /etc/fstab,and reboot.

# cd /usr/src# make installworld

5.5 Final Steps to Updating the System

If you have completed each of these steps, you can now reboot into the new system.Hopefully everything is working properly and you can keep the process moving. Your nextstep is to verify all of your software is up and running appropriately. Once you do, moveon to updating the files in /etc.

5.5.1 Mergemaster the /etc directory

When updating our system, we have to integrate changes to the /etc directory. Whileusing mergemaster, carefully select to either install the newly downloaded file (press ’i’) ordelete the new file and keep our existing file (press ’d’). If you pay attention and go slow,the process should be simple and not ruin anything.

# mergemaster -cv -w 120

26

5.5.2 Remove Build Leftovers

In the interest of saving space and keeping our compilation directories clean for the nextcompilation of world, we will go ahead and remove old files that are no longer needed.Be sure to type these commands probably as you could potentially ruin something if youmisplaced them ’rm -rf *’ command.

# cd /usr/obj# chflags -R noschg *# rm -rf *

5.6 Port Upgrades

In relation to us updating all of our sources on the system, we also received updates to theports tree. Because of this, we are able to update the port installed software on our system.If this is your first run through, be sure to actually install the portupgrade utility. Else,just follow the steps for updating the ports database and executing portupgrade.

NOTE: The portsdb command can take a while to execute, so be patient! After the portsdatabase updates, go ahead and start updating software using the portupgrade command.

# cd /usr/ports/ports-mgmt/portupgrade && make install clean# portsdb -Uu# portupgrade -aiRr

27

6 How-Tos

This section is dedicated to some notes on a few tasks that may need a small bit ofexplanation. These notes are possibly not fully complete, but are a great starting point forcompleting the task. Use them loosely to get an idea of what to do, and adapt as needed.It wouldn’t hurt if this document was updated regularly with new How-Tos so that theusefulness of this section continues to grow.

6.1 Add/Remove LDAP Users

Every staff or faculty member that we want to authenticate via LDAP, we need them tohave a template of an account present in our password database. Even though we don’tstore a password locally, we still have to have them created as a user. Without this step,LDAP users will not be able to login or do anything else.

To add a user for LDAP usage

1. Type: adduser

2. Type the my.emich username of the account you want added

3. Type the full name of the user you are adding (as best as you know it)

4. Leave the UID empty (just press enter)

5. The default login group for any staff or faculty is just staff, so press enter

6. If you have groups created and know the user is supposed to be added, type the groupname you want to add the user to

7. The login class for all staff and faculty is restricted, so just press enter

8. The default shell for every staff and faculty is rssh, so just press enter

9. The directory that you are given is correct, so just press enter

10. We want password-based authentication, so just press enter

11. We do want an empty password. This is necessary for LDAP authentication to beused. Just press enter

12. We do not want to lock out the account after creation, so just press enter

13. Verify all of the information is correct and then type ’yes’

14. If you want to add another user, type ’yes’. Else, type ’no’

15. Verify that the user was created successfully by checking /storage/ for their account

28

To remove a user

1. Be sure any files that the user has that may be needed by someone else in thedepartment or on campus is backed up securely

2. Type: chflags noschg /storage/user name/.bash*

3. Type: chflags nosappnd /storage/user name/.bash*

4. Type: rmuser user name

5. Verify that the record you are deleting is the correct one

6. Type ’yes’ if you want to remove that record for sure

7. Type ’yes’ to delete the user’s home directory (remember, verify you don’t need theirfiles first)

8. Edit /etc/group with your favorite text editor and remove the username you justdeleted from any groups for cleanliness

6.2 Add/Remove User Groups

For each user account that we add, we want to have at least one user group to add theminto. By default, every user is going to be placed into the ’staff’ user group as it gives usa standard place to have people. When editing user groups, you won’t actually see usersin the staff group, but they are. User groups are found in /etc/group and are easily editedwith any text editor.

To add a user group

1. Open /etc/group in a text editor of your choice

2. Go to the bottom of the list of groups and find the last user group number in the list(e.g. 1008)

3. Go to the next empty line and copy & paste the previous group template and changethe name of the group (the first parameter) and increment the user group number(third parameter)

4. Verify that your new user group is not a dupe of another group, not a dupe of agroup number, and has the correct syntax

5. Add users you want in that group after the last colon, and add each additional onewith a comma after the previous

6. Save /etc/group

29

To delete a user group

1. Open /etc/group in a text editor of your choice

2. Go to the line of the file for the user group you want to delete

3. Delete the line, end-to-end

4. Save /etc/group

6.3 Add/Remove Department Share

Since our server hosts multiple departments at the College of Technology, we want to beable to share files between members of each area. To do so, we have group shares that arebased on the department that the users are in.

Be careful removing a department share as it could contain a lot of valuable information topeople other than the one who requested you delete it. More often than not, a departmentshare should never be removed, unless the department goes away or the name of thedepartment changes. Either way, exercise great caution with this step.

To add a departmental group share

1. Type: ls -la /storage/shares and verify that your desired share doesn’t already exist

2. Type: mkdir /storage/shares/desired share

3. Type: chmod 770 /storage/shares/desired share

4. Type: chown root:department user group /storage/shares/desired share

5. You now have to create a symlink to each user that needs the department share access

6. Type: ln -s /storage/shares/desired share /storage/user account/

7. Type: chown user account:staff /storage/user account/desired share

8. Type: chmod 770 /storage/user account/desired share

To delete a departmental group share

1. Verify that you backup any files in the directory share you want to delete

2. Type: rm -rf /storage/shares/desired share/ (Be very careful)

3. Type: rm -f /storage/*/desired share (Be very careful)

30

6.4 Enabling Multi-Label Security on Your Partitions

If you decide to use MLS as part of your security plan on the new server, you will need toenable it on each partition that you want to use it on. For our purposes, you should onlybe enabling MLS on /, /storage, /home, and /backup.

1. Edit /etc/fstab and set / to ro

2. Reboot the machine

3. Type 4 at the menu screen (Single User Mode)

4. Type: /sbin/tunefs -l enable /

5. Reboot again

6. # Type: mount -urw /

7. Edit /etc/fstab and set / to rw

8. # Type: shutdown now

9. # Type: umount /storage && tunefs -l enable /storage

10. # Type: umount /home && tunefs -l enable /home

11. # Type: umount /backup && tunefs -l enable /backup

12. # Reboot a final time

6.5 Setting a “Secure Level”

While it would be ideal to user the FreeBSD secure levels, we would have to change someof how we do things on the system. As of this time, the secure level is at your standardInsecure Mode. This is because we use the system immutable and append-only flags onour system with user accounts. Unfortunately, in order to delete a user properly, we haveto remove these flags from their bash files.

Secure Level Alterations

1. Verify that either you are fine with some user files not being able to get delete or youare not settings schg or appnd flags on user bash files

2. Type: sysctl kern.securelevel=your number (Recommended: 3)

3. Add a line into /etc/sysctl.conf for this setting

31

6.6 Create an SSL Certificate For Apache

You must be running SSL on Apache when using Gollem. Do not skip this step as people’susernames and passwords will be easily vulnerable to being sniffed. If for some reason youhave to create a new SSL certificate for the server, do so via this method.

# cp -f /files/openssl.cnf /etc/ssl/# mkdir -p /root/ssl cert/certs /root/ssl cert/private /root/ssl cert/newcerts# touch /root/ssl cert/index.txt# openssl req -new -nodes -out cotdrive-req.pem -keyout private/cotdrive-key.pem -config /etc/ssl/openssl.cnf# openssl req -new -x509 -days 3650 -extensions v3 ca -keyout pri-vate/cakey.pem -out cacert.pem -config /etc/ssl/openssl.cnf# openssl ca -config /etc/ssl/openssl.cnf -out cotdrive-cert.pem -days 3650-infiles cotdrive-req.pem# mkdir /etc/ssl/crt# mkdir /etc/ssl/key# cp -f /root/ssl cert/cotdrive-cert.pem /etc/ssl/crt# cp -f /root/ssl cert/private/cotdrive-key.pem /etc/ssl/key

32