The University of California, San Diego
UCSD

Site Contents

Making a Read Only Filesystem to be served via NAS

These are Marvin's quick and dirty instructions (perhaps not so quick, but definitely very dirty), for making a Linux filesystem for serving read-only NAS-root (we use NFS-root). I used CentOS 5, but these steps should be very similar for other Linux distros (Identical for RHEL 5).

By the way, if you want to just skip all this and use my filesystem (or attempt to use), you can download it here. WARNING: It's a fairly straightforward CentOS 5 install, so it's about 500MB! You may also just want to look at it for reference in setting up your own. In addition, I would be remiss to not mention a very compact read-only Debian based distribution named Pebble Linux. I've used Pebble with success and can say it's quite nice and easy to use.

Create a Linux filesystem

Somehow, you have to get a complete Linux filesystem which you can export from your NAS box. If you're an expert at this or already have a filesystem ready to go, skip the steps in this section. Otherwise, do the following:

  1. If you're creating a Debian filesystem, you probably just want to use debootstrap to create your filesystem. I'm told Gentoo has a similar feature. Otherwise, install a system with your Linux distro (unless you already have a system with the distro you want without a bunch of extra configuration and packages). Doing this on a single partition is recommended, but not required. If you used debootsrtap, you're done with the steps in this section.
  2. After installation is complete, use the rescue mode from your installer (or any other rescue or live CD) to boot your system. Once booted from CD, mount the disk partition where you installed your new Linux system. You can then rsync the newly installed image wherever you want (even another place on the local disk). If you don't have your installation on a single partition, make sure you rsync all partitions. Here are the command's I used: 
$ mount /dev/sda1 /mnt
$ rsync -avH --progress /mnt/ my-nas-server:/exports/centos5-dom0

Notice that I sent mine to another machine. Of course, you'll have to setup networking in your CD boot session to do this. Otherwise, mount another partition of a local disk to which you can copy it (You could even later boot back into your newly installed Linux to copy it elsewhere if getting networking setup in your rescue session is too difficult).

Export it

Ok, you should now have a Linux filesystem to customize for serving read-only NAS-root. Go ahead and export your filesystem read-only from your NAS box to the nodes which will be mounting it. For me, using Linux NFS, I added the following to my /etc/exports file: 

/exports/centos5-dom0 192.168.0.0/24(ro,async,no_root_squash)

Don't forget to run exportfs -r after adding something to /etc/exports. You can use showmount to make sure all is well after doing this.

Notice that we export no_root_squash since this will be the root filesystem. Don't worry, we're exporting it read-only. For obvious reasons, I also don't have a password hash for root in /etc/shadow. I just have my ssh public key in /root/.ssh/authorized_keys.

Make it work when mounted read only

CentOS 5

CentOS 5 now has the Stateless Linux scripts merged into its startup scripts. Unfortunately, I can't seem to find any documentation on this, but reading the startup scripts reveals that all you need to do to enable the read-only filesystem setup is edit /etc/sysconfig/readonly-root file. Change both the READONLY and TEMPORARY_STATE variables to 'yes'. Then, edit /etc/rwtab to mount --bind any additional files or directories you'd like to be writable on the ramdisk. Anything after dirs will copy the entire directory from the read-only filesystem onto the ramdisk, then mount --bind it there. Anything after empty creates an empty directory in the ramdisk then does a mount --bind to make that a writable directory in your root filesystem. Finally, anything after files copies that file from the read-only filesystem into the ramdisk, then does a mount --bind to make the file writable.

Others

Unless your distribution provides a read-only option like CentOS 5, you'll need to do something similar to the following. This was done for CentOS 4.5.

Copy the rc.readonly script to the copied filesystem's /etc/ directory. Then, add the following to the filesystem's /etc/rc.d/rc.sysinit: 

if [ -f /etc/sysconfig/readonly-root ]; then
    . /etc/sysconfig/readonly-root

    if [ "$READONLY" = "yes" ]; then
        # Call rc.readonly to set up magic stuff needed for readonly root
        . /etc/rc.readonly
    fi
fi

You'll need to create /etc/sysconfig/readonly-root (or wherever you want it). You can just do an 

$ echo "READONLY=yes" > /etc/sysconfig/readonly-root

to do this.

Note, the path for the rc script may be different depending upon your Linux distribution. Also, if not using SysV init, you'll have to figure out how to get this script to run at the appropriate time during startup.

Next, modify the /etc/rc.readonly script to work for your setup.

Now, try it out and tweak as necessary. Below are a few additional tweaks I did, as well as a few things specific to our UCSD environment.

Additional Configuration

To really get things working well, I had to do a few additional things to my CentOS 5 filesystem. Here's a list of things I did:

  1. Added necessary device nodes to /dev for bootup (could also have added udev to initrd).
  2. Disabled lvm scan in /etc/lvm/lvm.conf (didn't need it and it was creating lots of error messages at boot time).
  3. Turned off onboot for eth0 in /etc/sysconfig/network-scripts/ifcfg-eth0 (already network booted).
  4. Edited /etc/fstab for nfs root env (In particular, removed local filesystems).
  5. Soft linked /etc/mtab to /proc/mounts (/etc/mtab -> /proc/mounts).
  6. Edited iptables to allow traffic to nfs server.
  7. Added several files and dirs to /etc/rwtab (i.e. files and dirs I want to mount --bind to in my ramdisk)
  8. Changed /etc/init.d/network to not shut down eth0 on system shutdown.
  9. Edited /etc/logrotate.conf and individual scripts in /etc/logrotate.d to be very aggressive about rotating logs and /etc/cron.daily/tmpwatch to be more aggressive about removing old files.

UCSD Customizations:

  1. Changed /etc/rc.d/rc to create nfs directories for startup scripts.
  2. changed /etc/xen/xend-config.sxp network-script to point to a script which doesn't break my network connection when xend starts since I also boot Domain-0 read-only nfs root.
  3. Edited /etc/xen/xend-config.sxp xend-relocation-hosts-allow parameter to allow my hosts to migrate to one another.

For Usher, I installed all dependencies for LNM listed on the Documentataion page.

Last Updated: 2007-08-29 by mmcnett
Report problems to: Marvin McNett