Using the Usher System

Here, I describe the details of using the Usher system as setup on our cluster at UCSD. For general Usher documentation, see the Usher Documentation page. This is one of many ways the system can be setup to manage virtual clusters.

Ush

Ush is an Usher client which provides a command line interface to the Usher system. At UCSD, an installation of ush resides on vm.ucsd.edu. To create virtual machines, first login to vm.ucsd.edu and run 'ush'. Next, connect to the Usher controller using the ush 'connect' command. A site-wide ush configuration file exists on the vm.ucsd.edu VMs, so you should not need to specify a controller name (i.e. simply run 'connect' and enter your SysNet password at the prompt). Finally, use the ush 'start' command to create new VMs. After that, you can also run ush from any of your created virtual machines.

Optionally, you can create an ush config file and set the environment variable 'USH_CONFIG' to point to it. There you can set connection options (including your password (watch out!)) to avoid having to explicity specify them to the 'connect' command. As mentioned, a site-wide ush config file already exists with everything except your password (in /etc/usher/ush.config).

For examples of using ush to connect to an Usher controller, start VMs, etc, check out the Screenshots tutorial.

Ush Commands

To see a list of possible ush commands, type '?' or 'help' at the 'ush>' prompt. To get command specific help, use: help <command>. At the time of this writing, the command list is (as output by the 'help' command in ush):

ush> help
Command         Brief Description         
=========================================
 ?              show this help message
 connect        connect to an Usher controller
 cycle          power off and restart VM(s)
 destroy        power off VM(s) and unregister
 help           show this help message
 list           list VMs
 migrate        migrate VM(s) to different VMM
 pause          pause execution of VM(s)
 quit           quit ush shell
 reboot         reboot VM(s)
 reconnect      reconnect to Usher Controller after disconnect
 resume         resume paused VM(s).  See 'start' for hibernated VMs.
 shutdown       shutdown VM(s)
 start          start VM(s) (creating if necessary)
 vmm_list       show list of VMM names

For verbose command specific help, use: help <command>

Always get the latest list of available commands by running help in ush.

VM Names

Your VMs will be created with the domain name

<your username>.<Usher system domain name>

and will have both forward and reverse name resolution. In addition, round robin DNS entries can be added for VMs using the '--rr' flag to 'start'. For example, starting '1.bluefish.seuss.usher.ucsdsys.net' and '2.bluefish.seuss.usher.ucsdsys.net' with the '--rr' option will create of two round robin dns entries named 'bluefish.seuss.usher.ucsdsys.net'. Then, logging into 'bluefish.seuss.usher.ucsdsys.net' will log into either '1.bluefish' or '2.bluefish' in a round robin manner. For this reason, it's important to not create a VM with the same name as a cluster in which you currently have VMs running with round robin dns entries. That is, don't create 'bluefish.seuss.usher.ucsdsys.net' if you have '1.bluefish.seuss.usher.ucsdsys.net' and '2.bluefish.seuss.usher.ucsdsys.net' running with round robin dns entries. If you do, you may have trouble connecting to 'bluefish.seuss.usher.ucsdsys.net' by name.

At UCSD, the Usher domain name is 'usher.ucsdsys.net'. You can create any level of subdomain in this domain. For example, user seuss can issue the command 'start redfish.sneetch' resulting in startup of the VM named 'redfish.sneetch.seuss.usher.ucsdsys.net'. Note, however, that you cannot start the VM <your username>.usher.ucsdsys.net, as this VM is actually in the usher.ucsdsys.net cluster (in which you do not have VM start permission).

Clusters in Usher

In Usher, clusters are essentially administrative domains, where administrative domain boundaries are defined by domain names. Each level in the domain name hierarchy defines a separate cluster. Currently, these clusters are not hierarchical (but could be made to be if the consensus is that it should). So, a VM with hostname 1.bluefish.seuss.usher.ucsdsys.net is a member of the "bluefish" cluster, whereas a VM named bluefish.seuss.usher.ucsdsys.net is a member of the "seuss" cluster. Hence, these VMs are in two separate administrative domains and users with login on bluefish.seuss.usher.ucsdsys.net cannot login to 1.bluefish.seuss.usher.ucsdsys.net and vice versa. However, users of bluefish.seuss.usher.ucsdsys.net would be able to log into 1.seuss.usher.ucsdsys.net and redfish.seuss.usher.ucsdsys.net because these are all in the seuss.usher.ucsdsys.net cluster. The following table should squelch any lingering confusion.

VM Domain Name	Cluster
1.seuss.usher.ucsdsys.net	seuss
bluefish.seuss.usher.ucsdsys.net	seuss
redfish.seuss.usher.ucsdsys.net	seuss
1.bluefish.seuss.usher.ucsdsys.net	bluefish
2.bluefish.seuss.usher.ucsdsys.net	bluefish

Using Your VMs (UCSD setup)

After creating a VM or two, you'll probably want to do something with them. Before delving too deep into the documentation below, here's a short list of common things you might want to do with links to the relevant section.

I Want To…

• Create a new user in one of my clusters
• Link an existing user into one of my clusters
• Create a new group in one of my clusters
• Manage users for my physical machines using one of my Usher cluster administrative domains
• Use a different home directory
• Mount a remote directory
• Run a service in my cluster (or on a single VM in my cluster)
• Install additional software
• Run a command as root
• SSH to my VMs as root (with or without password)
• Use my own filesystem as my VM root filesystem
• Use my own kernel

Special Groups and Sudo

Before I say anything else here, let me just say, "Do not run 'usher_' commands using sudo or as root" (see below for what the 'usher_' commands are).

OK, let's move on. Cluster creators are automatically added to the 'admin' group of every cluster they create. Users of the admin group can change other cluster users' attributes in LDAP using the 'usher_' commands (they cannot change attributes of users in other clusters owned by you)

If you need to run a command as root, 'admin' group members have been granted unrestricted sudo privileges. No default root password has been set for your VMs. Also, because our VMs are running ro NFS root, setting a root password will not persist between VM restarts. So, if you want to actually become root, you can do a 'sudo /bin/bash' or read the Public Keys section on how to ssh to your VMs as root.

There is also a lesser group named 'power' in every cluster. Users of this group have unrestricted sudo privileges, but are not free to change attributes of other users in LDAP. In this way, cluster admins can grant unrestricted sudo to a cluster without worrying about exposing cluster users' private information or giving rights to change that information.

For directions on registering public keys which will be copied to root's ~/.ssh/authorized_keys file, carefully read the Public Keys section. Note that any keys manually copied to root's ~/.ssh/authorized_keys file will be lost the next time the VM is restarted since we're running ro NFS root. For this reason, persistent keys there must be registered.

User and Group Management

In this and the following sections, 'Usher user' refers to users who are able to create clusters using the Usher system, rather than users who are merely able to login to clusters managed by Usher. 'Usher user' is used interchangably with 'cluster creators' below. 'Cluster users' refer to users in the cluster in question rather than all users in all clusters.

A command line suite has been written and installed on the VM filesystem for managing users. All commands in the suite begin with 'usher_'. Note that these commands modify LDAP entries and attributes and should not be run using sudo since LDAP doesn't typically care what your local credentials are (although our LDAP server is setup to verify that you're in the 'admin' group for the cluster before attempting to modify another cluster user's attributes). You'll always be prompted for your password when attempting to modify attributes. The commands are as follows:

• usher_groupadd Add a new group to a cluster
• usher_groupdel Delete a group from a cluster
• usher_grouplist List cluster groups
• usher_groupmod Modify a group (i.e. add/remove users, etc)
• usher_passwd Change/set user password
• usher_useradd Add users to your cluster
• usher_userdel Delete a user from a cluster
• usher_userlist List cluster users
• usher_userln Link an existing user (in any other cluster) into your cluster
• usher_usermod Modify a user's attributes

Verbose help for each command is made available by running the command with the '-h' flag.

Passwords

To set or modify a user password, use the usher_passwd command rather than the usual passwd command. Users are free to change their own passwords. Admin group members can change the password of anyone in the cluster. Usher users are automatically inserted into the admin group for clusters they create.

As mentioned above, because our VMs are running ro NFS root, setting a root password will not persist between VM restarts. If you are in the 'admin' group and want to become root, you can do a 'sudo /bin/bash' or read the following section on how to ssh to your VMs as root.

Public Keys

Users can register a public key or set of public keys with their account using the usher_usermod command. The point of doing this is to enable password-less login via ssh.

Note: The act of registering a key or set of keys will overwrite your ~/.ssh/authorized_keys file with your registered keys. So, be sure all the keys you want there are registered (if you decide to register public keys).

Root

Usher users (i.e. cluster creators) will also have their registered public keys copied to /root/.ssh/authorized_keys. As a result, they will also be able to login as root without password. Note that any keys manually copied to /root/.ssh/authorized_keys will be lost the next time the VM is restarted since we're running ro NFS root. For this reason, persistent keys for root must be registered. One caveat here is that registered keys are not copied to /root/.ssh/authorized_keys on already running VMs (although they will be the next time the VM is started/restarted). In this case, if you want them there, you'll either have to restart those VMs or copy the key(s) manually (using sudo).

As a cluster creator, if you do not what your public keys copied to /root/.ssh/authorized_keys, you can always manually edit your own ~/.ssh/authorized_keys file (i.e. don't register your keys).

If you want something more flexible, feel free to create any sort of startup script to add whatever you want to root's authorized_keys file and forgo registering keys (see Running Services on how to run things at VM start time).

Adding New Users

Adding new users to your cluster is quite easy using the usher_useradd command. Make sure you understand the scope of user access to you clusters as mentioned in the Clusters in Usher section above.

If there are users in another cluster (not necessarily a cluster owned by you), you can link them into your current cluster using the usher_userln command mentioned in the following section.

Linking to Existing Users

Rather than creating the same user over and over for every cluster to which you want the user to have access, you can simply create the user once, then link her into any additional clusters. This is done using the usher_userln command.

Note that linked users do not need to be in clusters which you've created. This makes it easy to give access to users you didn't initially create, including Usher users.

Caveat: A linked user's id number cannot conflict with any existing user's id number in your cluster. Talk to me if you need to change someone's id number, but don't know how (there's more to it than using the usher_usermod command).

Adding Groups

Use the usher_groupadd command to add new groups to your clusters. To add or remove users from a group, use the usher_groupmod command rather than the usher_usermod command (many UNIX systems stupidly have usermod provide this functionality).

Authentication for Physical Machines

Although handing out computing resources at the granularity of physical machines goes against the spirit of Usher, sometimes folks need a few. One cool thing about the way I've setup Usher for SysNet is that it creates a separate branch in its LDAP server for every cluster a user creates (which is only editable by that user). So, if user 'sneetch' wanted authentication for his physical machines to be handled by the LDAP branch created for his 'fire' cluster, he simply needs to point his machines at the branch for that cluster (Note: sneetch doesn't even have to have VMs running on that cluster, since this never gets deleted once created). The easiest way to do this (Using CentOS or RedHat) would be to copy the /etc/ldap.conf and /etc/openldap directory from one of his fire.sneetch.usher.ucsdsys.net VMs to his physical machine(s). (This is probably quite similar for Debian, but may not be exactly the same. Send me a note if you know for sure.) Once copied, sneetch can create and manage users and groups (using the 'usher_' scripts I wrote and placed on the Usher VM filesystem) for his physical machines himself.

Note: be sure you've enabled LDAP authentication for your physical machines. RedHat flavors have a nice program called 'authconfig'. I believe others have to edit their pam files and /etc/nsswitch.conf files.

Ramdisk

The root filesystem for VMs at UCSD is being served as read-only NFS root. Since Linux requires some files and directories to be writable, a ramdisk is setup at system startup where these files are copied and mounted using the mount '--bind' option to make them writable. This filesystem has been restricted to 64 MB in size.

Note that /tmp is one of the directories mounted on the ramdisk. If you need a larger /tmp directory (i.e. your applications really must write large files to /tmp), feel free to mount it on a remote nfs server. Make sure the directory on which you're going to mount /tmp has the sticky bit set and is rw by all. Assuming 'newtmp' is the exported directory, running a 'chmod 1777 newtmp' on the directory before mounting /tmp there will properly set the permissions.

Remote Filesystems

At UCSD, all Usher filesystems (with the exception of the ramdisk mentioned in the previous section) are remote. The root filesystem is being served as read-only NFS root. There are, however, writable filesystems mounted by your VMs to facilitate customization. The following section discusses these.

Usher Writable Filesystems

By default, your VMs will NFS mount (rw) three different filesystems. All mount points are under /net. Looking there, you'll find: /net/global, /net/cluster, and /net/local. Under each of these, you'll find and etc directory.

The purpose of these three filesystems is as follows:

1. /net/global This is where you'll install or store anything you want accessible by all of your clusters, independent of their name. The contents of /net/global will be the same for all virtual machines you create.
2. /net/cluster This is where you'll keep any files you want accessible by the current cluster only. The contents of /net/cluster on '1.bluefish.mmcnett.usher.ucsdsys.net' and '2.bluefish.mmcnett.usher.ucsdsys.net' will be the same, but '1.redfish.mmcnett.usher.ucsdsys.net' will mount a different filesystem for it's /net/cluster.
3. /net/local This is unique to the current VM only. The contents of /net/local will be different for every VM you create. This might be where you setup services and config files for your cluster to which you don't want all VMs in that cluster to have access.

Finally, you are given a home directory in the usher system. Automount will take care of mounting this directory upon login for all your VMs. See below for setting an alternate URI for your home directory if you'd rather not use the one provided by our Usher setup.

Please note: No quotas are in place here at UCSD (and hopefully will not need to be), although space is currently very limited. Please try to keep your combined disk usage in the above directories below 10GB. Temporarily going over this is fine, but please try to rsync data off at your earliest convenience.

Mounting

Since you are admin on your VMs, feel free to nfs mount anything you've exported to them. Another option is to use sshfs to mount directories on machines to which you have ssh access. In the simplest form, you can do:

nfs:

As root (assuming /exported/directory has been properly exported to your VM(s)):
mount -t nfs host:/exported/directory /mnt

sshfs:

No need to be root or have anything exported here:
mkdir ~/mnt
sshfs user@host:/dir/to/mount ~/mnt

As Home Directory

As mentioned, our Usher setup already provides an NFS exported home directory for you. However, you're not obligated to use this. Feel free to change the URI for your home directory provided your remote home directory has been properly exported to your VM(s). This can be done using the 'usher_usermod --homeuri=URI' in one of your Usher VMs. Currently, only NFS URIs are accepted in the form:

nfs://hostname/path/to/home

You should use a fqdn for the hostname. Note that this takes a URI, not nfs mount point. To see your current homeuri, run: 'usher_userlist -l' in a VM.

Keep in mind that if you merely need access to a few files on a remote server, you can simply nfs or sshfs mount the containing directory (see above for simple examples) without having to change your Usher home directory.

If you're using NFS, make sure that your user id number on the remote server matches your user id number in the Usher system. This is not a problem for sshfs, however. SysNet users have the same id number as their SysNet account.

home directory via sshfs If there's a demand, I'll make this work. Simply send mail to Marvin McNett. No need to type anything in the body.

As Root Filesystem

This isn't fully tested, but should work (or work with a minor tweak or two). You can specify this at VM start using the '--fs_uri' flag to start. Again, make sure the directory has been properly exported to your Usher VM(s).

Installed Software

Apart from a nearly 'kitchen sink' CentOS installation, a few additional packages have been installed. I list these here in hope that folks will not install them into their clusters without knowing they're already available. All additional packages have been installed into /usr/local/ and added to your path. These include:

• Java JDK 1.6.0
• Matlab
• Eclipse
• Acroread

If there's anything you feel should be part of the standard VM filesystem, let me know.

Installing Your Own Software

What good is all this functionality if you can't install and run your own software? Well, it's certainly not as good as if you can, so feel free to install any software you need that's not already part of the default VM filesystem. Depending upon what you want/need, there are four different locations for doing this as described in the Usher Writable Filesystems section.

If you want to run your installed software as a service, read the following section.

Running Services

Even though our VMs are running read-only NFS root, our Usher setup has been configured to allow users to start services at VM startup. In each of /net/global, /net/cluster, and /net/local, there exist:

• ./etc
• ./etc/rc.d
• ./etc/rc.d/init.d
• ./etc/rc.d/rc0.d
• ./etc/rc.d/rc1.d
• ./etc/rc.d/rc2.d
• ./etc/rc.d/rc3.d
• ./etc/rc.d/rc4.d
• ./etc/rc.d/rc5.d
• ./etc/rc.d/rc6.d

Unless the runlevel is changed by passing kernel options to your VMs at startup (using the '--kernel_parms' flag), all our VMs will start in runlevel 3. So, to start up a service in a particular cluster, you'd write its startup script into /net/cluster/etc/rc.d/init.d, then link to it in /net/cluster/etc/rc.d/rc3.d with a S<number> prepended to the name. The number specifies the order for services to startup, so lower numbers will be started first. See below for an example.

Know that many services need writable locations for saving logs, writing lock files, etc. You must configure the service to write to rw locations, or make certain points in our ro filesystem rw by mounting them using the '--bind' option to mount. Check out the mount man pages and the /etc/rc.readonly file for how to do this. Let's see an example.

It's pretty easy to configure services you've installed to write anywhere you want. However, there are already many services installed in the Usher filesystem which we don't need to install ourselves. Let's see an example of using the Apache web server already installed on our VMs. We'll start by seeing what happens when we try to run it.

  [mmcnett@dud.mmcnett.usher.ucsdsys.net ~]
  $ sudo /etc/init.d/httpd start
  Starting httpd:                                            [  OK  ]
  [mmcnett@dud.mmcnett.usher.ucsdsys.net ~]
  $ 

OK, looks like it worked. Apparently, the requisite directories and files are writable. A simple check reveals that all is well and we can access the site. But we want a custom httpd.conf file and have it start at system startup. Though we could get away with simply doing a mount bind on /etc/httpd/conf/httpd.conf, we may want more flexibility (like server ssl keys, etc). Let's go ahead and write a script to setup the environment we want and use that to start Apache for us at boot time. Here's the script:

#!/bin/sh
#
# Marvin McNett 2006

# We probably don't want to lose our log file on server restart or have it fill
# our ramfs, so let's make a log mount point if it's not there.
if [ ! -d /net/cluster/var/log/httpd ]; then
    mkdir -p /net/cluster/var/log/httpd
fi

# if we don't have an apache ServerRoot, copy it
if [ ! -d /net/cluster/etc/httpd ]; then
    rsync -aH /etc/httpd /net/cluster/etc
    # update these broken links
    ln -sf  /var/run /net/cluster/etc/httpd/run
    ln -sf /usr/lib/httpd/modules /net/cluster/etc/httpd/modules
fi

# now bind to the directories if we haven't already
if [ $1 = 'start' -o $1 = 'restart' ]; then
    if ! (df | grep /var/log/httpd > /dev/null); then
        mount -n --bind /net/cluster/var/log/httpd /var/log/httpd
    fi
    if ! (df | grep /etc/httpd > /dev/null); then
        mount -n --bind /net/cluster/etc/httpd /etc/httpd
    fi
fi

# now start it in the usual way
/etc/init.d/httpd $1

Granted, there are many other ways (probably better) of doing this, but this works. Now, all we need to do is save this script to /net/cluster/etc/rc.d/init.d/httpd and create a link to it in /net/cluster/etc/rc.d/rc3.d.

  cd /net/cluster/etc/rc.d/rc3.d
  ln -s ../init.d/httpd S85httpd

Now, all VMs in your cluster will startup with Apache running. As explained, if you only want one machine in your cluster to startup running Apache, put this in /net/local rather than /net/cluster.

If you're still having problems starting up a service, let me know.

Custom Kernels

Currently, this is low on the priority list, but that could change if there's a demand. Know that, until Usher is given HVM nodes to manage, you'll have to compile your kernel for the current version of Xen being used on the dom0 VMs. As of this writing, that is Xen 3.0.4-1. If you want to run a non-Xen kernel, let your advisor and Geoff know that Usher needs some HVM nodes.

If after reading the above, you still want/need this, send mail to Marvin McNett. No need to type anything in the body.

UCSD Usher Installation Details

Usher has been tested and run under the following setup here at UCSD (since 2007-01-17).

Usher Controller and Servers

On a single Dell PowerEdge 1750, CentOS 4.5 server (so RHEL 4.5 should work w/o modification):

• NFS server serving up read-only nfsroot VM filesystems
• LDAP server
• SQL server
• Usher Controller

The NFS server serves up read-only nfsroot filesystems to both privileged and unprivileged domains (in Xen). This facilitates adding physical nodes to the system and is really convenient for starting and stopping large numbers of VMs. The ro filesystems are all based on CentOS 4.4, but any Linux distro should work. Ramdisks are used for files and directories which need to be writable by Linux. There are a few ways to do this², but one of the easiest is to simply use UnionFS once to see what files and directories get created, then create and mount (see the '-bind' option to mount) those in the ramdisk at boot time.

Attempts were made to use either LVM snapshots or UnionFS over NFS as opposed to the read-only NFS with ramdisks. LVM snapshots served over NFS seemed to work for a few VMs, but maxed out somewhere between 30 and 40 (it's been a while so I don't recall the exact number). In addition, even with only a few VMs (~10), I would begin seeing filesystem corruption after only a few hours and eventually the snapshot would become unavailable to the LVM.

Using UnionFS, I attempted to use a single filesystem image onto which I would stack a rw filesystem for each VM (using UnionFS), then export the stacked filesystems to the VMs for NFS-root mounting. This seemed more promising, running several hours without incident. Unfortunately, the UnionFS module would kernel panic my NFS server at somewhere around 20 filesystems stacked atop a single read only filesystem.

The above attempts were made in May 2006, so perhaps the bugs have since been fixed. If anyone has more success with either of these methods, or an alternative, please let me know.

Local Node Managers

On several Dell PowerEdge (many models) servers:

• Xen 3.0.4-1
• Usher Local Node Manager running in Domain-0

As mentioned, the LNMs also run NFS-root. This facilitates easily adding and removing nodes as they become available or are needed for other projects. It also, eliminates the need to destroy disk contents. This way, if the machine is simply on loan from another user, we can easily return it to her just as she left it with us.

Clients

Not much to say here. The client library and shell (ush) can be run from anywhere, as long as it has network connectivity to the controller.

FAQ

Frequently asked questions for both Usher and virtual machines created by Usher.

Usher

1. Why does my client sometimes disconnect and reconnect?

UCSD Usher Virtual Machines (as setup at UCSD)

2. What are all those mounted directories when I run 'df'?
3. Why is `/tmp` so small?
4. Why can't I ssh to my VM from home?
5. My VM is running out of memory. How can I add swap space?

Answers

1. Why does my client sometimes disconnect and reconnect?
   Your client is setup to automatically try reconnecting to the Controller upon an unexpected disconnect with exponential backoff. Apart from an actual network problem, a Controller restart will also disconnect all clients. Since it's still early in the game and updates to the Controller code here at UCSD are still rather common, this may happen a bit more often. In time, this will be less frequent as the Controller code becomes more stable. Know that you can also try manually reconnecting with your same credentials using the 'reconnect' command.
2. What are all those mounted directories when I run 'df'?
   Those are files and directories which have been mounted on the VM's ramdisk using 'mount --bind' to make them writable.
3. Why is /tmp so small
   As mentioned in the ramdisk section /tmp is one of the directories which must be mounted read-write. The size of the ramdisk in which /tmp is mounted is 64MB. If this is too small for your needs, try the solution mentioned there in the ramdisk section.
4. Why can't I ssh to my VM from home?
   By default, our VMs are setup to start with an internal UCSD address (i.e. a 172.xxx.xxx.xxx address). So, you'll need to first connect to a UCSD machine, then ssh from there. The machines are, however, capable of connecting to off campus machines (via NAT). Globally routable ip addresses are available for a small fee.
5. My VM is running out of memory. How can I add swap space?
   Since Usher VMs don't use local disk, you'll have to mount a swap file over NFS. To do this, first create a file for your VM's swap space in its /net/local directory with:

       sudo dd if=/dev/zero of=/net/local/swapfile bs=1k count=262144
   

   for a 256MB swap space (Please be conservative as doing this often can quickly eat up space on our NFS server. Also, consider deleting your swap files when shutting your VM down for long periods of time.).
   

Next, mount the swap file on a loopback device (/dev/loop0 in this case) with:

    sudo losetup /dev/loop0 /net/local/swapfile

make it a swapfile

    sudo mkswap /dev/loop0

then mount it

    sudo swapon /dev/loop0

At this point, you should now have a 256MB swap space being served over NFS. Keep in mind that you can easily set this up in a statup script for your VM (and have it torn down on shutdown), using the method discussed in the Running Services section.

Last Updated: 2008-01-24 by mmcnett
Report problems to: Marvin McNett