The University of California, San Diego
UCSD

Site Contents

Documentation

  1. Software Overview
  2. Download
  3. Installation
    1. Requirements
      1. Hardware
      2. Usher Controller
      3. Usher Local Node Managers
      4. Usher Clients
    2. Recommended
    3. Procedure
  4. Configuration Files
  5. Plugins
  6. Clients

Usher Documentation

Usher is a virtual cluster management system designed to substantially reduce the administrative burden of managing clusters of virtual machines while simultaneously improving the ability of users to request, control, and customize their virtual computing environment.

Architecturally, Usher is designed to impose few constraints upon the computing environment under its management. No two sites have identical hardware and software configurations, user and application requirements, or service infrastructures. To facilitate its use in a wide range of environments, Usher combines a core set of interfaces that implement basic mechanisms, clients for using these mechanisms, and a framework for expressing and customizing administrative policies in extensible modules, or plugins.

For a complete overview and description of Usher, please see:

Usher: An Extensible Framework for Managing Clusters of Virtual Machines, Marvin McNett, Diwaker Gupta, Amin Vahdat, and Geoffrey M. Voelker, 21st Large Installation System Administration Conference, (LISA 2007).

Finally, to see how Usher is being used at UCSD, check out the UCSD Documentation.

Software Overview

This section is a very brief overview of the Usher software. For a more in-depth discussion, check out the paper mentioned above.

As shown in Figure 1 below, the Usher system consists of three main components (in violet):

  • Centralized Controller
  • Local Node Manager (LNM)
  • Client API
Usher Architecture
Figure 1: Usher Components

The centralized controller provides the mechanism for taking requests from clients and issuing commands to the LNMs. It communicates with the LNMs to collect usage data and manage virtual machines running on each physical node. All communication between clients and the controller and LNMs and the controller is encrypted via SSL. The controller also provides an interface for users to register plugins to be notified of events of interest in a running system.

LNMs run as a servers on each node running a virtual machine manager in the Usher system. The LNMs communicate directly with virtual machine managers (or hypervisors) to perform tasks such as creation, deletion, and migration of virtual machines on behalf of the controller. The local node managers also collect usage data from the hypervisors which they periodically send to the controller where it is assimilated for use by clients and plugins.

Applications use the Usher client API to interact with the Usher controller. This API provides methods for requesting or manipulating VMs and performing state queries. The client API provides the mechanism for clients to securely authenticate and connect to the Usher controller. All methods are asynchronous, event-based calls to the controller.

Download

See the Downloads page at http://usher.ucsd.edu/wiki/UsherDownloads for information about getting Usher, Usher clients, and Usher plugins.

Installation

Below are Usher installation requirements and the procedure for installing a stock (i.e. no plugins) Usher system. A more detailed, site specific installation experience (with plugins) has been documented on the Experiences page.

Requirements

Before installing Usher, make sure your setup meets the following requirements.

Hardware

Physically, you'll need:

  • One or more networked servers capable of running a Virtual Machine Monitor (the more the merrier).

Usher Controller

The Usher controller must be running on a machine with:

Usher Local Node Managers

LNMs must be running on machines with:

Usher Clients

Clients utilizing the Usher Client API must have (in addition to the latest Usher release):

We recommend:

  • An NAS server to serve root filesystems to all unprivileged virtual machines in the system (needed for migration). It's very convenient to make this a read-only root filesystem. The steps I took to do this for CentOS 5 are here.

Procedure

The following procedure should get a basic Usher installation up and running on your infrastructure.

Currently, all three of the main Usher components are bundled in a single package. This has the benefit of making it easy to maintain, package, and distribute. The down side is that the installation script installs everything when you may only want to install one component. However, the entire distribution is only about 800K installed.1 An additional benefit is that it makes the installation instructions below brief.

There are three locations where Usher needs to be installed: the centralized controller, virtual machine manager nodes (i.e. machines where VMs will be run), and clients. If you haven't already, begin by installing the dependencies listed above for each component on the respective machines:

  • The centralized controller node needs the Usher Controller dependencies
  • Virtual Machine Manager nodes need the LNM dependencies
  • Client nodes need the Usher Clients dependencies

After installing the dependencies, extract the latest Usher package (obtained from the downloads page) and run the setup.py script in the distribution root directory. 

$ tar xvzf usher-0.3-latest.tgz
$ cd usher-0.3-revXXX
$ python setup.py install

This should install the Usher package as well as init scripts and sample configuration files. There are two init scripts, one for the controller (usherctrl) and one for the LNMs (usherlnm). These are placed in /etc/init.d. Link to one of these (usherctrl for the controller and usherlnm for the LNMs) in the appropriate directory to have them start a boot time. Otherwise, you can manually start, stop, and restart the services. If on a Red Hat or CentOS machine, you can also use the chkconfig program to register these to be started/stopped in the desired runlevels.

The sample configuration files are placed in /etc/usher. Remove the .orig extension from the appropriate file (ctrl.config for the controller, lnm.config for the LNMs) there and edit for your infrastructure. Information on editing configuration files can be found in the configuration files themselves.

After you're done editing these files for the controller and LNMs, you're ready to fire up the servers. As an example, manually starting the Usher controller can be done using (you may need to edit the init script for your filesystem if you have problems): 

$ /etc/init.d/usherctrl start

It's a good idea to start the controller before the LNMs the first time. Otherwise, the controller won't know how to notify the LNMs that it's running.

After starting the services and seeing the the LNMs have successfully connected to the controller, we're ready to connect to the controller with a client and start managing VMs. For this, you'll need at least one Usher client. A good client with which to start is called Ush (the Usher Shell). Information on getting, installing, and using Ush can be found here.

Finally, for extended functionality, the Usher system supports plugins which can be registered for events in the Usher system. If Usher doesn't do exactly what you want/need, check out the list of available plugins, or consider writing and contributing your own plugin(s) to the Usher community.

Configuration Files

Each component of the Usher system has its own configuration file. Here we discuss points relevant to all Usher configuration files. Component specific details are given in the corresponding sample configuration file bundled with the Usher distribution. All valid configuration options for each component are documented in their corresponding sample configuration file. Likewise, plugins and clients typically include a sample configuration file with all valid options documented there.

Here are the key points of writing Usher configuration files:

  • Each configuration item is specified as a (key, value) pair. The format is "key = value" (without the quotes).
  • Comments begin with '#'
  • Configuration options must be specified in a section. Sections are denoted using square brackets. For example, the controller's configuration file options are placed in a section titled '[ctrl]'. Below is a snippet: 
    [ctrl]
# SSL server key file.
ssl_key_file = /etc/usher/usherctrl_key.pem

# SSL server certificate file.
ssl_cert_file = /etc/usher/usherctrl_cert.pem
...
  • Subsections are denoted with additional square brackets. For example, '[[subsection]]' is a subsection of '[section]' below: 
    [section]

var1 = val1

[[subsection]]

var2 = val2
    Above, var1 is not defined in the subsection. However, as shown in the following point, it is available for interpolation.
  • string interpolation is performed using the ${<varname>} syntax. For example: 
    [section]
var1 = val1

[[subsection]]
# var2 = val1 as well
var2 = ${var1}
  • Sections titled '__allsecs__' will have their variables added to all subsections in their parent section of the same nesting level (unless a variable of the same name is also defined there).
  • All values are interpreted as strings and cast according to the corresponding component or plugin's configuration file specification. Usher's configuration file parser2 handles casting values to their correct type.
  • With the few (and rare) exceptions listed below, you DO NOT need to quote your values, even if they have spaces between them. For example: 
      # GOOD
  foo = bar is good
  # value will be the string "bar is good"

  # BAD
  foo = 'bar is bad'
  # value will be the string "'bar is bad'" (note the additional quotes)
    Exceptions:
    • muti-line options are supported using triple quotes only. In this case, the quotes are stripped.
    • If you must assign the literal string 'None' to a variable, it must be quoted. In this case, the quotes are stripped. Otherwise, None will be converted to NoneType.

Plugins

Plugin documentation can be found on the Plugins page.

Clients

Client documentation can be found on the Clients page.

[1] If you're that concerned about disk space, you may not have an infrastructure on which Usher should be run anyway.
[2] Usher extends ConfigObj to parse its config files. Refer to their site for additional details.

Last Updated: 2008-06-21 by mmcnett
Report problems to: Marvin McNett