INTRODUCTION
============
-Autocluster is set of scripts for building virtual clusters to test
-clustered Samba. It uses Linux's libvirt and KVM virtualisation
-engine.
+Autocluster is a script for building virtual clusters to test
+clustered Samba.
-Autocluster is a collection of scripts, template and configuration
-files that allow you to create a cluster of virtual nodes very
-quickly. You can create a cluster from scratch in less than 30
-minutes. Once you have a base image you can then recreate a cluster
-or create new virtual clusters in minutes.
-
-The current implementation creates virtual clusters of RHEL5 nodes.
+It uses Vagrant (with the libvirt plugin) and Ansible to build and
+configure a cluster.
+This software is freely distributable under the GNU public license, a
+copy of which you should have received with this software (in a file
+called COPYING).
CONTENTS
========
+* SUPPORTED PLATFORMS
+
* INSTALLING AUTOCLUSTER
* HOST MACHINE SETUP
* CREATING A CLUSTER
-* BOOTING A CLUSTER
+* DESTROYING A CLUSTER
-* POST-CREATION SETUP
+* DEVELOPMENT HINTS
-* CONFIGURATION
-* DEVELOPMENT HINTS
+SUPPORTED_PLATFORMS
+===================
+
+Tested host platforms:
+
+* CentOS 7
+Tested guest platforms:
+
+* CentOS 7
+
+Tested cluster filesystems:
+
+* GPFS
INSTALLING AUTOCLUSTER
======================
Before you start, make sure you have the latest version of
autocluster. To download autocluster do this:
- git clone git://git.samba.org/tridge/autocluster.git autocluster
-
-Or to update it, run "git pull" in the autocluster directory
+ git clone git://git.samba.org/autocluster.git
You probably want to add the directory where autocluster is installed
to your PATH, otherwise things may quickly become tedious.
+Packages can also be built and installed.
+
HOST MACHINE SETUP
==================
-This section explains how to setup a host machine to run virtual
-clusters generated by autocluster.
-
-
- 1) Install kvm, libvirt, qemu-nbd and nbd-client.
-
- Autocluster creates virtual machines that use libvirt to run under
- KVM. This means that you will need to install both KVM and
- libvirt on your host machine. You will also need the qemu-nbd and
- nbd-client programs, which autocluster uses to loopback-nbd-mount
- the disk images when configuring each node.
-
- For various distros:
-
- * RHEL/CentOS
-
- For RHEL5/CentOS5, useful packages for both kvm and libvirt can
- be found here:
-
- http://www.lfarkas.org/linux/packages/centos/5/x86_64/
-
- You will need to install a matching kmod-kvm package to get the
- kernel module.
-
- RHEL5.4 ships with KVM but it doesn't have the SCSI disk
- emulation that autocluster uses by default. There are also
- problems when autocluster uses virtio on RHEL5.4's KVM. You
- should use a version from lfarkas.org instead. Hopefully this
- will change!
-
- qemu-nbd is in the kvm package.
-
- Unless you can find an RPM for nbd-client then you need to
- download source from:
-
- http://sourceforge.net/projects/nbd/
-
- and build it.
-
- * Fedora Core
-
- Useful packages ship with Fedora Core 10 (Cambridge) and later.
-
- qemu-nbd is in the kvm package.
+1. Install Ansible
- nbd-client is in the nbd package.
+2. Run: autocluster host <platform> setup
- * Ubuntu
+ Currently the only supported <platform> is "centos7"
- Useful packages ship with Ubuntu 8.10 (Intrepid Ibex) and later.
+ This will
- qemu-nbd is in the kvm package but is called kvm-nbd, so you
- need to set the QEMU_NBD configuration variable.
+ * Install and configure several packages, including Vagrant
- nbd-client is in the nbd-client package.
+ * Assume you want to serve repositories to guests from /home/mediasets/.
- For other distributions you'll have to backport distro sources or
- compile from upstream source as described below.
+ * Create a libvirt storage pool at /virtual/autocluster/ for VM
+ images/files.
- * For KVM see the "Downloads" and "Code" sections at:
+ * Create an SSH key for autocluster
- http://www.linux-kvm.org/
-
- * For libvirt see:
-
- http://libvirt.org/
-
- * As mentioned about, nbd can be found at:
-
- http://sourceforge.net/projects/nbd/
-
- You will need to add the autocluster directory to your PATH.
-
- You will need to configure the right kvm networking setup. The
- files in host_setup/etc/libvirt/qemu/networks/ should help. This
- command will install the right networks for kvm:
-
- rsync -av --delete host_setup/etc/libvirt/qemu/networks/ /etc/libvirt/qemu/networks/
-
- After this you might need to reload libvirt:
-
- /etc/init.d/libvirt reload
-
- or similar.
-
- You might also need to set:
-
- VIRSH_DEFAULT_CONNECT_URI=qemu:///system
-
- in your environment so that virsh does KVM/QEMU things by default.
-
- 2) You need a caching web proxy on your local network. If you don't
- have one, then install a squid proxy on your host. See
- host_setup/etc/squid/squid.conf for a sample config suitable for a
- virtual cluster. Make sure it caches large objects and has plenty
- of space. This will be needed to make downloading all the RPMs to
- each client sane
-
- To test your squid setup, run a command like this:
-
- http_proxy=http://10.0.0.1:3128/ wget <some-url>
-
- 3) Setup a DNS server on your host. See host_setup/etc/bind/ for a
- sample config that is suitable. It needs to redirect DNS queries
- for your virtual domain to your windows domain controller
-
- 4) Download a RHEL install ISO.
+ For speed, you may wish to mirror the guest distribution somewhere
+ under /home/mediasets/ or on another nearby machine.
+Depending on how your host machine is setup, you may need to run
+autocluster commands as root.
CREATING A CLUSTER
==================
-A cluster comprises a single base disk image, a copy-on-write disk
-image for each node and some XML files that tell libvirt about each
-node's virtual hardware configuration. The copy-on-write disk images
-save a lot of disk space on the host machine because they each use the
-base disk image - without them the disk image for each cluster node
-would need to contain the entire RHEL install.
-
-The cluster creation process can be broken down into 2 mains steps:
+Configuration file
+------------------
- 1) Creating the base disk image.
+The configuration file is a YAML file. If your cluster is to be
+called "foo" then the configuration file must be "foo.yml" in the
+current directory.
- 2) Create the per-node disk images and corresponding XML files.
+To see what options to set, try this:
-However, before you do this you will need to create a configuration
-file. See the "CONFIGURATION" section below for more details.
+ # autocluster cluster foo defaults
-Here are more details on the "create cluster" process. Note that
-unless you have done something extra special then you'll need to run
-all of this as root.
+This will show default the default configuration. This is the only
+cluster command that doesn't need a cluster configuration.
- 1) Create the base disk image using:
+It may also be worth looking at the file defaults.yml, which
+includes some useful comments.
- ./autocluster create base
+Add updated settings foo.yml. Try to set the minimum number of
+options to keep the configuration file small. See example.yml.
- The first thing this step does is to check that it can connect to
- the YUM server. If this fails make sure that there are no
- firewalls blocking your access to the server.
+Most items are fairly obvious. However, here are some details:
- The install will take about 10 to 15 minutes and you will see the
- packages installing in your terminal
+* networks
- The installation process uses kickstart. If your configuration
- uses a SoFS release then the last stage of the kickstart
- configuration will be a postinstall script that installs and
- configures packages related to SoFS. The choice of postinstall
- script is set using the POSTINSTALL_TEMPLATE variable, allowing you
- to adapt the installation process for different types of clusters.
+ Default: 10.0.0.0/24 10.0.1.0/24 10.0.2.0/24
- It makes sense to install packages that will be common to all
- nodes into the base image. This save time later when you're
- setting up the cluster nodes. However, you don't have to do this
- - you can set POSTINSTALL_TEMPLATE to "" instead - but then you
- will lose the quick cluster creation/setup that is a major feature
- of autocluster.
+ There should be at least 2 networks. The first network is a
+ private network, while the others can be used for CTDB public IP
+ addresses.
- When that has finished you should mark that base image immutable
- like this:
+* firstip
- chattr +i /virtual/ac-base.img
+ Default: 20
- That will ensure it won't change. This is a precaution as the
- image will be used as a basis file for the per-node images, and if
- it changes your cluster will become corrupt
+ This is the final octet of the first IP address used on each network.
- 2) Now run "autocluster create cluster" specifying a cluster
- name. For example:
+* node_list
- autocluster create cluster c1
+ Default: [nas, nas, nas, nas, test]
- This will create and install the XML node descriptions and the
- disk images for your cluster nodes, and any other nodes you have
- configured. Each disk image is initially created as an "empty"
- copy-on-write image, which is linked to the base image. Those
- images are then loopback-nbd-mounted and populated with system
- configuration files and other potentially useful things (such as
- scripts).
+ Each node is offset from firstip by its position in the list.
+ The above default will result in 5 nodes.
-BOOTING A CLUSTER
-=================
+ - The first 4 will be Clustered Samba NAS nodes (running CTDB,
+ Samba, NFS) with addresses on the first network from 10.0.0.20
+ to 10.0.0.23 (with similar static addresses on the other
+ networks).
-At this point the cluster has been created but isn't yet running.
-Autocluster provides a command called "vircmd", which is a thin
-wrapper around libvirt's virsh command. vircmd takes a cluster name
-instead of a node/domain name and runs the requested command on all
-nodes in the cluster.
+ - The 5th node will be a minimally installed/configured test node
+ that can be used as a CTDB test client, with address 10.0.0.24.
- 1) Now boot your cluster nodes like this:
+ Valid node types are:
- vircmd start c1
+ nas: Clustered Samba node with cluster filesystem, smbd, nfsd
+ ad: Samba Active Directory Domain Controller node
+ base: Base operaing system node
+ build: Build node for CTDB packages
+ cbuild: Build node for Samba, with cluster filesystem installed
+ storage: Cluster filesystem node that doesn't directly provide NAS services
+ test: CTDB test node, with CTDB packages installed
- The most useful vircmd commands are:
-
- start : boot a node
- shutdown : graceful shutdown of a node
- destroy : power off a node immediately
-
- 2) You can watch boot progress like this:
-
- tail -f /var/log/kvm/serial.c1*
-
- All the nodes have serial consoles, making it easier to capture
- kernel panic messages and watch the nodes via ssh
-
-
-POST-CREATION SETUP
-===================
+Cluster creation
+----------------
-Now you have a cluster of nodes, which might have a variety of
-packages installed and configured in a common way. Now that the
-cluster is up and running you might need to configure specialised
-subsystems like GPFS or Samba. You can do this by hand or use the
-sample scripts/configurations that are provided
+In theory this is easy:
- 1) Now you can ssh into your nodes. You may like to look at the
- small set of scripts in /root/scripts on the nodes for
- some scripts. In particular:
+ # autocluster cluster foo build
- mknsd.sh : sets up the local shared disks as GPFS NSDs
- setup_gpfs.sh : sets up GPFS, creates a filesystem etc
- setup_samba.sh : sets up Samba and many other system compoents
- setup_tsm_server.sh: run this on the TSM node to setup the TSM server
- setup_tsm_client.sh: run this on the GPFS nodes to setup HSM
+This runs several internal steps:
- To setup a SoFS system you will normally need to run
- setup_gpfs.sh and setup_samba.sh.
+1. `destroy` - Destroy any existing cluster of the same name
+2. `generate` - Generate metadata (for Vagrant, Ansible, SSH) from the
+ configuration
+3. `create` - Create the cluster nodes (using Vagrant)
+4. `ssh_config` - Configure SSH to allow direct access to nodes as root
+5. `setup` - Setup each node according to its type (using Ansible)
- 2) If using the SoFS GUI, then you may want to lower the memory it
- uses so that it fits easily on the first node. Just edit this
- file on the first node:
+DESTROYING A CLUSTER
+====================
- /opt/IBM/sofs/conf/overrides/sofs.javaopt
-
- 3) For automating the SoFS GUI, you may wish to install the iMacros
- extension to firefox, and look at some sample macros I have put
- in the imacros/ directory of autocluster. They will need editing
- for your environment, but they should give you some hints on how
- to automate the final GUI stage of the installation of a SoFS
- cluster.
-
-
-CONFIGURATION
-=============
-
-Basics
-======
-
-Autocluster uses configuration files containing Unix shell style
-variables. For example,
-
- FIRSTIP=30
-
-indicates that the last octet of the first IP address in the cluster
-will be 30. If an option contains multiple words then they will be
-separated by underscores ('_'), as in:
-
- ISO_DIR=/data/ISOs
-
-All options have an equivalent command-line option, such
-as:
-
- --firstip=30
-
-Command-line options are lowercase. Words are separated by dashes
-('-'), as in:
-
- --iso-dir=/data/ISOs
-
-Normally you would use a configuration file with variables so that you
-can repeat steps easily. The command-line equivalents are useful for
-trying things out without resorting to an editor. You can specify a
-configuration file to use on the autocluster command-line using the -c
-option. For example:
-
- autocluster -c config-foo create base
-
-If you don't provide a configuration variable then autocluster will
-look for a file called "config" in the current directory.
-
-You can also use environment variables to override the default values
-of configuration variables. However, both command-line options and
-configuration file entries will override environment variables.
-
-Potentially useful information:
-
-* Use "autocluster --help" to list all available command-line options
- - all the items listed under "configuration options:" are the
- equivalents of the settings for config files. This output also
- shows descriptions of the options.
-
-* You can use the --dump option to check the current value of
- configuration variables. This is most useful when used in
- combination with grep:
-
- autocluster --dump | grep ISO_DIR
-
- In the past we recommended using --dump to create initial
- configuration file. Don't do this - it is a bad idea! There are a
- lot of options and you'll create a huge file that you don't
- understand and can't debug!
-
-* Configuration options are defined in config.d/*.defconf. You
- shouldn't need to look in these files... but sometimes they contain
- comments about options that are too long to fit into help strings.
-
-Keep it simple
-==============
-
-* I recommend that you aim for the smallest possible configuration file.
- Perhaps start with:
-
- FIRSTIP=<whatever>
-
- and move on from there.
-
-* Use the --with-release option on the command-line or the
- with_release function in a configuration file to get default values
- for building virtual clusters for releases of particular "products".
- Currently there are only release definitions for SoFS.
-
- For example, you can setup default values for SoFS-1.5.3 by running:
-
- autocluster --with-release=SoFS-1.5.3 ...
-
- Equivalently you can use the following syntax in a configuration
- file:
-
- with_release "SoFS-1.5.3"
-
- So the smallest possible config file would have something like this
- as the first line and would then set FIRSTIP:
-
- with_release "SoFS-1.5.3"
-
- FIRSTIP=<whatever>
-
- Add other options as you need them.
-
- The release definitions are stored in releases/*.release. The
- available releases are listed in the output of "autocluster --help".
-
- NOTE: Occasionally you will need to consider the position of
- with_release in your configuration. If you want to override options
- handled by a release definition then you will obviously need to set
- them later in your configuration. This will be the case for most
- options you will want to set. However, some options will need to
- appear before with_release so that they can be used within a release
- definition - the most obvious one is the (rarely used) RHEL_ARCH
- option, which is used in the default ISO setting for each release.
- If things don't work as expected use --dump to confirm that
- configuration variables have the values that you expect.
-
-* The NODES configuration variable controls the types of nodes that
- are created. At the time of writing, the default value is:
-
- NODES="rhel_base:0-3"
-
- This means that you get 4 nodes, at IP offsets 0, 1, 2, & 3 from
- FIRSTIP, all part of the CTDB cluster. That is, with standard
- settings and FIRSTIP=35, 4 nodes will be created in the IP range
- 10.0.0.35 to 10.0.0.38.
-
- The SoFS releases use a default of:
-
- NODES="tsm_server:0 sofs_gui:1 sofs_front:2-4"
-
- which should produce a set of nodes the same as the old SoFS
- default. You can add extra rhel_base nodes if you need them for
- test clients or some other purpose:
-
- NODES="$NODES rhel_base:7,8"
-
- This produces an additional 2 base RHEL nodes at IP offsets 7 & 8
- from FIRSTIP. Since sofs_* nodes are present, these base nodes will
- not be part of the CTDB cluster - they're just extra.
-
- For many standard use cases the nodes specified by NODES can be
- modified by setting NUMNODES, WITH_SOFS_GUI and WITH_TSM_NODE.
- However, these options can't be used to create nodes without
- specifying IP offsets - except WITH_TSM_NODE, which checks to see if
- IP offset 0 is vacant. Therefore, for many uses you can ignore the
- NODES variable.
-
- However, NODES is the recommended mechanism for specifying the nodes
- that you want in your cluster. It is powerful, easy to read and
- centralises the information in a single line of your configuration
- file.
+ # autocluster cluster foo destroy
DEVELOPMENT HINTS
=================
-The -e option provides support for executing arbitrary bash code.
-This is useful for testing and debugging.
-
-One good use of this option is to test template substitution using the
-function substitute_vars(). For example:
-
- ./autocluster --with-release=SoFS-1.5.3 -e 'CLUSTER=foo; DISK=foo.qcow2; UUID=abcdef; NAME=foon1; set_macaddrs; substitute_vars templates/node.xml'
-
-This prints templates/node.xml with all appropriate substitutions
-done. Some internal variables (e.g. CLUSTER, DISK, UUID, NAME) are
-given fairly arbitrary values but the various MAC address strings are
-set using the function set_macaddrs().
-
-The -e option is also useful when writing scripts that use
-autocluster. Given the complexities of the configuration system you
-probably don't want to parse configuration files yourself to determine
-the current settings. Instead, you can ask autocluster to tell you
-useful pieces of information. For example, say you want to script
-creating a base disk image and you want to ensure the image is
-marked immutable:
-
- base_image=$(autocluster -c $CONFIG -e 'echo $VIRTBASE/$BASENAME.img')
- chattr -V -i "$base_image"
+The Ansible playbook for nodes has been structured in a way that
+should make it easy to add new platforms and cluster filesystems. Try
+to follow the pattern and keep task names as generic as possible.
- if autocluster -c $CONFIG create base ; then
- chattr -V +i "$base_image"
- ...
+To see facts about <node>:
-Note that the command that autocluster should run is enclosed in
-single quotes. This means that $VIRTBASE and $BASENAME will be expand
-within autocluster after the configuration file has been loaded.
+ ansible -i <node>, all -m setup
git.samba.org / tridge / autocluster.git / blobdiff