Welcome to elasticluster’s documentation!

Introduction

ElastiCluster aims to provide a user-friendly command line tool to create, manage and setup computing clusters hosted on cloud infrastructures (like Amazon’s Elastic Compute Cloud EC2, Google Compute Engine, or a private OpenStack cloud). Its main goal is to get a private cluster up and running with just a few commands; this video demoes ElastiCluster setting up a computational batch-queueing cluster.

Complete documentation for ElastiCluster is available on the Read The Docs website. General discussion over ElastiCluster’s usage, features, and bugs takes place on the elasticluster@googlegroups.com mailing-list (only subscribers can post).

The ElastiCluster project is an effort of the Services and Support for Science IT (S3IT) unit at the University of Zurich, licensed under the GNU General Public License version 3.

Features

ElastiCluster is in active development, and offers the following features at the moment:

• INI-style configuration file to define cluster templates
• Can start and manage multiple independent clusters at the same time * Automated setup of:
  • HPC clusters using SLURM or GridEngine;
  • Spark / Hadoop clusters with HDFS and Hive/SQL;
  • distributed storage clusters using GlusterFS, OrangeFS, or Ceph;
  • …or anything that you can install with an Ansible playbook!
• Growing and shrinking a running cluster.

ElastiCluster is currently in active development: please use the GitHub issue tracker to file enhancement requests and ideas, or the mailing-list for discussion.

We appreciate pull requests for new features and enhancements. Please use the master branch as starting point.

Overview

The architecture of elasticluster is quite simple: the configuration file ~/.elasticluster/config defines a set of cluster configurations and information on how to access a specific cloud service (including access id and secret keys).

Using the command line or a simple API, you can start a cluster (possibly overriding some of the default values, like the number of nodes you want to fire up) and configure it:

• ElastiCluster connects to the cloud provider indicated in the cluster configuration file, starts virtual machines, and waits until they are accessible via ssh.
• After all the VMs are up and running, ElastiCluster runs Ansible to configure the cluster.

Upon resize of the cluster [1], new virtual machines will be created and again Ansible will run on all the VMs, in order to properly add the new hosts to the cluster.

ElastiCluster commands export and import allow moving a running cluster’s definition and status data from one machine to the other, to allow controlling the same cluster from different places.

[1]	Currently, only growing a cluster is fully supported; shrinking a loaded cluster may remove nodes with running jobs and cause malfunctionings. See the remove-node command for a safer, albeit more low-level, way of shrinking clusters.

Table of Contents

• Installation
  • Quickstart
  • Prepare the environment for installation
  • Installing released code from PyPI
  • Installing development code from GitHub
  • Getting help
• Configuration
  • Basic syntax of the configuration file
  • Processing of configuration values
  • Cloud Section
  • Login Section
  • Setup Section
  • Cluster Section
  • Storage section
• Usage
  • The start command
  • The stop command
  • The list command
  • The list-nodes command
  • The list-templates command
  • The setup command
  • The resize command
  • The remove-node command
  • The ssh command
  • The sftp command
  • The export command
  • The import command
• Troubleshooting
  • Setup of a cluster fails and stops at task nfs-client: add to /etc/fstab
  • Downloading fails with error “Failed to validate the SSL certificate”
  • Running any elasticluster command fails with a version conflict about the requests package
  • Running any elasticluster command fails with a version conflict about the pbr package
  • Installation fails with ValueError: ('Expected version spec in' [...]
  • Installation fails complaining about version of setuptools
  • Installation fails with VersionConflict: ... Requirement.parse('setuptools>=17.1'))
  • Upgrading setuptools fails with ImportError: No module named extern
  • Installation fails with: “fatal error: ffi.h: No such file or directory”
  • Installation fails with: “fatal error: openssl/opensslv.h: No such file or directory”
  • Installation fails with: “fatal error: Python.h: No such file or directory”
  • Installation fails with: “unable to execute gcc: No such file or directory”
  • Installation fails with Too many levels of symbolic links
  • Setup of RHEL/CentOS 7 clusters fails immediately
  • Setup of Ubuntu 16.04 (“xenial”) clusters fails immediately
  • Issues when installing from source on MacOSX
  • Error “ImportError: No module named anyjson” on MacOSX
• Playbooks distributed with ElastiCluster
  • Setup variables
  • Compute clusters
  • Filesystems and storage
  • Add-on software
• Elasticluster programming API
  • elasticluster
  • elasticluster.cluster
  • elasticluster.conf
  • elasticluster.exceptions
  • elasticluster.gc3pie_config
  • elasticluster.__main__
  • elasticluster.providers
  • elasticluster.providers.ansible_provider
  • elasticluster.providers.azure_provider
  • elasticluster.providers.ec2_boto
  • elasticluster.providers.gce
  • elasticluster.providers.openstack
  • elasticluster.repository
  • elasticluster.subcommands
  • elasticluster.utils

Indices and tables

• Index
• Module Index
• Search Page