elasticluster.providers.gce

Cloud provider for the Google Compute Engine.

See <https://code.google.com/p/google-cloud-platform-samples/source/browse/python-client-library-example/gce.py?repo=compute> for reference.

elasticluster.providers.gce.GCE_SCOPE = 'https://www.googleapis.com/auth/compute'

the OAuth scope for the GCE web API

class elasticluster.providers.gce.GoogleCloudProvider(gce_project_id, gce_client_id='', gce_client_secret='', email='default', network='default', noauth_local_webserver=False, zone='us-central1-a', storage_path=None)[source]

Cloud provider for the Google Compute Engine.

Parameters:
  • gce_client_id (str) – Client ID to use in OAuth authentication.
  • gce_client_secret (str) – Client secret (password) to use in OAuth authentication.
  • gce_project_id (str) – Project name to log in to GCE.
  • noauth_local_webserver – do not pop up a webserver for auth
  • zone (str) – gce zone, default is us-central1-a
  • network (str) – network to use, default is default
  • email (str) – service email to use, default is default
  • storage_path (str) – path to store authentication data (oauth.dat file). If no path is specified, the login data has to be entered after every request.
IMAGE_NAME_SHORTCUTS = {'backports-debian': 'debian-cloud', 'centos': 'centos-cloud', 'container-vm': 'google-containers', 'coreos': 'coreos-cloud', 'cos': 'cos-cloud', 'debian': 'debian-cloud', 'rhel': 'rhel-cloud', 'sles': 'suse-cloud', 'ubuntu': 'ubuntu-os-cloud'}

Map image names to projects, based on prefix.

The image names and full resource URLs for several Google- provided images (debian, centos, etc.) follow a consistent pattern, and so ElastiCluster supports a short-hand of just an image name, such as:

"debian-7-wheezy-v20150526

The cloud project in this case is then debian-cloud.

get_ips(instance_id)[source]

Retrieves the ip addresses (public) from the cloud provider by the given instance id.

Parameters:instance_id (str) – id of the instance
Returns:list (ips)
Raises:InstanceError if the ip could not be retrieved.
is_instance_running(instance_id)[source]

Check whether the instance is up and running.

Parameters:instance_id (str) – instance identifier
Reutrn:True if instance is running, False otherwise
list_instances(filter=None)[source]

List instances on GCE, optionally filtering the results.

Parameters:filter (str) – Filter specification; see https://developers.google.com/compute/docs/reference/latest/instances/list for details.
Returns:list of instances
pause_instance(instance_id)[source]

Pauses the instance, retaining disk and config.

Parameters:instance_id (str) – instance identifier
Raises:InstanceError if instance cannot be paused
Returns:dict - information needed to restart instance.
resume_instance(paused_info)[source]

Restarts a paused instance, retaining disk and config.

Parameters:instance_id (str) – instance identifier
Raises:InstanceError if instance cannot be resumed.
Returns:dict - information needed to restart instance.
start_instance(key_name, public_key_path, private_key_path, security_group, flavor, image_id, image_userdata, cluster_name, username=None, node_name=None, boot_disk_type='pd-standard', boot_disk_size=10, tags=None, scheduling=None, accelerator_count=0, accelerator_type='default', allow_project_ssh_keys=True, local_ssd_count=0, local_ssd_interface='SCSI', min_cpu_platform=None, **kwargs)[source]

Starts a new instance with the given properties and returns the instance id.

Parameters:
  • key_name (str) – name of the ssh key to connect
  • public_key_path (str) – path to ssh public key
  • private_key_path (str) – path to ssh private key
  • security_group (str) – firewall rule definition to apply on the instance
  • flavor (str) – machine type to use for the instance
  • image_id (str) – image type (os) to use for the instance
  • image_userdata (str) – command to execute after startup
  • username (str) – username for the given ssh key, default None
  • node_name (str) – name of the instance
  • tags (str|Sequence) – “Tags” to label the instance. Can be either a single string (individual tags are comma-separated), or a sequence of strings (each string being a single tag).
  • scheduling (str) – scheduling option to use for the instance (“preemptible”)
  • accelerator_count (int) – Number of accelerators (e.g., GPUs) to make available in instance
  • accelerator_type (str) –

    Type of accelerator to request. Can be one of:

    • Full URL specifying an accelerator type valid for the zone and project VMs are being created in. For example, https://www.googleapis.com/compute/v1/projects/[PROJECT_ID]/zones/[ZONE]/acceleratorTypes/[ACCELERATOR_TYPE]
    • An accelerator type name (any string which is not a valid URL). This is internally prefixed with the string https://www.googleapis.com/compute/v1/projects/[PROJECT_ID]/zones/[ZONE]/acceleratorTypes/ to form a full URL.
  • allow_project_ssh_keys (bool) – When True (default), SSH login is allowed to a node using any of the project-wide SSH keys (if they are defined). When False, only the SSH key specified by ElastiCluster config’s [login/*] section will be allowed to log in (instance-level key).
  • local_ssd_count (int) – Number of local SSD disks (each 375GB size) to make available in instance
  • local_ssd_interface (int) – Attachment interface for local SSD disks; either 'SCSI' (default) or 'NVME'.
  • min_cpu_platform (str) –

    require CPUs of this type or better (e.g., “Intel Skylake”)

    Only used if accelerator_count is > 0.

Returns:

str - instance id of the started instance

stop_instance(node)[source]

Destroy a VM.

Parameters:node (Node) – A Node instance
Raises:InstanceError if instance can not be stopped
to_vars_dict()[source]

Return local state which is relevant for the cluster setup process.