|Deletions are marked like this.||Additions are marked like this.|
|Line 31:||Line 31:|
|1. Launch a new instance of the machine image created in Step 1 using Cloud Launch (either the one available at https://launch.usegalaxy.org/ or a privately deployed one). When !CloudMan comes up, choose ''Test cluster'' as the cluster type;||1. Launch a new instance of the machine image created in Step 1 using Cloud Launch (either the one available at https://launch.usegalaxy.org/ or one you have installed yourself from https://github.com/galaxyproject/cloudlaunch). When !CloudMan comes up, choose ''Test cluster'' as the cluster type;|
Hey - this page is under construction! Feel free to contribute and help in getting it wrapped up or bug someone on the CloudMan Team.
Launching a default version of CloudMan and Galaxy on the Cloud is a pretty straightforward process. The underlying system, however, is more complex and consists of a number of components. This page describes the steps required to build your own version of the components and deploy the system. You may want to do this if you are deploying the system on your own Cloud. If you would just like to have a custom version of the existing system on AWS, perhaps this can help?
The process of building your own instance of the system is time consuming (although we are continuing to simplify and streamline this) and will require some technical skills and understanding of the process. Before starting this endeavor, it is highly recommended that you read the following papers:
"Galaxy CloudMan: delivering cloud compute clusters" - which gives you an overview of the ideology behind what's being done here; and
"Building and Provisioning Bioinformatics Environments on Public and Private Clouds" - which gives many of the technical details of the overall build process
(optional) "A reference model for deploying applications in virtualized environments" - which gives you the technical background why things are being done they way they are being done.
The process of building the system consists of a number of steps. Each step creates a component that, when joined together, compose the complete system. To have a functional Galaxy on the Cloud system, it is required to have all the components in place; CloudMan alone requires the machine image (step 1) and an S3 bucket (or a Swift container) (step 4). Ideally, instances are launched via a launcher application, such as the one available at https://launch.usegalaxy.org/launch. The process of building the components has been automated via an Ansible playbook: galaxy-cloudman-playbook.
Step 1: the Machine Image
The machine image, often called the AMI (for Amazon Machine Image, although other cloud middlewares use the same term), represents the base operating system required to run the system; it contains the required system level applications and libraries as well as hooks for starting CloudMan.
To build your machine image:
From your cloud's console (or via the API), launch an instance of the base machine image - we'll call this the builder instance. When launching the builder instance, make sure to select the launch options that will permit ssh access to the instance (via Security Groups). This can be automatically set on your behalf if you use a launcher app, such as the CloudLaunch (just specify a custom AMI under the Advanced startup options). When launching the instance also provide the following value for user data: no_start: true (this will prevent CloudMan from automatically starting if the builder instance reboots during the build process). The image building process has been developed using Ubuntu 14.04 so any machine image based on Ubuntu 14.04 should be OK to use (on AWS, we currently use ami-98aa1cf0 from Canonical);
Once the instance has launched and you have the instance IP address, follow the playbook instructions to run the image building role;
After the playbook has run to completion, via your cloud's console, create a new machine image (or image snapshot, in OpenStack's parlance) from the builder instance.
Step 2: the galaxyFS
The Galaxy File System (galaxyFS) contains the Galaxy application, the PostgreSQL database, installed Galaxy tools, and the accompanying configurations. The aim here is to create a snapshot of the galaxyFS that can be replicated when instances of the system are launched while permitting the changes to the file system (e.g., user-uploaded data, analysis results). This replication is realized by CloudMan either by creating a volume from a volume snapshot or by downloading a tarball of the file system. The process of building galaxyFS snapshot is the following:
Launch a new instance of the machine image created in Step 1 using Cloud Launch (either the one available at https://launch.usegalaxy.org/ or one you have installed yourself from https://github.com/galaxyproject/cloudlaunch). When CloudMan comes up, choose Test cluster as the cluster type;
From CloudMan's Admin page, add a new volume-based file system called galaxy of desired size (default 10GB);
Follow the instructions from the CloudMan playbook to build the galaxyFS; note that this playbook has an option to automatically install Galaxy tools and genome reference data;
Create a snapshot of the file system: stop any services that might be running (e.g., Galaxy, PostgreSQL) and create a volume snapshot from the CloudMan's Admin page.
Step 3: the galaxyIndicesFS
Required step but no docs exist here yet...
Step 4: Hooking it all up
Once you have above components built, it is necessary to connect them. This is done by creating a publicly accessible S3 bucket/Swift container (e.g., my_cloudman) and placing a YAML specification file along with a copy of CloudMan's source packaged in a .tar.gz:
- Create a bucket/container, whether it be is on S3 or Swift
Place a copy of CloudMan's source code in the bucket. Call it cm.tar.gz. This can be your own modified version of CloudMan's source downloaded/forked from the source code repository or the default one, available from http://s3.amazonaws.com/cloudman/cm.tar.gz
Place a copy of CloudMan's boot script in the bucket. Call it cm_boot.py. This can be your own modified version of the script downloaded from the source code repository or the default one, available from http://s3.amazonaws.com/cloudman/cm_boot.py
Create a file called snaps.yaml and also place it in the bucket. See the main snaps.yaml for an example of what the structure of this file must look like. Just make sure to provide the snapshot ID's for the snapshots you built in the previous steps. Note that if you would like for others to be able to use your version of the system, it is necessary to make the provided snapshots public.
Finally, provide access a launcher application by installing and configuring Cloud Launch application from https://github.com/galaxyproject/cloudlaunch
Over time, the community has developed a few more documents and resources that can help with setting up the system and these are aggregated here: