In the Cloud

Continuing with the volume created in the post Sharing EBS Volumes Among Instances, in this post I show how to create a snapshot, create a new volume from that snapshot, and mount the new volume in an instance. Remember that the volume created in the previous post contained 1 file called “readme”.

Creating the Snapshot

First we look at the available volumes:

$ ec2-describe-volumes
VOLUME	vol-4001e429	1		us-east-1c	available	2008-09-25T09:51:48+0000

Then we make the snapshot:

$ ec2-create-snapshot vol-4001e429
SNAPSHOT	snap-cb7493a2	vol-4001e429	pending	2008-09-26T11:48:30+0000

It is “pending”. We check the status until it the snapshot creation has “completed”:

$ ec2-describe-snapshots snap-cb7493a2
SNAPSHOT	snap-cb7493a2	vol-4001e429	completed	2008-09-26T11:48:30+0000	100%

Creating a Volume from the Snapshot

Now that the snapshot is ready, we can create a new volume from it. Note that we create it in a different availability zone. The orginal volume resides in “us-east-1c”. The new volume will reside in “us-east-1a”.

$ ec2-create-volume --snapshot snap-cb7493a2 -z us-east-1a
VOLUME	vol-9f00e5f6	1	snap-cb7493a2	us-east-1a	creating	2008-09-26T11:52:37+0000

We wait until the volume is “available”:

$ ec2-describe-volumes
VOLUME	vol-9f00e5f6	1	snap-cb7493a2	us-east-1a	available	2008-09-26T11:52:37+0000
VOLUME	vol-4001e429	1		us-east-1c	available	2008-09-25T09:51:48+0000

Now we have two available volumes.

Mounting the New Volume in an Instance

Let’s launch an image so that we can verify that the newly created volume can be mounted and has the same contents as the original volume. Note that the instance is launched in the availability zone where the newly created volume resides.

$ ec2-run-instances ami-0757b26e -k gettingstarted-keypair -z us-east-1a

The AMI we use here is a public Ubuntu Desktop image.

$ ec2-describe-instances
RESERVATION	r-ff4d9e96	190912652296	default
INSTANCE	i-0fcf6c66	ami-0757b26e	ec2-67-202-35-79.compute-1.amazonaws.com	domU-12-31-38-00-6C-F6.compute-1.internal	running	gettingstarted-keypair	0		m1.small	2008-09-26T11:55:16+0000	us-east-1a	aki-a71cf9ce	ari-a51cf9cc

The instance is ready to be used. In another terminal we connect to the image and start the “user-setup” script. The GUI interaction to set up the user is not shown here.

$ ssh -i id_rsa-gettingstarted-keypair root@ec2-67-202-35-79.compute-1.amazonaws.com
The authenticity of host 'ec2-67-202-35-79.compute-1.amazonaws.com (67.202.35.79)' can't be established.
RSA key fingerprint is ab:df:4e:78:b7:4d:59:3e:ae:6c:81:32:80:eb:bd:78.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'ec2-67-202-35-79.compute-1.amazonaws.com,67.202.35.79' (RSA) to the list of known hosts.
Linux domU-12-31-38-00-6C-F6 2.6.21.7-2.fc8xen #1 SMP Fri Feb 15 12:39:36 EST 2008 i686

The programs included with the Ubuntu system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Ubuntu comes with ABSOLUTELY NO WARRANTY, to the extent permitted by
applicable law.

Amazon EC2 Ubuntu 7.10 gutsy base install AMI built by
Eric Hammond
For more information: http://ec2gutsy-desktop.notlong.com

root@domU-12-31-38-00-6C-F6:~# user-setup
Shadow passwords are now on.
Using `/usr/share/libgksu/debian/gconf-defaults.libgksu-sudo' to provide `libgksu-gconf-defaults'.

Time to attach the volume (in the orginal terminal):

$ ec2-attach-volume vol-9f00e5f6 -i i-0fcf6c66 -d /dev/sdh
ATTACHMENT	vol-9f00e5f6	i-0fcf6c66	/dev/sdh	attaching	2008-09-26T12:01:15+0000
$ ec2-describe-volumes
VOLUME	vol-9f00e5f6	1	snap-cb7493a2	us-east-1a	in-use	2008-09-26T11:52:37+0000
ATTACHMENT	vol-9f00e5f6	i-0fcf6c66	/dev/sdh	attached	2008-09-26T12:01:15+0000
VOLUME	vol-4001e429	1		us-east-1c	available	2008-09-25T09:51:48+0000

Let’s see whether it is available. Connect to the desktop as described in Preparing for Amazon AWS Usage.

In the terminal connected to the instance, we can mount the volume now:

root@domU-12-31-38-00-6C-F6:~# mkdir /mnt/my-volume
root@domU-12-31-38-00-6C-F6:~# mount /dev/sdh /mnt/my-volume

Now the volume should be mounted as “my-volume” and accessible. Let’s verify that by opening a file browser on that volume.

Indeed, the “readme” file that was on the original volume is also on the new volume created from the snapshot.

No comments

In this post I share an experiment to create an EBS volume, to attach it to an EC2 instance, to mount it in the instance, to put a file on it, to unmount it, and to detach it. Afterwards the volume will be mounted in another instance (while the first instance has been terminated, because attaching volumes to different instances at the same time is impossibe).

I followed the instructions given in the Elastic Block Storage Feature Guide.

Starting an Instance

Let’s see which AMIs are available:

$ ec2-describe-images -o self
IMAGE	ami-c6c622af	dehonk-gettingstarted/image.manifest.xml	190912652296	available	private		i386	machine

I launch ami-c6c622af with Elasticfox. Let’s check the status of the instance with the command line tools:

$ ec2-describe-instances
RESERVATION	r-9f3deef6	190912652296	default
INSTANCE	i-520faf3b	ami-c6c622af			pending	gettingstarted-keypair	0		m1.small	2008-09-25T09:50:01+0000	us-east-1c

Important to note for later is the availability zone in which the instance is running, because volumes can ony be attached to instances when they live in the same availability zone.

Create the Volume

Create a volume of 1 GB in the same availability zone in which the instance resides:

$ ec2-create-volume --size 1 -z us-east-1c
VOLUME	vol-4001e429	1		us-east-1c	creating	2008-09-25T09:51:48+0000

Check the status of the volume:

$ ec2-describe-volumes vol-4001e429
VOLUME	vol-4001e429	1		us-east-1c	available	2008-09-25T09:51:48+0000

The volume is available now. Time to use it!

Attaching the Volume

Attach the newly created volume as device /dev/sdh to the running instance:

$ ec2-attach-volume vol-4001e429 -i i-520faf3b -d /dev/sdh
ATTACHMENT	vol-4001e429	i-520faf3b	/dev/sdh	attaching	2008-09-25T09:59:14+0000

The command returns saying that the volume is attaching. Let’s check the status:

$ ec2-describe-volumes VOLUME	vol-4001e429	1		us-east-1c	in-use	2008-09-25T09:51:48+0000 ATTACHMENT	vol-4001e429	i-520faf3b	/dev/sdh	attached	2008-09-25T09:59:14+0000

While the volume was “available” and “attaching” before, now it is “in-use” and “attached”.

Formatting the Volume

Open another termnal. Connect to the instance via ssh:

$ ssh -i id_rsa-gettingstarted-keypair root@ec2-75-101-254-227.compute-1.amazonaws.com

Looking at the contents of /dev reveals that the volume is available as device “sdh”:

# ls /dev
MAKEDEV   port   ptyc1  ptye6  ptyqb  ptyt0  ptyv5  ptyxa  ptyzf    ttya2  ttyc7  ttyec  ttyr1  ttyt6  ttyvb  ttyy0
--- cut here for brevity ---
loop7     ptyb3  ptyd8  ptypd  ptys2  ptyu7  ptywc  ptyz1  sdh      ttyb9  ttyde  ttyq3  ttys8  ttyud  ttyx2  ttyz7
--- cut here for brevity ---

Since a new volume is not formatted, we do that first:

# yes | mkfs -t ext3 /dev/sdh
mke2fs 1.38 (30-Jun-2005)
/dev/sdh is entire device, not just one partition!
Proceed anyway? (y,n) Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
131072 inodes, 262144 blocks
13107 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=268435456
8 block groups
32768 blocks per group, 32768 fragments per group
16384 inodes per group
Superblock backups stored on blocks:
	32768, 98304, 163840, 229376

Writing inode tables: done
Creating journal (8192 blocks): done
Writing superblocks and filesystem accounting information: done

This filesystem will be automatically checked every 20 mounts or
180 days, whichever comes first.  Use tune2fs -c or -i to override.

Mounting the Volume

Finally, the volume is ready to be mounted in the instance:

# mkdir /mnt/data-store
# mount /dev/sdh /mnt/data-store

Let’s check whether everything is as expected:

# ls /mnt
data-store  lost+found
# ls /mnt/data-store/
lost+found

That looks okay.

Put a file on the volume

Using vi, I created a file named “readme” with this contents:

This is an example file to show that a file persists on an EBS volume after unmounting and detaching.

Unmounting the Volume

Before we stop the instance, we have to unmount the volume. From the Elastic Block Storage Feature Guide: A volume must be unmounted inside the instance before being detached. Failure to do so will result in damage to the file system or the data it contains.

# umount /mnt/data-store

Remember to cd out of the volume, otherwise you will get an error message “umount: /mnt/data-store: device is busy”

Detach the Volume

From the Feature Guide: An Amazon EBS volume can be detached from an instance by either explicitly detaching the volume or terminating the instance. Let’s do it by explicitly detaching it:

$ ec2-detach-volume vol-4001e429 -i i-520faf3b -d /dev/sdh
ATTACHMENT	vol-4001e429	i-520faf3b	/dev/sdh	detaching	2008-09-25T09:59:14+0000

Soon the status of the volume changes form “detaching” to “available”:

$ ec2-describe-volumes
VOLUME	vol-4001e429	1		us-east-1c	available	2008-09-25T09:51:48+0000

Mounting the Volume in Another Instance

Now do all the steps over again to start a new image and mount the volume. Because the volume resides in availability zone “us-east-1c” and instances and volumes have to live in the same availability zone, we have to launch the instance in “us-east-1c”.

$ ssh -i id_rsa-gettingstarted-keypair root@ec2-72-44-53-70.compute-1.amazonaws.com
Last login: Tue Sep  9 14:48:20 2008 from 213.49.236.209

	 __|  __|_  )  Rev: 2
	 _|  (     /
	___|\___|___|

 Welcome to an EC2 Public Image
                        

    Getting Started

    __ c __ /etc/ec2/release-notes.txt

[root@domU-12-31-39-01-5C-76 ~]# mkdir /mnt/data-store
[root@domU-12-31-39-01-5C-76 ~]# mount /dev/sdh /mnt/data-store
[root@domU-12-31-39-01-5C-76 ~]# cd /mnt/data-store
[root@domU-12-31-39-01-5C-76 data-store]# ls
lost+found  readme
[root@domU-12-31-39-01-5C-76 data-store]# cat readme
This is an example file to show that a file persists on an EBS volume after unmounting and detaching.
[root@domU-12-31-39-01-5C-76 ~]# umount /mnt/data-store
[root@domU-12-31-39-01-5C-76 ~]# exit

The file we created earlier was on the volume and we could read it. This proves that we can share volumes among instances.

To clean up:

$ ec2-detach-volume vol-4001e429 -i i-5a12b233
ATTACHMENT	vol-4001e429	i-5a12b233	/dev/sdh	detaching	2008-09-25T11:25:46+0000
$ ec2-describe-volumes
VOLUME	vol-4001e429	1		us-east-1c	available	2008-09-25T09:51:48+0000
$ ec2-terminate-instances i-5a12b233
INSTANCE	i-5a12b233	running	shutting-down
n$ ec2-describe-instances
RESERVATION	r-eb22f182	190912652296	default
INSTANCE	i-5a12b233	ami-c6c622af	ec2-72-44-53-70.compute-1.amazonaws.com	domU-12-31-39-01-5C-76.compute-1.internal	shutting-down	gettingstarted-keypair	0		m1.small	2008-09-25T11:23:01+0000	us-east-1c
$ ec2-describe-instances
RESERVATION	r-eb22f182	190912652296	default
INSTANCE	i-5a12b233	ami-c6c622af			terminated	gettingstarted-keypair	0		m1.small	2008-09-25T11:23:01+0000

4 comments

Section “Setting up the Tools” of the Amazon EC2 Getting Started Guide explains how to set up environment variables, so that the EC2 tools find themselves (EC2_HOME), Java ($JAVA_HOME), the private access key file (EC2_PRVATE_KEY) and the certificate file (EC2_CERT). And they also suggest to change the PATH variable, so that you can run EC2 commands from anywhere.

Of course, you do not want to set the environment variables every time you want to use the EC2 tools. I added the lines below to ~/.bash_login on my Mac, so that the environment variables are set everytime I open a terminal. I installed the tools in ~/AWS/ec2-api-tools-1.3-24159 and the private access key file and the certificate file reside in ~/.ec2.

export EC2_PRIVATE_KEY=~/.ec2/pk-<some id here>.pem
export EC2_CERT=~/.ec2/cert-<some id here>.pem
export EC2_HOME=~/AWS/ec2-api-tools-1.3-24159
export JAVA_HOME=/System/Library/Frameworks/JavaVM.framework/Home
export PATH=$PATH:$EC2_HOME/bin

No comments

In order to experiment with Amazon Web Services, I requested an AWS account and I installed a bunch of software to get started. Here is an overview of what I did to get up and running.

Setting Up an AWS Account and the EC2 Tools

I filled in the registration form to request a new AWS account. Then I followed the excellent Gettting Started Guide. It explains how to set up an account, how to install the EC2 tools, how to run an instance, and how to create your own image starting from an existing one. I immediately felt the power to EC2.

The AWS Service Health Dashboard shows the status of AWS. It even shows the status history of about 1 month.

The account activity (you need an AWS account to see this page) shows the costs involved with AWS usage.

Installing Useful FireFox Plug-Ins

Although the EC2 tools provide everything you need to manage images, instances, and other EC2 resources, it is easier to have a nice management GUI.

Elasticfox is a FireFox plug-in for interacting with EC2, the Elasic Compute Cloud. When installed, it is available from the Tools menu. The plug-in opens in a web page and shows the avalaible machne images (AMIs) and your instances. Running an instance is as simle as selecting an AMI and pressing the “Launch Instance(s)” button. The list of public images is long. Elasticfox allows to filter the list.

The running instances show up in the “Your Instances” list. For each running instance, the public DNS name is shown, so that you can use that to connect to the instance through SSH, HTTP or other means. In the “Your Instances” list you can terminate instances when they are no longer necessary. Terminated instances are kept in the list until about one hour after termination.

S3Fox Organizer for Amazon enables access to Amazon S3 (Simple Storage Service). It shows the contents of your buckets on S3 and you can download and upload files. Simply entering you account ID, the access key and the secret key, and you are ready to roll.

Installing Desktop Access to Instances

When you want to use a desktop image, so that you can use a GUI to manage your instances, you need software to access the desktop of the instances.

First you need a desktop image. I chose an Ubuntu GutsyDesktop AMI, which is also available as a public AMI and listed in Elasticfox. I use a Mac, so I downloaded the NX Client for Mac OSX (you can download NX Client for other operating systems from the same location). In no time, I was able to manage the instance through a GUI.

I selected “ami-0757b26e” from the list in Elasticfox and pressed the “Launch Instance(s)” button. In the dialog window that appeared, I entered my key pair and pressed the “Launch” button. When the instance was running, I opened a terminal and entered:

ssh -i /Users/Koen/ec2-keys/id_gettingstarted-keypair root@ec2-75-101-241-111.compute-1.amazonaws.com

Using the provided information, user “root” logs in automatically. Then I entered:

user-setup

and followed the instructions to enter a user name (I chose “koen”) and a password.

Then I started NX Client and configured it as follows:

After pressing “Login” and waiting for the connection to be setup, this was the result (I already opened the home folder before making the screen shot): the Ubuntu desktop in X11.

Cool.

5 comments