In the Cloud

Secure Shell is a great tool for securely connecting between several machines. In the past weeks, I am using it more and more, but I was getting tired of typing too much. I found a great article on setting up passwordless authentication using public/private keys and defining multiple SSH identities, but it still wasn’t enough.

I manage multiple Unix users on Dreamhost, a plethora of Linux virtual machines at work, running Hudson builders and two additional machines at home. With ssh-keygen, you can generate multiple different public/private keypairs (aka an identity). The section “Selecting Keys” of the above mentioned article describes how you can select a specific identity for connecting to a specific host. The example below shows how to connect to one of my DreamHost user accounts in a passwordless manner:

ssh -i ~/.ssh/dh-user1 user1@boba.dreamhost.com


If you have a long list of accounts, it would definitely be easy to use shortcuts for every combination of user@host and link that up to a specific SSH identity. Well, this is possible with the use of an SSH config file. I found out about this file here and then read more about it in the man page.

When you have user1 and user2 as accounts on your remote machine, in my case boba.dreamhost.com, and having different SSH identities for each user (dh-user1[.pub] and dh-user2[.pub]), how do you link everything together to be able to just type one of the following:

ssh dh-user1
ssh dh-user2


Actually, this is quite easy. Here is the ~/.ssh/config file in my local account (the account I’m making SSH connections from):

Host dh-user1
User user1
HostName boba.dreamhost.com
IdentityFile ~/.ssh/dh-user1

Host dh-user2
User user2
HostName boba.dreamhost.com
IdentityFile ~/.ssh/dh-user2


Every section in this file starts with Host ConnectionName followed by a number of connection parameters that are fully described in the man page. In my case, I specify the real host name, the username on the remote machine, and the identity file I want to link to that account.

Done!

1 comment

Lately, I have been reading a lot on various technical topics. Usually, after my reading session, I try to find out how to mix good products and/or tools for the benefits not just to add up, but to multiply.

The latest products and tools I have been examining are these:

• SpringSource dm Server
  
• OSGi in general
• Groovy
• Grails

I am not going to rephrase all that is marketed as shiny and exciting since you can read that on the respective web sites. What I will do here is finding out how close I can get to building a simple web application in Grails, using a backend service written in Groovy, where both modules are OSGi bundles and both deployed on the SpringSource dm Server.

Let’s start with the setup of all our tools!

First of all, go to the SpringSource website and download the Community edition of SpringSource dm Server. Follow the installation instructions, and launch the server. Here is the output you should get:

AtrisoBook:springsource-dm-server-1.0.0.RELEASE ringods$ bin/startup.sh
[2008-10-31 17:28:59.097] main                     <SPKB0001I> Server starting.
[2008-10-31 17:28:59.584] main                     <SPOF0001I> OSGi telnet console available on port 2401.
[2008-10-31 17:29:01.990] main                     <SPKE0000I> Boot subsystems installed.
[2008-10-31 17:29:02.817] main                     <SPKE0001I> Base subsystems installed.
[2008-10-31 17:29:03.962] server-dm-2              <SPPM0000I> Installing profile 'web'.
[2008-10-31 17:29:04.771] server-dm-2              <SPPM0001I> Installed profile 'web'.
[2008-10-31 17:29:04.860] server-dm-4              <SPSC0001I> Creating HTTP/1.1 connector with scheme http on port 8080.
[2008-10-31 17:29:04.889] server-dm-4              <SPSC0001I> Creating HTTP/1.1 connector with scheme https on port 8443.
[2008-10-31 17:29:04.905] server-dm-4              <SPSC0001I> Creating AJP/1.3 connector with scheme http on port 8009.
[2008-10-31 17:29:04.930] server-dm-4              <SPSC0000I> Starting ServletContainer.
[2008-10-31 17:29:05.417] server-dm-13             <SPPM0002I> Server open for business with profile 'web'.
[2008-10-31 17:29:05.435] fs-watcher               <SPDE0048I> Processing 'INITIAL' event for file 'server.admin.web-1.0.0.RELEASE.jar'.
[2008-10-31 17:29:06.243] fs-watcher               <SPSC1000I> Creating web application '/admin'.
[2008-10-31 17:29:06.638] async-delivery-thread-1  <SPSC1001I> Starting web application '/admin'.
[2008-10-31 17:29:07.044] fs-watcher               <SPDE0010I> Deployment of 'com.springsource.server.servlet.admin' version '1.0.0.RELEASE' completed.
[2008-10-31 17:29:07.048] fs-watcher               <SPDE0048I> Processing 'INITIAL' event for file 'server.admin.splash-1.0.0.RELEASE.war'.
[2008-10-31 17:29:07.582] fs-watcher               <SPSC1000I> Creating web application '/'.
[2008-10-31 17:29:07.714] async-delivery-thread-1  <SPSC1001I> Starting web application '/'.
[2008-10-31 17:29:07.878] fs-watcher               <SPDE0010I> Deployment of 'com.springsource.server.servlet.splash' version '0' completed.

So far so good! Now shutdown it again. During development, we will launch the server from within Eclipse.

Next step is to get our developer tools up and running. At the time of writing, I downloaded Eclipse Ganymede 3.4.1 (SR1) JEE edition. Launch Eclipse and add three update sites:

• Spring IDE
• SpringSource dm Server Tooling
• Groovy Eclipse Plugin

From these update sites, install all the tools offered.

After Eclipse relaunched, we define a local SpringSource dm Server that will be used to deploy our 2 projects. Start with opening the Servers view in Eclipse.

The New Server wizard starts, you select SpringSource dm Server v1.0, give a name and press Next.

In the second part of the wizard, you point to your installation folder of the SpringSource dm Server. You only have to do this once. The wizard will create a SpringSource dm Server Runtime Environment, similar to the different Java Runtime Environments you can have in Eclipse.

Press Finish and your server is now listed in the Servers view.

Using the buttons at the top of the Servers view, you can start and stop the SpringSource dm Server. All output should be accessible in the Console view within Eclipse. Go ahead and try it out.

In the next article, I will start with the development of the backend and web modules. So stay tuned for more!

No comments

Continuing with the volume created in the post Sharing EBS Volumes Among Instances, in this post I show how to create a snapshot, create a new volume from that snapshot, and mount the new volume in an instance. Remember that the volume created in the previous post contained 1 file called “readme”.

Creating the Snapshot

First we look at the available volumes:

$ ec2-describe-volumes
VOLUME	vol-4001e429	1		us-east-1c	available	2008-09-25T09:51:48+0000

Then we make the snapshot:

$ ec2-create-snapshot vol-4001e429
SNAPSHOT	snap-cb7493a2	vol-4001e429	pending	2008-09-26T11:48:30+0000

It is “pending”. We check the status until it the snapshot creation has “completed”:

$ ec2-describe-snapshots snap-cb7493a2
SNAPSHOT	snap-cb7493a2	vol-4001e429	completed	2008-09-26T11:48:30+0000	100%

Creating a Volume from the Snapshot

Now that the snapshot is ready, we can create a new volume from it. Note that we create it in a different availability zone. The orginal volume resides in “us-east-1c”. The new volume will reside in “us-east-1a”.

$ ec2-create-volume --snapshot snap-cb7493a2 -z us-east-1a
VOLUME	vol-9f00e5f6	1	snap-cb7493a2	us-east-1a	creating	2008-09-26T11:52:37+0000

We wait until the volume is “available”:

$ ec2-describe-volumes
VOLUME	vol-9f00e5f6	1	snap-cb7493a2	us-east-1a	available	2008-09-26T11:52:37+0000
VOLUME	vol-4001e429	1		us-east-1c	available	2008-09-25T09:51:48+0000

Now we have two available volumes.

Mounting the New Volume in an Instance

Let’s launch an image so that we can verify that the newly created volume can be mounted and has the same contents as the original volume. Note that the instance is launched in the availability zone where the newly created volume resides.

$ ec2-run-instances ami-0757b26e -k gettingstarted-keypair -z us-east-1a

The AMI we use here is a public Ubuntu Desktop image.

$ ec2-describe-instances
RESERVATION	r-ff4d9e96	190912652296	default
INSTANCE	i-0fcf6c66	ami-0757b26e	ec2-67-202-35-79.compute-1.amazonaws.com	domU-12-31-38-00-6C-F6.compute-1.internal	running	gettingstarted-keypair	0		m1.small	2008-09-26T11:55:16+0000	us-east-1a	aki-a71cf9ce	ari-a51cf9cc

The instance is ready to be used. In another terminal we connect to the image and start the “user-setup” script. The GUI interaction to set up the user is not shown here.

$ ssh -i id_rsa-gettingstarted-keypair root@ec2-67-202-35-79.compute-1.amazonaws.com
The authenticity of host 'ec2-67-202-35-79.compute-1.amazonaws.com (67.202.35.79)' can't be established.
RSA key fingerprint is ab:df:4e:78:b7:4d:59:3e:ae:6c:81:32:80:eb:bd:78.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'ec2-67-202-35-79.compute-1.amazonaws.com,67.202.35.79' (RSA) to the list of known hosts.
Linux domU-12-31-38-00-6C-F6 2.6.21.7-2.fc8xen #1 SMP Fri Feb 15 12:39:36 EST 2008 i686

The programs included with the Ubuntu system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Ubuntu comes with ABSOLUTELY NO WARRANTY, to the extent permitted by
applicable law.

Amazon EC2 Ubuntu 7.10 gutsy base install AMI built by
Eric Hammond
For more information: http://ec2gutsy-desktop.notlong.com

root@domU-12-31-38-00-6C-F6:~# user-setup
Shadow passwords are now on.
Using `/usr/share/libgksu/debian/gconf-defaults.libgksu-sudo' to provide `libgksu-gconf-defaults'.

Time to attach the volume (in the orginal terminal):

$ ec2-attach-volume vol-9f00e5f6 -i i-0fcf6c66 -d /dev/sdh
ATTACHMENT	vol-9f00e5f6	i-0fcf6c66	/dev/sdh	attaching	2008-09-26T12:01:15+0000
$ ec2-describe-volumes
VOLUME	vol-9f00e5f6	1	snap-cb7493a2	us-east-1a	in-use	2008-09-26T11:52:37+0000
ATTACHMENT	vol-9f00e5f6	i-0fcf6c66	/dev/sdh	attached	2008-09-26T12:01:15+0000
VOLUME	vol-4001e429	1		us-east-1c	available	2008-09-25T09:51:48+0000

Let’s see whether it is available. Connect to the desktop as described in Preparing for Amazon AWS Usage.

In the terminal connected to the instance, we can mount the volume now:

root@domU-12-31-38-00-6C-F6:~# mkdir /mnt/my-volume
root@domU-12-31-38-00-6C-F6:~# mount /dev/sdh /mnt/my-volume

Now the volume should be mounted as “my-volume” and accessible. Let’s verify that by opening a file browser on that volume.

Indeed, the “readme” file that was on the original volume is also on the new volume created from the snapshot.

No comments

In this post I share an experiment to create an EBS volume, to attach it to an EC2 instance, to mount it in the instance, to put a file on it, to unmount it, and to detach it. Afterwards the volume will be mounted in another instance (while the first instance has been terminated, because attaching volumes to different instances at the same time is impossibe).

I followed the instructions given in the Elastic Block Storage Feature Guide.

Starting an Instance

Let’s see which AMIs are available:

$ ec2-describe-images -o self
IMAGE	ami-c6c622af	dehonk-gettingstarted/image.manifest.xml	190912652296	available	private		i386	machine

I launch ami-c6c622af with Elasticfox. Let’s check the status of the instance with the command line tools:

$ ec2-describe-instances
RESERVATION	r-9f3deef6	190912652296	default
INSTANCE	i-520faf3b	ami-c6c622af			pending	gettingstarted-keypair	0		m1.small	2008-09-25T09:50:01+0000	us-east-1c

Important to note for later is the availability zone in which the instance is running, because volumes can ony be attached to instances when they live in the same availability zone.

Create the Volume

Create a volume of 1 GB in the same availability zone in which the instance resides:

$ ec2-create-volume --size 1 -z us-east-1c
VOLUME	vol-4001e429	1		us-east-1c	creating	2008-09-25T09:51:48+0000

Check the status of the volume:

$ ec2-describe-volumes vol-4001e429
VOLUME	vol-4001e429	1		us-east-1c	available	2008-09-25T09:51:48+0000

The volume is available now. Time to use it!

Attaching the Volume

Attach the newly created volume as device /dev/sdh to the running instance:

$ ec2-attach-volume vol-4001e429 -i i-520faf3b -d /dev/sdh
ATTACHMENT	vol-4001e429	i-520faf3b	/dev/sdh	attaching	2008-09-25T09:59:14+0000

The command returns saying that the volume is attaching. Let’s check the status:

$ ec2-describe-volumes VOLUME	vol-4001e429	1		us-east-1c	in-use	2008-09-25T09:51:48+0000 ATTACHMENT	vol-4001e429	i-520faf3b	/dev/sdh	attached	2008-09-25T09:59:14+0000

While the volume was “available” and “attaching” before, now it is “in-use” and “attached”.

Formatting the Volume

Open another termnal. Connect to the instance via ssh:

$ ssh -i id_rsa-gettingstarted-keypair root@ec2-75-101-254-227.compute-1.amazonaws.com

Looking at the contents of /dev reveals that the volume is available as device “sdh”:

# ls /dev
MAKEDEV   port   ptyc1  ptye6  ptyqb  ptyt0  ptyv5  ptyxa  ptyzf    ttya2  ttyc7  ttyec  ttyr1  ttyt6  ttyvb  ttyy0
--- cut here for brevity ---
loop7     ptyb3  ptyd8  ptypd  ptys2  ptyu7  ptywc  ptyz1  sdh      ttyb9  ttyde  ttyq3  ttys8  ttyud  ttyx2  ttyz7
--- cut here for brevity ---

Since a new volume is not formatted, we do that first:

# yes | mkfs -t ext3 /dev/sdh
mke2fs 1.38 (30-Jun-2005)
/dev/sdh is entire device, not just one partition!
Proceed anyway? (y,n) Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
131072 inodes, 262144 blocks
13107 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=268435456
8 block groups
32768 blocks per group, 32768 fragments per group
16384 inodes per group
Superblock backups stored on blocks:
	32768, 98304, 163840, 229376

Writing inode tables: done
Creating journal (8192 blocks): done
Writing superblocks and filesystem accounting information: done

This filesystem will be automatically checked every 20 mounts or
180 days, whichever comes first.  Use tune2fs -c or -i to override.

Mounting the Volume

Finally, the volume is ready to be mounted in the instance:

# mkdir /mnt/data-store
# mount /dev/sdh /mnt/data-store

Let’s check whether everything is as expected:

# ls /mnt
data-store  lost+found
# ls /mnt/data-store/
lost+found

That looks okay.

Put a file on the volume

Using vi, I created a file named “readme” with this contents:

This is an example file to show that a file persists on an EBS volume after unmounting and detaching.

Unmounting the Volume

Before we stop the instance, we have to unmount the volume. From the Elastic Block Storage Feature Guide: A volume must be unmounted inside the instance before being detached. Failure to do so will result in damage to the file system or the data it contains.

# umount /mnt/data-store

Remember to cd out of the volume, otherwise you will get an error message “umount: /mnt/data-store: device is busy”

Detach the Volume

From the Feature Guide: An Amazon EBS volume can be detached from an instance by either explicitly detaching the volume or terminating the instance. Let’s do it by explicitly detaching it:

$ ec2-detach-volume vol-4001e429 -i i-520faf3b -d /dev/sdh
ATTACHMENT	vol-4001e429	i-520faf3b	/dev/sdh	detaching	2008-09-25T09:59:14+0000

Soon the status of the volume changes form “detaching” to “available”:

$ ec2-describe-volumes
VOLUME	vol-4001e429	1		us-east-1c	available	2008-09-25T09:51:48+0000

Mounting the Volume in Another Instance

Now do all the steps over again to start a new image and mount the volume. Because the volume resides in availability zone “us-east-1c” and instances and volumes have to live in the same availability zone, we have to launch the instance in “us-east-1c”.

$ ssh -i id_rsa-gettingstarted-keypair root@ec2-72-44-53-70.compute-1.amazonaws.com
Last login: Tue Sep  9 14:48:20 2008 from 213.49.236.209

	 __|  __|_  )  Rev: 2
	 _|  (     /
	___|\___|___|

 Welcome to an EC2 Public Image
                        

    Getting Started

    __ c __ /etc/ec2/release-notes.txt

[root@domU-12-31-39-01-5C-76 ~]# mkdir /mnt/data-store
[root@domU-12-31-39-01-5C-76 ~]# mount /dev/sdh /mnt/data-store
[root@domU-12-31-39-01-5C-76 ~]# cd /mnt/data-store
[root@domU-12-31-39-01-5C-76 data-store]# ls
lost+found  readme
[root@domU-12-31-39-01-5C-76 data-store]# cat readme
This is an example file to show that a file persists on an EBS volume after unmounting and detaching.
[root@domU-12-31-39-01-5C-76 ~]# umount /mnt/data-store
[root@domU-12-31-39-01-5C-76 ~]# exit

The file we created earlier was on the volume and we could read it. This proves that we can share volumes among instances.

To clean up:

$ ec2-detach-volume vol-4001e429 -i i-5a12b233
ATTACHMENT	vol-4001e429	i-5a12b233	/dev/sdh	detaching	2008-09-25T11:25:46+0000
$ ec2-describe-volumes
VOLUME	vol-4001e429	1		us-east-1c	available	2008-09-25T09:51:48+0000
$ ec2-terminate-instances i-5a12b233
INSTANCE	i-5a12b233	running	shutting-down
n$ ec2-describe-instances
RESERVATION	r-eb22f182	190912652296	default
INSTANCE	i-5a12b233	ami-c6c622af	ec2-72-44-53-70.compute-1.amazonaws.com	domU-12-31-39-01-5C-76.compute-1.internal	shutting-down	gettingstarted-keypair	0		m1.small	2008-09-25T11:23:01+0000	us-east-1c
$ ec2-describe-instances
RESERVATION	r-eb22f182	190912652296	default
INSTANCE	i-5a12b233	ami-c6c622af			terminated	gettingstarted-keypair	0		m1.small	2008-09-25T11:23:01+0000

4 comments

Section “Setting up the Tools” of the Amazon EC2 Getting Started Guide explains how to set up environment variables, so that the EC2 tools find themselves (EC2_HOME), Java ($JAVA_HOME), the private access key file (EC2_PRVATE_KEY) and the certificate file (EC2_CERT). And they also suggest to change the PATH variable, so that you can run EC2 commands from anywhere.

Of course, you do not want to set the environment variables every time you want to use the EC2 tools. I added the lines below to ~/.bash_login on my Mac, so that the environment variables are set everytime I open a terminal. I installed the tools in ~/AWS/ec2-api-tools-1.3-24159 and the private access key file and the certificate file reside in ~/.ec2.

export EC2_PRIVATE_KEY=~/.ec2/pk-<some id here>.pem
export EC2_CERT=~/.ec2/cert-<some id here>.pem
export EC2_HOME=~/AWS/ec2-api-tools-1.3-24159
export JAVA_HOME=/System/Library/Frameworks/JavaVM.framework/Home
export PATH=$PATH:$EC2_HOME/bin

No comments

In order to experiment with Amazon Web Services, I requested an AWS account and I installed a bunch of software to get started. Here is an overview of what I did to get up and running.

Setting Up an AWS Account and the EC2 Tools

I filled in the registration form to request a new AWS account. Then I followed the excellent Gettting Started Guide. It explains how to set up an account, how to install the EC2 tools, how to run an instance, and how to create your own image starting from an existing one. I immediately felt the power to EC2.

The AWS Service Health Dashboard shows the status of AWS. It even shows the status history of about 1 month.

The account activity (you need an AWS account to see this page) shows the costs involved with AWS usage.

Installing Useful FireFox Plug-Ins

Although the EC2 tools provide everything you need to manage images, instances, and other EC2 resources, it is easier to have a nice management GUI.

Elasticfox is a FireFox plug-in for interacting with EC2, the Elasic Compute Cloud. When installed, it is available from the Tools menu. The plug-in opens in a web page and shows the avalaible machne images (AMIs) and your instances. Running an instance is as simle as selecting an AMI and pressing the “Launch Instance(s)” button. The list of public images is long. Elasticfox allows to filter the list.

The running instances show up in the “Your Instances” list. For each running instance, the public DNS name is shown, so that you can use that to connect to the instance through SSH, HTTP or other means. In the “Your Instances” list you can terminate instances when they are no longer necessary. Terminated instances are kept in the list until about one hour after termination.

S3Fox Organizer for Amazon enables access to Amazon S3 (Simple Storage Service). It shows the contents of your buckets on S3 and you can download and upload files. Simply entering you account ID, the access key and the secret key, and you are ready to roll.

Installing Desktop Access to Instances

When you want to use a desktop image, so that you can use a GUI to manage your instances, you need software to access the desktop of the instances.

First you need a desktop image. I chose an Ubuntu GutsyDesktop AMI, which is also available as a public AMI and listed in Elasticfox. I use a Mac, so I downloaded the NX Client for Mac OSX (you can download NX Client for other operating systems from the same location). In no time, I was able to manage the instance through a GUI.

I selected “ami-0757b26e” from the list in Elasticfox and pressed the “Launch Instance(s)” button. In the dialog window that appeared, I entered my key pair and pressed the “Launch” button. When the instance was running, I opened a terminal and entered:

ssh -i /Users/Koen/ec2-keys/id_gettingstarted-keypair root@ec2-75-101-241-111.compute-1.amazonaws.com

Using the provided information, user “root” logs in automatically. Then I entered:

user-setup

and followed the instructions to enter a user name (I chose “koen”) and a password.

Then I started NX Client and configured it as follows:

After pressing “Login” and waiting for the connection to be setup, this was the result (I already opened the home folder before making the screen shot): the Ubuntu desktop in X11.

Cool.

5 comments