Tutorials
Keypairs, bundling, and virtualization - oh my.
Amazon Products API Request Signing
Anyone using the Amazon Product Advertising API (previously known as the Amazon Associates Web Service) should be aware that on August 15th Amazon will start requiring all incoming requests to be signed in accordance with their API documentation. Subscribers to the service should have received an email from Amazon, but if not let this serve as a PSA as well as a tutorial. So far we have implemented solutions for a handful of clients using a variety of languages, and I wanted to share our results with the hope that it will save some headaches.
In this tutorial, we will cover signing requests using Python, PHP, and Visual Basic. If you look at the Product Advertising API documentation (or any of Amazon’s APIs really), you will see the details for signing REST requests. The process is straightforward, and more or less easy to implement.
- Add an ISO 8601 timestamp (in GMT) as the Timestamp parameters
- Sort all of the URL params
- Construct a string to sign
- Compute an HMAC signature using the SHA256 hash algorithm
- Base64 encode the resulting signature and set it as the Signature parameter
General Disclaimer
The following code snippets are not meant to be ingredients in copy-pasta, they are meant to illustrate the procedure for signing Amazon API requests and show users the appropriate functions/libraries/resources for doing so. None of the following code will run as-is.
SimpleDB Primer with Python and Boto
SimpleDB is Amazon's distributed, highly-available, super-awesome database system which runs on Amazon's cloud. It does not function or behave like a database in a traditional sense - it is more akin to a spreadsheet (Excel or OpenOffice Calc). It is, in fact, Schema-less. It is without Schema. You simply have Items which have key/value pairs. Items are grouped together into Domains, and an Item can have many Attributes (keys) and each Attribute can hold multiple Values. I know it's sort of confusing, so let's have an example. Consider the following data set:
We have 6 Items in a Domain called test–domain, and each Item has a name that is unique (this is the only requirement for Items in SimpleDB). We have defined four Attributes which will hold the data for these Items, however none of these Attributes are required. In fact, you cannot specify a required Attribute in SimpleDB (if you need this functionality it must be imposed on the application side).
Keep in mind, we have no Schema. This can be a difficult notion for database-savvy people to come to terms with, but it is a key part of what makes SimpleDB so great and so flexible.
Let's get on with the example. The following code creates a connection to SimpleDB and selects the test–domain Domain and inserts a few Items into it. Before you can do anything with SimpleDB, you need to have an AWS account and opt-in for access to SimpleDB. This takes only a moment, but I'm not waiting. Moving on.
Bundling A Custom AMI
You will see a slew of public images (including some of our own), and the list is actualy quite daunting. This is actually a very important step since this will be the foundation on which you build you AMI. We use (exclusively) the Alestic Debian AMIs (32-bit or 64-bit depending on the application). More information about what Alestic can be here. Search through the list, and find whichever base image you want to work with. If you are starting with Ubuntu or Debian, I really insist on using the Alestic images - they are frequently updated and are highly optimized for EC2.
Now that you’ve got your base AMI in hand, lets get to getting.
To begin, spawn your instance (I chose a Debian base AMI):
ec2-run-instances ami-67fe190e
Or spawn it with Elasticfox
Notice that when using a public AMI, you must spawn it using a keypair so you can access it. To read more about keypairs, check out our other tutorial. Once the instance has become available, ssh into it (Windows users check our article on using Putty and EC2).
ssh root@ec2-72-44-61-6.compute-1.amazonaws.com -i ~/.ssh/id-cloud-carpenters
Once logged in, make like a sysadmin and get to apting - or rather, apt to getting. You get the idea.
apt-get update apt-get upgrade apt-get install git-core apache2-mpm-worker libapache2-mod-python
Now that your instance is awesome, you need to bundle it. The Alestic AMIs come with the AMI tools already installed so there’s no need to download them. In order to bundle a image, you will need your private key and certificate issued by Amazon. You can find them on http://aws.amazon.com under “Your Account” and then “Access Identifiers”. When you are first prompted to generate a certificate file, you should recieve the cert-yourcert.pem and pk-yourpk.pem files. Again, this tutorial is assuming you have a basic knowledge of EC2 which includes this account identification stuff. If you’re having trouble with this, try the unofficial IRC channel (##aws), or read Amazon’s getting started guide. Once you have the private key and certificate files on the instance, start the bundle.
ec2-bundle-vol -c /root/.ec2/cert-yourcert.pem -k /root/.ec2/pk-yourpk.pem -d /mnt -u 0123456678901 -e /root/.ec2
A couple of notes about that command. Notice the pk and cert are in a directory under root called “.ec2”. I put them in their own directory so that I can explicitly exclude that directory from the bundle, because you probably don’t want you cert and private key on the newly created image (especially if you make it public). The “-u” is your 12 digit account number sans dashes from “Your Account” on http://aws.amazon.com, and “-e” is a comma-delimited list of directories you want excluded from the bundle.
The bundle shouldn’t take all that long (unless you installed a lot of stuff, or have some data files or something). It usually takes less than 5 minutes for our images to bundle (we rebundle often). After the bundling is complete, simply upload it to your S3 account.
ec2-upload-bundle -b bucket-for-new-image -m /mnt/image.manifest.xml-a yourawsaccesskeyid -s yourawssecretkey
The AWS Access Key Id and Secret Key can be found on your “Access Identifiers” page on Amazon’s website. I like to give my images meaningful names unlike “cloudcarpenters-the-best-image-ever” or “cloudcarpenters-my-ami-brings-all-the-boys-to-the-yard”.
The final step before testing you delicious new AMI is registering the manifesto manifest. This can be done with Elasticfox or with the EC2-API-tools.
ec2-register
Personally, I’ve found Elasticfox to be somewhat flaky in this department, so I typically will use the command line tools. Note: If you register an existing image manifest, it will override any previous AMI that used that manifest. Sometimes this is the desired functionality, sometimes not. If you need to do versioning with your images I would recommend a naming scheme like: my-lamp-image-0.1, my-lamp-image-0.2, etc.
Once the image is registered, launch it log in and make sure you didn’t leave anything important lying around - you know, like you certificate/private keyxt, your SSN, incriminating photos, etc.
You now have you’re very own AMI!
Accessing EC2 from Windows with Putty
Since Windows does not have a built in SSH client, logging into an EC2 instance with Windows is a little tricky. Here we present a (hopefully) straightforward approach to getting your keys setup and getting logged in with Putty.
You will need to download two executable files from Putty: putty.exe and puttygen.exe. Both are available for free on their website. There are other SSH clients for Windows, but Putty is the only decent free one (to my knowledge).
Assuming you’re already created a key (if not check out this tutorial), we need to first prep the key for use with Putty. Load Puttygen.exe and load your private key-pair file. The contents of you key should appear in the window (I removed it from the following screenshot).
Generating a Keypair for Amazon EC2
Before you do anything on EC2, you need a keypair plain and simple. There are two easy ways to generate a key: one is using Elasticfox (my preferred method), and the other is using Amazon’s EC2-API-tools. If you are reading this tutorial, I’m assuming you’re new to working with EC2. A very brief run down of how to get the EC2-API-tools working:
- Goto http://aws.amazon.com, sign up for EC2
- Under “Your Account” click “Access Identifiers” (you will likely come back here often)
- Generate a certificate (you should get two *.pem files cert-somelonghash.pem and pk-somelonghash.pem
- Make a directory in your home folder, call it something like “ec2” or “.ec2” or “aws”. Put the files here.
- Download the tools (link above)
- Install and configure the tools (Amazon has a good write-up on this)
