Vijay Ramachandran asked me, via twitter, how to test if an Amazon Elastic Load Balancer is really doing it’s job. Because 140 characters really isn’t sufficient space to handle this answer, I’ve created this post. Feel free to use any of this in any of your environment.

First, I’ll assume you’ve covered some of the basics with ELB.

• Scaling Out eith EC2, CloudWatch, AutoScaling and Elastic Load Balancing
• 3 Amazon Elastic Load Balancer Tips
• Elastic Load Balancing in Multiple Zones
• Elastic Load Balancing Commands

The default configuration you’ll end up with following my guides above is a stateless system that distributes the requests more or less evenly across all configured servers. However, when you do it the first time, it’s nice to see that it’s actually doing what you think it should be. The steps are simple

1. Verify each instance is working as expected
2. Verify the load balancer is distributing the requests across multiple instances
3. Verify the instances are working behind the load balancer

1. Verify each instance is working

This is far and away the easiest step. You can simply access each machine by the amazon assigned IP address for that specific instance and ensure that it’s doing what you expect. The only potential issue here is you might jump from one machine to a different machine if you are not watching your URL. For example, if you are on ec2-123-123-123-123.compute-1.amazonaws.com, access your application at that address and ensure it works as expected, if it jumps to a domain name because you’ve hard coded a link somewhere, you may not be testing the new server at all.

2. Verify the load balancer is distributing the requests across multiple instances

To test that requests are being distributed across multiple machines, I use a test file. I generate my test file automatically by running the following script as part of the boot-up routine. This simply saves the instance-id from the metadata into a text file. If you are uncomfortable placing this information in the web root, you can optionally place it behind basic authentication, put it into a script that hashes it (md5 or sha1) or some other application based logic to access it.

/usr/local/bin/curl http://169.254.169.254/latest/meta-data/instance-id
 > /var/www/html/instance-id.txt

Check the path for curl and the web root for your local system and adjust accordingly. This should work from RedHat flavored distributions.

Once you’ve run this on each of your instances, you can tell that requests are being distributed to both machines by simply requesting your load balancer address and verifying that it changes. (Obviously replace the following request with the correct address for your machine.)

http://applicationservers-123456789.us-east-1.elb.amazonaws.com/instance-id.txt

3. Verify the instances are working behind the load balancer

Now for the last and final test. Confident that your requests are being distributed across both machines, test that your application works as expected. First under the Amazon assigned name, applicationservers-123456789.us-east-1.elb.amazonaws.com in this example, then under your CNAME’d alias.

If everything still works, you can assume all is good.

4. Bonus Check

If you really, really, really want to know… you can also verify using your access logs. Check in /var/log/httpd/access_log or wherever your web server logs are kept to see that requests are being distributed to each machine.

DNS Tips:

1. Never use the real IP returned from dig or nslookup as an A record in DNS unless you automate checking it (and even still I wouldn’t) because the actual IP changes from time to time. Only use CNAME entries.

2. If you are using GoDaddy’s DNS tool, you can’t CNAME the root of a domain (ie .example.com). For this case I use one instance as a permanent instance with an elastic IP and point the root A record for my domains to this. I then assign www. as a CNAME for the load balancer’s AWS assigned domain. Last but not least, I use .htaccess and mod_rewrite to ensure requests are sent to www.example.com. This ensures traffic is being sent to the load balancer address.

Earlier this year, Amazon launched a suite of new services that replaced the need to work with a product like Scalr and RightScale for building scaleable applications on the EC2 platform. Those tools help you allocate more resources according to current application load. The key benefit of using a cloud based service is that you only pay for what you use. However, without one of the afore mentioned providers, and their additional costs, you were in a lurch designing a system that could detect the current load of your infrastructure and respond accordingly. Amazon has now made it very simple to create infrastructure that can expand AND contract very simply, of course only paying for what you use.

The Tools

Elastic Load Balancer

Solutions for load balancing were as varied as round robin selection DNS to running a load balancer on an instance (I’d been using Nginx on an m1.small instance $0.10/hr). Nginx worked well, with an assigned an elastic ip (static ip) that could move from machine to machine as needed and special scripts to manage the pool of servers (or do it manually). It worked, but was by no means efficient or even easy to maintain. Furthermore, there is a single point of failure with the Nginx host. Being proactive, it was possible to create a monitoring system to monitor Nginx, and then bring up and configure a new server before re-directing the elastic ip to the new host should it fail. It was a hack and certainly not elegant!

Enter elastic load balancing. You create an elastic load balancer and then add the instances to the load balancer. That’s it! Amazon handles the redundancy and the best part is that it’s only $0.025/hr that’s a savings of $54/month over running a load balancer instance. There is of course a drawback. With Nginx and other load balancers, you have the option to do intelligent load balancing. Advanced functionality like sticky sessions and response rewriting isn’t available for the Amazon solution. However, with a well designed application, this should be irrelevant.

CloudWatch

Monitoring the cloud is VERY important. Amazon has issues with all sorts of things from EBS stores going offline to instances being completely unavailable. Before CloudWatch, I used a mix of systems including SNMP monitoring and 3rd party service Pingdom to keep tabs on my instances. The CloudWatch product doesn’t replace these, but rather supplements the data I gather from them. CloudWatch is an additional $0.015 per server above the default instance cost, it takes about 2 minutes to come online and the statistics are available through the API almost immediately after that. CloudWatch provides access to monitored instance’s CPU utilization, disk read bytes, disk read operations, disk write bytes, disk write operations, network in, and network out. I find for my needs, CPU utilization is an excellent indicator of server performance and I use that to determine when to add a new server or take one away.

You can gather these statistics grouped by AMI, Instance Id, instance type and even AutoScaling group. If you can reliably detect your need to add an additional server based on these statistics, you’ll be able to take advantage of Auto Scaling; more on that in a minute. If not, it’s very simple to write a script that determines if it’s time to start a new server up to help with processing and register it with the load balancer. Oh, and did I mention for the load balancer you also get access to healthy host count, latency, request count, and unhealthy host count? These could be helpful metrics for rolling your own scaling scripts or may be sufficient for knowing when you need an additional server.

Auto Scaling

This is the glue that brings it all together. Auto Scaling monitors your statistics from CloudWatch and starts new instances when needed then turns them back off when no longer needed. Currently this is all offered for FREE if you are using CloudWatch! The setup is simple once you go through it the first time, but took me a couple of tries to get it right. So in my case, I monitor my application server pool and when I see that it’s stressed, I add another server. Because of the way it’s configured, I have some safe guards in place that keep me from starting thousands of instances too.

How To Do It

Background

This assumes you’ve installed all the Amazon CLI tools for Elastic Load Balancing, Auto Scaling and CloudWatch, your fairly comfortable at the command line and know how to make your own AMI. Now, you’ll need to determine what the best way for you to publish your code to a new server is. Some possible solutions to this are rsync, subversion, nfs mount, s3 or a mix of technologies. Some folks just bundle up the code in their AMI (works well if your codebase is static). Regardless, that’s a bit beyond the scope of this post. After you create your solution, you’ll create a server image (AMI) that can boot up and correctly get a copy of the code you’re running. If you already have that, you can of course just use that one. Once you create an instance that can be turned on and handle traffic…

The Process

1. Create the Load Balancer
2. Create the Auto Scale Launch Config
3. Create the Auto Scale Group
4. Create the Auto Scale Trigger(s)

Create the Load Balancer

The DNS-NAME that is returned is the point you’ll direct all traffic to. Add this as a CNAME in your DNS for your domain.

elb-create-lb ApplicationServer --availability-zones us-east-1a --listener "protocol=http,lb-port=80,instance-port=80"
DNS-NAME ApplicationServer-12345678.us-east-1.elb.amazonaws.com

Create the Auto Scale Launch Config

The AMI will of course be your AMI that knows how to come online and get a fresh copy of your code and you may be using different instance types. Definitely take a look over the documentation to ensure you are doing it all right

as-create-launch-config AppServerConfig --image-id ami-12345678 --instance-type m1.small --group default

Create the Auto Scale Group

I use a nice long cooldown period here (10 minutes) so that the servers don’t come online or go offline too quickly. If you expect an occasional slashdotting, you might want this to be shorter. This also provides some a boundry. There will always be at least 1 server and no more than 3. This also tells auto scaling that you want the new instance to join the load balancer.

as-create-auto-scaling-group AppServerGroup --launch-configuration AppServerConfig --availability-zones us-east-1a --min-size 1 --max-size 3 --cooldown 600 --load-balancers ApplicationServer

Create the Auto Scale Trigger(s)

You will likely spend a good bit of time working on this portion. What this basically does is if the average CPU utilization for my servers is above 70% for 10 minutes, bring a new server online. Then likewise, if it falls below 30% for 10 minutes, turn one off. The Auto Scaling Group we created ensures there is always at least 1 server online.

as-create-or-update-trigger AppServerTrigger --auto-scaling-group AppServerGroup --namespace "AWS/EC2" --measure CPUUtilization --statistic Average --dimensions "AutoScalingGroupName=AppServerGroup" --period 60 --lower-threshold 30 --upper-threshold 70 --lower-breach-increment=-1 --upper-breach-increment 1 --breach-duration 600

That is all there is to it! You now have an system that can grow your application servers up on demand! I hope this helps you build out an infrastructure that lets you scale up your next web application. You might want to look over the command line tool documentation before getting started.

Amazon’s Web Services are very handy, although sometimes the command line tool syntax is a little awkward to remember and the documentation, while extensive, is not quite as simple to navigate as I’d like. I’m providing these help files as a reference for anyone who might need them. As you are no doubt aware, you can also get this content directly by issuing <aws_command_name> --help in the shell. For me it’s much easier to have these up in a browser window so I can quickly toggle between it and the command line without losing my place. I’ve added references for Auto Scaling, CloudWatch, Elastic Compute Cloud and Elastic Load Balancing.