While I don’t have access to it yet, apparently Amazon has opened US-WEST-1 for EC2. Customers using enStratus have access already.
This is important because anyone leveraging platforms like Facebook or MySpace have just put themselves nearly 3000 miles closer to the key data centers where these platforms are running out of.
Earlier this year, Amazon launched a suite of new services that replaced the need to work with a product like Scalr and RightScale for building scaleable applications on the EC2 platform. Those tools help you allocate more resources according to current application load. The key benefit of using a cloud based service is that you only pay for what you use. However, without one of the afore mentioned providers, and their additional costs, you were in a lurch designing a system that could detect the current load of your infrastructure and respond accordingly. Amazon has now made it very simple to create infrastructure that can expand AND contract very simply, of course only paying for what you use.
Solutions for load balancing were as varied as round robin selection DNS to running a load balancer on an instance (I’d been using Nginx on an m1.small instance $0.10/hr). Nginx worked well, with an assigned an elastic ip (static ip) that could move from machine to machine as needed and special scripts to manage the pool of servers (or do it manually). It worked, but was by no means efficient or even easy to maintain. Furthermore, there is a single point of failure with the Nginx host. Being proactive, it was possible to create a monitoring system to monitor Nginx, and then bring up and configure a new server before re-directing the elastic ip to the new host should it fail. It was a hack and certainly not elegant!
Enter elastic load balancing. You create an elastic load balancer and then add the instances to the load balancer. That’s it! Amazon handles the redundancy and the best part is that it’s only $0.025/hr that’s a savings of $54/month over running a load balancer instance. There is of course a drawback. With Nginx and other load balancers, you have the option to do intelligent load balancing. Advanced functionality like sticky sessions and response rewriting isn’t available for the Amazon solution. However, with a well designed application, this should be irrelevant.
Monitoring the cloud is VERY important. Amazon has issues with all sorts of things from EBS stores going offline to instances being completely unavailable. Before CloudWatch, I used a mix of systems including SNMP monitoring and 3rd party service Pingdom to keep tabs on my instances. The CloudWatch product doesn’t replace these, but rather supplements the data I gather from them. CloudWatch is an additional $0.015 per server above the default instance cost, it takes about 2 minutes to come online and the statistics are available through the API almost immediately after that. CloudWatch provides access to monitored instance’s CPU utilization, disk read bytes, disk read operations, disk write bytes, disk write operations, network in, and network out. I find for my needs, CPU utilization is an excellent indicator of server performance and I use that to determine when to add a new server or take one away.
You can gather these statistics grouped by AMI, Instance Id, instance type and even AutoScaling group. If you can reliably detect your need to add an additional server based on these statistics, you’ll be able to take advantage of Auto Scaling; more on that in a minute. If not, it’s very simple to write a script that determines if it’s time to start a new server up to help with processing and register it with the load balancer. Oh, and did I mention for the load balancer you also get access to healthy host count, latency, request count, and unhealthy host count? These could be helpful metrics for rolling your own scaling scripts or may be sufficient for knowing when you need an additional server.
This is the glue that brings it all together. Auto Scaling monitors your statistics from CloudWatch and starts new instances when needed then turns them back off when no longer needed. Currently this is all offered for FREE if you are using CloudWatch! The setup is simple once you go through it the first time, but took me a couple of tries to get it right. So in my case, I monitor my application server pool and when I see that it’s stressed, I add another server. Because of the way it’s configured, I have some safe guards in place that keep me from starting thousands of instances too.
This assumes you’ve installed all the Amazon CLI tools for Elastic Load Balancing, Auto Scaling and CloudWatch, your fairly comfortable at the command line and know how to make your own AMI. Now, you’ll need to determine what the best way for you to publish your code to a new server is. Some possible solutions to this are rsync, subversion, nfs mount, s3 or a mix of technologies. Some folks just bundle up the code in their AMI (works well if your codebase is static). Regardless, that’s a bit beyond the scope of this post. After you create your solution, you’ll create a server image (AMI) that can boot up and correctly get a copy of the code you’re running. If you already have that, you can of course just use that one. Once you create an instance that can be turned on and handle traffic…
The DNS-NAME that is returned is the point you’ll direct all traffic to. Add this as a CNAME in your DNS for your domain.
elb-create-lb ApplicationServer --availability-zones us-east-1a --listener "protocol=http,lb-port=80,instance-port=80" DNS-NAME ApplicationServer-12345678.us-east-1.elb.amazonaws.com |
The AMI will of course be your AMI that knows how to come online and get a fresh copy of your code and you may be using different instance types. Definitely take a look over the documentation to ensure you are doing it all right
as-create-launch-config AppServerConfig --image-id ami-12345678 --instance-type m1.small --group default |
I use a nice long cooldown period here (10 minutes) so that the servers don’t come online or go offline too quickly. If you expect an occasional slashdotting, you might want this to be shorter. This also provides some a boundry. There will always be at least 1 server and no more than 3. This also tells auto scaling that you want the new instance to join the load balancer.
as-create-auto-scaling-group AppServerGroup --launch-configuration AppServerConfig --availability-zones us-east-1a --min-size 1 --max-size 3 --cooldown 600 --load-balancers ApplicationServer |
You will likely spend a good bit of time working on this portion. What this basically does is if the average CPU utilization for my servers is above 70% for 10 minutes, bring a new server online. Then likewise, if it falls below 30% for 10 minutes, turn one off. The Auto Scaling Group we created ensures there is always at least 1 server online.
as-create-or-update-trigger AppServerTrigger --auto-scaling-group AppServerGroup --namespace "AWS/EC2" --measure CPUUtilization --statistic Average --dimensions "AutoScalingGroupName=AppServerGroup" --period 60 --lower-threshold 30 --upper-threshold 70 --lower-breach-increment=-1 --upper-breach-increment 1 --breach-duration 600 |
That is all there is to it! You now have an system that can grow your application servers up on demand! I hope this helps you build out an infrastructure that lets you scale up your next web application. You might want to look over the command line tool documentation before getting started.
A few weeks ago I switched Sexii to use Amazon’s CloudFront content delivery network (CDN) instead of serving images through my own Apache server. The decision came after analysis of watching my EC2 instance get slammed through a couple of high volume peaks. I crunched my logs and found that over 80% of the traffic I had was serving images! Adding a new instance for handling images exclusively would certainly solve the problem, but at ~$75/month for a new instance, I figured there was probably a better way. My ultimate goal here to increase the potential work my infrastructure can do per instance hour, not necessarily save costs, but I don’t want to spend more than I need to.
Setting up CloudFront is like any AWS service, agree to the terms and costs and get going. There’s a full API for interacting with it, but rather than learn another new API, I opted for the point and click features of S3 Organizer. If you’re unfamiliar with the product, it gives you Finder or Explorer like functionality for managing the data you store on the Amazon S3 platform.
I started by creating a unique bucket in S3 and uploading my images. Now with my content happily hosted on S3, I actually had a CDN of sorts. I could have set the access control list (ACL) to allow anonymous reads for that bucket then updated my code accordingly and stopped there. S3 would handle the serving of the content and my server is free to do other tasks. Some popular websites are already doing this; for example Twitter serves your profile picture from S3. This actually accomplishes my primary goal, reducing the stress on my server. But, being the curious geek that I am, I wanted to try out the full blown CDN and with the pay per use pricing model, it was low risk.
With the bucket setup, S3 Fox was able to create the distribution with a couple of clicks (literally). What surprised me was how long it actually took (10 minutes) for the distribution to come online. Since my image URL’s are assembled on the fly in code, I updated my configuration file to reflect the new source and I was live. I did run into one problem, the URL’s for images had to be syntactically correct and I was bit by this. By default, Apache is more forgiving and just ignores the double forward slash.
Works: <img src="http://a5e3px8iw78h4.cloudfront.net/images/logo.png" /> Doesn't: <img src="http://a5e3px8iw78h4.cloudfront.net/images//logo.png" /> |
A nice feature, that I didn’t use, is the ability to map your own domain name to the delivery network; effectively masking a name like http://a5e3px8iw78h4.cloudfront.net with http://mycdn.example.com if you want. This wasn’t critical for me and so I skipped it, besides Amazon’s DNS servers are likely much more robust than mine.
Before dissecting the costs, its important to understand the content I moved. Sexii doesn’t actually have many images. It relies heavily on CSS and the layout as the design. It was done this way to eliminate hosting costs early on. MySpace applications that have home and profile surfaces incur a very high traffic volume with a low return on the investment. The assets hosted by the application, are a set of small icons used in a feature that facilitates flirting. The feature shows a grid of around 30 icons that the user can attach to a flirt. Each icon is a separate image file and all of these icons are shown at the same time resulting in about 30 requests for <1k images. See below for the consequences of this design mistake.
Costs are cumulative across the AWS platform and so are a bit difficult to break down on a per application basis, but I'm going to try to give you an overview of what the change to CloudFront cost me and where I can potentially save more money. I've skipped over the costs of uploading and hosting on S3 as they're insignificant for my data and usage pattern, less than $0.08/month.
During the roughly 4 1/2 day window analyised, the cost for CloudFront was $6.44, roughly $1.46/day. I served over 5.4 million images or 14.3 images per second to the 4 regions. If I had built a dedicated server and configured my site to use that for all images, it would have cost me around $11.44 including bandwidth for the same time period. The largest portion of the cost for me was the overall number of requests. By rewriting the icon code to use CSS positioning, I expect to reduce the request count more than 90% (-$4.95 or -$1.12/day). The bandwidth number should stay about the same because the larger icon file transfers the same amount of information as the 30 independent requests.
For those of you who are curious, the Japanese traffic was the smallest portion at 0.11% and Hong Kong was 0.16%.
Move more to CloudFront!
First step, after reducing the number of requests, is to move externally loaded CSS and JavaScript files. These are no-brainers (and are even suggested on the CloudFront site). Taking it a bit further, I’m considering moving static advertising iframe files, common in Social Network based applications. I can continue to maintaing my generic demographic targeting by creating a series of files that include the appropriate demographic distinctions. skyscraper_m_18.html, skyscraper_f_18.html, skyscraper_m_25.html, skyscraper_f_25.html and so on. I’ll be looking to see if this can be better handled through S3 alone to keep the costs of these low value CPM/CPC ads lower. MySpace hosts the actual application code on their servers, but sites like Hi5 and Orkut do not. I would strongly suggest moving your application.xml files to a CDN and only run the ajax response files on your application servers.
Where would it all end?
It’s conceivable you could host an entire brochureware site on CloudFront but that may be taking it too far.
I’m pleased with CloudFront. I’ve definitely increased the capacity of the application server which is especially useful during peak times. It also exposed a design weakness, a sad but important lesson to learn. This is a GREAT way for companies to get the speed benefits of a CDN and increase the processing capacity of an application server.
Amazon’s Web Services are very handy, although sometimes the command line tool syntax is a little awkward to remember and the documentation, while extensive, is not quite as simple to navigate as I’d like. I’m providing these help files as a reference for anyone who might need them. As you are no doubt aware, you can also get this content directly by issuing <aws_command_name> --help in the shell. For me it’s much easier to have these up in a browser window so I can quickly toggle between it and the command line without losing my place. I’ve added references for Auto Scaling, CloudWatch, Elastic Compute Cloud and Elastic Load Balancing.
Over the last few weeks I’ve been considering some options for cutting development costs for myself and a few clients. One of the continuing questions is how to manage the code base. Keeping a development server on hand is great during periods of active development and work, but when the site reaches maturity and only bug fixed are required, development servers sit idle for weeks on end without use. This got me thinking about how to best manage the source in a persistent way. This first post looks at how this might be accomplished using EC2.
I’ve been thinking about moving a development environment to Amazon EC2 from a dedicated server. The problem is, at least for this project, development only occurs a few hours per day and may go entire weeks without anyone working on it. Obviously a small instance at $0.10/hour is sufficient for load. That would cost roughly $72/month. But, even if I’m working on the application 40 hours per week, I should be able to reduce that charge to $16/month to cover the time the server is actually on. Additionally, after being burned with an instance failure last weekend, I want to be sure the data is securely backed up as well. I thought about using EBS but, as readers have pointed out, even they can fail. Furthermore, I don’t want to create a drive sized in GB if I only need a few MB of storage. Lastly, but not least, if I need to scale the drive – I don’t want to re-create the AMI each time to reflect new drive ids.
My initial thought was to start with a public Fedora Core instance and install PersistentFS, automating all of the startup and shutdown process to ensure data integrity. Next, configure subversion to use that mount point for file storage. Last but not least, I’ll create a script I can run from my local machine (or a remote server) that starts and stops an instance and binds a known elastic IP to that instance at boot time. I think my overall costs will be greatly reduced.
Estimate of costs on EC2:
| EC2 Small Instance Run Time (40hr week) | $16.00 |
| S3 Storage Cost (~10GB AMI) | $1.50 |
| S3 Storage Cost – Filesystem | $0.15 |
| S3 Bandwidth Cost (guess) | $2.00 |
| EC2 Bandwidth Cost (guess) | $2.00 |
| Total Cost (Monthly) | $21.90 |
The bare minimum – if no development work was done at all would be the storage costs of $1.65 – certainly cheap enough! However, the time to build the initial environment, create the scripts and the time lost during the startup and shutdown of the server each time made me think there may be a better alternative. Read more on subversion hosting in the second part.
Today I learned a few painful lessons: First, EC2 instances die and a simple reboot will not recover them. Second, unlike many web hosts – amazon doesn’t offer any level of monitoring. Third, backups are only useful if they’re current. 
When I originally built my EC2 instance to host this site (and a few other applications) I was learning about Amazon’s EC2 and so spent a good amount of time trying to be as thorough in my documentation of my server setup. The result was an install script, “install_server” that effectively did all the steps I did when I turned on the server. The script goes something like this:
#!/bin/sh yum -y upgrade yum -y install sendmail yum -y install httpd yum -y install php yum -y install php-mysql yum -y install php-pecl-memcache yum -y install memcached yum -y install subversion pear install HTTP_Request cp configs/freetds.conf /etc/freetds.conf cp configs/httpd.conf /etc/httpd/conf/httpd.conf cp configs/memcached /etc/sysconfig/memcached cp configs/php.ini /etc/php.ini cp configs/fstab /etc/fstab tar -xzf webroot.01.tar.gz /etc/init.d/memcached restart /etc/init.d/httpd restart /etc/init.d/sendmail restart echo configs/crontab.txt echo NOW SETUP CRONTAB!!! |
This effectively copies into place all of the settings I need for a server. This is nice, because I am able to bring a new server online within 10 minutes or so of it going down. I should probably just automate the creation of a server specific AMI once or twice a day, but I’m just not there yet. Also – I know there are some weak points in the startup script… I’ll be working those out soon now that I see it’s really a useful tool.
Monitoring of EC2 instances needs to be done by an external service. Fortunately, not too many people care what’s going on with this site over a Saturday night to Sunday morning. I’m looking at the following options in two realms, first – a basic alert that there’s a problem, and secondly a more proactive approach that can do some instance killing and restarting on it’s own.
Right now I’m looking at these services for quick and dirty SMS alerts about the status of my instances:
And I’m looking at these for a more holistic approach to monitoring but am gun shy on relying on these to manage the instances until I learn more about them:
Any experiences anyone has had with any of these products is always appreciated.
Last but not least are backups. I have another script, aptly named “backup_server” that makes a snapshot of the settings and configurations every 24 hours during off peak times storing the data on an elastic block storage device that I have mounted to the application server. That goes something like this:
crontab -l > configs/crontab.txt cp /etc/php.ini configs/php.ini cp /etc/httpd/conf/httpd.conf configs/httpd.conf cp /etc/sysconfig/memcached configs/memcached cp /etc/fstab configs/fstab tar -czvf backups/configs.01.tar.gz configs tar -czvf backups/webroot.01.tar.gz /mnt |
Where I was burned here is that my cron job only backed up my data every 24 hours at 4am. However, the application server crashed and burned at 2am EDT. Clearly I need to consider something like rsync to prevent this type of data loss. Rsync can grab the incremental changes hourly, thus reducing work losses during between the full backups every 24 hours. As a stop gap, I’ve increased the frequency of the backups until I can get back to the system and setup rsync.
Migrating to EC2? Here’s a gotcha that’s documented but not something I even thought to look up. You can see it in the System Output Log from bootup “INIT: Entering runlevel: 4″ The default runlevel for EC2 instances (at least if you’re using the Amazon provided Fedora AMI’s) is 4. The process for editing the scripts is of course the same, it’s just all in /etc/rc.d/rd4.d instead. Alternately, for less command line oriented editing, you can always use the /usr/sbin/setup tool to manage the runlevel – of course you’ll still need to create your own scripts for your non-default packages.
Update (3/3/2010): a better measure of RAID performance is available here.
While considering different options for a database server, I decided to do some digging into Amazon Web Services (AWS) as an alternative to dedicated servers from an ISP. I was most curious about the I/O of the Elastic Block Storage (EBS) on the Elastic Compute Cloud (EC2). What I tested was a number of different file systems EXT3, JFS, XFS, ReiserFS as single block devices and then some different software RAID configurations leveraging JFS. The tests were run using Bonnie++.
The configuration was vanilla, no special tuning was done, just the default values that are assigned by the tools. I used Fedora Core 9 as the OS from the default Amazon AMI and used “yum install” to aquire the necessary utilities (more on that below). I expect with further tuning, some increases in performance can still be obtained. I used the small instance for cost reasons, which includes “moderate” I/O performance. Running on a large or extra-large standard instance should perform even better with “high” I/O performance. You can get all the instance specifications from Amazon.
First I wanted to determine what the EBS devices would compare to in the physical world. I ran Bonnie against a few entry level boxes provided by a number of ISP’s and found the performance roughly matched a locally attached SATA or SCSI drive when formatted with EXT3. I also found that JFS, XFS and ReiserFS performed slightly better than EXT3 in most tests except block writes.
Again, let me re-iterate that some numbers may not be accurately reflected in your production environment. Amazon states, small instances have “moderate” I/O availability. Presumably if your running this for a production DB, you’ll want to consider a large or extra-large instance for the memory and so you should see slightly better performance from your configuration. Also note, that the drives I allocated were rather small (to keep testing costs low) so you may experience different results with larger capacities.
Note: The graph below is in KB, not bytes as titled.
| Size (Filesystem) | Output Per Char | Output Block | Output Re-write | Input Per Char | Input Block |
|---|---|---|---|---|---|
| 4×5Gb RAID5 (JFS) | 22,349 | 58,672 | 39,149 | 25,332 | 84,863 |
| 4×5Gb RAID0 (JFS) | 24,271 | 99,152 | 43,053 | 26,086 | 96,320 |
| 10Gb (XFS) | 20,944 | 43,897 | 24,386 | 25,029 | 65,710 |
| 10Gb (ReiserFS) | 22,864 | 57,248 | 17,880 | 21,716 | 44,554 |
| 10Gb (JFS) | 23,905 | 47,868 | 21,725 | 24,585 | 55,688 |
| 10Gb (EXT3) | 22,986 | 57,840 | 22,100 | 24,317 | 48,502 |
As expected, RAID 0 does best with read/write speed and RAID 5 does very well on reads (input block) as well. For InnoDB, the re-write and block read (input)/write (output) operations are the most critical values. Longer bars mean better values. To better understand what the test is doing, be sure to read the original Bonnie description of each field.
The process for making a device is simple. There are many tutorials on how to make this persistent and you can certainly build this into your own AMI when you’re done – this is not a tutorial on how to do that. To get a volume up and running you’ll follow these basic steps:
Remember, the speed and efficiency of the single EBS device is roughly comparable to a modern SATA or SCSI drive. Use of a different filesystem (other than EXT3) can increase different aspects of drive performance, just as it would with a physical hard drive. This isn’t a comparison of the pros and cons of different engines, but simply providing my findings during testing.
| JFS | yum install jfsutils |
| XFS | yum install xfsprogs |
| ReiserFS | yum install reiserfs-utils |
I didn’t test any other filesystems such as ZFS, because I’ve read some other filesystems are unstable on Linux and I’ll be running production on Linux so the extra time for the tests seemed unnecessary. I am interested in other alternatives that could increase performance if you have any to share I’d love to hear about them.
You can quickly get a volume setup with the following:
mkfs -t ext3 /dev/sdf mkdir /vol1 mount /dev/sdf /vol1 |
Next time you mount the volume, you won’t need to use “mkfs” because the drive is already formatted.
The default AMI already includes support for RAID, but if you needed to add them to your yum enabled system, it’s “yum install mdadm”. On the Fedora Core 9 test rig I was using, RAID 0, 1, 5, 6 were supported, YMMV.
To create a 4 disk RAID 0 volume, it’s simply:
mdadm --create --verbose /dev/md0 --level=0 --raid-devices=4 /dev/sdf /dev/sdg /dev/sdh /dev/sdi mkfs -t ext3 /dev/md0 mkdir /raid mount /dev/md0 /raid |
To create a 4 disk RAID 5 volume instead, it’s simply:
mdadm --create --verbose /dev/md0 --level=5 --raid-devices=4 /dev/sdf /dev/sdg /dev/sdh /dev/sdi mkfs -t ext3 /dev/md0 mkdir /raid mount /dev/md0 /raid |
This example assumes you have 4 EBS volumes attached to the system. AWS shows 7 possible mount points /dev/sdf – /dev/sdl in the web console, however, the documentation states you can use through /dev/sdp, which is 11 EBS volumes in addition to the non-persistent storage. This would be a theoretical maximum of 10TB of RAID 5 or 11TB of RAID 0 storage!
It’s clear from the numbers that software RAID offer a clear performance advantage over a ESB volume. Since with EBS you pay per Gb not per disk, it’s certainly cost effective to create a robust RAID volume. The question that remains is how careful do you need to be with your data? RAID 0 offered blistering fast performance but like a traditional array, without redundancy. You can always set it up as RAID 5, RAID 6 or RAID 10 but this of course requires more unusable space to handle the redundancy.
Since the volumes on EBS are theoretically invincible, it may be okay to run unprotected by a mirror or parity drive, however, I haven’t found anyone who would recommend this in production. If anyone knows of a good reason to ignore the saftey of RAID 10 or RAID 6 or RAID 5, I’d love to hear the reasoning.
I am also curious if these drives maintain a consistent throughput over the full capacity of the disk or will they slow down as the drive fills like a traditional drive? I did not test this. It remains open for another test (and subsequent blog post). Should anyone run ZCAV against a 100Gb+ drive and figure that out, please let me know.
Storage starts at a reasonable $0.10/GB-Month which is reasonable and is prorated for only the time you use it. A 1Tb RAID 0 volume made of 10×100Gb volumes would only cost $1,200 per year. Good luck getting performance/dollar costs for 1Tb like that from any SAN solution at a typical ISP. There are however some hidden costs in the I/O that you’ll need to pay attention to. Each time you read or write a block to disk, there’s an incremental cost. The pricing is $0.10 per million I/O requests – which seems cheap, but just running the simple tests I ran with Bonnie++ I consumed almost 2 million requests in less than 3 hours of instance time. If you have a high number of reads or writes, which you likely do if you’re considering reading this, you’ll need to factor these costs in.
The total AWS cost for running these tests was $0.71 of which $0.19 were storage related. The balance was the machine instances and bandwidth.
This morning my spam filter caught this message from Amazon announcing the new Amazon Block Storage service. I’m looking forward to seeing this work it’s way into implementations from hosting companies that are currently re-selling a service layer on top of the already interesting EC2.
One of the largest drawbacks with EC2 (and a major reason I’ve stayed away from it with a 10′ pole) has been the lack of persistent storage. If your just purchasing occasional horsepower – say for a large compute project – you would have to configure your instance – import your data – do your calculations etc and then bring all your data back down. How many times have you needed to wait until the middle of the night to run complex queries or analysis on your data and didn’t want to take down your database to do it? It’s been more than I can count for me. Now you can run the server only when you need it. I can see this a serious boon for folks wanting to do data analysis but don’t need the large EC2 container running over weekends and holidays but don’t want to pay for transfer costs to and from local sources (or deal with pushing data in and out of S3).
From the email:
Prior to Amazon EBS, block storage within an Amazon EC2 instance was tied to the instance itself so that when the instance was terminated, the data within the instance was lost. Now with Amazon EBS, users can chose to allocate storage volumes that persist reliably and independently from Amazon EC2 instances. Amazon EBS volumes can be created in any size between 1 GB and 1 TB, and multiple volumes can be attached to a single instance. Additionally, for even more durable backups and an easy way to create new volumes, Amazon EBS provides the ability to create point-in-time, consistent snapshots of volumes that are then stored to Amazon S3.
Amazon EBS is well suited for databases, as well as many other applications that require running a file system or access to raw block-level storage. As Amazon EC2 instances are started and stopped, the information saved in your database or application is preserved in much the same way it is with traditional physical servers. Amazon EBS can be accessed through the latest Amazon EC2 APIs, and is now available in public beta.
Recently I was helping a friend get his new movie review blog started and decided to review a few monetizing strategies. The offering from Amazon struck me as interesting and so I have posted it here for everyone to see. This item in particular is a great widget because it puts content that I couldn’t otherwise create directly into my site. It becomes a value add for me, the publisher, because I get a very rich experience for my users, that I can tailor to meet my content needs, while Amazon gets potential sales leads. Of course I get a kickback too…
Amazon also offers more traditional banner size advertising, product links and other integration points. Check them out as a possible option for monetizing your site.
