Yesterday Amazon announced their Spot pricing model. Effectively providing market driven pricing for instances on EC2. Depending on your product, this probably won’t impact you much, but it got me to thinking about pricing of the cloud. Amazon’s Web Services was a game changer when it launched. Buy the computing resources you need for only the time you needed them. However, your stuck with a very limited set of instances and therefore you need to architect your systems around their pre-defined instance sizes. While they expanded their instance offering to include high cpu and more recently high memory instances, you’re still stuck with a fairly rigid set of boxes from which to run your systems.

A specific weak spot I’m having with the pre-defined box sizes is Memcached. It turns out that Memcached is fairly light on the processor and requires essentially no disk I/O. Really the processor is just a go between for the memory and the network card. If you are looking at putting a 32Gb server online to manage the caching tier for your app, you’d need to buy the “High-Memory Double Extra Large Instance” for $1.20/hr (or $10,512/year) wait… what?! Okay, obviously we should pre-pay this, typical business model is to run the hardware over a 3 year cycle, so lets pay the $4,900 up front and then we enjoy a more comfortable $0.42/hr (or $3,679.20/year + $1,633.34/year for the pre-pay = $5,312.54 each year for 3 years). Obviously the $15,937.60 we pay over 3 years is easier to swallow than the $31,536 if we don’t pre-pay it.

Now, if your running your infrastructure in the cloud and considering using Memcached, you really can’t put a box in a rack somewhere else because the increased latency and unreliability means you may not be able to get data from your cache in a cost effective way so I’m not going to look at what buying a box with that kind of memory would cost, not to mention there is such variation in buying rack/ping/power that it would be too messy to calculate here.

This has me intruiged to see how other providers are doing their billing. I love the idea of a-la-carte servers paid by the hour. But really what would be great is allowing me to choose the CPU, memory, and I/O I need. This brings me to two smaller cloud providers who seem to have interesting offerings.

First up is 3Tera. 3Tera offers a completely different take on the cloud infrastructure model. The idea behind their offering is that you purchase hardware (or lease it) and then slice the box however you want. Basically, running your own virtual cloud! You can consider different hardware options, including stuffing a ton of RAM into weaker boxes and so on. Ultimately the product is a resource allocation tool. The dark side is that you have to pay for all that hardware, even if your not using it. Really this isn’t a cost savings over EC2. Although it’s an interesting idea if your system resource needs shift significantly over time, but are consistent enough to warrant buying or leasing hardware. I’m really interested in their technology and they have an impressive list of partners running the software that you can then lease the virtual images from.

The second provider is OpSource Cloud. OpSource charges a base fee for the VLAN service and then you build your infrastructure on top of that. The beauty is that it’s a-la-carte down to the cpu cycles and memory! Currently the memory footprint is limited to 8Gb and each machine needs between 1 and 4 CPU’s. However, this pricing model is interesting as you can provision a single CPU with 8Gb of RAM which comes out to roughly $0.24/hr (or $2,102.40/year). Starting 4 of these instances to hold the 32Gb of cache is only slightly cheaper than Amazon’s model coming in at a whopping $8,409.60/year. There are some cost savings available if you buy a silver, gold or platinum pricing tier for a monthly pre-pay. The pricing for those starts at $500/month and goes up; so you really need to have some significant hardware running to justify those costs. Another gotcha with this plan is that you need to provision a network which is $0.20/hr. I’m going to be keeping an eye on this provider. I think in the future they may have a winning solution.

Unfortunately, I don’t yet see a solution that fits my specific need. Perhaps I need to adjust my thinking and look at alternatives. It may be time to consider Amazon’s Simple DB, which provides simple key/value storage like Memcached, although as a service. Is it the answer for putting large amounts of data into a non-RDBMS? I’ll consider that in another post.

Creative Commons Photo by ArcticNomad

This is the second part of of an article looking at how to effectively host a small subversion based project that is no longer going through rapid development. The first part looked at using EC2 to run Subversion and S3 for persistent storage. While an intruiging solution, it raised some concerns.

The alternative solution is to look at outsourcing the hosting of Subversion and ticket management to another provider. The size of our repository is less than 1GB and so I’m using that as the price point. Additionally, there are 2-3 developers who’ll require access to the repository. There are many great “free” services including Google, but this is not an open sourced project so it’s out. In the hosted subversion realm, there are a number of providers with basic accounts to handle this size repository. The following table is a price comparison at the 1GB storage level. Many providers offer a free service for smaller projects with different limitations for bandwidth, tickets and so on so YMMV.

• ¹ Pricing dependent on storage and developers
• ² Offers a cheaper or free plan with less than 1GB of storage.

The real benefit of a hosted solution is the addition of services such as Trac, user management, automated backups and more. If you are looking at building a project with multiple developers who are not in the same physical location, hosting your project with a service is definitely the way to go. It’s cheaper and the overhead of configuring and maintaining your own EC2 instance (or even a dedicated server) increases the costs significantly.

Over the last few weeks I’ve been considering some options for cutting development costs for myself and a few clients. One of the continuing questions is how to manage the code base. Keeping a development server on hand is great during periods of active development and work, but when the site reaches maturity and only bug fixed are required, development servers sit idle for weeks on end without use. This got me thinking about how to best manage the source in a persistent way. This first post looks at how this might be accomplished using EC2.

I’ve been thinking about moving a development environment to Amazon EC2 from a dedicated server. The problem is, at least for this project, development only occurs a few hours per day and may go entire weeks without anyone working on it. Obviously a small instance at $0.10/hour is sufficient for load. That would cost roughly $72/month. But, even if I’m working on the application 40 hours per week, I should be able to reduce that charge to $16/month to cover the time the server is actually on. Additionally, after being burned with an instance failure last weekend, I want to be sure the data is securely backed up as well. I thought about using EBS but, as readers have pointed out, even they can fail. Furthermore, I don’t want to create a drive sized in GB if I only need a few MB of storage. Lastly, but not least, if I need to scale the drive – I don’t want to re-create the AMI each time to reflect new drive ids.

My initial thought was to start with a public Fedora Core instance and install PersistentFS, automating all of the startup and shutdown process to ensure data integrity. Next, configure subversion to use that mount point for file storage. Last but not least, I’ll create a script I can run from my local machine (or a remote server) that starts and stops an instance and binds a known elastic IP to that instance at boot time. I think my overall costs will be greatly reduced.

Estimate of costs on EC2:

The bare minimum – if no development work was done at all would be the storage costs of $1.65 – certainly cheap enough! However, the time to build the initial environment, create the scripts and the time lost during the startup and shutdown of the server each time made me think there may be a better alternative. Read more on subversion hosting in the second part.

I’ve been doing some research over the last few days and have found that niche search is far from solved. Many market verticals are lacking an effective aggregator for timely content. The tools everyone needs are known, but they haven’t been implemented. Instead of true solutions, we find scraping sites that just suck in content and spit it out un-validated, unverified and unorganized. Because of the high costs of building robust tools, what results is a poor, incomplete collection of information, which quickly becomes stale.

That coupled with inaccurate supporting information, primarily due to high data-set costs, provides a less than desirable experience for users. Mahalo saw this in the generalized search market and put into place human editors who comb the internet looking for the best information and vetting user submissions. The result are guides of information that are informative and helpful.

How long will it be before we see Mahalo style sites targeting niche verticals? Who out there is working on a white label Mahalo which can be reused in these smaller markets?

Some five or six years ago, I implemented an email data transfer process using PGP to handle file encryption leveraging self signed keys that relied on the freely available tools to handle the encryption of the data. All was good (or good enough). The data was secure, everything worked and I was happy.

Fast forward to present day.

I need to update the code so I can deploy on a new environment so I turn back to the company who was so much help last time and what do I find? The “free” product has been replaced by a 30 day trial version that cripples some features (actually locking you out of your data BTW). Licensing is ridiculously hard to understand – suddenly I realized why so many individuals and small businesses don’t bother to encrypt their data. Hell, many organizations don’t even do it as often as they should.

Unfortunately I can’t alter this process to veer from PGP, I am no longer the owner of it, but I think for all future implementations I’ll use GnuPG. I highly recommend anyone dealing with sensitive files make a strong effort to stay away from the larger vendors until they make a serious effort to make encryption available to individuals freely (and easily).

Cutting through wood with a hand saw is a hard job. Pushing the blade back and forth through the wood slowly but surely making the cut deeper and deeper until it’s completed is a time consuming, labor intensive task. Once completed, there’s almost always another piece that needs to be cut.

Often in our daily tasks, we find ourselves pushing and pulling the saw blade back and forth, moving a project along towards it’s expected completion date. As soon as that project is completed, often before it’s completed, another one presents itself and requires our attention. This is very similar to cutting through the log. We often feel taking even the shortest break will put us off our course, the work will pile up, and we’ll never get it all done.

You’re wrong. Sharp tools cut faster.

Franklin Covey has taught us this already. Taking a 10-20 minute break to sharpen the saw can make the cutting process go faster. Increasing productivity and of course resulting in completing the tasks quicker. Taking the time to obtain the needed skills is a hard discipline, but it must be done and done now!

Think about your current and future task list. Now, take 10 minutes to think about your most mundane task and research to see if there’s a book, course, webinar or other resource you can tap to learn more about it. Even the most cursory overview will teach you some nuance that improves your productivity – sometimes – you’ll learn your understanding of a topic is really far more basic than you realize. The added time spent on yourself will help you to complete that mundane task perhaps a little more efficiently, giving you more time to complete the task at hand.

What saw am I sharpening?

I’ve been writing simple, and some not so simple, CRUD (CReate Update Delete) SQL for most of my professional career. I find it boring and tedious, but a necessary evil in the Web 2.0 world. This week I started reading Refactoring SQL Applications by Stéphane Faroult and Joe Celko’s Thinking in Sets. What I found was that many applications I’ve seen (and written) hardly tap the power of SQL and that there are some major mistakes that the PHP/Perl/ColdFusion/ASP.NET/C# programming manuals proliferate in the over simplification of relational database design. I’m now realizing that the majority of code I’ve seen and worked on is actually using SQL as a giant persistent hashmap!

These books are causing me to think about not just the CRUD statements differently, but how my applications interact with the persistent storage engines web apps interact with every day.

Take a few minutes today to sharpen your saw.

Mashable had a great piece by guest author Michael Cerda for employees of small companies to help know when to jump ship. I highly recommend reading this for anyone working at a small(ish) company. I’ve seen evidence of this at a number of the places I’ve been in or worked with (not just tech companies either) and would be wary if you see anything in your current situation that you can relate to.

Brian Nadel recently wrote a nice article comparing broadband service from the big three wireless carriers, AT&T, Verizon and Nextel. Normally I have good connectivity within my house through my local cable carrier and don’t mind paying for airport WiFi once in a while. I’ll be traveling to the Northern Virginia area in the next month and want to have a broadband connect card as an option for connecting wirelessly. The $300/night hotel I’m staying in doesn’t offer free WiFi, instead it’s a $10/day T-Mobile hotspot. Add that together with the fees from the airport (Boingo) it comes up to a hefty $80 for internet service while away from home for a single week!

In Brian’s article he mentioned that AT&T had an $80 option for those adverse to a 1 or 2 year commitment. Intrigued I stopped by my local AT&T store and was given a bunch of misinformation that differed even from the information I was able to read on the AT&T site. 2 calls to AT&T later I was still unable to get the $80/month plan mentioned. I asked my sales consultant if there was an option to add it onto an existing line of service which could be upgraded and downgraded as needed based on my travel patterns and still no luck. At least a 1 year commitment is required. I was able to get an offer of a month to month with no discount on the card plan (which is to be expected) for $29.95/month with a 10Mb data allowance. Unfortunately the overage charges are outrageous ($0.06/kb). A single webpage might end up costing $2 to $3 to view!

There are companies that are catering to my more specific needs – a short term “rental” but they’re making a hefty profit on the service. I found two services, one which I wouldn’t even think about giving my credit card information to. The other, Rovair, seems to have a nice product offering, comparable with purchasing WiFi cost wise – but of course with the added convenience of your own dedicated connection.

Verizon and Nextel only offer plans with contracts as well. Verizon does have a 1 year options and Nextel locks you in for 2. So now I’m back to considering my options with pay as you go WiFi and spotty EDGE connectivity through my paired Blackberry and probably the most economical – going without <shudder />.

Nothing like a nice fluff piece in the monthly issue of Fortune Small Business, but I figured I’d check and sure enough Rochester, MN made the list of 100 Great Places to live and launch a small business (although with 100 cities – most major metro areas do). The winner was Bellevue, Washington. San Jose, CA ranked 66th. Other Minnesota cities with honors were Minneapolis, MN at #23 and St. Cloud, MN at #75.

Recently I had the opportunity to purge over 15 years of saved papers, files, magazines and other junk. With the family out of town for a few days I sat down with the goal of getting myself inline with GTD and purging the backlog of files. I had made an attempt about 6 years ago, thinning out old bank statements, but now it was time to get serious. With all of my necessary organization items at hand I started going through the files, sorting them into 2 boxes: one to shred and another to recycle. After many hours of processing all of this backlog, I ended up with a box 16″x16″x16″ (~41lbs.) of potentially sensitive papers.

I own a light weight shredder that will process 5 or 6 sheets at a time without much effort, but I didn’t want to dedicate an entire day or more to feeding paper and emptying the impossibly small receptacle that it’s attached to. I began the quest for a shredding service online by filling out 3 different forms for local shred on site services. Only one bothered to email me back (although another did put me on their email SPAM list) with a proposal. They actually recommended their ship and shred service as it would cost about 1/2 as much as the on-site document destruction process. With no other alternatives easily at hand, I opted to give it a try. I filled out the online form, provided some information about the contents of the box and where I live – the whole process only took about 10 minutes at the end of which I had a FedEx label to ship the box to St. Paul, MN about an hour away. The entire charge was right around $45 and was good for up to 65lbs.

I then used FedEx’s online tool to schedule a pickup for the package and 24 hours later it was gone. Checking the rates with FedEx’s site, the shipping costs for the box was around $15 (if I had shipped it with my corporate account) so the shredding portion fills out the rest at $30. Of course they’re saving a few bucks because the box is lighter and they presumably have worked out a better rate with FedEx – but I digress. This is considerably cheaper than even hiring someone at $5/hr to feed the paper into my shredder for me (which may or may not have gotten done in the 9 hours that would have paid for). According to the site, once it’s all done I’ll get a certificate of destruction that I can print from the website. Good to know if your a small business needing to purge some documents in compliance with federal law. As I see it, it saved me countless hours of shredding and I’ll use them again if I have a large quantity of documents to shred.