This is a quick and dirty way to clean up the values in an array, removing duplicates while preserving (as much as possible) the original order of the values. This method works well for smaller sorted data sets, having less than a few hundred elements in the array. Of course this type of processing is heavily dependent on your equipment so your mileage may vary.

$arr = array("a","b","c","c","d","e","e","f","g","h","h","h","i");
 
// removing the duplicates
$tmp = array();
while(count($arr) > 0){
	$item = array_pop($arr);
	if(!in_array($item, $tmp))
		array_unshift($tmp, $item);
}
$arr = $tmp;
unset($tmp);
 
// array now holds unique values
// array("a","b","c","d","e","f","g","h","i");

You can extend this to work with larger arrays if the original data is sorted by replacing the in_array() function. The in_array() function looks over the entire set of values in the array from the first element to the last until it finds a match, so the execution time increases with each additional element you find.

You can also use this method to work with complex array elements, such as an array of associative arrays, although the process is slightly different. In this case you will use a helper array (called $keys below) to keep track of the primary key value you are de duping against.

$arr = array(
	array("id"=>"1", "data"=>"apple", "qty"=>10),
	array("id"=>"1", "data"=>"apple", "qty"=>10),
	array("id"=>"2", "data"=>"orange", "qty"=>8),
	array("id"=>"2", "data"=>"orange", "qty"=>8),
	array("id"=>"3", "data"=>"pear", "qty"=>12),
	array("id"=>"3", "data"=>"pear", "qty"=>12),
);
 
// Remove duplicate values using 'id' as the unique value
$keys = array();
$tmp = array();
while(count($arr) > 0){
	$item = array_pop($arr);
	if(!in_array($item['id'], $keys)){
		array_unshift($keys, $item['id']);
		array_unshift($tmp, $item);
	}
}
$arr = $tmp;
unset($tmp);
unset($keys);

In case your wondering why the process starts at the end of the array – using array_pop() – and work towards the beginning, for any row that is a duplicate of the prior value in_array() will find the match quickly because it is the first element in the array. This of course doesn’t provide any real benefit for values that are not sorted and any value that doesn’t match still requires a scan of the entire $tmp or $keys array.

Last night I attended CloudCamp in Minneapolis. While there was much healthy discussion about the “cloud”, one thing became crystal clear for me. The cloud means different things to different people. George Reese summed it up well, there are three distinct types of clouds: Infrastructure, Platform and Software. I took away from the discussions that this distinction wasn’t clear for many people (including myself).

Infrastructure as a Service (IaaS):
Amazon and Rackspace are the two largest players in this space, but there are other solid offerings (including ReliaCloud) that compete with them. This is very similar in concept to leasing a dedicated server from an ISP, but with flexible pricing. Keith Schacht pointed out on my post about cloud pricing models, that some providers are offering non-virtualized infrastructure on a per-hour basis. Is there any benefit to choosing a virtualized machine vs. a real machine? I think that goes beyond the scope of this discussion. Something companies need to be aware of here is that running infrastructure in the cloud doesn’t reduce the need for good system administrators and that in terms of architecture very little has changed. Developers in this tier still need to be concerned with system capacity etc. The upside is many of these problems are well understood and the solutions for dealing with them are common place.

Platform as a Service (PaaS):
Salesforce and Google App Engine run platforms which you can build services on. These providers abstract everything away so that product can become the focus. Designing products for platforms doesn’t require an in-depth understanding of the sub-systems. Developers don’t need to know if MySQL, Oracle, MS SQL Server or some other storage engine are handling the data storage layer, they can just trust that the data is being stored and retrieve it when they need it. Of course this model has limitations and anyone building a product would be well served to learn about the best way to leverage the platform efficiently. The drawbacks are obvious as well. Google is an extremely reliable provider, however, they do have downtime. It’s is also extremely difficult to migrate platforms. None of the vendors currently provide import/export style functionality for data.

Software as a Service (Saas):
SaaS isn’t consumer offerings, such as those from 37 signals. Those are products or applications. What I’m referring to is a lower level software like MySQL, SQL Server, Amazon’s SQS and so on. Leveraging these services provides a unique opportunity to use the best solutions for each task, instead of a complete vendor lock-in. Developers interact with the tools and sub-systems they’re already familiar with. What the SaaS vendor does is abstract the management and scaleability tasks. Unfortunately this has a dark side. Reliance on multiple providers requires building systems that degrade gracefully when any single sub-system is no longer available. Zynga, developers of the massively popular Facebook game Farmville, build their scaleable systems in the cloud using the notion of degradeable services. Architecting the solution such that it’s dependence on external systems can be dialed back on demand. Building this into products up-front requires a different way of thinking about application design. Someone raised the point last night during the breakout about architecting for the cloud, that these are the same problems that were being solved in the 70′s. Designing networks of loosely coupled systems is not a new problem. However, it is a problem that many developers I’ve met haven’t spent much time thinking about… yet.

While at times PHP seems to be capable of 64 bit math, it’s important to understand what’s really going on. Beyond 32 bit integers, PHP is silently converting your integers to floats. While this usually isn’t a problem, many of the operations you might perform on an int, such as modulus choke when attempting to convert back to a 32 bit integer internally.

This is actually the cause of the sprintf / printf issue I encountered before. The code below provides the maximum signed value for integers between 1 and 64 bit in PHP along with the result of the built in modulus operand “%” and a function I wrote mod() which doesn’t go all the way to 64 bits, but gets us a lot closer leveraging the built in data types. If you can install external modules, you might review and test the performance of BCMath or GMP which can both handle much larger values.

<?php
 
// Find out what our internal values are capable of
print "PHP_INT_MAX: " . PHP_INT_MAX . "\n";
print "PHP_INT_SIZE: " . PHP_INT_SIZE . " bytes (" . (PHP_INT_SIZE * 8) . " bits)\n";
 
// Generate an array of maximum signed 32 bit values
$ints = array();
for($pwr = 0; $pwr < 64; $pwr++){ $ints[] = pow(2,$pwr) - 1; }
 
// Generate a table of values
print "bits\t%100\tmod()\t%s\n";
$bits = 0;
foreach($ints as $int){
	$bits++;
	printf("%d\t%s\t%s\t%s\n", $bits, $int%100, mod($int,100), $int);
}
 
// (60 bit) - 1 aware modulus function
function mod($val, $mod){ return $val - floor($val/$mod) * $mod; }
?>

This generates the following table. Notice the internal value for integers is capped at 2,147,483,647 and the modulus operation goes kaput beyond 32 bits. The function provided seems to holds up through 59 bits before failing to function properly at 60.

PHP_INT_MAX: 2147483647
PHP_INT_SIZE: 4 bytes (32 bits)
bits	%100	mod()	%s
1	0	0	0
2	1	1	1
3	3	3	3
4	7	7	7
5	15	15	15
6	31	31	31
7	63	63	63
8	27	27	127
9	55	55	255
10	11	11	511
11	23	23	1023
12	47	47	2047
13	95	95	4095
14	91	91	8191
15	83	83	16383
16	67	67	32767
17	35	35	65535
18	71	71	131071
19	43	43	262143
20	87	87	524287
21	75	75	1048575
22	51	51	2097151
23	3	3	4194303
24	7	7	8388607
25	15	15	16777215
26	31	31	33554431
27	63	63	67108863
28	27	27	134217727
29	55	55	268435455
30	11	11	536870911
31	23	23	1073741823
32	47	47	2147483647
33	-1	95	4294967295
34	-1	91	8589934591
35	-1	83	17179869183
36	-1	67	34359738367
37	-1	35	68719476735
38	-1	71	137438953471
39	-1	43	274877906943
40	-1	87	549755813887
41	-1	75	1099511627775
42	-1	51	2199023255551
43	-1	3	4398046511103
44	-1	7	8796093022207
45	-1	15	17592186044415
46	-1	31	35184372088831
47	-1	63	70368744177663
48	-1	27	1.4073748835533E+14
49	-1	55	2.8147497671066E+14
50	-1	11	5.6294995342131E+14
51	-1	23	1.1258999068426E+15
52	-1	47	2.2517998136852E+15
53	-1	95	4.5035996273705E+15
54	-1	91	9.007199254741E+15
55	0	84	1.8014398509482E+16
56	0	68	3.6028797018964E+16
57	0	32	7.2057594037928E+16
58	0	64	1.4411518807586E+17
59	0	32	2.8823037615171E+17
60	0	0	5.7646075230342E+17
61	0	0	1.1529215046068E+18
62	0	0	2.3058430092137E+18
63	0	0	4.6116860184274E+18
64	0	0	9.2233720368548E+18

Nothing can stifle development faster than putting a price tag on the development tools for your product. A case in point is email provider iContact who for whatever reason requires developers to register for a $9.95 per month account to create an application on their platform. Imagine for a minute how different the Facebook ecosystem would have been if they had charged developers to integrate with Platform. Many of iContact’s competitors including those featured on the iContact website, Constant Contact and Bronto, provide easy access to their development API’s. I don’t know if there’s a correlation, but these providers have been continuing to grow while iContact’s growth has remained relatively flat over the last 6 months.

There are business reasons why it’s probably important to control what applications make it onto a platform. Quality control, security of client data, inability to handle scale are all good reasons to keep people out. If this is what you need to do – don’t offer a public API.

The spectrum of current solutions for controlling access is varied. Apple’s often criticized application review process is one way to ensure that only “quality” applications become available to clients. On the other extreme, also not without criticism, Facebook’s more capitalistic approach allows users to ultimately decide what applications to use and thus drives the ecosystem from the other direction. Interestingly, regardless of the application approval process issues, anyone can create a FREE developer accounts, explore and interact with the SDK and Facebook API and start building applications. In fact, according to Alexa, only 2 of the Top 20 US Web Properties (ESPN and Disney’s GO) don’t offer free access to their API.

I strongly advise iContact and any other parties who sell their software as a service to provide an environment that encourages developers to interact with your service or don’t bother at all. Allowing developers to add value to your clients will always result in a happier customer and may even give you a competitive advantage in the marketplace, especially one as heavily saturated as email marketing.

Over the last few weeks I fought with getting my Mac OS X machine to mount a NFS share on Linux remotely. I’ve tried different port numbers, different security models, even SSH tunneling to no avail. I’ve read mailing lists that say it can’t be done, and others that say certain kernel versions work and others don’t. I’ve been able to actually edit files for 2-3 minutes before the connection would drop for no apparent reason. In other tests, I was able to view files, but due to some permissions errors, I was unable to make changes. After parsing through the miriad of freely given advice online, I began seeking alternatives.

The key requirements for me were simple:

1. Be able to browse the directory structure using Mac OS X finder.
2. Be able to save/copy/move/delete files without being prompted for my password.
3. Be able to edit files using any editor of my choosing.

ExpanDrive

I have found the tool I’ve been looking for! ExpanDrive allows me to use sftp, basically ssh to manage my files remotely but leverage my super handy Mac tools to do the editing. This is a huge win for productivity. The publisher provides a 30 day free trial, but I knew instantly this was what I’ve been looking for. The package runs $39.95. I expect it will pay for itself in less than a week of development. There’s lots of great support through getsatisfaction.com for common questions like, “how do I access ec2?” which didn’t directly answer my question, but got me sufficient information to troubleshoot my own SSH connection issues.

I’m still a little disappointed I was unable to get NFS working as I had hoped, but this solution definitely makes my life easier.

Meerkat

Another great tool that I discovered today is Meerkat. As you’ll recall I said I was trying to use SSH tunnels to access the NFS service. After trying 8 different tools (that were all junk) I found Meerkat. The software is dead simple and provides rich access to pretty much everything I could want. Using Meerkat you can leverage an SSH login into one machine to provide access to a port bound service somewhere else. I see myself using this to grab remote access to a MySQL server parked behind a firewall via an SSH tunnel to a server that can see it. Other tools may have more “options”, but Meerkat actually works. I haven’t bitten the bullet to buy this one yet, I want to see how much I actually use it first, but at $19.95 – it’s a bargain as well and may quickly pay for itself.

I also wanted to give credit to Alan Watson who’s blog post about using ExpanDrive even put them on the radar for me. He also blogged on Meerkat which is how I found him in the first place. Thank you Alan, you made my day!

Nate Haug provides a great script for installing memcached along with some very detailed instructions on setting up a sandbox environment. I’m not using his MAMP sandbox, instead opting for the built in PHP / Apache install, so I needed to change a few things from his tutorial. My system is a fully updated Intel MacBook Pro running OS X 10.5.5 with the Xcode tools installed – YMMV. PHP is currently reporting version 5.2.6.

1. I added my revised start script for memcached.
2. The PHP version that ships with OS X doesn’t have PECL, so I downloaded the source and compiled manually.
   phpize; configure; make; sudo make install
3. Edit to /etc/php.ini: Changed: extension_dir = /usr/lib/php/extensions/no-debug-non-zts-20060613/
4. Edit to /etc/php.ini: Added: extension=memcache.so

You can skip his Apache scripts. Restart apache by restarting Web Sharing in the System Preferences.

The major changes I made from Nate’s memcached startup script were the singular instance and binding the service to localhost (127.0.0.1) only. This keeps memcached slightly more secure by only having it listen on the loopback adapter. If you need more space, just change the -m attribute to be higher; it’s measured in MB.

# /bin/sh
memcached -m 1 -l 127.0.0.1 -p 11211 -d

NOTE: As with any other service running on your system, opening a web server exposes your system to potential attack and worse. Be sure to keep production data away from your test environment. Someone at Starbucks, sharing your WiFi connection, may be surfing your development site too. Consider yourself warned!

If you develop code in PHP or any other language take 5 minutes to read this. It raises excellent points regardless of the language you author in despite it’s focusing on PHP. I particularly liked #7 Use a PHP Framework & #8 Don’t use a PHP Framework. It harkens back to point #1 Use PHP Only When You Need It. Hammers are good at driving nails, but they make terrible screwdrivers. The same can be said of frameworks which are not the panacea that many tout them to be. There is a good tool for each job – the roll of technology folks is to accurately select and implement the correct tool for the task – because not everything is a nail.

For the last couple of weeks I’ve been developing and working completely remotely. Removed from my office, removed from my resources and often while in remote locations from the car with my family as we travel from destination to destination. It’s been enjoyable to be part of everything the family is doing but it has not been without difficulty. For example, it can be hard to debug complex problems with kids unhappy in the back seat or to read a terminal session with the sun washing out the screen. It can also be challenging to be working via SSH and have the connection drop because we travel through an area without data connectivity.

Those sorts of problems aside, the one thing I miss most is my reference library. I’ve amassed a number of books that are helpful references for obscure bits of knowledge I don’t keep locked away in my mind. Last year I travelled with my books and never used them. This year, however, I’ve found myself wanting them in a couple of instances while working. While I can use Google for the majority of the code snips and shortcuts I would need, it’s a highly inefficient, to continually switch windows. I’ve contemplated signing up for O’Reilly Safari but have found digital books don’t save any time over Google and the $22.99 or more per month seems expensive for materials generally available elsewhere and locatable through Google.

Mashable has a nice collection of 16 tools for designers and developers for plugging into Firefox. I’m personally a fan of Web Developer Toolbar and Y!Slow which are both great. The original Mashable article included the other two tools I find indispensable, ColorZilla and Firebug.

OpenSocial, a group effort to create a widget platform spear headed by Google, has a few glitches that I’d love to see fixed. They are largely comparison items from Facebook, who has recently announced that it will open source it’s own platform. Regardless of the motivation, there are a couple of items I’d like to see ported from Facebook to OpenSocial.

1. Support for FBML like syntax: “Hold on a second! Standard HTML is the benefit of OpenSocial over Facebook” folks are probably thinking. While that’s true, without an effective way to capture and store user information for more than 24 hours (pesky terms of service documentation) developers are left querying for friends pictures etc over and over and over again. While this isn’t inherently a problem, if you wanted to display a 1,000 profile pictures for some unknown reason, you need to call the API a large number of times. Then generate the HTML and pass it to the client. This makes applications painfully slow. It’s great to store the numeric id (12345678) and be able to pass it back to the pre-client for parsing. The best examples are and . These really are very handy.
2. Support for FQL like syntax: OpenSocial does a great job of providing methods for gathering most of the information you would want from the social graph, but it’s lacking in the ability to remix the data in new and interesting ways (easily). OpenSocial requires all of the heavy lifting to be done on the client (or the application backend if their API allows it). Most clients have a reasonable limit as to how big data structures can effectively be and have the application still function. Processing on the applications infrastructure negates another advantage of OpenSocial which is requiring very little in the way of hardware to operate.

There are also some learnings here for Facebook. I’d like to see a few OpenSocial conventions ported from OpenSocial to Facebook.

1. OAuth Signature: To be fair, Facebook does provide signed requests, but it would be great if they’d use a standards based signature instead of their own homegrown version.
2. External JavaScript Libraries: Facebook’s FBJS is powerful and provides most functionality that developers need. It’s even been open sourced so it can be used outside of the Facebook universe. However, developers who’ve been working with jQuery, ProtoType or any of the other numerous javascript libraries have to start at the beginning again. Additionally, they may be missing the functionality in the FBJS library that they need.

Both platforms still have a ways to go in terms of making developers life’s easier and users application experiences more robust. I think it’s great news that Facebook is opening their platform more. It’s really more symbolic than anything, because they still ultimately control what they do or do not implement on their platform. Bebo is the only other social network using Facebook’s model and it still requires some re-writing for developers because of syntax difference and lack of some features.