The amount of data available of publicly online is astounding. The US Government has done a pretty good job of providing detailed data to it’s citizens on pretty much anything they keep tabs on. The census data immediately comes to mind, but there is much much more if you take the time to look for it. I stumbled across the USASearch.gov search engine while looking for some population data for cities. The search engine is not as accurate as Google unless you know the government lingo for your data, but it’s pretty good. I suspect there’s a data set there somewhere that would enhance any businesses results and the only cost to you is figuring out how it applies to what you do.

I’ve been thinking a lot about topic specific search lately. Google has a product that lets you search within a specific vertical, for example Linux, for content that’s evergreen. This used to be more open so you could define your own search, although the link eludes me right now. This of course updates with Google’s typical frequency, which can vary depending on how popular your site is. But what about time sensitive content. What about items like Twitter feeds?

Sure there’s search.twitter.com, formerly Summize, but what about including Plurk, Yammer or any of the other life streaming content into it? Friendfeed attempts to bring all this content to you, and does a good job, but what about other verticals? What if you knit and what you care about is yarn. A quick Google search for ‘yarn blog’ brings up some blogs; some are even about knitting, but generally lacking are good portals about all things yarn.

While portals have fallen out of fashion over the last 5 years, they’re purpose is not lost. Tech Crunch is certainly an important technology portal, even if it’s just called a blog. Mike Arrington has gathered one of the most comprehensive dataset about web startups and hardened internet companies around. So what about those knitters who are constantly looking for new information about yarn? Where can they turn for quick, one stop shopping, reliable information, industry trends and other bits about yarn? Nowhere! There are blogs, ecommerce stores, even industry news sites, but nothing that brings it all together as a good 1990’s .com would have.

At very least, I’m sure some tech-savvy knitter is out there putting together a nice search engine to leverage the Google (or perhaps Yahoo!) search results to create a single destination for knitting or yarn, or perhaps even needles. Is it possible to create a search engine, on the fly, for every possible vertical? Is it possible to create a portal that has content specific to every need, but only that need? The google search I ran for yarn was good, but there were people who were telling tales, often called yarns, on their blogs. Once I’ve selected the appropriate subset, I’d like to see the correct information, not all data - and the most important and most recent information at the top.

Photo by LollyKnit, courtesy of Flickr

I’ve been doing some research over the last few days and have found that niche search is far from solved. Many market verticals are lacking an effective aggregator for timely content. The tools everyone needs are known, but they haven’t been implemented. Instead of true solutions, we find scraping sites that just suck in content and spit it out un-validated, unverified and unorganized. Because of the high costs of building robust tools, what results is a poor, incomplete collection of information, which quickly becomes stale.

That coupled with inaccurate supporting information, primarily due to high data-set costs, provides a less than desirable experience for users. Mahalo saw this in the generalized search market and put into place human editors who comb the internet looking for the best information and vetting user submissions. The result are guides of information that are informative and helpful.

How long will it be before we see Mahalo style sites targeting niche verticals? Who out there is working on a white label Mahalo which can be reused in these smaller markets?