Search on the web began with curation. People found something interesting, created directories pertaining to that interesting thing, and made those directories available to other web explorers. Within those directories, you and I could find what we were looking for. Soon, however, the web exploded in popularity. The amount of information being published each day couldn’t be kept up with by the organizations (Yahoo) that attempted to categorize it.
Up stepped Google with its algorithms. Google created equations, and unleashed minions called spiders that crawled the web and organized information based on keywords. The algorithms grew, advanced, got smarter. They included keywords. Then they began ranking sites, putting them into hierarchies based on authoritativeness. A site’s authority was based largely on links: the more sites that linked back to yours, the greater its authority, the higher its position in the rankings on Google. More and more, however, these algorithms have been exploited, and many technology analysts have begun to point out an increasing degree of corruption in the search giant’s organic listings.
“Any algorithm can be gamed; it’s only a matter of time,” wrote Paul Kedrosky in his blog, Infectious Greed. Google “has lost its mojo…in part because it is, for practical purposes, reverse-engineered, well-understood and operating in an adaptive content landscape.”
In essence, Google’s algorithms cannot keep up with the human forces that continually strive to figure them out, crack them, and exploit them for gain. The quest for the all-too-coveted spots atop Google’s listings has rendered the search engine’s answers increasingly useless.
Principally indicative of Google’s deteriorating search returns are the growing numbers of content farms in top-ranked positions. These are companies that employ large numbers of unprofessional writers to create articles that target specific keywords. The articles tend to be lacking in substance and credibility, and are therefore less helpful to searchers. Michael Arrington of TechCrunch refers to them as fast-food content. And it’s obvious they are prevalent when Google’s commercials–the ones where they build a little story by showing what a person queries on Google–turn up content farm after content farm.
Other harbingers exist as well (an entire industry built around search engine optimization being one of them). In Google’s case, the clutter is amplified because the SEO industry all but ignores Bing, Yahoo, and Google’s other feeble competitors. Indeed, the colorful metaphors envisioned by technology bloggers who’ve set out to highlight the disrepair of Google’s search are many. (Here’s a roundup in The Atalntic. My favorite is, “Google is a Flea Market with Sleazy Salesmen).
The Future of Search
So where does that leave search? It’s hard to tell, but some envision a new type of search engine, one that relies on a hybrid of algorithmic intuition coupled with crowdsourced curation. The two start-ups that come most immediately to mind are Blekko and Quora. Of the two, Blekko is the more traditional search engine. Blekko allows you to perform regular searches a la Google, but it also features what are called “slashtags,” which permit searchers to filter the information according to criteria. (For a complete write up on Blekko see here). Quora, on the other hand, solicits questions and answers from its members. Instead of specific answers generated from an algorithm, users can ask questions and get answers from the Quora community. This, of course, has its pitfalls (random jokers answering questions). But the engine does a decent job of getting expert answers too. Quora members are entreated to include their qualifications, which appear in plain sight below their names, so answer-seekers have a means of quality assurance.
What would be an interesting model is a search engine that successfully meshes algorithms and crowdsourcing: a blend of Quora with Blekko or Google. This could help cull content farms and SEO spam but keep spider bots running sorties for content across the entire web.