You can find Part 1 of this article here.
During (and before) Alta Vista’s blazing but brief reign at the top, several search engines were working their way up to takes its place. The most influential of these were Lycos, Excite and though not strictly a search engine, one cannot but mention Yahoo.
Lycos was created in May 1994 by Michael Mauldin, a researcher at Carnegie Mellon University, USA. It was the first major search engine to use links to a web page in order to determine the page’s relevance, a technique Google would later polish to perfection. It was also the first major engine to produce web page summaries in search results, as opposed to just listing links. For a short period in 1999, Lycos became the most popular online destination in the world. In May 2000, Lycos was sold to a Spanish telecom giant for $12.5 billion. This was at the height of the dot.com bubble. Four years later, after the bubble was well and truly burst, it was sold to a South Korean company for $100 million.
Excite was created by 6 Stanford students in 1994. It would be publicly launched in [month] 1995. Soon it would be locked in heated battle with Yahoo. Competition though makes for strange bedfellows. It would soon be in merger and acquisition talks with the same Yahoo and came very close to doing a deal but a broadband company called @Home soon entered the fray and offered Excite a better deal. Soon after @Home acquired excite, everything went to hell. Excite would ultimately file for bankruptcy.
The venerable Yahoo had the most ridiculous of starts. Yahoo founders Jerry yang and David Filo were PhD engineering students in their 4th year, who grew bored with their studies and were trying to avoid having to write their thesis. To distract themselves, they soon got caught up in an internet basketball fantasy league. David Filo would write a program similar to a search engine crawler to pull up data from as many basketball sites as possible. They ended up winning the league. That experience got them seriously thinking about the internet.
Soon after, Jerry Yang would begin obsessively surfing the web, noting sites he found interesting, putting them in a list. David Filo would soon write software that automating this list and before they knew it, they had built their first product; a web directory. They weren’t even close to thinking about starting a company.
All that changed when someone offered to publish their directory on a CD. They would turn the offer down and take fate into their own hands. Today, Yahoo is still very much with us, even though it has been through many tumultuous ups and downs, but sadly it is no longer the internet darling it once was.
In 1995 Google co-founder, Larry Page was a first year PhD student at Stanford University, frantically trying to come up with a good idea for a PhD thesis. He would eventually settle on one exploring the mathematical structure of the World Wide Web. The Web is an example of what mathematicians call a graph. A graph is essentially a structure that depicts a network of any kind. It consists of nodes, which in the case of Larry Page’s project would represent web pages, and edges which in this use case, would represent hyperlinks between pages.
You can use a graph to schematically depict any kind of network, whether it is a computer network, mobile telecom network, power grid network, bank branch network, social network (hence the name Facebook Social Graph) or even a genetic network.
Specifically, he was interested in the link structure of the web. He had noticed that it was very easy to go forward from one page to another via a link on the first page pointing to the second, but virtually impossible to go back from the second to the first. This meant the owner of the second page wasn’t aware of the first page or any other page pointing to his page. He thought that would be a useful feature for the web to have backlinks, so he set out to be build a tool to remedy that. At some early point Google’s other co-founder, brilliant but unfocused, second year PhD student Sergey Brin, joined the project, where his immense mathematical talent was put to good use. They were also concerned with trying to determine the importance of a web page. To do this, they borrowed the idea of citations from academic publishing. A web page was more important, the more links pointed to it and the more those links themselves were pointed to by other links. With this, they were able to come up with a reasonably decent ranking of web pages.
At some point, Larry and Sergey noticed that their tool (called BackRub) was behaving like a search engine, so they decided to put a user interface on backrub and they started writing search queries. The results returned were vastly superior to those returned by the leading search engines of the day like Excite and Alta Vista. By this time, the idea of a company was probably in the pipeline but they would stay on the Stanford campus till 1998. The rest as they say is history…well not quite. While the search engine was a blazing success from day one, it would take them another three years to come up with a viable business model that would turn Google into the corporate juggernaut we know today…but that is a story for another day.
Okay, so that was the history or past. It’s fair to ask “what’s the future?” Well its dangerous to speculate on the future but we do it anyway. What is clearly safe to say is that, based on present happenings, search will be significantly more AI-driven in the future. AI techniques will either complement or replace traditional search engine design techniques based on mathematical graph theory and citation analysis, thus making search understand user intent better, enabling more personalization in search results, enabling natural language understanding, thus enabling search to go beyond keyword matching etc.
One major AI advance that is likely to have much on the search industry as time goes on, is the integration of Generative AI into search engines. That is something Google as begun to experiment with. This will enable search engines generate entirely new content in response to user queries as opposed to the traditional keyword matching that merely pulls existing content. The new content cab now be presented in concise summaries that make search a delightfully more conversational event.
Further off and more speculatively, as the Internet of Things (IoT) becomes mainstream, every kind of device or artifact that can house a microchip will possibly have search capability eventually built in, making search ubiquitous and available everywhere and not just on your PC or mobile phone. Lost your luggage while switching flights? Search Google. Lost your car keys, purse, phone, dog? Search Google.
Whatever the future of search, one thing that is for sure is that it is going to be more interesting than its history.
Bibliography
- Batelle, John. 2005 The Search: How Google and its Rivals Rewrote the Rules of Business and Transformed Our Culture. New York: Portfolio
