I’ve been reading “The Anatomy of a Large-Scale Hypertextual Web Search Engine”, the original academic paper in which the co-founders of Google, Sergey Brin and Larry Page, outlined their search engine and its properties. It’s a fascinating read for various reasons, not least the evidence it presents of the pair’s originality. And at the end there are two Appendices, the first of which suggests an eerie prescience about the extent to which advertising would be a malignant business model for any enterprise aiming at objective search. Here it is:

Appendix A: Advertising and Mixed Motives

Currently, the predominant business model for commercial search engines is advertising. The goals of the advertising business model do not always correspond to providing quality search to users. For example, in our prototype search engine one of the top results for cellular phone is “The Effect of Cellular Phone Use Upon Driver Attention”, a study which explains in great detail the distractions and risk associated with conversing on a cell phone while driving. This search result came up first because of its high importance as judged by the PageRank algorithm, an approximation of citation importance on the web [Page, 98]. It is clear that a search engine which was taking money for showing cellular phone ads would have difficulty justifying the page that our system returned to its paying advertisers. For this type of reason and historical experience with other media [Bagdikian 83], we expect that advertising funded search engines will be inherently biased towards the advertisers and away from the needs of the consumers.

Since it is very difficult even for experts to evaluate search engines, search engine bias is particularly insidious. A good example was OpenText, which was reported to be selling companies the right to be listed at the top of the search results for particular queries [Marchiori 97]. This type of bias is much more insidious than advertising, because it is not clear who “deserves” to be there, and who is willing to pay money to be listed. This business model resulted in an uproar, and OpenText has ceased to be a viable search engine. But less blatant bias are likely to be tolerated by the market. For example, a search engine could add a small factor to search results from “friendly” companies, and subtract a factor from results from competitors. This type of bias is very difficult to detect but could still have a significant effect on the market. Furthermore, advertising income often provides an incentive to provide poor quality search results. For example, we noticed a major search engine would not return a large airline’s homepage when the airline’s name was given as a query. It so happened that the airline had placed an expensive ad, linked to the query that was its name. A better search engine would not have required this ad, and possibly resulted in the loss of the revenue from the airline to the search engine. In general, it could be argued from the consumer point of view that the better the search engine is, the fewer advertisements will be needed for the consumer to find what they want. This of course erodes the advertising supported business model of the existing search engines. However, there will always be money from advertisers who want a customer to switch products, or have something that is genuinely new. But we believe the issue of advertising causes enough mixed incentives that it is crucial to have a competitive search engine that is transparent and in the academic realm.

In today’s edition of his regular newsletter ‘Big’, Matt Stoeller reports something Rana Foroohar, author of Don’t Be Evil: the case against Big Tech (my review of which is here), said when he asked her what was the most surprising or weird thing she learned when working on her book. “I don’t know if it’s weird”, she replied,

but the most surprising thing I leaned while researching the book was that the founders of Google, Sergei and Larry, had basically predicted the key problems with surveillance capitalism and where they would lead us back in their original paper on search, written while they were Stanford grad students. At the very end, in the appendix, there’s a paragraph where they admit that the targeted advertising business model could be misused by companies or other entities in ways that would hurt users. This is kind of a bombshell revelation given that search engines say everything they do is for users. The fact that this paper hasn’t gotten more attention makes me think people aren’t reading….which is itself part of the problem of attention capture I describe in the book.

Matt’s book Goliath: The 100-year war between monopoly power and democracy is well worth a read, btw.