Google and the Dissemination of Knowledge
Because of the decentralized nature of the web the traditional methods of finding
information have been transformed. Previously people would use reference books
or established indexes to find material. These reference sources were created
by a large number of organizations with differing objectives. Some were primarily
commercial like telephone directories, some academic like Medline and some were
aiming at "lifestyle" issues like The Whole Earth Catalog.
So the person seeking information had a wide variety of approaches to pursue.
With the growth of the web this function has become consolidated into a very
few web search engines. At the present time just three have most of the traffic:
Google, Yahoo and Microsoft. This presents several problems for anyone seeking
There has been considerable discussion devoted to the commercial aspects of
search engines. Because of human nature, the items returned at the top of
the first page get a disproportionate amount of activity whether they are
relevant or not. So, many people spend a lot of effort trying to get their
sites list "first". The search engines know this and have created a lucrative
business model to provide prominent site display for a fee.
This is not the issue that I wish to discuss, however. What I'm interested in
is the access to non-commercial material. This can be scholarly, political or
even self expression. Without the assistance of search engines this material
is "invisible". It does not appear in printed indexes, and its only other
connection may be from other sites that have chosen to create a link to it.
Without being found by use of a search engine it doesn't exist. This has
serious implications for the dissemination of knowledge as online material
starts to replace print as the principal media for many.
If the search engines decline to index a certain site then it can't be found.
This can be done for commercial reasons. There is no reason why a search engine
site should expend funds on those sites for which it expects to receive no income.
So far these companies have included much non-commercial material as a way to
gain credibility and to drive viewers to their service. There is no guarantee
that this will continue in the future.
Government or Corporate Censorship
Another danger is that sites may present information that is found offensive to
the search engine owners. We see this problem already in broadcast media. Never
do negative comments about the parent company appear on NBC (General Electric)
or ABC (Disney), for example. Thus the news programs on these networks have been
compromised. Because of the interrelationship between business and government
sites that present unpopular political positions may be suppressed. The Chinese
government, for example, censors web traffic. Does this also mean that search
engines are required to remove references to sites the government is blocking?
How would anyone even know this was occurring? The search engines don't tell
what they are skipping.
Poor Retrieval Techniques
The methods that search engines use to organize the material is quite primitive.
There is a body of nearly fifty years of work on optimizing techniques for
improving the relevance of the material retrieved. One of the first lessons
learned was that Boolean searches on keywords is just about the poorest strategy.
This is a fairly technical area and if you wish to read my comments follow this link:
Search Techniques. The result of this
is that material that contains abstract concepts is overlooked. The ranking
techniques are closely tied to the popularity of the links being displayed rather
than the relevance. This may work fine for directing viewers to sites where a
keyword is highly selective. So, for example, searching on a famous person will
almost always lead to relevant pages since the name has high selectivity.
Thus, search engines appear to be doing a good job because the needs of the
majority of non-critical users are being met.
So not only are we not finding the material we are looking for, but we don't
even know what we are missing. Even in the darkest days of the Soviet Union
banned publications were circulated by being retyped and photocopied. They may
have reached a much smaller audience than officially sanctioned publications,
but they still existed and influenced the political discourse in the society.
The organization of electronic material is still in its early stages, but if
the promise of the free flow of knowledge is not to be subverted into yet
another tightly controlled method for promoting consumerism then a serious
discussion needs to take place on how to deal with the limited pathways into
Moral: Freedom of speech doesn't mean much if nobody
can hear you
to see all my essays in context.
If you have any comments you would like to add email me
Copyright © 2004 Robert D Feinman
Feel free to use the ideas, but the words are mine.