Surveillance vs Civil Liberties

The recent revelations about large-scale data mining in the US makes this a good time to reconsider the issues of surveillance. The most recent public plan for the use of new technology for surveillance was called "Total Information Awareness" and was canceled by the US congress. It has apparently reappeared in other departments.

There are three groups who are traditionally "of interest" to security operations. They are known political or social activist groups, foreign security and political operations, and unknown activists.

Let's treat each separately, in reverse order.


The third group is one the one now being most discussed in the press. Foreign and domestic "terrorists" are portrayed as presenting a unique threat to civilized countries. This threat is used as the rational for expanding into new areas of surveillance. The latest data mining efforts are just an enhanced version of all intelligence. If we only "connect the dots" from diverse sources we can catch people before they have a time to act. Recent history indicates that this has not worked well. We have had attacks in Spain, the UK, Indonesia and several other places just in the past year or two. None of these impending attacks was uncovered. Why not?

Here we get into the awful truth of all data mining and pattern recognition techniques. They only work when one knows what to look for. I spent the bulk of my career on looking at pattern recognition techniques in scientific research literature, but the issues remain the same. My favorite example is to find Einstein's original paper on Special Relativity published in 1905. Nowadays this is easy since subsequent authors in the field refer back to it in their citations thus making a connection obvious. Less successful is to try to use key words. This fails because the terminology that developed for this field was not yet in use when Einstein wrote his paper. So key word correlations fail.

Similar problems arise with uncovering unknown activists. The connections between them may exists, but where is the starting point. If all are "newcomers" there are no "citation" links. In addition they will use a made up vocabulary to communicate, hence no key words. The shoe bomber and the millennium bomber were both caught by chance.

The next problem with finding patterns of unknown activists is the volume of material that must be scanned. The 9/11 commission stressed the fact that there were links in the gathered data which showed connections between the aircraft hijackers. They assumed that the failure to make the connections was do to poor management, or "barriers" between intelligence agencies, or some other organizational failing. In point of fact the real reason was that there is just too much information and finding the parts that are significant is essentially impossible. We do not know if they have prevented any actions, since they won't talk about their successes, but if the system is not 100% effective than it is ultimately a failure in the eyes of the public.

Foreign Governments

The second group to be watched consists of foreign governments and their agents. This is usually considered the domain of the CIA and similar agencies. Here the difficulty lies with the methods needed for information gathering. The other countries are aware of the attempts to penetrate their intelligence and take active steps to conceal their information. As a result there is much opportunity for the deliberate releasing of dis-information by all parties. In addition with large governments there may be contradictory actions being taken at the same time. Not all branches have the same goals, or are even in control of events within their own administration. Once again we don't know about the successes, but the failures are even more outstanding, given the amount of resources devoted to the task. In the recent past we have seen intelligence failures about the existence of WMD in Iraq, the status of nuclear programs in Iran and North Korea, and going back slightly further, the total unpreparedness for the collapse of the Soviet Union. With 70 years of spying the CIA missed the biggest story of the last quarter of the 20th Century!

Whether the poor quality of intelligence reflects the difficulty in obtaining the data, or the difficulty of synthesizing something from it, or the existence of ideological biases which prevent accurate assessments is something for intelligence experts to debate. For our purposes it is enough to know that foreign spying is ineffective much of the time.

Domestic Activists

Lastly we come to the group of known activists. Unlike the prior two groups these people are generally not trying to hide their existence. Indeed one of their primary goals is to change public policy so as to be in conformance with their goals. Groups like the Sierra Club, or the ACLU depend upon publicity to help achieve their programs. The same is true for politically oriented groups, whether the Green Party or Gold Star Mothers for Peace. Some try to keep their membership lists private, but that is about the extent of their attempts to remain unknown.

For this class of groups, surveillance works well. The groups are known in advance, their publications and other activities are easily monitored and connections between them are easy to determine. It is this group that secret police have been most successful in following. There is a long history of abuse by police forces who monitor groups which advocate changes in society. In the USSR, for example the KGB was a famous example. In the beginning of the 20th Century the US government cracked down on labor organizations like the IWW (Wobblies) and the Palmer raids of 1919 imprisoned between 5,000 and 10,000 people for opposing WWI. During the 1950's the US went after communists and communist sympathizers. Once again a group of fairly outspoken activists was easily monitored.

Along with monitoring the secret police commonly infiltrate groups. This is to serve two purposes. The first is the obvious one of having an insider to gather information. The second, and more insidious, is to plant provocateurs within the group. This is done either to lead them into breaking the law by making suggestions for action, or sometimes to actually provoke such actions during demonstrations in order to give the police an excuse to act. There is recent evidence of such actions being undertaken at anti-war rallies in New York City and at the Republican national convention.

The net effect of all this is that governments use the excuse of foreign threats by unfriendly countries, or by "terrorists", to beef up the secret police sector. The fact that these goals are poorly met is not important, the real goal is to use this as a cover for suppressing political dissent, or at least monitoring it, to make sure it doesn't become too effective. At this task the secret police operations are usually quite effective, at least in the short term. A single historical example will suffice.

Starting in the 1870's there was a social upheaval in Russia. Starting in 1861 with the freeing of the serfs a new social dynamic started to develop. By the 1880's there were numerous groups working to overthrow the Czar. Indeed Alexander II was successfully assassinated in 1881. During this same period the secret police became much more powerful. Numerous political prisons were set up and the number of political activists that were arrested, tortured, executed or sent to Siberia rose precipitously. The activists were only partially trying to remain undercover. Many printed underground newspapers and pamphlets which they distributed while trying to elude the police. Their semi-public stance made them easy to trace. However, despite all the actions of the secret police the social pressures which had caused the rise of these political groups proved too strong and the entire Russian social system collapsed in 1917.

It is possible for the ruling elite to use a secret police mechanism to suppress dissent, but if the reasons for the social unrest are fundamental enough change will happen eventually. In the Soviet regime which followed the Czars there was a 70 year history of secret police and suppression of popular dissent. The result was 20 million people killed or imprisoned, a barely functioning economic system for most of the period, and, ultimately, another social collapse: this time of the USSR empire.

The US is now seeing the first stages of a worldwide realignment brought on by the population explosion and the decline in natural resources. This is leading to a downward pressure on the US standard of living and "superpower" status. The US will not be able to extract wealth from weaker powers as it once did or control foreign countries by brute force. The response seems to be a last gasp attempt at militarism combined with a rise in internal police powers and a weakening of civil rights. This trend, judging from history, can only lead to personal misery for many, economic decline and social unrest. An enslaved people do not make an effective workforce.

We are now in the midst of another one of those debates over whether the ends justify the means. Those who support increased surveillance ignore the lessons of history which caution about the effectiveness of enhanced secret policing. These people are either Utopians, believing this time it will work better, or ignorant of history, or members of the class which is feeling threatened by social change. The ultimate irony is that once a police state is functioning properly it is the early supporters who are most likely to suffer persecution. It is not that they are less loyal than formerly, it is just that they know too much about how the regime gained power and constitute a potential threat to their continued dominance. Once again history provides some examples: Leon Trotsky and the Soviet show trials of 1938 are a good place to start.

Only an open society can be a free society

Click here to see all my essays in context.

If you have any comments or for further discussions email me at
Copyright © 2005 Robert D Feinman
Feel free to use the ideas, but the words are mine.