GCHQ

Today, in somewhat cramped circumstances, GCHQ struggles with some of the most difficult issues of the twenty-first century. Not only is it the leading edge of Britain’s struggle against al Qaeda, it is also involved in fundamental issues of freedom and privacy that will shape the future of our society. Over the last decade, Britain has engaged with global e-commerce and finance more enthusiastically than perhaps any other country in the world. Our porous electronic borders present their own enormous problems. Globalisation, and in particular the global communications revolution, has brought many benefits, but it has also allowed miscreants to communicate and organise anonymously. The need for GCHQ to monitor both terrorists and organised crime means that the distinction between domestic and foreign communications has less meaning than it once had. GCHQ used to be a wholly outward-looking foreign intelligence service, but this is no longer the case.

Who will rule the internet? Will ordinary citizens be allowed genuinely confidential communication? Would ID cards erode our privacy or extend our security? These are some of the questions that GCHQ ponders daily at the beginning of the second decade of the twenty-first century. Britain is already one of the most watched societies in the world, and some would argue that it is now addicted to surveillance. In 2008, Britain announced a £12 billion project to modernise the interception of telephone calls and email. The following year GCHQ announced a remarkable project entitled ‘Mastering the Internet’ that collects the details of Britain’s communications and internet traffic for security purposes. Even Britain’s Director of Public Prosecutions thought things had gone too far. Tasked with taking the lead on technological aspects of intelligence, GCHQ now finds itself at the centre of controversies that are of immense public importance. Accordingly, the time is ripe to trace GCHQ’s long and secretive journey from the nissen huts of Bletchley Park – via the Cold War – towards what now looks increasingly like a Brave New World.

THE 1940s BLETCHLEY PARK AND BEYOND

1 Schooldays

‘How wonderful!’ I said. ‘Do you mean we’re overhearing Portsmouth ships trying to talk to each other – that we’re eavesdropping across half South England?’

‘‘Just that.’

Rudyard Kipling, ‘Wireless’, 1904¹

In December 1902, Guglielmo Marconi made history by sending the first wireless radio message across the Atlantic. Remarkably, only two years later, Rudyard Kipling foretold the possibility of exploiting such radio messages to gather intelligence. In 1904 he published a short story entitled ‘Wireless’ that focused on intercepting communications sent from Morse equipment on board Royal Navy ships off the Isle of Wight. Kipling is thought of as a quintessentially late-Victorian author, but here he looks to the future, more in the manner of H.G. Wells, as his characters fret over technical matters such as induction and radio frequencies. To the readers of this fictional first instance of radio interception, the process seemed utterly magical. The Morse instrument ‘ticked furiously’, and one of the listening party observes that it reminds him of a séance, with ‘odds and ends of messages coming out of nowhere’. His companion retorts that spiritualists and mediums ‘are all impostors’, whereas these naval messages that they are eavesdropping on are the real thing.²

Kipling’s ‘Wireless’ is the first public discussion of the secret business of signals intelligence, or ‘sigint’. The magical process of extracting information from the ether would be one of the twentieth century’s most closely guarded secrets. Initially, producing ‘sigint’ only required equipment that would allow a third party to eavesdrop on a conversation broadcast by a radio transmitter using ‘wireless telegraphy’, but as this possibility became more widely known, communicators often resorted to using cyphers to keep their messages private. Thereafter, producing sigint usually required skilled listeners to capture the message and then a team of code-breakers to unscramble it. If the message was sent by cable rather than wireless, the listening-in process could be no less difficult than the code-breaking, or ‘decyphering’.

What did Britain’s code-breakers make of Kipling’s public airing of their black arts? The simple answer is that there were none to ask. Indeed, there had been no British code-breakers for more than fifty years. In the distant past, Britain had possessed a ‘black chamber’ in which skilled ‘cryptanalysts’ had broken the codes contained in diplomatic correspondence and private letters. These arcane skills resided in the ‘Secret Department’ of the Post Office. However, in 1847 this was exposed in a scandalous episode when the House of Commons heard that the Home Secretary had ordered the interception of the private correspondence of the heroic Italian nationalist in exile, Giuseppe Mazzini. Shocked Members of Parliament ordered an inquiry, leading to the closure of the ‘Secret Department’, just as the telegraph initiated what we now understand as a Victorian communications revolution. By 1904, Britain had been without a code-breaking centre for more than half a century³

The immediate origins of MI5 and its sister service SIS (often known as MI6) can be traced to scares about German espionage in 1909. But British code-breaking was not revived until the very eve of the First World War. On 2 August 1914 the British Army set up a secret code-breaking section called MIlb. Soon, specialist Army units at various locations in Europe and the Middle East were busy intercepting German radio communications. One of the largest sites was the intercept station in Mesopotamia. In December 1916 the military code-breakers of MIlb were given a fabulous Christmas present when the drunken chief of the German signals organisation in the Middle East sent all his Radio Operators a seasonal greeting using the same obvious formula in no fewer than six different codes. Up until that point the British had only been able to read one of these codes, but with these clues they could read all six. In the First World War, the Second World War and again in the Cold War, poor discipline by the human operators often proved to be the great weakness in otherwise impregnable cypher systems.⁴

The Royal Navy code-breakers, who had established themselves in the Admiralty’s ‘Room 40’, achieved even greater success. Famously, they broke the ‘Zimmermann Telegram’, a message sent from the German Foreign Minister, Arthur Zimmermann, suggesting an alliance between Germany and Mexico against the United States. As an inducement, Mexico was to be offered the return of her lost territories in Texas, New Mexico and Arizona. These revelations, made public in March 1917, were central in bringing the United States into the First World War on the side of Britain and France. The American entry into the war, together with a tightening blockade, persuaded Germany to seek an armistice the following year. The code-breakers of Room 40 celebrated with champagne. There are few more significant examples of the direct impact of code-breaking upon international relations.⁵

In 1919 the British government’s Secret Service Committee, chaired by Lord Curzon, the rather formidable Foreign Secretary, recommended that a unified peacetime code-breaking agency should be created. This involved the difficult merger of two quite separate organisations. The head of the Army code-breakers, Major Malcolm Hay, was awkward and argumentative, while his naval equivalent, Commander Alastair Denniston, proved to be suave and diplomatic. Denniston secured the job as chief of a new combined code-breaking organisation, which initially consisted of around two dozen intelligence officers and a similar number of clerical staff, and found himself installed in splendid accommodation at Watergate House in The Strand, next to the Savoy Hotel. Formed on 1 November 1919, the new organisation was given the name ‘Government Code and Cypher School’, or GC&CS, which was not inappropriate, since the leading code-breakers devoted a great deal of time to the patient training of new initiates.⁶ Both during the First World War and in the interwar period about half the staff of GC&CS and its predecessors were women, mostly in the clerical grades.

Almost immediately, GC&CS adopted a disingenuous description of its duties that would remain in place until the 1980s. Publicly, its functions were described as merely defensive; in other words, it was to assist in the provision and protection of codes and cyphers used by government departments. However, its more secret duty was to give priority to offensive activity, namely attacking the cypher communications used by foreign powers. GC&CS gradually shifted its focus to diplomatic traffic, and at the suggestion of Lord Curzon it was transferred to the control of the Foreign Office. It seemed natural that within the Foreign Office structure it should be placed under the supervision of Britain’s traditional overseas intelligence service, SIS, which recruited human spies. But a subliminal naval influence remained. The talented Chief of SIS, Mansfield Cumming (known within the organisation as ‘C’, the name by which the head of SIS would continue to be called), was a former naval officer. Cumming died in harness in 1923 and was succeeded by another sailor, the former head of Naval Intelligence, Hugh ‘Quex’ Sinclair. Naval intelligence and naval signals officers continued to exercise a profound influence on GC&CS and its successors as late as the 1970s.

The means by which Britain collected its intelligence was changing. During the First World War, much of its intelligence work had involved overhearing military wireless messages by means of receiving stations scattered around Europe, the Mediterranean and the Middle East. The shift to diplomatic traffic meant undertaking more work on encyphered diplomatic telegrams sent by cable. Each country had teams of code clerks who carefully encyphered diplomatic messages before they were sent by telegram using a worldwide network of cables. Although government cable censorship had officially ended in 1918, a private arrangement meant that all the commercial cable companies secretly handed over their traffic to GC&CS for copying. Most of the foreign embassies in London used cable companies to send their encyphered messages, and British dominance of international telecommunications networks meant that many of the world’s messages travelled over British cables at some point. Private companies such as Standard Cable & Wireless Ltd were almost an integral part of the worldwide British sigint system. This secret state-private network remained hidden until it was exposed by the journalist Chapman Pincher in February 1967 in the Daily Mail under the headline ‘Cable Vetting Sensation’.⁷

In 1925 both SIS and GC&CS were moved into Sinclair’s new secret service headquarters at Broadway Buildings, opposite St James’s Park tube station, which its occupants thought ‘more dingy than sinister’. The walls of the corridors were painted dark brown to a height of about four feet from the floor, and the ancient lifts moved between the many storeys with a slow clatter. The code-breakers were given the third floor. From here, the sigint product, which consisted of the verbatim text (or sometimes summaries) of the messages of foreign governments was distributed around Whitehall in files with special blue jackets that became known as ‘BJs’. GC&CS worked on the cyphers of many countries in the interwar period, including those of France, the United States and Japan, since they all shed light on international affairs; but the most important were those of Russia.⁸

Both MI5 and SIS, together with intelligence officers from the three armed services, were obsessed with the threat from Bolshevik Russia in the interwar period. GC&CS followed suit. There were good reasons for making Moscow the pre-eminent target. Bolshevik agents were actively seeking to subvert the British Empire, and sigint produced operational intelligence that could be used to thwart these plots. Alastair Denniston enjoyed a major advantage, having recruited Ernst Fetterlein, the Tsar’s leading code-breaker, when he fled Russia after the Revolution of 1917, and in the 1920s GC&CS was successfully reading Soviet diplomatic cyphers. Several times during that decade the British government directly accused the Soviets of underhand activities in London, making use of these intercepts and referring to them openly. In 1923, for example, Lord Curzon publicly quoted Soviet messages intercepted by GC&CS stations in India. The Soviets responded by changing their cyphers, but Fetterlein simply broke them again.⁹

However, in May 1927, a year after the General Strike, a disastrous row erupted over secret support from Moscow for the strikers and the distribution of subversive propaganda in Britain. A veritable centre for Soviet subversion was being run under the cover of its Trade Mission, located in the Arcos building in Moorgate. The building was raided on 12 May, but advance warning allowed the Soviets to destroy most of the incriminating material. The Prime Minister, Stanley Baldwin, was embarrassed. He ardently desired to break off relations with Moscow, but having failed to garner any incriminating evidence from the Arcos raid, he turned to the priceless intercepts provided by GC&CS. To the dismay of the code-breakers, Baldwin and his Foreign Secretary, Neville Chamberlain, read out four decyphered Soviet telegrams in Parliament in order to make their case. Alastair Denniston was especially bitter about this flagrant compromise of GC&CS secrets.¹⁰

Henceforth, the Soviets changed their cyphers and deployed more secure systems for communications with diplomatic and commercial missions overseas, including their intelligence stations. They now used the ‘one-time pad’ for their more important communications. The one-time pad was a breakthrough system created by an American army officer, Major Joseph Mauborgne, during the First World War and widely adopted by other powers. It involved using a sheet of random numbers to encypher a message. Each letter in the message was given a number. Each number was then added to another from a stream of random numbers taken from a sheet on the one-time pad. The result was a sheet of text that consisted simply of groups of five numbers, one after another. Recipients could decode the message if they possessed the same sheet from the same one-time pad. If that sheet was used only once – hence the name – and for a single message, the lack of repetition prevented decryption. In short, the code was unbreakable. The disadvantage was that it was slow and cumbersome, and therefore it was reserved for high-grade secrets. Moreover, vast numbers of pads with lists of random numbers were required. No country, not even the security-obsessed Soviet Union, could send all its communications by this means.¹¹

Nevertheless, after 1927, few Soviet diplomatic messages were being read by GC&CS. The only high-grade Soviet traffic that was decyphered were the messages of the Comintern, the part of the Soviet Communist Party that dealt with relations with Communist parties overseas. This effort was led by John Tiltman, a brilliant major from the Indian Army who had been running a small but successful interception effort in north India during the 1920s. In 1929 he was brought back to London to lead an expanded operation against Comintern communications (which were code-named ‘Mask’). This allowed the British government to learn of the secret subsidies paid by Moscow to the Communist Party of Great Britain and its newspaper, the Daily Worker. It also contributed to important successes against major Comintern agents in imperial outposts and international centres such as Singapore and Shanghai.¹²

Faced with the real threat of active subversion throughout the British Empire by the Comintern, GC&CS paid limited attention to military matters or the rise of the Axis until the mid-1930s. Germany, Italy and Japan were a remarkably low priority. Admittedly, a small naval section of GC&CS had been set up in 1925, and its most important work was done overseas by naval officers like Eric Nave, based in Hong Kong. From here they had ample practice at following military operations, because of the extensive fighting in Manchuria during the 1930s. Italy’s attack on Abyssinia in 1936 provided a new target for British code-breakers in the Middle East, located at sites such as Habbaniya in Iraq and Sarafand in Palestine. Remarkably, and despite the growing importance of air power, GC&CS only developed an RAF section in 1936, under Josh Cooper, a young and talented code-breaker who had joined the organisation a decade earlier with a First in Russian from King’s College London.¹³

Cyphers were important to the Axis military powers. One-time pads were slow and cumbersome. Moreover, they were out of step with the emerging new methods of warfare. Blitzkrieg, for example, required armoured forces to move forward at lightning speed, coordinating their activities with artillery and air support. So the pressure was on to find a way of making the growing volume of military radio traffic unintelligible to the enemy. Most developed countries turned to cypher machines to make their immense volumes of traffic secure.¹⁴ Complex cypher machines had been pioneered by banks and businesses – banks had long used fairly simple cyphers to keep commercial matters secret. In the 1920s, the German military adapted a Dutch invention to produce the Enigma cypher machine as an alternative to laborious hand cyphers. In fact, the first Enigma machines were sold commercially, and were widely used by banks and businesses. Enigma was what we now recognise as a ‘commercial off-the-shelf solution’ to a difficult military problem.¹⁵

The Enigma machine itself looked like an early typewriter in a square wooden box, but with a keyboard set out in alphabetical order rather than the traditional ‘QWERTY’ arrangement. As each letter key was depressed a set of lights that corresponded to the alphabet lit up, seemingly at random. The innovation was the rotors, which looked like fat metal wheels, embedded in the top of the machine. These rotated and scrambled the message in a highly unpredictable way. There were initially three – later four – rotors, with twenty-six positions relating to the letters of the alphabet. These moved round in a stepping motion that generated a cypher with an enormous number of possibilities. Moreover the complex nature of the rotation caused subtle changes in the stream of material, creating substantial headaches for any would-be code-breaker. The Germans were not alone in developing cypher machines. The British and Americans developed similar devices, respectively called the Typex and Sigaba.¹⁶

Critical to the breaking of Enigma was assistance from the secret services of France and Poland. French intelligence employed a lugubrious German agent called Hans Schmidt, who worked in the German military cypher department. Fond of the finer things in life, which the French secret service supplied to him in abundance, Schmidt divulged many technical documents about Enigma, including messages in both clear and encyphered text. He was later betrayed, and would commit suicide using cyanide procured for him by his daughter. By 1938 these secrets were being shared with the British through ‘Biffy’ Dunderdale, the SIS station chief in Paris. However, when the French gave the British material on German Air Force communications a further secret was accidentally revealed, namely that the French were also working on Enigma in collaboration with the Poles. In January 1939 Alastair Denniston took two of his top code-breakers, Hugh Foss and Dilly Knox, to Paris to meet their French and Polish opposite numbers. Eventually they discovered that the Poles had completely reconstructed the German version of the Enigma machine.¹⁷

Remarkably, by 1938 the Polish code-breakers were able to read the majority of German Army Enigma messages. The Polish breakthrough had been to train professional mathematicians to help them, together with the use of a primitive processor called the ‘bomba’ or ‘bombe’ – so named because of the alarming ticking noise it made – to find the rotor settings. One of their first ‘bombes’ was a weird contraption that consisted of no fewer than six Enigma-type machines wired together to provide rapid processing of possible solutions. Polish resources were limited, and by late 1938 new advances in the Enigma machine were running ahead of the ability of the Poles to do their calculations. But the precious secrets that the Poles taught the British were enough to continue the unravelling of Enigma. The timing was an extraordinary stroke of luck, since the talented Polish cypher bureau was within two months of being broken up by the coordinated German–Soviet invasion of Poland in the autumn of 1939. Before the Polish secret service was forced to flee Warsaw, its agents had achieved the remarkable feat of stealing several examples of the military Enigma machine from the German factory where they were made.

In the late 1930s, Britain lived in the shadow of the aerial bomber. Following the tragic fate of the Spanish town of Guernica in the spring of 1937, the presumption was that the first few days of the approaching war with Germany would bring untold destruction from the air, levelling the cities of Europe. By the Munich Crisis of 1938, Whitehall had begun to make emergency preparations. Admiral Hugh Sinclair, the Chief of SIS, was busy looking for alternative wartime accommodation away from London for both SIS and GC&CS. He soon settled on a country house, Bletchley Park, near Milton Keynes in Buckinghamshire, as an ideal location for the code-breakers. Much has been made of Bletchley Park’s proximity to Oxford and Cambridge, but in fact the availability of good trunk cable communications was the dominant consideration. Bureaucratic bickering now erupted. Although GC&CS was run by the Foreign Office, its relocation was considered to be war contingency planning, so the diplomats insisted that the military pay the bill. Predictably, the War Office insisted that GC&CS was nothing to do with it, and emergency relocation for Britain’s most valuable wartime asset stalled. In the end, Hugh Sinclair bought Bletchley Park with his own money, paying over £7,500 (more than £330,000 at today’s prices). This remarkable act of generosity allowed the first wave of evacuated staff to arrive at Bletchley on 15 August 1939. Sinclair’s largesse did not stop there. He acquired a top chef from London to provide food to the code-breakers in a restaurant in the main hall, complete with full waitress service.¹⁸

The emphasis at Bletchley Park was distinctly military. The main body of GC&CS was initially broken up into Naval, Military and Air Sections and allocated to the ground floor of the main house, while SIS was given the top floor, indicating that it still ruled the roost. On the periphery, an ever-growing collection of numbered wooden huts – including the famous Hut Three and Hut Six – were being constructed. Particular activities were associated with each hut: typically, the core of the Enigma problem was worked on in Hut Six, while its exploitation for intelligence purposes was undertaken in Hut Three. One former code-breaker recalls that the main house was soon ‘too small for more than a handful of top brass and their immediate acolytes’. So Bletchley Park’s considerable garden, with its rosebeds and delightful maze, gradually disappeared beneath the expanding penumbra of temporary structures.¹⁹ The shadow of the bomber even reached out to Bletchley Park. The radio transmission infrastructure involved elaborate aerials which had the potential to give away the site’s location from the air. Accordingly, Bletchley Park’s own radio station was moved to nearby Whaddon Hall. As the operation gained momentum, other nearby premises were absorbed. Elmers School, a neighbouring boys’ boarding establishment, was requisitioned for the GC&CS Diplomatic Sections.

Bletchley Park was Admiral Sir Hugh Sinclair’s last bequest to Britain’s sigint community. Through the early autumn of 1939 it was clear that he was terminally ill with cancer. His deputy and heir apparent, Stewart Menzies, was not regarded as a great brain, and indeed despised intellectuals. Sir Alexander Cadogan, the Permanent Under-Secretary at the Foreign Office, vigorously resisted the idea that Menzies might succeed Sinclair, and argued for someone from outside SIS to shake the organisation up. Senior SIS officers, however, did not want ‘a new broom at this critical stage’.²⁰ Cadogan noted in his diary, ‘I am not satisfied that Menzies is the man,’ but Menzies did have a crude talent for furthering his own ambitions, which he soon demonstrated. On Sunday, 5 November he came to see Cadogan bearing the sad news of the death of Sinclair the previous day. Cadogan noted that he ‘gave me a sealed letter from “C” recommending him (M[enzies]) as successor’.²¹ Lord Halifax, the Foreign Secretary, pressed for Menzies, who was finally accepted as the new Chief on 28 November.²²

The arrival of Menzies was a problem for Bletchley Park because the code-breakers were still subordinate to SIS. Under Menzies the administration of SIS was ‘chaotic’, and its headquarters was in ‘a state of upheaval’ throughout 1940.²³ Cadogan maintained his view that Menzies was a mentally disorganised intriguer who devoted more time to protecting the interests of SIS than to serious intelligence-collection. Typically, in March 1941, after Cadogan had met Menzies and the Directors of Intelligence of the three armed services, he recorded in his diary: ‘“C” as usual, a bad advocate on his own behalf. He babbles and wanders, and gives the impression he is putting up a smokescreen of words and trying to put his questioners off the track.’²⁴ Cadogan longed to see a thorough overhaul of SIS, which he regarded as an organisational basket-case. However, as the war dragged on, he had less and less time for the politics of intelligence.²⁵ Quite understandably, SIS wanted to keep all code-breaking under its wing, since it was a form of foreign intelligence-gathering. Menzies was also adamant that he should retain personal control over Ultra.²⁶ If possible, he preferred to take this material to Churchill personally, basking in its reflected glory. But he did not know how to manage Bletchley Park, and as a result it was under-resourced.²⁷ In the words of one SIS contemporary, Menzies regarded anything to do with personnel or administration as ‘dirty work’, and would go to considerable lengths to avoid it.²⁸

Bletchley Park may have been chaotic, but it was a creative and innovative chaos that allowed the code-breakers to make a fresh start in the Buckinghamshire countryside.²⁹ The head of GC&CS, Alastair Denniston, spent the autumn of 1939 making detailed war preparations. His task was to find new cryptographers to fill out the ranks of Bletchley Park. His valuable contacts with the Poles and their success with the ‘bombe’ had led him to realise that he not only needed more code-breakers, he also needed mathematically-inclined individuals. Most of the current inhabitants of GC&CS were linguists with a penchant for Latin and Greek. He now needed people who loved maths and machines, and in September 1939 he was actively scouring the high tables of Oxbridge colleges for talent. The brilliant new mathematicians he recruited included Gordon Welchman from Trinity College, Cambridge, who would run the heart of the code-breaking operations in Hut Six. He brought with him Stuart Milner-Barry from the same college, who was the chess correspondent of The Times and who eventually took over as head of Hut Six. In turn, Milner-Barry brought fellow members of the British chess team, Hugh Alexander and Harry Golombek, to Bletchley.³⁰

These arrivals came not a moment too soon. Hitler’s attack on Poland had tipped Europe into all-out conflict, and Bletchley Park was now a fully operational war station. The pressure was on to make progress against Enigma. The most brilliant mind engaged in this task was Alan Mathison Turing, who made an early and important contribution. Despite understanding the abstract problems of Enigma some months into the war, GC&CS was having difficulty in breaking any real Enigma messages, and was not delivering much product. To have examples of the machine was not enough, since the security of the messages it sent depended on the ‘key’, in other words the settings of the machine, which changed each day. Turing was sent to see the remnants of the Polish code-breaking team, now residing near Paris, to try to work out what the British were doing wrong. The Poles explained that the British had failed to think through the way in which the wiring was attached to the rotors of the Enigma machine.

In early 1940, with this further helpful shove from its allies, Bletchley Park began breaking substantial amounts of Enigma traffic. There were many different Enigma cyphers, and to distinguish them, they were colour-coded. In February 1940, Bletchley Park began breaking ‘Red’, which was an invaluable system used for liaison between the German Army and the Luftwaffe. Periodically, a change to a German cypher system would cause the British code-breakers to lose it for a while, and quite often recovering it depended on second-guessing the lazy habits of the operators. German overconfidence in the improved Enigma machine led to basic mistakes that greatly simplified the task of those whose objective was to tease out the rotor setting for each day.³¹