Bletchley Park was now an empty shell in the Bedfordshire countryside. Barbara Abernethy, who had worked as Denniston’s personal assistant, recalls: ‘We just closed down the huts, put all the files away and sent them down to Eastcote. I was the last person left at Bletchley Park. I locked the gate and took the key down to Eastcote. That was it.’[68] Much of the machinery was broken up, including examples of the mighty ‘Colossus’ computational machine. However, Professor Max Newman, who had been central to its development, managed to secure two ‘Colossus’ machines for his new computing department at Manchester University. These were transported by the Ministry of War Transport at the price of thirty-four shillings a ton. Newman offered to send a junior university lecturer down ‘to sit on the van’ to make sure that the precious machines were not damaged in transit.[69] In fact, this was not quite the end of Bletchley Park’s active life in sigint, since GCHQ continued to use it for training courses as late as the 1960s.

The intention behind GCHQ’s post-war move to London was to service the centres of power in British government. Accordingly, in the autumn of 1945 Travis took the opportunity to look at how the sigint product – the ‘blue jackets’ or ‘BJs’– circulated around Whitehall. The Foreign Office was a big customer, receiving three sets of BJs daily. One set stayed with Ernest Bevin, the new Foreign Secretary and his war-weary Permanent Under-Secretary, Cadogan, ‘for their immediate information’. Another went to the Services Liaison Department, which worked closely with the JIC. The third went to the main departments. Virtually everyone in the operational core of the Foreign Office habitually saw BJs, but they were always kept separate from other documents in special boxes which were locked up overnight.[70]

In MI5, the ritual of sigint security was closely observed. Distribution was presided over by the redoubtable ‘Mrs Arbuthnot’, who recorded everything meticulously in her log. Security of BJs seems to have been at its most lax inside SIS, where batches of them circulated around sections for as long as six weeks before being returned. Nor were they properly logged. GCHQ noted that, quite uniquely, inside SIS BJs were never treated as requiring special security measures, and indeed in some cases had ‘found their way into the General Office for filing’. This broke the cardinal rule that sigint was never to mix with ordinary paperwork.[71]

The first major international crisis of the Cold War era was not long in coming. In June 1948, the Soviets decided to block road and railway access to the western sectors of Berlin, which were controlled by the British, the French and the Americans. The Berlin Blockade was defeated by a massive airlift of some four thousand tons of supplies a day. Hidden amongst the innumerable supply flights heading to Berlin were anonymous but highly secret aircraft collecting sigint for GCHQ, which provided some of the best intelligence during the crisis. Even before the crisis ended in May 1949, GCHQ had already been working hard on the ‘Russian problem’ for almost five years. The early onset of the Cold War had not only provided GCHQ with new targets, but had helped to perpetuate the wartime alliance between British code-breakers and their counterparts in allied countries. This, as we shall see, was fundamental to the post-war success of GCHQ.

4
The KGB and the Venona Project

… Paul [Guy Burgess], and Yan [Anthony Blunt] consider that the situation is serious.

Message from the KGB station in London to Moscow, February 1950[1]

The ‘Venona Project’ was possibly the most astounding code-breaking effort of the early Cold War.[2] Employing perhaps no more than a hundred people, it exploited a weakness in KGB communications and decoded some of the messages sent by Soviet intelligence. As a result, it revealed key Soviet agents and illuminated the unexpectedly vast scope and scale of KGB espionage in the West during the 1940s. This material was so significant that even though no new messages were collected after 1948, British and American code-breakers continued to work on the residue until October 1980. Initiated by the Americans, Venona collected new partners – first the British, and later the Australians, the Canadians, the Dutch and even the ‘neutral’ Swedes. It is justly famous for revealing some of the ‘giants’ of Russian espionage, including Klaus Fuchs and Donald Maclean, but the vast pool of messages that remain unsolved is also significant. Even now, it points unambiguously to many other cases yet to be resolved.

Anxiety about the compromise of sigint secrets was always central to the code-breaking profession. Back in 1927, Prime Minister Stanley Baldwin’s infamous exposure of the reading of Soviet high-grade systems in the House of Commons had taught a whole generation of interwar code-breakers the price of careless talk. Thereafter, anxiety about the Ultra secret persuaded more than ten thousand people to keep their wartime vow of silence for decades. However, Venona introduced an even greater level of paranoia, since it hinted at the possibility of hundreds of Soviet agents active inside the governments of the West, some in high positions. For this reason it is unlikely that Venona was ever made known to President Roosevelt, and it was three years before his successor, Harry Truman, was let into the secret. Clement Attlee, Britain’s first post-war leader, was not told until a major security case made it unavoidable in late 1947.[3] Nevertheless, the Venona project was compromised by several Soviet agents within five years of its initiation. This did not entirely negate its value, since the Soviets could not prevent the West from continuing work on the immense volume of KGB messages that had already been collected during the 1940s, patiently revealing the names of important agents. In the late 1950s, for example, GCHQ suddenly began to have success with Soviet Naval Intelligence messages, having used a new analytic technique.[4]

The extreme secrecy of the Venona project was its Achilles heel. Although the material often pointed to the identity of Soviet spies in the West, for security reasons it could not be shown to those arrested to persuade them to confess; still less could it be produced in court. Any sensible defence lawyer would seek to probe the nature of Venona, not only exposing its fragmentary nature, but also revealing sensitive secrets about sigint. Therefore, once spies had been identified by Venona, they had to be either caught red-handed meeting with their KGB controller, or successfully interrogated and broken. The result was a game of cat and mouse in which the mouse sometimes got away. In 1951, Donald Maclean, Guy Burgess and John Cairncross were among those who escaped by a whisker. Remarkably, Theodore Hall, an American Communist spy within the atomic programme, also brassed it out, despite close interrogation, escaping what would almost certainly have been death in the electric chair.[5] By contrast, in 1950 the atom spy Klaus Fuchs succumbed to repeated and patient questioning by MI5 after his arrest. He told his interrogators that he ‘supposed he would be shot’, and was pleasantly surprised when he wasn’t.[6]

Venona revealed the security-minded nature of the Soviets. Much of their traffic was encrypted using a one-time pad system. This was time-consuming and slow, but they were willing to put in vast effort to protect their communications. This required huge volumes of tear-off pads with sheet after sheet of random numbers. The difficulty of generating thousands of sheets of truly random numbers should not be underestimated, and no one is clear how the Soviets made them. One individual has recalled a room full of women simply shouting out any number that came into their heads, but this seems improbable. Others have described devices not unlike lottery machines, with numbered balls. Whatever system was used, the logistical difficulties of generating many thousands of one-time pads and distributing them proved too much for wartime Russia.[7] Some time in early 1942, with Moscow on the verge of evacuation and much of Soviet industry badly dislocated, operators began to run out of pads. The KGB department that printed them committed the fatal error of reprinting twenty-five thousand pages. This made a small proportion of the messages, which should have been unbreakable, vulnerable to cryptanalysis. Far worse, they were sent to KGB units as well as to military and diplomatic users.[8]

The Venona project that exploited this mistake began in Washington. The Americans had collected Soviet messages during the war, but they lacked time to work on them. On 1 February 1943 the US Army’s code-breaking service, called the Signals Intelligence Service, began a modest effort to see if it could exploit Soviet diplomatic communications. The telegrams had been collected at Arlington Hall, in Virginia, a former girls’ school which was commandeered by the Army as its main code-breaking centre. Interest increased dramatically when it was discovered that some of the streams of traffic related to espionage. In October 1943 a young code-breaker, Lieutenant Richard Hallock, a Signal Corps reserve officer who had been a peacetime archaeologist at the University of Chicago, was looking at Soviet commercial traffic when he realised that the Soviets had committed a terrible error and were reusing their pads. This was an astonishing discovery, and thereafter Venona slowly began to unravel some of the KGB’s most precious secrets.[9]

The US Army’s head of signals intelligence, Carter W. Clarke, was the main enthusiast for Venona. Clarke was a tough, impatient, hard-drinking individual who many regarded as uncouth, but he was also a lateral thinker. Like many military intelligence chiefs in both Britain and the United States, he nurtured a deep-seated distrust of the Soviets, asserting bluntly: ‘They’re your friends today and they’re your enemies tomorrow, and when they’re on your side find out as much as you can about them because you can’t when they become your enemy.’[10] The US Navy code-breakers also began work on Soviet traffic in the summer of 1943. The fact that by the autumn of 1944 the two rival armed services were both referring to all Soviet radio intercepts by the same code name of ‘Rattan’ suggests a directive from a high level. The following year the code name was changed to ‘Bourbon’.[11]

By 1944, another talented young American code-breaker, Meredith Gardner, was busy making the first breaks into KGB traffic and even some from Soviet military intelligence (GRU). Other code-breakers were now drafted in to help. One of them was Cecil Phillips, a chemistry student who was sent to Arlington Hall in June 1943, initially to work on Japanese naval messages. In May 1944 he was switched to Soviet diplomatic traffic. He quickly realised the scale of duplication, and made a number of progressions that led to wider breaks in the cypher system used by the KGB.

However, substantial activity had to await the end of the war with Japan, when larger numbers of staff could be transferred to work on ‘the Russian problem’.[12] Some of the Soviet messages were double-encrypted, and so represented a fantastic level of difficulty. Nevertheless, on 20 December 1946 Gardner decrypted a KGB message listing the names of scientists who had been working on the wartime development of the atomic bomb at Los Alamos, known as the ‘Manhattan Project’. In spring 1947 he decyphered a message that showed that the Soviets were being given highly classified material from inside the US War Department.[13] KGB agents were rarely referred to by their real names in the messages. The British spy Donald Maclean, for example, was ‘Homer’ or ‘Gomer’. Accordingly, their identities had to be figured out from their activities and from what material they were providing to the Soviets.[14]

Early accounts of Venona suggested that the first breaks were achieved as a result of the recovery of a partly burned Soviet codebook found in Finland and sold to America’s wartime intelligence agency, the Office of Strategic Services. Stories have long circulated about how American diplomats insisted that protocol required that it be returned to the Soviets. In fact, up until 1952, the progress made on Venona was probably driven by the pure sweat of mathematics, and represented a remarkable intellectual achievement. A little help was gained by intercepting Japanese traffic that contained Soviet material purchased from the Finns in 1944. The Finns had not been reading high-grade traffic, but had learned enough to be able to sort messages into homogeneous groups, the first stage of a cryptanalytical attack.[15] It was only in 1953 that the American team realised that one of the KGB systems it was working on related to a Soviet codebook that had been in their possession since 1945. At the end of the war TICOM Team 6, led by Lieutenant Colonel Paul Neff, had seized a copy of a partially burned Soviet codebook while exploring the German sigint centre at Burgscheidungen. The Germans had themselves seized the codebook from the Soviet Consulate in Petsamo in Finland during June 1941.[16]

The big shock was revelations about espionage within the Manhattan Project.This immediately raised the question of how the material might be employed for counter-espionage. Liaison was established with Robert Lamphere of the FBI’s Intelligence Division, which had responsibility for maintaining physical surveillance on Soviet espionage activities. Venona was of immense help to the FBI, but it was not a one-way street. Occasionally the Bureau undertook burglaries of Soviet premises and photographed Soviet documents. Over the next decade, attempts were made to match material from these ‘black bag jobs’ with Venona material, but sadly there were few connections. Nevertheless, Lamphere ensured a coordinated exploitation system with the code-breakers.[17]

Meredith Gardner recalls that tight security for Venona only crept in slowly. In the beginning, everyone in the branch where it was being worked on was potentially privy to it, and ‘no special treatment was given’. This was partly because crypt-analysts had to support each other by discussing problems, since systems were often related to each other. There were people who genuinely needed to know, and there were also ‘mere busy-bodies who perhaps considered themselves consultants at large for all’. The Army intelligence liaison man, Howard Barkley, heard that ‘there was something interesting going on’ and came for a look, even though he had not been formally indoctrinated. Knowledge of Venona ‘might have been picked up almost anywhere’ in the branch at Arlington.[18]

Yet Venona was ‘so sensational’ that eventually something unusual had to be done on the security front. The focus was less on restricting the knowledge that it existed than on tightly controlling the contents of the messages. However, counter-intelligence is a messy business. What the US Army code-breakers needed in order to identify the spies was background material from other government departments – so they were forced to work closely with a gradually expanding circle of people scattered across Washington. Typically, seven copies of one Venona message, issued on 30 August 1947 and entitled ‘Cover Names in Diplomatic Traffic’, were circulated. One went to GCHQ through its liaison, Colonel Patrick Marr-Johnson. The US Army code-breakers noted that the British surrounded the material with ‘rigid safeguards’. Two copies went to the heads of Army and Navy code-breaking. Four went to mainstream Army Intelligence, Naval Intelligence and FBI. The State Department was also an important collaborator. Given that informal secondary briefing must have taken place, this means that perhaps as many as thirty people may have been given information from one circulated Venona message.[19] By contrast, an understanding on Venona was only reached with the CIA in September 1948, and detailed cooperation on active cases did not occur until 1952. Remarkably, this was six years after the American code-breakers had fully indoctrinated the British at GCHQ.[20]

It was the TICOM raids of early 1945 into Nazi Germany that had forced the British and the Americans to reveal their respective hands on the ‘Russian problem’. TICOM was an Anglo–American project, and no one could disguise the fact that material on German code-breaking successes against the Soviet Union was one of its top priorities. TICOM led to some of the greatest successes of the early Cold War. During the autumn of 1945 some of its best results were coming from a Soviet encyphered teleprinter system code-named ‘Caviar’ which was almost certainly being broken with the help of the German team recovered by Rushworth and Norland on their foray into Germany. No less important was the breaking of a number of Soviet military machine cyphers that were not dissimilar to the Enigma machine, or its widely used Swedish equivalent, the ‘Hagelin’ machine. GCHQ code-named these machines the ‘Poets Systems’. The first success was with an encoded Soviet teletype system code-named ‘Coleridge’ that gave great administrative detail relating to the Red Army in Eastern Europe. Carefully combined with material from more basic techniques such as radio direction-finding, it provided a superbly detailed picture of the Soviet Army in Europe. Thereafter, a team of GCHQ cryptanalysts led by Gerry Morgan working with an American naval team helped to decrypt another Soviet system called ‘Longfellow’. Some of the best successes against Soviet machines were the product of the brilliant mind of Hugh Alexander, combined with the enormous computer power provided by GCHQ’s American allies. In the Far East, Soviet naval codes were beginning to yield, but immediately after the war, ‘Coleridge’ and ‘Longfellow’ were the most important Soviet systems being exploited by the West.[21]

Britain was told about the embryonic Venona project as early as August 1945, and thereafter John Tiltman, head of the Cryptographic Group at Eastcote, was kept informed of progress.[22] However, full cooperation came a little later. The young American code-breaker Cecil Phillips spent six months at GCHQ’s new location at Eastcote collaborating with Philip Howse. They focused on Soviet traffic that had been collected in Australia by monitoring Moscow’s Embassy in Canberra. More senior figures such as John Tiltman did not give them much attention, since Phillips and Howse initially thought much of the traffic to be low-level consular material. In 1947 GCHQ received a further briefing, this time from Meredith Gardner, the key American analyst of the Venona messages. However, GCHQ did not set up a proper Venona office at Eastcote until December 1947, sparked by the recognition that the Australian material was actually KGB traffic.[23] Eastcote was itself in a state of permanent revolution, with sections being constantly reformed and merged, to the extent that the ‘rumblings of reorganisation’ drew comment from figures like Joseph Wenger, Washington’s senior naval code-breaker.[24] The rumblings were the sounds of growth. From an establishment of just over a thousand in December 1945, GCHQ was nudging three thousand staff by 1948, and was already looking for new premises to accommodate its swelling numbers.[25]

The British had also collected plenty of interesting wartime KGB traffic. As early as June 1943, Alastair Denniston had met Colonel Ted Maltby of the Radio Security Service, together with Roger Hollis and John Curry of MI5, to discuss ‘the interception of certain apparently illicit transmissions from this country which have been “DF-ed” to the Soviet Embassy’. (‘DF’ referred to the technique of radio direction-finding by triangulating between several aerials, sometimes mounted on detector vans.) These messages had attracted interest because they had nothing in common with the old Comintern style of transmissions, and it was noted that they might well be KGB traffic as they showed ‘great technical skill’. Collecting this material stretched Britain’s interceptor resources, since the traffic had lasted for eight hours solid in every twenty-four-hour period. Meanwhile, it was also searching for an illegal Comintern radio station in Wimbledon, using a disguised Ford Thames van with direction-finding equipment and security personnel in civilian clothes.[26]

By 1948, the Venona teams at GCHQ and Arlington Hall were small but extremely integrated. Although the British employed a different code name for Venona, calling it ‘Bride’, they adopted a standard procedure for the translations. The British cell was superintended by William Bodsworth, one of the initial team that began studying Enigma in 1937.[27] Like so many interwar code-breakers, Bodsworth was a linguist, not a mathematician, having read Spanish at Cambridge. Cheerful and possessed of a gentle humour, he was dubbed ‘Snow White’ because of his mop of white hair. Bodsworth’s team undertook much of the laborious task of trying to reconstruct the Soviet codebooks. The seven dwarfs supplied almost enough nicknames for the Venona teams: by the end of 1950, the number of people at Eastcote working on ‘Bride’ remained at less than ten. For the Americans, British input was essential both to the efforts to track down the identity of figures like ‘Homer’ and to obtaining background material to allow the analysis of the KGB’s Canberra messages.[28]

It is almost certain that the first person to alert the Soviets to the existence of the Venona project in any detail was a KGB agent named William Weisband. Born in the Soviet Union in 1908, Voldya Weisband had emigrated with his family to the United States in the 1920s. In 1940 he had changed his name to William, and had registered at the American University in Washington DC. By 1942 he was serving as a lieutenant in a US Army code-breaking unit in the Middle East. He was posted back to Arlington Hall in July 1944, and was soon working in the Soviet section. Weisband had in fact been a KGB agent since 1934, and he certainly displayed all the traits of a classic agent. Gregarious and popular, he had friends throughout what was now called the Army Security Agency, and charmed the senior officers. His reputation as a problem-solver allowed him wide access within the Soviet section, and Meredith Gardner actually recalls him looking at a list of names derived from Venona material in late 1946. Weisband was not himself identified by Venona, but seeing the messages decrypted must had made him feel queasy, since his name – or at least his code name ‘Zhora’ – was certainly buried in traffic somewhere. In 1948 the Soviets summarised Weisband’s reports that had been fed back to KGB headquarters in Moscow. They contained worrying news:

For one year, a large amount of very valuable documentary material concerning the work of the Americans on deciphering Soviet cyphers, intercepting and analysing open-radio correspondence of Soviet Institutions was received … On the basis of Weisband material, our state security organs carried out a number of defensive measures, resulting in the reduced efficiency of the American deciphering service. This has led to a considerable current reduction in the amount of deciphering and analysis by the Americans.[29]

In May 1950 Weisband was named by another agent who had been revealed by Venona and interrogated by the FBI. Although Weisband was questioned, there was insufficient evidence to charge him. There was also a fear that a court case would advertise the work of signals intelligence to other countries, which might then take steps to upgrade their communications. He was never prosecuted for espionage.[30]

Yet Weisband caused immense damage to Western code-breaking. On Friday, 29 October 1948 the Soviets implemented a massive change in all their communications security procedures. American code-breakers referred to this fateful event as ‘Black Friday’. Many Soviet radio nets moved over to one-time pads, which henceforth were not reused. Much of the procedural material that had been sent ‘in clear’, or unencrypted, between operators running medium-grade Army, Navy, Air Force and Police systems, was now encrypted for the first time. Operator chatter was banned. In the space of twenty-four hours, most Soviet systems from which the West had been deriving intelligence were lost.[31] This affected the ‘Poets’ military systems which the British and Americans had been reading successfully as a result of their raids into Germany in 1945.[32] This was the most serious British intelligence loss of the early Cold War.

For the British, Venona was full of irony. As a joint programme with the Americans it symbolised the highest level of trust. However, its subsequent revelations damaged the most important parts of the transatlantic relationship, including agreements on code-breaking and atomic cooperation. This was because in early 1950 Venona uncovered Klaus Fuchs, who had come to Los Alamos as part of the British contribution to the Manhattan Programme, but was in fact an agent for the KGB. Venona also raised serious doubts about the possibility of Anglo–American–Commonwealth sigint and defence cooperation because of the number of KGB agents identified in Australia. Directly or indirectly, Venona also exposed four of the KGB’s top agents inside the British establishment: Kim Philby, Donald Maclean, Guy Burgess and John Cairncross. The main problem for the KGB was that it did not know how many of its previous messages had been broken by the Venona project, and which of its agents had been exposed. This made it hard for it to warn specific agents. Venona also contributed to Soviet paranoia about double agents who might be planting disinformation. The KGB’s strange tendency not to wholly trust even its best sources, including the SIS officer Kim Philby, was one manifestation of this.[33]

In August 1949 Philby returned from a posting in Istanbul to London. He was preparing to take over from Peter Dwyer as SIS liaison officer with the CIA in Washington, and was briefed by Sir Stewart Menzies, Chief of SIS, together with two of his senior officers, James Easton and Maurice Oldfield. Oldfield, whose responsibility was counter-intelligence, explained Venona to him in detail. Philby’s blood probably ran cold as Oldfield observed that they had broken about 10 per cent of the KGB’s Washington–Moscow telegrams and were now searching for a British diplomat working for the KGB and code-named ‘Homer’.[34] Philby immediately requested a conference with his KGB controller, Yuri Modin. The KGB station in the Soviet Embassy in London reported the crisis that now confronted them:

Stanley [Philby] asked to communicate that the Americans and the British had constructed a deciphering machine which in one day does ‘the work of a thousand people in a thousand years’. Work on deciphering is facilitated by three factors: (1) A one-time pad used twice; (2) our cipher resembles the cipher of our trade organisation in the USA; (3) a half-burnt codebook has been found in Finland and passed to the British and used to decrypt our communications. They will succeed within the next twelve months. The Charles [Klaus Fuchs] case has shown the counter-intelligence service the importance of knowing the past of civil servants … Stanley, Paul [Guy Burgess], and Yan [Anthony Blunt] consider that the situation is serious.[35]

Yuri Modin recalls that Venona ‘hung over us like the sword of Damocles’.[36] Nevertheless, he and Philby agreed gloomily that in the short term there was nothing they could do, ‘only wait and behave with extreme care and caution’.[37] Arriving in Washington in November 1949, Philby was offered a ringside seat on Venona. He was given Venona summaries by the GCHQ liaison officer in Washington, and was actually taken to Arlington and briefed on the project in detail several times.[38] Incredibly, in July 1950 he put in a successful request for GCHQ to give him an extra copy of any Venona-related material it was sending to the Americans in Washington, so he could peruse it at leisure. In any other circumstances this would have been an espionage triumph, but it caused Philby no joy. The arrests at this time of Julius and Ethel Rosenberg, perhaps the most famous figures ever to be charged with espionage for the Soviet Union, cannot have calmed his nerves.[39]

Understandably, the Americans had initially refused to show the British the KGB Washington–Moscow traffic.[40] This delayed the search for the Foreign Office spy code-named ‘Homer’, who eventually turned out to be Donald Maclean. In 1947, the earliest period of good code recovery, analysts knew that several messages from late March 1944 began with a stock preamble and greeting. Such standard openings were a gift for code-breakers. In this case it read: ‘To the 8th section. Material “G”.’ The Eighth section was thought to receive political intelligence, and short breaks in other KGB messages showed that the material concerned Britain’s Ambassador in Washington, Sir Archibald Clark Kerr. By December 1948, further work by Philip Howse revealed that it seemed to originate from telegrams sent by Churchill. In January 1949, after a month of frantic night-time digging in the registry of the Foreign Office (a daytime search would have alerted the regular diplomatic staff), the originals were found. The circle of suspects was gradually narrowing. The final breakthrough came in August 1950, when the Americans recovered two short stretches of material that referred to ‘Homer’ being entrusted with decyphering a telegram from ‘Boar’ [Churchill] to ‘Captain’ [Roosevelt]. This pointed directly to someone in Britain’s wartime Washington Embassy, and the finger of suspicion began to circle over the heads of a very few people. Further work on the messages suggested that ‘Homer’ was married. However, it was only on 30 March 1951 that the code-breakers were sure that ‘G’ and ‘Homer’ were the same. This information placed him in New York in June 1944.[41]