At this moment, Philby knew that Maclean had been identified. However, he also knew that MI5 would have to gather traditional evidence against him to support an arrest, so a window of opportunity existed. Philby’s friend and fellow Soviet agent, the diplomat Guy Burgess, was being sent home from Washington in disgrace after an especially embarrassing drunken episode, and Philby used him to pass a message to Yuri Modin, their KGB controller in London. On Friday, 25 May 1951, Burgess and Maclean fled from Britain on a ferry to St Malo. It was a narrow escape: MI5 had planned to confront Maclean when he turned up for work the following Monday. Once in France, a KGB contact handed them false papers which ensured that they could travel in relative safety across Europe towards Moscow. The false papers were essential, since by now every security service in Europe was looking for them. Inevitably, suspicion also fell on Philby, not least because Burgess had been lodging with him in Washington, but there was no hard evidence. Philby was recalled and forced into retirement, but no other action was taken against him.[42]

Venona also had ramifications in the British Commonwealth. In July 1947, Field Marshal Montgomery, now Chief of the Imperial General Staff, had held a meeting with Australia’s Minister for External Affairs, Dr H.V. Evatt, about joint weapons development in Australia. Montgomery noted that ‘good security precautions are very necessary’ because of the appearance in Australia of a spy who was connected to the Igor Gouzenkou case, in which a defecting GRU cypher clerk had revealed a major spy ring in Ottawa in 1946. But in November and December 1947 Venona revealed that despite enhanced security precautions, sensitive documents were regularly leaking from Canberra to the KGB.[43] These revelations soon made their way to the highest level. On 27 January 1948, Admiral Roscoe Hillenkoeter, Director of the CIA, warned President Truman: ‘Indications have appeared that there is a leak in high government circles in Australia, to Russia.’ He explained that MI5 was engaged in expansive undercover investigations to determine just where the leakages were.[44] Highly sensitive material had been passed to the KGB from the Department of External Affairs in Canberra. The Soviets considered it to be spectacular stuff, for it included copies of the ‘explosive’ future strategy papers drafted by the British Post Hostilities Planning Committee, or ‘PHP’. This was bare-faced anti-Soviet planning material, prepared with the encouragement of the British Chiefs of Staff, that had already resulted in rows in Whitehall. Anthony Eden, then Foreign Secretary, had banned its circulation abroad in late 1944, but by then it was too late, and the volatile PHP reports had already made their way via Canberra to Moscow.[45] The KGB chief in Australia considered the PHP papers to be such an important coup that he asked Moscow for permission to send them by cypher rather than courier. This was a bad mistake, for the two lengthy papers, ‘Security in the Western Mediterranean and the Eastern Atlantic’ and ‘Security of India and the Indian Ocean’, provided the code-breakers with a vast word-for-word ‘crib’ to get into other Soviet traffic.[46] Partly because it was relatively easy to identify which documents had been taken in Australia, the KGB Moscow–Canberra cables proved to be the most successful part of the Venona operation. Remarkably, by early 1948 so much progress had been made that GCHQ was virtually reading the messages in real time.[47]

London did not regard the Australians as competent enough to handle this security crisis. In February 1948 Sir Percy Sillitoe, the head of MI5, was despatched to Australia. With him came Roger Hollis, head of MI5’s C Division (later himself wrongly accused of working for the KGB), concerned with protective security and background checks, and another senior security officer, Roger Hemblys-Scales. With Courtney Young, MI5’s resident Security Liaison Officer in Australia, they persuaded the Prime Minister, Ben Chifley and Defence Minister, Frederick Shedden, to permit vigorous investigations. In July, following further discussions with British Prime Minister Clement Attlee, the Australians accepted British proposals for the creation of an Australian equivalent of MI5 later known as the Australian Security Intelligence Organisation (ASIO).[48] Sillitoe returned to London, but Hollis and Hemblys-Scales remained in Australia to set up ASIO and work on the list of Venona suspects, which numbered twelve.

Thereafter, ASIO was almost entirely focused on what it called ‘The Case’. Tracing documents quoted in KGB traffic indicated likely suspects, including a typist, Frances Bernie, who helped to run a Communist youth league and who worked personally for Dr Evatt, the Minister for External Affairs. It also pointed to two Australian diplomats with Communist leanings, Ian Milner and Jim Hill. Hollis and Courtney Young did not tell the Australians that the names came from intercepts, but the nature of the material led some of the more experienced ASIO hands to suspect sigint as the key source. Some of the suspects were referred to by code names rather than real names, and their identities could only be deduced by careful circumstantial guesswork. Milner and Hill, who were identified positively, refused to ‘come over’. William Skardon, MI5’s most experienced interrogator, made a soft approach to Hill when he visited London in 1950, trying to persuade him to ‘be sensible’ and ‘make a clean breast of it’, but Hill denied everything.[49]

The ‘Venona Twelve’ kept ASIO’s staff of close to two hundred busy well into the 1950s. Each new suspect opened a world of further associates and contacts who required separate examination. The task was difficult, since the Communist Party of Australia had long expected to be banned, and had built up a substantial underground organisation. Not unlike the Communist Party of India, seasoned by years of security attention, it had also achieved some infiltration of the police. Even the infiltration of ASIO seemed a possibility. ASIO’s staff worked around the clock watching and bugging the flats of suspect Soviet diplomats in Canberra. Each visitor was tailed and investigated. ASIO’s staff were learning the hardest lesson of counter-espionage and counter-subversion: working security cases really diligently only manufactured more leads and opened more cases.[50] Almost a quarter of the Venona messages relating to Canberra still remain classified, presumably because they relate to KGB agents not pursued or prosecuted.[51]

The British and Australians were not alone in suffering KGB penetration. Although headlines about Klaus Fuchs and Donald Maclean generated anxiety about Britain amongst the American elite, those on the inside knew Washington had its fair share of Soviet agents. Venona uncovered spies in the State Department, the Treasury, even in the White House. They included Harry Dexter White, a senior Treasury official, and Laughlin Currie, who had been a personal assistant to Franklin D. Roosevelt. This was not particularly surprising, since the vast influx of academics and scientists moving into government work during wartime had inevitably included some Communist Party sympathisers. The Office of Strategic Services, forerunner of the CIA, which had recruited heavily from the East Coast academic establishment, harboured perhaps a dozen people working for the Soviets.

Venona had profound implications for the development of the security state in America, Britain, Canada and Australia. Just at the moment when the public were anxious to throw off the claustrophobic constraints of wartime security, officials were confronted with irrefutable evidence of a massive programme of Soviet espionage. Selling strong security measures in the late 1940s was an uphill task. This was nowhere more true than in Australia. The creation of ASIO by a Labor Prime Minister, Ben Chifley, was a remarkable development. Like the British Labour Party, its Australian counterpart had historically been sceptical about surveillance, associating it with right-wing anti-union activities. In Britain too, Venona led indirectly to the introduction of detailed personal background checks, or ‘positive vetting’, for officials. British civil servants resisted the idea, but it was increasingly clear that without it, Anglo–American strategic cooperation on matters like atomic energy was likely to end.[52]

Venona represents a documentary source of high value, and has helped to resolve some of the most bitterly contested Cold War espionage cases. These include the famously controversial cases of the atomic scientist Julius Rosenberg and the diplomat Alger Hiss, who were both active espionage agents for the Soviets. In these important cases, Venona offers us what Nigel West has rightly called ‘a glimpse of the unvarnished truth’.[53] At the same time, much of the Venona material is rather fragmentary, and in 1995 it was further obfuscated by the lamentable decision of the British and American governments to blank out some names on grounds of potential political embarrassment. Some KGB code names for individuals were re-used and given to more than one person. Moreover, it is possible that a minority of the people who appear in the Venona cables did not knowingly have a relationship with Soviet intelligence officers, or were identified as possible targets for future recruitment, but were never actually recruited. The tendency of some intelligence officers to exaggerate their triumphs has also to be borne in mind. In short, Venona has provided us with fabulous revelations, but the full story awaits the moment when historians access the files of the KGB and Soviet military intelligence, or GRU, in Moscow. That will not happen for a long time yet.

5
UKUSA – Creating the Global Sigint Alliance

Much discussion about 100 per cent cooperation with the USA about SIGINT. Decided that less than 100 per cent cooperation was not worth having.

Admiral Andrew Cunningham, Chief of the Naval Staff, 21 November 1945 [1]

One of the most important legacies of the Second World War was the creation of the vast global signals intelligence alliance known as ‘UKUSA’. The signing of the UKUSA intelligence treaty between Britain, the United States, Canada, Australia and New Zealand has long been regarded as marking the birth of a secretive leviathan, a global multilateral alliance that has grown to embrace numerous countries and to command almost unlimited intelligence power. Its origins are often traced to a single landmark treaty between Britain, the United States and the Commonwealth deemed to be concluded in 1948. Indeed, the highly classified UKUSA treaty is widely considered to be nothing less than the linchpin of the West’s post-war intelligence system. UKUSA supposedly created a cosy Anglo-Saxon club sharing everything in the super-secret realm of sigint.

Remarkably, there is in fact no singular UKUSA ‘treaty’ of 1948, and none of the above assertions is true. Instead, UKUSA is less an alliance than a complex network of different alliances built up from many different overlapping agreements. It is the sum of a curious agglomeration of many understandings that were mostly between two countries only, that accumulated over more than two decades.[2] Britain and the United States concluded the main agreements in 1943 and 1946, together with a further convention in 1948. According to the historian Peter Hennessy they are still in force, and as recently as August 2006, some sixty years on, the authorities deemed them so sensitive that, after anxious deliberation, they announced that they could not be released.[3] Further agreements were added – and continue to be added – creating a complex spider’s web of cooperation. However, each agreement has its limits, and all parties have withheld sigint material from each other. In short, there is no common pooling of material. Moreover, relations between the various parties have often been tense, and latterly Washington has threatened some adherents, including Britain, Australia and New Zealand, with suspension or exclusion. If UKUSA is an alliance, its members are only ‘allies of a kind’.[4]

It is also wrong to think of UKUSA as exclusively concerned with sigint. It is, rather, a sigint and security network. Security agreements on physical control of the sigint product and on protecting the security of communications were perhaps the most important aspects of the UKUSA network. Sigint reports on particular subjects were rigidly compartmentalised and given ‘Codeword’ status, ensuring that they could only be seen by people cleared to see that series, and making them effectively ‘above Top Secret’. Venona is the best-known example of such a Codeword. Much of this obsessive secrecy was codified in a biblical tome entitled ‘International Regulations on Sigint’, or ‘IRSIG’, which had reached its third edition by 1967.[5] UKUSA was also about secretly undermining the communications security of other states, even neutrals and allies. Communications security, or ‘comsec’, is perhaps even more sensitive than sigint. The efforts of the UKUSA powers to control it have been among the darkest secrets of alliance politics in Western Europe. In short, the realm of sigint alliances is profoundly realist – at times even paranoid – with operators ‘taking what they can get’. While UKUSA might appear from the outside to represent a single powerful intelligence colossus, on the inside it was anything but unified.

The best example of allies spying on allies is provided by Finland. The end of the Second World War had not turned out well for the Finns, since their Russian enemy had returned to the Baltic in overwhelming strength. Anticipating the arrival of the Russians, the talented Finnish code-breakers decamped en masse to Sweden, complete with their relatives, equipment and support staff. There they began a veritable car-boot sale of their cryptographic wares, including the results of sixteen years of continuous work against Russian systems. The beauty of selling codes is that the same items can be sold many times over. Predictably, the Finns paid their ground rent by assisting the Swedish equivalent of Bletchley Park, the Förvarets Radioanstalt, or FRA. In the last days of the war they also sold complete Russian codebooks to the American wartime intelligence agency, the Office of Strategic Services, to Britain’s SIS and also to the Japanese. They also sold the Americans the details of the British codes they had broken, and work they had completed against some US State Department cyphers. The Americans were eager customers. This episode – known as the ‘Stella Polaris’ case because of its northern origins – underlines the duplicitous nature of friendships in the realm of code-breaking.[6]

In the autumn of 1945, even while the Stella Polaris case was ‘live’, President Roosevelt’s successor, Harry S. Truman, was engaged in the abolition of the Office of Strategic Services. Two years later its remnants would be revived to become the Central Intelligence Agency, but for now many of its intelligence officers were dispersed and its agents paid off. By contrast, Truman regarded sigint as indispensable, and secretly gave permission for the American code-breaking agencies to work on into the post-war period and ‘to continue collaboration in the field of communication intelligence between the United States Army and Navy and the British’.[7] All major countries desired the maximum world coverage. On 19 November 1945, Admiral Andrew Cunningham, Britain’s senior naval commander, attended a critical meeting of the British Chiefs of Staff. There was ‘Much discussion about 100 per cent cooperation with the USA about Sigint,’ he recorded, adding that they ‘Decided that less than 100 per cent was not worth having.’ In Ottawa, George Glazebrook, a senior Canadian diplomat, recommended to the Canadian Joint Intelligence Committee that Canada enhance her independent sigint effort in order to stake a claim in this secretive emerging cooperative system. ‘It is paramount,’ he insisted, ‘that Canada should make an adequate contribution to the general pool.’[8]

Yet a ‘general pool’ was not what emerged. Moreover, the way ahead was strewn with obstacles and tortuous negotiations. The complex package of agreements, letters and memoranda of understanding was not completed until 1953. In this process, Britain derived considerable benefit from her dominance over her Commonwealth partners and her imperial bases. GCHQ’s approach was to align her Commonwealth affiliates to create a critical mass before entering negotiations with the Americans. The story of Britain’s sigint relations with Australia illustrates this well. In March 1945, with the end of the European war looming, Edward Travis set off from Bletchley Park on a veritable world sigint tour. The possibility of transforming wartime cooperative arrangements into a post-war sigint alliance was already in his mind.[9] En route, he and his party visited major sigint centres at Heliopolis in Egypt and HMS Anderson in Ceylon. They arrived in Melbourne in early April, and spent time with the Australian code-breaking organisation there, called the Central Bureau. On 17 April they departed for New Zealand and then moved on to Hawaii, San Francisco and finally Washington. By the time they reached Hawaii they were running low on funds, and had to beg a cash advance from the Foreign Office before they could proceed further. At each stop, the possibility of continued post-war cooperation was gently raised.[10]

Relations with the Australians were somewhat awkward. Typically, London had reluctantly agreed that Sir Frederick Shedden, the new Australian Defence Minister, could be indoctrinated into the secrets of sigint, but only so he could use his power to prevent a reduction of Australian spending on intelligence. There had also been alleged leaks about intelligence in Canberra, and in September 1945 there were momentary doubts as to whether any cooperation with Australia on sigint would be authorised.[11] Indeed, by December 1945 a ‘dangerous position’ had developed, with the Australians seeming to want to go it alone with their own system; what was worse, there were rival elements within the Australian armed services. British liaison officers warned, ‘If we are not prompt to give a lead there may even be 3 or 4 rival shows in Australia with no hope of proper security.’ During the war, material collected in Australia and the Far East had often been sent back to Britain for analysis. However, there was now a possibility that the Australians might end up ‘insisting on full exploitation in Australia’. This was a situation that British code-breakers wanted to avoid at all costs, since final exploitation was power, and they wished to keep their Commonwealth associates in a subordinate position.[12]

The crucial moment in the creation of the global sigint alliance occurred on 22 February 1946, when Britain opened a two-week Commonwealth conference for ‘Signals Intelligence Authorities’. This gave them critical mass prior to concluding a deal with the Americans the following month. The attendance of Australia and Canada was a foregone conclusion, and given the significant contribution that New Zealand had made during the war to naval sigint, there were hopes that she would also join in.[13] The conference was also attended by senior officers from GCHQ’s regional centres. Bruce Keith, the commander of HMS Anderson, the massive sigint collection station in Ceylon, was there, accompanied by his deputy, Teddy Poulden.[14] At this conference Australia offered sixty-five operating teams, amounting to 417 personnel, from the three armed services as its contribution to a new global sigint network.[15] Australia was persuaded to set up a British-style Joint Intelligence Committee and, most importantly, a unitary Signals Intelligence Centre along the lines of GCHQ, which was given the cover name Defence Signals Branch.[16]

The big issue was the choice of the director of Australian sigint. The Australians fielded four candidates, all experienced wartime intelligence officers. However, Travis told them bluntly that it would be a British officer. Some Australians were affronted, but on balance Travis’s decision was probably correct, since it ensured that Australia would have good access to British sigint. Travis’s choice was Teddy Poulden, who had spent the last two years of the war as deputy to Bruce Keith, the commander of HMS Anderson, on Ceylon.[17] Poulden took over in April 1947, commanding a staff of around two hundred, about twenty of whom were GCHQ personnel on secondment. Although senior Australian sigint officers resented the fact that Poulden had his own private cypher for communicating with Travis, he was broadly considered to have done a good job. In the early 1950s he was succeeded by an Australian, Ralph Thompson, who remained in the position until 1978, making him easily the longest-serving Western sigint chief.[18] In January 1947 a further Commonwealth sigint conference was held in London, and the Chifley government gave final approval for the integration of Australian sigint into UKUSA at the end of the year.[19] However, this was delayed by the concerns raised by Verona about KGB spies in Australia. Therefore, Edward Travis imposed a ban on sending American material to Poulden’s new sigint organisation on 29 July 1948 and stopped GCHQ material a month later. Normal business was resumed in early 1950.[20]

Canada’s sigint organisation under the long-serving Lieutenant Colonel Edward Drake suffered similar ‘colonial’ treatment. Although Drake was a Canadian, his deputy was the stalwart British code-breaker and expert on Russian systems Geoffrey Evans, who arrived to take up his post in Ottawa in March 1946. A few weeks later, on 13 April, the Canadian Prime Minister Mackenzie King authorised the consolidation of a number of wartime organisations into a small post-war unit of about a hundred staff known as the Communications Branch of the National Research Council (CBNRC). A number of senior posts were filled by staff seconded by GCHQ, prompting locals to observe that CBNRC stood for ‘Communications Branch – No Room for Canadians’. By the early 1950s Drake had resolved to offset this British oligopoly by developing better relations with the US Army code-breakers.[21]

The two-week sigint conference that GCHQ had convened with the Commonwealth partners in February 1946 was a vital prelude to business with the Americans the following month.[22] Indeed, the Australians and Canadians had given GCHQ permission to negotiate on their behalf.[23] On 6 March, William Friedman, one of the US Army’s most senior code-breakers, arrived in London to complete a revised version of the previous wartime agreements between Britain and the United States. The main section of the agreement which followed this policy conference between the principals was only four pages long; however, a UK–USA Technical Conference followed in June 1946 which added many annexes and appendices. Much of this new material was about attempting to agree on security procedures for handling sigint.[24] The terms of the 1946 agreement are still highly secret. Both parties agreed to ‘pool their knowledge of foreign comint organizations’, and that in any future negotiations with other parties ‘every effort should be made to avoid disclosure of US/UK collaboration in the COMINT field’.[25] Joseph Wenger, the head of America’s naval code-breakers, accepted that the 1946 conference had only dealt with generalities, and this had generated ‘some criticism’. Nevertheless, his priority was to ‘set up the framework and establish the will to make it work’, so in his view it was a great success, and ‘laid the foundation of a very fruitful and important partnership’.[26]

For GCHQ, liaison with allies was all-important to its new status as a proper intelligence agency in its own right. By the spring of 1946, Edward Travis was operating with two deputies at Eastcote. Nigel de Grey was the senior deputy, and had responsibility for operational coordination between the five main groups at GCHQ, together with recruitment, training and security. Following the important allied sigint conferences of February and March 1946, Travis added a second deputy, a naval officer called Captain Edward Hastings who had much wartime experience of working with Canada. His responsibilities included liaison with the US, the Commonwealth and India, together with managing GCHQ’s overseas collection stations.[27]

GCHQ’s strategy for cooperation with the Americans was to rapidly reorientate its collection towards Russia. Typically, the vast Forest Moor wireless station near Harrogate in Yorkshire, with an aerial farm of some ten square miles, was switched from collecting German traffic from the Eastern Front to Russian traffic as soon as the war drew to an end. British field units in Germany, Austria and Italy joined the suborned Italians in collecting Russian military traffic. The re-established sigint stations in Singapore and Hong Kong also focused on Russian traffic, with the latter specialising in KGB messages. All this made Britain an attractive partner for the United States.[28] The core of Anglo–American cooperation was a ‘relentless attack’ on the wartime generation of Russian cyphers. Figures like John Tiltman and Hugh Alexander provided the code-breaking expertise, while the Americans provided most of the processing capability.[29]

All the three American armed services were routinely circulated with GCHQ finished product on Russia. A key instrument was the ‘Comintsum’, a digest of the latest ‘hot’ material which made its way around comint-cleared centres. London would send twenty copies of this sort of document to Washington on a regular basis, with two copies going to US Air Force intelligence, two to US Army intelligence and so forth.[30] On Russian military targets at least, the British and Americans operated smoothly as one machine. A very high priority was given to joint planning for the use of nuclear weapons in any future war. As early as 28 April 1948, General Charles Cabell, head of US Air Force intelligence, reviewed the intelligence arrangements in support of the current emergency atomic strike plan ‘Operation Halfmoon’. ‘At the present time,’ he noted with satisfaction, ‘there is complete interchange of communications intelligence information between the cognizant United States and British agencies. It is not believed that the present arrangements … could be improved.’[31] This was cemented by a further Anglo–American agreement on communications intelligence signed in June 1948.

However, the sharing of material on other parts of the world remained selective, reflecting the political tensions of the moment. In 1948, even while the UKUSA alliance was gradually being drawn together, Britain and America were at loggerheads over Palestine and the emerging state of Israel. There was anxiety in London about sharing intelligence on the Middle East with the Americans. On 15 February 1948, Britain’s Joint Intelligence Committee discussed the problem of circulating its own reports, which included material from SIS and GCHQ, to the newly formed CIA. Although British intelligence representatives in Washington were not aware of a specific ‘pro-Zionist bloc in the Central Intelligence Agency’, nevertheless they asserted that ‘Jewish sympathisers were no doubt included in its establishment’, and complained that there had been leaks. William Hayter, the Chair of the JIC, insisted that its material on Palestine should be shown in the first instance only to the Director of Central Intelligence in person. He added that ‘It should be explained to him that if he could not guarantee that they would not fall into pro-Zionist hands, then he could not be left with them.’ Even so, it was decided to withhold more sensitive recent reports on Palestine from the Americans.[32] Meanwhile, ‘Operation Gold’, run by US Navy intelligence, was intercepting the cable traffic of Jewish arms smugglers, but this was not being shared with Britain, or indeed acted upon.[33]

American links with the Commonwealth parties were also hesitant. The Americans were slow to do business with the Canadians. They worried about how much GCHQ had told the Canadians about Anglo–American agreements, and suspected that GCHQ was secretly giving Ottawa some American sigint.[34] During discussion of the 1949 CAN–USA sigint agreements, it became clear that the US Communications Intelligence Board was anxious to prevent an information free-for-all. It preferred to hand material to the Canadians on a ‘need to know’ basis, and was anxious to prevent a proliferation of sigint liaison officers.[35] Meanwhile, somewhat foolishly, Canada resisted the all-important standardisation of security procedures that was a foundation stone of the BRUSA agreement, so negotiations were ‘very difficult’, dragging on endlessly.[36] The Americans were even more wary of sigint cooperation with the Australians due to the KGB spy cases uncovered by Venona in the late 1940s. This, in turn, retarded the joint sigint effort against the newly formed People’s Republic of China from 1949. In late 1953, the advent of a Liberal (i.e. conservative) government in Australia triggered a full resumption of cooperation, formalised at a tripartite sigint conference between the Americans, British and Australians. New Zealand also came in as fifth partner. It was only at this point that the name ‘UKUSA’ was adopted at GCHQ’s request. UKUSA was revised a number of times and new sections added between 1953 and 1956. Because conflict with the Soviets seemed imminent, some sections dealt with the supply of comint to NATO commanders in wartime, or the emergency relocation of sigint units in the event of a sudden invasion.[37]