The author and publishers are committed to respecting the intellectual property rights of others and have made all reasonable efforts to trace the copyright owners of the images reproduced, and to provide appropriate acknowledgement within this book. In the event that any untraceable copyright owners come forward after the publication of this book, the author and publishers will use all reasonable endeavours to rectify the position accordingly.

All rights reserved under International and Pan-American Copyright Conventions. By payment of the required fees, you have been granted the non-exclusive, non-transferable right to access and read the text of this e-book on screen. No part of this text may be reproduced, transmitted, down-loaded, decompiled, reverse engineered, or stored in or introduced into any information storage and retrieval system, in any form or by any means, whether electronic or mechanical, now known or hereinafter invented, without the express written permission of HarperCollins.

Source ISBN: 9780007555475

Ebook Edition © April 2016 ISBN: 9780007555451

Version: 2017-03-29

Dedication

To Joanne and Libby

(two espionage experts)

Cover

Title Page

Copyright

Dedication

Abbreviations and Acronyms

Introduction

PART ONE: CREATING SECRET SERVICE

1 Herbert Asquith, David Lloyd George and Andrew Bonar Law (1908–1923)

2 Stanley Baldwin and Ramsay MacDonald (1923–1937)

PART TWO: THE WINDS OF WAR

3 Neville Chamberlain (1937–1940)

4 Winston Churchill (1940–1941)

5 Winston Churchill (1942–1945)

PART THREE: THE HOT COLD WAR

6 Clement Attlee (1945–1951)

7 Winston Churchill (1951–1955)

8 Anthony Eden (1955–1957)

9 Harold Macmillan (1957–1963)

10 Alec Douglas-Home (1963–1964)

PART FOUR: DÉTENTE AND DISSENT

11 Harold Wilson (1964–1970)

12 Edward Heath (1970–1974)

13 Harold Wilson (1974–1976)

14 James Callaghan (1976–1979)

15 Margaret Thatcher (1979–1990)

PART FIVE: TURBULENT TIMES

16 John Major (1990–1997)

17 Tony Blair (1997–2007)

18 Gordon Brown (2007–2010)

19 David Cameron (2010–2016)

Conclusion: Prime Ministers and the Future of Intelligence

Appendix I: Key Officials Since 1909

Appendix II: Key Intelligence and Security Machinery

‘Eat Before Reading’: A Short Essay on Methodology

Picture Section

Acknowledgements

Notes

Bibliography

Index

About the Authors

About the Publisher

Abbreviations and Acronyms

‘C’ – Chief of the British Secret Intelligence Service (SIS)

CCC – Churchill College Cambridge

CIA – Central Intelligence Agency [American]

CIGS – Chief of the Imperial General Staff

CND – Campaign for Nuclear Disarmament

Comint – Communications intelligence

Comsec – Communications security

COS – Chiefs of Staff

CPGB – Communist Party of Great Britain

CSC – Counter Subversion Committee

CX – Prefix for a report originating with SIS

DCI – Director of Central Intelligence, the head of the CIA

DIS – Defence Intelligence Staff

DMI – Director of Military Intelligence

DNI – Director of Naval Intelligence

D-Notice – Defence Notice to the media covering security issues

DOPC – Defence and Overseas Policy Committee

Elint – Electronic intelligence

FBI – Federal Bureau of Investigation [American]

FCO – Foreign and Commonwealth Office

GC&CS – Government Code and Cypher School

GCHQ – Government Communications Headquarters

GOC – General Officer Commanding

GRU – Soviet Military Intelligence

IRD – Information Research Department of the Foreign Office

ISC – Intelligence and Security Committee

ISI – Inter-Services Intelligence [Pakistan]

ISP – Internet Service Provider

JAC – Joint Action Committee

JIC – Joint Intelligence Committee

JTAC – Joint Terrorism Analysis Centre

LHCMA – Liddell Hart Centre for Military Archives

MI5 – Security service

MI6 – Secret Intelligence Service (also SIS)

MIT – Turkish Intelligence Service

MoD – Ministry of Defence

NATO – North Atlantic Treaty Organisation

NSA – National Security Agency [American]

NSC – National Security Council [American]

NUM – National Union of Mineworkers

OSS – Office of Strategic Services [American]

PKI – Indonesian Communist Party

PLO – Palestine Liberation Organisation

PSIS – Permanent Secretaries’ Committee on the Intelligence Services

PUSC – Permanent Under-Secretary’s Committee of the Foreign Office

PUSD – Permanent Under-Secretary’s Department

PV – Positive vetting

RAW – Research and Analysis Wing [Indian]

RUC – Royal Ulster Constabulary

SAS – Special Air Service

SAVAK – Iranian Security Service

SBS – Special Boat Service

Sigint – Signals intelligence

SIS – Secret Intelligence Service (also MI6)

SOE – Special Operations Executive

TASS – Soviet Press Agency

TUC – Trades Union Congress

Ultra – British classification for signals intelligence

UKUSA – UK–USA signals intelligence agreements 1948

WMD – Weapons of Mass Destruction

Introduction

This is my own true spy story …

Winston Churchill1

On Saturday, 6 September 1941, Winston Churchill stood on a pile of bricks outside the newly built Bletchley Park. Here, in the Buckinghamshire countryside, the mysteries of the German Enigma encryption machine were being patiently unravelled. Each day the codebreakers’ product was fed to a prime minister in Downing Street who was beside himself with anticipation. Now, with some emotion, Churchill expressed his profound gratitude and explained to the codebreakers how they had already transformed decision-making at the highest levels, and with it the course of the Second World War. A decade later – and now approaching his eightieth year – Churchill was back in Downing Street. His keen interest in intelligence had not diminished. In 1952, top-secret spy flights took pictures over Moscow at the express instruction of the prime minister. Over Minsk and Lvov, his airborne intelligence emissaries were greeted by a formidable wall of Soviet anti-aircraft fire.

Churchill also relished covert action. In 1953, he positively purred with enthusiasm over a joint CIA–MI6 plot that had overthrown the government of Iran. This underlines the way in which intelligence was not just a secret window on the world for Britain’s leaders, but also a discreet means of manipulating it. In 1956 Churchill’s successor, a furious Anthony Eden, neurotic and plagued by ill-health, barked into a telephone that he wanted Egyptian President Nasser destroyed by MI6. Harold Macmillan’s government drew up what he called a ‘formidable’ plan for Syria which involved assassinating several leaders. Alec Douglas-Home added Indonesia’s President Sukarno to the list of foreign leaders that prime ministers wished to see toppled using Britain’s intelligence agencies. However, when Harold Wilson asked for the liquidation of the Ugandan dictator Idi Amin, officials responded with horror, and refused to investigate the options. When secret intelligence took extreme risks, it was usually at the direction of Downing Street.

Harold Wilson evoked the dark side of intelligence. He was convinced that plotters within MI5, MI6 and especially renegade generals in the Ministry of Defence were out to undermine his government. Notably terrified of the South African secret service, known as ‘BOSS’, he chose to develop close personal relations with the Israeli secret service Mossad instead. Speaking with American officials who were inquiring into illegal activities by the CIA in the wake of Watergate, he agreed with them that the CIA failed to tell British authorities everything it did in London. Yet he remained fascinated by the secret world, and valued the intelligence machinery in Downing Street, engaging in academic debate with his intelligence analysts on points of detail like the Oxford junior research fellow he once was.

Intelligence imperilled more than one British prime minister. Within weeks of her arrival at Downing Street, Margaret Thatcher insisted on sitting in with the Joint Intelligence Committee to better understand how intelligence was prepared for those at the top. Only three years later, a major intelligence failure by the same mechanism over the Falkland Islands almost ended her government. John Major found himself confronted by the Arms to Iraq affair, in which ministers had sought to cover up the control of arms export companies by MI5 and MI6. The subsequent inquiry by Lord Scott revealed only part of the murky tale, and brought Major’s government close to defeat in the House of Commons. Tony Blair’s era was defined by vicious public arguments over intelligence. Despite his successful use of secret service during the creation of the Good Friday Agreement in Northern Ireland, it was accusations of the misuse of intelligence over Iraq that would leave his reputation in tatters. His bold decision to use intelligence publicly to justify the war on Iraq quickly backfired, and by 2005 the missing WMD fiasco threatened to end his government prematurely. Blair’s administration also left a toxic legacy of allegations about complicity in torture with which Gordon Brown and David Cameron have both struggled. Most recently, the deluge of secrets revealed by NSA contractor Edward Snowden has reshaped relations between David Cameron, Barack Obama and other world leaders including Angela Merkel. In an era when secret services are increasingly kept in check by whistleblowers and their remarkable revelations, British prime ministers live in constant fear of intelligence ‘blowback’.

For good or ill, intelligence matters to prime ministers. It may be only one factor shaping their thinking, but it can be critical and very personal, often revealing what their opponents think about them.2 Intelligence can provide warning of dangerous future events, and has certainly averted major terrorist attacks on London. It can help to prevent wars, or – once they begin – mean the difference between victory and defeat. Yet it can also be misused to justify preconceived policy desires. The secret world can help to resolve a prime minister’s most difficult dilemmas by providing a hidden hand that shapes events behind the scenes. It can also become a seductive apparent panacea, whereby secret service is employed to carry out a leader’s dirty work, at home and abroad. In its darkest moments, intelligence can consume prime ministers.

The relationship has become ever closer. The link between Number 10 and Britain’s intelligence agencies, as intimate as it is secret, lies at the heart of the British establishment. For almost a century, a stream of secret boxes – red for MI6 material and blue for intercepted communications from GCHQ – made their way to 10 Downing Street. Sifted by the cabinet secretary, the best material was placed in a special striped box, nicknamed ‘Old Stripey’, for the early attention of the prime minister. Since the creation of Britain’s intelligence agencies in 1909, the relationship between prime ministers and the secret world has moved from circumspection to centrality. One thing is clear from this story. Intelligence is not about rogue agents operating wildly and freely; nor is it an unaccountable business far removed from the corridors of political power. Many of the most hair-raising intelligence activities and many of the most dangerous covert operations have involved Number 10 directly. Intelligence is part of the beating heart of Britain’s core executive, and has long held a special place behind the famous black door of 10 Downing Street.

The use of secret intelligence is one of the dark arts of statecraft. Yet during the early years of the twentieth century, British prime ministers were notably inept practitioners. Relative strangers to the murky world of espionage and clandestine warfare, they treated the subject with either indifference or outright suspicion. When prime ministers did draw on intelligence, they demonstrated remarkable inexperience and naïvety in its use. While Herbert Asquith presided over the creation of the modern British intelligence services in 1909, he had little interest in secret matters other than his mistress. David Lloyd George and Stanley Baldwin enjoyed the first fruits of Britain’s revived wartime codebreaking operations, but squandered them by publicising not only German but also American and Russian intercepts, compromising the source and causing anguish among the denizens of secret service. Ramsay MacDonald, Britain’s first Labour prime minister, distanced himself from the secret services, fearing they may have been out to plot the demise of his government; while Neville Chamberlain arrogantly ignored invaluable secret intelligence provided by brave Germans who personally visited Downing Street at the risk of their own lives. Instead, he preferred to rely on diplomatic reports from those sympathetic to Hitler, which reinforced his own preconceptions.

In 1940, Churchill changed all this. Indeed, a Churchillian revolution in intelligence occurred during the middle of the twentieth century. For five years Britain’s fate hung in the balance, and intelligence, especially Ultra material provided by Bletchley Park, often proved the operational advantage. Churchill placed a premium on intelligence chiefs telling truth to power: he believed that craven intelligence officers accounted for some of Sir Douglas Haig’s failings during the First World War. ‘The temptation to tell a Chief in a great position the things he most likes to hear,’ Churchill later observed, ‘is one of the commonest explanations of mistaken policy.’3 As prime minister he developed close links with the secret world, especially through Stewart Menzies, chief of MI6, whom he sometimes summoned to his bedside in the middle of the night. The prime minister was obsessed with secret intelligence, and – more importantly – strained every sinew to make sure that the government used it to efficient purpose.

Churchill was impulsive, and so constituted a mixed blessing. As David Reynolds argues in his magisterial study, there were reasons why both Baldwin and Chamberlain had kept him out of office. He had imagination, industry, energy and eloquence, but he lacked judgement and wisdom. Lloyd George was of much the same opinion, arguing that in an emergency Churchill’s vision and imagination were essential, and should be used to the full, but at the same time he should be kept under a ‘vigilant eye’.4 Churchill’s over-enthusiasm led to friction with the chiefs of staff, who had the onerous duty of vigilance, and indeed to occasional blunders. Yet his determination to harness the power of intelligence transformed not only the centre of government, but also the very nature of British statecraft.5

Uniquely, Churchill brought with him vast intelligence experience. In charge of the Admiralty during the First World War, he had seen the work of Britain’s most proficient codebreakers at first hand. In the interwar years he had been deeply interested in intelligence, and repeatedly shaped its development. Once he became prime minister in May 1940, he was finally empowered to change procedure at the top. He demanded raw intelligence fresh from MI6 agents in the field or from Bletchley Park. This annoyed the chiefs of staff, and triggered a revolution in Whitehall’s intelligence assessment machinery – the Joint Intelligence Committee moved centre stage – laying the foundations of the successful body which still operates today. Moreover, the impetuous Churchill loved action. He not only nurtured Number 10’s direct connection with espionage, but also encouraged a new focus on special operations and covert warfare. His personal interest in ‘funnies’ and oddball units – ranging from the Chindits and the Commandos to the Special Operations Executive and most famously the SAS – changed how Britain approached warfare for decades to come.6

Churchill’s wartime government was itself a school for secret service. Around him, Britain’s future leaders learned the business of intelligence and its importance to the practice of statecraft. Thereafter, Britain’s next few prime ministers – all veterans of Churchill’s wartime government – understood the value of secret sources and the event-shaping power of intelligence. Trained in the sophisticated integration of intelligence and policy, his successors realised that it formed a crucial instrument for any leader in a perilous period of British decline. Secret service was part of the ‘fancy footwork’ of retreating empire that could potentially turn the tide in colonial bushfire wars, gain an upper hand in manoeuvrings against the Soviets, and even deceive Washington or Brussels.7

Churchill constituted an improbable double act with Clement Attlee. Serving as Churchill’s wartime deputy, Attlee took a quiet interest in intelligence from the very first week he entered office, including the sensitive matter of interning Nazi sympathisers.8 As early as the autumn of 1940, he was arguing for a single authority to coordinate and improve intelligence.9 Later, he led an inquiry into one of darkest episodes of the war, the Gestapo penetration of Britain’s resistance efforts. Despite his public reputation for timidity, Attlee hid an inner toughness. After 1945, he quietly sanctioned numerous covert operations behind the Iron Curtain and across the Middle East. He also recognised the importance of MI5, of positive vetting and of domestic counter-subversion – even at the risk of frustrating his own backbenchers. Most importantly, he completed Churchill’s project, developing new mechanisms that improved Downing Street’s control over secret service. Indeed, by 1951, deliberately seeking to capture corporate memory and learn the lessons of the preceding ten years, Attlee completed the Churchillian revolution, harnessing the power of the hidden hand to state policy and creating a connected British intelligence community for the first time.10

Anthony Eden and Harold Macmillan had also learned about the growing power of secret service during the war. Eden in particular had endured a series of vexatious confrontations with SOE, which appeared at times to be conducting a parallel foreign policy. In 1942, confronted with yet another improbable episode, he peevishly exclaimed, ‘Am I foreign secretary or am I not?’ Yet once ensconced in Downing Street, both Eden and Macmillan recognised the value of secret service. They also realised that the CIA was quietly being used to expand American influence while bypassing the political, economic and military constraints associated with overt intervention. Accordingly, as well as launching their own covert operations, the prime ministers’ relations with presidents Eisenhower and Kennedy often involved selling cooperation on Anglo–American secret wars against figures like Nasser and Sukarno to the White House.11

More recent prime ministers did not, of course, serve under Churchill or Attlee. But all experienced their legacy inside Number 10. Churchill brought the intelligence machine more firmly into Downing Street, while Attlee, as his partner and immediate successor, greatly strengthened the central control of the cabinet secretary over the secret world and began the creation of a refined national security apparatus. Together, they created an intelligence community and made it a familiar part of the working lives of prime ministers for the first time. Thereafter, intelligence became part of the growing ‘presidentialism’ of the British prime minister. While Harold Wilson may have nurtured a curious love–hate relationship with intelligence, Burke Trend, his cabinet secretary, continued Churchill’s drive to connect secret service with the centre. Under Wilson, the involvement of the Cabinet Office with the intelligence services became so great that Trend created a new post of Cabinet Office Intelligence Coordinator to help manage their business. To Wilson’s pleasure, Dick White, who had served as a much-admired director-general of MI5 and then chief of MI6, was the first incumbent.

The growing power of premiers, presidents and prime ministers in the twentieth century went hand in hand with secret service. In Britain, this meant the rise of special policy advisers, spin doctors and the increasing tendency of Downing Street to impinge on the territory of cabinet ministers. Typically, Edward Heath created a Central Policy Review Staff under Victor Rothschild, a wartime MI5 officer who also served as a back channel to the secret services. Margaret Thatcher took this further, appointing Percy Cradock to serve simultaneously as her personal foreign policy adviser and also as Chair of the Joint Intelligence Committee. Unhappy with the reluctance of the ‘wets’ in MI5 and the Foreign Office to confront what she saw as Britain’s enemies, Thatcher even created a network of private secret services around Downing Street that paralleled Ronald Reagan’s piratical adventures with the US National Security Council during Iran–Contra. In Whitehall and Washington, secret service now bestowed upon Western leaders their own hidden and deniable foreign policy.12

The twenty-first century witnessed the final convergence of intelligence and political leadership. Tony Blair waged wars from Bosnia to Iraq, Sierra Leone to Afghanistan. Doing so required the support of Britain’s secret services. Ironically, his failure to understand intelligence, his determination to exercise personal control and the calamities that followed prompted a further reshaping of the way in which secret service connects with British foreign policy. Gordon Brown and David Cameron formally institutionalised their relationship with the intelligence community, the latter using an American-style National Security Council. For the first time, Cameron met with the three agency chiefs once a week, together with key decision-makers, in an environment that promotes action. Moreover, he announced himself as ‘minister for the intelligence services’.13

One of the things that frightens British prime ministers most is the ghost of secret service past. A special security squad of weeders and censors constantly patrols the boundaries of Britain’s official past in the hope of protecting Downing Street from embarrassment. The intention is that intelligence scandals will only emerge slowly and in a controlled fashion, as the result of papers being opened after an embargo of seventy-five or even a hundred years. Eventually, in the National Archives at Kew, amid crumbling paper and crumbling academics, the sensations of yesteryear can be safely examined, since any possible witnesses who might speak out of turn have long passed away. But occasionally, premiers have had to confront skeletons rattling in the secret service cupboard of their immediate predecessor. None was as noisy or as dangerous as the ‘Arms to Iraq’ episode that came within a whisker of ending John Major’s government.14

‘Scandals are the nastiest experiences in politics,’ noted Bernard Donoughue, senior policy adviser to both Wilson and Callaghan. In September 1977, reflecting on the latest press revelations about alleged attempts by the intelligence services to subvert or even overthrow the prime minister, the so-called ‘Wilson plot’, he added that these affairs brought together ‘scared politicians, hysterical and self–righteous civil servants and hypocritical lying journalists’.15 Intelligence scandals have long caused sleepless nights in the flat above 10 Downing Street. Almost as soon as he entered office, Attlee was confronted with revelations about the ‘atom spies’, Alan Nunn May and Klaus Fuchs, two wartime scientists employed by the British who had seemingly passed the secrets of the West’s new wonder weapon to Moscow. When Stalin detonated an atomic bomb ahead of the British in August 1949, there were awkward public questions in Parliament, made worse by the mysterious flight of Donald Maclean and Guy Burgess to the Soviet Union in 1951. Security scandals had an alarming habit of appearing out of a clear blue sky, as Anthony Eden discovered just a few years later. In 1956, he clumsily attempted to bat away questions about the mysterious death of the navy frogman ‘Buster’ Crabb, who during a state visit to Britain by the Soviet leadership had embarked on a hazardous secret operation that MI6 had failed to clear properly with the political authorities. The Crabb fiasco triggered a major review of prime ministerial clearance for clandestine operations and new mechanisms for assessing attendant political risk.

The 1963 Profumo affair was a defining moment for Downing Street. It not only helped prompt Macmillan to terminate his premiership early, it also heralded a new attitude by the press towards matters of secrecy. Hitherto, the long shadow of the Second World War had ensured a degree of circumspection and security-consciousness by journalists when addressing intelligence matters. But by the 1960s, propelled by a range of scandals around CIA special operations, the press was hungry for stories about espionage. The shooting down of Gary Powers’ U-2 spy plane in May 1960 reverberated personally for Macmillan, collapsing a Cold War summit with Khrushchev that he had worked hard to create. This was followed by the CIA’s disastrous Bay of Pigs escapade, which marred the first months of Kennedy’s presidency and raised questions about his personal judgement. Profumo too had an Anglo–American dimension, with anxious FBI officers arriving in London to probe the delicate question of whether Kennedy had met Christine Keeler. For Macmillan, always keen to project the persona of Edwardian stability, the increasing connections made by the press between premiers, politics and the secret world became deeply disconcerting. It was a wind of change he did not welcome.

Harold Wilson had enjoyed tormenting Macmillan over Profumo. As leader of the opposition in 1963, he recognised that this episode was about sex and high society as well as secrets – a gift in terms of media coverage. Entering Downing Street in 1964, he was determined to protect himself against the same fate, and noted that his predecessor, Alec Douglas-Home, had already established a Security Commission to which he could refer malodorous matters when they came before Parliament, giving the prime minister the welcome appearance of actually doing something. For Downing Street, the tactic of creating long-running inquiries into secret matters gradually became a central mechanism for containing toxic security issues. But Wilson also wanted his own ‘security enforcer’, and appointed his paymaster general, George Wigg, to sniff out potential scandals within government. Cabinet ministers and MI5 loathed the freewheeling security inquiries that Wigg made into the most sensitive parts of Whitehall. All this failed to protect Wilson from hideous embarrassment over the interception of telegrams during the ‘D-Notice affair’ in 1967, and then the public flaunting of MI6 secrets by Kim Philby in his waspish memoirs published in 1968.

Labour leaders faced a peculiar predicament. They often found themselves publicly defending domestic security services that they privately feared might be seeking to undermine them. Attlee faced the awkward position of being a socialist prime minister at the onset of the Cold War. The Labour Party and MI5 were not, at this point, natural bedfellows, and Attlee had to balance issues of positive vetting with backbench accusations of launching an anti-communist witch-hunt. Attlee was always conscious of the tension between intelligence, security and liberty. He agonised over investigation into the personal backgrounds of Whitehall officials, and introduced it only under the American threat of ceasing nuclear cooperation, passing this into law as almost the last act of his administration. By the 1970s, Labour prime ministers were inspecting MI5 files on their own MPs and wondering what level of risk was involved in appointing them to government. It was now known by Downing Street that some MPs – like John Stonehouse – had actually worked for Eastern bloc intelligence services. Indeed, one of the longest-serving MPs in the House of Commons, Labour’s Bob Edwards, who represented Wolverhampton South-East until 1987, was a fully-paid-up KGB agent. Other MPs had worked closely and enthusiastically with the CIA or Mossad.16