The Black Door: Spies, Secret Intelligence and British Prime Ministers

The Asquith coalition government disintegrated in late 1916, as a result of its own incompetence and disorder. Lacking personal authority, Asquith had spent much of his time assembling factions and alliances. There was little planning, and astonishingly, letters sent by the prime minister to the King after each meeting formed the only record of cabinet discussions. When Bonar Law joined the coalition in 1915 he had been amazed by the lack of any method or even agenda for cabinet meetings. The failure of the Gallipoli campaign at the start of 1916 accelerated the decline of the Asquith government. Mercifully for the war effort, in June 1916 Lord Kitchener, the aged secretary of state for war, died at sea off the Orkneys in mysterious circumstances, an event that many attributed to a bewildering mixture of German, Irish or Russian saboteurs. He was replaced by the energetic David Lloyd George. At the end of the year, however, both Asquith and Lloyd George resigned in short order, while Bonar Law declined to form a government. Lloyd George became head of a new coalition two days later, establishing a Supreme War Council. Neither Asquith nor Bonar Law was mentally equipped to handle the range of decisions required by modern war. The main challenge for Lloyd George, now and for the rest of the war, was to try to wrest strategic control of the conflict from the military chiefs, a task that he never quite completed.56

Asquith had been notably detached from the business of war. He may have presided over the creation of the Secret Service Bureau and the rapid expansion of every kind of intelligence after 1914, but it had interested him very little. By contrast, in December 1916, David Lloyd George became the first prime minister to embrace intelligence, albeit often in an amateurish manner. This was partly to do with his nature, for he was by temperament a man of enormous energy and sudden impulses. But his initial mistakes also reflected the fact that British intelligence lacked a central brain. No system existed for sifting and interpreting intelligence for top policymakers. Despite a quantum leap in the organisation of Downing Street, and the creation of the Cabinet Office in late 1916, intelligence was deliberately left out. As a result, Lloyd George lacked context and made emotional responses to the raw intelligence he received – with unhappy results.57

His previous interactions with intelligence had been in the context of the ongoing spy-mania. In May 1915, as minister of munitions, he sought to confront the problem of factory explosions. Such disasters were almost invariably the result of primitive manufacturing processes, running at maximum capacity, which did not privilege safety. Like many others, however, Lloyd George was obsessed with the danger of the German ‘hidden hand’, and blamed saboteurs. His staff were allowed to set up a counter-intelligence unit to ferret out these imaginary enemies. Given the name P.M.S.2, it failed to find any spies, and slowly shifted its attention to trade union activity in the munitions factories.58

Lloyd George later admitted that he and some of his friends had deliberately encouraged rumours of saboteurs within the British munitions programme. This included the vast shell-filling factory at Chilwell on the banks of the River Trent in Nottingham, the largest concentration of high explosives anywhere in Britain. In January 1916, a Zeppelin was reported to be hunting up and down the Trent, supposedly hoping to bomb the factory. The next day, rumours circulated that Lloyd George’s friend Lord Chetwynd, who ran the huge complex, had caught three German spies in the act of trying to guide the Zeppelin to its target with hand-held torches and had shot them. Chetwynd exploited the false rumour by asking a labourer to dig three graves on the hillside by night, placing an anonymous black post at the head of each. This, recalled Lloyd George, ‘turned the rumour into unquestioned history’ and discouraged the curious from prying around the factory. Predictably, when it suffered a catastrophic explosion later in the war, it was blamed on yet more spies.59

In late 1916, shortly before Lloyd George became prime minister, the Germans asked the American ambassador in Berlin to explain to President Woodrow Wilson that they were ‘anxious to make peace’. But they did not wish to appear weak, so they secretly asked the United States, which was then neutral, to make a ‘spontaneous’ offer of mediation. Unfortunately for Germany, Britain’s Room 40 had decrypted the American message. When Lloyd George read it, he wrongly assumed that it signified collaboration between America and Germany. With the cabinet in disarray, and lacking intelligence-assessment machinery, the impulsive Lloyd George decided to act alone. He warned the American press against interference by Washington, and asserted that the war must be a ‘fight to the finish’. In reality, President Wilson was immersed in an election campaign and had no interest in peace initiatives at this point. Either way, diplomacy was not Lloyd George’s responsibility. When rebuked by the foreign secretary for meddling, he used the decrypts to defend himself. Far worse, he also alluded to secret information when explaining his actions in Parliament. Even as he assumed office in Downing Street in December 1916, more decrypts crossed his desk which he wrongly – and amateurishly – assumed suggested that the Kaiser and Wilson were still working together. This was not an auspicious beginning, and pointed to a wider failure around the assessment of intelligence at the centre of government.60

Lloyd George brought his undoubted talents for planning and organisation to the highest level of government. The most important part of this reform was the creation of a professional secretariat by Maurice Hankey, a former Royal Marine officer who became the first cabinet secretary. His background was in intelligence – as a junior officer assigned to HMS Ramillies, the flagship of the Mediterranean Fleet, he had engaged in unofficial reconnaissance. By 1902, he had joined the staff of the Naval Intelligence Department in Whitehall. An outstanding officer who spoke many languages, he was the perfect administrator. In 1909, he had written a report for the Committee of Imperial Defence that proposed a Secret Service Bureau.61 Now, in the newly created Cabinet Office, he was joined by Thomas Jones. Once described as ‘a disguised Bolshevik whom Lloyd George had discovered somewhere in a Welsh coal pit’, Jones was nevertheless an equally formidable organiser. It is difficult to capture the chaos that surrounded cabinet affairs before their arrival, and it is no exaggeration to say that they invented modern cabinet government.62

Hankey’s reforms were a triumph. They became central to the development of a modern British interdepartmental coordination system, with its labyrinthine sub-committees and orderly minutes focused on Downing Street. Cabinet meetings were no longer rambling conversations amongst twenty-three people, with no agenda. Instead, they became businesslike discussions at which decisions were made and properly recorded. Yet the reforms were a tragedy for secret service. Hankey created a central mechanism for everything except intelligence. Jones, his deputy, recalled that he had been insistent that the new Cabinet Secretariat should not become ‘an Intelligence department’,63 and although the design of the war cabinet at first envisaged ‘a comprehensive and regular gathering of intelligence’, this never happened.64 The lack of a central clearing house for assessing intelligence had been a constant criticism of government for some time, so while Hankey is celebrated as a moderniser of the government machine, he simultaneously retarded the British intelligence community by twenty years. The idea of a central intelligence machine located alongside Downing Street had to await a further world war, and the arrival of Winston Churchill as premier.65

Lloyd George’s personal record as a user of decrypts did not improve during the war. He was often left to deduce the story from individual intercepts, or ‘flimsies’ as they were called, because of the thin paper on which they were recorded, that arrived without context or comment. He was also given little guidance on the need for security. Thus in February 1917, when the American ambassador, Walter Page, visited Downing Street to convey a message from President Wilson, the prime minister could not restrain himself. He boasted that he had already seen Wilson’s message, attributing this to a leak. Page thought it possible that the telegram had been obtained by a ‘British spy service’ from an unreliable American official. In any case, knowing Wilson’s hatred of leaks, he did not inform Washington. Indeed, the president was obsessively secretive, and actually insisted on deciphering his more sensitive telegrams himself, sometimes with the help of his wife. The realisation that Lloyd George was reading his every word would not have endeared London to him.66

Surprisingly, even as Lloyd George put their work in peril, British codebreakers delivered the greatest intelligence coup of the First World War. They had intercepted what would soon become famous as the ‘Zimmermann telegram’. In this amazing message, Germany’s foreign minister, Arthur Zimmermann, promised Mexico the reward of three of America’s southern states if she joined the German cause and declared war on her northern neighbour, the US. The message was one of the most secret of the war, and was supposed to have been taken by safe hand on a German submarine. But the vessel broke down before leaving port, forcing Berlin to trust the safety of its ciphers.

The Zimmermann telegram is a rare example of a single piece of intelligence changing the course of history. The way in which the British exposed it was elegant, but had nothing to do with Downing Street. Instead, it was a cooperative venture between Room 40 and the Foreign Office – and perhaps for that reason it was not bungled. Lloyd George, doubtless kept abreast given his strong interest in American orientations, was a mere observer. President Woodrow Wilson had won his recent election campaign on the slogan ‘He kept us out of war’. But, provoked by the Zimmermann telegram, alongside Berlin’s aggressive submarine policy, he declared war on Germany in April 1917.67 In his memoirs, Lloyd George records his gratitude to the codebreakers of Room 40 and their ‘uncanny efficiency in the unearthing of German secrets’.68

By the end of the war everyone seemed to be aware of the British secret service. The famous German philosopher Hannah Arendt, for example, was fascinated by the rise of Britain’s professional spies. She wrote that after the First World War, British secret services ‘began to attract England’s best sons, who preferred serving mysterious forces all over the world to serving the common good of their country’, adding that as a result ‘the stage seemed to be set for all possible horrors’. However, she noted that with the British, unlike the Russians and Germans, ‘a minimum of human rights was always safeguarded’.69 Bertolt Brecht wrote that Britain controlled detective fiction and also controlled the world, clearly feeling that these two things were connected in some subterranean way.70

Room 40 was created as a wartime emergency. Nevertheless, its product proved too valuable for it to disband once the guns fell silent, and in peacetime Britain continued to read the secret communications of many countries. As the armistice talks opened in France, President Wilson sent his trusted confidant Colonel Edward House to join the negotiations. Lloyd George was able to read everything sent between House and Wilson. Yet strangely he had not been bitten by the intelligence bug, and useful as he found it, was not an enthusiast. Intelligence spending dwindled dramatically after the war, and the prime minister was happy to leave the reorganisation of peacetime intelligence matters to his cabinet colleagues. Accordingly, although Lloyd George established a governmental committee to review secret service in 1919, he did not join it. Instead, Churchill, an avid intelligence enthusiast, was the leading light in this important reordering.71 Its most important decision would be to maintain the wartime codebreaking effort and focus all resources in one unit: the Government Code and Cypher School, or GC&CS.72

Nonetheless, in February 1920, Lloyd George was required to revisit intelligence matters. He sought to end the festering conflict with the Bolsheviks, viewing Britain’s support for the White Russians as an unhappy extension of the First World War, and therefore a chapter that should now be closed. Although reluctant to offer recognition to the revolutionaries, the prime minister offered Moscow trade agreements as a path to restoring relations. His cabinet colleagues Churchill and Curzon were horrified, not least because of mounting evidence of Bolshevik subversion against the British Empire. Indeed, although there was clear evidence of Soviet funds going directly to the increasingly communist-dominated Labour Research Department during the early 1920s, it was Moscow’s interference in areas such as India that really got the British cabinet excited.73 MI6 had been active in this clandestine conflict too, and many military intelligence officers who had been heavily involved in the Russian Civil War were also dismayed.74

Once more, Lloyd George had access to his opponents’ decrypts. Not for the last time, a British leader was able to read the derogatory terms in which his opposite numbers discussed him. Lenin denounced the prime minister as a deceiving ‘swine’ and a man without scruples, and urged the Soviet trade delegation in London to repay him with even greater levels of deception. This, of course, was difficult, given that every line of Lenin’s instructions was being decoded by the British. Lloyd George declared himself ‘unruffled by Bolshevik intrigues’, which he considered amateurish and unimportant. He was also prepared to turn a blind eye to the war between Russia and Poland that developed in late 1919. But Moscow was a highly political and divisive issue, and by early 1920, some senior military chiefs had begun to question Lloyd George’s motives. On 15 January General Henry Wilson, the Chief of the Imperial General Staff, wrote in his diary: ‘I keep wondering if L.G. is a traitor & a Bolshevist, & I will watch him very carefully.’ Wilson was especially paranoid about Bolshevik plots, and made several similar entries to the same effect over the next few months.75

The activities of Soviet negotiators in Britain were inflammatory. In August 1920 Leonid Krasin, a Soviet trade representative, had arrived for talks accompanied by Lev Kamenev, head of the Moscow Communist Party. Decrypts clearly showed that Kamenev was establishing secret contact with the embryonic British Communist Party, and subsidising the far-left newspaper the Daily Herald, using smuggled diamonds. Every move was visible to the codebreakers, and General Wilson was incredulous that Lloyd George had not immediately ejected the trade mission. On 18 August, Wilson confided his worries to Winston Churchill and the director of military intelligence, General Sir William Thwaites. Over the next few days he did the rounds of the security chiefs, including Lord Trenchard, who commanded the RAF, Sir Basil Thompson from the Home Office, and Rear Admiral Hugh ‘Quex’ Sinclair, director of naval intelligence, and received a sympathetic hearing.76 On 24 August he noted in his diary: ‘Trenchard with whom I discussed the matter later and to whom I showed … the intercepts thinks like Basil Thompson, that Lloyd George is a traitor.’77

On 2 September, with Lloyd George away in Lucerne, his coalition partner Bonar Law summoned a meeting at Downing Street that included Balfour, Churchill and Basil Thompson. Thompson circulated the latest material from the codebreakers. Thomas Jones, the deputy cabinet secretary, recalled that ‘Everyone felt that the last intercept from Lenin where he lays down propaganda as the business of the Russian Delegation put the lid on and that there was nothing for it but to clear them out as quickly as possible.’78

Hankey, who had accompanied Lloyd George to Lucerne, argued exactly the opposite. He insisted that Wilson was absurdly alarmist, and that the activities of the Soviets in London were silly and easy to counter. His main worry was that if they ejected the Soviets they would have to publish the decrypts to justify their actions, placing the prime minister in a very difficult position. Britain would then lose its ‘most valuable and trustworthy source of secret information’, as no matter what they did to try to disguise the fact, Moscow would probably realise its communications were being read. He continued: ‘This particular cypher is a very ingenious one which was discovered by great cleverness and hard work. The key of the cypher is changed daily and sometimes as often as three times in one message. Hence if it becomes known that we decoded the messages all the governments of the world will probably discover that no messages are safe.’79

Back in London, Lloyd George forced a showdown with his critics. He freely admitted that the Soviets were engaged in ‘perfidy’ and ‘trickery’, but argued that good intelligence could be obtained by keeping the trade mission in place. Although he thought their activities ineffective, he could see that political opinion in cabinet was turning against him, and probably regretted not having taken a closer interest in secret service matters, since the intelligence chiefs had been part of this pressure to act. An opportunity to do so arose in early September, when Kamenev decided to return to Moscow and came to Downing Street to pay his respects. He walked into a diplomatic ambush. Lloyd George met him accompanied by Hankey, Jones and Bonar Law. The prime minister opened with a tirade about ‘gross bad faith’ and interference in British internal affairs, including Soviet attempts to recruit labour leaders and secretly subsidise the Daily Herald. He also accused Kamenev of attempting to turn Poland into a client Soviet state – but did not mention the decrypts. There was obviously a double game going on here, with the prime minister trying to clear himself of suspicions of being overly sympathetic to Bolshevism.80 Perhaps for audience effect, he warned Kamenev that if he did not leave Britain he would be deported.81

Lloyd George’s hard line resulted in his also being criticised by the left, so he was eventually pushed into publishing eight of the intercepts. He used a rather thin cover story, claiming that they had been provided by a neutral country. Churchill, showing his characteristic impulsiveness and little nuanced appreciation of the value of intelligence, had led the charge demanding publication. Amazingly, blinded by their ideological hatred of Moscow, the intelligence chiefs had agreed. Quex Sinclair, now head of MI6, who had superintended the best wartime codebreakers, insisted that ‘even if the publication of the telegrams was to result in not another message being decoded, then the present situation would fully justify it’. Intelligence officials leaked further decrypts to newspapers three weeks later. Lloyd George fired Basil Thompson shortly afterwards, leading some to suspect that it was he who was responsible for the leak. His departure was a positive move, and ensured that the Foreign Office and MI6 extended full control over foreign intelligence.82

The prime minister maintained his policy regardless, and secured an Anglo–Soviet trade agreement in 1921. Although a further surge of Soviet subversive activity was discovered in 1922, Lloyd George was overtaken by political scandal before he could respond, accusations of selling seats in the House of Lords forcing him to resign in October. In the summer his colleagues, led by Bonar Law, had chosen to make more Soviet decrypts public. Moscow predictably changed its cipher system, and an important stream of high-grade intelligence disappeared.83 Despite the deliberate and foolish revelations of 1920, 1922 and 1923, GC&CS continued to read much high-level Soviet traffic.84

Astoundingly, Soviet incompetence was even greater than that of the British. As early as July 1920, Kamenev and his colleagues had requested a replacement for their ‘Marta’ cipher system, believing it to be insecure. But their superiors resisted this request as requiring too much effort. They not only ignored the revelations in The Times but also strident warnings from senior Red Army commanders that their most secret correspondence in Europe ‘is known word for word to the English, who have organised a network of stations designed particularly for listening to our radio’. By the end of 1920, Georgy Chicherin, the long-serving Soviet foreign minister, seemed to have got the message, and warned his mission in London that very sensitive material should be sent only by courier. But many secret communications continued to be intercepted by GC&CS.85

GC&CS therefore remained the star in Britain’s interwar intelligence firmament. Between 1920 and 1924, it issued over 13,000 intercepted signals, approximately 290 a month, including more than half of all French traffic. Yet little of this went to Downing Street. If the prime minister was not regularly in receipt of these gems, where did they go? The answer is to the Foreign Office, which aggressively clawed back control over foreign intelligence from the military following the end of the war. Intercepts informed the negotiation of complex post-war settlements and treaties that followed the First World War, such as those at Sèvres and Lausanne, often giving British diplomats the upper hand. One of the most important customers was the foreign secretary, Lord Curzon. By 1922, Curzon also enjoyed power over distribution, deciding what secrets should go to the prime minister or to members of the cabinet. Unfortunately, the decrypts often revealed the fact that Lloyd George was pursuing a separate and secret foreign policy, entirely undeclared to Curzon, and was sometimes backing the Greeks in treaty negotiations against his own foreign secretary. Curzon was an emotional individual, and knowledge of the ‘dirty’ activities of his friends and allies stirred apoplectic outbursts. In 1922, meeting with the French prime minister Raymond Poincaré, Curzon had to be led weeping from the room, and refused to return until Poincaré apologised for his behaviour.86

Curzon deliberately kept much intelligence away from Downing Street. Things were made worse by the lack of a system for synthesising intelligence material and presenting it to leaders in a way that would allow it to inform strategy. Instead, attention often focused on single documents. This was typified by the impact of intelligence on talks about the future of Britain’s alliance with Japan during the early 1920s, a decision that would cast a long shadow. Intelligence focused less on Japan’s intentions and capabilities than on plots and nefarious activities: Lloyd George told cabinet that other friendly powers did things that were ‘infinitely worse’ than the Japanese secret service. Curzon argued for a renewal of the alliance, despite the fact that he considered the Japanese ‘insidious and unscrupulous’. Churchill opposed a continued alliance with Tokyo, and to back up his argument showed his cabinet colleagues a secret Japanese map of Gibraltar that had come into the hands of MI5. Intelligence was often about alarms and incidents, and rarely informed national assessments.87

Meanwhile, the Irish problem had been rumbling on. The Easter Rising and the First World War were swiftly followed by the Irish War of Independence in 1919. Peace negotiations began in 1921, culminating in the formation of the Irish Free State the following year. This time Lloyd George lacked signals intelligence, but he did draw upon other forms of intelligence to aid the British position. Unfortunately, the material reaching him and his ministers came from military intelligence officers who opposed the truce. Consequently, they offered biased worst-case scenarios in which Sinn Féin and especially the IRA intended to play for time before resuming their offensive, including ‘fantastical reports’ of chemical weapons. Fortunately for the peace process, a more confident outlook emanated from the civilian authorities in Dublin, and the ‘two opposing interpretations of the Irish situation battled it out from July to December 1921’. Lloyd George fell into the optimistic camp, and was determined to reach peace, but the alarmist intelligence came perilously close to collapsing the negotiations altogether. By early December, the prime minister had had enough. He issued a dramatic ultimatum: accept the peace or resume war in three days. This was a gamble. Contradictory and inaccurate intelligence meant that he had no idea whether Sinn Féin would accept – and Britain was in no position to resume war so soon. However, the starkly pessimistic intelligence coming out of Ireland offered enough warning about IRA rearmament to convince Lloyd George to set a strict deadline. The negotiations hung in the balance, and there was ‘palpable relief’ when the Irish signed the deal.88

Prime ministers must celebrate when they can; success is often short-lived. In May 1922, Lloyd George was livid when he was shown the new Irish government’s draft constitution, despairing that it represented ‘a complete evasion of the treaty’ he had gambled on securing. Intelligence uncovered numerous plots, some more real than others, against Northern Ireland and the British mainland, in which the Dublin government was complicit. Frustrated, Lloyd George feared that the Irish ‘may have to face re-conquest’. When IRA men assassinated Field Marshal Henry Wilson in London the following month, the government found itself under pressure to wipe out the IRA headquarters in Dublin. A single piece of intelligence wrongly linked the murder to a Dublin conspiracy, and an attack on the IRA headquarters by British troops would have wrecked Britain’s Irish policy ‘at one stroke’. Nevertheless, led by Churchill, the cabinet approved the strike. Once more, Churchill failed to appreciate the nuances of raw intelligence and exhibited his famous impetuousness. All this frustrated the prime minister, but fortunately for Lloyd George and his Ireland policy, the army refused to obey the order.89 This marked Lloyd George’s final dealings with the Ireland problem, but Churchill would have to return to it during the Second World War. More ominously, it was merely a taste of the troubles to come for future prime ministers, from Harold Wilson onwards.

Bonar Law, who replaced Lloyd George in 1922, was the shortest-serving prime minister of the twentieth century, spending only 211 days in office. He died of a throat tumour shortly afterwards, and was buried in Westminster Abbey near the Tomb of the Unknown Soldier. During the funeral, Asquith described him as ‘the unknown prime minister’. Hankey noted in his diary: ‘Poor Bonar never had the nerve for the job of prime minister. The responsibility preyed on his mind and, I feel sure, hastened on his cancer.’ Turning to Bonar Law’s successor, he added, quite correctly, ‘Baldwin has the nerve but scant capacity and I fear will not last long.’ Stanley Baldwin’s first term in office indeed lasted only eight months, and he stepped down in January 1924. Alongside these figures, Lloyd George appeared a political giant, and it was under him that British intelligence took its most adventurous step forward.90

A connected British intelligence community was not created in 1909, but it had already begun to centralise and professionalise. Indeed, it has been suggested that Asquith’s government changed the very meaning of the word ‘intelligence’. At the turn of the century, intelligence was something that existed in the far-flung service of empire, and meant information of almost any kind, so long as it impacted upon policy or made colonial rule more efficient. Much of it was supplied by an army of enterprising amateurs serving on ‘special duties’, supplemented by eccentrics who divided their time between collecting rare beetles or tulip bulbs and sketching Turkish fortifications. By 1918, intelligence was about secret work, and incorporated a strong emphasis on counter-intelligence. Most importantly, it had also become increasingly militarised, and had embraced the science of codebreaking. Room 40 had produced astonishing intelligence in just four years, but as yet no one knew how to interpret it or to use it securely.91

Lloyd George, perhaps assisted by the wise counsel of Maurice Hankey, had begun to learn his trade. In contrast to his early years as prime minister, by 1920 he knew that context was everything in interpreting decrypts. Accordingly, Lenin’s hot language had not alarmed him. He also came to understand that access to decrypts was acquired with difficulty, and given away easily – a lesson quickly forgotten in Number 10. Therefore, although he eventually discussed codebreaking in his memoirs, he said no more than had already appeared in the public domain, and worked closely with Hankey on the agreed text.92

By contrast, his cabinet colleagues performed poorly – even those with more intelligence experience. Lord Curzon, foreign secretary between 1919 and 1924, should have been a master of intelligence, having presided over a sophisticated espionage system in his previous existence as viceroy of India. Yet personal insults from the French, revealed in all their glory by intercepts, literally drove him to tears of rage. Churchill’s performance was even worse, and despite boundless enthusiasm for intelligence he was impulsive in its use. Moreover, when he rushed out his own history of the First World War in 1923, he made many references to British codebreaking capabilities. The Germans were soon avidly reading his account, and it is no coincidence that shortly afterwards they began to take an interest in a new and effective cipher machine called Enigma. In the mid-1920s, however, Germany had not yet resurfaced as a problem. The First World War was over, and a young man named Adolf Hitler had only recently established a nascent Nazi Party. Instead, it was the Russians who attracted the attention of Britain’s secret service. An early cold war of subversion and subterfuge was emerging, in which incoming prime ministers would need to use intelligence subtly and wisely. Inexperienced in handling the secret world and lacking an integrated intelligence assessment community, this may have been asking too much.