Lux Ex Umbra: History and its missing contents

It's no secret that the history of intelligence agencies is mostly hidden away in locked cabinets and encrypted data banks.

SIGINT agencies have traditionally been among the most secretive parts of that top-secret world. The darkness shrouding the Second World War SIGINT activities of the countries now known as the Five Eyes began lifting in a significant way only 30 years after the end of that war. Now, 30 years after the end of the Cold War, we are beginning to get a clearer picture of the outlines of SIGINT activities during that era, but huge gaps still remain. And most of the post-9/11 picture is even darker.

The closer we get to examining the present, the more we are forced to rely on information from investigative journalists, leakers, whistleblowers, the cryptic reports of oversight and review bodies, and, erm, researchers of an unusually obsessive nature to shed a glimmer of light on the vast territories of official darkness that swath these agencies.

But this is only a partial and far from satisfactory expedient. Over the longer term to get anywhere near a complete picture we need the agencies themselves to maintain detailed archives and oral history records and to make them available to historians and other researchers in essentially complete and unredacted form.

That, unfortunately, can take many, many decades. We're still waiting to get the full, entirely unredacted picture of the war against Hitler and Tojo. And while there may be little excuse for that, the agencies do have legitimate reasons to remain silent about many aspects of their work. Timidity, inertia, and a lack of resources for declassification also slow the process.

One useful step in the meantime is the production of official agency histories by security-cleared outside historians, often with the assistance of professional in-house staff. This compromise approach provides the historians involved access to complete or near-complete agency records and often to key individuals much earlier than we could otherwise expect.

The trade-off, of course, is that in-house histories are inevitably sanitized, for security purposes at least and for PR purposes at worst, and the picture they present is often distorted — sometimes deliberately — by the absence of still-forbidden topics.

Still, the better examples of such histories can contribute a lot to public and scholarly understanding, and so it's good to see that the Five Eyes SIGINT agencies are starting to step up and participate in such initiatives.

NSA has long had an active official history program that has produced a multitude of studies, including American Cryptology During the Cold War, 1945-1989, Thomas R. Johnson's four-part history of American SIGINT up to the fall of the Berlin Wall, which was written for internal use but later had large portions declassified and released to the public. A wide range of other public or subsequently declassified histories have also been written on specific aspects of the agency's history. A guide to the publicly available versions of these documents and other NSA history resources can be found here.

No other Five Eyes agency has anything on the scale of the NSA's program, but an official history of GCHQ (Behind the Enigma: The Authorised History of GCHQ, Britain’s Secret Cyber Intelligence Agency) is currently in the works in the U.K., with publication originally planned for this year but now expected in October 2020. Written by University of Calgary professor John Ferris, the book is designed for public release.

The Australian Signals Directorate also recently commissioned an official history, to be written by Australian National University professor John Blaxland, who previously co-wrote the official history of the Australian Security Intelligence Organisation. The ASD history is scheduled for release in 2022, coinciding with the agency's 75th anniversary.

[Update 18 September 2020: But it looks like it could be late. Blaxland was dumped from the ASD project in August 2020. According to the Sydney Morning Herald, "The agency will go ahead with an official history but security sources said it wanted more control over the project." That doesn't sound very promising, but at least there does seem to be a continued commitment to some sort of official history being produced.]

Which leaves just the Communications Security Establishment and New Zealand's Government Communications Security Bureau. GCSB hasn't announced a history project yet, as far as I know.

Where is CSE's history project?

What about CSE?

The agency's own 75th anniversary celebrations will be held in 2021. It would be very interesting to know what CSE has in mind for that milestone. Sadly, as far as I know no kind of substantial history is contemplated for release in time for the celebration — or indeed any time afterwards.

CSE has in recent years augmented the brief explanation of its origins posted on its website with two short faux news reels that highlight artifacts related to its history. These videos are interesting and fun to watch, and I hope the agency makes more of them. But they're really just an exercise in profile-raising, intended perhaps to soften the agency's intimidating image and maybe interest a few people in working there. I don't know what effect they're likely to have on the targets of CSE's recruiting efforts — or if 20-somethings have any idea what a news reel even is — but in any case neither CSE nor anyone else would mistake them for an actual history program.

CSE does have a seven-volume in-house history, History of CBNRC (CSE's original name), that covers the 29 years from 1946 to 1975. But 44 years have passed since 1975, and that document was written by retired senior officials, who although they knew their subject intimately had neither the training nor the objectivity of outside historians.

The closest Canadians have come to getting a professional history was in the late 1990s when the Privy Council Office contracted with professor Wesley Wark, then of the University of Toronto, to write an official history of the Canadian intelligence community. Unfortunately, that project fell apart a few years into the effort when the government decided that the document would not be made public.

If there's any bright spot in the Canadian picture, it is that the record on releasing intelligence-related documents in response to Access to Information requests has very gradually been improving, moving from absolutely abysmal in the early years to somewhere between terrible and occasionally useful, although still ridiculously slow and with an appeals process that has now essentially collapsed.

On the useful end of the spectrum, a significant portion of the History of CBNRC has been declassified and released to the public in recent years, still suffering from large redacted sections (including one entire volume) but far more complete than the skeletonized version released in the 1990s. Significant parts of the draft version of Wark's history also have been declassified and released, although again with very significant redactions.

Among other results, these recent releases have enabled Wark to write a fascinating and detailed account, published earlier this month, of the negotiations leading up to the CANUSA communications intelligence agreement reached by the U.S. and Canada in 1949. (See Wesley Wark, “The road to CANUSA: how Canadian signals intelligence won its independence and helped create the Five Eyes,” Intelligence and National Security, published online 7 November 2019; mirrored here.)

For those interested in the history of Canadian intelligence and its signals intelligence effort in particular, that's a highly welcome development.

But such articles are no substitute for an actual, professional, official history of the organization.

So how about it, CSE? Are you going to wait until New Zealand's tiny GCSB beats you to it too?

Update 19 February 2020:
On a related note, A declassification strategy for national security and intelligence records, a discussion paper written by Wesley Wark for the Office of the Information Commissioner (published by the OIC yesterday), is well worth reading.