Still Crays after all these years

Thirty years ago, in March 1985, CSE received its first supercomputer, a Cray X-MP/11, purchased as the linchpin of the agency's effort to revitalize its then-moribund cryptanalytic capabilities.

With a processing speed of about 200 MFlops, impressive at the time but absurdly slow now, that computer has long since been retired.

[Update 12 November 2020: According to the Canada Science and Technology Museum, which has three circuit boards from that computer, the X-MP was retired on 24 August 1995.]

But CSE has continued to acquire increasingly capable supercomputers in order to maintain cutting-edge high-performance computing capabilities, both for cryptanalysis and, in more recent years, for large-scale data mining. And although CSE has been reticent to talk about those computers, it is clear that many of them continue to carry the name Cray.

For example, this CSE presentation, leaked by Edward Snowden and published by the CBC in February, confirms that CSE was operating a Cray XMT supercomputer in 2010 (see page 17).

Cray's XMT model was first rolled out in 2008, and it is likely that CSE obtained its example not long afterward. According to this report, the XMT line "was originally targeted for high-end data mining and analysis for a particular set of government clients in the intelligence community.” Unlike the supercomputers optimized for speed, the XMT was

made to deal with really huge datasets—we’re talking terabytes—whether they be technical or non-technical in nature. But the XMT is actually designed for a specific flavor of data-intensive application: those that must deal with irregularly structured data at scale—what are sometimes referred to as graph analytics problems. These can be broken down further into two general categories. The first is the finding-the-needle-in-a-haystack problem, which involves locating a particular piece of information inside a huge dataset. The other is the connecting-the-dots problem, where you want to establish complex relationships in a cloud of seemingly unrelated data.

It's not hard to imagine why CSE might want a computer with capabilities like that.

Although evidence is scanty, it is likely that CSE has operated successive generations of Cray computers more or less continuously since 1985.

CSE's original Cray, the X-MP/11, was later upgraded to X-MP/14 status and then further augmented in the early 1990s.

The agency's next Cray was probably a T3E, purchased sometime around 1998. (The first T3E was produced in 1996.) This machine would have carried the agency through to the end of the pre-9/11 era.

In October 2001, in the immediate aftermath of 9/11, CSE was granted an immediate one-time budget boost of $37 million, of which $26 million was for "upgrading information technology infrastructure and enhancing processing and analysis".

It is likely that a significant part of this money went to the purchase of a 36-core Cray X1 system, which appears to have arrived at CSE in 2003. Described as a "Government" system, this computer was ranked number 278 on the "Top500" supercomputers in the world list in June 2003. With an Rmax of 404.3 GFlops, it was about 2000 times as fast as the original Cray X-MP. By November 2003, its speed had increased to 422.1 GFlops, but it was only 388th on the list.

The next generation of Cray purchased by CSE was most likely the XMT, which as noted above was probably obtained around 2008 and was certainly in place in 2010. Unlike the earlier systems, the XMT's design was optimized not for speed at single calculations, but for its ability to process massive data sets for data-mining applications. For this reason, no XMT has ever appeared on the Top500 list of supercomputing systems. But this doesn't mean that they weren't supercomputers: according to Cray, XMTs delivered "10 to 100 times better performance than conventional architectures on problems that exhibit irregular data access patterns."

This change in optimization may be an indication that the importance of data-mining had by that time eclipsed that of traditional cryptanalysis at CSE. However, it is also possible that separate systems optimized for cryptanalysis also continued to be purchased.

Finally, the completion in November 2011 of CSE's new high-performance computing centre, Pod 1 of the agency's new headquarters complex, was probably the occasion for the arrival of even newer computers, most likely including the XMT's successor, the XMT-2. As I noted here, Pod 1 was

a very expensive building for its size, costing $61.5 million according to CSE. A simple high-security office building of the size of Pod 1 would have cost about $25 million to build, so it’s probably a safe assumption that, in addition to covering the cost of electrical distribution systems, uninterruptible power supplies, and cooling systems required by a data centre, the building’s budget also covered the purchase of some pretty significant computer capabilities.

It is likely that the building actually contains multiple high-performance systems, some of which are probably still optimized for cryptanalysis. When Cray Inc. advertised for a Customer Service Systems Engineer last year for an unidentified “classified account headquartered in Ottawa, Canada”, the ad stated that the position would “provide hardware and software technical support and maintenance for Cray Inc. massively parallel (MPP) computer systems”. Note the plural in the word "systems".

In addition to Cray computers, CSE has also obtained high-performance systems produced by other manufacturers.

The November 2000 Top500 list, for example, noted that the "National Defense" department (read CSE) had a 60-core IBM SP Power3 system, also known as an RS/6000, with an RMax of 63.6 GFlops, ranking 405th in the world. This was then replaced, or augmented, in 2002 by a classified "Government" 688-processor IBM Netfinity Cluster PIII 1 with an RMax of 188 GFlops, ranking 358th.

Ten years later, the "Government" was reported to have a 17,472-core HP Cluster Platform 4000 BL465c with an RMax of 81.7 TFlops, over 400 times as fast as the 2002 system and 400,000 times as fast as the Cray X-MP, fast enough for 286th position in the world in June 2012. Later in 2012 it was upgraded to 25,472 cores and 118.1 TFlops, bumping it up temporarily to 227th in the world. By 2014, however, this system had fallen off the list.

CSE's high-performance computing capabilities were discussed by MP David Price and CSE Chief Keith Coulter during parliamentary hearings in March 2004:

Price: “When we were there in 2001, you had one of the best computer systems, actually, in the world at that point, and you were going away from that and into another system of daisy-chaining computers all together. You were in the process of expanding that on a rather large scale. Of the extra funds you got for post-9/11, would part of that money have gone into that type of equipment?”

Coulter: “Yes.”

Price: “Simple answer.”

Coulter: “High-powered computing is part of our business. You have to have state-of-the-art computing capacity to do what we do. We're always trying to increase it and looking for money opportunities to do that, because it's expensive equipment. Regarding the chained-together computers, it's all about the total computing capacity, in many respects. Some of the smaller systems can be flexible and they can do the job, but in the end, those big high-powered computers are where it's at. That's why we can bring the best PhDs in computer science in, because they know those are the most challenging and exciting computers.”

Price: “We are still one of the top ones, though, in the world with the system that we do have.”

Coulter: “Yes. Top in the world? We're definitely one of the top in the country. The National Security Agency has more computing power than any organization in the world.”

CSE's high-performance computing systems continue to evolve.

The completion of CSE's data warehouse at its new headquarters complex has probably augmented the agency's data processing capabilities considerably, as sophisticated storage systems can also be used as data processors.

But for high-end codebreaking and data-mining purposes, it seems likely that CSE still relies on the systems in Pod 1.

Cray no longer holds the monopoly on those systems.

But 30 years after its X-MP launched CSE into the supercomputing business, Cray continues to be one of the agency's key suppliers.


Update 28 December 2016:
As noted here, in November 2015, CSE reported that its new headquarters building has "The largest concentration of supercomputers in Canada."