Who has the most powerful supercomputers in Canada?
According to the well-known
Top500 list, the top supercomputers in Canada in terms of peak processing speed are operated by
SOSCIP/LKSAVI/University of Toronto, “
IT Service Provider C”,
SciNet/University of Toronto/Compute Canada, and
Calcul Canada/Calcul Québec/Université de Sherbrooke.
SOSCIP frequently
boasts that its supercomputer “is the fastest in Canada on the TOP500 list of the world's top supercomputers”.
But there is at least one Canadian institution that doesn’t report its computer capabilities to the Top500 list: the Communications Security Establishment.
[
Update 13 August 2015: However, its computers do occasionally turn up on the list described as classified "Government" systems, etc., as noted
here.]
In 1985, when CSE entered the supercomputing business, the
Cray X-MP/11 it acquired was definitely the most powerful computer in the country.
But that was a long time ago, and today that computer is just a piece of computer history.
Still, it is likely that CSE’s subsequent supercomputer acquisitions, including successive generations of Cray products, have kept the agency at or near the top of the Canadian list ever since.
In general terms, this is no secret.
In 2004, member of parliament David Price, noting the post-9/11 computer purchases made by CSE,
asked CSE Chief Keith Coulter if “we are still one of the top ones… in the world with the system that we do have.” Coulter’s reply was, “Yes. Top in the world? We're definitely one of the top in the country. The National Security Agency has more computing power than any organization in the world.”
CSE remains coy about the exact nature of its high performance computing capabilities, but as recently as 2013 it was willing to
state that “CSEC is Canada's centre for high performance computing”, operating “state-of-the-art equipment”. Its recruiting site
currently states that CSE operates “some of the most powerful computers in Canada”, and until 2010 job notices
specified that CSE “computer scientists utilize a variety of computer systems including SUN, HP and IBM servers, personal computers, DEC systems, and state-of-the-art computers such as the Cray.”
More specific claims occasionally turn up in news articles about the agency.
In 2012, it was
reported that CSE’s new headquarters would house “the three most powerful supercomputers in Canada”. And a QMI Agency report in 2013
stated that CSE’s new headquarters will house “the country’s five most powerful computers”.
In neither case were these claims attributed to a specific source, and CSE has never confirmed either claim, but it is difficult to believe that these reporters would have reported such specific information if they hadn’t heard it directly from what they considered to be an inside source.
The level of performance required to rank as the country’s most powerful computer is a constantly moving target, of course, but the claims seem entirely plausible.
In 2011, CSE completed a brand-new high-performance computing centre, the
Mid-Term Accommodation Project, now known as Pod 1 of CSE’s new headquarters complex.
Pod 1 was a very expensive building for its size, costing $61.5 million according to CSE. A simple high-security office building of the same size would have cost about $25 million to build, so it’s probably a safe assumption that, in addition to covering the cost of electrical distribution systems, uninterruptible power supplies, and cooling systems required by a data centre, the building’s budget also covered the purchase of some pretty significant computer capabilities.
It is also likely that substantial additional computer money has been made available since. CSE has not lacked for funds in recent years (see
here and
here), and there’s no reason to build a state-of-the-art computing centre if it’s not going to contain state-of-the-art computers.
As the news articles suggest, the building may well contain multiple high performance systems. (In addition, the data storage systems in the separate data warehouse also built at CSE’s new complex might also be considered a form of supercomputer.)
As the systems on the Top500 list show, a variety of different manufacturers produce supercomputing systems, and it is possible, perhaps even likely, that CSE has obtained systems from more than one company. It seems certain, however, that one or more Cray systems continue to be in use at CSE.
Cray has maintained a close relationship with the major Five Eyes SIGINT agencies throughout the history of the various companies that have borne that name, and as noted above, CSE was acknowledging its own continuing relationship with Cray as recently as 2010.
It is surely no coincidence that Cray Inc. is currently looking for a
Customer Service Systems Engineer to “provide hardware and software technical support and maintenance for Cray Inc. massively parallel (MPP) computer systems” at a “classified account headquartered in Ottawa, Canada”. According to the notice, Canadian citizenship is “a must” for the job, as is a “Top Secret (SBI) security clearance”.
Cray’s ad doesn’t reveal the name of its customer, but there’s only one Canadian agency that belongs to the
Cray Users Group.
CSE’s Australian counterpart, the Australian Signals Directorate (previously known as the Defence Signals Directorate),
acknowledged purchasing a $14.5 million Cray system in 2010.
Although no details of that system were released, at that cost and date it was probably a medium-sized XE6 system, or something with comparable performance, with a theoretical peak processing speed on the order of 300 teraFLOPS and consuming around 0.9 megawatts of electrical power. (This is a guess based on the
reported performance and
$45 million cost of the larger Cray Cielo system purchased by the U.S. that year.) If so, it was the most powerful supercomputer in Australia at the time and would have been roughly on par with the top publicly acknowledged supercomputer in Canada that same year.
[
Update 13 August 2015: Alternatively, it may have been a Cray XMT system, discussed
here.]
Did CSE purchase something similar, or more powerful, for its new high performance computing centre in 2011?
The two cooling towers on the roof of Pod 1 provide a bit of a clue (photo courtesy of Chuck Clark).
The two towers, built by Evapco, appear to be from the company’s AT-112-514 to 112-914 series, which means that each tower is capable of providing 494–574 tons of nominal cooling. If both towers were in full use, this would provide cooling for equipment consuming roughly 3.5 to 4 megawatts, of which the IT load might comprise around 3 megawatts. (At least, that’s what I think can be concluded; I would be grateful if readers would correct any errors in the preceding.)
If these conclusions are correct, then Pod 1 has the capability of supporting a much more capable computer system, or set of systems, than that apparently purchased by ASD in 2010. (Moreover, there is space available in the enclosure on the roof for an additional cooling tower, suggesting that the building was designed to accommodate even greater cooling capacity if it is ever required.)
If the building was using its full two-tower cooling capacity in 2011, it would have been capable of supporting the equivalent of the entire
“Hopper” system, with a theoretical peak performance of 1289 teraFLOPS, or three copies of the
“Gaea C2” system, each with a theoretical peak performance of 716 teraFLOPS. The latter would certainly have been the three most powerful supercomputers in Canada at the time.
Of course, it is likely that the systems actually in use in Pod 1 require less than the maximum amount of cooling that the facility is capable of providing—quite possibly a lot less.
The site that hosts the #1 and #3 Canadian systems on the current Top500 list was built to accommodate a 4-megawatt load, about the same as Pod 1, but those two systems currently require only about 1.3 megawatts (plus whatever cooling and other support load is required at various times).
Further complicating analysis based on power consumption is the fact that the ratio of performance to electrical consumption in supercomputer systems is very sensitive to the design and especially to the date of construction of the system. The #3 computer mentioned above, about 37% as fast as the #1 system but about five years older, requires more than three times as much power as the #1 system requires (more than eight times as much per calculation).
Overall, however, I suspect that these factors increase the likelihood that CSE has the country’s top supercomputers.
Given Pod 1’s more recent construction, and CSE’s generous budgets in recent years, it seems likely both that CSE’s systems are more up to date and thus more power-efficient than the #3 system mentioned above and that Pod 1’s capacity is more fully utilized than the SciNet site’s.
