Microsoft’s Belady on Immersion: ‘Air Cooling is Not Enough’

Microsoft has begun using immersion-cooled servers in production, the company said this week, providing an update on its strategy to manage rising power densities and heat in its cloud data centers. Microsoft had been test-driving cooling technology used in bitcoin mining facilities in which servers are dunked in tanks of cooling fluid to manage rising heat densities, as DCF reported last month.

In a blog post Tuesday, Microsoft disclosed that it has begun using immersion-cooled servers in a data center on its campus in Quincy, Washington. The single rack of 48 servers uses two-phase immersion cooling, in which servers are immersed in a coolant fluid that boils off as the chips generate heat, removing the heat as it changes from liquid to vapor. The vapor then condenses into liquid for reuse, all without a pump.

“We are the first cloud provider that is running two-phase immersion cooling in a production environment,” said Husam Alissa, a principal hardware engineer on Microsoft’s data center team. The Quincy installation is initially being used on Microsoft’s internal emails, rather workloads for its cloud computing customers.

A Validation for Immersion at Scale?

A single rack in a cloud data center may not seem like a huge deal. But Microsoft’s adoption is a major milestone for immersion cooling, a technology which offers advanced cooling capabilities but has lacked a high-profile customer that could kick-start adoption. For years, advocates of immersion cooling have argued that support from one of the large hyperscale computing operators could be a tipping point for broader use.

Microsoft has now emerged as that marquee customer, and is sharing its story with the world, first in a presentation by Azure CTO Mark Russinovich at Microsoft Ignite and now on its blog. Significantly, Microsoft project has the support of key decision makers, including Russinovich and Christian Belady, distinguished engineer and vice president of Microsoft’s datacenter advanced development group.

“Air cooling is not enough,” said Belady. “That’s what’s driving us to immersion cooling, where we can directly boil off the surfaces of the chip. Liquid cooling enables us to go denser, and thus continue the Moore’s Law trend at the datacenter level.”

Christian Belady, distinguished engineer and vice president of Microsoft’s datacenter advanced development group, stands next to a two-phase immersion cooling tank at a Microsoft data center. (Photo: Microsoft)

Moore’s Law is the famous prediction by Intel engineer Gordon Moore that the number of transistors per silicon chip roughly doubles every year. As processor sizes continue to shrink, engineers debate whether the industry is still able to deliver on Moore’s Law at the chip level.

But the same kind of massive productivity gains are still possible at the data center level, Belady asserts, and Microsoft is investing in bold research and development initiatives to deliver those gains. Other “moonshot” initiatives include:

• Underwater data centers for edge computing scenarios, complete with controlled environments that may also extend the life of servers, bring benefits to data center economics.
• The use of hydrogen fuel cells to power its data center servers, a first step toward adopting new technologies to fulfill its pledge to eliminate the use of diesel fuel in its emergency power systems
• Data storage using DNA and holograms to house massive amounts of data, and infrastructure that could disrupt how data centers are designed and operate.
• Containerized data modules that use satellite broadband connectivity, allowing capacity to be deployed in geographies with no fiber infrastructure.

An Opportunity for Immersion Cooling Specialists

Belady has been a key pioneer in the data center industry, playing a role in the development of energy efficiency metrics and early use of fresh air cooling. He has also had a front-row seat for the extraordinary growth of cloud data centers, a trend reflected in Microsoft’s planning for massive future cloud campuses.

Microsoft’s adoption of immersion hints that this hyperscale future may create a huge opportunity for high-density cooling vendors, who have historically deployed their gear at modest scale in a select number of high-performance computing (HPC) clients. While there are multiple technologies using liquid cooling, Microsoft says its testing has highlighted two-phase immersion as the most promising way forward.

The Microsoft testing lab’s implementation used a two-phase immersion system from Allied Control, a subsidiary of bitcoin specialist Bitfury that has recently been spun off as LiquidStack, with financial backing of hyperscale hardware maker Wiwynn. In this week’s blog post, Microsoft named Wiwynn as its technology vendor.

Other companies specializing in two-phase immersion include TMGcore, which also developed its technology in bitcoin mining applications.

“Because of spatial and technological constraints, the data center industry has been limited in its growth potential,” said John-David Enright, CEO of TMGcore, who said two-phase liquid immersion cooling “will move data centers closer to urban centers and significantly reduce environmental impact.”

There’s also ZutaCore, which uses a different two-phase technique that delivers the dielectric fluid directly to the processor, using a piping system to bring the liquid inside the server chassis, rather than immersing the entire server.

Immersion as an Edge Computing Solution

Recent weeks have also seen some traction for single-phase immersion specialists, most notably GRC Cooling, which is now partnering with data center equipment giant Vertiv on an immersion cooling enclosure rolling out in Asia-Pacific markets.

“Liquid immersion cooling is increasingly becoming an attractive option for many data center and colocation facilities, as it not only supports high-compute environments but also promises improved efficiencies, sustainability and reliability compared to traditional cooling solutions,” said Dhruv Varma, vice president of Business Development, APAC at GRC.

Vertiv says its Liebert VIC enclosure will be relevant in edge computing and Internet of Things scenarios. That potential is affirmed by Belady, who said Microsoft’s immersion technology could be optimized for low-latency, high-performance applications with low maintenance requirements. An immersion tank could be deployed under a 5G cellular communications tower in the middle of a city for applications such as self-driving cars.

For now, Microsoft will spend the next several months performing a series of tests to prove the viability of the tank and the technology.

“This first step is about making people feel comfortable with the concept and showing we can run production workloads,” Belady said.