When designing a data center, energy consumption, water consumption, and noise are all constraints that should be taken into consideration early. The main problems that arise when designing a data center revolve around cooling decisions and mechanisms. Servers convert nearly all the electricity they draw into heat as they process data; a single high-density compute rack has passed 125 kilowatts, per the U.S. Department of Energy’s data center design guide, and that heat must be dealt with by using air, water, or both.
Where the heat starts
Many data centers have a computer room air conditioner (CRAC). Hot air from the servers passes over a cooling coil inside the unit. The CRAC sends the cooled air under a raised floor, and the servers draw it back up through floor grates. Newer halls often drop the raised floor for fan walls and hot-aisle containment, and the heat path is the same. The server air passes over a coil, and the coil leads to the outdoors where a direct expansion loop or chilled water can be used to cool the coil.
Two ways to cool the coil
Direct expansion
A direct expansion loop works like a residential air conditioner. It has an evaporator, a compressor, and a condenser. The evaporator coil sits inside the CRAC. As hot server air flows over it, the refrigerant inside boils. “Evaporation is a heat absorbing process,” said Ryan Wirtanen, a chemical engineer at C-P Systems. The refrigerant takes the heat out of the air.
From the evaporator, the refrigerant vapor goes to the compressor and then to the condenser, which usually sits outdoors. A fan moves ambient air across the condenser coil, the refrigerant condenses back to a liquid, and it returns to the evaporator to repeat the cycle. Outside air carries the server heat away.
Chilled water
The second method replaces the refrigerant coil in the room with a coil of cold water. Hot server air passes over the coil, the heat moves from the air into the water, and the warmed water leaves the room for a central chiller. With chilled water, the room unit is often called a computer room air handler (CRAH) rather than a CRAC.
Which is better?
The difference between these two methods is based on where refrigeration happens. A direct expansion loop is compact and local to the room. A chilled water system carries the heat out of the room in water and rejects it at a chiller that can sit on the roof or in a central plant.
Direct expansion fits small and medium rooms, older sites, and retrofits where no chilled water plant exists: each unit carries its own refrigerant circuit and outdoor condenser, so capacity grows by adding units. Chilled water fits large halls and campuses: one central plant serves many air handlers, the loop can take a waterside economizer in cold weather, and it is the same facility water loop that cold plates and immersion systems tie into as rack densities rise.
Where the chilled water goes: air-cooled and water-cooled chillers
A direct expansion loop ends at its outdoor condenser. A chilled water loop ends at a chiller, and chillers come in two kinds.
An air-cooled chiller typically sits on the roof or in the equipment yard. “It’s basically a massive residential air conditioner,” Wirtanen said. It runs the same refrigerant cycle as a direct expansion loop, except its evaporator chills water instead of room air. Ambient air condenses the refrigerant in its coils, and the refrigerant chills the water returning from the data hall before the chiller sends it back.
A water-cooled chiller is built as two shells, typically one above the other. The evaporator sits in the lower shell, where the chilled water loop gives up its heat to the refrigerant. The condenser sits in the upper shell, and cold water rather than air condenses the refrigerant. That condenser water comes from outside the building.
Where the condenser water comes from
A water-cooled chiller can take its cold water from several sources:
- Most often a cooling tower, which rejects heat mainly by evaporating part of the water. The rest recirculates, with makeup water added to replace what evaporated.
- Well water or city water comes out of the same supply the surrounding community uses.
- River water, where a discharge permit allows it, is drawn, warmed in the condenser, and returned, often by way of a holding reservoir open to the air, where it sheds heat before it goes back to the river.
Cooling with liquid at the server
Air can only carry so much heat away from a dense rack. Two approaches put liquid closer to the chips.
Cold plates
Cold plates keep the servers in air and put the liquid only where the heat is densest. A cold plate mounted on each processor carries coolant through a technology cooling system (TCS) loop to a coolant distribution unit (CDU). The CDU is a heat exchanger: it passes the heat into the facility water system (FWS), the loop that feeds the chillers described above, without letting the two fluids mix. The coolant is treated water or a water-glycol mix in a sealed loop, kept clean for channels a fraction of a millimeter wide. The building water takes the heat from the other side of the exchanger and carries it out to the chillers, the same path the rest of the building’s heat takes. Cold plates are the usual answer for racks too dense for air to cool.
Immersion
The second approach removes the air step altogether. The servers sit in a dielectric liquid, a non-conductive fluid that absorbs their heat. In single-phase systems the fluid stays liquid and circulates across the equipment to a heat exchanger. In two-phase systems the fluid boils on the hot components and condenses on a coil above the bath. In either form, a CDU, heat exchanger, or condenser passes the heat into the facility water system, where the chillers or other heat rejection equipment described above carry it away. “Liquid is way better at transferring heat than air,” Wirtanen said.
Open Compute Project (OCP) requirements set minimums for the fluid’s flash point and auto-ignition temperature (how hot it can get before it ignites), its dielectric strength (how well it blocks electricity), and how long it keeps those properties in service. Under the safety standard UL 62368-1, a testing lab evaluates the fluid as a critical component, including its flammability and how it reacts with the materials it touches.
For a system assembled on site, a testing lab certifies a representative build under UL 2416, the standard for equipment cabinets and racks, and evaluates the insulating fluid as a critical component. The listing names the fluid, its container, and the materials the fluid touches, so an installer checks with the certifier before topping off with a different fluid or substituting a gasket material.
Letting the climate do part of the work
Free cooling, which operators usually call economizer mode, uses cold outside air, or water cooled by outside conditions, so the chillers can run less or stay off.
On the air side, cold outside air passes through an air handling unit and goes directly to the server room, with no CRAC in the path. In dry conditions, a fine mist in the airstream cools the air further by evaporation and adds a little humidity. The air picks up the server heat and leaves through an exhaust. An indirect version puts a heat exchanger between the outside air and the hall air, which keeps outdoor dust and humidity out of the hall. Dry regions suit evaporative cooling, Wirtanen said, because low-humidity air can carry more of it, and it is simpler to install than a chilled water system, with no refrigerant circuit or compressor to add.
On the water side, cold condenser water from a cooling tower, a well, or a river cools the chilled water loop through a heat exchanger, so the chillers run less or shut off.
A site can combine methods by season. “Here in Minnesota, we have four distinct seasons,” Wirtanen said. “In the winter you could use the colder air outside to cool your data center, and when it gets really warm in the summer you can use either a water-cooled or air-cooled chiller.” In his view, location rarely rules free cooling out: “There’s almost always a source of ambient or free cooling, regardless of where you are. It’s just a matter of what it is.”
Chip power sets the limit on how far that goes. The hotter the chip, the colder the facility water it needs, and the colder the water, the fewer hours a year the climate can supply it without a compressor. A server rated for warm water, the higher ASHRAE W classes, free cools for more of the year than one rated for the W27 class, which needs supply water at 27 °C (81 °F) or colder. An engineer sizing the plant reads the server’s rated facility water temperature alongside the site’s climate to find how many hours a year the chillers stay off.
Why most systems recirculate
Most of these systems recirculate their water. “You just need to add a little bit of makeup water every now and then,” Wirtanen said. Single-pass systems, which draw water, use it once, and discharge it, are less common, in part because they depend on a water source that stays cold year-round.
Why the cooling design comes first
Owners, permitting authorities, and neighbors ask the same three things about a data center: how loud it will be, how much power it will draw, and how much water it will use. The cooling design is the foundation the rest of the design is built on. Most other questions can be answered once it is addressed.
The noise comes from the outdoor equipment: fans on air-cooled chillers and cooling towers, the fans that move outside air through an air-side economizer, and the backup generators when they run. The building walls block the noise from indoor equipment. What neighbors hear is the equipment in the yard, and that is where the design choices matter: a water-cooled chiller fed by river, well, or city water rejects its heat through water, so it has no fan and makes none of that noise.
Every cooling method runs on electricity, and cooling demand peaks in the same summer months when home air conditioners run; free cooling cuts the hours the chillers run, and liquid cooling, by cold plate or immersion, moves server heat with less energy than air. Water use depends on the condenser water source, since a site either evaporates water in a tower, draws on the community’s supply, or returns river water warmer than it took it. It also depends on the loop, since recirculating loops draw far less water than single-pass systems, though a cooling tower consumes the water it evaporates.
Wirtanen recommends beginning the design of a site with geography. The engineer determines which locations are available and which cooling source each one has: cold air, cold water, or neither. From there, he looks at the systems that move heat with the least energy, and then at the power available to the site. Once an engineer has chosen the coil in the hall, the chiller behind it, and the water or air that takes the heat, that engineer has also decided how much water the site uses, how loud it is at the property line, and how many hours a year the chillers run. When the racks are dense enough for liquid, the engineer makes a second set of choices at the pipe: materials, cleanliness, relief, and glycol. The Liquid Cooling Scope Checklist describes each of these. For questions about cooling, piping, or utility systems on a data center project, contact the C-P Systems engineering team.
