In 2024, NVIDIA released the NVL72/NVL36 solution, which increased the demand for the construction of fully liquid-cooled data centers.
The two types of liquid cooling used on a large scale in the data center field are cold-plate and submerged liquid cooling. Other types such as spray cooling have not been deployed on a large scale.
Partial and overall renovation of the computer room
Why do switches need liquid cooling?
The focus of network is on switch liquid cooling, and the focus of server is on GPU liquid cooling.
Over the past 12 years, the power consumption of a typical switch system has increased by 22 times, Optics by 26 times, and ASIC Serdes by 25 times. The power consumption growth rate has exceeded the speed of improvement of air cooling capacity.
Power consumption growth of typical switch
Switch liquid cooling requirement 1: Taking the 128 x 3.2T DR16-DSP device as an example, optical transceiver consumes more power than the GPU and has a lower temperature resistance than the GPU. The space that can be added to the radiator is much smaller than that of the GPU.
Switch Liquid Cooling Requirements
Switch liquid cooling requirement 2: The power consumption and heat flux density of the switch chip are greater than those of the GPU.
Three Difficulties in Designing Liquid-Cooled Switches
Difficulty 1: The switch cold plate system belongs to “three-dimensional” assembly, and the GPU liquid cooling cold plate system belongs to “two-dimensional” assembly. The difficulty of three-dimensional is much greater than that of two-dimensional.
Single-layer cold plate for switches: The cold plate kit is assembled or tested by the radiator factory, and the ODM only needs to tighten the screws in the XY direction to assemble the whole machine.
Multi-layer cold plate for switches: strongly relies on the ODM factory’s XYZ “three-dimensional” direction to assemble and test the cold plate kit, and the ability required is far more than just tightening screws.
Difficulty 2: Hardware is bound to have wear and tear, and the same is true for liquid cooling. The impact of a single switch failure is far greater than that of a single server. How can we ensure that a single switch does not fail even if there is a leak?
Difficulty 3: Cold plate design of Belly-belly optics is easier, but the system cost is higher. Stack opitcs solution has lower cost, but as of 2024, there will be no cold plate solution of bottom optics in stack cage available on the market.
Alibaba and Ruijie jointly develop Liquid Tigatron switches
For the Difficulty 1: The customized automated liquid cooling assembly platform based on the S³IP UNP concept has been verified through mass delivery and can realize the “three-dimensional” rapid assembly and airtight testing of the TH5 Tigatron switch liquid cooling kit.
For the Difficulty 2: The TH5 Liquid Tigatron switch implements a design that separates the liquid circuit and the electric circuit. Liquid leakage in high-risk areas will not leak onto the circuit board, thus improving availability.
For the Difficulty 3: A low-cost solution with water pipes embedded in aluminum plates supports cascaded liquid cooling of multiple 2×4 stack cage upper and lower optics, solving industry difficulties (one cold plate covers 22 optical transceiver, ensuring good contact and heat dissipation for all modules, and thermal performance does not decrease after 50 to 100 plug-in and unplug cycles).
Besides bringing coolness, what else does liquid cooling do?
Liquid cooling is not just a simple heat dissipation method, but more like a lubricant for the “innovation engine” in the field of network equipment.
For example: the recently popular CPO “corrosion inhibitor” Vertical Switch design requires liquid cooling to continue to evolve.
Liquid-cooling-enable V-switch solution
Liquid-cooling-enable V-switch solution: single-layer PCB supports 128 pluggable ports with optimal SI performance and low system cost.
There is also the CPO switch that has been controversial in the industry for many years: cold plate liquid cooling is a mandatory requirement.
Liquid-cooling-enable CPO solution
Liquid cooling is about to usher in a larger-scale market explosion. Let us witness the emergence of more liquid cooling design cases.