How to Customize Optical Modules in Data Centers?

The 11th Network Platform Department Technology Summit ended successfully in Shenzhen on July 6, 2022. This summit focused on the four major areas of hardware research and development, hardware acceleration, network products, and network operations. It comprehensively demonstrated the continuous improvement of the network platform department’s research and development capabilities and exploration results.

Optical Modules and Applications

This is the first time that the optical module has appeared at the technology conference. The conference reviewed the development process of optical module products from commercial use to customization. It focused on the future, grasped critical technology paths, and explored new development models at the same time.

An optical module is a small system with complete functions and structure. The electrical signal input is analyzed by chips such as DSP, then enters the driver chip, and then drives the optical chip for modulation. The modulated light enters the passive optical system for coupling, and finally, the coupled light enters the optical fiber system for transmission.

Optical modules are not exclusive to data centers but are firstly used in telecommunications networks, especially long-distance transmission networks. However, with the development of data centers, including the increase in the bandwidth of equipment, optical modules have come into use in the data centers. Since the development speed of data centers is much faster than that of the traditional telecommunications industry, the growth in demand for optical modules in data centers has also far exceeded that of the latter one.

Figure1：Applications of Optical Modules

From commercial use to procurement

For the data center, the early 40G network is a closed system. For example, what the data center buys is the entire network system of the traditional equipment manufacturer, including software, switching equipment, and optical modules. The cost of commercial systems is high, and it is not easy to operate and maintain if there is a problem. These are important reasons for the data centers to switch from commercial use to procurement.

Combined with the online application of 25G servers, 25G SFP28, and 100G QSFP28 optical modules will inevitably face challenges in beginning the procurement. The first challenge is the adaptation among devices. Although there is no problem in the single optical module test, the data center has encountered many equipment compatibility problems in the procurement stage.

The data center found that the modules of different manufacturers have different performances on each port on different devices, and the performance on different devices is also different. A perfect adaptation scheme must be made to form a complex test system, to ensure that different modules can communicate well. It is often necessary to optimize port parameters one-to-one.

Figure2：End-to-end quality/cost control

But it will be profitable if the purchased optical modules can be managed very finely. A large number of 40G QSFP optical modules malfunctioned in the data center between 2015 and 2017. That is mainly because the system integrator outsourced all optical modules, who only had a shallow understanding of optical modules. Besides, the integrator depends on the module manufacturer to intercept the quality problem of the optical module.

If you combine your application scenarios, manage the end-to-end module technical solution, control and optimize its process, and implement the system’s quality management solution properly, you can completely avoid the above problems. Through the continuous optimization and control of the self-lighting module for 2 to 3 years, not only the cost has increased year by year in the data center, but also the quality has been systematically improved, and the failure rate of the production environment has been continuously reduced.

From procurement to customization

Another advantage of purchasing optical modules by yourself is that it can meet the diversified customization needs brought about by the diversified development of data center networks. For example, 10km spectrum monitoring scenarios require 10km modules to be interconnected. Due to the high price of integrated transceiver modules, data centers use the existing module platform to make some minor improvements to achieve low-cost solutions.

Another example is the requirement of DCI for 2km light splitting. The whole project is very profitable, but the technical solution is very challenging. It is necessary to combine the network environment of the data center and carry out in-depth design optimization of modules, which includes a series of upgrades at the optical part of the module and the chip level.

Figure3：Meet a variety of customized needs

With the launch of 100G servers in batches, the 200G network begins mass production applications. The optical modules used in the entire network are upgraded from 100G QSFP28 to 200G QSFP56 or 200G QSFP-DD. The entire industry chain is similar to a “rugby”, with a large middle and small ends. There are more than 200 companies in the entire industry that claim to be able to produce optical modules, but there may be no more than 10 chip manufacturers at the top of the industry chain, especially the ones with first-rate techniques such as DSP.

The number of downstream application parties or end-users is far less than the number of optical module manufacturers, showing a chaotic industrial structure. This also implies some other problems: unclear and unspecific demand, and unclear value of the chip. The first goal of optical modules is to break this ecology, that is, data centers and chip manufacturers directly discuss requirements for specification and cost (consumption), to achieve real end-to-end cost competitiveness.

The data center also launched the multi-party JDM mode, that is, the data center directly puts forward specifications demand, and some customized functions to the chip manufacturer and the manufacturer optimizes or re-develops according to the needs of the data center. The module scheme is jointly designed by the data center and module manufacturers, and the latter is responsible for production.

Figure4: “break the ecology”

Entering the self-research stage, the most important issue for data centers to consider is how to achieve maximum value in the design and selection of solutions. “Light” and “electricity” are fundamentally different in terms of transmission properties. Light has some special properties, such as optical power characteristics at different temperatures, bandwidth characteristics at different currents, and wavelength characteristics under different conditions. In the self-developed mode, the data center can continuously promote the optimization of the solution in combination with the system specifications, and enable the electronic chip to be more suitable for light, to exert greater technical advantages.

Optical modules are often perceived as standard products because the shell shape and input and output interface specifications are defined. However, the internal optical packaging is special, with various form factors. Each manufacturer has a different design and customized manufacturing equipment.

If the data center wants to quickly design and manufacture a self-developed module, the data center will face the problem of platform selection. In the self-developed optical module mode, making full use of the mature 100G QSFP28 optical module packaging platform, and doing some targeted upgrades at the same time, is the best solution to meet the demands of quality stability, development efficiency, and low cost.

Exploring the next stage

The network of the data center iterates every 2 to 3 years. Judging from the road signs of the switching chip, the comprehensive 112G network will come in 2023. Recently, people have been discussing the development of the next-generation 112Gbps-based network, including network cards and switching equipment.

At the optical module level, the data center has also done some technical research in the past two years. From 100G to 400G and 800G, the power consumption of the module will continue to increase. At the same time, BER (Bit Error Rate) will also face exponential degradation. Solving these problems requires increasing the design and optimization work at the chip level, and even participating in chip design. Optical packaging will also face a series of challenges, and various new solutions to improve optical integration need to be explored. In the transmission scheme, the data center also needs to continue to explore to meet the network scenario requirements of the data center.

Figure5: Evolution of Rate

Signal transmission distance is a basic attribute of interconnect hardware, and it is also a key constraint that must be discussed every time data centers discuss future technical solutions and development trends. With the continuous improvement of the basic speed, the physical distance of the computer room or IDC of the data center will not change much.

Therefore, when selecting a module or technical solution for connecting two devices, iteration needs to occur with the iteration of the transmission technology. There are three interfaces in the evolution of “distance”. From the short-distance access side, “optical” goes in and “copper” goes out. From the internal connection of the IDC room, it is single-mode instead of multi-mode. From the perspective of DCI connection, It is the continuous sinking of coherent technology.

In the above aspects, the data center has also made some advanced layouts, including the 112Gbpsl network card access, and the data center is being deployed based on the 112Gbpsl TAC technology and the corresponding chip cooperation development. In terms of connection inside the IDC room, the MM SR module may not be able to meet the connection within the data center room (Building Range), especially the cross-room connection. Here, the data center is also exploring single-mode instead of multi-mode solutions, especially Some joint development attempts have also been made in the single-mode fully integrated chip solution. The choice and replacement of IMDD and coherent, the data center is expected to arrive in the 1.6T or single-wavelength 400G QSFP-DD generation, so the future data center will do some work in the field of Coherent-lite, especially the oDSP algorithm.

Starting from the chip

A key difference between an optical module and traditional electrical devices/equipment is that it not only has “electrical” characteristics, but also “optical” characteristics, so it must face both the Moore’s Law of microelectronics and the Moore’s Law of optoelectronics. With the trend of light entering and copper retreating, the world is constantly integrating core optoelectronic resources. Manufacturers led by giants in the field of traditional electronic chips are also expanding in the field of optoelectronics to prepare for the future trend of optoelectronic integration. Similarly, data centers also need to do something to meet one challenge after another. The development of 112Gbps may be a long process. The 400G system based on 112G Serdes is the best opportunity for data centers to be deployed now and to be explored in the future.