With the continuous surge of data and the explosive growth of computing-intensive applications, traditional electronic interconnection technology gradually show its limitations when it comes to meeting multi-dimensional demands such as high bandwidth, low latency, and energy saving. In this context, optical interconnection technology has gradually become the focus of the industry with its unique performance advantages and is regarded as the key to breakthrough in future data center interconnection technology.
As a high-speed serial point-to-point interconnection standard widely adopted in the industry, PCIe has undergone many iterations and upgrades since its birth, and has now become an indispensable interconnection bridge between computing devices such as CPU, GPU, FPGA, and SSD. The PCIe 7.0 standard increases data transfer rates to an astonishing 32 GB/s (per lane).
PCI-SIG announced continued progress on the PCI Express 7.0 specification at the June 2022 U.S. Developers Conference (US DevCon). Version 0.3 was released in June 2023. The latest development is that PCIe 7.0 ver0.5 has been released, and it has been announced that the official SPEC specification will be released in 2025.
As accelerated computing becomes mainstream, the role of PCIe connections in systems becomes even more important. Direct GPU-to-GPU communication is critical for scaling complex computing tasks across multiple graphics processing units (GPUs) or accelerators within servers and computing pods. There is a growing recognition within the industry that there is a growing need for scalable, open architectures in high performance computing. As artificial intelligence and data-intensive applications continue to advance, demand for such technologies is expected to increase, making PCIe 7.0 a critical component of next-generation interface IP.
I/O Bandwidth Doubles Every 3 Years
In applications such as AI training, large-scale data analysis, and real-time rendering, data transmission speed and efficiency are key factors in determining system performance. The 128 GT/s bandwidth provided by PCIe 7.0 is double the previous generation, which not only means that data can flow between GPUs faster, but also supports more efficient resource allocation and workload balancing. For example, in large-scale machine learning model training scenarios, multiple GPUs are directly connected via PCIe 7.0, which can significantly shorten training time and increase model iteration speed, thereby accelerating the development and deployment of new algorithms and services.
AMD/Broadcom Expand the AMD Infinity Fabric
Although the PCIe protocol optimizes latency performance, the increased distance and interconnection levels within large-scale data centers will lead to accumulated latency. As data rates increase, interconnect power consumption becomes increasingly prominent, posing challenges to overall energy efficiency and thermal design.
In such an environment, the demand for high-speed data transmission within a data center and between different data centers has increased dramatically. AI applications such as deep learning and machine learning require processing massive amounts of data and have extremely high requirements for real-time performance and bandwidth. The traditional copper wire medium PCIe bus has physical limitations in transmission distance, bandwidth expansion and signal integrity, which is difficult to meet the long-distance, high-bandwidth data transmission requirements between large-scale data centers.
Therefore, the industry began to explore the application of PCIe protocol on optical links, the so-called PCIe over Optics technology. This technology converts PCIe signals into optical signals for transmission, utilizing the large capacity, low loss, and long-distance transmission characteristics of optical fiber to break through the distance and speed bottlenecks of traditional copper wire interfaces.
PCI-SIG has also established the PCIe Optical Workgroup, which is committed to developing standards for PCIe over Optical to achieve seamless integration with the existing PCIe ecosystem. For example, Synopsys is working with photonic technology vendors such as OpenLight to develop supporting optical solutions.
End to End PCle 7.0 over Optics at 128 Gbps
PCIe over Optics solutions usually include the following key links:
- Photoelectric conversion: First, advanced electro-optical transceivers are used to convert the electrical signals on the server motherboard into optical signals to ensure that the data can be transmitted over optical fibers.
- High-performance controller and PHY: Design dedicated high-performance PCIe controller and physical layer chip (PHY) to support the latest PCIe standards and optimize its connection with the optical interface to ensure the integrity and accuracy of data after optical transmission.
- Optical engine: Use a highly efficient optical engine, such as the Nubis XT1600 linear optical engine, which provides multi-channel, full-duplex data transmission capabilities while maintaining low latency and power consumption.
- Long-distance lossless transmission: Since optical fiber is not affected by electromagnetic interference, it can achieve kilometer-level lossless transmission, greatly expanding the connection range between server clusters and allowing equipment in the data center to be more flexibly distributed in a larger space without affecting data communication performance.
- Modularity and standardization: Although some solutions may be customized products, the overall trend is towards modularization and standardization, ensuring that products from different manufacturers are compatible with each other, making it easier for data center operators to flexibly choose and integrate them into existing or future data center architectures.
At PCI-SIG DevCon 2024, Cadence demonstrated an optical connectivity solution designed for the next-generation PCIe 7.0 standard. This is the industry’s first PCIe 7.0 demonstration based on optical technology, an achievement that marks a major leap in data transmission speed and interconnect technology.
Cadence built a complete subsystem for the test chip, which includes an eight-channel PHY (physical layer interface) and a complete 8-channel controller. Integrating the controller into the test chip significantly improves the efficiency and precision of characterization and stress testing. Cadence PCIe 7.0 PHY IP diagram:
Cadence PCIe 7.0 PHY IP Diagram
Cadence is able to simulate and test the entire protocol stack as a unit, which covers a wide range of configurations that customers use in real-world applications. A major advantage of doing so is that it can simulate real-world working conditions more comprehensively, identify and resolve problems that may exist in the early stages of the design process, and thus significantly reduce the various risks that customers may face in their system-on-chip (SoC) designs.
Simulate and test the entire protocol stack
Cadence successfully demonstrated the robust transmit (TX) and receive (RX) capabilities of its 128GT/s PCIe 7.0 IP under real-world application conditions, including a low-latency, retiming-free linear optical connection.
The Cadence team not only demonstrated the speed potential of this advanced interface, but also maintained a very stable pre-forward error correction (pre-FEC) bit error rate (BER) of approximately 3E-8 over two days of non-stop demonstrations. This value is far lower than the 1E-6 required by the PCIe specification, which means that even under extreme conditions, the system can provide sufficient redundancy to ensure accurate data transmission, leaving ample room for subsequent RS error correction coding. This means that even if there are tiny errors during data transmission, the system has the ability to correct itself and ensure the integrity of the data.
From the receiving-end PAM4 eye diagram histogram displayed by Cadence, it can be seen that the linearity and margin of the signal are excellent, which reflects its high stability and reliability when processing high-speed data transmission. The wide and clear eye diagram indicates that the signal can maintain high quality after long-distance transmission, which is critical to ensuring efficient data exchange in complex system environments.
This demonstration is not only a powerful proof of the potential of PCIe 7.0 technology, but also heralds a new breakthrough in optical interconnect technology in the field of high-speed data transmission. By overcoming the signal attenuation and interference problems of traditional copper cables at high speeds, optical connections open new doors for the widespread application of PCIe 7.0, especially in areas such as high-performance computing (HPC), artificial intelligence (AI), and data centers that have extremely high requirements for data transmission speed and latency.
With the advancement and commercialization of the PCIe 7.0 standard, such optical connection solutions are expected to become a key component of the next-generation computing platform and data center architecture, driving the entire industry towards a more efficient and faster data processing era.