Exploring PCI Express over Fiber: A Technical Journey

Welcome to the lab, where I'm thrilled to explore the concept of transporting PCI Express over fiber optics. This idea has been simmering in my mind for nearly a decade, but it's only in the past year that I've begun experimenting with it. My ultimate goal is to have a remote box on my workbench to host PCIe cards, connected to my main workstation across the room. While commercial solutions exist, they are often expensive and not widely available. Thunderbolt cables are an option, but they encapsulate PCI Express, adding complexity I'd prefer to avoid. Plus, I want to leverage the single-mode fiber bundles already running through my lab.

This project is as much about learning as it is about achieving a practical setup. Initially, I aimed for a modest Gen 2x1 link, but I've since set my sights on a Gen 3x4 link, which fits well with affordable optics like QSFP 40 gig L4 modules. These modules can handle four 10 gig links on a single fiber pair using wavelength multiplexing.

PCI Express is a high-speed serial link, and SFP optics have a similar high-speed interface. However, simply wiring them together isn't straightforward due to potential complications. One issue is whether SFP modules are transparent to PCIe data and can operate at different line rates. Fortunately, 10 gig modules generally work well, performing basic on-off keying and amplification without complex retiming.

Another challenge is the electrical interface, where AC coupling caps and termination styles are slightly out of spec for PCIe. This affects receiver detection, as PCIe uses common mode pulses to detect link partners, which SFP modules don't support. Out-of-band communication isn't a problem since PCIe doesn't rely on non-differential low-speed signals.

Spread spectrum clocking (SSC) is another hurdle. Many PCIe hosts modulate their clock to meet EMI standards, but since we're only sending data lanes through the SFP, not the reference clock, SSC becomes an issue. Additionally, Gen 3 introduces link equalization training, which the optical conversion disrupts.

To overcome these challenges, I've set up a test with a Raspberry Pi 5 and a PCIe breakout box, using a Blackmagic Decklink capture card as a test device. The setup involves custom cards with SFP modules and single-mode fiber. The goal is to achieve a working Gen 1 or Gen 2 link, with Gen 3 posing more challenges due to equalization requirements.

I've also embarked on a side quest to create a PCI Express analyzer using optical splitters. By splitting the signal, I can send one half to the device and the other to an FPGA dev board for analysis. This setup allows for convenient signal capture without the need for electrical splitting, which can degrade the signal over long distances.

Future steps include creating new test boards with improved red drivers and connectors, and extending the link to a 4x width using QSFP modules. I'm also working on making the PCIe analyzer more accessible and efficient, with plans to publish the setup for others to replicate.

This journey is ongoing, and I invite you to follow along as I continue to explore the possibilities of PCI Express over fiber. Your feedback and suggestions are welcome, and I look forward to sharing more updates as the project progresses.