Bringing up the Ceres FPGA dev board

In which we admit three embarrassing design errors, but snatch victory from the jaws of defeat, and have pretty pictures to show for it.

I received two Ceres prototypes from JLCPCB earlier this month! Not wanting to smoke-test them in my Linux machine’s PCI Express slot, I wired them up using a PCIe-to-M.2 adapter, then an M.2-to-Thunderbolt adapter to my Mac. I connected the DC power supply, and… pop! The 0.85V switching regulator for main power to the FPGA let out its magic smoke. Turns out its enable pin had an internal pullup to +5V, and wasn’t designed to tolerate +12V.

RIP Ceres serial number 1.

On Ceres serial number 2, the relevant trace was too short to cut (it didn’t stick out far enough from under the regulator die), so I continued the bringup with a 5V power supply instead of the 12V one I’d intended to use. This time, there was no magic smoke, the FPGA showed up in Vivado, and I was able to program the QSPI Flash memory with a bitstream. Hooray!

A PCI Express card sitting on a wooden table, with a DDR4 DRAM stick, a QSFP28 loopback module, two half-length SYZYGY modules, and a JTAG connector plugged into it. The card edge is plugged into a PCIe-to-M.2 adapter, which is in turn connected to a DC power supply and an M.2-to-Thunderbolt adapter.

Except that the PCIe link never came up, and the DDR4 controller failed to train. Apparently that happens if your PCIe TX pairs are all swapped with the corresponding RX pairs, and if all your DDR4 clock and data strobes are polarity-inverted. I really should’ve commissioned one of my fellow EE friends to do a design review; it would’ve been cheaper than a board spin and taken less time.

Fortunately, at least the SYZYGY and QSFP28 transceiver interfaces were connected properly, and I was able to test them using passive loopback modules. All four QSFP28 links were stable at 27.8125 Gbps with BER under 1e-12 and qualitatively good-looking statistical eye diagrams. The SYZYGY links were marginal at 28 Gbps with closed statistical eyes and BER around 1e-6, but rock-solid at 20 Gbps. (Samtec rates the SYZYGY TXR4 connectors for up to 19 Gbps.)

Eight statistical eye diagrams showing marginal links at 28Gbps for channels X0Y8 through X0Y11 and good links at 27.8125Gbps for channels X0Y12 through X0Y15. Four statistical eye diagrams shoing good links at 20Gbps for channels X0Y8 through X0Y11.

I also did some power-supply testing with a clip-on heatsink over the FPGA. For this test, I implemented an array of 65,536 ring oscillators to maximize power consumption, and a MicroBlaze RISC-V core running a proportional-control algorithm to keep supply voltage and junction temperature (as measured by the onboard ADC) and total system power (as measured by an INA226 current monitor) within limits. The plot below shows the number of active ring oscillators (“bbq”) over time, along with power consumption, Vccint supply voltage, and temperature.

A plot of power consumption, supply voltage, and temperature. Power increases rapidly from zero to its peak value in the first 40 seconds of the test, then plateaus around 22W until temperature reaches 100C two minutes later, at which point it drops off to about 15W. Voltage drops rapidly from 0.85V to 0.825V as power increases, then gradually recovers to about 0.833V.

With the FPGA at ambient temperature, total power is limited by supply voltage droop to about 24 W, likely due to IR drop on the PCB between the 0.85V supply and the FPGA pins. The FPGA takes about two minutes to heat up to 100 C, at which point total power levels out around 15 W. Assuming 85% efficiency for the power delvery system, about 13 W is actually being dissipated by the FPGA heatsink. For revision B, I’ve moved the 0.85V regulator’s sense connection closer to the FPGA and ordered an active fan/heatsink; I’m hoping for closer to 40W thermal design power.

Changelog for revision B

I’ve made some other changes to fix minor problems, and I’m just waiting for parts availability before I place an order for revision B. Fingers crossed that this is what we go to production with!