Why Are Some PCI Express Slots in my Intel Server or Workstation not Available?

Problem

• Some Xeon E5 workstation or server based systems may not have all PCI Express Slots available for use.
• When you plug a card into a Slot that is not available, the card is not detected at all in the BIOS or in Windows.

Cause

In a modern workstation or server system, the PCI Express Slots are almost without exception connected to the PCI Express Lanes on the system processors. This enables fast transfers and low latencies. Most Xeon E5-2400 or Xeon E5-2600 systems support up to two CPU sockets. Many (but not all) of these systems split the PCI Express slots in the system between the two processors.

This means that if you have a dual CPU socket system, but only one processor is fitted, then some PCI Express Slots may be unavailable.

Resolution

• Refer to the motherboards technical product specification (TPS) to see which slots are connection to which processor. Generally speaking, the PCI Express slots nearest to the system memory are available, as they run from Processor 1 on most motherboards. The PCI Express slot furthest away from the system memory may also be available, however this will be a PCH Slot and may suffer from increased latency.
• Use the Intel Support Page to find TPS documents for Intel Server boards or see the attached article for some commonly used board TPS documents.
• Fit add-in cards only to slots which are available.

More Information

Sample Layout Diagram 1 - Intel S2600CW

The first diagram shows the Slot availability as part of the chipset and processor layout; the second diagram shows the availability by physical slot.

Sample Layout Diagram 2 - Intel S2600WT in an Intel R2308WTTYS chassis

The first diagram shows the Slot availability as part of the chipset and processor layout; the second diagram shows the availability by physical riser slot.

The three riser diagrams from the System's Technical Product Specification show the PCI Express availability at each slot, and which processor provides the connectivity.

Terminology

PCH Slots

PCI Express Slots connected to the PCH are usually x4 or x8 PCI Express slots connected to the motherboard chipset. Because they don't have a direct connection to the processor, the performance of these slots is often slightly lower. However they will always be available for use in whichever processor configuration.

MUX Slots

MUX Slots are PCI Express Slots that share some or all of the PCI Express lanes with other slots or devices. For example, a PCI Express x8 MUX based slot might have four lanes coming directly from the processor, and four more lanes that go via a MUX. The MUX shares the PCI Express Lanes between that slot and another slot.

MUX based slots usually do not have any compatibility issues but bear in mind that the bandwidth on MUXPCI Express lanes is shared.

DMI Slots

PCI-DMI Slots are PCI Express Lanes running from a CPU. However, these lanes are normally used for CPU to Chipset communications. When implemented as PCI Express Lanes, the result is often a PCI Express slot that is slower or smaller, for example Gen.2 and x4.

The Difference between Physical or Mechanical, And Electrical

Some PCI Express Slots may not be fully wired. For example, a PCI Express x16 connector may, on some systems, only be x8 electrically wired. This means that half of the PCI Express lanes are unavailable. Plugging in a PCI Express x16 device may work if the device supports running with only half of the lanes connected, but it will run at reduced bandwidth.

PCI Express Generations

There are different generations of PCI Express available. PCI Express Generation 3 (or Gen.3) supports 8 Giga-tranfers per second, resulting in a little less than 1GByte a second of transfer performance per lane.

If you have a PCI Express Generation 3 device, always use PCI Express Generation 3 slots in preference to PCI Generation 2 slots wherever possible.

Applies to:

• Stone Xeon E5 based server system families, including S2400 and S2600 servers.