Intel Motherboard Hardware v2.

Author: Harry Li, Engineer Author: Amir Michael, Engineer

Scope
This document defines the technical specifications for the Intel motherboard used in Open Compute Project servers.

Contents
1 Scope ......................................................................................................................................... 2 2 Contents .................................................................................................................................... 2 3 Overview ................................................................................................................................... 5 3.1 3.2 4.1 4.2 4.3 4.4 4.5 5.1 5.2 5.3 5.4 5.5 6.1 6.2 6.3 6.4 6.5 6.6 6.7 6.8 6.9 License .......................................................................................................................... 5 CAD Models .................................................................................................................. 5 Block Diagram .............................................................................................................. 6 Placement and Form Factor ......................................................................................... 6 CPU and Memory ......................................................................................................... 8 Platform Controller Hub ............................................................................................... 8 PCB Stackup .................................................................................................................. 8 Block Diagram ............................................................................................................ 10 Placement and Form Factor ....................................................................................... 10 CPU and Memory ....................................................................................................... 12 Platform Controller Hub ............................................................................................. 12 PCB Stackup ................................................................................................................ 12 BIOS Interface and Size .............................................................................................. 13 BIOS Socket ................................................................................................................ 13 BIOS Source Code ....................................................................................................... 13 BIOS Configuration and Features ............................................................................... 13 BIOS Setup Menu ....................................................................................................... 14 PXE Boot ..................................................................................................................... 14 Other Boot Options .................................................................................................... 14 Remote BIOS Update .................................................................................................. 14 Event Log .................................................................................................................... 15

4 Entry Motherboard Features .................................................................................................... 6

5 Efficient Performance Motherboard Features ........................................................................ 10

6 BIOS ......................................................................................................................................... 13

June 23, 2011

Open Compute Project Intel Motherboard Hardware v2.0

7 Hardware Monitoring .............................................................................................................. 16 7.1 7.2 7.3 8.1 8.2 8.3 8.4 8.5 9.1 9.2 9.3 9.4 9.5 10.1 10.2 10.3 10.4 10.5 10.6 10.7 10.8 10.9 11.1 11.2 11.3 11.4 11.5 Thermal Sensors ......................................................................................................... 16 Fan Connection .......................................................................................................... 16 Fan Control Algorithm ................................................................................................ 17 PSU Connector ........................................................................................................... 17 Fan Connectors .......................................................................................................... 18 Motherboard Connectors .......................................................................................... 18 Motherboard Power-up Delay ................................................................................... 19 Hot Swap Controller ................................................................................................... 19 Input Voltage .............................................................................................................. 19 CPU Voltage Regulation Module (VRM) ..................................................................... 20 Hard Drive Power ....................................................................................................... 20 System VRM Efficiency ............................................................................................... 21 Power On .................................................................................................................... 21 PCIe x16 Slot/Riser Card ............................................................................................. 21 PCIe External Connector ............................................................................................. 23 PCIe Mezzanine Card .................................................................................................. 24 DIMM Connector ........................................................................................................ 26 Network ...................................................................................................................... 26 USB Interfaces ............................................................................................................ 26 SATA ........................................................................................................................... 27 Debug Header ............................................................................................................ 27 Switches and LEDs ...................................................................................................... 28 Fixed Locations ........................................................................................................... 30 PCB Thickness ............................................................................................................. 31 Heat Sinks ................................................................................................................... 31 Silkscreen ................................................................................................................... 31 DIMM Connector Color .............................................................................................. 31

8 Midplane ................................................................................................................................. 17

9 Power System .......................................................................................................................... 19

10 I/O System ............................................................................................................................... 21

11 Mechanical .............................................................................................................................. 30

12 Environmental Requirements ................................................................................................. 31

http://opencompute.org

12.1 13.1 13.2 13.3

Vibration and Shock ................................................................................................... 32 Disallowed Components ............................................................................................ 32 Capacitors and Inductors ........................................................................................... 32 Component De-rating ................................................................................................. 32

13 Prescribed Materials ............................................................................................................... 32

June 23, 2011

Open Compute Project Intel Motherboard Hardware v2.0

Overview
When data center design and hardware design move in concert, they can improve efficiency and reduce power consumption. To this end, the Open Compute Project is a set of technologies that reduces energy consumption and cost, increases reliability and choice in the marketplace, and simplifies operations and maintenance. One key objective is opennessthe project is starting with the opening of the specifications and mechanical designs for the major components of a data center, and the efficiency results achieved at facilities using Open Compute technologies. One component of this project is a custom motherboard. This document describes both Open Compute Project Intel motherboards: the Intel entry board and the Intel efficient performance board. The motherboard is power-optimized and barebones, designed to provide the lowest capital and operating costs. Many features found in traditional motherboards have been removed from the design.

3.1

License
As of June 23, 2011, the following persons or entities have made this Specification available under the Open Web Foundation Final Specification Agreement (OWFa 1.0), which is available at http://www.openwebfoundation.org/legal/the-owf-1-0agreements/owfa-1-0: Facebook, Inc. You can review the signed copies of the Open Web Foundation Agreement Version 1.0 for this Specification at http://opencompute.org/licensing/, which may also include additional parties to those listed above. Your use of this Specification may be subject to other third party rights. THIS SPECIFICATION IS PROVIDED "AS IS." The contributors expressly disclaim any warranties (express, implied, or otherwise), including implied warranties of merchantability, noninfringement, fitness for a particular purpose, or title, related to the Specification. The entire risk as to implementing or otherwise using the Specification is assumed by the Specification implementer and user. IN NO EVENT WILL ANY PARTY BE LIABLE TO ANY OTHER PARTY FOR LOST PROFITS OR ANY FORM OF INDIRECT, SPECIAL, INCIDENTAL, OR CONSEQUENTIAL DAMAGES OF ANY CHARACTER FROM ANY CAUSES OF ACTION OF ANY KIND WITH RESPECT TO THIS SPECIFICATION OR ITS GOVERNING AGREEMENT, WHETHER BASED ON BREACH OF CONTRACT, TORT (INCLUDING NEGLIGENCE), OR OTHERWISE, AND WHETHER OR NOT THE OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

3.2

CAD Models
The following CAD files are incorporated by reference as if fully set forth in this specification: TBD

http://opencompute.org

4
4.1

Entry Motherboard Features

Block Diagram
Figure 1 illustrates the functional block diagram of the entry motherboard.

Figure 1 Entry Motherboard Functional Block Diagram

4.2

Placement and Form Factor

The motherboard's form factor is 6.5x20 inches. Figure 2 illustrates board placement. The placement shows the relative positions of key components, while exact dimension and position information is available in the mechanical DXF file. The ODM should strictly follow the form factor, PCIe slot position, front IO port positions, PCIe mezzanine card connector position, power connector, and mounting holes, while other components can be shifted based on layout routing as long as relative position is maintained. As shown in Figure 25, one Open Compute chassis accommodates two motherboards. In order to remove and install one board without affecting the other board, the following internal connectors are placed as close as possible to front of the board in order to have easy frontal access: One vertical SATA signal connector and one SATA power connector. Debug card header.

June 23, 2011

Open Compute Project Intel Motherboard Hardware v2.0

Figure 2 Entry Board Board Placement

http://opencompute.org

4.3

CPU and Memory

The motherboard uses Intel next generation Intel Xeon processors. The motherboard supports these features: 2 Intel processors 1 full-width Intel QuickPath interconnect (QPI) link Single Processor Mode DDR3 direct attached memory support on cpu0 and cpu1

4.4

Platform Controller Hub

The Freedom entry board uses the next generation Intel platform controller hub (PCH), which supports the following features: USB ports Serial attached SCSI ports SPI interface SMBUS interface (master and slave)

4.5

PCB Stackup
The entry board's PCB stackup and impedance control are defined in the following tables.
Layer L1 L2 L3 L4 L5 L6 L7 L8 Plane Description Solder Mask TOP GND1 IN1 VCC1 VCC2 IN2 GND2 BOT Signal PrePreg Ground Core Signal PrePreg Power Core Power PrePreg Signal Core Ground PrePreg Signal Solder Mask Total
Figure 3 Entry Board PCB Stackup

Copper (oz) 0.5+1.0 2.0 1.0 2.0 2.0 1.0 2.0 0.5+1.0

Thickness (mil) 0.5 1.9 2.7 2.6 4.0 1.3 25.0 2.6 4.0 2.6 25.0 1.3 4.0 2.6 2.7 1.9 0.5 85.2

Dielectric (er) 3.8 3.5 3.7 4.4 4.1 4.4 3.7 3.5 3.8 Tolerance: +/-8mil

June 23, 2011

Open Compute Project Intel Motherboard Hardware v2.0

Trace Width (mil) 4.0 6.5 5.0 3.9 3.8 5.0 4.0 4.5 5.0 8.0 3.9 4.4 5.0

Air Gap Spacing (mil)

Impedance Type Single Single Single

Layer 1,8 1,8 1,8 1,8 1,8 1,8 3,6 3,6 3,6 3,6 3,6 3,6 3,6

Impedance Target (ohm) 50 40 43 83 95 85 53 50 48 38 85 95 85

Tolerance (+/%) 15.0 15.0 15.0 17.5 17.5 17.5 10.0 10.0 10.0 10.0 12.0 12.0 12.0

4.1 8.2 7.0

Differential Differential Differential Single Single Single Single

4.1 9.1 6.5

Differential Differential Differential

Figure 4 Entry Board PCB Impedance Control

http://opencompute.org

5
5.1

Efficient Performance Motherboard Features

Block Diagram
Figure 5 illustrates the functional block diagram of the efficient performance motherboard.

Figure 5 Efficient Performance Motherboard Functional Block Diagram

5.2

Placement and Form Factor

The motherboard's form factor is 6.5x20 inches. Figure 6 illustrates board placement. The placement shows the relative positions of key components, while exact dimension and position information is available in the mechanical DXF file. Form factor, PCIe slot position, front IO port positions, PCIe mezzanine card connector position, power connector, and mounting holes should be followed strictly, while other components can be shifted based on layout routing as long as relative position is maintained. As shown in Figure 25, one Open Compute chassis accommodates two motherboards. In order to remove and install one board without affecting the other board, the following internal connectors are placed as close as possible to front of the board in order to have easy frontal access: One vertical SATA signal connector and one SATA power connector. Debug card header.

10

June 23, 2011

Open Compute Project Intel Motherboard Hardware v2.0

Figure 6 Efficient Performance Board Placement

http://opencompute.org

11

5.3

CPU and Memory

The board uses next generation Intel Xeon processors. It supports these features: 2 Intel processors Single Processor Mode DDR3 direct attached memory support on cpu0 and cpu1

5.4

Platform Controller Hub

The Freedom efficient performance board uses the next generation Intel platform controller hub (PCH), which supports the following features: USB ports Serial attached SCSI ports SPI interface SMBUS interface (master and slave)

5.5

PCB Stackup
The efficient performance board's PCB stackup and impedance control are defined in the following tables.
Layer L1 L2 L3 L4 L5 L6 L7 L8 L9 L10 L11 Plane Description Solder Mask TOP GND1 IN1 GND2 IN2 VCC1 VCC2 IN3 GND3 IN4 GND4 Signal PrePreg Ground Core Signal PrePreg Ground Core Signal PrePreg Power Core Power PrePreg Signal Core Ground PrePreg Signal Core Ground PrePreg Solder Mask Total 1.0 1.0 1.0 1.0 2.0 2.0 1.0 1.0 1.0 0.5+1.0 1.0 Copper (oz) Thickness (mil) 0.5 1.9 2.7 1.3 4.0 1.3 7.7 1.3 4.0 1.3 12.0 2.6 4.0 2.6 12.0 1.3 4.0 1.3 7.7 1.3 4.0 1.3 2.7 0.5 85.2 3.5 3.8 Tolerance: +/-8mil 3.6 4.0 3.6 4.3 3.6 4.3 3.6 4.0 3.5 3.6 Dielectric (er) 3.8

12

June 23, 2011

Open Compute Project Intel Motherboard Hardware v2.0

Figure 7 Efficent Performance PCB Stackup

Trace Width (mil) 4.0 6.5 5.0 3.9 3.8 5.0 4.1 4.0 4.5 5.0 8.0 3.9 4.4 5.0 4.1

Air Gap Spacing (mil)

Impedance Type Single Single Single

Layer 1,12 1,12 1,12 1,12 1,12 1,12 1,12 3,5,8,10 3, 5,8,10 3, 5,8,10 3, 5,8,10 3, 5,8,10 3, 5,8,10 3, 5,8,10 3, 5,8,10

Impedance Target (ohm) 50 40 43 83 95 85 100 53 50 48 38 85 95 85 100

Tolerance (+/%) 15.0 15.0 15.0 17.5 17.5 17.5 10 10.0 10.0 10.0 10.0 12.0 12.0 12.0 10.0

4.1 8.2 7.0 13

Differential Differential Differential Differential Single Single Single Single

4.1 9.1 6.5 13

Differential Differential Differential Differential

Figure 8 Efficient Performance PCB Impedance Control

BIOS
The ODM is responsible for supplying and customizing a BIOS for the motherboard. The specific BIOS requirements are outlined in this section.

6.1

BIOS Interface and Size

The BIOS chip uses the platform controller hub's SPI interface. The ODM is responsible for selecting a specific BIOS chip that meets the required functionality. 16MB of space should be allocated for both the BIOS and the PCH management engine.

6.2

BIOS Socket
A socket on the motherboard must be used to mount the BIOS chip to ensure that the BIOS chip can be replaced manually. The BIOS socket is easily accessible; other components on the motherboard do not interfere with the insertion or removal of the BIOS chip. A DIP-type BIOS chip and compatible socket are used for easy insertion and removal.

6.3

BIOS Source Code

The BIOS source code comes from AMI EFI. The ODM is responsible for maintaining the BIOS source code to make sure it has the latest version from AMI and Intel.

6.4

BIOS Configuration and Features

The BIOS is tuned to minimize system power consumption. It has the following features: Unused devices disabled, including PCIe lanes, PCI lanes, USB ports, SATA/SAS ports

http://opencompute.org

13

Tuning CPU/chipset settings to reach minimized power consumption and best performance SPECpower is used as guidance for ODM to validate BIOS tuning results

6.5

BIOS Setup Menu

The ODM must provide a BIOS specification, which includes the complete BIOS, setup menu, and default settings. The setup menu allows its options to be configured before the operating system loads. The configuration options available through the boot menu include the following: Settings to adjust memory speed, QPI speed, Speed-step/Turbo mode, and CPU Cx power state. Setting for power feature after AC failure, default is set to power on. Settings for console redirection: o PCH Virtual COM port: With baud rate 115200, no flow control, and terminal type VT100. o SIO COM1: With baud rate 57600, no flow control and terminal type VT100. o Auto: The PCH virtual COM port is enabled by default. The BIOS switches to SIO COM1 automatically, depending on hardware strapping. Default option is "Auto". Setting for fan speed control (for SIO FSC enabled board only). Setting for altitude of server deployment location. Hardware health monitoring display. Setting for watchdog timer. Default is enabled and timeout value is 15 minutes. Event log viewing and clearing. Setting for ECC error threshold, available settings are 1, 4, 10, and 1000. If a CMOS checksum error occurs (for example, caused by BIOS update), the BIOS loads the system default automatically after displaying a message in the console for 5 seconds and rebooting the system to apply the update without waiting for user input. Setting to disable all "wait for keyboard input to continue" types of features.

6.6

PXE Boot
The BIOS supports Intel PXE boot. When PXE booting, the system first attempts to boot from the first Ethernet interface (eth0). If a PXE boot on the first Ethernet interface fails, the BIOS attempts to PXE boot from the second Ethernet interface (eth1).

6.7

Other Boot Options

The BIOS also supports booting from SATA/SAS and USB interfaces. The BIOS provides the capability to select boot options.

6.8

Remote BIOS Update

The BIOS can be updated remotely under these scenarios: Scenario 1: Sample/Audit BIOS settings o Return current BIOS settings, or o Save/export BIOS settings in a human-readable form that can be restored/imported (as in scenario 2) Scenario 2: Update BIOS with pre-configured set of BIOS settings o Update/change multiple BIOS settings

14

June 23, 2011

Open Compute Project Intel Motherboard Hardware v2.0

o Reboot
Scenario 3: BIOS/firmware update with a new revision o Load new BIOS/firmware on machine and update, retaining current BIOS settings o Reboot

Additionally, the update tools have the following capabilities: Update from the operating system over the LAN the OS standard is CentOS v5.2 Can complete BIOS update or setup change with a single reboot (no PXE boot, no multiple reboots) No user interaction (like prompts) BIOS updates and option changes do not take longer than five minutes to complete Can be scripted and propagated to multiple machines

6.9

Event Log
An event log is available through SMBIOS. Per SMBIOS specification Rev 2.6, the BIOS implements SMBIOS type 15 for an event log; the assigned area is large enough to hold more than 500 event records (assuming the maximum event record length is 24 bytes, then the size will be larger than 12KB), and follow the SMBIOS event log organization format for the event log. The ODM must provide a system access interface and application software to retrieve and clear the event log from the BIOS, including, at minimum, a Linux application for the CentOS operating system and driver as needed. The event log must be retrieved and stored as a readable text file that is easy to handle by a scripting language under Linux. Each event record includes enhanced information identifying the error source device's vendor ID and device ID.

6.9.1

Logged Errors CPU/Memory errors: Both correctable ECC and uncorrectable ECC errors should be logged into the event log. Error categories include DRAM, Link, and L3 cache. QPI errors: Any errors that have a status register should be logged into the event log. Fatal or non-fatal classification follows the chipset vendor's recommendation. PCIe errors: Any errors that have a status register should be logged into the event log, including root complex, endpoint device, and any switch upstream/downstream ports if available. Link disable on errors should also be logged. Fatal, non-fatal, or correctable classification follows the chipset vendor's recommendation. POST errors: All POST errors detected by the BIOS during POST are logged into the event log. Power errors: Two power errors are logged: o 12.5V DC input power failure that causes all power rails on motherboard to lose power, including standby power. o Unexpected system shutdown during system S0/S1 while 12.5V DC input is still valid. Error Threshold Settings An error threshold setting must be enabled for both correctable and uncorrectable errors. Once the programmed threshold is reached, an event should be triggered and logged.

6.9.2

http://opencompute.org

15

6.9.3

Memory Correctable ECC: The threshold value is 1000. When the threshold is reached, the BIOS should log the event including DIMM location information and output DIMM location code through the debug card. QPI errors: Follow the chipset vendor's suggestion. PCIe errors: Follow the chipset vendor's suggestion.

BIOS Error Codes MRC fatal error codes should be enabled for POST code output. The major and minor codes alternately display.

Hardware Monitoring
The motherboard does not employ a traditional out of band monitoring solution. The ODM needs to provide a system access interface and application to retrieve hardware monitoring sensor readings. Lm_sensors is the preferred tool for hardware monitoring under Linux; the ODM ensures Lm_sensors works. The sensors to be read include voltage, temperature, and fan speed. The NCT6681 serves as both the super IO (SIO) and hardware monitor.

7.1

Thermal Sensors
The motherboard has five thermal sensors: Two to monitor temperatures for CPU0 and CPU1, retrieved through the CPU's temperature sensor interface (PECI) PCH temperature, retrieved through the Intel controller hubs internal DTS, through PCH SMLink1 Inlet temperature, retrieved through the thermistor, and located in the front of the motherboard Outlet temperature, retrieved through the thermistor, and located in the rear of the motherboard The sensors should make sure that no CPU throttling is triggered due to thermal issues, under the following environmental conditions: Inlet temperature lower than 30C (including 30C), and 0 inch H2O pressure Inlet temperature higher than 30C but lower than 35C (including 35C), and 0.01 inch H2O pressure The sensors should make sure that the total airflow rate for the chassis is lower than 89CFM, including PSU. In the event that one fan fails, an inlet temperature of 30C with 0 inch H2O pressure environment is used to verify thermal sensors.

7.2

Fan Connection
The motherboard has fan tachometers and PWM connections to two system fans through the midplane. See section 8.2.

16

June 23, 2011

Open Compute Project Intel Motherboard Hardware v2.0

7.3

Fan Control Algorithm

The motherboard supports auto fan speed control for the system fans connected to it. The ODM must provide an optimized fan control algorithm based on the thermal solution of the system including fan, heat sink, and air duct. Fan speed control should set system fans running at lowest speed and provide enough damping to avoid speed vibration.

Midplane
The midplane is a PCB that functions as a bridge between the system fans, power supply (PSU), and both motherboards. Its form factor is 2x13 inches.

8.1

PSU Connector
The midplane has one FCI 51939-582 male right angle header, which is mated directly with the PSU for 12.5VDC input. Figure 9 shows the pin definition and direction based on the PSU.
Pin # P1, P2 P3, P4 A1 A2 B1 B2 C1 C2 D1 D2 Signal P12V GND AUX_RTN_GND BACKUP_N SHARE_SEL_1 SHARE_SEL_2 GREEN_LED_N YELLOW_LED_N RED_LED_N P5V_AUX Output Input Input Output Output Output Power Direction Power Power Description 12.5VDC Ground Signal return PSU backup mode indication PSU mode selection PSU mode selection Low active Low active Low active 5V for LED, 50mA limited Usage 12.5VDC Ground NC NC NC NC Connect to bi-color LED Connect to bi-color LED Connect to LED LED power

Figure 9 Midplane to PSU Connector Pin Definition

For the PSU LED, the midplane provides a 4-pin vertically shrouded 2.54mm pitch header with latch. This allows an LED cable to extend the PSU LED to the chassis front. The PSU connector pins C1 and C2 connect to one bi-color (green/yellow) LED with a common anode. Pin D1 is connected to one red LED. Pin D2 is 5V and used for an LED anode. Both are 3mm LEDs. A current limit resistor is required for each LED signal.
Pin 1 2 3 4 Description GREEN_LED_N YELLOW_LED_N RED_LED_N P5V_AUX

Figure 10 PSU LED Header Pin Definition

When the PSU's red LED blinks (at 1Hz, 50% duty-cycle), it indicates a PSU fan failure.

http://opencompute.org

17

8.2

Fan Connectors
The midplane has connectors for the four system fans. The connector signals comply both mechanically and electrically with the specifications defined in the 4-Wire Pulse Width Modulation (PWM) Controlled Fans Specification Revision 1.3 September 2005 published by Intel Corporation. Each fan is driven by a dedicated PWM signal. Figure 11 defines the proper pin out of the connector.
Pin 1 2 3 4 Description GND 12VDC Sense Control

Figure 11 Fan Header Pin-out

A fan tachometer signal from each fan is routed to acquire fan speed. The midplane directly delivers 12.5V power to the fan connector. If one motherboard is not powered on, then its two corresponding fans are turned off to save power.

8.3

Motherboard Connectors
The midplane has two FCI 51770-044 female right-angle power/signal connectors (2P+16S+2P: 4 power blades and 16 signals). The motherboard -- with the mated FCI 51730162 male right angle header -- slides in and mates with one of the FCI headers on the midplane. Figure 12 shows the pin definition of the 2P+16S+2P connector; the direction is based on the midplane.
Pin # P1, P2 Signals P12V Direction Description

Power Power Output Output Output Bi-direction Bi-direction Input Output Output Input Output Input

12.5VDC Ground SMBUS alert signal from hot-swap controller Reserved for extra fan tachometer on FAN1 Reserved for extra fan tachometer on FAN2 Reserved for future SMBUS CLOCK SMBUS DATA Indicates that motherboard starts powered on Indicates that PSU 12.5VDC output is ready System fan #1 tachometer System fan #1 PWM System fan #2 tachometer System fan #2 PWM

P3, P4 A1 A2 A3 A4 B1 B2 B3 B4 C1 C2 C3 C4

GND SMB_ALT_N TACH1A TACH2A RSVD SCLK SDATA MB_ON PSU_PG FAN1_TACH FAN1_PWM FAN2_TACH FAN2_PWM

18

June 23, 2011

Open Compute Project Intel Motherboard Hardware v2.0

D1 (short pin) MATED_N D2 D3 D4 MATED_GND_RTN MB_ID FAN_FAIL_N

Input

Low active, indicates motherboard is fully mated Connected to GND in midplane Motherboard ID = 0 (left), 1 (right) PSU fan failure detected

Output Output

Figure 12 Midplane to Motherboard Connector Pin Definition

8.4

Motherboard Power-up Delay

While running on AC power, in order to avoid both motherboards powering up at the same time and drawing larger than normal current, the midplane introduces a delay between the 12.5V power delivered to each of the two motherboards. The delay time can be set between 1 second and 1 minute, with 30 seconds as the default configuration. The power-up delay behaves as follows: When both MB0 and MB1 are installed and AC power is applied, MB0 powers on first, then after 30 seconds (the timer delay), MB1 powers on. When both MB0 and MB1 are operating, and you remove and re-insert a motherboard, there is no delay for it to power on again. When only MB0 is installed and AC power is applied, there is no delay when it powers on. When only MB1 is installed and AC power is applied, there is no delay when it powers on. With one motherboard is operating, and another motherboard is inserted, there is no delay when it powers on. If no motherboards are installed and AC power is applied, then both MB0 and MB1 are inserted, there is a 30 second timer delay between MB0 and MB1 powering on.

8.5

Hot Swap Controller

In order to have better control of the 12.5VDC power input to each motherboard, the ODM should include two hot swap controllers (one for each motherboard) on the midplane. The hot swap controller provides: Inrush current control when the motherboard is inserted and the server is powered on. Current limiting protection for short circuit. PMBUS interface to enable the PCH to report server input power.

9
9.1
9.1.1

Power System
Input Voltage
Input Voltage Level The nominal input voltage delivered by the power supply is 12.5VDC. The motherboard can accept and operate normally with an input voltage tolerance range between 10.8V and 13.2V. The motherboard's undervoltage protection level is 10V or less.

http://opencompute.org

19

9.1.2

Capacitive Load To ensure compatibility with the system power supply, the motherboard cannot have a capacitive load greater than 4000F. The capacitive load of the motherboard cannot exceed the maximum value of 4000F under any operating condition listed in section 12, which defines environmental conditions.

9.1.3

Input Connector The power input connector is an FCI 51733-009LF right-angle press-fit header.

9.2
9.2.1

CPU Voltage Regulation Module (VRM)

CPU M axim um Power The motherboard can handle a processor with a maximum thermal design power (TDP) of 95W.

9.2.2

CPU VRM Optim izations The CPU VRM is optimized to reduce cost and increase the efficiency of the power conversion system. The ODM should use only the minimum number of required phases to support the maximum CPU power defined in 9.2.1. A PSI (power state indicator) allows the shedding of unused phases, letting the VRM operate at its peak efficiency.

9.2.3

CPU VRM Efficiency The minimum efficiency for the CPU VRM is 91% over the 30% to 90% load range and 93% over the 50% to 70% load range, measured from the 12.5V input to the VRM output.

9.3

Hard Drive Power

The motherboard supplies power to the system's 14 hard drives. The drives require 12VDC and 5VDC power sources. Power is delivered through a traditional 4-pin floppy disk power connector described in Figure 13.
Pin 1

Motherboard component side

Pin 1 2 3 4

Description +5VDC GND GND +12VDC

Figure 13 Drive Power Connector

20

June 23, 2011

Open Compute Project Intel Motherboard Hardware v2.0

For SATA ports inside the miniSAS connector, power will be delivered through a 4-pin (2x2) ATX power connector, which fans out into 4 standard SATA power cables. Pin definition is described in Figure 14.
Pin 1 2 3 4 Description GND GND +5VDC +12VDC

Figure 14 4 Pin ATX Power Connector

9.3.1

Power Requirem ents In order for the motherboard to supply 12.5VDC power to the hard drives, the PCB traces must support 14A of continuous power (1A per drive) on the 12.5VDC power rail. In order for the system's 5VDC to supply power to the hard drives, its regulator must support an additional 10.5A (0.75A per drive) of continuous power on the 5VDC power rail. The motherboard must support the inrush current required to start each drive from idle.

9.3.2

Output Protection The 5V disk output power regulator protects against shorts and overload conditions.

9.3.3

Spin-up Delay When a hard drive spins up after the system powers on, it draws excessive current on both the 12V and 5V rails. The peak current may reach the 1.5A-2A range in 12V. Each of the 14 hard drives must spin up in sequence. The BIOS implements a 5 second delay between each hard drive spinning up. To enable the hard drive's spin-up delay function, set pin 11 of the SATA hard drive's power cable to NC (No Connection).

9.4

System VRM Efficiency

The ODM supplies high efficiency VRMs for all other voltage regulators over 20W not defined in this specification. All voltage regulation modules over 20W have 91% efficiency over the 30% to 90% load range.

9.5

Power On
The motherboard powers on upon application of power to the input connector. The use of a power button is not required. The motherboard always resumes operation upon restoration of power in a power failure event.

10

I/O System
This section describes the motherboard's I/O features.

10.1 PCIe x16 Slot/Riser Card

Both the entry and efficient performance motherboards have one PCIe x16 slot, which holds an x16 PCIe signal from the CPU. The slot location and detailed dimensions are described in the mechanical DXF file. The motherboard also has a PCIe riser card so two

http://opencompute.org

21

full-height PCIe cards can be inserted horizontally and locked in position. Its form factor is 2x4.66 inches.

Figure 15 PCIe Riser Card

22

June 23, 2011

Open Compute Project Intel Motherboard Hardware v2.0

The reserved pins on the PCI-E x16 slot on the motherboard are described in Figure 16.
Pin A7 A8 B12 A32 A33 A50 B82 B17 B31 B48 B81 Pin Defined LAN_SMB_CLK LAN_SMB_DAT LAN_SMB_ALERT_N CLK_100M_P CLK_100M_N SLOT0_CONFIG SLOT1_CONFIG SLOT1_CPRSNT1_N SLOT1_CPRSNT2_N SLOT0_CPRSNT1_N SLOT0_CPRSNT2_N Lower slot on riser card has 1x8 (high), 2 x4 (low) Higher slot on riser card has 1x8 (high), 2 x4 (low) CPRSNT1# for SLOT1 on PCIe riser card CPRSNT2# for SLOT1 on PCIe riser card CPRSNT1# for SLOT0 on PCIe riser card CPRSNT2# for SLOT0 on PCIe riser card Description SMBUS CLOCK from SMLINK0 of PCH SMBUS DATA from SMLINK0 of PCH SMBUS Alert signal to SMLINK0 of PCH Extra 100MHz clock for second PCIe slot on riser card

Figure 16 PCIe x16 Slot Reserved Pin Usage on Motherboard

The reserved pins on the PCIe x16 slot 0 (low) on the riser card are described in Figure 17.
Pins A32 A33 A50 B48 B81 Pin Defined LAN_SMB_CLK LAN_SMB_DAT LAN_SMB_ALERT_N SLOT0_CPRSNT1_N SLOT0_CPRSNT2_N Description SMBUS clock from SMLINK0 of PCH SMBUS data from SMLINK0 of PCH SMBUS alert signal to SMLINK0 of PCH CPRSNT1# for SLOT0 on PCIe riser card CPRSNT2# for SLOT0 on PCIe riser card

Figure 17 PCIe x16 Slot 0 (Low) Reserved Pin Usage on Riser Card

The reserved pins on the PCIe x16 slot 1 (high) on riser card are described in Figure 18.
Pin A32 A33 A50 B17 B31 Pin Defined LAN_SMB_CLK LAN_SMB_DAT LAN_SMB_ALERT_N SLOT1_CPRSNT1_N SLOT1_CPRSNT2_N Description SMBUS clock from SMLINK0 of PCH SMBUS data from SMLINK0 of PCH SMBUS alert signal to SMLINK0 of PCH CPRSNT1# for SLOT1 on PCIe riser card CPRSNT2# for SLOT1 on PCIe riser card

Figure 18 PCIe x16 Slot 1 (High) Reserved Pin Usage on Riser Card

To support OOB LAN access on the platform controller hub's management engine, a customized PCIe card is needed to use these redefined reserved pins.

10.2 PCIe External Connector

The motherboard has two PCIe x4 external connectors on the efficient performance motherboard and one PCIe x4 external connector on the entry board. These connectors can be used to build a PCIe connection between two systems.

http://opencompute.org

23

The PCIe x4 connector can be hot inserted and removed. A PCIe re-driver is used for PCIe external links and supports a miniSAS cable up to 2 meters long. The connector is a miniSAS-4i right-angle connector. External PCI Express target device is TBD. Figure 19 shows the external PCIe pin assignments. The design follows the PCI Express External Cabling 1.0 Specification (http://www.pcisig.com/members/downloads/specifications/pciexpress/PCI_Express_E xternal_Cabling_Rev1.0_updated.pdf).
Pin Numbers A2/A3, A5/A6, A13/A14, A16/A17 A1, A4, A7, A12, A15, A18 A8 A9 A10 A11 B2/B3, B5/B6, B13/B14, B16/B17 B1, B4, B7, B12, B15, B18 B8 B9 B10 B11 Signals PER{0..3}{P/N} GND CPRSNT# CPWRON CWAKE# CPERST# PET{0..3}{P/N} GND SCLK/TX SDATA/RX 3.3V/SYS_RST# SB_RTN Description Differential PCI Express receiver lanes Ground reference for Differential PCI Express lanes Cable installed/downstream subsystem powered up Upstream subsystem's power valid notification Power management signal for wakeup events (optional) Cable PERST# Differential PCI Express transmitter lanes Ground reference for Differential PCI Express lanes SMBUS (PCH SMLINK0) CLOCK (optional UART TX from SIO) SMBUS (PCH SMLINK0) DATA (optional UART RX from SIO) 3.3V standby with 0 ohm in series (Reset signal to trigger system reset) Signal return for single-ended sideband signals

Figure 19 External PCIe Pin Assignments

10.3 PCIe Mezzanine Card

The motherboard has one PCIe x8 mezzanine card connector that holds the x8 PCIe signal from cpu0 on both the entry and the efficient performance motherboards. The mezzanine card has two PCIe x4 external connectors (miniSAS) and one mSATA miniPCIe connector.
Pin Name P12V P12V P12V GND GND Pin 61 62 63 64 65 Pin 1 2 3 4 5 Pin Name MEZZ_PRSNT1_N P5V_AUX P5V_AUX P5V_AUX GND Pin Name GND MEZZ_TX_DP_C <1> MEZZ_TX_DN_ C<1> GND GND Pin 91 92 93 94 95 Pin 31 32 33 34 35 Pin Name MEZZ_RX_DN <0> GND GND MEZZ_RX_DP <1> MEZZ_RX_DN

24

June 23, 2011

Open Compute Project Intel Motherboard Hardware v2.0

<1> P3V3_AUX GND GND P3V3 P3V3 P3V3 P3V3 GND LAN_3V3STB_A LERT_N SMB_LAN_3V3 STB_CLK SMB_LAN_3V3 STB_DAT PCIE_WAKE_N DA_DSS GND SATA_TX+ SATA_TXGND GND CLK_100M_ME ZZ2_DP CLK_100M_ME ZZ2_DN GND GND MEZZ_TX_DP_ C<0> MEZZ_TX_DN_ 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 GND P3V3_AUX GND GND P3V3 P3V3 P3V3 P3V3 MEZZ_CPRSNT1_ N MEZZ_CPRSNT2_ N SSD_PRSNT_N RST_PLT_MEZZ_N MEZZ_SMCLK MEZZ_SMDATA GND GND SATA_RX+ SATA_RXGND GND CLK_100M_MEZZ 1_DP CLK_100M_MEZZ 1_DN GND GND MEZZ_TX_DP_C <2> MEZZ_TX_DN_ C<2> GND GND MEZZ_TX_DP_C <3> MEZZ_TX_DN_ C<3> GND GND MEZZ_TX_DP_C <4> MEZZ_TX_DN_ C<4> GND GND MEZZ_TX_DP_C <5> MEZZ_TX_DN_ C<5> GND GND MEZZ_TX_DP_C <6> MEZZ_TX_DN_ C<6> GND GND MEZZ_TX_DP_C <7> MEZZ_TX_DN_ C<7> GND GND 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 GND GND MEZZ_RX_DP <2> MEZZ_RX_DN <2> GND GND MEZZ_RX_DP <3> MEZZ_RX_DN <3> GND GND MEZZ_RX_DP <4> MEZZ_RX_DN <4> GND GND MEZZ_RX_DP <5> MEZZ_RX_DN <5> GND GND MEZZ_RX_DP <6> MEZZ_RX_DN <6> GND GND MEZZ_RX_DP <7> MEZZ_RX_DN

http://opencompute.org

25

C<0> GND 90 30 MEZZ_RX_DP<0> MEZZ_PRSNT2_ N 120 60

<7> GND

Figure 20 PCIe Mezzanine Card Connector Pin Definition

10.4 DIMM Connector

The motherboard uses a 30u" gold contact for the DDR3 DIMM connector.

10.5 Network
The motherboard has an Intel 82574L Ethernet interface to the front RJ45 connector. It has a PCIe x1 lane routed to the PCH. The motherboard has an Intel I350 dual port network chip. It has a single Ethernet interface to the front RJ45 connector. It has PCIe x2 lanes routed to the PCH on entry board, while it has PCIe x4 lanes routed to the PCH on efficient performance board. The BIOS supports PXE boot on all RJ45 ports on the motherboard. Each RJ45 connector has two built-in LEDs. While facing the RJ45 connector, the left LED is green single color; solid on means the link is active and blinking means activity. The right LED is green/yellow dual color; green means 100M link speed while yellow means 1000M link speed. 10.5.1 Reboot on W OL in S0 State Reboot on WOL (ROW) is a feature that repurposes the traditional Wake on LAN (WOL) signal to reboot the motherboard. While the system is in S0 state (running), when a WOL packet is received by the NIC, the wakeup signal generated by the NIC causes a hardware reboot of the motherboard. This is accomplished by tying the WOL interrupt pin of the NIC to the system's master reboot signal. ROW does not require the power supply to cycle its output. There is an optional ROW connection for the WAKE# signal from PCIe slot and external PCIe connector, which gives optional ROW support for add-in cards and external PCIe devices. ROW is enabled by the NIC EEPROM, so the appropriate NIC EEPROM for the 82574 and I350 interface must be used. The motherboard also supports ROW on both the PCIe LAN card and the mezzanine LAN card, which includes hardware circuit support and NIC EEPROM enabling.

10.6 USB Interfaces

The motherboard has two external USB ports located in the front of the motherboard. The BIOS supports the following USB devices: Keyboard and mouse USB flash drive (bootable) USB hard drive (bootable) USB optical drive (bootable)

26

June 23, 2011

Open Compute Project Intel Motherboard Hardware v2.0

10.7 SATA
The motherboard has a next generation Intel platform controller hub on board and supports the SATA ports and the miniSAS connectors. The HDDs attached to all the SATA connectors follow the spin-up delay described in section 9.3.3.

10.8 Debug Header

The motherboard includes a debug header on the front of the motherboard to display POST codes (see 10.8.1). The debug header supports hot plugging. The debug card has two 7-segment LED displays, one RS-232 serial connector, and one reset switch. The RS-232 serial port provides console redirection. The two 7-segment LED displays show BIOS POST code and DIMM error information. The reset switch triggers a system reset when pressed. The connector for the debug header is a 14-pin, shrouded, vertical, 2mm pitch connector. Figure 21 is an illustration of the headers. The debug card has a key to match with the notch to avoid pin shift when plugging it in.

Figure 21 Debug Header

Pin (CKT) 1 2 3 4 5 6 7 8 9 10 11 12 13 14

Function Low HEX character [0] least significant bit Low HEX character [1] Low HEX character [2] Low HEX character [3] most significant bit High HEX character [0] least significant bit High HEX character [1] High HEX character [2] High HEX character [3] most significant bit Serial transmit (motherboard transmit) Serial receive (motherboard receive) System reset Serial console select (1=SOL; 0=local) GND VCC (+5VDC)

http://opencompute.org

27

Figure 22 Debug Header Pin Definitions

10.8.1 Post Codes POST codes are sent to the debug header in hexadecimal format via two hex codes. The hex codes can be driven by either the legacy parallel port (port 80) on the SIO, or 8 GPIO pins. During the boot sequence, the BIOS initializes and tests each DIMM. If a module fails initialization or does not pass the BIOS test, one of the following POST codes will flash on the debug card to indicate which DIMM has failed. The first hex character indicates which CPU interfaces the DIMM module; the second hex character indicates the number of the DIMM module. The BIOS flashes the corresponding hex code indefinitely to allow time for a technician to service the system. The DIMM number count starts from the DIMM furthest from the CPU. 10.8.2 Serial Console The output stage of the system's serial console is contained on the debug card. The TX and RX signals from the SIO are sent to the debug header at the chip's logic levels (+3.3V). The debug card contains the RS-232 level shifter and the RS-232 D-9 connector. By default, the host does console redirection through serial over LAN (SOL). When the debug card is connected, debug card pin 12 is used to select console redirection between SOL and the local serial port on the card, as described in Figure 22.

10.9 Switches and LEDs

The motherboard includes a power switch, reset switch, power LED, HDD activity LED, and beep error LED. 10.9.1 Switches The front edge of the PCB has right angle pushbutton switches. One switch is used as the system's power button the second switch is used at the system's reset button. Note: If the ODM chooses to use smaller tactile switches, the push button actuator must be a minimum 2.5mm diameter and protrude at least 1.5mm from the switch's enclosure. If the power switch is depressed for less than four seconds, a power management event is issued, indicating that the power switch has been triggered. If the power switch is depressed for more than four seconds, the motherboard performs a hard power off. If the reset switch is depressed for any length of time, the motherboard performs a hard reset and begins executing the BIOS initialization code. Each switch is identified by a label on the motherboard's silkscreen. The power button is labeled PWR and the reset button is labeled RST. 10.9.2 LEDs The motherboard has 3 LEDs on the front edge. Figure 23 identifies each LED's color, function, and silkscreen label. The label describes the functionality of each LED.
LED Color Function Silkscreen Label

28

June 23, 2011

Open Compute Project Intel Motherboard Hardware v2.0

Blue

Power LED. This LED has the same functionality of a traditional PC power LED. It illuminates only if the motherboard is in the powered on state. Hard drive activity. This LED illuminates when there is activity on the motherboard's SATA hard drive interfaces. This LED replaces the functionality of the PC speaker. The motherboard causes the LED to illuminate for the same duration and sequence as the PC speaker would normally beep. The LED allows for easier diagnosis in a noisy data center environment.

PWR

Green Yellow

HDD BEEP

Figure 23 LED Functionality

http://opencompute.org

29

The beep error LED patterns are described in Figure 24.

Error Description Memory refresh timer error Base memory read/write test error Keyboard controller BAT test error General exception error Display memory error LED Patterns On (2s) On (2s) Off (0.25s) Off (0.25s) On (2s) On (2s) Off (0.25s) Off (0.25s) On (2s) On (0.25s) Off (3s) Off (3s) (repeat) (repeat)

On (0.25s) On (2s) On (0.25s)

Off (0.25s)

On (0.25s) On (0.25s) On (0.25s)

Off (0.25s)

On (2s)

Off (0.25s) Off (0.25s)

Off (0.25s) Off (0.25s)

On (0.25s) On (0.25s)

Off (3s)

(repeat)

Figure 24 Beep Error LED Patterns

11

Mechanical
Figure 25 shows the basic view of the Open Compute Project server chassis. Refer to mechanical step file provided for detailed information.

Figure 25 Open Compute Project Server Chassis for Intel Motherboards

11.1 Fixed Locations

Refer to the mechanical DXF file for fixed locations of the mounting hole, PCIe x16 slot, and power connector.

30

June 23, 2011

Open Compute Project Intel Motherboard Hardware v2.0

11.2 PCB Thickness

To ensure proper alignment of the FCI power connector and mounting within the mechanical enclosure, the boards should follow the PCB stackups described in sections 4.5 and 5.5 respectively and have 85mil (2.16mm) thickness. The mid-plane PCB thickness is also 85mil (2.16mm). The mezzanine card PCB thickness is 62mil (1.6mm).

11.3 Heat Sinks

The motherboard supports customized CPU heat sinks that are mounted according to the Intel specifications. The mounting device employs a backplate and receptacles for screw-down type heat sinks. The ODM must provide all keep out zones defined by Intel to ensure the heat sinks mount correctly on the board.

11.4 Silkscreen
The silkscreen is white in color and includes labels for these components: cpu0/cpu1 eth0/eth1 DIMM slot numbering, as described in 10.8.1 LEDs, as defined in 10.9.2 Switches, as defined in 10.9.1

11.5 DIMM Connector Color

Colored DIMM connectors indicate the first DIMM of each memory channel, whereas the remaining DIMM connectors on the same memory channel are a different color. The first DIMM on each channel is defined as the DIMM placed physically furthest from its associated CPU. This DIMM connector must be populated first when memory is only partially populated.

12

Environmental Requirements
The motherboard meets the following environmental requirements: Gaseous Contamination: Severity Level G1 per ANSI/ISA 71.04-1985 Ambient operating temperature range: -5C to +45C Operating and storage relative humidity: 10% to 90% (non-condensing) Storage temperature range: -40C to +70C Transportation temperature range: -55C to +85C (short-term storage) The full OCP system also meets these requirements. In addition, the full system has an operating altitude with no de-ratings of 1000m (3300 feet).

http://opencompute.org

31

12.1 Vibration and Shock

The motherboard meets shock and vibration requirements according to the following IEC specifications: IEC78-2-(*) and IEC721-3-(*) Standard & Levels. The testing requirements are listed in Figure 26.
Operating Vibration 0.5g acceleration, 1.5mm amplitude, 5 to 500 Hz, 10 sweeps at 1 octave / minute for each of the three axes (one sweep is 5 to 500 to 5 Hz) 6g, half-sine 11mS, 5 shocks for each of the three axes Non-Operating 1g acceleration, 3mm amplitude, 5 to 500 Hz, 10 sweeps at 1 octave / minute for each of the three axes (one sweep is 5 to 500 to 5 Hz) 12g, half-sine 11mS, 10 shocks for each of the three axes

Shock

Figure 26 Vibration and Shock Requirements

13

Prescribed Materials
The following components are not used in the design of the motherboard: Components disallowed by the European Union's Restriction of Hazardous Substances Directive (RoHS 6) Trimmers and/or potentiometers Dip switches

13.1 Disallowed Components

13.2 Capacitors and Inductors

The following limitations apply to the use of capacitors: Only aluminum organic polymer capacitors made by high quality manufacturers are used; they must be rated 105C All capacitors have a predicted life of at least 50,000 hours at 45C inlet air temperature, under worst conditions Tantalum capacitors are forbidden SMT ceramic capacitors with case size > 1206 are forbidden (size 1206 are still allowed when installed far from the PCB edge and with a correct orientation that minimizes risks of cracks) Ceramic material for SMT capacitors must be X7R or better material (COG or NP0 type should be used in critical portions of the design) Only SMT inductors may be used. The use of through hole inductors is disallowed.

13.3 Component De-rating

For inductors, capacitors, and FETs, de-rating analysis should be based on at least 20% de-rating.

32

June 23, 2011