Enterprise SSD Form Factor: Version 1.0a
Enterprise SSD Form Factor: Version 1.0a
Copyright © 2012 SSD Form Factor Work Group, All rights reserved.
2
Enterprise SSD Form Factor 1.0a
Table of Contents
1 OVERVIEW ................................................................................................................... 6
1.1 Overview ......................................................................................................................................... 6
1.2 Goals .............................................................................................................................................. 6
1.3 Technical Summary ........................................................................................................................ 6
1.3.1 Flexible Drive Backplane option .............................................................................................................7
1.4 Scope .............................................................................................................................................. 8
1.5 Outside of Scope ............................................................................................................................ 9
2 SIGNAL LIST .............................................................................................................. 10
2.1 Signal Lists ................................................................................................................................... 10
2.2 Connector pin out ......................................................................................................................... 12
2.3 PRSNT# - Presence Detect and IfDet# - Interface Detect Signal Definition ................................ 14
2.4 DualPortEn# - PCIe Dual Port Enable .......................................................................................... 14
2.5 Reserved(WAKE#/OBFF#), Reserved(ClkReq#/DevSLP) ........................................................... 15
2.6 ePERst[1:0]# - PCIe Reset ........................................................................................................... 15
2.7 RefClk[1:0][+/-] – Enterprise PCIe Clocks .................................................................................... 15
2.8 Activity - Activity indication signal ................................................................................................ 15
3 CONNECTOR MECHANICAL............................................................................................ 16
3.1 Connector Mechanical Drawing notes .......................................................................................... 17
3.2 Connector Plug Mechanical Drawings.......................................................................................... 17
3.3 Connector Plug Mechanical Drawings - Details ........................................................................... 19
3.4 Connector Receptacle Mechanical Drawings ............................................................................... 21
3.5 Cable Retention Mechanical Drawings ......................................................................................... 22
4 HOT PLUG AND REMOVAL ............................................................................................. 27
4.1 Limit current inrush ....................................................................................................................... 28
4.2 Detecting Enterprise PCIe SSD Insertion/Removal ..................................................................... 29
4.3 PCIe Hardware and Reset System Requirements ....................................................................... 29
4.4 Software Implications of Hot Swap (Informative) ......................................................................... 31
5 DUAL PORT OPERATION ............................................................................................... 33
5.1 Dual Port Mode (implications of DualPortEn#) ............................................................................. 34
6 SM-BUS OPERATION ................................................................................................... 35
6.1 Simple Accesses to Vital Product Data (VPD) ............................................................................. 35
6.2 Extended SM-Bus management................................................................................................... 35
7 ELECTRICAL (POINTER TO OTHER SPECIFICATIONS) .......................................................... 36
7.1 Electrical Channel ......................................................................................................................... 36
7.2 Presence and Interface Detect ..................................................................................................... 37
7.3 ePERst[1:0] ................................................................................................................................... 37
7.4 Power – 12V ................................................................................................................................. 37
7.5 Power 3.3Vaux ............................................................................................................................. 38
8 REFERENCES AND BIBLIOGRAPHY .................................................................................. 39
8.1 References ................................................................................................................................... 39
8.2 References Under Development .................................................................................................. 39
8.3 Formal Bibliography ...................................................................................................................... 39
9 SM-BUS - INFORMATIVE APPENDIX ................................................................................ 41
3
Enterprise SSD Form Factor 1.0a
4
Enterprise SSD Form Factor 1.0a
Table of Figures
Figure 1: Example configurations ...................................................................................................................................7
Figure 2: Flexible Backplane Implementation ................................................................................................................8
Figure 3: Signal List Summary (pin counts) ................................................................................................................. 10
Figure 4: Signal List Table (defining specification) ....................................................................................................... 11
Figure 5: Pin Out Drawing (Receptacle pin naming). ................................................................................................... 12
Figure 6: Drive type - Pin decoding .............................................................................................................................. 14
Figure 7: Connector Mechanical Overview .................................................................................................................. 16
Figure 8: Connector Examples ..................................................................................................................................... 16
Figure 9: Connector Plug Mechanical Drawing – Top View ......................................................................................... 17
Figure 10: Connector Plug Mechanical Drawing – Drive Insertion View ...................................................................... 17
Figure 11: Connector Plug Mechanical Drawing – Bottom View (outside of enclosure) ............................................... 18
Figure 12: Connector Plug Mechanical Drawing – End View (showing latch slot) ....................................................... 18
Figure 13: Connector Plug Mechanical Drawing – Detail A (key) ................................................................................. 19
Figure 14: Connector Plug Mechanical Drawing – Detail C (key pins) ......................................................................... 19
Figure 15: Connector Plug Mechanical Drawing – Detail B (Tongue) .......................................................................... 20
Figure 16: Connector Plug Mechanical Drawing – Section D-D (Tongue cross section).............................................. 20
Figure 17: Connector Plug Mechanical Drawing – Drive Keep-out Area ...................................................................... 21
Figure 18: Connector Receptacle Mechanical Drawing – Drive insertion view ............................................................ 21
Figure 19: Connector Receptacle Mechanical Drawing – Section E-E (Receptacle) ................................................... 22
Figure 21: Cable Retention Mechanical Drawing – Representative Cable Housing ..................................................... 24
Figure 22: Cable Release Clearance - Representative Cable Housing........................................................................ 25
Figure 23: Representative Cable Housing with flat ribbon cables ................................................................................ 26
Figure 24: Hot Plug Current Inrush Limiting ................................................................................................................. 28
Figure 25: System Reset Timing .................................................................................................................................. 30
Figure 26: Reset Timing Details (PCI SIG CEM, 2007) ................................................................................................ 31
Figure 27: Single Port and Dual Port example ............................................................................................................. 33
Figure 28: Typical SSD Form Factor Channel ............................................................................................................. 36
Figure 29: Power (12V) specifications.......................................................................................................................... 37
Figure 30: Power 3.3VAux specifications ..................................................................................................................... 38
Figure 31: SM Bus Vital Product Data.......................................................................................................................... 41
Figure 32: SM Bus Capability Definition ....................................................................................................................... 42
Figure 33: Dual Port Extended Capability Definition .............................................................................................. 42
Figure 34: RefClk Extended Capability Definition ................................................................................................... 43
Figure 35: SM Bus Temperature Extended Capability Definition ................................................................................. 44
Figure 36: Dual Port and Active/Passive Extended Capability Definition ............................................................ 45
Figure 37: Pin usage across Existing standards .......................................................................................................... 48
Figure 38: Pin usage across Emerging standards........................................................................................................ 49
Figure 39: Backplane for Enterprise SSD (PCIe x4), SATA & SATA Express (Client PCIe) ........................................ 50
Figure 40: Backplane for Enterprise SSD (PCIe x4), SAS & SATA.............................................................................. 51
Figure 41: Backplane for SAS X4, SAS & SATA .......................................................................................................... 51
Figure 42 Cross Standard Connector Keying ............................................................................................................... 53
5
Enterprise SSD Form Factor 1.0a
1 Overview
1.1 Overview
This specification defines the electrical and mechanical requirements for a PCI Express connection to the
existing standard 2.5” and 3.5” disk drive form factors. This is intended for PCIe connections to SSDs
(Solid State Drives) for the enterprise market of servers and storage systems. This provides a new PCIe
form factor that is storage friendly, leveraging both the existing PCIe specification and the existing 2.5”
and 3.5” drive mechanical specifications. This standard allows system designs that can support a flexible
mix of new enterprise PCI Express drives, and existing SAS and SATA drives.
Note: The References and Bibliography are listed at the end, in section 8 on page 39. The
Specification Conventions are listed at the end, in section 11 on page 54. This includes the standard
material such as the definition of “Shall”, “Should”, “May”, etc. and bibliographic material to reference
specifications. Also included are definitions of terms such as “PCIe”, “PCIe Gen 3”, “PCIe x4”, “SFF”, and
“SSD”. The bibliographic reference use the original author name, for example (SFF-8482, 2006) while
the section 8.3 give the current designator, for example EIA-966.
1.2 Goals
The following are the characteristics of the product envisioned using this specification:
1
PCIe connection to Enterprise Solid State Drives (SSDs).
Standardize connector and form factor – but enabling innovation using the PCIe model.
Fits in existing drive mechanical enclosures using a disk backplane. Supports both 2.5” drive
mechanical enclosures and 3.5” drive mechanical enclosures.
Support customer expectation for storage device:
o Externally accessible & Hot Swappable (with surprise removal)
o Support existing OEM’s existing drive infrastructure (device detection and indicators)
o Support existing OEM thermal architectures, and support for future enhancements.
Support for both single port (typical servers), or dual port (typically storage systems).
Meet projected system storage device performance requirements for systems introduced in the
2H 2011 to 2016 timeframe.
Enables flexible system designs that support Enterprise PCIe Express SSDs and SAS or SATA
drives using the same connector family, allow systems to support flexible mix of capacity (SAS or
SATA hard drives) or Enterprise PCIe SSDs.
1.3 Technical Summary
A technical summary of the “SSD Form Factor Working Group” is:
Focus is on connector which is supported in both the existing 2.5” and 3.5” form factors.
2
Supports the 2.5” drive (SFF-8223, 2006) specification unchanged, or supports the 3.5” drive
(SFF-8301, 2010) specification unchanged.
Compatible with existing SAS connector (SFF-8680, 2012). Meaning the connectors allow
interoperation, see Section 10. For example a new backplane receptacle would accept new or
existing drives. The connector is doubled sided extension to existing SAS connector. New pins
are 0.8mm pitch across the complete B-side and on top of SAS key.
Support for PCIe x4 (Gen2 and Gen3) signaling. References PCIe specifications for high speed
electrical specifications – unchanged.
The electrical channel is assumed to be similar to SAS-3 channel – short board trace, longer
cable, trace on storage backplane, SSD.
1
Definition of PCIe and reference specification are included in section 8 on page 40
2
Definition of SAS, SATA, SFF and reference specifications including official designations (EIA, ANSI,
INCITIS numbers) are included in section 8 on page 40
6
Enterprise SSD Form Factor 1.0a
25W connector limit, with 12V only delivery. Expecting Enterprise PCIe SSDs at range of power
levels, and expecting Enterprise PCIe SSDs to support software settable power limit, and
temperature monitoring.
There is optional support for 3.3V aux to allow probing device information before system power
on (Vital Product Data – VPD – over SM bus.)
Pins list is the merging of SATA/SAS signals and the PCIe x4 plus PCIe side band. The actual
high speed lanes for SAS and SATA are separate pins from the Enterprise PCIe pins. This
enables direct connection to SAS or SATA controller and separate pins to high performance
Enterprise PCIe upstream device without an intervening repeater, mux, or controller.
Support Single x4 or Dual x2 port as (2@x2) with dual PCIe side band
Supports Hot swap support details, basics same as SAS
Supports SM bus for out of band discovery
Supports PCIe power management features
Figure 1 shows example configurations.
Figure 1: Example configurations
Disk Bay
Backplane Connector
PCIe upstream 2.5”
conn
Connector
A 2.5”
SSD Form Factor
Device
conn
7
Enterprise SSD Form Factor 1.0a
The concept is that a single system or backplane design would support either Enterprise PCIe SSD or
SAS/SATA drives allowing an optimal balance of performance and/or capacity to be achieved. The
assumption is that SAS expanders allow adding SAS ports at low additional cost. This is shown in Figure
2.
The drawing of Flexible Backplane (Figure 2) is not the only implementation since a Hard Disk Drive
(HDD) optimized using just SAS/SATA and other implementations for just PCIe are also expected. There
are multiple variations of flexible backplane based on the capabilities of the supporting controllers. The
informative appendix, Section 10.2 on page 50, shows multiple other options.
The capability to build a flexible backplane is based on extending the existing SAS connector (SFF-8680,
2012) sharing functions like power and giving it the ability to meet SAS MultiLink (SFF-8630, 2012)
functionality as well as giving it the capability to deliver four ports of high speed PCIe. SFF-8630 does not
accommodate the additional 2 x PCIe ports nor does it provide the PCIe sidebands.
new
PCIe x4 ePCIe
SAS
SAS
New
PCIe
SATA
Controller,
SATA
Root port
Expanders
SAS
SAS
SAS
SAS
SAS
SATA
SAS SATA
Controller
1.4 Scope
This is a form factor specification that focuses on extending the existing connector for PCIe use. The new
connector is a backward compatible extension of the existing SAS connector (SFF-8482, 2006). The
8
Enterprise SSD Form Factor 1.0a
overall mechanical form factor is compliant with the 2.5” form factor (SFF-8223, 2006) or 3.5” form factor
(SFF-8301, 2010).
This specification defines the following:
Pin list and pin out of the new connector.
Mechanical definition of the new connector, including latching, keying, and retention.
Limited electrical specification (largely reference PCIe CEM and SAS Specifications).
Hardware hooks for Hot Swap
Signal definitions for Dual Port
Definitions of a basic product data accessible from SM-Bus
1.5 Outside of Scope
This specification is only normative for connector and related definitions (just listed)
The actual usage is not defined. While we envisioned having an SSD using NVM Express interface
(NVM Express, 2011) or SOP (SCSI over PCIe) (T10, SOP, Draft 2012), these are not required and other
PCIe device interface models would work.
The specification does give possible usage (aka. implementation notes) for Hot Swap, Dual Port, and SM-
bus but the details are system implementation specific. These are just meant to illustrate a possible
usage.
Specifically this specification is informative (not normative) in the following areas:
The drive outline is defined by EIA/ECA-720 (SFF-8223, 2006) for 2.5” drives or by EIA-740
(SFF-8301, 2010) for 3.5” drives. The drive carrier and storage enclosure are implementation
specific.
Mechanical drawings normative document is SFF-8639 (SFF-8639, 2012). Section 3 is
informative, for ease of reference.
SAS signal definitions and specifications – normative are the SAS Specifications (T10, Various).
Note: SAS-x4 signal definitions in this specification are a draft proposal to T10. This includes
SOP (SCSI over PCIe).
PCIe signal definitions and specifications – normative are the PCIe Specifications (PCI SIG,
various).
SATA/SATA Express signal definitions and specification – normative are the SATA Specifications
(SATA-IO, various).
Note: SATA Express signal definitions in this specification are a proposal to the SATA-IO Cabcon
SM Bus signal definitions and specifications – normative are the SM Bus specifications (SMBus,
2000)
PCIe device functionality and register details and the SM-bus registers are defined by the specific
device specifications. Likely referencing the PCIe Specification for standard PCIe device
discovery and setup.
Specific system implementation of presence pins (hot plug)
Specific system implementation of dual port functionality
References to other related connectors - specifically keying and overlaying of pins.
This is envisioned as a PCIe SSD specification and there is no standardization for additional connections
beyond the drive PCIe connector. There are both backplane and cable versions of the connector
receptacle.
9
Enterprise SSD Form Factor 1.0a
2 Signal List
The signal list is a combination of SAS/SATA and PCIe signals. Figure 3: Signal List Summary gives an
overview of the signal groups. Figure 4: Signal List Table gives the complete signal list, usage, and which
document is normative for the signal definition. Most of the signals used in this specification are defined
in other documents. Figure 5 gives the mapping of signal pins to connector pins.
2.1 Signal Lists
Figure 3: Signal List Summary (pin counts)
Usage Signals Contacts
10
Enterprise SSD Form Factor 1.0a
11
Enterprise SSD Form Factor 1.0a
12
Enterprise SSD Form Factor 1.0a
SATA Express pin placement is only a proposal on how to align this specification’s Enterprise
PCIe pin-out with proposals for pins in SATA Express. The SATA Express signals are not yet
standardized, and this document is not the normative document on pin out or pin definition for
SATA Express.
SAS-x4 pin placement is a proposal on how to align this specification’s Enterprise PCIe pin-out
with proposals for pins to support SAS-x4
13
Enterprise SSD Form Factor 1.0a
2.3 PRSNT# - Presence Detect and IfDet# - Interface Detect Signal Definition
The Presence (PRSNT#) is similar to Presence on the existing SFF-8482 connector. There is a
redefinition of the IfDet# (pin P4) to encode drive type. The usage of the combined signals is to detect a
drive is present and the drive type. Figure 6 shows how the drive type can be decoded from PRSNT# and
IfDet# pins. .
Figure 6: Drive type - Pin decoding
1
SATA Express Gnd (drv) Open (drv) Open(NC) PCIe on SATA/SAS lanes (S1-S14)
(Client PCIe) sPERst# is used, no RefClk used
(P4 is NC, E25 is NC)
Enterprise PCIe Open (drv) Gnd (drv) High (by PCIe are on new Enterprise lanes (E7-E15,
(SFF-8639) system) S16-S28, E17-E22)
RefClk0 and ePERst0# are used
Dual Port Open (drv) Gnd (drv) Low (by PCIe are on new Enterprise lanes (
Enterprise PCIe system) PortA = E7- E15, S16-S22
(SFF-8639) PortB = S22-S28, E17-E22)
RefClk0, ePERst0#, and
RefClk1, ePERst1# are used
No drive Open (drv) Open (drv) - All open
present
1. SATA Express may define IfDet# as a pull up & resistance. The SATA Express documentation is
normative.
Either P10 or P4 being low indicates a drive is present. Further decoding is when P4 is open, then the
PCIe lanes use the SATA pins (S1-S14) as defined for SATA Express.
In-band signal discovery is used to determine SATA vs. SAS. In-band signal discovery is used for width
(x1, x2, x4) determination for SAS and PCIe. This is consistent with the existing SAS and PCIe standards.
14
Enterprise SSD Form Factor 1.0a
The electrical signal characteristics of DualPortEn# follows the sideband interface signals of PCIe as
stated in PCIe CEM Specification 2.6 Auxiliary Signal Parametric Specifications (PCI SIG CEM,
2007).
Section 5.1 on page 33 gives more details on Dual Port operation.
2.5 Reserved(WAKE#/OBFF#), Reserved(ClkReq#/DevSLP)
The Reserved(Wake#/OBFF#), Reserved(ClkReq#/DevSLP), signals are all optional and are used in
device power management. These signals are defined by the PCIe Specification (PCI SIG ECNs,
various). At this time the pin definition and usage in Enterprise PCIe SSD is not completely defined, so
the pins are listed as reserved.
2.6 ePERst[1:0]# - PCIe Reset
The Enterprise PCIe Reset (ePERst[1:0]#) are logically the same as defined in the PCIe Specification
(PCI SIG CEM, 2007) but for Enterprise PCIe SSD there are additional input current requirements. This
allows for a drive to have pull up or pull down circuitry to support hot removal. This is defined in Section
7.3. There are two resets to allow for dual port operation, as defined in Section 5.
The Enterprise PCIe Reset (ePERst0#) is required to be supplied by the system for the Enterprise PCIe
SSD.
If DualPort# is asserted, then Port B Enterprise PCIe Reset (ePERst1#) is required to be supplied by the
system for the Port B of the Enterprise PCIe SSD.
2.7 RefClk[1:0][+/-] – Enterprise PCIe Clocks
The Enterprise PCIe Reference Clock (RefClk0[+/-]) should be supplied by the system for all Enterprise
PCIe SSDs.
If DualPort# is asserted, then Port B Enterprise PCIe Reference Clock (RefClk1[+/-]) should be supplied
by the system for all Port B (second port) of the Enterprise PCIe SSDs.
Implementation Note: RefClk is required for systems that use Spread Spectrum Clocking (SSC) to track
the changes in the reference clock. In a typical server system it is expected that Enterprise PCIe SSD will
need to use RefClk# for correct operation. This is the same as PCIe slots. RefClk does not need to be
supplied if both the platform and drive support SRIS - Separate RefClk with Independent SSC (PCI SIG
ECNs, various) or SRNS - Separate RefClk with No SSC (independent clocking).
Implementation Note: RefClk is required for Enterprise PCIe SSD but note that SATA-Express is
pursuing a PCI SIG specification change in future root ports that will enable no RefClk for SATA-Express.
2.8 Activity - Activity indication signal
The Activity signal asserts with drive activity. The blink patterns the same as for existing SAS drives, and
may have vendor defined blink patterns. The system use of the Activity signal is optional.
Implementation Note: An overview is the Activity signal is asserted when drive is active, but blinks off
"occasionally" during periods of continuous activity to distinguish it from other LED states. The Activity
signal is called “Ready” in the SAS documentation and the definition is in the SAS Protocol Layer - 2
(SPL-2) Specification (T10, SPL 2, 2011) in Section 9.4.1 SAS Ready signal behavior.
15
Enterprise SSD Form Factor 1.0a
3 Connector Mechanical
Note: this section with the connector mechanical drawing is informative. The normative specification is
SFF-8639(SFF-8639, 2012).
Figure 7 gives an overview of the mechanical layout of the connector. The new signals are shown in red.
Figure 8 gives an isometric view of representative examples of the connectors from multiple angles.
Figure 7: Connector Mechanical Overview
16
Enterprise SSD Form Factor 1.0a
17
Enterprise SSD Form Factor 1.0a
Figure 11: Connector Plug Mechanical Drawing – Bottom View (outside of enclosure)
Figure 12: Connector Plug Mechanical Drawing – End View (showing latch slot)
18
Enterprise SSD Form Factor 1.0a
19
Enterprise SSD Form Factor 1.0a
Figure 16: Connector Plug Mechanical Drawing – Section D-D (Tongue cross section)
20
Enterprise SSD Form Factor 1.0a
21
Enterprise SSD Form Factor 1.0a
22
Enterprise SSD Form Factor 1.0a
23
Enterprise SSD Form Factor 1.0a
24
Enterprise SSD Form Factor 1.0a
25
Enterprise SSD Form Factor 1.0a
26
Enterprise SSD Form Factor 1.0a
Implementation Note: Surprise Hot Removal is a must have requirement from system vendors for PCIe
SSD devices. More specifically, it is expected that system vendors will require Enterprise PCIe SSD to
retain all “committed data” following a surprise hot removal. This implies that the Enterprise PCIe SSD
locally detect hot removal and store/destage any cache data. For a typical implementation this means the
following is implemented
Having power sustaining capability, such as a battery or super cap.
Detecting removal (by loss of 12V input or link loss)
27
Enterprise SSD Form Factor 1.0a
Copying all “committed data” from volatile storage (DRAM based write back cache) to non-volatile
storage. The details of this operation are complex, and implementation specific. The definition of
“committed data” is specific to the higher level interface but typically means acknowledged write
data. This “committed data” includes all indirection pointers to the data.
This copying of data has to be recovered upon PCIe SSD power on or hot addition. It is common
for data or indirection tables to be written in to a staging location to simplify power down. This has
to be reconstructed on power on/hot add to recover the full committed PCIe SSD state.
From a practical perspective for the Enterprise PCIe SSD, surprise hot removal is not that different from
surprise power fail. In system power fail 12V likely has a longer power down ramp, but that is not
guaranteed.
Implementation Note: Limiting the inrush and hot plug support require merging information from a
number of sources. SFF-8639 (SFF-8639, 2012) has the mechanical specifications for blind mate
tolerance and the 0.5mm difference in mating pin length. The SATA Specification (SATA-IO, various)
st
gives a typical existing system insertion rates of 3ms delay from the 1 precharge 12V power to
28
Enterprise SSD Form Factor 1.0a
remaining power pins contacting. Given these parameters and the 12Vcap specification in Section 7.4,
the backplane resistance (RL in Figure 24) can be determined.
4.2 Detecting Enterprise PCIe SSD Insertion/Removal
There are multiple methods to detect PCIe SSD insertion or removal.
Use the drive type pins (PRSNT# pin P10, and IfDet#, pin P4). Note drive type can be
determined as described in Section 2.3.
Use of In-Band signaling. The PCIe or SAS links may be enabled to train periodically,
succeeding when device is plugged in. The PCIe or SAS links are lost when device is removed.
How system software is signaled is beyond the scope of this specification.
Implementation Note: There are three broad classes of usage of Presence (PRSNT#) and SATA Detect
(IfDet#) during hot insertion or removal.
1. A common implementation is expected to be that PRSNT# and IfDet# are routed to the existing
storage enclosure microcontroller and the microcontroller sends specific drive insertion or
removal information to system software using the management stack. This is a common SAS
implementation extended to two pins for 3 type encodings.
2. Alternatively the PRSNT# & IfDet# signals are simply turned into a general storage attention
interrupt, enabling the system software to probe the system for changes. This can be augmented
by examining if any PCIe link has changed status, or used as a trigger to enable Enterprise PCIe
SSD to test for link training.
3. Optionally an existing storage enclosure microcontroller can poll and read the SM-bus to read
vital product data (VPD) to learn specific information about the inserted device.
4. It is acceptable for the PRSNT# and IfDet# signals to be ignored completely by system hardware.
Enterprise PCIe SSD insertion detection can be done in-band by detecting the training of the
PCIe link. This may be complicated by PCIe links being disabled (powered down), so periodic re-
arming may be required to probe for newly added Enterprise PCIe SSDs. In this case Enterprise
PCIe SSD removal is detected by loss of PCIe link or PCIe link errors. From a timing perspective
st
the PCIe link error/loss is the common 1 detection of surprise removal since it is tightly coupled
with ongoing operation.
4.3 PCIe Hardware and Reset System Requirements
Enterprise PCIe SSD Reset/power on and PCIe Hot Add follows the PCIe specification (PCI SIG CEM,
2007). Enterprise PCIe SSDs must generate their own internal power-on reset by detecting the power
ramp on 12V.
In all cases the Root Complex and system software shall allow at least 1.0 second after a Conventional
Reset of an Enterprise PCIe SSD, before determining that an Enterprise PCIe SSD which fails to return a
Successful Completion status for a valid Configuration Request, is an unresponsive or failed Enterprise
PCIe SSD. This 1.0 second is true for both PCIe and SMbus operations. This period is independent of
how quickly Link training completes.
PCIe Reset Card Requirements
● Normal System Operation (not Hot Insertion/Removal)
o An Enterprise PCIe SSD shall adhere to PCIe system reset and card reset requirements
(Figure 25 and Figure 26)
● Hot Insertion
o If reset asserted then an Enterprise PCIe SSD must adhere to PCIe system reset and
slot reset requirements.
This also means that the system & root complex must adhere to these
requirements
o If reset negated
Power and clock shall be stable 10ms after assertion of IfDet# (Enterprise PCIe
SSD’s Presence Detect)
PRSNT# and IfDet# is in the last (3rd) group of pins to mate
29
Enterprise SSD Form Factor 1.0a
This consumes 10ms of the 100ms power stable time in Figure 25 and
Figure 26.
Enterprise PCIe SSD shall generate its own internal reset
Logically PCIe reset is considered to have occurred 100ms after PRSNT# pin
mating
An Enterprise PCIe SSD must adhere to PCIe system reset
requirements from this point on
This simply shifts timing parameters on Figure 26 by 100ms
o This shift must also be comprehended by the platform if hot
insertion occurs during a normal PCIe system reset sequence.
● Hot Removal
o On a Hot Removal the Enterprise PCIe SSD shall perform any necessary clean-up and
retain on the PCIe SSD any previously acknowledged write operations
Figure 25: System Reset Timing
30
Enterprise SSD Form Factor 1.0a
31
Enterprise SSD Form Factor 1.0a
o If Enterprise PCIe SSD is not notified before removal, then it is surprise removal, see
next section.
● Optionally use management system to identify the Enterprise PCIe SSD to remove. For example
blink a red light for failure indication.
● User safely removes the Enterprise PCIe SSD.
If the removal is “surprise” meaning that the Enterprise PCIe SSD is active as it is being removed then the
system software implications grow. These include, but are not limited to, the handling of errors during
surprise removal event.
The software implications vary with the PCIe root port implementation and OS/SW versions. The
software implications are beyond the scope of this specification.
Implementation Note: A complication of Hot Removal is discovering and testing all the boundary cases
of the hardware/software interaction. In general a clean HW/SW interface and a clean software
implementation are warranted – for example not a legacy interrupt model. The basic mechanism for an
error detected by an instruction (ld/st) failure is to fault then handle the fault with by aborting the transfer.
It is expected that any data in flight at the time of removal is lost or corrupted. Hardware is only expected
to retain “acknowledged and committed data”.
Implementation Note: It is expected that older versions of the operating system and PCIe root port will
not be able to support surprise Hot Removal. With older OS/HW the errors during surprise removal might
generate a fatal error.
New versions of the PCIe root port will be design to signal to the system software (BIOS and OS) to
handle the hot removal event. The details are beyond the scope of this specification. The documents are
not yet released. This detail is expected in documents such as the System BIOS Writers Guide for the
root port component.
Implementation Note: Windows client OS is surprise hot remove capable because it was designed to
support ExpressCard form factor. The Slot Capabilities Register contains a bit called Hot Plug Surprise
which is a form factor specific bit. Express Card was expected to set this bit to indicate to Windows that
card could be removed without notification and software stack should handle this without failure or data
loss. For example, any writes to media behind such a card should never be cached by OS.
Since the Server OS shares a common code base with the Client OS at the kernel level, the basic
capability exists in Windows Server but is not validated.
An outline of how Enterprise PCIe SSD Hot Swap will work:
- The OS gets signaled that an Enterprise PCIe SSD is inserted. (Using PCIe Hot Plug
mechanism, a custom OEM leveraged mechanism, or detecting a change in PCIe Link Status.)
- OS enables read caching but not software write caching since the Enterprise PCIe SSD is
surprise remove capable (if this impacts performance, then IT has to ensure they go through
software UI to prep for ejection first).
- When the Enterprise PCIe SSD is removed cleanly, the error handling will abort any active
transfers, then the presence detect toggle will indicate the hot remove event to OS
- OS will unload the corresponding driver stack and complete all pending IO request packets.
32
Enterprise SSD Form Factor 1.0a
Cmd Q
Connector
PCIe upstream PCIe x4 SSD Form Factor
Device Data
DataQQ
(e.g. CPU x4 Data Q
Data Q
PCIe port) (Enterprise PCIe SSD)
A
SSD Form Factor Data Q
Data Q
Device
PCIe x2 (Enterprise PCIe SSD)
Controller Cmd Q
B Data
DataQQ
Data Q
Data Q
.
The two PCIe links must operate independently. It is expected that both ports are fully PCIe compliant
devices and as such each port presents a fully a PCIe compliant endpoint.
It is possible that the PCIe device is a multi-root PCIe switch, but that is not expected to be the common
solution. In this solution resets and failures (such as errors and clock failures) on one port must not affect
operation on the other port.
Implementation Note: In the most common dual port usage the Enterprise PCIe SSD appears as a
totally independent Enterprise PCIe SSD on each bus sharing media. In Figure 27 both Controller A and
Controller B see standard PCIe devices. There is not expected to be a PCIe-probe SW discoverable
connection to the other port. There are two completely independent interfaces – such as two independent
sets of PCIe configuration space, memory mapped control registers and queues. A higher level
mechanism (such as reporting the same GUID on both links, or inherent knowledge of the topology) allow
high level storage software to detect that the two Enterprise PCIe SSD controllers are sharing a common
pool of storage and the higher level software coordinates the allocation of resources.
33
Enterprise SSD Form Factor 1.0a
Implementation Note: All Enterprise PCIe Devices have a single supply and single ground return plane.
For signal integrity there MUST be a single ground plane. Any redundant supply is handled outside the
Enterprise PCIe SSD.
This specification places no access restriction or interlocks on the usage of the two ports. Specifically
writing independently to the same storage block from both ports the ordering is not guaranteed. Higher
level software or system architecture is assumed to prevent any unsafe overlapping commands.
5.1 Dual Port Mode (implications of DualPortEn#)
The Enterprise PCIe SSD may be configured to train as either a single x4 port (port A) or dual x2 ports
(i.e., ports A and B). When DualPortEn# (pin E25) is asserted (typically grounded by storage backplane),
then the Enterprise PCIe SSD shall be configured for dual port operation. DualPortEn# is left
unconnected or undriven by the system for single port operation. The Enterprise PCIe SSD shall pull
DualPortEn# high (deasserted) if left unconnected by the system.
When configured for single port operation, the following rules apply:
● Enterprise PCIe SSD shall not train as two independent links.
● Enterprise PCIe SSD may train as a single x4, x2 or x1 link. Support for x2 is optional for both
the system and the PCIe SSD.
● Enterprise PCIe SSD shall use ePERst0# to control the PCIe interface as defined by PCIe
specifications (PCI SIG CEM, 2007).
● Enterprise PCIe SSD may use RefClk0 (if operating in common clock mode)
● Enterprise PCIe SSD shall ignore dual port signals, RefClk1, ePERst1#.
When configured for dual port operation, the following rules apply:
● Port A shall use PCIe lanes 0 and 1. Port B shall use PCIe lanes 2 and 3.
● Enterprise PCIe SSD shall not train as a single x4 link.
● Enterprise PCIe SSD shall train as a dual independent x2 if both upstream controllers are active
and the Enterprise PCIe SSD supports dual port.
● Either port may train only as x2 or x1.
● Enterprise PCIe SSD may train one port, both ports, or neither port.
● Each PCIe port shall operate completely independently
o ePERst0# assertion/negation or loss of RefClk0 or errors affecting PCIe port A shall not
affect the operation on the PCIe Port B.
o ePERst1# assertion/negation or loss of RefCLk1 or errors affecting PCIe port B shall not
affect the operation on the PCIe Port A.
● When an Enterprise PCIe SSD is not capable of dual port operation but is configured for dual port
operation, the following rules apply:
o Enterprise PCIe SSD should train on either port A or Port B in x2 or x1. This is to allow
just a single upstream controller active. This requires the appropriate PCIe Clock and
PCIe Reset to be used. Specifically if Port A is trained then RefClk0 and ePERst0# are
used, and if Port B is trained then RefClk1 and ePERst1# are used. For a single port
PCIe SSD support for Port B is optional, and support for Port A is required.
o Enterprise PCIe SSD shall not drive or terminate the signals on the port that is not
trained. Specifically: If Port A is trained, then Enterprise PCIe SSD shall not drive or
terminate Port B signals; or if only Port B is trained, then PCIe shall not drive or terminate
Port A signals.
Implementation Note: For a single port device that is presented with DualPortEn# asserted, it is
important to disable driving and terminating the unused pins because some upstream ports may detect
the electrical drive or termination as an electrical presence and try to train the port which would create
training errors. This capability can be implemented using standard PCIe IP block sitting on Port A, and
using the force width capability to disable the unused lanes, or by a separate HW disable derived from
DualPortEn# to disable lanes 2-3.
Implementation Note: There is a partial solution called Active/Passive mode. This is defined in Section
9.2.4.
34
Enterprise SSD Form Factor 1.0a
6 SM-Bus Operation
The Enterprise PCIe SSD should support SM-Bus operation. The SM-Bus protocol and signaling are
defined in the SM-Bus Specification (SMBus, 2000). SM-Bus operation is at 100khz or less.
SM-Bus is defined as independent of the PCIe links. There is no defined relationship between SM-Bus
accesses and accesses on the PCIe buses.
There are two level of SM-Bus access.
1. Single reads and writes to a PROM device holding VPD (Vital Product Data) used for Enterprise
PCIe SSD discovery and power allocation. Supported on 3.3Vaux power and during normal
operation. The register definitions are listed in Section 9.1 SM-Bus Vital Product Data (VPD).
Each Enterprise PCIe SSD’s VPD is expected to be independently addressed on a unique SM-
Bus segment with VPD at a fixed address. SM-Bus ARP is not supported for VPD access. An
Enterprise PCIe SSD may require 3.3Vaux to be supplied to access this VPD data.
2. Extended management and access using MCTP-over-SMbus and this mode is only supported
during normal operation (e.g., when 12V is present). MTCP Commands are defined in Section
9.2
Implementation Note: While SM-Bus is optional for Enterprise PCIe SSD operation, it is expected to be
included in the Enterprise PCIe SSD purchase criteria by many system vendors.
Implementation note: This is specified to allow implementation with an I2C PROM connected to SM-Bus
and to 3.3Vaux. It is allowed to have the VPD with 3.3Vaux applied but required to apply 12V to access
optional extended management capabilities.
Implementation Note: The SM-Bus is not replicated for dual port operation. In dual controller
configuration, this is envisioned as going to a shared management agent, using the multi-master
characteristics of SM-Bus, or independent SM-Buses merged external to the Enterprise PCIe SSD.
6.1 Simple Accesses to Vital Product Data (VPD)
Enterprise PCIe SSD should support simple Reads and Write to Vital Product Data. This is defined in
Section 9.1 SM-Bus Vital Product Data (VPD) in summary it contains:
Basic inventory information such as type and size of Enterprise PCIe SSD, manufacturer, date,
revision, and GUID.
Power management data such as power level and power modes
Vendor specific data
Implementation note: This is likely implemented using a SM-Bus PROM connected to SM-Bus and to
3.3Vaux.
If a system vendor wants access to this data, then 3.3V shall be supplied to 3.3Vaux pins. If the system
vendor does not want access to this data via SM-bus, then 3.3Vaux may be optional.
6.2 Extended SM-Bus management
An Enterprise PCIe SSD may support full MCTP device access over SMbus (MCTP-over-SMbus). This
is only possible during normal operation (e.g. when 12V present).
The MCTP commands are beyond the scope of this specification.
Implementation Note:
The MCTP-over-SMbus is implementation dependent but is likely to support:
Access to VPD using MCTP semantics
Access to Enterprise PCIe SSD control registers
Access to underlying media (with protection mechanisms)
MCTP-over-SMbus is likely to be nearly symmetric with MCTP-over-PCIe.
35
Enterprise SSD Form Factor 1.0a
d g
Conn
Motherboard
or Adaptor
Connector
ASIC
card SSD Form
Cable Factor Device
f
Conn
Storage
Backplane
36
Enterprise SSD Form Factor 1.0a
1
12Vamp Max continuous current 2.45A (max) Higher than PCI CEM
2
12Vpeak-amp Max peak current 4.5A (max)
3
12Vcap Max cap load 5uF (max)
12Vdrop Voltage drop across connector 80mV
SM-Bus Delay from 12V being within spec 20ms (min) The Enterprise PCIe
delay before SM-bus access to any SSD has up to 1.0s
1.0s (max)
SM-bus slave address other than before responding to
VPD. SM-bus transactions.
(Section 4.3)
Delay from PRSNT# connector 20ms (min)
mating to an SM-bus access to
1.0s (max)
any SM-bus slave address other
than VPD.
1. Maximum continuous current is defined as the highest averaged current value over any one
second period.
2. Maximum current to limit connector damage and limit instantaneous power.
3. Maximum capacitance presented by the Enterprise PCIe SSD on the 12V power rail at the
backplane connector.
4. System Software can set a different lower limit using the PCIe Express Slot Capabilities register
defined in the PCIe Specifications (PCI SIG 3.0, 2010). This value can be set before an
37
Enterprise SSD Form Factor 1.0a
Enterprise PCIe SSD is installed, and the value will be sent to the device just after link training
and overrides this default. This is standard PCIe behavior.
5. No voltage tolerance is specified. Due diligence is need to confirm system and drive have
compatible specifications.
Implementation Note: To accommodate all drive types possible with a flexible backplane using SFF-
8639 connector, a 12V voltage tolerance of ±5% would conservatively support shipping SATA and SAS
drives. The SATA and SAS standards do not specify 12V voltage tolerance, but many existing drives
specify ±5% tolerance. The PCIe CEM (PCI SIG CEM, 2007) specifies ±9% on 12V in the standard PCIe
card edge form factor. System vendors commonly specify a voltage tolerance in their purchasing
specifications, commonly in the ±5-±8% range but not always symmetrical.
Note that Rev 1.0 of this specification specified a 12V voltage tolerance of ±15%, but this changed to
unspecified in Rev 1.0a. This is due to no single value being optimal for all flexible backplane
configurations.
7.5 Power 3.3Vaux
Enterprise PCIe SSDs may support 3.3V operation. SM-bus access is supported only when 3.3Vaux is
supplied.
The system may choose to not supply 3.3Vaux if limitation to SM-bus access is acceptable. The
Enterprise PCIe SSD shall operate even with no 3.3Vaux supplied. Enterprise PCIe SSD operation shall
not be dependent on 3.3Vaux except for SM-bus.
Implementation Note: Systems may connect 3.3VAux to either a standard 3.3V rail, or a special rail that
is powered even when system is in a standby state with main 12V rail disabled.
Implementation Note: A typical Enterprise PCIe SSD could connect a small PROM to SM-Bus and
power this device with 3.3Vaux. This VPD is only accessible if 3.3Vaux is applied by the system. SM-
Bus could also be routed to the controller ASIC and these SM-bus operations would be dependent on
12V power rail.
Figure 30: Power 3.3VAux specifications
Parameter Definition Value Comment
3.3VAuxtol Voltage Tolerance (at pin) ±15% (max) Relaxed from PCIe CEM
1
3.3VAuxamp Max continuous current 20µA Lower than PCI CEM
(SM-Bus inactive)
1mA
12V applied so system
(SM-Bus active
not in S5-Soft Off (S5)
or 12V applied)
2
3.3VAuxcap Max cap load 5µF (max)
SM-Bus delay Delay from 3.3V being within spec 20ms (min) The Enterprise PCIe SSD
before SM-Bus access to VPD serial has up to 1.0s before
1.0s (max)
EEPROM may be performed. responding to SM-bus
transactions. (Section
Delay from PRSNT# & IfDet# 20ms (min) 4.3)
connector mating to when SM-bus
1.0s (max)
access to VPD serial EEPROM may
be performed.
1. Maximum continuous current is defined as the highest averaged current value over any one
second period.
2. Maximum capacitance presented by the PCIe SSD on the 12V power rail at the backplane
connector.
38
Enterprise SSD Form Factor 1.0a
NVM Express. (2011, July 12). NVM Express Revision 1.0b. Retrieved December 13, 2011, from NVM
Express: http://www.nvmexpress.org/
PCI SIG 2.1. (2009, March 4). PCIe® Base Specification, Revision 2.1. Retrieved from PCI SIG:
http://www.pcisig.com/specifications/pciexpress/
PCI SIG 3.0. (2010, November 10). PCIe® Base Specification Revision 3.0. Retrieved from PCI SIG:
http://www.pcisig.com/specifications/pciexpress/
PCI SIG CEM. (2007, April 11). PCI Express® Card Electromechanical Specification Revision 2.0.
Retrieved from PCI SIG: http://www.pcisig.com/specifications/pciexpress/
PCI SIG ECNs. (various). PCI Express Specifications & ECNs. Retrieved from PCI SIG:
http://www.pcisig.com/specifications/pciexpress/specifications/#ecn2
39
Enterprise SSD Form Factor 1.0a
40
Enterprise SSD Form Factor 1.0a
41
Enterprise SSD Form Factor 1.0a
Byte
3 2 1 0
Next Capability
0xA5
Vendor Specific Capability Address
Vendor Specific PCI-SIG Vendor ID
Optional parameters, implementation values
Byte
3 2 1 0
Next Capability
Capability ID = 0xA0
Dual Port Capability Address
Capability ID value of 0xA0 specifies that Enterprise PCIe SSD support Dual Port Mode as
defined in Section 5.
Next Capability Address is offset to next capability with 0x0 meaning end of capability list.
42
Enterprise SSD Form Factor 1.0a
Byte
3 2 1 0
Next Capability
Capability ID = 0xA1
RefClk Capability Address
reserved RefClk Vector
Capability ID value of 0xA1 specifies that SSD supports extended RefClk functionality.
Next Capability Address is offset to next capability with 0x0 meaning end of capability list.
RefClk Vector is a bit vector decoded as follows: (Set all that apply for a device)
o Bit 0 – 0x01 – Common RefClk required as defined – default.
o Bit 1 – 0x02 – Separate RefClk with no SSC (SRNS)
o Bit 2 – 0x04 – Separate RefClk with SSC (SRIS)
o Bit 3 – 0x08 – Can select Common RefClk pin or Separate RefClk using Vendor specific
mechanism.
o Bit 4 - 0x10 – Will automatically sense RefClk pin and use if provided, otherwise will use
Separate RefClk.
43
Enterprise SSD Form Factor 1.0a
Byte
3 2 1 0
Next Capability
Capability ID = 0xA2
Temperature Sensor Capability Address
SM-Bus Sensor
reserved
Address Type
Overtemp Threshold Warning Threshold
44
Enterprise SSD Form Factor 1.0a
Byte
3 2 1 0
Next Capability
Capability ID = 0xA3
RefClk Capability Address
Dual Port
reserved
Vector
Capability ID value of 0xA3 specifies that SSD supports extended Dual Port functionality.
Next Capability Address is offset to next capability with 0x0 meaning end of capability list.
Dual Port Vector is a bit vector decoded as follows: (Set all that apply for a device)
o Bit 0 – 0x01 – Dual Port mode as specified.
o Bit 1 – 0x02 – Active/Passive mode
45
Enterprise SSD Form Factor 1.0a
SATA Express pin placement is only for reference on how to align this specification’s
Enterprise PCIe pin-out with SATA Express standard (SATA-IO, Draft 2011).
SAS-x4 pin placement is only for reference on how to align this specification’s Enterprise
PCIe pin-out with SAS x4 specification (SFF-8630, 2012).
46
Enterprise SSD Form Factor 1.0a
o SAS-x4 may redefine support pins for example layering a second Active (SAS-Active2)
on existing Reserved(WAKE#/OBFF#) signal. If this is done the system must be able to
gate the signal usage based on drive type.
47
Enterprise SSD Form Factor 1.0a
GND
48
Enterprise SSD Form Factor 1.0a
Gray=unused Gray=unused
49
Enterprise SSD Form Factor 1.0a
Figure 39: Backplane for Enterprise SSD (PCIe x4), SATA & SATA Express (Client PCIe)
New
PCIe
X1-x2
SATA, SATA
New
New
50
Enterprise SSD Form Factor 1.0a
Figure 40: Backplane for Enterprise SSD (PCIe x4), SAS & SATA
Storage
Backplane
Flexible Bays
Enterprise
SSD and PCIe or SAS or SATA
SAS Not SATA Express
(Client PCIe),
X4
PCIe x4 Enterprise
New
New
not SAS x4
PCIe
SAS
Enterprise SAS
New
New
PCIe x4
SATA
Controller, SATA
Root port Expanders
SAS
SAS
SAS
SAS
SAS SAS
SATA
Controller SATA
SAS x4 SAS
Controller
SATA
SATA
SAS x4
Expanders
(Supports SAS, SATA)
Existing Bays: SAS or SATA
SAS
SAS
SAS
SAS
SAS
SAS
SATA
SATA
51
Enterprise SSD Form Factor 1.0a
SATA Express keying features are only a proposal on how to align this specification’s with proposals for
SATA Express. The SATA Express keying features are not yet standardized, and this document is not
the normative document for SATA Express.
52
Enterprise SSD Form Factor 1.0a
Mates-Nonfunctional
Mates-nonfunctional
Works with STP (requires STP+) Works
(carrier key)
SAS backplane (carrier key)
Mates-Nonfunctional Mates-nonfunctional
Works Works
(requires STP+) & no detent retention
SAS cable
53
Enterprise SSD Form Factor 1.0a
11 Specification Conventions
11.1 Definitions
11.1.1 PCI Express® (PCIe)
PCIe refer to the PCIe Express IO bus standards. Please see the 8.1 References on page 39 for a list of
the most relevant PCIe documents (PCI SIG, various).
11.1.2 PCI Express® Generation 2 (PCIe Gen2)
In this context Gen 2 is used to refer to a link speed of 5Gbps using 8b/10b encoding.
11.1.3 PCI Express® Generation 3 (PCIe Gen3)
In this context Gen 3 is used to refer to a link speed 8Gbps using a 128/130 encoding with scrambling.
11.1.4 PCI Express® x4 (PCIe x4)
x4 means using 4 PCIe lanes for communication. This is 4 differential pairs in each direction. Total
signals is 4 lanes times 2 (differential) times 2 (Transmit and receive) = 16 signals.
11.1.5 SFF standards
Small Form Factor Committee is an ad hoc group that defined the 2.5” & 3.5” drive and related
specifications. Documents created by the SFF Committee are submitted to bodies such as EIA (Electronic
Industries Association) or an ASC (Accredited Standards Committee).
Please see the 8.1 References on page 39 for a list of the most relevant (SFF, various).
11.1.6 SSD
Solid State Disk.
11.2 Keywords
Several keywords are used to differentiate between different levels of requirements.
11.2.1 mandatory
A keyword indicating items to be implemented as defined by this specification.
11.2.2 may
A keyword that indicates flexibility of choice with no implied preference.
11.2.3 optional
A keyword that describes features that are not required by this specification. However, if any optional
feature defined by the specification is implemented, the feature shall be implemented in the way defined
by the specification.
11.2.4 R, RSVD
“R” and RSVD are used as an abbreviation for “reserved” when the figure or table does not provide
sufficient space for the full word “reserved”.
11.2.5 reserved
A keyword indicating reserved signal pins, bits, bytes, words, fields, and opcode values that are set-aside
for future standardization. Their use and interpretation may be specified by future extensions to this or
other specifications. A reserved bit, byte, word, field, or register shall be cleared to zero, or in accordance
with a future extension to this specification. The recipient shall not check reserved bits, bytes, words, or
fields.
54
Enterprise SSD Form Factor 1.0a
11.2.6 shall
A keyword indicating a mandatory requirement. Designers are required to implement all such mandatory
requirements to ensure interoperability with other products that conform to the specification.
11.2.7 should
A keyword indicating flexibility of choice with a strongly preferred alternative. Equivalent to the phrase “it
is recommended”.
55