Description

Introduces four new types for quantization / computation to speed up deep learning models on GPU.

First pair for NVIDIA:

• FLOATE8E4M3FN: float 8 bits with 4 bits for the exponent and 3 for the mantissa, usually used for the coefficients, support nan values
• FLOATE8E5M2: float 8 bits with 5 bits for the exponent and 2 for the mantissa, usually used for the gradients, supports nan values and infinities

Second pair for GraphCore:

• FLOATE8E4M3FNUZ: float 8 bits with 4 bits for the exponent and 3 for the mantissa, usually used for the coefficients, only (no infinities)
• FLOATE8E5M2FNUZ: float 8 bits with 5 bits for the exponent and 2 for the mantissa, usually used for the gradients, supports nan values only (no infinities)

Suffix FN means no infinities, UZ means the negative zero is used to represent nan values.

The PR modifies operator Cast, CastLike, QuantizeLinear, DequantierLinear to make them support the four new types. It adds function to cast from/to float32.

Motivation and Context

Latest NVIDIA, Arm, Inter, GraphCore introduces float 8 for faster computation.

• FP8 FORMATS FOR DEEP LEARNING: specifications for Nvidia, Arm, Intel
• 8-bit Numerical Formats For Deep Neural Networks (GraphCore)

Other related papers:

• FP8 Formats for Deep Learning
• RFC: FP8 in XLA