Hello,

I hit an issue with FusedLayerNorm raising error when using cpu only.
RuntimeError: input must be a CUDA tensor (layer_norm_affine at csrc/layer_norm_cuda.cp:148)

The use case is either:

• Run some unit test in cpu
• Load model in cpu for inference

The issue is already reported here but personally I think it would make more sense for apex to make sure the Modules works on both cpu and gpu.