Graph Compiler For Multi-stream (NVIDIA#1886)

* Graph compiler for multi-stream.

Co-authored-by: Wil Kong <[email protected]>
Co-authored-by: Rundong Li <[email protected]>

* Reuse streams across torch compile graphs.

---------

Co-authored-by: Rundong Li <[email protected]>

Enhance Distributed Fused Adam (NVIDIA#1794)

source of apex in nvcr.io/nvidia/pytorch:23.08-py3

source of apex in nvcr.io/nvidia/pytorch:23.07-py3

source of apex in nvcr.io/nvidia/pytorch:23.06-py3

source of apex in nvcr.io/nvidia/pytorch:23.05-py3

[transformer] Update `build_model` function to support encoder&decode…

…r model (NVIDIA#1307)

* update build_model to support enc&dec model

* fix typo: cur_sargs -> cur_args

* enc&dec path: correctly update pre/post process