Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Commit 52e18c8

Browse files
timmoon10timmoon10
authored
Distributed optimizer support for multiple dtypes (#1721)
* Distopt support for multiple dtypes Rough draft. Signed-off-by: Tim Moon <[email protected]> * Add test for distopt with multiple dtypes Signed-off-by: Tim Moon <[email protected]> * Debug typos Signed-off-by: Tim Moon <[email protected]> * Support for contiguous buffers with multiple dtypes Signed-off-by: Tim Moon <[email protected]> * Fix bug when calling bf16 remainder kernel Signed-off-by: Tim Moon <[email protected]> * Return distopt checkpoint on all ranks Signed-off-by: Tim Moon <[email protected]> * Tweak docs Signed-off-by: Tim Moon <[email protected]> * Black formatting Handle case where we load old checkpoints without multi-dtype support Signed-off-by: Tim Moon <[email protected]> * Tweak logic for BF16 param remainder kernel Signed-off-by: Tim Moon <[email protected]> * Skip distopt test if import fails Signed-off-by: Tim Moon <[email protected]> --------- Signed-off-by: Tim Moon <[email protected]>
1 parent 730376f commit 52e18c8

2 files changed

Lines changed: 534 additions & 396 deletions

File tree

0 commit comments

Comments
 (0)