Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Commit d62b1c1

Browse files
pritamdamania87pritamdamaniabrianjo
authored
Tutorial for pipeline parallelism. (#1343)
* Tutorial for pipeline parallelism. Summary: The tutorial demonstrates training Transformer models on 2 GPUs using pipeline parallelism. * Add some checks * Some refinements. * Add thumbnail Summary: Test Plan: Reviewers: Subscribers: Tasks: Tags: * Fix legacy torchtext * Rebase and rerun circleci * Use configurable way to build pipeline. * Tweak TensorPipe options. * Add main method * Fix indent * Revert "Fix indent" This reverts commit a39b6c7. * Revert "Add main method" This reverts commit a8a8c26. * Fix Tensorpipe options. Co-authored-by: pritam <[email protected]> Co-authored-by: Brian Johnson <[email protected]>
1 parent 50ecf40 commit d62b1c1

3 files changed

Lines changed: 470 additions & 0 deletions

File tree

‎_static/img/thumbnails/cropped/Training-Transformer-models-using-Pipeline-Parallelism.png‎

34.9 KB
Loading

‎index.rst‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -413,6 +413,13 @@ Welcome to PyTorch Tutorials
413413
:link: advanced/rpc_ddp_tutorial.html
414414
:tags: Parallel-and-Distributed-Training
415415

416+
.. customcarditem::
417+
:header: Training Transformer models using Pipeline Parallelism
418+
:card_description: Walk through a through a simple example of how to train a transformer model using pipeline parallelism.
419+
:image: _static/img/thumbnails/cropped/Training-Transformer-models-using-Pipeline-Parallelism.png
420+
:link: intermediate/pipeline_tutorial.html
421+
:tags: Parallel-and-Distributed-Training
422+
416423
.. customcarditem::
417424
:header: Training Transformer models using Distributed Data Parallel and Pipeline Parallelism
418425
:card_description: Walk through a through a simple example of how to train a transformer model using Distributed Data Parallel and Pipeline Parallelism
@@ -629,6 +636,7 @@ Additional Resources
629636
intermediate/dist_pipeline_parallel_tutorial
630637
intermediate/rpc_async_execution
631638
advanced/rpc_ddp_tutorial
639+
intermediate/pipeline_tutorial
632640
advanced/ddp_pipeline_tutorial
633641

634642
.. toctree::

0 commit comments

Comments
 (0)