Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Commit 2294ad7

Browse files
hunkimchsasank
hunkim
authored and
chsasank
committed
Added Data Parallelism tutorial in blitz (#174)
* Added Data Parallelism tutorial in blitz * Added and note in the cifar10 tutorial * Kill line and all warnings
1 parent 761e369 commit 2294ad7

5 files changed

Lines changed: 265 additions & 2 deletions

File tree

‎_static/img/data_parallel.png‎

15.5 KB
Loading

‎beginner_source/blitz/cifar10_tutorial.py‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -297,6 +297,11 @@ def forward(self, x):
297297
# - Understanding PyTorch's Tensor library and neural networks at a high level.
298298
# - Train a small neural network to classify images
299299
#
300+
# Training on multiple GPUs
301+
# -------------------------
302+
# If you want to see even more MASSIVE speedup using all of your GPUs,
303+
# please check out :doc:`data_parallel_tutorial`.
304+
#
300305
# Where do I go next?
301306
# -------------------
302307
#

‎beginner_source/blitz/data_parallel_tutorial.py‎

Lines changed: 254 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,254 @@
1+
"""
2+
Optional: Data Parallelism
3+
==========================
4+
**Authors**: `Sung Kim <https://github.com/hunkim>`_ and `Jenny Kang <https://github.com/jennykang>`_
5+
6+
In this tutorial, we will learn how to use multiple GPUs using ``DataParallel``.
7+
8+
It's very easy to use GPUs with PyTorch. You can put the model on a GPU:
9+
10+
.. code:: python
11+
12+
model.gpu()
13+
14+
Then, you can copy all your tensors to the GPU:
15+
16+
.. code:: python
17+
18+
mytensor = my_tensor.gpu()
19+
20+
Please note that just calling ``mytensor.gpu()`` won't copy the tensor
21+
to the GPU. You need to assign it to a new tensor and use that tensor on the GPU.
22+
23+
It's natural to execute your forward, backward propagations on multiple GPUs.
24+
However, Pytorch will only use one GPU by default. You can easily run your
25+
operations on multiple GPUs by making your model run parallelly using
26+
``DataParallel``:
27+
28+
.. code:: python
29+
30+
model = nn.DataParallel(model)
31+
32+
That's the core behind this tutorial. We will explore it in more detail below.
33+
"""
34+
35+
36+
######################################################################
37+
# Imports and parameters
38+
# ----------------------
39+
#
40+
# Import PyTorch modules and define parameters.
41+
#
42+
43+
import torch
44+
import torch.nn as nn
45+
from torch.autograd import Variable
46+
from torch.utils.data import Dataset, DataLoader
47+
48+
# Parameters and DataLoaders
49+
input_size = 5
50+
output_size = 2
51+
52+
batch_size = 30
53+
data_size = 100
54+
55+
56+
######################################################################
57+
# Dummy DataSet
58+
# -------------
59+
#
60+
# Make a dummy (random) dataset. You just need to implement the
61+
# getitem
62+
#
63+
64+
class RandomDataset(Dataset):
65+
66+
def __init__(self, size, length):
67+
self.len = length
68+
self.data = torch.randn(length, size)
69+
70+
def __getitem__(self, index):
71+
return self.data[index]
72+
73+
def __len__(self):
74+
return self.len
75+
76+
rand_loader = DataLoader(dataset=RandomDataset(input_size, 100),
77+
batch_size=batch_size, shuffle=True)
78+
79+
80+
######################################################################
81+
# Simple Model
82+
# ------------
83+
#
84+
# For the demo, our model just gets an input, performs a linear operation, and
85+
# gives an output. However, you can use ``DataParallel`` on any model (CNN, RNN,
86+
# Capsule Net etc.)
87+
#
88+
# We've placed a print statement inside the model to monitor the size of input
89+
# and output tensors.
90+
# Please pay attention to what is printed at batch rank 0.
91+
#
92+
93+
class Model(nn.Module):
94+
# Our model
95+
96+
def __init__(self, input_size, output_size):
97+
super(Model, self).__init__()
98+
self.fc = nn.Linear(input_size, output_size)
99+
100+
def forward(self, input):
101+
output = self.fc(input)
102+
print(" In Model: input size", input.size(),
103+
"output size", output.size())
104+
105+
return output
106+
107+
108+
######################################################################
109+
# Create Model and DataParallel
110+
# -----------------------------
111+
#
112+
# This is the core part of the tutorial. First, we need to make a model instance
113+
# and check if we have multiple GPUs. If we have multiple GPUs, we can wrap
114+
# our model using ``nn.DataParallel``. Then we can put our model on GPUs by
115+
# ``model.gpu()``
116+
#
117+
118+
model = Model(input_size, output_size)
119+
if torch.cuda.device_count() > 1:
120+
print("Let's use", torch.cuda.device_count(), "GPUs!")
121+
# dim = 0 [30, xxx] -> [10, ...], [10, ...], [10, ...] on 3 GPUs
122+
model = nn.DataParallel(model)
123+
124+
if torch.cuda.is_available():
125+
model.cuda()
126+
127+
128+
######################################################################
129+
# Run the Model
130+
# -------------
131+
#
132+
# Now we can see the sizes of input and output tensors.
133+
#
134+
135+
for data in rand_loader:
136+
if torch.cuda.is_available():
137+
input_var = Variable(data.cuda())
138+
else:
139+
input_var = Variable(data)
140+
141+
output = model(input_var)
142+
print("Outside: input size", input_var.size(),
143+
"output_size", output.size())
144+
145+
146+
######################################################################
147+
# Results
148+
# -------
149+
#
150+
# When we batch 30 inputs and 30 outputs, the model gets 30 and outputs 30 as
151+
# expected. But if you have GPUs, then you can get results like this.
152+
#
153+
# 2 GPUs
154+
# ~~~~~~
155+
#
156+
# If you have 2, you will see:
157+
#
158+
# .. code:: bash
159+
#
160+
# # on 2 GPUs
161+
# Let's use 2 GPUs!
162+
# In Model: input size torch.Size([15, 5]) output size torch.Size([15, 2])
163+
# In Model: input size torch.Size([15, 5]) output size torch.Size([15, 2])
164+
# Outside: input size torch.Size([30, 5]) output_size torch.Size([30, 2])
165+
# In Model: input size torch.Size([15, 5]) output size torch.Size([15, 2])
166+
# In Model: input size torch.Size([15, 5]) output size torch.Size([15, 2])
167+
# Outside: input size torch.Size([30, 5]) output_size torch.Size([30, 2])
168+
# In Model: input size torch.Size([15, 5]) output size torch.Size([15, 2])
169+
# In Model: input size torch.Size([15, 5]) output size torch.Size([15, 2])
170+
# Outside: input size torch.Size([30, 5]) output_size torch.Size([30, 2])
171+
# In Model: input size torch.Size([5, 5]) output size torch.Size([5, 2])
172+
# In Model: input size torch.Size([5, 5]) output size torch.Size([5, 2])
173+
# Outside: input size torch.Size([10, 5]) output_size torch.Size([10, 2])
174+
#
175+
# 3 GPUs
176+
# ~~~~~~
177+
#
178+
# If you have 3 GPUs, you will see:
179+
#
180+
# .. code:: bash
181+
#
182+
# Let's use 3 GPUs!
183+
# In Model: input size torch.Size([10, 5]) output size torch.Size([10, 2])
184+
# In Model: input size torch.Size([10, 5]) output size torch.Size([10, 2])
185+
# In Model: input size torch.Size([10, 5]) output size torch.Size([10, 2])
186+
# Outside: input size torch.Size([30, 5]) output_size torch.Size([30, 2])
187+
# In Model: input size torch.Size([10, 5]) output size torch.Size([10, 2])
188+
# In Model: input size torch.Size([10, 5]) output size torch.Size([10, 2])
189+
# In Model: input size torch.Size([10, 5]) output size torch.Size([10, 2])
190+
# Outside: input size torch.Size([30, 5]) output_size torch.Size([30, 2])
191+
# In Model: input size torch.Size([10, 5]) output size torch.Size([10, 2])
192+
# In Model: input size torch.Size([10, 5]) output size torch.Size([10, 2])
193+
# In Model: input size torch.Size([10, 5]) output size torch.Size([10, 2])
194+
# Outside: input size torch.Size([30, 5]) output_size torch.Size([30, 2])
195+
# In Model: input size torch.Size([4, 5]) output size torch.Size([4, 2])
196+
# In Model: input size torch.Size([4, 5]) output size torch.Size([4, 2])
197+
# In Model: input size torch.Size([2, 5]) output size torch.Size([2, 2])
198+
# Outside: input size torch.Size([10, 5]) output_size torch.Size([10, 2])
199+
#
200+
# 8 GPUs
201+
# ~~~~~~~~~~~~~~
202+
#
203+
# If you have 8, you will see:
204+
#
205+
# .. code:: bash
206+
#
207+
# Let's use 8 GPUs!
208+
# In Model: input size torch.Size([4, 5]) output size torch.Size([4, 2])
209+
# In Model: input size torch.Size([4, 5]) output size torch.Size([4, 2])
210+
# In Model: input size torch.Size([2, 5]) output size torch.Size([2, 2])
211+
# In Model: input size torch.Size([4, 5]) output size torch.Size([4, 2])
212+
# In Model: input size torch.Size([4, 5]) output size torch.Size([4, 2])
213+
# In Model: input size torch.Size([4, 5]) output size torch.Size([4, 2])
214+
# In Model: input size torch.Size([4, 5]) output size torch.Size([4, 2])
215+
# In Model: input size torch.Size([4, 5]) output size torch.Size([4, 2])
216+
# Outside: input size torch.Size([30, 5]) output_size torch.Size([30, 2])
217+
# In Model: input size torch.Size([4, 5]) output size torch.Size([4, 2])
218+
# In Model: input size torch.Size([4, 5]) output size torch.Size([4, 2])
219+
# In Model: input size torch.Size([4, 5]) output size torch.Size([4, 2])
220+
# In Model: input size torch.Size([4, 5]) output size torch.Size([4, 2])
221+
# In Model: input size torch.Size([4, 5]) output size torch.Size([4, 2])
222+
# In Model: input size torch.Size([4, 5]) output size torch.Size([4, 2])
223+
# In Model: input size torch.Size([2, 5]) output size torch.Size([2, 2])
224+
# In Model: input size torch.Size([4, 5]) output size torch.Size([4, 2])
225+
# Outside: input size torch.Size([30, 5]) output_size torch.Size([30, 2])
226+
# In Model: input size torch.Size([4, 5]) output size torch.Size([4, 2])
227+
# In Model: input size torch.Size([4, 5]) output size torch.Size([4, 2])
228+
# In Model: input size torch.Size([4, 5]) output size torch.Size([4, 2])
229+
# In Model: input size torch.Size([4, 5]) output size torch.Size([4, 2])
230+
# In Model: input size torch.Size([4, 5]) output size torch.Size([4, 2])
231+
# In Model: input size torch.Size([4, 5]) output size torch.Size([4, 2])
232+
# In Model: input size torch.Size([4, 5]) output size torch.Size([4, 2])
233+
# In Model: input size torch.Size([2, 5]) output size torch.Size([2, 2])
234+
# Outside: input size torch.Size([30, 5]) output_size torch.Size([30, 2])
235+
# In Model: input size torch.Size([2, 5]) output size torch.Size([2, 2])
236+
# In Model: input size torch.Size([2, 5]) output size torch.Size([2, 2])
237+
# In Model: input size torch.Size([2, 5]) output size torch.Size([2, 2])
238+
# In Model: input size torch.Size([2, 5]) output size torch.Size([2, 2])
239+
# In Model: input size torch.Size([2, 5]) output size torch.Size([2, 2])
240+
# Outside: input size torch.Size([10, 5]) output_size torch.Size([10, 2])
241+
#
242+
243+
244+
######################################################################
245+
# Summary
246+
# -------
247+
#
248+
# DataParallel splits your data automatically and sends job orders to multiple
249+
# models on several GPUs. After each model finishes their job, DataParallel
250+
# collects and merges the results before returning it to you.
251+
#
252+
# For more information, please check out
253+
# http://pytorch.org/tutorials/beginner/former\_torchies/parallelism\_tutorial.html.
254+
#

‎beginner_source/deep_learning_60min_blitz.rst‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -24,6 +24,7 @@ Goal of this tutorial:
2424
/beginner/blitz/autograd_tutorial
2525
/beginner/blitz/neural_networks_tutorial
2626
/beginner/blitz/cifar10_tutorial
27+
/beginner/blitz/data_parallel_tutorial
2728

2829
.. galleryitem:: /beginner/blitz/tensor_tutorial.py
2930
:figure: /_static/img/tensor_illustration_flat.png
@@ -37,6 +38,9 @@ Goal of this tutorial:
3738
.. galleryitem:: /beginner/blitz/cifar10_tutorial.py
3839
:figure: /_static/img/cifar10.png
3940

41+
.. galleryitem:: /beginner/blitz/data_parallel_tutorial.py
42+
:figure: /_static/img/data_parallel.png
43+
4044
.. raw:: html
4145

4246
<div style='clear:both'></div>

‎intermediate_source/dist_tuto.rst‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
Writing Distributed Applications with PyTorch
2-
===============================
2+
=============================================
33
**Author**: `Séb Arnold <http://seba1511.com>`_
44

55
In this short tutorial, we will be going over the distributed package of PyTorch. We'll see how to set up the distributed setting, use the different communication strategies, and go over some the internals of the package.
@@ -373,7 +373,7 @@ world.
373373
dist.all_reduce(param.grad.data, op=dist.reduce_op.SUM)
374374
param.grad.data /= size
375375
376-
*Et voilà *! We successfully implemented distributed synchronous SGD and
376+
*Et voilà*! We successfully implemented distributed synchronous SGD and
377377
could train any model on a large computer cluster.
378378

379379
**Note:** While the last sentence is *technically* true, there are `a

0 commit comments

Comments
 (0)