Thanks to visit codestin.com
Credit goes to github.com

Skip to content
This repository was archived by the owner on Nov 17, 2023. It is now read-only.

Use In-place operator to prevent memory spikes in optimizer updates#13960

Merged
eric-haibin-lin merged 1 commit into
apache:masterfrom
anirudhacharya:opt_mem
Feb 15, 2019
Merged

Use In-place operator to prevent memory spikes in optimizer updates#13960
eric-haibin-lin merged 1 commit into
apache:masterfrom
anirudhacharya:opt_mem

Conversation

@anirudhacharya
Copy link
Copy Markdown
Member

@anirudhacharya anirudhacharya commented Jan 22, 2019 •
edited
Loading

Description

The update rules in Nadam, Adadelta, Adamax and SGLD optimizers have been changed to using in-place operators to prevent memory spikes during execution.

Checklist

Essentials

Please feel free to remove inapplicable items for your PR.

  • The PR title starts with [MXNET-$JIRA_ID], where $JIRA_ID refers to the relevant JIRA issue created (except PRs with tiny changes)
  • Changes are complete (i.e. I finished coding on this PR)
  • All changes have test coverage:
  • Unit tests are added for small changes to verify correctness (e.g. adding a new operator)
  • Nightly tests are added for complicated/long-running ones (e.g. changing distributed kvstore)
  • Build tests will be added for build configuration changes (e.g. adding a new build option with NCCL)
  • Code is well-documented:
  • For user-facing API changes, API doc string has been updated.
  • For new C++ functions in header files, their functionalities and arguments are documented.
  • For new examples, README.md is added to explain the what the example does, the source of the dataset, expected performance on test set and reference to the original paper if applicable
  • Check the API doc at http://mxnet-ci-doc.s3-accelerate.dualstack.amazonaws.com/PR-$PR_ID/$BUILD_ID/index.html
  • To the my best knowledge, examples are either not affected by this change, or have been fixed to be compatible with this change

@anirudhacharya
Copy link
Copy Markdown
Member Author

anirudhacharya commented Jan 22, 2019 •
edited
Loading

Some statistics from profiler.dump() on running with an mnist example.

Adadelta Old

Profile Statistics.
	Note that counter items are counter values and not time units.
Device Storage
=================
Name                          Total Count        Time (ms)    Min Time (ms)    Max Time (ms)    Avg Time (ms)
----                          -----------        ---------    -------------    -------------    -------------
Memory: cpu/0                        1908         627.2000           0.0000        2007.0400        1003.5200

Adadelta New

Profile Statistics.
	Note that counter items are counter values and not time units.
Device Storage
=================
Name                          Total Count        Time (ms)    Min Time (ms)    Max Time (ms)    Avg Time (ms)
----                          -----------        ---------    -------------    -------------    -------------
Memory: cpu/0                        1764         627.2000           0.0000        1606.1440         803.0720

Adamax Old

Profile Statistics.
	Note that counter items are counter values and not time units.
Device Storage
=================
Name                          Total Count        Time (ms)    Min Time (ms)    Max Time (ms)    Avg Time (ms)
----                          -----------        ---------    -------------    -------------    -------------
Memory: cpu/0                        1728         627.2000           0.0000        2009.6000        1004.8000

Adamax New

Profile Statistics.
	Note that counter items are counter values and not time units.
Device Storage
=================
Name                          Total Count        Time (ms)    Min Time (ms)    Max Time (ms)    Avg Time (ms)
----                          -----------        ---------    -------------    -------------    -------------
Memory: cpu/0                        1656         627.2000           0.0000        1606.6560         803.3280

@anirudhacharya
Copy link
Copy Markdown
Member Author

anirudhacharya commented Jan 22, 2019

@mxnet-label-bot add [pr-awaiting-review]

@szha @eric-haibin-lin

@marcoabreu marcoabreu added the pr-awaiting-review PR is waiting for code review label Jan 22, 2019
@anirudhacharya
Copy link
Copy Markdown
Member Author

anirudhacharya commented Jan 25, 2019 •
edited
Loading

For 5 batches of training with a deep embeddings example the decrease in memory consumption is ~0.27 factor for Nesterov Adam optimizer.

Old Nadam -

old_nadam

New Nadam( with in-place operators) -

new_nadam

vandanavk
vandanavk approved these changes Feb 5, 2019
View reviewed changes
Copy link
Copy Markdown
Contributor

@vandanavk vandanavk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@vandanavk
Copy link
Copy Markdown
Contributor

vandanavk commented Feb 5, 2019

@mxnet-label-bot update [pr-awaiting-merge]

@marcoabreu marcoabreu added pr-awaiting-merge Review and CI is complete. Ready to Merge and removed pr-awaiting-review PR is waiting for code review labels Feb 5, 2019
@ankkhedia
Copy link
Copy Markdown
Contributor

ankkhedia commented Feb 14, 2019

@sandeep-krishnamurthy Could you please review/merge this PR?

@eric-haibin-lin eric-haibin-lin merged commit a4e249b into apache:master Feb 15, 2019
@anirudhacharya anirudhacharya deleted the opt_mem branch February 15, 2019 01:29
@anirudhacharya
Copy link
Copy Markdown
Member Author

anirudhacharya commented Feb 15, 2019

Thanks for merging and thanks to @szha for the tip here - #13683 (comment)

stephenrawls pushed a commit to stephenrawls/incubator-mxnet that referenced this pull request Feb 16, 2019
@anirudhacharya @stephenrawls
In-place updates for Nadam, Adadelta, Adamax and SGLD (apache#13960)
f51c8cf
drivanov pushed a commit to drivanov/incubator-mxnet that referenced this pull request Mar 4, 2019
@anirudhacharya @drivanov
In-place updates for Nadam, Adadelta, Adamax and SGLD (apache#13960)
f39148c
vdantu pushed a commit to vdantu/incubator-mxnet that referenced this pull request Mar 31, 2019
@anirudhacharya @vdantu
In-place updates for Nadam, Adadelta, Adamax and SGLD (apache#13960)
88af4fe
haohuanw pushed a commit to haohuanw/incubator-mxnet that referenced this pull request Jun 23, 2019
@anirudhacharya
In-place updates for Nadam, Adadelta, Adamax and SGLD (apache#13960)
7874b31
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Reviewers

2 more reviewers

@eric-haibin-lin eric-haibin-lin eric-haibin-lin approved these changes

@vandanavk vandanavk vandanavk approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

pr-awaiting-merge Review and CI is complete. Ready to Merge

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@anirudhacharya @vandanavk @ankkhedia @eric-haibin-lin @marcoabreu