Thanks to visit codestin.com
Credit goes to github.com

Skip to content

enable custom activation functions#65

Merged
tgale96 merged 3 commits into
databricks:mainfrom
vchiley:custom_act_fn
Dec 19, 2023
Merged

enable custom activation functions#65
tgale96 merged 3 commits into
databricks:mainfrom
vchiley:custom_act_fn

Conversation

@vchiley

@vchiley vchiley commented Dec 19, 2023

Copy link
Copy Markdown
Contributor

No description provided.

tgale96
tgale96 reviewed Dec 19, 2023
View reviewed changes

@tgale96 tgale96 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks awesome!

Comment thread megablocks/layers/glu.py Outdated
Comment thread megablocks/layers/glu.py
Comment thread megablocks/layers/mlp.py Outdated
Comment thread megablocks/layers/glu.py Outdated
Comment thread megablocks/layers/mlp.py Outdated
Comment thread megablocks/layers/mlp.py Outdated
Comment thread megablocks/layers/mlp.py Outdated
Comment thread megablocks/layers/mlp.py Outdated
Comment thread megablocks/layers/mlp.py Outdated
Comment thread megablocks/layers/arguments.py
@tgale96

tgale96 commented Dec 19, 2023

Copy link
Copy Markdown
Contributor

LGTM! Tests passing? Ok to merge?

@vchiley vchiley marked this pull request as ready for review December 19, 2023 18:19
@vchiley vchiley force-pushed the custom_act_fn branch 2 times, most recently from 825671d to ff9c2de Compare December 19, 2023 18:30
@vchiley

vchiley commented Dec 19, 2023

Copy link
Copy Markdown
Contributor Author

LGTM! Tests passing? Ok to merge?

the tests you have seem to pass
It runs in llm-foundry for configs

  1. mlp_type: glu + grouped_mlp: true
  2. mlp_type: glu + grouped_mlp: false
  3. mlp_type: mlp + grouped_mlp: false + memory_optimized_mlp: false
  4. mlp_type: mlp + grouped_mlp: true + memory_optimized_mlp: false
  5. mlp_type: mlp + grouped_mlp: false + memory_optimized_mlp: true
  6. mlp_type: mlp + grouped_mlp: true + memory_optimized_mlp: true

@vchiley vchiley requested a review from tgale96 December 19, 2023 18:33
@vchiley

vchiley commented Dec 19, 2023

Copy link
Copy Markdown
Contributor Author

The only functionality being used from megablocks.layers import gelu is gelu.gelu_backward_
@tgale96 how much do you think that buy us?

@tgale96

tgale96 commented Dec 19, 2023

Copy link
Copy Markdown
Contributor

The only functionality being used from megablocks.layers import gelu is gelu.gelu_backward_ @tgale96 how much do you think that buy us?

Probably not much if anything. I think this was a holdover from when the approx gelu wasn't supported in Torch, which I then reused for gelu remat.

@tgale96 tgale96 merged commit 44d9743 into databricks:main Dec 19, 2023
@vchiley vchiley deleted the custom_act_fn branch December 19, 2023 20:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@tgale96 tgale96 Awaiting requested review from tgale96

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@vchiley @tgale96