Thanks to visit codestin.com
Credit goes to github.com

Skip to content

updated applyChatTemplate with lazy memoization #218

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 4 commits into from
Sep 2, 2025
Merged

updated applyChatTemplate with lazy memoization #218

merged 4 commits into from
Sep 2, 2025

Conversation

tpae
Copy link
Contributor

@tpae tpae commented Sep 1, 2025

Based on #217.

Cache compiled Jinja Template instances in PreTrainedTokenizer keyed by the template string; reuse in applyChatTemplate. No API or behavior changes.

Measured performance: Repeated calls with the same template are ~6x faster.

Cached (reused template): ~0.00049 s (0.49 ms) average
Uncached (unique template each call): ~0.00292 s (2.92 ms) average
Speedup: ~5.9–6.2x

Raw results:

Test Case '-[TokenizersTests.ChatTemplateTests testApplyChatTemplatePerformanceCached]' started.
swift-transformers/Tests/TokenizersTests/ChatTemplateTests.swift:276: Test Case '-[TokenizersTests.ChatTemplateTests testApplyChatTemplatePerformanceCached]' measured [Clock Monotonic Time, s] average: 0.000, relative standard deviation: 7.781%, values: [0.000339, 0.000385, 0.000339, 0.000385, 0.000316], performanceMetricID:com.apple.dt.XCTMetric_Clock.time.monotonic, baselineName: "", baselineAverage: , polarity: prefers smaller, maxPercentRegression: 10.000%, maxPercentRelativeStandardDeviation: 10.000%, maxRegression: 0.000, maxStandardDeviation: 0.000
Test Case '-[TokenizersTests.ChatTemplateTests testApplyChatTemplatePerformanceCached]' passed (0.986 seconds).
Test Case '-[TokenizersTests.ChatTemplateTests testApplyChatTemplatePerformanceUncached]' started.
swift-transformers/Tests/TokenizersTests/ChatTemplateTests.swift:287: Test Case '-[TokenizersTests.ChatTemplateTests testApplyChatTemplatePerformanceUncached]' measured [Clock Monotonic Time, s] average: 0.003, relative standard deviation: 2.207%, values: [0.002790, 0.002796, 0.002700, 0.002880, 0.002734], performanceMetricID:com.apple.dt.XCTMetric_Clock.time.monotonic, baselineName: "", baselineAverage: , polarity: prefers smaller, maxPercentRegression: 10.000%, maxPercentRelativeStandardDeviation: 10.000%, maxRegression: 0.000, maxStandardDeviation: 0.000
Test Case '-[TokenizersTests.ChatTemplateTests testApplyChatTemplatePerformanceUncached]' passed (0.932 seconds).

@FL33TW00D
Copy link
Collaborator

@tpae great contribution thank you.

If you could format it with the pinned version of swiftformat we can merge!

brew install [email protected]
swiftformat --config .swiftformat .

pcuenca
pcuenca approved these changes Sep 1, 2025
View reviewed changes
Copy link
Member

@pcuenca pcuenca left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks very nice, thank you @tpae! 🙌

@tpae
Copy link
Contributor Author

tpae commented Sep 1, 2025

Amazing! Thank you!

My brew couldn't find [email protected], but was able to find 0.57.2, and I ran the formatting.

@tpae tpae requested a review from pcuenca September 1, 2025 17:08
@tpae
Copy link
Contributor Author

tpae commented Sep 1, 2025

Test was passing for me locally 🤔 i think 429 is rate limiting

@pcuenca
Copy link
Member

pcuenca commented Sep 2, 2025

Tests pass for me locally as well, and the CI worked for this PR too. So I think the token may not be passed to the CI from forks. I'll take a quick look, otherwise I'll merge and resolve later.

@pcuenca
Copy link
Member

pcuenca commented Sep 2, 2025

I'm merging. Also opened #222 which could be interesting for actual use cases, not just the CI.

@pcuenca
Copy link
Member

pcuenca commented Sep 2, 2025

Thanks a lot @tpae!

@pcuenca pcuenca merged commit ed54f3c into huggingface:main Sep 2, 2025
1 of 3 checks passed
@FL33TW00D FL33TW00D mentioned this pull request Sep 2, 2025
Closed
This was referenced Sep 2, 2025
Closed
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@pcuenca pcuenca Awaiting requested review from pcuenca

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@tpae @FL33TW00D @pcuenca