Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Add streaming decode API #3

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
pragyanatvade merged 30 commits into from
Nov 28, 2025
Merged

Add streaming decode API #3

pragyanatvade merged 30 commits into from
Nov 28, 2025

Conversation

@pragyanatvade
Copy link
Contributor

@pragyanatvade pragyanatvade commented Nov 28, 2025

Adds event-based streaming decode for memory-efficient processing of large TOON documents.

What's new

  • events / events-ch - decode to lazy seq or async channel of parse events
  • events->value - reconstruct values from event streams
  • lines->value - decode from pre-split lines
  • encode-lines - streaming encode (returns seq of lines)

Other changes

  • Feature parity with TS: wasQuoted on keys, length on arrays
  • Strict mode validation for array counts
  • Refactored internal naming to follow Stuart Sierra conventions
  • Reduced CLJ/CLJS duplication in stream module

All 520 tests pass.

@pragyanatvade
feat(deps): add core.async dependency for streaming decode
cdafa89
@pragyanatvade
feat(decode): add event types for streaming decoder
d94af92
@pragyanatvade
feat(decode): implement event-based streaming decoder
5d7f209
@pragyanatvade
feat(decode): add streaming decode API with sync and async support
e19de93
@pragyanatvade
feat(api): expose streaming decode functions in public API
0915dda
@pragyanatvade
test(decode): add comprehensive tests for streaming decoder
3157d4c
@pragyanatvade
feat(decode): add value builder for reconstructing values from event …
ed51c92 
…streams
@pragyanatvade
feat(api): expose events, events-ch, and events->value in public API
474ad7e
@pragyanatvade
feat(events): add length parameter to start-array and was-quoted to k…
2877ab1 
…ey-event
@pragyanatvade
feat(parser): change key-token to return map with key and was-quoted …
1a3291b 
…info
@pragyanatvade
feat(event-builder): pass array length and key was-quoted to events
eecd568
@pragyanatvade
fix(objects): extract key from key-token map in decode path
de62acf
@pragyanatvade
test(parser): update key-token tests to expect map format
bb33759
@pragyanatvade
test(stream): add tests for array length and key was-quoted properties
6a86537
@pragyanatvade
feat(api): add lines->value function for decoding pre-split lines
87dfc19
@pragyanatvade
test(interface): add comprehensive tests for lines->value function
f7d2770
@pragyanatvade
feat(api): add encode-lines function for streaming line output
fe738f3
@pragyanatvade
test(encode): add comprehensive tests for encode-lines function
2b099f3
@pragyanatvade
fix(decode): support list arrays with nested objects in streaming
2c3cad4 
List items with inline fields and nested children now correctly emit
complete objects combining both. Cursor tracking fixed to properly
advance through child fields before processing next list item.
@pragyanatvade
test(decode): add streaming equivalence tests for all structure types
8045fef 
Add 8 comprehensive tests ensuring streaming decode produces identical
results to regular decode for: simple objects, nested objects, mixed
structures, list arrays, and all primitive types (number, string,
boolean, null)
@pragyanatvade
fix(events): rename key to event-key to avoid shadowing clojure.core/key
21bcac1
@pragyanatvade
refactor(events): simplify event API, make was-quoted optional
91b1981
@pragyanatvade
fix(decode): add strict mode validation for array counts in streaming
662abb7
@pragyanatvade
refactor(stream): reduce CLJ/CLJS duplication with shared helpers
0a69fe2
@pragyanatvade
feat(scanner): add next-cursor and get-blank-lines-in-range functions
823165f
@pragyanatvade
test(stream): consolidate tests, add was-quoted and length assertions
98a7aa9
@pragyanatvade
refactor(event-builder): use noun naming, extract helpers, simplify s…
afbd9a6 
…tructure
@pragyanatvade
refactor(stream): rename to cursor->events and source-lines per namin…
a78ad54 
…g conventions
@pragyanatvade
refactor(items): rename detect-list-item-type to list-item-type
235521f
@pragyanatvade
style: apply linter formatting fixes
4f6f54c
@pragyanatvade pragyanatvade merged commit 8127248 into main Nov 28, 2025
2 checks passed
@pragyanatvade pragyanatvade deleted the feat/streaming-decode-api branch December 8, 2025 14:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@pragyanatvade