Pre-process iCalendars line by line to avoid OOM on large files #92

ArnyminerZ · 2025-09-28T13:05:28Z

Context

Right now we are loading the whole iCalendar into memory when applying the preprocessors, which breaks the whole point to use Readers in the first place. This can lead to Out Of Memory exceptions on super large iCalendar documents, or in devices with limited memory.

More info and reproduction in #90

Changes

Added a new test (ICalPreprocessorInstrumentedTest) that generates a very large iCalendar file.
With Int.MAX_VALUE events.
Changed the way ICalPreprocessor.preprocessStream works:
- Instead of applying the StreamPreprocessors on the whole document at once, they are applied line-wise.
- If the Reader given to the ICalPreprocessor support reset, the result of this function will be a SequenceReader. This is a new class, that converts sequences into a Reader. Note that this function does not support reset, by the way sequences work in Kotlin.
- If the Reader doesn't support reset, it will have to be loaded fully into memory anyway.
Since StreamPreprocessor is now simpler (most logic has been moved into ICalPreprocessor), they are now interfaces.

Note

It is possible to run the pre-processors in a line-basis is because they are applied line-wise. If at some point we require a preprocessor that needs to fix multiple lines at once (maybe description fixes? which allow multi-line), we might have to re-do this.

Copilot

Pull Request Overview

This PR addresses Out Of Memory (OOM) issues when processing super large iCalendar files by implementing chunked processing instead of loading entire files into memory. The changes move from a Reader-based preprocessing approach to a line-by-line chunked processing system that processes iCalendar data in groups of 1000 lines to maintain memory efficiency while preserving functionality.

Key changes:

Refactored stream preprocessing to process data in configurable chunks rather than loading entire files
Converted StreamPreprocessor from abstract class to interface with simplified API
Added SequenceReader utility class to convert processed line sequences back to Reader interface

Reviewed Changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
SequenceReader.kt	New utility class that converts String sequences into Reader interface
StreamPreprocessor.kt	Simplified from abstract class to interface, removing preprocessing logic
ICalPreprocessor.kt	Refactored to implement chunked processing with configurable chunk sizes
FixInvalidUtcOffsetPreprocessor.kt	Updated to implement StreamPreprocessor interface
FixInvalidDayOffsetPreprocessor.kt	Updated to implement StreamPreprocessor interface
ICalPreprocessorInstrumentedTest.kt	Added test with large iCalendar file generator to verify memory efficiency

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

lib/src/main/kotlin/at/bitfire/synctools/utils/SequenceReader.kt

lib/src/main/kotlin/at/bitfire/synctools/icalendar/validation/ICalPreprocessor.kt

...oidTest/kotlin/at/bitfire/synctools/icalendar/validation/ICalPreprocessorInstrumentedTest.kt

ArnyminerZ · 2025-09-28T13:46:19Z

@bitfireAT/app-dev should be ready :)

A bit ugly, but I don't know how to simplify it to be honest. Mainly SequenceReader is a bit verbose.

sunkup

Pretty cool, but unfortunately not very necessary and might introduce new problems. The current focus for synctools is the refactoring for stability, which is already breaking things a bit ... So it's not really a good time to be adding this. I will do an actual review if @rfc2822 thinks we should add it to synctools now anyways.

rfc2822

Because the large file / OOM was a real problem, I think it's a good idea to make the preprocessors more stable, too.

Some comments.

lib/src/main/kotlin/at/bitfire/synctools/icalendar/validation/StreamPreprocessor.kt

lib/src/main/kotlin/at/bitfire/synctools/icalendar/validation/ICalPreprocessor.kt

lib/src/main/kotlin/at/bitfire/synctools/utils/SequenceReader.kt

Signed-off-by: Arnau Mora <[email protected]>

rfc2822

Some more comments :)

...oidTest/kotlin/at/bitfire/synctools/icalendar/validation/ICalPreprocessorInstrumentedTest.kt

...src/main/kotlin/at/bitfire/synctools/icalendar/validation/FixInvalidDayOffsetPreprocessor.kt

lib/src/main/kotlin/at/bitfire/synctools/icalendar/validation/StreamPreprocessor.kt

lib/src/main/kotlin/at/bitfire/synctools/icalendar/validation/ICalPreprocessor.kt

# Conflicts: # gradle/libs.versions.toml

rfc2822 · 2025-11-06T09:57:55Z

I tried it out with 100,000 events and processing the stream in chunks of 1000 lines was not faster than line by line (both ~ 10 sec). So I'll remove the chunking for the benefit of having an easy-to-understand pre-processer interface (processes only one line).

rfc2822

@ArnyminerZ I did some last modifications, please double-check and wait for @sunkup's review before merging (the PR is quite important because it will be applied to every single synced iCalendar).

Copilot

Pull Request Overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated 1 comment.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

lib/src/main/kotlin/at/bitfire/synctools/icalendar/validation/ICalPreprocessor.kt

ArnyminerZ · 2025-11-06T12:28:19Z

I did some last modifications

I've checked them, they look good. Let's wait for @sunkup's approval

sunkup

Looks a bit scary tbh, but I think you covered a good amount of test cases so lets hope it goes well :)

As far as I can see the repairs are applied line by line, so this note in the PR description is confusing. Might want to update the PR description.

Note: to avoid having to apply regex conditions (which can get expensive) hundred of thousands of times, the lines are chunked in groups of 1000 lines (arbitrary, can be adjusted).

ArnyminerZ added 2 commits September 28, 2025 14:57

Fix OOM issues

af8f8f6

Remove reset

9692929

ArnyminerZ self-assigned this Sep 28, 2025

ArnyminerZ added the refactoring Quality improvement of existing functions label Sep 28, 2025

ArnyminerZ requested a review from a team as a code owner September 28, 2025 13:05

ArnyminerZ linked an issue Sep 28, 2025 that may be closed by this pull request

OOM Exception with large icalendar files #90

Closed

ArnyminerZ requested a review from Copilot September 28, 2025 13:07

Copilot AI reviewed Sep 28, 2025

View reviewed changes

ArnyminerZ marked this pull request as draft September 28, 2025 13:08

ArnyminerZ added 4 commits September 28, 2025 15:11

Get rid of Android's Log

84e33f3

Fix function calling

fd65367

Ignore exception in try-catch

dad980b

Fix tests

6f52535

ArnyminerZ marked this pull request as ready for review September 28, 2025 13:46

sunkup reviewed Oct 1, 2025

View reviewed changes

rfc2822 reviewed Oct 24, 2025

View reviewed changes

ArnyminerZ added 12 commits October 25, 2025 09:39

Merge branch 'main' into 90-oom-exception-with-large-icalendar-files

beaf42b

Get rid of regexpForProblem in StreamPreprocessor

73380e7

Improve KDoc

0da87e2

Rename argument

4ef91b5

Move warning

bb47bc0

Closing readers

1b17033

Make it clear why we use LF

fca4cc1

Use Guava's CharSource and get rid of SequenceReader

20483d9

Optimize imports

6652463

Signed-off-by: Arnau Mora <[email protected]>

Disable inspection

3910eef

Signed-off-by: Arnau Mora <[email protected]>

Added Spotbugs annotations

a257c2c

Signed-off-by: Arnau Mora <[email protected]>

Add annotations

7948036

Signed-off-by: Arnau Mora <[email protected]>

rfc2822 reviewed Oct 25, 2025

View reviewed changes

rfc2822 reviewed Oct 29, 2025

View reviewed changes

lib/src/main/kotlin/at/bitfire/synctools/icalendar/validation/ICalPreprocessor.kt Outdated Show resolved Hide resolved

Merge branch 'main' into 90-oom-exception-with-large-icalendar-files

0bda829

# Conflicts: # gradle/libs.versions.toml

rfc2822 self-assigned this Nov 6, 2025

[WIP] Process line by line

43cbcc3

rfc2822 added 5 commits November 6, 2025 11:00

KDoc

1f7298a

Move ICalPreprocessorInstrumentedTest to unit tests

17ca072

KDoc

cc53c37

Add test

68187c4

Make some fields internal

08a1d6b

rfc2822 approved these changes Nov 6, 2025

View reviewed changes

rfc2822 requested review from Copilot and sunkup November 6, 2025 10:55

Copilot AI reviewed Nov 6, 2025

View reviewed changes

lib/src/main/kotlin/at/bitfire/synctools/icalendar/validation/ICalPreprocessor.kt Outdated Show resolved Hide resolved

rfc2822 changed the title ~~Fix OOM issues with super large iCalendar files~~ Pre-process iCalendars line by line to avoid OOM on large iCalendar files Nov 6, 2025

rfc2822 changed the title ~~Pre-process iCalendars line by line to avoid OOM on large iCalendar files~~ Pre-process iCalendars line by line to avoid OOM on large files Nov 6, 2025

KDoc

c1acb29

sunkup approved these changes Nov 10, 2025

View reviewed changes

rfc2822 merged commit 7af49dd into main Nov 12, 2025
7 checks passed

rfc2822 deleted the 90-oom-exception-with-large-icalendar-files branch November 12, 2025 09:29

Pre-process iCalendars line by line to avoid OOM on large files #92

Pre-process iCalendars line by line to avoid OOM on large files #92

Uh oh!

Conversation

ArnyminerZ commented Sep 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Context

Changes

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ArnyminerZ commented Sep 28, 2025

Uh oh!

sunkup left a comment

Choose a reason for hiding this comment

Uh oh!

rfc2822 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

rfc2822 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

rfc2822 commented Nov 6, 2025

Uh oh!

rfc2822 left a comment

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Uh oh!

Uh oh!

ArnyminerZ commented Nov 6, 2025

Uh oh!

sunkup left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

ArnyminerZ commented Sep 28, 2025 •

edited

Loading