Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Implement lazy geom col #458

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 6 commits into from
Apr 27, 2025
Merged

Implement lazy geom col #458

merged 6 commits into from
Apr 27, 2025

Conversation

anitagraser
Copy link
Collaborator

@anitagraser anitagraser commented Apr 15, 2025 •
edited
Loading

Init only

Points (Million) Original Init Time (s) Lazy Init Time (s)
0.1 0.888077 0.041898
1.0 7.927111 0.172050
5.0 36.693478 0.755375
10.0 66.308508 1.577410

Init + ValueChangeSplitter

Points (Million) Original Init + ValueChangeSplitter (s) Lazy Init + ValueChangeSplitter (s)
0.01 2.118076 0.639527
0.1 15.885158 6.281623
0.5 78.102128 30.691467

@anitagraser
Copy link
Collaborator Author

anitagraser commented Apr 16, 2025 •
edited
Loading

On Laptop (columns in million points):

Traj init Splitter 10k 100k 500k
Lazy Init None 0.025283 0.057060 0.143990
Original Init None 0.122563 1.038553 4.748427
Lazy Init No min_length 0.852815 8.193364 44.912145
Lazy Init min_length=1000 1.800314 17.491231 86.674723
Lazy Init min_length=1000, early skip 0.886468 8.292531 41.281583
Lazy Init min_length=100000 1.744545 16.783528 90.314423
Lazy Init min_length=100000, early skip 1.790124 9.830566 45.494739
Lazy Init min_length=10000000 1.041278 11.236167 51.147402
Lazy Init min_length=10000000, early skip 1.486809 14.290635 71.029866
Original Init No min_length 1.859516 18.173769 97.446402
Original Init min_length=1000 2.729446 18.731986 95.853996
Original Init min_length=100000 2.066168 18.096591 99.708181
Original Init min_length=10000000 0.942515 8.959574 44.724029

In the worst-case scenario, when the original implementation discards all trajectories as too short right from the start, the new approach is worse. But otherwise there's quite some improvement:

image

@anitagraser anitagraser merged commit 8184855 into main Apr 27, 2025
18 checks passed
@anitagraser
Copy link
Collaborator Author 

### Original Trajectory init

--- 0.01 million points ---
TrajectoryCollection with 4 trajectories
Trajectory init:  0.122563s

--- 0.1 million points ---
TrajectoryCollection with 4 trajectories
Trajectory init:  1.038553s

--- 0.5 million points ---
TrajectoryCollection with 4 trajectories
Trajectory init:  4.748427s

--- 1.0 million points ---
TrajectoryCollection with 4 trajectories
Trajectory init:  9.569846s

--- 5.0 million points ---
TrajectoryCollection with 4 trajectories
Trajectory init:  45.446195s

--- 10.0 million points ---
TrajectoryCollection with 4 trajectories
Trajectory init:  87.278555s


### Original init + Value Change Splitter

#### Without min_length

--- 0.01 million points ---
TrajectoryCollection with 4 trajectories
Trajectory init + ValueChangeSplitter:  1.859516s

--- 0.1 million points ---
TrajectoryCollection with 4 trajectories
Trajectory init + ValueChangeSplitter:  18.173769s

--- 0.5 million points ---
TrajectoryCollection with 4 trajectories
Trajectory init + ValueChangeSplitter:  97.446402s

#### with min_length 1000 

--- 0.01 million points ---
TrajectoryCollection with 4 trajectories
Trajectory init + ValueChangeSplitter:  2.729446s

--- 0.1 million points ---
TrajectoryCollection with 4 trajectories
Trajectory init + ValueChangeSplitter:  18.731986s

--- 0.5 million points ---
TrajectoryCollection with 4 trajectories
Trajectory init + ValueChangeSplitter:  95.853996s

#### with min_length 100_000 

--- 0.01 million points ---
TrajectoryCollection with 4 trajectories
Trajectory init + ValueChangeSplitter:  2.066168s

--- 0.1 million points ---
TrajectoryCollection with 4 trajectories
Trajectory init + ValueChangeSplitter:  18.096591s

--- 0.5 million points ---
TrajectoryCollection with 4 trajectories
Trajectory init + ValueChangeSplitter:  99.708181s

#### with min_length 10_000_000 

--- 0.01 million points ---
TrajectoryCollection with 0 trajectories
Trajectory init + ValueChangeSplitter:  0.942515s

--- 0.1 million points ---
TrajectoryCollection with 0 trajectories
Trajectory init + ValueChangeSplitter:  8.959574s

--- 0.5 million points ---
TrajectoryCollection with 0 trajectories
Trajectory init + ValueChangeSplitter:  44.724029s


### Lazy Trajectory init 

--- 0.01 million points ---
TrajectoryCollection with 4 trajectories
Trajectory init:  0.025283s

--- 0.1 million points ---
TrajectoryCollection with 4 trajectories
Trajectory init:  0.057060s

--- 0.5 million points ---
TrajectoryCollection with 4 trajectories
Trajectory init:  0.143990s

--- 1.0 million points ---
TrajectoryCollection with 4 trajectories
Trajectory init:  0.286819s

--- 5.0 million points ---
TrajectoryCollection with 4 trajectories
Trajectory init:  1.463101s

--- 10.0 million points ---
TrajectoryCollection with 4 trajectories
Trajectory init:  2.458700s



### Lazy init + Value Change Splitter
#### Without min_length

--- 0.01 million points ---
TrajectoryCollection with 4 trajectories
Trajectory init + ValueChangeSplitter:  0.852815s

--- 0.1 million points ---
TrajectoryCollection with 4 trajectories
Trajectory init + ValueChangeSplitter:  8.193364s

--- 0.5 million points ---
TrajectoryCollection with 4 trajectories
Trajectory init + ValueChangeSplitter:  44.912145s


#### with min_length 1000 

--- 0.01 million points ---
TrajectoryCollection with 4 trajectories
Trajectory init + ValueChangeSplitter:  1.800314s

--- 0.1 million points ---
TrajectoryCollection with 4 trajectories
Trajectory init + ValueChangeSplitter:  17.491231s

--- 0.5 million points ---
TrajectoryCollection with 4 trajectories
Trajectory init + ValueChangeSplitter:  86.674723s


#### with min_length 1000 and early skipping

--- 0.01 million points ---
TrajectoryCollection with 4 trajectories
Trajectory init + ValueChangeSplitter:  0.886468s

--- 0.1 million points ---
TrajectoryCollection with 4 trajectories
Trajectory init + ValueChangeSplitter:  8.292531s

--- 0.5 million points ---
TrajectoryCollection with 4 trajectories
Trajectory init + ValueChangeSplitter:  41.281583s

#### with min_length 100_000 

--- 0.01 million points ---
TrajectoryCollection with 4 trajectories
Trajectory init + ValueChangeSplitter:  1.744545s

--- 0.1 million points ---
TrajectoryCollection with 4 trajectories
Trajectory init + ValueChangeSplitter:  16.783528s

--- 0.5 million points ---
TrajectoryCollection with 4 trajectories
Trajectory init + ValueChangeSplitter:  90.314423s

#### with min_length 100000 and early skipping

--- 0.01 million points ---
TrajectoryCollection with 4 trajectories
Trajectory init + ValueChangeSplitter:  1.790124s

--- 0.1 million points ---
TrajectoryCollection with 4 trajectories
Trajectory init + ValueChangeSplitter:  9.830566s

--- 0.5 million points ---
TrajectoryCollection with 4 trajectories
Trajectory init + ValueChangeSplitter:  45.494739s

#### with min_length 10_000_000 (worst-case scenario: min_length longer than trajectories)

--- 0.01 million points ---
TrajectoryCollection with 0 trajectories
Trajectory init + ValueChangeSplitter:  1.041278s

--- 0.1 million points ---
TrajectoryCollection with 0 trajectories
Trajectory init + ValueChangeSplitter:  11.236167s

--- 0.5 million points ---
TrajectoryCollection with 0 trajectories
Trajectory init + ValueChangeSplitter:  51.147402s

#### with min_length 10_000_000 and early skipping (worst-case scenario: min_length longer than trajectories)

--- 0.01 million points ---
TrajectoryCollection with 0 trajectories
Trajectory init + ValueChangeSplitter:  1.486809s

--- 0.1 million points ---
TrajectoryCollection with 0 trajectories
Trajectory init + ValueChangeSplitter:  14.290635s

--- 0.5 million points ---
TrajectoryCollection with 0 trajectories
Trajectory init + ValueChangeSplitter:  71.029866s

@anitagraser anitagraser mentioned this pull request May 7, 2025
Merged
@anitagraser anitagraser deleted the lazy branch May 11, 2025 17:24
@minrk minrk mentioned this pull request May 15, 2025
Merged
@anitagraser anitagraser mentioned this pull request Jun 11, 2025
Closed
3 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@anitagraser