Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Added config parameter to stream with batch size #322

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 1 commit into from
Jan 3, 2022
Merged

Added config parameter to stream with batch size #322

merged 1 commit into from
Jan 3, 2022

Conversation

@bdbernardy
Copy link
Contributor

The pool.stream method is prohibitively slow when working with large amounts of data because the QueryStream is configured to fetch 100 rows at a time. Being able to control the number of rows fetched would allow us to find the right balance between execution speed and memory consumption.

This PR adds a config parameter to the pool.stream method which allows us to specify the batchSize used in the QueryStream.

@coveralls
Copy link

Coverage Status

Coverage increased (+0.1%) to 88.97% when pulling 3b54f62 on bdbernardy:add-stream-batch-size into c6ddd79 on gajus:master.

@gajus gajus merged commit 2eb206b into gajus:master Jan 3, 2022
@gajus
Copy link
Owner

gajus commented Jan 3, 2022

Thank you

@gajus
Copy link
Owner

gajus commented Jan 3, 2022

🎉 This PR is included in version 27.1.0 🎉

The release is available on:

Your semantic-release bot 📦🚀

@gajus gajus added the released label Jan 3, 2022
@bdbernardy
Copy link
Contributor Author

bdbernardy commented Jan 4, 2022 •
edited
Loading

Awesome! Thank you. I just tested it and it works wonders. Streaming 6 million rows now only takes 6 minutes instead of 2 hours :).

@bdbernardy bdbernardy deleted the add-stream-batch-size branch January 4, 2022 08:46
@gajus
Copy link
Owner

gajus commented Jan 4, 2022

yay, thank you for contributing

@matthew-white matthew-white mentioned this pull request Aug 4, 2023
Open
@raine
Copy link

raine commented Oct 12, 2023

Is batchSize supposed to control how many rows you get in the data event? Because I'm getting a single one even with batchSize: 1000. Or is it some internal thing?

@bdbernardy
Copy link
Contributor Author

Is batchSize supposed to control how many rows you get in the data event? Because I'm getting a single one even with batchSize: 1000. Or is it some internal thing?

It's the number of rows that you fetch from the database at a time. The higher the number, the faster it will be, but it will also increase the memory consumption.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@bdbernardy @coveralls @gajus @raine