Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Conversation

@driusan
Copy link
Collaborator

@driusan driusan commented Sep 12, 2024

This adds \LORIS\Database\Query class which is both an iterator an countable to replace the return value of Database->pselect(). This improves the memory consumption of large datasets, because the iterator only needs to load the row that is being foreach'd over, instead of loading the entire query result into memory.

Places that currently depend on it being an array are wrapped in iterator_to_array (They are mostly queries returning small results, so this isn't expensive. The optimization is mostly useful for places like the dataquery module which iterate over large amounts of data.)

@kongtiaowang kongtiaowang added the Passed manual tests PR has been successfully tested by at least one peer label Sep 12, 2024
@kongtiaowang
Copy link
Contributor

@driusan It passed the manual tests.

Copy link
Contributor

@kongtiaowang kongtiaowang left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@driusan driusan merged commit 25c0ef6 into aces:main Sep 12, 2024
maximemulder pushed a commit to maximemulder/Loris that referenced this pull request Sep 25, 2024
aces#9334)

This adds `\LORIS\Database\Query` class which is both an iterator an
countable to replace the return value of `Database->pselect()`. This
improves the memory consumption of large datasets, because the iterator
only needs to load the row that is being foreach'd over, instead of
loading the entire query result into memory.

Places that currently depend on it being an array are wrapped in
`iterator_to_array` (They are mostly queries returning small results, so
this isn't expensive. The optimization is mostly useful for places like
the dataquery module which iterate over large amounts of data.)
ZhichGaming pushed a commit to ZhichGaming/Loris that referenced this pull request Nov 25, 2024
aces#9334)

This adds `\LORIS\Database\Query` class which is both an iterator an
countable to replace the return value of `Database->pselect()`. This
improves the memory consumption of large datasets, because the iterator
only needs to load the row that is being foreach'd over, instead of
loading the entire query result into memory.

Places that currently depend on it being an array are wrapped in
`iterator_to_array` (They are mostly queries returning small results, so
this isn't expensive. The optimization is mostly useful for places like
the dataquery module which iterate over large amounts of data.)
driusan added a commit that referenced this pull request Feb 3, 2025
This updates DBRowProvisioner to remove the strval (and fixes #9335).

After doing that, it became clear that the inner class could be replaced
by a generator while still fulfilling the contract of returning a
`\Traversable`. Doing this made it more obvious that since #9334,
pselect could be used directly while maintaining the lazy evaluation
instead of bypassing the Database object to use the PDO directly.
However this required injecting a `LorisInstance` object to the
constructor to get the database connection.

With access to the `LorisInstance` object, the `DBRowProvisioner` can
also get a new database connection and disable query buffering to lazily
retrieve rows since PR #9344. This may theoretically improve performance
and memory usage on large provisioners but has not been tested.
(Performance did not decrease in raisinbread but it is not a large
dataset where it would be noticeable.)
driusan pushed a commit that referenced this pull request Apr 11, 2025
This wraps the $db->pselect() call with iterator_to_array so that it can be processed with .map in DiagnosisEvolution.js. This is necessary following the change in #9334

    Resolves #9711
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Passed manual tests PR has been successfully tested by at least one peer

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants