-
Notifications
You must be signed in to change notification settings - Fork 146
Comparing changes
Open a pull request
base repository: man-group/ArcticDB
base: v6.2.1
head repository: man-group/ArcticDB
compare: master
- 16 commits
- 753 files changed
- 3 contributors
Commits on Sep 8, 2025
-
Configuration menu - View commit details
-
Copy full SHA for 3e71b32 - Browse repository at this point
Copy the full SHA 3e71b32View commit details
Commits on Sep 10, 2025
-
Upgrade node version in CI to 24 (#2636)
#### Reference Issues/PRs <!--Example: Fixes #1234. See also #3456.--> #### What does this implement or fix? - Azure CI problems as seen here - https://github.com/man-group/ArcticDB/actions/runs/17551831701/job/49846790143 -- This is fixed by upgrading the Node version and fixing how ports are provisioned for the Azurite sum - Skips the problematic Linux Conda test for now - https://github.com/man-group/ArcticDB/actions/runs/17551831697/job/49845948723 #### Any other comments? #### Checklist <details> <summary> Checklist for code changes... </summary> - [ ] Have you updated the relevant docstrings, documentation and copyright notice? - [ ] Is this contribution tested against [all ArcticDB's features](../docs/mkdocs/docs/technical/contributing.md)? - [ ] Do all exceptions introduced raise appropriate [error messages](https://docs.arcticdb.io/error_messages/)? - [ ] Are API changes highlighted in the PR description? - [ ] Is the PR labelled as enhancement or bug so it appears in autogenerated release notes? </details> <!-- Thanks for contributing a Pull Request to ArcticDB! Please ensure you have taken a look at: - ArcticDB's Code of Conduct: https://github.com/man-group/ArcticDB/blob/master/CODE_OF_CONDUCT.md - ArcticDB's Contribution Licensing: https://github.com/man-group/ArcticDB/blob/master/docs/mkdocs/docs/technical/contributing.md#contribution-licensing -->
Configuration menu - View commit details
-
Copy full SHA for 68603dc - Browse repository at this point
Copy the full SHA 68603dcView commit details
Commits on Sep 15, 2025
-
[9898131742] Fix arrow projection with dynamic schema (#2630)
#### Reference Issues/PRs Monday ref: 9898131742 #### What does this implement or fix? This PR modifies `NullReducer` code to not rely on the slice index and by preserving a `column_block_offset_` state avoids an unneeded `log(n)` search for the offset. #### Any other comments? `NullReducer` code was assuming that `len(slice_and_keys) = len(row_slices_per_column)` when using `dynamic_schema=True`. That is not true if we use projections. E.g. for the following projection our slicing would look like: ``` Given: TD key 1: index A 1 1 2 2 TD key 2: index A B 3 3 1 4 4 2 TD key 3: index B 5 3 6 4 And we do a projection like `q.apply("C", q["A"] + q["B"])` our slicing would look like: Slice 1: TD key 1 Slice 2: TD key 2 Slice 3: index C 3 4 4 6 Slice 4: TD key 3 ``` #### Checklist <details> <summary> Checklist for code changes... </summary> - [ ] Have you updated the relevant docstrings, documentation and copyright notice? - [ ] Is this contribution tested against [all ArcticDB's features](../docs/mkdocs/docs/technical/contributing.md)? - [ ] Do all exceptions introduced raise appropriate [error messages](https://docs.arcticdb.io/error_messages/)? - [ ] Are API changes highlighted in the PR description? - [ ] Is the PR labelled as enhancement or bug so it appears in autogenerated release notes? </details> <!-- Thanks for contributing a Pull Request to ArcticDB! Please ensure you have taken a look at: - ArcticDB's Code of Conduct: https://github.com/man-group/ArcticDB/blob/master/CODE_OF_CONDUCT.md - ArcticDB's Contribution Licensing: https://github.com/man-group/ArcticDB/blob/master/docs/mkdocs/docs/technical/contributing.md#contribution-licensing -->
Configuration menu - View commit details
-
Copy full SHA for 1b13f92 - Browse repository at this point
Copy the full SHA 1b13f92View commit details -
Only one version when running storage tests (#2643)
#### Reference Issues/PRs <!--Example: Fixes #1234. See also #3456.--> #### What does this implement or fix? Currently for storage tests we run all OS-Python combinations, although storage tests are executed only with Python 3.11 on Win and Linux. That creates a mix of results where majority 3.8-3.10, 3.12, 3.13 are LMDB tests and only 3.11 are real storage tests. That creates lots of confusion. The ideal solution would be to limit to 3.11 the runs so that only results from real storage tests are in the runs. For example see this link of a run of real_tetsts: https://github.com/man-group/ArcticDB/actions/runs/17659474240. It has so many LMDB runs and only 2 are Real Storages. Can you guess which ones? A run for GCPXML: https://github.com/man-group/ArcticDB/actions/runs/17643448437 Note: Linux - 3.8 still exists, and that is because it uses matrix-include which cannot be created with a condition. Still all others are gone from the run #### Any other comments? #### Checklist <details> <summary> Checklist for code changes... </summary> - [ ] Have you updated the relevant docstrings, documentation and copyright notice? - [ ] Is this contribution tested against [all ArcticDB's features](../docs/mkdocs/docs/technical/contributing.md)? - [ ] Do all exceptions introduced raise appropriate [error messages](https://docs.arcticdb.io/error_messages/)? - [ ] Are API changes highlighted in the PR description? - [ ] Is the PR labelled as enhancement or bug so it appears in autogenerated release notes? </details> <!-- Thanks for contributing a Pull Request to ArcticDB! Please ensure you have taken a look at: - ArcticDB's Code of Conduct: https://github.com/man-group/ArcticDB/blob/master/CODE_OF_CONDUCT.md - ArcticDB's Contribution Licensing: https://github.com/man-group/ArcticDB/blob/master/docs/mkdocs/docs/technical/contributing.md#contribution-licensing -->
Configuration menu - View commit details
-
Copy full SHA for a10ecd7 - Browse repository at this point
Copy the full SHA a10ecd7View commit details -
[9898177828] Use
default_value
inNullReducer
for `OutputFormat::……ARROW` (#2633) #### Reference Issues/PRs Monday ref: 9898177828 #### What does this implement or fix? When doing aggregation we explicitly default `sum=0` for slices with no underlying values. For arrow this means to not set the validity bitmap in this case and to default initialize the values. The change includes: - Small refactor of `NullReducer` to extract common parts between `reduce` and `finalize` in `backfill_up_to_frame_offset` - Modification of `Column::default_initialize` to work across several blocks - Removes broken `memset` method from `ChunkedBuffer` and instead provides a new `util::initialize` method which can initialize a `ChunkedBuffer` across blocks #### Any other comments? #### Checklist <details> <summary> Checklist for code changes... </summary> - [ ] Have you updated the relevant docstrings, documentation and copyright notice? - [ ] Is this contribution tested against [all ArcticDB's features](../docs/mkdocs/docs/technical/contributing.md)? - [ ] Do all exceptions introduced raise appropriate [error messages](https://docs.arcticdb.io/error_messages/)? - [ ] Are API changes highlighted in the PR description? - [ ] Is the PR labelled as enhancement or bug so it appears in autogenerated release notes? </details> <!-- Thanks for contributing a Pull Request to ArcticDB! Please ensure you have taken a look at: - ArcticDB's Code of Conduct: https://github.com/man-group/ArcticDB/blob/master/CODE_OF_CONDUCT.md - ArcticDB's Contribution Licensing: https://github.com/man-group/ArcticDB/blob/master/docs/mkdocs/docs/technical/contributing.md#contribution-licensing -->
Configuration menu - View commit details
-
Copy full SHA for ffe58bc - Browse repository at this point
Copy the full SHA ffe58bcView commit details
Commits on Sep 17, 2025
-
Apply formatting rules (#2649)
#### Reference Issues/PRs Monday ref: 10048929527 #### What does this implement or fix? #### Any other comments? #### Checklist <details> <summary> Checklist for code changes... </summary> - [ ] Have you updated the relevant docstrings, documentation and copyright notice? - [ ] Is this contribution tested against [all ArcticDB's features](../docs/mkdocs/docs/technical/contributing.md)? - [ ] Do all exceptions introduced raise appropriate [error messages](https://docs.arcticdb.io/error_messages/)? - [ ] Are API changes highlighted in the PR description? - [ ] Is the PR labelled as enhancement or bug so it appears in autogenerated release notes? </details> <!-- Thanks for contributing a Pull Request to ArcticDB! Please ensure you have taken a look at: - ArcticDB's Code of Conduct: https://github.com/man-group/ArcticDB/blob/master/CODE_OF_CONDUCT.md - ArcticDB's Contribution Licensing: https://github.com/man-group/ArcticDB/blob/master/docs/mkdocs/docs/technical/contributing.md#contribution-licensing -->
Configuration menu - View commit details
-
Copy full SHA for 7acd347 - Browse repository at this point
Copy the full SHA 7acd347View commit details -
[10026766759] Correct sparse handling for Aggregation clauses (#2644)
#### Reference Issues/PRs Monday ref: 10026766759 #### What does this implement or fix? - Makes Aggregation clauses like `Mean` and `Count` respect input column sparsity - Fixes `CopyToBufferTask` to respect sparsity for arrow - Adds a similar test for resampling - Adds an xfail test for monday issue: 10029194063 #### Any other comments? Commits can be reviewed individually #### Checklist <details> <summary> Checklist for code changes... </summary> - [ ] Have you updated the relevant docstrings, documentation and copyright notice? - [ ] Is this contribution tested against [all ArcticDB's features](../docs/mkdocs/docs/technical/contributing.md)? - [ ] Do all exceptions introduced raise appropriate [error messages](https://docs.arcticdb.io/error_messages/)? - [ ] Are API changes highlighted in the PR description? - [ ] Is the PR labelled as enhancement or bug so it appears in autogenerated release notes? </details> <!-- Thanks for contributing a Pull Request to ArcticDB! Please ensure you have taken a look at: - ArcticDB's Code of Conduct: https://github.com/man-group/ArcticDB/blob/master/CODE_OF_CONDUCT.md - ArcticDB's Contribution Licensing: https://github.com/man-group/ArcticDB/blob/master/docs/mkdocs/docs/technical/contributing.md#contribution-licensing -->
Configuration menu - View commit details
-
Copy full SHA for 803c91e - Browse repository at this point
Copy the full SHA 803c91eView commit details -
Fix git blame after reformatting (#2652)
#### Reference Issues/PRs <!--Example: Fixes #1234. See also #3456.--> #### What does this implement or fix? #### Any other comments? #### Checklist <details> <summary> Checklist for code changes... </summary> - [ ] Have you updated the relevant docstrings, documentation and copyright notice? - [ ] Is this contribution tested against [all ArcticDB's features](../docs/mkdocs/docs/technical/contributing.md)? - [ ] Do all exceptions introduced raise appropriate [error messages](https://docs.arcticdb.io/error_messages/)? - [ ] Are API changes highlighted in the PR description? - [ ] Is the PR labelled as enhancement or bug so it appears in autogenerated release notes? </details> <!-- Thanks for contributing a Pull Request to ArcticDB! Please ensure you have taken a look at: - ArcticDB's Code of Conduct: https://github.com/man-group/ArcticDB/blob/master/CODE_OF_CONDUCT.md - ArcticDB's Contribution Licensing: https://github.com/man-group/ArcticDB/blob/master/docs/mkdocs/docs/technical/contributing.md#contribution-licensing -->
Configuration menu - View commit details
-
Copy full SHA for 1ab46e7 - Browse repository at this point
Copy the full SHA 1ab46e7View commit details -
Change version filter to 5.3.0 for tags (#2654)
Updated version filtering to require tags of version 5.3.0 or higher due to deprecated Numpy issues. #### Reference Issues/PRs <!--Example: Fixes #1234. See also #3456.--> #### What does this implement or fix? #### Any other comments? #### Checklist <details> <summary> Checklist for code changes... </summary> - [ ] Have you updated the relevant docstrings, documentation and copyright notice? - [ ] Is this contribution tested against [all ArcticDB's features](../docs/mkdocs/docs/technical/contributing.md)? - [ ] Do all exceptions introduced raise appropriate [error messages](https://docs.arcticdb.io/error_messages/)? - [ ] Are API changes highlighted in the PR description? - [ ] Is the PR labelled as enhancement or bug so it appears in autogenerated release notes? </details> <!-- Thanks for contributing a Pull Request to ArcticDB! Please ensure you have taken a look at: - ArcticDB's Code of Conduct: https://github.com/man-group/ArcticDB/blob/master/CODE_OF_CONDUCT.md - ArcticDB's Contribution Licensing: https://github.com/man-group/ArcticDB/blob/master/docs/mkdocs/docs/technical/contributing.md#contribution-licensing -->
Configuration menu - View commit details
-
Copy full SHA for 90505d1 - Browse repository at this point
Copy the full SHA 90505d1View commit details -
Fix range intersection calculation in query.cpp (#2632)
#### Reference Issues/PRs Monday refs: - 9943908001 - 10031061308 #### What does this implement or fix? - Fixes reading of indexes with old format (Monday ref: 9943908001) - Main fix: cpp/arcticdb/pipeline/query.cpp - tests: cpp/arcticdb/pipeline/test/test_query.cpp and python/tests/integration/arcticdb/version_store/test_num_storage_operations.py - Fixes a bug where delete_range was not updating the index correctly (Monday ref: 10031061308) - Main fix: cpp/arcticdb/version/version_core.cpp - Tests: https://github.com/man-group/ArcticDB/pull/2632/files#diff-7a61fb14e9865723e613017a346654e7c2694ffdb5eb13617e1665d1ad1ccba5R784 - Also tries to fix timeouts in macos builds #### Any other comments? #### Checklist <details> <summary> Checklist for code changes... </summary> - [ ] Have you updated the relevant docstrings, documentation and copyright notice? - [ ] Is this contribution tested against [all ArcticDB's features](../docs/mkdocs/docs/technical/contributing.md)? - [ ] Do all exceptions introduced raise appropriate [error messages](https://docs.arcticdb.io/error_messages/)? - [ ] Are API changes highlighted in the PR description? - [ ] Is the PR labelled as enhancement or bug so it appears in autogenerated release notes? </details> <!-- Thanks for contributing a Pull Request to ArcticDB! Please ensure you have taken a look at: - ArcticDB's Code of Conduct: https://github.com/man-group/ArcticDB/blob/master/CODE_OF_CONDUCT.md - ArcticDB's Contribution Licensing: https://github.com/man-group/ArcticDB/blob/master/docs/mkdocs/docs/technical/contributing.md#contribution-licensing -->
Configuration menu - View commit details
-
Copy full SHA for 59551d3 - Browse repository at this point
Copy the full SHA 59551d3View commit details
Commits on Sep 18, 2025
-
Use s3 for sccache in ASV related builds (#2657)
#### Reference Issues/PRs Monday ticket ref: 10079294063 #### What does this implement or fix? Changed the ASV builds to use S3 for sccache as is done in the regular builds. Also fixes the [VCPKG caching in one of the builds](https://github.com/man-group/ArcticDB/pull/2657/files#diff-8f568f466457a96303cbb4eed3d01446c3d3e5f8e522f58ffa4e78a13f04d64eR164). Tested manually in [this build](https://github.com/man-group/ArcticDB/actions/runs/17821717217/job/50665565352). #### Any other comments? #### Checklist <details> <summary> Checklist for code changes... </summary> - [ ] Have you updated the relevant docstrings, documentation and copyright notice? - [ ] Is this contribution tested against [all ArcticDB's features](../docs/mkdocs/docs/technical/contributing.md)? - [ ] Do all exceptions introduced raise appropriate [error messages](https://docs.arcticdb.io/error_messages/)? - [ ] Are API changes highlighted in the PR description? - [ ] Is the PR labelled as enhancement or bug so it appears in autogenerated release notes? </details> <!-- Thanks for contributing a Pull Request to ArcticDB! Please ensure you have taken a look at: - ArcticDB's Code of Conduct: https://github.com/man-group/ArcticDB/blob/master/CODE_OF_CONDUCT.md - ArcticDB's Contribution Licensing: https://github.com/man-group/ArcticDB/blob/master/docs/mkdocs/docs/technical/contributing.md#contribution-licensing -->
Configuration menu - View commit details
-
Copy full SHA for d78b668 - Browse repository at this point
Copy the full SHA d78b668View commit details -
v1 API tests for add_to_snapshot and remove_from_snapshot + other (#2651
) #### Reference Issues/PRs <!--Example: Fixes #1234. See also #3456.--> #### What does this implement or fix? Documentation is changed to reflect actual behavior Added new test for add_to_snapshot and remove_from_snapshot methods to checks behavior when we supply non existing symbol or version. Additional simple test for listing and removing incompletes also added as all coverage was for v2 #### Any other comments? #### Checklist <details> <summary> Checklist for code changes... </summary> - [ ] Have you updated the relevant docstrings, documentation and copyright notice? - [ ] Is this contribution tested against [all ArcticDB's features](../docs/mkdocs/docs/technical/contributing.md)? - [ ] Do all exceptions introduced raise appropriate [error messages](https://docs.arcticdb.io/error_messages/)? - [ ] Are API changes highlighted in the PR description? - [ ] Is the PR labelled as enhancement or bug so it appears in autogenerated release notes? </details> <!-- Thanks for contributing a Pull Request to ArcticDB! Please ensure you have taken a look at: - ArcticDB's Code of Conduct: https://github.com/man-group/ArcticDB/blob/master/CODE_OF_CONDUCT.md - ArcticDB's Contribution Licensing: https://github.com/man-group/ArcticDB/blob/master/docs/mkdocs/docs/technical/contributing.md#contribution-licensing --> --------- Co-authored-by: Georgi Rusev <Georgi Rusev>
Configuration menu - View commit details
-
Copy full SHA for bcf2b2f - Browse repository at this point
Copy the full SHA bcf2b2fView commit details -
Flexible execution and easier analysis with autogenerated Marks (#2609)
#### Reference Issues/PRs <!--Example: Fixes #1234. See also #3456.--> #### What does this implement or fix? Currently our tests lack metadata to help quicly filtered out tests or do in depth analysis of existing tests: - how many tests we have for each one of storage types we test against - how many tests we have for each test type/level - unit, integration - how many test we have in different cross sections of marks With this PR several things are introduced: - dynamic assignments of marks based on physical structure of directories - from there we can obtain which tests are at what level - dynamic assignment of marks based on fixture usage - each test is marked with specific mark if certain storage fixture is used - lmdb, s3, real_s3 etc - dynamic assignment of marks based on library options like dynamic_schema, dynamic_strings etc, inclusing arctic encoding type that allows quick queries over our tests like: ``` pytest -s --co -m "(lmdb and unit) or (lmdb and integration)" ``` to obtain better understanding where and what tests we have. Those marks will be possible to be used further in test execution selection Overall this approach adds good benefit for adding with little effort important metadata to the tests. Further that metadata could be enhanced significantly when needed. Like taking info from external sources - like databases. xls. github to dynamically mark tests with certain properties, like flaky, quaranteen etc. In other words there is no longer need marks to be added and maintained all the time by the team. Effectivley marks now are combination of all those we add explicitly to the test and those assigned to a test by external resources. Most important files to review: conftest.py (here we do dynamic assignment) marking.py (small helper class for better handling of Marks on large scale projects) Additionally this PR introduces small additions to Marks namagement: - ability to assign many marks to a test on a single line - ability to group marks and avoidance of misspelling of marks through Mark and Marks classes As there is no way currently to obtain list of unique tests only a small cmd line utility is also available.: ``` $ . ../build_tooling/list_pytests.sh Usage: -bash <pytest_mark_expression> Example: -bash "pipeline and real_s3" $ . ../build_tooling/list_pytests.sh pipeline and real_s3 2025-08-25 16:03:18,353 - client_utils - INFO - VERSION with AZURE and GCP 240/16608 tests collected (16368 deselected) in 3.67s python/tests/integration/arcticdb/test_arctic.py::test_read_with_read_request_form python/tests/integration/arcticdb/test_arctic_batch.py::test_delete_version_with_snapshot_batch python/tests/integration/arcticdb/test_arctic_batch.py::test_read_batch_overall_query_builder python/tests/integration/arcticdb/test_arctic_batch.py::test_read_batch_overall_query_builder_and_per_request_query_builder_raises python/tests/integration/arcticdb/test_arctic_batch.py::test_read_batch_per_symbol_query_builder python/tests/integration/arcticdb/test_arctic_batch.py::test_read_batch_query_builder_missing_keys python/tests/integration/arcticdb/test_arctic_batch.py::test_read_batch_query_builder_symbol_doesnt_exist python/tests/integration/arcticdb/test_arctic_batch.py::test_read_batch_query_builder_version_doesnt_exist python/tests/integration/arcticdb/test_read_batch_more.py::test_read_batch_multiple_symbols_all_types_data_query_metadata python/tests/integration/arcticdb/test_read_batch_more.py::test_read_batch_multiple_wrong_things_at_once python/tests/integration/arcticdb/test_read_batch_more.py::test_read_batch_query_and_columns python/tests/integration/arcticdb/test_read_batch_more.py::test_read_batch_query_with_and ``` IMPORTANT: this approach can we switched on and off. By default it is switched on. To switch it off use ```ARCTICDB_EXTENDED_MARKS=0``` #### Any other comments? #### Checklist <details> <summary> Checklist for code changes... </summary> - [ ] Have you updated the relevant docstrings, documentation and copyright notice? - [ ] Is this contribution tested against [all ArcticDB's features](../docs/mkdocs/docs/technical/contributing.md)? - [ ] Do all exceptions introduced raise appropriate [error messages](https://docs.arcticdb.io/error_messages/)? - [ ] Are API changes highlighted in the PR description? - [ ] Is the PR labelled as enhancement or bug so it appears in autogenerated release notes? </details> <!-- Thanks for contributing a Pull Request to ArcticDB! Please ensure you have taken a look at: - ArcticDB's Code of Conduct: https://github.com/man-group/ArcticDB/blob/master/CODE_OF_CONDUCT.md - ArcticDB's Contribution Licensing: https://github.com/man-group/ArcticDB/blob/master/docs/mkdocs/docs/technical/contributing.md#contribution-licensing --> --------- Co-authored-by: Georgi Rusev <Georgi Rusev>
Configuration menu - View commit details
-
Copy full SHA for f30c921 - Browse repository at this point
Copy the full SHA f30c921View commit details
Commits on Sep 19, 2025
-
additional check for batch_metadata_multi (#2653)
#### Reference Issues/PRs <!--Example: Fixes #1234. See also #3456.--> #### What does this implement or fix? Several additional tests for batch meadata mutli #### Any other comments? #### Checklist <details> <summary> Checklist for code changes... </summary> - [ ] Have you updated the relevant docstrings, documentation and copyright notice? - [ ] Is this contribution tested against [all ArcticDB's features](../docs/mkdocs/docs/technical/contributing.md)? - [ ] Do all exceptions introduced raise appropriate [error messages](https://docs.arcticdb.io/error_messages/)? - [ ] Are API changes highlighted in the PR description? - [ ] Is the PR labelled as enhancement or bug so it appears in autogenerated release notes? </details> <!-- Thanks for contributing a Pull Request to ArcticDB! Please ensure you have taken a look at: - ArcticDB's Code of Conduct: https://github.com/man-group/ArcticDB/blob/master/CODE_OF_CONDUCT.md - ArcticDB's Contribution Licensing: https://github.com/man-group/ArcticDB/blob/master/docs/mkdocs/docs/technical/contributing.md#contribution-licensing --> Co-authored-by: Georgi Rusev <Georgi Rusev>
Configuration menu - View commit details
-
Copy full SHA for 617ac1c - Browse repository at this point
Copy the full SHA 617ac1cView commit details -
Pin sparrow 1.1.0 on conda (#2662)
This makes it in line with our vcpkg dependency #### Reference Issues/PRs <!--Example: Fixes #1234. See also #3456.--> #### What does this implement or fix? #### Any other comments? #### Checklist <details> <summary> Checklist for code changes... </summary> - [ ] Have you updated the relevant docstrings, documentation and copyright notice? - [ ] Is this contribution tested against [all ArcticDB's features](../docs/mkdocs/docs/technical/contributing.md)? - [ ] Do all exceptions introduced raise appropriate [error messages](https://docs.arcticdb.io/error_messages/)? - [ ] Are API changes highlighted in the PR description? - [ ] Is the PR labelled as enhancement or bug so it appears in autogenerated release notes? </details> <!-- Thanks for contributing a Pull Request to ArcticDB! Please ensure you have taken a look at: - ArcticDB's Code of Conduct: https://github.com/man-group/ArcticDB/blob/master/CODE_OF_CONDUCT.md - ArcticDB's Contribution Licensing: https://github.com/man-group/ArcticDB/blob/master/docs/mkdocs/docs/technical/contributing.md#contribution-licensing -->
Configuration menu - View commit details
-
Copy full SHA for 4776764 - Browse repository at this point
Copy the full SHA 4776764View commit details -
Cleanup CI buckets/containers (#2646)
#### Reference Issues/PRs <!--Example: Fixes #1234. See also #3456.--> #### What does this implement or fix? Release storage on regular basis. We leak storage due to different types of failures that cause symbols and libraries not to be cleaned after tests. This PR adds a step for cleaning the storages - AWS S3, GCP, Azure once a week. By default all data older than 28 days will be cleaned, thus presevrving info for the last moth. The PR adds small boto and azure library for obtaining info and cleaning buckets/containers. This library can be further reused and enhanced if needed for more bucket management that we will need Prior to this PR the leftovers were as follows: ``` AWS TOTAL SIZE : 891,874,609,188 GCP TOTAL SIZE : 3,363,272,750 AZURE TOTAL SIZE: 2,706,787,373 ``` Cleanup buckets with no more than 1 month old data (initial run) : https://github.com/man-group/ArcticDB/actions/runs/17733685971/job/50390234882 Next run after the buckets are clean: https://github.com/man-group/ArcticDB/actions/runs/17756785784/job/50460916726 ``` 2025-09-16 06:28:50,748 - __main__ - INFO - Cleaning before: 2025-08-19 06:28:50.748631+00:00 2025-09-16 06:28:50,749 - __main__ - INFO - Cleaning-up GCP storage 2025-09-16 06:29:05,352 - __main__ - INFO - GCP TOTAL SIZE: 339770117 2025-09-16 06:29:19,172 - utils.bucket_management - INFO - Found 0 objects to delete before 2025-08-19 06:28:50.748631+00:00 2025-09-16 06:29:30,408 - __main__ - INFO - GCP TOTAL SIZE: 339770117 2025-09-16 06:29:30,408 - __main__ - INFO - Cleaning-up Azure storage 2025-09-16 06:29:35,005 - __main__ - INFO - AZURE TOTAL SIZE: 3591350 2025-09-16 06:29:38,719 - utils.bucket_management - INFO - Found 0 blobs to delete before 2025-08-19 06:28:50.748631+00:00 2025-09-16 06:29:42,433 - __main__ - INFO - AZURE TOTAL SIZE: 3591350 2025-09-16 06:29:42,433 - __main__ - INFO - Cleaning-up S3 storage 2025-09-16 06:29:43,227 - __main__ - INFO - AWS S3 TOTAL SIZE: 0 2025-09-16 06:29:43,418 - __main__ - INFO - AWS S3 TOTAL SIZE: 0 ``` #### Any other comments? #### Checklist <details> <summary> Checklist for code changes... </summary> - [ ] Have you updated the relevant docstrings, documentation and copyright notice? - [ ] Is this contribution tested against [all ArcticDB's features](../docs/mkdocs/docs/technical/contributing.md)? - [ ] Do all exceptions introduced raise appropriate [error messages](https://docs.arcticdb.io/error_messages/)? - [ ] Are API changes highlighted in the PR description? - [ ] Is the PR labelled as enhancement or bug so it appears in autogenerated release notes? </details> <!-- Thanks for contributing a Pull Request to ArcticDB! Please ensure you have taken a look at: - ArcticDB's Code of Conduct: https://github.com/man-group/ArcticDB/blob/master/CODE_OF_CONDUCT.md - ArcticDB's Contribution Licensing: https://github.com/man-group/ArcticDB/blob/master/docs/mkdocs/docs/technical/contributing.md#contribution-licensing --> --------- Co-authored-by: Georgi Rusev <Georgi Rusev>
Configuration menu - View commit details
-
Copy full SHA for 0076efd - Browse repository at this point
Copy the full SHA 0076efdView commit details
This comparison is taking too long to generate.
Unfortunately it looks like we can’t render this comparison for you right now. It might be too big, or there might be something weird with your repository.
You can try running this command locally to see the comparison on your machine:
git diff v6.2.1...master