Thanks to visit codestin.com
Credit goes to github.com

Skip to content

feat(storage): init s3 backend for blob storage#466

Merged
darkskygit merged 7 commits into
masterfrom
0614/init-s3
Jun 15, 2023
Merged

feat(storage): init s3 backend for blob storage#466
darkskygit merged 7 commits into
masterfrom
0614/init-s3

Conversation

@thorseraq
Copy link
Copy Markdown
Contributor

@thorseraq thorseraq commented Jun 14, 2023 •
edited
Loading

We plan to put blob data into s3 storage, and keep local db to make metadata querying speed unaffected.

This draft PR plans to adapt S3Storage to implement BlobStorage, so we can delegate relevant operations to it. It's worth noting that some behavior under BlobStorage may not be needed, and may need further discussion.

Upstream issue: #374, This PR will close #467

S3Storage was renamed to BucketStorage, which may represent more general key-value like storage medium

@vercel
Copy link
Copy Markdown

vercel Bot commented Jun 14, 2023 •
edited
Loading

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name Status Preview Comments Updated (UTC)
octobase ✅ Ready (Inspect) Visit Preview 💬 Add feedback Jun 15, 2023 11:23am

@thorseraq thorseraq temporarily deployed to development June 14, 2023 09:37 — with GitHub Actions Inactive
@thorseraq thorseraq temporarily deployed to development June 14, 2023 09:49 — with GitHub Actions Inactive
@thorseraq thorseraq temporarily deployed to development June 14, 2023 09:49 — with GitHub Actions Inactive
@thorseraq thorseraq temporarily deployed to development June 14, 2023 09:49 — with GitHub Actions Inactive
@codecov
Copy link
Copy Markdown

codecov Bot commented Jun 14, 2023 •
edited
Loading

Codecov Report

Patch coverage: 26.42% and project coverage change: -0.32 ⚠️

Comparison is base (c94cf10) 69.96% compared to head (42aab0b) 69.65%.

❗ Current head 42aab0b differs from pull request most recent head cd3e4fb. Consider uploading reports for the commit cd3e4fb to get more accurate results

Additional details and impacted files 
@@            Coverage Diff             @@
##           master     #466      +/-   ##
==========================================
- Coverage   69.96%   69.65%   -0.32%     
==========================================
  Files         161      164       +3     
  Lines       17732    17872     +140     
==========================================
+ Hits        12407    12448      +41     
- Misses       5325     5424      +99
Impacted Files Coverage Δ
libs/jwst-storage/src/entities/bucket_blobs.rs 0.00% <0.00%> (ø)
.../jwst-storage/src/storage/blobs/bucket_local_db.rs 0.00% <0.00%> (ø)
libs/jwst-storage/src/storage/blobs/local_db.rs 79.00% <0.00%> (ø)
libs/jwst-storage/src/storage/blobs/mod.rs 92.98% <ø> (ø)
libs/jwst-storage/src/types.rs 0.00% <ø> (ø)
libs/jwst/src/lib.rs 54.54% <ø> (ø)
libs/jwst/src/types/blob.rs 0.00% <ø> (ø)
.../src/m20230614_000001_initial_bucket_blob_table.rs 79.54% <79.54%> (ø)
libs/jwst-storage/src/migration/src/lib.rs 100.00% <100.00%> (ø)
libs/jwst-storage/src/migration/src/schema.rs 100.00% <100.00%> (ø)

... and 5 files with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Do you have feedback about the report comment? Let us know in this issue.

@thorseraq thorseraq mentioned this pull request Jun 14, 2023
Closed
@github-actions
Copy link
Copy Markdown

github-actions Bot commented Jun 14, 2023 •
edited
Loading

Benchmark Results

Linux

group                                  master                                 pr
-----                                  ------                                 --
codec/jwst decode var_int (32 bit)     1.06    346.6±5.09µs        ? ?/sec    1.00    327.4±4.82µs        ? ?/sec
codec/jwst decode var_int (64 bit)     1.00    316.2±0.32µs        ? ?/sec    1.22    386.9±2.83µs        ? ?/sec
codec/jwst decode var_uint (32 bit)    1.00    272.1±3.21µs        ? ?/sec    1.01    274.8±2.23µs        ? ?/sec
codec/jwst decode var_uint (64 bit)    1.03    292.6±4.10µs        ? ?/sec    1.00    284.8±1.55µs        ? ?/sec
codec/jwst encode var_int (32 bit)     1.00    392.0±0.66µs        ? ?/sec    1.00    391.9±0.35µs        ? ?/sec
codec/jwst encode var_int (64 bit)     1.10    434.6±5.97µs        ? ?/sec    1.00    395.9±4.35µs        ? ?/sec
codec/jwst encode var_uint (32 bit)    1.00    315.1±4.47µs        ? ?/sec    1.00    314.1±4.62µs        ? ?/sec
codec/jwst encode var_uint (64 bit)    1.00    313.9±4.04µs        ? ?/sec    1.00    313.9±3.63µs        ? ?/sec
codec/lib0 decode var_int (64 bit)     1.00    270.8±1.35µs        ? ?/sec    1.03    278.9±5.49µs        ? ?/sec
codec/lib0 decode var_uint (32 bit)    1.03    264.2±2.72µs        ? ?/sec    1.00    255.9±7.15µs        ? ?/sec
codec/lib0 decode var_uint (64 bit)    1.06    276.6±5.98µs        ? ?/sec    1.00    262.0±5.12µs        ? ?/sec
codec/lib0 encode var_int (64 bit)     1.00    384.5±5.86µs        ? ?/sec    1.25   479.3±14.62µs        ? ?/sec
codec/lib0 encode var_uint (32 bit)    1.00    300.9±7.29µs        ? ?/sec    1.02    305.8±2.50µs        ? ?/sec
codec/lib0 encode var_uint (64 bit)    1.00    298.5±3.20µs        ? ?/sec    1.15    342.9±0.91µs        ? ?/sec
operations/jwst array insert           1.00    255.4±0.46µs        ? ?/sec    1.01    256.7±0.20µs        ? ?/sec
operations/jwst array insert range     1.00   469.4±29.96µs        ? ?/sec    1.00   469.3±29.04µs        ? ?/sec
operations/yrs array insert            1.00     85.0±1.31µs        ? ?/sec    1.07     91.1±0.26µs        ? ?/sec
operations/yrs array insert range      1.00    113.3±0.83µs        ? ?/sec    1.04    118.0±0.62µs        ? ?/sec
update/parse with jwst/basic.bin       1.00     60.2±0.21µs    90.7 MB/sec    1.01     60.8±0.22µs    89.8 MB/sec
update/parse with jwst/database.bin    1.00     65.3±0.39µs    50.8 MB/sec    1.00     65.6±0.23µs    50.5 MB/sec
update/parse with jwst/large.bin       1.01      4.1±0.01ms    55.2 MB/sec    1.00      4.1±0.01ms    55.9 MB/sec
update/parse with yrs/basic.bin        1.00     41.4±0.15µs   132.1 MB/sec    1.00     41.2±0.04µs   132.4 MB/sec
update/parse with yrs/database.bin     1.00     33.3±0.02µs    99.4 MB/sec    1.00     33.2±0.02µs    99.8 MB/sec
update/parse with yrs/large.bin        1.00      2.2±0.00ms   105.3 MB/sec    1.00      2.2±0.01ms   105.0 MB/sec

Windows

group                                  master                                 pr
-----                                  ------                                 --
codec/jwst decode var_int (32 bit)     1.00   342.1±18.70µs        ? ?/sec    1.05   357.6±23.90µs        ? ?/sec
codec/jwst decode var_int (64 bit)     1.00   339.3±14.32µs        ? ?/sec    1.00   340.8±17.44µs        ? ?/sec
codec/jwst decode var_uint (32 bit)    1.02   303.0±18.13µs        ? ?/sec    1.00   298.5±15.16µs        ? ?/sec
codec/jwst decode var_uint (64 bit)    1.02   309.8±18.73µs        ? ?/sec    1.00   303.4±14.69µs        ? ?/sec
codec/jwst encode var_int (32 bit)     1.00   414.5±27.03µs        ? ?/sec    1.06   439.5±37.72µs        ? ?/sec
codec/jwst encode var_int (64 bit)     1.01   407.5±25.24µs        ? ?/sec    1.00   404.2±22.01µs        ? ?/sec
codec/jwst encode var_uint (32 bit)    1.00   274.4±12.39µs        ? ?/sec    1.03   283.9±19.43µs        ? ?/sec
codec/jwst encode var_uint (64 bit)    1.02   279.3±20.99µs        ? ?/sec    1.00   273.3±12.72µs        ? ?/sec
codec/lib0 decode var_int (64 bit)     1.00   276.9±13.25µs        ? ?/sec    1.01   278.6±14.03µs        ? ?/sec
codec/lib0 decode var_uint (32 bit)    1.01   243.8±14.11µs        ? ?/sec    1.00   242.0±13.78µs        ? ?/sec
codec/lib0 decode var_uint (64 bit)    1.02   239.9±12.36µs        ? ?/sec    1.00    234.1±8.65µs        ? ?/sec
codec/lib0 encode var_int (64 bit)     1.02   413.9±30.75µs        ? ?/sec    1.00   404.7±19.49µs        ? ?/sec
codec/lib0 encode var_uint (32 bit)    1.00   274.4±17.13µs        ? ?/sec    1.06   292.1±12.53µs        ? ?/sec
codec/lib0 encode var_uint (64 bit)    1.07   300.4±25.05µs        ? ?/sec    1.00   280.6±13.81µs        ? ?/sec
operations/jwst array insert           1.10   319.3±38.72µs        ? ?/sec    1.00   289.4±11.57µs        ? ?/sec
operations/yrs array insert            1.01    100.7±3.71µs        ? ?/sec    1.00     99.3±3.41µs        ? ?/sec
update/parse with jwst/basic.bin       1.00    113.9±4.60µs    47.9 MB/sec    1.03    116.9±4.53µs    46.7 MB/sec
update/parse with jwst/database.bin    1.00     91.5±3.06µs    36.2 MB/sec    1.00     91.8±3.34µs    36.1 MB/sec
update/parse with jwst/large.bin       1.02      5.8±0.30ms    39.1 MB/sec    1.00      5.7±0.26ms    39.9 MB/sec
update/parse with yrs/basic.bin        1.00     66.0±3.46µs    82.7 MB/sec    1.03     67.7±3.37µs    80.6 MB/sec
update/parse with yrs/database.bin     1.00     50.8±1.87µs    65.3 MB/sec    1.04     52.6±2.31µs    62.9 MB/sec
update/parse with yrs/large.bin        1.01      3.1±0.17ms    73.0 MB/sec    1.00      3.1±0.13ms    74.0 MB/sec

@Xuanwo Xuanwo mentioned this pull request Jun 15, 2023
Closed
@thorseraq thorseraq temporarily deployed to development June 15, 2023 04:05 — with GitHub Actions Inactive
@thorseraq thorseraq temporarily deployed to development June 15, 2023 04:05 — with GitHub Actions Inactive
@thorseraq thorseraq temporarily deployed to development June 15, 2023 08:19 — with GitHub Actions Inactive
@thorseraq thorseraq temporarily deployed to development June 15, 2023 08:26 — with GitHub Actions Inactive
github-advanced-security[bot]
github-advanced-security AI found potential problems Jun 15, 2023
View reviewed changes
Comment thread libs/jwst-storage/src/storage/blobs/bucket_local_db.rs Fixed
@thorseraq thorseraq temporarily deployed to development June 15, 2023 08:45 — with GitHub Actions Inactive
@thorseraq thorseraq temporarily deployed to development June 15, 2023 08:45 — with GitHub Actions Inactive
@thorseraq thorseraq temporarily deployed to development June 15, 2023 09:12 — with GitHub Actions Inactive
@thorseraq thorseraq temporarily deployed to development June 15, 2023 09:13 — with GitHub Actions Inactive
@darkskygit darkskygit temporarily deployed to development June 15, 2023 11:23 — with GitHub Actions Inactive
@darkskygit darkskygit temporarily deployed to development June 15, 2023 11:23 — with GitHub Actions Inactive
@darkskygit darkskygit temporarily deployed to development June 15, 2023 11:23 — with GitHub Actions Inactive
@darkskygit darkskygit merged commit 110b30e into master Jun 15, 2023
@darkskygit darkskygit deleted the 0614/init-s3 branch June 15, 2023 11:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@darkskygit darkskygit darkskygit approved these changes

Assignees

@thorseraq thorseraq

Labels

None yet

Projects

AFFiNE Project
Archived in project

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

add s3 database migration

3 participants

@thorseraq @darkskygit @github-advanced-security