Noyez zsh histdb import #393

noyez · 2022-05-06T18:41:44Z

This adds import support for zsh-histdb. There are a few caveats.

Caveat 1, async

Edit: This has been addressed by the recent code change to add async traits to the importer, thanks @conradludgate !

Zsh-histdb uses a sqlite file to store history, which can easily be ready by sqlx-rs. However, the Importer trait has methods parse() and histfile(), none are async (i tried to use #[async-trait] with no luck). The parse() method needs to read the sqlite database using sqlx-rs async runtime, but that cannot be done from a non-async function. As a workaround, i added a method called populate(), which gets called before parse() when a zsh-histdb installation is detected. The method populate() is async, and will populate a global Vec<History>. Then parse() will simply read the data from the Vec<History>. This is kind of an ugly hack to get around the async issues, but it works.
I'm open to other ideas

Caveat 2, sqlite and non UTF-8 codes.

Edit: I believe this has been addressed by using Vec<u8> instead of String Thanks to @conradludgate for the suggestion!

While testing on my installation, i had some entries in my history that would crash sqlx-rs with 'invalid utf-8 squence'. I'm not sure this is fault of sqlx-rs, and i wasn't sure how to isolate those issues. One such entry was created by running the command "cd noyez �s\ iMac.sparsebundle". I tried to reproduce this in a smaller db with no luck, sqlx-rs worked on that smaller DB. I don't know how this entry got into my history, whether it was from zsh-histdb, or an import script that i used when i converted from regular zsh-history to zsh-histdb. Without more test data it is hard to test.

~~It might be possible that you'll see bug reports with this error, and if so, perhaps the sqlx-rs project could help.~~

~~If sqlx-rs cannot handle a particular entry the following error is produced:~~

thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: error occurred while decoding column "argv": invalid utf-8 sequence of 1 bytes from index 96

Caused by:
    invalid utf-8 sequence of 1 bytes from index 96

Import compiles passes tests, but doesn't run b/c of async runtime. zsh-histdb uses sqlite, and sqlx-rs is async, but import code is sync.

conradludgate · 2022-05-07T10:06:27Z

Thanks, I appreciate the effort. I'll try and fix it up for you

conradludgate · 2022-05-07T15:16:48Z

Hey, so I've been refactoring over on #395 if you want to take a look. It should be general enough to support sqlx queries. If not, let me know over there 🙏

ellie · 2022-05-09T06:47:46Z

Hey @noyez! Thanks for all the effort here! 🚀

Just gonna need a rebase on top of @conradludgate's work I just merged. Hopefully things are easier now :)

noyez · 2022-05-09T11:12:59Z

Great! Should be a lot easier, I guess I should have waited a week :).it’ll take me a few days to get the time, but it’ll get done.

…

On May 9, 2022, at 02:47, Ellie Huxtable ***@***.***> wrote: Hey @noyez <https://github.com/noyez>! Thanks for all the effort here! 🚀 Just gonna need a rebase on top of @conradludgate <https://github.com/conradludgate>'s work I just merged. Hopefully things are easier now :) — Reply to this email directly, view it on GitHub <#393 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAJ3VFMRHTMYCMLFKJ25YJLVJCYJ3ANCNFSM5VI6MVPQ>. You are receiving this because you were mentioned.

…port

noyez · 2022-05-10T01:50:59Z

Code merged, updated from latest main.

ellie

Woo! Thanks for this! 🚀 <3

conradludgate

Minor comments, if you could delete the test DB files too please 🙏

conradludgate · 2022-05-10T22:26:49Z

atuin-client/Cargo.toml

 sha2 = { version = "0.10", optional = true }
 rmp-serde = { version = "1.0.0", optional = true }
 base64 = { version = "0.13.0", optional = true }
+tokio = { version = "1", features = ["full"] }


I don't think this is needed anymore

atuin-client/src/import/zsh_histdb.rs

ellie · 2022-05-10T22:33:26Z

Although the test actually loads that DB. I'd be happy to keep it as a test fixture

- Rewriting tests to eliminate depencency on local file system - Using `Vec<u8>` for command strings instead of `String` to eliminate the utf8 errors i was seeing previously. Seems to be working.

noyez · 2022-05-13T16:50:35Z

New code merged,

non FS depending tests
using Vec instead of string.

ellie

Approved! Tysm for the work here, it's great to get another importer in! 🚀

06ac958 Show current version on server index (#436) 706b1af Disable ARM docker builds (#438) f2abc23 Update README.md 3c2b055 Noyez fix dir hostname utf8 (#430) 3f5350d [feature] Add scroll wheel support to interactive history search (#435) dcdde22 Fix text outline for dark mode 9ac0c60 Implement cursor (#412) 119ab9e Adds password prompt for register and login (#424) e5df809 Noyez zsh histdb import (#393) b08e254 Improve default fish keybindings (#420) 4096c6e Update README.md cd2a3ab Add fish shell to key binding docs (#418) b278211 Bump clap_complete from 3.1.3 to 3.1.4 (#397) ee66c0a Bump axum from 0.5.5 to 0.5.6 (#415) 4297e26 Bump tokio from 1.18.1 to 1.18.2 (#396) dbd9ca5 Bump clap from 3.1.16 to 3.1.18 (#401) a7c9d19 Bump tower-http from 0.3.2 to 0.3.3 (#399) 3b79f68 Bump axum from 0.5.4 to 0.5.5 (#402) f340771 Cleanup dependencies – disable unnecessary or unused features (#407) ab294cd Don't pollute shell environment - remove 'id' variable (#408) 14b3060 Allow to build atuin server without client (#404) 5e4e8d1 Don't create config dir for server in default location if not needed (#406) b7946cc Update Chinese version README.md (#403) e0291f6 Update README.md 301190e Build ARM docker image in GitHub Actions using QEMU (#400) 1d030b9 Importer V3 (#395) d3a4ff9 Bump clap from 3.1.15 to 3.1.16 (#392) e9d2ec4 Add ctrl-k and ctrl-j for up and down (#394) 25afb5b Bump serde_json from 1.0.80 to 1.0.81 (#387) 4a839da Adds stats summary (#384) 7a394b0 Bump serde from 1.0.136 to 1.0.137 (#375) edd3f81 Bump clap_complete from 3.1.2 to 3.1.3 (#377) d85d03d Bump log from 0.4.16 to 0.4.17 (#382) dc3b7ef Bump tokio from 1.18.0 to 1.18.1 (#383) 12440c1 Bump serde_json from 1.0.79 to 1.0.80 (#376) 731042f Bump tower-http from 0.3.1 to 0.3.2 (#378) 82505e6 Bump clap from 3.1.12 to 3.1.15 (#381) e05c19d Add Chinese documentation translation & Fix spelling mistakes (#373) 6e280e2 Add Russian documentation translation (#365) 40efdd1 Bump http from 0.2.6 to 0.2.7 (#368) 8bc5bec Bump tower-http from 0.3.0 to 0.3.1 (#367) 172ac8d Create FUNDING.yml 7cdd00b Bump tokio from 1.17.0 to 1.18.0 (#357) 9d2e9ea Search: Allow specifiying the limited of returned entries (#364) 93ab4e7 ignore JetBrains IDEs, tidy-up imports (#348) 2cb4cb3 Bump axum from 0.5.3 to 0.5.4 (#355) 796644e Add created_at column to users (#354) f8233bc SQLx cannot run this migration OK (#353) d8ef5dd fix db range query (#351) 5926ea6 fix import auto for bash (#352) 43d299f bump tui (#346) 8ac6571 Remove all select * from the server queries (#347) 4030de4 Add btree index on history table (#345) b692e0c Bump tower-http from 0.2.5 to 0.3.0 (#343) 3680f4a Bump clap from 3.1.11 to 3.1.12 (#342) 7f5310a history list (#340)

noyez added 2 commits May 2, 2022 11:19

Attempting to implement zsh-histdb import

ad36770

Import compiles passes tests, but doesn't run b/c of async runtime. zsh-histdb uses sqlite, and sqlx-rs is async, but import code is sync.

More working on importing histdb

c545a08

Merge remote-tracking branch 'upstream/main' into noyez_zsh_histdb_im…

49791ca

…port

ellie previously approved these changes May 10, 2022

View reviewed changes

conradludgate reviewed May 10, 2022

View reviewed changes

Rewriting tests and using Vec<u8> instead of String

5c52392

- Rewriting tests to eliminate depencency on local file system - Using `Vec<u8>` for command strings instead of `String` to eliminate the utf8 errors i was seeing previously. Seems to be working.

noyez dismissed ellie’s stale review via 5c52392 May 13, 2022 16:47

Running fmt

47e3c47

ellie approved these changes May 20, 2022

View reviewed changes

ellie merged commit e5df809 into atuinsh:main May 20, 2022

norpol mentioned this pull request May 31, 2022

zsh-histdb import #93

Closed

ellie mentioned this pull request Jun 6, 2022

Release v0.10.0 #440

Merged

Uh oh!

Uh oh!

Noyez zsh histdb import #393

Noyez zsh histdb import #393

Uh oh!

Conversation

noyez commented May 6, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Caveat 1, async

Caveat 2, sqlite and non UTF-8 codes.

Uh oh!

conradludgate commented May 7, 2022

Uh oh!

conradludgate commented May 7, 2022

Uh oh!

ellie commented May 9, 2022

Uh oh!

noyez commented May 9, 2022 via email

Uh oh!

noyez commented May 10, 2022

Uh oh!

ellie left a comment

Choose a reason for hiding this comment

Uh oh!

conradludgate left a comment

Choose a reason for hiding this comment

Uh oh!

conradludgate May 10, 2022

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

ellie commented May 10, 2022

Uh oh!

noyez commented May 13, 2022

Uh oh!

ellie left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

noyez commented May 6, 2022 •

edited

Loading