A Python CLI that wraps common Linux commands for both local and cloud files using cloudpathlib.
Cloud Storage Provider dependencies:
pip install -U cloudsh
# Install for different cloud storage providers
pip install -U cloudsh[gcs] # Google Cloud Storage
pip install -U cloudsh[aws] # Amazon S3
pip install -U cloudsh[azure] # Azure Blob Storage
# Install for all cloud storage providers
pip install -U cloudsh[all]cloudsh provides common Linux commands that work with both local and cloud files. Currently supported commands include:
cat: Concatenate and print filescp: Copy files and directorieshead: Output the first part of filesless: Display file contents with forward and backward navigationls: List directory contentsmkdir: Make directoriesmore: Display file contents page by pagemv: Move files and directoriesrm: Remove files and directoriestail: Output the last part of filestouch: Create empty files
And two additional commands:
complete: Generate shell completion scriptssink: Redirect output to a file
See: https://cloudpathlib.drivendata.org/stable/authentication/ for details on how to authenticate with cloud storage providers.
$ cloudsh ls /tmp
$ cloudsh cp /tmp/file.txt /tmp/file2.txt$ cloudsh ls gs://my-bucket
$ cloudsh touch gs://my-bucket/file.txt$ cloudsh cp /tmp/file.txt gs://my-bucket/file.txt
$ cloudsh mv gs://my-bucket/file.txt /tmp/file.txt# It is easy to redirect output to a local file
$ echo "Hello, World!" > /tmp/hello.txt
# But it is not so easy to redirect output to a cloud file, so we use `sink`
$ echo "Hello, World!" | cloudsh sink gs://my-bucket/hello.txt
# Append to a cloud file
$ echo "Hello, World!" | cloudsh sink -a gs://my-bucket/hello.txtSince the commands work on local files as well, you can make aliases to use cloudsh as a drop-in replacement for the GNU/Linux commands.
alias cat='cloudsh cat'
alias cp='cloudsh cp'
alias head='cloudsh head'
alias ls='cloudsh ls'
alias mkdir='cloudsh mkdir'
alias mv='cloudsh mv'
alias rm='cloudsh rm'
alias tail='cloudsh tail'
alias touch='cloudsh touch'What if I want to use the original GNU/Linux commands?
# alias ls='cloudsh ls'
ls -- -l # actually executes `/usr/bin/ls -l`cloudsh provides shell completion support, including the subcommands, options and both local and cloud paths, for bash, zsh and fish. To enable it:
# For bash
mkdir -p ~/.local/share/bash-completion
cloudsh complete --shell bash > ~/.local/share/bash-completion/cloudsh
activate-global-python-argcomplete --user
# Restart your shell# For zsh
# Create the completions directory
mkdir -p ~/.zsh/completions
# Generate the Zsh script
cloudsh complete --shell zsh > ~/.zsh/completions/_cloudsh
# Update ~/.zshrc
echo 'fpath=(~/.zsh/completions $fpath)' >> ~/.zshrc
echo 'autoload -Uz compinit && compinit' >> ~/.zshrc
# Restart your shell# For fish
cloudsh complete --shell fish > ~/.config/fish/completions/cloudsh.fishBecause of the latency when completing cloud paths, a message 'fetching ...' will be shown when completing cloud paths. export CLOUDSH_COMPLETE_NO_FETCHING_INDICATOR=1 to disable it.
# Only cache the paths at depth 2 in the bucket
cloudsh complete --update-cache --depth 2 gs://my-bucketNote
Remember to update the cache when the bucket structure changes. You can set up a cron job to update the cache periodically.
Tip
For the first time you are using cached cloud path completion, a warning message will be shown to remind you that you are using a cached completion. The warning will only be shown when <tmpdir>/cloudsh_caching_warned does not exist, which will be created after the first warning. To disable the warning permanently, try export CLOUDSH_COMPLETE_CACHING_WARN=1.