A lightweight and portable jq style cli for doing jq queries/filters/maps/transforms on YAML/TOML/JSON documents by converting to JSON and passing data to jq. Output is raw jq output which can optionally be mapped to TOML or YAML.
Via cargo:
cargo install lqor download a prebuilt from releases either manually, or via binstall:
cargo binstall lqNote: Requires jq.
- arbitrary
jqusage on any input format (yaml/toml/json) by going through json and jq - same syntax, same filters, types, operators, conditionals, regexes, assignment, modules, etc
- matches
jq's cli interface (only some extra input/output format controlling flags) - supports
jqoutput formatters such as-c,-r, and-j(compact, raw, joined output resp)
- supports multidoc yaml input, handles yaml merge keys (expanding tags)
- supports multidoc document splitting into expression based filenames
- supports in-place edits of documents
- maintains key order even while roundtripping between formats
- reads from stdin xor file (file if last arg is a file)
- filetype format inference when passing files
- quick input/output flags:
-y(YAML out) or-t(TOML out),-T(TOML in),-J(JSON in)
- ~1MB in binary (for small CI images / binstalled ci actions)
- 99% replacement of python-yq (with
yqnamed/linked tolq)
- Shells out to
jq(not standalone - for now) - Expands YAML tags (input is singleton mapped -> recursively, then merged) - so tags are not preserved in the output
- Does not preserve indentation (unsupported in serde_yaml)
- Halts on duplicate keys in the input document
- Formats require a serde implementation.
- Limited format support. No XML/CSV/RON support (or other more exotic formats). KDL wanted.
Use as jq either via stdin:
$ lq '.[3].kind' -r < test/deploy.yaml
Service
$ lq -y '.[3].metadata' < test/deploy.yaml
labels:
app: controller
name: controller
namespace: defaultor from a file arg (at the end):
$ lq '.[3].kind' -r test/deploy.yaml
$ lq -y '.[3].metadata' test/deploy.yamlInfers input format from extension, or set explicitly via -T or --input=toml.
$ lq '.package.categories[]' -r Cargo.toml
command-line-utilities
parsingconvert jq output back into toml (-t):
$ lq -t '.package.metadata' Cargo.toml
[binstall]
bin-dir = "lq-{ target }/{ bin }{ format }"
pkg-url = "{ repo }/releases/download/{ version }lq-{ target }{ archive-suffix }"convert jq output to yaml (-y) and set explicit toml input when using stdin (-T):
$ lq -Ty '.dependencies.clap' < Cargo.toml
features:
- cargo
- derive
version: 4.4.2jq style compact output:
$ lq '.profile' -c Cargo.toml
{"release":{"lto":true,"panic":"abort","strip":"symbols"}}To shortcut passing input formats, you can add alias tq='lq --input=toml' in your .bashrc / .zshrc (etc).
Infers input format from extension, or set explicitly via -J or --input=json.
$ lq -Jy '.ingredients | keys' < test/guacamole.json
- avocado
- coriander
- cumin
- garlic
- lime
- onions
- pepper
- salt
- tomatoesUsing JSON input is kind of like talking to jq directly, with the benefit that you can change output formats, or do inplace edits.
Default is going from yaml input to jq output to allow further pipes into jq.
- Input flags are upper case ::
-Jjson input,-Ttoml input (shorthands for--input=FORMAT) - Output flags are lower case ::
-yyaml output,-ttoml output (shorthands for--output=FORMAT)
Ex;
lq:: yaml -> jq outputlq -t:: yaml -> tomllq -y:: yaml -> yamllq -Tt:: toml -> tomllq -Jy:: json -> yamljq -Ty:: toml -> yamljq -Jt:: json -> toml
Output formatting such as -y for YAML or -t for TOML will require the output from jq to be parseable json.
If you pass on -r,-c or -c for raw/compact output, then this output may not be parseable as json.
Two things you cannot do in jq:
Split a bundle of yaml files into a yaml file per Kubernetes .metadata.name key:
mkdir -p crds
curl -sSL https://github.com/prometheus-operator/prometheus-operator/releases/download/v0.82.1/stripped-down-crds.yaml \
| lq . -y --split '"crds/" + (.metadata.name) + ".yaml"'Patch a json file (without multiple pipes):
lq -i '.SKIP_HOST_UPDATE=true' ~/.config/discord/settings.jsonAny weird things you can do with jq works. Some common (larger) examples:
Select on yaml multidoc:
$ lq '.[] | select(.kind == "Deployment") | .spec.template.spec.containers[0].ports[0].containerPort' test/deploy.yaml
8000Escaping keys with slashes etc in them:
lq '.updates[] | select(.["package-ecosystem"] == "cargo") | .groups' .github/dependabot.ymlYou can import jq modules e.g. k.jq:
$ lq 'include "k"; .[] | gvk' -r -L$PWD/test/modules < test/deploy.yaml
v1.ServiceAccount
rbac.authorization.k8s.io/v1.ClusterRole
rbac.authorization.k8s.io/v1.ClusterRoleBinding
v1.Service
apps/v1.DeploymentThe project respects RUST_LOG when set, and sends these diagnostic logs to stderr:
$ RUST_LOG=debug lq '.version' test/circle.yml
2023-09-18T23:17:04.533055Z DEBUG lq: args: Args { input: Yaml, output: Jq, yaml_output: false, toml_output: false, in_place: false, jq_query: ".version", file: Some("test/circle.yml"), compact_output: false, raw_output: false, join_output: false, modules: None }
2023-09-18T23:17:04.533531Z DEBUG lq: found 1 documents
2023-09-18T23:17:04.533563Z DEBUG lq: input decoded as json: {"definitions":{"filters":{"on_every_commit":{"tags":{"only":"/.*/"}},"on_tag":{"branches":{"ignore":"/.*/"},"tags":{"only":"/v[0-9]+(\\.[0-9]+)*/"}}},"steps":[{"step":{"command":"chmod a+w . && cargo build --release","name":"Build binary"}},{"step":{"command":"rustc --version; cargo --version; rustup --version","name":"Version information"}}]},"jobs":{"build":{"docker":[{"image":"clux/muslrust:stable"}],"environment":{"IMAGE_NAME":"lq"},"resource_class":"xlarge","steps":["checkout",{"run":{"command":"rustc --version; cargo --version; rustup --version","name":"Version information"}},{"run":{"command":"chmod a+w . && cargo build --release","name":"Build binary"}},{"run":"echo versions"}]},"release":{"docker":[{"image":"clux/muslrust:stable"}],"resource_class":"xlarge","steps":["checkout",{"run":{"command":"rustc --version; cargo --version; rustup --version","name":"Version information"}},{"run":{"command":"chmod a+w . && cargo build --release","name":"Build binary"}},{"upload":{"arch":"x86_64-unknown-linux-musl","binary_name":"${IMAGE_NAME}","source":"target/x86_64-unknown-linux-musl/release/${IMAGE_NAME}","version":"${CIRCLE_TAG}"}}]}},"version":2.1,"workflows":{"my_flow":{"jobs":[{"build":{"filters":{"tags":{"only":"/.*/"}}}},{"release":{"filters":{"branches":{"ignore":"/.*/"},"tags":{"only":"/v[0-9]+(\\.[0-9]+)*/"}}}}]},"version":2}}
2023-09-18T23:17:04.533650Z DEBUG lq: jq args: [".version"]
2023-09-18T23:17:04.538606Z DEBUG lq: jq stdout: 2.1
2.1Because yaml is the default input language, you can use it as your top level yq executable with a symlink or alias:
# globally make yq be lq
ln -s $(which lq) /usr/local/bin/yq
# alias yq to lq in shell environment only
alias yq=lqIt is mostly compatible with python-yq (which uses jq syntax) but differs from go yq (which invents its own syntax).
(This use-case was the first use-case for this tool, i.e. to get rid of heavy python deps in CI images)