Verdin is a tiny bird, and also a Tinybird SDK for Python.
pip install verdin
Python 3.10+
# the tinybird module exposes all important tinybird concepts
from verdin import tinybird
client = tinybird.Client("p.mytoken")
query = client.sql("select * from my_datasource__v0")
# run the query with `FORMAT JSON` and receive a QueryJsonResult
response: tinybird.QueryJsonResult = query.json()
# print records returned from the pipe
print(response.data)You can also run, e.g., query.get(format=OutputFormat.CSV) to get the raw response with CSV data.
from verdin import tinybird
client = tinybird.Client("p.mytoken")
pipe = client.pipe("my_pipe")
# query the pipe using dynamic parameters
response: tinybird.PipeJsonResponse = pipe.query({"key": "val"})
# print records returned from the pipe
print(response.data)from verdin import tinybird
client = tinybird.Client("p.mytoken")
# will access my_datasource__v0
datasource = client.datasource("my_datasource", version=0)
# query the pipe using dynamic parameters
datasource.append([
("col1-row1", "col2-row1"),
("col1-row2", "col2-row2"),
])The DataSource object also gives you access to /v0/events, which is the high-frequency ingest, to append data.
Use the send_events method and pass JSON serializable documents to it.
datasource.send_events(records=[
{"key": "val1"},
{"key": "val2"},
...
])Verdin provides a way to queue and batch data continuously:
from queue import Queue
from threading import Thread
from verdin import tinybird
from verdin.worker import QueuingDatasourceAppender
client = tinybird.Client("p.mytoken")
records = Queue()
appender = QueuingDatasourceAppender(records, client.datasource("my_datasource"))
Thread(target=appender.run).start()
# appender will regularly read batches of data from the queue and append them
# to the datasource. the appender respects rate limiting.
records.put(("col1-row1", "col2-row1"))
records.put(("col1-row2", "col2-row2"))The DataSource and Pipes objects presented so far are high-level abstractions that provide a convenience Python API
to deal with the most common use cases. Verdin also provides more low-level access to APIs via client.api.
The following APIs are available:
/v0/datasources:client.api.datasources/v0/events:client.api.events/v0/pipes:client.api.pipes/v0/sql:client.api.query/v0/tokens:client.api.tokens/v0/variables:client.api.variables
Note that for some (datasources, pipes, tokens), manipulation operations are not implemented as they are typically done through tb deployments and not through the API.
Also note that API clients do not take care of retries or rate limiting. The caller is expected to handle fault tolerance.
You can query a pipe through the pipes API as follows:
from verdin import tinybird
client = tinybird.Client(...)
response = client.api.pipes.query(
"my_pipe",
parameters={"my_param": "..."},
query="SELECT * FROM _ LIMIT 10",
)
for record in response.data:
# each record is a dictionary
...You can use the HFI endpoint /v0/events through the events api. As records, you can pass a list of JSON serializable
documents.
from verdin import tinybird
client = tinybird.Client(...)
response = client.api.events.send("my_datasource", records=[
{"id": "...", "value": "..."},
...
])
assert response.quarantined_rows == 0Create the virtual environment, install dependencies, and run tests
make venv
make test
Run the code formatter
make format
Upload the pypi package using twine
make upload