Henge is a Python package for building data storage and retrieval interfaces for arbitrary data. Henge is based on the idea of decomposable recursive unique identifiers (DRUIDs), which are hash-based unique identifiers for data derived from the data itself. For arbitrary data with any structure, Henge can mint unique DRUIDs to identify data, store the data in a key-value database of your choice, and provide lookup functions to retrieve the data in its original structure using its DRUID identifier.

Henge was intended as a building block for sequence collections, but is generic enough to use for any data type that needs content-derived identifiers with database lookup capability.

pip install henge

Create a Henge object by providing a database and a data schema. The database can be a Python dict or backed by persistent storage. Data schemas are JSON-schema descriptions of data types, and can be hierarchical.

import henge

schemas = ["path/to/json_schema.yaml"]
h = henge.Henge(database={}, schemas=schemas)

Insert items into the henge. Upon insert, henge returns the DRUID (digest/checksum/unique identifier) for your object:

druid = h.insert({"name": "Pat", "age": 38}, item_type="person")

Retrieve the original object using the DRUID:

h.retrieve(druid)
# {'age': '38', 'name': 'Pat'}

For a comprehensive walkthrough covering basic types, arrays, nested objects, and advanced features, see the tutorial notebook.

DRUIDs are a special type of unique identifiers with two powerful properties:

• Decomposable: Identifiers in henge automatically retrieve structured data (tuples, arrays, objects). The structure is defined by a JSON schema, so henge can be used as a back-end for arbitrary data types.

• Recursive: Individual elements retrieved by henge can be tagged as recursive, meaning these attributes contain their own DRUIDs. Henge can recurse through these, allowing you to mint unique identifiers for arbitrary nested data structures.

A DRUID is ultimately the result of a digest operation (such as md5 or sha256) on some data. Because DRUIDs are computed deterministically from the item, they represent globally unique identifiers. If you insert the same item repeatedly, it will produce the same DRUID -- this is true across henges as long as they share a data schema.

Use a Python dict as the database for testing or ephemeral use:

h = henge.Henge(database={}, schemas=schemas)

For persistent storage with SQLite:

from sqlitedict import SqliteDict

mydict = SqliteDict('./my_db.sqlite', autocommit=True)
h = henge.Henge(mydict, schemas=schemas)

Requires: pip install sqlitedict

For production use with MongoDB:

1. Start MongoDB with Docker:

docker run --network="host" mongo

For persistent storage, mount a volume to /data/db:

docker run -it --network="host" -v /path/to/data:/data/db mongo

2. Connect henge to MongoDB:

import henge

h = henge.Henge(henge.connect_mongo(), schemas=schemas)

Requires: pip install pymongo mongodict