Harmonia is a LLM-based agent that leverages the bdi-kit library to perform data harmonization with the user in the loop.

First, add your OpenAI API key to the environment:

export OPENAI_API_KEY=your key goes here

Then use docker compose to build and run the BDIKit Beaker context:

docker compose build
docker compose up -d

Navigate to localhost:8888 to open the UI.

The docker image includes a file named dou.csv by default that can be used. You can experiment with the following script that loads the file and executes a few harmonization tasks.

Load the file dou.csv as a dataframe and subset it to the following columns: Country, Histologic_Grade_FIGO, Histologic_type, FIGO_stage, BMI, Age, Race, Ethnicity, Gender, Tumor_Focality, Tumor_Size_cm.

Please match this to the GDC schema using the 'ct_learning' method, and fix any results that don't look correct.

Find alternative mappings for Histologic_type.

Find alternative mappings for Tumor_Size_cm.

Find value mappings for the columns Country, Histologic_Grade_FIGO, Histologic_type, FIGO_stage, Race, Ethnicity, Gender, Tumor_Focality. If there are any errors in the mappings, please provide suggestions.

Please create a final harmonized table based on the discovered column and value mappings and save it at "dou_harmonized.csv".

Show dou_harmonized.csv and the initial subsetted dou.csv file one after the other for comparison.

Currently the agent supports multiple bdi-kit tools, including match_schema(), match_values(), and materialize_mapping(). Tools are implemented defined in src/bdikit_context/agent.py. Additional tools can easily be added by copying the template for the match_schema tool.

One thing to note is that @tools are managed by Archytas. Archytas allows somewhat restricted argument types and does not allow direct passing of pandas.DataFrame. Instead, dataframes should be referenced by their variable names as a str. The actual code procedure that is executed (see procedures/python3/match_schema.py) treats the arguments from the @tool as variable names; when they should actually be strings they should be wrapped in quotes as in the match_schema.py example. Procedures invoked by tools can have their arguments passed in using Jinja templating. For example:

column_mappings = bdi.match_schema({{ dataset }}, target="{{ target }}", method="{{ method }}")

Here {{ dataset }} is the string name of a pandas.DataFrame and is interpreted as a variable, where as "{{ target }}" is treated as a string such as "gdc".

There are two main places to edit the agent's prompt. In src/bdikit_context/context.py the auto_context is a place to provide additional context. Currently the tools are enumerated here though this isn't strictly necessary. Additionally, prompt can be edited/managed in the agent.py BDIKitAgent docstring.