feat: Add integration tests for dbt import with local dbt project #5877
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,84 @@ | ||
| name: dbt-integration-tests | ||
|
|
||
| # Run dbt integration tests on PRs | ||
| on: | ||
| pull_request: | ||
| paths: | ||
| - 'sdk/python/feast/dbt/**' | ||
| - 'sdk/python/tests/integration/dbt/**' | ||
| - 'sdk/python/tests/unit/dbt/**' | ||
| - '.github/workflows/dbt-integration-tests.yml' | ||
|
|
||
| jobs: | ||
| dbt-integration-test: | ||
| runs-on: ubuntu-latest | ||
| strategy: | ||
| matrix: | ||
| python-version: ["3.11", "3.12"] | ||
| env: | ||
| PYTHON: ${{ matrix.python-version }} | ||
| steps: | ||
| - uses: actions/checkout@v4 | ||
|
|
||
| - name: Setup Python | ||
| uses: actions/setup-python@v5 | ||
| with: | ||
| python-version: ${{ matrix.python-version }} | ||
| architecture: x64 | ||
|
|
||
| - name: Install the latest version of uv | ||
| uses: astral-sh/setup-uv@v5 | ||
| with: | ||
| enable-cache: true | ||
|
|
||
| - name: Install dependencies | ||
| run: make install-python-dependencies-ci | ||
|
|
||
| - name: Install dbt and dbt-duckdb | ||
| run: | | ||
| uv pip install --system dbt-core dbt-duckdb | ||
|
|
||
| - name: Run dbt commands | ||
| run: | | ||
| cd sdk/python/tests/integration/dbt/test_dbt_project | ||
| dbt deps | ||
| dbt build | ||
| dbt test | ||
|
|
||
| - name: Setup Feast project for dbt import test | ||
| run: | | ||
| cd sdk/python/tests/integration/dbt | ||
| mkdir -p feast_repo | ||
| cd feast_repo | ||
| cat > feature_store.yaml << EOF | ||
| project: feast_dbt_test | ||
| registry: data/registry.db | ||
| provider: local | ||
| online_store: | ||
| type: sqlite | ||
| path: data/online_store.db | ||
| EOF | ||
| mkdir -p data | ||
|
|
||
| - name: Test feast dbt import | ||
| run: | | ||
| cd sdk/python/tests/integration/dbt/feast_repo | ||
| feast -c feature_store.yaml dbt import \ | ||
| -m ../test_dbt_project/target/manifest.json \ | ||
| -e driver_id \ | ||
| -d file \ | ||
| --tag feast | ||
|
|
||
| - name: Verify feast objects were created | ||
| run: | | ||
| cd sdk/python/tests/integration/dbt/feast_repo | ||
| feast -c feature_store.yaml feature-views list | ||
| feast -c feature_store.yaml entities list | ||
|
|
||
| - name: Run dbt integration tests | ||
|
Member
There was a problem hiding this comment. @copillot we should also run the standard dbt commands; i.e,. dbt run
dbt build
dbt testafter that we should then test the new feast cli and test that feast materialization works from a dbt model into a sqlite online store.
Contributor
Author
There was a problem hiding this comment. Added dbt commands (deps, build, test) and Feast CLI testing with SQLite online store in commit ba38da0. The workflow now:
|
||
| run: | | ||
| cd sdk/python | ||
| python -m pytest tests/integration/dbt/test_dbt_integration.py -v --tb=short | ||
|
|
||
| - name: Minimize uv cache | ||
| run: uv cache prune --ci | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,96 @@ | ||
| # dbt Integration Tests | ||
|
|
||
| Integration tests for Feast's dbt integration feature, which allows importing dbt models as Feast FeatureViews. | ||
|
|
||
| ## Overview | ||
|
|
||
| These tests verify the complete workflow of: | ||
| 1. Parsing dbt `manifest.json` files using dbt-artifacts-parser | ||
| 2. Extracting model metadata (columns, types, tags, descriptions) | ||
| 3. Creating Feast objects (Entity, DataSource, FeatureView) from dbt models | ||
| 4. Generating Python code with Feast definitions | ||
|
|
||
| ## Test Coverage | ||
|
|
||
| ### TestDbtManifestParsing | ||
| - Parse manifest metadata (dbt version, project name) | ||
| - Extract all models from manifest | ||
| - Filter models by dbt tags | ||
| - Filter models by name | ||
| - Parse model properties (database, schema, table, description, tags) | ||
| - Parse column metadata (name, type, description) | ||
|
|
||
| ### TestDbtToFeastMapping | ||
| - Create BigQuery data sources from dbt models | ||
| - Create Snowflake data sources from dbt models | ||
| - Create File data sources from dbt models | ||
| - Create Feast entities | ||
| - Create FeatureViews with proper schema | ||
| - Exclude entity and timestamp columns from features | ||
| - Handle custom excluded columns | ||
| - Create all objects together (Entity + DataSource + FeatureView) | ||
|
|
||
| ### TestDbtDataSourceTypes | ||
| - Test all supported data source types (bigquery, snowflake, file) | ||
| - Verify unsupported types raise errors | ||
|
|
||
| ### TestDbtCodeGeneration | ||
| - Generate Python code from dbt models | ||
| - Generate code for different data source types | ||
| - Verify generated code structure and imports | ||
|
|
||
| ### TestDbtTypeMapping | ||
| - Map dbt types to Feast types correctly: | ||
| - STRING → String | ||
| - INT32 → Int32 | ||
| - INT64 → Int64 | ||
| - FLOAT32 → Float32 | ||
| - FLOAT64 → Float64 | ||
| - TIMESTAMP → UnixTimestamp | ||
|
|
||
| ### TestDbtIntegrationWorkflow | ||
| - End-to-end workflow with multiple models | ||
| - Code generation workflow with file output | ||
|
|
||
| ## Test Data | ||
|
|
||
| Tests use a pre-generated dbt manifest from `test_dbt_project/`: | ||
| - 3 models: driver_features, customer_features, product_features | ||
| - Various column types and tags for comprehensive testing | ||
| - No external dependencies (database, dbt CLI) required | ||
|
|
||
| ## Running Tests | ||
|
|
||
| Run all dbt integration tests: | ||
| ```bash | ||
| pytest sdk/python/tests/integration/dbt/ -v | ||
| ``` | ||
|
|
||
| Run specific test class: | ||
| ```bash | ||
| pytest sdk/python/tests/integration/dbt/test_dbt_integration.py::TestDbtManifestParsing -v | ||
| ``` | ||
|
|
||
| Run specific test: | ||
| ```bash | ||
| pytest sdk/python/tests/integration/dbt/test_dbt_integration.py::TestDbtManifestParsing::test_parse_manifest_metadata -v | ||
| ``` | ||
|
|
||
| ## Dependencies | ||
|
|
||
| Required: | ||
| - `dbt-artifacts-parser>=0.6.0` - For parsing dbt manifest files | ||
| - `feast` - Core Feast SDK with all data source types | ||
|
|
||
| ## CI/CD | ||
|
|
||
| Tests run in GitHub Actions: | ||
| - `.github/workflows/dbt-integration-tests.yml` - Dedicated dbt test workflow | ||
| - `.github/workflows/unit_tests.yml` - As part of general unit tests | ||
|
|
||
| ## Related Code | ||
|
|
||
| - `feast/dbt/parser.py` - dbt manifest parser | ||
| - `feast/dbt/mapper.py` - dbt to Feast object mapper | ||
| - `feast/dbt/codegen.py` - Python code generator | ||
| - `feast/cli/dbt_import.py` - CLI commands for dbt import |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1 @@ | ||
| # Integration tests for dbt import functionality |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,10 @@ | ||
| """ | ||
| Conftest for dbt integration tests. | ||
|
|
||
| This is a standalone conftest that doesn't depend on the main Feast test infrastructure. | ||
| """ | ||
|
|
||
| import pytest | ||
|
|
||
| # This conftest is minimal and doesn't import the main feast conftest | ||
| # to avoid complex dependency chains for dbt-specific tests |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@copilot we should use the Makefile to install dependencies here
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Updated to use
make install-python-dependencies-ciin commit 621cefb.