Codestin Search App

NOTE: Datafold needs catalog-level permissions in your Databricks workspace to read and write table data, stage cross-database diff transfers, query system tables, and deploy migration bundles. You will need workspace admin access to create a service principal and grant the required permissions.

If your workspace storage account is firewalled to block public access, see Securing Connections → Azure. Query results over 1 MB are downloaded directly from workspace storage via Cloud Fetch, which typically requires its own private endpoint in addition to the SQL warehouse connection.

Steps to complete:

If you plan to run Datafold Migration Agent inference on your own Databricks model serving endpoints, there are two extra prerequisites — see Databricks AI for migrations.

Create a service principal and configure authentication

Create a dedicated service principal for the Datafold integration. This is the identity Datafold will use to connect to your workspace.

Go to Settings → Identity and access → Service principals
Click Add service principal and give it a name (e.g., datafold)
Select the service principal, go to the Secrets tab, and click Generate secret
Save the Client ID and Secret — the secret is only shown once

OAuth secrets are valid for up to 730 days. You can have a maximum of 5 active secrets per service principal. Rotate secrets before expiry to avoid connection interruptions.

Datafold also supports Personal Access Tokens as an alternative authentication method. PATs are considered legacy by Databricks — see the Databricks authentication documentation for details.

Retrieve SQL warehouse connection details

Navigate to SQL Warehouses under the SQL section in the left sidebar. Choose the preferred warehouse and copy the following fields from its Connection Details tab:

Server hostname
HTTP path

You also need to grant the service principal access to the SQL warehouse:

On the warehouse page, click the Permissions tab
Add the service principal and grant Can Use permission

Grant permissions

Run the following SQL statements to grant Datafold the permissions it needs. Replace <catalog_name> and <service_principal_id> with your values. Replace <schema_name> with the schema where you want to store the DMA bundle volume (e.g., default).

The <service_principal_id> is the application ID (also called Client ID) of your service principal. In Databricks SQL, service principal identifiers must be enclosed in backticks.

-- Catalog access
GRANT USE CATALOG ON CATALOG <catalog_name> TO `<service_principal_id>`;
GRANT USE SCHEMA ON CATALOG <catalog_name> TO `<service_principal_id>`;
GRANT SELECT ON CATALOG <catalog_name> TO `<service_principal_id>`;
GRANT CREATE TABLE ON CATALOG <catalog_name> TO `<service_principal_id>`;
GRANT MODIFY ON CATALOG <catalog_name> TO `<service_principal_id>`;

-- Temporary schema for Datafold
CREATE SCHEMA IF NOT EXISTS <catalog_name>.datafold_tmp;
GRANT CREATE VOLUME ON SCHEMA <catalog_name>.datafold_tmp TO `<service_principal_id>`;

-- System tables access
GRANT USE CATALOG ON CATALOG system TO `<service_principal_id>`;
GRANT USE SCHEMA ON CATALOG system TO `<service_principal_id>`;
GRANT SELECT ON CATALOG system TO `<service_principal_id>`;

-- UC Volume for DMA bundle deployment
CREATE VOLUME IF NOT EXISTS <catalog_name>.<schema_name>.datafold_bundles;
GRANT READ VOLUME ON VOLUME <catalog_name>.<schema_name>.datafold_bundles TO `<service_principal_id>`;
GRANT WRITE VOLUME ON VOLUME <catalog_name>.<schema_name>.datafold_bundles TO `<service_principal_id>`;

(Optional) Additional grant for migrations

If you use Datafold to run migrations (its agents deploy and run Databricks Asset Bundles on this connection), also grant permission to create schemas. Each migration run materializes into a fresh schema created at deploy time. This is not needed for data diffing, monitoring, or lineage.

GRANT CREATE SCHEMA ON CATALOG <catalog_name> TO `<service_principal_id>`;

Authentication for migrations depends on the method. A service principal using M2M OAuth needs no additional token scope. If you use a Personal Access Token (legacy) instead, it must have the all-apis scope, because bundle deployment uploads files via the Databricks workspace-files API, which Databricks gates on all-apis even when narrower scopes are present.

Databricks AI for migrations

This section applies only if the Datafold Migration Agent will run inference on your Databricks model serving endpoints. Skip it for data diffing, monitoring, lineage, and CI, and for migrations that use Datafold-managed inference or a different LLM provider.

Databricks-served inference reuses the data connection above — there is no second service principal, endpoint URL, or API key to create. Two things still need arranging on the Databricks side.

Grant Can Query on the serving endpoints

Serving endpoint access is a separate ACL from SQL warehouse and Unity Catalog access, so the grants above do not cover it. For each endpoint you approve for inference:

Navigate to Serving under the Machine Learning section in the left sidebar
Open the endpoint and click its Permissions tab
Add the Datafold service principal and grant Can Query

Inference requires the connection to authenticate as a service principal — M2M OAuth, a Personal Access Token, or Azure Entra ID. Per-user OAuth connections cannot be used, because the agent runs without a user context.

If your workspace serves pay-per-token foundation models as Unity Catalog model services (system.ai.*), Datafold lists them from the Unity Catalog model services API. When the service principal cannot read that API, those models are missing from the model picker; endpoints already configured keep serving inference.

Raise the AI Gateway throughput limit

The default AI Gateway limit is roughly 200,000 input tokens per minute, which throttles a real migration heavily. Ask your Databricks account team to raise it for the models you approve — request at least 2–3M input tokens per minute. Tier 2 allows up to 10M.

Raising the limit is free: the default is a guardrail, not a paid tier. Databricks typically takes a few days to apply the change, so request it early.

Once Can Query is granted and the limit is raised, connect the endpoints in Datafold by adding the Databricks LLM provider integration.

Configure in Datafold

Select M2M OAuth / Service Principal (Recommended) as the authentication method and fill in the following fields:

Field	Description
Connection name	A name for this data connection within Datafold
Host	The Server hostname from the warehouse Connection Details tab
HTTP path	The HTTP path from the warehouse Connection Details tab
Authentication method	Select M2M OAuth / Service Principal (Recommended)
Client ID	The Client ID of the service principal
Client Secret	The secret generated in the authentication step
Catalog	The default catalog name (e.g., `hive_metastore` or your Unity Catalog name)
Schema path for temporary tables	The temp schema as `<catalog_name>.datafold_tmp` (e.g., `demo.datafold_tmp`)

Click Create. Your data connection is ready!

Validate your setup

Run these queries to verify that permissions are configured correctly:

-- Check all grants for the service principal
SHOW GRANTS TO `<service_principal_id>`;

-- Verify warehouse access (run from the SQL warehouse the service principal will use)
SELECT 1;

-- Verify temp schema access
SHOW TABLES IN <catalog_name>.datafold_tmp;

-- Verify system table access (should return a row)
SELECT * FROM system.query.history LIMIT 1;

-- Verify volume access
LIST '/Volumes/<catalog_name>/<schema_name>/datafold_bundles';

-- Verify volume creation in the temp schema (run as the service principal)
CREATE VOLUME IF NOT EXISTS <catalog_name>.datafold_tmp.datafold_permission_check;
DROP VOLUME IF EXISTS <catalog_name>.datafold_tmp.datafold_permission_check;

INTRODUCTION

DATA PLATFORM MIGRATIONS

DATA DIFFS

CI/CD TESTING

DATA MONITORS

DATA EXPLORER

INTEGRATIONS

DEPLOYMENT

SECURITY

SUPPORT

Databricks

Create a service principal and configure authentication

Retrieve SQL warehouse connection details

Grant permissions

(Optional) Additional grant for migrations

Databricks AI for migrations

Grant Can Query on the serving endpoints

Raise the AI Gateway throughput limit

Configure in Datafold

Validate your setup

​Create a service principal and configure authentication

​Retrieve SQL warehouse connection details

​Grant permissions

​(Optional) Additional grant for migrations

​Databricks AI for migrations

​Grant Can Query on the serving endpoints

​Raise the AI Gateway throughput limit

​Configure in Datafold

​Validate your setup

Create a service principal and configure authentication

Retrieve SQL warehouse connection details

Grant permissions

(Optional) Additional grant for migrations

Databricks AI for migrations

Grant Can Query on the serving endpoints

Raise the AI Gateway throughput limit

Configure in Datafold

Validate your setup