| Title | Fabric Metadata-Driven Framework (FMD) Overview |
|---|---|
| Description | Discover the architecture, components, and deployment guidance for the Fabric Metadata-Driven Framework (FMD) on Fabric SQL Database. |
| Date | 07/2025 |
| Author | edkreuk |
The Fabric Metadata-Driven Framework (FMD) is a scalable, extensible solution for managing, integrating, and governing data using a metadata-driven approach on Fabric SQL Database. This article provides an overview of the FMD Framework, including its architecture, core components, workspace structure, supported data sources, and deployment guidance.
The FMD Framework enables organizations to streamline data operations by leveraging metadata to drive dynamic data pipelines and parameterized notebooks. Built on Fabric SQL Database, the framework supports secure, flexible, and modern data management at scale.
Tip
The FMD Framework is designed for rapid deployment and extensibility. You can use it out-of-the-box or customize it to meet your organization's evolving data needs.
- Comprehensive data governance: Maintain detailed metadata for improved data quality, consistency, and compliance.
- Scalability and flexibility: Seamlessly scale with organizational growth and adapt to changing data requirements.
- Streamlined data integration: Integrate diverse data sources for a unified data landscape.
- Cost efficiency: Optimize data processes and reduce redundancy to achieve cost savings.
The FMD Framework uses a modular architecture that separates data, code, and orchestration for enhanced security and manageability.
The recommended workspace structure ensures clear separation of responsibilities and access control.
| Workspace Type | Purpose | Examples |
|---|---|---|
| Data | Manage and store data | Data Landing Zone, Bronze, Silver |
| Code | Develop and manage code artifacts | Data Pipelines, Notebooks, Spark Environment, Variable Library |
| Gold | Manage and store data | Gold and Semantic Model |
| Reporting | Create reports for end users | Reports |
| Orchestration & logging | Orchestrate and monitor data operations | Fabric SQL Database, Auditing, Logging |
Note
For organizations with a Gold Layer, create a separate workspace for reports to restrict access appropriately.
The framework implements the Medallion Architecture using Lakehouses:
| Layer | Description |
|---|---|
| Data Landing Zone: | Stores raw, unstructured, and structured data with incremental loads and no enforced schema. |
| Bronze layer: | Deduplicates and structures data, applying schema for consistency. |
| Silver layer: | Maintains historical, validated data with enforced quality rules. |
| Gold layer: | Stores curated, business-ready data for analytics, including dimensions and facts tables. |
The Variable Library centralizes the management of variables used throughout the FMD Framework. It is integrated with the PL_LOAD_ALL process, allowing Variables(parameters) to be passed dynamically during execution.
Parameters are added for the ConnectionString and Database name for the SQL_FMD_FRAMEWORK. These parameters are necessary for the built-in logging. Parameters are updated according to your environment during execution of the Setup Notebook.
Currently supported variables:
key_vault_nameLakehouse_schema_enabled
Additional variables will be supported in future releases.
To deploy the default taskflow:
- Import
FMD_FABRIC_TASKFLOW.jsonfrom the Taskflow folder into your workspace. - Assign the correct artifacts to the taskflows as shown in the documentation.
The FMD Framework supports integration with the following sources:
- SQL Server (on-premises or cloud)
- Azure Data Lake Gen2
- SFTP / FTP
- Azure Data Factory
- Onelake tables and files
- Custom Notebooks
- Oracle (only with On Premises Data Gateway)
More data sources will be supported in future releases.
To get started:
- Review the FMD Framework Deployment Guide.
- Import the taskflow and configure your workspaces as recommended.
- Refer to additional resources for data model, pipelines, and logging.
Check the link below to get started on how to implement and configure Data Cleansing functions
| Resource | Description |
|---|---|
| FMD Integration Framework reference | Overview on how to add sources to the FMD Framework |
| FMD Data Model reference | Overview of the data model used in the FMD Framework |
| FMD Framework Data Pipelines | Guide to configuring data pipelines in the FMD Framework |
| Configure and load demo data | Instructions for loading demo data into the FMD Framework |
| Auditing and logging | Information on auditing and logging within the FMD Framework |
-
Fabric SQL Database limitations:
If deployment fails, verify that you haven't exceeded the allowed number of Fabric Databases in your tenant. Trial capacities are limited to three databases. -
Error handling:
If you encounter errors during deployment (for example, database creation failures), re-run the notebook or manually create the database to diagnose the issue.
We welcome contributions! To suggest improvements, open an issue or submit a pull request.
If opening a pull request, please follow these steps:
- Fork the repository.
- Create a feature branch.
- Commit your changes.
- Push to the feature branch.
- Create a pull request and add documentation on what you have changed.
This project is licensed under the MIT License - see the LICENSE file for details.
Contributors:
Erwin de Kreuk