Data Warehousing Project
2025-04-14 23:56
| #### Tags: data-engineering |
|---|
Project Epics
| DWH Epics | Status |
|---|---|
| Requirement Analysis | Not Started |
| Design Data Architecture | Not Started |
| Project Initialization | Not Started |
Project Tasks
| DWH Epics | Tasks | Status |
|---|---|---|
| Requirement Analysis | [[##### Analyse & Understand the Requirement]] | Not Started |
| Design Data Architecture | Choose Data Management Approach | Done, going with DW |
| Design Data Architecture | [[##### Designing of The Layers]] | Not Started |
| Design Data Architecture | [[##### Draw the Data Architecture]] | Done |
| Project Initialization | Create git repo & prepare the structure | Not Started |
| Project Initialization | Create DB & Schema | Not Started |
| Project Initialization | Project Naming Convenstion | Not Started |
Analyse & Understand the Requirement
Draw the Data Architecture
For this project, we will go with Medallion Architecture
Designing of The Layers
| Bronze Layer | Silver Layer | Gold Layer | |
|---|---|---|---|
| Definition | Raw, unprocessed data as-is from sources | Clean & Standardized Data | Business-Ready Data |
| Objective | Traceability & Debugging | (Intermediate Layer) Prepare Data for Analysis | Provide data to be consumed by Downstream for reporting and Analytics |
| Object Types | Tables | Tables | Views |
| Load Method | Full Load (Truncate & Insert) |
Full Load (Truncate & Insert) |
None |
| Data Transformation | None (as-is) | - Data Cleaning - Data Standardization - Data Normalization - Derived Columns - Data Enrichment |
- Data Integration - Data Aggregation - Business Logic & Rules |
| Data Modeling | None (as-is) | None (as-is) | - Start Schema - Aggregated Objects - Flat Tables |
| Target Audience | Data Engineer | - Data Analysts - Data Engineers |
- Data Analysts - Business Users |
Note: Best practices say, for the last layers in your data warehouse make it View, as it gives a lot of dynamic to build it.