Change Data Capture is a mechanism in SQL Server to capture the changes in the database for specified retain period (retention window). You can find more information here. In this example i create something like POC (Proof of Concept) to implement CDC in my data ingestion design. In few words: Tables: […]
Database
52 posts
In my projects i need to design a transactional database and since the beginning i know that it will be the source of an ETL, feeding a Data Warehouse. I add Primary Key (single column or composite) and standardized metadata columns in all the tables. The date-tracking columns: Option 1 […]
Just a quick note: pay attention when you hash different data types. The result is different. Also the lower/upper case: Keep it simple :-)
The MERGE statement was introduced in SQL Server 2008 and the developers embraced it immediately. Later a feedback came and the statement was criticized as not very efficient. Aaron Bertrand collected some links about this here. MERGE statement anatomy The MERGE statement joins ‘source’ and ‘target’ tables and runs INSERT, […]
Set-based Database Development As database developer, we need to switch our mindsets to “Set-based thinking”. In simple words we need to manipulate the data as a “set”. An example of set-based database development is: JOIN tables Aggregate the result Calculate the portion (%) of a row to the total of […]
An ETL that i build recently instigated me to share the following excerpt of python code. The players in this ETL are: Apache Kafka (Source) Azure Data Factory (ETL app) Azure Databricks (Extract and Transform with Python) Azure Data Lake Storage (File storage) Cosmos DB (Destination) In this example i […]
Let say that we need the following transformation in SSRS report: One of the tools to do this is List. I will explain only the tricky part and will give the .rdl in the end. Drag-and-drop List in the report body: Right click the top left corner of the list […]
XML is dead, long live JSON! Even before i knew JSON, i thought “XML contains more metadata than data”. Today in my new development, i prefer JSON and avoid using XML, but in some cases we need to fight with XML too. In this quick example i show how to […]
Entity-Atribute-Value (EAV) data model gives us the flexibility to store the data in a way that we have dynamic number of: tables columns in the simplest EAV we have 3 tables: Entity Attribute Value To simplify the understanding of the model we declare: Table Entity holds the list of the […]