Change Data Capture is a mechanism in SQL Server to capture the changes in the database for specified retain period (retention window). You can find more information here. In this example i create something like POC (Proof of Concept) to implement CDC in my data ingestion design. In few words: Tables: […]
Database Architecture
12 posts
In my projects i need to design a transactional database and since the beginning i know that it will be the source of an ETL, feeding a Data Warehouse. I add Primary Key (single column or composite) and standardized metadata columns in all the tables. The date-tracking columns: Option 1 […]
An ETL that i build recently instigated me to share the following excerpt of python code. The players in this ETL are: Apache Kafka (Source) Azure Data Factory (ETL app) Azure Databricks (Extract and Transform with Python) Azure Data Lake Storage (File storage) Cosmos DB (Destination) In this example i […]
Entity-Atribute-Value (EAV) data model gives us the flexibility to store the data in a way that we have dynamic number of: tables columns in the simplest EAV we have 3 tables: Entity Attribute Value To simplify the understanding of the model we declare: Table Entity holds the list of the […]
SSISDB is the database that is collaborating with SSIS. When we execute a package, the execution information is logged in SSISDB. If a job is scheduled, we need to know if there are errors in the executions. As a last step in SSIS package, I run stored procedure that searches […]
In the last posts we played with Change Tracking (CT): T-SQL: Change Tracking – What is this and how it works T-SQL: Change Tracking Retention Period – How Long the Changes are Kept? Let’s execute few queries and see what do we have after the retention period expires.
As we mentioned in Change Tracking – What is this and how it works, the retention period is the one that determines how long the changes are kept.
Change tracking was introduced in SQL Server 2008. This is a tool that keeps the fact that there are changes in tracked tables for a specified period of time. It doesn’t show what the changes are, it tells that a data has been changed. It answers if a change has been […]
While working on ETL, I build a list of tables and their corresponding columns that need to be created on the destination server. In this case the square brackets around SchemaName and TableName are mandatory. I started adding the square brackets with the built-in function QUOTENAME(), but soon I got […]
In some scenarios like ETL or load data into a DB, we need to insert the data into a staging table, where all the columns are String data type and on the next step to cleanse and manipulate the data. I heard a discussion about the data type of the […]