Change Data Capture is a mechanism in SQL Server to capture the changes in the database for specified retain period (retention window). You can find more information here. In this example i create something like POC (Proof of Concept) to implement CDC in my data ingestion design. In few words: Tables: […]
ETL
9 posts
In my projects i need to design a transactional database and since the beginning i know that it will be the source of an ETL, feeding a Data Warehouse. I add Primary Key (single column or composite) and standardized metadata columns in all the tables. The date-tracking columns: Option 1 […]
An ETL that i build recently instigated me to share the following excerpt of python code. The players in this ETL are: Apache Kafka (Source) Azure Data Factory (ETL app) Azure Databricks (Extract and Transform with Python) Azure Data Lake Storage (File storage) Cosmos DB (Destination) In this example i […]
SSISDB is the database that is collaborating with SSIS. When we execute a package, the execution information is logged in SSISDB. If a job is scheduled, we need to know if there are errors in the executions. As a last step in SSIS package, I run stored procedure that searches […]
In the last posts we played with Change Tracking (CT): T-SQL: Change Tracking – What is this and how it works T-SQL: Change Tracking Retention Period – How Long the Changes are Kept? Let’s execute few queries and see what do we have after the retention period expires.
As we mentioned in Change Tracking – What is this and how it works, the retention period is the one that determines how long the changes are kept.
Change tracking was introduced in SQL Server 2008. This is a tool that keeps the fact that there are changes in tracked tables for a specified period of time. It doesn’t show what the changes are, it tells that a data has been changed. It answers if a change has been […]
While working on ETL, I build a list of tables and their corresponding columns that need to be created on the destination server. In this case the square brackets around SchemaName and TableName are mandatory. I started adding the square brackets with the built-in function QUOTENAME(), but soon I got […]
In a lot of situations we may not have permissions to use SQL Server Agent for automations of our back-end processes. I built a run-around that uses a Windows Scheduled Task, SQLCMD and Stored Procedure to run task. Work flow: 1. Windows Scheduled Task calls a .bat file at specific […]