Change Data Capture is a mechanism in SQL Server to capture the changes in the database for specified retain period (retention window). You can find more information here. In this example i create something like POC (Proof of Concept) to implement CDC in my data ingestion design. In few words: Tables: […]
Backend
12 posts
In my projects i need to design a transactional database and since the beginning i know that it will be the source of an ETL, feeding a Data Warehouse. I add Primary Key (single column or composite) and standardized metadata columns in all the tables. The date-tracking columns: Option 1 […]
All the posts about Fabric Framework are under tag "fabric-framework". After we create the Fabric Workspace, you need to create the objects that will run the automation. Upload folder “one_time_exec” and run pipeline “pl_one_time_exec_master” Diagram This pipeline has to be ran once. It calls notebooks in order: Configuration File You […]
All the posts about Fabric Framework are under tag "fabric-framework". I started working on a Fabric Framework. in few words this is: Diagram Repository You can find an early version of the source code in GitHub. Dataflow The starting point is an hourly scheduled pipeline “pl_master” that: Folders and Files […]
Entity-Atribute-Value (EAV) data model gives us the flexibility to store the data in a way that we have dynamic number of: tables columns in the simplest EAV we have 3 tables: Entity Attribute Value To simplify the understanding of the model we declare: Table Entity holds the list of the […]
In this example we’ll create SQL Server Common Language Runtime (CLR) to read/write from/to REST API. Technology used: Visual Studio 2019 (C# Windows .NET Class Library) SQL Server 2019 (on premises) Authentication: Basic (base64 encoded Username:Password) Requests: HTTP SQL GET SELECT POST INSERT Visual Studio 2019 Create new C#, Windows, […]
In this example we create ADF pipeline that extracts from SQL Server and saves in CSV files in Data Lake. The point is just to demonstrate the logic so you can edit it as you need. The extract is driven by SQL Server table with all the parameters for a […]
SSISDB is the database that is collaborating with SSIS. When we execute a package, the execution information is logged in SSISDB. If a job is scheduled, we need to know if there are errors in the executions. As a last step in SSIS package, I run stored procedure that searches […]
In the last posts we played with Change Tracking (CT): T-SQL: Change Tracking – What is this and how it works T-SQL: Change Tracking Retention Period – How Long the Changes are Kept? Let’s execute few queries and see what do we have after the retention period expires.
As we mentioned in Change Tracking – What is this and how it works, the retention period is the one that determines how long the changes are kept.