Blog Masonry

Welcome to My Blog

Here is a masonry blog layout with no sidebar

The Epic list of features

Here we dump all the features we have built to date as we build them. We are currently working on catching the documentation/list up with what we have already built, while we are still adding even more features.  So its highly likely there are more features in the...

Persistent Staging Area with ODE

In ODE version 5 we have added SSIS (MS SQL Server Integration Services) data loads and CDC (Change Data Capture) loads.This functionality is enough to build Persistent Staging Area with ODE. PSA is a copy of the source database, but all the changes to all the records...

Create a Data Vault with ODE v5

ODE is available as a download from GitHub, which has the instructions for implementing the Configuration Database. Configuration is a heart and brain of ODE Data Vault. Other organs include stage and vault databases. From ODE version 5 these databases should have a...

Preparation for Data Vault Modeling

Pre-requisite for creating Data Vault ensembles and links is an ODE Config database installed (refer here for ODE installation instructions), stage and vault databases created and have an audit table in there (refer here for the Data Vault database installation...

SSIS Packages Data Load Type

In ODE version prior to 5 you could only load the data into Data Vault from the source database on the same server where the configuration database is implemented. ODE version 5 could get the data into the Data Vault from another server via SSIS (Microsoft SQL Server...

Building a Data Vault with ODE v5

Data Vault with ODE Data Vault is a Data Warehousing methodology. Learn more about it here. ODE is a config-driven engine for building and managing Data Vaults. At the moment ODE only works with MS SQL Server. Refer here for the latest ODE installation. Modelling a...

Satellite Change Data Capture Functions

In the Data Vault satellite is a container for business concept's attributes and it has a SCD type 2 structure for storing the data. This means that history of record changes is stored in the satellite alongside with its current state. One have to use a set of...

Release Config with SSIS package

ODE configuration release mechanism includes a step of writing into the file. We found that some organisations limit developer security permissions, making this function unavailable for them. We have implemented this SSIS package to be used in this case. Configuration...

How To Monitor and Troubleshoot Data Loads?

How to Monitor? You can set up a scheduled email to be sent to an administrator or to a group of administrators responsible for monitoring the Data Vault loads. ODE sends an email on procedure [dv_scheduler].[dv_report_email_manifest_progress] execution. SQL Server...

How To Version Business Rules in ODE v4

ODE attaches all Processing Rules to “Source Tables”. Each Source Table can have a number of Source Table Versions, only one of which may be "Current". The rest are retained for history and tracking purposes. As data is loaded into the Vault, the objects – Hubs, Links...

How To Create a Calculated Field in ODE v4

In ODE v4 we have introduced a new feature, we call it functions. Functions could be used in satellites to calculate attributes on the fly (during the load actually) from other satellite's columns, i.e. calculated columns. At the moment most of the load logic is...

How To Implement Delta Load in ODE v4

ODE supports two types of data loads, Full load and Delta load. The difference is there are no deletions in Delta mode. So that Delta loads could be used for quick loads of small datasets as opposed to long overnight Full loads where full datasets are checked for new,...

Metrics Vault

Metrics Vault is a Data Vault for the data about the data. Metrics Vault is managed by ODE configuration database. It could be added to the Configuration at any time, and it will start collecting record stats, Vault data load time and record counts. Also Data Vault...

How To Implement Same-As Link

Same-as and hierarchical links represent recursive relationships. Same-as link let us know that two records in a Hub table both represent the same core concept. Hierarchical link let us know that there is a form of hierarchy  between two records in the same Hub table....