Table of Contents

• What is Data Quality Management?
• Importance of Continuous Data Quality Monitoring
• Introduction to re_data
• Integrating re_data with dbt
• Ensuring Data Accuracy

What is Data Quality Management?

Data quality refers to the extent to which information is accurate and reliable for its intended use. An organization's ability to maintain high data quality is a reflection of its effectiveness in gathering and utilizing precise information that meets its operational needs.

There are two main strategies for managing data quality. A reactive approach focuses on identifying issues only after they occur, typically addressing immediate concerns rather than long-term improvements. In contrast, a proactive approach treats data quality as an ongoing process, requiring continuous oversight of the data pipeline from inception to end-use. This proactive strategy is vital for organizations that rely heavily on data for critical operations, such as healthcare, finance, intelligence, and retail.

Importance of Continuous Data Quality Monitoring

In today's data-driven world, maintaining high data quality is crucial for organizational success. With the explosion of big data, the speed and volume of data generation have reached unprecedented levels. Effective data quality management helps uncover errors, rectify discrepancies, and ensure data accuracy. Proactively identifying and addressing poor data can mitigate or eliminate associated business risks.

A successful data management strategy encompasses four key components: monitoring, modeling, measuring, and improving. Monitoring involves detecting issues in existing data by identifying anomalies that may signal errors in data collection or storage. Modeling focuses on diagnosing these abnormalities and implementing preventive measures. Measuring tracks the impact of these adjustments, while improving is an iterative process of continuous enhancement.

Introduction to re_data

While dbt offers a documentation user interface (UI), re_data significantly elevates this experience. Let's delve into the various components involved in re_data.

The re_data UI presents four primary features: detailed ownership and type information, SQL code display and customization, and relationship visualization. While dbt's offerings are impressive, re_data enhances these capabilities further.

Another screenshot illustrates the re_data UI, which includes additional functionalities.

Key Features of re_data

For each actively monitored table, re_data calculates essential base metrics, including row count, freshness, and schema changes. Column-level metrics such as minimum, maximum, average, variance, and null percentage can also be included. If you need a specific metric that isn't available, re_data allows you to create custom metrics by adding them to the macros folder in your dbt project, following the naming convention: re_data_metric_(your_name).

Anomaly Detection Capabilities

Re_data facilitates anomaly detection, enabling you to spot unusual data patterns effortlessly using Z-scores. This statistical measure indicates how far a data point deviates from the mean. Additionally, re_data employs boxplots to characterize variations, providing insights into locality, spread, and skewness across numerical data. By calculating upper and lower bounds, you can identify outlier values.

2. re_data Notifications

The re_data notify command sends alerts whenever a new row is added to the re_data_alerts. Perhaps the most advantageous feature is that notifications can be sent directly to Slack, allowing you to monitor your data in real time.

Integrating re_data with dbt

To implement re_data, you need to add the package to your packages.yml file located in your dbt folder:

# packages.yml

packages:

• package: re-data/re_data

  version: [">=0.7.0", "<0.8.0"]

Next, install the new library in your environment using dbt deps. Configure the tables for monitoring by adding a configuration to each model's SQL file:

{{

config(

re_data_monitored=true,

re_data_time_filter='timestamp',

)

}}

select ...

If you haven't set up a dbt project yet, run: re_data init <your_project_name> to create one with the necessary files. To start calculations, execute:

re_data run --start-date <your_date> --end-date <your_date> --interval <your_interval>

This command performs calculations to create or update the data.

Ensuring Data Accuracy

To keep your data current, you must rerun the calculations. You can utilize dbt Cloud to run jobs, or employ tools like Prefect or Airflow. Personally, I use Prefect regularly, allowing me to easily add:

dbt_task(command='dbt run -m re_data', task_args={"name": "re_data monitoring"})

to my existing dbt tasks for Prefect.