#data engineering #dbt #pipelines Lint, Test, Deploy: Building a dbt CI Pipeline with SQLFluff Style Checks Enhancing dbt Development with Automated PR Validation in Azure PipelinesIn the fast-paced world of data engineering, maintaining high-quality, consistent code is crucial. As teams grow and projects scale, the need for thorough validation of pull requests (PRs) becomes even more important. This ensures minimal errors, enforces best practices, and guarantees
#data engineering #data analysis #data science Snowflake Notebooks: What I Love and What I Hope to See Next I recently explored Snowflake Notebooks, which have recently became generally available, and found it quite interesting. In this article, I’ll cover the following topics: What Snowflake Notebook is.4 key features which I love in notebooks.3 additional features I’d like to see.Ideas for integrating it into
#data engineering #evaluating data sources Establishing a Framework for Evaluating Data Sources Quantitatively Data has become one of the most critical assets in any business setting with significant influence on a business' success of failure. Ease of access to quality data enables business stakeholders to make accurate data-driven decisions and, depending on the timeliness of the data, gives an edge over competitors to
#data engineering #data quality #snowflake Text Similarity Analysis at Scale Think of your data as a giant puzzle, where the pieces often wear clever disguises. As a data engineer, you're often left trying to put the pieces back together, but there's a catch - you have thousands of pieces to sift through. Here's where text similarity can be a handy
#data engineering #data quality Data Quality Testing - The What, How, Why ... Tools and Practices This is a long read, so let me first tell you what exactly you’ll be getting out of reading this article. Understanding the importance of data quality and data testingExplore the main differences between two of the most prominent data testing tools (Great Expectations and DBT) with a data
