The dropdown and multiselect components are among the most fundamental building blocks in our design system. In a data-heavy application where users constantly filter, search, and select from large datasets, these components need to be performant, accessible, and flexible enough to handle diverse use cases across our product.

Component History

Our dropdown and multiselect components have evolved through several distinct phases, each representing a different approach to solving the same fundamental problem.

Phase 1: In-House Implementation

The first phase saw us building dropdown components completely in-house, from scratch. This initial implementation served us well for basic use cases, but as our product scaled and requirements became more complex, the limitations of this approach became evident.

Building everything from scratch meant we owned every piece of logic, state management, keyboard navigation, accessibility, filtering, virtualization, and portal rendering. While this gave us complete control, it also meant we were responsible for maintaining and evolving all of this complexity ourselves.

As new requirements surfaced, support for large lists, custom filtering, add-item functionality, tree structures, and various styling variants, each feature addition became increasingly risky and time-consuming. The component grew into a monolithic structure where changes in one area could have unexpected consequences in another.

Phase 2: Third-Party Dependency

The second phase took us to the opposite end of the spectrum: adopting a third-party solution (react-select) to satisfy our dropdown needs. This approach initially seemed promising. we could leverage a mature, well-tested library and focus on customization rather than core functionality.

However, we quickly discovered that third-party packages, while powerful, came with their own set of constraints. Customization was limited to what the library exposed through its API. When we needed behaviors that weren't supported out of the box, like specific keyboard navigation patterns, custom item rendering, or unique filtering logic. we found ourselves fighting against the library's architecture rather than building on top of it.

The component became a source of friction. Tasks that should have been straightforward required workarounds, and the bundle size grew with features we didn't always need. Most importantly, we lost the ability to evolve the component in directions that matched our specific product needs.

Phase 3: Hybrid Approach

This realization led us to our current approach: a hybrid solution that combines the best of both worlds. We leverage a headless utility library for the heavy lifting, state management, accessibility, keyboard navigation, and complex interaction patterns, while maintaining full control over the UI and UX through our own design system components.

We chose Downshift as our headless utility, wrapping it with our design system's Input, Label, and other primitives to create a component that feels native to our design language while benefiting from battle-tested interaction logic.

Motive: Why Rebuild?

As the product continued to scale, the existing components started hitting their limits. Adding new features became risky and time-consuming. Fixing bugs often turned into deep investigations. The dropdown was used extensively across the product, and the lack of confidence in such a critical component began to slow the team down.

Specific pain points included:

• Inflexible customization: Third-party solutions couldn't accommodate our unique design requirements
• Performance issues: Large lists caused performance problems without proper virtualization
• Maintenance burden: The in-house implementation required constant attention and refactoring
• Inconsistent behavior: Different implementations across the codebase led to inconsistent user experiences

Why Downshift?

The options in the headless dropdown space are relatively limited, but several libraries have emerged as strong contenders. We took the time to thoroughly evaluate multiple alternatives before committing to such a significant change.

Key Factors in Our Decision

• Accessibility-first: Built with ARIA attributes and keyboard navigation as core features
• Headless architecture: Complete control over UI rendering without opinionated styling
• Flexible state management: Exposes state and state change handlers for custom logic, including a powerful stateReducer pattern
• Active maintenance: Regular updates and community support
• Small bundle size: Focused on logic, not UI—critical for our performance requirements
• Composable hooks: useCombobox and useMultipleSelection work seamlessly together for multiselect scenarios

The tool checked all our boxes, and we decided it was time to rebuild our dropdown and multiselect components.

Implementation and Design Decisions

Building design-system components, especially ones as complex as dropdowns and multi-selects is not just about implementation details or elegant abstractions. It's about offering a clear, self-explanatory interface and a developer experience that enables progress instead of forcing repeated reconsideration of fundamental decisions.

The design was guided by three core principles: modularity, separation of concerns, and composability.

Architecture: Base and Composition

Our new architecture separates concerns into distinct layers:

Dropdown/
├── Dropdown.tsx           # Public API, composes base with UI elements
├── DropdownBase/
│   └── Dropdown.base.tsx  # Headless logic wrapper (Downshift integration)
└── TaggedListItem/        # Specialized list item variant

Multiselect/
├── Multiselect.tsx        # Public API
├── Multiselect.base.tsx   # Core logic (Downshift + useMultipleSelection)
├── MultiselectInput.tsx   # Input with tag display
└── MultiselectList.tsx    # Menu with virtualization

The DropdownBase component handles all the interaction logic using Downshift's useCombobox hook. It manages:

• Input value and filtering
• Menu open/close state
• Keyboard navigation (arrow keys, enter, escape)
• Item highlighting and selection
• Virtualization for large lists
• Portal rendering for modals

The Dropdown component composes the base with our design system primitives:

• Input or CompactInput for the trigger
• Label for list items
• Custom styling through our BEM utility system
• Loading states and empty states
• Text highlighting: Search terms are automatically highlighted in list items

This separation allows developers to use the base component directly if they need custom UI, or use the composed Dropdown for standard use cases.

Virtualization for Performance

One of the key improvements in the new implementation is virtualization using @tanstack/react-virtual. For dropdowns with hundreds or thousands of options, rendering all items at once would cause significant performance issues.

The virtualizer only renders the visible items plus a small buffer, dramatically improving performance for large lists while maintaining smooth scrolling and keyboard navigation.

Floating UI for Positioning

We use @floating-ui/react for intelligent menu positioning. This ensures the dropdown menu:

• Stays within viewport bounds
• Adjusts position based on available space
• Matches the width of the input trigger
• Works correctly inside modals and scrollable containers

Multiselect: Extending the Pattern

The multiselect component builds on the same foundation but uses Downshift's useMultipleSelection hook in combination with useCombobox. This allows us to:

• Manage multiple selected items as an array
• Display selected items as removable tags
• Filter out selected items from the dropdown list
• Support keyboard navigation between tags (left/right arrows)
• Handle backspace/delete to remove items

The component maintains the same separation of concerns, with MultiselectInput handling the tag display and MultiselectList managing the dropdown menu.

Auto-Complete Mode

One notable feature of the multiselect is the auto-complete mode. When enabled, the dropdown menu only opens after the user has typed at least two characters. This pattern is particularly useful for scenarios with large datasets where showing all options upfront would be overwhelming or perform poorly. It provides a more focused, search-first experience that guides users to type before browsing, reducing cognitive load and improving performance for large option lists.

Customization Through Composition

Rather than trying to anticipate every possible use case through props, we designed the components to be composable. Developers can:

• Provide custom list item renderers with custom Type
• Implement custom filtering logic
• Add "create new item" functionality
• Use the base components directly for completely custom UIs
• Extend through TypeScript generics for type-safe item handling

This approach keeps the API surface manageable while providing escape hatches for edge cases.

Releasing Strategy

Creating reliable and robust components was only part of the challenge. For high-impact components used across multiple applications, the question of how to release them to production without disrupting delivery or degrading the user experience became equally important.

Our solution was two-fold: incremental rollout and gradual deprecation.

Phase 1: Parallel Implementation

We built the new components alongside the existing ones, ensuring feature parity for common use cases. This allowed us to validate the new architecture without disrupting existing functionality.

Phase 2: Gradual Migration

We began migrating existing implementations incrementally

This gradual approach allowed us to:

• Validate the new components in production
• Gather feedback from developers
• Fix issues as they arose
• Build confidence in the new architecture

Phase 3: Deprecation

Once we had sufficient confidence and adoption, we marked the old components as deprecated.

The old-dd-base folder remains in the codebase for the TreeDropdown component, which still uses the legacy base components. This is a known technical debt that we plan to address by migrating TreeDropdown to the new architecture.

Workarounds and Customizations

1. Custom State Reducer

Purpose: Preserves highlighted index on input click instead of resetting it.

Workaround:

function stateReducer<T>(state, actionAndChanges) {
  switch (type) {
    case useCombobox.stateChangeTypes.InputClick:
      return {
        ...changes,
        isOpen: true,
        highlightedIndex: state.highlightedIndex >= 0
          ? state.highlightedIndex
          : changes.highlightedIndex,
      };
    default:
      return changes;
  }
}

2. Show Full List on Menu Open

Purpose: Override Downshift's default to show full options list when menu opens, not filtered list.

Workaround:

onIsOpenChange: ({ isOpen, selectedItem }) => {
  setFilteredList(optionList); // Reset to full list

  // rest of the code
}

3. Manual Filtering Logic

Purpose: Custom client-side filtering (Downshift has no built-in filtering).

Workaround:

onInputValueChange: ({ inputValue }) => {
  // custom filtering handler
}

4. Custom Item Identification

Purpose: Handle duplicate display values by using a separate identifier field.

Workaround:

const idField = identifierField || fieldToDisplay;

// Use idField for all item matching
const index = optionList.findIndex(op => op[idField] === selectedItem[idField]);

5. Virtualization Integration

Purpose: Render large lists efficiently using @tanstack/react-virtual.

Workaround:

const rowVirtualizer = useVirtualizer({
  count: filteredList.length,
  getScrollElement: () => listRef.current,
  estimateSize: () => 40,
});

onHighlightedIndexChange: ({ highlightedIndex }) => {
  rowVirtualizer.scrollToIndex(Number(highlightedIndex));
}

6. Floating UI Integration

Purpose: Intelligent menu positioning with viewport boundary detection.

Workaround:

const { refs, floatingStyles } = useFloating({
  whileElementsMounted: autoUpdate,
  middleware: [sizeMW({
    apply({ elements, rects }) {
      elements.floating.style.width = `${rects.reference.width}px`;
    },
  })],
});

7. Portal Rendering

Purpose: Render menu in portal for modals and z-index management.

Workaround:

{portalTarget ? (
  createPortal(
    <div ref={refs.setFloating} style={{ ...floatingStyles, zIndex: 9999 }}>
      {menuElementRenderer}
    </div>,
    portalTarget
  )
) : (
  <>{menuElementRenderer}</>
)}

8. Custom Tab Key Handling

Purpose: Select highlighted item before closing menu on Tab press.

Workaround:

const keydownHandler = (event) => {
  if (event.key === 'Tab' && isOpen && optionList.length && highlightedIndex >= 0) {
    const isTheSameItem = selectedItem &&
      filteredList[highlightedIndex][idField] === selectedItem[idField];

    if (!isTheSameItem) {
      setSelectedItem(filteredList[highlightedIndex]);
    }
    closeMenu();
  }
};

9. Auto-Open Menu on Focus

Purpose: Open menu automatically when input receives focus.

Workaround:

getInputProps({
  onFocus: () => {
    if (!isOpen) openMenu();
  },
})

Result: Beyond the Code

The most meaningful outcome was a shift in how the team related to the dropdown and multiselect components. Tasks that were once dreaded became routine, as confidence in the components grew.

Performance Improvements

• Virtualization: Large lists (1000+ items) now render smoothly
• Bundle size: Reduced by leveraging headless utilities
• Render performance: Optimized re-renders through proper React patterns

Developer Experience

• Type safety: Full TypeScript support with generics
• Clear API: Self-documenting prop names and structure
• Flexibility: Easy to extend without modifying core logic
• Consistency: Same patterns across dropdown and multiselect

User Experience

• Accessibility: WCAG 2.1 AA compliant out of the box
• Keyboard navigation: Full keyboard support for power users
• Visual feedback: Clear loading, empty, and error states
• Responsive: Works correctly in modals, scrollable containers, and mobile

That shift removed friction and allowed the dropdown and multiselect to function as what design-system components should be: reliable primitives the team can build on with confidence.

Looking Forward

The hybrid approach has proven successful, but the work is never done. We continue to:

• Monitor usage: Track component adoption and identify pain points
• Gather feedback: Listen to developers using the components daily
• Iterate: Add features as needed while maintaining the core architecture
• Migrate: Gradually move remaining legacy implementations to the new components

The table is one of the core components in our product, especially in a data‑heavy application, where multiple user flows depend on it being both performant and predictable.

Component history

The table has always been an evolving component in our library, going through several phases and heading in multiple directions over time.

The first phase of this journey relied entirely on third‑party packages to satisfy our table needs. Two different packages were introduced, each serving a distinct purpose.

The first was built using react-data-table-component and served as the first reusable table component in our design system. Shortly after, another table was built using handsontable, with the sole purpose of providing a reusable way to create spreadsheet‑like experiences.

The second phase took us to the opposite end of the spectrum: building the component completely in‑house, from scratch.

The result of this iteration -Table (V1)- served us well for a couple of years. However, as the scale we operated at increased rapidly, more complex requirements began to surface, and the limitations of this approach became evident.

This ushered in the era of the enhanced Table (V2), built using the team’s accumulated experience and by leveraging newer tools and approaches from the design systems and component‑library ecosystem. This time, we adopted a hybrid approach, combining third‑party utilities with in‑house customization. We’ll dive into this in more detail later.

Motive: Why Rebuild?

As the product continued to scale, the existing component -DS/Table (V1) -started to hit its limits. Adding new features became risky and time‑consuming, and fixing bugs often turned into deep, time‑draining investigations.

At that point, this was no longer just a technical inconvenience, it was a scalability risk. The table was used extensively across the product, and the lack of confidence in such a critical component began to slow the team down.

Exploration and Alternatives

We began exploring different approaches and carefully weighing their trade‑offs. Given the complexity and scale of our requirements, choosing the wrong direction would have long‑term consequences, so this decision was made deliberately and collectively as a team.

Naturally, the options spanned the same spectrum we had already experienced: on one end, total dependency on third‑party libraries; on the other, a fully in‑house implementation. Neither extreme proved to be the right answer for us.

Total dependency on third‑party packages represented the phase of “just making it work.” At that stage, concerns such as bundle size, customization flexibility, design consistency, and long‑term maintainability were not top priorities. The focus was purely on utility.

As it became clear that the third‑party approach was too rigid and could not fully satisfy our needs, we were pushed toward the other extreme: building everything in‑house. This came with an entirely different scope and level of complexity. Creating such a component from scratch required covering all the foundational pieces of a reusable design‑system table.

We had to build our own state management, virtualization, sorting, and filtering logic. This led to a large and complex component structure, where we owned every piece of logic and were responsible for ensuring that all of it worked correctly together.

Eventually, we realized that we didn’t need to reinvent the wheel when it came to core logic and algorithms, but we did need to own the steering wheel when it came to UI and UX. This realization led us to adopt a hybrid approach: relying on a third‑party utility for the heavy lifting, while wrapping it with our own design‑system components.

Based on our experience with both extremes, we aligned on this approach and chose TanStack Table as our headless table utility.

Why TanStack Table?

The options in the headless table space are relatively limited. The TanStack ecosystem has emerged as a strong reference point in this area, with tools such as TanStack Query playing a significant role in shaping modern frontend development practices over the past few years.

Nevertheless, we took the time to thoroughly evaluate the utility before committing to such a significant change. We reviewed its feature set to ensure that our core use cases were covered out of the box, identified potential gaps that might require additional work, and assessed community adoption and real‑world production usage. We also reviewed the source code itself to validate its quality.

The tool checked all our boxes, and we decided it was time to rebuild our table.

Implementation and Design Decisions

Building a design-system component -especially one as complex as a table- is not just about implementation details or elegant abstractions. It’s about offering a clear, self-explanatory interface and a developer experience that enables progress instead of forcing repeated reconsideration of fundamental decisions.

Drawing from our previous experience, we knew we needed to strike a balance between a robust yet rigid component that solved our current use cases, and a flexible one that could evolve over time without introducing regressions.

The design was guided by three core principles: modularity, separation of concerns, and opt‑in features.

Since the heavy logic was handled by TanStack Table, our focus shifted toward structuring the component in a way that aligned with these principles.

The 📁 components folder contains the complete collection of UI building blocks that make up the table. Developers can pick what suits their needs without worrying about inconsistent design or duplicated logic.

The 📁 types folder holds the TypeScript types required for the component to function with full type safety—both internally (for internal components) and externally (for consumers of the design system).

We decided not to export TanStack types directly. By wrapping them in custom types, we prevent "leaky abstractions," ensuring that if we ever replace the underlying utility, the external API remains stable.

The 📁 hooks folder contains the core logic hooks used to build the table, including:

• Preparing the core configuration object required by TanStack Table.
• Building the context used to share table‑specific values across the component tree.
• Defining advanced custom filtering utilities.
• Implementing a custom infinite‑scrolling mechanism.

Not all tables are created equal, and not every table is expected to support the same functionality. For that reason, we chose to leave feature composition to the consumer of the component. This is where the 📁 middleware folder comes in.

By design, the base table ships with no additional features such as sorting or filtering. Each feature is opt‑in and encapsulated in its own hook. Calling one of these hooks returns a partial configuration that augments the overall table configuration passed to TanStack Table. Developers can freely choose which features they need and pass them as an array-middleware: [sortingMW, colPinningMW, colVisMW, globalFilterMW]-to the hook responsible for assembling the final configuration.

Releasing Strategy

Creating a reliable and robust component was only part of the challenge. For a high‑impact component used across multiple applications-each relying on different variations-the question of how to release it to production without disrupting delivery or degrading the user experience became equally important.

Our solution was two‑fold: incremental rollout and gradual deprecation.

The first step was to mark Table V1 as deprecated using the @deprecated JSDoc tag, ensuring that it was clearly struck-through by editors and avoided in new usage. We then ensured that all new stories were built using Table V2.

After a few weeks in production, where the component faced real‑world usage and demand, we gained the confidence to begin phasing out the remaining legacy implementations, both third‑party and in‑house.

Result: Beyond the Code

The most meaningful outcome was a shift in how the team related to the table. Tasks that were once dreaded became routine, as confidence in the component grew.

With a more predictable foundation, feature delivery sped up and hesitation around changes faded. We no longer fight the component; we leverage it.

That shift removed friction and allowed the table to function as what a design-system component should be: a reliable primitive the team can build on with confidence.

How I Built a Multi-Agent System That Automatically Transforms Raw Snowflake Data into Polished Business Reports.

Across the organization, stakeholders regularly need answers: KPI summaries, financial comparisons, and model performance insights. These questions often come up ahead of meetings, reviews, or decision points, and they usually need answers now, not days later.

In practice, getting those answers is slow. A single question typically triggers a long manual pipeline: writing SQL, validating results, generating charts, moving data between tools, and finally summarizing everything into a report. By the time the analysis is ready, the discussion it was meant to support may already be over.

What stakeholders needed was a faster and more reliable way to turn questions into insights without waiting on manual analysis cycles or dealing with technical complexity. The goal was an automated system that could take an analytics question, work through the data step by step, and deliver a clear, decision-ready report in minutes. By abstracting away query writing, visualization, and analytical interpretation, the system enables stakeholders of all technical backgrounds to access insights independently, without relying on data teams or manual workflows.

The result was LangGraph-Insights: a multi-agent system that plans analyses, writes (and fixes) its own SQL, generates charts, and delivers a polished PDF report directly to stakeholders. In this post, I'll walk through how I built it using LangGraph, Anthropic Claude, and Snowflake.

What This Project Is (and Isn't)

This isn't a chatbot that answers analytics questions in plain text.
It's an autonomous analytics system that:

• Breaks down complex business questions
• Iteratively queries real enterprise data
• Detects and fixes its own errors
• Produces artifacts humans actually use (charts, PDFs, summaries)
• Notifies users when results are ready

That difference guided how I designed the system from the ground up.

Prompts vs. Agents: Why We Chose Agents

To see why this architecture works, it helps to first understand the difference between a prompt and an agent.
A prompt is a single instruction sent to an LLM: "Here is some data, write a SQL query." It's linear and stateless. If the output is wrong, the process fails unless a human steps in.
An agent, on the other hand, is a system that uses an LLM as a reasoning engine to perform actions. It has:

• Tools: Capabilities to interact with the world (run SQL, search the web, write files).
• Looping: The ability to observe the output of its actions and decide what to do next.
• Memory: Context of what it has already tried and learned.

We chose an agentic approach because analytics is rarely a linear process. It requires exploration, error correction, and multi-step reasoning capabilities that a simple prompt-response cycle simply cannot provide.

The Problem: Why Single Prompts Fail at Analytics

Early on, I tried treating the problem as a simple prompt: give the model the schema, ask a question, get an answer. That worked for trivial cases, but it fell apart the moment the question required real analysis.
Real analytics problems break single-prompt approaches in three ways:

• Complexity: A question like "Compare operating expenses between Q2 and Q3 and explain the main drivers of change" involves multiple steps: querying raw transactions, aggregating by category, validating totals, and synthesizing insights.
• Reliability: LLMs will inevitably produce invalid SQL or make incorrect assumptions about the data. One mistake is enough to break the whole process.
• State & Structure: A single prompt can’t reliably track intermediate results, chart requests, report sections, or retry logic.

I needed a system that could reason iteratively, fail safely, and recover. This is where LangGraph shines.

The Architecture: A Hierarchical Multi-Agent System

Instead of one monolithic "AI brain", I designed a hierarchical agent graph, inspired by how real analytics teams operate.

• Supervisor Agent - Oversees the entire workflow and delegates tasks
• Lead Analyst Agent - Plans and executes the analysis
• Report Writer Agent - Turns analytical results into a structured report with visualizations
• Reviewer Agent - Evaluates correctness, clarity, and alignment with the business question

All agents communicate through a shared state object that acts as the system's memory.

class StateAgent(TypedDict):
    steps: list[dict]        # History of {goal, sql, output}
    report: str              # Evolving markdown report
    chart_requests: list
    analysis_feedback: str

This shared state is what allows the system to stay coherent across multiple reasoning steps.

Deep Dive: How the Agents Actually Work

1. Planner Node - The Brain

The Planner doesn't write SQL; it decides what to do next.
Based on the user request and the current state, it creates a step-by-step plan:

• "Query the production table"
• "Aggregate results by month"
• "Compare forecasted vs actual values"
• "The data is sufficient, write the report"

This transforms analytics from a one-shot prompt into a deliberate reasoning loop.

2. Executor Node - The Hands

The Executor takes the generated SQL and runs it against Snowflake.
And as expected, SQL sometimes fails - syntax errors, missing columns, or Snowflake-specific issues.

3. Query Fixer Node - The Self-Healing Mechanism

When a query fails, the system doesn't crash.
A conditional router detects the error and sends the failed SQL and error message to a Query Fixer Agent. This agent analyzes what went wrong, rewrites the query, and retries execution.
To prevent the system from getting stuck in an infinite retry loop, each query has a maximum retry limit. If the limit is reached, the failure is surfaced to the Reviewer Agent and reflected transparently in the final report.

Key insight: Reliability comes from bounded feedback loops - not perfect prompts.

Making the System Adaptable: Dynamic Configuration with YAML

To support multiple analytics use cases such as financial comparisons, KPI tracking, and model evaluations, I avoided hard-coding prompts or schemas.
Instead, the system is driven by dynamic YAML configuration files, stored in Azure Blob Storage. Each use case defines its own:

• Table schemas
• SQL templates
• System prompts for each agent
• Report structure and tone

use_case_name: financial_comparison
prompts:
  planner_sys_node: |
    You are an expert financial analyst. Your task is to plan ...
  report_writer_node: |
    Summarize the findings in a clear and professional tone ...

# ... other parameters

This design allows the system to support new use cases without changing the code or graph structure, only the configuration changes. As a result, the same multi-agent architecture can adapt to different business domains with minimal effort.

Deployment & Azure Infrastructure

The system is fully deployed on Azure using a cloud-native, scalable setup:

• Azure Container Registry (ACR): Stores versioned Docker images for the application.
• Azure Container Apps: Hosts the FastAPI service and LangGraph runtime, with automatic scaling and scale-to-zero for cost efficiency.
• Azure Queue Storage: Enables asynchronous, queue-based processing so multiple report requests can be handled in parallel.
• Azure Blob Storage: Store (YAML configuration files, Generated reports and summaries (PDFs), Logs)

Once a report is generated, the system automatically emails the user, providing a notification along with a summary of the findings and a link to the final PDF.

Results & Impact

The impact of the system was immediate:

• Speed: Reports that previously took hours of manual work now generate in minutes.
• Accessibility: Non-technical stakeholders can now get a comprehensive analysis without needing to write SQL queries, create visualizations, or interpret raw data. The system abstracts all technical complexity, democratizing access to data insights.
• Accuracy: SQL errors are auto-corrected, retries are bounded, and the Reviewer Agent ensures alignment between numbers and narrative.
• Scalability: Queue-based processing and container scaling allow the system to handle spikes in demand efficiently.
• Usability: Stakeholders receive clean, professional PDFs directly in their inbox, no dashboards or tools required.

Disclaimer

This project relies on AI models that can occasionally produce incomplete, inaccurate, or outdated information. All outputs (including SQL queries, metrics, and narrative explanations) should be treated as decision support, not as a single source of truth. For any critical, financial, regulatory, or high-impact decisions, results must be independently validated and reviewed by qualified humans before being acted upon.

Conclusion

Building LangGraph-Insights has demonstrated that the true potential of Generative AI lies not just in text generation, but in intelligent orchestration. By moving beyond simple prompts to a robust, multi-agent architecture; we can solve complex, multi-step business problems with a level of reliability, and autonomy that was previously unattainable.

This system proves that with the right design that combines specialized agents, shared state, and self-correction mechanisms, we can bridge the gap between raw data and actionable business intelligence. As these autonomous systems continue to evolve, the focus will shift from simply querying data to enabling deeper, more strategic decision-making across the enterprise.

The Problem: A Growing Codebase, Growing Concerns

Let's start by asking ourselves why do we need unit tests?

• Catch Bugs Early – Identify issues in individual components before integration.
• Improve Code Quality – Enforce better coding practices and maintainability as writing code for testing ensures adhering to the coding architecture and when a function is testable it's usually readable and less-prone to errors.
• Facilitate Refactoring – Allow us to refactor with confidence.
• Enhance Development Speed – You might say "How is that! I am spending more time writing the code itself then the tests!" and that's true when looking at one simple task, however, when you're building a big project, through multiple iterations, it definitely reduces your testing time overhead and it makes later additions easier.
• Support Continuous Integration – Ensure code stability in automated pipelines, as unit tests are very very fast to run, so it is a convenient step to add to your pipeline without worrying about performance and ensuring your code maintains its functionality with every release.

Now that we've talked about the importance of unit tests, it looks very appealing but there are some downfalls we wanted to avoid and some specific goals we wanted to achieve within our application.

Goals:

• Easy to maintain and grow - We wanted them to be easy to edit or require no edits when there is a code change. We were very cautious approaching this project; unit tests have a bad reputation regarding their maintainability. When developers only think about code coverage and a specific percentage to adhere to, it ruins the whole point.
• Valuable tests only - what do we mean by valuable tests? A valuable test is a test that prevents regression, ensures business logic is achieved and is not trivial.

The Journey: Finding Our Testing Sweet Spot :

Understanding Our Ecosystem

Before diving into our approach, it’s important to understand the structure of our application. We use .NET (C#) with EF Core as our ORM, and our architecture follows a clean separation of concerns:

• Controllers – Responsible for routing only, containing no business logic.
• Services – Contain all business logic and act as the core of the application.
• External Services - Handle interactions with any external service, like third party integrations, cloud solutions, data sources, etc.
• Repositories – Handle all interactions with the database.
• Dependency Inversion – Ensures loose coupling between these components.

Given this structure, the natural question was: Where should we focus our unit tests? And what is our sweet spot?

Choosing what to test?

Since business logic resides in the service layer, it became clear that our primary focus should be testing services. Testing repositories would mean interacting with the database, which is better suited for integration tests rather than unit tests. Controllers, on the other hand, only handle routing, making them less critical for unit testing. Thus, our solution was straightforward: unit test the service layer in isolation while mocking dependencies like repositories and external services. This allows us to:

• Validate business rules without relying on actual database calls.
• We achieve better maintainability since services encapsulate the application's domain logic.
• Keep tests fast and deterministic.
• Prevent unnecessary breakage when implementation details change as we reduce test coupling to implementation details, ensuring tests are resilient to changes in other layers... - YES, such dreams can come true!

Best Practices And Guidelines for Test Creation:

1. Identify Core Logic: Focus on methods with significant business logic like calculations, validations, and core operations.
2. Ensure Single Responsibility: Each test should validate one behavior.
3. Keep Tests Independent: Ensure tests do not depend on each other or shared state.
4. Mock External Dependencies: Use mocks for database access and external dependencies. Keep them light and specific to the test.
5. Validate Outcomes: Focus assertions on the results rather than internal implementation details.
6. Fail Fast: Write tests that fail clearly and give enough information to debug quickly.
7. Define Test Scenarios: Identify edge cases and common use cases for each method. Ensure all critical logic paths are tested.
8. Refactor for Testability: If needed, refactor code to simplify dependencies and improve testability.
9. Use Descriptive Names: Use clear and meaningful test names that describe the scenario and the expected outcome.

Adhering to these standards ensures robust, maintainable, and effective unit tests.

The Solution: A Practical Testing Approach

Technical Components Overview

• Testing Framework: XUnit Docs
• Mocking Framework: FakeItEasy
• Assertion Library: FluentAssertions

A problem we faced when deciding how the new testing project would access the service, since it is by design, not accessible. We came across a couple of solutions but the easiest, one-time, and neat solution was to expose the internals of the project to be tested to be seen by the testing project. Here is how you can define it.

The Apps.Tests project has access to internal members of Apps.Services by using the InternalsVisibleTo attribute in the Apps.Services.csproj file:

<ItemGroup>
    <AssemblyAttribute Include="System.Runtime.CompilerServices.InternalsVisibleTo">
        <_Parameter1>Apps.Tests</_Parameter1>
    </AssemblyAttribute>
</ItemGroup>

Writing Tests

Let's examine a real-world example of testing a document upload service to demonstrate our testing principles in action.

Service Under Test (SUT)

We'll look at a document upload service that handles:

• Document uploading and replacement
• Metadata management
• Integration with external storage
• Document attribute management

This is an excellent example as it contains:

• Core business logic (document naming rules, attribute management)
• External dependencies (storage service, repository)
• Complex validation rules
• Multiple paths of execution

public async Task UploadDocument(DocumentMetaData docData, bool replace, FileData file)
{
    var transactionTime = DateTime.UtcNow;
    var userId = _ctx.CurrentUserId();

    await ValidateDocumentsAttributes(docData.AttributesIds);
    await ValidateDocumentType(docData.DocTypeId);
    ValidateAssetType(docData.AssetType);
    AssertFileExtensionIsCorrect(docData.Extension);

    ConcatenateExtensionToDocName(docData.Extension, docData);

    if (replace)
    {
        return await ReplaceDocumentAndUpdateMetaData(docMetaData, file, userId, transactionTime);
    }

    List<Document> docWithName = await _repository.GetDocumentByName(docMetaData.DocName);

    AssertDocumentIsNotDuplicatedForAsset(docMetaData, docWithName);

    docMetaData.DocumentEntities.Add(new DocumentAssociation
    {
        AssetId = docMetaData.AssetId,
        AssetType = docMetaData.AssetType
    });

    return await CreateDocumentAndUploadToCloud(docMetaData, file, userId, transactionTime);
}

private async Task<int> ReplaceDocumentAndUpdateMetaData(DocumentMetaData docMetaData, FileData file, int userId, DateTime modifiedAt)
{
    var dbDocument = await _repository.GetDocumentByNameAndAssetAssociation(docMetaData.DocName, docMetaData.AssetType, docMetaData.AssetId);

    Assert(dbDocument != null, "documentNotFound");

    await _documentStorageExternalService.UploadDocument(dbDocument.Name, dbDocument.NameInStorage, file, ExternalStorageContext.ApplicationDocuments);

    using(var transaction = _dbContext.Database.BeginTransaction())
    { 
        await UpdateDocumentAttributes(dbDocument, docMetaData.AttributesIds, userId, modifiedAt);
        await ReplaceDocumentMetaData(file, dbDocument, docMetaData, userId, modifiedAt);
        transaction.Commit();
    }

    return dbDocument.Id;
}

private async Task<int> CreateDocumentAndUploadToCloud(DocumentMetaData docMetaData, FileData file, string extension, int userId, DateTime createdAt)
{
    string documentNameInStorage = GenerateDocumentName();
    var absoluteUriInStorage = await _documentStorageExternalService.UploadDocument(docMetaData.DocName, documentNameInStorage, file, ExternalStorageContext.ApplicationDocuments);

    return await _repository.InsertDocument(file, docMetaData, documentNameInStorage, absoluteUriInStorage, extension, userId, createdAt);
}

Setting Up the Test Class

public class DocumentsServiceTests
{
    private readonly IDocumentsRepository _repository;
    private readonly IContext _ctx;
    private readonly DBContext _dbContext;
    private readonly IDocumentStorageExternalService _documentStorageExternalService;

    private readonly IDocumentsService _service;

    public DocumentsServiceTests()
    {
        _repository = A.Fake<IDocumentsRepository>();
        _documentStorageExternalService = A.Fake<IDocumentStorageExternalService>();
        _ctx = A.Fake<IContext>();
        _dbContext = A.Fake<DBContext>();

        _service = new DocumentsService(_repository, _ctx, _dbContext, _documentStorageExternalService);
    }
}

  A.CallTo(() => _repository.UpdateDocumentMetaData(A<Document>._)).Returns(1);

  A.CallTo(() => _repository.GetAttributesByIds(A<List<int>>._));
  A.CallTo(() => _repository.GetDocumentType(A<int>._));

  A.CallTo(() => _documentStorageExternalService.UploadDocument(A<string>._, A<string>._, A<FileData>._)).Returns("absoluteUriInCloud");
  
  A.CallTo(() => _repository.GetDocumentByNameAndAssetAssociation(A<string>._, A<string>._, A<int>._)).Returns(new Document { Id = 1, Name = "DocName.docx" });

public static IEnumerable<object[]> UploadDocumentSuccess_Data =>
    new List<object[]>
    {
         new object[]
         {
             new DocumentMetaData {
	             Name = "DocumentName",
	             AttributesId = new List<int> { 1, 2, 3 },
	             Extension = "xlsx",
	             DocTypeId = 1,
	             DocType = AssetsWithDocuments.Deal
             },
             new List<DocAttribute> {
                 new DocAttribute { Id = 1, Name = "atr1" },
                 new DocAttribute { Id = 2, Name = "atr2" },
                 new DocAttribute { Id = 3, Name = "atr3" }
             },
         }
    };

result.Should().NotBeNull();
result.ValidRecords.Should().HaveCount(1);

Example Tests

Here's how we test different aspects of the document upload functionality:

1. Validation Rules Testing

[Fact]
public async Task UploadDocumentReplaceCannotChangeNameError()
{
    // Arrange
    A.CallTo(() => _repository.UpdateDocumentMetaData(A<Document>._)).Returns(1);
    A.CallTo(() => _repository.GetAttributesByIds(A<List<int>>._));
    A.CallTo(() => _repository.GetDocumentType(A<int>._));
    A.CallTo(() => _documentStorageExternalService.UploadDocument(A<string>._, A<string>._, A<FileData>._))
        .Returns("absoluteUriInCloud");
        
    A.CallTo(() => _repository.GetDocumentByNameAndAssetAssociation(A<string>._, A<string>._, A<int>._))
        .Returns(new Document { Id = 1, Name = "DocName.docx" });

    // Act & Assert
   await _service.Invoking(s => s.UploadDocument(
       new DocumentMetaData {
           "CHANGEDdocName",
           1,
           A.Dummy<List<int>>(),
           "docx",
           true,
           1,
           AssetsWithDocuments.Deal
       },
       A.Dummy<List<DocumentAssociation>>(),
       A.Dummy<FileData>()
       ))
	.Should().ThrowAsync<CustomException>()
    .Where(e => e.ErrorKey == ErrorKey.DocumentNameChangeException);
}

This test demonstrates several of our best practices:

• Single Responsibility: Tests one specific business rule (document name cannot change during replacement)
• Clear Arrangement: Dependencies are mocked with specific returns
• Explicit Assertion: Tests for the exact error message
• Focused Scope: Only mocks what's necessary for this specific test

Also, note the naming convention MethodName_StateUnderTest_ExpectedBehavior

2. Complex Operation Testing

[Fact]
public async Task UploadDocument_Replace_UpdateIsCalledSuccess()
{
    // Arrange
    A.CallTo(() => _repository.UpdateDocumentMetaData(A<Document>._)).Returns(1);
    A.CallTo(() => _repository.GetAttributesByIds(A<List<int>>._))
        .Returns(new List<DocAttribute> {
            new DocAttribute { Id = 1, Name = "atr1" },
            new DocAttribute { Id = 2, Name = "atr2" },
            new DocAttribute { Id = 3, Name = "atr3" }
        });
    A.CallTo(() => _repository.GetDocumentType(A<int>._));
    A.CallTo(() => _documentStorageExternalService.UploadDocument(A<string>._, A<string>._, A<FileData>._))
        .Returns("absoluteUriInCloud");
        
    A.CallTo(() => _repository.GetDocumentByNameAndAssetAssociation(A<string>._, A<string>._, A<int>._))
        .Returns(new Document
        {
            Id = 1,
            Name = "DocName.docx",
            DocumentAttributes = new List<DocumentAttribute> {
                new DocumentAttribute { AttributeId = 1 },
                new DocumentAttribute { AttributeId = 2 },
                new DocumentAttribute { AttributeId = 5 }
            }
        });

    // Act
    var result = await _service.UploadDocument(
        new DocumentMetaData {
            Name = DocName",
            1,
            A.Dummy<List<int>>(),
            "docx",
            Replace = true,
            1,
            AssetsWithDocuments.Deal
        }
        ,
        A.Dummy<List<DocumentAssociation>>(),
        new FileData { FileName = "", FileContent = new byte[] { 20 } }
    );

    // Assert
    A.CallTo(() => _repository.UpdateDocumentMetaData(A<Document>._))
        .MustHaveHappenedOnceExactly();
}

This test showcases:

• Complex State Setup: Multiple dependencies configured with specific data
• Behavior Verification: Ensures the update method is called exactly once

3. External Integration Testing

[Fact]
public async Task UploadDocument_CallsAzureBlobStorageUpload()
{
    // Arrange
    A.CallTo(() => _repository.InsertDocument(
        A<FileData>._,
        A<DocumentMetaData>._,
        A<string>._,
        A<string>._,
        A<string>._,
        1,
        DateTime.UtcNow
    )).Returns(1);
    
    A.CallTo(() => _repository.GetAttributesByIds(A<List<int>>._));
    A.CallTo(() => _repository.GetDocumentType(A<int>._));
    
    A.CallTo(() => _repository.GetDocumentByName(A<string>._))
        .Returns(new List<Document> { new Document { } });
        
    A.CallTo(() => _documentStorageExternalService.UploadDocument(
        A<string>._,
        A<string>._,
        A<FileData>._
    )).Returns("documentUrl");

    // Act
    var result = await _service.UploadDocument(
        "docname",
        1,
        A.Dummy<List<int>>(),
        "docx",
        false,
        1,
        AssetsWithDocuments.Deal,
        A.Dummy<List<DocumentAssociation>>(),
        A.Dummy<FileData>()
    );

    // Assert
    A.CallTo(() => _documentStorageExternalService.UploadDocument(
        A<string>._,
        A<string>._,
        A<FileData>._
    )).MustHaveHappenedOnceExactly();
}

This test illustrates:

• External Service Integration: Verifies interaction with blob storage
• Minimal Mocking: Only mocks what's necessary for the test
• Clear Intent: Test name clearly indicates what's being verified

Key Takeaways from These Examples

1. Focus on Business Rules: Each test validates a specific business requirement or behavior
2. Clear Arrangement: Dependencies are explicitly mocked with relevant data
3. Precise Assertions: Tests verify exact behaviors or outcomes
4. Independent Tests: Each test stands alone and doesn't depend on others
5. Maintainable Structure: Tests are organized by functionality and follow a consistent pattern
6. Descriptive naming: to make test intentions clear:

Running Tests

Locally

Use the Test Explorer in Visual Studio to run and debug tests.

In the Pipeline

• Tests run automatically in the build pipeline.
• Results are visible in the build summary under the Tests section.

Key Takeaways: What We Learned

1. Focus on What Matters: Not all code needs the same level of testing. Focus on your business logic.
2. Test Behavior, Not Implementation: Our tests validate what the code should do, not how it does it.
3. Keep Tests Simple: Each test should tell a clear story about a specific requirement.
4. Mock Wisely: Mock external dependencies, but keep the mocks simple and focused.

Remember, the goal isn't 100% code coverage - it's having confidence that your critical business logic works correctly.

In the fast-paced world of data engineering, maintaining high-quality, consistent code is crucial. As teams grow and projects scale, the need for thorough validation of pull requests (PRs) becomes even more important. This ensures minimal errors, enforces best practices, and guarantees seamless deployments.

This blog post explores how to build a robust Continuous Integration (CI) pipeline specifically tailored for dbt projects. By automating PR validation with Azure Pipelines, dbt tests, and SQLFluff, teams can significantly improve efficiency, solidify code consistency, and reduce the risk of errors slipping into production.

Understanding Azure Pipelines

Azure Pipelines, similar to GitHub Actions, GitLab CI, and Jenkins, is Microsoft's cloud-based CI/CD service that automates building, testing, and deploying code. It works by defining workflows in YAML files, which specify the steps needed to validate and deploy code changes. These pipelines can be triggered automatically when code is pushed or pull requests are created, making them ideal for automated validation workflows.

What is dbt?

At its core, dbt (data build tool) is the data engineer’s Swiss Army knife for transforming raw data into analytics-ready gold. It’s an open-source tool designed to help teams write modular, version-controlled SQL code that’s easy to understand, test, and share.

Think of dbt as the bridge between your data warehouse and meaningful insights. By leveraging the power of SQL and a little bit of software engineering magic, dbt allows you to:

• Transform Data: Turn messy, raw data into clean, modeled tables ready for analysis.
• Version-Control Code: Keep your transformation logic organized and trackable with Git integration.
• Test Your Work: Implement rigorous testing to ensure your models deliver consistent and accurate results.
• Document Effortlessly: Automatically generate comprehensive documentation for your data models.

dbt shines in environments where data pipelines can grow messy or unwieldy. It encourages software engineering best practices, like modularity and testing, while remaining approachable for SQL-savvy analysts.

In short, dbt isn’t just a tool; it’s a movement empowering teams to own their transformation layer with confidence and agility. Now, let’s dive into how you can supercharge your dbt workflows with automated PR validation.

Automated PR Validation Pipeline: Addressing Key Challenges

Implementing an automated PR validation pipeline tackles several challenges head-on. By leveraging Azure Pipelines, teams can:

• Validate dbt models to ensure data integrity.
• Enforce SQL standards for consistent code quality across the project.
• Focus code reviews on business logic rather than syntax.
• Prevent resource conflicts through sequential execution.
• Target validation of only modified files for enhanced efficiency.

This fosters a more streamlined and error-free development process.

Overview of the Build Validation Pipeline

The build validation pipeline automates the verification of dbt models and SQL code to maintain quality and consistency before changes are merged.

Key Steps:

1. Identify Changes: Uses Git to identify modified SQL and YAML files, optimizing validation.
2. Validate Models: Executes dbt parse and dbt run --empty to verify schema definitions and documentation.
3. Enforce Code Quality: Uses SQLFluff to ensure SQL formatting standards.
4. Sequential Execution: Ensures orderly processing to prevent conflicts.

Pipeline Breakdown

1. Triggering and Setup

The pipeline is triggered by PR creation. Initial steps include:

• Secure Configuration: Downloads the dbt_profiles.yml file for secure credentials.
• Repository Checkout: Performs a deep clone of the repository.

2. Identifying Changes

The pipeline efficiently identifies modified SQL and YAML files using Git:

- script: |
  cd $(project_root)
  changed_sql_files=$(git diff --name-only --diff-filter=AM origin/$(System.PullRequest.targetBranchName) |
    grep '^$(project_root)/models/.*\.sql$' |
    tr '\n' ' ')
  echo "##vso[task.setvariable variable=changed_sql_files]$changed_sql_files"
  displayName: 'Detect Changed SQL Files'

This targeted approach optimizes validation by focusing on specific changes.

3. Dependency Installation

To validate dbt models, the pipeline installs necessary dependencies:

- script: |
    pip install dbt-core==$(dbt_core_version) dbt-snowflake==$(dbt_snowflake_version) sqlfluff-templater-dbt==$(sqlfluff_version)
  displayName: 'Install Dependencies'

This process ensures:

• Version consistency across tools.
• Proper error handling for smooth execution.
• Efficient setup tailored to the project’s requirements.

Validating dbt Models and Code Quality

4. Model Validation

To ensure the integrity and accuracy of dbt models, the pipeline employs two key validation steps:

Syntax Validation:

Verifies the correctness of syntax and references within dbt models.

Employs dbt parse to analyze the model structure.

Identifies potential errors early in the development process.

Implementation:

- script: |
    cd $(project_root)
    dbt parse
  displayName: 'Validate Syntax with dbt Parse'

dbt Model Validation:

Syntax Validation: Uses dbt parse to check for errors in dbt model syntax.

Dry-Run Testing: Executes dbt run --empty to simulate model execution without reading production data.

Utilizes dbt run with the --empty flag to simulate execution without real data.

Validates model logic and identifies potential issues.

Implementation:

- script: |
    dbt run --select $(changed_sql_files_base) --target CI --full-refresh --empty
  displayName: 'Dry-Run Validation'

5. Code Quality with SQLFluff

SQLFluff is a powerful linter that enforces consistent SQL formatting and best practices:

- script: |
  cd $(project_root)
  sqlfluff lint $(changed_sql_files)
  displayName: 'Lint SQL Files with SQLFluff'

By enforcing consistent formatting and identifying potential issues early, SQLFluff significantly improves:

• Code Readability: Consistent formatting enhances code understanding.
• Maintainability: Standardized code is easier to maintain and modify.
• Overall Quality: Early identification of issues reduces the risk of errors.

6. Setting Source Database

To ensure the accuracy of schema definitions and documentation, the pipeline manually sets Source Database:

- script: |
  cd $(project_root)
  find . -name "*.yml" -type f | while read -r file; do
    sed -i 's/{{target\.database}}/$(prod_database)/g' "$file"
  done
  displayName: 'Validate YAML Files'

This step ensures:

1. Data Accuracy: Ensures the pipeline uses the correct and up-to-date source database.
2. Consistency: Keeps schema and data definitions consistent across environments.
3. Schema Integrity: Guarantees accurate schema and documentation for downstream processes.

7. Sequential Execution

To maintain order and prevent resource conflicts, the pipeline enforces sequential processing:

trigger: none
lockBehavior: sequential
variables:
- group: dbt_build_validation

This configuration guarantees:

• Resource Efficiency: Prevents multiple pipeline instances from running concurrently.
• Reliable Validation: Ensures that validation processes are completed in a predictable order.
• Consistent Results: Maintains the integrity of validation results.

By combining these validation steps and enforcing sequential execution, the pipeline ensures the quality, consistency, and reliability of dbt models and SQL code. This approach is especially critical because dbt Core does not support concurrent runs out of the box, meaning multiple runs at the same time can lead to conflicts or unexpected behavior. Sequential execution avoids these pitfalls, ensuring a smooth and predictable validation process.

Full Pipeline Configuration

trigger: none

# Input variables:
lockBehavior: sequential # Ensure that only one instance of the pipeline runs at a time
variables:
- group: dbt_build_validation 

pool:
  vmImage: ubuntu-latest

steps:

# Download secure dbt profiles.yml only if there are changed files
- task: DownloadSecureFile@1
  name: dbt_profile
  displayName: 'Download dbt profiles.yml'
  inputs:
    secureFile: 'CI_profiles.yml'

    
# Checkout the repository first to get the necessary files for comparison
- checkout: self
  fetchDepth: 0 
  persistCredentials: true

# Determine changed and added files from the current branch
- script: |
    cd $(project_root)
    
    # Get full paths and store as space-separated list for changed SQL files
    changed_sql_files=$(git diff --name-only --diff-filter=AM origin/$(System.PullRequest.targetBranchName) |
      grep '^$(project_root)/models/.*\.sql$' |
      sed  's|raisa_dbt|.|g' |
      tr '\n' ' ')
    
    # Get base names without .sql extension and wrap with +fileName+
    changed_sql_files_base=$(git diff --name-only --diff-filter=AM origin/$(System.PullRequest.targetBranchName) | 
      grep '^$(project_root)/models/.*\.sql$' | 
      xargs -I {} basename {} .sql | 
      sed 's/.*/"@"&" "/' | 
      tr '\n' ' ' | 
      sed 's/"//g')

    echo "##vso[task.setvariable variable=changed_sql_files]$changed_sql_files"
    echo "##vso[task.setvariable variable=changed_sql_files_base]$changed_sql_files_base"
    
    # Output for debugging
    echo "Changed and added files (full paths): $changed_sql_files"
    echo "Changed and added files (base names): $changed_sql_files_base"
  displayName: 'Get changed SQL files'



# Determine changed and added YAML files from the current branch
- script: |
    cd $(project_root)
    
    # Get full paths and store as space-separated list for changed YAML files
    changed_yaml_files=$(git diff --name-only --diff-filter=AM origin/$(System.PullRequest.targetBranchName) | grep '^$(project_root)/models/.*\.yml$' | tr '\n' ' ')

    echo "##vso[task.setvariable variable=changed_yaml_files]$changed_yaml_files"
    
    # Output for debugging
    echo "Changed and added YAML files (full paths): $changed_yaml_files"
  displayName: 'Get changed YAML Files'

# Install required dependencies for dbt and sqlfluff only if there are changed files
- script: |
    set -ex
    pip install dbt-core==$(dbt_core_version) dbt-snowflake==$(dbt_snowflake_version) sqlfluff-templater-dbt==$(sqlfluff_version)
  displayName: 'Install dbt and sqlfluff dependencies'
  condition: and(succeeded(), or(ne(variables['changed_sql_files'], ''), ne(variables['changed_yaml_files'], '')))

# Prepare dbt profiles.yml only if there are changed files
- script: |
    cd $(project_root)
    mkdir -p ~/.dbt
    cp $(dbt_profile.secureFilePath) ~/.dbt/profiles.yml
    dbt deps
  displayName: 'Copy dbt profiles.yml'
  condition: and(succeeded(), or(ne(variables['changed_sql_files'], ''), ne(variables['changed_yaml_files'], '')))

# Only parse dbt models if there are changed files
- script: |
    cd $(project_root)
    dbt parse
  displayName: 'Parse dbt models for changed and added files'
  condition: and(succeeded(), or(ne(variables['changed_sql_files'], ''), ne(variables['changed_yaml_files'], '')))

# Lint changed and added files with sqlfluff only if there are changed files
- script: |
    cd $(project_root)
    echo $(changed_sql_files)
    sqlfluff lint $(changed_sql_files)
  displayName: 'Lint issues in changed files with sqlfluff'
  condition: and(succeeded(), ne(variables['changed_sql_files'], ''))

# Run dbt for changed and added files with full-refresh only if there are changed files
- script: |
    cd $(project_root)
    echo $(changed_sql_files_base)

    find . -name "*.yml" -type f | while read -r file; do
      sed -i 's/{{target\.database}}/$(prod_database)/g' "$file"
    done

    echo "Replacement complete for all .yml files."

    dbt run --select $(changed_sql_files_base) --target CI --full-refresh --empty
  displayName: 'Empty run dbt for changed and added files'
  condition: and(succeeded(), ne(variables['changed_sql_files'], ''))

Key Advantages

• Faster Feedback Loops: Rapid identification and resolution of issues.
• Improved Code Quality: Consistent formatting, enhanced readability, and adherence to best practices.
• Efficient Collaboration: Streamlined PR reviews and reduced manual effort.
• Reduced Risk of Errors: Early detection and prevention of potential problems.

Best Practices for Pipeline Optimization

• Regular Maintenance: Keep the pipeline up to date with the latest dbt and SQLFluff versions.
• Team Collaboration: Establish clear guidelines for PR submissions and code reviews.
• Performance Optimization: Fine-tune the pipeline to minimize execution time and resource consumption.

Conclusion

By automating PR validation, organizations can significantly improve their data engineering workflows. This streamlined approach leads to higher code quality, faster development cycles, and reduced risk of errors.

Next Steps

To further enhance your dbt development practices, consider these additional strategies:

• Expand Test Coverage: Implement comprehensive testing scenarios to identify potential issues early.
• Customize SQLFluff Rules: Tailor linting rules to match specific organizational standards.
• Integrate with End-to-End Pipelines: Seamlessly integrate the validation pipeline with deployment and monitoring processes.
• Automate Documentation Validation: Ensure documentation stays up-to-date and accurate.
• Lint and Validate Python Models: Ensure Python models are properly linted and validated as part of the pipeline.
• Parse if MD Files Changed: Detect and handle changes in markdown files to trigger appropriate actions.
• Seed --full-refresh if Seed Files Change: Implement a full-refresh for seed files when changes are detected.

By embracing automated PR validation, you can elevate your data engineering team's efficiency and deliver high-quality data products.

I recently explored Snowflake Notebooks, which have recently became generally available, and found it quite interesting. In this article, I’ll cover the following topics:

• What Snowflake Notebook is.
• 4 key features which I love in notebooks.
• 3 additional features I’d like to see.
• Ideas for integrating it into our data pipelines.

What is Snowflake Notebooks

Snowflake Notebooks, part of Snowflake Projects alongside tools like Worksheets, Streamlit, and Dashboards, is an interactive, cell-based environment within Snowflake. Similar to Jupyter Notebooks, it enables seamless data analysis, building data pipelines, and training machine learning models using both SQL and Python, all within Snowflake’s infrastructure without moving data outside the platform.

Snowflake Notebooks offers a wide range of features. From the ones I’ve tried so far, in the next section, I’ll share the top 4 features I love about Snowflake Notebooks.

My Favorite Features in Snowflake Notebooks

Feature #1: Compute Control

One of the features I appreciate most in Snowflake Notebooks is the ability to fully control the compute instance used by the notebook. You can choose between:

Existing Snowflake Warehouses (WH):

These are the classical warehouses that come in a variety of sizes, ranging from X-Small to 6X-Large, allowing you to scale based on your needs.

Container Compute Pools:

For even more flexibility, you can leverage compute pools that offer diverse configurations, such as:

• Basic CPUs for general-purpose tasks.
• Memory-Intensive CPUs for data-heavy operations.
• GPUs for machine learning and other computationally intensive workloads.

Configuring these options is remarkably simple, thanks to Snowflake’s high level of abstraction. As with all Snowflake features, the user experience is designed to be intuitive, making it easy to manage and adjust these settings effortlessly.

Feature #2: Enhanced Cells Interaction

Another feature I really like in Snowflake Notebooks is the ability to work with SQL cells alongside the standard Python and Markdown cells. This adds powerful functionality for data analysis and exploration. Here’s what makes it stand out:

Interaction Between Cells:

• Referencing SQL Cells in Other SQL Cells: You can use the name of a SQL cell as if it was a Common Table Expression (CTE). This allows you to query the output of one SQL cell directly in another SQL cell without having to re-run the query or export data.
• Accessing SQL Output in Python: The output of a SQL cell can be accessed in a Python cell by simply referencing the SQL cell's name. You can then convert the output into either a Snowpark DataFrame or a Pandas DataFrame for further processing.

Accessing Python DataFrames in SQL

To use a Python DataFrame in a SQL cell, you can simply save it as a temporary table. Once saved, the temporary table can be queried in any SQL cell within the same session.

Using Python Variables in SQL Queries

Python string variables can be seamlessly integrated into SQL queries within SQL cells. This allows for dynamic and parameterized queries, making the interaction between Python and SQL even more flexible.

Feature #3: Customized Conda Environments

Another feature I appreciate in Snowflake Notebooks is the ability to easily define Conda package dependencies for each notebook. This allows you to create a customized Conda environment tailored to the specific requirements of your notebook.

• Custom Environments for Each Notebook: Each notebook can have its own unique set of dependencies, making it easy to work on multiple projects with varying library requirements without conflicts.
• Support for Custom Packages: In addition to using the pre-defined packages from the Snowflake Conda channel, you can also add your own custom packages, providing flexibility to include specialized libraries or tools.
• Current Limitation: At the moment, the functionality is limited to packages available in the Snowflake Conda channel, which may restrict the availability of some libraries.

Feature #4 Flexible Execution Options

One of the features I really like about Snowflake Notebooks is the flexibility in how they can be executed. Snowflake provides several ways to run notebooks, making it easy to integrate them into different workflows:

• Interactive Execution: You can execute the notebook directly from the Snowflake UI while opening it, enabling an interactive experience for analyzing or exploring the data.
• SQL-Based Execution: Notebooks can be executed through SQL code using the EXECUTE NOTEBOOK command. This SQL command can be run either directly from the Snowflake UI or from external tools such as dbt or Airflow, allowing seamless integration into automated data pipelines.
• Scheduled Execution: Snowflake Notebooks can be scheduled as Snowflake Tasks, allowing them to run periodically based on a defined schedule. One standout feature of this functionality is the ability to pass parameters to notebooks when executed as a task. For example, during interactive execution, you might run the notebook on a sample of the data for faster testing. However, when scheduled as a task, you can pass a parameter to process the entire dataset, enabling full-scale production processing seamlessly.

Features I'd Like to See in Snowflake Notebooks

Feature #1 Real-Time Collaboration

I would love to see robust real-time collaboration capabilities in Snowflake Notebooks, allowing multiple users to edit the same notebook simultaneously. Features like shared cursors, presence indicators, and real-time updates would greatly improve teamwork and enhance the overall user experience.

Feature #2 Expanded Package Management

I would like to see the ability to support additional Conda channels and install packages from pip repositories in Snowflake Notebooks. This feature would provide greater flexibility in accessing specialized libraries while maintaining robust environment management and effectively handling dependency conflicts.

Feature #3 Parameters Passing for SQL Execution

Adding the ability to pass parameters to a notebook when executing it through SQL would allow greater flexibility. This would enable external tools, not just the Snowflake scheduler, to dynamically send parameters, making integrations with third-party platforms more powerful.

Integrating Snowflake Notebooks into Our Data Pipelines

Interactive Development and Testing for Data Pipelines

One way we can leverage Snowflake Notebooks in our data pipeline is by using them to develop and test subsets of our workflows interactively, allowing us to incorporate stakeholder feedback before deploying them to production.

Building Interactive Analytics Dashboards

Another use for Snowflake Notebooks is to build custom interactive data analytics dashboards, leveraging the Streamlit integration to provide a rich, interactive user experience. These dashboards can also be kept refreshed as part of our data pipelines, ensuring they always display up-to-date information for stakeholders.

Building and Experimenting with Machine Learning Models

Another use case for Snowflake Notebooks is the ability to build machine learning models and conduct experiments without needing to move data outside Snowflake. While allowing users to visualize model results and interact with them directly within the notebook environment.

Prototyping and Scheduling Intermediate Pipeline Components

Another use for Snowflake Notebooks is to write quick, experimental components of our data pipelines, such as data models or data transformations, and schedule them to run periodically using tools like dbt and Airflow. These components can serve as an intermediate stage, allowing us to test and refine them before fully integrating them into our production pipelines.

Conclusion

Snowflake Notebooks provide an exciting opportunity to streamline data workflows, enhance collaboration, and bridge the gap between experimentation and production. By building on their current features and introducing additional functionality, such as broader package support, advanced collaboration tools, and better integration with external platforms, they can support a wider range of use cases. These enhancements would make Snowflake Notebooks easier to use and become powerful tool for managing and optimizing our data workflows.

Data has become one of the most critical assets in any business setting with significant influence on a business' success of failure. Ease of access to quality data enables business stakeholders to make accurate data-driven decisions and, depending on the timeliness of the data, gives an edge over competitors to take those decisions in a quicker and more strategic manner.

The value of the data you use does not lie merely in the information provided, it's also highly dependent on where you get this information from. The sources you choose to rely on could either hinder or build the credibility of your insights. Data sources could vary from Web scraping, API calling, Public Data Providers, Databases, etc. Accordingly, when multiple sources provide similar data, how do we choose the most suitable one?

As a Data Engineer (DE), you are typically responsible for extracting data from these various sources. The objective of this post is to establish a framework for DEs that assigns quantitative scores to Data Providers based on specific, predefined quality metrics. These scores are designed to align closely with the business's interests and reflect the priorities within its Data Domain.

At the end of the article, since we are an oil and gas company, we will be applying this framework on an oil and gas example. But for now, for illustrative purposes, let's assume we work for an athletics summary company tasked with identifying the world's top athletes to feature on our app, ProStats, which showcases their records and statistics. We are currently evaluating two hypothetical Public Data Providers as potential data sources: ProAthlete and AthlaWorld.

Taking a step back

Before evaluating any data providers, it's crucial to first turn the focus inward. We need to clearly define our business data needs, priorities, and expectations. What specific data do we require? What are our priorities when it comes to data quality, accuracy, and timeliness?

Establishing a well-defined and concise understanding of our data requirements is essential. These definitions should be objective, independent, and not tied to a particular data provider, ensuring that it remains applicable regardless of which provider/source we choose now or in the future.

A Data Dictionary serves as the central repository for our data requirements across its various data domains. It can be categorized into two primary entities:

• Attributes: An attribute is a single piece of information associated with a particular record—essentially, think of it as a column in a table.
• Dimensions: Dimensions are collections of related attributes that pertain to the same field of knowledge, much like the structure of a table.

The Data Dictionary will likely consist of multiple Dimensions, each with its own distinct grouping of attributes. For example, we might have a dimension called ATHLETE_RECORDS, which contains attributes that describe an athlete's records throughout their career. As outlined in the table above, this dimension would include attributes detailing world record statistics, with a clear definition for each attribute, the required formatting, expected data type, and the applicable unit of measure.

Initially, a Data Dictionary can be built by reflecting the Dimensions and Attributes found within our organization's databases, BI dashboards, data warehouses, and other data sources. These sources provide valuable insight into our Data Domain and lay the groundwork for defining the necessary Dimensions and Attributes.

A well-constructed Data Dictionary should also facilitate collaboration across all teams and stakeholders in ProStats, allowing us to contribute to its development. This collaborative approach helps set priorities for each attribute and dimension, ensuring that the dictionary aligns with the broader business needs.

To enhance this process, interviews with business stakeholders can be conducted to capture their insights and translate their business terminology into specific attributes within the Data Dictionary. This ensures that the data needs and priorities of the business are properly reflected per team. These prioritizations can then influence the final scoring of each attribute and dimension—the higher the priority, the greater its impact on the overall score.

Attribute priorities could be defined as follows:

1. Must have: is a deal-breaker if attribute is not available
2. Nice to have: Has business value but not a deal-breaker
3. Not Needed: not critical to have

each of those priority levels could then be assigned a number and averaged out to assign a weight to each attribute's score factor. For simplicity's sake, let's assign them as 3, 2, 1, respectively, with 3 as the highest priority and 1 as lowest.

It is important to note that the weighting process is not mandatory and might cause cloudiness to the final score's business meaning. It might sometimes make more business sense to give all attributes the same weight.

The Data Dictionary will function as a contract between the Data Engineering (DE) team and the business stakeholders, clearly outlining and aligning data expectations. For instance, if a marketing team member needs to know the number of world records currently held by an athlete, they should be able to find this information in an agreed-upon data storage location—such as a data warehouse. Specifically, they would look for it in the ATHLETE_RECORDS table, under the column HELD_WR, with the predefined data type and format.

Intermediary Abstraction Layer

Now that we have clear definitions of ProStats' data requirements through the Data Dictionary, the next step would be to start mapping those Dimensions and attributes to both Data Providers, thus, creating an Intermediary Abstraction Layer (IAL). Since the data representation coming from both Data Providers will not necessarily be the same, the IAL will work as a common denominator between them, transforming them into a more familiar and agreed-upon representation, hence, facilitating an apples-to-apples comparison.

This process could be a simple one-to-one mapping between each attribute and its corresponding attribute found in the Data Provider or a complicated process where calculations are required to reach the exact format defined in the Data Dictionary.

For example, in the case of the NAME attribute, columns Athlete_name and Full_name could be found in our Data Providers ProAthlete and AthlaWorld, respectively, which will then be directly mapped to said attribute. However, for the attribute LATEST_WR_DATE, the required data might not be as readily available and necessitates a transformation operation for both Data Providers. The transformation could be in the following form, assuming that for AthlaWorld, there exists a table called WORLD_RECORDS that contains all athletes' world records:

SELECT ATHLETE_ID, MAX(WORLD_RECORD_DATE) AS LATEST_WR_DATE
FROM ATHLAWORLD.WORLD_RECORDS
GROUP BY ATHLETE_ID

This query would then be used to map the data from AthlaWorld to the LATEST_WR_DATE attribute. Similarly, a query for ProAthlete might also be required to map its data to the same attribute. This is merely a simplified example of how transformation operations could be used to map data coming from the Data Providers to our Data Dictionary, it is important to note that in other instances the queries could be higher in complexity.

The outcome of this step should result in two versions of the Data Dictionary, each populated using one of the Data Providers, reflecting the unique mappings and transformations required.

Foundational Tables

Now that we have the IAL set up, we could start performing some basic comparisons between the IAL tables coming from each Data Provider. In the Foundational tables (FT), we compare the most granular levels of data together for each attribute in all dimensions to set the stage for higher level scoring (next section, Quality Metrics). These comparisons could take many forms depending on the type of data to be compared.

Main Types of Comparisons

1. String Comparisons:
   To ensure accurate string comparisons, it is essential to perform string cleanups beforehand. This process helps eliminate unnecessary variations and ensures that only meaningful data is used for comparison. For example, Michael Phelps’ name might appear as michael_phelps in AthlaWorld and as Michael Phelps # in ProAthlete. Without cleaning, these variations could lead to mismatches during data comparison.
   
   The cleanup process typically involves several key steps:
   1. Remove special characters (e.g., underscores, hashtags).
   2. Convert all letters to lowercase to standardize case differences.
   3. Remove extra spaces to avoid formatting inconsistencies.
   
   By applying these and other techniques, we can minimize data noise and improve the accuracy of string comparisons, ensuring that identical records are correctly matched across different data providers.
   
   There are several Algorithms that could be implemented afterwards, such as, Levenshtein distance, Jaro-Winkler Similarity, along with many others, however, this topic is beyond the scope of this post so I will leave it to the reader's own discretion to decide which is most appropriate.
2. Boolean:
   Boolean data are straightforward to compare as there only two possibilities, either they match or don't.
3. Enum:
   Enum is not an explicit data type in a database, however, it does implicitly show up in many of the attributes. For example, when mapping the attribute SPORT_TYPE, we might imagine that there are only 3 types of sports: Swimming, Running and Cycling. These values could be considered as an enum and values coming from both data providers should be mapped to one of these three values. Similarly to Boolean, after mapping, comparison should be straightforward
4. Integers:
   Integer comparison could be performed through a simple subtraction of attributes coming from both providers, in a consistent order. As an example, comparing athletes' world records recorded by both data providers using the formula
   ATHLAWORLD.HELD_WR - PROATHLETE.HELD_WR
   would result in:
   0 if both report the same number of Held world records
   >0 AthlaWorld reports more world records
   <0 if Proathlete reports more world Records
5. Dates:
   Date comparison will be executed similarly to integers with the added step of needing to express the result in specific time units—such as months, days, or minutes.

The resulting Foundational table of the Dimension mentioned, ATHLETE_RECORDS should look something comparable to the following table

When the comparison done produces nondefinitive result, ex. string A is 70% like string B, it is important to define a tolerance threshold for each of these comparisons. For example, any string with less than 70% similarity will be considered mismatching. The threshold could be defined using statistical analysis technics by, for instance, defining the mean and standard deviation of the differences or calculating the z-score for the differences.

Quality Metrics and Clusters

Quality Metrics

With all comparisons done and ready in the Foundational Tables, we need to utilize them in a way to describe the data coming from each Data provider through defined quality metrics. Each quality metric tests for a certain character in the data.

Generally, there are 3 main quality metrics:

Accuracy

Fundamentally, Accuracy is about identifying which data provider reports more accurate data for each attribute. However, calculating accuracy can be complex due to the challenge of determining the correct data source when discrepancies arise. When faced with mismatches between the two data providers, the possible scenarios are:

1. AthlaWorld is correct.
2. ProAthlete is correct.
3. Neither source is correct.

Determining which scenario is right requires careful consideration. Here are three possible approaches to tackle this:

Using a Third Data Provider as a Tiebreaker:

• Introduce a third data provider as a reference point. If all three providers agree on a value, all are awarded a point for accuracy.
• If the two primary data providers (AthlaWorld and ProAthlete) disagree, but one of them matches the tiebreaker, the matching provider is awarded a point.
• If none of the providers match, none are awarded points.
• This point system can help quantify and score the accuracy of each provider.

Manual Verification with a Source of Truth:

• Define a small, representative subset of the data where disagreements exist between the two data providers.
• Compare this subset against a trusted, authoritative source of truth. For example, if evaluating the HELD_WR attribute, the Olympics website could be used as a reference.
• Although effective, this method can be time-consuming and labor-intensive due to its manual nature, especially for large datasets.

Leveraging Existing Organizational Data:

• Compare the data from the providers with information that already exists in the organization's data storage, such as data warehouses. This existing data has typically been vetted, reviewed, and used by Subject matter experts (SMEs) from within Prostats, making it a trusted reference point.

Using any of the mentioned approaches, each attribute should then have a score ranging between 0-1 representing the accuracy score of the Data Provider with respect to the attribute. Giving ProAthlete a score of 0.7 in HELD_WR means 70% of the data in this attribute was verified to be accurate.

Completeness

Completeness measures the amount of data in each attribute that lacks any business value. These values could range from nonsensical values, such as having an athlete with LATEST_WR_DATE set as 1300-01-01 or SPORT_TYPE set as NULL. However, it's important to note NULL or empty values should not always be regarded as incomplete values. For example, an athlete could have LATEST_WR_DATE set as NULL not because it is missing, but a result of not having any world records. Taking that into account, it is critical to identify what defines a record as incomplete for each attribute before performing any calculations.

Using the FT layer, each attribute should then have a score ranging between 0-1 representing the completeness score of a Data Provider for each attribute. Giving AthlaWorld a score of 0.9 in LATEST_WR_DATE essentially means that 90% of the data found in this attribute is of business value.

Timeliness

Not only is having data available in our Data Provider essential, but the time it takes for the Data Provider to deliver that data for processing by our pipelines is equally critical. Timeliness serves as a key measure of the data's relevance, reflecting how quickly the Data Provider can provide it for downstream consumption. Ensuring timely delivery directly impacts how current and actionable the data is for decision-making, making it a crucial factor in evaluating the Data Provider's overall effectiveness.

Unlike accuracy and completeness, timeliness operates independently of both IAL and FT layers. To measure timeliness effectively, a separate layer is established using Slowly Changing Dimension (SCD) Type 2 tables. This SCD layer will track attributes that we deem time sensitive.

The SCD layer will consume data from both Data Providers (Data Providers) and record three types of data events:

• Inserts: When a new record is made available by the Data Provider.
• Updates: When an existing record is updated to reflect the latest information, ensuring outdated data is replaced.
• Deletes: When a record is removed, preventing the use of obsolete data.

By comparing the movement of data within each Data Provider, we can gain a clear understanding of what to expect if they choose to subscribe to one provider over the other. For instance, if Michael Phelps breaks a world record, and AthlaWorld reports it within three days, but ProAthlete manages to report it within minutes, ProAthlete would have a significant edge in timeliness.

Since this metric is time-sensitive, it is crucial to set up the SCD layer early in the evaluation process to ensure that enough data is captured for meaningful analysis. This will allow ProStats to make informed decisions based on how quickly each provider can deliver actionable data.

Clusters

How many athletes in total throughout the 20th and 21st century do you think exist? Probably a couple of thousands of athletes. Unfortunately, not all of them are relevant in our current day and age. Now imagine when calculating the accuracy quality metric for both AthlaWorld and ProAthlete it was determined that they had 0.95 and 0.75 accuracy, respectively. Intuitively, our stakeholders would be racing to AthlaWorld due to their obvious edge in accuracy. However, one important thing to consider is what data is AthlaWorld 0.95 accurate in?

From those thousands of athletes, we are most likely interested in only the most recent and relevant current athletes to feature on our App, consequently, it would make sense to cluster the data provided by both Data Providers to only the athletes that conforms to our business needs. Let's assume as well that we are mainly interested in new current Events, thus excluding old irrelevant sporting events such as Standing High Jump and Walking Events. It would only be reasonable for us to limit our quality metrics checks to Clusters that would represent Athletes that are competing in the 21st century, defined as Cluster A and Athletes that are competing in currently held events, defined as Cluster B. Each of those clusters would then be assigned its own Quality metrics for each Data Provider.

After this clustering process, it could be the case that AthlaWorld and ProAthlete are in fact 0.80 and 0.90 accurate in Clusters A and B, respectively, giving ProAthlete the lead in accuracy in data that have business value for us.

Clusters could be as simple or as complicated as required to give the quality metrics accurate and relevant business meaning.

Score Cards

Presenting quality metrics in a way that’s both intuitive and easy to understand is just as important as the metrics themselves. For non-technical business stakeholders—those who will ultimately decide which data provider to choose—complex data needs to be distilled into a handful of key numbers that clearly communicate the pros and cons of each option. This is where the scorecard comes in.

A scorecard serves as the perfect bridge between all the detailed technical work and the decision-makers. It could be as simple as an excel sheet with tabs for each defined dimension. In each tab, all attributes are listed as columns and each row contains the number of records scored by a provider. An additional row is also added to give the total population of data used to measure the attribute's quality metrics. An example for Accuracy would be as follows:

Using the table above, we could now calculate the accuracy of each Data Provider by dividing the Data Provider's row with the totals row, resulting in the following table

Now, referring to the weights we assigned to each attribute when first designing the Data Dictionary, we could perform a weighted average for each Data Provider's accuracy across all attributes in this dimension, for example, the results for ATHLETE_RECORDS dimension would be

AthlaWorld

(0.243)0.900 + (0.162)0.970 + (0.189)0.322 + (0.182)0.333 + (0.216)0.857=0.682416

ProAthlete

(0.243)0.936 + (0.162)0.963 + (0.189)0.874 + (0.182)0.996 + (0.216)0.286=0.791688

From those results, we could assume that ProAthlete would be better performant when it comes to accuracy in this dimension. The same process will then be repeated for each quality Metric per Dimension for every cluster. The end score card should be comparable to the following:

To simplify things further, all dimensions could be averaged to produce a single score per quality metric for each cluster. This would allow stakeholders to quickly assess overall performance at a glance. For a more detailed view, the scorecard could be structured in an accordion format, where each quality metric can be drilled through to reveal more granular results.

The scorecard could also contain other essential details, like contract prices, the effort required to integrate the Data Provider with ProStats systems, and the potential impact on existing infrastructure which will all factor into the decision-making process.

Update and Maintain

Maintaining the scorecard framework is essential to ensure that it remains relevant, accurate, and aligned with evolving business needs. Making sure that the data dictionary reflects new business data needs makes future reevaluation process easier and more readily available. The update and maintenance step could be done by:

1. Keep Data Dictionary Up-to-date: As business needs evolve, new quality metrics or dimensions may become necessary. The scorecard should incorporate these changes as they become available. If a new attribute or dimension becomes critical, ensure it’s integrated, and that stakeholders are informed about how it impacts overall scores.
2. Periodic Data Provider Evaluation: based on a predefined period, the framework should be used to reevaluate the currently used Data Provider and decide if a switch is necessary.

Wrapping Up

To bring this framework into the world of oil and gas, since we are an oil and gas company, let's consider how we might choose between two public data providers—Fofi and DellKalaTaste—for well data. Following the ProStats example's steps, we will need to start by defining our requirements first.

First, our oil and gas company would create a Data Dictionary that defines our expected dimensions and attributes:

• Header: General definitions and metadata related to a well
• Geometry: geometric characteristics of a well.
• Production: well’s production performance.

Data Dictionary

Due to its nature, we would then proceed to set up the SCD tables for the Timeliness quality metric to have sufficient time to gather useful and actionable data. Next, an Intermediary Abstraction Layer (IAL) would be built to harmonize the data coming from Fofi and DellKalaTaste, ensuring that comparisons are made on an apples-to-apples basis. For example, the WELL_NAME attribute from both providers would be mapped to a common format in the IAL, making it easier to compare.

SCD Table Example for Header

Once the IAL is set up, we would start comparing the data using Foundational Tables (FT). These tables would perform comparisons between the attributes of each dimension, such as evaluating how accurately each provider reports well locations or production volumes. Comparisons would be tailored to the data type, using algorithms for string matching, integer comparisons, or date comparisons where necessary. Moreover, using the SCD tables, we would then calculate and compare time span for each Data Provider to insert new wells or old well's data, such as production data, update outdated data or deleted irrelevant data. This will serve in calculating the Timeliness quality metrics.

Intermediary Abstraction Layer for Header

Reflecting on the business needs and requirements, we could group the wells into the following 2 Clusters:

1. Cluster A: defined as all well that our company owns in, as it is critical to have the correct information about our wells
2. Cluster B: defined as all horizontal wells existing in the Permian Basin assuming that we as a company are only interested in buying wells in this basin

The results of these comparisons grouped by the defined clusters would feed into Quality Metrics such as Accuracy, Completeness, and Timeliness. For example, accuracy could be measured by comparing well production data from both providers with a trusted third source, like internal records of owned wells. Completeness would evaluate how much useful data each provider delivers, while timeliness would measure how quickly new data becomes available.

Finally, the scores would be aggregated in a Scorecard. For example, Cluster A would look like:

Scorecard For Header

By consolidating the scores, stakeholders can quickly see which provider excels in the areas that matter most to the business. The scorecard can also include contract prices, integration costs, qualitative impacts, and the impact on existing systems to ensure all factors are considered in the decision-making process.

To maintain the relevance and accuracy of the scorecard, it should be periodically updated to reflect any changes in business needs or data priorities. New attributes can be added to the Data Dictionary, and the scorecard framework should be used regularly to reevaluate the current data providers. This ensures that the company remains agile, consistently using the best data sources available.

This article proposes a method for significantly faster Arps decline curve fitting on large datasets using TensorFlow on GPUs. It tackles the limitations of traditional approaches, which rely on iterative curve fitting, like SciPy's curve_fit, applied to each data point one at a time. By leveraging TensorFlow's parallel processing and vectorization capabilities on GPUs, this method can fit multiple Arps decline curves simultaneously, achieving substantial speed improvements for large datasets.

Arps decline curve analysis (DCA) is a vital tool in the oil and gas industry for representing and predicting the future production of wells. Developed by J.J. Arps in 1945, it analyzes historical production data relying on the observation that oil and gas production typically follows a declining trend over time as the reservoir pressure depletes. By fitting the data to parametric equations, engineers can forecast how oil and gas flow will decline over time. This information is crucial for tasks like estimating reserves, planning production strategies, and making investment decisions. Essentially, Arps DCA helps maximize efficiency and profitability by providing a roadmap for oil and gas well behavior.

Arps divides the well production life into mainly two partitions:

1. A hyperbolic curve representing the segment that starts after peak production, this is the initial stage of production characterized by a rapid decline in production rate.
2. An exponential curve representing the later stage of production where the decline rate becomes more gradual and approaches a constant value over time.

The curve function is summarized as follows:

where

• q(t)  is the production rate in the form of production per unit time
• t is time
• q_i is the initial production
• d_i is the initial decline in the hyperbolic part of the equation
• b is the hyperbolic factor controlling the rate of change of the decline

After reaching a certain decline rate d_f  (usually 7% secant decline) at time t_h, the curve is represented by an exponential function using q_h until end of production.

As seen in Code Snippet 1, SciPy's curve_fit is usually used by defining the Arps decline function arps_production, the production time period xdata, and the production values ydata. Based on domain knowledge or engineering estimates, the fitting can be sped up by providing adequate initial values for the parameters p0, and the output can be constrained by adding the boundaries for each parameter bounds.

params, params_covariance = optimize.curve_fit(arps_production,
xdata=np.arange(len(actual_production)),
ydata=actual_production, 
p0=[initial_qi, initial_di,initial_b], 
bounds=([0,DF_PERCENT_SEC,B_FACTOR_LOWER_BOUND],[np.inf,SECANT_DI_UPPER_BOUND,B_FACTOR_UPPER_BOUND]))

SciPy's curve_fit employs a least squares minimization algorithm. This technique iteratively adjusts the parameters of the Arps equation to minimize the sum of squared residuals between the predicted and actual production rates (ydata). Residuals represent the difference between these values. In each iteration, the algorithm calculates the gradient of the cost function (sum of squared residuals) with respect to each parameter and uses this information to update the parameters in the direction that leads to a better fit, ultimately minimizing the overall difference between predicted and observed production rates.

Conventional curve fitting algorithms, like SciPy's curve_fit, struggle with large datasets. These methods typically fit each data point individually using a separate optimization instance, making them computationally expensive for massive datasets. Additionally, SciPy's curve_fit relies on few optimization algorithms in its base package (Trust Region Reflective, dogleg and Levenberg-Marquardt's least squares minimization algorithms), which might not always be the best choice for complex models or large data volumes.

This article introduces a TensorFlow-based approach to accelerate Arps decline curve fitting by leveraging its distributed computing, complex optimization algorithms and vectorization. Here's the workflow:

• Load production data for all wells containing time t and monthly production rates q into two Tensors.
• Define the Arps decline curve equation within a TensorFlow function.
• This function will take time t as input and utilize initial production rate q_i, initial decline d_i, and B-factor b as variables.
• The function outputs the predicted production rate based on the chosen model parameters.
• Define a loss function that calculates the difference between the predicted production rate from the model and the actual production rate q.
• Common loss choices include mean squared error (MSE) or mean absolute error (MAE).

def loss():return tf.reduce_mean(tf.square(arps_production_tf_vectorized(t,[qi, di, b]) - q))

• Utilize a TensorFlow optimizer (e.g., Nadam optimizer) to minimize the loss function across all data points simultaneously over (n) epochs.

optimizer = tf.keras.optimizers.Nadam(learning_rate=LEARNING_RATE, beta_1=BETA_1, beta_2=BETA_1)

• Define the constraints for each of the model parameters.
• This single optimization step fits a different Arps decline curve to each data point in the entire dataset.

for _ in range(NUMBER_OF_EPOCHS):  # number of optimization steps
	optimizer.minimize(loss, params) # add constraints 
	qi.assign(tf.clip_by_value(qi, clip_value_min=0, 	clip_value_max=float('inf')))
	di.assign(tf.clip_by_value(di, clip_value_min=DF_PERCENT_SEC_T, 
	clip_value_max=SECANT_DI_UPPER_BOUND_T)) 
    b.assign(tf.clip_by_value(b, 
	clip_value_min=B_FACTOR_LOWER_BOUND, clip_value_max=B_FACTOR_UPPER_BOUND))

• Evaluate the fitted model performance using metrics like Mean Absolute Percentage Error (MAPE) or Mean Absolute Error (MAE).
• Visualize the fitted curve compared to the actual production data for validation.

The core strength of TensorFlow lies in two key features. The first is its vectorized distributed computing architecture. Unlike SciPy's curve_fit which is confined to a single CPU core, TensorFlow efficiently distributes the workload across the multiple GPUs available in your system, fitting the Arps decline equation to millions of data points simultaneously. The second is its wider range of optimization functions, while SciPy's curve_fit often relies on few optimization algorithm in its base package, which might not always find the absolute best solution, especially with massive datasets and potential complexities. TensorFlow offers a vast arsenal of optimization algorithms (Adam, Nadam, SGD, etc.). Even though SciPy provides a broader optimization library (scipy.optimize which adds a few extra optimizers like Nelder-Mead simplex algorithm and Differential Evolution), utilizing these alternatives requires manual implementation of the optimization loop.

A crucial aspect of this approach is evaluating its effectiveness against traditional curve fitting methods. Our investigation focused on two key areas: fitting error and time performance. Interestingly, both approaches yielded comparable fitting errors (SciPy's curve_fit scored an average MAPE of 0.65% across all datasets, while the Tensorflow method scored an average MAPE of 0.67%), indicating that the TensorFlow method accurately replicates the curve fitting behavior of SciPy's curve_fit. However, the true advantage lies in time performance.

To achieve a fair comparison, we employed two machines from Azure with similar budgets. The first machine, a GPU-based system (2 V100 GPUs, 32 GiBs GPU memory, and 224 GiBs RAM ), served as the platform for the TensorFlow approach using the Nadam optimizer. The second machine, a CPU-based system (72 vCPUs,  and 144 GiBs RAM ), ran the traditional method using the Trust Region Reflective algorithm's least squares minimization and code based parallelization where each core was responsible for fitting a chunk of the dataset. Our findings revealed that the TensorFlow method on the GPU significantly outperformed the CPU-based approach with larger datasets. This performance improvement becomes more pronounced as the dataset size increases, making the TensorFlow method ideal for large-scale production analysis.

This plot shows the improvement in performance over different dataset sizes for the two machines (NC12s v3 VS F72s v2):

Due to the overhead incurred by TensorFlow's initialization, there isn't a noticeable improvement compared to the CPU-based method for smaller datasets. However, as the dataset grows larger, the improvement over the CPU method exponentially increases. Once the dataset becomes larger than the GPU's memory, the improvement factor slightly decreases as a new overhead is introduced when the dataset has to be broken down into chunks based on the GPU memory limit. The improvement factor then plateaus since TenforFlow now must process the chunks sequentially. We can further increase the improvement factor and delay the plateau by increasing the number of GPUs or using GPUs with more memory.

Conclusion

TensorFlow offers a compelling approach to Arps decline curve fitting, particularly for large datasets. It leverages GPU parallelization and vectorization to deliver significant speedups compared to traditional techniques. This enhanced efficiency scales well as data size increases, making it ideal for real-world scenarios. Additionally, TensorFlow provides flexibility in customizing the Arps decline model, loss function, and optimizer, enabling tailored analysis to address specific needs. This empowers engineers to analyze larger datasets faster, leading to quicker production forecasting and optimized reservoir management decisions.

References

• Arps, J. J. (1945). Analysis of decline curves. Transactions of the AIME, 160(1), 228-247.
• SciPy curve_fit: https://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.curve_fit.html
• https://www.tensorflow.org/
• Microsoft. (2023, March 28). Azure Virtual Machines. https://learn.microsoft.com/en-us/azure/virtual-machines/
  

‌‌

Think of your data as a giant puzzle, where the pieces often wear clever disguises. As a data engineer, you're often left trying to put the pieces back together, but there's a catch - you have thousands of pieces to sift through.

Here's where text similarity can be a handy tool. It can be your magnifying glass, helping you spot the hidden connections more efficiently and making your insights sharper.

In this article, we will explore different approaches to find the extent to which certain words/sentences can be similar. We will delve into the world of text similarity analysis, leveraging the power of Snowflake, a versatile data warehouse. We chose Snowflake for this task because of its exceptional scalability, both vertically and horizontally, making it the ideal tool to tackle text similarity on a massive scale.

Topics

1. Understanding Text Similarity
2. Text Similarity Metrics
   1. Jaccard Index
   2. Levenshtein Distance
   3. Jaro-Winkler
3. Use Cases
4. Benchmarking for Differnt Warehouse Sizes
5. Conclusion

Understanding Text Similarity

Similarity is the process of determining the extent to which two items are similar. For example, consider two cars: you have many ways to compare them, such as color, shape, brand and various other features.

When it comes to Text Similarity specifically, you are exploring similarity on two different dimensions - semantically and lexically.

• Semantic similarity focusses on the meaning behind the words.
• Lexical similarity focusses on the words by themselves, considering factors like letter and word frequency, as well as the presence of any missing letters (e.g. William -> Williams).

For example: The mouse ate the cheese.

1. The cat ate the mouse. (Similar in wording but different in meaning)
2. The rat consumed the cheese. (Different in wording but Similar in meaning).

In this article, We will focus on the Lexical similarity between two sentences/words.

Text Similarity Metrics

There are multiple metrics to find if two strings are similar. We will use sql on Snowflake Data Cloud to discuss the most commonly used metrics, leveraging their text similarity built-in functions' capabilities.

1. Jaccard Index

Jaccard Index (also known as Jaccard Similarity coefficient) is used to compare between two distinct sets of words. It provides a metric ranging from 0% to 100%. The higher the metric the closer the two strings are to each other.

It's calculated by dividing the intersection of the two sets against their union.
$$ J(A,B) = { |A\cap B| \over |A\cup B| } $$
Example:

A: "Raisa Energy is a leading oil & gas investment and management company"
B: "Raisa Energy company is focused on oil & gas investment"

$$ J(A,B) = {|{A\cap B}| \over |{A}{\cup} {B}| } = {{8} \over {4}+{8}+{2}} = 57\% $$

Snowflake Example:

CREATE TEMP TABLE JACCARD_INDEX_EXAMPLES (TEXT_A VARCHAR, TEXT_B VARCHAR);
INSERT INTO JACCARD_INDEX_EXAMPLES (TEXT_A,TEXT_B) 
VALUES 
    ('I love dogs and cats','I love cats'),
    ('Coding is great','I Hate Coding'),
    (
        'Raisa Energy is a leading oil & gas investment and management company', 
        'Raisa Energy company is focused on oil & gas investment'
    ),
    ('Well Name 10 15 20H','Name Well 10 15 J 20HX'),
    ('Did you sleep well ?', 'Are you well ?')
;

SELECT 
    TEXT_A,
    TEXT_B,
    -- split by the spaces and take only the distinct words
    ARRAY_DISTINCT(SPLIT(LOWER(TEXT_A),' ')) TEXT_A_TOKENS,
    ARRAY_DISTINCT(SPLIT(LOWER(TEXT_B),' ')) TEXT_B_TOKENS,
    -- the intersection between the two texts
    ARRAY_INTERSECTION(TEXT_A_TOKENS,TEXT_B_TOKENS) TEXT_INTERSECTED_TOKENS,
    -- the union by concatenating the two texts tokens and get the distinct tokens
    ARRAY_DISTINCT(ARRAY_CAT(TEXT_A_TOKENS,TEXT_B_TOKENS)) TEXT_UNION_TOKENS,
    ARRAY_SIZE(TEXT_INTERSECTED_TOKENS) / ARRAY_SIZE(TEXT_UNION_TOKENS) * 100 JACCARD_INDEX
FROM JACCARD_INDEX_EXAMPLES;

SPLIT:Splits a given string with a specified separator and returns the result as an array of strings.
LOWER: Returns the input string with all characters converted to lowercase.
ARRAY_DISTINCT: Excludes any duplicate elements that are present in the input array.
ARRAY_INTERSECTION: Returns an array that contains the matching elements in the two input arrays.
ARRAY_CAT: Returns a concatenation of two arrays.

Note:
keep in mind that the Jaccard similarity is a basic measure of similarity that simply considers the presence or absence of words in the texts. The frequency or order of the words are not taken into consideration. You might want to look at more complex similarity measurements depending on your particular use case.

2. Levenshtein Distance

Levenshtein Distance (aka Edit Distance) is used to compare between two texts. It gives a score from 0 to infinity (∞). The lower the score the more similar the two texts are to each other.

It counts the minimum number of single character changes (insertions, deletions or substitutions) to convert the source text to the target text.

Example:

1. raesa → raisa (substitution of 'e' for 'i')
2. raisas → raisa (deletion of 's' at the end)
3. aisa → raisa (insertion of 'r' at the beginning)

All of those examples have a distance of 1.

Snowflake Example:

CREATE TEMP TABLE LEVENSHTEIN_DISTANCE_EXAMPLES (TEXT_A VARCHAR, TEXT_B VARCHAR);
INSERT INTO LEVENSHTEIN_DISTANCE_EXAMPLES (TEXT_A,TEXT_B) 
VALUES 
    ('william','williams'),
    ('ntiresys','interest'),
    ('carray','bark'),
    ('North','South'),
    ('Energy', 'Inrgy')
;

SELECT 
    TEXT_A,
    TEXT_B,
    -- Get the L. distance between the two texts by snowflake built-in function
    EDITDISTANCE(TEXT_A,TEXT_B) L_DISTANCE
FROM LEVENSHTEIN_DISTANCE_EXAMPLES;

EDITDISTANCE: Computes the Levenshtein distance between two input strings. It is the number of single-character insertions, deletions, or substitutions needed to convert one string to another.

Note:
Unlike some other metrics (e.g. Damerau-Levenshtein distance), character transpositions are not considered.

3. Jaro-Winkler

Jaro-Winkler Similarity is used to compare between two texts, providing a score from 0% to 100%. The higher the score the more similar the two texts are to each other.

It's similar to Levenshtein Distance, but takes into consideration the number of transpositions between characters.

Note:
There is a metric called 'Jaro' which 'Jaro-Winkler' is based upon.
The difference is that 'Jaro-Winkler' takes into account the common substring prefix between the two texts.

$$
Jaro~Similarity =
sim_j = \begin{cases}
0 & \text{if } m = 0 \\
{1 \over 3}({m \over |s_1|}+{m \over |s_2|}+{m-t \over m}) & \text{otherwise}
\end{cases}
$$

$$
JaroWinkler =
sim_w = sim_j + l * p * (1-sim_j)
$$

• S_i: Length of word.
• m: Number of matching characters.
• t: Number of transpositions.
• l: length of common prefix at the start of the string up to max of 4.
• p: Constant Scaling Factor should not exceed .25(if l is max at 4), Standard Value = 0.1.

Example:
Src Word: 'risaa', Target Word: 'raisa'

$$
sim_j ={1 \over 3}({5 \over 5}+{5 \over 5}+{5-2 \over 5})= 86.67\%
$$
$$
sim_w = .8667 + 1 * 0.1 * (1-.8667)= 88\%
$$

Snowflake Example:

CREATE TEMP TABLE JARO_W_EXAMPLES (TEXT_A VARCHAR, TEXT_B VARCHAR);
INSERT INTO JARO_W_EXAMPLES (TEXT_A,TEXT_B) 
VALUES 
    ('william','williams'),
    ('risaa','raisa'),
    ('carray','bark'),
    ('North Pole','North'),
    ('Energy', 'Inrgy')
;

SELECT 
    TEXT_A,
    TEXT_B,
    -- Get the jaro-winkler score between the two texts by snowflake built-in function
    JAROWINKLER_SIMILARITY (TEXT_A,TEXT_B) JARO_W_SCORE
FROM JARO_W_EXAMPLES;

JAROWINKLER_SIMILARITY: Computes the Jaro-Winkler similarity between two input strings. The function returns an integer between 0 and 100, where 0 indicates no similarity and 100 indicates an exact match.

Note:

• The similarity computation is case-insensitive.
• For short strings, Runtime Complexity is O(|S₁|*|S₂|).
• For long strings, Runtime Complexity is O(MAX(|S₁|,|S₂|)).

Use Cases

1. Jaccard Index - Tailored for Relevance 📜:
   • Search Engines: It excels in search engines by measuring the relevance of documents to user queries, helping deliver precise search results based on common terms.
   • Recommendation Systems: Jaccard Index shines in e-commerce, suggesting products based on user interests and attributes, enhancing personalized recommendations.
2. Levenshtein Distance – Typo Terminator 🤖:
   • Typo Correction: Levenshtein Distance is the go-to tool for spell-checkers and auto-correct systems, quietly fixing typing errors in various applications.
   • Auto-Complete: It speeds up text input by predicting the next word you're likely to type, making your typing experience smoother.
   • Bioinformatics: In genetics, it's an indispensable detective, uncovering tiny mutations in DNA or protein sequences.
3. Jaro-Winkler - The Data Matchmaker 🕵️‍:
   • Record Linkage: Jaro-Winkler's strength lies in data integration and record linkage tasks, where it excels at identifying matching records across different datasets, even when there are minor differences or typos.
   • Name Matching: It's a powerful tool for finding similar names in databases, ensuring accurate identification, and reducing duplicates.
   • Geographic Data Matching: In geographic information systems (GIS), Jaro-Winkler ensures precise matching of place names, even when there are slight variations, enhancing data accuracy.

Benchmarking Jaro-Winkler Similarity Scores for Different Warehouse Sizes

We've explored different similarity techniques, navigating the accuracy trade-offs that each one introduces based on varying use-cases. Now, let’s dive into a real-world scenario. Imagine you have a sample dataset with n rows, and the desire to find the similarity of each of these rows against a fixed reference dataset with m rows. The plan? A classic cross join of n and m, calculating a score for each pair, and cherry-picking the highest one. But, of course, this comes with a potential price tag.

If we opt for Jaro-Winkler as the chosen similarity technique for our use-case, what's the price tag for using different sample datasets against a substantial 2.5M rows reference dataset? And how does this cost fluctuate with the size of our warehouse? Let's dig into the numbers and draw some insights.

This experiment shows a positive correlation between scaling up the warehouse size and performance improvement, thanks to augmented computing resources. A larger warehouse size means more processing power, memory, and storage capacity. When dealing with extensive cross joins and similarity score calculations, having more resources allows the database to handle the computational load more efficiently. However, it's crucial to note that this enhancement comes at a monetary cost.

In weighing the pros and cons, if budget constraints or time considerations are factors, and expanding computational resources might not align with your goals, this approach may warrant reconsideration - especially if your sample data scales considerably. As with any technological choice, the decision ultimately depends on the specific needs, priorities, and constraints of your project. Evaluating the trade-offs and considering alternative approaches will guide you to the most suitable solution for your text similarity requirements.

Conclusion

In conclusion, exploring text similarity in Snowflake can unlock a range of possibilities. By utilizing powerful algorithms and leveraging Snowflake's user-friendly environment, you can easily implement text similarity analysis without the need for extensive programming. Snowflake's streamlined approach empowers users to uncover insights, make data-driven decisions, and extract valuable information from textual data efficiently. Snowflake provides a simplified yet robust solution for text similarity tasks, enabling users to harness the power of text analysis with ease.

Our goal was to introduce you to the practical examples of text similarity in Snowflake. These algorithms are powerful and usually involve complex programming in any language, even with the available libraries, but Snowflake makes their implementation incredibly straightforward.

Want to Know More?

Check out these links:

• Comparing string similarity algorithms for different apps
• Online string similarity comparison app, using various metrics
• Introduction to semantic similarity using NLP and Python

Overview

Have you ever wondered how to make data more accessible to other employees in your organization without them having to deal with the headache of SQL queries? We would like to share with you our attempt at solving this problem by making a cost-efficient Text-to-SQL chatbot at Raisa.

In this article we begin by introducing the Text-to-SQL task, current best approaches to the problem, and how our implementation balances accuracy with cost-effectiveness. Without further ado, let's get into business!

Introduction

The main goal of Text-to-SQL chatbots is to provide users with a natural language interface to relational databases. This eliminates the burden of learning SQL and makes data more accessible to users.

The general steps of a Text-to-SQL chatbot are to take the user question in English, create the corresponding SQL query that answers the question, and finally execute the query on the database and return the result to the user.

Historically, researchers have tried to tackle the Text-to-SQL task using multiple approaches that can be categorized into rule-based approaches, training/fine-tuning approaches, and in-context learning approaches tied to LLMs.

Rule-based approaches require manually created logic, are difficult to implement, and only achieve good accuracy in specific cases. On the other hand, training/fine-tuning approaches revolve around training language models or fine-tuning pre-trained models on a more specific task. Finally, in-context learning involves the user instructing a language model on how to perform tasks through the prompt itself. This approach has recently flourished with the emergence of LLMs.

Since rule-based approaches depend on manual logic which can be difficult and impractical to apply, we can say that the following table summarizes the most promising approaches along with the different methods of deployment.

Literature Survey

When starting a new project in Data Science/Machine Learning it is very instructive to spend some time researching and exploring related benchmarks in order to get a sense of recent advancements and state-of-the-art approaches. In the Text-to-SQL task an important and popular benchmark is the Spider benchmark. Another benchmark that has recently gained popularity is called BIRD which targets more realistic use cases than the Spider benchmark.

Observing the leaderboards for both Spider ^[1] and BIRD ^[2], one can notice that LLMs have recently been dominating the top ranks and achieved state of the art results using in-context learning, without the need for any training whatsoever.

Since LLMs show the most promising results, we decided go with in-context learning using efficient prompting techniques rather than using fine-tuning approaches. However, there is still one more question to answer: should we host our own LLM (like Llama 2) on-premises or use LLMs through APIs (like OpenAI)? One way to answer that question is to do a cost-benefit analysis of going down each path, and we can almost always find that hosting LLMs in-house will give a low return on investment when model usage is low (as was our case) ^[3].

In both the BIRD and Spider benchmarks the leading approach at the time of our research and implementation was the DIN-SQL ^[4] approach equipped with GPT-4, which we will discuss in detail below.

DIN-SQL Approach

The main idea of the DIN-SQL approach is to decompose the Text-to-SQL problem into smaller sub-tasks which can improve LLM performance by allowing it to solve easier sub-tasks rather than solving the original task all at once. The approach also utilizes chain-of-thought prompting which has been shown to drastically improve LLM performance on various tasks ^[5].

Let's delve more into each step of the DIN-SQL approach:

Schema Linking

The first sub-task is responsible for identifying the columns that will be used from each table as well as condition values for these columns. To do this, the LLM is first given the database schema and the user question, then it is instructed to think step by step and identify the columns and condition values.

To give the LLM a better idea of the required sub-task, 10 examples are given to the model as well. Below, you can find one such example that was given in the prompt.

// Input
Table course, columns = [*,course_id,title]
Table instructor, columns = [*,ID,name,salary]
Table teaches, columns = [*,ID,course_id,semester,year]
Foreign_keys = [course.course_id = teaches.course_id, instructor.ID = teaches.ID]
Q: "Find the id of instructors who taught a class in Fall 2009 but not in Spring 2010."

// Output
A: Let’s think step by step. In the question "Find the id of instructors who taught a class in Fall 2009 but
not in Spring 2010.", we are asked:
"id of instructors who taught " so we need column = [teaches.id]
"taught a class in" so we need column = [teaches.semester,teaches.year]
Based on the columns and tables, we need these Foreign_keys = [].
Based on the tables, columns, and Foreign_keys, The set of possible cell values are =
[Fall,2009,Spring,2010]. So the Schema_links are:
Schema_links: [teaches.id,teaches.semester,teaches.year,Fall,2009,Spring,2010]

Classification & Decomposition

In this step, the model tries to predict the complexity of the required SQL which can vary drastically depending on the user question. For example, some SQL queries are very simple while others are more complex and require joining tables and nested queries. Determining the complexity of the required SQL query can be crucial for generating the correct solution.

To do the classification, the LLM is given the columns and the condition values from the Schema Linking step along with the user question. It is then instructed to think step by step and classify the question based on joining and nesting into one of the following types:

• "Easy" query: a simple query that needs one table and doesn't need any nested queries.
• "Non-Nested" query: a query that needs joining multiple tables but doesn't need any nested queries.
• "Nested" query: a query that needs nested queries (may or may not need joining)

10 examples are given to the model here as well, two of which are shown below.

// Input
Q: "Find the buildings which have rooms with capacity more than 50."
schema_links: [classroom.building,classroom.capacity,50]

// Output
A: Let’s think step by step. The SQL query for the question "Find the buildings which have rooms with
capacity more than 50." needs these tables = [classroom], so we don’t need JOIN.
Plus, it doesn’t require nested queries with (INTERSECT, UNION, EXCEPT, IN, NOT IN), and we need
the answer to the questions = [""].
So, we don’t need JOIN and don’t need nested queries, then the SQL query can be classified as
"EASY".
Label: "EASY"

// Input
Q: "Give the name and building of the departments with greater than average budget."
schema_links: [department.budget,department.dept_name,department.building]

// Output
A: Let’s think step by step. The SQL query for the question "Give the name and building of the
departments with greater than average budget." needs these tables = [department], so we don’t need JOIN.
Plus, it requires nested queries with (INTERSECT, UNION, EXCEPT, IN, NOT IN), and we need the
answer to the questions = ["What is the average budget of the departments"].
So, we don’t need JOIN and need nested queries, then the the SQL query can be classified as "NESTED".
Label: "NESTED"

SQL generation

In this step, the LLM is given the user question and the outputs of all the previous steps and is asked to generate the SQL query. In order to increase the accuracy of SQL generation, the LLM is instructed differently depending on the complexity prediction from the previous step.

• "Easy" query: the model is simply instructed to generate the SQL.
• "Non-Nested" query: the model is instructed to first think step by step about the tables that must be joined, then to generate the SQL.
• "Nested" query: the model is first instructed to think step by step about any necessary sub-queries, then to generate the final SQL.

10 examples are given to the model in each type. The example shown below is for the "Nested" query since this involves the most steps.

// Input
Q: "Give the name and building of the departments with greater than average budget."
Schema_links: [department.dept_name,department.building,department.budget]

// Output
A: Let’s think step by step. "Give the name and building of the departments with greater than average
budget." can be solved by knowing the answer to the following sub-question "What is the average budget
of departments?".
The SQL query for the sub-question "What is the average budget of departments?" is SELECT avg(budget)
FROM department
So, the answer to the question "Give the name and building of the departments with greater than average
budget." is =
Intermediate_representation: select department.dept_name , department.building from department where
@.@ > avg ( department.budget )
SQL: SELECT dept_name , building FROM department WHERE budget > (SELECT avg(budget) FROM department)

Self-Correction

The purpose of this final step is to check for any bugs in the generated SQL. The LLM is given the full database schema, the user question, the SQL output from the previous step, and a set of hints, then it is instructed to correct the bugs in the SQL (if any).

// Input
#### For the given question, use the provided tables, columns, foreign keys, and primary keys to fix the given SQLite SQL QUERY for any issues. If there are any problems, fix them. If there are no issues, return the SQLite SQL QUERY as is.
#### Use the following instructions for fixing the SQL QUERY:
1) Use the database values that are explicitly mentioned in the question.
2) Pay attention to the columns that are used for the JOIN by using the Foreign_keys.
3) Use DESC and DISTINCT when needed.
4) Pay attention to the columns that are used for the GROUP BY statement.
5) Pay attention to the columns that are used for the SELECT statement.
6) Only change the GROUP BY clause when necessary (Avoid redundant columns in GROUP BY).
7) Use GROUP BY on one column only.

Table department, columns = [*,Department_ID,Name]
Table manager, columns = [*,Manager_ID,Department_ID,Name,Age]
Foreign_keys = [manager.Department_ID = department.Department_ID]
Primary_key = [department.Department_ID, manager.Manager_ID]
#### Question: How many managers of the departments are older than 56 ? 
#### SQL Query
SELECT COUNT(*) FROM department WHERE Age > 56

// Output
#### Fixed SQL Query
SELECT COUNT(*) FROM manager WHERE Age > 56

Proposed Approach

Although the DIN-SQL paper achieves incredible accuracy, it does so at a high monetary cost. For example, the paper suggests using GPT-4 to get the highest accuracy. However, the price of using the GPT-4 API is more than 20 times the cost of GPT-3.5 ^[6], which raises an important question: do we really need to use GPT-4 in every step to achieve good accuracy? Furthermore, are all the steps in the DIN-SQL approach really necessary?

To decrease the cost, we propose two modifications:

1. Use GPT-3.5 in as many steps as possible while reserving GPT-4 for the most difficult steps.
2. Eliminate and simplify steps which use GPT-3.5 (to make it easier for the model).

Let's discuss the changes in more details:

Simplifying the Steps:

Columns Selection

This step replaces the Schema Linking step in the DIN-SQL approach. The main difference is that the model is only tasked with selecting appropriate columns without having to also identify the condition values for these columns. We also give the model a detailed description for each column which helps a lot with ambiguous column names.

// Input
CREATE TABLE [department] (
	[Department_ID] INTEGER NOT NULL, --The ID of the department, a unique id for each department
	[Name] varchar(64) NULL, --The name of the department.
	CONSTRAINT [PK_department] PRIMARY KEY ([Department_ID])
)
CREATE TABLE [manager] (
	[Manager_ID] INTEGER NOT NULL, --The ID of the manager, a unique id for each manager
	[Department_ID] INTEGER NOT NULL, --The ID of the department, a unique id for each department
	[Name] varchar(64) NULL, --The name of the manager.
	[Age] INTEGER NULL, --The age of the manager.
	CONSTRAINT [PK_manager] PRIMARY KEY ([Manager_ID])
)
Foreign_keys = [manager.Department_ID = department.Department_ID]
Q: "How many managers of the departments are older than 56 ?"

// Output
A: Let’s think step by step. In the question "How many managers of the departments are older than 56 ?", we
are asked:
"older" so we need column = [manager.Age]
Based on the columns and tables, we need these Foreign_keys = [].
So the needed columns are:
Columns: {"manager": ["Age"]}

Classification & Decomposition

For our specific use case, we can further simplify this step since the number of tables used in our POC is no more than three. Any joins between these tables are also all done using the same foreign key.

For these reasons, we can classify query complexity based only on nesting into one of the following types:

• "Non-Nested" query: a query that does not require nested queries.
• "Nested" query: a query that requires nested queries.

This modification simplifies the decomposition step, making it easier to use GPT-3.5, and reduces the number of necessary examples in the prompt, thereby decreasing the cost even more.

// Input
Q: "Find the buildings which have rooms with capacity more than 50."
schema:
CREATE TABLE [classroom] (
	[building] varchar(10) NOT NULL, -- The building name, e.g. "Packard"
	[capacity] integer NULL, -- The capacity of the building, e.g. "500"
)

// Output
A: Let’s think step by step. The SQL query for the question "Find the buildings which have rooms with
capacity more than 50." doesn’t require nested queries with (INTERSECT, UNION, EXCEPT, IN, NOT IN), and we need
the answer to the questions = [""].
So, we don’t need nested queries, then the SQL query can be classified as
"NON-NESTED".
Label: "NON-NESTED"

SQL generation

Similar to the SQL generation step in the original approach, the instructions given to the LLM depend on the predicted SQL complexity.

• "Non-Nested" query: the model is simply instructed to generate the SQL, just like Easy type queries in the DIN paper.
• "Nested" query: the instructions for the Nested queries are kept the same as those in the DIN paper.

// Input
Q: "Find the buildings which have rooms with capacity more than 50."
Schema:
CREATE TABLE [classroom] (
	[building] varchar(10) NOT NULL, -- The building name, e.g. "Packard"
	[capacity] integer NULL, -- The capacity of the building, e.g. "500"
)

// Output
SQL: SELECT DISTINCT building FROM classroom WHERE capacity > 50

Self-correction

This step is similar to the DIN-SQL paper except it's only applied if the generated SQL query gives an error when we try to execute it.

Modifications Summary

The table below summarizes the changes done to the task breakdown of the DIN-SQL paper.

Blending between GPT-3.5 & GPT-4:

Since using the GPT-4 API costs 20 times as much as using the GPT-3.5 API, we recommend delegating most of the steps to GPT-3.5 and restricting GPT-4 to only the most difficult steps. We argue that the Nested SQL generation step is the hardest step and therefore only use GPT-4 for generating nested queries.

To gauge our approach's performance, we created a small dataset of around 120 examples, each composed of a user question and a target SQL query. The table below summarizes the results of using three different methods:

1. Using GPT-3.5 for all steps
2. Using GPT-4 for all steps
3. Using GPT-4 only for the Nested SQL generation and using GPT-3.5 otherwise

We can see that the last method reduces the cost significantly compared to using GPT-4 in all the steps while maintaining high accuracy.

References

[1]: Spider: Yale Semantic Parsing and Text-to-SQL Challenge | Tao Yu et al.

[2]: BIRD-bench | Jinyang Li et al.

[3]: Why choose Llama 2 over ChatGPT? | Damien Benveniste

[4]: DIN-SQL: Decomposed In-Context Learning of Text-to-SQL with Self-Corretion | Mohammadreza Pourreza and Davood Rafiei

[5]: Chain-of-Thought Prompting Elicits Reasoning in Large Language Models | Jason Wei et al.

[6]: Pricing for OpenAI models

Imagine logging into your application's database and finding a change in a data record that just doesn’t make any sense. You trace the application code, check for any bugs, sit with the users, try reproducing the change and all of this doesn't get you anywhere!  

Wouldn't it give you a head start in your debugging process if there was a way to find out when the record changed and how it looked like before that change?

There are many ways to store your application’s historical data in SQL Server. Some people prefer doing this directly from the application level, others prefer using some of SQL Server's features such as Temporal Tables, Change Data Capture, Triggers and Change Tracking. In this article we will be talking about two approaches: saving historical records directly through your application code versus doing so using SQL Server Temporal Tables.

Example:
The example we will discuss in this article is the database for school students’ grades. We will be focusing on the table that keeps the final grades and how the history of the grades per student can be recorded. The grades are entered by teachers along the semester, and they are reflected on the FinalGrade table for each student to see where they stand. For simplicity I will not use any foreign keys. The FinalGrade table will look something like this:

This table contains each student’s final grades for the semester. Each time a teacher enters a new grade, say a quiz worth 10 points, the final grade is updated to reflect this addition. All these records were created on the first day of the semester, January 10th in our case. All the grades are originally set to 0 for all students. At the end of the semester these records will look something like this:

Goal: Catch the history that led the records to be the way they are. For example, Layla has a total of 533 points. If we want to know the history of how this record reached the current state, how would we save these steps?

Application Side History Recording:

In this approach, the design simply revolves around the idea that the logic of saving the historical record is done from the application side. There are two ways of keeping the historical data in the database:

1. We can either keep them in a separate table, for instance “FinalGradeHistory”, and choose the table’s schema as we want. We can also have a foreign key in the history table to the record in the main table for which this history record was inserted. This separate table can be in the same database or in a different one for better performance.
2. We can keep them in the same table and use “Effective Dating”. That is, each record will have an Effective Start Date and an Effective End Date. The most recent record in this case will have the Effective End Date = null. In some cases, if the dates are not needed, we can just have a boolean value for each record “IsEffective” to identify whether it is effective or not.

Applying the "Separate Table" way:

The goal is to save the history data to the FinalGradeHistory table.

1. Create the database table in which the history will be kept. As mentioned before, we can use whichever schema we want. I chose not to add the CreatedBy and CreatedAt columns because the data would be redundant. The CreatedBy and CreatedAt in our case are going to be the same for the lifetime of each record.

2. Now to the application level, each time there is an update or delete statement applied to the FinalGrade table, there must be an insert statement to the FinalGradeHistory table to capture the old values. The insertion to the history table will simply be a copy of the already existing record before the update. The update function would look something like this:

3.  After some updates to the FinalGrade table, querying the FinalGradeHistory table by running

should give us something like this:

Pros:

• Flexibility and control over what to save and what to discard. For example, if we choose to keep the data in a separate table, we can choose the schema we want as some data from the main table might be irrelevant and redundant if kept in the history table.
• Ability to record who did the modification. To do that, you could simply get the user id from the database context in your application level and save it in the historical record before inserting it to the database. We will see that this is not as simple in the Temporal Tables approach.

Cons:

• As they say, with great power comes great responsibility. While this approach gives the developer control over what to save, they need to be sure that each time the application level writes into the main table, it saves the history as well. The logic needs to be applied with every update and delete.
• In case of running scripts directly on the database, changes are not going to be reflected automatically on the history table. It will be the responsibility of the developer to cover recording the history in this case as well. This is a problem we will not face with Temporal Tables.
• If “Effective Dating” is used, the main table can get too big, which might eventually affect performance.
• It does not tell you directly what changed. If you need to know which field(s) specifically changed, you will need to compare the latest record with the latest history record.  

Final thoughts:
Although using the application-level approach might work as a charm for some systems, it might be a headache for others. The goal is to choose the better fit for your system; what works best for your data, what exactly do you want to capture, how much effort are you willing to put to maintain these tables and so on.

Now that we are aware of this approach's strengths and weaknesses, what advantages do Temporal Tables have to offer? Which disadvantages raise a flag and could be a reason to why you would want to use this way over the other? Let's talk more about Temporal Tables in Part 2!

We often need to expand or generalize the output of clustering algorithms for unseen data points in training without recomputing the clustering. Some clustering algorithms support this by nature, which is called Inductive Clustering. This article will discuss extending the non-inductive clustering algorithms to classify new instances and classify data samples with lower dimensions than those used for creating clusters.

Keywords: Inductive Clustering, Clustering Generalization

Problem Definition

Let us start by introducing some concepts from the oil and gas industry.

• Completion Features: features that describe the process of making a well ready for production after drilling operations
• Proved Developed Producing Oil and Gas Reserves (PDPs): oil and gas wells that are proven to have a production and already producing
• Proven Undeveloped Oil and Gas Reserves (PUDs): oil and gas Reserves that are proven to have oil production but still some development and execution are remaining before the oil is produced.

Some Context

In Rasia, we use a mix of learning and rule-based algorithms for clustering PDPs. These clusters are made from the wells' location, and geological features. We can then use these clusters to verify the production prediction for a given well by comparing it to other wells within its cluster.

Goal

We aim to generalize these clusters for PUDs to predict the production and completion feature patterns for a given PUD by assigning the PUD to one (or more) of the existing clusters. To achieve this, we will need to tackle the following challenges:

• For PUDs, we do not have as many features as we do for PDPs.
• Try not to modify the clustering scheme as much as possible.
• Try to assign multiple clusters for each well with a given probability for each cluster to capture the difference between the dimensions of PUDs and PDPs

We can formulate this by repreasenting a PUD feature vector \( x \in R^{n-m} \) and set of the cluster \( \{ c_1, c_2, ..., c_k \} \) that is generated from PDPs feature vectors \( v \in R^{n} \).
Our goal is to assign a probability for each cluster \( \{ p_1, p_2, ..., p_k \} \) that represents how likely this PUD will be in this cluster after completion.

Inductive Clustering Algorithms

Some clustering algorithms induce functions for classifying the whole space of interest; these methods are called inductive clustering. An example of these algorithms is the fuzzy c-means clustering algorithm. Instead of putting each data point into separate clusters, a probability of that point being in that cluster is assigned. In fuzzy clustering, also called soft clustering, each data point can belong to multiple clusters along with its probability score or likelihood.

In our problem, using any of the soft clustering algorithms will have two main downsides: firstly, it will require a complete change in our clustering scheme. Secondly, it will only work if the test data sample has the same features as the training data or \( R^{n-m} = R^{n}\) .

Extending Clustering With Estimators

Instead of changing the clustering scheme, we can use the clustering output to train an inductive model with a classifier and use this classifier to scale to new data. This approach has several benefits:

• It works independently from the used clustering algorithm, which makes this a modular approach that is consistent over time.
• It allows us to use the inferential characters of the classifier to describe or explain the clusters, more on this later.

Another essential benefit for our use case is that we can train the classifier using only the location features to use the model for PUDs.

Choosing the classifier

The choice of the classification algorithm will influence the properties of generated clustering regions. As we cannot determine how the clustering regions will fill all the input space from the input data, a proper understanding of both the decision boundaries generated from different algorithms and the nature of the problem you are trying to solve are essential for choosing the best algorithm for the problem.

Implementation Walkthrough

Looking at the data

This dataframe contains the PDP data that is used for generating clusters, and the output cluster for each well, the description of each feature is as follows:

• API: Id for each well
• Latitude: location information for well
• Longitude: location information for well
• totalDepthTVD: total true vertical depth
• GOR90: gas to oil ratio (geological feature)
• WOR90: water to oil ratio (geological feature)
• Label: Id for the cluster to which the well belongs

As we mentioned earlier, we will only use Latitude and Longitude features to train our model as they are the only available features for PUDs case.

Plotting clusters with location features

Before training our classifier, we need to know the amount of lost information by using only the location features. We can discern it by showing the overlap between clusters when projected onto the lat-long 2D plane.

Classifier Training

We will start our experiments by training a KNN classifier using Longitude and Latitude as the features and see our performance on the validation set. We use RobustScaler from sklearn for features scaling to minimize the effect of outliers and make_pipeline to combine features preprocessing, and model in one interface.

X = df[["Longitude", "Latitude"]].values
y = df['Label'].values

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.33, random_state=42)

knn = KNeighborsClassifier()

knn_pipe = make_pipeline(RobustScaler(), clf)
knn_pipe.fit(X_train, y_train)
print(f"Validation Accuracy: round({knn_pipe.score(X_test, y_test)}, 3)")
>>>>> Validation Accuracy: 0.987

Top-3 Accuracy Improvement

Instead of predicting one cluster, we can use the predict_proba function to predict a probability for every cluster, these probabilities can be used to determine if the well is in an area with overlapping clusters.

from sklearn.metrics import top_k_accuracy_score
y_hat = pipe.predict_proba(X_test)
print(f"Validation top-3 Accuracy: {top_k_accuracy_score(y_test, y_hat, k=3)}")
>>>>> Validation top-3 Accuracy: 1.0

Comparing The Clustering Regions From KNN and RF

To compare clustering boundaries from KNN and RF models. We will train an RF model the same as we did for KNN, then use the model to predict all the points in the space and plot  the outputs.

rf = RandomForestRegressor(n_estimators=10)
rf_pipe = make_pipeline(RobustScaler(), clf)
rf_pipe.fit(X_train, y_train)

To create an input representing all input space, we need to create a grid with a specific resolution using min and max values for Longitude and Latitude the pass this grid as input to the two models.

x0_min, x0_max = df["Longitude"].min(), df["Longitude"].max()
x1_min, x1_max = df["Latitude"].min(), df["Latitude"].max()

xx0, xx1 = np.meshgrid(
np.linspace(x0_min, x0_max, 800),
np.linspace(x1_min, x1_max, 800),
)

X_grid = np.c_[xx0.ravel(), xx1.ravel()]
knn_response = knn_pipe.predict(X_grid)
rf_response = rf_pipe.predict(X_grid)

Plotting the resulted boundaries using contourf function from matplotlip

fig, axs = plt.subplots(1, 2)
fig.set_size_inches((14, 6))

axs[0].contourf(xx0, xx1, rf_response, levels=len(data["Label"].unique()), alpha=0.6)
axs[0].set_title('Random Forest')

axs[1].contourf(xx0, xx1, knn_response, levels=len(data["Label"].unique()), alpha=0.6)
axs[1].set_title('knn')

for ax in fig.get_axes():
    ax.label_outer()

plt.show()

Comparing the two outputs, we found that the regions created from the KNN model are more suitable for our use case as they are more understandable and can be visualized easily with maps applications.

Conclusion

In this article, we discussed the concept of inductive and non-inductive clustering and how to extend non-inductive algorithms for classification over all input space. We also discussed handling cases when we need to cluster data with dimensions lower than the data used for creating clusters and how this can be useful for solving a real business problem.

References

• Miyamoto, Sadaaki. (2012). Inductive and non-inductive methods of clustering. Proceedings - 2012 IEEE International Conference on Granular Computing, GrC 2012. 12-17. 10.1109/GrC.2012.6468710.
• James C. Bezdek, Robert Ehrlich, William Full, FCM: The fuzzy c-means clustering algorithm, Computers & Geosciences, Volume 10, Issues 2–3, 1984, Pages 191-203, ISSN 0098-3004, https://doi.org/10.1016/0098-3004(84)90020-7.
• Inductive Clustering sklearn documentation: https://scikitlearn.org/stable/auto_examples/cluster/plot_inductive_clustering.html

According to a new market pulse survey conducted by Matillion and International Data Group (IDG), which drew insight from 200 enterprise data professionals, companies are drawing from an average of over 400 different data sources. It doesn't stop there, but the survey found that, on average, enterprise data volumes are growing by 63 percent a month. This survey also didn’t take into account internal datasets generated internally by the enterprises.

With this growing rate of data assets, it becomes almost impossible to track the data lifecycle and design the correct workflows.

That's when Data Lineage comes in as it essentially gives you a bird's eye view of a data flow which in return facilitates your workflows’ orchestration.

In this article we are going to discuss

• Data Lineage and its use cases
• Workflow orchestration and how data lineage play a role in it
• How did we manage to get full data lineage model and use it on our workflow orchestration
• How we implemented task orchestration guided with data lineage in our system

What is Data Lineage?

Data lineage is the process of tracking the data lifecycle in your system, from data sources to consumption and any transformation in between. This tracking is then visualized as a graph with data assets represented as nodes, thus giving us visibility of how each data asset was built and aggregated and who is responsible for it and each source transformation.

Data Lineage use cases

Identify important resources

Think about a scenario where you are subscribed to multiple data sources and now you want to evaluate these sources and see if it’s worth keeping your subscription there.

The best way to do that is to take a look at the data that is actually being consumed by your clients and how they impact your business. Afterwards, you can decide which data is useful to your business and accordinly identify what sources are worth keeping.

Impact Analysis

Think of another scenario where you want to change a certain data asset. You will have to keep in mind what other data assets will be affected by that change. You will need to be very careful with your change decisions and will even need to notify and coordinate with all the affected data asset consumers.

Troubleshooting

Another very important use case is troubleshooting and diagnosis. Say you see a weird behavior on a certain dashboard, the data lineage visualization will allow you to trace back to the source of any change and pinpoint the problem.

What is Workflow orchestration?

Now that we understand data lineage and its importance we can move on to the next part of the project - workflow orchestration.

A workflow is a set of sequences of tasks that need to be executed in a specific order due to their dependency. Orchestration is the process of automatically running your workflows on schedule and monitor their performance.

How do all the parts fall together…

Since our tasks are reading from data assets and creating other data assets, they are essentially building the data life cycle in our system and are tightly coupled with the data lineage.

If we want to know how your workflow should be, you can take a look at your data lineage and it will tell you. For example, if Task B consumes a data asset generated by Task A, then Task B should definitely be executed after Task A. Accordingly, we can say that data lineage can guide in deciding our task hierarchy and thus our workflows.

Creating your own data lineage: Steps breakdown

In order to create our lineage we ask the following questions:

• Where does our data live? In which databases or warehouses?
• What tools or programming languages are used in data transformation?
• How can we collect these queries while assigning them to whatever client dispatched this query?

Once we know the answer to these questions we can start designing our pipeline. For example, if you know that your data lives on Snowflake, you can start utilizing that cloud platforms features to your advantage and search for ways to pull out the query execution history by each client. However, if you use DBT for transformations, then DBT does the work for us as it keeps track of the data lineage.

Raisa Data Lineage pipeline

As for our system we know the data lineage comes from two sources:

• Data stored in SQL server has its transformations done through python tasks.
• Data stored in Snowflake has its transformations done with DBT

Let’s discuss how we are going to handle each of those and then how we are going to merge them to get full lineage.

Generic pipeline

A proposed pipeline for a custom lineage driven task orchestration system would be as follow:

Python tasks with SQL Server lineage

For our python tasks we needed to create our own data lineage.

Let's see how we made use of this pipeline!

• For gathering queries, we used SQL Extended Events to get the query history as well as get the python task that dispatched the query.
• We implemented a SQL queries parser to collect tables names and generate metadata for each python task.

Now we are going to discuss both of these steps in more detail.

Extended Events

SQL server extended events is a tool built inside SQL server used for monitoring specified events and their associated data.

Main building blocks

• Events

An Event is the triggers on which the extended session start taking specified actions. For our use case, the Events are SQL statement or stored procedure execution, so upon the execution of a SQL query the session is triggered.

• Actions

An Action is a programmed response to an event. We wanted to capture SQL text, database name and client application name, where both the database name and the client application name can be defined in the connection strings

• Targets

A Target is the event consumer. With each event trigger, data is gathered and this data needs to be stored somewhere.

Extended Events provide multiple targets that can be used to direct event output to a file or memory buffer. We used files on disk as a target, and later loaded the files into a table.

• Predicate

A Predicate is a filter applied to events such as capturing SQL queries dispatched by client applications with specific name or database related queries excluding server setup such as setting up a connection or checking for locks.

CREATE EVENT SESSION [PythonTasksLogs] ON SERVER 
ADD EVENT sqlserver.sql_statement_starting(
ACTION(sqlserver.client_app_name,sqlserver.database_name,sqlserver.sql_text)
WHERE (
        [sqlserver].[like_i_sql_unicode_string]([sqlserver].[client_app_name],N'%python_task::%') AND 
        [sqlserver].[like_i_sql_unicode_string]([sqlserver].[sql_text],N'%select%') 
    ))
ADD TARGET package0.event_file(SET filename=N'logs.xel',max_file_size=(10),max_rollover_files=(3))
WITH (
    MAX_MEMORY=4096 KB,
    EVENT_RETENTION_MODE=NO_EVENT_LOSS,
    MAX_DISPATCH_LATENCY=30 SECONDS,
    MAX_EVENT_SIZE=0 KB,
    MEMORY_PARTITION_MODE=NONE,
    TRACK_CAUSALITY=ON,
    STARTUP_STATE=ON
)
GO

These excel files are then parsed and loaded into a SQL table.

Advantages:

• Gathered Queries are raw, independent of the method of executing this query. This means that it is independent of the package or method that executed the query and even the programming language used.
• Minimal code change, only adding an App field to the connection string, so the session will be able to recognize the client executing this query. Other methods such as logging will require manually adding logging lines for each executed query and will also require maintenance in case the query changed or adding new functions programmers would add the logging lines themselves.

cnxn = pyodbc.connect(\
    "Driver={DRIVER_NAME};"
    "Server=server_name;"
    "Database=database_name;"
    "uid=username;"
    "PWD=password;"
    "App=pyhton_task::task1;")

Limitations:

• Limited file size as you can only ever specify limited size for your target files and although that protects your disk space it might rollover previous logs that in some cases you didn’t get the chance to parse and save
• Inserts with some pandas functions are translated into parameterized stored procedures which causes a lot of unnecessary logging and in case of large data frames insertion might rollover other logs
• Logging a lot of data to the disk will potentially increase task execution time

Parsing Queries and Generating Metadata

Now that we have the logs table, we can parse executed queries by each task to get the tables names. For this we built a class on top of sql_metadata package.

The package handles getting table names in nested queries, excluding common table expression (CTE) names (not considering them as tables), ignoring inline comments as well as block comments.

We need to also Identify whether the query was reading or writing so we assign the table type (input or output) according to the last SQL keyword (SELECT, INSERT.. etc.) that appeared before the table name.

At the end of this step we are able to gather for each task its input database objects and output database objects and these objects type (BASE TABLE or VIEW).

"task1": {
    "input": [
        {
            "_entity_name": "table_a",
            "_entity_type": "BASE TABLE",
            "_database_name": "db",
            "_schema_name": "dbo",
            "_generated_by": null
        },
        {
            "_entity_name": "table_b",
            "_entity_type": "BASE TABLE",
            "_database_name": "db",
            "_schema_name": "dbo",
            "_generated_by": null
        }
    ],
    "output": [
        {
            "_entity_name": "table_c",
            "_entity_type": "BASE TABLE",
            "_database_name": "db",
            "_schema_name": "dbo",
            "_generated_by": "python_task"
        }
    ]
}

DAGs

We want to represent our data lineage and tasks workflow in the form of a graph with dependencies as edges that has logical processing order for its nodes.

We must ensure that our graph has no cycles otherwise we won’t be able to have this processing order, and to achieve that we used topological sorting algorithm.

Note: processing tasks means executing them, while for data nodes it means generating or updating this database object.

Data Lineage DAG

Assuming that each task is responsible for creating one table as output, then all the input tables to this task had contributed to the creation of this output table.

So we start building the DAG in which the output database objects are considered nodes, where if task1 read from table_a and  table_b to generate table_c then we can imply that table_c is dependent on table_a and table_b.

nodes:
{
    table_c:{
        task_name: "task1",
        repo_url: "repo_url",       
        ins: 2
        outs: [table_d]
    }
    .
    .
},
edges: [ (table_a, table_c), (table_b, table_c),(table_c, table_d) ]        

Tasks workflow DAG

From Data Lineage DAG we can easily create similarly structured objects, but with tasks as nodes , since we already have database object dependencies and we also know which task is responsible for which database object creation.  

In the previous example task1 generates table_c. So if task2 reads from table_c then task2 is dependent on task1.

nodes:
{
    task1:{
        repo_url: "repo_url",       
        ins: 0
        outs: [task2]
    }
    .
    .
},
edges: [ (task1, task2) ]        

Data Build Tool (DBT) lineage

DBT provides an easy way to create, transform, and validate the data within a data warehouse.

As mentioned above the second source of data lineage is DBT transformations done on Snowflake, we need also to get its lineage and merge it with python tasks data lineage.

How to keep track of DBT lineage?

DBT keeps track of its lineage and writes it to a manifest.json file. We just needed to parse the file and convert it to the same DAG format we used for our python lineage in order to get the full lineage.

Now we can merge them together and have a DAG object with the full lineage!

Visualizing DAGs with pyvis

The pyvis library is meant for quick generation of visual network graphs with minimal python code. It is designed as a wrapper around the popular JavaScript visJS library.

For our lineage we used hierarchical layout for the graph to demonstrate the order in which either tasks should be executed or the database objects should be generated and refreshed.

def visaulize_dag_hierarichal(edges: List[List], nodes: dict,
                              nodes_order: List[List], dir: str) -> None:

    # setting layout to true makes the network have heirarichal structure
    nt = Network(height='100%', width='100%', directed=True,
                 layout=True, bgcolor='#021522', font_color="#cccecf")
                 
    # nodes order is list of lists returned by topological sort
    # each list represent a level
    # set level attribute for each network node
    for idx, sub_group in enumerate(nodes_order):
        for node in sub_group:
            color = "#0692b1" if nodes[node].type == 'BASE TABLE' or 'DBT Task' else "#77b84d"
            nt.add_node(node, color=color, name=node, level=idx,
                        shape='box', title="Text to show on hover over node")
                        
    for edge in edges:
        nt.add_edge(*edge)

    nt.show_buttons(filter_=["physics"])
    nt.show(dir+'.html')
    

The Visualization output is a html web page where you can zoom into nodes, drag them around and also highlight clicked node.

Visualizing our full Lineage using DBT docs

DBT docs offers a very powerful visualization for its lineage with very rich features such as

• Edges overlapping is highly optimized, you won’t see overlapping edges unless there is no other way, Also Nodes placement is very smart as they are always as near as possible to its upstream and downstream data assets where as our hierarchical visualization didn’t offer either and nodes where placed systematically on each level
• Focusing on a certain node and only view related downstream and upstream nodes
• Tagging nodes with a certain tag and then show them only or even hide them
• View columns of each asset, some statistics (like last materialization date)

To take advantage of these features we decided to inject our lineage into the DBT manifest.json and catalog.json which its docs engine uses to create the user interface.

So after we Inspected these files(manifest.json and catalog.json) we added our data nodes to it in the same format. Also we unified our data source names in both python and DBT in order for the lineage to be correct and if DBT considered a python generated table a source we changed it into a model as we now have this sort of knowledge.

After the full injection of python data assets we just served the docs and displayed the full lineage in the browser.

Orchestration tools: Airflow

Introduction

Airflow is a tool to programmatically author, schedule and monitor workflows.

It has a lot of useful functionalities including:

• Running multiple tasks in parallel either on the same computer node or remote nodes
• Metrics to control task retries, running instances of the same workflow, maximum time for a task to complete…etc.
• Rich Web UI, to gain visibility about the status of your workflows

Airflow core concepts

• DAG

Directed Acyclic Graph. An Airflow DAG is a workflow defined as a graph, where all dependencies between nodes are directed and nodes do not self-reference.

• Task

A step in a DAG describing a single unit of work.

• Operator

Describes a task behavior. For example, Python Operator takes a python callable function to execute where as a Bash Operator takes a bash command to execute.

• Sensors

Special operators, which pause the execution of dependent tasks until some criteria is met.

Our requirements

Since our tasks are either

• python tasks that has isolated repositories that run on a certain conda environment
• DBT models to materialize

We decided to rely on the bash operator to execute a script that:

• Pulls the most recent repository
• Activates a repository conda environment
• Runs a python file with the command or dbt command to materialize a model

if [[ ! -e task1 ]]; then
    git clone https://username:pwd@repo_url
elif [[ ! -d task1 ]]; then
    git pull https://username:pwd@repo_url
fi

conda activate python_env
cd ~/airflow/repos/task1
python `find ./ -name "main.py"`

source ~/miniconda3/etc/profile.d/conda.sh
conda activate dbt_env
dbt run --select dbt_model1

Automating Airflow DAG creation

Since we want to have minimal code and minimum maintenance to execute our workflows in the right order we made use of the Dag Factory package.

The package uses a yml file which is simply a template that holds:

• Airflow DAG configuration which can be configured dynamically by a parametrized python
• Tasks dependency which we already conclude from our Lineage
• Bash command which is also parametrized with tasks names and types

dag_yml:
  catchup: false
  default_args:
    owner: example_owner
    retries: 1
    retry_delay_sec: 300
    start_date: '2022-11-01'
  description: this is an example dag!
  schedule_interval: '*/1 * * * *'
  max_active_runs: 1
  tasks:
    task1:
      bash_command: '/home/airflow_user/airflow/run_task1.sh '
      operator: airflow.operators.bash_operator.BashOperator
    task2:
      bash_command: '/home/v/airflow/run_task2.sh '
      dependencies:
      - task1
      operator: airflow.operators.bash_operator.BashOperator
    task3:
      bash_command: '/home/airflow_user/airflow/run_task3.sh '
      dependencies:
      - task2
      operator: airflow.operators.bash_operator.BashOperator
    task4:
      bash_command: '/home/airflow_user/airflow/run_task4.sh '
      dependencies:
      - task3
      operator: airflow.operators.bash_operator.BashOperator
    task5:
      bash_command: '/home/airflow_user/airflow/run_task5.sh '
      operator: airflow.operators.bash_operator.BashOperator

This way each time we execute our data lineage pipeline, our tasks DAG will be up to date with the data lineage.

Summary

In this article, we discussed the importance of data lineage and how it can be used to create our tasks as an airflow DAG dynamically. Then, we went through how we actually implemented this in our system.

Reference Links

https://www.matillion.com/resources/blog/matillion-and-idg-survey-data-growth-is-real-and-3-other-key-findings

https://www.castordoc.com/blog/what-is-data-lineage

https://www.astronomer.io/blog/what-is-data-lineage/#task-hierarchy-versus-data-lineage

https://learn.microsoft.com/en-us/sql/t-sql/statements/create-event-session-transact-sql?view=sql-server-ver16

https://www.mssqltips.com/sqlservertip/4148/manage-extended-events-for-sql-server-2016-analysis-services-in-management-studio/#:~:text=The%20event%20retention%20mode%20is,performance%20degradation%20on%20the%20server.

https://www.sqlservercentral.com/blogs/sql-querying-xml-attributes-from-xml-column

https://learn.microsoft.com/en-us/sql/relational-databases/extended-events/quick-start-extended-events-in-sql-server?view=sql-server-ver16

https://www.askpython.com/python/examples/customizing-pyvis-interactive-network-graphs

https://medium.com/analytics-vidhya/getting-started-with-airflow-b0042cae3ebf

https://mlopshowto.com/an-apache-airflow-mvp-complete-guide-for-a-basic-production-installation-using-localexecutor-beb10e4886b2