Data Architecture

Data Cataloging

A centralized inventory of data assets across the organization — capturing technical metadata, business context, data lineage, ownership, and classification to enable discovery, governance, and trust.

⏱ 9 min read

What it is

A data catalog is a metadata management system that inventories an organization's data assets — databases, tables, columns, dashboards, ML models, data pipelines — and enriches them with technical metadata (schema, statistics, sample data), business metadata (descriptions, ownership, business glossary terms), operational metadata (last updated, row count, freshness), and governance metadata (classification, sensitivity, data steward). It serves as the "yellow pages" for data: when an analyst needs customer purchase data, they search the catalog to discover it exists, understand what it contains, assess its quality and freshness, and know who to contact.

Key catalog capabilities: Data lineage — which upstream sources feed a table, and which downstream reports or models consume it; Business glossary — organization-wide definition of terms like "active customer" or "revenue" tied to the physical columns that implement them; Data classification — labeling columns as PII, sensitive, restricted; Data discovery — search and browse interface for finding datasets; Quality scores — freshness, completeness, accuracy metrics surfaced inline.

Common tools: Apache Atlas (open-source, Hadoop ecosystem, strong lineage); DataHub (open-source, LinkedIn-originated, modern REST/GraphQL API, pre-built connectors); Amundsen (open-source, Lyft-originated, discovery-focused); Alation, Collibra, Atlan (commercial); AWS Glue Data Catalog, Google Dataplex, Microsoft Purview (cloud-native managed).

Why it exists

As organizations accumulate data across warehouses, lakes, operational databases, SaaS tools, and streaming systems, the cost of not knowing what data exists and how to trust it grows faster than the data itself. Analysts waste hours searching for the right table and asking colleagues for context. Duplicate datasets with subtly different definitions proliferate. A new GDPR deletion request arrives and no one knows all the systems where a user's data resides. An ML model is trained on a table that turns out to be a deprecated staging table with incomplete data. The data catalog solves organizational data opacity: it makes data discoverable, understandable, and governable without requiring tribal knowledge.

When to use

Growing data landscape: When the organization has more than a handful of data sources and analysts spend meaningful time finding and understanding data rather than analyzing it.
Regulatory compliance: GDPR, CCPA, HIPAA, and similar regulations require knowing where personal or sensitive data resides, who accesses it, and being able to execute deletion requests — a catalog with classification is foundational.
Data mesh implementation: Each domain team publishes their data products with catalog metadata; the catalog is the discovery layer that makes domain data products findable across the organization.
ML governance: Track which training datasets, feature tables, and model versions were used together — required for model auditing, reproducibility, and detecting data drift.
Impact analysis: Before modifying a table schema or deprecating a dataset, lineage in the catalog shows all downstream consumers (reports, dashboards, pipelines) that will be affected.

When not to use

Don't implement a full enterprise catalog before there's organizational commitment to maintain it — a catalog without data stewardship quickly becomes inaccurate and is abandoned; tool alone doesn't solve the governance problem.
Don't use a catalog as a substitute for data contracts between producers and consumers — the catalog describes data as-is; data contracts define obligations about how it will behave.
Don't expect automated metadata harvesting to replace domain knowledge — automated crawlers can discover schema and statistics but cannot infer business meaning, data quality context, or ownership without human curation.
Don't add a catalog before the underlying data architecture is reasonably stable — cataloging a rapidly changing schema creates maintenance overhead that discredits the catalog before it gains adoption.

Typical architecture


  Metadata Collection (automated + manual):
  ┌───────────────┐   ┌─────────────────┐   ┌──────────────┐
  │  Data Sources │   │  ETL/ELT Pipelines│   │  BI Tools    │
  │  (Snowflake,  │   │  (dbt, Airflow,  │   │  (Looker,    │
  │   Redshift,   │──▶│   Spark)        │──▶│   Tableau)   │
  │   Postgres)   │   └────────┬────────┘   └──────┬───────┘
  └───────┬───────┘            │                    │
          │    Connectors / Metadata extractors      │
          └──────────────┬─────────────────────────┘
                         ▼
               ┌─────────────────┐
               │   Data Catalog   │ (DataHub, Apache Atlas, Amundsen)
               │  ┌─────────────┐│
               │  │  Metadata   ││  - Tables, columns, schema
               │  │  Graph      ││  - Lineage (column-level if available)
               │  └─────────────┘│  - Business glossary terms
               │  ┌─────────────┐│  - Ownership, stewards
               │  │  Search &   ││  - PII / sensitivity classification
               │  │  Discovery  ││  - Quality scores / freshness
               │  └─────────────┘│
               └────────┬────────┘
                        │ APIs
          ┌─────────────┼─────────────┐
          ▼             ▼             ▼
     Data Analysts  Data Engineers  Compliance Team
     (discovery)    (impact analysis) (GDPR requests)

Pros and cons

Pros

Reduces data discovery time: analysts find the right dataset in minutes rather than hours of ad-hoc Slack messages and tribal knowledge hunting.
Enables data trust: data quality scores, lineage, and documentation let consumers assess fitness for their use case without relying on data producers.
GDPR and compliance readiness: classification metadata enables automated PII discovery and deletion request scope analysis across heterogeneous systems.
Impact analysis for changes: before deprecating or modifying a table, lineage shows all downstream consumers — preventing unannounced breaking changes.
Business-IT alignment: the business glossary bridges the gap between business domain terms and the physical tables implementing them — analysts and engineers share common vocabulary.

Cons

Requires organizational commitment: a catalog that nobody maintains becomes incorrect faster than it was built — data stewardship is a continuous process, not a one-time project.
Automated lineage gaps: column-level lineage requires instrumentation in pipeline tools (dbt lineage is excellent; ad-hoc SQL is nearly impossible to auto-parse correctly).
Integration complexity: connecting the catalog to every data source, pipeline, and BI tool requires connectors and ongoing maintenance as source systems evolve.
Classification accuracy: automated PII detection using pattern matching has false positives and negatives; a column named "data" containing email addresses may not be detected without sampling and ML classification.
Adoption friction: the catalog provides value only if users actually use it — adoption programs, gamification, and integration into existing workflows are needed to change habits.

Implementation notes

Start with automated metadata harvesting before manual curation: configure connectors to ingest schema, statistics (row counts, null rates, value distributions), and usage patterns (query logs identify popular and frequently-joined tables) from your primary data sources. Automated lineage from dbt models is particularly high-value — dbt's manifest.json exposes full model dependency graphs that catalogs like DataHub and Atlan ingest natively. Prioritize cataloging the 20% of datasets that 80% of queries touch, rather than attempting complete coverage before demonstrating value.

For GDPR compliance metadata, implement a data classification taxonomy at minimum: PII (directly identifies individuals), Quasi-PII (indirectly identifies when combined), Sensitive (financial, health, credentials), Internal, Public. Apply classifications at column level via a combination of automated pattern detection (regex for email, SSN, credit card formats), ML-based column name and sample value classifiers, and manual review by data stewards. Integrate classification into access control: classify first, then trigger access policy review for any newly discovered PII columns. Track classifications in version control alongside schema definitions rather than only in the catalog UI.

Common failure modes

Catalog without stewardship: The catalog is deployed, assets are auto-discovered, but no ownership model is established; descriptions remain empty, classifications are wrong, and within 6 months analysts stop trusting and using it.
Lineage black holes: Lineage is tracked within dbt models but ad-hoc SQL Stored Procedures and Jupyter notebooks that transform data are invisible — the lineage graph has gaps at critical transformation points.
Business glossary inconsistency: Multiple conflicting definitions for "customer" exist in the catalog (the marketing definition includes trials, the finance definition excludes them); no reconciliation process exists; analysts pick whichever definition suits their current report.
Over-ambitious initial scope: A team attempts to catalog 10,000 tables across 15 systems before launch; the project takes 18 months; by the time it launches, half the metadata is already stale.
Classification without enforcement: Columns are classified as PII but the classification isn't integrated with access control systems; any analyst can query PII columns regardless of classification.

Decision checklist

Is there an identified data stewardship model — who owns which domains and is responsible for keeping catalog metadata accurate?
Are the 10–20 highest-value data assets (most queried, most business-critical) prioritized for initial cataloging rather than attempting complete coverage?
Are automated connectors available for the primary data sources (Snowflake, Redshift, dbt, Airflow) to reduce manual metadata entry burden?
Is there a data classification taxonomy and an automated + manual classification process aligned with compliance requirements?
Is the catalog's classification metadata integrated with access control (column-level security) so that PII classification triggers access policy?
Is there a plan for driving adoption — integration into analyst workflows, making catalog search the default entry point for data discovery?

Example use cases

GDPR deletion request fulfillment: A user submits a right-to-erasure request. The data privacy team queries the catalog for all assets classified as containing that user's PII, filtered by the user's user_id column. The catalog returns 14 tables across 5 systems with column-level lineage. Engineers execute targeted deletion scripts against each asset within the 30-day regulatory window.
New analyst onboarding: A new data analyst joins and needs to find revenue data for a sales dashboard. They search "monthly recurring revenue" in the catalog's business glossary, find the canonical definition, click through to the physical columns in the fct_subscriptions table in Snowflake, see the dbt model that computes it, review the 4-day freshness SLA, and identify the data steward to contact with questions — all without asking a colleague.
Schema change impact analysis: An engineer needs to rename a column in a frequently-used table. Before making the change, they use the catalog's lineage view to identify 12 downstream dbt models, 3 Looker dashboards, and 1 Airflow pipeline that reference that column — producing a precise migration checklist.

Data Contracts — Data contracts define producer obligations; the catalog records what actually exists and how it's being used.
Data Mesh — Data catalogs are the discovery layer in Data Mesh architectures, enabling self-service access to domain-owned data products.
Change Data Capture — CDC-fed pipelines should emit lineage events to the catalog so that real-time derived datasets are tracked.