Data Governance for Analytics

What it is

Data governance for analytics is the set of policies, processes, tools, and organisational structures that ensure data assets in a data platform are trustworthy, secure, discoverable, and compliant with regulations. It answers the questions: Who owns this dataset? Who is allowed to access which columns? Where did this data come from? Does it contain personally identifiable information? Governance in analytics platforms must scale to hundreds of datasets, dozens of teams, and cross-platform data flows — it cannot be managed manually with access control lists in a spreadsheet.

Why it exists

As data platforms grow, three failure modes emerge: (1) data swamps — no one knows what datasets exist, who owns them, or whether they are reliable; (2) compliance breaches — PII ends up in places it shouldn't (analytical tables, ML training sets, BI exports) due to lack of column-level controls; and (3) unexplained data quality issues — a metric changes and no one can trace which upstream pipeline caused it, because lineage is not tracked. Formal governance addresses all three.

Typical architecture

DATA GOVERNANCE ARCHITECTURE
──────────────────────────────
  ┌─────────────────────────────────────────┐
  │            Metadata Catalog             │
  │      (Apache Atlas / DataHub / Alation) │
  │  - Dataset discovery and search         │
  │  - Ownership and steward assignments    │
  │  - Classification tags (PII, Sensitive) │
  │  - Business glossary linkage            │
  └──────────────────────┬──────────────────┘
                         │
           ┌─────────────┼─────────────┐
           │             │             │
    Lineage Tracking  Access Control  Data Quality
    (OpenLineage/     (Ranger/Unity   (Great Expectations
     Marquez/Datahub)  Catalog)        / Monte Carlo)

DOMAIN OWNERSHIP (DATA MESH PRINCIPLE)
────────────────────────────────────────
  Central data team owns platform (tooling, standards, self-service)
  Domain teams own their data products (datasets, pipelines, quality SLAs)

  Data Product Owner responsibilities:
  - Maintains schema contracts (Avro/JSON Schema in Schema Registry)
  - Assigns data classification tags (PUBLIC, INTERNAL, CONFIDENTIAL, PII)
  - Defines SLA: freshness (updated every 4h), availability (99.5%),
    quality (< 0.1% null rate on required columns)
  - Acknowledges downstream consumer impact before schema changes

ROW AND COLUMN LEVEL SECURITY
───────────────────────────────
  Apache Ranger (Hadoop ecosystem: Hive, Spark, HDFS, Kafka):
  -- Grant analytics team read on orders, but mask email column
  policy:
    resource: hive.analytics.orders
    user: analytics_team
    permissions: read
    masking: HASH(email)  -- SHA-256 hash replaces plaintext email

  BigQuery column-level security:
  -- Tag columns with policy tags (PII → requires healthcare_analyst role)
  CREATE TABLE `project.dataset.patients` (
    patient_id STRING,
    name STRING OPTIONS (tags='PII'),    -- requires PII reader role
    diagnosis STRING OPTIONS (tags='PHI') -- requires PHI reader role
  );

  Databricks Unity Catalog (cloud-native fine-grained access):
  GRANT SELECT ON TABLE catalog.schema.orders TO `data-analysts@company.com`;
  -- Row-level filtering (only own region):
  CREATE ROW FILTER finance_region_filter ON catalog.silver.transactions
  USING (region = current_user_region());

PII DETECTION AND MASKING (PRESIDIO)
──────────────────────────────────────
  Microsoft Presidio: open-source NLP-based PII detection

  Supported entities: PERSON, EMAIL_ADDRESS, PHONE_NUMBER, CREDIT_CARD,
                      IBAN_CODE, IP_ADDRESS, DATE_TIME, NRP, LOCATION, etc.

  Python pipeline integration:
  from presidio_analyzer import AnalyzerEngine
  from presidio_anonymizer import AnonymizerEngine

  analyzer = AnalyzerEngine()
  anonymizer = AnonymizerEngine()

  results = analyzer.analyze(text=row['free_text'], language='en')
  anonymized = anonymizer.anonymize(text=row['free_text'], analyzer_results=results)
  # Replaces "John Smith called from +44 207 123 4567" with
  # " called from "

  Scan Bronze layer on ingestion → tag columns containing PII →
  enforce column masking policies before Silver promotion.

DATA LINEAGE (OPENLINEAGE)
───────────────────────────
  OpenLineage: open standard for data lineage metadata emission
  Supported: Apache Airflow, Spark (OpenLineage Spark integration),
             dbt, Flink, Great Expectations, and more

  Lineage event (simplified):
  {
    "eventType": "COMPLETE",
    "job": { "namespace": "my-pipeline", "name": "orders_silver" },
    "inputs":  [{ "namespace": "s3", "name": "s3://bronze/orders" }],
    "outputs": [{ "namespace": "delta", "name": "silver.orders" }]
  }

  Storage backends: Marquez (open-source lineage server),
                    DataHub (LinkedIn's data catalog, also lineage),
                    Apache Atlas

  Use cases:
  - Impact analysis: "if I change orders.amount schema, which Gold tables break?"
  - Root cause: "daily revenue metric dropped — which upstream pipeline changed?"
  - Compliance: "show me all tables that derive from raw_customers (PII source)"

METADATA CATALOG COMPARISON
─────────────────────────────
  Apache Atlas:
  + Hadoop ecosystem native (Hive, HBase, Kafka, Spark)
  + Built-in Ranger integration for policy enforcement
  + OpenLineage compatible
  - Complex deployment; less user-friendly UI
  - Primarily on-premises/hybrid

  DataHub (LinkedIn, open-source):
  + Modern UI; strong lineage graph visualisation
  + Schema history, changelog, impact analysis
  + Multi-platform: BigQuery, Snowflake, Redshift, Hive, Kafka
  + Active development; large community
  - Self-hosted; significant ops overhead for small teams

  Managed alternatives: Alation, Atlan, Collibra
  Cloud-native: BigQuery Data Catalog, AWS Glue Data Catalog

Pros and cons

Implementation notes

Progressive governance: Attempting to implement full governance (catalog + lineage + column-level security + PII scanning) simultaneously creates paralysis. Start with the highest-value intervention: classify PII columns in your most sensitive datasets and enforce column-level access policies on those columns first. Add lineage tracking as the second priority (high ROI for debugging). Catalog and stewardship are third.

Common failure modes

• Governance without enforcement: Publishing data classification tags in a catalog but not enforcing them with query-engine-level access controls — analysts bypass the catalog entirely by querying tables directly.
• No data product owner: Datasets without clear owners become orphaned — schema changes break downstream pipelines with no one accountable for fixing them or communicating the change.

Decision checklist

• Does every dataset in the platform have a named owner (team or individual)?
• Are PII-containing columns tagged and protected with column-level access policies?
• Is lineage tracked for critical pipelines (Gold tables, ML feature tables) using OpenLineage or equivalent?
• Is there a data catalog (DataHub, Atlas, or cloud-native equivalent) where engineers can discover datasets and their owners?
• Is there a data schema change notification process so downstream consumers are alerted before breaking changes?

Example use cases

• Healthcare analytics platform (HIPAA): Bronze → Presidio PII scan on admit notes → PHI columns tagged in Atlas → Unity Catalog column filter enforced → only users with PHI role see unmasked values; OpenLineage tracks which ML training datasets derive from PHI source tables for audit.
• GDPR right-to-erasure: Customer deletion request → DataHub lineage graph identifies all derived tables containing customer_id → automated deletion propagation script runs against each table.

Related patterns

• Data Pipeline Patterns — Medallion Bronze/Silver/Gold layers where governance policies are enforced at each tier boundary.
• Data Mesh — Domain ownership model that underpins decentralised data governance at scale.
• Security Architecture — Access control, encryption, and compliance patterns that complement data governance.

Further reading

• OpenLineage Documentation
• DataHub Documentation
• Microsoft Presidio Documentation