Ingestion & Connectors¶

Four hard breaks here: the pipelineStatuses shape, the Databricks Pipeline connection, the log-stream SSE payload, and the progress payload. Plus a Great Expectations major-version jump.

pipelineStatuses is now an array¶

Breaking · Affects: every client reading ingestion pipelines, dashboards, health checks, SDK users

  "pipelineStatuses": {
-   "description": "Last of executions and status for the Pipeline.",
-   "$ref": "#/definitions/pipelineStatus"
+   "description": "List of the most recent executions and status for the Pipeline.",
+   "type": "array",
+   "items": { "$ref": "#/definitions/pipelineStatus" }
  }

The list endpoint now returns the last five statuses per pipeline (which also removes an N+1 query), instead of a single most-recent status object.

{ "name": "my_pipeline", "pipelineStatuses": { "runId": "…", "pipelineState": "success" } }

{ "name": "my_pipeline",
  "pipelineStatuses": [ { "runId": "…", "pipelineState": "success" },
                        { "runId": "…", "pipelineState": "failed" } ] }

The 2.0.0 migration also strips any stale single-object pipelineStatuses value that a GET → PUT round-trip may have persisted into stored entity JSON, so it cannot break deserialisation server-side:

UPDATE ingestion_pipeline_entity
SET json = JSON_REMOVE(json, '$.pipelineStatuses')
WHERE JSON_CONTAINS_PATH(json, 'one', '$.pipelineStatuses');

Databricks Pipeline connection requires authType¶

Breaking · Affects: Databricks Pipeline services created via API/IaC

- "required": ["hostPort", "token"]
+ "required": ["hostPort", "authType"]

The top-level token property is removed and moves under authType:

{ "hostPort": "…", "token": "dapi…" }

{ "hostPort": "…", "authType": { "token": "dapi…" } }

The 2.0.0 migration rewrites existing services in place:

UPDATE pipeline_service_entity
SET json = JSON_INSERT(JSON_REMOVE(json, '$.connection.config.token'),
                       '$.connection.config.authType',
                       JSON_OBJECT('token', JSON_EXTRACT(json, '$.connection.config.token')))
WHERE serviceType = 'DatabricksPipeline'
  AND JSON_EXTRACT(json, '$.connection.config.token') IS NOT NULL
  AND NOT JSON_CONTAINS_PATH(json, 'one', '$.connection.config.authType');

Log stream SSE payload changed¶

Breaking — flagged BREAKING CHANGE: in the commit trailer

GET /v1/services/ingestionPipelines/logs/{fqn}/stream/{runId}

emits one JSON LogStreamEvent per frame instead of one raw log line.

{
  "eventType": "logs | complete | error",
  "runId": "scheduled__2026-08-01T00:00:00+00:00",
  "logs": "…appended log content…",
  "after": "opaque-cursor",
  "replay": false,
  "endReason": "runFinished | idleTimeout | maxDuration | maxBytes"
}

• after is an opaque resume cursor — pass it back as the after query parameter to resume without re-reading delivered content.
• endReason is set only on a complete event. A stream that ends without one was cut short — reconnect from the last cursor.
• replay: true marks chunks replayed from the server buffer because the stream was already running when this client connected.
• The path now accepts an id or FQN, and runId is a free-form String, so Airflow's scheduled__<ts> run ids no longer 404 before reaching a backend.

Ingestion progress payload changed¶

Breaking · Affects: consumers of GET /v1/services/ingestionPipelines/progress/{fqn}/stream/{runId}

progressUpdate.progress changes from a flat per-entity-type map to a hierarchical tree:

{
  "label": "",                 // display name; empty for the run root
  "entityType": "Database",    // type of children this node counts
  "processed": 3,
  "expected": 12,              // null when the producer was iterated lazily
  "active": true,
  "overflow": 0,               // active children beyond the per-parent display cap
  "children": [ /* active or relevant child nodes only */ ]
}

New service-level progress stream¶

GET /v1/services/ingestionPipelines/progress/service/{serviceType}/{serviceFqn}/stream

with a serviceProgressEvent payload, so the Services page can show live progress across all agents for a service.

Hierarchical progress tracking and SSE¶

Connectors now declare progress totals, the runner counts, and the registry reports. Coverage added in 2.0:

• Database connectors generally, plus Databricks and Unity Catalog explicitly
• Airflow REST and dbt Cloud pipeline totals
• Looker and Tableau via MANUAL progress mode
• Profiler and auto-classification workflows
• Total assets, run-start emit, and agent-on-discovery events

Third-party logs, warnings and uncaught exceptions are captured in the streamable logger.

Dependency changes¶

Great Expectations 1.3+ required¶

Breaking · Affects: the great-expectations plugin

- "great-expectations": "great-expectations~=0.18.0",
- "great-expectations-1xx": "great-expectations~=1.0",
+ "great-expectations": "great-expectations~=1.3",

The separate great-expectations-1xx extra is removed. 1.3 is the floor because GX only gained the validation-action registry there — on 1.0–1.2 Checkpoint.actions is a closed union that rejects the OpenMetadata action outright.

Ingestion images require Python 3.12¶

Breaking · Affects: custom ingestion and Airflow images

The 2.0 ingestion image uses python:3.12-slim-trixie, and the bundled Airflow image uses apache/airflow:3.3.0-python3.12. Custom images and binary wheels built only for CPython 3.10 must be rebuilt for Python 3.12. This image change first shipped in 1.13.4, so no additional action is needed when upgrading from that patch release.

Other dependency floors raised¶

· Affects: pinned/air-gapped installs

New plugin extras: prefect (uses requests, no extra deps) and sftp.

Contributor-facing: black, isort, pylint and pycln are replaced by ruff ~=0.15.12; basedpyright is pinned to 1.39.3; typing stubs are added for Google APIs, requests, pandas and scipy. Exasol integration-test deps move to a dedicated exasol-test extra.

New connectors¶

New connection schemas: sapBw4HanaConnection.json, omniConnection.json, prefectConnection.json (+ prefect/cloudAuth.json, prefect/serverAuth.json), sapBw4HanaPipelineConnection.json.

Connector configuration changes¶

Default filter patterns now exclude system objects¶

Behavioural · Affects: existing services too

SSIS databaseConnection is now optional¶

Relaxed requirement

- "required": ["databaseConnection", "packageConnection"]
+ "required": ["packageConnection"]

Supports file-only SSIS mode. Existing configurations are unaffected.

Snowflake¶

• includeSemanticViews (default false) ingests Snowflake semantic views; tableType gains SemanticView.
• Opt-in ACCESS_HISTORY lineage path.
• policyAgentConfig is declared with defaults and backfilled onto existing services — see Data Governance.
• Private key field gains uiFieldType: fileOrInput and is marked recommended.

Other connector changes¶

Stale-entity cleanup API¶

New pattern for connectors

DELETE /v1/tables/deleteStale

{
  "scopeFqn": "service.database.schema",
  "scopeEntityType": "databaseSchema",
  "seenFqns": ["…", "…"],
  "dryRun": false,
  "hardDelete": false,
  "recursive": true
}

Entities inside the scope that are not in seenFqns are considered stale and soft-deleted (default). dryRun: true lists what would be deleted without deleting.

Available on 18 entity types: tables, databases, database schemas, stored procedures, dashboards, charts, dashboard data models, pipelines, topics, ML models, search indexes, containers, API collections, API endpoints, and the four drive types.

Ingestion pipeline management¶

Connector framework internals¶

· Affects: custom connector authors only

2.0 continues the BaseConnection migration and related refactors:

• mlmodel, metadata, messaging, storage, search and drive connectors migrated to BaseConnection.
• get_connection_dict dropped from BaseConnection; data-diff gated via a Protocol.
• Duplicated test-connection helpers consolidated; Athena migrated to the declarative test-connection framework; a shared GetPipelines step added to the pipeline test-connection vertical.
• ClassifiableEntityAdapter replaces scattered isinstance checks.
• A TagRegistry domain layer is introduced.
• SamplerInterface collapses to a single typed config object.
• ConnectionsRouterClassBase enables pluggable connection routing in the UI.
• Comprehensive diagnostics added for the Airflow connection check.