dlt.dataset.lineage

create_sqlglot_schema

def create_sqlglot_schema(schema_map: Mapping[str, Sequence[dlt.Schema]],
                          dialect: TSqlGlotDialect) -> SQLGlotSchema

View source on GitHub

Create an SQLGlot schema from multiple dlt schemas grouped by dataset name.

Each key in schema_map becomes a top-level qualifier (SQL schema / catalog) that scopes all tables underneath it. Tables from multiple dlt schemas that share a dataset name are merged via Schema.unify_schemas; the first schema in each sequence is treated as the default and wins on column-level collisions.

Arguments:

• schema_map - Mapping of dataset_name to a list of dlt schemas. The dataset name is used as the qualifying namespace in the generated SQLGlot schema.
• dialect - SQLGlot dialect for the target destination.

compute_columns_schema

def compute_columns_schema(
        expression: sge.Expression,
        sqlglot_schema: SQLGlotSchema,
        dialect: TSqlGlotDialect,
        allow_partial: bool = False) -> Tuple[TTableSchemaColumns, sge.Query]

View source on GitHub

Compute the dlt columns schema of the output of an SQL SELECT query. No case-folding or quoting is performed on the query.

Columns that come from tables unknown to sqlglot_schema, or whose type sqlglot cannot derive, have no data_type.

Arguments:

• expression sge.Expression - Parsed SQL query, with identifiers in the dlt schema namespace. It is not modified.
• sqlglot_schema SQLGlotSchema - Schema of the tables the query reads, as created by create_sqlglot_schema. Provides column types and dlt column hints.
• dialect TSqlGlotDialect - SQLGlot dialect used to qualify the query.
• allow_partial bool - If True, columns that cannot be resolved have no data_type and a * over a table unknown to sqlglot_schema is skipped.

Returns:

Tuple[TTableSchemaColumns, sge.Query]: The dlt columns schema of the query output, keyed by output column name, and the qualified query with every column tied to its table, stars expanded and every projection aliased.

Raises:

• LineageFailedException - If the query is not a SELECT or, unless allow_partial is set, a column cannot be resolved or a * selects from a table unknown to sqlglot_schema.