dlt.transformations.lineage

create_sqlglot_schema

def create_sqlglot_schema(schema: Schema,
                          sql_client: SqlClientBase[Any],
                          dialect: Optional[DialectType] = "duckdb") -> Any

View source on GitHub

Create an SQLGlot schema using a dlt Schema and the destination capabilities.

The SQLGlot schema automatically includes the database and catalog names if available. This can allow cross-dataset transformations on the same physical location.

compute_columns_schema

def compute_columns_schema(sql_query: str,
                           sqlglot_schema: SQLGlotSchema,
                           dialect: Optional[DialectType] = None,
                           infer_sqlglot_schema: bool = True,
                           allow_anonymous_columns: bool = True,
                           allow_partial: bool = True) -> TTableSchemaColumns

View source on GitHub

Compute the expected dlt columns schema for the output of an SQL SELECT query.

Arguments:

• infer_sqlglot_schema bool - If False, all columns and tables referenced must be derived from the SQLGlot schema. If True, allow columns and tables not found in SQLGlot schema
• allow_anonymous_columns bool - If False, all columns in final selection must have an explicit name or alias. If True, the name of columns from the final selection can be generated by the dialect
• allow_partial bool - If False, raise exceptions if the schema returned is incomplete. If True, this function always returns a dictionary, even in cases of SQL parsing errors, missing table reference, unresolved SELECT *, etc.