dlt.transformations.lineage

create_sqlglot_schema

def create_sqlglot_schema(
        schema: Schema,
        dataset_name: str,
        dialect: Optional[DialectType] = "duckdb") -> SQLGlotSchema

View source on GitHub

Create an SQLGlot schema using a dlt Schema and the destination capabilities.

The SQLGlot schema automatically scopes the tables to the dataset_name. This can allow cross-dataset transformations on the same physical location. No name translation nor case folding is performing. All identifiers correspond to identifiers in dlt schema.

compute_columns_schema

def compute_columns_schema(
    sql_query: str,
    sqlglot_schema: SQLGlotSchema,
    dialect: Optional[DialectType] = None,
    infer_sqlglot_schema: bool = True,
    allow_anonymous_columns: bool = True,
    allow_partial: bool = True
) -> Tuple[TTableSchemaColumns, Optional[sge.Query]]

View source on GitHub

Compute the expected dlt columns schema for the output of an SQL SELECT query. No case-folding or quoting is performed on the query.

Arguments:

• infer_sqlglot_schema bool - If False, all columns and tables referenced must be derived from the SQLGlot schema. If True, allow columns and tables not found in SQLGlot schema

• allow_anonymous_columns bool - If False, all columns in final selection must have an explicit name or alias. If True, the name of columns from the final selection can be generated by the dialect

• allow_partial bool - If False, raise exceptions if the schema returned is incomplete. If True, this function always returns a dictionary, even in cases of SQL parsing errors, missing table reference, unresolved SELECT *, etc.

• Returns - tuple of dlt columns schema and qualified sql_query