dlt.common.libs.sqlglot

from_sqlglot_type

def from_sqlglot_type(sqlglot_type: DATA_TYPE) -> TColumnType

View source on GitHub

Convert a SQLGlot DataType to dlt column hints.

reference: https://dlthub.com/docs/general-usage/schema/#tables-and-columns

to_sqlglot_type

def to_sqlglot_type(dlt_type: TDataType,
                    precision: Optional[int] = None,
                    scale: Optional[int] = None,
                    timezone: Optional[bool] = None,
                    nullable: Optional[bool] = None,
                    use_named_types: bool = False) -> DataType

View source on GitHub

Convert the dlt data_type and column hints to a SQLGlot DataType expression.

if use_named_type is True: use "named" types with fallback on "parameterized" types. else: use "parameterized" types everywhere.

Named types:

• have some set attributes, e.g., DataType.Type.DECIMAL64 has precision=16 and scale=4.
• can be referenced directly in SQL (table definition, CAST(), which could make SQLGlot transpiling more reliable.
• may not exist in all dialects and will be casted automatically during transpiling
• can have limited expressivity, e.g., named types for timestamps only included precision 0, 3, 9

Parameterized types:

• instead of DECIMAL64, a generic DataType.Type.DECIMAL is used with DataTypeParam expressions attached to represent precision and scale. SQLGlot is responsible for properly compiling and transpiling them.
• parameters might be handled differently across dialects and would require greater testing on the dlt side.

reference: https://dlthub.com/docs/general-usage/schema/#tables-and-columns

set_metadata

def set_metadata(sqlglot_type: sge.DataType,
                 metadata: TColumnSchema) -> sge.DataType

View source on GitHub

Set a metadata dictionary on the SQGLot DataType object.

By attaching dlt hints to a DataType object, they will be propagated until the DataType is modified.

get_metadata

def get_metadata(sqlglot_type: sge.DataType) -> TColumnSchema

View source on GitHub

Get a metadata dictionary from the SQLGlot DataType object.

query_is_complex

def query_is_complex(parsed_select: Union[sge.Select, sge.Union],
                     columns: Set[str]) -> bool

View source on GitHub

Return True unless the query is provably “simple”. A simple query

1. references exactly one physical table,
2. contains no complex constructs (CTEs, sub-queries, derived tables, unions, window functions, GROUP BY, DISTINCT, etc.),
3. projects either • a plain/qualified star with only constant literals after it, or • the full, explicit list of all columns with only constant literals after it. Anything we cannot prove to be simple is conservatively flagged as complex.