dlt.common.destination.dataset
SupportsReadableRelation Objects
class SupportsReadableRelation(Protocol)
A readable relation retrieved from a destination that supports it
columns_schema
Returns the expected columns schema for the result of the relation. Column types are discovered with sql glot query analysis and lineage. dlt hints for columns are kept in some cases. Refere to <docs-page> for more details.
query
def query() -> Any
Returns the sql query that represents the relation
Returns:
Any- The sql query that represents the relation
df
def df(chunk_size: int = None) -> Optional[DataFrame]
Fetches the results as arrow table. Uses the native pandas implementation of the destination client cursor if available.
Arguments:
chunk_sizeint, optional - The number of rows to fetch for this call. Defaults to None which will fetch all rows.
Returns:
Optional[DataFrame]- A data frame with query results.
arrow
def arrow(chunk_size: int = None) -> Optional[ArrowTable]
Fetches the results as arrow table. Uses the native arrow implementation of the destination client cursor if available.
Arguments:
chunk_sizeint, optional - The number of rows to fetch for this call. Defaults to None which will fetch all rows.
Returns:
Optional[ArrowTable]- An arrow table with query results.
iter_df
def iter_df(chunk_size: int) -> Generator[DataFrame, None, None]
Iterates over data frames of 'chunk_size' items. Uses the native pandas implementation of the destination client cursor if available.
Arguments:
chunk_sizeint - The number of rows to fetch for each iteration.
Returns:
Generator[DataFrame, None, None]: A generator of data frames with query results.
iter_arrow
def iter_arrow(chunk_size: int) -> Generator[ArrowTable, None, None]
Iterates over arrow tables of 'chunk_size' items. Uses the native arrow implementation of the destination client cursor if available.
Arguments:
chunk_sizeint - The number of rows to fetch for each iteration.
Returns:
Generator[ArrowTable, None, None]: A generator of arrow tables with query results.
fetchall
def fetchall() -> List[Tuple[Any, ...]]
Fetches all items as a list of python tuples. Uses the native dbapi fetchall implementation of the destination client cursor.
Returns:
List[Tuple[Any, ...]]: A list of python tuples with query results.
fetchmany
def fetchmany(chunk_size: int) -> List[Tuple[Any, ...]]
Fetches the first 'chunk_size' items as a list of python tuples. Uses the native dbapi fetchmany implementation of the destination client cursor.
Arguments:
chunk_sizeint - The number of rows to fetch for this call.
Returns:
List[Tuple[Any, ...]]: A list of python tuples with query results.
iter_fetch
def iter_fetch(chunk_size: int) -> Generator[List[Tuple[Any, ...]], Any, Any]
Iterates in lists of Python tuples in 'chunk_size' chunks. Uses the native dbapi fetchmany implementation of the destination client cursor.
Arguments:
chunk_sizeint - The number of rows to fetch for each iteration.
Returns:
Generator[List[Tuple[Any, ...]], Any, Any]: A generator of lists of python tuples with query results.
fetchone
def fetchone() -> Optional[Tuple[Any, ...]]
Fetches the first item as a python tuple. Uses the native dbapi fetchone implementation of the destination client cursor.
Returns:
Optional[Tuple[Any, ...]]: A python tuple with the first item of the query results.
limit
def limit(limit: int, **kwargs: Any) -> Self
Returns a new relation with the limit applied.
Arguments:
limitint - The number of rows to fetch.**kwargsAny - Additional keyword arguments to pass to the limit implementation of the destination client cursor.
Returns:
Self- The relation with the limit applied.
head
def head(limit: int = 5) -> Self
By default returns a relation with the first 5 rows selected.
Arguments:
limitint - The number of rows to fetch.
Returns:
Self- The relation with the limit applied.
select
def select(*columns: str) -> Self
Returns a new relation with the given columns selected.
Arguments:
*columnsstr - The columns to select.
Returns:
Self- The relation with the columns selected.
__getitem__
def __getitem__(columns: Union[str, Sequence[str]]) -> Self
Returns a new relation with the given columns selected.
Arguments:
columnsUnion[str, Sequence[str]] - The columns to select.
Returns:
Self- The relation with the columns selected.
__getattr__
def __getattr__(attr: str) -> Any
get an attribute of the relation
Arguments:
attrstr - The attribute to get.
Returns:
Any- The attribute of the relation
__copy__
def __copy__() -> Self
create a copy of the relation object
Returns:
Self- The copy of the relation object
DBApiCursor Objects
class DBApiCursor(SupportsReadableRelation)
Protocol for DBAPI cursor
native_cursor
Cursor implementation native to current destination
SupportsReadableDataset Objects
class SupportsReadableDataset(Generic[TReadableRelation], Protocol)
A readable dataset retrieved from a destination, has support for creating readable relations for a query or table
schema
@property
def schema() -> Schema
Returns the schema of the dataset, will fetch the schema from the destination
Returns:
Schema- The schema of the dataset
dataset_name
@property
def dataset_name() -> str
Returns the name of the dataset
Returns:
str- The name of the dataset
__call__
def __call__(query: Any) -> SupportsReadableRelation
Returns a readable relation for a given sql query
Arguments:
queryAny - The sql query to base the relation on
Returns:
SupportsReadableRelation- The readable relation for the query
__getitem__
def __getitem__(table: str) -> TReadableRelation
Returns a readable relation for the table named table
Arguments:
tablestr - The name of the table
Returns:
TReadableRelation- The readable relation for the table
__getattr__
def __getattr__(table: str) -> TReadableRelation
Returns a readable relation for the table named table
Arguments:
tablestr - The name of the table
Returns:
TReadableRelation- The readable relation for the table
ibis
def ibis() -> IbisBackend
Returns a connected ibis backend for the dataset. Not implemented for all destinations.
Returns:
IbisBackend- The ibis backend for the dataset
row_counts
def row_counts(*,
data_tables: bool = True,
dlt_tables: bool = False,
table_names: List[str] = None) -> SupportsReadableRelation
Returns the row counts of the dataset
Arguments:
data_tablesbool, optional - Whether to include data tables. Defaults to True.dlt_tablesbool, optional - Whether to include dlt tables. Defaults to False.table_namesList[str], optional - The names of the tables to include. Defaults to None. Will override data_tables and dlt_tables if set
Returns:
SupportsReadableRelation- The row counts of the dataset as ReadableRelation