sources.filesystem.helpers
Helpers for the filesystem resource.
fsspec_from_resource
def fsspec_from_resource(
filesystem_instance: DltResource) -> AbstractFileSystem
Extract authorized fsspec client from a filesystem resource
add_columns
def add_columns(columns: List[str],
rows: List[List[Any]]) -> List[Dict[str, Any]]
Adds column names to the given rows.
Arguments:
columns
List[str] - The column names.rows
List[List[Any]] - The rows.
Returns:
List[Dict[str, Any]]: The rows with column names.
fetch_arrow
def fetch_arrow(file_data, chunk_size: int) -> Iterable[TDataItem]
Fetches data from the given CSV file.
Arguments:
file_data
DuckDBPyRelation - The CSV file data.chunk_size
int - The number of rows to read at once.
Yields:
Iterable[TDataItem]
- Data items, read from the given CSV file.
fetch_json
def fetch_json(file_data, chunk_size: int) -> List[Dict[str, Any]]
Fetches data from the given CSV file.
Arguments:
file_data
DuckDBPyRelation - The CSV file data.chunk_size
int - The number of rows to read at once.
Yields:
Iterable[TDataItem]
- Data items, read from the given CSV file.