sources.filesystem_pipeline

stream_and_merge_csv

def stream_and_merge_csv() -> None

[view_source]

Demonstrates how to scan folder with csv files, load them in chunk and merge on date column with the previous load

read_custom_file_type_excel

def read_custom_file_type_excel() -> None

[view_source]

Here we create an extract pipeline using filesystem resource and read_csv transformer

copy_files_resource

def copy_files_resource(local_folder: str) -> None

[view_source]

Demonstrates how to copy files locally by adding a step to filesystem resource and the to load the download listing to db