common.storages.data_item_storage
DataItemStorage Objects
class DataItemStorage(ABC)
write_empty_items_file
def write_empty_items_file(load_id: str, schema_name: str, table_name: str,
columns: TTableSchemaColumns) -> DataWriterMetrics
Writes empty file: only header and footer without actual items. Closed the empty file and returns metrics. Mind that header and footer will be written.
import_items_file
def import_items_file(load_id: str,
schema_name: str,
table_name: str,
file_path: str,
metrics: DataWriterMetrics,
with_extension: str = None) -> DataWriterMetrics
Import a file from file_path
into items storage under a new file name. Does not check
the imported file format. Uses counts from metrics
as a base. Logically closes the imported file
The preferred import method is a hard link to avoid copying the data. If current filesystem does not support it, a regular copy is used.
Alternative extension may be provided via with_extension
so various file formats may be imported into the same folder.
close_writers
def close_writers(load_id: str, skip_flush: bool = False) -> None
Flush, write footers (skip_flush), write metrics and close files in all
writers belonging to load_id
package
closed_files
def closed_files(load_id: str) -> List[DataWriterMetrics]
Return metrics for all fully processed (closed) files
remove_closed_files
def remove_closed_files(load_id: str) -> None
Remove metrics for closed files in a given load_id