Skip to main content
Version: devel

common.storages.data_item_storage

DataItemStorage Objects

class DataItemStorage(ABC)

[view_source]

write_empty_items_file

def write_empty_items_file(load_id: str, schema_name: str, table_name: str,
columns: TTableSchemaColumns) -> DataWriterMetrics

[view_source]

Writes empty file: only header and footer without actual items. Closed the empty file and returns metrics. Mind that header and footer will be written.

import_items_file

def import_items_file(load_id: str,
schema_name: str,
table_name: str,
file_path: str,
metrics: DataWriterMetrics,
with_extension: str = None) -> DataWriterMetrics

[view_source]

Import a file from file_path into items storage under a new file name. Does not check the imported file format. Uses counts from metrics as a base. Logically closes the imported file

The preferred import method is a hard link to avoid copying the data. If current filesystem does not support it, a regular copy is used.

Alternative extension may be provided via with_extension so various file formats may be imported into the same folder.

close_writers

def close_writers(load_id: str, skip_flush: bool = False) -> None

[view_source]

Flush, write footers (skip_flush), write metrics and close files in all writers belonging to load_id package

closed_files

def closed_files(load_id: str) -> List[DataWriterMetrics]

[view_source]

Return metrics for all fully processed (closed) files

remove_closed_files

def remove_closed_files(load_id: str) -> None

[view_source]

Remove metrics for closed files in a given load_id

This demo works on codespaces. Codespaces is a development environment available for free to anyone with a Github account. You'll be asked to fork the demo repository and from there the README guides you with further steps.
The demo uses the Continue VSCode extension.

Off to codespaces!

DHelp

Ask a question

Welcome to "Codex Central", your next-gen help center, driven by OpenAI's GPT-4 model. It's more than just a forum or a FAQ hub – it's a dynamic knowledge base where coders can find AI-assisted solutions to their pressing problems. With GPT-4's powerful comprehension and predictive abilities, Codex Central provides instantaneous issue resolution, insightful debugging, and personalized guidance. Get your code running smoothly with the unparalleled support at Codex Central - coding help reimagined with AI prowess.