common.data_writers.writers

FileWriterSpec Objects

class FileWriterSpec(NamedTuple)

[view_source]

file_format

format of the output file

data_item_format

format of the input data

supports_schema_changes

File format supports changes of schema: True - at any moment, Buffer - in memory buffer before opening file, False - not at all

DataWriter Objects

class DataWriter(abc.ABC)

[view_source]

item_format_from_file_extension

@classmethod
def item_format_from_file_extension(cls, extension: str) -> TDataItemFormat

[view_source]

Simple heuristic to get data item format from file extension

ImportFileWriter Objects

class ImportFileWriter(DataWriter)

[view_source]

May only import files, fails on any open/write operations

ArrowToObjectAdapter Objects

class ArrowToObjectAdapter()

[view_source]

A mixin that will convert object writer into arrow writer.

is_native_writer

def is_native_writer(writer_type: Type[DataWriter]) -> bool

[view_source]

Checks if writer has adapter mixin. Writers with adapters are not native and typically decrease the performance.

resolve_best_writer_spec

def resolve_best_writer_spec(
        item_format: TDataItemFormat,
        possible_file_formats: Sequence[TLoaderFileFormat],
        preferred_format: TLoaderFileFormat = None) -> FileWriterSpec

[view_source]

Finds best writer for item_format out of possible_file_formats. Tries preferred_format first. Best possible writer is a native writer for item_format writing files in preferred_format. If not found, any native writer for possible_file_formats is picked. Native writer supports item_format directly without a need to convert to other item formats.

get_best_writer_spec

def get_best_writer_spec(item_format: TDataItemFormat,
                         file_format: TLoaderFileFormat) -> FileWriterSpec

[view_source]

Gets writer for item_format writing files in {file_format}. Looks for native writer first

create_import_spec

def create_import_spec(
        item_file_format: TLoaderFileFormat,
        possible_file_formats: Sequence[TLoaderFileFormat]) -> FileWriterSpec

[view_source]

Creates writer spec that may be used only to import files