dlt.common.data_writers.writers
FileWriterSpec Objects
class FileWriterSpec(NamedTuple)
file_format
format of the output file
data_item_format
format of the input data
supports_schema_changes
File format supports changes of schema: True - at any moment, Buffer - in memory buffer before opening file, False - not at all
file_max_items
Set an upper limit on the number of items in one file
DataWriter Objects
class DataWriter(abc.ABC)
item_format_from_file_extension
@classmethod
def item_format_from_file_extension(cls, extension: str) -> TDataItemFormat
Simple heuristic to get data item format from file extension
ImportFileWriter Objects
class ImportFileWriter(DataWriter)
May only import files, fails on any open/write operations
ModelWriter Objects
class ModelWriter(DataWriter)
Writes incoming items row by row into a text file and ensures a trailing ;
ArrowToObjectAdapter Objects
class ArrowToObjectAdapter()
A mixin that will convert object writer into arrow writer.
is_native_writer
def is_native_writer(writer_type: Type[DataWriter]) -> bool
Checks if writer has adapter mixin. Writers with adapters are not native and typically decrease the performance.
resolve_best_writer_spec
def resolve_best_writer_spec(
item_format: TDataItemFormat,
possible_file_formats: Sequence[TLoaderFileFormat],
preferred_format: TLoaderFileFormat = None) -> FileWriterSpec
Finds best writer for item_format out of possible_file_formats. Tries preferred_format first.
Best possible writer is a native writer for item_format writing files in preferred_format.
If not found, any native writer for possible_file_formats is picked.
Native writer supports item_format directly without a need to convert to other item formats.
get_best_writer_spec
def get_best_writer_spec(item_format: TDataItemFormat,
file_format: TLoaderFileFormat) -> FileWriterSpec
Gets writer for item_format writing files in {file_format}. Looks for native writer first
create_import_spec
def create_import_spec(
item_file_format: TLoaderFileFormat,
possible_file_formats: Sequence[TLoaderFileFormat]) -> FileWriterSpec
Creates writer spec that may be used only to import files