Skip to main content

Parquet file format

Apache Parquet is a free and open-source column-oriented data storage format in the Apache Hadoop ecosystem. dlt is capable of storing data in this format when configured to do so.

To use this format, you need a pyarrow package. You can get this package as a dlt extra as well:

pip install "dlt[parquet]"

Supported Destinationsโ€‹

Supported by: BigQuery, DuckDB, Snowflake, filesystem, Athena, Databricks, Synapse

By setting the loader_file_format argument to parquet in the run command, the pipeline will store your data in the parquet format at the destination:

info =, loader_file_format="parquet")

Destination AutoConfigโ€‹

dlt uses destination capabilities to configure the parquet writer:

  • It uses decimal and wei precision to pick the right decimal type and sets precision and scale.
  • It uses timestamp precision to pick the right timestamp type resolution (seconds, micro, or nano).

Writer settingsโ€‹

Under the hood, dlt uses the pyarrow parquet writer to create the files. The following options can be used to change the behavior of the writer:

  • flavor: Sanitize schema or set other compatibility options to work with various target systems. Defaults to None which is pyarrow default.
  • version: Determine which Parquet logical types are available for use, whether the reduced set from the Parquet 1.x.x format or the expanded logical types added in later format versions. Defaults to "2.6".
  • data_page_size: Set a target threshold for the approximate encoded size of data pages within a column chunk (in bytes). Defaults to None which is pyarrow default.
  • timestamp_timezone: A string specifying timezone, default is UTC.
  • coerce_timestamps: resolution to which coerce timestamps, choose from s, ms, us, ns
  • allow_truncated_timestamps - will raise if precision is lost on truncated timestamp.

Default parquet version used by dlt is 2.4. It coerces timestamps to microseconds and truncates nanoseconds silently. Such setting provides best interoperability with database systems, including loading panda frames which have nanosecond resolution by default

Read the pyarrow parquet docs to learn more about these settings.


# the default values

Or using environment variables:


Timestamps and timezonesโ€‹

dlt adds timezone (UTC adjustment) to all timestamps regardless of a precision (from seconds to nanoseconds). dlt will also create TZ aware timestamp columns in the destinations. duckdb is an exception here

Disable timezones / utc adjustment flagsโ€‹

You can generate parquet files without timezone adjustment information in two ways:

  1. Set the flavor to spark. All timestamps will be generated via deprecated int96 physical data type, without the logical one
  2. Set the timestamp_timezone to empty string (ie. DATA_WRITER__TIMESTAMP_TIMEZONE="") to generate logical type without UTC adjustment.

To our best knowledge, arrow will convert your timezone aware DateTime(s) to UTC and store them in parquet without timezone information.

This demo works on codespaces. Codespaces is a development environment available for free to anyone with a Github account. You'll be asked to fork the demo repository and from there the README guides you with further steps.
The demo uses the Continue VSCode extension.

Off to codespaces!


Ask a question

Welcome to "Codex Central", your next-gen help center, driven by OpenAI's GPT-4 model. It's more than just a forum or a FAQ hub โ€“ it's a dynamic knowledge base where coders can find AI-assisted solutions to their pressing problems. With GPT-4's powerful comprehension and predictive abilities, Codex Central provides instantaneous issue resolution, insightful debugging, and personalized guidance. Get your code running smoothly with the unparalleled support at Codex Central - coding help reimagined with AI prowess.