Skip to main content

Deploy with Kestra

Introduction to Kestra

Kestra is an open-source, scalable orchestration platform that enables engineers to manage business-critical workflows declaratively in code. By applying  infrastructure as code best practices to data, process, and microservice orchestration, you can build and manage reliable workflows.

Kestra facilitates reliable workflow management, offering advanced settings for resiliency, triggers, real-time monitoring, and integration capabilities, making it a valuable tool for data engineers and developers.

Kestra features

Kestra provides a robust orchestration engine with features including:

  • Workflows accessible through a user interface, event-driven automation, and an embedded visual studio code editor.
  • It also offers embedded documentation, a live-updating topology view, and access to over 400 plugins, enhancing its versatility.
  • Kestra supports Git & CI/CD integrations, basic authentication, and benefits from community support.

To know more, please refer to Kestra's documentation.

Building Data Pipelines with dlt

dlt is an open-source Python library that allows you to declaratively load data sources into well-structured tables or datasets. It does this through automatic schema inference and evolution. The library simplifies building data pipeline by providing functionality to support the entire extract and load process.

How does dlt integrate with Kestra for pipeline orchestration?

To illustrate setting up a pipeline in Kestra, we’ll be using the following example: From Inbox to Insights AI-Enhanced Email Analysis with dlt and Kestra.

The example demonstrates automating a workflow to load data from Gmail to BigQuery using the dlt, complemented by AI-driven summarization and sentiment analysis. You can refer to the project's github repo by clicking here.


For the detailed guide, please take a look at the project's README section.

Here is the summary of the steps:

  1. Start by creating a virtual environment.

  2. Generate an .env File: Inside your project repository, create an .env file to store credentials in "base64" format, prefixed with 'SECRET_' for compatibility with Kestra's secret() function.

  3. As per Kestra’s recommendation, install the docker desktop on your machine.

  4. Ensure Docker is running, then download the Docker compose file with:

     curl -o docker-compose.yml \
  5. Configure Docker compose file: Edit the downloaded Docker compose file to link the .env file for environment variables.

    image: kestra/kestra:develop-full
    - .env
  6. Enable Auto-Restart: In your docker-compose.yml, set restart: always for both postgres and kestra services to ensure they reboot automatically after a system restart.

  7. Launch Kestra Server: Execute docker compose up -d to start the server.

  8. Access Kestra UI: Navigate to http://localhost:8080/ to use the Kestra user interface.

  9. Create and Configure Flows:

    • Go to 'Flows', then 'Create'.
    • Configure the flow files in the editor.
    • Save your flows.
  10. Understand Flow Components:

    • Each flow must have an id, namespace, and a list of tasks with their respective id and type.
    • The main flow orchestrates tasks like loading data from a source to a destination.

By following these steps, you establish a structured workflow within Kestra, leveraging its powerful features for efficient data pipeline orchestration.


For detailed information on these steps, please consult the in the dlt-kestra-demo repo.

Additional Resources

  • Ingest Zendesk data into Weaviate using dlt with Kestra: here.
  • Ingest Zendesk data into DuckDb using dlt with Kestra: here.
  • Ingest Pipedrive CRM data to BigQuery using dlt and schedule it to run every hour: here.

This demo works on codespaces. Codespaces is a development environment available for free to anyone with a Github account. You'll be asked to fork the demo repository and from there the README guides you with further steps.
The demo uses the Continue VSCode extension.

Off to codespaces!


Ask a question

Welcome to "Codex Central", your next-gen help center, driven by OpenAI's GPT-4 model. It's more than just a forum or a FAQ hub – it's a dynamic knowledge base where coders can find AI-assisted solutions to their pressing problems. With GPT-4's powerful comprehension and predictive abilities, Codex Central provides instantaneous issue resolution, insightful debugging, and personalized guidance. Get your code running smoothly with the unparalleled support at Codex Central - coding help reimagined with AI prowess.