2024 dlt Recap: Moments, Mentions, and Milestones
- Adrian Brudaru,
Co-Founder & CDO - Aman Gupta,
Jr. Data Engineer
2024 was a remarkable year for dltHub. Together with our users and partners, we streamlined workflows, introduced powerful capabilities, and laid a stronger foundation for the future.
Strengthening Partnerships
- Tower.dev: Improved data orchestration for more efficient workflows. (Link)
- Cognee: Enhanced data ingestion to support faster, more accurate analysis. (Link)
Community Milestones
- 1,000 Production Users: A testament to the trust and reliance our community has placed in dltHub. (Link)
- dlt v1 Release: A major step forward in stability, features, and performance. (Link)
Enriching the Data Ecosystem
dlt wants to integrate with your data stack, not replace it. As testament, here are some interesting integrations or demos
Integrations:
- Dagster: https://dagster.io/blog/expanding-dagsters-embedded-elt-ecosystem-with-dlthub-for-data-ingestion
- SQLMesh: https://sqlmesh.readthedocs.io/en/stable/integrations/dlt/
- dbt: https://dlthub.com/blog/dbt-gen
Demos:
- Modal: https://modal.com/blog/analytics-stack
Community articles and mentions
Getting Started with dltHub โ Introductions and First Impressions
- ๐ย data load tool (dlt): An introduction by Crux Data
- ๐ย Python ELT with dlthubโ why I both love and hate data load tool (dlt) by Hugo Lu
- ๐ย Data Ingestion with Data Loads Tool (dlt): Be the Magician in Data Engineering ๐ by Kang Zhi Yong
- ๐ย dltHub First Look by Bhavani Ravi
- ๐ย Moving Data with Python and dlt: A Guide for Data Engineers by Datacamp
Integrations with Dagster, Airflow, Databricks, and Other Orchestration Tools
- ๐ย Expanding the Dagster Embedded ELT Ecosystem with dltHub for Data Ingestion by Colton Padden
- ๐ย Using dltHub together with dlt and DABs by molkke
- ๐ย Data Ingestion with dlthub and Dagster From Hubspot to Bigquery by Noonanlabs
- ๐ย Using data load tool (dlthub) together with Databricks by StephTheChef
- ๐ย Open source library to build data pipelines with YAML - a configuration layer for Dagster by -infinite-
- (๐) Now that @dagster 1.7 has landed, Iโm excited to share our official integration with @dltHub. by Colton Padden
- ๐ย Hello, everyone! We did it, again! ...contributing support for dlt assets partitions and metadata by Edson Nogueira
- ๐ย Great run-through of how Dagster and dltHub can be used together to save money on ELT by Pete Hunt
Real-World Use Cases, Building Pipelines, and Tooling
- (๐) Built a pipeline using @dltHub and @motherduck and it's very nice by Garrett McClintock
- (๐) dlt has had a major update to v1.0.0! by ikki / stableไปฃ่กจ
- (๐) Analytics stack at modal labs by Kenny Ning
- (๐) time for some @dltHub to go into mdsinabox by Jacob Matson
- (๐) ingest data from REST APIs by Miguel Peredo Z
- (๐) Data ingestion with @dltHub by Ayoola
- ๐ย Schema Evolution with dlt ๐ฎ by Alejandro Gonzรกlez Bueno
- ๐ Exploring DuckDB in Personal Projects โจ by Alejandro Gonzรกlez Bueno
- ๐ย dltHub has a good thing going! ๐ by Jarich Braeckevelt
- ๐ Data Ingestion with dltHub: A Fantastic Learning Experience! ๐ by Joseph Udum
- ๐คย Search Reddit posts using dlt and LanceDB by Lance DB
- ๐ย Building data pipelines is easier now with dltHub by Manikandan Bellan
- ๐ญย "Shitty Python code and an API" by Federico Romeo
- ๐ญย It's been a while since I wrote a connector... after seeing the great work at dltHub by Hugo Lu
- ๐ญย I love exploring data tools. ...We live in exciting times! by Yuki Kakegawa
- ๐ย Efficient Shopify Data Transfer with dltHub by Manuel Schmidbauer
Workshops, Events, and Community Engagement
- (๐) About dlthub hosting a free hands-on workshop as part of the Data Engineering Zoomcamp! by Alexey Grigorev
- ๐ญ Are you following any DataTalksClub (free) courses? by Diogo Costa
- ๐ญย Comment nous selections les bons outils... Dans quels cas nous recommandons dltHub ? by Matthieu Rousseau
- ๐ญย Want to learn about an easy way to load data with Python? Then check dltHub! by Nevenka Lukic
- ๐ญย In less than one week we're starting Data Engineering Zoomcamp by Alexey Grigorev
- ๐คย dlt Paris community meetup by Christophe Blefari
- ๐ญย If you missed the tutorial at #PyDataNYC... try it yourself! by Deepyaman Datta
- ๐คย The Open Data Lakes(ide) gathering at Foundation Capital... by Foundation Capital
- ๐คย Check out this cool use case from the dltHub team, combining dltHub, Kestra, and OpenAI by Kestra
- ๐คย Hey folks, we have a new Tower.dev case study... Taktile used Tower.dev & dltHub by Serhii Sokolenko
- ๐ญย My most recent post details how I built a pipeline to summarize Slack threads by Thierry Jean
Ecosystem Outlook, Comparisons, and Future Directions
- ๐คย โWe want to be Hugging Face, not Fivetranโ by Quentin Churet
- ๐ญย Week 4 Recap: #LLMs #VectorEmbeddings #Dlthub by Abhishek Nakka
- ๐คย Excited to Launch a Full-Stack Data Engineering Project! by Ahmad Darwich
- ๐ญย Thank you, Daniyar Mussakulov, for your support... by Ajit Gupta
- ๐คย Exploring project and dataset setup with sqlmesh by David Jayatillake
- ๐ญ Open Source data ingestion tools that should be on your radar: dltHub by Noel Gomez
- ๐ญย It's not new news that Fivetran is expensive... by Paul Dudley
- ๐ย The dlt Project, Data Science at the Command Line by Rami Krispin
- ๐ย dltHub Is giving me the same, "making me feel more competent at my job", feels as dbt Labs did back in the day by Yale Newman
Founderโs note: Our Community-Driven journey in 2024
Looking back at 2024, what stands out most isn't the technical milestones, but how our community has shaped and strengthened every aspect of dlt. When we started, we built on a clear vision: data engineers needed a better way to build pipelines in Python. What's been truly inspiring is seeing this vision resonating and evolving through the contributions of our growing community.
Each mention in this recap represents someone who invested their time and expertise in making dlt better for everyone. From engineers exploring new integrations with modern data warehouses to teams implementing dlt at fast-growing startups, to the countless thoughtful blog posts and tutorials, every shared experience has helped shape dlt's path forward.
The integration stories with various other tools showcase the power of community-driven development. Whether it's detailed write-ups about expanding workflow ecosystems or practical guides about combining dlt with different orchestrators, these experiences reinforce our commitment to building open, composable tools that work within existing data stacks.
Educational impact is another big topic. From our own courses, to the engagement with industry-leading learning platforms, the enthusiasm around community workshops, to the energy at our local meetups, these initiatives have shown how knowledge sharing is central to our community's success. Seeing experienced engineers and newcomers alike using dlt to learn and teach modern data engineering has been incredibly rewarding.
Going from a barely known library to an emerging de-facto standard for pythonic ingestion in 2024 was a milestone that reflects the trust our community has placed in dlt.
But the real achievement lies in the quality of engagement: the detailed feedback, the creative implementations, and the continuous improvements suggested by users.
Every enhancement that built towards dlt v1.0.0 came from this collaborative spirit and deep understanding of real-world needs.
Looking ahead, we're more committed than ever to our founding principles: Solving real world problems, learning from our community, and making data engineering more accessible. The strength of dlt lies in our community of engineers who use it, improve it, integrate with it, and share their knowledge with others.
Thank you for being part of this journey and for helping us build something truly meaningful together. Here's to an even more collaborative 2025!
โ Adrian, data engineer and co-founder at dltHub.