Build → Deploy → Share: A roadmap for sharing on dltHub

You want connectors, and you want them to be many, high quality and customisable? A man can dream? here’s our roadmap to making those dreams a reality, and how you can help us today.

We’re launching scaffold contribution

We previously launched scaffolds - a stepping stone towards pipeline generation. Scaffolds are a distillation of documentation in just the right format so an LLM can stand up working code.

We imagine the following process

1. We generate a scaffold ← This is where your current contribution trickles down
2. You generate a running pipeline using dlt’s scaffolds
3. You debug it: look for duplicates or gaps, check that only new rows arrive, confirm the primary keys land in the right place, make sure each column looks how it should, and verify the row count isn’t crazy high or low
4. Now you have an amazing pipeline - and you might think - can i share it to fast-forward the next guy?

Yes! You can share back to fast-forward the next guy, so they don’t have to reinvent the flat tyre either. Before we get into the details of how, and what will happen to your contribution, let me add more context by discussing our short term sharing roadmap

This is just the start of something bigger

Scaffolds as a concept have some utility as a stepping stone to running code and as a guide to customise it, but what is really useful, what we all really want, is the actual correctly-working code.

This correct code is the result of multiple development steps, from reading the docs, to crafting the boilerplate code, and finally testing the outputs and adjusting for correctness and for our needs.

🔴 Currently, scaffolds are an alternative way to do the basic work, the “read the docs” from the image above. You still have to generate the running code, and finally adjust and debug it.

🟠 Next, we will push towards making running code - after all, this would more or less emulate what you already do in cursor in “yolo mode” (where you add no input and simply let the LLM solve the problem based on traces).

🟢 Finally, debugging, adjusting for business case, and getting credentials will never be automated - this is where you come in.

Finally, in the workspace app we will offer a one click share option to share back your work to the broader community:

Deployment and Sharing coming to the workspace app

The final piece of the puzzle is you. Soon you’ll be able to deploy pipelines on dltHub and share any debugged pipeline to other private users or back to the broader community.

By June, the community had spun up 100.000+ private pipelines, from niche SaaS APIs to public data sources. LLM-powered apps are hungry for that data, but wiring every source by hand just doesn’t scale.

By coupling what you already do naturally, with better tooling to help you do it faster, and share it in a simple way, we are building dlthub: 100.000’s pipelines and datasets, that are run and shared all in one place.

For the first time, you’ll be able to publish high-quality working code and datasets.

Each pipeline that has run produces a unique fingerprint or trace. This informs about how the pipeline is configured, the structure and volume of data transacted and other useful metadata. By sharing this trace and metadata back to us, you would be able to choose what you send us

• the working configuration of your pipeline, that enables us to re-generate the same code.
• the schema of the data, that we can use to generate a synthetic data preview - since some schemas reflect private information, sending this will be optional.
• a sample dataset with actual data: Real data is better than synthetic, so if the information is not private, you can share a sample that helps others build on or preview the pipeline.

By leveraging different amounts of metadata, we would be able to

• guarantee that the code runs, as evidenced by a run trace - so even if we do not have credentials to test it ourselves, we can know from you that it works.
• provide you with test results on the data, and previews to help you decide

Finally, those of you who decide to deploy your pipelines on dltHub, can opt in to share your traces ongoing, enabling us to keep a health pulse of the pipeline and share any fixes with others.

How you can help now

As we described, we are early in the process of building this out. What we want to hear from you

• Any feedback you have of the overall process, so pick up a scaffold, smash it and let us know
• Any scaffolds you might want to see - tell us which.
• Any successful builds - if you offer us the correct code, we can use it now to improve the scaffold.

We think there’s much that can be done to improve what we have now and we are looking forward to your contributions and feedback as we move towards the really good stuff. Don’t be gentle, give us your criticism, if something feels clunky, break it and tell us where it hurts.

And we know, it would be really cool to share your code on the hub - we heard you and we’re getting there. In the meantime, write us an email on [email protected] or fill this temporary form to send us your feedback.