Highlights
- Composable data stack with dlt: Dentolo’s data ingestion solution, powered by dlt, demonstrates how a composable data stack avoids the all-or-nothing problem. Dentolo’s team can seamlessly integrate new data sources into the existing infrastructure without a complete overhaul. As the team grew to love dlt’s power, and the friendly, technical community support, it could transition more components across.
- Seamless integration of custom data sources: Dentolo initially struggled with integrating a crucial proprietary data source using their existing data ingestion setup, primarily reliant on Airbyte and a proprietary paid solution. Migrating to dlt allowed them to create a custom connector easily, leading to a smoother and more efficient integration process without extensive manual troubleshooting.
- Avoiding vendor lock-in: Previous solutions included use of a tool from a paid proprietary data source vendor. By moving to dlt, Dentolo’s data team can connect new data sources of any type, saving time, cost, and de-risked the vendor by moving to an open source solution with a busy community.
- Democratization of data access: With the introduction of dlt, the data ingestion setup now is much easier to maintain. It also became much easier to share the work with all Python developers and get reviewed by colleagues.
Data Stack
- Data sources: Postgres and other API-based sources (related to marketing, sales, operations, etc.)
- Destination: Postgres
- Orchestration: Airflow
- Transformation: dbt
Challenge
Dentolo conducts daily analysis of data from numerous sources. In the original setup, Airbyte and custom Python scripts primarily covered their original data ingestion setup. Due to ongoing challenges with plugging in data from a custom source, a proprietary API crucial to their operations, Dentolo began to look for other solutions.
Daniel Le, Data Engineer at Dentolo, recalls initially attempting to integrate this custom source using the Airbyte custom connector. However, he soon discovered that building and managing this custom integration with Airbyte did work acceptably, integrating a custom source is difficult.
Airbyte's documentation was lacking, and its support community didn't provide necessary assistance for more difficult questions. Technical issues and bugs were met with generic AI-generated responses instead of real assistance to help solve complex problems, leading to prolonged troubleshooting and mounting frustration.
Moreover, Airbyte’s frequent updates often caused configuration issues, disrupting Dentolo’s workflow and requiring extensive manual adjustments. This constant need to resolve problems became unsustainable.
Dentolo needed a lightweight solution specifically for their custom source integration without migrating all their existing data sources. Already struggling with vendor lock-in on one API, and not willing to embrace a new and heavy SaaS platform, Dentolo was seeking more control and flexibility over their tools. The solution needed to be easy to implement, maintain, while being able to adapt to unique needs.
Meet dlt
When the Dentolo data team first discovered dlt, they were intrigued but showed sensible cautiousness. Although the data community was buzzing about dlt, risk-averse companies require specific reassurances. Initial hesitation stemmed from concerns about the library's maturity and readiness for production, as it was still in its early versions.
While applying caution, Dentolo’s data team decided to try dlt, finding the library easy to work with, especially appreciating its good documentation and responsive community support.
The tipping point to start trying out dlt was that when I had a question, I posted it in the community, and it got answered right away. All the following questions were also answered within the day. I remember I told my manager: “this response time is great, and important.” And we became confident about dlt.
— Daniel Le, Data Engineer at Dentolo
Another crucial area of confidence came from knowing that dlt had backing from established organizations like Hugging Face, reassuring them of its longer-term reliability. This combination of positive experiences and support ultimately led Dentolo to adopt dlt for their data needs.
Solution
Dentolo's data ingestion solution, powered by dlt, is a game-changer for their data management. Starting with one custom source built with dlt, the team not only implemented all new sources with dlt, but also started to migrate existing ones.
Daniel and his team built a lightweight deployment integration, which allowed Python engineers to easily deploy new dlt connectors on EC2 allowing quick prototyping and testing. All integrated sources use ECS, managed by Airflow, for efficient scaling and cost savings. This setup allows the Dentolo data team to easily maintain and review the code. Euan Johnston, Senior Data Analyst in Dentolo highlighted how straightforward it is to jump in and make adjustments.
One of the most important aspects of Dentolo's data ingestion solution is its composable data stack. This means that they didn't need to overhaul their entire existing infrastructure or embark on a lengthy and complex development process to migrate all their data sources to the new technology. Instead, new data sources were seamlessly plugged into their current setup and old sources removed or replaced as needed.
The team was able to avoid the vendor lock-in issues it experienced, while avoiding some of the limitations with Airbyte.
Result
Overall, using dlt has lowered the technical barriers, simplified maintenance, and improved the efficiency of Dentolo’s data pipelines, delivering much smoother loading and ingestion, while reducing the complexity and maintenance burden of its data pipelines.
The ease of deploying and managing data sources has allowed their analytics engineers to work more efficiently, leading to quicker iterations and improvements.
Furthermore, Dentolo democratized data access, allowing any Python developer to write and maintain data sources.
In insurance, there is no compromise on data integrity or security. dlt's approach allowed us to enhance our data pipeline piece by piece, integrating custom sources alongside our existing systems. It's given our data team the flexibility they needed, while letting us squeeze more value out of the data we already had in place. I appreciate the entire team’s work in making this transition so seamless. We’re now positioned to scale and adapt as our needs evolve.
- Erol Mesut Gün, Data lead in Dentolo
Additionally, the flexibility of dlt has enabled incremental loading and seamless integration with their existing infrastructure, leading to better performance and cost savings. Overall, this transition has not only streamlined Dentolo’s data processes and positioned the company for scalability and future growth.
About Dentolo
Dentolo is the supplementary dental insurance that makes dental health a priority - and focuses on correct and regular preventative care. Innovative, simple, digital and always there when it's important.
As a brand of Dentolo Deutschland GmbH, dentolo is a subsidiary of the Zurich Group and a partner of DA Direkt.