Loading Data from Cisco Meraki
to CockroachDB
Using dlt
in Python
We will be using the dlt PostgreSQL destination to connect to CockroachDB. You can get the connection string for your CockroachDB database as described in the CockroachDB Docs.
Join our Slack community or book a call with our support engineer Violetta.
Cisco Meraki
is a cloud-managed IT solution that simplifies the management of wireless, switching, security, and mobile device management. It provides powerful tools for network visibility, control, and automation. With an intuitive dashboard, Cisco Meraki
allows organizations to manage their entire network from a single interface, enhancing operational efficiency and security. It offers features like real-time monitoring, remote troubleshooting, and automated updates to ensure optimal network performance.
This documentation explains how to load data from Cisco Meraki
to CockroachDB
using the open source python library called dlt
. CockroachDB
is a distributed, cloud-native SQL database that is Kubernetes compatible and free up to 5GB and 1vCPU. By leveraging dlt
, users can efficiently transfer and manage their Cisco Meraki
data within CockroachDB
.
dlt
Key Features
- Pipeline Metadata:
dlt
pipelines leverage metadata to provide governance capabilities, including load IDs that enable incremental transformations and data vaulting. Learn more - Schema Enforcement and Curation: Ensure data consistency and quality by enforcing and curating schemas, which guide the processing and loading of data. Read more
- Schema Evolution:
dlt
alerts users to schema changes, allowing proactive governance by notifying stakeholders of modifications in the source data’s schema. Learn more - Scaling and Finetuning: Offers several mechanisms and configuration options to scale up and finetune pipelines, such as running extraction, normalization, and load in parallel. Read more
- Supported Data Types:
dlt
supports a variety of data types including text, double, bool, timestamp, date, time, bigint, binary, complex, decimal, and wei. Learn more
Getting started with your pipeline locally
dlt-init-openapi
0. Prerequisites
dlt
and dlt-init-openapi
requires Python 3.9 or higher. Additionally, you need to have the pip
package manager installed, and we recommend using a virtual environment to manage your dependencies. You can learn more about preparing your computer for dlt in our installation reference.
1. Install dlt and dlt-init-openapi
First you need to install the dlt-init-openapi
cli tool.
pip install dlt-init-openapi
The dlt-init-openapi
cli is a powerful generator which you can use to turn any OpenAPI spec into a dlt
source to ingest data from that api. The quality of the generator source is dependent on how well the API is designed and how accurate the OpenAPI spec you are using is. You may need to make tweaks to the generated code, you can learn more about this here.
# generate pipeline
# NOTE: add_limit adds a global limit, you can remove this later
# NOTE: you will need to select which endpoints to render, you
# can just hit Enter and all will be rendered.
dlt-init-openapi cisco_meraki --url https://raw.githubusercontent.com/dlt-hub/openapi-specs/main/open_api_specs/Business/cisco_meraki.yaml --global-limit 2
cd cisco_meraki_pipeline
# install generated requirements
pip install -r requirements.txt
The last command will install the required dependencies for your pipeline. The dependencies are listed in the requirements.txt
:
dlt>=0.4.12
You now have the following folder structure in your project:
cisco_meraki_pipeline/
├── .dlt/
│ ├── config.toml # configs for your pipeline
│ └── secrets.toml # secrets for your pipeline
├── rest_api/ # The rest api verified source
│ └── ...
├── cisco_meraki/
│ └── __init__.py # TODO: possibly tweak this file