Telemetry
dlt
collects and reports anonymous usage information. This information is essential to figure
out how we should improve the library. Telemetry does not send any personal data. We create random
tracking cookie that is stored in your ~./dlt
directory. You can disable telemetry at any moment
or send it to your own servers instead.
How to opt-out
You can disable telemetry by adding --disable-telemetry
to any dlt
command.
This command will disable telemetry both in the current project and globally for the whole machine:
dlt --disable-telemetry
While this command will also permanently disable telemetry and then initialize the chess
pipeline:
dlt --disable-telemetry init chess duckdb
You can check the current telemetry status with this command:
dlt telemetry
The other way to disable telemetry is to set the runtime.dlthub_telemetry
option in config.toml
file in .dlt
folder.
[runtime]
dlthub_telemetry=false
What we send when
Anonymous telemetry is sent when:
- Any
dlt
command is executed from the command line. The data contains the command name. In the case ofdlt init
command, we also send the requested destination and data source names. - When
pipeline.run
is called, we send information when extract, normalize and load steps are completed. The data contains the destination name (e.g.duckdb
), hashes of: dataset name, pipeline name, default schema name, destination fingerprint (which is a hash of selected destination configuration fields), elapsed time, and if the step succeeded or not. - When
dbt
andairflow
helpers are used
Here is an example dlt init
telemetry message:
{
"anonymousId": "933dd165453d196a58adaf49444e9b4c",
"context": {
"ci_run": false,
"cpu": 8,
"exec_info": [],
"library": {
"name": "dlt",
"version": "0.2.0a25"
},
"os": {
"name": "Linux",
"version": "4.19.128-microsoft-standard"
},
"python": "3.8.11"
},
"event": "command_init",
"properties": {
"destination_name": "bigquery",
"elapsed": 3.1720383167266846,
"event_category": "command",
"event_name": "init",
"pipeline_name": "pipedrive",
"success": true
}
}
Example for load
pipeline run step:
{
"anonymousId": "570816b273a41d16caacc26a797204d9",
"context": {
"ci_run": false,
"cpu": 3,
"exec_info": [],
"library": {
"name": "dlt",
"version": "0.2.0a26"
},
"os": {
"name": "Darwin",
"version": "21.6.0"
},
"python": "3.10.10"
},
"event": "pipeline_load",
"properties": {
"destination_name": "duckdb",
"destination_fingerprint": "",
"pipeline_name_hash": "OpVShb3cX7qQAmOZSbV8",
"dataset_name_hash": "Hqk0a3Ov5AD55KjSg2rC",
"default_schema_name_hash": "Hqk0a3Ov5AD55KjSg2rC",
"elapsed": 2.234885,
"event_category": "pipeline",
"event_name": "load",
"success": true,
"transaction_id": "39c3b69c858836c36b9b7c6e046eb391"
}
}
The message context
The message context contains the following information:
anonymousId
: a random tracking cookie stored in~/.dlt/.anonymous_id
.ci_run
: a flag indicating if the message was sent from a CI environment (e.g.Github Actions
,Travis CI
).cpu
: contains number of cores.exec_info
: contains a list of strings that identify execution environment: (e.g.kubernetes
,docker
,airflow
).- The
library
,os
, andpython
give us some understanding of the runtime environment of thedlt
.
Send telemetry data to your own tracker
You can setup your own tracker to receive telemetry events. You can create scalable, globally distributed
edge service using dlt
and Cloudflare.
Once your tracker is running, point dlt
to it. You can use global config.toml
to redirect all pipelines on
a given machine.
[runtime]
dlthub_telemetry_endpoint="telemetry-tracker.services4745.workers.dev"
Track events with Segment
You can send the anonymous telemetry to your own Segment
account. You need to create a HTTP Server source and generate a WRITE KEY, which you then pass to
the config.toml
like this:
[runtime]
dlthub_telemetry_endpoint="https://api.segment.io/v1/track"
dlthub_telemetry_segment_write_key="<write_key>"