Files Python API Docs | dltHub

Gemini Files API is a platform for uploading, storing (temporary), and managing media and document files for use with the Gemini (Generative Language) models. The REST API base URL is https://generativelanguage.googleapis.com/v1beta and All requests require an API key (x-goog-api-key header or ?key= query parameter); OAuth2 bearer tokens are also supported..

dlt is an open-source Python library that handles authentication, pagination, and schema evolution automatically. dlthub provides AI context files that enable code assistants to generate production-ready pipelines. Install with uv pip install "dlt[workspace]" and start loading Files data in under 10 minutes.

---

What data can I load from Files?

Here are some of the endpoints you can load from Files:

---

How do I authenticate with the Files API?

The Files API accepts an API key sent as the x-goog-api-key HTTP header or as the key query parameter. For Google Cloud authenticated clients, standard OAuth2 Bearer tokens (Authorization: Bearer ) can be used.

1. Get your credentials

1. Sign in to Google AI Studio or Google Cloud Console. 2) For API keys: navigate to the API Keys page (https://aistudio.google.com/apikey) and create a new API key. 3) For OAuth/service accounts: create a service account in Google Cloud Console, grant the required project permissions, generate a key, and obtain an access token via gcloud auth or OAuth flows. 4) Store the API key or access token securely (see secrets.toml example).

2. Add them to .dlt/secrets.toml

Copy
[sources.files_source]
api_key = "YOUR_GEMINI_API_KEY_HERE"

dlt reads this automatically at runtime — never hardcode tokens in your pipeline script. For production environments, see setting up credentials with dlt for environment variable and vault-based options.

---

How do I set up and run the pipeline?

Set up a virtual environment and install dlt:

Copy
uv venv && source .venv/bin/activate
uv pip install "dlt[workspace]"

1. Install the dlt AI Workbench:

Copy
dlt ai init --agent <your-agent> # <agent>: claude | cursor | codex

This installs project rules, a secrets management skill, appropriate ignore files, and configures the dlt MCP server for your agent. Learn more →

2. Install the rest-api-pipeline toolkit:

Copy
dlt ai toolkit rest-api-pipeline install

This loads the skills and context about dlt the agent uses to build the pipeline iteratively, efficiently, and safely. The agent uses MCP tools to inspect credentials — it never needs to read your secrets.toml directly. Learn more →

3. Start LLM-assisted coding:

Use /find-source to load data from the Files API into DuckDB.

The rest-api-pipeline toolkit takes over from here — it reads relevant API documentation, presents you with options for which endpoints to load, and follows a structured workflow to scaffold, debug, and validate the pipeline step by step.

4. Run the pipeline:

Copy
python files_pipeline.py

If everything is configured correctly, you'll see output like this:

Copy
Pipeline files_pipeline load step completed in 0.26 seconds
1 load package(s) were loaded to destination duckdb and into dataset files_data
The duckdb destination used duckdb:/files.duckdb location to store data
Load package 1749667187.541553 is LOADED and contains no failed jobs

Inspect your pipeline and data:

Copy
dlt pipeline files_pipeline show

This opens the Pipeline Dashboard where you can verify pipeline state, load metrics, schema (tables, columns, types), and query the loaded data directly.

---

Python pipeline example

This example loads files and file_search_documents from the Files API into DuckDB. It mirrors the endpoint and data selector configuration from the table above:

Copy
import dlt
from dlt.sources.rest_api import RESTAPIConfig, rest_api_resources

@dlt.source
def files_source(api_key=dlt.secrets.value):
    config: RESTAPIConfig = {
        "client": {
            "base_url": "https://generativelanguage.googleapis.com/v1beta",
            "auth": {
                "type": "api_key",
                "api_key": api_key,
            },
        },
        "resources": [
            {"name": "files", "endpoint": {"path": "v1beta/files", "data_selector": "files"}},
            {"name": "file", "endpoint": {"path": "v1beta/files/{name}", "data_selector": "file"}}
        ],
    }
    yield from rest_api_resources(config)


def get_data() -> None:
    pipeline = dlt.pipeline(
        pipeline_name="files_pipeline",
        destination="duckdb",
        dataset_name="files_data",
    )
    load_info = pipeline.run(files_source())
    print(load_info)

To add more endpoints, append entries from the resource table to the "resources" list using the same name, path, and data_selector pattern.

---

How do I query the loaded data?

Once the pipeline runs, dlt creates one table per resource. You can query with Python or SQL.

Python (pandas DataFrame):

Copy
import dlt

data = dlt.pipeline("files_pipeline").dataset()
sessions_df = data.files.df()
print(sessions_df.head())

SQL (DuckDB example):

Copy
SELECT * FROM files_data.files LIMIT 10;

In a marimo or Jupyter notebook:

Copy
import dlt

data = dlt.pipeline("files_pipeline").dataset()
data.files.df().head()

See how to explore your data in marimo Notebooks and how to query your data in Python with dataset.

---

What destinations can I load Files data to?

dlt supports loading into any of these destinations — only the destination parameter changes:

Change the destination in dlt.pipeline(destination="snowflake") and add credentials in .dlt/secrets.toml. See the full destinations list.

---

Troubleshooting

Authentication failures

If you get 401/403: verify you are sending a valid API key (x-goog-api-key header or ?key=). If using OAuth, ensure your Bearer token is valid and has the required scopes and project access.

Rate limits / Quotas

The Files API is subject to project quotas (uploads, requests). If you receive 429 or quota errors, back off and retry with exponential backoff and check your project quota in Google Cloud Console.

Upload / resumable upload issues

Resumable uploads use the /upload/v1beta/files flow. If you don't receive an x-goog-upload-url in the initial start response, re-check the start request headers (X-Goog-Upload-Protocol: resumable and X-Goog-Upload-Command: start) and Content-Type. If upload fails mid-stream, use the upload URL and X-Goog-Upload-Offset to resume.

Pagination quirks

List endpoints may be paginated. Use the returned pageToken/nextPageToken in subsequent requests to iterate all files. The SDKs (Python/JS) provide iterator helpers — prefer SDK iterators when possible.

File lifetime and access

Uploaded File objects are temporary (stored ~48 hours) and cannot be downloaded directly; use the returned file.uri to reference the file in model requests. Delete files when done to free temporary storage.

Ensure that the API key is valid to avoid 401 Unauthorized errors. Also, verify endpoint paths and parameters to avoid 404 Not Found errors.

---

Next steps

Continue your data engineering journey with the other toolkits of the dltHub AI Workbench:

• data-exploration — Build custom notebooks, charts, and dashboards for deeper analysis with marimo notebooks.
• dlthub-runtime — Deploy, schedule, and monitor your pipeline in production.

Copy
dlt ai toolkit data-exploration install
dlt ai toolkit dlthub-runtime install

• How to deploy a pipeline
• How to explore your data in marimo Notebooks
• How to query your data in Python with dataset
• Join the dlt community on Slack