dlt.common.storages.fsspecs.google_drive

GoogleDriveFileSystem Objects

class GoogleDriveFileSystem(AbstractFileSystem)

View source on GitHub

__init__

def __init__(credentials: GcpCredentials = None,
             trash_delete: bool = True,
             access: Optional[Literal["full_control",
                                      "read_only"]] = "full_control",
             spaces: Optional[Literal["drive", "appDataFolder",
                                      "photos"]] = "drive",
             **kwargs: Any)

View source on GitHub

Google Drive as a file-system.

The gdrive url has following format: gdrive://<root_file_id>/<file_path> Where <root_file_id> is a file id of the folder where the <file_path> is present.

Google Drive provides consistency when file ids are used. Changes are reflected immediately. In case of listings (ls) the consistency is eventual. Changes are reflected with a delay. As ls is used to retrieve file id from file name, we will be unable to build consistent filesystem with google drive API. Use this with care.

Based on original fsspec Google Drive implementation: https://github.com/fsspec/gdrivefs

Arguments:

• credentials GcpCredentials - Google Service credentials. If not provided, anonymous credentials are used
• trash_delete bool - If True sends files to trash on rm. If False, deletes permanently. Note that permanent delete is not available for shared drives access (Optional[Literal["full_control", "read_only"]]): One of "full_control", "read_only". spaces (Optional[Literal["drive", "appDataFolder", "photos"]]): Category of files to search, can be 'drive', 'appDataFolder' and 'photos'. Of these, only the first is general. **kwargs: Passed to the parent.

connect

def connect() -> None

View source on GitHub

Connect to Google Drive.

mkdir

def mkdir(path: str, create_parents: Optional[bool] = True) -> None

View source on GitHub

Create a directory.

Arguments:

• path str - The directory to create. create_parents (Optional[bool]): Whether to create parent directories if they don't exist. Defaults to True.

makedirs

def makedirs(path: str, exist_ok: Optional[bool] = True) -> None

View source on GitHub

Create a directory and all its parent components.

Arguments:

• path str - The directory to create.
• exist_ok Optional[bool] - Whether to raise an error if the directory already exists. Defaults to True.

rm

def rm(path: str,
       recursive: Optional[bool] = True,
       maxdepth: Optional[int] = None) -> None

View source on GitHub

Remove files or directories.

Arguments:

• path str - The file or directory to remove.
• recursive Optional[bool] - Whether to remove directories recursively. Defaults to True.
• maxdepth Optional[int] - The maximum depth to remove directories.

rmdir

def rmdir(path: str) -> None

View source on GitHub

Remove a directory.

Arguments:

• path str - The directory to remove.

export

def export(path: str, mime_type: str) -> Any

View source on GitHub

Convert a Google-native file to other format and download

mime_type is something like "text/plain"

ls

def ls(path: str,
       detail: Optional[bool] = False,
       refresh: Optional[bool] = False) -> Any

View source on GitHub

List files in a directory.

Arguments:

• path str - The directory to list.
• detail Optional[bool] - Whether to return detailed file information. Defaults to False.

Returns:

• Any - Files in the directory data.

path_to_file_id

def path_to_file_id(path: str,
                    parent_id: Optional[str] = None,
                    parent_path: str = "") -> str

View source on GitHub

Get the file ID from a path.

Arguments:

• path str - The path to get the file ID from.
• parent_id Optional[str] - The parent directory id to search.
• parent_path Optional[str] - Path corresponding to parent id

Returns:

• str - The file ID.

GoogleDriveFile Objects

class GoogleDriveFile(AbstractBufferedFile)

View source on GitHub

__init__

def __init__(fs: GoogleDriveFileSystem,
             path: str,
             mode: Optional[str] = "rb",
             block_size: Optional[int] = DEFAULT_BLOCK_SIZE,
             autocommit: Optional[bool] = True,
             **kwargs: Any)

View source on GitHub

A Google Drive file.

Arguments:

• fs AbstractFileSystem - The file system to open the file from.
• path str - The file to open.
• mode Optional[str] - The mode to open the file in. Defaults to "rb".
• block_size Optional[str] - The block size to use. Defaults to DEFAULT_BLOCK_SIZE.
• autocommit Optional[bool] - Whether to automatically commit the file. Defaults to True.
• **kwargs - Passed to the parent.

commit

def commit() -> None

View source on GitHub

If not auto-committing, finalize the file.

discard

def discard() -> None

View source on GitHub

Cancel in-progress multi-upload.