common.storages.fsspec_filesystem

FileItem Objects

class FileItem(TypedDict)

[view_source]

A DataItem representing a file

fsspec_filesystem

def fsspec_filesystem(
    protocol: str,
    credentials: FileSystemCredentials = None,
    kwargs: Optional[DictStrAny] = None,
    client_kwargs: Optional[DictStrAny] = None
) -> Tuple[AbstractFileSystem, str]

[view_source]

Instantiates an authenticated fsspec FileSystem for a given protocol and credentials.

Please supply credentials instance corresponding to the protocol. The protocol is just the code name of the filesystem i.e.:

• s3
• az, abfs, abfss, adl, azure
• gcs, gs

also see filesystem_from_config

prepare_fsspec_args

def prepare_fsspec_args(config: FilesystemConfiguration) -> DictStrAny

[view_source]

Prepare arguments for fsspec filesystem constructor.

Arguments:

• config FilesystemConfiguration - The filesystem configuration.

Returns:

• DictStrAny - The arguments for the fsspec filesystem constructor.

fsspec_from_config

def fsspec_from_config(
        config: FilesystemConfiguration) -> Tuple[AbstractFileSystem, str]

[view_source]

Instantiates an authenticated fsspec FileSystem from config argument.

Authenticates following filesystems:

• s3
• az, abfs, abfss, adl, azure
• gcs, gs
• sftp

All other filesystems are not authenticated

Returns: (fsspec filesystem, normalized url)

FileItemDict Objects

class FileItemDict(DictStrAny)

[view_source]

A FileItem dictionary with additional methods to get fsspec filesystem, open and read files.

__init__

def __init__(mapping: FileItem,
             credentials: Optional[Union[FileSystemCredentials,
                                         AbstractFileSystem]] = None)

[view_source]

Create a dictionary with the filesystem client.

Arguments:

• mapping FileItem - The file item TypedDict.
• credentials Optional[FileSystemCredentials], optional - The credentials to the filesystem. Defaults to None.

fsspec

@property
def fsspec() -> AbstractFileSystem

[view_source]

The filesystem client is based on the given credentials.

Returns:

• AbstractFileSystem - The fsspec client.

local_file_path

@property
def local_file_path() -> str

[view_source]

Gets a valid local filesystem path from file:// scheme. Supports POSIX/Windows/UNC paths

Returns:

• str - local filesystem path

open

def open(mode: str = "rb",
         compression: Literal["auto", "disable", "enable"] = "auto",
         **kwargs: Any) -> IO[Any]

[view_source]

Open the file as a fsspec file.

This method opens the file represented by this dictionary as a file-like object using the fsspec library.

Arguments:

• mode Optional[str] - Open mode.
• compression Optional[str] - A flag to enable/disable compression. Can have one of three values: "disable" - no compression applied, "enable" - gzip compression applied, "auto" (default) - compression applied only for files compressed with gzip.
• **kwargs Any - The arguments to pass to the fsspec open function.

Returns:

• IOBase - The fsspec file.

read_bytes

def read_bytes() -> bytes

[view_source]

Read the file content.

Returns:

• bytes - The file content.

glob_files

def glob_files(fs_client: AbstractFileSystem,
               bucket_url: str,
               file_glob: str = "**") -> Iterator[FileItem]

[view_source]

Get the files from the filesystem client.

Arguments:

• fs_client AbstractFileSystem - The filesystem client.
• bucket_url str - The url to the bucket.
• file_glob str - A glob for the filename filter.

Returns:

• Iterable[FileItem] - The list of files.