common.storages.fsspec_filesystem
FileItem Objects
class FileItem(TypedDict)
A DataItem representing a file
fsspec_filesystem
def fsspec_filesystem(
protocol: str,
credentials: FileSystemCredentials = None,
kwargs: Optional[DictStrAny] = None,
client_kwargs: Optional[DictStrAny] = None
) -> Tuple[AbstractFileSystem, str]
Instantiates an authenticated fsspec FileSystem
for a given protocol
and credentials.
Please supply credentials instance corresponding to the protocol.
The protocol
is just the code name of the filesystem i.e.:
- s3
- az, abfs, abfss, adl, azure
- gcs, gs
also see filesystem_from_config
prepare_fsspec_args
def prepare_fsspec_args(config: FilesystemConfiguration) -> DictStrAny
Prepare arguments for fsspec filesystem constructor.
Arguments:
config
FilesystemConfiguration - The filesystem configuration.
Returns:
DictStrAny
- The arguments for the fsspec filesystem constructor.
fsspec_from_config
def fsspec_from_config(
config: FilesystemConfiguration) -> Tuple[AbstractFileSystem, str]
Instantiates an authenticated fsspec FileSystem
from config
argument.
Authenticates following filesystems:
- s3
- az, abfs, abfss, adl, azure
- gcs, gs
- sftp
All other filesystems are not authenticated
Returns: (fsspec filesystem, normalized url)
FileItemDict Objects
class FileItemDict(DictStrAny)
A FileItem dictionary with additional methods to get fsspec filesystem, open and read files.
__init__
def __init__(mapping: FileItem,
credentials: Optional[Union[FileSystemCredentials,
AbstractFileSystem]] = None)
Create a dictionary with the filesystem client.
Arguments:
mapping
FileItem - The file item TypedDict.credentials
Optional[FileSystemCredentials], optional - The credentials to the filesystem. Defaults to None.
fsspec
@property
def fsspec() -> AbstractFileSystem
The filesystem client is based on the given credentials.
Returns:
AbstractFileSystem
- The fsspec client.
local_file_path
@property
def local_file_path() -> str
Gets a valid local filesystem path from file:// scheme. Supports POSIX/Windows/UNC paths
Returns:
str
- local filesystem path
open
def open(mode: str = "rb",
compression: Literal["auto", "disable", "enable"] = "auto",
**kwargs: Any) -> IO[Any]
Open the file as a fsspec file.
This method opens the file represented by this dictionary as a file-like object using the fsspec library.
Arguments:
mode
Optional[str] - Open mode.compression
Optional[str] - A flag to enable/disable compression. Can have one of three values: "disable" - no compression applied, "enable" - gzip compression applied, "auto" (default) - compression applied only for files compressed with gzip.**kwargs
Any - The arguments to pass to the fsspec open function.
Returns:
IOBase
- The fsspec file.
read_bytes
def read_bytes() -> bytes
Read the file content.
Returns:
bytes
- The file content.
glob_files
def glob_files(fs_client: AbstractFileSystem,
bucket_url: str,
file_glob: str = "**") -> Iterator[FileItem]
Get the files from the filesystem client.
Arguments:
fs_client
AbstractFileSystem - The filesystem client.bucket_url
str - The url to the bucket.file_glob
str - A glob for the filename filter.
Returns:
Iterable[FileItem]
- The list of files.