kaiju_files.abc module

class AbstractFileTransportInterface[source]

Bases: ABC

Connector to an external file storage.

abstract async has_new_files()[source]

Return True if new downloadable files found in shared folders.

Return type:

bool

abstract async list()[source]

List all files in shared folders.

Return type:

AsyncGenerator[Path, None]

abstract async download(uri)[source]

Download a file to a local temp dir and return the location.

Parameters:

uri (Path) –

Return type:

NamedTemporaryFile

abstract async delete(uri)[source]

Remove a downloaded file from a shared directory.

Parameters:

uri (Path) –

abstract async mark_failed(uri, reason, message='')[source]

Mark a shared file as failed and (optionally) move it to another shared location.

Parameters:

  • uri (Path) –

  • reason (ErrorCodes) –

  • message (str) –

class AbstractFileLoader[source]

Bases: AbstractFileOperation, Service, ABC

File uploading and metadata interface.

class Settings[source]

Bases: Settings

File loader settings.

filename_mask
directory_mask
ext
meta
output_extension
__init__(app, *args, logger=None, **kws)[source]

Initialize.

abstract async upload(data, **metadata)[source]

Upload a file from local temp dir and sets its metadata.

Parameters:

data (NamedTemporaryFile) –

class AbstractFileConverter[source]

Bases: AbstractFileOperation, ABC

File converter/normalizer which is supposed to be run in a thread / process.

class Settings[source]

Bases: Settings

File converter settings.

filename_mask
directory_mask
ext
meta
output_extension
READ_MODE = 'rb'
WRITE_MODE = 'wb'
MAX_PROCESSING_TIME = 300
__init__(dir='.', read_mode='rb', write_mode='wb', max_processing_time=300, settings=None)[source]

Initialize.

Parameters:

  • dir (str) – path to a temp local data storage

  • read_mode – read mode for opening original files

  • write_mode – write mode for writing new (converted) files

  • max_processing_time – maximum file processing time in seconds

  • settings (dict | List[dict] | None) – converter specific settings

convert(file, return_exceptions=False, **metadata)[source]

Call a converter and returns created file paths and metadata.

Parameters:

file (Path | str | NamedTemporaryFile) –

Return type:

List[Tuple[NamedTemporaryFile, dict]]

exception FileOperationConfigurationError[source]

Bases: ValueError

An error due to invalid file converter settings.