ExporterPool

class resilient_exporters.exporters.ExporterPool(exporters: Optional[Iterable[resilient_exporters.exporters.exporter.Exporter]], transform: Optional[Callable] = None, num_threads: int = 1, wait_for_result: bool = True, use_memory: bool = True, manual_reexport: bool = False, *, tmp_file: Optional[Union[str, pathlib.Path, BinaryIO]] = None, save_unsent_data: bool = True, name: Optional[str] = None)

Enables pooling of exporters for improved efficiency and performance. All the exporters will be managed by the pool, including the saving of unsent data, with only one send call to be used. It also offers a multithreading option to run the exporters’ send functions in parallel, to speed up the execution of ExporterPool.send.

Parameters

• exporters (Iterable[Exporter]) – a list of resilient_exporters.exporters.Exporter.

• transform (Callable) – a function to be invoked at each send call on the passed data. It must return data or exceptions will be raised.

• num_threads (int) – the number of threads to use for send calls. Must be greater than 1. If 1, multithreading is disabled. Default to 1.

• wait_for_result (bool) – value to decide if the instance has to wait for the results of send calls or not when multithreading is enabled.

• manual_reexport (bool) – if True, the user is responsible to call the function send_unsent_data when appropriate. If False, the instance will manage that automatically by assessing the necessity to call the function at each send call; the criterias are 1) there’s unsent data, 2) there’s an Internet connection. Default to False.

Note

If using parallelism, the default behaviour is to wait for the results of all calls. One can disable this behaviour with wait_for_result set to False, and ExporterPool.send will be then non-blocking, but will return None.

Raises

InvalidConfigError – if num_threads is < 1.

Example

import resilient_exporters as rex

exporter1 = rex.exporters.FileExporter("local_file.txt")
exporter2 = rex.exporters.FileExporter("/path/to/network/file.txt")
pool = ExporterPool([exporter1, exporter2], num_threads=2)

line = "A string to be written in a file"
pool.send(line)

num_threads

number of threads used by the instance.

Type

int

wait_for_result

value to decide if the instance has to wait for the results of send calls or not when multithreading is enabled.

Type

bool

property exporters

A dictionary of the contained exporters.

add_exporter(exporter: resilient_exporters.exporters.exporter.Exporter) → None

Adds an exporter to the pool. Use this function to add an exporter after the pool has been initialised. It removes the responsability to run the transform method from the exporter, and to save unsent data.

Parameters

exporter (Exporter) – an exporter.

send(data: Any, **kwargs) → List[resilient_exporters.exporters.ExportResult]

Runs the send method of all its exporters. If the pool’s num_threads attribute is > 1, it will execute all the calls in separate threads.

Parameters

• data (Any) – the data to export.

• **kwargs (Any) – the keyword arguments to pass down to the exporters’ send methods.

Returns

a list of the exporters’ results.

Return type

List[ExportResult]

send_unsent_data() → List[resilient_exporters.exporters.ExportResult]

Tries to send the previously saved, unsent data.

Returns

list of the results of the export jobs.

Return type

List[ExportResult]

has_unsent_data() → bool

Assesses if there’s saved, unsent data.

Returns

True if there’s saved, unsent data. False otherwise.

Return type

bool

save_unsent_data(data: Any, kwargs: dict, exporter_name: str) → None

Saves the data in memory or disk, depending on the value of use_memory of the instance, so it can be sent later.

Parameters

• data (Any) – the core data.

• kwargs (dict) – the keyword arguments to pass to send for the given data.

• exporter_name (str) – the name of the exporter.

Returns

None

transform(data: Any) → Any

Applies transform to the given data.

Parameters

data (Any) – the data to pass to transform.

Returns

the output of transform

Return type

Any

Raises

Exception – if the output of transform(data) is None.

property use_memory

The value use_memory of the instance.

When set to True, it loads the data into memory if it was previously using a file (previous value was False), or vice versa.

Parameters

new_val (bool) – new value for use_memory.

Returns

its value

Return type

bool