Karana.KUtils.multirun

Karana.KUtils.multirun#

This package contains the classes and functions needed to run multiple commands simultaneously and aggregate their results.

Attributes#

`ParamValue`
`ParamMap`
`FileLinkage`
`RunStatus`

Classes#

`FileResource`	Descriptor for a file to be used in runs.
`Run`	Handle to a Run task and its metadata.
`AbstractRunner`	Interface for runner classes.
`RunStatistics`	Statistical data about the Runs.
`ParallelLocalRunner`	Runner implementation that does runs in parallel locally.

Functions#

`displayStatus`(runner)	Display and asynchronously update a run status bar.
`getRunParams`(→ Karana.KUtils.multirun._types.ParamMap)	Get the parameters for this run.
`postRunResults`(results)	Submit the results of the run.
`multirun`(*, runs, cmd[, output_path, input_files, runner])	Run a command with varying parameters in distinct directories.
`multirunAsync`(*, runs, cmd[, output_path, ...])	Run a command with varying parameters in distinct directories.

Package Contents#

async Karana.KUtils.multirun.displayStatus(runner: Karana.KUtils.multirun._runner.AbstractRunner)[source]#

Display and asynchronously update a run status bar.

Parameters:: runner (AbstractRunner) – The runner to monitor

Karana.KUtils.multirun.getRunParams() → Karana.KUtils.multirun._types.ParamMap[source]#

Get the parameters for this run.

Returns:: A map of the parameters for this run
Return type:: ParamMap

Karana.KUtils.multirun.postRunResults(results)[source]#

Submit the results of the run.

Parameters:: results – A picklable object to submit as results

class Karana.KUtils.multirun.FileResource[source]#

Descriptor for a file to be used in runs.

path#

The path to the existing file

Type:: Path

linkage#

How to make the file available to runs - “copy”: copy the file into each run directory - “shared”: symlink to a shared copy in a common area - “link”: symlink to the given file path

Type:: multirun.FileLinkage

rename#

Filename or path of the file in the run directory

Type:: str | Path | None

path: pathlib.Path#

linkage: Karana.KUtils.multirun._types.FileLinkage = 'shared'#

rename: str | pathlib.Path | None = None#

property name: pathlib.Path#

Get the filename in the run directory.

Returns:: The filename
Return type:: Path

Karana.KUtils.multirun.multirun(*, runs: collections.abc.Iterable[Karana.KUtils.multirun._types.ParamMap], cmd: list[str], output_path: pathlib.Path | str | None = None, input_files: list[FileResource | pathlib.Path | str] | None = None, runner: Karana.KUtils.multirun._runner.AbstractRunner | None = None)[source]#

Run a command with varying parameters in distinct directories.

Parameters:

runs (Iterable[ParamMap]) – The parameters for each run
cmd (list[str]) – The command to run
output_path (Path | str | None) – A new directory to store the runs in. Defaults to a directory timestamped subdirectory in the current directory.
input_files (list[FileResource | Path | str] | None) – A list of external files to copy/link into each run directory
runner (AbstractRunner | None) – A runner used to execute the runs. Defaults to using multirun.ParallelLocalRunner.

async Karana.KUtils.multirun.multirunAsync(*, runs: collections.abc.Iterable[Karana.KUtils.multirun._types.ParamMap], cmd: list[str], output_path: pathlib.Path | str | None = None, input_files: list[FileResource | pathlib.Path | str] | None = None, runner: Karana.KUtils.multirun._runner.AbstractRunner | None = None)[source]#

Run a command with varying parameters in distinct directories.

Parameters:

runs (Iterable[ParamMap]) – The parameters for each run
cmd (list[str]) – The command to run
output_path (Path | str | None) – A new directory to store the runs in. Defaults to a directory timestamped subdirectory in the current directory.
input_files (list[FileResource | Path | str] | None) – A list of external files to copy/link into each run directory. If given as a Path or str, will be converted to a FileResource with default attributes.
runner (AbstractRunner | None) – A runner used to execute the runs. Defaults to using multirun.ParallelLocalRunner.

class Karana.KUtils.multirun.Run[source]#

Handle to a Run task and its metadata.

task#

Awaitable task handle once one has been assigned

Type:: asyncio.Task | None

status#

Current state of the run - “skipped”: The run will not be executed - “pending”: The run is waiting to be executed - “running”: The run is currently being executed - “success”: The run completed without error - “failure”: The run completed with an error

Type:: RunStatus

task: asyncio.Task | None = None#

status: Karana.KUtils.multirun._types.RunStatus = 'pending'#

class Karana.KUtils.multirun.AbstractRunner[source]#

Bases: abc.ABC

Interface for runner classes.

abstract computeStatistics() → RunStatistics[source]#

Compute statistics about the runs.

Returns:: Container for statistics about the runs
Return type:: RunStatistics

abstract add(cmd: list[str], cwd: pathlib.Path = Path('.')) → Run[source]#

Add a run.

Parameters:

cmd (list[str]) – The command to run
cwd (Path) – The run directory. Defaults to the current directory.

Returns:

A handle to the run task and metadata

Return type:

Run

abstract gather()[source]#

Async:

Wait for all runs to complete.

class Karana.KUtils.multirun.RunStatistics[source]#

Statistical data about the Runs.

skipped_count#

Number of runs with “skipped” status

Type:: int

pending_count#

Number of runs with “pending” status

Type:: int

running_count#

Number of runs with “running” status

Type:: int

success_count#

Number of runs with “success” status

Type:: int

failure_count#

Number of runs with “failure” status

Type:: int

complete_count#

Number of runs with any terminal status

Type:: int

total_count#

Total number of runs

Type:: int

skipped_count: int = 0#

pending_count: int = 0#

running_count: int = 0#

success_count: int = 0#

failure_count: int = 0#

complete_count: int = 0#

total_count: int = 0#

class Karana.KUtils.multirun.ParallelLocalRunner(*, max_parallel: int = 4)[source]#

Bases: AbstractRunner

Runner implementation that does runs in parallel locally.

semaphore#

runs = []#

computeStatistics() → RunStatistics[source]#

Compute statistics about the run.

Returns:: The statistics about the run.
Return type:: RunStatistics

add(cmd: list[str], cwd: pathlib.Path = Path('.')) → Run[source]#

Add a run.

Parameters:

cmd (list[str]) – The command to run.
cwd (Path) – The working directory to run it in.

Returns:

A Run instance with the tasks.

Return type:

Run

async gather()[source]#: Gather the results of the runs.

Karana.KUtils.multirun.ParamValue#

Karana.KUtils.multirun.ParamMap#

Karana.KUtils.multirun.FileLinkage#

Karana.KUtils.multirun.RunStatus#