Karana.KUtils.multirun ====================== .. py:module:: Karana.KUtils.multirun .. autoapi-nested-parse:: Classes and functions needed to run multiple commands simultaneously and aggregate their results. Attributes ---------- .. autoapisummary:: Karana.KUtils.multirun.ParamValue Karana.KUtils.multirun.ParamMap Karana.KUtils.multirun.FileLinkage Karana.KUtils.multirun.RunStatus Classes ------- .. autoapisummary:: Karana.KUtils.multirun.FileResource Karana.KUtils.multirun.Run Karana.KUtils.multirun.AbstractRunner Karana.KUtils.multirun.RunStatistics Karana.KUtils.multirun.ParallelLocalRunner Functions --------- .. autoapisummary:: Karana.KUtils.multirun.displayStatus Karana.KUtils.multirun.getRunParams Karana.KUtils.multirun.postRunResults Karana.KUtils.multirun.multirun Karana.KUtils.multirun.multirunAsync Package Contents ---------------- .. py:function:: displayStatus(runner: Karana.KUtils.multirun._runner.AbstractRunner) :async: Display and asynchronously update a run status bar. :param runner: The runner to monitor :type runner: AbstractRunner .. py:function:: getRunParams() -> Karana.KUtils.multirun._types.ParamMap Get the parameters for this run. :returns: A map of the parameters for this run :rtype: ParamMap .. py:function:: postRunResults(results) Submit the results of the run. :param results: A picklable object to submit as results .. py:class:: FileResource Descriptor for a file to be used in runs. .. attribute:: path The path to the existing file :type: Path .. attribute:: linkage How to make the file available to runs - "copy": copy the file into each run directory - "shared": symlink to a shared copy in a common area - "link": symlink to the given file path :type: multirun.FileLinkage .. attribute:: rename Filename or path of the file in the run directory :type: str | Path | None .. py:attribute:: path :type: pathlib.Path .. py:attribute:: linkage :type: Karana.KUtils.multirun._types.FileLinkage :value: 'shared' .. py:attribute:: rename :type: str | pathlib.Path | None :value: None .. py:property:: name :type: pathlib.Path Get the filename in the run directory. :returns: The filename :rtype: Path .. py:function:: multirun(*, runs: collections.abc.Iterable[Karana.KUtils.multirun._types.ParamMap], cmd: list[str], output_path: pathlib.Path | str | None = None, input_files: list[FileResource | pathlib.Path | str] | None = None, runner: Karana.KUtils.multirun._runner.AbstractRunner | None = None) Run a command with varying parameters in distinct directories. :param runs: The parameters for each run :type runs: Iterable[ParamMap] :param cmd: The command to run :type cmd: list[str] :param output_path: A new directory to store the runs in. Defaults to a directory timestamped subdirectory in the current directory. :type output_path: Path | str | None :param input_files: A list of external files to copy/link into each run directory :type input_files: list[FileResource | Path | str] | None :param runner: A runner used to execute the runs. Defaults to using multirun.ParallelLocalRunner. :type runner: AbstractRunner | None .. py:function:: multirunAsync(*, runs: collections.abc.Iterable[Karana.KUtils.multirun._types.ParamMap], cmd: list[str], output_path: pathlib.Path | str | None = None, input_files: list[FileResource | pathlib.Path | str] | None = None, runner: Karana.KUtils.multirun._runner.AbstractRunner | None = None) :async: Run a command with varying parameters in distinct directories. :param runs: The parameters for each run :type runs: Iterable[ParamMap] :param cmd: The command to run :type cmd: list[str] :param output_path: A new directory to store the runs in. Defaults to a directory timestamped subdirectory in the current directory. :type output_path: Path | str | None :param input_files: A list of external files to copy/link into each run directory. If given as a Path or str, will be converted to a FileResource with default attributes. :type input_files: list[FileResource | Path | str] | None :param runner: A runner used to execute the runs. Defaults to using multirun.ParallelLocalRunner. :type runner: AbstractRunner | None .. py:class:: Run Handle to a Run task and its metadata. .. attribute:: task Awaitable task handle once one has been assigned :type: asyncio.Task | None .. attribute:: status Current state of the run - "skipped": The run will not be executed - "pending": The run is waiting to be executed - "running": The run is currently being executed - "success": The run completed without error - "failure": The run completed with an error :type: RunStatus .. py:attribute:: task :type: asyncio.Task | None :value: None .. py:attribute:: status :type: Karana.KUtils.multirun._types.RunStatus :value: 'pending' .. py:class:: AbstractRunner Bases: :py:obj:`abc.ABC` Interface for runner classes. .. py:method:: computeStatistics() -> RunStatistics :abstractmethod: Compute statistics about the runs. :returns: Container for statistics about the runs :rtype: RunStatistics .. py:method:: add(cmd: list[str], cwd: pathlib.Path = Path('.')) -> Run :abstractmethod: Add a run. :param cmd: The command to run :type cmd: list[str] :param cwd: The run directory. Defaults to the current directory. :type cwd: Path :returns: A handle to the run task and metadata :rtype: Run .. py:method:: gather() :abstractmethod: :async: Wait for all runs to complete. .. py:class:: RunStatistics Statistical data about the Runs. .. attribute:: skipped_count Number of runs with "skipped" status :type: int .. attribute:: pending_count Number of runs with "pending" status :type: int .. attribute:: running_count Number of runs with "running" status :type: int .. attribute:: success_count Number of runs with "success" status :type: int .. attribute:: failure_count Number of runs with "failure" status :type: int .. attribute:: complete_count Number of runs with any terminal status :type: int .. attribute:: total_count Total number of runs :type: int .. py:attribute:: skipped_count :type: int :value: 0 .. py:attribute:: pending_count :type: int :value: 0 .. py:attribute:: running_count :type: int :value: 0 .. py:attribute:: success_count :type: int :value: 0 .. py:attribute:: failure_count :type: int :value: 0 .. py:attribute:: complete_count :type: int :value: 0 .. py:attribute:: total_count :type: int :value: 0 .. py:class:: ParallelLocalRunner(*, max_parallel: int = 4) Bases: :py:obj:`AbstractRunner` Runner implementation that does runs in parallel locally. .. py:attribute:: semaphore .. py:attribute:: runs :value: [] .. py:method:: computeStatistics() -> RunStatistics Compute statistics about the run. :returns: The statistics about the run. :rtype: RunStatistics .. py:method:: add(cmd: list[str], cwd: pathlib.Path = Path('.')) -> Run Add a run. :param cmd: The command to run. :type cmd: list[str] :param cwd: The working directory to run it in. :type cwd: Path :returns: A Run instance with the tasks. :rtype: Run .. py:method:: gather() :async: Gather the results of the runs. .. py:data:: ParamValue .. py:data:: ParamMap .. py:data:: FileLinkage .. py:data:: RunStatus