Scheduler

Scheduler#

Here is the template class to build your Scheduler based on your needs as well as the already implemented and tested scheduler.

class bindflow.orchestration.generate_scheduler.FrontEnd(cluster_config: None = None, out_dir: PathLike | str | bytes = '.', prefix_name: str = '', snake_executor_file: str | None = None)[source]#

__cluster_validation__()[source]#: Each scheduler should validate if the necessary options, as partition, CPUs, etc are in cluster_config.

__init__(cluster_config: None = None, out_dir: PathLike | str | bytes = '.', prefix_name: str = '', snake_executor_file: str | None = None) → None[source]#

Constructor of the class

Parameters:

cluster_config (dict) – All the necessary information for the specific schedular
out_dir (PathLike, optional) – Where all files will be exported and executed, by default ‘.’
prefix_name (str, optional) – A prefix append to the jobs names for easy identification, by default ‘’
snake_executor_file (str, optional) – The name/path of the file that will be used for execution of the workflow, by default None

build_snakemake(jobs: int = 12, latency_wait: int = 360, verbose: bool = False, debug_dag: bool = False, rerun_incomplete: bool = True, keep_incomplete: bool = True, keep_going: bool = True) → str[source]#

Build the snakemake command TODO Consider to put it in the parent class

Parameters:

jobs (int, optional) – Use at most N CPU cluster/cloud jobs in parallel. For local execution this is an alias for –cores. Note: Set to ‘unlimited’ in case, this does not play a role. For cluster this is just a limitation. It is advise to provided a big number in order to do not wait for finishing of the jobs rather that launch all in the queue, by default 100000
latency_wait (int, optional) – Wait given seconds if an output file of a job is not present after the job finished. This helps if your filesystem suffers from latency, by default 120
verbose (bool, optional) – Print debugging output, by default False
debug_dag (bool, optional) – Print candidate and selected jobs (including their wildcards) while inferring DAG. This can help to debug unexpected DAG topology or errors, by default False
rerun_incomplete (bool, optional) – Re-run all jobs the output of which is recognized as incomplete, by default True
keep_incomplete (bool, optional) – TODO !!! This could let to undesired effects but it is needed for GROMACS continuation Do not remove incomplete output files by failed jobs, by default True.
keep_going (bool, optional) – Go on with independent jobs if a job fails, by default True

Returns:

The snakemake command string. It also will set self._snakemake_str_cmd to the command string value

Return type:

str

submit(only_build: bool = False, new_cluster_config=None, job_prefix=None) → str[source]#

Submit to the workstation the snake_executor_file

Parameters:

only_build (bool, optional) – Only create the file to submit to the Frontend but it will not be executed, by default False
job_prefix (new_cluster_config and) – only added for compatibility and readability. This allows the current signature used on: bindflow.orchestration.flow_builder.approach_flow() during submission In reality it will not be used at all for this class

Returns:

The output of the submit command or None.

Return type:

str

Raises:

RuntimeError – If snake_executor_file is not present. You must declare it during initialization

class bindflow.orchestration.generate_scheduler.Scheduler(cluster_config: dict, out_dir: PathLike | str | bytes = '.', prefix_name: str = '', snake_executor_file: str | None = None)[source]#

Abstract Base Class to build an Schedular

Class variables#

submit_commandstr: The command used for your scheduler to launch jobs
cancel_commandstr: Command used to cancel jobs
shebangstr: Used to build script and detect properly the environment E.g: #!/bin/bash, #!/bin/sh, … This will be used to make the snake_executor_file executable.

abstract __cluster_validation__()[source]#: Each scheduler should validate if the necessary options, as partition, CPUs, etc are in cluster_config.

__init__(cluster_config: dict, out_dir: PathLike | str | bytes = '.', prefix_name: str = '', snake_executor_file: str | None = None) → None[source]#

Constructor of the class

Parameters:

cluster_config (dict) – All the necessary information for the specific schedular
out_dir (PathLike, optional) – Where all files will be exported and executed, by default ‘.’
prefix_name (str, optional) – A prefix append to the jobs names for easy identification, by default ‘’
snake_executor_file (str, optional) – The name/path of the file that will be used for execution of the workflow, by default None

abstract build_snakemake(jobs: int)[source]#

Function to create the snakemake command

Parameters:: jobs (int) – Number of snakemake jobs. Passed to the flag –jobs

abstract submit(only_build: bool, new_cluster_config: dict, job_prefix: str)[source]#

Command to update and execute the snake_executor_file.

Check the example implementations:

bindflow.orchestration.generate_scheduler.SlurmScheduler.submit()

bindflow.orchestration.generate_scheduler.FrontEnd.submit()

Parameters:

only_build (bool, optional) – Only create the file to submit but it will not be executed, by default False
job_prefix (new_cluster_config and) – This makes it compatible with the current submission of bindflow.orchestration.flow_builder.approach_flow()

to_json(out_file: str = 'cluster.json')[source]#

Method to write all the attributes of the BaseCluster class to a JSON file

Parameters:: out_file (str, optional) – Name of the output JSON file, by default “cluster.config”.

class bindflow.orchestration.generate_scheduler.SlurmScheduler(cluster_config: dict, out_dir: PathLike | str | bytes = '.', prefix_name: str = '', snake_executor_file: str | None = None)[source]#

__cluster_validation__()[source]#: Each scheduler should validate if the necessary options, as partition, CPUs, etc are in cluster_config.

__init__(cluster_config: dict, out_dir: PathLike | str | bytes = '.', prefix_name: str = '', snake_executor_file: str | None = None) → None[source]#

Constructor of the class

Parameters:

cluster_config (dict) – All the necessary information for the specific schedular
out_dir (PathLike, optional) – Where all files will be exported and executed, by default ‘.’
prefix_name (str, optional) – A prefix append to the jobs names for easy identification, by default ‘’
snake_executor_file (str, optional) – The name/path of the file that will be used for execution of the workflow, by default None

build_snakemake(jobs: int = 100000, latency_wait: int = 360, verbose: bool = False, debug_dag: bool = False, rerun_incomplete: bool = True, keep_incomplete: bool = True, keep_going: bool = True) → str[source]#

Build the snakemake command TODO Consider to put it in the parent class

Parameters:

jobs (int, optional) – Use at most N CPU cluster/cloud jobs in parallel. For local execution this is an alias for –cores. Note: Set to ‘unlimited’ in case, this does not play a role. For cluster this is just a limitation. It is advise to provided a big number in order to do not wait for finishing of the jobs rather that launch all in the queue, by default 100000
latency_wait (int, optional) – Wait given seconds if an output file of a job is not present after the job finished. This helps if your filesystem suffers from latency, by default 120
verbose (bool, optional) – Print debugging output, by default False
debug_dag (bool, optional) – Print candidate and selected jobs (including their wildcards) while inferring DAG. This can help to debug unexpected DAG topology or errors, by default False
rerun_incomplete (bool, optional) – Re-run all jobs the output of which is recognized as incomplete, by default True
keep_incomplete (bool, optional) – TODO !!! This could let to undesired effects but it is needed for GROMACS continuation Do not remove incomplete output files by failed jobs, by default True.
keep_going (bool, optional) – Go on with independent jobs if a job fails, by default True

Returns:

The snakemake command string. It also will set self._snakemake_str_cmd to the command string value

Return type:

str

submit(only_build: bool = False, new_cluster_config: dict | None = None, job_prefix: str = '') → str[source]#

Submit to the cluster the snake_executor_file

Parameters:

only_build (bool, optional) – Only create the file to submit to the cluster but it will not be executed, by default False
new_cluster_config (dict, optional) – New definition of the cluster. It could be useful to run the snakemake command with different resources as the one used on the workflow. For example, if the cluster has two partition deflt and long with 2 and 5 days as maximum time, we could run in the long partition the snakemake job and only ask for 1 CPU and in deflt the computational expensive calculations. If nothing is provided, cluster_config (passed during initialization) will be used, by default None
job_prefix (bool, optional) – It will be added as {job_prefix}.RuleThemAll , by default False

Returns:

The output of the submit command or None.

Return type:

str

Raises:

RuntimeError – If snake_executor_file is not present. You must declare it during initialization

bindflow.orchestration.generate_scheduler.slurm_validation(cluster_config: dict) → dict[source]#

Validate the provided user slurm keywords

Parameters:

cluster_config (dict) – A dictionary with key[SBATCH keyword]: value[SBATCH value]

Returns:

Corrected dictionary. Keywords like: c or p are translated to cpu-per-task and partition respectively.

Return type:

dict

Raises:

ValueError – Invalid Slurm keywords
ValueError – It was not provided necessary Slurm keywords

Scheduler

Contents

Scheduler#

Class variables#