Ensure the log and results folder are there before snakemake rules are executed

For now the way the log_directories and cluster_config are created with the bash commands.

Create log directories

os.makedirs( os.path.join( os.getcwd(), config['log_dir'], ), exist_ok=True)

if cluster_config: os.makedirs( os.path.join( os.getcwd(), os.path.dirname(cluster_config['default']['out']), ), exist_ok=True)

Perhaps we can consider using an extra labeling : on_start, which ensures that the log_folders and results are there before any rule is executed. This is not urgent so we can deal with it later.

added Future label

changed title from Ensure the log and results folder is there before snakemake rules are executed to Ensure the log and results folder are there before snakemake rules are executed

removed Future label

changed milestone to %pipeline

assigned to @katsanto

on_start: option to create directories

Here is the documentation: https://snakemake.readthedocs.io/en/stable/snakefiles/rules.html#onstart-onsuccess-and-onerror-handlers

onstart, onsuccess and onerror are the options

added upstream label

added To Do label

changed milestone to %upstream

added Doing label and removed To Do label

removed upstream label

added pipeline label

snakemake doc: https://snakemake.readthedocs.io/en/v6.0.5/snakefiles/configuration.html?highlight=cluster%20config#cluster-configuration-deprecated

"Note that a workflow shall never depend on a cluster configuration, because this would limit its portability. Therefore, it is also not intended to access the cluster configuration from within the workflow." Therefore the cluster log folders should not be created within the snakefile. This should be fixed once we replace the cluster configs with profiles as suggested (see #153 (closed))

unassigned @katsanto

added Blocked label

Blocked by #153 (closed)

added To Do label and removed Doing label

removed Blocked label

This is actually not related to Snakemake profiles; profiles are just another mechanism to encapsulate all the command line arguments, cluster submission options, jobs status queries etc and provide this all in one directory called profile. Parameters are provided in YAML and JSON plus there are some scripts that enable snakemake communication with the cluster.
However, this is all separate from how, where and when we create directories used in the workflow.

mentioned in merge request !97 (merged)

See my comment in merge request 97

@katsanto : I think our recent changes in !97 (merged) also solve this issue and it can be closed <?>

Fixed in !97 (merged)

closed