Skip to content

Nextflow outputs

Nextflow creates a least 2 directories: .nexflow and work in the current directory. You can change the working directory with the -work-dir option. Depending on the workflow you run, usually you also have a results directory.

$ ls -la
drwxr-xr-x  3 bleuet BIOINFO  4096 23 dec.  12:11 .nextflow        => internal files
-rw-r--r--  1 bleuet BIOINFO   108 23 dec.  12:07 nextflow.config  => config file
-rw-r--r--  1 bleuet BIOINFO 10962 23 dec.  12:11 .nextflow.log    => log file with all intermediate directories
drwxr-xr-x  2 bleuet BIOINFO  4096 23 dec.  12:11 results          => final results
drwxr-xr-x  8 bleuet BIOINFO  4096 23 dec.  12:10 work             => working and temporary files

work directory

Each job execution has a sub-directory. If a job fails, you can go into the directory and check the executed command. We can have a look to the directory corresponding to the failed job and have access to the log and command file.

$ ls -la work/40/944143ebbcc45aa0e4bf2f8ba9dab6/
total 4
drwxr-xr-x 3 bleuet BIOINFO 4096 24 mars  10:00 ..
-rw-r--r-- 1 bleuet BIOINFO 2514 24 mars  10:00 .command.run   
-rw-r--r-- 1 bleuet BIOINFO   36 24 mars  10:00 .command.sh    => the command to execute
-rw-r--r-- 1 bleuet BIOINFO    0 24 mars  10:00 .command.begin
-rw-r--r-- 1 bleuet BIOINFO    0 24 mars  10:00 .command.err   => error file
-rw-r--r-- 1 bleuet BIOINFO   13 24 mars  10:00 .command.log
-rw-r--r-- 1 bleuet BIOINFO   13 24 mars  10:00 .command.out   => output file
drwxr-xr-x 2 bleuet BIOINFO 4096 24 mars  10:00 .
-rw-r--r-- 1 bleuet BIOINFO    1 24 mars  10:00 .exitcode

See the log file to get the execution error .command.log.

The executed command is in file .command.sh.

Tips

You can find in the FAQ "How to find reason of an error? " how to retrieve the directory associated with a process.

results directory

Depending on the executed workflow the results files will be organized differently in results directory. For the moment none of the executed workflow has generate a result directory.

Manage disk space

Large workflows can generate plenty of tempory file.

Check the disk space used in current directory : 

du -sh *

The command nextflow clean provide the best way to save space after running a pipeline.

Dive into outputs

  • Find the last run of tutorial.nf with nextflow log
  • Find the working directory of the second execution of CONVERTTOUPPER
  • Go into this work directory.
  • Print content of all files .command.sh .command.log .command.out
  • Which word is in out file ?
Solution
  • nextflow log
  • identify the line corresponding to nextflow run tutorial.nf ... and retrieve the run name
  • launch nextflow log -f task_id,hash,name,status,exit,duration,realtime,pcpu,pmem RUN_NAME
  • find the hash corresponding to the line `CONVERTTOUPPER (2)̀
  • cd work/c2/8270e1 + tabulation to complete directory from hash
  • cd work/c2/8270e1905ee5ef64ef95178b7313ef/
  • more .command.sh
  • more .command.log
  • more .command.out
  • more .command.sh
  • the word is "WORLD!"

Clean outputs

  • Go back to nextflow_tutorial directory
  • Which nextflow command can clean directory ? Get help on this command
  • Perform a dry run of nextflow clean, does the work directory you entered into would be delete ?
  • Execute nextflow clean without dry run
Solution
  • cd ../../..
  • nextflow help clean
  • nextflow clean -n, yes
  • nextflow clean -f