Enjoy the Colorful Fluid Dynamics!

A Slurm script generator for running OpenFOAM jobs in HPC clusters

OpenFOAM OpenFOAM 10/06/2018

This article was published 591 days ago, and the content may be outdated.

Introduction

The Slurm (aka Simple Linux Utility for Resource Management or SLURM) job scheduler is employed by many of the HPC clusters. It is annoying when modifying the Slurm script for OpenFOAM parallel cases. Then, a Slurm script generator based on the Bash language has been developed.

Slurm Script Generator

The Bash commands shown below will generate a Slurm job script which requests a parallel job with 32 threads spread over 2 node and 12 hours. Some individual items, e.g., email, flow_solver or proj, need to be modified before running the generator.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
#!/bin/bash

#--------------------------------------------------------------------
# Slurm script generator for running OpenFOAM cases on HPC clusters.
#
# Note:
#   Please put this file in the root directory of the case.

# Author:
#   Dezhi Dai (dezhi.dai@mavs.uta.edu), MAE Department, UTA.
#--------------------------------------------------------------------

#-Modify this part if needed-----------------------------------------
# Set the email, replace this by your own email address
email="xxx@xxx"
# Set the flow solver
flow_solver="xxxFoam"
# Set the No. of nodes requested and total mpi tasks
n_nodes="2"
n_mpis="32"
# Set the project to charge
proj="xxx"
# Set request time
time_request="12:00:00"
#--------------------------------------------------------------------

# Obtain current case directory and rootcase name
case_dir=`pwd` case_name=`basename "$case_dir"`
# Set the file name of the SLURM job script
s_name="auto_slurm_job"

# Write options and commands to the Slurm script
# Header
echo "#!/bin/bash" > $s_name

# Slurm options
echo "#SBATCH -J $case_name" >> $s_name
echo "#SBATCH -o $case_name.out" >> $s_name
echo "#SBATCH -e $case_name.err" >> $s_name
echo "#SBATCH -N $n_nodes -n $n_mpis" >> $s_name
echo "#SBATCH -t $time_request" >> $s_name
echo "#SBATCH -A $proj" >> $s_name
echo "#SBATCH --mail-user=$email" >> $s_name
echo "#SBATCH --mail-type=all" >> $s_name

# OpenFOAM commands
echo "decomposePar -case $case_dir -fileHandler collated" >> $s_name
echo "mpirun -np $n_mpis $flow_solver -parallel -case $case_dir -fileHandler collated" >> $s_name
#echo "reconstructPar -case $case_dir -fileHandler collated" >> $s_name
echo "touch $case_dir/$case_name.foam" >> $s_name

It should be noted that the reconstructPar utility only support a serial job, which means it will waste your Service Unit budget if you run it in a parallel request.

Usage

The usage of the generator is simple. Save the commands as xxxx (any name you like) and run the following commands in the terminal:

1
2
$ bash xxxx
$ sbatch auto_slurm_job
avatar

  1. 1. Introduction
  2. 2. Slurm Script Generator
  3. 3. Usage