Skip to content

TACC Lonestar6

1. About

1.1 Compute Nodes

Lonestar6 is an HPC cluster operated by the Texas Advanced Computing Center (TACC). It comprises 560 compute nodes, each delivering up to 5 TFlops of peak performance and equipped with 256 GB of DRAM. The specifications of the compute node are listed below (the table is from Lonestar6 User Guide).

Specification Value
CPU 2x AMD EPYC 7763 64-Core Processor ("Milan")
Total cores per node 128 cores on two sockets (64 cores/socket)
Hardware threads per core 1 per core
Hardware threads per node 128 x 1 = 128
Clock rate 2.45 GHz (Boost up to 3.5 GHz)
RAM 256 GB (3200 MT/s) DDR4
Local storage 288 GB /tmp partition on a 288 GB SSD

1.2 Citizenship

Lonestar6 and also any other HPC systems are shared by all of the users. Therefore what you do on the systems does affect others. Exercise good citizenship to ensure that your activity does not adversely impact the system and other users. Any of the following operations but not limited to will lead warnings/access privileges revocations/project terminations/criminal prosecutions:

Prohibited Operations

  • Mining cryptocurrency and/or use block-chain technology for personal gain
  • Installing or using any protected software or data
  • Sharing user credentials
  • Infringing upon someone else's copyright
  • Circumventing login procedures to gain access where you are not allowed
  • Scanning or probing any information deliberately without prior authorization
  • Any usage for commercial purposes or personal gain
  • Modifying file access for the purpose of circumventing purge policies
  • Running jobs on the login nodes
  • Stressing the file systems
The last two operations are common reasons for a ticket. I have received a ticket in 2019 regarding file system stress.

Ticket No. 50733

Recent job(s) and filesystem related issues

  • TACC Hello, Over this past weekend the $WORK filesystem was under heavy stress. This was affected system wide. The system admins traced some of the issues back to jobs you submitted. While there were other factors involved some of your jobs raised some concerns. Can you please provide a brief summary of what you are attempting to run and how IO is being handled? A full directory path to your job script and log/error files would be helpful. Any additional information is helpful as well. Best,
  • ME Dear ###, I am sorry for this issue. Our job submitted to the system is a parallel CFD simulation case with OpenFOAM flow solver (10 nodes, 240 mpi threads allocated, will write data to disk every 10 minutes approximately). I noticed the heavy stress in the file system around 2:00 am this morning, because the simulation became very very slow (never happened before), the solver is always waiting for data writing. At that time, I just thought that might be the system issues. Because we did many parallel simulation before, and everything was OK. This morning, I killed the job (ID 2005083). The log/error files for this job can be found in '###.out' and '###.err'. The slurm job script is '###'. We are always following the usage policy of TACC. If you found the reason, please tell us in details since we can work this out together. Also please do let me know if you have any other questions. Sincerely, Dezhi Dai
  • ME What's more, in this case, since the mesh will be decomposed to 240 sub ones, the solver will write simulation results to the 240 sub folders at the same time. Hope this info helps for identifying.
  • ME Dear ###, I might found the reason. Our flow solver is based on OpenFOAM and no change to the IO modules. Here is the info about IO for parallel jobs in OpenFOAM (from 'https://openfoam.com/releases/openfoam-v1712/parallel.php'): OpenFOAM’s input and output (I/O) system has received an update that provides users with the choice of how to read and write OpenFOAM data files. All I/O is now redirected through an object called fileHandler. This file handler can be selected through a standard run-time selection model. This release ships with three file handlers which differ in how they handle parallel operation: uncollated : This is the normal behaviour where in parallel every processor writes its own processorXXX directory. masterUncollated : Special version of uncollated that performs all I/O on the master, and therefore does not require NFS (but is slower). collated : This writes all processors into a single file. This drastically reduce the number of files generated for parallel runs. For each output time a single field file is assembled, as opposed to all fields being written per processor, e.g. Looking at the $FOAM_CASE directory, instead of seeing directories processor0 … processorN, a single processors directory is written. The 'uncollated' scheme has been used in Job 2005083. Please do let me know if you have some new findings. Thanks, Dezhi Dai
  • TACC Dezhi, Thank you for providing all the information. What had occurred over the weekend was that a number of users across the different systems were all utilizing the /work filesystem. For the most part, the jobs on their own wouldn't cause quite the damage. However, with all these jobs running and utilizing /work there was many requests being made to the Meta Data Servers which caused a slowdown. Again, while not detrimental on their own all these different jobs together created the "perfect storm" which resulted in the filesystem to go down. However, your last comment about the uncollated scheme leads me to also think that added additional work on the filesystem that may not have been seen with another method. We advise users to not run out of /work but to utilize /scratch (https://portal.tacc.utexas.edu/user-guides/stampede2#overview-filesystems). I understand this is a link to the stampede2 user guide but the same information applies. If you are on a system that doesn't have a /scratch filesystem, such as maverick2, then if possible use /tmp. Essentially, use /scratch when possible as it avoids added stress to /work which is mounted across all tacc systems and prone to more users on it. Here are some additional tips about use of the filesystems: https://portal.tacc.utexas.edu/user-guides/stampede2#using-citizenship-filesystems Best,

Login Node Usages

  • The login nodes are for
    • Editing and managing files
    • Compiling code (small scale)
    • Submitting and tracking jobs
  • Do NOT
    • Run jobs
    • Perform intensive computational activities, even for post-processing

1.3 File Systems

The most common used directories are $HOME, $WORK and $SCRATCH. The $HOME and $WORK are for storage and keeping track of important items. Actual job activity, reading and writing to disk, should be offloaded to your resource's $SCRATCH file system.

Tip

To check the absolute paths of the $HOME, $WORK and $SCRATCH, run the following commands in the terminal

1
2
3
echo $HOME
echo $WORK
echo $SCRATCH
File System Quota Best Storage Practices Best Activities

$HOME
10 GB/200,000 files		 cron jobs
small scripts
environment settings		 compiling
editing

$WORK
1 TB/3,000,000 files		 store software installations
original datasets that can't be reproduced
job scripts and templates		 staging datasets

$SCRATCH
no quota		 Temporary Storage
I/O files
job files
temporary datasets		 all job I/O activity
see TACC's Scratch File System Purge Policy.

TACC's Scratch File System Purge Policy

The $SCRATCH file system, as its name indicates, is a temporary storage space. Files that have not been accessed in ten days are subject to purge. Deliberately modifying file access time (using any method, tool, or program) for the purpose of circumventing purge policies is prohibited.

The operating system updates a file's access time when that file is modified on a login or compute node. Reading or executing a file/script on a login node does not update the access time, but reading or executing on a compute node does update the access time. This approach helps us distinguish between routine management tasks (e.g. tar, scp) and production use. Use the command ls -ul to view access times.

2. Prepare for Build

Why using $WORK instead of $HOME?

Due to disk space and file count limitations in the $HOME directory, this guide uses the $WORK directory for building OpenFOAM. The $WORK directory offers higher quotas for both storage and the number of files, making it more suitable for large software builds.

2.1 Download OpenFOAM-v2312

The OpenFOAM packages are available at https://dl.openfoam.com. For OpenFOAM-v2312:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
cd $WORK

[ -d OpenFOAM ] || mkdir OpenFOAM
cd OpenFOAM

[ -f OpenFOAM-v2312.tgz ] || wget https://dl.openfoam.com/source/v2312/OpenFOAM-v2312.tgz
[ -d OpenFOAM-v2312 ] || tar -xzf OpenFOAM-v2312.tgz

[ -f ThirdParty-v2312.tgz ] || wget https://dl.openfoam.com/source/v2312/ThirdParty-v2312.tgz
[ -d ThirdParty-v2312 ] || tar -xzf ThirdParty-v2312.tgz

2.2 Update $WORK/OpenFOAM/OpenFOAM-v2312/etc/bashrc

Modify the default installation path in $WORK/OpenFOAM/OpenFOAM-v2312/etc/bashrc by changing the base directory from $HOME to $WORK. This ensures that all user-specific OpenFOAM files are stored under the $WORK directory.

Line No. From To
125 projectDir="$HOME/OpenFOAM/OpenFOAM-$WM_PROJECT_VERSION" projectDir="$WORK/OpenFOAM/OpenFOAM-$WM_PROJECT_VERSION"
175 $HOME/$WM_PROJECT/${USER:-user} $FOAM_USER_APPBIN $FOAM_USER_LIBBIN \ $WORK/$WM_PROJECT/${USER:-user} $FOAM_USER_APPBIN $FOAM_USER_LIBBIN \
183 export WM_PROJECT_USER_DIR="$HOME/$WM_PROJECT/${USER:-user}-$WM_PROJECT_VERSION" export WM_PROJECT_USER_DIR="$WORK/$WM_PROJECT/${USER:-user}-$WM_PROJECT_VERSION"

2.3 Update Relevant Configuration Files

The system uses the following compilers, libraries, and tools:

Program
Command to Load
 Note
GCC 11.2.0 module load gcc/11.2.0 GCC, the GNU Compiler Collection
Boost 1.86.0 module load boost/1.86.0 Boost is a set of libraries for the C++ programming language that provides support for tasks and structures such as linear algebra, pseudorandom number generation, multithreading, image processing, regular expressions, and unit testing.
SCOTCH 7.0.7 module load scotch/7.0.7-i32 SCOTCH is a set of programs and libraries which implement the static mapping and sparse matrix reordering algorithms developed within the SCOTCH project.
METIS 5.1.0.3 module load metis/5.1.0.3 METIS is a set of serial programs for partitioning graphs, partitioning finite element meshes, and producing fill reducing orderings for sparse matrices.
CMake 3.24.2 module load cmake/3.24.2 CMake is a free, cross-platform, software development tool for building applications via compiler-independent instructions.

Update the following configuration files to match the system's compiler and library settings:

File Line No. From To
$WM_PROJECT_DIR/etc/bashrc 99 #export WM_COMPILE_CONTROL="+strict" export WM_COMPILE_CONTROL="+openmp"
$WM_PROJECT_DIR/etc/bashrc 106 export WM_MPLIB=SYSTEMOPENMPI export WM_MPLIB=OPENMPI
$WM_PROJECT_DIR/etc/config.sh/scotch 38 SCOTCH_VERSION=scotch_6.1.0 SCOTCH_VERSION=scotch-system
$WM_PROJECT_DIR/etc/config.sh/scotch 39 export SCOTCH_ARCH_PATH=$WM_THIRD_PARTY_DIR/platforms/... export SCOTCH_ARCH_PATH=$TACC_SCOTCH_DIR
$WM_PROJECT_DIR/etc/config.sh/metis 36 METIS_VERSION=metis-5.1.0 METIS_VERSION=metis-system
$WM_PROJECT_DIR/etc/config.sh/metis 37 export METIS_ARCH_PATH=$WM_THIRD_PARTY_DIR/platforms/... export METIS_ARCH_PATH=$TACC_METIS_DIR
$WM_PROJECT_DIR/etc/config.sh/CGAL 46 boost_version=boost_1_74_0 boost_version=boost-system
$WM_PROJECT_DIR/etc/config.sh/CGAL 49 export BOOST_ARCH_PATH="$WM_THIRD_PARTY_DIR/platforms/..." export BOOST_ARCH_PATH=$TACC_BOOST_DIR

Append the following lines to $WM_PROJECT_DIR/etc/bashrc:

1
2
3
4
5
6
# Load system compilers, libraries, and tools
module load gcc/11.2.0
module load boost/1.86.0
module load scotch/7.0.7-i32
module load metis/5.1.0.3
module load cmake/3.24.2
Complete contents of the modified $WM_PROJECT_DIR/etc/bashrc file 
  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
#----------------------------------*-sh-*--------------------------------------
# =========                 |
# \\      /  F ield         | OpenFOAM: The Open Source CFD Toolbox
#  \\    /   O peration     |
#   \\  /    A nd           | www.openfoam.com
#    \\/     M anipulation  |
#------------------------------------------------------------------------------
#     Copyright (C) 2011-2016 OpenFOAM Foundation
#     Copyright (C) 2016-2023 OpenCFD Ltd.
#------------------------------------------------------------------------------
# License
#     This file is part of OpenFOAM, distributed under GPL-3.0-or-later.
#
# File
#     etc/bashrc
#
# Description
#     The OpenFOAM environment for POSIX shell (eg, bash,dash,zsh,...).
#     Source manually or from the ~/.bashrc or ~/.profile files.
#
#     Most settings can be overridden with a <prefs.sh> file:
#
#       - Base values (eg, from system admin):
#         -  $WM_PROJECT_DIR/etc/prefs.sh
#
#       - User or group values (first file found):
#         -  ~/.OpenFOAM/$FOAM_API/prefs.sh
#         -  ~/.OpenFOAM/prefs.sh
#         -  $WM_PROJECT_SITE/$FOAM_API/etc/prefs.sh
#         -  $WM_PROJECT_SITE/etc/prefs.sh
#
#     Some settings can also be overridden on the command-line when
#     sourcing this file. For example,
#
#       . /path/etc/bashrc WM_COMPILER=Clang WM_LABEL_SIZE=64
#
# Environment
#     FOAM_CONFIG_ETC
#     - alternative/additional location for OpenFOAM etc/ directory
#
#     FOAM_CONFIG_MODE (search mode for etc config files - see foamEtcFile)
#         - eg, FOAM_CONFIG_MODE="o" to only use OpenFOAM config files
#         - no influence on OpenFOAM applications, just the config files
#
#     FOAM_VERBOSE (set/unset)
#         - add extra verbosity when sourcing files
#
#     WM_PROJECT_SITE (optional directory)
#         - local site-specific directory, uses WM_PROJECT_DIR/site if unset
#
# Note
#     Changes made to this file will be lost with the next upgrade.
#
#------------------------------------------------------------------------------

# [WM_PROJECT_VERSION] - A human-readable version name
# A development version is often named 'com' - as in www.openfoam.com
export WM_PROJECT_VERSION=v2312

#------------------------------------------------------------------------------
# Configuration environment variables.
# Override with <prefs.sh> instead of editing here.

# [WM_COMPILER_TYPE] - Compiler location:
# = system | ThirdParty
export WM_COMPILER_TYPE=system

# [WM_COMPILER] - Compiler:
# = Gcc | Clang | Icc | Icx | Amd | Arm | Cray | Fujitsu | Nvidia |
#   Gcc<digits> | Clang<digits>
export WM_COMPILER=Icx

# [WM_PRECISION_OPTION] - Floating-point precision:
# = DP | SP | SPDP
export WM_PRECISION_OPTION=DP

# [WM_LABEL_SIZE] - Label size in bits:
# = 32 | 64
export WM_LABEL_SIZE=32

# [WM_COMPILE_OPTION] - Optimised(default), debug, profiling, other:
# = Opt | Dbg | Debug | Prof
export WM_COMPILE_OPTION=Opt

# [WM_COMPILE_CONTROL] - additional control for compilation/linking
#   +gold   : use gold linker
#   +link-ld: use   ld linker [clang]
#   +lld    : use  lld linker [clang]
#   +mold   : use mold linker [clang]
#   ~libz   : without libz compression
#   ~rpath  : without rpath handling [MacOS]
#   +openmp : with openmp
#   ~openmp : without openmp
#   +ccache : use ccache
#   +xcrun  : use xcrun and native compilers [MacOS]
#   +strict : more deprecation warnings (may generate *many* warnings)
#   ccache=... : ccache command (unquoted, single/double or <> quoted)
#   version=... : compiler suffix (eg, version=11 -> gcc-11)
export WM_COMPILE_CONTROL="+openmp"

# [WM_MPLIB] - MPI implementation:
# = SYSTEMOPENMPI | OPENMPI | SYSTEMMPI | MPI | MPICH | MPICH-GM |
#   HPMPI | CRAY-MPICH | FJMPI | QSMPI | SGIMPI | INTELMPI | USERMPI
# Specify SYSTEMOPENMPI1, SYSTEMOPENMPI2 for internal tracking (if desired)
# Can also use INTELMPI-xyz etc and define your own wmake rule
export WM_MPLIB=OPENMPI


#------------------------------------------------------------------------------
# (advanced / legacy)

# [WM_PROJECT] - This project is "OpenFOAM" - do not change
export WM_PROJECT=OpenFOAM

# [projectDir] - directory containing this OpenFOAM version.
# \- When this file is located as $WM_PROJECT_DIR/etc/bashrc, the next lines
#    should work when sourced by BASH or ZSH shells. If this however fails,
#    set one of the fallback values to an appropriate path.
#
#    This can be removed if an absolute path is provided for WM_PROJECT_DIR
#    later on in this file
# --
projectDir="${BASH_SOURCE:-${ZSH_NAME:+$0}}";
[ -n "$projectDir" ] && projectDir="$(\cd $(dirname $projectDir)/.. && \pwd -L)" ||\
projectDir="$WORK/OpenFOAM/OpenFOAM-$WM_PROJECT_VERSION"
# projectDir="/opt/openfoam/OpenFOAM-$WM_PROJECT_VERSION"
# projectDir="/usr/local/OpenFOAM/OpenFOAM-$WM_PROJECT_VERSION"
################################################################################
# Or optionally hard-coded (eg, with autoconfig)
# projectDir="@PROJECT_DIR@"
: # Safety statement (if the user removed all fallback values)

# [FOAM_SIGFPE] - Trap floating-point exceptions.
#               - overrides the 'trapFpe' controlDict entry
# = true | false
#export FOAM_SIGFPE=true

# [FOAM_SETNAN] - Initialize memory with NaN
#               - overrides the 'setNaN' controlDict entry
# = true | false
#export FOAM_SETNAN=false

# [FOAM_ABORT] - Treat exit() on FatalError as abort()
# = true | false
#export FOAM_ABORT=false

# [FOAM_CODE_TEMPLATES] - dynamicCode templates
# - unset: uses 'foamEtcFile -list codeTemplates/dynamicCode'
##export FOAM_CODE_TEMPLATES="$WM_PROJECT_DIR/etc/codeTemplates/dynamicCode"

# [FOAM_JOB_DIR] - location of jobControl
#                - unset: equivalent to ~/.OpenFOAM/jobControl
# export FOAM_JOB_DIR="$HOME/.OpenFOAM/jobControl"

# [WM_OSTYPE] - Operating System Type (set automatically)
# = POSIX | MSwindows
#export WM_OSTYPE=POSIX

# [WM_ARCH_OPTION] - compiling with -m32 option on 64-bit system
# = 32 | 64
#   * on a 64-bit OS this can be 32 or 64
#   * on a 32-bit OS this option is ignored (always 32-bit)
#export WM_ARCH_OPTION=64

# [FOAM_EXTRA_CFLAGS, FOAM_EXTRA_CXXFLAGS, FOAM_EXTRA_LDFLAGS]
# Additional compilation flags - do not inherit from the environment.
# Set after sourcing or via <prefs.sh> to avoid surprises.
unset FOAM_EXTRA_CFLAGS FOAM_EXTRA_CXXFLAGS FOAM_EXTRA_LDFLAGS

################################################################################
# NO (NORMAL) USER EDITING BELOW HERE

# Capture values of old directories to be cleaned from PATH, LD_LIBRARY_PATH
foamOldDirs="$WM_PROJECT_DIR $WM_THIRD_PARTY_DIR \
    $WORK/$WM_PROJECT/${USER:-user} $FOAM_USER_APPBIN $FOAM_USER_LIBBIN \
    $WM_PROJECT_SITE $FOAM_SITE_APPBIN $FOAM_SITE_LIBBIN \
    $FOAM_MODULE_APPBIN $FOAM_MODULE_LIBBIN"

# [WM_PROJECT_DIR] - Location of this OpenFOAM version
export WM_PROJECT_DIR="$projectDir"

# [WM_PROJECT_USER_DIR] - Location of user files
export WM_PROJECT_USER_DIR="$WORK/$WM_PROJECT/${USER:-user}-$WM_PROJECT_VERSION"

# [WM_PROJECT_SITE] - Location of site-specific (group) files
# Default (unset) implies WM_PROJECT_DIR/site
# Normally defined in calling environment


# Finalize setup of OpenFOAM environment
if [ -d "$WM_PROJECT_DIR" ]
then
    if [ -n "$FOAM_VERBOSE" ] && [ -n "$PS1" ]
    then
        echo "source $WM_PROJECT_DIR/etc/config.sh/setup" 1>&2
    fi
    . "$WM_PROJECT_DIR/etc/config.sh/setup" "$@"
else
    echo "Error: did not locate installation path for $WM_PROJECT-$WM_PROJECT_VERSION" 1>&2
    echo "No directory: $WM_PROJECT_DIR" 1>&2
fi

# Cleanup variables (done as final statement for a clean exit code)
unset foamOldDirs projectDir


# Load system compilers, libraries, and tools
module load gcc/11.2.0
module load boost/1.86.0
module load scotch/7.0.7-i32
module load metis/5.1.0.3
module load cmake/3.24.2

#------------------------------------------------------------------------------

This configuration compiles successfully, but the resulting applications encounter a Segmentation fault (core dumped) at runtime.

Program
Command to Load
 Note
ICC 24.1 module load intel/24.1 ICC, Intel® C++ Compiler Classic. Loading it will also automatically load GCC 13.2.0.
IMPI 21.12 module load impi/21.12 IMPI, Intel® MPI Library
SCOTCH 7.0.7 module load scotch/7.0.7-i32 SCOTCH is a set of programs and libraries which implement the static mapping and sparse matrix reordering algorithms developed within the SCOTCH project.
METIS 5.2.1.1 module load metis/5.2.1.1 METIS is a set of serial programs for partitioning graphs, partitioning finite element meshes, and producing fill reducing orderings for sparse matrices.
CMake 3.24.2 module load cmake/3.24.2 CMake is a free, cross-platform, software development tool for building applications via compiler-independent instructions.

Update the following configuration files to match the system's compiler and library settings:

File Line No. From To
$WM_PROJECT_DIR/etc/bashrc 71 export WM_COMPILER=Gcc export WM_COMPILER=Icx
$WM_PROJECT_DIR/etc/bashrc 99 #export WM_COMPILE_CONTROL="+strict" export WM_COMPILE_CONTROL="+openmp"
$WM_PROJECT_DIR/etc/bashrc 106 export WM_MPLIB=SYSTEMOPENMPI export WM_MPLIB=INTELMPI
$WM_PROJECT_DIR/etc/config.sh/scotch 38 SCOTCH_VERSION=scotch_6.1.0 SCOTCH_VERSION=scotch-system
$WM_PROJECT_DIR/etc/config.sh/metis 36 METIS_VERSION=metis-5.1.0 METIS_VERSION=metis-system

Add the following lines to $WM_PROJECT_DIR/etc/config.sh/settings after Line 220 (which is case "$WM_COMPILER_TYPE-$WM_COMPILER" in)

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
system-Icx)
    gmpDir="$archDir/${gmp_version}"
    mpfrDir="$archDir/${mpfr_version}"
    mpcDir="$archDir/${mpc_version}"
    gslDir="$archDir/${gsl_version}"    # used by waves2Foam
    _foamAddLibAuto "$gmpDir" && export GMP_ARCH_PATH="$gmpDir"
    _foamAddLibAuto "$mpfrDir" && export MPFR_ARCH_PATH="$mpfrDir"
    _foamAddLibAuto "$mpcDir"
    _foamAddLibAuto "$gslDir"
    ;;

Append the following lines to $WM_PROJECT_DIR/etc/bashrc:

1
2
3
4
5
6
7
8
# Load system compilers, libraries, and tools
module load intel/24.1
module load impi/21.12
module load scotch/7.0.7-i32
module load metis/5.2.1.1
module load cmake/3.24.2

export LD_PRELOAD="/scratch/projects/compilers/intel24.1/oneapi/mpi/2021.12/lib/libmpi.so${LD_PRELOAD:+:$LD_PRELOAD}"
Complete contents of the modified $WM_PROJECT_DIR/etc/bashrc file 
  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
#----------------------------------*-sh-*--------------------------------------
# =========                 |
# \\      /  F ield         | OpenFOAM: The Open Source CFD Toolbox
#  \\    /   O peration     |
#   \\  /    A nd           | www.openfoam.com
#    \\/     M anipulation  |
#------------------------------------------------------------------------------
#     Copyright (C) 2011-2016 OpenFOAM Foundation
#     Copyright (C) 2016-2023 OpenCFD Ltd.
#------------------------------------------------------------------------------
# License
#     This file is part of OpenFOAM, distributed under GPL-3.0-or-later.
#
# File
#     etc/bashrc
#
# Description
#     The OpenFOAM environment for POSIX shell (eg, bash,dash,zsh,...).
#     Source manually or from the ~/.bashrc or ~/.profile files.
#
#     Most settings can be overridden with a <prefs.sh> file:
#
#       - Base values (eg, from system admin):
#         -  $WM_PROJECT_DIR/etc/prefs.sh
#
#       - User or group values (first file found):
#         -  ~/.OpenFOAM/$FOAM_API/prefs.sh
#         -  ~/.OpenFOAM/prefs.sh
#         -  $WM_PROJECT_SITE/$FOAM_API/etc/prefs.sh
#         -  $WM_PROJECT_SITE/etc/prefs.sh
#
#     Some settings can also be overridden on the command-line when
#     sourcing this file. For example,
#
#       . /path/etc/bashrc WM_COMPILER=Clang WM_LABEL_SIZE=64
#
# Environment
#     FOAM_CONFIG_ETC
#     - alternative/additional location for OpenFOAM etc/ directory
#
#     FOAM_CONFIG_MODE (search mode for etc config files - see foamEtcFile)
#         - eg, FOAM_CONFIG_MODE="o" to only use OpenFOAM config files
#         - no influence on OpenFOAM applications, just the config files
#
#     FOAM_VERBOSE (set/unset)
#         - add extra verbosity when sourcing files
#
#     WM_PROJECT_SITE (optional directory)
#         - local site-specific directory, uses WM_PROJECT_DIR/site if unset
#
# Note
#     Changes made to this file will be lost with the next upgrade.
#
#------------------------------------------------------------------------------

# [WM_PROJECT_VERSION] - A human-readable version name
# A development version is often named 'com' - as in www.openfoam.com
export WM_PROJECT_VERSION=v2312

#------------------------------------------------------------------------------
# Configuration environment variables.
# Override with <prefs.sh> instead of editing here.

# [WM_COMPILER_TYPE] - Compiler location:
# = system | ThirdParty
export WM_COMPILER_TYPE=system

# [WM_COMPILER] - Compiler:
# = Gcc | Clang | Icc | Icx | Amd | Arm | Cray | Fujitsu | Nvidia |
#   Gcc<digits> | Clang<digits>
export WM_COMPILER=Icx

# [WM_PRECISION_OPTION] - Floating-point precision:
# = DP | SP | SPDP
export WM_PRECISION_OPTION=DP

# [WM_LABEL_SIZE] - Label size in bits:
# = 32 | 64
export WM_LABEL_SIZE=32

# [WM_COMPILE_OPTION] - Optimised(default), debug, profiling, other:
# = Opt | Dbg | Debug | Prof
export WM_COMPILE_OPTION=Opt

# [WM_COMPILE_CONTROL] - additional control for compilation/linking
#   +gold   : use gold linker
#   +link-ld: use   ld linker [clang]
#   +lld    : use  lld linker [clang]
#   +mold   : use mold linker [clang]
#   ~libz   : without libz compression
#   ~rpath  : without rpath handling [MacOS]
#   +openmp : with openmp
#   ~openmp : without openmp
#   +ccache : use ccache
#   +xcrun  : use xcrun and native compilers [MacOS]
#   +strict : more deprecation warnings (may generate *many* warnings)
#   ccache=... : ccache command (unquoted, single/double or <> quoted)
#   version=... : compiler suffix (eg, version=11 -> gcc-11)
export WM_COMPILE_CONTROL="+openmp"

# [WM_MPLIB] - MPI implementation:
# = SYSTEMOPENMPI | OPENMPI | SYSTEMMPI | MPI | MPICH | MPICH-GM |
#   HPMPI | CRAY-MPICH | FJMPI | QSMPI | SGIMPI | INTELMPI | USERMPI
# Specify SYSTEMOPENMPI1, SYSTEMOPENMPI2 for internal tracking (if desired)
# Can also use INTELMPI-xyz etc and define your own wmake rule
export WM_MPLIB=INTELMPI


#------------------------------------------------------------------------------
# (advanced / legacy)

# [WM_PROJECT] - This project is "OpenFOAM" - do not change
export WM_PROJECT=OpenFOAM

# [projectDir] - directory containing this OpenFOAM version.
# \- When this file is located as $WM_PROJECT_DIR/etc/bashrc, the next lines
#    should work when sourced by BASH or ZSH shells. If this however fails,
#    set one of the fallback values to an appropriate path.
#
#    This can be removed if an absolute path is provided for WM_PROJECT_DIR
#    later on in this file
# --
projectDir="${BASH_SOURCE:-${ZSH_NAME:+$0}}";
[ -n "$projectDir" ] && projectDir="$(\cd $(dirname $projectDir)/.. && \pwd -L)" ||\
projectDir="$WORK/OpenFOAM/OpenFOAM-$WM_PROJECT_VERSION"
# projectDir="/opt/openfoam/OpenFOAM-$WM_PROJECT_VERSION"
# projectDir="/usr/local/OpenFOAM/OpenFOAM-$WM_PROJECT_VERSION"
################################################################################
# Or optionally hard-coded (eg, with autoconfig)
# projectDir="@PROJECT_DIR@"
: # Safety statement (if the user removed all fallback values)

# [FOAM_SIGFPE] - Trap floating-point exceptions.
#               - overrides the 'trapFpe' controlDict entry
# = true | false
#export FOAM_SIGFPE=true

# [FOAM_SETNAN] - Initialize memory with NaN
#               - overrides the 'setNaN' controlDict entry
# = true | false
#export FOAM_SETNAN=false

# [FOAM_ABORT] - Treat exit() on FatalError as abort()
# = true | false
#export FOAM_ABORT=false

# [FOAM_CODE_TEMPLATES] - dynamicCode templates
# - unset: uses 'foamEtcFile -list codeTemplates/dynamicCode'
##export FOAM_CODE_TEMPLATES="$WM_PROJECT_DIR/etc/codeTemplates/dynamicCode"

# [FOAM_JOB_DIR] - location of jobControl
#                - unset: equivalent to ~/.OpenFOAM/jobControl
# export FOAM_JOB_DIR="$HOME/.OpenFOAM/jobControl"

# [WM_OSTYPE] - Operating System Type (set automatically)
# = POSIX | MSwindows
#export WM_OSTYPE=POSIX

# [WM_ARCH_OPTION] - compiling with -m32 option on 64-bit system
# = 32 | 64
#   * on a 64-bit OS this can be 32 or 64
#   * on a 32-bit OS this option is ignored (always 32-bit)
#export WM_ARCH_OPTION=64

# [FOAM_EXTRA_CFLAGS, FOAM_EXTRA_CXXFLAGS, FOAM_EXTRA_LDFLAGS]
# Additional compilation flags - do not inherit from the environment.
# Set after sourcing or via <prefs.sh> to avoid surprises.
unset FOAM_EXTRA_CFLAGS FOAM_EXTRA_CXXFLAGS FOAM_EXTRA_LDFLAGS

################################################################################
# NO (NORMAL) USER EDITING BELOW HERE

# Capture values of old directories to be cleaned from PATH, LD_LIBRARY_PATH
foamOldDirs="$WM_PROJECT_DIR $WM_THIRD_PARTY_DIR \
    $WORK/$WM_PROJECT/${USER:-user} $FOAM_USER_APPBIN $FOAM_USER_LIBBIN \
    $WM_PROJECT_SITE $FOAM_SITE_APPBIN $FOAM_SITE_LIBBIN \
    $FOAM_MODULE_APPBIN $FOAM_MODULE_LIBBIN"

# [WM_PROJECT_DIR] - Location of this OpenFOAM version
export WM_PROJECT_DIR="$projectDir"

# [WM_PROJECT_USER_DIR] - Location of user files
export WM_PROJECT_USER_DIR="$WORK/$WM_PROJECT/${USER:-user}-$WM_PROJECT_VERSION"

# [WM_PROJECT_SITE] - Location of site-specific (group) files
# Default (unset) implies WM_PROJECT_DIR/site
# Normally defined in calling environment


# Finalize setup of OpenFOAM environment
if [ -d "$WM_PROJECT_DIR" ]
then
    if [ -n "$FOAM_VERBOSE" ] && [ -n "$PS1" ]
    then
        echo "source $WM_PROJECT_DIR/etc/config.sh/setup" 1>&2
    fi
    . "$WM_PROJECT_DIR/etc/config.sh/setup" "$@"
else
    echo "Error: did not locate installation path for $WM_PROJECT-$WM_PROJECT_VERSION" 1>&2
    echo "No directory: $WM_PROJECT_DIR" 1>&2
fi

# Cleanup variables (done as final statement for a clean exit code)
unset foamOldDirs projectDir


# Load system compilers, libraries, and tools
module load intel/24.1
module load impi/21.12
module load scotch/7.0.7-i32
module load metis/5.2.1.1
module load cmake/3.24.2

export LD_PRELOAD="/opt/intel/compilers_and_libraries_2020.4.304/linux/mpi/intel64/lib/release/libmpi.so${LD_PRELOAD:+:$LD_PRELOAD}"

#------------------------------------------------------------------------------

This configuration compiles successfully, but the resulting applications encounter a Segmentation fault (core dumped) at runtime.

Program
Command to Load
 Note
GCC 11.2.0 module load gcc/11.2.0 GCC, the GNU Compiler Collection
Boost 1.86.0 module load boost/1.86.0 Boost is a set of libraries for the C++ programming language that provides support for tasks and structures such as linear algebra, pseudorandom number generation, multithreading, image processing, regular expressions, and unit testing.
MVAPICH2 2.3.7 module load mvapich2/2.3.7 A BSD-licensed implementation of the MPI standard developed by Ohio State University.
SCOTCH 7.0.7 module load scotch/7.0.7-i32 SCOTCH is a set of programs and libraries which implement the static mapping and sparse matrix reordering algorithms developed within the SCOTCH project.
METIS 5.2.1.1 module load metis/5.2.1.1 METIS is a set of serial programs for partitioning graphs, partitioning finite element meshes, and producing fill reducing orderings for sparse matrices.
CMake 3.24.2 module load cmake/3.24.2 CMake is a free, cross-platform, software development tool for building applications via compiler-independent instructions.

Update the following configuration files to match the system's compiler and library settings:

File Line No. From To
$WM_PROJECT_DIR/etc/bashrc 99 #export WM_COMPILE_CONTROL="+strict" export WM_COMPILE_CONTROL="+openmp"
$WM_PROJECT_DIR/etc/bashrc 106 export WM_MPLIB=SYSTEMOPENMPI export WM_MPLIB=mvapich2
$WM_PROJECT_DIR/etc/config.sh/scotch 38 SCOTCH_VERSION=scotch_6.1.0 SCOTCH_VERSION=scotch-system
$WM_PROJECT_DIR/etc/config.sh/scotch 39 export SCOTCH_ARCH_PATH=$TACC_SCOTCH_DIR export SCOTCH_ARCH_PATH=$TACC_SCOTCH_DIR
$WM_PROJECT_DIR/etc/config.sh/metis 36 METIS_VERSION=metis-5.1.0 METIS_VERSION=metis-system
$WM_PROJECT_DIR/etc/config.sh/metis 37 export METIS_ARCH_PATH=$TACC_METIS_DIR export METIS_ARCH_PATH=$TACC_METIS_DIR
$WM_PROJECT_DIR/etc/config.sh/CGAL 46 boost_version=boost-system boost_version=boost-system
$WM_PROJECT_DIR/etc/config.sh/CGAL 49 export BOOST_ARCH_PATH=$TACC_BOOST_DIR export BOOST_ARCH_PATH=$TACC_BOOST_DIR
$WM_PROJECT_DIR/etc/config.sh/mpi 325 export MPI_ARCH_PATH="$TACC_MPI_DIR" export MPI_ARCH_PATH="$TACC_MPI_DIR"
$WM_PROJECT_DIR/wmake/rules/General/Gcc/c 6 cc := gcc$(COMPILER_VERSION) cc := mpicc
$WM_PROJECT_DIR/wmake/rules/General/Gcc/c++ 9 CC := g++$(COMPILER_VERSION) -std=c++14 CC := mpicxx -std=c++14

Append the following lines to $WM_PROJECT_DIR/etc/bashrc:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
# Load system compilers, libraries, and tools
module load gcc/11.2.0
module load boost/1.86.0
module load mvapich2/2.3.7
module load scotch/7.0.7-i32
module load metis/5.2.1.1
module load cmake/3.24.2

export WM_CC=mpicc
export WM_CXX=mpicxx
Complete contents of the modified $WM_PROJECT_DIR/etc/bashrc file 
  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
#----------------------------------*-sh-*--------------------------------------
# =========                 |
# \\      /  F ield         | OpenFOAM: The Open Source CFD Toolbox
#  \\    /   O peration     |
#   \\  /    A nd           | www.openfoam.com
#    \\/     M anipulation  |
#------------------------------------------------------------------------------
#     Copyright (C) 2011-2016 OpenFOAM Foundation
#     Copyright (C) 2016-2023 OpenCFD Ltd.
#------------------------------------------------------------------------------
# License
#     This file is part of OpenFOAM, distributed under GPL-3.0-or-later.
#
# File
#     etc/bashrc
#
# Description
#     The OpenFOAM environment for POSIX shell (eg, bash,dash,zsh,...).
#     Source manually or from the ~/.bashrc or ~/.profile files.
#
#     Most settings can be overridden with a <prefs.sh> file:
#
#       - Base values (eg, from system admin):
#         -  $WM_PROJECT_DIR/etc/prefs.sh
#
#       - User or group values (first file found):
#         -  ~/.OpenFOAM/$FOAM_API/prefs.sh
#         -  ~/.OpenFOAM/prefs.sh
#         -  $WM_PROJECT_SITE/$FOAM_API/etc/prefs.sh
#         -  $WM_PROJECT_SITE/etc/prefs.sh
#
#     Some settings can also be overridden on the command-line when
#     sourcing this file. For example,
#
#       . /path/etc/bashrc WM_COMPILER=Clang WM_LABEL_SIZE=64
#
# Environment
#     FOAM_CONFIG_ETC
#     - alternative/additional location for OpenFOAM etc/ directory
#
#     FOAM_CONFIG_MODE (search mode for etc config files - see foamEtcFile)
#         - eg, FOAM_CONFIG_MODE="o" to only use OpenFOAM config files
#         - no influence on OpenFOAM applications, just the config files
#
#     FOAM_VERBOSE (set/unset)
#         - add extra verbosity when sourcing files
#
#     WM_PROJECT_SITE (optional directory)
#         - local site-specific directory, uses WM_PROJECT_DIR/site if unset
#
# Note
#     Changes made to this file will be lost with the next upgrade.
#
#------------------------------------------------------------------------------

# [WM_PROJECT_VERSION] - A human-readable version name
# A development version is often named 'com' - as in www.openfoam.com
export WM_PROJECT_VERSION=v2312

#------------------------------------------------------------------------------
# Configuration environment variables.
# Override with <prefs.sh> instead of editing here.

# [WM_COMPILER_TYPE] - Compiler location:
# = system | ThirdParty
export WM_COMPILER_TYPE=system

# [WM_COMPILER] - Compiler:
# = Gcc | Clang | Icc | Icx | Amd | Arm | Cray | Fujitsu | Nvidia |
#   Gcc<digits> | Clang<digits>
export WM_COMPILER=Icx

# [WM_PRECISION_OPTION] - Floating-point precision:
# = DP | SP | SPDP
export WM_PRECISION_OPTION=DP

# [WM_LABEL_SIZE] - Label size in bits:
# = 32 | 64
export WM_LABEL_SIZE=32

# [WM_COMPILE_OPTION] - Optimised(default), debug, profiling, other:
# = Opt | Dbg | Debug | Prof
export WM_COMPILE_OPTION=Opt

# [WM_COMPILE_CONTROL] - additional control for compilation/linking
#   +gold   : use gold linker
#   +link-ld: use   ld linker [clang]
#   +lld    : use  lld linker [clang]
#   +mold   : use mold linker [clang]
#   ~libz   : without libz compression
#   ~rpath  : without rpath handling [MacOS]
#   +openmp : with openmp
#   ~openmp : without openmp
#   +ccache : use ccache
#   +xcrun  : use xcrun and native compilers [MacOS]
#   +strict : more deprecation warnings (may generate *many* warnings)
#   ccache=... : ccache command (unquoted, single/double or <> quoted)
#   version=... : compiler suffix (eg, version=11 -> gcc-11)
export WM_COMPILE_CONTROL="+openmp"

# [WM_MPLIB] - MPI implementation:
# = SYSTEMOPENMPI | OPENMPI | SYSTEMMPI | MPI | MPICH | MPICH-GM |
#   HPMPI | CRAY-MPICH | FJMPI | QSMPI | SGIMPI | INTELMPI | USERMPI
# Specify SYSTEMOPENMPI1, SYSTEMOPENMPI2 for internal tracking (if desired)
# Can also use INTELMPI-xyz etc and define your own wmake rule
export WM_MPLIB=mvapich2


#------------------------------------------------------------------------------
# (advanced / legacy)

# [WM_PROJECT] - This project is "OpenFOAM" - do not change
export WM_PROJECT=OpenFOAM

# [projectDir] - directory containing this OpenFOAM version.
# \- When this file is located as $WM_PROJECT_DIR/etc/bashrc, the next lines
#    should work when sourced by BASH or ZSH shells. If this however fails,
#    set one of the fallback values to an appropriate path.
#
#    This can be removed if an absolute path is provided for WM_PROJECT_DIR
#    later on in this file
# --
projectDir="${BASH_SOURCE:-${ZSH_NAME:+$0}}";
[ -n "$projectDir" ] && projectDir="$(\cd $(dirname $projectDir)/.. && \pwd -L)" ||\
projectDir="$WORK/OpenFOAM/OpenFOAM-$WM_PROJECT_VERSION"
# projectDir="/opt/openfoam/OpenFOAM-$WM_PROJECT_VERSION"
# projectDir="/usr/local/OpenFOAM/OpenFOAM-$WM_PROJECT_VERSION"
################################################################################
# Or optionally hard-coded (eg, with autoconfig)
# projectDir="@PROJECT_DIR@"
: # Safety statement (if the user removed all fallback values)

# [FOAM_SIGFPE] - Trap floating-point exceptions.
#               - overrides the 'trapFpe' controlDict entry
# = true | false
#export FOAM_SIGFPE=true

# [FOAM_SETNAN] - Initialize memory with NaN
#               - overrides the 'setNaN' controlDict entry
# = true | false
#export FOAM_SETNAN=false

# [FOAM_ABORT] - Treat exit() on FatalError as abort()
# = true | false
#export FOAM_ABORT=false

# [FOAM_CODE_TEMPLATES] - dynamicCode templates
# - unset: uses 'foamEtcFile -list codeTemplates/dynamicCode'
##export FOAM_CODE_TEMPLATES="$WM_PROJECT_DIR/etc/codeTemplates/dynamicCode"

# [FOAM_JOB_DIR] - location of jobControl
#                - unset: equivalent to ~/.OpenFOAM/jobControl
# export FOAM_JOB_DIR="$HOME/.OpenFOAM/jobControl"

# [WM_OSTYPE] - Operating System Type (set automatically)
# = POSIX | MSwindows
#export WM_OSTYPE=POSIX

# [WM_ARCH_OPTION] - compiling with -m32 option on 64-bit system
# = 32 | 64
#   * on a 64-bit OS this can be 32 or 64
#   * on a 32-bit OS this option is ignored (always 32-bit)
#export WM_ARCH_OPTION=64

# [FOAM_EXTRA_CFLAGS, FOAM_EXTRA_CXXFLAGS, FOAM_EXTRA_LDFLAGS]
# Additional compilation flags - do not inherit from the environment.
# Set after sourcing or via <prefs.sh> to avoid surprises.
unset FOAM_EXTRA_CFLAGS FOAM_EXTRA_CXXFLAGS FOAM_EXTRA_LDFLAGS

################################################################################
# NO (NORMAL) USER EDITING BELOW HERE

# Capture values of old directories to be cleaned from PATH, LD_LIBRARY_PATH
foamOldDirs="$WM_PROJECT_DIR $WM_THIRD_PARTY_DIR \
    $WORK/$WM_PROJECT/${USER:-user} $FOAM_USER_APPBIN $FOAM_USER_LIBBIN \
    $WM_PROJECT_SITE $FOAM_SITE_APPBIN $FOAM_SITE_LIBBIN \
    $FOAM_MODULE_APPBIN $FOAM_MODULE_LIBBIN"

# [WM_PROJECT_DIR] - Location of this OpenFOAM version
export WM_PROJECT_DIR="$projectDir"

# [WM_PROJECT_USER_DIR] - Location of user files
export WM_PROJECT_USER_DIR="$WORK/$WM_PROJECT/${USER:-user}-$WM_PROJECT_VERSION"

# [WM_PROJECT_SITE] - Location of site-specific (group) files
# Default (unset) implies WM_PROJECT_DIR/site
# Normally defined in calling environment


# Finalize setup of OpenFOAM environment
if [ -d "$WM_PROJECT_DIR" ]
then
    if [ -n "$FOAM_VERBOSE" ] && [ -n "$PS1" ]
    then
        echo "source $WM_PROJECT_DIR/etc/config.sh/setup" 1>&2
    fi
    . "$WM_PROJECT_DIR/etc/config.sh/setup" "$@"
else
    echo "Error: did not locate installation path for $WM_PROJECT-$WM_PROJECT_VERSION" 1>&2
    echo "No directory: $WM_PROJECT_DIR" 1>&2
fi

# Cleanup variables (done as final statement for a clean exit code)
unset foamOldDirs projectDir


# Load system compilers, libraries, and tools
module load gcc/11.2.0
module load boost/1.86.0
module load mvapich2/2.3.7
module load scotch/7.0.7-i32
module load metis/5.2.1.1
module load cmake/3.24.2

export WM_CC=mpicc
export WM_CXX=mpicxx

#------------------------------------------------------------------------------

2.4 Download geometricVofExt

The geometricVofExt package is an unofficial extension to OpenFOAM for geometric Volume of Fluid (VOF) interface capturing. It provides a collection of utilities and solvers designed for accurate and conservative simulation of two-phase flows with sharp interface resolution on general unstructured meshes.

Get geometricVofExt from an online reposity: 

1
2
3
4
5
6
7
8
9
source $WORK/OpenFOAM/OpenFOAM-v2312/etc/bashrc

[ -d $WM_PROJECT_USER_DIR ] || mkdir $WM_PROJECT_USER_DIR
cd $WM_PROJECT_USER_DIR

[ -d modules ] || mkdir modules
cd modules

git clone https://github.com/daidezhi/geometricVofExt.git

3. Build

Start an idev (Interactive DEVelopment) session before building

  • Do NOT build OpenFOAM on the login nodes. These nodes have a strict process limit (maximum of 300), which is insufficient for compiling large-scale software.
  • Use the idev utility to request an interactive session on a compute node. This provides the necessary resources for building and running MPI-based applications like OpenFOAM.
  • To start an interactive session, run: 
    1
    idev -N 1 -n 16 -p vm-small -t 48:00:00
    This command allocates 16 cores in a compute node in the vm-small queue for 48 hours. vm-small is designed for users who only need a subset of a node's entire 128 cores in the normal queue.

Compile OpenMPI 4.1.2 if the system MPI libraries are not used

1
2
3
4
5
source $WORK/OpenFOAM/OpenFOAM-v2312/etc/bashrc

cd $WM_THIRD_PARTY_DIR

./makeOPENMPI

3.1 Compile CGAL with GMP and MPFR Using System GCC

The GNU Multiple Precision Arithmetic (GMP) and GNU Multiple Precision Floating-Point Reliably (MPFR) Libraries are highly recommended for use with CGAL. Otherwise, the setVofField and setShapedFields utilities will operate in a low-efficiency mode.

Library Note
GMP-6.2.1 GMP: GNU Multiple Precision Arithmetic Library
MPFR-4.1.0 MPFR: GNU Multiple Precision Floating-Point Reliably Library
MPC-1.2.1 MPC: GNU Multiple-Precision C Library

Build CGAL-4.14.3 with the following commands:

1
2
3
4
5
source $WORK/OpenFOAM/OpenFOAM-v2312/etc/bashrc

cd $WM_PROJECT_USER_DIR/modules/geometricVofExt

./compileCGALWithGMPAndMPFRUsingSystemGCC

3.2 Build ccmio 2.6.1

ccmio 2.6.1 is a C++ I/O library developed by CD-adapco (now part of Siemens) for reading and writing files in the CCM+ format used by STAR-CCM+. It provides an API to access mesh, solution data, and metadata from *.ccm files.

1
2
3
4
5
6
7
8
9
source $WORK/OpenFOAM/OpenFOAM-v2312/etc/bashrc

cd $WM_THIRD_PARTY_DIR

[ -f libccmio-2.6.1.tar.gz ] || wget -nv https://sourceforge.net/projects/foam-extend/files/ThirdParty/libccmio-2.6.1.tar.gz --no-check-certificate

[ -d libccmio-2.6.1 ] || tar -xzf libccmio-2.6.1.tar.gz

./makeCCMIO lib

3.3 Build OpenFOAM-v2312

1
2
3
4
5
6
7
source $WORK/OpenFOAM/OpenFOAM-v2312/etc/bashrc

cd $WM_PROJECT_DIR

./Allwmake -j16 -k

./Allwmake -j16 -k

3.4 Enable Initial Time Sampling (optional)

1
2
3
4
5
source $WORK/OpenFOAM/OpenFOAM-v2312/etc/bashrc

cd $WM_PROJECT_USER_DIR/modules/geometricVofExt

./enableInitialTimeWriting

3.5 Build geometricVofExt

1
2
3
4
5
6
7
8
9
source $WORK/OpenFOAM/OpenFOAM-v2312/etc/bashrc

cd $WM_PROJECT_USER_DIR/modules/geometricVofExt

./Allwmake -j16

mkdir -p $WM_PROJECT_USER_DIR/applications/utilities

./compileWaves2FoamInOpenFOAM2312