A Permanent Head Damage Student in Colorful Fluid Dynamics

Install OpenFOAM v1806 on TACC Lonestar 5 with Intel Compiler

Source code preparation

Download and unpack the source code to $WORK/OpenFOAM. TACC’s $WORK filesystem is used.

1
2
3
4
5
6
7
8
cdw
mkdir OpenFOAM
cd OpenFOAM
wget -O OpenFOAM-v1806.tgz https://sourceforge.net/projects/openfoamplus/files/v1806/OpenFOAM-v1806.tgz/download --no-check-certificate
wget -O ThirdParty-v1806.tgz https://sourceforge.net/projects/openfoamplus/files/v1806/ThirdParty-v1806.tgz/download --no-check-certificate
tar -xzf OpenFOAM-v1806.tgz
tar -xzf ThirdParty-v1806.tgz
rm -rf *.tgz

Configuration

Before compiling OpenFOAM, some file modifications are needed to specify the working directory, compiler, MPI library, optimization flags, and CGAL library building options.

The version v1806 needs the latest Intel compiler, the default one may return errors when compiling return static_cast<const searcher&>(finder); in \src\OpenFOAM\db\dictionary\dictionarySearch.C, which is a new source file in OpenFOAM :

1
module load intel/18.0.2

The configuration files which need modifications are listed below:

File Description
1 /OpenFOAM/OpenFOAM-v1806/etc/bashrc specify installation directory, compiler and MPI library name
2 /OpenFOAM/OpenFOAM-v1806/etc/config.sh/mpi specify the system MPI library
3 /OpenFOAM/OpenFOAM-v1806/etc/config.sh/settings specify the MPI parallel compiler and optimization flags
4 /OpenFOAM/OpenFOAM-v1806/wmake/rules/linux64Icc/c specify the compiling command with a parallel one
5 /OpenFOAM/OpenFOAM-v1806/wmake/rules/linux64Icc/c++ specify the compiling command with a parallel one
6 /OpenFOAM/ThirdParty-v1806/makeCGAL add support of intel toolset when building CGAL library

/OpenFOAM/OpenFOAM-v1806/etc/bashrc

line 43
from

1
FOAM_INST_DIR=$HOME/$WM_PROJECT

to

1
FOAM_INST_DIR=$WORK/$WM_PROJECT

line 62
from

1
export WM_COMPILER=Gcc

to

1
export WM_COMPILER=Icc

line 85
from

1
export WM_MPLIB=SYSTEMOPENMPI

to

1
export WM_MPLIB=MYMPI

line 109
from

1
$HOME/$WM_PROJECT/$USER $FOAM_USER_APPBIN $FOAM_USER_LIBBIN \

to

1
$WORK/$WM_PROJECT/$USER $FOAM_USER_APPBIN $FOAM_USER_LIBBIN \

line 127
from

1
export WM_PROJECT_USER_DIR=$HOME/$WM_PROJECT/$USER-$WM_PROJECT_VERSION

to

1
export WM_PROJECT_USER_DIR=$WORK/$WM_PROJECT/$USER-$WM_PROJECT_VERSION

/OpenFOAM/OpenFOAM-v1806/etc/config.sh/mpi

Add

1
2
3
4
MYMPI)
export FOAM_MPI=mpi
export MPI_ARCH_PATH="${MPICH_HOME}"
;;

after line 32: case "$WM_MPLIB" in


/OpenFOAM/OpenFOAM-v1806/etc/config.sh/settings

lines 24-25
from

1
2
export WM_CC="gcc"
export WM_CXX="g++"

to

1
2
export WM_CC="mpicc"
export WM_CXX="mpicxx"

lines 52-54
from

1
2
3
export WM_CFLAGS="$WM_CFLAGS -m64"
export WM_CXXFLAGS="$WM_CXXFLAGS -m64"
export WM_LDFLAGS="-m64"

to

1
2
3
export WM_CFLAGS="$WM_CFLAGS -O2"
export WM_CXXFLAGS="$WM_CXXFLAGS -O2"
export WM_LDFLAGS="-O2"

/OpenFOAM/OpenFOAM-v1806/wmake/rules/linux64Icc/c

line 3
from

1
cc          = icc

to

1
cc          = mpicc

/OpenFOAM/OpenFOAM-v1806/wmake/rules/linux64Icc/c++

line 3
from

1
CC          = icpc -std=c++11 -fp-trap=common -fp-model precise

to

1
CC          = mpicxx -std=c++11 -fp-trap=common -fp-model precise

/OpenFOAM/ThirdParty-v1806/makeCGAL

lines 235-241
from

1
2
3
4
5
6
7
./bootstrap.sh \
--prefix=$BOOST_ARCH_PATH \
--libdir=$boostLib \
--with-libraries=thread \
--with-libraries=system \
&amp;&amp; ./bjam toolset=$WM_CC -j $WM_NCOMPPROCS install \
&amp;&amp; echo "Built: boost"

to

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
case $WM_COMPILER in
Gcc*)
TOOLSET=gcc
;;
Icc)
TOOLSET=intel
;;
Clang)
TOOLSET=clang
;;
esac

./bootstrap.sh \
--prefix=$BOOST_ARCH_PATH \
--libdir=$boostLib \
--with-libraries=thread \
--with-libraries=system \
&amp;&amp; ./bjam toolset=$TOOLSET -j $WM_NCOMPPROCS install \
&amp;&amp; echo "Built: boost"

Environment check

Add

1
2
3
module load intel/18.0.2

source $WORK/OpenFOAM/OpenFOAM-v1806/etc/bashrc

to the end of user .bashrc file and reinitialize it by:

1
2
cdh
. ~/.bashrc

Build CGAL package

Compile CGAL package provided in ThirdParty:

1
2
3
4
cdw
cd $WM_THIRD_PARTY_DIR
module load cmake
./makeCGAL

Once the terminal returns Done CGAL, go next step.


Build OpenFOAM

Compile OpenFOAM and it may take 6-8 hours:

1
2
3
4
5
6
7
cdh
. ~/.bashrc
cdw
foam
module load cmake
wmSet $FOAM_SETTINGS
./Allwmake

Finally, the terminal should return:



Validate the build by running:

1
foamInstallationTest

and the returning message should be:




This post is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International (CC BY-NC-SA 4.0) License.

OpenFOAM Job Submission on TACC Lonestar 5

  1. 1. Source code preparation
  2. 2. Configuration
    1. 2.1. /OpenFOAM/OpenFOAM-v1806/etc/bashrc
    2. 2.2. /OpenFOAM/OpenFOAM-v1806/etc/config.sh/mpi
    3. 2.3. /OpenFOAM/OpenFOAM-v1806/etc/config.sh/settings
    4. 2.4. /OpenFOAM/OpenFOAM-v1806/wmake/rules/linux64Icc/c
    5. 2.5. /OpenFOAM/OpenFOAM-v1806/wmake/rules/linux64Icc/c++
    6. 2.6. /OpenFOAM/ThirdParty-v1806/makeCGAL
  3. 3. Environment check
  4. 4. Build CGAL package
  5. 5. Build OpenFOAM