Setting up DL_MESO

Summary

These practical exercises are intended to introduce you to some of the capabilities of Dissipative Particle Dynamics (DPD) and the Lattice Boltzmann Equation (LBE) method. They will involve the use of the CCP5 simulation code DL_MESO. Before undertaking the exercises, you will need to compile DL_MESO.

Background

The wide range of applications for DPD and LBE has led to development of software to carry out simulations. The CCP5 simulation code suite DL_MESO [Seaton2013] is one such piece of software, consisting of simulation codes that can apply different mesoscale modelling methods, including LBE (DL_MESO_LBE) and DPD (DL_MESO_DPD). These codes can be run on many different computers, including large supercomputers and clusters as well as standalone desktops and laptops.

DL_MESO is available from DL_MESO’s website free of charge from UKRI STFC Daresbury Laboratory under licence for academics and researchers. (A licence is also available for commercially-funded research in exchange for an annual fee.) It is supplied as source code, allowing more advanced users to make modifications to implement their own functionalities without needing to write codes from scratch.

The package comes with its own graphical user interface (GUI) written in Java, which is rather basic but can help new users to set up DL_MESO simulations. It is particularly helpful in setting up boundary conditions for the LBE code.

Requirements

To undertake these exercises you will need a PC running either MS Windows, a flavour of Unix/Linux or macOS (Mac OS X). The main requirement is availability of compilers that can compile the source code into executables:

  • a Fortran compiler for the DPD code and most of its associated utilities
  • a C++ compiler for the LBE code, its utilities and two DPD utilities

Several options exist for all operating systems: the GNU Compiler Collection (GCC) compilers are generally a good set to use and are available for all operating systems. The program GNU Make is also required to automate the compilation process by using makefiles to compile individual code modules and link them together.

It is possible to speed up calculations by splitting them across the available threads in a processor: in DL_MESO this is enabled using a standard called OpenMP, which is available in recent versions of most Fortran compilers by means of a compiler flag. [1] If you have access to multiple processor cores, you can split the calculation among the cores using MPI: to compile the parallel form of DL_MESO that uses MPI, an implementation of MPI (e.g. OpenMPI, MPICH2) needs to be installed to provide ‘compiler wrappers’ to automatically link codes to MPI libraries.

The Java GUI requires a Java Runtime Environment (JRE), which is normally already installed on most desktop and laptop computers. If you need to compile the GUI’s source code, you will also need a Java Development Kit (JDK) for the relevant version of Java (which includes the compiler javac).

While not part of the package itself, the DPD exercises can make use of some specially written Python3 scripts to carry out a simulation workflow and plot results. These will require a recent version of Python3 (not Python2, although the two versions of Python can exist side-by-side) and the graphical user interface toolkit Qt 5.

For Unix/Linux distros, many of the above requirements are either already pre-installed (e.g. GNU Make) or can be downloaded and installed using a software package manager. For computers with macOS, we recommend downloading and installing a package manager, e.g. MacPorts or HomeBrew, which can be used to download and install any of the above required components.

Windows 10 users can use the Windows Linux Subsystem, install any missing requirements using a package manager and follow the instructions for Unix/Linux. Alternatively, the required components will need to be downloaded and installed individually - here are some useful links:

Compiling DL_MESO

The source code is obtained by registering using the webform available on the DL_MESO website and following the instructions contained in the email you subsequently receive to download the package. You will need to unpack the software and compile the executable programs for the DPD code itself and its various utilities.

Open a terminal (command line) window and go to the directory where you have downloaded the DL_MESO package and/or intend to unpack it. Type the following command:

unzip dl_meso_2.7.zip

which will unpack the software once you have entered the decryption password supplied in the email. Now go into the directory you have just unpacked:

cd dl_meso

You will observe that the main DL_MESO directory contains several sub-directories:

  1. The DEMO directory, which contains input files and sample outputs for various test cases;
  2. The DPD directory, which contains the source code for the DPD code and its associated utilities (as well as makefiles to help compile them);
  3. The JAVA directory, which contains the source code for the Java GUI;
  4. The LBE directory, which contains the source code for the LBE code and its associated utilities;
  5. The MAN directory, which contains a PDF copy of the user manual;
  6. The WORK directory, which is a sample working directory and contains a makefile for compiling the utilities and a script to run the Java GUI.

Compiling DL_MESO_DPD

To compile DL_MESO_DPD, go into the WORK directory and copy one of the makefiles from the DPD/makefiles directory, renaming it Makefile e.g.

copy ..\DPD\makefiles\Makefile-serial Makefile
cp ../DPD/makefiles/Makefile-serial ./Makefile

(The first command is for Windows machines, the second is for Unix/Linux and macOS.) There are several makefiles available, of which the most useful are:

  1. Makefile-serial for serial running without any parallelisation;
  2. Makefile-OMP for serial running with OpenMP multithreading;
  3. Makefile-MPI for parallel running with MPI;
  4. Makefile-MPIOMP for parallel running with MPI and OpenMP multithreading.

These have been created to compile and link together the various modules in DL_MESO_DPD: different versions of certain modules exist to enable/disable MPI parallelism or OpenMP multithreading and can be slotted together appropriately. The supplied makefiles assume GCC compilers - gfortran - and a standard implementation of MPI with mpifort as the ‘wrapped’ compiler command: if you are using anything else, you will need to modify references to these compilers/wrappers in the makefile. (Hint: older implementations of MPI might use the wrapped compiler command mpif90 instead of mpifort.)

To compile DL_MESO_DPD, invoke the makefile:

make

If the process is successful, an executable called dpd.exe will be created in the WORK directory. You can safely remove any module or object files by typing one of the following commands (the first for Windows, the second for Unix/Linux or macOS):

del *.mod *.o
rm *.mod *.o

Compiling DL_MESO_LBE

To compile DL_MESO_LBE, go into the WORK directory and type one of the two following two commands:

c++ -O3 -o lbe.exe ../LBE/slbe.cpp
mpicxx -O3 -o lbe.exe ../LBE/plbe.cpp

The first command will compile the single-core (serial) version of DL_MESO_LBE with slbe.cpp as the main source file, while the latter will compile the multicore (parallel) version that uses MPI with plbe.cpp as the main source file. The compiler flag -O3 applies a reasonable amount of optimisation to the code, while -o specifies the name of the executable (lbe.exe). The compiler commands c++ and mpicxx should be substituted with those required for the installed compiler and MPI implementation, e.g. g++ is the command for the GCC C++ compiler.

To speed up the calculation by using all available threads, a flag can be included in the above commands to invoke OpenMP and make use of directives given in DL_MESO_LBE’s source code. For the GCC C++ compiler (and its MPI-wrapped equivalent), this flag is -fopenmp and can be inserted before the reference to the source code file.

Compiling utilities

To compile the utilities, a makefile (Makefile-utils) is supplied in the WORK directory - again this makefile assumes use of GCC compilers (gfortran and g++ for Fortran and C++), so you may wish to edit it if you are using other compilers. This makefile can be invoked using the command:

make -f Makefile-utils

Once you have compiled the utilities, you can safely remove all of the module and object files used to compile and link the programs.

Compiling Java GUI

A pre-compiled Java package file for the GUI, GUI.jar, is supplied with DL_MESO, so ordinarily you will not need to compile it. However, if you want or need to compile the Java GUI, go into the JAVA directory and invoke the following commands:

cd ../JAVA
javac *.java
jar -cfm GUI.jar manifest.mf *.class

For Unix/Linux and Mac users, the last two commands are included in a runscript makegui and can be used instead as a shortcut.

Running DL_MESO

To run either the DPD code (DL_MESO_DPD) or the LBE code (DL_MESO_LBE), you will need some input files in the directory from where you launch the code’s executable, dpd.exe or lbe.exe. (See Using DL_MESO_DPD or Using DL_MESO_LBE for more details.)

To run the serial version of e.g. DL_MESO_DPD, with or without OpenMP multithreading, use one of the following commands (first one for Windows, second one for other operating systems):

dpd.exe
./dpd.exe

If you have compiled a parallel version with MPI, the above command will only run DL_MESO on a single processor core. To run the code on multiple cores, you will need to precede the executable with mpirun and a flag indicating the number of cores you wish to use, e.g. for 8 cores

mpirun -np 8 ./dpd.exe

(Note: By default, DL_MESO_DPD will not print to the screen or standard output while running. You can use the command tail -f OUTPUT in another terminal window to continuously look at the end of this standard output file as the calculation runs.)

If you want to use the Java GUI, you can either use the rungui script in the WORK directory or the command:

java -jar ../java/GUI.jar

Chapter 3 of the DL_MESO User Manual explains what the buttons and menus do.

The utilities for preparing simulation inputs and analysing/processing simulation outputs can be run in the same way as the serial versions of DL_MESO_DPD or DL_MESO_LBE, although they can also take command-line options to change how they operate. Each utility can take the command-line option -h to show what the utility does and what the other command-line options happen to be, e.g.

./traject_vtf.exe -h
[Seaton2013]MA Seaton, RL Anderson, S Metz and W Smith, DL_MESO: highly scalable mesoscale simulations, Molecular Simulation, 39, p. 796-821, 2013, doi: 10.1080/08927022.2013.772297.

Footnotes

[1]For DL_MESO_DPD, the default OpenMP implementation requires additional memory to assign forces to particles without different threads trying to do this at the same time for a particular particle. This implementation is likely to work best on processors running at low clock speeds with multiple threads. There is an input file option available to change this implementation to one that assigns the forces serially, avoiding the use of additional memory. Alternatively you may want to use the threads as though they were virtual cores (known as hyperthreading) and make use of additional MPI parallelism.