src/Tips

Parallel runs with MPI

To compile with MPI parallelism enabled, you need to use something like:

CC99='mpicc -std=c99' qcc -Wall -O2 -D_MPI=1 example.c -o example -lm

where mpicc calls the MPI compiler on your system. The resulting executable can then be run in parallel using something like

mpirun -np 8 ./example

The details may vary according to how the MPI compiler is setup on your system.

Using Makefiles

The “manual” way above is automated if you use the standard Makefiles provided by Basilisk. You can then compile and run the example above on eight processes using:

CC='mpicc -D_MPI=8' make example.tst

This assumes that mpicc and mpirun are available on your system.

Running on supercomputers

A simple way to run Basilisk code on a supercomputer is to first generate a portable (ISO C99) source code on a machine where qcc is installed i.e.

%localmachine: qcc -source -D_MPI=1 example.c

Then copy the portable source code *_example.c* (don’t forget the underscore!) on the supercomputer and compile it:

%localmachine: scp _example.c login@supercomputer.org:
%localmachine: ssh login@supercomputer.org
%supercomputer.org: mpicc -Wall -std=c99 -O2 -D_MPI=1 _example.c -o example -lm

where the -std=c99 option sets the version of the language to C99. Note that this option may change depending on the compiler (the options shown above are valid for gcc or icc, the Intel compiler).

You will then need to use the job submission system of the supercomputer to set the number of processes and run the executable. See also the following examples:

• Parallel scalability
• Atomisation of a pulsed liquid jet
• Forced isotropic turbulence in a triply-periodic box

Non-cubic domains

It is possible to define non-cubic domains when running with MPI but also when running on GPUs or in OpenMP or serial mode. It is not possible yet to define non-cubic domains on tree grids.

Not that for MPI multigrid, each of the parallel sub-domain is cubic, so that the number of processes n_p must verify the relation \displaystyle n_p = 2^{di}\prod_d \text{dimensions}_d with i an integer, d the number of spatial dimensions and \text{dimensions}_d the domain dimensions specified using the dimensions() function.

Physical dimensions and spatial resolution

As usual, the physical dimension of the domain is set by calling the size() function. What is set is the size along the x-axis of the entire domain (not of the individual subdomains). This allow one to vary the number of processes while keeping the physical size constant (i.e. do the same simulation on a different number of processes).

The spatial resolution (N) is handled in a similar way, with the added constraint that it must be a multiple of n_x\times 2^n with n an integer.

Examples

A [0:1]\times[0:1/3]\times[0:1/3] channel, periodic along the x-axis (with slip walls), with 96 points along the x-axis:

#include "grid/multigrid3D.h"
...
int main() {
  dimensions (nx = 3, ny = 1, nz = 1);
  periodic (right);
  init_grid (128);
  ...
}

The simulation above can run in OpenMP/GPU/serial or on 3, 24, 192, 1536 etc. MPI processes.

A cube centered on the origin, of size 1.5, all-periodic, with 512 points along each axis, on 8 = 2³ or 64 = 4³ or 512 = 8³ or 4096 = 16³ etc… processes:

#include "grid/multigrid3D.h"
...
int main() {
  size (1.5);
  origin (-0.75, -0.75, -0.75);
  foreach_dimension()
    periodic (right);
  init_grid (512);
  ...
}