49. Gromacs -- hangs on multicore when doing normal mode analysis

Symptom:
when doing
mdrun -s nm.tpr -mtx nm.mtx -v -deffnm nm
on a system with 637 atoms you end up with:

...Finish step 636 out of 637

and it hangs there with all cores running at 100%

Reason:
For some reason the normal mode analysis of at least this particular system won't run on multiple cores.

Solution:
Use an mpi compiled version of mdrun (see previous posts on compiling _dd, _mpi and _ddmpi versions of gromacs) and force the use of ONE core.

mpd --ncpus=4 &
mpdrun -n 1 mdrun_mpi -s nm.tpr -mtx nm.mtx -v -deffm nm

works!

Confirmation
This was confimed by running it on four computers:
64 bit: a six core AMD 64 using a compiled version of gromacs. Hangs.
64 bit: a four core intel i5 using both the debian version and a compiled version of gromacs. Hangs.
64 bit: an older four core intel using a compiled version of gromacs. Hangs.
32 bit: an old single-core laptop using the debian version of gromacs. Works.

Next, three single-core virtual machines were set up -- a stable 32 bit, a testing 32 bit and a testing 64 bit machine, all with the debian version of gromacs (sudo apt-get install gromac). They all worked, as they only had a single core.

46. A VERY rough approach to "benchmarking" gromacs (unscientific) on debian

Here's a comparison between different hardware and binaries which were built as described in http://verahill.blogspot.com/2012/01/debian-testing-64-wheezy-compiling.html

The simulation in question is a 100,000 step 100 ps simulation of a carbonate ion in water. Check back later for more details.

grompp -f carbonate.mdp -c carbonate.pdb -p carbonate.top -o carbonate.tpr

time mdrun -v -deffnm carbonate

Conclusions: 1) Double precision is slower by 25-35 % than single precision. 2) On a single machine there's no gain in using mpi. 3) Sadly, it appears that  intel i5-2400 X4 3.1GHz is more expensive than AMD Phenom II X6 3.1GHz for a reason.

Machines:
Be -- Phenom II X6, 8Gb RAM.
B --   Athlon X3 3.1 GHz 3 core, 4Gb RAM
Ta -- Optiplex 990 (i5 2400 3.1 GHz 4 core, 8Gb RAM). This machine was running a full gnome3/gnome-shell desktop at the same time as the tests were carried out. Take the results with a grain of salt.

Using mdrun (no mpi, single precision)
-------------------------------
Run   Be (6)     Ta (4)   B ( 3)
-------------------------------
1       1m27s    1m1s    1m48s
2       1m28s    1m1s    1m46s
3       1m35s    1m1s    1m47s
4       1m32s    1m1s    1m47s
5       1m33s    1m1s    1m47s


Using mdrun_dd (no mpi, double precision)
-------------------------------
Run   Be (6)     Ta (4)   B ( 3)
-------------------------------
1       1m49s    1m15s    2m25s
2       1m47s    1m15s    2m26s
3       1m51s    1m15s    2m26s
4       1m59s    1m15s    2m24s
5       1m58s    1m15s    2m26s



Using mdrun_mpi (mpi, single precision)

Machine: Be (Phenom II X6, 8Gb RAM).
(5 cores doesn't work)
---------------------------------
Cores/Run:   1            2      
---------------------------------
1                  4m11s   4m13s
2                  2m12s   2m15s
3                  1m46s   1m43s
4                  1m31s   1m31s
5                  ----------------
6                  1m28s   1m35s

Machine: Ta (Optiplex 990 (i5 3.1 GHz 4 core, 8Gb RAM).
---------------------------------
Cores/Run:   1            2           3
--------------------------------
1                  3m20s   3m20s   3m20s
2                  1m39s   1m38s   1m40s
3                  1m12s   1m13s   1m12s
4                  1m01s   1m01s   1m00s


Machine: Athlon X3 3.1 GHz 3 core, 4Gb RAM.
---------------------------------
Cores/Run:   1            2           3
--------------------------------
1                  4m32s   4m33s   4m36s
2                  2m28s   2m28s   2m27s
3                  1m49s   1m50s   1m49s

Using mdrun_ddmpi (mpi, double precision):

Machine: Phenom II X6, 8Gb RAM.
---------------------------------
Cores/Run:   1            2    
---------------------------------
1                  5m23s   5m25s
2                  2m56s   2m54s
3                  2m11s   2m11s
4                  1m56s   1m57s
5                  -----------------
6                  1m51s   1m52s

Machine: Optiplex 990 (i5 3.1 GHz 4 core, 8Gb RAM).
---------------------------------
Cores/Run:   1            2           3
--------------------------------
1                  4m14s   4m13s   4m13s
2                  2m09s   2m09s   2m10s
3                  1m33s   1m33s   1m33s
4                  1m16s   1m16s   1m16s


Machine: Athlon X3 3.1 GHz 3 core, 4Gb RAM.
---------------------------------
Cores/Run:   1            2           3
--------------------------------
1                  5m01s   5m52s   5m50s
2                  3m17s   3m17s   3m18s
3                  2m31s   2m31s   2m31s

45. Compiling gromacs with mpich2 ver 1.2 on debian testing

If you are using mpich2 1.2.1.1-5 -- read the ** comment. Otherwise don't worry.
** In my example I've used mpich2 ver 1.2.1.1-5 -- install mpich2 and libmpich2-dev version 1.2.1.1-5 according to http://verahill.blogspot.com/2012/01/debian-testing-64-wheezy-nwhchem.html -- do everything in between "Edit these two files.." and "exit" if you want the same system as I've used. **

Start here:
This is basically a condensed and annotated version of http://www.gromacs.org/Downloads/Installation_Instructions

Have a look at
http://www.gromacs.org/Downloads
to see what file to download

Also, you may want to do
sudo apt-get install build-essential gfortran fftw3

Next, use the console:

mkdir ~/tmp
cd ~/tmp

wget ftp://ftp.gromacs.org/pub/gromacs/gromacs-4.5.5.tar.gz

tar -xvf gromacs-4.5.5.tar.gz

aptitude search fftw
I have libfftw3-3 so I'll pull libfftw3-dev
sudo apt-get install libfftw3-dev

cd gromacs-4.5.5/

create buildconf.sh and put the following in it to build four different versions of gromacs L
(without mpi and single precision, with mpi and single precision (_mpi), without mpi and double precision (_dd), with mpi and double precision (_ddmpi)).
Change N in make -jN to equal the number of cores+1, in my case six cores => N=7, so -j7.
##########################
./configure --with-fft=fftw3
make -j7
sudo make install
 ./configure --with-fft=fftw3 --enable-mpi --program-suffix=_mpi 
make -j7 mdrun
sudo make install

make distclean

./configure --with-fft=fftw3 --disable-float --program-suffix=_dd
make -j7
sudo make install
 ./configure --with-fft=fftw3 --enable-mpi --disable-float --program-suffix=_ddmpi 
make -j7 mdrun
sudo make install

##########################

Then run

sh buildconf.sh

Next, in ~/.bashrc put

PATH=$PATH:/usr/local/gromacs/bin

or, to install for everyone, put the above line in /etc/profile (and then do source /etc/profile)

Then run
source ~/.bashrc

You can download a test set from http://www.gromacs.org/Downloads/Test-Set
Currently the newest one is ftp://ftp.gromacs.org/pub/tests/gmxtest-4.0.4.tgz

Or using git, if you have that installed:
git clone git://git.gromacs.org/regressiontests.git