Showing posts with label building. Show all posts
Showing posts with label building. Show all posts

17 October 2013

520. New node: AMD FX 8350/32 Gb RAM/990 FX

Update 5 Nov 2012: Note that the motherboard doesn't support the CPU and this leads to spontaneous reboots under certain conditions. Make sure to look at the list over supported CPUs for the motherboard you use (in retrospect, obvious -- but as a linux person you get used to ignoring those things since everything's for just OSX or Win).

See here for the troubleshooting thread:

Also see this thread:
I'll need to read up on...stuff...but the bottom line seems to be that one would expect issues with this board/cpu combo:

Still only a 4+1 phase board the FX chips pull a bit more power than that can put out comfortably and stable. [..] Those would be your three best to choose from all are the better 8+2 phase designs...
my opinion is to stay away from the asus FX ive seen many people asking why their boards are throttling at full load, vrm protection causes voltages to drop at full load when vrms hit a certain temp.

and it seemed that low (CPU) voltages precipitated crashes.

Original post:
So I built a new node at the beginning of October 2013, using the following parts:
  • AMD FX 8350 CPU
  • 4*8 Gb GSkill RAM
  • ASRock 990FX Extreme3 motherboard
  • 1 Tb Seagate Barracuda HDD
  • MSI N210 graphics card
  • ASUS NX1101 Gigabit NIC
  • Corsair GS700 PSU
  • Antec GX700 case
NOTE that I'm having issues with spontaneous reboots during extended periods (days) of heavy load (100% CPU) which do not appear to be associated with faulty RAM, so you might want to think twice before using the exact same permutation of parts as is listed above. Most likely there's a power issue -- either the PSU isn't supplying enough juice to the Mobo, or the Mobo isn't supplying enough power to the CPU. Note also that the CPU isn't listed as an officially supported CPU by the motherboard manufacturer.*

See here for my troubleshooting thread:

So the main value of this post are the photos which show how easy it is to build a computer. Just don' this particular computer -- use a different motherboard.

The other value of the post is purely personal -- I just wanted to write down the steps to take whenever installing a new node in my little cluster.

There will be a post later on troubleshooting (and hopefully fixing) the issue of the spontaneous reboots.

*I've built a number of computers for myself as well as for other people and haven't had any issues (other than bad RAM) before. I got lazy this time and am paying for it.

The first step was assembly:

I like the case -- it's metal and feels robust. Having two fans on top is a definite plus as well as it works well with my home-built rack.

Note that the case doesn't come with a printed manual -- to get the manual you need to go online. And it still falls short -- there's no guide as to how to use the many, many cables it comes with. However, it's not rocket science either. Turns out that the case has a molex plug for powering the case fans. So plug in the molex plug to the PSU, then plug in the fans to the four weird plug/cables that the case comes with. Note that the mobo has no plug for the internal connector USB 3 cable that the case comes with.

See here for more details re the case:
Case, closed

The glorious innards of The Case.

The 700GX does not come with a mobo panel -- not that they tend to be useful anyway

Luckily all (most?) mobos come with their own panels -- push it in place before doing anything else. It can need a bit of negotiation in order to snap in properly.

The case came with riser nuts, four PSU screws and lots of screws for the mobo.

Put the riser nuts in the case -- they are the golden thingies

And here's the mother board. I don't know if there are universally accepted recommendations, but I prefer to install the CPU and RAM onto the mother board before installing it in the case -- you have more space to manouver and the risk of breaking the board is smaller.

The heatsink (left) and CPU (right)

The heatsink comes with thermal paste pre-applied. Don't touch it -- you want it to be as smooth and even as possible.

Get the CPU out

Note the yellow triangle in the bottom right corner in the picture

That should match up with the triangle in the bottom left of this picture. Note the raised level on the right side of the CPU socket.

The CPU in place. Note the raised lever. There should be no pushing -- the CPU should fit perfectly without any force whatsoever. If you bend a pin...then good luck.

The lever is in the locked position.

Next put the heatsink on. Give this a bit of thought as you won't want to have to reseat it several times (in the worst case you'll have to go buy some thermal paste, clean the heatsink and reapply the paste). So make sure you line up the fasteners before pushing the heatsink in place.

Everything is locked down.

The motherboard with the processor and heatsink in place. In the picture the CPU fan is attached to the WRONG connector. Look for a connector saying 'CPU FAN' (in the picture it's attached to POWER FAN)

Open the RAM slot fasteners, and push the RAM sticks in place firmly, but without excessive violence. Once the fasteners snap shut by themselves the sticks are properly seated. Improperly seated RAM sticks tend to prevent you from booting and leads to a lot of noise.

All four RAM sticks in place, and the motherboard attached to the case via seven screws that screw into the riser nuts.

The PSU is in place.

Main power and auxiliary power cables attached.

This particular case has a special tray for the hard drives.

Hard drive in place

SATA data and power cables attached

The other end of the SATA data cable attaches to the motherboard (SATA 1)

After a bit of rewiring. 

PCI NIC and PCI-E graphics cards in place.
And below is a picture of the cluster -- each node is connected to a gigabit WAN ( router  and a gigabit LAN switch ( 8/32 means 8 cores, 32 gb ram. The cluster 'runs' on the LAN. Each of the four nodes in the picture (there are two three-core nodes in addition) are connected to a KVM. Jobs are managed using SGE.

It's questionable whether one can really call it a cluster though since I run each job on a single node for performance reasons. It still attracts attention from visitors to my office though.

I then installed debian wheezy on it. During the installation I was notified that I might want to consider enabling non-free to get the r8169 and tg3 firmwares

So after enabling non-free in the sources I did:
sudo apt-get install firmware-realtek firmware firmware-linux-nonfree

Didn't seem to change anything though -- everything was working fine before too.

I also installed amd64-microcode which, if I understand things correctly, should obviate the need for some of the full BIOS updates.

Other little housekeeping things:
I first sorted out
INIT: Id "co" respawning too fast: disabled for 5 minutes
as shown here:

I then installed a few basic thing:
sudo apt-get install vim screen sinfo gawk lm-sensors

and made a ~/.vimrc:
set number set pastetoggle=<f3> nnoremap <f4> :set nonumber!<CR>

And set vim to the default editor in lieu of nano:
sudo update-alternatives --config editor

I edited /etc/default/sinfo to make it use the correct network:
OPTS="${OPTS} --quiet --bcastaddress="
I set up 'static' dhcp on the WAN router.

On the node, I then sorted out /etc/network/interfaces  to use dhcp on eth1 and on eth0, and to route everything properly (i.e. local traffic over eth0, and everything else over eth1):

auto lo
iface lo inet loopback

auto eth1
iface eth1 inet dhcp

auto eth0
iface eth0 inet static

post-up ip route flush all
post-up route add default eth1
post-up route add -net netmask gw eth0

SGE won't work properly unless you edit /etc/hosts:       localhost
#      oxygen   oxygen

The way my cluster works is that every node has its own shared folder.
mkdir ~/oxygen
mkdir ~/scratch
chmod 777 ~/oxygen

Export it as shown here:
Set up ssh key login in both directions:
vim ~/.ssh/authorized_keys

Then add the new node to the cluster:
Build nwchem as shown here:
Set up gaussian as shown here:
Fix shmem:

Finally, to address this issue regarding corrupt packages during SSH sessions I then added to /etc/rc.local/sbin/ethtool -K eth1 rx off tx off

15 March 2012

108. Building local version of sinfo without root/sudo on ROCKS/CentOS

Edit 04/04/2012: there were several errors and omissions. These have been fixed now.

Because I don't want to mess up a cluster which is on a different continent I'm trying to use my superuser powers as little as possible.

Here's how to make a local version of sinfo -- you'll still need to make sinfod runs as a service on all the nodes.

There's no reason the instructions here shouldn't work on most linux distros, including Debian.

cd ~/tmp
tar -xvf boost_1_49_0.tar.gz
cd boost_1_49_0/
./ --prefix=/export/home/me/.libboost

Edit tools/build/user-config.jam and add
using mpi ;
The space between mpi and ; is needed.

Start installation:
./b2 install

cd /export/home/me/.libboost/lib
ln -s
ln -s
ln -s
ln -s
ln -s

cd ~/tmp
wget ""
tar -xvf asio-1.5.3.tar.bz2
cd asio-1.5.3/

./configure --prefix=/export/home/me/.asio --with-boost=/export/home/me/.libboost/include
make install

tar -xvf sinfo-0.0.45.tar.gz
cd sinfo-0.0.45/

export LIBS=-L/export/home/me/.libboost/lib
export CPPFLAGS="-I/export/home/me/.libboost/include -I/export/home/me/.asio/include/"
./configure --prefix=/export/home/me/.sinfo --disable-IPv6

make install 

Getting started:
In order to make something happen at boot you need sudo/root access. However, HPC clusters are rarely rebooted, so even if you launch something as a user it will persist for a long time. If you're lucky the right ports are open -- and they should be open between nodes.

You also need to add this to your ~/.bashrc:
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/export/home/me/.libboost/lib

Start sinfod (the daemon) using:
~/.sinfo/sbin/./sinfod --quiet

ps aux |grep sinfod 
will show it it's running

And check that everything is ok using

01 March 2012

88. Building Apache 2.4.1 on Debian Testing

Don't remove your existing installation of apache2 without thinking. gnome-core depends on it.

Updated 1 March 2012. Added missing information which prevented building and now provide information about auto-start using an /etc/init.d/ script. Original post at 2012-02-23

 A new version of apache is kind of a big deal. Here's how to build it.

sudo apt-get install libapr1-dev uuid-dev libaprutil1-dev libmysqlclient-dev libpq-dev libsqlite3-dev rcconf
tar -xvf httpd-2.4.1.tar.gz
cd httpd-2.4.1/
make -j5

where 5 is the number of cores+1. Four-core i5 => 4+1=5.

To install run
sudo make install
sudo checkinstall
sudo dpkg -i *.deb


sudo checkinstall won't work unless you first
sudo mkdir -p /usr/local/apache2/modules
(see comment below by Y&S who pointed this out)

If you don't create the directory first, you get
/usr/share/apr-1.0/build/libtool --silent --mode=install install /usr/local/apache2/modules/
ranlib: could not create temporary file whilst writing archive: No more archived files
make[3]: *** [install-modules-yes] Error 1
make[3]: Leaving directory `/home/me/tmp/httpd-2.4.1/modules/aaa'
make[2]: *** [install-recursive] Error 1
make[2]: Leaving directory `/home/me/tmp/httpd-2.4.1/modules/aaa'
make[1]: *** [install-recursive] Error 1
make[1]: Leaving directory `/home/me/tmp/httpd-2.4.1/modules'
make: *** [install-recursive] Error 1
****  Installation failed. Aborting package creation.

Anyway, your compiled httpd is now in /usr/local/apache2/bin/httpd
In my case /usr/local/apache2/bin is not in my PATH -- whether you want to add it or not is a matter of choice.

In order for you to bind your own httpd to port 80 you need to stop apache2 if it is running
sudo service apache2 stop

Test your build:
me@tantalum:~$ /usr/local/apache2/bin/httpd -v
Server version: Apache/2.4.1 (Unix)
Server built:   Feb 23 2012 11:51:26
So far, very easy.

I've tried it on amd64 and i386 debian testing machines.

Replacing old version of apache2
Don't try to remove apache2.2-bin since gnome-core depends on it:

The easiest way to deal with this is to do
sudo rcconf 
and de-select apache2. This way it's still installed, but won't run as a rc service.

Making it boot -- init.d script
I had a look at this:

And here's my /etc/init.d/httpd script

# description: apache2 httpd 2.4.1 server
# Start the service httpd
start() {
        /usr/local/apache2/bin/httpd &
        echo "Up and running"
# Restart the service httpd
stop() {
        killall httpd
        echo "Killing httpd"
### main logic ###
case "$1" in
        echo $"Usage: $0 {start|stop}"
        exit 1
exit 0
Don't forget to chmod+x

sudo rcconf
and select httpd (it'll be at the bottom of the list)

It's not the prettiest of scripts, and you can probably do better by editing /etc/init.d/apache2

23 February 2012

71. Building Thunderbird 10.0.2 on debian testing

I use evolution for email, contacts and calendar because it integrates well with gnome and because it looks a whole lot prettier than the version of Thunderbird (i.e. Icedove) in the debian repos (3.1.16-1).

Well, sometimes you've got to check out the alternatives. Here's how to build thunderbird 10.0.2 from source.


sudo apt-get install libdbus-glib-1-dev gir1.2-notify-0.7 libnotify-dev  yasm checkinstall libzip-dev zip

cd ~/tmp

tar -xvf thunderbird-10.0.2.source.tar.bz2 

cd comm-release/

./configure --disable-necko-wifi
updating cache ../../.././config.cache
creating ./config.status
creating Makefile
creating config/Makefile
creating config/
creating ldap/Makefile
creating ldap/clients/tools/Makefile
creating ldap/include/Makefile
creating ldap/libraries/Makefile
creating ldap/libraries/libldap/Makefile
creating ldap/libraries/libprldap/Makefile
creating ldap/libraries/libldif/Makefile
creating ldap/libraries/liblber/Makefile
creating ldap/libraries/libiutil/Makefile
creating ldap/libraries/libssldap/Makefile
creating ldap/libraries/libutil/Makefile

make -jN

make[4]: Leaving directory `/home/me/tmp/comm-release/mail/test/mozmill'
make[3]: Leaving directory `/home/me/tmp/comm-release/mail'
make[2]: Leaving directory `/home/me/tmp/comm-release'
make[1]: Leaving directory `/home/me/tmp/comm-release'
if test -d ./mozilla/dist/bin ; then touch ./mozilla/dist/bin/.purgecaches ; fi

where N is the number of cores +1 -- in my case it's 7 since I have a six-core CPU. Be aware that building does take a while.

sudo make install

(sudo checkinstall ended with segfault for some reason)

You are now done.

Make sure that /usr/local/bin is in your PATH

me@beryllium:~/tmp/comm-release$ which thunderbird

Interesting observation:
while thunderbird starts thunderbird the home-built version seems to be referred to as earlybird:

What's ugly or not is subjective, but you may want to use this add-on:

checking MOZ_PANGO_CFLAGS... -pthread -I/usr/include/pango-1.0 -I/usr/include/glib-2.0 -I/usr/lib/x86_64-linux-gnu/glib-2.0/include -I/usr/include/freetype2
checking MOZ_PANGO_LIBS... -pthread -lpangoft2-1.0 -lpango-1.0 -lfreetype -lfontconfig -lgobject-2.0 -lgmodule-2.0 -lgthread-2.0 -lrt -lglib-2.0
checking for gnome-vfs-2.0 >= 2.0 gnome-vfs-module-2.0 >= 2.0... checking for gconf-2.0 >= 1.2.1 gobject-2.0 ... checking for dbus-glib-1 >= 0.60... Package dbus-glib-1 was not found in the pkg-config search path. Perhaps you should add the directory containing `dbus-glib-1.pc' to the PKG_CONFIG_PATH environment variable No package 'dbus-glib-1' found
configure: error: Library requirements (dbus-glib-1 >= 0.60) not met; consider adjusting the PKG_CONFIG_PATH environment variable if your libraries are in a nonstandard prefix so pkg-config can find them.

sudo apt-get install libdbus-glib-1-dev


checking MOZ_PANGO_LIBS... -pthread -lpangoft2-1.0 -lfreetype -lfontconfig -lpangocairo-1.0 -lpango-1.0 -lcairo -lgobject-2.0 -lgmodule-2.0 -lgthread-2.0 -lrt -lglib-2.0
checking for gnome-vfs-2.0 >= 2.0 gnome-vfs-module-2.0 >= 2.0... checking for gconf-2.0 >= 1.2.1 gobject-2.0 ... checking for libnotify >= 0.4... Package libnotify was not found in the pkg-config search path. Perhaps you should add the directory containing `libnotify.pc' to the PKG_CONFIG_PATH environment variable No package 'libnotify' found
configure: error: Library requirements (libnotify >= 0.4) not met; consider adjusting the PKG_CONFIG_PATH environment variable if your libraries are in a nonstandard prefix so pkg-config can find them

sudo apt-get install gir1.2-notify-0.7 libnotify-dev


checking MOZ_DBUS_GLIB_LIBS... -pthread -ldbus-glib-1 -ldbus-1 -lpthread -lgobject-2.0 -lgthread-2.0 -lrt -lglib-2.0
checking __attribute__ ((aligned ())) support... trying 64
configure: error: yasm is a required build tool for this architecture when webm is enabled. You may either install yasm or --disable-webm (which disables the WebM video format). See for more details.
configure: error: ./configure failed for mozilla

sudo apt-get install yasm

20 January 2012

55. Building/compiling OpenMM4.0 from source on debian testing

sudo apt-get install swig doxygen openjdk-7-jdk

Also, patience...the build can be temperamental. However, go through the steps enough times and it will work. Why, I don't know.

1. download (see error 4 below) py-dom-xpath-0.1 from here:

sudo apt-get install python-setuptools

2. Unzip it
3. Change to the directory py-dom-xpath-0.1/ and run
sudo python install

add this to your ~/.bashrc or /etc/profile

export LD_LIBRARY_PATH=/lib/openmm:/usr/lib/nvidia-cuda-toolkit:/usr/lib/nvidia:$LD_LIBRARY_PATH
export OPENMM_PLUGIN_DIR=/usr/local/openmm/lib/plugins
export OPENMM_ROOT_DIR=/usr/local/openmm
Finally (and this is truly ridiculous):

mkdir ~/tmp/OpenMM4.0-Source/docs/UsersGuide
cp OpenMM4.0-Source/docs/OpenMMUsersGuide.pdf OpenMM4.0-Source/docs/UsersGuide/

Start here:

As usual we'll do our work in ~/tmp
mkdir ~/tmp 
if that directory doesn't exist

To download the file you need to register at

Put the file in ~/tmp

cd ~/tmp
cd OpenMM4.0-Source/
cmake CMakeList.txt
make OpenMM
sudo make install

If all goes well (it took me three hours to iron out the details...) you'll see
-- Installing: /usr/local/openmm/include/openmm/internal/AmoebaUreyBradleyForceImpl.h
-- Installing: /usr/local/openmm/include/openmm/internal/AmoebaTorsionTorsionForceImpl.h
-- Installing: /usr/local/openmm/include/openmm/internal/AmoebaStretchBendForceImpl.h
-- Installing: /usr/local/openmm/lib/
-- Removed runtime path from "/usr/local/openmm/lib/"
-- Installing: /usr/local/openmm/lib/plugins/
-- Removed runtime path from "/usr/local/openmm/lib/plugins/"
-- Installing: /usr/local/openmm/lib/plugins/
-- Removed runtime path from "/usr/local/openmm/lib/plugins/"
-- Installing: /usr/local/openmm/include/openmm/RPMDIntegrator.h
-- Installing: /usr/local/openmm/include/openmm/RpmdKernels.h
-- Installing: /usr/local/openmm/lib/
-- Removed runtime path from "/usr/local/openmm/lib/"
-- Installing: /usr/local/openmm/lib/plugins/
-- Removed runtime path from "/usr/local/openmm/lib/plugins/"
-- Installing: /usr/local/openmm/include/openmm/serialization/SerializationNode.h
-- Installing: /usr/local/openmm/include/openmm/serialization/SerializationProxy.h
-- Installing: /usr/local/openmm/include/openmm/serialization/XmlSerializer.h
-- Installing: /usr/local/openmm/lib/
-- Removed runtime path from "/usr/local/openmm/lib/"

And you're are done! SUCCESS!

Errors and solutions:
I hate cmake, possibly because I don't really understand it.
Anyway, every time you hit one of the errors below you need to do the following after fixing it:
make clean
cmake CMakeList.txt
make OpenMM
or you'll get error # 6

1. Doxygen missing:

Swig version must be 1.3.39 or greater! (You have 0.0.0)
CMake Error at /usr/share/cmake-2.8/Modules/FindPackageHandleStandardArgs.cmake:91 (MESSAGE):
  Could NOT find Doxygen (missing: DOXYGEN_EXECUTABLE)
Call Stack (most recent call first):
  /usr/share/cmake-2.8/Modules/FindPackageHandleStandardArgs.cmake:252 (_FPHSA_FAILURE_MESSAGE)
  /usr/share/cmake-2.8/Modules/FindDoxygen.cmake:80 (FIND_PACKAGE_HANDLE_STANDARD_ARGS)
  wrappers/python/CMakeLists.txt:82 (find_package)
sudo apt-get install doxygen

2. Swig missing:
Swig version must be 1.3.39 or greater! (You have 0.0.0)
-- Java version configured successfully!
CMake Error at /usr/share/cmake-2.8/Modules/FindPackageHandleStandardArgs.cmake:91 (MESSAGE):
  Could NOT find Java (missing: Java_JAR_EXECUTABLE Java_JAVAC_EXECUTABLE)
  (found version "")
Call Stack (most recent call first):
  /usr/share/cmake-2.8/Modules/FindPackageHandleStandardArgs.cmake:252 (_FPHSA_FAILURE_MESSAGE)
  /usr/share/cmake-2.8/Modules/FindJava.cmake:174 (find_package_handle_standard_args)
  wrappers/python/CMakeLists.txt:85 (find_package)
sudo apt-get install swig

3. Java found and missing at the same time
-- Java version configured successfully!
CMake Error at /usr/share/cmake-2.8/Modules/FindPackageHandleStandardArgs.cmake:91 (MESSAGE):
  Could NOT find Java (missing: Java_JAR_EXECUTABLE Java_JAVAC_EXECUTABLE)
  (found version "")
Call Stack (most recent call first):
  /usr/share/cmake-2.8/Modules/FindPackageHandleStandardArgs.cmake:252 (_FPHSA_FAILURE_MESSAGE)
  /usr/share/cmake-2.8/Modules/FindJava.cmake:174 (find_package_handle_standard_args)
  wrappers/python/CMakeLists.txt:85 (find_package)
sudo apt-get install openjdk-7-jdk

4. No module named xpath
Execution time: 1072 milliseconds
Memory used: 21270968
NamePool contents: 89 entries in 86 chains. 6 prefixes, 6 URIs
[ 87%] Creating OpenMM Python swig input files...
Traceback (most recent call last):
  File "/home/me/tmp/OpenMM4.0-Source/python/src/swig_doxygen/", line 15, in <module>
    import xpath
ImportError: No module named xpath
make[2]: *** [python/src/swig_doxygen/swig_lib/python/pythonprepend.i] Error 1
make[1]: *** [wrappers/python/CMakeFiles/BuildModule.dir/all] Error 2
make: *** [all] Error 2
Download py-dom-xpath-0.1.tar.gz:


Untar (tar -xvf) in ~/tmp
cd py-dom-xpath-0.1/
sudo python install

What didn't work:
sudo apt-get install libxml-xpath-perl
sudo apt-get install libghc-hxt-xpath-dev

5. Missing OpenMMUsersGuide.pdf
-- Installing: /usr/local/openmm/docs/API Reference.html
CMake Error at cmake_install.cmake:119 (FILE):
  file INSTALL cannot find

make: *** [install] Error 1

WTF??? The build can fail over something so trivial???
mkdir ~/tmp/OpenMM4.0-Source/docs/UsersGuide
cp ~/tmp/OpenMM4.0-Source/docs/OpenMMUsersGuide.pdf ~/tmp/OpenMM4.0-Source/docs/UsersGuide/

6. Linking CXX shared library failure
[  6%] Building CXX object CMakeFiles/OpenMM.dir/src/cuda/kCalculateLocalForces.cu_OpenMMCuda_generated.cpp.o
Linking CXX shared library
CMakeFiles/OpenMM.dir/src/cuda/kCalculateCDLJForces.cu_OpenMMCuda_generated.cpp.o: In function `__device_stub__Z29kFindBlockBoundsCutoff_kernelv()':
kCalculateCDLJForces.cu_OpenMMCuda_generated.cpp:(.text+0x200): multiple definition of `__device_stub__Z29kFindBlockBoundsCutoff_kernelv()'
CMakeFiles/OpenMM.dir/src/cuda/kCalculateNonbondedSoftcore.cu_OpenMMFreeEnergyCuda_generated.cpp.o:kCalculateNonbondedSoftcore.cu_OpenMMFreeEnergyCuda_generated.cpp:(.text+0x1b0): first defined here
CMakeFiles/OpenMM.dir/src/cuda/kCalculateAmoebaCudaUtilities.cu_OpenMMAmoebaCuda_generated.cpp.o: In function `kFindInteractionsWithinBlocksPeriodic_kernel(unsigned int*)':
kCalculateAmoebaCudaUtilities.cu_OpenMMAmoebaCuda_generated.cpp:(.text+0x1d0): multiple definition of `kFindInteractionsWithinBlocksPeriodic_kernel(unsigned int*)'
CMakeFiles/OpenMM.dir/src/cuda/kCalculateNonbondedSoftcore.cu_OpenMMFreeEnergyCuda_generated.cpp.o:kCalculateNonbondedSoftcore.cu_OpenMMFreeEnergyCuda_generated.cpp:(.text+0x3a0): first defined here
collect2: ld returned 1 exit status
make[2]: *** [] Error 1
make[1]: *** [CMakeFiles/OpenMM.dir/all] Error 2
make: *** [all] Error 2

You had one of errors 1-5, you solved it and tried to run make again. You need to do

make clean
cmake CMakeList.txt

first before proceeding with make or you end up with this error.

54. Compiling GROMACS with GPU support using OpenMM (OpenMM 4.0 source or 3.1.1 Linux 64 binaries) on debian testing

To compile gromacs without GPU support (and that's probably what you should do) look here:
GPU support is most likely NOT going to be useful for you.

For a better way (out-of-tree) of building a newer openMM, see:

The method below describes an in-tree build of OpenMM. Do an out of tree build (e.g. see link above) to avoid headaches.

OK, enough with the 'bold'. Basically, I wrote the post below at a time when I was at a very early stage of learning how to compile my own programs. I'm obviously still learning, but I have published better methods now -- see the links above for those.
Be aware that in-tree (when you build your package in the root of the source tree) building is not supported for OpenMM and you will probably be told so if posting on forums asking for help. Out-of-tree (when you build in a directory outside the source tree structure) is supported and is described above. So, while I'm leaving this post here for posterity, use it as inspiration, but don't follow it blindly.

/25th of September 2012.

Original Post:
First read this: 1. Use EITHER the OpenMM3.1.1-Linux64 binaries
 2. OR, which is better, compile OpenMM4.0 from source -- see here -- if you do this you can skip step 3.

Do NOT use: 1. the Open4.0MM-Linux64 binaries or the OpenMM3.1.1-Source source -- at least I have had issues with those versions.

The following guide was last tested: 20/01/2012
It has not been checked for typos -- whenever you use a command with 'sudo' in it, try to understand what it does first.

Finally: if it doesn't work the first time, try a few more times. I've never managed to build on the first try, but restarting with make clean, cmake CMakeList.txt, make works in the end every time. Sigh...

You may also want to remove CMakCache.txt in the source root.

1. Things to install first

Install cmake and autoconf if you haven't already

sudo apt-get install cmake autoconf

2. edit ~/.bashrc  or /etc/profiles
Put this at the end of your ~/.bashrc  (or /etc/profiles for multi-user systems)

export LD_LIBRARY_PATH=/lib/openmm:/usr/lib/nvidia-cuda-toolkit:/usr/lib/nvidia:$LD_LIBRARY_PATH
export OPENMM_PLUGIN_DIR=/usr/local/openmm/lib/plugins
export OPENMM_ROOT_DIR=/usr/local/openmm 

load its content by
source ~/.bashrc

3. OpenMM
To download OpenMM you need to register with Registering is all thing considered fairly easy and quick.

Either follow this guide for compiling OpenMM4.0 from source (preferred) or follow the steps below to use the 3.1.1 binaries:

Download the file (if applicable).

Put the .zip file in ~/tmp

cd ~/tmp/

sudo mkdir -p /lib/openmm/plugins
sudo mkdir -p /lib/openmm/include

sudo cp lib/*.so /lib/openmm
sudo cp include/* -R /lib/openmm/include
sudo cp plugins/* -R /lib/openmm/plugins

This is the structure you're aiming for:

|-- include
|   |-- internal
|   |-- openmm
|   `-- serialization
`-- plugins


sudo mkdir /usr/local/openmm
sudo cp ~/tmp/OpenMM3.1.1/* -R /usr/local/openmm

So that you end up with

|-- bin
|-- docs
|   `-- api
|       `-- search
|-- include
|   `-- openmm
|       |-- internal
|       `-- serialization
|-- lib
|   `-- plugins
|-- licenses
`-- python
    |-- simtk
    |   |-- openmm
    |   `-- unit
    `-- src
        `-- swig_doxygen
            |-- doxygen
            `-- swig_lib
                `-- python
4. Gromacs-4.5.5
Get and untar gromacs:
cd ~/tmp
tar -xvf gromacs-4.5.5.tar.gz
mv gromacs-4.5.5/ gromacs_gpu/

Let's start preparing our build:
cd gromacs_gpu/
make clean
export OPENMM_ROOT_DIR=/usr/local/openmm

EDIT/COMMENT:  I think autoconf /automake and cmake are mutually exclusive. I only let autoconf stay here since I did invoke it when I did my build.

The output of a successful run of cmake follows below -- note the binary suffix, as well as the Found CUDA and Found OpenMM.

-- Using default binary suffix: "-gpu"
-- Using default library suffix: "_gpu"
-- No external FFT libraries needed for the OpenMM build, switching to fftpack!
-- Switching off CPU-based acceleration, the OpenMM build does not support/need any!
-- Found CUDA: /usr (Required is at least version "3.1")
-- Found OpenMM: /usr/local/openmm
-- Looking for fseeko
-- Looking for fseeko - found
-- Using internal FFT library - fftpack
-- Configuring done
-- Generating done
-- Build files have been written to: /home/me/tmp/gromacs_mopac

Next, some editing - first we copy two of the files that the DGMX_OPENMM switch on the cmake command above created, then we remove two offending lines.

This command should be on a single line:
cp gromacs_gpu/src/kernel/gmx_gpu_utils/CMakeFiles/ ~/tmp

This command is also a single line:
cp gromacs_gpu/src/kernel/gmx_gpu_utils/CMakeFiles/ ~/tmp

We now have two *.cu.o files in our ~/tmp
Open each one and remove all instances of -fexcess-precision=fast in them. Then copy the files back to their original location:

(This command should be on a single line:)
cp gromacs_gpu/src/kernel/gmx_gpu_utils/CMakeFiles/

(This command is also a single line:)
cp gromacs_gpu/src/kernel/gmx_gpu_utils/CMakeFiles/

We cope and edit since these files get regenerated every time you do the cmake -DGMX_OPENMM=ON command.
Quick aside: a clue to  -fexcess-precision:
Here: "On x86 targets, code containing floating-point calculations may run significantly slower when compiled with GCC 4.5 in strict C99 conformance mode than they did with earlier GCC versions. This is due to stricter standard conformance of the compiler and can be avoided by using the option -fexcess-precision=fast." * Here: "-fexcess-precision=fast # disables the GCC 4.5 strict floating point C99 standards conformance for improved floating point performance "* I'm running gcc (Debian) 4.6.2-12 and g++ 4.6.2-4* I've tried with both -fexcess-precision=fast and -fexcess-precision=standard. Neither works.
* Not all excess-precision options work with all languages.
* Using g++ 4.5 someone got "cc1plus: sorry, unimplemented: -fexcess-precision=standard for C++" and comes to the conclusion that "Bah. Still, at least the -ffloat-store workaround still helps, for this
case at any rate. Also, if you get GCC to use SSE instructions, there's no
issue with excess precision".
I think the conclusion is that removing -fexcess-precision=fast MAY lead to a speed penalty (up to 2x) but will not change the results of the sim/calc.
OK, time to build!
cd gromacs_gpu/

and then install:
sudo make install

If all went well you'll now have mdrun-gpu in your /usr/local/gromacs/bin folder together with 93 other -gpu bin files!

Files for benchmarking can be downloaded from here:

If you get:

Program mdrun-gpu, VERSION 4.5.5
Source code file: /home/me/tmp/gromacs_gpu/src/kernel/openmm_wrapper.cpp, line: 1324
Fatal error:
The selected GPU (#0, GeForce GT 430) is not supported by Gromacs! Most probably you have a low-end GPU which would not perform well, or new hardware that has not been tested with the current release. If you still want to try using the device, use the force-device=yes option.
For more information and tips for troubleshooting, please check the GROMACS
website at

when trying to benchmark, use

mdrun-gpu -device "OpenMM:deviceid=0, force-device=yes" -v

to force execution.

Make sure to monitor your GPU temperature -- it will easily reach 80 degrees for a fan-less video card.

Early 'results':
Test                             GPU: GT 430    GT520      CPU (6 core AMD Phenom II, 2.8 GHz)
dhfr-impl-1nm.bench           31.8          19.5            27.2 ns/day
dhfr-impl-2nm.bench           31.9          19.9            5.8 ns/day
dhfr-solv-PME.bench           2.7             1.7             8.6 ns/day

Two significant problems are present:
1. The card heats up quickly to 80 degrees Celsius. It may thus be throttled.
2. The video card is in use by GNOME at the same time. Will rerun later without x.

The difference in speed for the 2nm text is significant.

5. Optional
If you haven't already, add this to ~/.bashrc (or /etc/profile)

then run
source ~/.bashrc
to load the new settings

Addendum: Errors and solutions
Do a fresh cycle of make clean, cmake, make and make install after each fix. Also, remove CMakeCache.txt

1. Missing libxml2-dev

[ 89%] Building C object src/mdlib/CMakeFiles/md.dir/gmx_qhop_xml.c.o
/home/me/tmp/gromacs_gpu/src/mdlib/gmx_qhop_xml.c:48:27: fatal error: libxml/parser.h: No such file or directory
compilation terminated.
make[3]: *** [src/mdlib/CMakeFiles/md.dir/gmx_qhop_xml.c.o] Error 1
make[2]: *** [src/mdlib/CMakeFiles/md.dir/all] Error 2
make[1]: *** [src/kernel/CMakeFiles/mdrun.dir/rule] Error 2
make: *** [mdrun] Error 2

sudo apt-get install libxml2-dev

2. Missing OpenMM.h
Linking C shared library
[ 98%] Built target gmxpreprocess
Scanning dependencies of target openmm_api_wrapper
[ 98%] Building CXX object src/kernel/CMakeFiles/openmm_api_wrapper.dir/openmm_wrapper.cpp.o
/home/me/tmp/gromacs_gpu/src/kernel/openmm_wrapper.cpp:59:20: fatal error: OpenMM.h: No such file or directory
compilation terminated.
make[3]: *** [src/kernel/CMakeFiles/openmm_api_wrapper.dir/openmm_wrapper.cpp.o] Error 1
make[2]: *** [src/kernel/CMakeFiles/openmm_api_wrapper.dir/all] Error 2
make[1]: *** [src/kernel/CMakeFiles/mdrun.dir/rule] Error 2
make: *** [mdrun] Error 2

 sudo mkdir /usr/local/openmm/lib/include
sudo cp ~/tmp/OpenMM3.1.1-Source/openmmapi/include/* -R /usr/local/openmm/include/

3. Missing Kernel.h
Linking CXX static library libgmx_gpu_utils.a
[  0%] Built target gmx_gpu_utils
[  0%] Building CXX object src/kernel/CMakeFiles/openmm_api_wrapper.dir/openmm_wrapper.cpp.o
In file included from /usr/local/openmm/include/OpenMM.h:36:0,
                 from /home/me/tmp/gromacs_gpu/src/kernel/openmm_wrapper.cpp:59:
/usr/local/openmm/include/openmm/BrownianIntegrator.h:36:27: fatal error: openmm/Kernel.h: No such file or directory
compilation terminated.
make[3]: *** [src/kernel/CMakeFiles/openmm_api_wrapper.dir/openmm_wrapper.cpp.o] Error 1
make[2]: *** [src/kernel/CMakeFiles/openmm_api_wrapper.dir/all] Error 2
make[1]: *** [src/kernel/CMakeFiles/mdrun.dir/rule] Error 2
make: *** [mdrun] Error 2
make: *** No rule to make target `mdrun-install'.  Stop.

Kernel.h was missing from the openmmapi folder when I compiled OpenMM myself. Use the pre-compiled version

4. cc1plus: Unrecognised option -fexcess-precision=fast

 (the % depends on whether you used make, make mdrun, make gmx_gpu_utils etc)
[  0%] Building NVCC (Device) object src/kernel/gmx_gpu_utils/./
cc1plus: error: unrecognized command line option "-fexcess-precision=fast"
CMake Error at CMakeFiles/ (message):
  Error generating

make[3]: *** [src/kernel/gmx_gpu_utils/./] Error 1
make[2]: *** [src/kernel/gmx_gpu_utils/CMakeFiles/gmx_gpu_utils.dir/all] Error 2
make[1]: *** [src/kernel/CMakeFiles/mdrun.dir/rule] Error 2
make: *** [mdrun] Error 2

You need to edit and and remove all instances of -fexcess-precision=fast
The files in src/kernel/gmx_gpu_utils/ will be overwritten every time you run cmake so make copies of the two edited files to keep around if you need to re-build.

Here's the diff (edit vs unedited) for

< set(CMAKE_HOST_FLAGS  -Wall -Wno-unused   )
> set(CMAKE_HOST_FLAGS  -fexcess-precision=fast -Wall -Wno-unused   )

Here's the diff (edit vs unedited) for

< set(CMAKE_HOST_FLAGS  -Wall -Wno-unused   )
> set(CMAKE_HOST_FLAGS  -fexcess-precision=fast -Wall -Wno-unused   )

5. Nonbonded kernel
[ 78%] Building C object src/gmxlib/CMakeFiles/gmx.dir/nonbonded/nb_kernel_x86_64_sse/nb_kernel400_x86_64_sse.c.o
/home/me/tmp/gromacs_gpu/src/gmxlib/nonbonded/nb_kernel_x86_64_sse/nb_kernel400_x86_64_sse.c: In function ‘nb_kernel400nf_x86_64_sse’:
/home/me/tmp/gromacs_gpu/src/gmxlib/nonbonded/nb_kernel_x86_64_sse/nb_kernel400_x86_64_sse.c:629:32: error: ‘gmx_invsqrt_exptab’ undeclared (first use in this function)
/home/me/tmp/gromacs_gpu/src/gmxlib/nonbonded/nb_kernel_x86_64_sse/nb_kernel400_x86_64_sse.c:629:32: note: each undeclared identifier is reported only once for each function it appears in
/home/me/tmp/gromacs_gpu/src/gmxlib/nonbonded/nb_kernel_x86_64_sse/nb_kernel400_x86_64_sse.c:629:59: error: ‘gmx_invsqrt_fracttab’ undeclared (first use in this function)
make[3]: *** [src/gmxlib/CMakeFiles/gmx.dir/nonbonded/nb_kernel_x86_64_sse/nb_kernel400_x86_64_sse.c.o] Error 1
make[2]: *** [src/gmxlib/CMakeFiles/gmx.dir/all] Error 2
make[1]: *** [src/kernel/CMakeFiles/mdrun.dir/rule] Error 2
make: *** [mdrun] Error 2

Solution: instead of just using cmake, use cmake ../gromacs_gpu/ -DGMX_OPENMM=ON -DGMX_THREADS=OFF

Links to this page:;cid=1358868814898-115

15 December 2011

29. Compiling/Building nwchem with mpich on debian testing 64 bit (Wheezy -- 15/12/2011)

So, as seen in the previous post, mpich2 ver 1.4 and nwchem 6.0 don't play nicely together.

Continuing with the virtual machine in the previous post, I added a line referencing the stable version to /etc/apt/sources.list:
deb stable main contrib non-free

Important: I ADDED that line -- all lines referencing the testing version (or wheezy) are left untouched.
sudo apt-get update
Since the versions in stable are older adding that line won't affect your system.

apt-cache showpkg mpich2

Under provides it should say:

sudo apt-get install mpich2=

You'll get a warning that you're about to downgrade, which is what we're trying to do.

sudo apt-get autoremove (will downgrade dependent packages. Just go with it)
aptitude search mpich2
check what's installed and what version
aptitude show libmpich2-dev
If it's 1.4.1 or not installed, make sure to set it to just like for mpich2

Run sh (see here for the script). Seems to build ok. All the mpd tools are where they should be.
NOTE: according to this post mpd isn't needed in newer (>=1.3) versions.

In summary, this seems to be the way to build nwchem on wheezy -- by downgrading the mpich2 and mpich2-dev packages. Since downgrading those packages may affect other packages as well, it may cause problems, but so far so good.

Testing the built binary:
mpd --ncpus=4 &
mptrace -l
mpdrun -n 4 ./nwchem ../../examples/dirdyvtst/h3/h3tr1.nw

All is good

EDIT (16/12/2011):
So, you've installed an older version of a package. apt-get will want to upgrade it, so you should put the packages on 'hold'. Every time you upgrade your system apt-get will warn you that there are packages on hold, so don't worry about forgetting about it

sudo su
echo "mpich2 hold"|dpkg --set-selections
echo "libmpich2-dev hold"|dpkg --set-selections

(taken from

As an aside, you may want to downgrade gromacs-mpich to use mpich2-1.2 as well:
sudo apt-get install gromacs-mpich=4.0.7-3

28. Compiling/Building nwchem with mpich on debian 64 bit (15/12/2011) -- observations of squeeze, wheezy, sid and experimental

** NOTE: the nwchem build in the debian repos (6.0-1) does not support mpich. It will not run in parallel. It seems like 6.0-2 will, but it's not in the repos yet, and looking at the package status page I get a bit worried about the likelihood of seeing a finished build for amd64 **

** The solution to building on debian testing and above (and presumably ubuntu >10.10) is here: **

** NOTE: according to this post mpd isn't needed in newer (>=1.3) versions. It is needed -- and provided -- by mpich2 1.2**

So, I've had problems building nwchem on debian testing for about a year now. Actually, building nwchem is pretty straightforward, but building it with mpich2 support doesn't seem to work.

I also noted that mpd doesn't refer to an mpich daemon anymore, leading me to suspect that maybe the version of mpich2 (now at 1.4; was at 1.2 when I built on ubuntu early in the year) could be part of the problem.

Long story short: it builds with mpich2 support on debian stable (squeeze), but not debian testing (wheezy).

Here's what I did: I installed debian stable 64 bit in virtualbox from the business-card iso (standard tools + ssh. No desktop environment). I made sure the system (squeeze) was up to date, downloaded nwchem-6.0 and extracted it in /home/myhome/nwchem/nwchem-6.0  and create a build file,, with the following content:
export TCGRSH=/usr/local/bin/ssh
export NWCHEM_TOP=/home/myhome/nwchem/nwchem-6.0
export USE_MPI=y
export USE_MPIF=y
export MPI_LOC=/usr
export MPI_INCLUDE=$MPI_LOC/include/mpich2
export LIBMPI="-lfmpich -lmpich"
cd $NWCHEM_TOP/src
make nwchem_config
make FC=gfortran

Only thing remaining before building is:
sudo apt-get install mpich2 gfortran

mpich2: version 1.2

sh then starts the build process which takes about 10-20 minutes in a virtual machine. Everything good. Works as should -- I only have one core in the virtual machine, so can't do much testing. It builds without a hitch though. In addition mpd, mpdtrace etc are all present.

 Time to take a snapshot. 1.7 Gb.
I next edited /etc/apt/sources.list and replaced all instances of squeeze with wheezy (including commenting out the last two lines (wheezey-updates / squeeze-updates), did sudo apt-get update and sudo apt-get dist-upgrade

mpich2: version 1.4.1.
mpd and mpdtrace are now gone. Only mpi* around.

Tried sh again (added a make clean before make nwchem_config). It builds for a long time - 10-20 minutes -- ultimately the build fails with
/usr/lib/ undefined reference to `MPL_trid'
/usr/lib/ undefined reference to `MPL_trinit'
collect2: ld returned 1 exit status
make: *** [all] Error 1

The undefined references are: MPL_trid, MPL_trvalid, MPL_env2int, MPL_trrealloc, MPL_trspace, MPL_trDebugLevel, MPL_TrSetMaxMem, MPL_trlevel, MPL_trmalloc, MPL_putenv, MPL_env2bool, MPL_env2range, MPL_trcalloc, MPL_trfree, MPL_en2str, MPL_trstrdup, MPL_trdump and MPL_trinit

So no luck with Wheezy, which is the system I run on all my computers. I've actually tried quite a few approaches under Wheezy and have managed to complete the odd build, but without getting proper functionality i.e. using mpiexec -n 6 six instances are created instead of one instance spread across six cores so that the system is solved six times instead of just one.

Time to take another snapshot followed by an upgrade.
The wheezy snapshot is 2.2 Gb -- wonder where the 0.5 Gb come from?

Replacing all instances of wheezy with sid using vim (:%s/wheezy/sid/g). Fails to fetch during sudo apt-get update. Oh well. Remove that line from sources.list. Now update works.
Then sudo apt-get dist-upgrade. 140 Mb of archives to get.

As a general thought -- maybe it'd be worth keeping a copy of sid in a virtual machine just to see what's around the corner?

Upgrade done, sudo shutdown -r now
mpich2: version 1.4.1, same as wheezy. No mpd or mpdtrace.

Try building and...starts at 14:17, fails at 14:45. Same errors as for Wheezy.

Last chance -- snapshot (1.9 Gb...), add a ref to experimental updates. They are the same. Not worth trying thus.

So there you go -- nwchem builds ok on Squeeze using mpich2 ver 1.2, but not on any of the more up-to-date distros. I also wonder about the lack of mpd/mpdtrace -- in Squeeze mpd is the mpich daemon, while in Wheezy and above it's the Music Player Daemon. Something is odd here.

Coming next: can you build nwchem on Wheezy if you pull mpich2 ver 1.2 from the archives? Of course you can. The answer and how-to is here: