Showing posts with label wheezy. Show all posts
Showing posts with label wheezy. Show all posts

11 January 2013

316. Briefly: Automated chroot/sandbox creation

EDIT:
There are plenty of reasons to use chroots, but security is not one of them.

For a practical how-to see e.g
http://pen-testing.sans.org/blog/2012/06/06/escaping-restricted-linux-shells

For a bit of yelling, see
http://yarchive.net/comp/linux/chroot.html

chroot will improve your security by creating an obstacle which may filter out some would-be crackers, but it will not make it secure by any standard. (in spite of what I may have written elsewhere on this blog).

Original post:
I've been using chroot to compile and test stuff so much lately that I figure it was time to automate the process.

Before creating your chroot you'll need a few packages:
sudo apt-get install debootstrap coreutils x11-xserver-utils


The scripts
makechroot.sh
mkdir $HOME/tmp/jail/$1 -p sudo debootstrap --arch amd64 testing $HOME/tmp/jail/$1 http://ftp.au.debian.org/debian/ sudo cp setupchroot.sh $HOME/tmp/jail/$1/

setupchroot.sh
rm /etc/apt/sources.list echo 'deb http://ftp.au.debian.org/debian/ wheezy main contrib non-free' >> /etc/apt/sources.list apt-get update apt-get install locales sudo vim echo 'export LC_ALL="C"'>>/etc/bash.bashrc echo 'export LANG="C"'>>/etc/bash.bashrc echo 'export DISPLAY=:0.0' >> /etc/bash.bashrc echo '127.0.0.1 beryllium >> /etc/hosts' source /etc/bash.bashrc adduser sandbox usermod -g sudo sandbox echo 'Defaults !tty_tickets' >> /etc/sudoers

launchchroot.sh
xhost + sudo mount -o bind /proc $1/proc sudo cp /etc/resolv.conf $1/etc/resolv.conf sudo chroot $HOME/tmp/jail/$1

How to use
To set up the chroot:
sh makechroot.sh mynewchroot
sudo chroot mynewchroot
root@beryllium:/# sh setupchroot.sh

To use the chroot:
sh launchchroot.sh mynewchroot

Once you're done with the chroot and logged out, do
sudo umount $HOME/tmp/jail/mynewchroot/proc

to unmount the /proc -- you can now delete, copy etc. the directory structure of you chroot.

315. Briefly: Compile firefox 18 on debian testing/wheezy

For no better reason than pure curiosity, here's how to build firefox 18. My regular build environment has so many packages installed these days that it's useless for working out dependencies for other people, so I tried it in a fresh chroot to hammer out the current dependencies beyond a doubt. In other words: it will work for you.

In retrospect I'm not sure if you really need the dev package for libgtk2.0.

sudo apt-get install bzip2 build-essential python zip unzip pkg-config libgtk2.0-dev gconf2 libdbus-1-dev libdbus-glib-1-dev yasm libasound2-dev libcurl4-gnutls-dev libxt-dev mesa-common-dev
mkdir ~/tmp/firefox -p
cd ~/tmp/firefox
wget ftp://ftp.mozilla.org/pub/firefox/releases/18.0/source/firefox-18.0.source.tar.bz2
tar xvf firefox-18.0.source.tar.bz2
mkdir build/
cd build/
../mozilla-release/./configure --prefix=$HOME/.firefox18
make
make install
echo "alias firefox18='$HOME/.firefox18/bin/firefox'" >> $HOME/.bashrc
source $HOME/.bashrc

Launch by typing
firefox18


05 January 2013

303. Compiling Simon on Debian Testing (Simon 0.4)

Here's how to get Simon 0.4.0 compiled. Simon does speech recognition which allows you do voice-control your computer. It does NOT do transcription, e.g. like Dragon Naturally Speaking.

See e.g. here, here and here for more information about Simon.

Simon uses cmake. I hate cmake since it makes life a lot more difficult than it needs to be.

Also, note that Simon relies heavily on KDE, so you need a number of KDE related files -- if not the whole desktop -- installed. Obviously, it runs fine under GNOME, which is where I'm using it.

This post is limited to describing how to compile it, not how to use it -- that may come later.



First get the deendencies. The list of deps is taken from http://userbase.kde.org/Simon/Development_Environment#Requirements, and expanded (e.g. libboost).

sudo apt-get install qt4-qtconfig kdelibs5-dev libxtst-dev libsamplerate-dev kdepimlibs5-dev libboost-dev
sudo apt-get install build-essential cmake gettext kdeartwork libqwt-dev libqt4-sql-sqlite libphonon-dev libattica0 libattica-dev zlib1g-dev libasound2-dev 


Next sort out Julius
I think it used to be in the repos, but it's not there anymore from what I can see).

mkdir ~/tmp/simon -p
cd ~/tmp/simon
wget http://jaist.dl.sourceforge.jp/julius/56549/julius-4.2.2.tar.gz
tar xvf julius-4.2.2.tar.gz
cd julius-4.2.2/
./configure --prefix=~/.simon/julius
**************************************************************** Julius/Julian libsent library rev.4.2.2: - Audio I/O primary mic device API : alsa (Advanced Linux Sound Architecture) available mic device API : alsa oss supported audio format : RAW and WAV only NetAudio support : no - Language Modeling class N-gram support : yes - Libraries file decompression by : zlib library - Process management fork on adinnet input : no Note: compilation time flags are now stored in "libsent-config". If you link this library, please add output of "libsent-config --cflags" to CFLAGS and "libsent-config --libs" to LIBS. ****************************************************************
make make install cd ../


HTK
You'll need HTK to use adaptive models with the Voxforge base. It's not necessary otherwise.
To get HTK, register here: http://htk.eng.cam.ac.uk/register.shtml

You should receive a password immediately afterwards. Then go to http://htk.eng.cam.ac.uk/download.shtml and download the Linux sources.Put the HTK-3.4.1.tar.gz file in ~/tmp/simon.
cd ~/tmp/simon
tar xvf HTK-3.4.1.tar.gz
cd htk/
./configure --prefix=$HOME/.simon/htk --disable-hslab
make all
mkdir ~/.simon/htk
make install

For some reason it's built as 32 bit, not 64. Also, it wants the parent install directory to exist before running make install.

Next, sphinx:

wget http://sourceforge.net/projects/cmusphinx/files/sphinxbase/0.8/sphinxbase-0.8.tar.gz
tar xvf sphinxbase-0.8.tar.gz 
cd sphinxbase-0.8/
./configure --prefix=$HOME/.simon/sphinxbase
make
make install
cd ../


wget http://sourceforge.net/projects/cmusphinx/files/pocketsphinx/0.8/pocketsphinx-0.8.tar.gz
tar xvf pocketsphinx-0.8.tar.gz
cd pocketsphinx-0.8/
./configure --prefix=$HOME/.simon/pocketsphinx SphinxBase_CFLAGS=-I$HOME/.simon/sphinxbase/include/sphinxbase SphinxBase_LIBS=-L$HOME/.simon/sphinxbase/lib
make
make install
cd ../

wget http://sourceforge.net/projects/cmusphinx/files/sphinxtrain/1.0.8/sphinxtrain-1.0.8.tar.gz
tar xvf sphinxtrain-1.0.8.tar.gz
cd sphinxtrain-1.0.8/
./configure --prefix=$HOME/.simon/sphinxtrain SphinxBase_CFLAGS=-I$HOME/.simon/sphinxbase/include/sphinxbase SphinxBase_LIBS=-L$HOME/.simon/sphinxbase/lib
make
make install


I couldn't sort out OpenCV, so this Simon won't be watching you.


Finally, Simon.
wget http://mirrors.mit.edu/kde/stable/simon/0.4.0/src/simon-0.4.0.tar.bz2
tar xvf simon-0.4.0.tar.bz2
mkdir build/
cd build/
cmake -DCMAKE_INSTALL_PREFIX=~/.simon -DPOCKETSPHINX_LIBRARIES=$HOME/.simon/pocketsphinx/lib/libpocketsphinx.so -DPOCKETSPHINX_INCLUDE_DIR=$HOME/.simon/pocketsphinx/include/ -DSphinxBase_INCLUDE_DIR=$HOME/.simon/sphinxbase/include/ -DSphinxBase_LIBRARY=$HOME/.simon/sphinxbase/lib/libsphinxbase.so -DPJULIUS=$HOME/.simon/julius -DPSPHINX=$HOME/.simon/sphinxbase ../simon-0.4.0/

at which points you should see:
----------------------------------------------------------------------------- -- The following external packages were located on your system. -- This installation will have the extra features provided by these packages. ----------------------------------------------------------------------------- * LibSampleRate - Resampling library * KDE PIM Libs - KDE Libraries for PIM * Sphinxbase - Open source toolkit for speech recognition * PocketSphinx - PocketSphinx is a small-footprint continuous speech recognition system ----------------------------------------------------------------------------- -- The following OPTIONAL packages could NOT be located on your system. -- Consider installing them to enable more features from this software. ----------------------------------------------------------------------------- * qaccessibilityclient KDE client-side accessibility library Required to enable ATSPI plugin. * OpenCV OpenCV (Open Source Computer Vision) is a library of programming functions for real time computer vision Required for Simon Vision -----------------------------------------------------------------------------


And now a bit of a hackjob -- we give the sphinx include dirs above, and while it works fine for most include files (they use e.g. sphinxbase/includeme.h), a number of files (prim_type, sphnx_config.h etc.) point towards the wrong directory (e.g. #include ). Two solutions -- either edit the files or put symmlinks where the files are looking. Symmlinks work well for me.


ln -s ~/.simon/sphinxbase/include/sphinxbase/sphinx_config.h ~/.simon/sphinxbase/include/sphinx_config.h
ln -s ~/.simon/pocketsphinx/include/pocketsphinx/pocketsphinx_export.h ~/.simon/pocketsphinx/include/pocketsphinx_export.h
ln -s ~/.simon/pocketsphinx/include/pocketsphinx/cmdln_macro.h ~/.simon/pocketsphinx/include/cmdln_macro.h
ln -s ~/.simon/pocketsphinx/include/pocketsphinx/ps_lattice.h ~/.simon/pocketsphinx/include/ps_lattice.h
ln -s ~/.simon/pocketsphinx/include/pocketsphinx/ps_mllr.h ~/.simon/pocketsphinx/include/ps_mllr.h
ln -s ~/.simon/pocketsphinx/include/pocketsphinx/fsg_set.h ~/.simon/pocketsphinx/include/fsg_set.h

make
make install
cd ~/.simon/bin
export PATH=$PATH:/$HOME/.simon/julius/bin:$HOME/.simon/sphinxtrain/bin:$HOME/.simon/htk/bin:$HOME/.simon/bin
simond &
simon

I haven't really explored it yet, but here are the first few dialogues:




There's an official video which you may want to watch before you get started:



Issues:
Stuff that doesn't work: Configure Audio from within simon, Configure Acoustic Model from within Simon
Stuff that does work: speech recognition.

Note:
If you set it up 'wrong' the first time around (or it crashed coding samples as above)) and want the wizard to run again, delete all the configuration files:
rm ~/.kde/share/apps/simon* -rf
rm ~/.kde/share/config/simon*

Links to this post:
http://www.forum-raspberrypi.de/Thread-speech-to-text?pid=13465

23 December 2012

300. Briefly: Sharing a folder using SAMBA on Debian

I don't ever use samba, but it's not a bad thing to know how to set up in case you need to share files with someone using Windows in a pinch.

First install samba:
sudo apt-get install samba samba-common smbclient

To get a share up with samba, create an /etc/samba/smb.conf and stick the following in it:


[global]
workgroup=WORKGROUP
guest account=nobody
security=shared


[asharedfolder]
path=/home/lindqvist/shared
guest ok=yes
read only=no
writable=yes
browsable=yes
comment= SMB share

Restart samba:
sudo service samba restart

1. This is an insecure share i.e. <b>anyone can access it</> and edit everything.
2.. Also, by omitting "netbios name=  " you can use the IP address of the server as the hostname, but you could also specify e.g. "netbios name=niobium" and use that as the hostname in nautilus when you connect to the host server.

To set up a user- and password-based share, do



[global]
workgroup=WORKGROUP
security=user


[asharedfolder]
path=/home/lindqvist/shared
guest ok=no
read only=no
writable=yes
browsable=yes
comment= SMB share

You need to add and set the samba password, and enable the linux user you want to give access as well:
sudo smbpasswd -L -a -e lindqvist


There are a lot of other options that can be set. Two of the more interesting ones are probably

[asharedfolder]
createmask=0755
valid users=me myself irene

which means that any new files created in that share via samba gets chmod 755, and only the users me, myself and irene can connect.

But often a basic smb.conf is easier to manage and will do what you want it to.

19 December 2012

296. Building Wine 1.5.19 on Debian (Wheezy/Testing) without errors

UPDATE 16 May 2013: See here for Wine 1.5.30: http://verahill.blogspot.com.au/2013/05/416-wine-1530-in-chroot.html

UPDATE (10th Jan 2013): See here for Wine 1.5.21 using the multiarch approach: http://verahill.blogspot.com.au/2013/01/308-compiling-wine-1521-on-debian.html

This time we'll be cheating and using the carbon-dev.org debian rules to build wine 1.5.19, which works amazingly well with a tiny bit of editing. In spite of what it looks like, this build is easy.



This is also my first encounter with multiarch since libgsm was causing issues (think it's moved out of ia32). Ultimately, multiarch will be the proper way to build 32-bit wine anyway, but I'll deal with that some other day.

Interesting fact: if you try to build with ... 1>build.log 2> build.err the build will fail since files are changing in the main directory during build. Make sure that you do e.g. .. 1> ../build.log 2> ../build.err if you want to track the build.

Finally, I've also built wine 1.5.18 this way, in addition to wine 1.5.19.

The build starts here:

Here's my current best guess at dependencies (Note that multiarch on ubuntu is a little bit different from Debian.):
sudo dpkg --add-architecture i386
sudo apt-get update
sudo apt-get install ia32-libs ia32-libs-dev bison flex gcc libc6-dev libfontconfig-dev libfreetype6-dev libglu-dev libgsm1-dev libice-dev libjpeg-dev libldap-dev libmpg123-dev libncurses5-dev libopenal-dev libpng-dev libsm-dev libssl-dev libusb-dev libx11-dev libxcomposite-dev libxcursor-dev libxext-dev libxi-dev libxinerama-dev libxml2-dev libxrandr-dev libxrender-dev libxslt-dev libxt-dev libxxf86vm-dev make libcapi20-dev liblcms-dev libsane-dev libhal-dev libdbus-1-dev valgrind prelink libcups2-dev opencl-dev lib32opencl1 oss4-dev gettext lib32v4l-dev lib32ncurses5-dev lib32asound2-dev libtiff4-dev libgphoto2-2-dev libxkbfile-dev libxxf86dga-dev freeglut3-dev unixodbc-dev gcc-multilib
sudo apt-get install libgsm1:i386
Reading package lists... Done Building dependency tree Reading state information... Done The following extra packages will be installed: gcc-4.7-base:i386 libc6:i386 libc6-i686:i386 libgcc1:i386 Suggested packages: glibc-doc:i386 locales:i386 The following NEW packages will be installed: gcc-4.7-base:i386 libc6:i386 libc6-i686:i386 libgcc1:i386 libgsm1:i386 0 upgraded, 5 newly installed, 0 to remove and 2 not upgraded. Need to get 5,412 kB of archives. After this operation, 11.9 MB of additional disk space will be used. [..]
Once you've got that sorted, it is time to download the sources:
mkdir -p ~/tmp/wine_1.5.19_carbon/
cd ~/tmp/wine_1.5.19_carbon/
wget http://mirrors.ibiblio.org/wine/source/1.5/wine-1.5.19.tar.bz2 -O wine-unstable_1.5.19.orig.tar.bz2
tar xvf wine-unstable_1.5.19.orig.tar.bz2
cd wine-1.5.19/
wget http://dev.carbon-project.org/debian/wine-unstable/wine-unstable_1.5.5-0.1.debian.tar.bz2
tar xvf wine-unstable_1.5.5-0.1.debian.tar.bz2
rm wine-unstable_1.5.5-0.1.debian.tar.bz2

We could do the editing the proper way, or the quick way. Since I'm not really that familiar with build debian packages the right way (I cheat using checkinstall) we're doing this the quick and dirty way:
sed -i 's/1.5.5/1.5.19/g' debian/changelog
sed -i 's/\-4.5//g' debian/rules
sed -i 's/\-4.5//g' debian/control
sed -i 's/\-4.5//g' debian/control.in

Next, edit debian/patches/series and change it from
  1 debian-changes-from-1.1.32-1.patch
  2 readd_xpm.patch
  3 function_grep.patch
  4 Bug#29669_proposed-fix.patch
  5 fix-winegcc-paths.patch
  6 Bug#28898_squashed-proposed-patches.patch
  7 Bug#28201_proposed-fix-modified.patch
to
  1 debian-changes-from-1.1.32-1.patch
  2 readd_xpm.patch
  3 function_grep.patch

And build:
time dpkg-buildpackage -uc -us

The entire build takes about 40 minutes and gives the following files:
../libwine-alsa-unstable_1.5.19-0.1_amd64.deb
../libwine-dbg-unstable_1.5.19-0.1_amd64.deb
../libwine-ldap-unstable_1.5.19-0.1_amd64.deb
../libwine-sane-unstable_1.5.19-0.1_amd64.deb
../libwine-bin-unstable_1.5.19-0.1_amd64.deb
../libwine-dev-unstable_1.5.19-0.1_amd64.deb
../libwine-openal-unstable_1.5.19-0.1_amd64.deb
../libwine-unstable_1.5.19-0.1_amd64.deb
../libwine-capi-unstable_1.5.19-0.1_amd64.deb
../libwine-gl-unstable_1.5.19-0.1_amd64.deb
../libwine-oss-unstable_1.5.19-0.1_amd64.deb
../wine-bin-unstable_1.5.19-0.1_amd64.deb
../libwine-cms-unstable_1.5.19-0.1_amd64.deb
../libwine-gphoto2-unstable_1.5.19-0.1_amd64.deb
../libwine-print-unstable_1.5.19-0.1_amd64.deb
../wine-unstable_1.5.19-0.1_amd64.deb

To install,
cd ../
sudo dpkg -i *.deb

And you're done.
To confirm what version you're using (and get a screenshot like above) do
winecfg
and click on 'about'.

Links to this post:
http://www.debian-srbija.iz.rs/p/kako-da.html

14 December 2012

291. Another nautilus-open-terminal related bug

We recently had issues with nautilus-open-terminal opening the terminal in $HOME no matter from which directory it was called: http://verahill.blogspot.com.au/2012/11/fix-for-nautilus-open-terminal-opens-in.html

In that post the fix was to do
gsettings set org.gnome.desktop.default-applications.terminal exec gnome-terminal

Which worked fine as long as the exec-arg remained '-x'. It seems to have changed to '-e' now, so during the past few days (since nautilus 3.4.2) 'open terminal here' doesn't open a terminal at all.

Diagnosis:
Looking at this: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=693894


gsettings get org.gnome.desktop.default-applications.terminal exec
'gnome-terminal'
gsettings get org.gnome.desktop.default-applications.terminal exec-arg
'-e'
Fix:
gsettings set org.gnome.desktop.default-applications.terminal exec x-terminal-emulator

It should now work normally.

29 November 2012

287. Compiling Wine 1.5.5 from source using the carbon-dev deb patches

UPDATE 16 May 2013: See here for Wine 1.5.30: http://verahill.blogspot.com.au/2013/05/416-wine-1530-in-chroot.html

UPDATE (10th Jan 2013): See here for Wine 1.5.21 using the multiarch approach: http://verahill.blogspot.com.au/2013/01/308-compiling-wine-1521-on-debian.html
. Use the updated build instead of what follows below.


Here's how to build the wine 1.5.5 packages without any errors.

I'm cheating and using the debian rules from dev.carbon-project.org.-- it's a small file and is a better solution than downloading the deb files from there.

Note that you will probably need more packages than the ones listed here.

1. Download the source and debian build patches

sudo apt-get install bzip2
cd ~/tmp
mkdir wine-1.5.5_carbon/
cd wine-1.5.5_carbon/
wget http://sourceforge.net/projects/wine/files/Source/wine-1.5.5.tar.bz2
mv wine-1.5.5.tar.bz2 wine-unstable_1.5.5.orig.tar.bz2
tar xvf wine-unstable_1.5.5.orig.tar.bz2
cd wine-1.5.5/
wget http://dev.carbon-project.org/debian/wine-unstable/wine-unstable_1.5.5-0.1.debian.tar.bz2
tar xvf wine-unstable_1.5.5-0.1.debian.tar.bz2
rm wine-unstable_1.5.5-0.1.debian.tar.bz2 

2.  Edit control, control.in and rules
Allow any version of gcc (this may obviously backfire if your version is no good)
Edit debian/control and control.in

 27  gcc-4.5-multilib [amd64 kfreebsd-amd64], gcc-4.5 [amd64 kfreebsd-amd64]
 28  g++-4.5 [amd64 kfreebsd-amd64],

so that is says
 27  gcc-multilib [amd64 kfreebsd-amd64], gcc [amd64 kfreebsd-amd64],
 28  g++ [amd64 kfreebsd-amd64],

Edit debian/rules
 53 ifeq ($(DEB_HOST_GNU_CPU),x86_64)
 54 CC=gcc-4.5
 55 CXX=g++-4.5
 56 else
 57 CC=gcc
 58 CXX=g++

to say
 53 ifeq ($(DEB_HOST_GNU_CPU),x86_64)
 54 CC=gcc
 55 CXX=g++
 56 else
 57 CC=gcc
 58 CXX=g++

2. Get dependencies
Note that this list is based on a very old post -- some packages may not be needed.


sudo apt-get install ia32-libs ia32-libs-dev bison flex gcc libc6-dev libfontconfig-dev libfreetype6-dev libglu-dev libgsm1-dev libice-dev libjpeg-dev libldap-dev libmpg123-dev libncurses5-dev libopenal-dev libpng-dev libsm-dev libssl-dev libusb-dev libx11-dev libxcomposite-dev libxcursor-dev libxext-dev libxi-dev libxinerama-dev libxml2-dev libxrandr-dev libxrender-dev libxslt-dev libxt-dev libxxf86vm-dev make libcapi20-dev liblcms-dev libsane-dev libhal-dev libdbus-1-dev valgrind prelink libcups2-dev opencl-dev lib32opencl1 oss4-dev gettext lib32v4l-dev lib32ncurses5-dev lib32asound2-dev libtiff4-dev libgphoto2-2-dev
sudo apt-get install libxkbfile-dev libxxf86dga-dev freeglut3-dev unixodbc-dev gcc-multilib dpkg-dev

Note: I used the following list on debian stable/squeeze.
sudo apt-get install ia32-libs ia32-libs-dev bison flex gcc libc6-dev libfontconfig-dev libfreetype6-dev libglu-dev libgsm1-dev libice-dev libjpeg-dev libldap-dev libmpg123-dev libncurses5-dev libopenal-dev libpng-dev libsm-dev libssl-dev libusb-dev libx11-dev libxcomposite-dev libxcursor-dev libxext-dev libxi-dev libxinerama-dev libxml2-dev libxrandr-dev libxrender-dev libxslt-dev libxt-dev libxxf86vm-dev make libcapi20-dev liblcms-dev libsane-dev libhal-dev libdbus-1-dev valgrind prelink libcups2-dev oss4-dev gettext lib32v4l-dev lib32ncurses5-dev lib32asound2-dev libtiff4-dev libgphoto2-2-dev libxkbfile-dev libxxf86dga-dev freeglut3-dev unixodbc-dev gcc-multilib dpkg-dev debhelper libxmu-dev dctrl-tools lzma sharutils lib32z1-dev libgstreamer-plugins-base0.10-dev
sudo apt-get install lib32opencl1


Then continue as normal.

If you're on Wheezy/Testing you should be following this post instead anyway.

3. Compile
in wine-1.5.5/:
dpkg-buildpackage -us -uc
If all went well you saw:
dpkg-deb: building package `wine-unstable' in `../wine-unstable_1.5.5-0.1_amd64.deb'.
dpkg-deb: building package `wine-bin-unstable' in `../wine-bin-unstable_1.5.5-0.1_amd64.deb'.
dpkg-deb: building package `libwine-dbg-unstable' in `../libwine-dbg-unstable_1.5.5-0.1_amd64.deb'.
dpkg-deb: building package `libwine-dev-unstable' in `../libwine-dev-unstable_1.5.5-0.1_amd64.deb'.
dpkg-deb: building package `libwine-unstable' in `../libwine-unstable_1.5.5-0.1_amd64.deb'.
dpkg-deb: building package `libwine-alsa-unstable' in `../libwine-alsa-unstable_1.5.5-0.1_amd64.deb'.
dpkg-deb: building package `libwine-bin-unstable' in `../libwine-bin-unstable_1.5.5-0.1_amd64.deb'.
dpkg-deb: building package `libwine-capi-unstable' in `../libwine-capi-unstable_1.5.5-0.1_amd64.deb'.
dpkg-deb: building package `libwine-cms-unstable' in `../libwine-cms-unstable_1.5.5-0.1_amd64.deb'.
dpkg-deb: building package `libwine-gl-unstable' in `../libwine-gl-unstable_1.5.5-0.1_amd64.deb'.
dpkg-deb: building package `libwine-gphoto2-unstable' in `../libwine-gphoto2-unstable_1.5.5-0.1_amd64.deb'.
dpkg-deb: building package `libwine-ldap-unstable' in `../libwine-ldap-unstable_1.5.5-0.1_amd64.deb'.
dpkg-deb: building package `libwine-openal-unstable' in `../libwine-openal-unstable_1.5.5-0.1_amd64.deb'.
dpkg-deb: building package `libwine-oss-unstable' in `../libwine-oss-unstable_1.5.5-0.1_amd64.deb'.
dpkg-deb: building package `libwine-print-unstable' in `../libwine-print-unstable_1.5.5-0.1_amd64.deb'.
dpkg-deb: building package `libwine-sane-unstable' in `../libwine-sane-unstable_1.5.5-0.1_amd64.deb'.
[..]
dpkg-source: info: using options from wine-1.5.5/debian/source/options: --compression=bzip2 --compression-level=9
dpkg-source: info: unapplying Bug#28201_proposed-fix-modified.patch
dpkg-source: info: unapplying Bug#28898_squashed-proposed-patches.patch
dpkg-source: info: unapplying fix-winegcc-paths.patch
dpkg-source: info: unapplying Bug#29669_proposed-fix.patch
dpkg-source: info: unapplying function_grep.patch
dpkg-source: info: unapplying readd_xpm.patch
dpkg-source: info: unapplying debian-changes-from-1.1.32-1.patch
dpkg-buildpackage: full upload (original source is included)

Your packages will be found in the wine-1.5.5_carbon/ directory.
../libwine-alsa-unstable_1.5.5-0.1_amd64.deb
../libwine-bin-unstable_1.5.5-0.1_amd64.deb
../libwine-capi-unstable_1.5.5-0.1_amd64.deb
../libwine-cms-unstable_1.5.5-0.1_amd64.deb
../libwine-dbg-unstable_1.5.5-0.1_amd64.deb
../libwine-dev-unstable_1.5.5-0.1_amd64.deb
../libwine-gl-unstable_1.5.5-0.1_amd64.deb
../libwine-gphoto2-unstable_1.5.5-0.1_amd64.deb
../libwine-ldap-unstable_1.5.5-0.1_amd64.deb
../libwine-openal-unstable_1.5.5-0.1_amd64.deb
../libwine-oss-unstable_1.5.5-0.1_amd64.deb
../libwine-print-unstable_1.5.5-0.1_amd64.deb
../libwine-sane-unstable_1.5.5-0.1_amd64.deb
../libwine-unstable_1.5.5-0.1_amd64.deb
../wine-bin-unstable_1.5.5-0.1_amd64.deb
../wine-unstable_1.5.5-0.1_amd64.deb

If something went wrong, it's probably because of a missing dependency.

To install just run
sudo dpkg -i *.deb in the wine-1.5.5_carbon/ directory.

07 November 2012

277. Compiling LSDALTON on debian testing/wheezy

I'm writing this as a separate post even though it's really an integral part of the compilation of Dalton 2011 which I described here: http://verahill.blogspot.com.au/2012/11/compiling-dalton-2011-on-debian.html

LSDALTON supports Open MP which is neat -- it means you can't run across nodes, but it'll automatically take advantage of the resources on the node it's run on.

Anyway.

Assuming you've followed that post, you're now ready to compile LSDALTON.

cd ~/tmp/Dalton2011_release/LSDALTON/
./configure

There are fewer questions this time so I won't list them -- basically, use gfortran and gcc, and compile with OpenMP support.

This gives you a Makefile.config -- edit it as shown. Note that I have the debian libblas3 and libgomp1 packages installed.

ARCH        = linux
FMMDIR      = mm
#
#
CPPFLAGS      = -DSYS_LINUX -D_FILE_OFFSET_BITS=64 -D'INSTALL_BASDIR="/opt/dalton/basis"' -DGFORTRAN=471 -DVAR_LINSCA -DIMPLICIT_NONE
F77            = gfortran
F90            = gfortran
FLNK           = gfortran
CC             = gcc
RM             = rm -f
FFLAGS         = -march=native -O1 -ffast-math -funroll-loops -ftree-vectorize -ffloat-store -fno-whole-file
F90OPTFLAGS    = -march=native -O1 -ffast-math -funroll-loops -ftree-vectorize -ffloat-store -I. -x f95-cpp-input -ffloat-store -fopenmp -fno-whole-file
SAFEFFLAGS     = -march=native -O1 -ffast-math -funroll-loops -ftree-vectorize -ffloat-store -fno-whole-file
CFLAGS         = -march=native -O1 -ffast-math -funroll-loops -ftree-vectorize -std=c99 -DRESTRICT=restrict -DFUNDERSCORE=1 -ffloat-store -DUSE_UNDERSCORES
INCLUDES       =  
LIBS           = -lblas -lgomp 
INSTALLDIR     = /opt/dalton/bin
PDPACK_EXTRAS  = linpack.o eispack.o gp_dlapack.o gp_zlapack.o
AR             = ar
ARFLAGS        =  rvs
#
default : linux.x
#
# Suffix rules C
#
.SUFFIXES : .F90 .f90 .F .o .c

.F90.o:
        $(F90) $(INCLUDES) $(CPPFLAGS) $(F90OPTFLAGS) -c $*.F90

.f90.o:
        $(F90) $(INCLUDES) $(CPPFLAGS) $(F90OPTFLAGS) -c $*.f90

.F.o:
        $(F77) $(INCLUDES) $(CPPFLAGS) $(FFLAGS) -c $*.F

.c.o:
        $(CC) $(INCLUDES) $(CPPFLAGS) $(CFLAGS) -c $*.c

Now compile!
make

Test your installation:
cd test/
 ./TEST all
[..]
-----------------------------------------------------------
TEST ENDED PROPERLY

#####################################################################
                              Summary
#####################################################################

ALL TESTS ENDED PROPERLY!
date and time         : Wed Nov  7 14:57:29 EST 2012

06 November 2012

276. Compiling Dalton 2011 on Debian Testing/Wheezy

UPDATE: To deal with basis sets and 'GPOPEN' errors, see bottom of this post
UPDATE2: Because of the basis set issue the code doesn't run in parallel!
UPDATE 3: All issues are solved by -O0 or -O1. The code now works in parallel and you can define basis sets the usual way. Performance-wise? No idea. So you can compile with -O3 or -O2 but the code doesn't read basis sets the intended way, or you use -O1 or -O0 and it works.

THIS WORKS NOW  :)

Original post:
I've been wanting to use dalton for a long time, but it's been difficult to compile dalton 2.0, and I didn't realise until a few days ago that there's a newer version.
See here for a description of how to compile on ROCKS 5.4.3 (i.e. Centos 5.6) which uses gfortran v 4.1. The main difference between compiling on CentOS 5.6 and Debian Wheezy is in how you edit the Makefile.config. More specifically, compile works a whole lot better with -fno-whole-file and -march=native..

Other than that the steps are the same.

In terms of running, there's an issue with the discoverability of the basis sets which I don't really understand. There's a solution to that at the end of the file.

Before you get started you may want to compile ATLAS as shown here: http://verahill.blogspot.com.au/2012/05/compile-atlas-blas-on-debian-testing.html

Alternatively, you should get the ACML libraries.

NOTE: The compile went without a hitch on my AMD II X3, AMD Phenom II X6 and Intel i5-2400.
My AMF FX 8150 is a trickier story though: it failed to compile with acml libs (gfortran64_fma4_int64) for -O3 and -O2, but compiled with -O1 and -O0. The -O1 binary segfaults though. Never tried the -O0 binary.

WARNING: If you run dalton in parallel it will -- for some reason -- delete your scratch folder when the run is over. The scratch directory is defined in the  /opt/dalton/bin/dalton script (TMPDIR)

License:
First go to http://daltonprogram.org/licence/ and fill out the license agreement. Once that's done you'll get an automated email with a license form, which you should print, sign, scan and email to the email address you're given. Once your form has been processed you'll be sent another email with a user name and password. I received my user name and password the next business day.

Go online and download the source file, Dalton2011_release_v0.tgz, and put it in ~/tmp. Sort out where you want your program to end up
sudo mkdir /share/apps/dalton
sudo chown $USER /share/apps/dalton
mkdir /share/apps/dalton/bin /share/apps/dalton/basis /share/apps/dalton/lsdalton

Next,
cd ~/tmp
tar xvf Dalton2011_release_v0.tgz
cd Dalton2011_release/DALTON
./configure 

and answer all the questions:
./configure

------------------------------------------------------------------
   Configuring the DALTON Makefile.config and "dalton" run script
------------------------------------------------------------------

INFO: Operating system from 'uname -s' : Linux
INFO: Processor type   from 'uname -m' : x86_64
No architecture specified, attempting auto-configuration:
This appears to be a -linux architecture. Is this correct? [Y/n] 
--> Installing DALTON on a -linux computer


Note that 64-bit integers are desirable for Cholesky and very large
scale CI, otherwise the most important effect is that some files will be bigger.

If you choose 64-bit integers, be careful that any system library
routines (incl. MPI) also use 64-bit integers!

Do you want 64-bit integers? [y/N] Do you want to install the program in a parallel MPI version? [Y/n] 
-->WARNING: Makefiles for MPI architecture are difficult to guess
   Please compare the generated Makefile.config with local documentation.

   Checking for Fortran compiler ...
   from this list: mpif90 mpiifort ifort pgf95 pgf90 gfortran g95 

Compiler /usr/bin/mpif90 found, use this compiler? [Y/n] 
-->Compiler mpif90 found and accepted.
Is backend compiler gfortran ? [Y/n] 
   Checking for C compiler ...
   from this list: mpicc  mpiicc   icc ecc pgcc gcc 

Compiler /usr/bin/mpicc found, use this compiler? [Y/n] 
-->Compiler mpicc found and accepted.

Testing existence of libraries in this order:
 libacml.a libmkl.so libmkl_p3.a libatlas.a libblas.a
Directory search list for libraries:
  /opt/ATLAS/lib /home/me/tmp/ATLAS/build/lib /lib /usr/local/lib /usr/lib /usr/local/lib/ATLAS /lib64 /usr/lib64 /usr/local/lib64 

Do you want to replace this with your own directory search list? [y/N] Found /opt/ATLAS/lib/libatlas.a, use it? [Y/n] 
-->The following mathematical library(ies) will be used:
   -L/opt/ATLAS/lib -llapack -llapack -lf77blas -latlas


DALTON uses almost 100 Megabytes of static
allocations, in addition to the dynamic allocation.

DALTON has the possibility to reserve an amount of static memory
for storing two-electron integrals in direct and parallel calculations
Storing some or all of the 2-el. integrals in memory will speed up
direct and parallel calculations (and in particular the latter).
NOTE: This will increase the static memory allocation used by DALTON

Would you like to activate the possibility of storing 2-el.int. in memory? [y/N] How many MB to use for storing 2-el. integrals? 
-->Program will be installed with 300 MB (39000000 words) used for storing 2-el. integrals

Maximum amount of work memory for dynamic allocations can be changed
at run time with the environment variable WRKMEM (in REAL*8 words = megabytes/8)
or by using the -M option to the run script: "dalton -M mb ..." (in megabytes).
We recommend at least 200 MB work memory,
larger for correlated calculations, but it should for maximum
efficiency NOT exceed available physical memory per CPU in parallel calculations.

How many MB to use as default for work memory (hit return for default of 1000 MB)? 
-->Program will be installed with a default work memory of 3900 MB (511000000 words)

-->Current directory is /home/me/tmp/Dalton2011_release/DALTON

Use default ../bin as installation directory for DALTON binaries and scripts? [Y/n] Please enter another installation directory: 
-->DALTON executable and script will be placed in /opt/dalton/bin directory


-->Default basis set directory will be /home/me/tmp/Dalton2011_release/DALTON/../basis/

Use this directory as default basis set directory? [Y/n] 
Please choose another default basis set directory (must end with /) 
-->Default basis set directory will be /opt/dalton/basis/


-->Job specific directories under $SCRATCH/$USER
-->will be used for temporary files when running DALTON

Use SCRATCH=/work as default root scratch space in "dalton" run script? [Y/n] Please enter default root scratch directory: 
-->Creating Makefile.config ...
gfortran version 471 prc=x86_64
INFO: Compiling with 32-bit integers.
INFO: Make sure pre-compiled BLAS, MPI etc. libraries are also with 32-bit integers!!!

Proper 64-bit file access detected.

-->Creating the DALTON run-script in /opt/dalton/bin

   The configuration of DALTON has finished succesfully.
   Check compiler flags etc. in Makefile.config and run "make" to get executable.

which generates Makefile.config. Edit it and

  • change the -march to native. 
  • add -fno-whole-file to avoid internal compiler errors
  • change optimisation level to -O1 (O0 is ok, O2 and O3 give GPOPEN problems)

Like this:

ARCH        = linux
#
#
CPPFLAGS      = -DVAR_GFORTRAN -DSYS_LINUX -DVAR_MFDS -D'INSTALL_WRKMEM=131000000' -D'INSTALL_MMWORK=65000000' -D_FILE_OFFSET_BITS=64 -DVAR_MPI -DGFORTRAN=471 -DIMPLICIT_NONE
F90           = mpif90
CC            = mpicc
LOADER        = mpif90
RM            = rm -f
FFLAGS        = -march=native -O1 -ffast-math -funroll-loops -ftree-vectorize -fbacktrace -fno-whole-file
SAFEFFLAGS    = -march=native -O1 -ffast-math -funroll-loops -ftree-vectorize -fbacktrace -fno-whole-file
CFLAGS        = -march=native -O1 -ffast-math -funroll-loops -ftree-vectorize -std=c99 -DRESTRICT=restrict -DFUNDERSCORE=1
INCLUDES      = -I../include 
MODULES       = -J../modules
LIBS          = -L/opt/ATLAS/lib -llapack -llapack -lf77blas -latlas 
INSTALLDIR    = /opt/dalton/bin
PDPACK_EXTRAS = linpack.o eispack.o gp_zlapack.o gp_dlapack.o
GP_EXTRAS     = 
AR            = ar
ARFLAGS       = rvs
# flags for ftnchek on Dalton /hjaaj
CHEKFLAGS  = -nopure -nopretty -nocommon -nousage -noarray -notruncation -quiet  -noargumants -arguments=number  -usage=var-unitialized
# -usage=var-unitialized:arg-const-modified:arg-alias
# -usage=var-unitialized:var-set-unused:arg-unused:arg-const-modified:arg-alias
#
default : dalton linuxparallel.x
SAFE_FFLAGS_for_ifort = $(FFLAGS)
#
# Parallel initialization
#
MPI_INCLUDE_DIR = 
MPI_LIB_PATH    = -L/usr/lib
MPI_LIB         = -lmpi
#
#
# Suffix rules
# hjaaj Oct 04: .g is a "cheat" suffix, for debugging.
#               'make x.g' will create x.o from x.F or x.c with -g debug flag set.
#
.SUFFIXES : .F .F90 .c .o .i .g .s

.F.o:
        $(F90) $(INCLUDES) $(MODULES) $(CPPFLAGS) $(FFLAGS) -c $*.F 

.F.i:
        $(F90) $(INCLUDES) $(MODULES) $(CPPFLAGS) -E $*.F > $*.i

.F.g:
        $(F90) $(INCLUDES) $(MODULES) $(CPPFLAGS) $(SAFEFFLAGS) -g -c $*.F 

.F.s:
        $(F90) $(INCLUDES) $(MODULES) $(CPPFLAGS) $(FFLAGS) -S -g -c $*.F 

.F90.o:
        $(F90) $(INCLUDES) $(MODULES) $(CPPFLAGS) $(FFLAGS) -c $*.F90 

.F90.i:
        $(F90) $(INCLUDES) $(MODULES) $(CPPFLAGS) -E $*.F90 > $*.i




make
make install

Now just copy the basis sets and ecp data to the proper location:
cd ../
cp basis/* -R /opt/dalton/basis

and edit your ~/.bashrc;
export PATH=$PATH:/opt/dalton/bin

And you should be good to go.


So far I haven't run all the tests, but
./TEST -dalton /opt/dalton/bin/dalton short

gave
#####################################################################
                              Summary
#####################################################################

ALL TESTS ENDED PROPERLY!

date and time         : Wed Nov  7 11:57:02 EST 2012



GPOPEN errors and how to get around them.

To make the story short: if you use -O3 or -O2 for some reason Dalton can't find the basis sets if you declare them the normal way (-O0 and -O1 take care of the problem). However, using ATOMBASIS it works.

Here's an example. Typically you'd specify the basis set for a whole molecule in your .mol file:

BASIS
STO-3G
DFT PROPERTIES TEST 
This doesn't work with O3
AtomTypes=2 Angstrom
        8.    1     
O -0.141254 0.0998816 0.00000
        1.    2     
H 0.589315 0.718039 0.00000
H -0.922641 0.652406 0.00000

but that leads to errors on the debian (but not centos) builds:
   0: Directories for basis set searches:
     /jobs/dalton:/opt/dalton/basis

 MPI node no.:     0
 Reason: ERROR (GPOPEN) UPON OPENING A FILE


 Node      0:  --- SEVERE ERROR, PROGRAM WILL BE ABORTED ---
 ERROR (GPOPEN) UPON OPENING A FILE

and
  Atomic type no.    1
  --------------------
  Nuclear charge:   8.00000
  Number of symmetry independent centers:    1
  Number of basis sets to read;    2
  Basis set file used for this atomic type with Z =   8 :
     "/opt/dalton/basis/                                                                                "


--> ERROR (GPOPEN) UPON TRYING TO OPEN FILE ON UNIT 11
--> with filename /opt/dalton/basis/
--> IOSTAT ERROR CODE RETURNED      21


 QTRACE dump of internal trace stack

 ========================
      level    module
 ========================
          7    GPOPEN      
          6    BASLIB      
          5    READ_MOL    
          4    READIN      
          3    HERMIT      
          2    DALTON      
          1    DALTON main 

whereas

ATOMBASIS
DFT PROPERTIES TEST 
This works with O3
AtomTypes=2 Angstrom
        8.    1    basis=STO-3G 
O -0.141254 0.0998816 0.00000
        1.    2     basis=STO-3G
H 0.589315 0.718039 0.00000
H -0.922641 0.652406 0.00000

works and gives
   0: Directories for basis set searches:
     /opt/dalton/basis:/opt/dalton/basis

 NOTE:    1 informational messages have been issued.
 Check output, result, and error files for "INFO".

and a normal exit:
 CPU time statistics for ABACUS
 ------------------------------

 LINRES     00:00:02      77 %

 TOTAL      00:00:03     100 %


 >>>> Total CPU  time used in ABACUS:   3.21 seconds
 >>>> Total wall time used in ABACUS:   3.22 seconds


                   .-------------------------------------------.
                   | End of Static Property Section (ABACUS) - |
                   `-------------------------------------------'

 >>>> Total CPU  time used in DALTON:   6.04 seconds
 >>>> Total wall time used in DALTON:   6.06 seconds

 
     Date and time (Linux)  : Tue Nov  6 14:54:24 2012
     Host name              : beryllium   

02 November 2012

274. Kernel 3.7-rc3

There's a specific driver (Silicom devices) included in the new kernel (3.7) that interests a friend. I do not suggest people in general run an -rc kernel, so I'm not going to put any effort into making this post user-friendly.

Having said that, in general you should be fine. In general.

sudo apt-get install kernel-package fakeroot build-essential
mkdir ~/tmp
cd ~/tmp
wget http://www.kernel.org/pub/linux/kernel/v3.0/testing/linux-3.7-rc3.tar.bz2
tar xvf linux-3.7-rc3.tar.bz2 
cd linux-3.7-rc3/
cat /boot/config-`uname -r`>.config
make oldconfig

At this stage you'll be asked questions about all the shiny new things that are found in the kernel. See bottom of the post for a list of the questions.

If you regret the answer to a question you can change your mind later by doing
make menuconfig

Once you're done answering questions, compile!
make-kpkg clean
time fakeroot make-kpkg -j3 --initrd --revision=3.7.0 --append-to-version=-rc3 kernel_image kernel_headers

It takes 43 minutes on my old AMD X3.



To install the kernel:
mv ../linux-*-3.7.0-rc3*.deb .
sudo dpkg -i *.deb

And you're done!

New stuff (compared to kernel 3.6.3):

* CPU/Task time and stats accounting
*
Cputime accounting
> 1. Simple tick based cputime accounting (TICK_CPU_ACCOUNTING) (NEW)
  2. Fine granularity task level IRQ time accounting (IRQ_TIME_ACCOUNTING)
choice[1-2]: 1

Consider userspace as in RCU extended quiescent state (RCU_USER_QS) [N/y/?] (NEW) 
Module signature verification (MODULE_SIG) [N/y/?] (NEW)  
Legacy cpb sysfs knob support for AMD CPUs (X86_ACPI_CPUFREQ_CPB) [Y/n/?] (NEW) 
Packet: sockets monitoring interface (PACKET_DIAG) [N/m/y/?] (NEW)
IPv6: GRE tunnel (IPV6_GRE) [N/m/y/?] (NEW) 
IPv4 NAT (NF_NAT_IPV4) [N/m/?] (NEW) 
IPv6 NAT (NF_NAT_IPV6) [N/m/?] (NEW) 
OMAP OCP2SCP DRIVER (OMAP_OCP2SCP) [N/m/y/?] (NEW) 
Maximum expected bad eraseblock count per 1024 eraseblocks (MTD_UBI_BEB_LIMIT) [20] (NEW)
UBI Fastmap (Experimental feature) (MTD_UBI_FASTMAP) [N/y/?] (NEW)
Calxeda Highbank SATA support (SATA_HIGHBANK) [N/m/?] (NEW) 
Virtual eXtensible Local Area Network (VXLAN) (VXLAN) [N/m/y/?] (NEW) Y
PCH PTP clock support (PCH_PTP) [N/y/?] (NEW)
Solarflare SFC9000-family PTP support (SFC_PTP) [Y/n/?] (NEW)
Drivers for Atheros AT803X PHYs (AT803X_PHY) [N/m/?] (NEW)
MAX310X support (SERIAL_MAX310X) [N/y/?] (NEW)
SCCNXP serial port support (SERIAL_SCCNXP) [N/m/y/?] (NEW)
TPM HW Random Number Generator support (HW_RANDOM_TPM) [M/n/?] (NEW) 
TPM Interface Specification 1.2 Interface (I2C - Infineon) (TCG_TIS_I2C_INFINEON) [N/m/?] (NEW) 
NXP SC18IS602/602B/603 I2C to SPI bridge (SPI_SC18IS602) [N/m/?] (NEW)
OMAP HDQ driver (HDQ_MASTER_OMAP) [N/m/?] (NEW)
Analog Devices ADT7410 (SENSORS_ADT7410) [N/m/?] (NEW)
Maxim MAX197 and compatibles (SENSORS_MAX197) [N/m/y/?] (NEW)
generic cpu cooling support (CPU_THERMAL) [N/y/?] (NEW) Y
Fairchild FAN53555 Regulator (REGULATOR_FAN53555) [N/m/?] (NEW)
Media USB Adapters (MEDIA_USB_SUPPORT) [N/y/?] (NEW)
Media PCI Adapters (MEDIA_PCI_SUPPORT) [N/y/?] (NEW) 
Media test drivers (V4L_TEST_DRIVERS) [N/y] (NEW) 
ISA and parallel port devices (MEDIA_PARPORT_SUPPORT) [N/y/?] (NEW) 
Autoselect tuners and i2c modules to build (MEDIA_SUBDRV_AUTOSELECT) [Y/n/?] (NEW) 
Maximum debug level (NOUVEAU_DEBUG) [5] (NEW)
Default debug level (NOUVEAU_DEBUG_DEFAULT) [3] (NEW)
Backlight Driver for LM3630 (BACKLIGHT_LM3630) [N/m/?] (NEW)
Backlight Driver for LM3639 (BACKLIGHT_LM3639) [N/m/?] (NEW)
Sony PS3 BD Remote Control (HID_PS3REMOTE) [N/m/?] (NEW)
HID Sensors framework support (HID_SENSOR_HUB) [N/m/?] (NEW)
ZTE USB serial driver (USB_SERIAL_ZTE) [N/m/?] (NEW)
Functions for loading firmware on EZUSB chips (USB_EZUSB_FX2) [M/y/?] (NEW) 
OMAP USB2 PHY Driver (OMAP_USB2) [N/m/y/?] (NEW)
LED support for LM3642 Chip (LEDS_LM3642) [N/m/?] (NEW)
LED support for LM355x Chips, LM3554 and LM3556 (LEDS_LM355x) [N/m/?] (NEW)
LED CPU Trigger (LEDS_TRIGGER_CPU) [N/y/?] (NEW)
Dallas DS2404 (RTC_DRV_DS2404) [N/m/y/?] (NEW) 
Silicom devices (NET_VENDOR_SILICOM) [Y/n/?] (NEW) Y
Silicom BypassCTL library support (SBYPASS) [N/m/?] (NEW)m
Silicom BypassCTL net support (BPCTL) [N/m/?] (NEW) m
Cambridge Electronic Design 1401 USB support (CED1401) [N/m/?] (NEW)
Digi Realport driver (DGRP) [N/m/y/?] (NEW)
STE-Modem remoteproc support (STE_MODEM_RPROC) [N/m/y/?] (NEW)
SMB2 network file system support (EXPERIMENTAL) (CIFS_SMB2) [N/y/?] (NEW)
Red-Black tree test (RBTREE_TEST) [N/m/?] (NEW)
Interval tree test (INTERVAL_TREE_TEST) [N/m/?] (NEW)
CAST5 (CAST-128) cipher algorithm (x86_64/AVX) (CRYPTO_CAST5_AVX_X86_64) [N/m/y/?] (NEW)
CAST6 (CAST-256) cipher algorithm (x86_64/AVX) (CRYPTO_CAST6_AVX_X86_64) [N/m/y/?] (NEW)
Asymmetric (public-key cryptographic) key type (ASYMMETRIC_KEY_TYPE) [N/m/y/?] (NEW)

As for Intel devices the pre-existing settings were:

Intel devices (NET_VENDOR_INTEL) [Y/n/?] y
      Intel(R) PRO/100+ support (E100) [M/n/y/?] m
      Intel(R) PRO/1000 Gigabit Ethernet support (E1000) [M/n/y/?] m
      Intel(R) PRO/1000 PCI-Express Gigabit Ethernet support (E1000E) [M/n/y/?] m
      Intel(R) 82575/82576 PCI-Express Gigabit Ethernet support (IGB) [M/n/y/?] m
        Direct Cache Access (DCA) Support (IGB_DCA) [Y/n/?] y
        PTP Hardware Clock (PHC) (IGB_PTP) [N/y/?] n
      Intel(R) 82576 Virtual Function Ethernet support (IGBVF) [M/n/y/?] m
      Intel(R) PRO/10GbE support (IXGB) [M/n/y/?] m
      Intel(R) 10GbE PCI Express adapters support (IXGBE) [M/n/y/?] m
        Intel(R) 10GbE PCI Express adapters HWMON support (IXGBE_HWMON) [Y/n/?] y
        Direct Cache Access (DCA) Support (IXGBE_DCA) [Y/n/?] y
        Data Center Bridging (DCB) Support (IXGBE_DCB) [Y/n/?] y
        PTP Clock Support (IXGBE_PTP) [N/y/?] n
      Intel(R) 82599 Virtual Function Ethernet support (IXGBEVF) [M/n/y/?] m
      Intel (82586/82593/82596) devices (NET_VENDOR_I825XX) [Y/n/?] y
        Zenith Z-Note support (EXPERIMENTAL) (ZNET) [N/m/y/?] n

28 October 2012

268. Compiling and testing GAMESS US on debian testing (wheezy)

Update 3: 9 May 2013. Fixed a couple of mistakes e.g. related to mpi. I also switched from ATLAS to acml -- when I build with ATLAS a lot of the example inputs do not converge.

Update 2: Pietro (see posts below) identified some odd behaviour when running test exam44 in which the scf failed to converge. The (temporary) fix for that has been included in the instructions below (change line 1664 in the file 'comp') -- most likely it's only a single file which needs to be compiled with -O0, but it will take a while to identify which one that is. Having to use -O0 on a performance critical piece of software is obviously unfortunate.

Update:
I've done this on ROCKS 5.4.3/ Centos 5.6 as well. Be aware that because of the ancient version of gfortran (4.1.2) on ROCKS there will be some limitations:

   Alas, your version of gfortran does not support REAL*16,
   so relativistic integrals cannot use quadruple precision.
   Other than this, everything will work properly.
Other than that, follow the instructions below (including editing lked)

Original post:
Solvation energies using implicit solvation is a tough nut to crack. I like working with NWChem, but there's only one solvation model (COSMO) implemented, it has had a history of giving results which are wildly different (>20 kcal/mol! It's fixed now -- using b3lyp/6-311++g** with the cosmo parameters in that post I got 63.68 kcal/mol for Cl-) from that of other software packages (partly due to a bug which was fixed in 2011), and I'm still not sure how to properly use the COSMO module (is rsolv 0 a reasonable value?). Obviously, my own unfamiliarity with the method is another issue, but that's where the idea of sane defaults come in. So, time to test and compare with other models. Reading Cramer, C. J.; Truhlar, D. G. A Acc. Chem. Res. 2008, 41, 760–768 got me interested in GAMESS US again.

Gaussian is not really an attractive option for me anymore for performance reasons (caveat: as seen by me on my particular systems using precompiled binaries). Free (source code + cost) is obviously also always attractive. Being a linux sort of person also plays into it.

So, here's how to get your cluster set up for gamess US:
1. Go to http://www.msg.chem.iastate.edu/GAMES S/download/register/
Select agree, then pick your version -- in my case
GAMESS version May 1, 2012 R1 for 64 bit (x86_64 compatible) under Linux with gnu compilers

Once you've completed your order you're told you may have to wait for up to a week before your registration is approved, but I got approved in less than 24 hours.

[2. Register for GAMESSPLUS at http://comp.chem.umn.edu/license/form-user.html
Again, it may take a little while to get approved -- in my case it was less than 24 hours. Also, it seems that you don't need a separate GAMESSPLUS anymore]

3. Download gamess-current.tar.gz as per the instructions and put it in /opt/gamess (once you've created the folder)

4. If you're using AMD you're in luck -- set up acml on your system. In my case I put everything in /opt/acml/acml5.2.0

I've had bad luck with ATLAS.

5. Compile
sudo apt-get install build-essential gfortran openmpi-bin libopenmpi-dev libboost-all-dev
sudo mkdir /opt/gamess
sudo chown $USER /opt/gamess
cd /opt/gamess
tar xvf gamess-current.tar.gz 
cd gamess/

You're now ready to autoconfigure.

The lengthy autoconfigure.
 Note that
* the location of your openmpi libs may vary -- the debian libs are put in /usr/bin/openmpi/lib by default, but I'm using my own compiled version which I've put in /opt/openmpi
* gamess is linked against the static libraries by default, so if you compiled atlas as is described elsewhere on this blog, you'll be fine.

./config
This script asks a few questions, depending on your computer system,
to set up compiler names, libraries, message passing libraries,
and so forth.
 
You can quit at any time by pressing control-C, and then .
 
Please open a second window by logging into your target machine,
in case this script asks you to 'type' a command to learn something
about your system software situation.  All such extra questions will
use the word 'type' to indicate it is a command for the other window.
 
After the new window is open, please hit  to go on.

   GAMESS can compile on the following 32 bit or 64 bit machines:
axp64    - Alpha chip, native compiler, running Tru64 or Linux
cray-xt  - Cray's massively parallel system, running CNL
hpux32   - HP PA-RISC chips (old models only), running HP-UX
hpux64   - HP Intel or PA-RISC chips, running HP-UX
ibm32    - IBM (old models only), running AIX
ibm64    - IBM, Power3 chip or newer, running AIX or Linux
ibm64-sp - IBM SP parallel system, running AIX
ibm-bg   - IBM Blue Gene (P or L model), these are 32 bit systems
linux32  - Linux (any 32 bit distribution), for x86 (old systems only)
linux64  - Linux (any 64 bit distribution), for x86_64 or ia64 chips
           AMD/Intel chip Linux machines are sold by many companies
mac32    - Apple Mac, any chip, running OS X 10.4 or older
mac64    - Apple Mac, any chip, running OS X 10.5 or newer
sgi32    - Silicon Graphics Inc., MIPS chip only, running Irix
sgi64    - Silicon Graphics Inc., MIPS chip only, running Irix
sun32    - Sun ultraSPARC chips (old models only), running Solaris
sun64    - Sun ultraSPARC or Opteron chips, running Solaris
win32    - Windows 32-bit (Windows XP, Vista, 7, Compute Cluster, HPC Edition)
win64    - Windows 64-bit (Windows XP, Vista, 7, Compute Cluster, HPC Edition)
winazure - Windows Azure Cloud Platform running Windows 64-bit
    type 'uname -a' to partially clarify your computer's flavor.
please enter your target machine name: linux64

Where is the GAMESS software on your system?
A typical response might be /u1/mike/gamess,
most probably the correct answer is /home/me/tmp/gamess
 
GAMESS directory? [/opt/gamess] /opt/gamess

Setting up GAMESS compile and link for GMS_TARGET=linux64
GAMESS software is located at GMS_PATH=/home/me/tmp/gamess
 
Please provide the name of the build locaation.
This may be the same location as the GAMESS directory.
 
GAMESS build directory? [/opt/gamess] /opt/gamess

Please provide a version number for the GAMESS executable.
This will be used as the middle part of the binary's name,
for example: gamess.00.x

Version? [00] 12r1

Linux offers many choices for FORTRAN compilers, including the GNU
compiler set ('g77' in old versions of Linux, or 'gfortran' in
current versions), which are included for free in Unix distributions.
 
There are also commercial compilers, namely Intel's 'ifort',
Portland Group's 'pgfortran', and Pathscale's 'pathf90'.  The last
two are not common, and aren't as well tested as the others.
 
type 'rpm -aq | grep gcc' to check on all GNU compilers, including gcc
type 'which gfortran'  to look for GNU's gfortran (a very good choice),
type 'which g77'       to look for GNU's g77,
type 'which ifort'     to look for Intel's compiler,
type 'which pgfortran' to look for Portland Group's compiler,
type 'which pathf90'   to look for Pathscale's compiler.
Please enter your choice of FORTRAN: gfortran

gfortran is very robust, so this is a wise choice.

Please type 'gfortran -dumpversion' or else 'gfortran -v' to
detect the version number of your gfortran.
This reply should be a string with at least two decimal points,
such as 4.1.2 or 4.6.1, or maybe even 4.4.2-12.
The reply may be labeled as a 'gcc' version,
but it is really your gfortran version.
Please enter only the first decimal place, such as 4.1 or 4.6:  
4.6

   Good, the newest gfortran can compile REAL*16 data type.
hit <return> to continue to the math library setup.

Linux distributions do not include a standard math library.
 
There are several reasonable add-on library choices,
       MKL from Intel           for 32 or 64 bit Linux (very fast)
      ACML from AMD             for 32 or 64 bit Linux (free)
     ATLAS from www.rpmfind.net for 32 or 64 bit Linux (free)
and one very unreasonable option, namely 'none', which will use
some slow FORTRAN routines supplied with GAMESS.  Choosing 'none'
will run MP2 jobs 2x slower, or CCSD(T) jobs 5x slower.
 
Some typical places (but not the only ones) to find math libraries are
Type 'ls /opt/intel/mkl'                 to look for MKL
Type 'ls /opt/intel/Compiler/mkl'        to look for MKL
Type 'ls /opt/intel/composerxe/mkl'      to look for MKL
Type 'ls -d /opt/acml*'                  to look for ACML
Type 'ls -d /usr/local/acml*'            to look for ACML
Type 'ls /usr/lib64/atlas'               to look for Atlas
 
Enter your choice of 'mkl' or 'atlas' or 'acml' or 'none': acml

Type 'ls -d /opt/acml*' or 'ls -d /usr/local/acml*'
and note the the full path, which includes a version number.
enter this full pathname: /opt/acml/acml5.2.0
Math library 'acml' will be taken from /opt/acml/acml5.2.0/gfortran64_int64/lib
please hit <return> to compile the GAMESS source code activator
gfortran -o /home/me/tmp/gamess/build/tools/actvte.x actvte.f
unset echo
Source code activator was successfully compiled.
 
please hit  to set up your network for Linux clusters.

If you have a slow network, like Gigabit Ethernet (GE), or
if you have so few nodes you won't run extensively in parallel, or
if you have no MPI library installed, or
if you want a fail-safe compile/link and easy execution,
     choose 'sockets'
to use good old reliable standard TCP/IP networking.
 
If you have an expensive but fast network like Infiniband (IB), and
if you have an MPI library correctly installed,
     choose 'mpi'.
 
communication library ('sockets' or 'mpi')? mpi

The MPI libraries which work well on linux64/Infiniband are
      Intel's MPI (impi)
      MVAPICH2
      SGI's mpt from ProPack, on Altix/ICE systems
Other libraries may work, please see 'readme.ddi' for info.
The choices listed above will compile and link easily,
and are known to run correctly and efficiently.

Enter 'sockets' if you just changed your mind about trying MPI.

Enter MPI library (impi, mvapich2, mpt, sockets): openmpi
MPI can be installed in many places, so let's find openmpi.
The person who installed your MPI can tell you where it really is.
 
impi     is probably located at a directory like
              /opt/intel/impi/3.2
              /opt/intel/impi/4.0.1.007
              /opt/intel/impi/4.0.2.003
         include iMPI's version numbers in your reply below.
mvapich2 could be almost anywhere, perhaps some directory like
              /usr/mpi/gcc/mvapich2-1.6
openmpi  could be almost anywhere, perhaps some directory like
              /usr/mpi/openmpi-1.4.3
mpt      is probably located at a directory like
              /opt/sgi/mpt/mpt-1.26
Please enter your openmpi's location: /opt/openmpi/1.6

Your configuration for GAMESS compilation is now in
     /home/me/tmp/gamess/build/install.info
Now, please follow the directions in
     /opt/gamess/machines/readme.unix



I next did this:

cd /opt/gamess/ddi
./compddi
cd ../

Edit the file 'comp' and change it from

1664       set OPT='-O2'
to
1664       set OPT='-O0'

or test case exam44.inp in tests/standard will fail due to lack of SCF convergence. (I've tried -O1 as well with no luck)

Continue your compilation:
./compall

Running 'compall' reads "install.info" which I include below:
#!/bin/csh
#   compilation configuration for GAMESS
#   generated on beryllium
#   generated at Friday 21 September  08:48:09 EST 2012
setenv GMS_PATH            /opt/gamess
setenv GMS_BUILD_DIR       /opt/gamess
#         machine type
setenv GMS_TARGET          linux64
#         FORTRAN compiler setup
setenv GMS_FORTRAN         gfortran
setenv GMS_GFORTRAN_VERNO  4.6
#         mathematical library setup
setenv GMS_MATHLIB         acml
setenv GMS_MATHLIB_PATH    /opt/acml/acml5.2.0//gfortran64_int64/lib
#         parallel message passing model setup
setenv GMS_DDI_COMM        mpi
setenv GMS_MPI_LIB         openmpi
setenv GMS_MPI_PATH        /opt/openmpi/1.6

Note that you can't change the gfortran version here either -- 4.7 won't be recognised.

Anyway, compilation will take a while -- enough for some coffee and reading.

In the next step you may have problems with openmpi -- lked looks in e.g. /opt/openmpi/1.6/lib64 but you'll probably only have /opt/openmpi/1.6/lib

Edit lked and change
 958             case openmpi:
 959                set MPILIBS="-L$GMS_MPI_PATH/lib64"
 960                set MPILIBS="$MPILIBS -lmpi"
 961                breaksw
to
 958             case openmpi:
 959                set MPILIBS="-L$GMS_MPI_PATH/lib"
 960                set MPILIBS="$MPILIBS -lmpi -lpthread"
 961                breaksw



Generate the runtime file:
./lked gamess 12r1 >&  lked.log

Done!


To compile with openblas:
1. edit install.info
#!/bin/csh
#   compilation configuration for GAMESS
#   generated on tantalum
#   generated at Friday 21 September  14:01:54 EST 2012
setenv GMS_PATH            /opt/gamess
setenv GMS_BUILD_DIR       /opt/gamess
#         machine type
setenv GMS_TARGET          linux64
#         FORTRAN compiler setup
setenv GMS_FORTRAN         gfortran
setenv GMS_GFORTRAN_VERNO  4.6
#         mathematical library setup
setenv GMS_MATHLIB         openblas
setenv GMS_MATHLIB_PATH    /opt/openblas/lib
#         parallel message passing model setup
setenv GMS_DDI_COMM        mpi
setenv GMS_MPI_LIB         openmpi
setenv GMS_MPI_PATH        /opt/openmpi/1.6

2. edit lked
Add lines 462-466 which sets up the openblas switch.

 453       endif
 454       set BLAS=' '
 455       breaksw
 456 
 457    case acml:
 458       #     do a static link so that only compile node needs to install ACML
 459       set MATHLIBS="$GMS_MATHLIB_PATH/libacml.a"
 460       set BLAS=' '
 461       breaksw
 462 case openblas:
 463        #     do a static link so that only compile node needs to install openblas
 464        set MATHLIBS="$GMS_MATHLIB_PATH/libopenblas.a"
 465        set BLAS=' '
 466        breaksw
 467 
 468    case none:
 469    default:
 470       echo "Warning.  No math library was found, you should install one."
 471       echo "    MP2 calculations speed up about 2x with a math library."
 472       echo "CCSD(T) calculations speed up about 5x with a math library."
 473       set BLAS='blas.o'
 474       set MATHLIBS=' '
 475       breaksw

3. Link
./lked gamess 12r2 >&  lked.log

You now have gamess.12r1.x which uses ATLAS, and gamess.12r2.x which uses openblas.

To run:
The rungms file was a bit too 'clever' for me, so I boiled it down to a file called gmrun which made executable (chmod +X gmrun) and put in /opt/gamess:

#!/bin/csh
set TARGET=mpi
set SCR=$HOME/scratch
set USERSCR=/scratch
set GMSPATH=/opt/gamess
set JOB=$1
set VERNO=$2
set NCPUS=$3

if ( $JOB:r.inp == $JOB ) set JOB=$JOB:r
echo "Copying input file $JOB.inp to your run's scratch directory..."
cp $JOB.inp $SCR/$JOB.F05

setenv TRAJECT $USERSCR/$JOB.trj
setenv RESTART $USERSCR/$JOB.rst
setenv INPUT $SCR/$JOB.F05
setenv PUNCH $USERSCR/$JOB.dat
if ( -e $TRAJECT ) rm $TRAJECT
if ( -e  $PUNCH ) rm $PUNCH
if ( -e  $RESTART ) rm $RESTART
source $GMSPATH/gms-files.csh

setenv LD_LIBRARY_PATH /opt/openmpi/1.6/lib:$LD_LIBRARY_PATH
set path= ( /opt/openmpi/1.6/bin $path )
/opt/openmpi/1.6/bin/mpiexec -n $NCPUS $GMSPATH/gamess.$VERNO.x|tee $JOB.out
cp $PUNCH .

Note that I actually do have two scratch directories -- one ~/scratch and one in /scratch. Note that the SCR directory should be local to the node as well as spacious, while USERSCR can be a networked,smaller directory.

Finally do
echo 'export PATH=$PATH:/opt/gamess' >> ~/.bashrc

Anyway.
Navigate to your tests/standard folder where there's a lot of exam*.inp files and do

gmrun exam12 12r1 4

where exam01 (or exam01.inp) is the name of the input file, 12r1 is the version number (that you set above) and 4 is the number of processors/threads .
 
          ---------------------
          ELECTROSTATIC MOMENTS
          ---------------------

 POINT   1           X           Y           Z (BOHR)    CHARGE
                 0.000000   -0.000000    0.000000       -0.00 (A.U.)
         DX          DY          DZ         /D/  (DEBYE)
    -0.000000    0.000000   -0.000000    0.000000
 ...... END OF PROPERTY EVALUATION ......
 CPU     0: STEP CPU TIME=     0.02 TOTAL CPU TIME=        2.2 (    0.0 MIN)
 TOTAL WALL CLOCK TIME=        2.3 SECONDS, CPU UTILIZATION IS  97.78%
  $VIB   
          IVIB=   0 IATOM=   0 ICOORD=   0 E=      -76.5841347569
 -6.175208802E-40-6.175208802E-40-4.411868660E-07 6.175208802E-40 6.175208802E-40
  4.411868660E-07-1.441225933E-40-1.441225933E-40 1.672333111E-06 1.441225933E-40
  1.441225933E-40-1.672333111E-06
 -4.053383177E-34 4.053383177E-34-2.257541709E-15
 ......END OF GEOMETRY SEARCH......
 CPU     0: STEP CPU TIME=     0.00 TOTAL CPU TIME=        2.2 (    0.0 MIN)
 TOTAL WALL CLOCK TIME=        2.3 SECONDS, CPU UTILIZATION IS  97.35%
               990473  WORDS OF DYNAMIC MEMORY USED
 EXECUTION OF GAMESS TERMINATED NORMALLY Fri Sep 21 14:27:17 2012
 DDI: 263624 bytes (0.3 MB / 0 MWords) used by master data server.

 ----------------------------------------
 CPU timing information for all processes
 ========================================
 0: 2.160 + 0.44 = 2.204
 1: 2.220 + 0.20 = 2.240
 2: 2.212 + 0.32 = 2.244
 3: 4.240 + 0.04 = 4.244
 4: 4.260 + 0.00 = 4.260
 5: 4.256 + 0.08 = 4.264
 ----------------------------------------


Done!


Looking at another test case (acetate w/ cosmo) I get the following scaling on a single node as a function of processors:


shmmax issue:
Anyone who has been using nwchem will be familiar with this
 INPUT CARD> $END                                                                           
 DDI Process 0: shmget returned an error.
 Error EINVAL: Attempting to create 160525768 bytes of shared memory.
 Check system limits on the size of SysV shared memory segments.

 The file ~/gamess/ddi/readme.ddi contains information on how to display
 the current SystemV memory settings, and how to increase their sizes.
 Increasing the setting requires the root password, and usually a sytem reboot.

 DDI Process 0: error code 911

The fix is the same. First do

cat /proc/sys/kernel/shmmax

and look at the value. Then set it to the desired value according to this post: http://verahill.blogspot.com.au/2012/04/solution-to-nwchem-shmmax-too-small.html
e.g.
sudo sysctl -w kernel.shmmax=6269961216

gfortran version issue:
Even though you likely have version 4.7.x of gfortran, pick 4.6 or you will get:

Please type 'gfortran -dumpversion' or else 'gfortran -v' to
detect the version number of your gfortran.
This reply should be a string with at least two decimal points,
such as 4.1.2 or 4.6.1, or maybe even 4.4.2-12.
The reply may be labeled as a 'gcc' version,
but it is really your gfortran version.
Please enter only the first decimal place, such as 4.1 or 4.6:  
4.7

The gfortran version number is not recognized.
It should only have one decimal place, such as 4.x

The reason is this (code from config):
      switch ($GMS_GFORTRAN_VERNO)
         case 4.1:
         case 4.2:
         case 4.3:
         case 4.4:
         case 4.5:
            echo "   Alas, your version of gfortran does not support REAL*16,"
            echo "   so relativistic integrals cannot use quadruple precision."
            echo "   Other than this, everything will work properly."
            breaksw
         case 4.6:
            echo "   Good, the newest gfortran can compile REAL*16 data type."
            breaksw
         default:
            echo "The gfortran version number is not recognized."
            echo "It should only have one decimal place, such as 4.x"
            exit 4
            breaksw
      endsw

267. ECCE client connecting to remote site via reverse port/local port forwarding

The situation I'm about describe is quite specific, yet I don't think it's that unusual.

A. I've got a computer at work which is behind a firewall so that I can't connect directly to it from the outside. This will be referred to as Work.

B. I've got a laptop at home which is connected to a wireless  router. This will be referred to as Home.

C. The router is a Linksys/tomato router, which is accessible from the outside (myrouter.com). This will be referred to as Router.

I'd like to connect from home to my ecce server at work so that I can monitor and submit jobs.

At Work:
ssh -R 19997:localhost:8096 root@myrouter.com

At Home:
ssh -L 5555:localhost:19997 root@myrouter.com

We're basically tying together port 5555 at Home with port 8096 at Work, via an intermediary server.

At Home, edit your ecce/apps/siteconfig/Dataservers and change the relevant lines to

<eccedata>
  <ecceserver>
    <url>http://localhost:5555/Ecce</url>
    <desc>ECCE Data Server--remote</desc>
  </ecceserver>

  <basisset>http://localhost:5555/Ecce/system/GaussianBasisSetLibrary</basisset>
</eccedata>

Note that submission actually happens from your ecce client, not your server (i.e. from Home, not Work), so to get your submission scripts in order you may have to do a bit of fiddling. E.g. if you ecce server is also the queue master for an SGE batch system:

Work:
ssh -R 19999:localhost:22 root@myrouter.com

Home:
ssh -L 5454:localhost:19999 root@myrouter.com

Home:
Edit /apps/siteconfig/remote_shells.site and add
ssh_p5454: ssh -XC -p 5454|scp -P 5454|xterm

But you can read more about that here: http://verahill.blogspot.com.au/2012/05/port-redirection-with-eccenwchem.html

25 October 2012

266. Back-up your gmail using getmail on debian testing/wheezy

Since I'm thinking about moving universities the issue of backing up my work email (which is hosted by gmail) account is weighing at the back of my mind.

I'm just following this post: https://wiki.archlinux.org/index.php/Backup_Gmail_with_getmail

This post is duplicating most of what's done there (I like the Arch tutes -- well written and thorough) so this is more of a 'yes, I followed it and it works' kind of posts.

Don't forget that gmail has an imap bandwidth limit of ca 2.5 GB per day: https://support.google.com/a/bin/answer.py?hl=en&answer=1071518

First install getmail4. You can probably ignore bug =#633799 (I'm assuming that you use apt-listbugs and receive the warning below -- if not, you probably shouldn't be using testing...)

sudo apt-get install getmail4
#633799 - getmail causes irrecoverable mail corruption when using mbox
mkdir ~/.getmail touch ~/.getmail/getmailrc chmod og-rwx ~/.getmail/getmailrc mkdir /media/backups/workmail cd /media/backups/workmail mkdir cur new tmp

If you don't make the last three folders getmail will complain.
Edit your ~/.getmail/getmailrc
[retriever] type = SimpleIMAPSSLRetriever server = imap.gmail.com mailboxes = ("[Gmail]/All Mail",) username = firstname.lastname@mycompany.com password = myPassword [destination] type = Maildir path = /media/backups/workmail/ [options] verbose = 2 message_log = ~/.getmail/log # retrieve only new messages # if set to true it will re-download ALL messages every time! #read_all = false # do not alter messages delivered_to = false received = false
Note that you may have a folder called [Google Mail/All Mail] instead

More options here: http://pyropus.ca/software/getmail/configuration.html

Next run getmail

NOTE: if you interrrupt (by e.g. ctrl+c) and then resume by running getmail again you'll download all emails again. If you let it run to completion, everything will work properly, however, and in the future it will only download new emails. 

getmail
msg 1/5053 (5621 bytes) msgid 648042553/1 from someone@somewhere delivered to Maildir /media/backups/workmail/ [..] msg 772/5053 (10839 bytes) from someone@somewhere delivered to Maildir /media/backups/workmail/ msg 773/5053 (4377 bytes) from someone@somewhere delivered to Maildir /media/backups/workmail [..] 5053 messages (783074371 bytes) retrieved, 0 skipped Summary: Retrieved 5053 messages (783074371 bytes) from SimpleIMAPSSLRetriever:firstname.lastname@mycompany.com@imap.gmail.com:993
If that all worked ok, edit your crontab and make it run e.g. once per day:

crontab -e
00 23 * * * getmail -q
Since I have a private gmail account as well I created a file called privatemailrc and use it with
getmail -r ~/.getmail/privatemailrc -q

I also created folders /media/backups/privatemail/cur, /media/backups/privatemail/tmp and /media/backups/privatemail/new and point to those in my privatemailrc file.

24 October 2012

265. shmmax revisited -- and shmall, shmmni

I've upgraded two of my nodes -- my old 4 core node with 8 GB ram now has 4x4=16 GB RAM, while my old 8 core, 16 GB ram now has 4*8=32 GB ram.

When using nwchem you eventually will run into an shmmax problem:


******************* ARMCI INFO ************************
The application attempted to allocate a shared memory segment of 44498944 bytes in size. This might be in addition to segments that were allocated succesfully previously. The current system configuration does not allow enough shared memory to be allocated to the application.

This is most often caused by:
1) system parameter SHMMAX (largest shared memory segment) being too small or
2) insufficient swap space.
Please ask your system administrator to verify if SHMMAX matches the amount of memory needed by your application and the system has sufficient amount of swap space. Most UNIX systems can be easily reconfigured to allow larger shared memory segments,
see http://www.emsl.pnl.gov/docs/global/support.html
In some cases, the problem might be caused by insufficient swap space.
*******************************************************
0:allocate: failed to create shared region : -1
(rank:0 hostname:boron pid:17222):ARMCI DASSERT fail. shmem.c:armci_allocate():1082 cond:0

I haven't gotten that in a while since I increased shmmax to 6572498432, but running a frequency calculation on a large molecule with unrestricted DFT triggered it again on my 32 GB node. So I hit google. These posts were informative:
http://www.pythian.com/news/245/the-mysterious-world-of-shmmax-and-shmall/
http://padmavyuha.blogspot.com.au/2010/12/configuring-shmmax-and-shmall-for.html
http://yuji.wordpress.com/2011/11/03/what-is-shmmax-shmall-shmmni-shared-memory-max/


me@neon:~$  cat /proc/sys/kernel/shmall
2097152
me@neon:~$ cat /proc/sys/kernel/shmni
4096
me@neon:~$ cat /proc/sys/kernel/shmmax
6572498432

That works out to (4096 bytes/page*2097152)*(1/(1024*1024*1024) bytes per gigabyte) pages=8.192 GB. And they are the same on all my nodes in spite of the memory available varying.

Another way of looking at it:
ipcs -lm

------ Shared Memory Limits --------
max number of segments = 4096
max seg size (kbytes) = 6418455
max total shared memory (kbytes) = 8388608
min seg size (bytes) = 1


Your shmmall is the number of pages total, the shmmni is the page size and the shmmax is the largest contigouos chunk of RAM available.

 So if I get things right, and parroting what's said on the pages above, your shmmall should approach but not exceed your total physical memory, you shmni is better left alone, and your shmmax can be anywhere up to your total RAM.

The links above cite Oracle recommendations which state that (for 32 bit system) it should be 4 GB - 1 byte OR half your RAM, whichever is smaller. I'll show that case here, but will be testing using 80% of my RAM for my calcs.

 So for my boxes:

32 GB RAM => shmmax=16GB, shmmall=(32-2 GB)/4095, shmni=4096
sudo sysctl -w kernel.shmmax=17179869184
sudo sysctl -w kernel.shmall=7340032
ipcs -lm

------ Shared Memory Limits -------- max number of segments = 4096 max seg size (kbytes) = 16777216 max total shared memory (kbytes) = 29360128 min seg size (bytes) = 1
16 GB RAM => shmmax=8GB, shmmall=(16-2 GB)/4096, shmni=4096
sudo sysctl -w kernel.shmmax=8589934592
sudo sysctl -w kernel.shmall=3670016


If you're happy with those values, make them permanent by editing your sysctl.conf and adding the relevant lines:
kernel.shmmax=17179869184
kernel.shmall=7340032


So here are the formulae (assuming that you set shmmax to half your ram and leave 2 gb out of shmall):
shmmax=RAM (bytes)/2
shmni=4096
shmmall=(RAM(bytes)-2147483648)/shmni

18 October 2012

259. Compiling firefox 16.0.1 in debian testing/wheezy

I actually mainly use chromium/chrome for my normal browsing needs, and have set up an ssh tunnel with an American university for iceweasel so I can access sites that filter on IP.

There's nothing particularly wrong with iceweasel, but it's nice to check out the latest and shiniest versions of different projects every now and again, so here's how to compile the latest firefox:

The source is just a hair under 90 Mb and the build takes around 40 minutes on a three-core AMD.

I've got so many -dev packages installed on my system that I don't really know what the dependencies are anymore, but here's a guess:
sudo apt-get install build-essential gfortran gawk libpango1.0-dev libgconf2-dev libgnomevfs2-dev libcogl-pango-dev libpangomm-1.4-dev libgconfmm-2.6-dev libnotify-dev libgnomeui-dev libdbus-1-dev libdbus-glib-1-dev 

Anyway:
cd ~/tmp
wget ftp://ftp.mozilla.org/pub/firefox/releases/16.0.1/source/firefox-16.0.1.source.tar.bz2
tar xvf firefox-16.0.1.source.tar.bz2
mkdir firefox_build
cd firefox_build/
../mozilla-release/./configure --prefix=$HOME/.firefox --disable-necko-wifi
make
make install

I'm getting tired of the clutter in my PATH, so
echo "alias firefox16='$HOME/.firefox/bin/firefox'" >> ~/.bashrc



Note that iceweasel puts a firefox in/usr/bin/, hence the use of firefox16

16 October 2012

258. Briefly: Throttle CPU between certain hours on linux

Because university administrators apparently don't consider academics trustworthy enough, at my university each office has an AC unit which cannot be adjusted. Sure, we can set the temperature -- but the unit cannot be turned on or off. Instead it shuts itself off at 5 pm and goes back on at 8 am. Same goes for weekends -- the unit shuts itself off on Friday at 5 pm and turns itself back on on Monday morning.

Obviously all the windows are sealed shut.

Which is fair enough in terms of saving energy -- no administrator would be in their office over the weekend. But most academics are constantly fighting to keep themselves afloat in terms of work, and most of us are in on weekends too.

But this is Australia -- the winters get cold enough that it's unpleasant to sit in an unheated office (no insulation) and the summers hot enough that computers get unhappy.

And I'm running a cluster in my office, so on a warmish weekend (above 20 degrees) my tiny office gets really hot. Last night one of my nodes overheated and shut itself down.

So, in want of a better solution, I figure I will throttle the CPUs overnight to give the fans a fighting chance to keep the cpus cool when the aircon is off. In particular weekends, with no air-conditioning from 5 pm on Friday until 8 am on Monday, are a challenge.


Enough talking! 

Put the following files in /etc/cron.d

throttle.sh
#!/bin/bash
## minimise heating overnight
/usr/bin/cpufreq-set -c 0 -g powersave
/usr/bin/cpufreq-set -c 1 -g powersave
/usr/bin/cpufreq-set -c 2 -g powersave
/usr/bin/cpufreq-set -c 3 -g powersave
/usr/bin/cpufreq-set -c 4 -g powersave
/usr/bin/cpufreq-set -c 5 -g powersave

slowdown.sh
#!/bin/bash
## minimise heating overnight
/usr/bin/cpufreq-set -c 0 -g userspace
/usr/bin/cpufreq-set -c 1 -g userspace
/usr/bin/cpufreq-set -c 2 -g userspace
/usr/bin/cpufreq-set -c 3 -g userspace
/usr/bin/cpufreq-set -c 4 -g userspace
/usr/bin/cpufreq-set -c 5 -g userspace
/usr/bin/cpufreq-set -c 0 -f 2.2G
/usr/bin/cpufreq-set -c 1 -f 2.2G
/usr/bin/cpufreq-set -c 2 -f 2.2G
/usr/bin/cpufreq-set -c 3 -f 2.2G
/usr/bin/cpufreq-set -c 4 -f 2.2G
/usr/bin/cpufreq-set -c 5 -f 2.2G

unthrottle.sh
#!/bin/bash
## maximise performance during the day
/usr/bin/cpufreq-set -c 0 -g ondemand
/usr/bin/cpufreq-set -c 1 -g ondemand
/usr/bin/cpufreq-set -c 2 -g ondemand
/usr/bin/cpufreq-set -c 3 -g ondemand
/usr/bin/cpufreq-set -c 4 -g ondemand
/usr/bin/cpufreq-set -c 5 -g ondemand

Now edit /etc/crontab
00 08   * * 1-5 root    sh /etc/cron.d/unthrottle.sh
00 17   * * 1-4 root    sh /etc/cron.d/slowdown.sh
00 17 * * 5 root sh /etc/cron.d/throttle.sh


And you're good to go.

15 October 2012

257. Wine 1.5.15 on debian testing/wheezy --finally sorting out that annoying libjpeg

UPDATE 16 May 2013: See here for Wine 1.5.30: http://verahill.blogspot.com.au/2013/05/416-wine-1530-in-chroot.html

UPDATE (10th Jan 2013): See here for Wine 1.5.21 using the multiarch approach: http://verahill.blogspot.com.au/2013/01/308-compiling-wine-1521-on-debian.html

Pretty much just follow this post:
The big difference here is that it seems to compile ok with libjpeg.

And here are the release notes

Install the pre-requisites 
(haven't checked this list for a while -- there may be a few too many packages)
sudo apt-get install ia32-libs ia32-libs-dev bison flex gcc libc6-dev libfontconfig-dev libfreetype6-dev libglu-dev libgsm1-dev libice-dev libjpeg-dev libldap-dev libmpg123-dev libncurses5-dev libopenal-dev libpng-dev libsm-dev libssl-dev libusb-dev libx11-dev libxcomposite-dev libxcursor-dev libxext-dev libxi-dev libxinerama-dev libxml2-dev libxrandr-dev libxrender-dev libxslt-dev libxt-dev libxxf86vm-dev make libcapi20-dev liblcms-dev libsane-dev libhal-dev libdbus-1-dev valgrind prelink libcups2-dev opencl-dev lib32opencl1 oss4-dev gettext lib32v4l-dev lib32ncurses5-dev lib32asound2-dev libtiff4-dev libgphoto2-2-dev


Get new source:
cd ~/tmp
wget http://sourceforge.net/projects/wine/files/Source/wine-1.5.15.tar.bz2
tar xvf wine-1.5.15.tar.bz2
cd wine-1.5.15/

To avoid
configure: WARNING: libjpeg 32-bit development files not found, JPEG won't be supported.

edit /usr/include/jpeglib.h and delete

24   #ifndef JCONFIG_INCLUDED        /* in case jinclude.h already did */
25   #include "jconfig.h"            /* widely used configuration options */
26   #endif

Then do 

./configure

which yields

configure: libOSMesa 32-bit development files not found (or too old), OpenGL rendering in bitmaps won't be supported.
configure: OpenCL 32-bit development files not found, OpenCL won't be supported.
configure: gstreamer-0.10 base plugins 32-bit development files not found, gstreamer support disabled
configure: libgsm 32-bit development files not found, gsm 06.10 codec won't be supported.
configure: Finished.  Do 'make' to compile Wine.

Neither of these issues are fatal -- depending on what you use wine for.

Time to install:
make
sudo checkinstall --install=no
sudo dpkg -i wine-1.5.15-1_amd64.deb

And you're done!

256. Briefly: Setting up ganglia on a debian beowulf cluster

On all your nodes (e.g. using clusterssh):
sudo apt-get install ganglia-monitor

On your front node only:
sudo apt-get install ganglia-webfrontend
sudo cp /etc/ganglia-webfrontend/apache.conf /etc/apache2/sites-enabled/ganglia.conf
sudo service apache2 restart

Edit /etc/ganglia/gmetad.conf and at a minimum add
data_source "debiwolf" 10 beryllium tantalum boron neon
or
data_source "debiwolf" 10 192.168.1.1 192.168.1.131 192.168.1.150 192.168.1.120
You can also specify additional information

gridname "rupert"
authority "http://192.168.1.1/ganglia/"

Go to http://localhost/ganglia in your browser. If you get
There was an error collecting ganglia data (127.0.0.1:8652): fsockopen error: Connection refused

Then do
sudo service gmetad start

Otherwise you should see something like this:

On all your nodes:
You may want to do
sudo service ganglia-monitor restart

if some of your nodes aren't showing.

Customize:
Edit your /etc/ganglia/gmond.conf:

cluster {
        name = "Debiwolf"
        owner = "Me"
        latlong = "unspecified"
        url = "None"
}