20 October 2012

262. chroot/jail your iceweasel (firefox)

Update: in my youth I thought that chroot=security. I now know better.
http://yarchive.net/comp/linux/chroot.html
http://pen-testing.sans.org/blog/2012/06/06/escaping-restricted-linux-shells

In other words, don't rely on chroot for security -- it's not meant to be a security tool. I personally use it to browse at work without leaving a too obvious a trail (nothing shady -- part of my job is to apply for permanent jobs, but it is not necessarily in the interest of my current employer to see me succeed. Academia is a weird place.)

Original post:
There are many reasons why you'd want to secure your browser -- ranging from paranoia to justifiable cautiousness (you're probably visiting the wrong kind of sites...).

A chroot environment doesn't make you anonymous -- it just helps sandbox your applications. To protect your anonymity you'll want to prevent your browser from setting cookies, and use a proxy via an ssh tunnel to encrypt your traffic and hide your true IP address. There's no silver bullet for these things.

I've chroot:d things in the past when compiling 32 bit applications on 64 bit machines, but for this guide I'm leaning heavily on this post: https://help.ubuntu.com/community/BasicChroot

sudo apt-get install coreutils debootstrap

Time to get rocking. Be aware that you should have a bit of free space on your target drive.


sudo debootstrap --arch amd64 wheezy /media/chroot/ http://ftp.au.debian.org/debian/

which gets stuff underway:


I: Retrieving InRelease
I: Checking Release signature
I: Valid Release signature (key id 9FED2BCBDCD29CDF762678CBAED4B06F473041FA)
I: Retrieving Packages
I: Validating Packages
I: Resolving dependencies of required packages...
I: Resolving dependencies of base packages...
I: Checking component main on http://ftp.au.debian.org/debian...
I: Retrieving libacl1
I: Validating libacl1
I: Retrieving apt
[..]
I: Configuring dpkg-dev...
I: Configuring build-essential...
I: Base system installed successfully.

So far, so good.

Create a shell script, e.g. chroot.sh, with the following in it:
xhost +
sudo mount -o bind /proc /media/chroot/proc
sudo cp /etc/resolv.conf /media/chroot/etc/resolv.conf
sudo chroot /media/chroot

Run it everytime you want to enter you chroot environment:
sh chroot.sh

In the chroot shell, do
apt-get install locales sudo vim
echo 'export LC_ALL="C"'>>/etc/bash.bashrc
echo 'export LANG="C"'>>/etc/bash.bashrc
echo 'DISPLAY=:0.0' >> /etc/bash.bashrc
source /etc/bash.bashrc
adduser sandbox
usermod -g sudo sandbox
echo 'Defaults !tty_tickets' >> /etc/sudoers
apt-get install iceweasel
su sandbox
cd ~

Launch iceweasel/firefox:

sandbox@beryllium:/$ firefox


And you're pretty much done.
Next time you want to launch a sandboxed version of firefox just do:

me@beryllium:~$ sh chroot.sh 
access control disabled, clients can connect from any host
root@beryllium:/# su sandbox
sandbox@beryllium:/$ firefox

261. Disable tracker-miner-fs

Looking at my ~/.local/share/tracker folder I've got 1.5 Gb wasting away. Also, tracker has a way of eating up CPU cycles,. Given that I never do any desktop searches -- I use either 'locate' or 'find' in the terminal and they do exactly what I need -- I don't see much point in allowing tracker to run.

1. If you're in a gui you can run tracker-preferences  and disable the indexing of all folders.

2. To delete existing tracker logs you need to do
tracker-control -r

This kills all running tracker processes and removes the databases.

3. Finally, edit /etc/xdg/autostart/tracker-miner-fs.desktop and tracker-store.desktop and change

X-GNOME-Autostart-enabled=true

to

X-GNOME-Autostart-enabled=false

19 October 2012

260. Compiling Dalton 2.0 QM on Debian -- in progress

UPDATE: I've since successfully compiled Dalton 2011 and lsdalton on both debian and CentOS/ROCKS:
http://verahill.blogspot.com.au/2012/11/compiling-lsdalton-on-debian.html
http://verahill.blogspot.com.au/2012/11/compiling-dalton-2011-on-debian.html
http://verahill.blogspot.com.au/2012/11/compiling-dalton-2011-on-rocks-543centos.html

I will revisit this post at a later date.

NOTE:  This is an update of an old post in which the compilation of Dalton was described, but I couldn't get the finished binary to run. Seems the main problem was that I don't know how to use dalton properly. I've got enough data now to show that dalton is working in serial mode though.

1. Sign up for a license.

2. Once you get approved you'll get an email with a link, a username and a password.
Download the cam-enabled package.

3.
cd ~/tmp
mv ~/Download/dalton-2.0-ca.tar.gz .
tar xvf dalton-2.0-cam.tar.gz
cd dalton-2.0-cam/
sudo mkdir /opt/dalton
sudo chown $USER:$USER /opt/dalton
mkdir /opt/dalton/basis
mkdir /opt/dalton/bin

Also, you may want to compile the netlib blas and lapack, and set them up. See here: http://verahill.blogspot.com.au/2012/09/compiling-netlibs-lapack-and-blas-on.html

Once you're done with that, start running configure:
./configure
------------------------------------------------------------------
   Configuring the DALTON Makefile.config and "dalton" run script
------------------------------------------------------------------

No architecture specified, attempting auto-configuration:
This appears to be a -linux architecture. Is this correct?  [y/n] y

-->Installing DALTON on a -linux computer

Do you want to install the program in a parallel MPI version? [y/n] y

-->WARNING: Makefiles for MPI architecture are difficult to guess
   Please compare the generated Makefile.config with local documentation.

   Checking for Fortran compiler ...
   from this list: mpif90 mpif77 ifort ifc efc pgf77 gfortran g77 

Compiler mpif90 found, use this compiler? [y/n] y

-->Compiler mpif90 found and accepted.

   Checking for C compiler ...
   from this list: mpicc  icc ecc pgcc gcc 

Compiler mpicc found, use this compiler? [y/n] y

-->Compiler mpicc found and accepted.

Testing existence of libraries in this order:
 libacml.a libmkl.so libmkl_p3.a libatlas.a libblas.a
Directory search list for libraries:
  /opt/gaussian/g03/ATLAS/compiled/lib /opt/gaussian/g03/ATLAS/build/lib /opt/ATLAS/lib /home/me/tmp/ATLAS/build/lib /lib /usr/local/lib /usr/lib /usr/local/lib/ATLAS /lib64 /usr/lib64 /usr/local/lib64 

Found /opt/gaussian/g03/ATLAS/compiled/lib/libatlas.a, use it ? [y/n] n
Found /opt/gaussian/g03/ATLAS/build/lib/libatlas.a, use it ? [y/n] n
Found /opt/ATLAS/lib/libatlas.a, use it ? [y/n] y

-->The following mathematical library(ies) will be used:
   -L/opt/ATLAS/lib -llapack -lf77blas -latlas

Does your file system allow files bigger than 2 Gb? [y/n] y

DALTON uses almost 7 million *8 words (56 Megabytes) of static
allocations, in addition to the dynamic allocation.

DALTON has the possibility to reserve an amount of static memory
for storing two-electron integrals in direct and parallel calculations
Storing some or all of the 2-el. integrals in memory will speed up
direct and parallel calculations (and in particular the latter).
NOTE: This will increase the static memory allocation used by DALTON

Would you like to activate the possibility of storing 2-el.int. in memory? [y/n] y
How many millions of *8 words to use for storing 2-el.integrals? 1000

-->Program will be installed with 1000000000 words used for storing 2-el.integrals

Maximum amount of work memory for dynamic allocations can be
changed at run time with the environment variable WRKMEM
or by using the -M option to the run script: "dalton -M mb ...".
We recommend at least 20 million *8 words work memory,
larger for correlated calculations, but it should for maximum
efficiency NOT exceed available physical memory.

How many millions of *8 words to use as default for work memory? 1000

-->Program will be installed with a default work memory of 1000000000 words

-->Current directory is /home/me/tmp/dalton-2.0-cam

Use ./bin as installation directory for binaries and scripts? [y/n] n
Please enter another installation directory: /opt/dalton/bin

-->DALTON executable and script will be placed in /opt/dalton/bin directory


-->Default basis set directory will be /home/me/tmp/dalton-2.0-cam/basis/


Use this directory as default basis set directory? [y/n] n

Please choose another default basis set directory (must end with /) /opt/dalton/basis/

-->Default basis set directory will be /opt/dalton/basis/


-->Job specific directories under $SCRATCH/$USER
-->will be used for temporary files when running DALTON

Use SCRATCH=/work as default root scratch space in "dalton" run script? [y/n] n
Please enter default root scratch directory: /scratch

-->Creating Makefile.config ...
Backup: renaming Makefile.config to Makefile.config.0
Is backend compiler gfortran ? [y/n] y
gfortran version 471 prc=x86_64
Proper 64-bit file access detected.

-->Creating the DALTON run-script in /opt/dalton/bin

   The configuration of DALTON has finished succesfully.
   Check compiler flags etc. in Makefile.config and run "make" to get executable.


So far so good. Ignore the math libs above.

The Makefile.config looks like this now:

ARCH        = linux
#
#
CPPFLAGS      = -DVAR_GFORTRAN -DSYS_LINUX -DVAR_MFDS -D'INSTALL_WRKMEM=1000000000' -D'INSTALL_MMWORK=1000000000' -D_FILE_OFFSET_BITS=64 -DVAR_MPI -DGFORTRAN=471 -DIMPLICIT_NONE -DINSTALL_BASDIR='"/opt/dalton/basis/"'
F77           = mpif90
CC            = mpicc
LOADER        = mpif90
RM            = rm -f
FFLAGS        = -march=native -O1 -ffast-math -funroll-loops -ftree-vectorize -fno-range-check -fno-whole-file
SAFEFFLAGS    = -march=native -O1 -ffast-math -funroll-loops -ftree-vectorize -fno-range-check -fno-whole-file
CFLAGS        = -march=native -O1 -ffast-math -funroll-loops -ftree-vectorize -std=c99 -DRESTRICT=restrict -DFUNDERSCORE=1
INCLUDES      = -I../include 
LIBS          = -L/opt/ATLAS/lib -llapack -llapack -lf77blas -latlas
INSTALLDIR    = /opt/dalton/bin
PDPACK_EXTRAS = linpack.o eispack.o gp_zlapack.o
GP_EXTRAS     = 
AR            = ar
ARFLAGS       = rvs
# flags for ftnchek on Dalton /hjaaj
CHEKFLAGS  = -nopure -nopretty -nocommon -nousage -noarray -notruncation -quiet  -noargumants -arguments=number  -usage=var-unitialized
# -usage=var-unitialized:arg-const-modified:arg-alias
# -usage=var-unitialized:var-set-unused:arg-unused:arg-const-modified:arg-alias
#
default : linuxparallel.x
SAFE2_FFLAGS = $(FFLAGS)
#
# Parallel initialization
#
MPI_INCLUDE_DIR = 
MPI_LIB_PATH    = 
MPI_LIB         = 
#
#
# Suffix rules
# hjaaj Oct 04: .g is a "cheat" suffix, for debugging.
#               'make x.g' will create x.o from x.F or x.c with -g debug flag set.
#
.SUFFIXES : .F .c .o .i .g .s

.F.o:
        $(F77) $(INCLUDES) $(CPPFLAGS) $(FFLAGS) -c $*.F 

.F.i:
        $(F77) $(INCLUDES) $(CPPFLAGS) -E $*.F > $*.i

.F.g:
        $(F77) $(INCLUDES) $(CPPFLAGS) $(FFLAGS) -g -c $*.F 

.F.s:
        $(F77) $(INCLUDES) $(CPPFLAGS) $(FFLAGS) -S -g -c $*.F 

.c.o:
        $(CC) $(INCLUDES) $(CPPFLAGS) $(CFLAGS) -c $*.c 

.c.i:
        $(CC) $(INCLUDES) $(CPPFLAGS) $(CFLAGS) -E $*.c > $.i

.c.g:
        $(CC) $(INCLUDES) $(CPPFLAGS) $(CFLAGS) -g -c $*.c 

.c.s:
        $(CC) $(INCLUDES) $(CPPFLAGS) $(CFLAGS) -S -g -c $*.c 


Start the compilation by
make -f Makefile.in


Note that
LIBS   =-L/opt/acml5.1.0/gfortran64/lib -lacml
also works, while I couldn't get openblas or the debian lapack/blas libs to work.

If all goes well:
...
a - amfi.o
a - symtra.o
make[1]: Leaving directory `/home/me/tmp/dalton-2.0-cam/amfi'
---------------> Linking sequential dalton.x ...
mpif90 -march=x86-64 -O3 -ffast-math -funroll-loops -ftree-vectorize -fno-range-check -fno-whole-file \
        -o /opt/dalton/bin/dalton.x abacus/dalton.o cc/crayio.o abacus/linux_mem_allo.o \
        abacus/herpar.o eri/eri2par.o amfi/amfi.o amfi/symtra.o gp/mpi_dummy.o -Labacus -labacus -Lrsp -lrsp -Lsirius -lsirius -labacus -Leri -leri -Ldensfit -ldensfit -Lcc  -lcc -Ldft -ldft -Lgp -lgp -Lpdpack -lpdpack -L/opt/ATLAS/lib -llapack -lf77blas -latlas 
---------------> Linking parallel dalpar.x ...
mpif90 -march=x86-64 -O3 -ffast-math -funroll-loops -ftree-vectorize -fno-range-check -fno-whole-file \
        -o /opt/dalton/bin/dalpar.x abacus/dalton.o cc/crayio.o abacus/linux_mem_allo.o \
        abacus/herpar.o eri/eri2par.o amfi/amfi.o amfi/symtra.o -Labacus -labacus -Lrsp -lrsp -Lsirius -lsirius -labacus -Leri -leri -Ldensfit -ldensfit -Lcc  -lcc -Ldft -ldft -Lgp -lgp -Lpdpack -lpdpack -L/opt/ATLAS/lib -llapack -lf77blas -latlas  \
        -L/usr/lin/openmpi/lib -lmpi 



Copy the basis sets:
cp basis/* /opt/dalton/basis/

Throw
export PATH=$PATH:/opt/dalton/bin
in your ~/.bashrc and source it (source ~/.bashrc)

You're now ready to see how well it went.

cd test/
./dft_b3lyp_cart
mv dft_b3lyp_cart.dal DALTON.INP && mv dft_b3lyp_cart.mol MOLECULE.INP
mpirun -n 1 dalpar.x DALTON.INP MOLECULE.INP > DALTON.OUT

Looking at the output in DALTON.OUT:

 ***************************************************************************
 ************************ FINAL RESULTS FROM ABACUS ************************
 ***************************************************************************


 
     Date and time (Linux)  : Fri Oct 19 09:55:42 2012
     Host name              : beryllium                               



                         Molecular geometry (au)
                         -----------------------

 O          0.0000000000            0.0000000000           -0.1231063545
 H1   1     0.0000000000            1.4300000000            0.9768936455
 H1   2     0.0000000000           -1.4300000000            0.9768936455





                    Molecular wave function and energy
                    ----------------------------------

     Spin multiplicity  1     State number       1     Total charge       0


     Total energy        -73.9502446565 au (Hartrees)
                         -2012.28846267 eV
                           -194156.3395 kJ/mol




                         Relativistic corrections
                         ------------------------

     Darwin correction:                          0.1095380827 au
     Mass-velocity correction:                  -0.1391457352 au

     Total relativistic correction:             -0.0296076525 au (0.0400%)
     Non-relativistic + relativistic energy:   -73.9798523089 au




                              Dipole moment
                              -------------

                    0.783423 au           1.991263 Debye




                         Dipole moment components
                         ------------------------

                               au             Debye

                    z      0.78342341      1.99126348

                        1 a.u. =   2.54175 Debye 





   Interatomic separations (in Angstroms):
   ---------------------------------------

            O           H1   1      H1   2
            ------      ------      ------
 O     :    0.000000
 H1   1:    0.954707    0.000000
 H1   2:    0.954707    1.513447    0.000000


  Max interatomic separation is    1.5134 Angstroms
  between atoms "H1   2" and "H1   1".


  Bond distances (angstroms):
  ---------------------------

                  atom 1     atom 2       distance
                  ------     ------       --------
  bond distance:  H1   1     O            0.954707
  bond distance:  H1   2     O            0.954707


  Bond angles (degrees):
  ----------------------

                  atom 1     atom 2     atom 3         angle
                  ------     ------     ------         -----
  bond angle:     H1   1     O          H1   2       104.863




 CPU time statistics for ABACUS
 ------------------------------





 >>>> Total CPU  time used in ABACUS:   0.01 seconds
 >>>> Total wall time used in ABACUS:   0.00 seconds

- End of Static Property Section

 >>>> Total CPU  time used in DALTON:   1.56 seconds
 >>>> Total wall time used in DALTON:   0.00 seconds

 
     Date and time (Linux)  : Fri Oct 19 09:55:42 2012
     Host name              : beryllium         



So it's clearly working. However, -n 2 leads to a string of errors:

2abe8f67d000-2abe8f87c000 ---p 00005000 08:01 1839615                    /usr/lib/openmpi/lib/openmpi/mca_dpm_orte.so
2abe8f87c000-2abe8f87d000 rw-p 00004000 08:01 1839615                    /usr/lib/openmpi/lib/openmpi/mca_dpm_orte.so
2abe8f87d000-2ac06c5e3000 rw-p 00000000 00:00 0 
2ac070000000-2ac070021000 rw-p 00000000 00:00 0 
2ac070021000-2ac074000000 ---p 00000000 00:00 0 
7fffc3ce9000-7fffc3d0c000 rw-p 00000000 00:00 0                          [stack]
7fffc3dff000-7fffc3e00000 r-xp 00000000 00:00 0                          [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]

Program received signal SIGABRT: Process abort signal.

Backtrace for this error:
^Cmpirun: killing job...

--------------------------------------------------------------------------
mpirun noticed that process rank 0 with PID 2846 on node beryllium exited on signal 0 (Unknown signal 0).
--------------------------------------------------------------------------
mpirun: clean termination accomplished




Errors:


Error:
Warning: Type mismatch in argument 'indsq' at (1); passed REAL(8) to INTEGER(4)

cc3_omeg3.F:3322.33:
                  CALL CC3_RACC3(DUMMY,OMEGA2M,WORK(KRMAT4),ISYMB,B,  
                                 1
Warning: Rank mismatch in argument 'omega2p' at (1) (rank-1 and scalar)
cc3_omeg3.F:3352.27:
            CALL CC3_RACC3(DUMMY,OMEGA2M,WORK(KRMAT3),ISYMD,D,ISYRES,
                           1
Warning: Rank mismatch in argument 'omega2p' at (1) (rank-1 and scalar)
cc3_omeg3.F: In function ‘cc3_racc3_.constprop.0’:cc3_omeg3.F:1319:0: internal compiler error: in expand_assignment, at expr.c:4731Please submit a full bug report,
with preprocessed source if appropriate.
See <file:///usr/share/doc/gcc-4.7/README.Bugs> for instructions.
make[1]: *** [cc3_omeg3.o] Error 1
make[1]: Leaving directory `/home/me/tmp/dalton-2.0-cam/cc'
make: *** [CC_OBJ] Error 2

Solution:
add -fno-whole-file to FFLAGS in Makefile.config

Error:
make[1]: Leaving directory `/home/me/tmp/dalton-2.0-cam/cc'
cd dft && make all
make[1]: Entering directory `/home/me/tmp/dalton-2.0-cam/dft'
gfortran -I../include  -DVAR_GFORTRAN -DSYS_LINUX -DVAR_MFDS -D'INSTALL_WRKMEM=1000000000' -D'INSTALL_MMWORK=1000000000' -D_FILE_OFFSET_BITS=64 -DVAR_MPI -DGFORTRAN=471 -DIMPLICIT_NONE -DINSTALL_BASDIR='"/opt/dalton/basis/"' -march=x86-64 -O3 -ffast-math -funroll-loops -ftree-vectorize -fno-range-check -fno-whole-file -c dft_den.F
gfortran -I../include  -DVAR_GFORTRAN -DSYS_LINUX -DVAR_MFDS -D'INSTALL_WRKMEM=1000000000' -D'INSTALL_MMWORK=1000000000' -D_FILE_OFFSET_BITS=64 -DVAR_MPI -DGFORTRAN=471 -DIMPLICIT_NONE -DINSTALL_BASDIR='"/opt/dalton/basis/"' -march=x86-64 -O3 -ffast-math -funroll-loops -ftree-vectorize -fno-range-check -fno-whole-file -c dft_grid.F
gfortran -I../include  -DVAR_GFORTRAN -DSYS_LINUX -DVAR_MFDS -D'INSTALL_WRKMEM=1000000000' -D'INSTALL_MMWORK=1000000000' -D_FILE_OFFSET_BITS=64 -DVAR_MPI -DGFORTRAN=471 -DIMPLICIT_NONE -DINSTALL_BASDIR='"/opt/dalton/basis/"' -march=x86-64 -O3 -ffast-math -funroll-loops -ftree-vectorize -fno-range-check -fno-whole-file -c dft_ksm.F
dft_ksm.F:239:0: fatal error: mpif.h: No such file or directorycompilation terminated.
make[1]: *** [dft_ksm.o] Error 1
make[1]: Leaving directory `/home/me/tmp/dalton-2.0-cam/dft'
make: *** [DFT_OBJ] Error 2
Solution: 
Make sure you set
MPI_INCLUDE_DIR = -I/usr/lib/openmpi/include

It may not be necessary for the mpiXXX compilers, but e.g. gfortran would not know where to look for the mpi headers.

On ROCKS/CentOS:

Error:
mpif90 -I../include  -DVAR_GFORTRAN -DSYS_LINUX -DVAR_MFDS -D'INSTALL_WRKMEM=100000000' -D'INSTALL_MMWORK=100000000' -D_FILE_OFFSET_BITS=64 -DVAR_MPI -DGFORTRAN=412 -DIMPLICIT_NONE -DINSTALL_BASDIR='"/home/me/.dalton/basis"' -march=x86-64 -O3 -ffast-math -funroll-loops -ftree-vectorize -fno-range-check -c gphjj.F
 In file gphjj.F:416
      WRITE (LUPRIN,'(T6,2A)') 'Date and time (Linux)  : ',FDATE()    
                                                          1
 In file gphjj.F:89
                 
                   2
Error: Global name 'fdate' at (1) is already being used as a SUBROUTINE at (2)
 In file gphjj.F:934
            IF(IR-IB) 340,310,320                                    
                                                                       1
Warning: Obsolete: arithmetic IF statement at (1)
 In file gphjj.F:1390
      DATA C/8H********/                                              
              1
Warning: Extension: Hollerith constant at (1)
 In file gphjj.F:1390
      DATA C/8H********/                                              
                      1
Warning: Extension: Conversion from HOLLERITH to REAL(8) at (1)
make[1]: *** [gphjj.o] Error 1
make[1]: Leaving directory `/home/me/tmp/dalton-2.0-cam/gp'
make: *** [GP_OBJ] Error 2
Solution:
???

18 October 2012

259. Compiling firefox 16.0.1 in debian testing/wheezy

I actually mainly use chromium/chrome for my normal browsing needs, and have set up an ssh tunnel with an American university for iceweasel so I can access sites that filter on IP.

There's nothing particularly wrong with iceweasel, but it's nice to check out the latest and shiniest versions of different projects every now and again, so here's how to compile the latest firefox:

The source is just a hair under 90 Mb and the build takes around 40 minutes on a three-core AMD.

I've got so many -dev packages installed on my system that I don't really know what the dependencies are anymore, but here's a guess:
sudo apt-get install build-essential gfortran gawk libpango1.0-dev libgconf2-dev libgnomevfs2-dev libcogl-pango-dev libpangomm-1.4-dev libgconfmm-2.6-dev libnotify-dev libgnomeui-dev libdbus-1-dev libdbus-glib-1-dev 

Anyway:
cd ~/tmp
wget ftp://ftp.mozilla.org/pub/firefox/releases/16.0.1/source/firefox-16.0.1.source.tar.bz2
tar xvf firefox-16.0.1.source.tar.bz2
mkdir firefox_build
cd firefox_build/
../mozilla-release/./configure --prefix=$HOME/.firefox --disable-necko-wifi
make
make install

I'm getting tired of the clutter in my PATH, so
echo "alias firefox16='$HOME/.firefox/bin/firefox'" >> ~/.bashrc



Note that iceweasel puts a firefox in/usr/bin/, hence the use of firefox16

16 October 2012

258. Briefly: Throttle CPU between certain hours on linux

Because university administrators apparently don't consider academics trustworthy enough, at my university each office has an AC unit which cannot be adjusted. Sure, we can set the temperature -- but the unit cannot be turned on or off. Instead it shuts itself off at 5 pm and goes back on at 8 am. Same goes for weekends -- the unit shuts itself off on Friday at 5 pm and turns itself back on on Monday morning.

Obviously all the windows are sealed shut.

Which is fair enough in terms of saving energy -- no administrator would be in their office over the weekend. But most academics are constantly fighting to keep themselves afloat in terms of work, and most of us are in on weekends too.

But this is Australia -- the winters get cold enough that it's unpleasant to sit in an unheated office (no insulation) and the summers hot enough that computers get unhappy.

And I'm running a cluster in my office, so on a warmish weekend (above 20 degrees) my tiny office gets really hot. Last night one of my nodes overheated and shut itself down.

So, in want of a better solution, I figure I will throttle the CPUs overnight to give the fans a fighting chance to keep the cpus cool when the aircon is off. In particular weekends, with no air-conditioning from 5 pm on Friday until 8 am on Monday, are a challenge.


Enough talking! 

Put the following files in /etc/cron.d

throttle.sh
#!/bin/bash
## minimise heating overnight
/usr/bin/cpufreq-set -c 0 -g powersave
/usr/bin/cpufreq-set -c 1 -g powersave
/usr/bin/cpufreq-set -c 2 -g powersave
/usr/bin/cpufreq-set -c 3 -g powersave
/usr/bin/cpufreq-set -c 4 -g powersave
/usr/bin/cpufreq-set -c 5 -g powersave

slowdown.sh
#!/bin/bash
## minimise heating overnight
/usr/bin/cpufreq-set -c 0 -g userspace
/usr/bin/cpufreq-set -c 1 -g userspace
/usr/bin/cpufreq-set -c 2 -g userspace
/usr/bin/cpufreq-set -c 3 -g userspace
/usr/bin/cpufreq-set -c 4 -g userspace
/usr/bin/cpufreq-set -c 5 -g userspace
/usr/bin/cpufreq-set -c 0 -f 2.2G
/usr/bin/cpufreq-set -c 1 -f 2.2G
/usr/bin/cpufreq-set -c 2 -f 2.2G
/usr/bin/cpufreq-set -c 3 -f 2.2G
/usr/bin/cpufreq-set -c 4 -f 2.2G
/usr/bin/cpufreq-set -c 5 -f 2.2G

unthrottle.sh
#!/bin/bash
## maximise performance during the day
/usr/bin/cpufreq-set -c 0 -g ondemand
/usr/bin/cpufreq-set -c 1 -g ondemand
/usr/bin/cpufreq-set -c 2 -g ondemand
/usr/bin/cpufreq-set -c 3 -g ondemand
/usr/bin/cpufreq-set -c 4 -g ondemand
/usr/bin/cpufreq-set -c 5 -g ondemand

Now edit /etc/crontab
00 08   * * 1-5 root    sh /etc/cron.d/unthrottle.sh
00 17   * * 1-4 root    sh /etc/cron.d/slowdown.sh
00 17 * * 5 root sh /etc/cron.d/throttle.sh


And you're good to go.

15 October 2012

257. Wine 1.5.15 on debian testing/wheezy --finally sorting out that annoying libjpeg

UPDATE 16 May 2013: See here for Wine 1.5.30: http://verahill.blogspot.com.au/2013/05/416-wine-1530-in-chroot.html

UPDATE (10th Jan 2013): See here for Wine 1.5.21 using the multiarch approach: http://verahill.blogspot.com.au/2013/01/308-compiling-wine-1521-on-debian.html

Pretty much just follow this post:
The big difference here is that it seems to compile ok with libjpeg.

And here are the release notes

Install the pre-requisites 
(haven't checked this list for a while -- there may be a few too many packages)
sudo apt-get install ia32-libs ia32-libs-dev bison flex gcc libc6-dev libfontconfig-dev libfreetype6-dev libglu-dev libgsm1-dev libice-dev libjpeg-dev libldap-dev libmpg123-dev libncurses5-dev libopenal-dev libpng-dev libsm-dev libssl-dev libusb-dev libx11-dev libxcomposite-dev libxcursor-dev libxext-dev libxi-dev libxinerama-dev libxml2-dev libxrandr-dev libxrender-dev libxslt-dev libxt-dev libxxf86vm-dev make libcapi20-dev liblcms-dev libsane-dev libhal-dev libdbus-1-dev valgrind prelink libcups2-dev opencl-dev lib32opencl1 oss4-dev gettext lib32v4l-dev lib32ncurses5-dev lib32asound2-dev libtiff4-dev libgphoto2-2-dev


Get new source:
cd ~/tmp
wget http://sourceforge.net/projects/wine/files/Source/wine-1.5.15.tar.bz2
tar xvf wine-1.5.15.tar.bz2
cd wine-1.5.15/

To avoid
configure: WARNING: libjpeg 32-bit development files not found, JPEG won't be supported.

edit /usr/include/jpeglib.h and delete

24   #ifndef JCONFIG_INCLUDED        /* in case jinclude.h already did */
25   #include "jconfig.h"            /* widely used configuration options */
26   #endif

Then do 

./configure

which yields

configure: libOSMesa 32-bit development files not found (or too old), OpenGL rendering in bitmaps won't be supported.
configure: OpenCL 32-bit development files not found, OpenCL won't be supported.
configure: gstreamer-0.10 base plugins 32-bit development files not found, gstreamer support disabled
configure: libgsm 32-bit development files not found, gsm 06.10 codec won't be supported.
configure: Finished.  Do 'make' to compile Wine.

Neither of these issues are fatal -- depending on what you use wine for.

Time to install:
make
sudo checkinstall --install=no
sudo dpkg -i wine-1.5.15-1_amd64.deb

And you're done!

256. Briefly: Setting up ganglia on a debian beowulf cluster

On all your nodes (e.g. using clusterssh):
sudo apt-get install ganglia-monitor

On your front node only:
sudo apt-get install ganglia-webfrontend
sudo cp /etc/ganglia-webfrontend/apache.conf /etc/apache2/sites-enabled/ganglia.conf
sudo service apache2 restart

Edit /etc/ganglia/gmetad.conf and at a minimum add
data_source "debiwolf" 10 beryllium tantalum boron neon
or
data_source "debiwolf" 10 192.168.1.1 192.168.1.131 192.168.1.150 192.168.1.120
You can also specify additional information

gridname "rupert"
authority "http://192.168.1.1/ganglia/"

Go to http://localhost/ganglia in your browser. If you get
There was an error collecting ganglia data (127.0.0.1:8652): fsockopen error: Connection refused

Then do
sudo service gmetad start

Otherwise you should see something like this:

On all your nodes:
You may want to do
sudo service ganglia-monitor restart

if some of your nodes aren't showing.

Customize:
Edit your /etc/ganglia/gmond.conf:

cluster {
        name = "Debiwolf"
        owner = "Me"
        latlong = "unspecified"
        url = "None"
}






255. Google Blogger Stats gone

Update: And...most of the stats are back. (03:19 GMT 15 October 2012)


02:30 GMT 15 October 2012
Did the google stats data just get reset/deleted?

Suddenly my stat counter shows just above 30 visitors -- all-time. That's a bit fewer than the 72k I was expecting. Oh well. I've at least got statcounter too keep me entertained.


It seems to have happened around 0:00 GMT 15 October 2012, although looking at the stats on blogger it may have happened around 9-10 am Eastern Aussie Daylight Savings time.

Update:
The angry posts are already beginning to appear: http://productforums.google.com/forum/#!topic/blogger/eOZyb9ws2IU
which bumps stuff in this direction:
http://productforums.google.com/forum/#!topic/blogger/dSFrwW2SWgE%5B1-25%5D
which has this statement from Blogger:
"We are aware of this issue and working to resolve it. "
And here's the official Blogger blog: http://knownissues.blogspot.com.au/

Not that I've ever had much use for the 'official' google stats much anyway -- although I obviously care enough about it to 1) notice that they're gone and 2) blog about it. But whatever.

12 October 2012

254. Compiling Thunderbird 16 on Debian Wheezy

I've posted how to compile thunderbird (12 and 13) in the past. Here's v 16.0.1:

First you need to sort out the dependencies:

sudo apt-get install libdbus-glib-1-dev gir1.2-notify-0.7 libnotify-dev yasm checkinstall libzip-dev zip libgtk2.0-dev

As usual, I prefer to do the building in ~/tmp
If you have a ~/tmp/comm-release directory, make sure to delete it first:

rm ~/tmp/comm-release -rf

Now download the new source (106 Mb):

cd ~/tmp
wget ftp://ftp.mozilla.org/pub/mozilla.org/thunderbird/releases/16.0.1/source/thunderbird-16.0.1.source.tar.bz2

Untar it, and create a new directory for out-of-tree building:

tar xvf thunderbird-16.0.1.source.tar.bz2
mkdir thunderbird16
cd thunderbird16/

Time to configure:
../comm-release/./configure --disable-necko-wifi

And build (40 minutes on a triple core AMD II)

make -j4

where -j4 indicates that it's built in parallel on a 3 core (3+1=4) processor. Note that this has nothing to do with running the finished binaries in parallel -- it's just a way of speeding up the compilation.


Make sure that you don't have an older version of thunderbird install via dpkg i.e.

aptitude search thunderbird|grep ^i

should come up blank. If not, uninstall that package.

Finally, install your new binaries:

sudo make install

And you're done.

09 October 2012

253. Leadtek DTV 1000s in kernel 3.6 (debian wheezy)

My Leadtek DTV 1000s has been working fine for a long time after the initial set-up, but when upgrading from kernel 3.4.0 to 3.6.0 on my tv computer' it suddenly stopped working.
me-tv 1.3.7-3

lspci shows
01:06.0 Multimedia controller: Philips Semiconductors SAA7130 Video Broadcast Decoder (rev 01)
but

 lsmod|grep saa 

returns nothing and there's nothing that looks right in /dev

Finally
sudo modprobe saa7134
FATAL: Module saa7134 not found.

 ls /lib/modules/3.4.0-amd64/kernel/drivers/media
common  dvb  media.ko  radio  rc  video
but

ls /lib/modules/3.6.0-amd64/kernel/drivers/media
common  dvb
Basically, saa7134.ko is never built.

I installed kernel 3.6 from deb files compiled on another computer. My first step was to uninstall linux-headers-3.6.0-amd64 and linux-image-3.6.0-amd64


Next, to get the saa7134 module-- roughly follow this post we get

wget http://www.kernel.org/pub/linux/kernel/v3.0/linux-3.6.tar.bz2
tar xvf linux-3.6.tar.bz2
cd linux-3.6/
cat /boot/config-`uname -r`>.config
make oldconfig

Make sure to answer in the affirmative here:
* Multimedia support
*
Multimedia support (MEDIA_SUPPORT) [M/n/y/?] Y
*
* Multimedia core support
*
Cameras/video grabbers support (MEDIA_CAMERA_SUPPORT) [N/y/?] (NEW) Y
Analog TV support (MEDIA_ANALOG_TV_SUPPORT) [N/y/?] (NEW)
Digital TV support (MEDIA_DIGITAL_TV_SUPPORT) [N/y/?] (NEW) Y
AM/FM radio receivers/transmitters support (MEDIA_RADIO_SUPPORT) [N/y/?] (NEW)
Remote Controller support (MEDIA_RC_SUPPORT) [N/y/?] (NEW) Y

Then build as usual
time fakeroot make-kpkg -j2 --initrd --revision=3.6.0 --append-to-version=-amd64 kernel_image kernel_headers

You can check that it built by

me@lithium:~/tmp/linux-3.6$ ls drivers/media/video/saa7134/*.ko -lah
drivers/media/video/saa7134/saa6752hs.ko
drivers/media/video/saa7134/saa7134-alsa.ko
drivers/media/video/saa7134/saa7134-dvb.ko
drivers/media/video/saa7134/saa7134-empress.ko
drivers/media/video/saa7134/saa7134.ko

All that's left now is to install the kernel:
mv ../linux*3.6*.deb . 
sudo dpkg -i *.deb

and you're done (see the original post for loading the driver: i.e. put options saa7134 tuner=48 card=175 in your /etc/modules) -- unless you have an nvidia graphics card, in which case

me@lithium:~/tmp/linux-3.6$ sudo dpkg-reconfigure nvidia-kernel-dkms

-------- Uninstall Beginning --------
Module:  nvidia
Version: 304.48
Kernel:  3.4.0-amd64 (x86_64)
-------------------------------------

Status: Before uninstall, this module version was ACTIVE on this kernel.

nvidia.ko:
 - Uninstallation
   - Deleting from: /lib/modules/3.4.0-amd64/updates/dkms/
 - Original module
   - No original module was found for this module on this kernel.
   - Use the dkms install command to reinstall any previous module version.

depmod.......

DKMS: uninstall completed.

-------- Uninstall Beginning --------
Module:  nvidia
Version: 304.48
Kernel:  3.6.0-amd64 (x86_64)
-------------------------------------

Status: Before uninstall, this module version was ACTIVE on this kernel.

nvidia.ko:
 - Uninstallation
   - Deleting from: /lib/modules/3.6.0-amd64/updates/dkms/
 - Original module
   - No original module was found for this module on this kernel.
   - Use the dkms install command to reinstall any previous module version.

depmod....

DKMS: uninstall completed.

------------------------------
Deleting module version: 304.48
completely from the DKMS tree.
------------------------------
Done.
Loading new nvidia-304.48 DKMS files...
Building for 3.4.0-amd64 and 3.6.0-amd64
Building initial module for 3.4.0-amd64
Done.

nvidia:
Running module version sanity check.
 - Original module
   - No original module exists within this kernel
 - Installation
   - Installing to /lib/modules/3.4.0-amd64/updates/dkms/

depmod....

DKMS: install completed.
Building initial module for 3.6.0-amd64
Done.

nvidia:
Running module version sanity check.
 - Original module
   - No original module exists within this kernel
 - Installation
   - Installing to /lib/modules/3.6.0-amd64/updates/dkms/

depmod....

DKMS: install completed.

08 October 2012

252. Molecular weight calculator in python

Here's the molecular weight part of the isotopic pattern calculator in a previous post.

Most people won't need a full molecular weight calculator with plotting of isotopic composition, so I'm publishing the molecular weight part as a separate program.

The actual algorithm is fairly simple and is more or less contained in the formulaExpander function below. It looks messy because of the definition of the PeriodicTable dictionary at the beginning, but it's simple.

Copy the code, past it into a file (call it e.g. molcalc), put it in e.g. /usr/local/bin and chmod +x it.

Usage:
molcalc 'Mg2(PO4)3'
returns
The mass of Mg2P3O12 is 333.524247 and the calculated charge is -5.
The charge is based on my default oxidation states -- depending on what kind of chemistry you do the oxidation states you encounter are likely to differ.

#!/usr/bin/python2.7
#########################################################################
# Principal author of current version: Me
# Isotopic abundances and masses were copied from Wsearch32.
#
#
# Dependencies:
# To be honest I'm not quite certain. At a minimum you will need python2.7,
# python-numpy
#
#########################################################################

import re #for regular expressions
import sys
from numpy import matrix,transpose # for molw calc
try:
 molecules=sys.argv[1]
except:
 quit()

#slowly changed to IUPAC 1997 isotopic compositions and IUPAC 2007 masses
# see http://pac.iupac.org/publications/pac/pdf/1998/pdf/7001x0217.pdf for
# natural variations in isotopic composition
PeriodicTable ={
   'H':[1,1,[1.0078250321,2.0141017780],[0.999885,0.0001157]], # iupac '97 in water
   'He':[2,0,[3.0160293097,4.0026032497],[0.00000137,0.99999863]], # iupac iso '97
   'Li':[3,1,[6.0151233,7.0160040],[0.0759,0.9241]], # iupac '97
   'Be':[4,2,[9.0121821],[1.0]], # iupac '97
   'B':[5,3,[10.0129370,11.0093055],[0.199,0.801]], # iupac' 97
                        'C':[6,4,[12.0,13.0033548378],[0.9893,0.0107]], # iupac '97
                        'N':[7,5,[14.0030740052,15.0001088984],[0.99632,0.00368]], # iupac '97
                        'O':[8,-2,[15.9949146221,16.99913150,17.9991604],[0.99757,0.00038,0.00205]], # iupac '97
                        'F':[9,-1,[18.99840320],[1.0]], # iupac '97
                        'Ne':[10,0,[19.9924401759,20.99384674,21.99138551],[0.9048,0.0027,0.0925]], # iupac '97 in air
                        'Na':[11,1,[22.98976967],[1.0]], #iupac '97
                        'Mg':[12,2,[23.98504190,24.98583702,25.98259304],[0.7899,0.10,0.1101]], #iupac '97
                        'Al':[13,3,[26.98153844],[1.0]], #iupac '97
                        'Si':[14,4,[27.9769265327,28.97649472,29.97377022],[0.92297,0.046832,0.030872]],#iupac '97
                        'P':[15,5,[30.97376151],[1.0]], #iupac '97
                        'S':[16,-2,[31.97207069,32.97145850,33.96786683,35.96708088],[0.9493,0.0076,0.0429,0.0002]], #iupac '97
                        'Cl':[17,-1,[34.96885271,36.96590260],[0.7578,0.2422]], #iupac '97
                        'Ar':[18,0,[35.96754628,37.9627322,39.962383123],[0.003365,0.000632,0.996003]],#iupac '97 in air
                        'K':[19,1,[38.9637069,39.96399867,40.96182597],[0.932581,0.000117,0.067302]], #iupac '97
                        'Ca':[20,2,[39.9625912,41.9586183,42.9587668,43.9554811,45.9536928,47.952534],[0.96941,0.00647,0.00135,0.02086,0.00004,0.00187]], #iupac '97
                        'Sc':[21,3,[44.9559102],[1.0]], #iupac '97
                        'Ti':[22,4,[45.9526295,46.9517638,47.9479471,48.9478708,49.9447921],[0.0825,0.0744,0.7372,0.0541,0.0518]], #iupac '97
                        'V':[23,5,[49.9471628,50.9439637],[0.00250,0.99750]], #iupac '97
                        'Cr':[24,2,[49.9460496,51.9405119,52.9406538,53.9388849],[0.04345,0.83789,0.09501,0.02365]], #iupac '97
                        'Mn':[25,2,[54.9380496],[1.0]], #iupac '97
                        'Fe':[26,3,[53.9396148,55.9349421,56.9353987,57.9332805],[0.05845,0.91754,0.02119,0.00282]], #iupac '97
                        'Ni':[27,3,[57.9353479,59.9307906,60.9310604,61.9283488,63.9279696],[0.680769,0.262231,0.011399,0.036345,0.009256]], #iupac '97
                        'Co':[28,2,[58.933195],[1.0]], #iupac '97
                        'Cu':[29,2,[62.9296011,64.9277937],[0.6917,0.3083]], #iupac '97
                        'Zn':[30,2,[63.9291466,65.9260368,66.9271309,67.9248476,69.925325],[0.4863,0.2790,0.0410,0.1875,0.0062]], #iupac '97
                        'Ga':[31,3,[68.925581,70.9247050],[0.60108,0.39892]], #iupac '97
                        'Ge':[32,2,[69.9242504,71.9220762,72.9234594,73.9211782,75.9214027],[0.2084,0.2754,0.0773,0.3628,0.0761]], #iupac '97
                        'As':[33,3,[74.9215964],[1.0]], #iupac '97
                        'Se':[34,4,[73.9224766,75.9192141,76.9199146,77.9173095,79.9165218,81.9167000],[0.0089,0.0937,0.0763,0.2377,0.4961,0.0873]], #iupac '97
                        'Br':[35,-1,[78.9183376,80.916291],[0.5069,0.4931]],#iupac '97
                        'Kr':[36,0,[77.920386,79.916378,81.9134846,82.914136,83.911507,85.9106103],[0.0035,0.0228,0.1158,0.1149,0.5700,0.1730]], #iupac '97 in air
                        'Rb':[37,1,[84.9117893,86.9091835],[0.7217,0.2783]], #iupac '97
                        'Sr':[38,2,[83.913425,85.9092624,86.9088793,87.9056143],[0.0056,0.0986,0.0700,0.8258]], #iupac '97
                        'Y': [39,3,[88.9058479],[1.0]], #iupac '97
                        'Zr': [40,4,[89.9047037,90.9056450,91.9050401,93.9063158,95.908276],[0.5145,0.1122,0.1715,0.1738,0.0280]],#iupac '97
                        'Nb':[41,5,[92.9063775],[1.0]], #iupac '97
                        'Mo':[42,6,[91.906810,93.9050876,94.9058415,95.9046789,96.9060210,97.9054078,99.907477],[0.1484,0.0925,0.1592,0.1668,0.0955,0.2413,0.0963]], #checked, iupac '97
                        'Tc': [43,2,[96.906365,97.907216,98.9062546],[1.0]], #no natural abundance
                        'Ru': [44,3,[95.907598,97.905287,98.9059393,99.9042197,100.9055822,101.9043495,103.905430],[0.0554,0.0187,0.1276,0.1260,0.1706,0.3155,0.1862]], #iupac '97
                        'Rh':[45,2,[102.905504],[1.0]], #iupac '97
                        'Pd':[46,2,[101.905608,103.904035,104.905084,105.903483,107.903894,109.905152],[0.0102,0.1114,0.2233,0.2733,0.2646,0.1172]], #iupac '97
                        'Ag':[47,1,[106.905093,108.904756],[0.51839,0.48161]], #iupac '97
                        'Cd':[48,2,[105.906458,107.904183,109.903006,110.904182,111.9027572,112.9044009,113.9033581,115.904755],[0.0125,0.0089,0.1249,0.1280,0.2413,0.1222,0.2873,0.0749]],#iupac '97
                        'In':[49,3,[112.904061,114.903878],[0.0429,0.9571]], #iupac '97
                        'Sn':[50,4,[111.904821,113.902782,114.903346,115.901744,116.902954,117.901606,118.903309,119.9021966,121.9034401,123.9052746],[0.0097,0.0066,0.0034,0.1454,0.0768,0.2422,0.0859,0.3258,0.0463,0.0579]], #iupac '97
                        'Sb':[51,3,[120.9038180,122.9042157],[0.5721,0.4279]], #iupac '97
                        'Te':[52,4,[119.904020,121.9030471,122.9042730,123.9028195,124.9044247,125.9033055,127.9044614,129.9062228],[0.0009,0.0255,0.0089,0.0474,0.0707,0.1884,0.3174,0.3408]],#iupac '97
                        'I':[53,-1,[126.904468],[1.0]], #iupac '97
                        'Xe':[54,0,[123.9058958,125.904269,127.9035304,128.9047795,129.9035079,130.9050819,131.9041545,133.9053945,135.907220],[0.0009,0.0009,0.0192,0.2644,0.0408,0.2118,0.2689,0.1044,0.0887]], #iupac '97
                        'Cs':[55,1,[132.905447],[1.0]], #iupac '97
   'Ba':[56,2,[129.906310,131.905056,133.904503,134.905683,135.904570,136.905821,137.905241],[0.00106,0.00101,0.02417,0.06592,0.07854,0.11232,0.71698]], #iupac '97
   'La':[57,3,[137.907107,138.906348],[0.00090,0.99910]],#iupac '97
   'Ce':[58,3,[135.907140,137.905986,139.905434,141.909240],[0.00185,0.00251,0.88450,0.11114]],#iupac '97
                        'Pr':[59,3,[140.907648],[1.0]], #iupac '97
   'Nd':[60,3,[141.907719,142.909810,143.910083,144.912569,145.913112,147.916889,149.920887],[0.272,0.122,0.238,0.083,0.172,0.057,0.056]],#iupac '97
   'Pm':[61,3,[144.91270],[1.0]], #no natural occurence
   'Sm':[62,3,[143.911995,146.914893,147.914818,148.917180,149.917271,151.919728,153.922205],[0.0307,0.1499,0.1124,0.1382,0.0738,0.2675,0.2275]], #iupac '97
   'Eu':[63,3,[150.919846,152.921226],[0.4781,0.5219]], #iupac '97
   'Gd':[64,3,[151.919788,153.920862,154.922619,155.922120,156.923957,157.924101,159.927051],[0.0020,0.0218,0.1480,0.2047,0.1565,0.2484,0.2186]],#iupac '97
                        'Tb':[65,4,[158.925343],[1.0]], #iupac '97
   'Dy':[66,3,[155.924278,157.924405,159.925194,160.926930,161.926795,162.928728,163.929171],[0.0006,0.0010,0.0234,0.1891,0.2551,0.2490,0.2818]], #iupac '97
   'Ho':[67,3,[164.930319],[1.0]], #iupac '97
   'Er':[68,3,[161.928775,163.929197,165.930290,166.932045,167.932368,169.935460],[0.0014,0.0161,0.3361,0.2293,0.2678,0.1493]], #iupac '97
   'Tm':[69,3,[168.934211],[1.0]], #iupac '97
                        'Yb':[70,3,[167.933894,169.934759,170.936322,171.9363777,172.9382068,173.9388581,175.942568],[0.0013,0.0304,0.1428,0.2183,0.1613,0.3183,0.1276]], #iupac '97
   'Lu':[71,3,[174.9407679,175.9426824],[0.9741,0.0259]],#iupac '97
   'Hf':[72,4,[173.940040,175.9414018,176.9432200,177.9436977,178.9458151,179.9465488],[0.0016,0.0526,0.1860,0.2728,0.1362,0.3508]], #iupac '97
   'Ta':[73,5,[179.947466,180.947996],[0.00012,0.99988]], #iupac '97
   'W':[74,6,[179.946704,181.9482042,182.9502230,183.9509312,185.9543641],[0.0012,0.2650,0.1431,0.3064,0.2843]], #iupac  '97
                        'Re':[75,2,[184.9529557,186.9557508],[0.3740,0.6260]],#iupac '97
   'Os':[76,4,[183.952491,185.953838,186.9557479,187.9558360,188.9581449,189.958445,191.961479],[0.0002,0.0159,0.0196,0.1324,0.1615,0.2626,0.4078]],#iupac '97
   'Ir':[77,4,[190.960591,192.962924],[0.373,0.627]], #iupac '97
   'Pt':[78,4,[189.959930,191.961035,193.962664,194.964774,195.964935,197.967876],[0.00014,0.00782,0.32967,0.33832,0.25242,0.07163]],#iupac '97
   'Au':[79,3,[196.966552],[1.0]], #iupac '97
                        'Hg':[80,2,[195.965815,197.966752,198.968262,199.968309,200.970285,201.970626,203.973476],[0.0015,0.0997,0.1687,0.2310,0.1318,0.2986,0.0687]], #iupac '97
   'Tl':[81,1,[202.972329,204.974412],[0.29524,0.70476]], #iupac '97
   'Pb':[82,2,[203.973029,205.974449,206.975881,207.976636],[0.014,0.241,0.221,0.524]],#
   'Bi':[83,3,[208.980383],[1.0]], #iupac '97
   'Po':[84,4,[209.0],[1.0]],
   'At':[85,7,[210.0],[1.0]],
                        'Rn':[86,0,[220.0],[1.0]],
   'Fr':[87,1,[223.0],[1.0]],
   'Ra':[88,2,[226.0],[1.0]],
   'Ac':[89,3,[227.0],[1.0]],
   'Th':[90,4,[232.0380504],[1.0]], #iupac '97
   'Pa':[91,4,[231.03588],[1.0]],
                        'U':[92,6,[234.0409456,235.0439231,236.0455619,238.0507826],[0.000055,0.007200,0.0,0.992745]], #iupac '97
   'Np':[93,5,[237.0],[1.0]],
   'Pu':[94,3,[244.0],[1.0]],
   'Am':[95,2,[243.0],[1.0]],
   'Cm':[96,3,[247.0],[1.0]],
   'Bk':[97,3,[247.0],[1.0]],
   'Cf':[98,0,[251.0],[1.0]],
                        'Es':[99,0,[252,.0],[1.0]],
   'Fm':[100,0,[257.0],[1.0]],
   'Md':[101,0,[258.0],[1.0]],
   'No':[102,0,[259.0],[1.0]],
   'Lr':[103, 0,[262.0],[1.0]],
   'Rf':[104, 0,[261.0],[1.0]],
   'Db':[105, 0,[262.0],[1.0]],
   'Sg':[106, 0,[266.0],[1.0]]
}

#######################################
# Collect properties
#######################################
def getMass(x):
 atom=re.findall('[A-Z][a-z]*',x)
 number=re.findall('[0-9]+', x)
 if len(number) == 0:
  multiplier = 1
 else:
  multiplier = float(number[0])
 atomic_mass=float(matrix(PeriodicTable[atom[0]][2])*transpose(matrix(PeriodicTable[atom[0]][3])))
# That's right -- the molecular weight is based on the isotopes and ratios
 return (atomic_mass*multiplier)

def getCharge(x):
 atom=re.findall('[A-Z][a-z]*',x)
 number=re.findall('[0-9]+', x)
 if len(number) == 0:
  multiplier = 1
 else:
  multiplier = float(number[0])
 atomic_charge=float(PeriodicTable[atom[0]][1])
 return (atomic_charge*multiplier)


#####################################################
# Iterate over expanded formula to collect property
#####################################################
def molmass(formula):
 mass=0
 while (len(formula)>0):
  segments = re.findall('[A-Z][a-z]*[0-9]*',formula)
  for i in range(0, len(segments)):
   mass+=getMass(segments[i])
  formula=re.sub(formula, '', formula)
 return mass

def molcharge(formula):
 charge=0
 while (len(formula)>0):
  segments = re.findall('[A-Z][a-z]*[0-9]*',formula)
  for i in range(0, len(segments)):
   charge+=getCharge(segments[i])  
  formula=re.sub(formula, '', formula)
 return charge


################################################################################
#expands ((((M)N)O)P)Q to M*N*O*P*Q
################################################################################

def formulaExpander(formula):
 while len(re.findall('\(\w*\)',formula))>0:
  parenthetical=re.findall('\(\w*\)[0-9]+',formula)
  for i in parenthetical:
   p=re.findall('[0-9]+',str(re.findall('\)[0-9]+',i)))
   j=re.findall('[A-Z][a-z]*[0-9]*',i)
   oldj=j
   for n in range(0,len(j)):
    numero=re.findall('[0-9]+',j[n])
    if len(numero)!=0:
     for k in numero:
      nu=re.sub(k,str(int(int(k)*int(p[0]))),j[n])
    else:
     nu=re.sub(j[n],j[n]+p[0],j[n])
    j[n]=nu
   newphrase=""
   for m in j:
    newphrase+=str(m)
   formula=formula.replace(i,newphrase)
  if (len((re.findall('\(\w*\)[0-9]+',formula)))==0) and (len(re.findall('\(\w*\)',formula))!=0):
   formula=formula.replace('(','')
   formula=formula.replace(')','')
 return formula


#######
# main #
########
if __name__ == '__main__':
 molecules=molecules.split(',')
 for element in molecules:
  element=formulaExpander(element)
  print ('The mass of %(substance)s is %(Mass)f and the calculated charge is %(Charge)i.' % {'substance': \
   element, 'Mass': molmass(element), 'Charge': molcharge(element)})



02 October 2012

251. Isotopic pattern and molecular weight calculator in Python for Linux

UPDATE: I've moved this code to https://sourceforge.net/projects/pyisocalc/

I'm not answering questions about this code -- it's a work in progress (updated every other day) and if you can't figure out how to use it  on your own, you're not the (currently) intended audience. For example, I've only had time to add a small subsection of the elements.

I originally implemented a very different solution -- a very exact and shiny one. The problem is that the number of permutations increases too rapidly, so that anything larger than e.g. B3(NO3)4 would use up 8 GB of RAM or more. 'Easy' molecules like C18 didn't use that much RAM, but still introduced a noticeable delay. Trimming the list of permutations introduces errors (small, hopefully) but speeds things up orders of magnitude.

In other words: this calculator is moderately fast (python), and very accurate (as far as I can tell). As I keep on looking at more and more complex examples for validation I find that I need to introduce various trimming functions to keep the matrices small.

Having said that, it's still kind of neat. Here's RuCl5^2- by my program and Matt Monroe's calculator (which I trust):


Monroe's output:


And plotting on top (scaled Monroe's by 1.08 to compensate for the error in scaling in Monroe's program which gives 108% abundance):


I removed the figures of W6O19^- since the error in the y axis scale in Monroe's program (went to 120%) made it a less good example, and the list of peaks is too long for easy comparison.
Here's another figure:
A hypothetical W6^- molecule


Anyway, here are a couple of syntax examples:

  Usage:
 ./isocalc 'Al2(NO3)3'
 ./isocalc 'Al2(NO3)3' -1
 ./isocalc 'Al2(NO3)3' -1 output.dat
 ./isocalc Al2N3O9 
  ./isocalc Al(NO3)3(OH)1
  ./isocalc Al(NO3)3(OH)
./isocalc Al

See here for the source code:
https://sourceforge.net/projects/pyisocalc/

01 October 2012

250. Compiling Kernel 3.6 on Debian Testing/Wheezy

IMPORTANT: a lot of drivers need to be explicitly enabled now, in particular those related to video devices (e.g. saa*, uvcvideo)

Nothing difficult or weird about compiling your own kernel, and 3.6 is no exception. This post is pretty much identical to my previous kernel compile posts, but with 3.6 replacing 3.x.
I've successfully compiled 3.6.0, 3.6.2, 3.6.3 and 3.6.6 this way.

If you're wondering whether it's worth it, you can have a look here:
 http://www.h-online.com/open/features/What-s-new-in-Linux-3-6-1714690.html

If you've never compiled a kernel before you might need to:
sudo apt-get install kernel-package fakeroot build-essential

Then get on with it:

wget http://www.kernel.org/pub/linux/kernel/v3.0/linux-3.6.3.tar.bz2
tar xvf linux-3.6.3.tar.bz2
cd linux-3.6/
cat /boot/config-`uname -r`>.config
make oldconfig

You'll get a whole bunch of questions about what to include. See the end of the post for a list.

Video device drivers
Chances are that you will want to do
make menuconfig

and enable the compilation of various video device drivers e.g. go to Device drivers, Multimedia support, star (*) Cameras/video grabbers support, which enables Video Capture Adapters. Go to Video Capture Adapters and enable e.g. V4L USB devices/USB Video Class (m) to get your webcam working. You may need to enable other things too.

 Next:

make-kpkg clean
time fakeroot make-kpkg -j3 --initrd --revision=3.6.3 --append-to-version=-amd64 kernel_image kernel_headers

Building takes a while -- about 39 minutes on an AMD II X3.

mv ../*3.6.3*.deb .
sudo dpkg -i *.deb

And you're done!


What's new:

Expose hardware/virtual IRQ mapping via debugfs (IRQ_DOMAIN_DEBUG) [N/y/?] (NEW) *
Memory Resource Controller for Control Groups (MEMCG) [N/y/?] (NEW)
HugeTLB Resource Controller for Control Groups (CGROUP_HUGETLB) [N/y/?] (NEW)
Virtual (secure) IP: tunneling (NET_IPVTI) [N/m/?] (NEW)
NFQUEUE integration with Connection Tracking (NETFILTER_NETLINK_QUEUE_CT) [N/y/?] (NEW)
CAN Identifier (NET_EMATCH_CANID) [N/m/?] (NEW)     IPset (NET_EMATCH_IPSET) [N/m/?] (NEW)
Three-wire UART (H5) protocol support (BT_HCIUART_3WIRE) [N/y/?] (NEW)
Trace all mac80211 debug messages (MAC80211_MESSAGE_TRACING) [N/y/?] (NEW)   *
Driver for Broadcom BCM8706 and BCM8727 PHYs (BCM87XX_PHY) [N/m/?] (NEW) 
rt2800pci - Include support for rt3290 devices (EXPERIMENTAL) (RT2800PCI_RT3290) [Y/n/?] (NEW) 
MELFAS MMS114 touchscreen (TOUCHSCREEN_MMS114) [N/m/?] (NEW) 
EDT FocalTech FT5x06 I2C Touchscreen support (TOUCHSCREEN_EDT_FT5X06) [N/m/?] (NEW) 
Analog Devices AD-FMCOMMS1-EBZ SPI-I2C-bridge driver (SPI_XCOMM) [N/m/?] (NEW) 
BT8XX GPIO abuser (GPIO_BT8XX) [N/m/y/?] (NEW)   
AMD 8111 GPIO driver (GPIO_AMD8111) [N/m/y/?] (NEW)  
4096-Bit Addressable 1-Wire EEPROM with PIO (DS28E04-100) (W1_SLAVE_DS28E04) [N/m/?] (NEW)
Adaptive Voltage Scaling class support (POWER_AVS) [N/y/?] (NEW) *
Honeywell Humidicon HIH-6130 humidity/temperature sensor (SENSORS_HIH6130) [N/m/?] (NEW) 
BCMA Broadcom GBIT MAC COMMON core driver (BCMA_DRIVER_GMAC_CMN) [N/y/?] (NEW)
Support Wolfson Microelectronics Arizona platform with I2C (MFD_ARIZONA_I2C) [N/m/?] (NEW)
Support Wolfson Microelectronics Arizona platform with SPI (MFD_ARIZONA_SPI) [N/m/y/?] (NEW)
Cameras/video grabbers support (MEDIA_CAMERA_SUPPORT) [N/y/?] (NEW)  
Analog TV support (MEDIA_ANALOG_TV_SUPPORT) [N/y/?] (NEW) 
Digital TV support (MEDIA_DIGITAL_TV_SUPPORT) [N/y/?] (NEW)  
AM/FM radio receivers/transmitters support (MEDIA_RADIO_SUPPORT) [N/y/?] (NEW) 
Remote Controller support (MEDIA_RC_SUPPORT) [N/y/?] (NEW) 
Enable framebuffer console under vmwgfx by default (DRM_VMWGFX_FBCON) [N/y/?] (NEW)  
User-space I/O driver support for HID subsystem (UHID) [N/m/?] (NEW)  
Lenovo ThinkPad USB Keyboard with TrackPoint (HID_LENOVO_TPKBD) [N/m/?] (NEW) 
LED support for LM3556 Chip (LEDS_LM3556) [N/m/?] (NEW)  
LED support for the BlinkM I2C RGB LED (LEDS_BLINKM) [N/m/?] (NEW) 
LED One-shot Trigger (LEDS_TRIGGER_ONESHOT) [N/m/y/?] (NEW) 
EDAC legacy sysfs (EDAC_LEGACY_SYSFS) [Y/n/?] (NEW)  
VFIO Non-Privileged userspace driver framework (VFIO) [N/m/y/?] (NEW) *
Xen platform mcelog (XEN_MCE_LOG) [N/y/?] (NEW) *
Comedi misc drivers (COMEDI_MISC_DRIVERS) [N/y/?] (NEW) *
Comedi PCI drivers (COMEDI_PCI_DRIVERS) [N/y/?] (NEW) *
Comedi PCMCIA drivers (COMEDI_PCMCIA_DRIVERS) [N/y/?] (NEW) *
Comedi USB drivers (COMEDI_USB_DRIVERS) [N/y/?] (NEW) *
CSR wireless driver (CSR_WIFI) [N/m/?] (NEW) *
Pulse-Width Modulation (PWM) Support (PWM) [N/y/?] (NEW) *
Log kernel console messages (PSTORE_CONSOLE) [N/y/?] (NEW)   
Provide swap over NFS support (NFS_SWAP) [N/y/?] (NEW) 
Notifier error injection (NOTIFIER_ERROR_INJECTION) [N/m/y/?] (NEW)
Set upper limit of TLB entries to flush one-by-one (DEBUG_TLBFLUSH) [N/y/?] (NEW)
Serpent cipher algorithm (x86_64/AVX) (CRYPTO_SERPENT_AVX_X86_64) [N/m/y/?] (NEW) 
Twofish cipher algorithm (x86_64/AVX) (CRYPTO_TWOFISH_AVX_X86_64) [N/m/y/?] (NEW)   *
TCM_VHOST fabric module (EXPERIMENTAL) (TCM_VHOST) [N/m/?] (NEW) #

Links to this post:
http://forumubuntusoftware.info/viewtopic.php?f=117&t=8832
http://linux.org.ru/forum/linux-hardware/8339795
http://www.taringa.net/comunidades/ubuntuparataringeros/6513772/_ayuda_compilar-kernel-3_6-en-crunchbang.html
http://kenan.abdullahoglu.com/index.php/bilisim
http://crunchbang.org/forums/viewtopic.php?id=24484
http://forum.teneon.de/index.php?yshout&file=home&history&n=1000

29 September 2012

249. Quick but precise isotopic pattern (isotope envelope) calculator in Octave

UPDATE: Below is an accurate calculator,  but it is impractically slow for large molecules. A practical AND accurate calculator is found here:http://verahill.blogspot.com.au/2012/10/isotopic-pattern-caculator-in-python.html

Use the post below to learn about the fundamental theory, but then look at the other post to understand how to implement it.

Old post:
Getting fast and accurate isotopic patterns can be tricky using tools available online, for download or which form part of commercial packages. A particular problem is that different tools give slightly different values -- so which one to trust?

The answer: the tool for which you know that the algorithm is sound.

The extreme conclusion of that way of thinking is to write your own calculator.
Below is the conceptual process of calculating the isotopic pattern of a molecule using GNU Octave.

You need the linear algebra package:
sudo apt-get install octave octave-linear-algebra

b is the isotopic distribution for an element, and bb are the masses of those isotopes.

Once you've got a computational engine it's not too difficult to expand it for more general cases, account for charge, and instrument resolution.


Molecule: Cl4

b=[0.7578,0.2422];
bb=[34.96885,36.96885];
e=prod(cartprod(b,b,b,b),2);
ee=sum(cartprod(bb,bb,bb,bb),2);
n=4;
g=histc([ee e],linspace(min(ee),max(ee),n*(max(ee)-min(ee)+1)),2);
h=linspace(min(ee),max(ee),n*(max(ee)-min(ee)+1));
distr=e'*g;
plot(h,100.*distr/max(distr))
[h' (100.*distr/max(distr))']
Here's the output for n=1:
   139.87540    78.22048
   140.87540     0.00000
   141.87540   100.00000
   142.87540     0.00000
   143.87540    47.94141
   144.87540     0.00000
   145.87540    10.21502
   146.87540     0.00000
   147.87540     0.81620

And here's the output from Matt Monroe's calculator:
Isotopic Abundances for Cl4
  Mass/Charge Fraction  Intensity
   139.87541 0.3297755   78.22
   140.87541 0.0000000    0.00
   141.87541 0.4215974  100.00
   142.87541 0.0000000    0.00
   143.87541 0.2021197   47.94
   144.87541 0.0000000    0.00
   145.87541 0.0430662   10.22
   146.87541 0.0000000    0.00
   147.87541 0.0034411    0.82


Another molecule: Li2Cl2

Here's the code:
a=[0.0759,0.9241];
aa=[6.01512,7.01512];
b=[0.7578,0.2422];
bb=[34.96885,36.96885];
e=prod(cartprod(a,a,b,b),2);
ee=sum(cartprod(aa,aa,bb,bb),2);
n=1;
g=histc([ee e],linspace(min(ee),max(ee),n*(max(ee)-min(ee)+1)),2);
h=linspace(min(ee),max(ee),n*(max(ee)-min(ee)+1));
distr=e'*g;
plot(h,100.*distr/max(distr))
[h' (100.*distr/max(distr))']

ans =

    81.96794     0.67170
    82.96794    16.35626
    83.96794   100.00000
    84.96794    10.45523
    85.96794    63.71604
    86.96794     1.67079
    87.96794    10.17116

vs Matt Monroe's calculator:
Isotopic Abundances for Li2Cl2
  Mass/Charge Fraction  Intensity
    81.96795 0.0033082    0.67
    82.96795 0.0805564   16.36
    83.96795 0.4925109  100.00
    84.96795 0.0514932   10.46
    85.96795 0.3138084   63.72
    86.96795 0.0082288    1.67
    87.96795 0.0500941   10.17

We can then expand the code to allow for plotting
a=[0.0759,0.9241];
aa=[6.01512,7.01512];
b=[0.7578,0.2422];
bb=[34.96885,36.96885];
e=prod(cartprod(a,a,b,b),2);
ee=sum(cartprod(aa,aa,bb,bb),2);
n=1;

g=histc([ee e],linspace(min(ee),max(ee),n*(max(ee)-min(ee)+1)),2);
h=linspace(min(ee),max(ee),n*(max(ee)-min(ee)+1));
distr=e'*g;
gauss= @(x,c,r,s) r.*1./(s.*sqrt(2*pi)).*exp(-0.5*((x-c)./s).^2);
k=100.*distr/max(distr);

npts=1000;
resolution=0.25;

x=linspace(min(ee)-1,max(ee)+1,npts);
l=cumsum(gauss(x,h',k',resolution));
l=100*l./max(l(rows(l),:));
plot(x,l(rows(l),:))

which gives:

Compare with Matt Monroe's calculator:

28 September 2012

248. Matt Monroe's Molecular Weight Calculator under Wine on Linux

I've downloaded the source code to Matt Monroe's molecular weight calculator in the past, and having replaced wsearch32 (+wine) with OpenChrom I figured I'd go online, download it and see what Mono can do for me. I had a vague recollection that the source code was only freely available online for a short while, and as it turns out I couldn't find it this time.

Anyway, not finding the source code I decided to update my Molecular Weight calculator from version 6.46 to 6.49 which (finally) allows you to set the charge of an ion WITHOUT having the mass of a H+ added for each charge. It's not difficult to compensate for, but it's always confusing to new students.

1. Install Wine and winetricks, add dlls
You can either install wine from the repos (old version)
sudo apt-get install wine-bin

Or you can download a newer, unstable version from dev.carbon-project.org:
http://verahill.blogspot.com.au/2012/01/debian-testingwheezy-64-bit-installing.html

Or you can compile your own:
http://verahill.blogspot.com.au/2013/01/308-compiling-wine-1521-on-debian.html



The mono step was a right headache and would fail unless I nuked everything winetricks and wine knew about each other/

To get winetricks and set everything up:

sudo apt-get install cabextract
wget http://winetricks.org/winetricks
sudo mv winetricks /usr/local/bin/
sudo chmod +x /usr/local/bin/winetricks
wget http://downloads.sourceforge.net/project/wine/Wine%20Mono/0.0.4/wine-mono-0.0.4.msi
wine msiexec /i wine-mono-0.0.4.msi
You're now asked whether to download and install mono...sigh...more often that not this has failed in the past.
winetricks vcrun6sp6
Download the file from the browser window that just opened
cd ~/.cache/winetricks/vcrun6sp6 
mv ~/Downloads/Vs6sp6.exe .
winetricks vcrun6sp6
winetricks corefonts
winetricks riched30
wine uninstaller --remove '{E45D8920-A758-4088-B6C6-31DBB276992E}'
winetricks dotnet20
cd ~/.cache/winetricks/dotnet20/
mv ~/Downloads/dotnetfx.exe .
winetricks dotnet20
Ignore this error. Installation will take a while after that. Have patience. Like 10 minutes kind of patience.
And finally:
winetricks wsh57


2. Download the molecular weight calculator and install
If you go to http://www.alchemistmatt.com/mwtwin.html
you get redirected to here: http://omics.pnl.gov/software/MWCalculator.php


cd ~/tmp
mkdir molw
cd molw/
wget http://omics.pnl.gov/installers/MolecularWeightCalculator_Installer.zip
wget http://omics.pnl.gov/installers/MwtWinDll_SourceAndSupportingDLLs.zip
ls *.zip|xargs -I {} unzip {}
unzip MwtWinDll_Source_v3.4.4518.zip

You'll get some warning about Revisionhistory.txt etc. being overwritten. That's fine.

Launch the install with
wine msiexec /i MolecularWeightCalculator.msi




If you try to launch the mol weight calculator at this point you'll get an error about a missing MwtWinDLL.dll:

So sort that out:

cd ~/tmp/molw/bin
regsvr32 MwtWinDll.dll
Successfully registered DLL MwtWinDll.dll
[If I tried to copy the dll to the wine structure first and register that copy I got:
DllRegisterServer not implemented in DLL C:\windows\system\MwtWinDll.dll]
If it seems weird to install wine-mono and then remove it as is done above, it's to get around a bug which causes dotnet20 installation to fail/


Anyway, you're pretty much done:
 wine ~/.wine/drive_c/Program\ Files/Molecular\ Weight\ Calculator/mwtwin.exe

Yay!






Comment:
Getting there was a bit of a trek, passing though a whole lot of different sets of dlls:
winetricks msflxgrd
winetricks vcrun2005
winetricks vb6run
winetricks mdac28
winetricks comctl32ocx
winetricks comctl32

The solution above should suffice though.

I even ended up installing mol weight calc on a windows box and using dependency walker, but not even that sorted it out -- googling for scrrun did it in the end.

In particular this last error was bloody annoying:
"Object doesn't support this action." What, saving?

"Error saving default options file. Use the /X switch at the command line to prevent this error."
But it was solved by doing winetricks wsh57

27 September 2012

247. Setting up Openchrom (and using it to open Agilent .D ESI-MS files on Linux)

I've been using Wsearch (http://www.wsearch.com.au/wsearch32/wsearch32.htm) to process agilent chemstation ESI-MS spectra for the past few years. It and Matt Monroe's Molecular Weight calculator (why, oh why is there no comparable molecular weight calculator for linux?) have been the only two reasons why I've bother with Wine under Linux. Openchrom is written in java and so will run on both Good (Linux) and Evil (OS X and Win) operating systems.

Having finally discovered openchrom (v 0.6 so still early days) I can now finally retire wsearch from my own computers (it's still a good piece of software, but it's crippled to encourage the purchasing of a 'full' version, and I've had no luck purchasing a license from the author in spite of having tried several times during the past couple of years). OpenChrom can export an entire agilent experiment as a '3D' csv file which makes processing a lot more fun.

As an aside: I hate proprietary file formats since they prevent me from using my own tools (cat, sed, gawk, gnuplot, octave) when processing -- or at a minimum make it more difficult. Most universities and grant agencies now add a provision regarding data management in their grant acceptance agreements/work conduct policies. In general these provisions state that the data shall be made available publicly and /or managed by a university repository. What is REALLY missing is a clause about using open formats -- and that should be taken into account when acquiring new instrumentation. All else being equal, an instrument which is 'open' will be a lot cheaper to manage in the long run since you won't have to feel locked in in terms of software. That's incidentally a reason why I like Metrohm since they provide details of their RS-232 interface allowing you to write your own software.

Anyway, here's how to get set up:


1. Install Java v1.7 (need > 1.6)
You can either use openjdk 7 or (Oracle) Java. See here for a general guide to installing Oracle/Sun Java.

As for openjdk, you can easily install it:
sudo apt-get install openjdk-7-jdk

(the openjdk-7-jre package is enough if you don't want the full developer's kit)

Anyway.

Make sure that you've selected the right version:
 sudo update-alternatives --config java
There are 7 choices for the alternative java (providing /usr/bin/java).

  Selection    Path                                            Priority   Status
------------------------------------------------------------
  0            /usr/lib/jvm/java-6-openjdk-amd64/jre/bin/java   1061      auto mode
  1            /usr/bin/gij-4.4                                 1044      manual mode
  2            /usr/bin/gij-4.6                                 1046      manual mode
  3            /usr/bin/gij-4.7                                 1047      manual mode
  4            /usr/lib/jvm/j2re1.6-oracle/bin/java             314       manual mode
  5            /usr/lib/jvm/j2sdk1.6-oracle/jre/bin/java        315       manual mode
  6            /usr/lib/jvm/java-6-openjdk-amd64/jre/bin/java   1061      manual mode
 *7            /usr/lib/jvm/java-7-openjdk-amd64/jre/bin/java   1051      manual mode



2. Get openchrom
cd ~/tmp
wget http://sourceforge.net/projects/openchrom/files/REL-0.6.0/openchrom_linux.gtk.x86_64_0.6.0.zip
unzip openchrom_linux.gtk.x86_64_0.6.0.zip
cd linux.gtk.x86_64/OpenChrom/
sudo mkdir /opt/openchrom
sudo chown $USER /opt/openchrom 
cp * -R /opt/openchrom
chmod +x /opt/openchrom/openchrom

Stick

alias openchrom='/opt/openchrom/openchrom'

in your ~/.bashrc and source it.




3. Get plugins
On first boot you're asked whether you want to get additional plugins using the 'Openchrom marketplace'. Since I'm mainly processing data from an Agilent ESI-MS, I wanted the plugin for Agilent files. The website says that you need a license key for plugins BUT that it's free to register for one.

This is a 30-days trial version. Afterwards, you need a valid serial key.
You can get a free serial key after registration on http://www.openchrom.net.
You can use the converter for commercial or non-commercial purposes free of charge, but you are not allowed to redistribute this software without my permission.

Note, that clicking on links on the website didn't lead me to a link to download the plugin. Instead, in OpenChrom click on the Plug-ins menu:






As always, make sure you trust your suppliers.


And then you're done installing.

There's nothing odd about registering other than this: you will receive an email with a confirmation of your registration in clear text WITH YOUR PASSWORD. So...be aware of that.


4. You can now browse in the tree to the left and select your .D folder:

There's a bit of clever thinking when it comes to the functionality of the program. The upside of this I think will eventually be that it's easy to get a consistent experience for a set of users (not unimportant for a research group). The downside is that it's a bit clunky getting started. Play with it for an hour and you'll get the hang of it, so it's not really that much of a hurdle. Also, too many options seem to be context sensitive -- I am having real trouble finding various options under the 'Accurate' perspective which I can find under the 'default' perspective.


5. Some comments:

It's still early days for OpenChrom (v 0.6) , and there are a few minor issues which may or may not affect you:

* Registration keys. They are easy enough to get (register online, log in, click on the plug in online that you want and you'll see the key), but if you have installed a new plug in and open your first spectrum right after that you'll be asked for registration keys. It won't tell you for which plug in the dialogue you're seeing is though, so if you've just installed three different plug ins you'll have to do some trial-and-error. This is fixed in the upcoming version.

* Raw/gaussian plot of mass spectrum. This took a while to figure out, but you have to use perspectives. The default (heavily zoomed in) view looks like this:
This might be good enough for those organic types...us 108 element inorganic types want more detail
If you go to the top right corner, click on 'other' (perspectives)
and select accurate you get
Bingo!
And then there's the obligatory wish list:

* A good quality isotopic pattern calculator would be nice. Anyone who has compared the output from different pieces of software will have discovered that different calculators may yield very different patterns. I think some of it boils down to truncation rather than incorrect isotopic ratios, but that just highlights how difficult it can be to implement a seemingly simple concept. The only calculator which I trust AND find useful is Matt Monroe's calculator -- the predicted patterns look good, and you get proper Gaussian broadening which means that it looks 'right'. This would be perfect as a plug-in. If only I knew how to properly implement it...

* A good quality ion generator -- some pieces of software (Hi Matt) allow you to select a handful of elements or fragments, pick a range of charges, input an m/z value and based on that spits out a list over possible identities for your signal. It's a good thing to have by your side the first time you look at a complex mixture trying to figure out what products may be present. This would be perfect as a plug-in. I've written this type of programmes before, but in python using for-loops...a vectorized version should be faster and maybe even easier to write.