10 August 2011

15. ORTEP-3 for Windows (Winortep) under Wine

Note: for those who use Mac OS X and get sent here by google, you can also use wine to run ortep3v2. See here for instructions on how to install wine: http://wiki.winehq.org/MacOSX

The installation file can be downloaded from http://www.chem.gla.ac.uk/~louis/software/ortep3/. Install in the same fashion as you would with any other windows program under wine. I will assume that you have installed it in c:\ortep3v2

Request a license the usual way. Make a file called Ortep3-license in the c:\ortep3v2 directory (/home/you/.wine/drive_c/ortep3v2). Put the license information in it:

Program license for : Ortep3 for Windows
User name  : XXXXXXXXXX
User email : XXXX@XXXXX.XXX
User affiliation : Department of Chemistry, XXXXX, XXXXX
Windows version  : Windows XP
License number   : XXXXXXXXXX

So far everything is the same as for a regular windows installation.
The one difference arrives when it comes to setting the ORTEP3DIR variable:
go to the linux shell, type regedit
Navigate to HKEY_CURRENT_USER/Environment
If you don't have the Environment directory, create a new key with that name, then create a string value which you call ORTEP3DIR. Double-click on the data field, and set it to c:\\ortep3v2

You should now be ready to go.

Winortep can write pov-ray files -- and they are well-written, logical and very easy to read/modify

WinOrtep is a full-featured program which can do just about anything.

08 August 2011

14. Evolution email and Google Calendar(s)

The process is already described here:

It's not much of a post, but it may be of help to someone who tried the Google option when setting up the calendar sync in Evolution and found it less than satisfactory.

Edit 16/12/2011:
Recently (last two days, version 3.0.3-3) been getting error messages in evolution when trying to connect to my google calendar. While using the method described above for a while I switched to using 'google' instead of 'caldav' as the calendar type at some point. Well, it doesn't work anymore.

Here's how to use google calendar in Evolution:
New -> Calendar
Type -- CalDav

URL: caldav://www.google.com/calendar/dav/myemail%40gmail.com/events/
Use SSL: yes
User name: myemail@gmail.com
User email: myemail@gmail.com

Then clock 'Browse server for calendar and select the
+ /calendar/dav/myemail%40gmail.com/
         - myemail@gmail.com

Whether  you want to copy content offline or use it as default calendar is a personal choice.

Edit: 16/12/2011 -- 2 hours later
Well, it's not working anymore. Same TLS handshake BS as before.

"Error loading calendar
Cannot open calendar: Unexpected HTTP status code 6 returned (Peer failed to perform TLS handshake)"

What I can't understand is why there is no good description of this problem on the web, given that Evolution is the default email and calendar client for gnome. I don't like thunderbird/icedove+sunbird/iceowl much, but things seem to work better in general there.

07 August 2011

13. phpSheduleIt - basic installation

phpScheduleIt is a nice web-driven instrument scheduling system, which I first come into contact with as an NMR user at the University of California at Davis, where it is used to manage the scheduling of the instruments in the NMR facility there.

You will need to have a LAMP server set up (see e.g. http://www.howtoforge.com/ubuntu_debian_lamp_server). You will need apache2, php and mysql to play nicely together. Basically follow steps one and two, then sudo /etc/init.d/apache2 restart.

Download phpScheduleIt from http://www.php.brickhost.com/

A basic way to get it up and running goes as follows:

1. create a directory under /var/www. In our example it will be /var/www/nmr
sudo mkdir /var/www/nmr. Unzip the file you downloaded from www.php.brickhost.com and put the files in the /var/www/nmr directory

2. in your terminal, go to /var/www/nmr and chmod o+x all files (including subdirectories)

3. cp /var/www/nmr/config/config.php.new /var/www/nmr/config/config.php

4. Edit it.
change $conf['app']['weburi'] = 'http://localhost/phpScheduleIt';

change $conf['app']['weburi'] = 'http://localhost/nmr'; 
edit other relevant items, such as adminEmail, defaultLanguage, timeFormat, emailType, defaultPassword etc,

The admin email is important, since this will become the administrator account.

5. go to http://localhost/nmr/install
If nothing happens, or you get an error message, it's time to start trouble-shooting. Going to the terminal and
cd /var/www/nmr/install
php index.php
might give some useful information. Most likely you have either a) made an error in config/config.php, or b) you haven't chmod:d the subdirectories.

6.  If all goes well, you are asked for your root user name and root mysql password (which you set during the installation). Thus, login as root with the correct password. Once you're in, hit the button saying 'create tables'. If all went well you can now delete the /var/www/nmr/install directory.

7. Go to http://localhost/nmr
You should be greeted with a log in window. Important: select create new user and enter the email address you gave as the administrator email in the config.php file. This will then automatically become an administrator account. It has nothing to do with the order you create accounts in - it's all about what email address you put in the config.php file. 

8.  Once you've set up the account you are able to create instruments, user groups, booking restrictions etc. 
Anyone who wants to use the facilities can create their own new registered users, BUT only the administrator(s) can assign resources to them. 

9. Be careful with your GMT setting...

This page has some information as well: http://www.m-osaka.com/jp/plaza/readme.html

31 July 2011

12. Pulseaudio - controlling where the output goes

The non-free flash plugin can be a bit of a headache, since even if you install it and flash videos play nicely, you may not get any sound. On a laptop, you might not notice it, since the default output seems to be the laptop speakers. However, you may have problems piping the sounds through any attached USB speakers. Same goes for desktops, where the default output may be the sounds ports on your motherboard.

The fix is simple (assuming you have pulseaudio installed)
create an /etc/asound.conf file or a ~/.asoundrc file, with the following in it:

pcm.!default.type pulse
ctl.!default.type pulse

Rebooting should take care of things

11. Sorting out problems with NVIDIA or ATI graphics cards on Debian Testing

This bug has been a problem for a while now when running debian testing - if you do a dist-upgrade, chances are your prorietary drivers get borked.

Symptoms include gdm not starting i.e. no Gnome login window. Instead you get dumped into the terminal.

There are two fixes - one quick and dirty one and one which is more long term.

1. Quick and dirty
If you want Gnome up and running without necessarily using graphics acceleration, simply log in in your terminal and rename your xorg.conf.

sudo mv xorg.conf xorg.conf.old

This will allow you to use the graphics card on your motherboard again. It won't help with a PCI card much.

2. A real fix
SMXI. It may seems scary, but is fairly straightforward. Log in (presumably in the terminal)

sudo su
cd /usr/local/bin && wget -Nc smxi.org/smxi.zip && unzip smxi.zip && smxi

The sudo su logs you in as root. You could also sudo each command individually above.

Follow the instructions and let the smxi script install your graphics drivers.

28 July 2011

10. Combining MS data into a matrix

The following script takes a series of files with comma-delimited data named 0v.csv, 10v.csv etc., extracts everything in column 4 (in my case it contains relative abundance values from MS) from row 7 to the bottom, and stores those columns in files called 0.dat, 10.dat etc.

The second column (in my case it contains m/z values) is extracted from one of the csv files, and stored in mz.x.

Next, all the .dat files are pasted together, with the columns side by side, and the mz.x data added as the first column.

The data is rotated using the rotate script I've published earlier.

A file with cone-voltage values, cv.xy, has already been prepared by hand (single column), and is pasted together with the m/z data.

for e in 0 10 20 30 40 50 60 70 80 90 100 110 120 130 140 150 160 170 180 190 200 220 240 260 280 300
tail -n +8 $e\v.csv | sed 's/\,/\t/g'| gawk '{print $4}' > $e.dat

#get m/z
tail -n +8  0v.csv | sed 's/\,/\t/g'| gawk '{print $2}' > mz.x

paste  0.dat 10.dat 20.dat 30.dat 40.dat 50.dat 60.dat 70.dat 80.dat 90.dat 100.dat 110.dat 120.dat 130.dat 140.dat 150.dat 160.dat 170.dat 180.dat 190.dat 200.dat 220.dat 240.dat 260.dat 280.dat 300.dat > all.dat
paste mz.x all.dat > $1dat
rotate $1.dat > $1\rot.dat
paste cv.xy $1\rot.dat > $1\tot.dat

26 July 2011

9. rotating a matrix

I lifted this off of some website at some point, but can't remember where and so can't give credit.

gawk '
    for (i=1; i<=NF; i++)  {
        a[NR,i] = $i
NF>p { p = NF }
END {  
    for(j=1; j<=p; j++) {
        for(i=2; i<=NR; i++){
            str=str" "a[i,j];
        print str
}' $1

8. Very rudimentary peak-finding

The following scripts looks through a file with one x column and several y columns, and returns values larger than a certain cutoff.

findpeaks filename columnnumber cutoffvalue
findpeaks test.dat 3 0.01

import sys



for spectrum in spectra:
spectrum=spectrum.split(' ')
# print spectrum

if spectrum[colno]>cutoff:
print spectrum[1],' ',spectrum[colno]

if spectrum[colno]<minsig:
if spectrum[colno]>maxsig:

print "not working: "

print "max sig: ",maxsig,", minsig: ",minsig

20 July 2011

7. Processing 1D Bruker nmr data

Bruker 1D binary NMR files can be processed using a combination of cat, grep, sed, gawk and od, together with python and octave (w/ octave-optim) for some fancy line-fitting.

#usage: brukdig2asc
SW=`cat acqus | grep 'SW_h' | sed 's/\=/\t/g' | gawk '{print $2}'| tr -d '\n'`
TD=`cat acqus | grep 'TD=' | sed 's/\=/\t/g' | gawk '{print $2}'| tr -d '\n'`
O=`cat acqus | grep '$O1=' | sed 's/\=/\t/g' | gawk '{print $2}'`
SFO=`cat acqus | grep 'SFO1=' | sed 's/\=/\t/g' | gawk '{print $2}'`
#AQ=`echo "1/(23809.5238095238/(16384/2))" | bc -lq`
cp fid fid.bin
ls fid.bin | cpio -o | cpio -i --swap -u
od -An -t dI -v -w8 fid.bin| gawk '{print NR,$1,$2}'| sed '1,64d' >fid.asc1
pynmr $SW $TD $O $SFO

import sys
#print str(sys.argv)

#print str(sweepwidth),str(nopts)
for line in f:
    line=line.split(' ')
#    print line


octave --silent --eval "fid=load('fid.asc');
#make xaxis
[nopts b]=size(fid);


spec=[fid(:,1) real(fftshift(fft(fid(:,2)+i*fid(:,3)))) imag(fftshift(fft(fid(:,2)+i*fid(:,3))))];
[a b]=size(spec); spec(a/2,2:3)=[0 0];
for n=1:180;
        localmax=max( real( (spec(:,2)+i*spec(:,3)).*exp(i*phc(n)) ));
        if (localmax>maxsig)
#simple baseline
[f m kvg iter corp covp covr stdresid z r2]=leasqr(fid(:,4),real((spec(:,2)+i*spec(:,3)).*exp(i*phc(k))),guess,absd);

#make spectrum
spectrum=[fid(:,4) real((spec(:,2)+i*spec(:,3)).*exp(i*phc(k)))-m imag((spec(:,2)+i*spec(:,3)).*exp(i*(phc(k)+pi/2)))-m];

pkg load optim
[a b]=max(spectrum(:,2));
guess=[10 max(spec(:,2))]; #centre width height
[f p r2]=leasqr(fid((b-150):(b+150),4),spectrum((b-150):(b+150),3),guess,lorentzian);

#filter out artefacts from fitting set
filtered=[0 0];
for l=(b-ceil(5*p(1)/res)):(b+5*ceil(p(1))/res)

if (delta>(lorentzian(fid(l,4),p))/1.2)
# do nothing
filtered=[filtered; fid(l,4) spectrum(l,2)];

filtered=[ filtered(2:size(filtered(:,2)),1) filtered(2:(size(filtered(:,2))),2)  ];
[f p r2]=leasqr(filtered(:,1),filtered(:,2),p,lorentzian);

params=[centre centre/67.8 max(lorentzian(fid(:,4),p)) p(1) 1.000 p(2)];
spex=[fid(:,4) real((spec(:,2)+i*spec(:,3)).*exp(i*phc(k)))-m imag((spec(:,2)+i*spec(:,3)).*exp(i*(phc(k)+pi/2)))-m lorentzian(fid(:,4),p)];
save spectrum.dat spex;"

6. Miscellaneous scripts

The following two scripts look through a set of files with paired xy values but not necessarily the same number of x values in each file, extracts all the x values from all the files (script numero uno), then makes sure that all x values are present in all files (and zero-fills to make up for missing data). In short, it's a way of creating files that can be used to generate a matrix of values where all rows have the same number of fields.

The first script:
for e in {0..200..10} {220..300..20}
        cat $e.dat | gawk '{ printf("%s\t",$1) }'
        echo ""

The second script:

import sys


for line in f:
        for i in line:
            if i not in arr:


last = mylist[-1]
for i in range(len(mylist)-2, -1, -1):
    if last == mylist[i]:
        del mylist[i]
        last = mylist[i]


for n in voltages:
    print "voltage: ',n,'\n'

    for line in f:
        line=line.split(' ')
            #do nothing
            print "fail"

    for i in range(0,len(arrx)-1):


    for i in range(0,len(myys)-1):


12 June 2011

5. Debian testing - Error cracking CSS key. Check regionset

I don't normally buy or watch DVDs, since the movie studios and distributors are not treating the linux community fairly. However, I was recently given a movie to watch, which turned out to not be as straight-forward as I would've liked it to be.

Long story short, in spite of following the usual routine of installing libdvdcss2 and other relevant packages, the movie would not play. Starting a movie player from the command line, I could see what error messages where associated with the failure to play, and the key was "Error cracking css key".

After having tried a few different things, I ended up installing regionset. When running regionset (sudo regionset) it did not echo a value for the current region which my dvd player was set to - apparently the device had not been locked to a region yet. Setting the region of my dvd player to the region of the dvd caused playback to work for all movie player clients.

Whether playback did not work because the
1. region was not set at all
2. region was not set to the same region as the DVD
remains to be found out.

Typically, you can only change the region of your playback device a limited number of times (3-5), so you will want to change it as few times as possible.

31 May 2011

4. Getting mouse and keyboard to work on Lenovo Thinkpad SL410 under Debian Testing/Wheezy

For the past few months (it seems) Debian Testing will install just fine on system. On the first boot, however, neither (internal or external) mouse nor keyboard will be responsive and to a casual observer it will appear that the system has hanged. This is fairly well documented and the solution is simple if you have an EXTERNAL keyboard and mouse available (but I'll show it here anyway):
1. Disconnect and reconnect your mouse and keyboard, or, on a laptop, connect a usb keyboard, so that you can log in.
2. Open the terminal  or a console (e.g. ctrl+alt+f1) and proceed to rename /run e.g. sudo mv /run /xrun

You could also delete the folder instead of renaming it. At this point you simply have to reboot and everything should be working without any reconnecting.

NOTE that everything will be responsive during the inital part of the boot - the mouse and keyboard do not stop working until startx/gdm3 starts.

If you do not have an external keyboard you would be advised to install openssh-server during the installation of Debian. This way you should be able to ssh to your laptop from another computer and fix it remotely.

I've set up four systems - three desktops and one laptop - using this approach, and all has been well. The past week a laptop that was set up this way suddenly booted into gdm3 with an unresponsive mouse and keyboard. Plugging in an external keyboard (which I do anyway to slow down the onset of carpal tunnel from using a keyboard at weird angles - I'm not much of a fan of touchpads either) made my laptop usable, but the problem has been annoying - a system which you use for work should be in, well, working order.

At any rate, the /run folder was back. I tried just removing /run/udev, and it didn't help. I couldn't easily remove /run, since it was locked, but by attaching the hdd to another system I could remove it. On boot I got a number of error messages saying that various folders under /var/run/ could not be accessed, and I never made it to either gdm3 or a console. Booting by selecting the 'rescue/recover' option in grub dropped me into a terminal, but I could little beyond creating a /run folder (sudo mkdir /run).

ls /var/run -lah showed that /var/run was linked to /run. /var/lock had also shown up in various error messages. I renamed the /var/run and the /var/lock files (xrun and xlock), made sure that there was no /run folder, and rebooted. Still not happy and no console - complains about various things missing. Booting using the recover option (or is it rescue? Whatever, you'll know when you see it) created the necessary files, but did not create a /run folder. Rebooting and selecting the normal startup option in grub now went off without a hitch, with gdm3 behaving and the touchpad working. It took a few nervous seconds after the start of gdm before the mouse became responsive, but work it did.

So, if you're having the same problem, and you haven't had much luck deleting /run, have a look at /var and see if there are any links.

Now, the big question is: why was /run recreated in the first place?
Links to this post:

28 January 2011

3. Compiling nwchem on Ubuntu 10.10 64 bit

** See post on 15/12/2011 for information about Debian 64 bit. It builds fine on Squeeze but not Wheezy. This seems to have to do with the version of mpich2.**

Figuring out how to compile nwchem with mpich support took a little while, but this seems to have worked:

First mpich2 and gfortran need to be installed (since it was not installed on a virgin system there may have been other required packages already installed)
sudo apt-get install mpich2 gfortran

I created a file called myconfig.sh in the nwchem directory, with the following content:

setenv TCGRSH /usr/local/bin/ssh
setenv NWCHEM_TOP /work/nwchem
setenv USE_MPI y
setenv USE_MPIF y
setenv MPI_LOC /usr
setenv MPI_LIB $MPI_LOC/lib
setenv MPI_INCLUDE $MPI_LOC/include/mpich2
setenv LIBMPI "-lfmpich -lmpich"
cd $NWCHEM_TOP/src
make nwchem_config
make FC=gfortran >& make.log

do a csh myconfig.sh and you should be good to go.

I then added the following to the end of my ~/.bashrc and sourced it:

export NWCHEM_EXECUTABLE=/work/nwchem/bin/LINUX64/nwchem

Jobs can then be submitted (assuming that mpd is up) by
mpdrun -n 2 nwchem nameofjob.nw

09 January 2011

2. Setting up a cheap-ish ubuntu cluster - the hardware

I've recently  become interested in setting up a small computational cluster for simple QM and MD work. Since it doesn't hurt sharing your experience with others, I'll post the details on this blog.
I'm not a computer/linux/computational chemistry expert - the goal of this project is as much to learn how to go about setting up a cluster, as getting a useful computational tool.

First, the hardware - the idea was to get something fairly cheap, but also reasonably useful. I opted for the following hardware (prices in USD):

30.99    V7 NS1132-N6 10/100/1000Mbps Unmanaged Networking Switch 5 x RJ45 2K MAC Address Table

Each node
39.99    BIOSTAR MCP6P3 AM3 NVIDIA GeForce 6150 / nForce 430 Micro ATX AMD Motherboard
77.00    AMD Athlon II X3 445 Rana 3.1GHz 3 x 512KB L2 Cache Socket AM3 95W Triple-Core Desktop Processor 
2*21.99    Mushkin Enhanced 2GB 240-Pin DDR3 SDRAM DDR3 1333 (PC3 10666) Desktop Memory Model 20-146-744
7.99    NIC Zonet ZEN3301E 10/ 100/ 1000Mbps PCI 32-Bit Network Adapter 1 x RJ45
24.99    APEX TX-381-C Black Steel Micro ATX Tower Computer Case (needs full size ATX PSU)
13.99    LOGISYS Computer PS480D 480W ATX12V Power Supply
3.99    APEVIA CF12S-BK 120mm Case Fan 

I ordered everything from Newegg, and all the parts arrived after three days, as promised.

I already had two SATA harddrives, and two IDE CD ROM drives, in addition to two IDE/PATA cables. The CD ROM drives aren't important, as the mobo bios allows for boot from USB, so you can install your OS that way (at least Ubuntu 10.10).

Assembling the two systems was easy, even though it was the first time I had done anything more complicated than installing a RAM stick or changing a hard drive. I'm not particularly good with my hands, so that really says something.

Here are the steps, in case someone is as useless as I am with hardware:
Preparing the motherboard:
1. The CPU was fitted on the motherboard - being an AM3 socket it was a simple matter of gently dropping the cpu in the socket with the yellow marking on the cpu being aligned with the yellow marking on the mobo.
2. Next the heatsink/fan was fitted. The heatsink arrived with a smooth layer of thermal grease already applied. While it's difficult to see whether the heatsink was in perfect position over the cpu, I did my best and everything went well.
3. The RAM memory was installed. Nothing complicated here.

Preparing the case:
1. The PSU was installed - nothing complicated. Just four screws to attach the PSU to the case.
2. The motherboard panel at the back of the case didn't fit my mobo, so I installed the one which came in the motherboard box. One of the PCI slot covers on the case was removed to make space for the gigabit ethernet card
3. The brass spacers that came with the case were attached to the wall of the case. The spacers are for the mobo to rest on. If you don't install them the mobo won't align with the openings at the back of the case, so there's little risk that you'll forget to do this.
4. A CD drive (IDE) and a harddrive (SATA) were installed, but not connected.

1. The motherboard was attached to the case by resting it on the brass spacers and aligning the holes in the mobo, then attaching the mobo with screws.
2. All the cables connected to the case were clearly labelled, as were the corresponding connections on the motherboard, so it was just a matter of matching one to the other.
3. The PSU cables were fairly simple to connect two - two cables were attached to the corresponding connections on the mobo (just going by shape), the sata power cable was connected to the HDD and a molex connector was connected to the IDE CD ROM drive.
4. The sata cable that came with the case was used to connect the motherboard with the harddrive. I had a couple of IDE cables around, and used it to connect the CD ROM drive to the motherboard.  It took two attempts to get the IDE thingy right as I first attached the cable the wrong way round - for an IDE cable with three plugs (one on each end, one slightly offset from the middle of the cable) you want to connect the end with the 'middle' connector closer to it to the CD ROM, and the other end to the MOBO. Makes sense, but getting it right the first time would save you a few minutes. Of course, an easier way of getting around this is probably by setting the jumper to cable-select.

One system was installed by booting from a Ubuntu 10.10 64 CD, and the other by booting from a USB stick prepared using usb-creator-gtk. I could've settled on minimal/server installs instead of full Ubuntu Desktop installs, but as I 1) will later modify grub to boot to terminal instead of starting Gnome, 2) don't have space contraints and 3) want to be able to use the computers as back-up workstations I opted for full installs.

The systems were updated after the install.

02 January 2011

1. Setting up a cheap-ish ubuntu cluster - basic set-up

Note: I have two nodes - one master (beryllium) and one slave (lithium). These names will be used in the examples

Once all the hardware was assembled and linux up and running, I installed a host of programs that I'd be using:

Cluster tools:
sudo apt-get install mpich2 vnstat sinfo ganglia-monitor openmpi-*

For compiling:
sudo apt-get install build-essential gfortran gpp cpp 

Generally useful software:
sudo apt-get install gnuplot mpqc-openmpi gromacs-mpich gomacs-mpich octave octave-parallel qtoctave gnome-activity-journal maxima

I downloaded the nwchem source from here. We'll deal with the compilation and installation in the next post.

I followed this post to get up and running (scroll down) with nfs. Here's my take on it:

On the master node:
sudo apt-get install nfs-kernel-server nfs-common portmap

    We don't want portmap to use, so
sudo dpkg-reconfigure portmap

sudo /etc/init.d/portmap restart
sudo mkdir /work
sudo chmod a+wrx /work

sudo nano /etc/exports
  Add the following line to the end of the file:
 /work *(rw,sync)

sudo /etc/init.d/nfs-kernel-server restart
sudo exportfs -a

One the slave node:
sudo apt-get install portmap nfs-common
sudo mkdir /work

sudo nano /etc/fstab
   Add the following line:
beryllium:/work /work nfs   rw   0   0

To get it up and running immediately
sudo mount /work

Next, create a file called mpd.hosts - you can use the /work directory or the home directory. List the hosts/nodes and add, without spaces, the number of processors on each node:


On beryllium
mpd --ncpus=3 &
mpdtrace -l

which returns something akin to beryllium_12345 ( 12345 is the port number, which will change each time you start mpd

On lithium
mpd --ncpus=3 -h beryllium -p 12345 &

NOTE: I had problems before specifying --ncpus - gave an error about mpich2 consecutive number/IDs. The down-side is that when you submit a job it fills up the master node, then starts jobs on the slave node(s).