11 January 2013

316. Briefly: Automated chroot/sandbox creation

There are plenty of reasons to use chroots, but security is not one of them.

For a practical how-to see e.g

For a bit of yelling, see

chroot will improve your security by creating an obstacle which may filter out some would-be crackers, but it will not make it secure by any standard. (in spite of what I may have written elsewhere on this blog).

Original post:
I've been using chroot to compile and test stuff so much lately that I figure it was time to automate the process.

Before creating your chroot you'll need a few packages:
sudo apt-get install debootstrap coreutils x11-xserver-utils

The scripts
mkdir $HOME/tmp/jail/$1 -p sudo debootstrap --arch amd64 testing $HOME/tmp/jail/$1 http://ftp.au.debian.org/debian/ sudo cp setupchroot.sh $HOME/tmp/jail/$1/

rm /etc/apt/sources.list echo 'deb http://ftp.au.debian.org/debian/ wheezy main contrib non-free' >> /etc/apt/sources.list apt-get update apt-get install locales sudo vim echo 'export LC_ALL="C"'>>/etc/bash.bashrc echo 'export LANG="C"'>>/etc/bash.bashrc echo 'export DISPLAY=:0.0' >> /etc/bash.bashrc echo ' beryllium >> /etc/hosts' source /etc/bash.bashrc adduser sandbox usermod -g sudo sandbox echo 'Defaults !tty_tickets' >> /etc/sudoers

xhost + sudo mount -o bind /proc $1/proc sudo cp /etc/resolv.conf $1/etc/resolv.conf sudo chroot $HOME/tmp/jail/$1

How to use
To set up the chroot:
sh makechroot.sh mynewchroot
sudo chroot mynewchroot
root@beryllium:/# sh setupchroot.sh

To use the chroot:
sh launchchroot.sh mynewchroot

Once you're done with the chroot and logged out, do
sudo umount $HOME/tmp/jail/mynewchroot/proc

to unmount the /proc -- you can now delete, copy etc. the directory structure of you chroot.

315. Briefly: Compile firefox 18 on debian testing/wheezy

For no better reason than pure curiosity, here's how to build firefox 18. My regular build environment has so many packages installed these days that it's useless for working out dependencies for other people, so I tried it in a fresh chroot to hammer out the current dependencies beyond a doubt. In other words: it will work for you.

In retrospect I'm not sure if you really need the dev package for libgtk2.0.

sudo apt-get install bzip2 build-essential python zip unzip pkg-config libgtk2.0-dev gconf2 libdbus-1-dev libdbus-glib-1-dev yasm libasound2-dev libcurl4-gnutls-dev libxt-dev mesa-common-dev
mkdir ~/tmp/firefox -p
cd ~/tmp/firefox
wget ftp://ftp.mozilla.org/pub/firefox/releases/18.0/source/firefox-18.0.source.tar.bz2
tar xvf firefox-18.0.source.tar.bz2
mkdir build/
cd build/
../mozilla-release/./configure --prefix=$HOME/.firefox18
make install
echo "alias firefox18='$HOME/.firefox18/bin/firefox'" >> $HOME/.bashrc
source $HOME/.bashrc

Launch by typing

10 January 2013

314. Briefly: Installing talkd on ROCKS 5.4.3

I was asked to set up talkd on our Rocks 5.4.3 cluster (Centos 5.4) . There's no talkd or talk-server packages in the repos on that server.

Note: The general consensus seems to be that talk is
1. insecure and
2. outdated.

To install:
wget http://mirror.centos.org/centos/5/os/x86_64/CentOS/talk-server-0.17-31.el5.x86_64.rpm
yum localinstall talk-server-0.17-31.el5.x86_64.rpm
sudo iptables -A INPUT -p udp --dport 517 -s -j ACCEPT
sudo iptables -A INPUT -p udp --dport 518 -s -j ACCEPT

Above I've added, but haven't yet tried, -s to limit connections from the local computer (localhost). If it doesn't work with -s, then try without -- but then be aware of the security implications. These rules also aren't permanent and will be lost on reboot. To make them permanent, edit /etc/sysconfig/iptables.

I couldn't get talk to work before opening the ports and would get
Error on read from talk daemon : Connection refused
Note that talkd uses Xinet and not init -- it will not run as  memory resident daemon, but instead be launched by xinet only when talkd is needed (traffic is detected to the ports associated with talkd). Xinet.d is a bit like a concierge, waking up whomever seems to be the adressee.

Edit both /etc/xinetd.d/talk and /etc/xinetd.d/ntalk. Change to:
# default: off # description: The talk server accepts talk requests for chatting with users \ # on other systems. service talk { flags = IPv4 disable = no socket_type = dgram wait = yes user = nobody group = tty server = /usr/sbin/in.talkd }
Finally, restart xinet.d (doing chkconfig talk on wasn't enough):
sudo /etc/init.d/xinet.d restart

Note: If you or the other user have several terminals open you should figure out which terminal to use. If you're user2, do
PID TTY TIME CMD 5455 pts/23 00:00:00 bash 9321 pts/23 00:00:00 ps
user1 can then do
[user1@host ~]$ talk user2@localhost pts/23

and user2 will see the following in that terminal:
Message from Talk_Daemon@host at 14:49 ... talk: connection requested by user1@localhost.localdomain. talk: respond with: talk user1@localhost.localdomain
If you are user1 and have no idea on what terminal user2 is logged on, you can try
w|grep user2
user2 pts/8 remote:S.0 04Nov12 59:00 0.04s 0.04s /bin/bash user2 pts/9 remote:S.1 04Nov12 17:28 0.03s 0.03s /bin/bash user2 pts/10 remote Mon12 2days 0.03s 0.00s ssh -X -v volde user2 pts/11 local Mon12 2days 0.29s 0.26s perl eccejobmon user2 pts/23 remote 14:30 7.00s 0.00s 0.00s -bash

09 January 2013

313. Which Office for linux users? Libreoffice, Openoffice, Lotus Symphony, Google Docs, Microsoft Live/Skydrive, Wine/Office2003 and Office2003 in Virtualbox, WPS Office

UPDATE 6 Apr 2013: I've added WPS Office

UPDATE 10 Feb 2013: I've tried Libreoffice 4.0 as well -- no improvement in terms of rendering vs Libreoffice 3.5.4 (below). The start-up times are greatly improved though.

UPDATE 4 Feb. 2013: Equation Editor in Office 2003 on Wine used to be supported -- however, while fixing the editor in office 2010 support for older office versions broke. This will (hopefully) be fixed in future versions and the codeweaver and wine devs are aware of the issue. Bug reports to that effect:
http://bugs.winehq.org/show_bug.cgi?id=32697 and http://www.codeweavers.com/support/tickets/browse/?ticket_id=931759

Most of my publications are via collaborations, and I'm more or less the only one who uses Linux. When I'm in charge, I use latex -- but you can be a zealot, or you can be a pragmatist. Life tends to be easier for the pragmatic. I draw the line at actually using a native installation of Windows though.

A few things are important: figures should render ok, endnote should render ok, I need to use track changes and I should be able to insert equations. Oh, and the templates from the major publishers (RSC, Wiley, ACS) should work.

So, once and for all (well, I should revise this for each major version I suppose) here's an evaluation of  where each solution on Linux falls short and where it does well.

I mocked up a simple article style doc file using Word 2003, Endnote X and Microsoft Equation Editor 3.0 and the Angewandte Chemie template, and then opened it in various alternative solutions.

I tried:

  • LibreOffice
  • OpenOffice
  • Lotus Symphony
  • Abiword
  • Google Docs/Drive
  • Microsoft Live/Skydrive

Here's what I found:

NOTE: when I say 'good' or 'bad' I mean how well they reproduce the look which is found in Word 2003. It's not a judgement on the quality of the word processor as such. I also recognise that for a developer it isnt much fun spending time working on reproducing someone else's crappy and poorly documented software, when you could be implementing new and wonderful things instead.

Outcome: all non-Word 2003 solutions at a minimum had the figure move slightly relative to the text. All solutions did the Endnote references ok, if not the location of them. Some solutions had issues with the greek letters in the equation. All non-MS solutions rendered the equation as an image which couldn't be edited.

See for yourself...

Office 2003/Virtualbox
This is the canonical version -- this is how it was created and how it should look.
Word 2003 in Virtualbox

Libreoffice 1:3.5.4+dfsg-4 
Looks fine, but
* The numbering for the equation (put it there using a tab stop) is too far left
* part of a paragraph now separates the figure from its caption -- the picture hasn't moved, but the first column on the first page seems to be one line shorter, which pushes the text of the previous paragraph past the figure.
* The last paragraph doesn't end on the first page, but in the first column of the second page..
* The second page also looks different, with the endnote bit moved to the second column.

Openoffice 3.4.1-1
I grabbed the latest debs from http://www.openoffice.org/download/other.html#tested-full
On the second page the Endnote references are in the right place (c.f. Libreoffice).

* Greek letters in Equation replaced by squares (missing font?). (greek letters in text is ok)
* The numbering for the equation (put it there using a tab stop) is too far left
* part of a paragraph now separates the figure from its caption -- the picture hasn't moved, but the first column on the first page seems to be one line shorter, which pushes the text of the previous paragraph past the figure.
* The last paragraph doesn't end on the first page, but in the first column of the second page..


Lotus Symphony 3.0.1
I downloaded the Lucid i386 deb and installed it in an Ubuntu Lucid virtual machine. Symphony is based on OpenOffice.

* Greek letters in Equation disappeared completely. (greek letters in text is ok)
* The numbering for the equation (put it there using a tab stop) is too far left
* part of a paragraph now separates the figure from its caption -- the picture hasn't moved, but the first column on the first page seems to be one line shorter, which pushes the text of the previous paragraph past the figure.
* The last paragraph doesn't end on the first page, but in the first column of the second page..
* The second page also looks different, with the endnote bit moved to the second column.

Lotus Symphony

Office 2003/Wine
See here for installation: http://verahill.blogspot.com.au/2013/01/307-installing-office-2003-in-wine-on.html
Everything looks fine with one pretty important exception:
* The equation is missing completely. Note that other people may have better luck installing Equation editor under wine -- I haven't managed to install it at all.
Word 2003 in Wine
Plenty of issues:
* The author box has move to the top left corner.
* Can't find the table
* Greek letters missing in equation
* Greek letter in text missing
* Lots of stuff missing or misplaced (e.g. Keyword background picture)

Google Docs
This was the big surprise -- I've always thought that google docs was essentially as 'good' as Microsoft Office. They have a financial incentive to be, after all. Turns out that the document was all but unrecognisable when opened in Google Docs. By far the worst solution. Note: the out-of-date warning is triggered by Chrome 18. Most similar to Abiword.
Google Docs

Microsoft Live 
Yes, that's right -- Microsoft Live works on Linux. This is the free version you get via Hotmail/Skydrive -- not their Office 365 or whatever they use now.
There are more surprises -- it doesn't render the document faithfully.

* The numbering for the equation (put it there using a tab stop) is too far left. (like Libreoffice)
* part of a paragraph now separates the figure from its caption -- the picture hasn't moved, but the first column on the first page seems to be one line shorter, which pushes the text of the previous paragraph past the figure. (like Libreoffice)
* The last paragraph doesn't end on the first page, but in the first column of the second page. (like Libreoffice)
* The last paragraph doesn't end on the first page, but in the first column of the second page. (like Openoffice)
In other words -- Libreoffice/Openoffice do just as well as Microsoft Live

The interface for the View mode

full -- and no, you can't zoom in and out...

In editing mode all it looks like a mess -- see figures.
In addition to the obvious formatting mess,
* greek letters in text show up as squares (problem with fonts on my computer or problems with live?)
* The equation cannot be edited.
However, once you go back to View mode the formatting is restored -- but it'd leave you flying blind while trying to make changes.

Editing mode

The full document -- yes, there's no Zoom...
WPS (Kingsoft)
I only discovered this by accident. Kingsoft (金山软件) is a legitimate Chinese company (Star Office was big in the 90s) and somehow there's a linux version (i386) of their Office suite: http://linux.wps.cn/. The bad news is that it's in Chinese, but that doesn't stop me from evaluating it -- it's such a close copy to Office 2007 that you can pretty much guess how to use it.

You can download it by clicking on this link: http://wdl.cache.ijinshan.com/wps/download/Linux/unstable/wps-office_8.1.0.3724~b1p2_i386.deb

It seems to be reproducing the file pretty well, with the exception of how it renders the equation and the missing stars in the author affiliation section. However, the location of the figure caption is correct, and the second page looks good.

And here's how it looks:

312. Tau + OpenMPI profiling on Debian Testing/Wheezy

Still searching for a way to easily look at the execution of parallel jobs I came across TAU: http://www.cs.uoregon.edu/Research/tau/home.php

You can download without registering, but please do register as the number of registered users tend to be important for funding and evaluation of software development in academia: http://www.cs.uoregon.edu/Research/tau/downloads.php

I'm not really sure about how to use PDT, and I've used Tau without it before without any problems.

The compilation order below is also important -- pdt won't build without libpdb.a which is generated by tau -- but you can't configure tau with -pdt if it doesn't exist.

sudo mkdir /opt/tau
sudo chown $USER /opt/tau
cd /opt/tau

wget http://tu-dresden.de/die_tu_dresden/zentrale_einrichtungen/zih/forschung/software_werkzeuge_zur_unterstuetzung_von_programmierung_und_optimierung/otf/dateien/OTF-1.12.2salmon.tar.gz
tar xvf OTF-1.12.2salmon.tar.gz
cd /OTF-1.12.2salmon/
./configure --prefix=/opt/tau/OTF
make install
cd ../

wget http://tau.uoregon.edu/tau.tgz
tar xvf tau.tgz
cd tau-2.22-p1/
./configure -mpilib=/usr/lib/openmpi/lib -prefix=/opt/tau -openmp -TRACE -iowrapper -otf=/opt/tau/OTF -pthread
make install
cd ../

wget http://tau.uoregon.edu/pdt.tar.gz
tar xvf pdt.tar.gz
cd pdtoolkit-3.18.1/
./configure -prefix=/opt/tau/pdt
make install

cd ../tau-2.22-p1/
./configure -mpilib=/usr/lib/openmpi/lib -prefix=/opt/tau -openmp -TRACE -iowrapper -pthread -otf=/opt/tau/OTF -pdt=/opt/tau/pdt

make install

Time to try it out on something parallel.

First set the path


I used nwchem with this input file, co2.nw:
title "co nmr" geometry c 0 0 0 o 0 0 1.13 end basis * library "6-311+G*" end property shielding end dft direct grid fine mult 1 xc HFexch 0.05 slater 0.95 becke88 nonlocal 0.72 vwn_5 1 perdew91 0.81 end task dft property

and ran it using
mpirun -n 3 tau_exec nwchem co2.nw

which ends with
Total times cpu: 4.8s wall: 7.6s
It's obviously a bit too short, but will do for illustration purposes.

That generates a set of files, profile.*.0.0 -- one for each thread i.e. profile.1.0.0, profile.2.0.0 and profile.3.0.0 in this particular case. There are a lot of options for tracing, using hardware counters etc. -- see http://www.cs.uoregon.edu/Research/tau/docs/newguide/
pprof -s
Reading Profile files in profile.* FUNCTION SUMMARY (total): --------------------------------------------------------------------------------------- %Time Exclusive Inclusive #Call #Subrs Inclusive Name msec total msec usec/call --------------------------------------------------------------------------------------- 100.0 15,813 25,931 3 14276 8643959 .TAU application 18.8 4,870 4,870 10272 0 474 MPI_Barrier() 12.1 3,138 3,138 3 0 1046279 MPI_Init() 8.1 2,090 2,090 818 0 2556 MPI_Recv() 0.0 9 9 3 0 3173 MPI_Finalize() 0.0 3 3 24 0 128 MPI_Bcast() 0.0 2 2 6 0 463 MPI_Comm_dup() 0.0 1 1 790 0 2 MPI_Comm_size() 0.0 0.872 0.872 818 0 1 MPI_Send() 0.0 0.294 0.294 841 0 0 MPI_Comm_rank() 0.0 0.17 0.17 674 0 0 MPI_Get_count() 0.0 0.111 0.111 3 0 37 MPI_Comm_free() 0.0 0.026 0.026 3 0 9 MPI_Errhandler_set() 0.0 0.024 0.024 6 0 4 MPI_Group_rank() 0.0 0.02 0.02 6 0 3 MPI_Comm_compare() 0.0 0.015 0.015 4 0 4 MPI_Comm_group() 0.0 0.008 0.008 4 0 2 MPI_Group_size() 0.0 0.004 0.004 1 0 4 MPI_Group_translate_ranks() FUNCTION SUMMARY (mean): --------------------------------------------------------------------------------------- %Time Exclusive Inclusive #Call #Subrs Inclusive Name msec total msec usec/call --------------------------------------------------------------------------------------- 100.0 5,271 8,643 1 4758.67 8643959 .TAU application 18.8 1,623 1,623 3424 0 474 MPI_Barrier() 12.1 1,046 1,046 1 0 1046279 MPI_Init() 8.1 696 696 272.667 0 2556 MPI_Recv() 0.0 3 3 1 0 3173 MPI_Finalize() 0.0 1 1 8 0 128 MPI_Bcast() 0.0 0.926 0.926 2 0 463 MPI_Comm_dup() 0.0 0.436 0.436 263.333 0 2 MPI_Comm_size() 0.0 0.291 0.291 272.667 0 1 MPI_Send() 0.0 0.098 0.098 280.333 0 0 MPI_Comm_rank() 0.0 0.0567 0.0567 224.667 0 0 MPI_Get_count() 0.0 0.037 0.037 1 0 37 MPI_Comm_free() 0.0 0.00867 0.00867 1 0 9 MPI_Errhandler_set() 0.0 0.008 0.008 2 0 4 MPI_Group_rank() 0.0 0.00667 0.00667 2 0 3 MPI_Comm_compare() 0.0 0.005 0.005 1.33333 0 4 MPI_Comm_group() 0.0 0.00267 0.00267 1.33333 0 2 MPI_Group_size() 0.0 0.00133 0.00133 0.333333 0 4 MPI_Group_translate_ranks()

...which I can't pretend to understand. Reasonably, the first line would be the cpu time and the wall time (4.8 and 7.6 s vs 5,271 and 8,643 ms).

A visual representation can be had by launching paraprof:

Now it's time to explore...

The one thing that doesn't seem to work is visualisation of the communication matrix...

Failed attempt to build with vampirtrace
sudo mkdir /opt/tau
sudo chown $USER /opt/tau
cd /opt/tau

wget http://tu-dresden.de/die_tu_dresden/zentrale_einrichtungen/zih/forschung/software_werkzeuge_zur_unterstuetzung_von_programmierung_und_optimierung/otf/dateien/OTF-1.12.2salmon.tar.gz
tar xvf OTF-1.12.2salmon.tar.gz
cd /OTF-1.12.2salmon/
./configure --prefix=/opt/tau/OTF
make install
cd ../

wget http://tu-dresden.de/die_tu_dresden/zentrale_einrichtungen/zih/forschung/software_werkzeuge_zur_unterstuetzung_von_programmierung_und_optimierung/vampirtrace/dateien/VampirTrace-5.14.1.tar.gz
tar xvf VampirTrace-5.14.1.tar.gz
cd VampirTrace-5.14.1/
./configure --prefix=/opt/tau/vampirtrace --with-mpi-dir=/usr/lib/openmpi/lib --with-extern-otf-dir=/opt/tau/OTF
make install

wget http://tau.uoregon.edu/tau.tgz
tar xvf tau.tgz
cd tau-2.22-p1/
./configure -mpilib=/usr/lib/openmpi/lib -prefix=/opt/tau -openmp -TRACE -iowrapper -otf=/opt/tau/OTF -vampirtrace=/opt/tau/vampirtrace
make install

It builds fine, but during execution of mpirun -n 2 tau_exec... I get
Error: No matching binding for 'mpi' in directory /opt/tau/x86_64/lib
Available bindings (/opt/tau/x86_64/lib):
Error: No matching binding for 'mpi' in directory /opt/tau/x86_64/lib
Available bindings (/opt/tau/x86_64/lib):

311. Compiling MPE for MPI profiling

I've been wanting to get an overview of how my parallel (nwchem, gamess, dalton, etc.) programs are running, and whether there are any obvious bottlenecks other than the network and slow harddrives that I can sort out.

The Australia high performance computer facility in Canberrra uses http://ipm-hpc.sourceforge.net/, but I'm not having much luck compiling it, and the lack of recent updates makes me somewhat less willing to invest too much effort into it.

So I stumbled across MPE instead: http://www.mcs.anl.gov/research/projects/perfvis/download/index.htm#MPE

The problem is that almost all of the links of that page are broken, including those pointing towards the documentation, so I don't actually know how to use it properly. The presence of mpecc in /opt/mpe/bin suggests that it's used as a stand-in for mpicc when compiling, which I'll test some day.

cd ~/tmp
wget ftp://ftp.mcs.anl.gov/pub/mpi/mpe/mpe2.tar.gz
tar xvf mpe2.tar.gz
cd mpe2-1.3.0/
./configure MPI_CC=mpicc MPI_F77=mpif77 --prefix=/opt/mpe
sudo make install

I found what I looked for in Tau instead: 

310. Remote mounting using sshfs

I've run out of USB ports on my work desktop, so I occasionally cheat and attach USB drives to one of my compute nodes and transfer the files across the network to my desktop. Since I've got a gigabit switch set up, the speeds are quite acceptable.

NFS isn't really a solution here. Instead, sshfs is the tool to use.

The local and remote computer will be referring to Desktop and Node, respectively. The specific example I'm using here is that of a USB drive manually mounted on the Node, which contains pictures that I want to transfer to my Desktop.

On the Node
The plugged in USB device is found as /dev/sdb, and holds only one partition, /dev/sdb1.

sudo mkdir /media/usbdrive
sudo mount /dev/sdb1 /media/usbdrive

On the Desktop

sudo apt-get install sshfs
sudo mkdir /media/remote
sudo sshfs $USER@Node:/media/usbdrive /media/remote -o allow_other

That's about it. To unmount do
sudo umount /media/remote

sudo umount /media/usbdrive


08 January 2013

309. Installing Office 2003 in Wine on Debian Testing

UPDATE 4 Feb 2013: Equation Editor under Office 2003 used to be support on Wine/Crossover. The devs are aware of it and looking into fixing the regression.

Bug reports to that effect:
http://bugs.winehq.org/show_bug.cgi?id=32697 and http://www.codeweavers.com/support/tickets/browse/?ticket_id=931759

UPDATE: I've since tried this using CrossOver 12 as well. The installation goes much more smoothly (the .exe file works fine), and you basically don't need to do anything except click 'Next' a couple of time. The Equation editor still doesn't work though. There's a free 15 day trial version of CrossOver available at http://www.codeweavers.com/products/. If running Office 2003 is your goal there seems to be little difference between CrossOver and Wine. However, I ended up buying a copy of crossover anyway since at this stage of my life and career it's not that much money and by supporting crossovre you indirectly support wine.

Since I would guess at the lack of Office for Linux as the single most problematic issue facing those wishing to switch from Windows or Mac to Linux, and since it would be something attempted very early on in someone's experience of Linux, it doesn't hurt to show the steps in detail -- especially since there are a few stumbling stones.

The linux community like to suggest new adopters to run Word under Wine, or better yet, to use libre/openoffice. It's an open secret that there are issues though. Most casual users won't notice most of the time,  but problems do exist.

In the particular case of Office 2003 under Wine, my main issue is that there's no way of installing Microsoft Equation Editor from what I can see, and it is a feature I do use when publishing together with other people (sweet, sweet TeX...). Another issue is that figures tend not to show up well or at all (complaints about memory -- most of our collaborative files are >5 Mb) when I use Word under Wine. These aren't issues when running in a virtual machine, which is the way I usually suffer through my bouts with Office.

An added head ache is this: http://wine.1045685.n5.nabble.com/Bug-26358-New-Office-2003-installer-crashes-immediately-td3414859.html and  http://bugs.winehq.org/attachment.cgi?id=35449. Using any of the versions of Wine above I just can't install using the setuppro.exe. This bug report ( http://bugs.winehq.org/show_bug.cgi?id=31942) showed me the simple solution though: using pro11.msi instead.

Anyway, just be aware of the limitations -- you can't install office, then do wine control, "add/remove programs", and add e.g. Microsoft Equation Editor. Likewise, if you try to be smart and do a custom install, and choose advanced customization, the installation will fail and complain about missing CAB files (in the case of equation editor: 01561403.cab). I've also tried installing MathType 4, which shows up under Objects, but is reported as missing when selected.

Otherwise this works fine.

To install the official wheezy version of wine:
sudp dpkg --add-architecture i386
sudo apt-get update
sudo apt-get install wine wine-bin:i386 libwine-bin:i386 libwine:i386

You can also see the following posts on this blog for how to install/compile wine:
1.5.21 (using multiarch)
1.5.19 (using the dev.carbon-project.org rules)
1.5.5 (using the dev.carbon-project.org rules)
1.5.5 (downloading precompiled debs)

For this installation I've used Wine 1.5.19.

First set up cabextract and winetricks and install corefonts etc.
sudo apt-get install cabextract ttf-mscorefonts-installer
wget http://winetricks.org/winetricks
chmod +x winetricks
./winetricks corefonts fontfix

If you have Office 2003 as an OFFICE11.iso file (likely if you're at e.g. a university) then loop mount that file
In the directory where your iso file is:

sudo mkdir /media/OFFICE11
sudo mount -o loop OFFICE11 /media/OFFICE11
mount: warning: /media/OFFICE11/ seems to be mounted read-only.
wine msiexec /i /media/OFFICE11/pro11.msi

If you get any questions about Gecko or Mono, go ahead and let them install. Anyway, this launches the installation -- just do what you'd do under Windows, with the following caveats:

Important: You can't go back and install missing programs, so install all that you want from the beginning. Also, if you try to do advanced customisation and install Equation Editor (maybe other programs too) the installation will fail and complain about a missing CAB file. It hasn't always been so  apparently.

Launch it as any other gnome or kde program, or from the terminal do
wine ~/.wine/drive_c/Program\ Files/Microsoft\ Office/OFFICE11/WINWORD.EXE

308. Compiling Wine 1.5.21 with multiarch on Debian Testing -- clean multiarch build

UPDATE 16 May 2013: See here for Wine 1.5.30: http://verahill.blogspot.com.au/2013/05/416-wine-1530-in-chroot.html

Another update (28/1/2013): It seems like your mileage in terms of how well Wine will work for you will vary using this method. It works fine for me and I suspect it's because I've built earlier Wine versions of my systems and have ia32-libs-dev installed. This package is no longer available in Testing. ia32-libs installs the libs you need, but does not provide symlinks between .so and .so.X files -- you can probably do that by hand if necessary though. It's not for beginners.

An additional thing to remember is that the 1.5 series of Wine is a development series -- 1.4 is the stable series which is meant for end users. Progress made in 1.5 will be included in 1.6.

Update2 (26 Jan 2013): there are a couple of issues, and they will depend on how your system is set up.
* libosmesa6-dev:i386 will require mesa-common-dev:i386 which will remove libosmesa6-dev:amd64 and more, which is bad.
* libglu1-mesa-dev:i386 depends on libgl1-mesa-dev:i386 which will remove libgl1-mesa-dev:amd64 and more, which is bad.
*  libgstreamer-plugins-base0.10-dev:i386 depends on libgstreamer0.10-dev:i386 which depends on gir1.2-gstreamer-0.10:i386 which depends on gir1.2-freedesktop:i386 which depends on gir1.2-glib-2.0:i386 which depends on libgirepository-1.0-1:i386 which will remove a whole lot of packages (132 on one of my systems, including gnome, gdm3 etc.)

I'm working on figuring out what's triggering this on some systems but not others. Be aware that that doesn't necessarily mean that I will be successful in doing so -- any help is appreciated.

In the mean time see http://verahill.blogspot.com.au/2013/01/308-compiling-wine-1521-on-debian.html to see how to build wine in a chroot, which is safe. You can then install that .deb package on your normal system and HOPEFULLY there won't be any broken dependencies. You won't need the -dev:i386 packages on the install target.

ldd `which wine` 
linux-gate.so.1 => (0x55573000) libwine.so.1 => /usr/local/lib/libwine.so.1 (0x55576000) libpthread.so.0 => /lib32/libpthread.so.0 (0x556ec000) libc.so.6 => /lib32/libc.so.6 (0x55705000) libdl.so.2 => /lib32/libdl.so.2 (0x55867000)
so you really don't seem to need much in the way of shared libs installed.

Old Update: This build is safe now, and there are no remaining errors (i.e. opencl has been taken care of) (see update2 above for more recent information)

I've been building wine (e.g. http://verahill.blogspot.com.au/2012/12/building-wine-1519-on-debian.html) on a system that was set up a few years ago.

Ergo, since ia32libs are on their way out and being replaced by multiarch libs it's time to have a look at what a build from scratch looks like, and what dependencies are needed.

This time we'll build wine the old-fashioned way using configure/make/make install. The biggest challenge was to overcome the annoying
configure: error: Cannot build a 32-bit program, you need to install 32-bit development libraries.
error. The solution to that was simply to install gcc-multilib. 

To work everything out from scratch I've set up a an amd64 chroot of a clean debian testing/wheezy. , It's increasingly appearing as if this is the easiest and safest thing to do.  Hopefully the release of Jessie will get the develop/improvement of multiarch back on the road.

To follow what's new in wine, look here: http://linux.softpedia.com/progChangelog/Wine-Changelog-148.html

...and the obligatory screenshot

The chroot environment
Basically as I've shown before:

sudo apt-get install debootstrap coreutils
mkdir -p $HOME/tmp/architectures/wine
cd $HOME/tmp/architectures
sudo debootstrap --arch amd64 testing $HOME/tmp/architectures/wine/ http://ftp.au.debian.org/debian/
sudo chroot wine/
apt-get install locales sudo vim
echo 'export LC_ALL="C"'>>/etc/bash.bashrc
echo 'export LANG="C"'>>/etc/bash.bashrc
source /etc/bash.bashrc
adduser winebuild
usermod -g sudo winebuild
echo 'Defaults !tty_tickets' >> /etc/sudoers

Edit /etc/hosts and add your hostname (here beryllium) to the end of the line e.g. localhost beryllium

Exit, and the log in again
sudo chroot wine

Then change user:
su winebuild
cd ~

Change at the very least the country code for the repo above (n red) so that you're using the closest/fastest one.

Compiling Wine
Set up multiarch (we're using amd64 but also want to support i386). I'm setting this up from scratch.

sudo dpkg --add-architecture i386
sudo apt-get update

Get the wine dependencies:

sudo apt-get install libx11-dev:i386 libfreetype6-dev:i386 libxcursor-dev:i386 libxi-dev:i386 libxxf86vm-dev:i386 libxrandr-dev:i386 libxinerama-dev:i386 libxcomposite-dev:i386 libglu-dev:i386 libosmesa-dev:i386 libglu-dev:i386 libosmesa-dev:i386 libdbus-1-dev:i386 libgnutls-dev:i386 libncurses-dev:i386 libsane-dev:i386 libv4l-dev:i386 libgphoto2-2-dev:i386 liblcms-dev:i386 libgstreamer-plugins-base0.10-dev:i386 libcapi20-dev:i386 libcups2-dev:i386 libfontconfig-dev:i386 libgsm1-dev:i386 libtiff-dev:i386 libpng-dev:i386 libjpeg-dev:i386 libmpg123-dev:i386 libopenal-dev:i386 libldap-dev:i386 libxrender-dev:i386 libxml2-dev:i386 libxslt-dev:i386 libhal-dev:i386 libcurl4-openssl-dev:i386

sudo apt-get install gettext prelink bzip2 bison flex oss4-dev checkinstall ocl-icd-libopencl1:i386 opencl-headers

sudo apt-get install gcc-multilib

mkdir ~/tmp/
wget http://mirrors.ibiblio.org/wine/source/1.5/wine-1.5.21.tar.bz2 -O wine-unstable_1.5.21.orig.tar.bz2
tar xvf wine-unstable_1.5.21.orig.tar.bz2
cd wine-1.5.21/
config.status: executing include/wine commands config.status: executing Makefile commands configure: Finished. Do 'make' to compile Wine.
time make -j4

where -j4 is the number threads to launch, normally number of cores+1 (see here). Took around 15 minutes on a 3 core AMD.

sudo checkinstall

to build a .deb package and install it (preferred solution) or
sudo make install

to just install.

Note that while this build works without issue, the transition to multiarch in debian is not complete and you will likely encounter this problem:
p11-kit: couldn't load module: /usr/lib/i386-linux-gnu/pkcs11/gnome-keyring-pkcs11.so: /usr/lib/i386-linux-gnu/pkcs11/gnome-keyring-pkcs11.so: cannot open shared object file: No such file or directory
If you try to fix it you get
sudo apt-get install gnome-keyring:i386 libcap-ng0:i386 libgck-1-0:i386 libgcr-3-1:i386 gcr:i386 libcap2-bin:i386
The following packages have unmet dependencies: libgcr-3-1:i386 : Depends: libgcr-3-common:i386 but it is not installable
This is known. The question is what the consequences of this are -- in this thread it's been suggested that it prevents any program needing to use SSL under wine from working. More at wine HQ here and here. More at Debian here. If I'm reading this right -- and I might not be -- we'll at the least have to wait for gnome-keyring 3.6 to make it out of experimental to sid, then from sid to testing (which is frozen).

 For no particular reason I think this may be what's causing issues with the installation of Office 2003 (and a few other programs, including Agilent HP Chemstation).

Links to this page:

307. Burning audio CDs from the command line in debian testing/wheezy

I've got a CD burner on a headless box, so burning audio CDs from the command line is a necessity.

I also normally end up burning flash videos that I've converted to mp3s, so I'll show that too. This post is essentially covered (although not very well) already here: http://verahill.blogspot.com.au/2012/01/debian-testing-64-wheezy-small-fixes.html

First install the necessary programmes
sudo apt-get install ffmpeg wodim mpg123

Converting flv to mp3
To batch-convert flv files to mp3, do
ls *.flv|xargs -I {}  ffmpeg -i {} -ar 44100 -ab 160k -ac 2 {}.mp3

Preparing the files
Rename your files to 01.mp3, 02.mp3 etc. to make the songs burn in that order (since you're using *wav below).

Convert the mp3s to wav files (you could've gone straight from flv to wav above)
for i in *.mp3; do mpg123 --rate 44100 --stereo --buffer 3072 --resync -w "`basename "$i" .mp3`".wav "$i"; done

Burn with
wodim -v -pad speed=1 dev=/dev/cdrw1 -dao -swab *.wav

assuming that cdrw1 is the correct device.

Eject your cd when done.
eject cdrom1

[There's also no shortage of terminal music players, such as cplay.]

PS. You can burn anything you want from the command line using burn, e.g. an .iso file
sudo apt-get install burn
sudo burn -I -n myiso.iso

If the device you want to burn on is /dev/cdrom1 instead of /dev/cdrom, you can edit that in /etc/burn.conf

07 January 2013

306. Insync with Google Drive and Google Docs on Debian Testing/Wheezy

The problem:
1. It would be nice to be able to use Google Docs as a replacement for Microsoft Word until Libre/OpenOffice catch up (post about that later) or the world switches to LaTeX and
2. for that to happen there needs to be an easier way to sync documents between google docs and your harddrive than using email.

The closest thing to that is using Google Drive to keep documents synced, and opening them in Google Docs using your browser.

It's been more than half a year since promising that Google Drive would be available for linux, and Google has yet to actually release anything (here) and it almost looks like vaporware (here).

In lieu of an official solution, there are a few options. One is Grive -- which seems to work with Google Drive but not Google Docs -- and another one is Insync, which isn't open source as it is owned by a start-up. It's the most promising and full-featured solution though, so we'll go with that.

There used to be gdocsfs, but it doesn't seem to be maintained.

The usual caveats about installing things from outside the repos apply, and even more so in this case since the source code is not available.

Setting up Insync
sudo apt-get install xdotool python-nautilus libxdo2 gir1.2-nautilus-3.0
mkdir -p ~/tmp/insync
cd ~/tmp/insync
wget http://s.insynchq.com/builds/insync-beta-gnome-cinnamon-common_0.9.34_amd64.deb
wget http://s.insynchq.com/builds/insync-beta-gnome_0.9.34_all.deb
sudo dpkg -i *.deb

If all went well you'll find InSync installed (move mouse to top-right corner, type insync and it should be there). Clicking on it opens a browser tab, in which you're asked to select the gmail account you wish to use.

You're next asked to allow InSync to do various things:

Confirm (you'll then get an email) and associate your machine with the account.

You should now have a new set of folders in $HOME:
`-- me@gmail.com

If you create a directory either in ~/Insync/me@gmail or in google drive using your browser, the directory should show up in both places (i.e. it's synced) -- assuming that you've got insync running:
insync --headless > /dev/null &

will keep it running in the background. Any doc file copied to the insync folder will now be editable in Google docs by pointing your browser to https://drive.google.com/#my-drive

Simple as that.

305. make -jN -- should N equal number of cores or N+1 cores? Optimal number of threads per core

Update: I repeated this test by compiling kernel 3.7.2 using different settings (http://verahill.blogspot.com.au/2013/01/321-compiling-kernel-372-on-debian.html)  -- given the length of the compile and it's reliance on CPU grunt it is probably a better test case. It came out showing that N -- or even N-1 -- was better than over-committing.

The original commentator also offered this explanation:
Historically N+1 or even N*1.5 was used & worked better on memory / I-O constrained systems where the available cache was used as a short lasting one to feed the extra committed threads / processes while I-O was in progress. As you've observed correctly this is not the case on machines that have an abundance of RAM, where this acts as a long lasting cache, no data that got written to disk will be read back > spawning additional threads / processes has therefore a detrimental effect on efficiency due to (much) more rescheduling / TLB shootdown interrupts. In short, when available ram is larger then total disk-space needed for build N = amount of logical cpu's if not N = logical cpu's + 1 Setting the global environment variable (CONCURRENCY_LEVEL) instead of fixed values for -j for automated builds using the previously mentioned #export CONCURRENCY_LEVEL=`getconf _NPROCESSORS_ONLN` is always the safest bet, especially when using server grade machines and high speed 0 seek time solid state disks ...
I think the conclusion is the one offered above -- stick to N for optimal performance, unless you have a compelling reason not to. I should also emphasize that I don't have a background in computing of any sort, whereas the poster is a professional in the HPC field.

So if I'm allowed to paraphrase and make conclusions:
for a very short compile, like the one in this post, you may find that N+1 seemingly gives a better result since disk I/O plays a big part relative to the code generation (and whatever else a compiler does). For a longer, more 'normal' compilation disk I/O play a smaller part.

If your RAM is too small and you need to cache to disk repeatedly, then that obviously increases the disk I/O as well.

In the end, the penalty for over-committing (http://verahill.blogspot.com.au/2013/01/321-compiling-kernel-372-on-debian.html) is large enough that it's a better bet to just got for N threads.

I really shouldn't be surprised -- it's the same effect you see when launching a computational job: you do NOT want to launch more threads than cores.

Original post:
I got a comment recently regarding the number of threads that should be used for make:
make -j7 is the number of cores +1 

Stop copy paste nonsense.... sigh...

make -j1 will spawn 1 worker process
-j7 will spawn 7. 


makes adding -jjob unnecessary 
on an i7 this is the same as -j8

When in doubt check top.....

So the question is whether for N cores, should you spawn N threads or N+1? The poster has a valid point -- there's not that much data on what really is the best configuration and while most people keep repeating the (mostly) accepted N+1 (or 1.5*N)  wisdom, we really need more hard numbers.

So here's my real-world unscientific benchmark for compiling Gromacs 4.5.5 on a six core AMD Phenom II  1055T with 8 Gb RAM and a slow 5400 rpm hard drive (disk I/O plays into things as well). I'm using gcc 4.7.2-4 and Debian Wheezy/Testing.

To get the data I used this script, maketest.sh:

make distclean
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/lib/openmpi/lib:/opt/openblas/lib
export LDFLAGS="-L/opt/fftw/fftw-3.3.2/single/lib -L/opt/openblas/lib -lopenblas"
export CPPFLAGS="-I/opt/fftw/fftw-3.3.2/single/include -I/opt/openblas/include"
./configure --disable-mpi --enable-float --with-fft=fftw3 --with-external-blas --with-external-lapack --program-suffix=_sp --prefix=/opt/gromacs/gromacs-4.5.5
time make -j$1

which I called with e.g.
sh maketest.sh 6

Admittedly, this is a fairly short build but it is a 'real' one.

N    Time (real)
1    9 m 52 s
2    5 m 18 s
3    3 m 48 s
4    3 m 02 s
5    2 m 24 s
6    2 m 16 s
7    2 m 05 s
8    2 m 06 s
9    2 m 07 s
10   2 m 07 s
11   2 m 08 s
12   2 m 09 s
Or as a plot:
The buiild time decreases roughly exponentially with the number of threads. The blue line is at 125 seconds i.e. dx/dy=0.
I'm actually quite surprised at how N+1 turned out to be the best configuration, although in general it seems that you don't suffer any penalty for using more threads, so 1.5*N works just as well.

I also ran sar (sysstat; sar -u 1 180 |gawk '{print $3,$5,$8}' |tee n7.dat ) for -j7 to see how the load varies with time during make (I collected a little bit of data before and after make, hence the flat line at the end):
The black/blue (user/idle) lines are what are interesting here
The build is very evidently not perfectly parallel at all stages, and that will also affect the optimal number of threads/core.

Raw results
real    9m51.519s
user    6m43.316s
sys     0m44.092s
real    5m18.359s
user    7m3.548s
sys     0m46.112s
real    3m47.850s
user    7m22.732s
sys     0m47.064s
real    3m2.131s
user    7m56.068s
sys     0m41.744s
real    2m24.258s
user    7m53.140s
sys     0m34.928s
real    2m16.429s
user    8m15.088s
sys     0m27.160s
real    2m5.361s
user    7m50.200s
sys     0m28.280s
real    2m5.820s
user    7m52.380s
sys     0m27.548s
real    2m7.266s
user    7m54.344s
sys     0m28.340s
real    2m7.057s
user    7m56.628s
sys     0m27.872s
real    2m7.728s
user    7m58.276s
sys     0m27.332s
real    2m8.819s
user    8m0.600s
sys     0m27.544s

06 January 2013

304. Getting started with Simon 0.4 on Debian Wheezy/Testing (very basic)

Here's how to get started with Simon 0.4 -- although be warned that I've never used Simon before, and that there are likely better resources out there.

In the few cases where I use the command line, I have presumed that you are using the same locations as shown in this post: http://verahill.blogspot.com.au/2013/01/303-building-simon-04-speech.html

In case you screw up, to wipe all previous settings, try:

find ~/.kde -name "simon*"|xargs -I {} rm {} -rf

If you're really desperate, nuke the entire ~/.kde folder, although that obviously has repercussions if you're actually using KDE and not GNOME. Also look under ~/tmp/$USER-kde -- I had simond put files there too.


Running simon
simond &

If you need to kill simond you can do
kill %1

assuming that you don't have any other background procs in that terminal.

1. Scenarios:
Click on Open, select Download and pick the scenarios which you are interested in. To make sure that things are working I'm more or less following the video.
In Scenarios, select a couple of H4W scenarios  (e.g. keyboard, mouse etc.) -- BUT NOT THE FIREFOX ONE, which causes trouble.

2. Speech Models:
Click on Open model, select Download and pick the Speech Model which you want. Pick the  HUB4 model since from the Youtube video it appears that you should match your scenarios and Speech models e.g. VF with VF and H4W with HUB4.

3. Server
Nothing weird here:

4. Sound Devices
I run my stuff via pulseaudio -- i.e. whatever the input source is there, will be used.

5. Volume
Do a bit of talking and see how the volume pans out. Ideally you should have any amplification of your microphone turned off, since that causes higher noise levels.

Julius problem:
If after clicking Finish you see this, you will want to work out what went wrong:
Make sure that you
1. compiled Simon with -DPJULIUS correctly set, and
2. you export the directory:
export PATH=$PATH:$HOME/.simon/julius/bin

Then restart simond and simon e.g. quit simon and kill simond, then launch simond in the background and then simon. To test if it's working correctly, go to Actions/Synchronize. No error means that it's working.

6. Get dictionary
Do e.g.
cd ~/.simon/
mkdir files
cd files
wget http://www.repository.voxforge1.org/downloads/SpeechCorpus/Trunk/Lexicon/VoxForge.tgz
tar xvf VoxForge.tgz

then select e.g. Standard, and click on Open "Standard"

  Click on Import Dictionary:
Select Shadow Dictionary and Next:
Pick HTK lexicon:

Import your file:
You should now see it compile the model and there should be stuff under Shadow Dictionary:

At this point you should be able to test whether you're being recognised. Make sure that the button says 'Activated' and not 'Activate'. Try speaking a few commands e.g. 'One', 'Two', 'Three','Chaos','Control' etc.

You'll see if it works by looking at "Last recognition results:".

Stuff that doesn't work: Configure Audio from within simon, Configure Acoustic Model from within Simon. Adaptive training with the VoxForge base model (complaining about grammar)

Stuff that does work: speech recognition, adaptive training for the HUB4 base model (and beautifully so).

While it's easy enough to get started, it does appear that there's no easy way of running the wizard again in case you want to change e.g. the base model or install more scenarios via download.

Obviously such a piece of software is fairly complex, and will have a high error rate, yet this is the type of software which should ideally just sit in the background and not really be noticed by the user -- it should (ideally) not require more work than e.g. using a mouse or a keyboard.

We're not quite there yet -- hence why we're at 0.4 and 1.0. Anyway, it's still kind of neat when my garbled utterances are correctly (most of the time) recognised by Simon. Now, how to get it to actually do stuff for me?

A. Get Scenarios later
visit http://kde-files.org/index.php?xcontentmode=692&PHPSESSID=0e48f2edd26bf70e676459a5465ec675

B. Get Base models later
visit http://kde-files.org/index.php?xcontentmode=648

05 January 2013

303. Compiling Simon on Debian Testing (Simon 0.4)

Here's how to get Simon 0.4.0 compiled. Simon does speech recognition which allows you do voice-control your computer. It does NOT do transcription, e.g. like Dragon Naturally Speaking.

See e.g. here, here and here for more information about Simon.

Simon uses cmake. I hate cmake since it makes life a lot more difficult than it needs to be.

Also, note that Simon relies heavily on KDE, so you need a number of KDE related files -- if not the whole desktop -- installed. Obviously, it runs fine under GNOME, which is where I'm using it.

This post is limited to describing how to compile it, not how to use it -- that may come later.

First get the deendencies. The list of deps is taken from http://userbase.kde.org/Simon/Development_Environment#Requirements, and expanded (e.g. libboost).

sudo apt-get install qt4-qtconfig kdelibs5-dev libxtst-dev libsamplerate-dev kdepimlibs5-dev libboost-dev
sudo apt-get install build-essential cmake gettext kdeartwork libqwt-dev libqt4-sql-sqlite libphonon-dev libattica0 libattica-dev zlib1g-dev libasound2-dev 

Next sort out Julius
I think it used to be in the repos, but it's not there anymore from what I can see).

mkdir ~/tmp/simon -p
cd ~/tmp/simon
wget http://jaist.dl.sourceforge.jp/julius/56549/julius-4.2.2.tar.gz
tar xvf julius-4.2.2.tar.gz
cd julius-4.2.2/
./configure --prefix=~/.simon/julius
**************************************************************** Julius/Julian libsent library rev.4.2.2: - Audio I/O primary mic device API : alsa (Advanced Linux Sound Architecture) available mic device API : alsa oss supported audio format : RAW and WAV only NetAudio support : no - Language Modeling class N-gram support : yes - Libraries file decompression by : zlib library - Process management fork on adinnet input : no Note: compilation time flags are now stored in "libsent-config". If you link this library, please add output of "libsent-config --cflags" to CFLAGS and "libsent-config --libs" to LIBS. ****************************************************************
make make install cd ../

You'll need HTK to use adaptive models with the Voxforge base. It's not necessary otherwise.
To get HTK, register here: http://htk.eng.cam.ac.uk/register.shtml

You should receive a password immediately afterwards. Then go to http://htk.eng.cam.ac.uk/download.shtml and download the Linux sources.Put the HTK-3.4.1.tar.gz file in ~/tmp/simon.
cd ~/tmp/simon
tar xvf HTK-3.4.1.tar.gz
cd htk/
./configure --prefix=$HOME/.simon/htk --disable-hslab
make all
mkdir ~/.simon/htk
make install

For some reason it's built as 32 bit, not 64. Also, it wants the parent install directory to exist before running make install.

Next, sphinx:

wget http://sourceforge.net/projects/cmusphinx/files/sphinxbase/0.8/sphinxbase-0.8.tar.gz
tar xvf sphinxbase-0.8.tar.gz 
cd sphinxbase-0.8/
./configure --prefix=$HOME/.simon/sphinxbase
make install
cd ../

wget http://sourceforge.net/projects/cmusphinx/files/pocketsphinx/0.8/pocketsphinx-0.8.tar.gz
tar xvf pocketsphinx-0.8.tar.gz
cd pocketsphinx-0.8/
./configure --prefix=$HOME/.simon/pocketsphinx SphinxBase_CFLAGS=-I$HOME/.simon/sphinxbase/include/sphinxbase SphinxBase_LIBS=-L$HOME/.simon/sphinxbase/lib
make install
cd ../

wget http://sourceforge.net/projects/cmusphinx/files/sphinxtrain/1.0.8/sphinxtrain-1.0.8.tar.gz
tar xvf sphinxtrain-1.0.8.tar.gz
cd sphinxtrain-1.0.8/
./configure --prefix=$HOME/.simon/sphinxtrain SphinxBase_CFLAGS=-I$HOME/.simon/sphinxbase/include/sphinxbase SphinxBase_LIBS=-L$HOME/.simon/sphinxbase/lib
make install

I couldn't sort out OpenCV, so this Simon won't be watching you.

Finally, Simon.
wget http://mirrors.mit.edu/kde/stable/simon/0.4.0/src/simon-0.4.0.tar.bz2
tar xvf simon-0.4.0.tar.bz2
mkdir build/
cd build/
cmake -DCMAKE_INSTALL_PREFIX=~/.simon -DPOCKETSPHINX_LIBRARIES=$HOME/.simon/pocketsphinx/lib/libpocketsphinx.so -DPOCKETSPHINX_INCLUDE_DIR=$HOME/.simon/pocketsphinx/include/ -DSphinxBase_INCLUDE_DIR=$HOME/.simon/sphinxbase/include/ -DSphinxBase_LIBRARY=$HOME/.simon/sphinxbase/lib/libsphinxbase.so -DPJULIUS=$HOME/.simon/julius -DPSPHINX=$HOME/.simon/sphinxbase ../simon-0.4.0/

at which points you should see:
----------------------------------------------------------------------------- -- The following external packages were located on your system. -- This installation will have the extra features provided by these packages. ----------------------------------------------------------------------------- * LibSampleRate - Resampling library * KDE PIM Libs - KDE Libraries for PIM * Sphinxbase - Open source toolkit for speech recognition * PocketSphinx - PocketSphinx is a small-footprint continuous speech recognition system ----------------------------------------------------------------------------- -- The following OPTIONAL packages could NOT be located on your system. -- Consider installing them to enable more features from this software. ----------------------------------------------------------------------------- * qaccessibilityclient KDE client-side accessibility library Required to enable ATSPI plugin. * OpenCV OpenCV (Open Source Computer Vision) is a library of programming functions for real time computer vision Required for Simon Vision -----------------------------------------------------------------------------

And now a bit of a hackjob -- we give the sphinx include dirs above, and while it works fine for most include files (they use e.g. sphinxbase/includeme.h), a number of files (prim_type, sphnx_config.h etc.) point towards the wrong directory (e.g. #include ). Two solutions -- either edit the files or put symmlinks where the files are looking. Symmlinks work well for me.

ln -s ~/.simon/sphinxbase/include/sphinxbase/sphinx_config.h ~/.simon/sphinxbase/include/sphinx_config.h
ln -s ~/.simon/pocketsphinx/include/pocketsphinx/pocketsphinx_export.h ~/.simon/pocketsphinx/include/pocketsphinx_export.h
ln -s ~/.simon/pocketsphinx/include/pocketsphinx/cmdln_macro.h ~/.simon/pocketsphinx/include/cmdln_macro.h
ln -s ~/.simon/pocketsphinx/include/pocketsphinx/ps_lattice.h ~/.simon/pocketsphinx/include/ps_lattice.h
ln -s ~/.simon/pocketsphinx/include/pocketsphinx/ps_mllr.h ~/.simon/pocketsphinx/include/ps_mllr.h
ln -s ~/.simon/pocketsphinx/include/pocketsphinx/fsg_set.h ~/.simon/pocketsphinx/include/fsg_set.h

make install
cd ~/.simon/bin
export PATH=$PATH:/$HOME/.simon/julius/bin:$HOME/.simon/sphinxtrain/bin:$HOME/.simon/htk/bin:$HOME/.simon/bin
simond &

I haven't really explored it yet, but here are the first few dialogues:

There's an official video which you may want to watch before you get started:

Stuff that doesn't work: Configure Audio from within simon, Configure Acoustic Model from within Simon
Stuff that does work: speech recognition.

If you set it up 'wrong' the first time around (or it crashed coding samples as above)) and want the wizard to run again, delete all the configuration files:
rm ~/.kde/share/apps/simon* -rf
rm ~/.kde/share/config/simon*

Links to this post:

02 January 2013

302. Surfraw on debian (and goosh)

I like shells, so I was pretty chuffed to discover this: http://goosh.org
This is basically a google shell simulator running in a gui browser -- not a true shell in any respect, so while it's really neat, but not terribly useful in itself. See below for how to get it set up.

It got me looking for true command line alternatives though, such as surfraw. While it was originally started by Julian Assange in 2000 it has been maintained by others during the past ten years.

Regardless, surfraw really does bring the power of the command line and the web together in a useful way.

To get set up with surfraw, do
sudo apt-get install surfraw
sr wikipedia surfraw

which opens a new window in your default browser -- e.g. chromium if that's what you have. This means that while you shoot of your command in the terminal, you'll then have to switch to your desktop environment. To force surfraw to use a more sensible alternative, e.g. w3m, install w3m and then create ~/.surfraw.conf and put
SURFRAW_text_browser=w3m SURFRAW_text_browser_args=-dump SURFRAW_graphical=no
in it. The -dump allows you to grep through the output but makes the whole experience less interactive -- remove that line to make things open properly in w3m instead.

w3m takes a bit of getting used to, so a cheat sheet might help: http://wiki.titan2x.com/index.php?title=W3m_cheat_sheet

A lot of it makes sense if you're used to vim though.

The next step is to explore what modules (elvi) are available e.g.
sr -elvi
acronym -- Look for acronyms definitions (www.acronymfinder.com) ask -- Question the web using Ask Jeeves (www.ask.com) bbcnews -- Search BBC News (news.bbc.co.uk) bing -- Search the web using Microsoft's Bing (www.bing.com) currency -- Convert currencies with the Universal Currency Converter (www.xe.net/ucc) debbugs -- Search the debian BTS (bugs.debian.org) duckduckgo -- Securely search the web using duckduckgo (www.duckduckgo.com) google -- Search the web using Google (www.google.com) pgpkeys -- Search the PGP key database scholar -- Search Google Scholar (scholar.google.com) wikipedia -- Search the free encyclopedia wikipedia youtube -- Search YouTube (www.youtube.com)

Hours of fun to be had:
sr wikipedia surfraw|less
sr google -results=100 -q "nwchem cosmo"|grep verahill

etc. It's probably not going to change your life, but time will tell whether there's an advantage of using surfraw over just (e)links or w3m.

There's an unofficial script which will help you install it. If you haven't got $EDITOR set, you can do export EDITOR=/usr/bin/vim or export EDITOR=/usr/bin/nano first. For some reason the value set using update-alternatives --config editor doesn't translate into $EDITOR. The editor is needed to edit the config file during installation.
cd ~/tmp
wget "https://raw.github.com/tolecnal/goosh-installer/master/goosh.sh"
chmod +x goosh.sh
sudo mkdir /var/www/goosh
sudo cp $HOME/goosh/index.php /var/www/goosh

and open http://localhost/goosh in your browser. This offers  basically no advantage over just navigating to http://goosh.org

23 December 2012

301. Building Mosquitto 1.0.5 on debian stable

Here's what I did in a chrooted Stable while writing up a reply for this post: http://forums.debian.net/viewtopic.php?f=10&t=90444

I've changed the commands a little bit from http://mosquitto.org/2012/11/making-mosquitto-packages-for-debian-yourself/ to be more faithful to my own style. Edit your /etc/apt/sources.list so that it has both stable and testing e.g.

deb http://ftp.au.debian.org/debian stable main
deb http://ftp.au.debian.org/debian testing main

and edit /etc/apt/preferences
Package: *
Pin: release a=stable
Pin-Priority: 990

Package: *
Pin: release a=testing
Pin-Priority: 800

Followed by

sudo apt-get update
sudo apt-get install build-essential python quilt libwrap0-dev libssl-dev devscripts python-setuptools
sudo apt-get install libssl-dev=1.0.1c-4 python3


mkdir ~/tmp
cd ~/tmp
wget http://mosquitto.org/files/source/mosquitto-1.0.5.tar.gz -O mosquitto_1.0.5.orig.tar.gz
tar xvf mosquitto-1.0.5.orig.tar.gz
cd mosquitto-1.0.5/
wget http://mentors.debian.net/debian/pool/main/m/mosquitto/mosquitto_1.0.5-1.debian.tar.gz
tar xvf mosquitto_1.0.5-1.debian.tar.gz
debuild -us -uc
sudo dpgk -i ../*mosquitto*.deb

and you'll find the .debs in the parent folder.

Links to this page:

300. Briefly: Sharing a folder using SAMBA on Debian

I don't ever use samba, but it's not a bad thing to know how to set up in case you need to share files with someone using Windows in a pinch.

First install samba:
sudo apt-get install samba samba-common smbclient

To get a share up with samba, create an /etc/samba/smb.conf and stick the following in it:

guest account=nobody

guest ok=yes
read only=no
comment= SMB share

Restart samba:
sudo service samba restart

1. This is an insecure share i.e. <b>anyone can access it</> and edit everything.
2.. Also, by omitting "netbios name=  " you can use the IP address of the server as the hostname, but you could also specify e.g. "netbios name=niobium" and use that as the hostname in nautilus when you connect to the host server.

To set up a user- and password-based share, do


guest ok=no
read only=no
comment= SMB share

You need to add and set the samba password, and enable the linux user you want to give access as well:
sudo smbpasswd -L -a -e lindqvist

There are a lot of other options that can be set. Two of the more interesting ones are probably

valid users=me myself irene

which means that any new files created in that share via samba gets chmod 755, and only the users me, myself and irene can connect.

But often a basic smb.conf is easier to manage and will do what you want it to.