Showing posts with label debian. Show all posts
Showing posts with label debian. Show all posts

07 November 2013

528. Briefly: Setting up 64 bit pre-built firefox (25) binaries on debian

/usr/local/lib is just a suggestion (and probably not a very good one).

Anyway, here's the quick and easy way to get set up with 64 bit firefox. Start it by calling firefox25

mkdir ~/tmp
cd ~/tmp
wget ftp://ftp.mozilla.org/pub/firefox/releases/25.0/linux-x86_64/en-GB/firefox-25.0.tar.bz2
tar xvf firefox-25.0.tar.bz2 
cd firefox/
sudo mkdir /usr/local/lib/firefox-25.0
sudo cp * -R /usr/local/lib/firefox-25.0
sudo ln -s /usr/local/lib/firefox-25.0/firefox /usr/bin/firefox25

527. Briefly: setting up thunderbird 24.1.0 on Debian (binaries)

Not much to say, other than that building thunderbird is a bit more complex these days than simply doing a configure/make/make install pass. For once I decided that rolling my own version wasn't worth it, and grabbed the pre-built binaries instead.

Here's a very brief description of how to get them set up:

mkdir ~/tmp
cd ~/tmp
wget ftp://ftp.mozilla.org/pub/mozilla.org/thunderbird/releases/24.1.0/linux-x86_64/en-GB/thunderbird-24.1.0.tar.bz2
tar xvf thunderbird-24.1.0.tar.bz2 
cd thunderbird/
sudo mkdir /usr/local/lib/thunderbird-24.1.0
sudo cp * -R /usr/local/lib/thunderbird-24.1.0
sudo rm /usr/local/bin/thunderbird
sudo ln -s /usr/local/lib/thunderbird-24.1.0/thunderbird /usr/local/bin/thunderbird

29 August 2013

505. Testing KDE on Debian (if you're a gnome user) -- getting set up

Every now and again I get ticked off with something to do with planned 'features' in gnome 3. I'm still not happy about gnome-screenshot, but I patched and compiled my own version, so I'm doing alright. I don't really like the transition of menus from the, well, menu bar of applications to the top bar of the gnome desktop, but it's not a great issue.

Oh, and I don't really use vanilla gnome 3 anyway -- I've got a desktop set up with a top panel, and bottom panel, static desktops etc. The latest 'feature' which is making me want to move to a different desktop is the planned change in the use of the middle-button so that it no longer can be used to paste text.

(of course there's a bit of inconsistency here: KDE turns out to have a remapped middle-click, the screenshot application needs a few more clicks than the old gnome-screenshot etc.)

I've toyed with the idea of changing desktops in the past, and for a while was running nothing but KDE on my laptop (I'd rather switch on my 'casual' computer first, before making the transition on my work computer), but for some reason I switched back. I actually can't remember why, but something really annoyed me about KDE.

Before I forget: a good reason to try another desktop is simply to reassure yourself that you do have alternatives if your chose DE suddenly turns into something you can't work with.

Anyway, time to give it another go. But what desktop? I like the idea of going for KDE since it's 'the other desktop' -- it's been around longer than gnome, whereas lxde and xcfe are mere babies in comparison. Alternatively, one could always piece together a custom desktop using e.g. fluxbox. Or switch to something completely different, like xmonad. Or something a little bit different, like Enlightenment. I think I should probably be going for LXDE (I instinctively don't like XFCE), but let's go for KDE first.

Finally, the main danger is switching desktops and expecting the new one to act exactly like the old one (minus the bad parts, obviously). Which is pretty much how windows users (myself included once upon a time) behave when trying out linux for the first time.

Anyway, time for a week in KDE (4.10.5).

Here's my gnome desktop at the time of switching:

In addition, I'm using Guake, and my network manager is wicd.

So let's see what KDE will look like at the end -- whether I'll be able to use KDE as KDE, instead of as an ersatz gnome.

Step 1. Install KDE
sudo apt-get install kde-full wicd-kde kdesudo yakuake

I selected kdm.

Step 2. First boot
Reboot and make sure to click the little sun icon to select the desktop you want to use, in my case KDE/plasma.

2.1 Switching workspaces
I next set up proper workspace switching so that it behaves like in...gnome (falling into the old trap of making the new desktop just like the old one)...i.e. ctrl+alt right/left arrow.

Go to System Settings/WorksSpace behaviour, Virtual Desktops -- I incrased the number of desktops to 6 (1 row), then clicked on the Switching tab -- I set shortcuts for Switch One Desktop to the Left and Switch One Desktop to the Right to ctrl+alt+left and ctrl+at+right.

2.2 Disable nepomuk
At this point I noticed that I was using 100% of the CPU. virtuoso-t was chewing up a lot, as was nepomukservices. I had a look at this post: https://forum.kde.org/viewtopic.php?f=66&t=98892



I first went to Search and Indexing (in the menu), unchecked Monitor file and directory changes, Enable when running on battery, Only when computer is not being used.  You can also go to System Settings/Desktop Search and uncheck Enable Nepomuk Semantic Desktop.

2.3 konqueror crashing
I also had repeated issues with konqueror crashing when clicking on links, but it seems to have resolved itself after disabling nepomuk.

2.4 guake/yakuake
Guake worked from the beginning -- I didn't have to set it up at all. However, I still ended up changing to yakuake since I figure it's a bit more KDE like.To get yakuake to remember changed settings (e.g. height, width) you'll need to close it manually. Restarting it will use the new settings. Not sure how I got yakuake to autostart -- I think it just happened.


2.5 The bottom panel
I had a difficult time figuring out how to add shortcuts to the panel, and once I had accidentally figured it out, I promptly screwed up the entire bottom panel, then managed to fix it again.

2.6 Keyboard shortcuts
I then had to figure out how to bind shortcuts to opening applications -- right-click on the menu ('kicker'), click Edit Applications, select your application, advanced, Advanced, Current Shortcut Key.
A more general approach to creating shortcuts was shown to me by an anonymous commentator (see below this post) -- I've posted the screenshot provided to me below:
shortcuts, gestures and whatnots


2.7 Update notification
I promptly removed the Update Notification widget -- it was driving me crazy and reminds me too much of windows. I first tried simply disabling the update notifier:



I then did apt-get autoremove update-notifier-kde. The problem -- even after a reboot I still get notifications about updates! I then did System Settings/Software Management, and click on the tiny, tiny button on the far right (level with the search field), and then click on settings and set Check for updates to never.  This finally seems to have done it.

2.8 Theme
I also changed from the default 'air' theme to 'oxygen'.


2.9 Gnome/GTK programs in KDE
To make gnome (well, gtk) programs look acceptable under KDE, install gtk2-engines-oxygen and gtk3-engines-oxygen and reboot.

2.10 Tap-to-click with synaptics
I can't get synaptics tap to click to work. Based on the odd post I tried gpointing-device-settings, but while there's a checkbox to disable tap to click, it won't enable it. I then installed kde-config-touchpad and was happy again. Funnily enough this made me discover that middle-click is mapped in KDE -- the one reason I wanted to explore something other than gnome...oh well.


2.11 Conky
To get conky to work properly I only had to make some minimal changes. /etc/conky/conky.conf:
background no own_window yes own_window_type normal own_window_argb_visual true own_window_transparent yes own_window_hints undecorated,below,sticky,skip_taskbar,skip_pager double_buffer yes alignment top_right background yes border_width 1 cpu_avg_samples 2 default_color white default_outline_color white default_shade_color white default_graph_size 20 200 draw_borders no draw_graph_borders yes draw_outline no draw_shades no use_xft yes xftfont DejaVu Sans Mono:size=12 gap_x 20 gap_y 60 minimum_size 5 5 net_avg_samples 2 no_buffers yes out_to_console no out_to_stderr no extra_newline no own_window_class Conky own_window_transparent yes stippled_borders 0 update_interval 1.0 uppercase no use_spacer none show_graph_scale no show_graph_range no TEXT ${exec lsb_release -ds} ${execi 5 acpi|gawk '{print $3,$4}'} ${addr eth0}/${addr wlan0} $hr ${color grey}Uptime:$color $uptime CPU:$alignc $cpu% $alignc$color ${cpugraph 10,200 ffff00 ff0000 -t} RAM:$alignc $mem/$memmax $alignc$color ${memgraph 10,200 ffff00 ff0000 -t} I/O:$alignc $diskio_read/$diskio_write $alignc$color ${diskiograph 10,200 ffff00 ff0000 -t} ${color grey}Frequency (in GHz):$color ${freq_g 1}, ${freq_g freq_g2} ${color grey}Temperature: $color CPU: ${acpitemp}°C $hr ${color grey}File systems:$alignr $color/ ${fs_used /}/${fs_size /} $alignr /home $color${fs_used /home}/${fs_size /home} ${color grey}Networking: ${color grey}eth0 ${color red} ${upspeed eth0}/${color green} ${downspeed eth0} ${color grey} ${color grey} ${upspeedgraph eth0 10,100 ffff00 ff0000 -t} ${color grey} ${downspeedgraph eth0 10,100 0000ff 00ff00 -t} ${color grey} ${color grey}wlan0 ${color red} ${upspeed wlan0}/${color green} ${downspeed wlan0} ${color grey} ${color grey} ${upspeedgraph wlan0 10,100 ffff00 ff0000 } ${color grey} ${downspeedgraph wlan0 10,100 0000ff 00ff00 } ${color grey} $hr ${color grey}Name CPU% MEM% ${color} ${top name 1} ${top cpu 1} ${top mem 1} ${color} ${top name 2} ${top cpu 2} ${top mem 2} ${color} ${top name 3} ${top cpu 3} ${top mem 3} $hr ${color}City Ping $alignr Time $font${color}San Francisco ${execi 60 ping -c 1 sfgate.com -n|grep icmp_seq|sed 's/=/\t/g'|gawk '{print $10,$11}'} $alignr${tztime America/Los_Angeles %a %H:%M} $font${color}Copenhagen $alignr${tztime Europe/Copenhagen %a %H:%M} $font${color}Shanghai $alignr${tztime Asia/Shanghai %a %H:%M} $font${color}Melbourne $alignr${tztime Australia/Melbourne %a %H:%M


And I've discovered that I really like the hot corner (top left) in gnome. There is a hot corner in kde too, but it does slightly different things.

KDE is pretty but it still gives me the occasional feeling that I'm using windows due to the bottom panel, the menu on the left and the task tray on the right, but these are things I'll hopefully grow out of. But yes, KDE is pretty -- very pretty. And it's pretty fast too.
My old gnome desktop (same as at the beginning of this post)

and my new KDE desktop
And one just to show how versatile KDE is -- you can even make it look (almost) like gnome 3

So...now it just remains to see how I feel after a week or so. The baby duck syndrome is pretty powerful...and I'm already well on my way to turning KDE into GNOME. But then, isn't the versatility of KDE the real selling point?

Postscript:
I ended up re-enabling nepomuk and spent some time struggling with akonadi to get kmail working -- the key to getting akonadi to work was to install the same server as the backend. In my case I did:
sudo apt-get install akonadi-backend-mysql mysql-server

21 August 2013

497. Compiling Wine 1.7 in a chroot on debian

Here's a generic way of building Wine 1.7 which is the new testing branch. And yes, it's the instructions for 1.5.28-1.6 recycled, with a few small amendments.

See here for information about 3D acceleration using libGL/U with Wine: http://verahill.blogspot.com.au/2013/05/429-briefly-wine-libglliubglu-blender.html

Getting started:
If you set up a e.g. chroot to build 1.6 you don't need to set up a new chroot to build 1.7. In that case, skip the set-up step below and instead re-enter your existing chroot like this:

sudo mount -o bind /proc wine32/proc
sudo cp /etc/resolv.conf wine32/etc/resolv.conf
sudo chroot wine32
su sandbox
cd ~/tmp

And skip to 'Building wine'.

Otherwise do this:
Setting up the Chroot
sudo apt-get install debootstrap
mkdir $HOME/tmp/architectures/wine32 -p
cd $HOME/tmp/architectures
sudo debootstrap --arch i386 wheezy $HOME/tmp/architectures/wine32 http://ftp.au.debian.org/debian/
sudo mount -o bind /proc wine32/proc
sudo cp /etc/resolv.conf wine32/etc/resolv.conf
sudo chroot wine32

You're now in the chroot:
apt-get update
apt-get install locales sudo vim
echo 'export LC_ALL="C"'>>/etc/bash.bashrc
echo 'export LANG="C"'>>/etc/bash.bashrc
echo '127.0.0.1 localhost beryllium' >> /etc/hosts
source /etc/bash.bashrc
adduser sandbox
usermod -g sudo sandbox
echo 'Defaults !tty_tickets' >> /etc/sudoers
su sandbox
cd ~/

Replace 'beryllium' with the name your host system (it's just to suppress error messages)

Building Wine
While still in the chroot, continue (the i386 is ok; don't worry about it -- you don't actually need it):

sudo apt-get install libx11-dev:i386 libfreetype6-dev:i386 libxcursor-dev:i386 libxi-dev:i386 libxxf86vm-dev:i386 libxrandr-dev:i386 libxinerama-dev:i386 libxcomposite-dev:i386 libglu-dev:i386 libosmesa-dev:i386 libglu-dev:i386 libosmesa-dev:i386 libdbus-1-dev:i386 libgnutls-dev:i386 libncurses-dev:i386 libsane-dev:i386 libv4l-dev:i386 libgphoto2-2-dev:i386 liblcms2-dev:i386 libgstreamer-plugins-base0.10-dev:i386 libcapi20-dev:i386 libcups2-dev:i386 libfontconfig-dev:i386 libgsm1-dev:i386 libtiff-dev:i386 libpng-dev:i386 libjpeg-dev:i386 libmpg123-dev:i386 libopenal-dev:i386 libldap-dev:i386 libxrender-dev:i386 libxml2-dev:i386 libxslt-dev:i386 libhal-dev:i386 gettext:i386 prelink:i386 bzip2:i386 bison:i386 flex:i386 oss4-dev:i386 checkinstall:i386 ocl-icd-libopencl1:i386 opencl-headers:i386 libasound2-dev:i386 build-essential
mkdir ~/tmp
cd ~/tmp
wget http://prdownloads.sourceforge.net/wine/wine-1.7.0.tar.bz2

tar xvf wine-1.7.0.tar.bz2
cd wine-1.7.0/
./configure
time make -j3
sudo checkinstall --install=no
checkinstall 1.6.2, Copyright 2009 Felipe Eduardo Sanchez Diaz Duran This software is released under the GNU GPL. The package documentation directory ./doc-pak does not exist. Should I create a default set of package docs? [y]: Preparing package documentation...OK Please write a description for the package. End your description with an empty line or EOF. >> wine 1.7.0 >> ***************************************** **** Debian package creation selected *** ***************************************** This package will be built according to these values: 0 - Maintainer: [ root@beryllium ] 1 - Summary: [ wine 1.7.0] 2 - Name: [ wine ] 3 - Version: [ 1.7.0] 4 - Release: [ 1 ] 5 - License: [ GPL ] 6 - Group: [ checkinstall ] 7 - Architecture: [ i386 ] 8 - Source location: [ wine-1.7.0 ] 9 - Alternate source location: [ ] 10 - Requires: [ ] 11 - Provides: [ wine ] 12 - Conflicts: [ ] 13 - Replaces: [ ]
Checkinstall takes a little while (In particular this step: 'Copying files to the temporary directory...').
********************************************************************** Done. The new package has been saved to /home/sandbox/tmp/wine-1.7.0/wine_1.7.0-1_i386.deb You can install it in your system anytime using: dpkg -i wine_1.7.0-1_i386.deb **********************************************************************


Installing Wine

Exit the chroot
sandbox@beryllium:~/tmp/wine-1.7.0$ exit
exit
root@beryllium:/# exit
exit
me@beryllium:~/tmp/architectures$ 

On your host system
 Enable multiarch* and install ia32-libs, since you've built a proper 32 bit binary:

sudo dpkg --add-architecture i386
sudo apt-get update
sudo apt-get install ia32-libs libgstreamer-plugins-base0.10-0

*At some point I think ia32-libs may be replaced by proper multiarch packages, but maybe not. So we're kind of doing both here.

 Copy the .deb package and install it
sudo cp wine32/home/sandbox/tmp/wine-1.7.0/wine_1.7.0-1_i386.deb .
sudo chown $USER wine_1.7.0-1_i386.deb
sudo dpkg -i wine_1.7.0-1_i386.deb

07 August 2013

491. Compiling ECCE v7.0 (on debian)

See here for more information about what's new:
http://ecce.pnl.gov/using/releasenotes.shtml

Building ECCE 7.0 is done in the same way as previous versions:
http://verahill.blogspot.com.au/2013/01/325-compiling-ecce-64-on-debian-testing.html

While ECCE is distributed as binaries as well, only a 32 bit version is available. To run on 32 bit linux you need to compile your own copy.

See here for Fedora 18 and here for Scientific Linux 6.3. Note that you no longer need to fix the GetOpts issue -- that's already fixed in the new version.

To add new basis sets, read scripts/gbs.README in the ecce source directory, or in apps/scripts/gbs.REAME in the installation directory.

Anyway, building on debian wheezy is simple:
First go to http://ecce.pnl.gov/using/download.shtml and give
sudo apt-get install bzip2 build-essential autoconf libtool ant pkg-config gtk+-2.0-dev libxt-dev csh gfortran openjdk-6-jdk python-dev libjpeg-dev imagemagick xterm
mkdir ~/tmp
cd ~/tmp
tar xvf ecce-v7.0-src.tar.bz2
cd ecce-v7.0/
export ECCE_HOME=`pwd`
cd build/
./build_ecce
Checking prerequisites for building ECCE... If any of the following tools aren't found or aren't the right version, hit -c at the prompt and either find or install the tool before re-running this script. The whereis command is useful for finding tools not in your path. Found gcc in: /usr/bin/gcc ECCE requires gcc 3.2.x or 4.x.x This version: gcc (Debian 4.7.2-5) 4.7.2 [..]. Found java in: /usr/bin/java ECCE requires java 1.5.x or 1.6.x This version: java version "1.7.0_25" Hit return if this java is OK... Found javac in: /usr/bin/javac ECCE requires javac to be the same version as java above This version: javac 1.7.0_25 [..] Found xterm in: /usr/bin/xterm Hit return if xterm was found... The /home/verahill/tmp/ecce/ecce-v7.0/scripts/sysdir script identifies the build platform directory as: Because this value is no platform-specific parent directory will be created for ECCE executables, libraries, etc. This works fine unless your site needs support for multiple platforms. Finished checking prerequisites for building ECCE. Do you want to skip these checks for future build_ecce invocations (y/n)? Y

If you're having issues with e.g. mismatching java and javac versions, use e.g.
sudo update-alternatives --config javac 
to correct it.

If you hit Y at the last question, when you re-run ./build_ecce you'll then start building:
./build_ecce
Xerces built
./build_ecce
Mesa OpenGL built
./build_ecce
wxWidgets built
./build_ecce
running build_ext wxPython built
./build_ecce
Apache HTTP server built
./build_ecce
Copying JMS server distribution jms_server.tar.bz2 Making server tar file ecce_server.tar Copying WebHelp tar file eccewebhelp.tar.bz2 Making combined tar file ecce.v7.0.tar Copying NWChem distribution Concatenating install script and combined tar file ecce.v7.0.tar create_ecce_bin finished ECCE built and distribution created in /home/verahill/tmp/ecce-v7.0
You can now install ~/tmp/ecce-v7.0/install_ecce.v7.0.csh as shown in this post: http://verahill.blogspot.com.au/2013/08/487-version-70-of-ecce-out-now.html

02 August 2013

487. Version 7.0 of ECCE out now

What's new:
1. There's a whole new type of calculation added, called 'Condensed Phase Reaction Study':

The previous reaction module has been renamed 'Gas Phase Reaction Study'. The real difference between the two types is that the Condensed Phase module uses the plan wave module in NWChem and thus uses explicit solvation, while the Gas Phase reaction module uses DFT/MP2. The latter can obviously use implicit solvation if requested.

2. The Def2-series of basis sets (Weigen and Alrichs, Phys. Chem. Chem. Phys, 2005, 7, 3297-3305) have been added to ECCE, and tools for adding basis sets on your own are now distributed with the ECCE sources. One of them is based on http://sourceforge.net/projects/nwbas2ecce/ , the scripts of which have been renamed gbsNWChemConverter and gsbDescriber. To add new basis sets, look at scripts/gbs.README in the ecce source directory, or in apps/scripts/gbs.REAME in the installation directory.

3. The precision of basis sets explicitly defined using coefficients and exponents has been greatly increased. Before, the energies obtained using the implicit nwchem basis sets and those obtained by defining the basis set through exponents and coefficients in the input could differ by 0.1 milli-Hartree in extreme cases. That should no longer be the case.

4. The COSMO parameter Rsolv can now be set in the Theory dialogue.

5. Other
There are other, non-obvious changes under the hood, such as an updated (2.2.25) apache httpd. The realease notes detail most of these.

Note that ECCE only runs on linux, and uses python 2.7. I haven't had much luck compiling it on Arch (due to python 3 seemingly being the default in Arch), but it runs fine on Debian Squeeze, Wheezy and Jessie, Fedora 18 and Scientific Linux 6.4 and Ubuntu something-or-other.


To install the binaries:

Note that there are only 64 bit binaries available. If you want to use ECCE on a 32 bit OS, you can, but you will have to compile the sources yourself. Luckily, this is very easy. Until I make a separate post, see here: http://verahill.blogspot.com.au/2013/01/325-compiling-ecce-64-on-debian-testing.html

To get the binaries (or sources) go to http://ecce.pnl.gov/using/download.shtml

Give your email and name so that the PNNL can track the number of users, as these things often impact on funding for different projects.

On the next page, select the version to download. Most likely you want the full binaries: install_ecce.v7.0.rhel5-gcc4.1.2-m64.csh

Download that file to e.g. ~/Downloads, and fire up your terminal -- in my case I'm doing an upgrade from a pre-release version of ECCE 7.0:
sudo apt-get install csh openjdk-7-jre
csh install_ecce.v7.0.rhel5-gcc4.1.2-m64.csh
Extracting ECCE distribution from install_ecce.v7.0.rhel5-gcc4.1.2-m64.csh... Main ECCE installation menu =========================== 1) Help on main menu options 2) Prerequisite software check 3) Full install 4) Full upgrade 5) Application software install 6) Application software upgrade 7) Server install 8) Server upgrade IMPORTANT: If you are uncertain about any aspect of installing or running ECCE at your site, please refer to the detailed ECCE Installation and Administration Guide at http://ecce.pnl.gov/docs/installation/2864B-Installation.pdf Hit at prompts to accept the default value in brackets. Selection: [1] Host name: [beryllium] New application installation directory: [/home/verahill/tmp/ecce/7.0-rc_final/ecce-v7.0/apps] Existing application directory to upgrade: New server installation directory: [/home/verahill/.ecce/e-v7.0b/server] Existing server directory to upgrade: Backup existing server user data (yes/no)? [yes] Yes ECCE v7.0 will be installed using the settings: Installation type: [full upgrade] Host name: [beryllium] Application installation directory: [/home/verahill/.ecce/ecce-v7.0b/apps] Application directory to upgrade: [/home/verahill/.ecce/ecce-v7.0/apps] Server installation directory: [/home/verahill/.ecce/ecce-v7.0b/server] Server directory to upgrade: [/home/verahill/.ecce/ecce-v7.0/server] Backup existing server user data: [yes] Are these choices correct (yes/no/quit)? [yes] Installing ECCE application software in /home/verahill/.ecce/ecce-v7.0b/apps... Extracting application distribution... Extracting NWChem distribution... Extracting client WebHelp distribution... Configuring application software... Configuring NWChem... Installing ECCE server in /home/verahill/.ecce/e-v7.0b/server... Extracting data server in /home/verahill/.ecce/e-v7.0b/server/httpd... Extracting data libraries in /home/verahill/.ecce/e-v7.0b/server/data... Extracting Java Messaging Server in /home/verahill/.ecce/e-v7.0b/server/activemq... Configuring ECCE server... Copying user data from server to be upgraded... Copying share data from server to be upgraded... ECCE installation succeeded. *************************************************************** !! You MUST perform the following steps in order to use ECCE !! -- Unless only the user 'verahill' will be running ECCE, start the ECCE server as 'verahill' with: /home/verahill/.ecce/e-v7.0b/server/ecce-admin/start_ecce_server -- To register machines to run computational codes, please see the installation and compute resource registration manuals at http://ecce.pnl.gov/using/installguide.shtml -- Before running ECCE each user must source an environment setup script. For csh/tcsh users add this to ~/.cshrc: if ( -e /home/verahill/.ecce/e-v7.0b/apps/scripts/runtime_setup ) then source /home/verahill/.ecce/e-v7.0b/apps/scripts/runtime_setup endif For sh/bash users, add this to ~/.profile or ~/.bashrc: if [ -e /home/verahill/.ecce/e-v7.0b/apps/scripts/runtime_setup.sh ]; then . /home/verahill/.ecce/e-v7.0b/apps/scripts/runtime_setup.sh fi ***************************************************************

Edit your ~/.bashrc and add the following lines (edited for your system):
export ECCE_HOME=/home/verahill/.ecce/ecce-v7.0b/apps alias stopecce='/home/verahill/.ecce/ecce-v7.0b/server/ecce-admin/stop_ecce_server' alias startecce='/home/verahill/.ecce/ecce-v7.0b/server/ecce-admin/start_ecce_server'
You can then start ecce by typing...ecce.

19 July 2013

479. Compiling Wine 1.6 on Debian (using a chroot)

Update:
I noticed
configure: libOSMesa 32-bit development files not found (or too old), OpenGL rendering in bitmaps won't be supported.

popping up at the end of ./configure. I've added a fix for it based on http://forum.winehq.org/viewtopic.php?f=2&t=17713

Original post:
Here's a generic way of building Wine 1.6 which is now stable. And yes, it's the instructions for 1.5.28-1.6-rcX recycled.

See here for information about 3D acceleration using libGL/U with Wine: http://verahill.blogspot.com.au/2013/05/429-briefly-wine-libglliubglu-blender.html

Getting started:
If you set up a e.g. chroot to build 1.5.28 you don't need to set up a new chroot to build 1.6. In that case, skip the set-up step below and instead re-enter your existing chroot like this:

sudo mount -o bind /proc wine32/proc
sudo cp /etc/resolv.conf wine32/etc/resolv.conf
sudo chroot wine32
su sandbox
cd ~/tmp

And skip to 'Building wine'.

Otherwise do this:
Setting up the Chroot
sudo apt-get install debootstrap
mkdir $HOME/tmp/architectures/wine32 -p
cd $HOME/tmp/architectures
sudo debootstrap --arch i386 wheezy $HOME/tmp/architectures/wine32 http://ftp.au.debian.org/debian/
sudo mount -o bind /proc wine32/proc
sudo cp /etc/resolv.conf wine32/etc/resolv.conf
sudo chroot wine32

You're now in the chroot:
apt-get update
apt-get install locales sudo vim
echo 'export LC_ALL="C"'>>/etc/bash.bashrc
echo 'export LANG="C"'>>/etc/bash.bashrc
echo '127.0.0.1 localhost beryllium' >> /etc/hosts
source /etc/bash.bashrc
adduser sandbox
usermod -g sudo sandbox
echo 'Defaults !tty_tickets' >> /etc/sudoers
su sandbox
cd ~/

Replace 'beryllium' with the name your host system (it's just to suppress error messages)

Building Wine
While still in the chroot, continue (the i386 is ok; don't worry about it -- you don't actually need it):

sudo apt-get install libx11-dev:i386 libfreetype6-dev:i386 libxcursor-dev:i386 libxi-dev:i386 libxxf86vm-dev:i386 libxrandr-dev:i386 libxinerama-dev:i386 libxcomposite-dev:i386 libglu-dev:i386 libosmesa-dev:i386 libglu-dev:i386 libosmesa-dev:i386 libdbus-1-dev:i386 libgnutls-dev:i386 libncurses-dev:i386 libsane-dev:i386 libv4l-dev:i386 libgphoto2-2-dev:i386 liblcms-dev:i386 libgstreamer-plugins-base0.10-dev:i386 libcapi20-dev:i386 libcups2-dev:i386 libfontconfig-dev:i386 libgsm1-dev:i386 libtiff-dev:i386 libpng-dev:i386 libjpeg-dev:i386 libmpg123-dev:i386 libopenal-dev:i386 libldap-dev:i386 libxrender-dev:i386 libxml2-dev:i386 libxslt-dev:i386 libhal-dev:i386 gettext:i386 prelink:i386 bzip2:i386 bison:i386 flex:i386 oss4-dev:i386 checkinstall:i386 ocl-icd-libopencl1:i386 opencl-headers:i386 libasound2-dev:i386 build-essential
mkdir ~/tmp
cd ~/tmp
wget http://prdownloads.sourceforge.net/wine/wine-1.6.tar.bz2

tar xvf wine-1.6.tar.bz2
cd wine-1.6/


Optional:
To avoid getting the

configure: libOSMesa 32-bit development files not found (or too old), OpenGL rendering in bitmaps won't be supported.

message, do the following:
1. Edit configure
 9450 LIBS="-lOSMesa -lGLU -lGL $X_LIBS $X_PRE_LIBS $XLIB -lm $X_EXTRA_LIBS $LIBS"

2. Also change
 9473     *) ac_cv_lib_soname_OSMesa=libOSMesa.so

Does it change anything? I don't know. But it removes the error message which is triggered by missing symbols so I think it does since the symbols are found in GLU/GL.
End optional.

Then do
./configure
time make -j3
sudo checkinstall --install=no
checkinstall 1.6.2, Copyright 2009 Felipe Eduardo Sanchez Diaz Duran This software is released under the GNU GPL. The package documentation directory ./doc-pak does not exist. Should I create a default set of package docs? [y]: Preparing package documentation...OK Please write a description for the package. End your description with an empty line or EOF. >> wine 1.6 >> ***************************************** **** Debian package creation selected *** ***************************************** This package will be built according to these values: 0 - Maintainer: [ root@beryllium ] 1 - Summary: [ wine 1.6] 2 - Name: [ wine ] 3 - Version: [ 1.6] 4 - Release: [ 1 ] 5 - License: [ GPL ] 6 - Group: [ checkinstall ] 7 - Architecture: [ i386 ] 8 - Source location: [ wine-1.6 ] 9 - Alternate source location: [ ] 10 - Requires: [ ] 11 - Provides: [ wine ] 12 - Conflicts: [ ] 13 - Replaces: [ ]
Checkinstall takes a little while (In particular this step: 'Copying files to the temporary directory...').

Installing Wine

Exit the chroot
sandbox@beryllium:~/tmp/wine-1.6$ exit
exit
root@beryllium:/# exit
exit
me@beryllium:~/tmp/architectures$ 

On your host system
 Enable multiarch* and install ia32-libs, since you've built a proper 32 bit binary:

sudo dpkg --add-architecture i386
sudo apt-get update
sudo apt-get install ia32-libs

*At some point I think ia32-libs may be replaced by proper multiarch packages, but maybe not. So we're kind of doing both here.

 Copy the .deb package and install it
sudo cp wine32/home/sandbox/tmp/wine-1.6/wine_1.6-1_i386.deb .
sudo chown $USER wine_1.6-1_i386.deb
sudo dpkg -i wine_1.6-1_i386.deb

17 July 2013

478. Briefly: proftpd on debian

I need to transfer raw mass spec files off of the computer controlling our waters zmd, and it seems like I may be the only one in the department wishing to do so.

Since the computer is running Windows NT 4 and doesn't support USB drives out of the box, and I'm a bit worried about installing new software (e.g. old versions of filezilla via oldapps) on a computer on which a lot of people rely, I have two options:

* use SMB i.e. a windows share
or
* use ftp

I'm having all sorts of trouble getting my samba to work well at work -- my computers are sitting on a 192.168.2.0/24 LAN behind a router connected to the corporate network which has proper IP addresses (i.e. not using a reserved private network address space). I haven't managed to get my computer behind the router to 'see' the other computers and their shares at work beyond my router . I can, however, connect directly to the computers using e.g. smbclient -- they just won't show up in e.g. nautilus under windows network or using nmblookup. At any rate, connection directly to the target computer prompts me for a password and it seems that there are no open, accessible shares on that computer, only password protected ones.

Win NT has a DOS ftp client, so I finally decided to set up a quick and dirty ftp server on my workstation in my office so that I could transfer a couple of data files to figure out my other issue -- whether I have any piece of software that can actually open the masslynx .raw files. Turns out that neither wsearch32 nor openchrom can, so the exercise has been somewhat futile, although it has to be said that I'd like to be in charge of any raw data that leads to publications, and so I should be able to manage the storage of it myself.

Note: ftp is an inherently unsafe method since it doesn't use encryption. Use a separate user for this with no privileges, change the password of that user regularly, and close port 21 whenever you aren't using it in order to not advertise that you are running an ftp server. Use ssh/sftp if at all possible.

Anyway, setting up an ftp server was easy.

This method follows this post, http://ubuntuforums.org/showthread.php?t=79588, almost verbatim.

First install proftpd

sudo apt-get install proftpd

Edit /etc/shells:
# /etc/shells: valid login shells /bin/csh /bin/sh #/usr/bin/es #/usr/bin/ksh #/bin/ksh #/usr/bin/rc #/usr/bin/esh /bin/dash /bin/bash /bin/rbash #/usr/bin/screen #/bin/tcsh #/usr/bin/tcsh #/bin/ksh93 /bin/false

sudo adduser ftpuser

su ftpuser
cd ~
mkdir download
mkdir upload
exit

Edit /etc/proftpd/proftpd.conf. In addition to what was already there, I added
UserAliasOnly on UserAlias spinebill ftpuser ExtendedLog /var/log/ftp.log TransferLog /var/log/xferlog SystemLog /var/log/syslog.log AllowStoreRestart on <Directory /home/ftpuser> Umask 022 022 AllowOverwrite off <Limit MKD STOR DELE XMKD RNRF RNTO RMD XRMD> DenyAll </Limit> </Directory> <Directory /home/ftpuser/download/*> Umask 022 022 AllowOverwrite off <Limit MKD STOR DELE XMKD RNEF RNTO RMD XRMD> DenyAll </Limit> </Directory> <Directory /home/ftpuser/upload/> Umask 022 022 AllowOverwrite on <Limit READ RMD DELE> DenyAll </Limit> <Limit STOR CWD MKD> AllowAll </Limit> </Directory> Include /etc/proftpd/conf.d/

su ftpuser
chsh -s /bin/false
exit

Check the syntax:
sudo proftpd -td5

Test:
ftp `hostname`
Connected to beryllium. 220 ProFTPD 1.3.4a Server (Debian) [192.168.1.1] Name (beryllium:me): spinebill 331 Password required for spinebill Password: 230 User spinebill logged in Remote system type is UNIX. Using binary mode to transfer files. ftp>
I have since tested this from the Win NT 4 computer and everything is working well. I had to familiarise myself with the windows ftp client first: http://www.nsftools.com/tips/MSFTP.htm

01 July 2013

469. Intel compiler (icc, icpc, ifort) on Debian

I've heard it said that MKL is faster than ACML even on AMD cpus. I've also heard it said that Intel compile + mkl beats everything else, even on AMD cpus.

So let's test the veracity of that statement. I'm in particular looking forward to seeing how this affects my amd fx 8150.

Note that the ACML libs are available as separate packages for different compilers -- download the right libs when linking (i.e. the gnu ones for gcc, and the intel ones for intel composer)

But first we need to install and set-up the intel compiler suite, and that's what we'll do in this post.

In the example below I've installed in on an AMD Athlon II X3, hence the message about non-Intel architecture.

Installation:

Register for the Intel Parallel Studio XE as touched upon in this post: http://verahill.blogspot.com.au/2013/06/465-intel-mkl-math-kernel-library-on.html

Download. It's about 2 Gb. Then extract, and run install.sh

sudo apt-get install build-essential
sudo sh install.sh
Step no: 1 of 7 | Welcome -------------------------------------------------------------------------------- Welcome to the Intel(R) Parallel Studio XE 2013 Update 3 for Linux* installation program. -------------------------------------------------------------------------------- You will complete the steps below during this installation: Step 1 : Welcome Step 2 : License Step 3 : Activation Step 4 : Intel(R) Software Improvement Program Step 5 : Options Step 6 : Installation Step 7 : Complete -------------------------------------------------------------------------------- Press "Enter" key to continue or "q" to quit: -------------------------------------------------------------------------------- Checking the prerequisites. It can take several minutes. Please wait... -------------------------------------------------------------------------------- Step no: 1 of 7 | Options > Missing Optional Pre-requisite(s) -------------------------------------------------------------------------------- There are one or more optional unresolved issues. It is highly recommended to resolve them all before you continue the installation. You can fix them without exiting from the installation and re-check. Or you can quit from the installation, fix them and run the installation again. -------------------------------------------------------------------------------- Missing optional pre-requisites -- Intel(R) VTune(TM) Amplifier XE 2013 Update 5: unsupported OS -- Intel(R) Inspector XE 2013 Update 5: unsupported OS -- Intel(R) Advisor XE 2013 Update 2: unsupported OS -- Intel(R) Composer XE 2013 Update 3 for Linux*: unsupported OS -------------------------------------------------------------------------------- 1. Skip missing optional pre-requisites [default] 2. Show the detailed info about issue(s) 3. Re-check the pre-requisites h. Help b. Back to the previous menu q. Quit -------------------------------------------------------------------------------- Please type a selection or press "Enter" to accept default choice [1]: Step no: 2 of 7 | License -------------------------------------------------------------------------------- As noted in the Intel(R) Software Development Product End User License Agreement, the Intel(R) Software Development Product you install will send Intel [..] -------------------------------------------------------------------------------- Do you agree to be bound by the terms and conditions of this license agreement? Type "accept" to continue or "decline" to back to the previous menu: accept Step no: 3 of 7 | Activation -------------------------------------------------------------------------------- If you have purchased this product and have the serial number and a connection to the internet you can choose to activate the product at this time. Activation is a secure and anonymous one-time process that verifies your software licensing rights to use the product. Alternatively, you can choose to evaluate the product or defer activation by choosing the evaluate option. Evaluation software will time out in about one month. Also you can use license file, license manager, or remote activation if the system you are installing on does not have internet access activation options. -------------------------------------------------------------------------------- 1. I want to activate my product using a serial number [default] 2. I want to evaluate my product or activate later 3. I want to activate either remotely, or by using a license file, or by using a license manager h. Help b. Back to the previous menu q. Quit -------------------------------------------------------------------------------- Please type a selection or press "Enter" to accept default choice [1]: Note: Press "Enter" key to back to the previous menu. Please type your serial number (the format is XXXX-XXXXXXXX): -------------------------------------------------------------------------------- Activation completed successfully. -------------------------------------------------------------------------------- Press "Enter" key to continue: Step no: 4 of 7 | Intel(R) Software Improvement Program -------------------------------------------------------------------------------- Help improve your experience with Intel(R) software Participate in the design of future Intel software. Select 'Yes' to give us permission to learn about how you use your Intel software and we will do the rest. - No Personal contact information is collected - There are no surveys or additional follow-up emails by opting in - You can stop participating at any time Learn more about Intel(R) Software Improvement Program http://software.intel.com/en-us/articles/software-improvement-program With your permission, Intel may automatically receive anonymous information about how you use your current and future Intel software. -------------------------------------------------------------------------------- 1. Yes, I am willing to participate and improve Intel software. (Recommended) 2. No, I don't want to participate in the Intel(R) Software Improvement Program at this time. b. Back to the previous menu q. Quit -------------------------------------------------------------------------------- Please type a selection: Step no: 5 of 7 | Options -------------------------------------------------------------------------------- You are now ready to begin installation. You can use all default installation settings by simply choosing the "Start installation Now" option or you can customize these settings by selecting any of the change options given below first. You can view a summary of the settings by selecting "Show pre-install summary". -------------------------------------------------------------------------------- 1. Start installation Now 2. Change install directory [ /opt/intel ] 3. Change components to install [ All ] 4. Change advanced options 5. Show pre-install summary h. Help b. Back to the previous menu q. Quit -------------------------------------------------------------------------------- Please type a selection or press "Enter" to accept default choice [1]: -------------------------------------------------------------------------------- Checking the prerequisites. It can take several minutes. Please wait... -------------------------------------------------------------------------------- Step no: 5 of 7 | Options > Missing Optional Pre-requisite(s) -------------------------------------------------------------------------------- There are one or more optional unresolved issues. It is highly recommended to resolve them all before you continue the installation. You can fix them without exiting from the installation and re-check. Or you can quit from the installation, fix them and run the installation again. -------------------------------------------------------------------------------- Missing optional pre-requisites -- Intel(R) VTune(TM) Amplifier XE 2013 Update 5: The system does not have an Intel Architecture processor -------------------------------------------------------------------------------- 1. Skip missing optional pre-requisites [default] 2. Show the detailed info about issue(s) 3. Re-check the pre-requisites h. Help b. Back to the previous menu q. Quit -------------------------------------------------------------------------------- Please type a selection or press "Enter" to accept default choice [1]: Step no: 7 of 7 | Complete -------------------------------------------------------------------------------- Thank you for installing and using the Intel(R) Parallel Studio XE 2013 Update 3 for Linux* Reminder: Intel(R) VTune(TM) Amplifier XE users must be members of the "vtune" permissions group in order to use Event-based Sampling. To register your product purchase, visit https://registrationcenter.intel.com/RegCenter/registerexpress.aspx?clientsn=N43 3-4XGWTJLB To get started using Intel(R) VTune(TM) Amplifier XE 2013 Update 5: - To set your environment variables: source /opt/intel/vtune_amplifier_xe_2013/amplxe-vars.sh - To start the graphical user interface: amplxe-gui - To use the command-line interface: amplxe-cl - For more getting started resources: /opt/intel/vtune_amplifier_xe_2013/ documentation/en/welcomepage/get_started.html. To get started using Intel(R) Inspector XE 2013 Update 5: - To set your environment variables: source /opt/intel/inspector_xe_2013/inspxe-vars.sh - To start the graphical user interface: inspxe-gui - To use the command-line interface: inspxe-cl - For more getting started resources: /opt/intel/inspector_xe_2013/ documentation/en/welcomepage/get_started.html. To get started using Intel(R) Advisor XE 2013 Update 2: - To set your environment variables: source /opt/intel/advisor_xe_2013/advixe-vars.sh - To start the graphical user interface: advixe-gui - To use the command-line interface: advixe-cl - For more getting started resources: /opt/intel/advisor_xe_2013/ documentation/en/welcomepage/get_started.html. To get started using Intel(R) Composer XE 2013 Update 3 for Linux*: - Set the environment variables for a terminal window using one of the following (replace "intel64" with "ia32" if you are using a 32-bit platform). For csh/tcsh: $ source /opt/intel/bin/compilervars.csh intel64 For bash: $ source /opt/intel/bin/compilervars.sh intel64 To invoke the installed compilers: For C++: icpc For C: icc For Fortran: ifort To get help, append the -help option or precede with the man command. - For more getting started resources: /opt/intel/composer_xe_2013/Documentation/en_US/get_started_lc.htm. /opt/intel/composer_xe_2013/Documentation/en_US/get_started_lf.htm. To view movies and additional training, visit http://www.intel.com/software/products. -------------------------------------------------------------------------------- q. Quit [default] -------------------------------------------------------------------------------- Please type a selection or press "Enter" to accept default choice [q]:


The Files and Setup:
The compiler binaries are now found in /opt/intel/composer_xe_2013.3.163/bin/intel64/ . Of particular interest are ifort, icc and icpc (fortran, c and c++).

In addition, you'll need the lib and include files, which are found in /opt/intel/composer_xe_2013.3.163/compiler/lib/intel64/ and /opt/intel/composer_xe_2013.3.163/compiler/include/intel64.

You can either simply add the libs using LD_LIBRARY_PATH, but a perhaps easier and better method is to create a file: /etc/ld.so.conf.d/intel.conf
/opt/intel/composer_xe_2013.3.163/compiler/lib/intel64 /opt/intel/composer_xe_2013.4.183/mkl/lib/intel64
Once that's done, run
sudo ldconfig

Then do
echo 'PATH=$PATH:/opt/intel/composer_xe_2013.3.163/bin/intel64' >> ~/.bashrc
source ~/.bashrc

Testing:
See this post for an example of how to compile nwchem using ifort: http://verahill.blogspot.com.au/2013/07/470-very-briefly-compiling-nwchem-63.html

468. Kernel 3.10 on Debian

NOTE I: As of 3.10.2 the nvidia module will still not build.  I've also tried 3.10.5 and it also does not work.

NOTE II: I'm getting random slowdowns on my SL410 laptop with intel graphics. Not sure if it's the same issue as this: http://verahill.blogspot.com.au/2013/03/368-slow-mouse-and-keyboard-triggered.html
Once kworker shows up in top everything grinds to a slow crawl. I also notice that I never used 3.9 on that laptop, so the issue may be present there too.

There are several ways of building a kernel. The easiest (a purely subjective statement) is to use kernel-package i.e. make-kpkg. However, every now and again I see people writing that it's been deprecated.

Either way, start by doing
sudo apt-get install fakeroot build-essential ncurses-dev
mkdir ~/tmp
cd ~/tmp
wget https://www.kernel.org/pub/linux/kernel/v3.x/linux-3.10.tar.xz
tar xvf linux-3.10.tar.xz
cd linux-3.10/
cat /boot/config-`uname -r`>.config
make oldconfig
Timer tick handling 1. Periodic timer ticks (constant rate, no dynticks) (HZ_PERIODIC) (NEW) > 2. Idle dynticks system (tickless idle) (NO_HZ_IDLE) (NEW) 3. Full dynticks system (tickless) (NO_HZ_FULL) (NEW) choice[1-3]: Memory placement aware NUMA scheduler (NUMA_BALANCING) [N/y/?] (NEW) Simple CPU accounting cgroup subsystem (CGROUP_CPUACCT) [N/y/?] (NEW) Group CPU scheduler (CGROUP_SCHED) [N/y/?] (NEW) Automatic process group scheduling (SCHED_AUTOGROUP) [N/y/?] (NEW) Choose SLAB allocator 1. SLAB (SLAB) (NEW) > 2. SLUB (Unqueued Allocator) (SLUB) choice[1-2?]: Linux guest support (HYPERVISOR_GUEST) [N/y/?] (NEW) Timer frequency 1. 100 HZ (HZ_100) 2. 250 HZ (HZ_250) (NEW) 3. 300 HZ (HZ_300) > 4. 1000 HZ (HZ_1000) choice[1-4?]: Memory Hotplug (ACPI_HOTPLUG_MEMORY) [N/y/?] (NEW) AMD frequency sensitivity feedback powersave bias (X86_AMD_FREQ_SENSITIVITY) [N/m/?] (NEW) Kernel support for scripts starting with #! (BINFMT_SCRIPT) [Y/n/m/?] (NEW) InfiniBand media type support (TIPC_MEDIA_IB) [N/y/?] (NEW) Network Coding (BATMAN_ADV_NC) [N/y/?] (NEW) NETLINK: mmaped IO (NETLINK_MMAP) [N/y/?] (NEW) NETLINK: socket monitoring interface (NETLINK_DIAG) [N/m/y/?] (NEW) Dummy IRQ handler (DUMMY_IRQ) [N/m/y/?] (NEW) Generic on-chip SRAM driver (SRAM) [N/y/?] (NEW) ME Enabled Intel Chipsets (INTEL_MEI_ME) [N/m/y/?] (NEW) Block device as cache (BCACHE) [N/m/y/?] (NEW) Qualcomm Atheros AR816x/AR817x support (ALX) [N/m/y/?] (NEW) QLOGIC QLCNIC 83XX family SR-IOV Support (QLCNIC_SRIOV) [Y/n/?] (NEW) Realtek RTL8152 Based USB 2.0 Ethernet Adapters (USB_RTL8152) [N/m/?] (NEW) Atheros ath9k rate control (ATH9K_LEGACY_RATE_CONTROL) [N/y/?] (NEW) rt2800usb - Include support for rt55xx devices (EXPERIMENTAL) (RT2800USB_RT55XX) [N/y/?] (NEW) y Realtek RTL8188EE Wireless Network Adapter (RTL8188EE) [N/m/?] (NEW) IMS Passenger Control Unit driver (INPUT_IMS_PCU) [N/m/?] (NEW) Qualcomm Single-wire Serial Bus Interface (SSBI) (SSBI) [N/m/y/?] (NEW) Analog Devices ADT7310/ADT7320 (SENSORS_ADT7310) [N/m/y/?] (NEW) National Semiconductor LM95234 (SENSORS_LM95234) [N/m/?] (NEW) Nuvoton NCT6775F and compatibles (SENSORS_NCT6775) [N/m/y/?] (NEW) generic cpu cooling support (CPU_THERMAL) [N/y/?] (NEW) y ChromeOS Embedded Controller (MFD_CROS_EC) [N/m/y/?] (NEW) Silicon Laboratories 4761/64/68 AM/FM radio. (MFD_SI476X_CORE) [N/m/?] (NEW) System Controller Register R/W Based on Regmap (MFD_SYSCON) [N/y/?] (NEW) TI TPS65912 Power Management chip (MFD_TPS65912) [N/y/?] (NEW) Conexant cx25821 support (VIDEO_CX25821) [N/m/?] (NEW) Cypress firmware helper routines (CYPRESS_FIRMWARE) [N/m] (NEW) QXL virtual GPU (DRM_QXL) [N/m/?] (NEW) Apple infrared receiver (HID_APPLEIR) [N/m/?] (NEW) Enable USB persist by default (USB_DEFAULT_PERSIST) [Y/n/?] (NEW) USB-Wishbone adapter interface driver (USB_SERIAL_WISHBONE) [N/m/?] (NEW) USB Physical Layer drivers (USB_PHY) [N/y/?] (NEW) PXA 27x (USB_PXA27X) [N/m/?] (NEW) MARVELL PXA2128 USB 3.0 controller (USB_MV_U3D) [N/m/?] (NEW) LED Support for TI LP5562 LED driver chip (LEDS_LP5562) [N/m/?] (NEW) LED Camera Flash/Torch Trigger (LEDS_TRIGGER_CAMERA) [N/m/y/?] (NEW) y iSCSI Extentions for RDMA (iSER) target support (INFINIBAND_ISERT) [N/m/?] (NEW) Set system time from RTC on startup and resume (RTC_HCTOSYS) [Y/n/?] (NEW) Set the RTC time based on NTP synchronization (RTC_SYSTOHC) [Y/n/?] (NEW) RTC used to set the system time (RTC_HCTOSYS_DEVICE) [rtc0] (NEW) WIS GO7007 MPEG encoder support (VIDEO_GO7007) [N/m/?] (NEW) DesignWare USB2 DRD Core Support (USB_DWC2) [N/m/?] (NEW) pvpanic device support (PVPANIC) [N/m/y/?] (NEW) Reset Controller Support (RESET_CONTROLLER) [N/y/?] (NEW) XFS Verbose Warnings (XFS_WARN) [N/y/?] (NEW) Btrfs will run sanity tests upon loading (BTRFS_FS_RUN_SANITY_TESTS) [N/y/?] (NEW) Btrfs debugging support (BTRFS_DEBUG) [N/y/?] (NEW) EFI Variable filesystem (EFIVAR_FS) [N/m/y/?] (NEW) torture tests for RCU (RCU_TORTURE_TEST) [N/m/y/?] (NEW) Ring buffer startup self test (RING_BUFFER_STARTUP_TEST) [N/y/?] (NEW) Test functions located in the string_helpers module at runtime (TEST_STRING_HELPERS) [N/m/y] (NEW) CMAC support (CRYPTO_CMAC) [N/m/y/?] (NEW) SHA256 digest algorithm (SSSE3/AVX/AVX2) (CRYPTO_SHA256_SSSE3) [N/m/y/?] (NEW) y SHA512 digest algorithm (SSSE3/AVX/AVX2) (CRYPTO_SHA512_SSSE3) [N/m/y/?] (NEW) y Camellia cipher algorithm (x86_64/AES-NI/AVX2) (CRYPTO_CAMELLIA_AESNI_AVX2_X86_64) [N/m/y/?] (NEW) m Serpent cipher algorithm (x86_64/AVX2) (CRYPTO_SERPENT_AVX2_X86_64) [N/m/y/?] (NEW) m KVM legacy PCI device assignment support (KVM_DEVICE_ASSIGNMENT) [Y/n/?] (NEW) VHOST_SCSI TCM fabric driver (VHOST_SCSI) [N/m/?] (NEW)
make menuconfig

You can now enable any additional modules by navigating the menu structure. Note that most likely you don't have to enable anything in this step, but it can come in handy if there's a major transition (e.g. the way multimedia was handled changed between kernel 3.5 and 3.6) or if you want to enable a previously disabled option.

Then pick either method 1 or 2 below.

If you only want to compile modules that are currently in use (not a good idea if you want to use the same kernel on a range of computers, or have USB devices that aren't currently plugged in) you can do that by using make localmodconfig instead of make oldconfig. I wouldn't recommend it -- in most cases it won't make a faster kernel, and space and memory tends not to be much of in the way of issues these days.


Method 1. kernel-package
Below, change -j2 to -jX, where X is the number of cores in your CPU (not cores+1 or anything funny like that. See other posts on this blog for compilation performance tests)

sudo apt-get install kernel-package
make-kpkg clean
time fakeroot make-kpkg -j2 --initrd kernel_image kernel_headers
sudo dpkg -i ../linux-image-3.10.0_3.10.0-10.00.Custom_amd64.deb ../linux-headers-3.10.0_3.10.0-10.00.Custom_amd64.deb


Took 49 minutes on a 3-core AMD Athlon II, and used ca 7 Gb.

The files are shown below:
-rw-r--r--  1 me me 8.4M Jul  1 16:02 linux-headers-3.10.0_3.10.0-10.00.Custom_amd64.deb
-rw-r--r--  1 me me  32M Jul  1 16:00 linux-image-3.10.0_3.10.0-10.00.Custom_amd64.deb


Method 2. make deb-pkg

make clean
time make deb-pkg -j2
sudo dpkg -i ../linux-firmware-image_3.10.0-2_amd64.deb ../linux-headers-3.10.0_3.10.0-2_amd64.deb ../linux-libc-dev_3.10.0-2_amd64.deb ../linux-image-3.10.0_3.10.0-2_amd64.deb

It took ca 50 minutes, it used ca 7 Gb, and generated the following files:
-rw-r--r-- 1 me me 1.1M Jul  1 16:58 ../linux-firmware-image_3.10.0-2_amd64.deb
-rw-r--r-- 1 me me 9.7M Jul  1 16:58 ../linux-headers-3.10.0_3.10.0-2_amd64.deb
-rw-r--r-- 1 me me 458M Jul  1 17:13 ../linux-image-3.10.0_3.10.0-2_amd64.deb
-rw-r--r-- 1 me me 920K Jul  1 16:58 ../linux-libc-dev_3.10.0-2_amd64.deb

Note the size of the linux-image-3.10.0_3.10.0-2_amd64.deb package.


Difference:
There are a few differences. One method, the kpgk one, is supposedly 'deprecated', but it's been working fine in the past and will work fine in the future (at least for some time).

The make dev-pkg method also generates a much bigger image file -- by a factor of 15 or so.

Otherwise, the chief difference, from what I can see, is if you want to uninstall the kernel. Remove the libc-dev package requires you to downgrade the package using apt-get, by specifying a version e.g.

apt-cache policy linux-libc-dev
linux-libc-dev: Installed: 3.10.0-1 Candidate: 3.10.0-1 Version table: *** 3.10.0-1 0 100 /var/lib/dpkg/status 3.9.6-1~bpo70+1 0 100 http://ftp.iinet.net.au/debian/debian/ wheezy-backports/main amd64 Packages 100 http://ftp.debian.org/debian/ wheezy-backports/main amd64 Packages 3.2.46-1 0 500 http://ftp.iinet.net.au/debian/debian/ wheezy/main amd64 Packages 3.2.41-2+deb7u2 0 500 http://security.debian.org/ wheezy/updates/main amd64 Packages
sudo apt-get install linux-libc-dev=3.9.6-1~bpo70+1

I personally prefer the kernel-package approach.

27 June 2013

465. The Intel MKL (Math Kernel Library) on Linux (Debian) -- for free

I've been living under the impression that the Intel MKL weren't free.

In all fairness, since I'm using AMD almost exclusively and the ACML is free -- and OpenBlas has worked fine on my i5-2400-based node -- I haven't had enough motivation to really dig into this. However, as part of a post on compiling GAMESS US, Kirill Berezovsky mentions in passing that you can get both the Intel MKL and ifortran for free for non-commercial use.

To be fair, that's free in the Windows sense, not in the Linux sense. You've still got enough restrictions to make Stallman weep, but it's free enough that we have a fighting chance at evaluation the software -- and use it if it's good enough.

See here for ACML: http://verahill.blogspot.com.au/2013/05/422-set-up-acml-on-linux.html
See here for OpenBlas: http://verahill.blogspot.com.au/2013/05/423-openblas-on-debian-wheezy.html

Anyway, this is about MKL:


0. Register a request
Go to http://software.intel.com/en-us/non-commercial-software-development
Click on what you want to download -- in this case the Intel MKL. Fill out the form and hit submit. Note that you'll need to enable cookies for this to work.


[while I was at it I got the intel parallel studio xe as well -- that's 2 Gb though, so don't get it unless you want it and have a lot of bandwidth to spare]


1. Download
You'll get an email with a link. Click it:


The MKL file is 609 MB, which is a bit bigger than the ACML files which tend to be around 67-70 Mb (each).

Note: the email you get has a serial number. From what I understand it's valid for one year if you want to download updates and new releases. However, nowhere does it say that any installed software will expire, so I presume that you can continue using the MKL libraries indefinitely.


2. Install
In my case the file is called l_mkl_11.0.4.183.tgz and it was downloaded to ~/Downloads
mkdir ~/tmp
cd ~/tmp
cp ~/Downloads/l_mkl_11.0.4.183.tgz .
tar xvf l_mkl_11.0.4.183.tgz
cd l_mkl_11.0.4.183
sudo sh install.sh
Step no: 1 of 7 | Welcome -------------------------------------------------------------------------------- Welcome to the Intel(R) Math Kernel Library 11.0 Update 4 for Linux* installation program. The Flagship of HPC Math Libraries. This library contains highly optimized, extensively threaded, mathematical functions for engineering, scientific, and financial applications that require maximum performance. -------------------------------------------------------------------------------- You will complete the steps below during this installation: Step 1 : Welcome Step 2 : License Step 3 : Activation Step 4 : Intel(R) Software Improvement Program Step 5 : Options Step 6 : Installation Step 7 : Complete -------------------------------------------------------------------------------- Step no: 1 of 7 | Options > Missing Optional Pre-requisite(s) -------------------------------------------------------------------------------- There are one or more optional unresolved issues. It is highly recommended to resolve them all before you continue the installation. You can fix them without exiting from the installation and re-check. Or you can quit from the installation, fix them and run the installation again. -------------------------------------------------------------------------------- Missing optional pre-requisites -- unsupported OS -------------------------------------------------------------------------------- 1. Skip missing optional pre-requisites [default] 2. Show the detailed info about issue(s) 3. Re-check the pre-requisites h. Help b. Back to the previous menu q. Quit -------------------------------------------------------------------------------- Please type a selection or press "Enter" to accept default choice [1]: 1 [..] 13. THIRD PARTY PROGRAMS. The Materials may include third party programs or materials that are governed by the third party's license terms, including without limitation, open source software. The license terms associated with such third party programs or materials govern your use of same, and Intel is not liable for such third party programs or materials. * Other names and brands may be claimed as the property of others -------------------------------------------------------------------------------- Do you agree to be bound by the terms and conditions of this license agreement? Type "accept" to continue or "decline" to back to the previous menu: accept Step no: 3 of 7 | Activation -------------------------------------------------------------------------------- If you have purchased this product and have the serial number and a connection to the internet you can choose to activate the product at this time. Activation is a secure and anonymous one-time process that verifies your software licensing rights to use the product. Alternatively, you can choose to evaluate the product or defer activation by choosing the evaluate option. Evaluation software will time out in about one month. Also you can use license file, license manager, or remote activation if the system you are installing on does not have internet access activation options. -------------------------------------------------------------------------------- 1. I want to activate my product using a serial number [default] 2. I want to evaluate my product or activate later 3. I want to activate either remotely, or by using a license file, or by using a license manager h. Help b. Back to the previous menu q. Quit -------------------------------------------------------------------------------- Please type a selection or press "Enter" to accept default choice [1]: 1 Note: Press "Enter" key to back to the previous menu. Please type your serial number (the format is XXXX-XXXXXXXX): -------------------------------------------------------------------------------- Activation completed successfully. -------------------------------------------------------------------------------- Press "Enter" key to continue: Step no: 4 of 7 | Intel(R) Software Improvement Program -------------------------------------------------------------------------------- Help improve your experience with Intel(R) software Participate in the design of future Intel software. Select 'Yes' to give us permission to learn about how you use your Intel software and we will do the rest. - No Personal contact information is collected - There are no surveys or additional follow-up emails by opting in - You can stop participating at any time Learn more about Intel(R) Software Improvement Program http://software.intel.com/en-us/articles/software-improvement-program With your permission, Intel may automatically receive anonymous information about how you use your current and future Intel software. -------------------------------------------------------------------------------- 1. Yes, I am willing to participate and improve Intel software. (Recommended) 2. No, I don't want to participate in the Intel(R) Software Improvement Program at this time. b. Back to the previous menu q. Quit -------------------------------------------------------------------------------- Please type a selection: 2 Step no: 5 of 7 | Options -------------------------------------------------------------------------------- You are now ready to begin installation. You can use all default installation settings by simply choosing the "Start installation Now" option or you can customize these settings by selecting any of the change options given below first. You can view a summary of the settings by selecting "Show pre-install summary". -------------------------------------------------------------------------------- 1. Start installation Now 2. Change install directory [ /opt/intel/composer_xe_2013.4.183 ] 3. Change components to install [ All ] 4. Show pre-install summary h. Help b. Back to the previous menu q. Quit -------------------------------------------------------------------------------- Please type a selection or press "Enter" to accept default choice [1]: 1 Step no: 6 of 7 | Installation -------------------------------------------------------------------------------- Each component will be installed individually. If you cancel the installation, components that have been completely installed will remain on your system. This installation may take several minutes, depending on your system and the options you selected. -------------------------------------------------------------------------------- Installing Intel Math Kernel Library 11.0 Update 4 on IA-32 component... done -------------------------------------------------------------------------------- Installing Intel Math Kernel Library 11.0 Update 4 on Intel(R) 64 component... done -------------------------------------------------------------------------------- Finalizing installation... done -------------------------------------------------------------------------------- Press "Enter" key to continue Step no: 7 of 7 | Complete -------------------------------------------------------------------------------- Thank you for installing and for using the Intel(R) Math Kernel Library 11.0 Update 4 for Linux*. Support services start from the time you install or activate your product, so please create your support account now in order to take full advantage of your product purchase. Your Subscription Service support account provides access to free product updates interactive issue management, technical support, sample code, and documentation. To create your support account, please visit the Subscription Services web site https://registrationcenter.intel.com/RegCenter/registerexpress.aspx?clientsn=NBJ N-87S2RM3P To get started using Intel(R) Math Kernel Library 11.0 Update 4 located in /opt/intel/composer_xe_2013.4.183 visit: install-dir/Documentation/en_US/mkl/get_started.html. -------------------------------------------------------------------------------- q. Quit [default] -------------------------------------------------------------------------------- Please type a selection or press "Enter" to accept default choice [q]:

3. Usage
I'll demonstrate using nwchem. I've repeated the build instructions over and over on this blog, but here it goes again:

sudo apt-get install build-essential gfortran python2.7-dev libopenmpi-dev openmpi-bin
sudo mkdir /opt/nwchem -p
sudo chown $USER:$USER /opt/nwchem
cd /opt/nwchem
wget http://www.nwchem-sw.org/download.php?f=Nwchem-6.3.revision1-src.2013-05-28.tar.gz -O Nwchem-6.3.revision1-src.2013-05-28.tar.gz
tar xvf Nwchem-6.3.revision1-src.2013-05-28.tar.gz
cd nwchem-6.3-src.2013-05-28/
export NWCHEM_TOP=`pwd`
export LARGE_FILES=TRUE
export TCGRSH=/usr/bin/ssh
export NWCHEM_TOP=`pwd`
export NWCHEM_TARGET=LINUX64
export NWCHEM_MODULES="all python"
export PYTHONVERSION=2.7
export PYTHONHOME=/usr

export BLASOPT="-L/opt/intel/composer_xe_2013.4.183/mkl/lib/intel64/ -lmkl_sequential -lmkl_core -lmkl_intel_ilp64"
export LIBRARY_PATH="$LIBRARY_PATH:/usr/lib/openmpi/lib:/opt/intel/composer_xe_2013.4.183/mkl/lib/intel64/"

export USE_MPI=y
export USE_MPIF=y
export USE_MPIF4=y
export MPI_LOC=/usr/lib/openmpi/lib
export MPI_INCLUDE=/usr/lib/openmpi/include
export LIBMPI="-lmpi -lopen-rte -lopen-pal -ldl -lmpi_f77 -lpthread"
export ARMCI_NETWORK=SOCKETS

cd $NWCHEM_TOP/src

make clean
make nwchem_config
make FC=gfortran 1> make.log 2>make.err

cd $NWCHEM_TOP/contrib
export FC=gfortran
./getmem.nwchem


I had a look at http://www.ichec.ie/support/tutorials/mkl.pdf to better understand what the different libraries do. Still not quite clear though, so I might not be getting optimal performance.


4. Performance
I basically repeated the test job shown here: http://verahill.blogspot.com.au/2013/05/430-briefly-crude-comparison-of.html

The machine has 16 gb ram and an intel i5-2400 cpu (4 cores)

ACML: Total times cpu: 1644.3s wall: 1655.5s
MKL:    Total times  cpu:     1550.0s     wall:     1563.8s

It certainly looks good. While 100 seconds in less than two minutes, and 1550 is 94 %  of 1644, it's a difference of about one day per fortnight, which doesn't make much of a difference on a small single-user cluster, it would make a bit of cumulative difference on our 1400 core multi-user cluster.

24 June 2013

462. Olex2 (1.1) on Debian

Olex2 is an open-source program for solving and refining crystal structures. I am not a crystallographer -- I have never (successfully= publication quality output) solved a crystal structure in my life -- and I have no idea whether Olex2 is an alternative or a complement to SHELX.

What I do now is that Olex2 is more of a point-and-click type program with a snazzy GUI, and so it would seem that the learning curve may be shallower than that of SHELX. However, as always you must always be cautious when doing science -- just because it's easy to do, doesn't mean that you're doing it right (I'm looking at you, biologists/physicians/social scientists using statistical software).

But maybe Olex2 makes getting started just easy enough that you can slowly work your way towards actually slowly get to the point where you know and understand what you are doing?

That's my hope anyway.

Quick note: there's no source code on the http://www.olex2.org/ website. However, there's a sourceforge SVN repository: http://sourceforge.net/projects/olex2/

Ideally I should show you how to download and compile and make a .deb package from the source, but sadly I haven't managed to figure it out (I'll add troubleshooting data to this post later). I even tried using the deb rules by http://jupiter.plymouth.edu/~jsduncan/software/olex2.php (again, I might add troubleshooting data later)

Anyway, using the pre-built binaries is easy enough. Only the older version 1.1 seems to be working ok on Debian Wheezy.

Installation:
Note that I've generated a list of dependencies the lazy way -- by looking at the ldd output.

Anyway.

You'll need a raft of libraries, including libGL. You can get this from different sources depending on your graphics card (or lack thereof). Install libgl1-mesa-glx or libgl1-fglrx-glx (ati) or libgl1-nvidia-glx (nvidia). Then continue.

Again, note that you may want to change the versions of the packages below that you install. Anyway, here's a list that may work:

sudo apt-get install libpng3 libglu1-mesa libgtk2.0-0 libatk1.0-0 libgdk-pixbuf2.0-0 libpango1.0-0 libglib2.0-0 libfreetype6 libxrender1 libfontconfig1 libx11-6 libxext6 libpng12-0 libxinerama1 libxxf86vm1 libsm6 lib32z1 zlib1g libc6 libpython2.6 libstdc++6 libgcc1 libxcomposite1 libxdamage1 libxfixes3 libcairo2 libxi6 libxrandr2 libxcursor1 libffi5 libexpat1 libpcre3 libxcb1 libice6 libuuid1 libssl1.0.0 libpixman-1-0 libxcb-shm0 libxcb-render0 libselinux1 libxau6 libxdmcp6

Get the compiled binary:
cd ~
wget http://www.olex2.org/olex2-distro/1.1/olex2-linux64.zip
unzip olex2-linux64.zip
cd ~/olex2
sed -i 's./work/distro/olex2-new.$HOME/olex2.g' start

Create the file ~/.local/share/applications/olex2.desktop
[Desktop Entry] Name=Olex2 GenericName=Olex2 Comment=Software for refinement of crystal structures Exec=sh /home/verahill/olex2/start Terminal=false Type=Application Categories=Science Version=1.1
You're now ready to use Olex2. If you launch it by hand, use the start script. Otherwise just launch is from your desktop.



Olex2 is very pretty, but  feels a bit incomplete and buggy though (again, I don't know if these are real issues or not -- and I'm no expert). The issue is that the program seems to throw errors every now and again. For example when clicking on electron density map in the Solve window:
Map sigma 0.000 : CalcFourier {diff=}{r=0.1}{m=} N4esdl25TInvalidArgumentExceptionE mask size at [xlib/fracmask.cpp(Init):22] calcFourier -diff -r=0.1 -m




Problem with version 1.2:
Start Olex2. 1.2 takes a lot longer than 1.1 to start, but that's fine. Click on Tutorials. Click on Maps and Masks. CLick Next five times (the step where you actually create the mask). It triggers the following error:
An error occured running the function/macro next_demo_item Traceback (most recent call last): File "olexFunctions.py", line 480, in func retVal = f(*args, **kwds) File "/home/verahill/olex2_1.2/etc/scripts/Tutorials.py", line 138, in next_demo_item self.run_demo_item() File "/home/verahill/olex2_1.2/etc/scripts/Tutorials.py", line 318, in run_demo_item flash_gui_control(control) File "gui/tools/__init__.py", line 123, in flash_gui_control OV.Refresh() File "guiFunctions.py", line 61, in Refresh olx.Refresh() File "/home/verahill/.olex2/data/4f11cd71424c9e8484c8c4f91644e3b6/olx/__init__.py", line 1319, in Refresh for arg in args: RuntimeError: [repository/pyext.cpp(runOlexFunctionEx):354]: Function 'html.SetImage' failed: wrong html object name: 'IMG_H3-H3-MASKS'Key variable values: args = () kwds = {} al = []

Problems with compiling:
sudo apt-get install libwxgtk2.8-dev
mkdir ~/tmp/svn_co
cd ~/tmp/svn_co
svn checkout svn://svn.code.sf.net/p/olex2/code/branches/1.2 olex2-1.2
cd olex2-1.2/
make 
/home/verahill/tmp/svn_co/olex2-1.2/gxlib/gxapp.cpp: In member function ‘esdl::TUndoData* gxlib::TGXApp::Name(gxlib::TXAtom&, const olxstr&, bool)’: /home/verahill/tmp/svn_co/olex2-1.2/gxlib/gxapp.cpp:2003:73: error: no matching function for call to ‘gxlib::TGXApp::SynchroniseBonds(gxlib::TXAtomPList)’ /home/verahill/tmp/svn_co/olex2-1.2/gxlib/gxapp.cpp:2003:73: note: candidate is: [..] /home/verahill/tmp/svn_co/olex2-1.2/gxlib/gxapp.cpp: In member function ‘esdl::TUndoData* gxlib::TGXApp::Name(const olxstr&, const olxstr&, bool, bool)’: /home/verahill/tmp/svn_co/olex2-1.2/gxlib/gxapp.cpp:2136:71: error: no matching function for call to ‘gxlib::TGXApp::SynchroniseBonds(gxlib::TXAtomPList)’ /home/verahill/tmp/svn_co/olex2-1.2/gxlib/gxapp.cpp:2136:71: note: candidate is: [..[ make[1]: *** [/home/verahill/tmp/svn_co/olex2-1.2/obj/gxapp.s] Error 1 make[1]: *** Waiting for unfinished jobs.... make[1]: *** [/home/verahill/tmp/svn_co/olex2-1.2/obj/gxmacro.s] Error 1 make[1]: Leaving directory `/home/verahill/tmp/svn_co/olex2-1.2' make: *** [all] Error 2


There's a CMakeLists.txt file, but when doing e.g.
mkdir build_olex
cd build_olex/
cmake ../olex2-1.2

I get
CMake Error: Error in cmake code at /home/verahill/tmp/svn_co/olex2-1.2/CMakeLists.txt:49: Parse error. Expected a command name, got unquoted argument with text "${olex2_html_SRCS}". -- Configuring incomplete, errors occurred!

12 June 2013

450. Tor and Chrome on Debian

Note:
* For the Tor bundle see http://verahill.blogspot.com.au/2013/05/408-briefly-tor-on-debian-quick-option.html
* For securing your dropbox, see http://verahill.blogspot.com.au/2013/04/398-securing-your-dropbox.html
* For encrypting your filesystem with encfs, see http://verahill.blogspot.com.au/2013/05/408-briefly-tor-on-debian-quick-option.html
* For one-time passwords (OTPW), see http://verahill.blogspot.com.au/2013/04/385-otpw-connecting-from-insecure.html
* For encryption in general using PGP/GPG, OTR, SRTP for chat, email, voice and video, see http://verahill.blogspot.com.au/2013/04/381-encrypting-chat-voice-video.html
* For truecrypt with dropbox, see http://verahill.blogspot.com.au/2012/04/using-truecrypt-with-dropbox.html

Post begins:
I think it's fair to say that online privacy is in the spotlight again, temporarily,  in particular if you are not living in the US. After all, the rest of the world is offered no protection from US agencies.

There are two levels of snooping that (can) go on:
Case 1:  outright intercept of communications
In this case your emails are read, your browsing data is intercepted and your phone conversations tapped. This is the most intrusive form, and I think even in the US a warrant is required for the intercept of this type of data (whether that's too easy of difficult to get is another question entirely).

Case 2: mining of 'meta-data'
In this case data such as recipient/sender of emails, URLs that you've been visiting, and whom you have been calling/called by are collected. In addition, e.g. cell phone tower records can be collected to track your whereabouts 24/7.

While the contents of your conversations isn't known, your entire social and professional life can be charted.
As far as I understand this is what NSA has been engaging in. Likewise, knowing exactly where you are at any given point in time, a pretty detailed picture of your life can be painted.

Begin Rant
I don't have anything to hide, but I am not too keen on the government having better records of my life than I do myself. And I should be the one deciding what to share as long as the presumption of innocence holds.

Also, we're making the presumption that the government is benign, and as has been shown repeatedly, it isn't always. That goes for the US government, the UK government and just about any bloody imaginable government, and for a simple reason: the government is made up of people. In particular people who are keen on 'leading' i.e. controlling others. Even a benign despot is a despot.

There's no use being naive -- in either direction. There are legitimate reasons for clandestine organisations wanting to mine data, and there are legitimate reasons for why we should not give them a carte blanche.

Whether you use PGP/GPG or not won't affect the mining of meta-data. Nor will OTR, although it might in theory give you a somewhat better level of deniability (but not really).

Using PGP/GPG, OTR and encryption of data in general will only protect the content of your conversations, not the fact that they occurred. Not that it's easy getting people to start using encryption of their email, especially not since hotmail and gmail provided the final push into getting people to do all their email processing in the browser rather than using a more capable email client. Obviously Google would not be pleased if all communication was PGP encrypted, since this would create issues with targeted ads.

Finally, what really irks me is the fact that because John Doe won't use encryption -- or learn how to do it -- I also cannot use it. Instead we have to play according to the rules of the least technologically informed.
End Rant

Anyway. There are a few things you can do -- at least to make you feel better. Whether they have any real impact on your privacy depends on what other sources of information leakage there are in your life.

The simplest thing you can do is to do all your browsing anonymously, including setting up and checking your email. And the easiest way to do that is by using Tor.

It's easy enough to use the Tor Bundle, e.g. http://verahill.blogspot.com.au/2013/05/408-briefly-tor-on-debian-quick-option.html

However, I for some forsaken reason like using Chrome.

To set up Proxy SwitchySharp I'm following this post:
http://lifehacker.com/5614732/create-a-tor-button-in-chrome-for-on+demand-anonymous-browsing

NOTE: there are many layers to managing your privacy, and you're only as anonymous as your worst habits allow you to be. I'm a pessimist -- I think it is virtually impossible to protect yourself against a determined adversary. However, trying won't hurt.


Step 0. Block cookies by default and install an ad blocker

Cookies
Pretending to be anonymous won't help if you give the game away by exposing cookies that you acquired while surfing without Tor.

You'll be surprised how many websites require you to accept cookies -- however, it's up to you whether you want to put up with that. I only allow cookies with services that I've signed up to and that I trust. I refuse to allow in particular commercial sites to require cookies for me to simply visit.

In Chrome, go to Settings, Content Settings, and check:
* Block sites from setting any data
* Block third-party cookies and site data
* Clear cookies and other site and plug-in data when I close my browser

Disable:
* Allow local data to be set

You may want to restrict e.g. image loading, javascript, pop-ups, plugins etc. as well. It's down to you to weight inconvenience vs privacy.

Set Cookie and Site Data exceptions manually, and make sure to distinguish between Session Only and Allow.

Ads
Also, install e.g. simple adblock:
https://chrome.google.com/webstore/search/simple%20adblock



Step 1. Install the HTTPS everywhere extension
https://chrome.google.com/webstore/search/https%20everywhere?hl=en-GB



Step 2. Install Proxy SwitchySharp
https://chrome.google.com/webstore/search/proxy%20switchysharp?hl=en-GB

Set up a profile called Tor to use SOCK 5 with 127.0.0.1:9050
Go to the General Tab and enable Quick Switch.
Make sure to drag both Tor and Direct Connection into the Quick Switch field.



Step 3. Install Tor and Vidalia
Add the following to your /etc/apt/sources.list
deb http://deb.torproject.org/torproject.org wheezy main

Then do
sudo apt-get update
sudo apt-get install deb.torproject.org-keyring
sudo apt-get update
sudo apt-get install vidalia

Tor should run in the background whether you start Vidalia or not.

Step 4. Prevent DNS leaks:
[for fun, do
sudo apt-get install tcpdump
sudo tcpdump -pni eth0 'port domain'
before turning off prefetching. ]

To make sure that your DNS requests aren't being read (i.e. providing meta-data to your ISP), you will need to turn of DNS pre-fetching in Chrome.

Google is sneaky about it though -- to turn off prefetching go to Settings/Under the Bonnet and uncheck "Predict network actions to improve page load performance".

[If you set up tcpdump before you'll see how suddenly the IPs and URLs stop streaming by.]


Step 5. Start Tor/Vidalia
You don't seem to be able to launch Vidalia from the terminal, so launch Vidalia from within e.g. gnome.
In fact, you probably don't have launch vidalia as Tor should be run in the background.
Then open Chrome and navigate to e.g. whatsmyip.org or ipchicken.com:


You can turn on and off the proxy by clicking on the icon in the top right corner.

Step 6. Enable private browsing:
You don't want to risk one website being able to see what another website left behind. It shouldn't happen, but it has happened in the past.

Anyway, it's easy: open an Incognito window (ctrl + shift + N).


Done.
As far as I can tell this should give you some privacy. However, the question is how effective this is in the long run since it's difficult to maintain enough discipline to prevent any information leakage to occur.