Install MPICH2-1.3a1 for CPU affinity

I tested MPICH2-1.3a1, which uses hydra as the default process manager.
The test env. is the IBM H22 Intel Nehalem blades.
>>./configure --prefix=/home/mydir/mpich2-1.3a
>>make
>>make install

No special configuration option is required. (In 1.2.1, we need --with-pm=hydra)
Setup the 'myhost' file as
intel01:1 binding=user:0
intel02:1 binding=user:0
intel03:1 binding=user:0
intel04:1 binding=user:0

>>LD_LIBRARY_PATH=../socIntel/goto:$LD_LIBRARY_PATH mpiexec -f myhost -n 4 ./main 62 62 tests/
I have to say that there is no process migration among cpus. However,
I cannot say this installation really has cpu affinity, because when I
use
-binding user:2,4 processes are not really binded to cpu2, and 4. Even
if I use intel01:4 binding=user:4,5,6,7. I see cpus 0,1,2,3, are busy.

Nevertheless, this is the best result I can get from the Bluegrit. On
it, the OpenMPI can do cpu affinity only on one node, because of TCP
firewall. Besides, MVAPICH2 cannot really support cpu affinity, since
there is no IB, iWARP, etc. Last, early version of MPICH does not
support core binding. It is really hard to get core mapping as a
non-root. I don't know why the admin are reluctant to install these
for the users. I wasted a lot of time on that!

Install MVAPICH with HWLOC as a non-root

Here is descriptions of installing MVAPICH with hwloc as a non-root.
With hwloc, we can make mvapich support cpu affinity.
Download hwloc from http://www.open-mpi.org/software/hwloc
Download mvapich2, and untar both to hwloc-0.9.3 and mvapich-1.4.1
>>mkdir hwloc0.9.3
>>mkdir mvapich1.4.1
>>cd hwloc-0.9.3
>>./configure --prefix=/home/username/hwloc0.9.3
>>make
>>make install
>>cd ../mvapich-1.4.1
>>export LDFLAGS='-L/home/username/hwloc0.9.3/lib'
>>export CPPFLAGS='-I/home/username/hwloc0.9.3/include'
>>./configure --with-hwloc --with-rdma=gen2 --prefix=/home/username/mvapich1.4.1 --disable-f90
>>make
>>make install
After that, add the mvapich1.4.1 path to .bashrc

Plot data file with PGFPlot

Tikz and pgfplot can plot data file in Tex.

\begin{figure}

\begin{center}

\begin{tikzpicture}[scale=0.3]

\begin{axis}[xlabel=nstep, ylabel=WaveHeight]

\addplot[mark=none] file {case14.data};   % Here is the data file

\addplot[color=green,mark=none] file {case14xb.data};

\legend{True, Predicted};

\end{axis}

\end{tikzpicture}

\begin{tikzpicture}[scale=0.3]

\begin{axis}[xlabel=nstep, ylabel=error(2-norm)]

\addplot[mark=none] file {case14err.data};

\end{axis}

\end{tikzpicture}

\end{center}

\caption{(a)True and predicted wave height at the domain center; (b)2-norm of numerical error on the whole domain}

\end{figure}

The data file's format can be multiple columns separated by space. Each column is for one variable. Such as
# data file
0.00001 2.123243
0.00003 3.123452
......

Sometimes, Matlab's fprintf function may help to generate such data file.

For example:

a=load('err14.dat');

fid = fopen('case14err.data','w');

b=1/650:6.5/650:6.5;

y=[b ; a];

fprintf(fid,'%1.5f %1.5f\n', y);

fclose(fid);

A script to create shared lib

   A good script to create shared lib

   From: http://anaturb.net/create_shared.htm

   1. With root privileges: create directories /usr/local/lib and /usr/local/include (if not exist)

   2. add pass for /usr/local/lib in file /etc/ld.so.conf (if not exists)

   3. run script: create_shared_lib.sh (text below)

      f.e. you have money.c file that is prepared for creating library "money".

      $sh create_shared_lib money

   4. copy your money.h to /usr/local/include

   5. to use your library:

      include money.h in global area of your program (#include &ltmoney.h>)

      gcc -Wall -g -o my_prog.exe my_prog.c -I/usr/local/include -lmoney

   1. Without root privileges: create directory lib and include in your $HOME directory

   2. change script's line 'mv "$libname".* /usr/local/lib/' for 'mv "$libname".* ~/lib/'

   3. comment line 'ldconfig &'

   4. run script (as I explained above)

   5. copy your header file into ~/include directory

   6. to use your library:

      include money.h in global area of your program (#include &ltmoney.h>)

      gcc -Wall -g -o my_prog.exe my_prog.c -I$HOME/include -L$HOME/lib -lmoney

create_shared_lib.sh

#!/bin/bash

ARG=1

BADARG=65

if [ "$#" -ne "$ARG" ] 

then

echo -e "\t\tUsages: `basename $0` name_of_shared_library"

exit $BADARG

fi

libname=lib"$1"

##########################################################################

# uncomment next 2 lines and comment 3-4 lines if you create C++ library #

##########################################################################

#g++ -fPIC -c "$1".cpp          

#g++ -shared -Wl,-soname,"$libname".so.1 -o "$libname".so.1.0 "$1".o

gcc -fPIC -c "$1".c

gcc -shared -Wl,-soname,"$libname".so.1 -o "$libname".so.1.0 "$1".o

ln -s "$libname".so.1.0 "$libname".so.1

ln -s "$libname".so.1 "$libname".so

echo -e "\t\tSHARED LIBRARY $libname IS CREATED"

echo -e "\t\t==================================\n"

echo

mv "$libname".* /usr/local/lib/

echo -e "\t\tldconfig is working"

ldconfig &

exit 0

Node plot and position in Tikz

It is difficult to create a neat, accurate and fancy scientific figure. Tikz/pgf, pstricks, metapost are not easy to learn, nor for me to have a full knowledge of their capabilities. On the other hand, gimp or photoshop is not suitable to produce figures from calculated data. I hope there will be a very handy tool that can help me using Tikz, or metapost easily. But I tested several java based front end. None of them would be a choice.

Here is an example of plotting nodes and point them to each other with arrows, with position determination. It is mainly about '\node', '\draw' and their decoration. It is easy to understand how it is produced, hence, I put no comments here.

-----------------------

\begin{tikzpicture}

\draw[dash pattern=on 2pt off 3pt on 4pt off 4pt](2,.2) -- (2,3.2);

\draw[dash pattern=on 2pt off 3pt on 4pt off 4pt](4,.2) -- (4,3.2);

\draw[dash pattern=on 2pt off 3pt on 4pt off 4pt](6,.2) -- (6,3.2);

\node at (2,2.9) [fill=blue!50,draw,circle, drop shadow,

label=left:\tiny{$x_k^b$}] (n1){};

\node at (2,1.9) [fill=red!50,draw,circle, drop shadow,label=left:\tiny{$x_k^a$}] (n2){};

\node at (2,.9) [fill=green!50,draw,circle, drop shadow,label=left:\tiny{$y_k^o$}] (n3){};

\node at (4,1.9) [fill=red!50,draw,circle, drop shadow,label=left:\tiny{$x_{k+1}^a$}] (n4){};

\node at (4,2.9) [fill=blue!50,draw,circle, drop shadow,label=left:\tiny{$x_{k+1}^b$}] (n5){};

\node at (4,.9) [fill=green!50,draw,circle, drop shadow,label=left:\tiny{$y_{k+1}^o$}] (n6){};

\node at (6,1.9) [fill=red!50,draw,circle, drop shadow,label=left:\tiny{$x_{k+2}^a$}] (n7){};

\node at (6,.9) [fill=green!50,draw,circle, drop shadow,label=left:\tiny{$y_{k+2}^o$}] (n8){};

\node at (6,2.9) [fill=blue!50,draw,circle, drop shadow,label=left:\tiny{$x_{k+2}^b$}] (n9){};

\node at (8,2.9) [fill=blue!20, anchor=base](n10){$\cdots$};

\draw[->, line width=1pt] (1,0.2) -- (10,0.2) node[right]{time};

\node at (2,0.2)[below]{\tiny{$k$}};

\node at (4,0.2)[below]{\tiny{$k+1$}};

\node at (6,0.2)[below]{\tiny{$k+2$}};

\end{tikzpicture}

\begin{tikzpicture}[overlay]

\path[->] (n2) edge [bend right] (n5);

\path[->] (n4) edge [bend right] (n9);

\path[->] (n7) edge [bend right] (n10);

\end{tikzpicture}

\vspace{7mm}

\begin{itemize}

\item \tikz\node [fill=blue!50,draw,circle]{}; Background state $x^b$

\item \tikz\node [fill=red!50,draw,circle]{}; Analysis state $x^a$

\item \tikz\node [fill=green!50,draw,circle]{}; Observation $y^o$

\end{itemize}

Install TeXLive on Ubuntu

To install texlive2008

1. sudo mount -o loop textlive2008-20080822.iso /mnt

2. cd /mnt

3. ./install-tl -gui

If there is error about "Cannot load Tk, maybe something is missing" or "perl/Tk unusable, cannot create main windows.",

Follow its suggestion, visit  http://tug.org/texlive/distro.html#perltk

4. sudo apt-get install perl-tk

5. sudo ./install-tl -gui

Make wise choice, default dir /usr/local/texlive/, click Install TeX Live

Download TexMaker, Install it. In configure, add correct path for each executable file, 

such as /usr/local/texlive/2008/bin/i386-linux/latex. In this way, those programs can be found.

6. sudo vi ~/.bashrc  and/or sudo vi /root/.bashrc

Add one line at last

export PATH=$PATH:/usr/local/texlive/2008/bin/i386-linux

To launch the tlmgr, 

7. sudo su
8. tlmgr -gui

If the path is not set in the bashrc, there is error when you launch the tlmgr, even if you execute it from its directory. The error are like:

Can't exec "kpsewhich"

Can't locate TeXLive/TLPOBJ.pm in @INC

So just ass the path of TexLive installation directory.

It is better to install texlive to somewhere does not require root. i.e. /home/yourname/textlive . But the 7th step solves this problem.

mpdtrace and using multiple nodes to run mpi

In a cluster, which the user may need to launch the mpd manually, here are descriptions of to-dos. The situation that one may consider to do so, is when you launch a executable program on multiple nodes, however, there is only the local node is used. Or you see some error like MPI connection or communication error. It is the time to check if all the nodes listed in the hosts file are able to communicate with each other. Error like: mpiexec: unable to start all procs; may have invalid machine names remaining specified hosts.

Following description works for MPICH2, using mpiexec ot mpirun. That is how I tested. 

On the node where you launch the program, type

>> mpdtrace -l

It gives you  <node name>_<port>(IP)

blade50_51094 (IP **)

blade49_56382 (IP **)

blade47_35763 (IP **)

blade48_49526 (IP **)

blade51_53029 (IP **) 

If not all nodes are listed here, for example, if blade46 is in the hosts file, and is available. ssh to blade46, type

>>mpd -h blade50 -p 51094 &

If you want to start more mpd

>>mpd -h blade50 -p 51094 -n &

Then the blade46 can be used.

To clean up mpd daemon, use mpdcleanup

Besides, if you want to launch m consecutive ranks on the same node, use mpd --ncpus=m
For example:
mpd --ncpus=2 &
or
mpd --ncpus=2 -h blade50 -p 51094 &


QUOTE"
If an mpd is started with the --ncpus option, then when it is its turn to start a process, it will start several application processes rather than just one before handing off the task of starting more processes to the next mpd in the ring. For example, if the mpd is started with

mpd --ncpus=4

then it will start as many as four application processes, with consecutive ranks, when it is its turn to start processes. This option is for use in clusters of SMP's, when the user would like consecutive ranks to appear on the same machine. (In the default case, the same number of processes might well run on the machine, but their ranks would be different.) (A feature of the --ncpus=[n] argument is that it has the above effect only until all of the mpd's have started n processes at a time once; afterwards each mpd starts one process at a time. This is in order to balance the number of processes per machine to the extent possible.)
"END of QUOTE

Animate image files

With a sequence of image files, we can code them into a video file. Suppose that they are named beginning from 1, use

ffmpeg -qscale 1 -r 20 -b 96000 -i %08d.png animate.mp4

If the file names do not start from 1, we can use a script do rename them in batch.

#!/bin/bash

d=1

for fname in *.png

do

  mv $fname `printf "%08d.png" $d`

  d=$(($d+1))

done

To enable mp3 etc.
>>sudo apt-get install ffmpeg libavcodec-extra-52
One example of extracting audio from mp4:
>>ffmpeg -ss 00:05:00:00 -t 00:02:00:00 -i input.mp4 -acodec libmp3lame -ab 128k output.mp3
ss: time offset from beginning of input in hh:mm:ss:frames.
t: duration of encode
ab: audio bitrate

GotoBlas and Lapack_wrapper

Gotoblas + lapack + lapack_wrapper
Rod Heylen has a good lapack wrapper, which can be used with ATLAS.
Find it here (http://itf.fys.kuleuven.be/~rob/computer/lapack_wrapper/index.html)
However, ATLAS is very hard and time consuming (>6hr for me) to
install. While, GotoBlas is said to have better performance than ATLAS
does, and is easy to install.
I tested to replace ATLAS with GotoBlas, and compile lapack_wrapper
examples successfully, on Ubuntu9.10, 32bit intel Core2DUO 2.2GHz,
gcc4.4. Of course, we need to change the lapack_wrapper a little. This
is because of some functions and variable declarations are
incompatible in ATLAS and GotoBlas.
Download GotoBLAS2-1.10.tar.gz.
Untar it to a directory, i.e. /home/shiming/GOTO
>>tar xvfz GotoBLAS2-1.10.tar.gz
cd to that directory
>> make
>>sudo cp libgoto2_* /usr/lib
>>sudo mkdir /usr/local/include/goto
>>sudo cp cblas.h /usr/local/include/goto
Download CLAPACK3.1.1, untar it, and copy CLAPACK-3.1.1/INCLUDE/f2c.h
and CLAPACK-3.1.1/INCLUDE/clapack.h to /usr/local/include/goto
>>cd /usr/local/include/goto
>>sudo chmod 755 *
>>cd /usr/lib
>>sudo ln -s libgoto2_*.a libgoto.a
>>sudo ln -s libgoto2_*.so libgoto.so
Change a little in lapack_wrapper.c. I do not list them here. To see
the difference, just download the original one, and use 'diff'
The changed version is available here
(http://sites.google.com/site/chuanese/lapack_wrapper_for_GotoBlas.tar?attredirects=0)

Also edit the cblas.h by adding one line
#define blasint int
And edit the f2c.h by changing the line on the 10th line to
typedef int integer;

To compile, use
>>gcc -o lapack_example lapack_example.c lapack_wrapper.c -lm -lgoto -lgfortran
>>./lapack_example
Everything works well.

In a machine that I have no 'root' right, there is a little difference to run the code.
For example, a PPC blade, RHEL, 64bit, gcc4.1.2. Just compile to GotoBlas and copy necessary header files and lib files to somewhere, i.e. /home/shiming1/goto, and chmod 755 *.

In the directory where lapack_example.c exists, compile with
>> gcc -o test lapack_example.c lapack_wrapper.c -lm -L../soc/goto -lgoto2 -lgfortran
Run it with
>>LD_LIBRARY_PATH=goto/:$LD_LIBRARY_PATH ./test

Here is a good article about using dynamic library
(http://tldp.org/HOWTO/Program-Library-HOWTO/shared-libraries.html)
--------------------------------------------------------------------
One example about cblas_dgemv. Suppose matrix A is a 3 by 4 double, stored as a 'cvec' (row major).
A=[1 3 2; 6 4 1; 2 8 7, 3 4 5].
x1=[1; 1; 1]
x2=[1; 1; 1; 1]
y =Ax1

cblas_dgemv(CblasRowMajor, CblasNoTrans, 4, 3, 1.0, A, 4, x1, 1, 0.0, y, 1);
If y= A'x2
cblas_dgemv(CblasRowMajor, CblasTrans, 3, 4, 1.0, A, 3, x2, 1, 0.0, y, 1);

A Matlab file saving format

Let me take this as a good start.
There is one special requirement for saving Matlab vectors into a file with certain format: updated vector Vec in each step should be saved like,
[1, 2, 3, 4, 5, 6]
[2, 3, 4, 5, 6, 7]
......
Which mean each vector is enclosed with [ and ], each entry separated by a comma. No comma after the last entry.
There could be more special requirements for the file format. Other commands' combination can accomplish more complicated task, such as "dlmwrite", "csvwrite".

A snippet Matlab code is:

fid = fopen('Hdata.dat','wb');
% Inside a loop body
% vec is updated
fprintf(fid, '[');
fprintf(fid,'%12.8f,', vec); % add a comma after each entry
fseek(fid,-1,0); % file pointer rewind one
fprintf(fid, ']\n'); % cover the last comma with a ]
% end of a loop body

fclose(fid);

There are only a few days off the deadline.

Google Blog Created

Google is powerful.