sphinx tips: add subsubsection into pdf compiled by make all-pdf-ja

When sphinx compiles latex files, it chooses cls file which written in conf.py.

# Grouping the document tree into LaTeX files. List of tuples
# (source start file, target name, title, author, documentclass [howto/manual]).
latex_documents =
   ('index', 'sample_document.tex', u'This is sample',
     u'Author Name', 'manual'),
]

If 'manual' is specified in conf.py, sphinx read sphinxmanual.cls when it compiles latex file. So, if you want to enable subsubsection, you should edit sphinxmanual.cls or other specified cls file. In my environment, original sphinxmanual.cls is located in /usr/local/lib/python2.6/dist-packages/Sphinx-1.0.7-py2.6.egg/sphinx/texinputs/sphinxmanual.cls. And you can enable subsubsection with editing this cls file like below.

before:
\setcounter{secnumdepth}{2}

after:
\setcounter{secnumdepth}{3}

enjoy.

sphinx tips: generate center-aligned table in pdf document

When we type 'make latex' command in sphinx root directory, sphinx generates set of latex files from rst files based on '/usr/share/pyshared/sphinx/writers/latex.py' which defines rules for generating latex files. If you do not like default behavior of latex generator, you can change it by modifying the code.

Sometimes I was annoyed by left-aligned table which is automatically created by using csv-table directive when creating a pdf document. So, I changed some part of latex.py. Below is difference between original latex.py and bit-of modified latex.py which can generate center-aligned table by using csv-table.

( ... just added '\begin{center}' and '\end{center}' command )

617c617
<             self.body.append('\n\\begin{threeparttable}\n'
---
>             self.body.append('\n\\begin{center}\n\\begin{threeparttable}\n'
664c664
<             self.body.append('\\end{threeparttable}\n\n')
---
>             self.body.append('\\end{threeparttable}\n\\end{center}\n\n')

postgres tips: enable plpgsql for implementing stored procedure

This entry is just a memo for me 8^)
If we want to implement stored procedure to any database, it is necessary to type below command.

createlang -d dbname plpgsql

python tips: draw beta distribution with matplotlib

import numpy as np
import numpy as np
import pylab as pl
import scipy.special as ss

def beta(a, b, mew):
    e1 = ss.gamma(a + b)
    e2 = ss.gamma(a)
    e3 = ss.gamma(b)
    e4 = mew ** (a - 1)
    e5 = (1 - mew) ** (b - 1)
    return (e1/(e2*e3)) * e4 * e5

def plot_beta(a, b):
    Ly = []
    Lx = []
    mews = np.mgrid[0:1:100j]
    for mew in mews:
        Lx.append(mew)
        Ly.append(beta(a, b, mew))
    pl.plot(Lx, Ly, label="a=%f, b=%f" %(a,b))

def main():
    plot_beta(0.1, 0.1)
    plot_beta(1, 1)
    plot_beta(2, 3)
    plot_beta(8, 4)
    pl.xlim(0.0, 1.0)
    pl.ylim(0.0, 3.0)
    pl.legend()
    pl.show()

if __name__ == "__main__":
    main()

postgres tips: calculate degree with stored procedure

-- User-Defined-Function for calculate degree
-- ------------------------------------------

-- This function calculates cos from vector(0,1)

-- .. code-block:: sql
--    :linenos:

CREATE OR REPLACE FUNCTION GET_COS
    (v2_x FLOAT, v2_y FLOAT)
RETURNS FLOAT AS
$$
DECLARE
    v1_x FLOAT;
    v1_y FLOAT;
    v1_v2 FLOAT;
    length_v1 FLOAT;
    length_v2 FLOAT;
    cos FLOAT;
BEGIN
    v1_x := 0.;
    v1_y := 1.;
    v1_v2 := (v1_x * v2_x) + (v1_y * v2_y);
    length_v1 := sqrt(power(v1_x,2) + power(v1_y,2));
    length_v2 := sqrt(power(v2_x,2) + power(v2_y,2));
    cos := v1_v2 / (length_v1 * length_v2);
    RETURN cos;
END;
$$ language plpgsql;

-- This function calculates sin from vector(0,1)

-- .. code-block:: sql
--    :linenos:

CREATE OR REPLACE FUNCTION GET_SIN
    (v2_x FLOAT, v2_y FLOAT)
RETURNS FLOAT AS
$$
DECLARE
    v1_x FLOAT;
    v1_y FLOAT;
    det FLOAT;
    length_v1 FLOAT;
    length_v2 FLOAT;
    sin FLOAT;
BEGIN
    v1_x := 0.;
    v1_y := 1.;
    det := (v1_x * v2_y) - (v1_y * v2_x);
    length_v1 := sqrt(power(v1_x,2) + power(v1_y,2));
    length_v2 := sqrt(power(v2_x,2) + power(v2_y,2));
    sin := det / (length_v1 * length_v2);
    RETURN sin;
END;
$$ language plpgsql;

-- This function calculates degree from vector(0,1)

-- .. code-block:: sql
--    :linenos:

CREATE OR REPLACE FUNCTION GET_DEGREE
    (x FLOAT, y FLOAT)
RETURNS FLOAT AS
$$
DECLARE
    sin FLOAT;
    cos FLOAT;
    degree FLOAT;
BEGIN
    sin := GET_SIN(x, y);
    cos := GET_COS(x, y);
    IF sin >= 0 AND cos >= 0 THEN
        degree := degrees(acos(cos));
    ELSEIF sin >= 0 AND cos < 0 THEN
        degree := degrees(acos(cos));
    ELSEIF sin < 0 AND cos < 0 THEN
        degree := 180. - degrees(asin(sin));
    ELSEIF sin < 0 AND cos >= 0 THEN
        degree := 360. + degrees(asin(sin));
    END IF;
    RETURN degree;
END;
$$ language plpgsql;

memo for my low-memory brain: calculate intersection from two lines in 2-D.

This is my memo for basic geometry-formula.
Now, I want to calculate intersection of two lines(Line1 and Line2).

Line1: passes point (x1=0, y1=9) and (x2=2, y2=5).
Line2: passes point (x3=1, y3=1) and (x4=3, y4=0).

Firstly, we can express these lines as following:
Line1:
Line2:

Secondly, we can get intersection (x,y) by solving below linear algebra:










Finally, using inverse matrix, we can easily calculate intersection:

sphinx tips: how to easily write a document for several scripts?

Today, I explain how to write a sphinx document for several scripts in an easy way. Assume that you are now writing postgres scripts and need to write document for them. Now, you have two postgres scripts named sample{1,2}.sql and these scripts are written like below.

-- Sample1.sql
-- -----------

-- This SQL does something!!
-- A definition of sample_table shown below.

-- =============== =========== ============
-- Column Name     Type        Description
-- =============== =========== ============
-- foo             TEXT        foo?
-- bar             TEXT        bar?
-- baz             FLOAT       baz?
-- =============== =========== ============

-- :NOTE: just return 10 records
-- :TODO: just return 11 records

-- .. code-block:: sql
--    :linenos:

SELECT
    *
FROM
    sample_table
LIMIT 10;

-- Sample2.sql
-- -----------

-- This SQL extract data from columns named foo, bar...

-- :NOTE: just return 10 records
-- :TODO: just return 11 records

-- .. code-block:: sql
--    :linenos:

SELECT
    foo,
    bar
FROM
    sample_table
LIMIT 11;

As you can see above, there are rst formatted comments in sql scripts. If you write this kind of comments in sql scripts, you can easily convert them to html/latex-styled pdf! Below shell scripts can convert above sql scripts into one rst document.

#!/bin/sh
# convert_sql2rst.sh

output_rst="sample.rst"

if [ -e ${output_rst} ]; then
    rm ${output_rst}
fi

echo "=======================" >> ${output_rst}
echo "This is sample chapter!" >> ${output_rst}
echo "=======================" >> ${output_rst}
echo "" >> ${output_rst}

for sql_file in `ls *.sql`
do
    echo "converting ${sql_file} to rst-formatted document..."
    sed ${sql_file} -e 's/^/    /g' | \
    sed -e 's/^    -- //g' >> ${output_rst}
    echo "" >> ${output_rst}
    echo "" >> ${output_rst}
done

Now, put all of above scripts into same sphinx-root directory and edit your index.rst for including ${output_rst}. Next, do this:

yaboo@maniac:~/Projects/SqlProject$ sh convert_sql2rst.sh 
converting sample.sql to rst-formatted document...
converting sample2.sql to rst-formatted document...
yaboo@maniac:~/Projects/SqlProject$ make html
sphinx-build -b html -d _build/doctrees   . _build/html
Making output directory...
Running Sphinx v1.0.7
loading pickled environment... not yet created
building [html]: targets for 2 source files that are out of date
updating environment: 2 added, 0 changed, 0 removed
reading sources... [100%] sample                                                
looking for now-outdated files... none found
pickling environment... done
checking consistency... done
preparing documents... done
writing output... [100%] sample                                                 
writing additional files... genindex search
copying static files... done
dumping search index... done
dumping object inventory... done
build succeeded.

Build finished. The HTML pages are in _build/html.

Finally, you can get document like this.

python tips: draw poisson pdf graph with matplotlib

this is a code for drawing a graph of poisson pdf with matplotlib.

#!/usr/bin/env python                                                           
#coding: utf-8                                                                  
                                                                                
import pylab as pl                                                              
import numpy as np                                                              
                                                                                
def get_factorial(n):                                                           
    if n == 0:                                                                  
        return 1                                                                
    else:                                                                       
        return n * get_factorial(n-1)                                           
                                                                                
def get_probability(p, n, r):                                                   
    mew = float(n) * p                                                          
    denom = (mew ** r) * np.exp(-1 * mew)                                       
    molec = get_factorial(r)                                                    
                                                                                
    return denom/float(molec)                                                   
                                                                                
def plot_poisson_distrib(p, n, lt, lab):                                        
    xl = [nr for nr in range(0, 20, 1)]                                         
    yl = [get_probability(p, n, nr) for nr in range(0, 20, 1)]                  
    pl.plot(xl, yl, lt, label=lab)                                              
                                                                                
def main():                                                                     
    p = 0.02                                                                    
                                                                                
    mew = 1.0                                                                   
    plot_poisson_distrib(p, (mew / p), 'b-', 'P(%f,%f)' %(mew,p))               
                                                                                
    mew = 2.5                                                                   
    plot_poisson_distrib(p, (mew / p), 'r-', 'P(%f,%f)' %(mew,p))               
                                                                                
    mew = 5.0                                                                   
    plot_poisson_distrib(p, (mew / p), 'g-', 'P(%f,%f)' %(mew,p))               
                                                                                
    pl.ylabel('p(x)')                                                           
    pl.xlabel('x')                                                              
    pl.legend()                                                                 
    pl.show()                                                                   
                                                                                
if __name__ == "__main__":                                                      
    main()

and output of above code is:

python tips: draw binary pdf graph with matplotlib

this is a code for drawing a graph of binary probability distribution with matplotlib.

#!/usr/bin/env python                                                           
#coding: utf-8                                                                  
                                                                                
import pylab as pl                                                              
                                                                                
def combination(n, r):                                                          
    if (r == 0) or (n == r):                                                    
        return 1.0                                                              
                                                                                
    denom = reduce(lambda x, y: x*y, [i for i in range(1, n+1)])                
    mol_l = reduce(lambda x, y: x*y, [i for i in range(1, n-r+1)])              
    mol_r = reduce(lambda x, y: x*y, [i for i in range(1, r+1)])                
    return float(denom / (mol_l * mol_r))                                       
                                                                                
def get_probability(p, n, r):                                                   
    q = 1 - p                                                                   
    return combination(n, r) * (p ** r) * (q ** (n-r))                          
                                                                                
def plot_bin_distrib(p, n, lt, lab):                                            
    xl = [nr for nr in range(0, n+1, 1)]                                        
    yl = [get_probability(p, n, nr) for nr in range(0, n+1, 1)]                 
    pl.plot(xl, yl, lt, label=lab)                                              
                                                                                
def main():                                                                     
    p = 1.0/3.0                                                                 
    n = 10                                                                      
    plot_bin_distrib(p, n, 'b-', 'Bin(%s,%f)' %(n,p))                           
    n = 20                                                                      
    plot_bin_distrib(p, n, 'r-', 'Bin(%s,%f)' %(n,p))                           
    n = 30                                                                      
    plot_bin_distrib(p, n, 'g-', 'Bin(%s,%f)' %(n,p))                           
    n = 40                                                                      
    plot_bin_distrib(p, n, 'y-', 'Bin(%s,%f)' %(n,p))                           
                                                                                
    pl.ylabel('p(x)')                                                           
    pl.xlabel('x')                                                              
    pl.legend()                                                                 
    pl.show()                                                                   
                                                                                
if __name__ == "__main__":                                                      
    main()

and here is output from above code.

python tips: use of chain generator

The book titled "Expert Python Programming" introduces pipeline of generators.
This concept is very useful for dividing loop codes to small parts.
Below is my example code that prints digit pyramid.

def change_value():
    for step in range(1, 10):
        yield step

def create_box(steps):
    for step in steps:
        yield (' ').join( str(step) for i in range(2 * step - 1) )

def create_pyramid(height):
    boxes = create_box(change_value())
    for step, box in enumerate(boxes):
        if step == height:
            break
        else:
            print ('').join(' ' for i in range((height - step)*2)) + box

Result of above code is ...

>>> create_pyramid(9)
                  1
                2 2 2
              3 3 3 3 3
            4 4 4 4 4 4 4
          5 5 5 5 5 5 5 5 5
        6 6 6 6 6 6 6 6 6 6 6
      7 7 7 7 7 7 7 7 7 7 7 7 7
    8 8 8 8 8 8 8 8 8 8 8 8 8 8 8
  9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9

postgres tips: stored procedure for calculating distance on the earth

CREATE FUNCTION GET_DISTANCE
    (alat FLOAT, alon FLOAT, lat FLOAT, lon FLOAT)
RETURNS FLOAT AS
$$
DECLARE
    radius_earth FLOAT;
    radian_lat FLOAT;
    radian_lon FLOAT;
    distance_v FLOAT;
    distance_h FLOAT;
    distance FLOAT;
BEGIN
    -- Insert earth radius
    SELECT INTO radius_earth 6378.137;

    -- Calculate difference between lat and alat
    SELECT INTO radian_lat radians(lat - alat);

    -- Calculate difference between lon and alon
    SELECT INTO radian_lon radians(lon - alon);

    -- Calculate vertical distance
    SELECT INTO distance_v (radius_earth * radian_lat);

    -- Calculate horizontal distance
    SELECT INTO distance_h (cos(radians(alat)) * radius_earth * radian_lon);

    -- Calculate distance(km) and convert it to distance(meter)
    SELECT INTO distance sqrt(pow(distance_h,2) + pow(distance_v,2)) * 1000;

    -- Returns distance
    RETURN DISTANCE;
END;
$$ language plpgsql;

> createlang plpgsql mytestdb
> psql -d mytestdb
mytestdb=# CREATE FUNCTION GET_DISTANCE
mytestdb-#     (alat FLOAT, alon FLOAT, lat FLOAT, lon FLOAT)
mytestdb-# RETURNS FLOAT AS
mytestdb-# $$
mytestdb$# DECLARE
mytestdb$#     radius_earth FLOAT;
mytestdb$#     radian_lat FLOAT;
mytestdb$#     radian_lon FLOAT;
mytestdb$#     distance_v FLOAT;
mytestdb$#     distance_h FLOAT;
mytestdb$#     distance FLOAT;
mytestdb$# BEGIN
mytestdb$#     -- Insert earth radius
mytestdb$#     SELECT INTO radius_earth 6378.137;
mytestdb$#
mytestdb$#     -- Calculate difference between lat and alat
mytestdb$#     SELECT INTO radian_lat radians(lat - alat);
mytestdb$#
mytestdb$#     -- Calculate difference between lon and alon
mytestdb$#     SELECT INTO radian_lon radians(lon - alon);
mytestdb$#
mytestdb$#     -- Calculate vertical distance
mytestdb$#     SELECT INTO distance_v (radius_earth * radian_lat);
mytestdb$#
mytestdb$#     -- Calculate horizontal distance
mytestdb$#     SELECT INTO distance_h (cos(radians(alat)) * radius_earth * radian_lon);
mytestdb$#
mytestdb$#     -- Calculate distance(km) and convert it to distance(meter)
mytestdb$#     SELECT INTO distance sqrt(pow(distance_h,2) + pow(distance_v,2)) * 1000;
mytestdb$#
mytestdb$#     -- Returns distance
mytestdb$#     RETURN DISTANCE;
mytestdb$# END;
mytestdb$# $$ language plpgsql;
CREATE FUNCTION
mytestdb=# select GET_DISTANCE(34.701909, 135.4949770, 35.681382, 139.766084);
   get_distance
------------------
 405807.810663345
(1 行)

latex tips: how to set up UTF-8 latex environment in Ubuntu 10.04LTS/amd64

I show you utf-8 latex installation steps for utf-8 encodings in Ubuntu-10.04 LTS (amd64) 2011/08/07. I am not sure this is the most effective way, but this can work very well on Ubuntu-10.04 LTS ;-) If you through below steps, you can compile utf-8 encoded tex files and also compile sphinx document!! (And..., if you want to convert sphinx document to pdf format and get ExtBabel error when uses "make all-pdf-ja" command, you should also check here)

Step1. install dvipsk-ja (default package is broken at 2011.06.11 JST)

> sudo add-apt-repository ppa:cosmos-door/dvipsk-ja
> sudo aptitude update
> sudo aptitude upgrade
> sudo aptitude install dvipsk-ja

Step2. install packages

> sudo aptitude install texlive texlive-math-extra texlive-latex-extra texlive-latex-extra-doc texlive-fonts-extra texlive-fonts-extra-doc texlive-fonts-recommended texlive-fonts-recommended-doc texlive-formats-extra texlive-latex-recommended texlive-latex-recommended texlive-extra-utils texlive-font-utils texlive-doc-ja ptex-bin jbibtex-bin mendexk okumura-clsfiles latex-cjk-japanese cmap-adobe-japan1 cmap-adobe-japan2 cmap-adobe-cns1 cmap-adobe-gb1 gs-cjk-resource ghostscript xdvik-ja dvi2ps dvi2ps-fontdesc-morisawa5 jmpost latexmk latex-mk pybliographer yatex

Step3. update latex environment

> updmap
> sudo mktexlsr
> sudo updmap-sys
> sudo dpkg-reconfigure ptex-jisfonts
> sudo jisftconfig add

Step4. get iso image of texlive2010 (this includes platex for utf8)

Step5. mount iso image of texlive2010

sudo mount -o loop texlive2010-20100826.iso /mnt3

Step6. install texlive2010

> sudo ./install-tl
...
Enter command: O [options]
Enter command: L [create symlinks in standard directories]
New value for binary directory [/usr/local/bin]: Enter
New value for man directory    [/usr/local/man]: Enter
New value for info directory   [/usr/local/info]: Enter
Enter command: R
Enter command: I
Installing to: /usr/local/texlive/2010
Installing [0001/2133, time/total: ??:??/??:??]: 12many [376k]
Installing [0002/2133, time/total: 00:00/00:00]: 2up [66k]
Installing [0003/2133, time/total: 00:00/00:00]: ANUfinalexam [3k]
Installing [0004/2133, time/total: 00:00/00:00]: AkkTeX [16k]
Installing [0005/2133, time/total: 00:00/00:00]: Asana-Math [433k]
Installing [0006/2133, time/total: 00:00/00:00]: ESIEEcv [137k]
Installing [0007/2133, time/total: 00:01/18:36]: FAQ-en [4943k]
Installing [0008/2133, time/total: 00:01/03:12]: HA-prosper [266k]
Installing [0009/2133, time/total: 00:01/03:03]: IEEEconf [188k]
Installing [0010/2133, time/total: 00:01/02:58]: IEEEtran [1325k]
Installing [0011/2133, time/total: 00:01/02:28]: MemoirChapStyles [669k]
Installing [0012/2133, time/total: 00:02/04:32]: SIstyle [338k]
Installing [0013/2133, time/total: 00:02/04:22]: SIunits [284k]
....
Installing [2129/2133, time/total: 03:33/03:33]: zhmetrics [66k]
Installing [2130/2133, time/total: 03:33/03:33]: zhspacing [165k]
Installing [2131/2133, time/total: 03:33/03:33]: ziffer [3k]
Installing [2132/2133, time/total: 03:33/03:33]: zwgetfdate [242k]
Installing [2133/2133, time/total: 03:33/03:33]: zwpagelayout [520k]
Time used for installing the packages: 03:33
running mktexlsr /usr/local/texlive/2010/texmf-dist /usr/local/texlive/2010/texmf
mktexlsr: Updating /usr/local/texlive/2010/texmf-dist/ls-R...
mktexlsr: Updating /usr/local/texlive/2010/texmf/ls-R...
mktexlsr: Done.
writing fmtutil.cnf data to /usr/local/texlive/2010/texmf-var/web2c/fmtutil.cnf
writing updmap.cfg to /usr/local/texlive/2010/texmf-config/web2c/updmap.cfg
writing language.dat data to /usr/local/texlive/2010/texmf-var/tex/generic/config/language.dat
writing language.def data to /usr/local/texlive/2010/texmf-var/tex/generic/config/language.def
writing language.dat.lua data to /usr/local/texlive/2010/texmf-var/tex/generic/config/language.dat.lua
running mktexlsr /usr/local/texlive/2010/texmf-var
mktexlsr: Updating /usr/local/texlive/2010/texmf-var/ls-R...
mktexlsr: Done.
running updmap-sys...done
re-running mktexlsr /usr/local/texlive/2010/texmf-var
mktexlsr: Updating /usr/local/texlive/2010/texmf-var/ls-R...
mktexlsr: Done.
pre-generating all format files (fmtutil-sys --all), be patient...done
running path adjustment actions
finished with path adjustment actions
running package specific postactions
finished with package specific postactions

 See
   /usr/local/texlive/2010/index.html
 for links to documentation.  The TeX Live web site (http://tug.org/texlive/)
 contains updates and corrections.

 TeX Live is a joint project of the TeX user groups around the world;
 please consider supporting it by joining the group best for you. The
 list of groups is available on the web at http://tug.org/usergroups.html.

 Add /usr/local/texlive/2010/texmf/doc/man to MANPATH, if not dynamically determined.
 Add /usr/local/texlive/2010/texmf/doc/info to INFOPATH.

 Most importantly, add /usr/local/texlive/2010/bin/x86_64-linux
 to your PATH for current and future sessions.

 Welcome to TeX Live!
Logfile: /usr/local/texlive/2010/install-tl.log

Step7. check platex can corporate with utf-8 encoded text (see -kanji=STRING option)

> platex --help
Usage: ptex [option] texfile
     : ptex [option] &format texfile

-fmt=NAME               use NAME instead of program name or %&format.
-halt-on-error          stop processing at the first error
[-no]-file-line-error   disable/enable file:line:error style messages
-ini                    be iniptex.
-interaction=STRING     set interaction mode (STRING=batchmode|nonstopmode|
                          scrollmode|errorstopmode)
-ipc                    send DVI output to a socket as well as the usual
                          output file
-ipc-start              as -ipc, and also start the server at the other end
-jobname=STRING         set the job name to STRING
-kanji=STRING           set Japanese encoding (STRING=euc|jis|sjis|utf8)
-kpathsea-debug=NUMBER  set path searching debugging flags according to
                          the bits of NUMBER
[-no]-mktex=FMT         disable/enable mktexFMT generation (FMT=tex/tfm)
-mltex                  enable MLTeX extensions such as \charsubdef
-output-comment=STRING  use STRING for DVI file comment instead of date
-output-directory=DIR   use DIR as the directory to write files to
[-no]-parse-first-line  disable/enable parsing of the first line of the
                          input file
-progname=STRING        set program (and fmt) name to STRING
-recorder               enable filename recorder
[-no]-shell-escape      disable/enable \write18{SHELL COMMAND}
-src-specials           insert source specials into the DVI file
-src-specials=WHERE     insert source specials in certain places of
                          the DVI file. WHERE is a comma-separated value
                          list: cr display hbox math par parend vbox
-translate-file=TCXNAME use the TCX file TCXNAME
-help                   print this message and exit.
-version                print version information and exit.

Email bug reports to ptex-staff@ml.asciimw.jp.

hive tips: convert default separator

hive's default separator is like '^A'.
to convert '^A' to ',', use below sed script.

/opt/hadoop/bin/hadoop dfs -cat /user/test/* | sed 's/[Ctrl-V][Ctrl-A]/,/g' > ${outputfile}

machine learning: curve-fitting

#coding: utf-8                                                                  
                                                                                
import numpy as np                                                              
from pylab import *                                                             
import sys                                                                      
                                                                                
M = 9                                                                           
def y(x, wlist):                                                                
    ret = wlist[0]                                                              
    for i in range(1, M+1):                                                     
        ret += wlist[i] * (x ** i)                                              
    return ret                                                                  
                                                                                
def estimate(xlist, tlist, lam):                                                
    A = []                                                                      
    for i in range(M+1):                                                        
        for j in range(M+1):                                                    
            temp = (xlist**(i+j)).sum()                                         
            if i == j:                                                          
                temp += lam                                                     
            A.append(temp)                                                      
    A = array(A).reshape(M+1, M+1)                                              
                                                                                
    T = []                                                                      
    for i in range(M+1):                                                        
        T.append(((xlist**i) * tlist).sum())                                    
    T = array(T)                                                                
                                                                                
    wlist = np.linalg.solve(A, T)                                               
    return wlist                                                                
                                                                                
def rms(xlist, tlist, wlist):                                                   
    E_w = 0                                                                     
    N = len(xlist)                                                              
                                                                                
    for n in range(0, N):                                                       
        sum = 0                                                                 
        for j in range(0, M+1):                                                 
            sum += wlist[j] * (xlist[n] ** j)                                   
        E_w += ((sum - tlist[n]) ** 2)/2                                        
                                                                                
    return str(np.sqrt(2 * E_w / N))                                            

def example1():                                                                 
    # number of training data                                                   
    N = 10                                                                      
                                                                                
    # generate training data                                                    
    xlist = np.linspace(0, 1, N) # extract N-points                             
    tlist = np.sin(2 * np.pi * xlist) + np.random.normal(0, 0.2, xlist.size)    
                                                                                
    # estimate parametaer w                                                     
    wlist = estimate(xlist, tlist, np.exp(-18.0))                               
    #print wlist                                                                
    print "E_RMS for training data: %s" %  rms(xlist, tlist, wlist)             
                                                                                
    N = 100                                                                     
    xlist2 = np.linspace(0, 1, N) # extract N-points                            
    tlist2 = np.sin(2 * np.pi * xlist2) + np.random.normal(0, 0.2, xlist2.size) 
    print "E_RMS for test data: %s" % rms(xlist2, tlist2, wlist)                
                                                                                
    # generate original data                                                    
    xs = np.linspace(0, 1, 1000)                                                
                                                                                
    # ideal and model                                                           
    ideal = np.sin(2 * np.pi * xs)                                              
    model = [y(x, wlist) for x in xs]                                           
                                                                                
    # plot training data and original data                                      
    plot(xlist, tlist, 'bo')                                                    
    plot(xlist2, tlist2, 'rd')                                                  
    plot(xs, ideal, 'g-')                                                       
    plot(xs, model, 'r-')                                                       
    xlim(0.0, 1.0)                                                              
    ylim(-1.5, 1.5)                                                             
    show()                                                                      
                                                                                
if __name__ == "__main__":                                                      
    example1()

shell tips: get absolute path

$ cat abspath.sh
#!/bin/sh

echo $(cd $(dirname $0) && pwd)/$(basename $0)

android tips: what to do before install self-compiled apk

yaboo@maniac:~/Projects/Android/UseLocation$ jarsigner -verbose -keystore ~/.android/debug.keystore bin/MainActivity-unsigned.apk androiddebugkey
キーストアのパスワードを入力してください: 
   追加中: META-INF/MANIFEST.MF
   追加中: META-INF/ANDROIDD.SF
   追加中: META-INF/ANDROIDD.RSA
  署名中: res/layout/main.xml
  署名中: AndroidManifest.xml
  署名中: resources.arsc
  署名中: res/drawable-hdpi/icon.png
  署名中: res/drawable-ldpi/icon.png
  署名中: res/drawable-mdpi/icon.png
  署名中: classes.dex

android tips: check build-target from android-sdk command

first, check installed android-sdk supports my build-target.

yaboo@maniac:~/Projects/Android$ android list targets
Available Android targets:
id: 1 or "android-3"
     Name: Android 1.5
     Type: Platform
     API level: 3
     Revision: 4
     Skins: QVGA-P, HVGA-L, QVGA-L, HVGA-P, HVGA (default)
id: 2 or "android-4"
     Name: Android 1.6
     Type: Platform
     API level: 4
     Revision: 3
     Skins: WVGA800 (default), WVGA854, QVGA, HVGA
id: 3 or "android-7"
     Name: Android 2.1-update1
     Type: Platform
     API level: 7
     Revision: 2
     Skins: WQVGA400, WVGA800 (default), WVGA854, QVGA, WQVGA432, HVGA
id: 4 or "android-8"
     Name: Android 2.2
     Type: Platform
     API level: 8
     Revision: 2
     Skins: WQVGA400, WVGA800 (default), WVGA854, QVGA, WQVGA432, HVGA
id: 5 or "android-9"
     Name: Android 2.3.1
     Type: Platform
     API level: 9
     Revision: 2
     Skins: WQVGA400, WVGA800 (default), WVGA854, QVGA, WQVGA432, HVGA
id: 6 or "android-10"
     Name: Android 2.3.3
     Type: Platform
     API level: 10
     Revision: 1
     Skins: WQVGA400, WVGA800 (default), WVGA854, QVGA, WQVGA432, HVGA
id: 7 or "android-11"
     Name: Android 3.0
     Type: Platform
     API level: 11
     Revision: 1
     Skins: WXGA (default)
id: 8 or "android-12"
     Name: Android 3.1
     Type: Platform
     API level: 12
     Revision: 2
     Skins: WXGA (default)
id: 9 or "android-13"
     Name: Android 3.2
     Type: Platform
     API level: 13
     Revision: 1
     Skins: WXGA (default)

now, create android project using 'android' command

yaboo@maniac:~/Projects/Android$ android create project --target 6 --path ./myProject --activity MyActivity --package com.yablog.myproject
Created project directory: ./myProject
Created directory /home/yaboo/Projects/Android/myProject/src/com/yablog/myproject
Added file ./myProject/src/com/yablog/myproject/MyActivity.java
Created directory /home/yaboo/Projects/Android/myProject/res
Created directory /home/yaboo/Projects/Android/myProject/bin
Created directory /home/yaboo/Projects/Android/myProject/libs
Created directory /home/yaboo/Projects/Android/myProject/res/values
Added file ./myProject/res/values/strings.xml
Created directory /home/yaboo/Projects/Android/myProject/res/layout
Added file ./myProject/res/layout/main.xml
Created directory /home/yaboo/Projects/Android/myProject/res/drawable-hdpi
Created directory /home/yaboo/Projects/Android/myProject/res/drawable-mdpi
Created directory /home/yaboo/Projects/Android/myProject/res/drawable-ldpi
Added file ./myProject/AndroidManifest.xml
Added file ./myProject/build.xml
Added file ./myProject/proguard.cfg

python tips: convert ip-address returned by android api to readable format

In SL4A, wifiGetConnectionInfo contains ip-address of an android device. But it is formatted in integer, difficult to understand. Below script can convert integer formatted ip-addr to human-friendly format 8^)

>>> ip = 67807424
>>> format(ip, 'b')
'100000010101010100011000000'
>>> len(format(ip, 'b'))
27
>>> ipstr = '00000' + format(ip, 'b')
>>> L = []
>>> for i in range(0, 32, 8):
...     L.append(str(int(ipstr[i:i+8], 2)))
... 
>>> L.reverse()
>>> L
['192', '168', '10', '4']
>>> ('.').join(L)
'192.168.10.4'

python tips: python scripting on android and remotecontrol

* Overview
SL4A(Script Layer for Android) enables you to execute several script language (such as python, ruby, ...) on your android devices. Today I tried to SL4A and remotecontrol which enables you to remotely execute python code from your non-android computer (such as linux, windows...).

- SL4A installation steps are introduced here
- RemoteControl setup steps are described here
- SL4A API is also described here

* Get location and geocode from android device (RemoteControl)

yaboo@maniac:~/Downloads/android-sdk-linux_x86/platform-tools$ python
Python 2.6.5 (r265:79063, Apr 16 2010, 13:57:41) 
[GCC 4.4.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import android
>>> droid = android.Android(('192.168.10.4', 46522)) # public server mode
>>> droid.startLocating()
Result(id=0, result=None, error=None)
>>> loc = droid.readLocation().result
>>> loc
{u'network': {u'altitude': 0, u'provider': u'network', u'longitude': 139.***69349999998, u'time': 1311439075865, u'latitude': 35.***094566666668, u'speed': 0, u'accuracy': 75}, u'gps': {u'altitude': 58.9000244140625, u'provider': u'gps', u'longitude': 139.***20321081579, u'time': 1311439081000, u'latitude': 35.***270288087428, u'speed': 0, u'accuracy': 55}}
>>> n = loc['network']
>>> la = n['latitude']
>>> lo = n['longitude']
>>> la
35.***094566666668
>>> lo
139.***69349999998
>>> address = droid.geocode(la, lo).result
>>> print repr(address).decode('unicode-escape')
[{u'thoroughfare': u'*丁目', u'locality': u'**市', u'admin_area': u'**県', u'feature_name': u'２７', u'country_code': u'JP', u'country_name': u'日本'}]
>>> droid.stopLocating()
Result(id=3, result=None, error=None)

* Testing TelephonyManager functionality

>>> print droid.getDeviceId().result
3568***********
>>> print droid.getCellLocation().result
{u'lac': 145, u'cid': 99008715}
>>> print droid.getDeviceSoftwareVersion().result
01
>>> print droid.getNeighboringCellInfo().result
[]
>>> print droid.getNetworkOperator().result
44010
>>> print droid.getNetworkOperatorName().result
NTT DOCOMO
>>> print droid.getPhoneType().result
gsm
>>> print droid.getSimCountryIso().result
jp
>>> print droid.getSimOperator().result
44010
>>> print droid.getSimOperatorName().result

>>> print droid.getSimSerialNumber().result
89811**************
>>> print droid.getSimState().result
ready
>>> print droid.SubscriberId().result
com.googlecode.android_scripting.rpc.RpcError: Unknown RPC.
None

* Testing Batterymanager API

>>> droid.batteryStartMonitoring()
Result(id=24, result=None, error=None)
>>> print droid.batteryCheckPresent().result
True
>>> print droid.batteryGetHealth().result
2
>>> print droid.batteryGetLevel().result
74
>>> print droid.batteryGetPlugType().result
2
>>> print droid.batteryGetPlugType().result
1
>>> print droid.batteryGetStatus().result
2
>>> print droid.batteryGetTechnology().result
Li-ion
>>> print droid.batteryGetTemperature().result
360
>>> print droid.batteryGetVoltage().result
4057
>>> print droid.readBatteryData().result
{u'status': 2, u'temperature': 380, u'level': 74, u'battery_present': True, u'plugged': 1, u'health': 2, u'voltage': 4057, u'technology': u'Li-ion'}
>>> droid.batteryStopMonitoring()
Result(id=35, result=None, error=None)

* Testing CommonIntentsFacade

>>> droid.viewMap(u"*****") # android device open google map!!!
Result(id=40, result=None, error=None)

* Testing ContactsFacade

>>> droid.contactsGetCount().result
301

* Testing WifiFacade

>>> print droid.wifiGetConnectionInfo().result
{u'ssid': u'**********', u'bssid': u'00:0d:02:f7:**:**', u'network_id': 0, u'supplicant_state': u'completed', u'link_speed': 54, u'mac_address': u'04:**:65:**:**:18', u'rssi': -57, u'ip_address': 67***24, u'hidden_ssid': False}
>>> print droid.wifiGetScanResults().result
[{u'capabilities': u'[WPA-PSK-TKIP+CCMP][WPA2-PSK-TKIP+CCMP]', u'frequency': 2467, u'ssid': u'logitec51', u'bssid': u'00:01:8e:f2:**:c0', u'level': -82}, {u'capabilities': u'[WEP]', u'frequency': 2442, u'ssid': u'**********', u'bssid': u'00:0d:02:f7:c0:bc', u'level': -55}]

python tips: print list contains japanese characters

>>> L = [u"ほげ", u"まげ"]
>>> L
[u'\u307b\u3052', u'\u307e\u3052']
>>> print repr(L).decode('unicode-escape')
[u'ほげ', u'まげ']
>>> 

python tips: creating polar graph using matplotlib

#!/usr/bin/env python                                                                                                                                                                                      
                                                                                                                                                                                                           
import sys                                                                                                                                                                                                 
import csv                                                                                                                                                                                                 
import codecs                                                                                                                                                                                              
                                                                                                                                                                                                           
import matplotlib                                                                                                                                                                                          
import numpy as np                                                                                                                                                                                         
from matplotlib.pyplot import figure, show, rc, grid                                                                                                                                                       
                                                                                                                                                                                                           
def main():                                                                                                                                                                                                
    argvs = sys.argv                                                                                                                                                                                       
    argc = len(argvs)                                                                                                                                                                                      
                                                                                                                                                                                                           
    if (argc != 4):                                                                                                                                                                                        
        print "Usage: python %s input_file output_file title" %(argvs[0])                                                                                                                                  
        sys.exit(0)                                                                                                                                                                                        
                                                                                                                                                                                                           
    in_file = argvs[1]                                                                                                                                                                                     
    out_file = argvs[2]                                                                                                                                                                                    
    title = argvs[3]                                                                                                                                                                                       
                                                                                                                                                                                                           
    csvfile = codecs.open(in_file, 'r', 'utf-8')                                                                                                                                                           
                                                                                                                                                                                                           
    list_x = []                                                                                                                                                                                            
    list_y = []                                                                                                                                                                                            
                                                                                                                                                                                                           
    for row in csv.reader(csvfile):                                                                                                                                                                        
        radian = 0                                                                                                                                                                                         
                                                                                                                                                                                                           
        if float(row[0]) < 0:                                                                                                                                                                              
            radian = (float(row[0])+360.0)/180                                                                                                                                                             
        else:                                                                                                                                                                                              
            radian = float(row[0])/180                                                                                                                                                                     
                                                                                                                                                                                                           
        list_x.append(np.pi*radian)                                                                                                                                                                        
        list_y.append(float(row[1]))                                                                                                                                                                       
                                                                                                                                                                                                           
    init_graph_settings()                                                                                                                                                                                  
    draw_polar_graph(list_x, list_y, out_file, title)                                                                                                                                                      
    csvfile.close()                                                                                                                                                                                        
                                                                                                                                                                                                           
def init_graph_settings():                                                                                                                                                                                 
    # radar green, solid grid lines                                                                                                                                                                        
    rc('grid', color='#316931', linewidth=0.5, linestyle='--')                                                                                                                                             
    rc('xtick', labelsize=15)                                                                                                                                                                              
    rc('ytick', labelsize=10)                                        

def draw_polar_graph(list_x, list_y, out_file, title):                                                                                                                                                     
    """                                                                                                                                                                                                    
    draws polar graph                                                                                                                                                                                      
                                                                                                                                                                                                           
    :param list_x: list contains x values                                                                                                                                                                  
    :param list_y: list contains y values                                                                                                                                                                  
    :param title: title for a graph                                                                                                                                                                        
    :type list_x: list contains float values                                                                                                                                                               
    :type list_y: list contains float values                                                                                                                                                               
    :type title: string.                                                                                                                                                                                   
    """                                                                                                                                                                                                    
                                                                                                                                                                                                           
    # force square figure and square axes looks better for polar, IMO                                                                                                                                      
    width, height = matplotlib.rcParams['figure.figsize']                                                                                                                                                  
    size = min(width, height)                                                                                                                                                                              
                                                                                                                                                                                                           
    # make a square figure                                                                                                                                                                                 
    fig = figure(figsize=(size, size))                                                                                                                                                                     
                                                                                                                                                                                                           
    ax = fig.add_axes([0.1, 0.1, 0.8, 0.8], polar=True, axisbg='#d5de9c')                                                                                                                                  
    ax.plot(list_x, list_y, 'r.', markersize=2)                                                                                                                                                            
    ax.set_rmax(10000.0)                                                                                                                                                                                   
    ax.set_title(title, fontsize=20)                                                                                                                                                                       
    grid(True)                                                                                                                                                                                             
    fig.savefig(out_file, dpi=100)                                                                                                                                                                         
                                                                                                                                                                                                           
if __name__ == "__main__":                                                                                                                                                                                 
    main()

sphinx tips: sphinx.ext.pngmath plugin

Below is a sample code for checking how sphinx pngmath plugin works. If you ompile below code and check output with browser, you can see TeX like expression is converted to mathematical expression. You need to install dvipng for using pngmath plugin :-)

====================
sphinx.ext.math test
====================

.. math::

   (a + b)^2 = a^2 + 2ab + b^2

   (a - b)^2 = a^2 - 2ab + b^2

.. math::

   (a + b)^2  &=  (a + b)(a + b) \\
              &=  a^2 + 2ab + b^2

.. math:: (a + b)^2 = a^2 + 2ab + b^2

python tips: googley style docstring

I surveyed some comment style in python. One is sphinx style and the other is google style (they are introduced in this site). However sphinx style is suitable for documentation using sphinx, its code legibility is poorer than google style. Below code is commented in google style.

#!/usr/bin/env python                                                           
#                                                                               
# Copyright 2011 Yuki OYABU Inc.                                                
#                                                                               
# Licensed under the Apache License, Version 2.0 (the "License");               
# you may not use this file except in compliance with the License.              
# You may obtain a copy of the License at                                       
#                                                                               
#     http://www.apache.org/licenses/LICENSE-2.0                                
#                                                                               
# Unless required by applicable law or agreed to in writing, software           
# distributed under the License is distributed on an "AS IS" BASIS,             
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.      
# See the License for the specific language governing permissions and           
# limitations under the License.                                                


def fetch_bigtable_rows(big_table, keys, other_silly_variable=None):
    """                                                                         
    Fetches rows from a Bigtable.                                               
                                                                                
    Retrieves rows pertaining to the given keys from the Table instance         
    represented by big_table. Silly things may happen if                        
    oahter_silly_variable is not None.                                          
                                                                                
    Args:                                                                       
        big_table:                                                              
            An open Bigtable Table instance.                                    
        keys:                                                                   
            A sequence of strings representing the key of each table            
            row to fetch.                                                       
        other_silly_variable:                                                   
            Another optional variable, that has a much longer name than         
            the other args, and which does nothing.                             
                                                                                
    Returns:                                                                    
        A dict mapping keys to the corresponding table row data                 
        fetched. Each row is represented as a tuple of strings. For             
        example::                                                               
                                                                                
            {'Serak' : ('Rigel VII', 'Preparer'),                               
             'Zim': ('Irk', 'Invader'),                                         
             'Lrrr': ('Omicron Persei 8', 'Emperor')}                           
                                                                                
        If a key from the keys argument is missing from the dictionary,         
        then that row was not found in the table.                               
                                                                                
    Raises:                                                                     
        IOError:                                                                
            An error occured accessing the bigtable. Table object.              
    """
    pass


class SampleClass(object):
    """                                                                         
    Summary of class here.                                                      
                                                                                
    Longer class information....                                                
    Longer class information....                                                
                                                                                
    Attributes:                                                                 
        likes_spam:                                                             
            A boolean indicating if we like SPAM or not.                        
        eggs:                                                                   
            An integer count of the eggs we have laid.                          
    """

    def __init__(self, likes_spam=False):
        """Inits SampleClass with blah."""
        self.likes_spam = likes_spam
        self.eggs = 0

    def public_method(self):
        """Performs operation blah."""

ubuntu tips: how to add repository using apt-key

sometime i cannot access to keyserver.ubuntu.com, and cannnot automatically update my package repository using add-repository command. here I show you how manually update my package repogitory.

# get gpgkey from anywhere... 
> sudo apt-key add ${gpgkey_file}
# edit /etc/apt/sources.list for update repositories
> sudo apt-get update 

shpinx tips: avoid ExtBabel class error for compiling japanese latex

i don't know why below error occurs...

yaboo@sp2:~/DocProject$ make latex
sphinx-build -b latex -d _build/doctrees   . _build/latex
Running Sphinx v1.0.7
loading translations [ja]... done
loading pickled environment... done
building [latex]: all documents
updating environment: 0 added, 0 changed, 0 removed
looking for now-outdated files... none found
processing testProject.tex... index intro code bd sd 
resolving references...
writing... 
Exception occurred:
  File "/usr/local/lib/python2.6/dist-packages/Sphinx-1.0.7-py2.6.egg/sphinx/writers/latex.py", line 193, in __init__
    babel = ExtBabel(builder.config.language)
TypeError: __init__() takes exactly 3 arguments (2 given)
The full traceback has been saved in /tmp/sphinx-err-q8bibE.log, if you want to report the issue to the developers.
Please also report this if it was a user error, so that a better error message can be provided next time.
Either send bugs to the mailing list at ,
or report them in the tracker at . Thanks!
make: *** [latex] エラー 1

modify /usr/local/lib/python2.6/dist-packages/Sphinx-1.0.7-py2.6.egg/sphinx/writers/latex.py
(just comment out non-japanese process....)

if builder.config.language:
    self.elements['classoptions'] += ',english,dvipdfm'
    self.elements['babel'] += r'\newcount\pdfoutput\pdfoutput=0'
    self.elements['preamble'] += \
        r'\AtBeginDvi{\special{pdf:tounicode EUC-UCS2}}'
    self.elements['shorthandoff'] = ''
    self.elements['fncychap'] = '\\usepackage[Sonny]{fncychap}'
else:

sphinx tips: sphinx extenstions and configurations

# Abstraction

I installed blockdiag and seqdiag provided by tk0miya.

This post show you how to install and setup *diag tools and autodoc.

# blockdiag and seqdiag installation

- blockdiag installation

sudo easy_install sphinx

sudo easy_install blockdiag
sudo easy_install sphinxcontrib-blockdiag

- seqdiag installation

sudo easy_install seqdiag
sudo easy_install sphinxcontrib-seqdiag

# additional settings into conf.py

# Enabled extensions (blockdiag and seqdiag, autodoc!)
extensions = ['sphinxcontrib.blockdiag','sphinxcontrib.seqdiag','sphinx.ext.autodoc']

# If extensions (or modules to document with autodoc) are in another directory,
# add these directories to sys.path here. If the directory is relative to the
# documentation root, use os.path.abspath to make it absolute, like shown here.
sys.path.insert(0, os.path.abspath('./codes'))

# configure default autodoc's action
autodoc_default_flags = ['members', 'undoc-members', 'show-inheritance']

# Fontpath for blockdiag (truetype font)
blockdiag_fontpath = '/usr/share/fonts/truetype/ipafont/ipagp.ttf'
blockdiag_tex_image_format = 'PNG'

# Fontpath for seqdiag (truetype font)
seqdiag_fontpath = '/usr/share/fonts/truetype/ipafont/ipagp.ttf'
seqdiag_tex_image_format = 'PNG'

sphinx tips: useful links

sphinx is "python documentation generation" tool.
sphinx is one of powerful documentation tool I have ever used.

Here, I list useful link how to use sphinx and reStructuredText.

sphinx

• sphinx official website
• sphinx: html theming support

reStructuredText

• reStructuredText official website
• quick reStructuredText
  
• reStructuredText cheat sheet

reStructuredText add-on

• blockDiag
• how to use autodoc
• autodoc usage introduced by sphinx official website
  

latex tips: texlive2010 installation

Download texlive2010 iso image from a nearest CTAN site
http://ftp.yz.yamagata-u.ac.jp/pub/CTAN/systems/texlive/Images/

Mount the iso image like this:

sudo mount -o loop texlive2010-20100826.iso /mnt3

Install texlive2010

cd /mnt3
sudo sh install-tl

python tips: draw gaussian pdf graph with matplotlib

*** Definition of gaussian distribution ***

*** Sample code for drawing gaussian distribution ***

#!/usr/bin/env python                                                           
#coding:utf-8                                                                   

import pylab as pl
import numpy as np
from enthought.mayavi import mlab

def gauss2d(x):
    return 1/np.sqrt(2*np.pi) * np.exp(-(x**2)/2)

def gauss3d(x, y):
    return 1/np.sqrt(2*np.pi) * np.exp(-(x**2 + y**2)/2)

def plot_gauss2d():
    x = np.mgrid[-4:4:100j]

    # matplotlib functions                                                      
    pl.plot(gauss2d(x), 'bo-')
    pl.ylabel('gauss2d(x)')
    pl.xlabel('x')
    pl.show()

def plot_gauss3d():
    x, y = np.mgrid[-4:4:100j, -4:4:100j]
    """                                                                         
    x, y = np.mgrid[-2:2:5j, -2:2:5j]                                           
    array([[[-2., -2., -2., -2., -2.],                                          
            [-1., -1., -1., -1., -1.],                                          
            [ 0.,  0.,  0.,  0.,  0.],                                          
            [ 1.,  1.,  1.,  1.,  1.],                                          
            [ 2.,  2.,  2.,  2.,  2.]],                                         
           [[-2., -1.,  0.,  1.,  2.],                                          
            [-2., -1.,  0.,  1.,  2.],                                          
            [-2., -1.,  0.,  1.,  2.],                                          
            [-2., -1.,  0.,  1.,  2.],                                          
            [-2., -1.,  0.,  1.,  2.]]])                                        
    x[0,0] ==> -2.0                                                             
    y[0,0] ==> -2.0                                                             
    """

    # matplotlib functions                                                      
    pl.plot(gauss3d(x, y))
    pl.ylabel('gauss3d(x)')
    pl.xlabel('x')
    pl.show()

def plot_gauss3d_mayavi():
    x, y = np.mgrid[-4:4:100j, -4:4:100j]
    z = gauss3d(x, y)

    mlab.surf(z, warp_scale='auto')
    mlab.outline()
    mlab.axes()
    mlab.show()

def main():
    #plot_gauss2d()                                                             
    #plot_gauss3d()                                                             
    plot_gauss3d_mayavi()

if __name__ == "__main__":
    main()

*** Sample code output ***

• output from gauss2d function

• output from gauss3d function

• output from gauss3d_mayavi function

python tips: matplotlib plot option

#/usr/bin/env python
# coding:utf-8

import pg
import sys
import datetime
from pylab import *

def main():
    argvs = sys.argv
    argc = len(argvs)

    """ check argument """
    if (argc != 1 + 6):
        argument = "dbname dbuser dbpass fromdate todate dbtable_prefix"
        print "Usage: python %s %s" % (argvs[0], argument)
        sys.exit(1)

    dbname=argvs[1]
    dbuser=argvs[2]
    dbpass=argvs[3]
    fromdate=argvs[4]
    todate=argvs[5]
    prefix=argvs[6]

    """ initialization """
    initGraphParams()
    dbobj = getConnection(dbname, dbuser, dbpass)

    """ get X label """
    Lx = makeX(fromdate, todate)
    Lx2 = makeX2(fromdate, todate)


    """ get X,Y data """
    classes = getClassId()
    styles = ('-', 'x-', '+-', '*-')
    colors = ('b', 'g', 'r', 'c', 'm', 'y', 'k')
    count = 0
    for cls in classes:
        Ly = makeY(dbobj, Lx, prefix, 'class', cls, 'rows')
        style = styles[count % len(styles)]
        color = colors[count % len(colors)]
        plot(Lx2, Ly, style+color, label=cls)
        count += 1

    title('number of XXX')
    xlabel('date')
    ylabel('number')
    grid(True)
    legend(bbox_to_anchor=(1.01, 1), loc=2, ncol=2, borderaxespad=0.)
    show()
    dbobj.close()

Realtime Big Data at Facebook with Hadoop and HBase

Tokyo HBase Meetup - Realtime Big Data at Facebook with Hadoop and HBase (ja)

View more presentations from tatsuya6502

python tips: create line graph with matplotlib

*** sample code (mypg.py) ***

#/usr/bin/env python                                                       
# coding:utf-8                                                             

# need to install python-pygresql                                          
import pg
import sys
from pylab import *

def main():
argvs = sys.argv
argc = len(argvs)

# check argument                                                       
if (argc != 4):
   print "Usage: python %s dbname dbuser dbpass" % (argvs[0])
   sys.exit(1)

dbname=argvs[1]
dbuser=argvs[2]
dbpass=argvs[3]
dbobj = getConnection(dbname, dbuser, dbpass)

# input data from database tables                                      
Lx = getValueX()
L1 = collectData(dbobj, '1')
L2 = collectData(dbobj, '2')
L3 = collectData(dbobj, '3')
L4 = collectData(dbobj, '4')

# plot data using matplotlib                                           
plot(Lx, L1, label='id=1')
plot(Lx, L2, label='id=2')
plot(Lx, L3, label='id=3')
plot(Lx, L4, label='id=4')

# set labels and title                                                 
xlabel('hour')
ylabel('random_value')
title('plot test')

# draw legend                                                          
legend()

# draw graph                                                           
show()

dbobj.close()

def getValueX():
    L = []
    hour = 0
    while (hour <= 23):
        L.append(hour)
        hour += 1
    return L

# collecting data if same id                                                    
def collectData(dbobj, target):
    L = []
    sn = 0
    while sn <= 23:
        dbtable='public.table_' + str(sn)

        sql="SELECT * FROM %s WHERE id = '%s';" % (dbtable, target)
        for row in dbobj.query(sql).dictresult():
            if row['id'] == target:
                L.append(int(row['value']))
        sn += 1

    return L
# debug function                                                                
def showTables(dbobj):
    for tbl in dbobj.get_tables():
        print tbl

# initialize db connection                                                      
def getConnection(dbname, dbuser, dbpass):
    pg.set_defhost('localhost')
    pg.set_defport(5432)
    pg.set_defbase(dbname)
    return pg.DB(user=dbuser, passwd=dbpass)

if __name__ == "__main__":
    main()

*** output from above code ***

yaboo@maniac:~/$ python mypg.py mydb test test

*** matplotlib reference pages ***
ぐうたらの部屋
Matplotlib サンプル集 - Kaiseki
http://www.ike-dyn.ritsumei.ac.jp/~uchida/scipy-lecture-notes/intro/index.html

shell script tips: bash and function

*** motivation ***
sh script does not have ${RANDOM} value which generates random value if used in bash script. Below is sample code explains how to use ${RANDOM} and define functions in bash scripts.

*** sample code (makeData.sh) ***

#!/bin/bash                                                                   

function create_db(){
  dbname=$1
  dbuser=$2
  dbpass=$3
  PGPASSWORD=${dbpass} dropdb -U ${dbuser} ${dbname}
  PGPASSWORD=${dbpass} createdb -U ${dbuser} ${dbname}
}

function create_tbl(){
  id=0
  dbname=$1
  dbuser=$2
  dbpass=$3
  dbtable=$4

  sql="CREATE TABLE ${dbtable} (id text, value int);"
  PGPASSWORD=${dbpass} psql -U ${dbuser} -d ${dbname} -c "${sql}"

  while [ ${id} -le 3 ]
  do
      id=`expr ${id} + 1`
      sql="INSERT INTO ${dbtable} VALUES (${id}, ${RANDOM});"
      PGPASSWORD=${dbpass} psql -U ${dbuser} -d ${dbname} -c "${sql}"
  done
}

function select_tbl(){
  dbname=$1
  dbuser=$2
  dbpass=$3
  dbtable=$4

  sql="SELECT * FROM ${dbtable};"
  PGPASSWORD=${dbpass} psql -U ${dbuser} -d ${dbname} -q -c "${sql}"
}

function main(){
  dbname=$1
  dbuser=$2
  dbpass=$3
  create_db ${dbname} ${dbuser} ${dbpass}

  sn=0
  prefix="table_"

  while [ ${sn} -le 23 ]
  do
      dbtable=${prefix}${sn}
      create_tbl ${dbname} ${dbuser} ${dbpass} ${dbtable}
      select_tbl ${dbname} ${dbuser} ${dbpass} ${dbtable}
      sn=`expr ${sn} + 1`
  done
}


# Program starts here!!!
if [ $# -ne 3 ]; then
  echo "usage: bash $0 dbname dbuser dbpass"
  exit
fi

main $1 $2 $3

*** output from above script ***

yaboo@maniac:~$ bash makeData.sh mydb test test 
id | value                                                                   
----+-------                                                                  
1  |   632                                                                   
2  |  5999                                                                   
3  |  3495                                                                   
4  |  3127                                                                   
(4 行)                                                                        
                                                                            
id | value                                                                   
----+-------                                                                  
1  | 29475                                                                   
2  |  7581                                                                   
3  | 17044                                                                   
4  | 12726                                                                   
(4 行)                                                                        
                                                                            
id | value                                                                   
----+-------                                                                  
1  | 13619                                                                   
2  | 25982                                                                   
3  | 21835                                                                   
4  | 23054                                                                   
(4 行)                                                                        
                                                                            
id | value                                                                   
----+-------                                                                  
1  |   726                                                                   
2  | 15405                                                                   
3  | 16549                                                                   
4  | 16405                                                                   
(4 行)                                                                        
                                                                            
id | value                                                                   
----+-------                                                                  
1  | 14362                                                                   
2  | 22030                                                                   
3  | 16364                                                                   
4  | 24379                                                                   
(4 行)                                                                        

...

id | value                                                                   
----+-------                                                                  
1  |  8468                                                                   
2  | 10413                                                                   
3  | 12345                                                                   
4  | 30637                                                                   
(4 行)                                                                        
                                                                            
id | value                                                                   
----+-------                                                                  
1  |  8372                                                                   
2  | 16466                                                                   
3  | 31683                                                                   
4  | 23073                                                                   
(4 行)                                                                        
                                                                            
yaboo@maniac:~$

postgres tips: enable md5 authentication

step1. Add user

postgres@maniac:~$ createuser test
新しいロールをスーパーユーザとしますか? (y/n) n
新しいロールにデータベース作成権限を与えますか? (y/n) y
新しいロールにロールを作成する権限を与えますか? (y/n) y

step2. Set password for user

postgres@maniac:~$ psql
psql (8.4.8)
"help" でヘルプを表示します.

postgres=# alter user postgres with encrypted password 'test';
ALTER ROLE

step3. Edit pg_hba.conf (hba: host based authentication)
[default configuration]

# Database administrative login by UNIX sockets
local   all         postgres                          ident

# TYPE  DATABASE    USER        CIDR-ADDRESS          METHOD

# "local" is for Unix domain socket connections only
local   all         all                               ident
# IPv4 local connections:
host    all         all         127.0.0.1/32          md5
# IPv6 local connections:
host    all         all         ::1/128               md5

[configuration enables md5 authentication for user 'postgres']

# Database administrative login by UNIX sockets
local   all         postgres                          ident
local   all         test                              md5

# TYPE  DATABASE    USER        CIDR-ADDRESS          METHOD

# "local" is for Unix domain socket connections only
local   all         all                               ident
# IPv4 local connections:
host    all         all         127.0.0.1/32          md5
# IPv6 local connections:
host    all         all         ::1/128               md5

step4. Restart postgresql

sudo /etc/init.d/postgresql-8.4 restart
* Restarting PostgreSQL 8.4 database server    [ OK ]
yaboo@maniac:~$ createdb -U test sampledb
パスワード: 
yaboo@maniac:~$ psql -U test sampledb
ユーザ test のパスワード: 
psql (8.4.8)
"help" でヘルプを表示します.

sampledb=> 

postgres tips: copy and \copy

Below explanation is quoted from http://wiki.postgresql.org/wiki/COPY

COPY is the Postgres method of data-loading. Postgres's COPY comes in two separate variants, COPY and \COPY: COPY is server based, \COPY is client based.

COPY will be run by the PostgreSQL backend (user "postgres"). The backend user requires permissions to read & write to the data file in order to copy from/to it.You need to use an absolute pathname with COPY.

\COPY on the other hand, runs under the current $USER, and with that users environment. And \COPY can handle relative pathnames. The psql \COPY is accordingly much easier to use if it handles what you need.

postgres tips: How to Execute PostgreSQL Commands Inside Unix Shell Scripts

I've used to use -c option (method1) when execute postgreSQL command inside unix shell scripts. However, method1 supports single postgreSQL command and requires backslash '\' when postgreSQL is written by multiple lines. On the contrary, method2 supports multiple command at a time and does not require any backslash 8^). I think method2 is better than what I used to do(method1)

Method1: Using -c option

#!/bin/sh

dbname="mydb"
dbuser="postgres"
table1="test1"
table2="test2"

psql -U ${dbuser} -d ${dbname} -c "
DROP TABLE ${table1};"
psql -U ${dbuser} -d ${dbname} -c "
DROP TABLE ${table2};"
psql -U ${dbuser} -d ${dbname} -c "
CREATE TABLE ${table1} \
  (id int, \
  name text);"

psql -U ${dbuser} -d ${dbname} -c "
CREATE TABLE ${table2} \
  (id int, \
  name text);"

Method2: Using EOF operator

#!/bin/sh

dbname="mydb"
dbuser="postgres"
table1="test1"
table2="test2"

psql -d ${dbname} -U ${dbuser} << EOF
    DROP TABLE ${table1};
    DROP TABLE ${table2};
    CREATE TABLE ${table1}
      (id int,
      name text);
    CREATE TABLE ${table2}
      (id int,
      name text);
EOF

postgres tips: postgis operations

What is PostGIS?
PostGIS adds support for geographic objects to the PostgreSQL object-relational database. In effect, PostGIS "spatially enables" the PostgreSQL server, allowing it to be used as a backend spatial database for geographic information systems (GIS), much like ESRI's SDE or Oracle's Spatial extension

PostGIS Initialization

createdb ${dbname}
createlang plpgsql ${dbname}
psql -d ${dbname} -f /usr/share/postgresql/8.4/contrib/postgis-1.5/postgis.sql
psql -d ${dbname} -f /usr/share/postgresql/8.4/contrib/postgis-1.5/spatial_ref_sys.sql
psql -d ${dbname} -f /usr/share/postgresql/8.4/contrib/postgis_comments.sql

PostGIS Tips

1. Create geometry data from x, y

-- 4326 means WGS84
CREATE TABLE sample_data AS
  SELECT
    ST_GeomFromText('POINT('||x||' '||y||')', 4326) AS the_geom
  FROM
    ${table}

2. Create spatial index to geometry data

-- GiST: Generalized Search Trees Indexes
CREATE INDEX
  geom_index
ON
  sample_data
USING GIST (the_geom);"

3. Create intersection and its area

-- intersection(geom_1, geom_2) returns intersection
CREATE TABLE intersections AS
  SELECT
    Area(intersection(a.the_geom, b.the_geom)) AS area,
    intersection(a.the_geom, b.the_geom) AS the_geom
  FROM
    geom_data_1 as a
  INNER JOIN
    geom_data_2 as b
  ON
    ST_Intersects(a.the_geom, b.the_geom);

shell script tips: date command

"date" command is useful when we write a shell script which does per-day-processing.

#/bin/sh

fromdate="20110101"
todate="20110501"

date=${fromdate}
while [ ${date} -le ${todate} ]
do
  # do somewhat you want

  date=`date -d "$(date +2011-06-30) +1 day" +%Y%m%d`
done