ipython setup on windows

Your life on windows platform will be much more convenient with ipython. Here is a short memo to setup ipython with enhanced console.

$ python ez_setup.py

$ easy_install pip

$ pip install pyreadline

$ pip install ipython

public key authentication

Sometimes you would like to login remote system without password authentication. You can make use of public key authentication for this sake.

Here is a short memo to set up local & remote system.

Let us generate rsa key in case that you have none on localhost.

$ ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (~/.ssh/id_rsa): <ENTER>
Enter passphrase (empty for no passphrase): <ENTER>
Enter same passphrase again: <ENTER>
Your identification has been saved in ~/.ssh/id_rsa.
Your public key has been saved in ~/.ssh/id_rsa.pub.
The key fingerprint is:
ab:6b:44:71:2f:48:22:09:85:d2:1b:18:a5:93:03:97 USER@HOST

As seen, private key (id_rsa) and public key (id_rsa.pub) is generated respectively.

$ file .ssh/*
.ssh/id_rsa:      PEM RSA private key
.ssh/id_rsa.pub:  OpenSSH RSA public key
.ssh/known_hosts: ASCII text

Now create .ssh directory on target host.

$ ssh USER@TARGET_HOST mkdir -p .ssh

Copy public key of localhost onto remote target host and save it as authorized_keys so that remote host shall recognize localhost as authorized one.

$ cat .ssh/id_rsa.pub |ssh USER@TARGET_HOST 'cat >> .ssh/authorized_keys'

Now you can connect target host without password. Here is an excerpt from debug message yielded by ssh command!

debug1: Authentications that can continue: publickey,password
debug1: Next authentication method: publickey
debug1: Offering RSA public key: ~/.ssh/id_rsa
debug1: Server accepts key: pkalg ssh-rsa blen 279
debug1: read PEM private key done: type RSA
debug1: Authentication succeeded (publickey).
Authenticated to TARGET_HOST ([TARGET_HOST]:22).

automatic upgrade

Sometimes you would like to configure your ubuntu host to upgrade automatically without any manual intervention.

You can achieve this by unattended-upgrades package. Here is a short memo to do so.

Install unattended-upgrades package (if not there).

$ sudo apt-get install unattended-upgrades

Enable automatic upgrade.

$ sudo dpkg-reconfigure unattended-upgrades

You will be prompted and see message like this.

Applying updates on a frequent basis is an important part of keeping systems secure. By default, updates need to be applied manually using package management tools. Alternatively, you can choose to have this system automatically download and install security updates.
Automatically download and install stable updates?

After enabling unattended-upgrades, you will have configuration like this.

$ more /etc/apt/apt.conf.d/20auto-upgrades
APT::Periodic::Update-Package-Lists "1";
APT::Periodic::Unattended-Upgrade "1";

According to history records yielded in /var/log/apt/history.log, it seems that automatic update may take place around 08:00.

Start-Date: 2013-12-05  07:58:10
Install: linux-headers-3.11.0-14-generic:amd64 (3.11.0-14.21, automatic), linux-headers-3.11.0-14:amd64 (3.11.0-14.21, automatic), linux-image-extra-3.11.0-14-generic:amd64 (3.11.0-14.21, automatic), linux-image-3.11.0-14-generic:amd64 (3.11.0-14.21, automatic)
End-Date: 2013-12-05  08:00:54

vmplayer on ubuntu 13.10

I have recently upgrade ubuntu machine from 13.04 to 13.10. (As has been anticipated,) vmplayer would not start up stating that it requires to build.

Ok, let us build as we have been doing for a while upon upgrade of Kernel.

$ uname -r
3.11.0-13-generic
$ sudo ln -s /usr/src/linux-headers-`uname -r`/include/generated/uapi/linux/version.h /usr/src/linux-headers-`uname -r`/include/linux/version.h
$ sudo vmware-modconfig --console --install-all

Mmmmm, something strange. Build would not succeed and abort in the middle.

Googling with this symptom and found similar report.

Ok, I need to patch some code change before building vmplayer module. Here is what I followed.

Download following files at first.

• procfs.patch
• vmblock.3.10.patch
• vmblock.3.11.patch

And then,

$ cd /usr/lib/vmware/modules/source
$ sudo tar -xf vmnet.tar
$ sudo tar -xf vmblock.tar
$ cd vmnet-only
$ sudo patch -p1 < ~/Downloads/procfs.patch
$ cd ../vmblock-only
$ sudo patch -p1 < ~/Downloads/vmblock.3.10.patch
$ sudo patch -p1 < ~/Downloads/vmblock.3.11.patch
$ cd ..
$ sudo tar -cf vmblock.tar vmblock-only
$ tar -cf vmnet.tar vmnet-only
$ vmware-modconfig --console --install-all

Ok, now my vmplayer comes up without any problem. And original issue has been reported in vm communities.

cifs continued

In addition to previous post as to CIFS.

You may encounter “permission denied” error upon adding or modifying files.

You can avoid this error by adding following options in /etc/fstab.

uid=<user>,gid=<group>

Replace each value respectively by your own user and default group.

//<MS_HOST>/<ShareName>  /windows/server1  cifs credentials=<HOME>/cifs/passwd,uid=<user>,gid=<group>,iocharset=utf8,sec=ntlm  0  0

root your android

Sometimes you would like to obtain root privilege on android device. For example, in case that you install applications, which requires root privilege (e.g. bash).

I hereby share with you a quick procedure to do so.

Please be note that unlocking boot loader will lose warrant of your device. I remind you that this procedure shall be done totally under your own risk :-)

$ sudo fastboot oem unlock

$ fastboot boot image/CF-Auto-Root-<device_name>-nakasi-nexus<*>.img

mount windows file system remotely

Sometime you would like to mount Windows File System from ubuntu so that you are able to Windows files by making use of command line tools of linux.

You can access via CIFS and here is a quick procedure to do so.

$ net share <shareName>

$ sudo apt-get install cifs-utils

$ sudo mkdir -p /windows/server1

username=<MS_UID>
password=<MS_PASSWD>

//<MS_HOST>/<ShareName>  /windows/server1  cifs credentials=<HOME>/cifs/passwd,iocharset=utf8,sec=ntlm  0  0

$ sudo mount -v -a

view next line of matched word

You sometimes would like to show the very next line(s), which contain specific word(s).

I have to confess that I did not know there is a useful option in grep to achieve this sake... I have been using, say, some script to extract lines, which I would like to check.

You can achieve this with -A option. For example, if you would like to show the next line, which contains specific word, then following command give you the answer:

$ grep <WORD> -A 1 <FILE>

If you would like to see more line(s), increase argument after -A option.

So easy...

a X window trouble

When I connected from localhost to an instance of ubuntu in lxc, I encountered following message upon login via ssh.

X11 forwarding request failed on channel 0

That is, I connected to this host with -X option for the purpose of X forwarding. (so as to use gvim)

$ ssh -X -l ubuntu <lxc_instance_IP>

Let us dig into further by adding verbose option, -v as argument of ssh.

$ ssh -X -v -l ubuntu <lxc_instance_IP>

Then I found following debug message, which states that relevant xauth application does not exist.

debug1: Remote: No xauth program; cannot forward with spoofing.

Ok, let us install xauth on target host side.

$ sudo apt-get install xauth

After installation of xauth, this message never comes up. And I am able to forward X sessions to host without any problem.

insert characters in multiple lines

Sometime you would like to insert same characters into multiple lines. I have to confess that I’ve been using sed command to achieve this, say,

$ sed -e "s/^/> /g" in.txt > out.txt

You can do the same within vim.

• press ctrl + v
• press shift + i (I)
• input character(s) of your choice
• press esc

So easy...

start mongo

It occurred to me that I want some place to store data like CSV file. Ok, let us use NoSQL like mongodb.

Like other tools, set up process is quite easy for anyone.

Installation requires only one line of command execution. It depends upon existing packages though, it may take time to download all the necessary package.

$ sudo apt-get install mongodb

Installation will automatically starts up daemon, mongod.

$ ps -ef|grep mongod
mongodb    759     1  0 03:47 ?        00:00:27 /usr/bin/mongod --config /etc/mongodb.conf

Configuration file shall be /etc/mongodb.conf. According to configuration file, database repository seems to reside under /var/lib/mongodb directory.

# Where to store the data.
dbpath=/var/lib/mongodb

mongod listesn on default port, TCP 27107.

$ sudo lsof -nPi:27017
COMMAND PID    USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
mongod  759 mongodb    6u  IPv4 113195      0t0  TCP 127.0.0.1:27017 (LISTEN)

OK, let us import CSV file into mongodb. You can import CSV file into collection, an quivalent of page in spreadsheet(?), under database.

$ mongoimport -d <DB_NAME> -c <COLLECTION_NAME> --type csv --file <CSV_FILE> --headerline

After installation, you will check its content from interactive shell.

$ mongo
MongoDB shell version: 2.2.4
connecting to: test
> use <DB_NAME>
switched to db salary
> db.<COLLECTION_NAME>.find().pretty()

port forwarding

Now you have working wiki in lxc network. Let’s configure instances to launch automatically upon system boot and can be accessed from outside network.

lxc Instance be started by creating symbolic link under /etc/lxc/auto directory, which points config file for each instance.

$ sudo ln -s /var/lib/lxc/wiki/config /etc/lxc/auto/wiki.conf
$ sudo ln -s /var/lib/lxc/proxy/config /etc/lxc/auto/proxy.conf

You will see AUTOSTART flag as set to YES.

$ sudo lxc-ls --fancy
NAME   STATE    IPV4       IPV6  AUTOSTART
------------------------------------------
proxy  RUNNING  10.0.3.20  -     YES
wiki   RUNNING  10.0.3.21  -     YES

By default, no one can access from outside.

$ sudo iptables -t nat -L
Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination

Configure routing tables so that access to TCP port 443 shall be forwarded to reverse proxy instance.

$ sudo iptables -t nat -A PREROUTING -i eth0 -p tcp --dport 443 -j DNAT --to 10.0.3.20:443
$ sudo iptables -t nat -L
Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination
DNAT       tcp  --  anywhere             anywhere             tcp dpt:https to:10.0.3.20:443

Now you can access wiki by IP address of host.

This change of routing table is not persistent and will be lost upon reboot. So, for example, add such lines as follows in /etc/rc.local file.

NETWORK_IF=eth0
WIKI_IP=10.0.3.20
WIKI_PORT=443

iptables -t nat -A PREROUTING -i $NETWORK_IF -p tcp --dport $WIKI_PORT -j DNAT --to $WIKI_IP:$WIKI_PORT

caching content on nginx

You can cache content on front-end reverse proxy so as to provide content more effectively for requests and reduce traffic against back-end web server.

Let us get started.

http {

    ##
    # cache config
    ##
    proxy_cache_path /tmp/nginx/cache levels=1 keys_zone=wiki:4m inactive=1h max_size=10m;

    include /etc/nginx/conf.d/\*.conf;
    include /etc/nginx/sites-enabled/\*;
}

/tmp/nginx/cache
├── 0
│   └── 64b08f8ec9459d892a1a80bea5d2d400
├── 1
│   └── 4bbed59225358625d11842e1ec069b81
├── 2
│   └── 47c32bbfbc8c08c9047c8a8271893f02

location / {

    proxy_cache       wiki;
    proxy_cache_valid 200 10m;
}

password authentication on nginx

I hereby demonstrate a simple procedure to set password authentication against content on nginx, which is quite similar to that of apache.

First, you need to install apache2-utils for the purpose of creating password file.

$ sudo apt-get install apache2-utils

Create password file with htpasswd command.

$ sudo htpasswd -c /home/ubuntu/wiki/.htpasswd lupin

Incorporate newly created password file into virtual host configuration.

location / {
    auth_basic "Open Sesame!";
    auth_basic_user_file /home/ubuntu/wiki/.htpasswd;
}

Reflect modified configuration into running nginx process.

$ sudo /etc/init.d/nginx reload

enable encryption on nginx

Sometimes you would like to configure nginx instance to communicate with clients in secured connection. You can issue self signed certificate and make use of it for encryption.

Here is a quick procedure to do it.

$ cd /etc/nginx
$ sudo mkdir cert
$ cd cert

$ sudo apt-get install openssl
$ sudo openssl genrsa -des3 -out server.key 1024
Generating RSA private key, 1024 bit long modulus
......................................++++++
............................++++++
e is 65537 (0x10001)
Enter pass phrase for server.key:
Verifying - Enter pass phrase for server.key:
$ file server.key
server.key: PEM RSA private key

$ sudo openssl req -new -key server.key -out server.csr
$ file server.csr
server.csr: PEM certificate request

$ sudo cp server.key server.key.org
$ sudo openssl rsa -in server.key.org -out server.key

$ sudo openssl x509 -req -days 365 -in server.csr -signkey server.key -out server.cert
$ file server.cert
server.cert: PEM certificate

server {

    listen 443 ssl;

    ssl on;
    ssl_certificate     cert/server.cert;
    ssl_certificate_key cert/server.key;

}

$ sudo lsof -nPi:443
COMMAND PID     USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
nginx   385     root    8u  IPv4 127726      0t0  TCP \*:443 (LISTEN)
nginx   387 www-data    8u  IPv4 127726      0t0  TCP \*:443 (LISTEN)

reverse proxy

Create an instance of plain reverse proxy instance with nginx. It’s quite simple.

Create an instance of ubuntu container as usual. Install nginx and add virtual host entry, say, “reverse” under /etc/nginx/sites-available.

server {
    listen 80;

    access_log  /home/ubuntu/reverse/logs/access.log;
    error_log   /home/ubuntu/reverse/logs/error.log debug;

    location / {
        proxy_set_header X-Real-IP  $remote_addr;
        proxy_set_header X-Forwarded-For $remote_addr;
        proxy_set_header Host $host;
        proxy_pass http://10.0.3.21:80/;
    }
}

Create symbolic link under /etc/nginx/sites-enabled directory to point it. And remove default symbolic link.

Make certain that you create log directory for nginx instance.

$ cd
$ mkdir -p reverse/logs

That’s all

quick wiki configuration

Now you have container to store your content. So let us setup sphinx so that we can deploy our contents onto it.

$ sudo apt-get install build-essential
$ sudo apt-get install python-setuptools
$ sudo easy_install pip

$ sudo apt-get install python-sphinx
$ sudo pip install sphinxcontrib-blockdiag sphinxcontrib-nwdiag sphinxcontrib-seqdiag sphinxcontrib-actdiag
$ sudo apt-get install python-matplotlib
$ sudo apt-get install ttf-dejavu

#extensions = []
extensions = ['sphinxcontrib.blockdiag',
              'sphinxcontrib.nwdiag',
              'sphinxcontrib.seqdiag',
              'sphinxcontrib.actdiag']

blockdiag_fontpath = '/usr/share/fonts/truetype/ttf-dejavu/DejaVuSansMono.ttf'
nwdiag_fontpath    = '/usr/share/fonts/truetype/ttf-dejavu/DejaVuSansMono.ttf'
seqdiag_fontpath   = '/usr/share/fonts/truetype/ttf-dejavu/DejaVuSansMono.ttf'
actdiag_fontpath   = '/usr/share/fonts/truetype/ttf-dejavu/DejaVuSansMono.ttf'

$ sudo apt-get install inotify-tools

#!/usr/bin/env bash

ROOT=/home/ubuntu/wiki
WATCH=source

cd $ROOT

while inotifywait -r -e create,modify,delete $WATCH; do
    sphinx-build -b html -d build/doctrees source build/html
done

#!/usr/bin/env bash

nohup /home/ubuntu/script/build.sh 1> /dev/null 2>&1 &

static web server in lxc

Let us create web site in lxc environment. That is, with rough design as follows.

$ sudo lxc-create -n web -t ubuntu
$ sudo lxc-start -n web -d
$ ssh -l ubuntu `cut -d " " -f3 /var/lib/misc/dnsmasq.lxcbr0.leases`

$ sudo ln -sf /usr/share/zoneinfo/Asia/Tokyo /etc/localtime
$ tail -8 /etc/network/interfaces
auto eth0
#iface eth0 inet dhcp
iface eth0 inet static
address 10.0.3.21
network 10.0.3.0
netmask 255.255.255.0
broadcast 10.0.3.255
gateway 10.0.3.1

$ sudo apt-get install nginx

$ tree
.
├── conf.d
├── fastcgi_params
├── koi-utf
├── koi-win
├── mime.types
├── naxsi_core.rules
├── naxsi.rules
├── nginx.conf
├── proxy_params
├── scgi_params
├── sites-available
│   └── default
├── sites-enabled
│   └── default -> /etc/nginx/sites-available/default
├── uwsgi_params
└── win-utf

$ diff nginx.conf nginx.conf.org
2c2
< worker_processes 1;
---
> worker_processes 4;

$ update-rc.d nginx defaults

server {
    listen 80;

    access_log  /home/ubuntu/wiki/logs/access.log;
    error_log   /home/ubuntu/wiki/logs/error.log debug;

    location / {
        root   /home/ubuntu/wiki/build/html/;
        index  index.html;
    }
}

$ sudo ln -s /etc/nginx/sites-available/wiki .
$ rm -i default

search for package

You may encounter such a situation that some shared libraries are stripped when you debug core file. For example,

(gdb) info shared
From        To          Syms Read   Shared Object Library
0xb778f430  0xb77a6054  Yes (*)     /lib/i386-linux-gnu/libncurses.so.5
0xb75ef350  0xb7728f4e  Yes         /lib/i386-linux-gnu/libc.so.6
0xb75d2ad0  0xb75d3a9c  Yes         /lib/i386-linux-gnu/libdl.so.2
0xb75b87d0  0xb75c2c24  Yes (*)     /lib/i386-linux-gnu/libtinfo.so.5
0xb77c9830  0xb77e1e7c  Yes         /lib/ld-linux.so.2
(*): Shared library is missing debugging information.

You would like to search for package, which relevant shared object is included. With library, which contains symbol table and you can examine more closely core file.

$ sudo apt-get install apt-file

$ apt-file update

$ apt-file search libncurses.so.5
lib64ncurses5: /lib64/libncurses.so.5
lib64ncurses5: /lib64/libncurses.so.5.9
libncurses5: /lib/i386-linux-gnu/libncurses.so.5
libncurses5: /lib/i386-linux-gnu/libncurses.so.5.9
libncurses5-dbg: /usr/lib/debug/lib/i386-linux-gnu/libncurses.so.5.9
libncurses5-dbg: /usr/lib/debug/lib64/libncurses.so.5.9
libncurses5-dbg: /usr/lib/debug/libncurses.so.5
libncurses5-dbg: /usr/lib/debug/libncurses.so.5.9
libncurses5-dbg: /usr/lib/debug/usr/libx32/libncurses.so.5.9
libx32ncurses5: /usr/libx32/libncurses.so.5
libx32ncurses5: /usr/libx32/libncurses.so.5.9

$ sudo apt-get install libncurses5-dbg

(gdb) info shared
From        To          Syms Read   Shared Object Library
0xb778f430  0xb77a6054  Yes         /lib/i386-linux-gnu/libncurses.so.5
0xb75ef350  0xb7728f4e  Yes         /lib/i386-linux-gnu/libc.so.6
0xb75d2ad0  0xb75d3a9c  Yes         /lib/i386-linux-gnu/libdl.so.2
0xb75b87d0  0xb75c2c24  Yes         /lib/i386-linux-gnu/libtinfo.so.5
0xb77c9830  0xb77e1e7c  Yes         /lib/ld-linux.so.2

resize disk size of guest on KVM

Sometimes you regret that you created guest OS with too small disk space. sigh... And you would like to increase disk space a little.

You can do it with some steps and I hereby quote a sample procedure of such operation. In this sample, OS of guest instance is Windows 7.

$ du -sh /var/lib/libvirt/images/win7.img
21G /var/lib/libvirt/images/win7.img

$ sudo qemu-img resize /var/lib/libvirt/images/win7.img +20G
Image resized.

plotting

Sometimes you would like to create a plot out of some text file. You can achieve it by making use of matplotlib.

I have to confess that I am not quite accustomed with matplotlib though, I hereby demonstrate a sample.

$ sudo apt-get install python-matplotlib

201201,1477,994
201202,1462,988
201203,1437,985

plot

You can plot a graph, for example, as follows.

import csv
from datetime import datetime
from matplotlib import pyplot as plt
from matplotlib import dates

if __name__ == "__main__":

    raw_data = "sample.csv"
    fmt = ["prepay", "month", "bonus"]

    # prepayment date
    prepay = []
    # lists to store each values
    monthCal = []
    monthAct = []
    year     = []

    with open(raw_data, "rb") as f:
        reader = csv.reader(f)
        for row in reader:
            prepay.append(dates.date2num(datetime.strptime(row[fmt.index("prepay")], "%Y%m")))
            monthCal.append(int(row[fmt.index("month")]) + int(row[fmt.index("bonus")])/6) 
            monthAct.append(int(row[fmt.index("month")]))
            year.append(int(row[fmt.index("month")])*12 + int(row[fmt.index("bonus")])*2) 

    # let's start plotting
    fig = plt.figure()

    # monthly graph
    graph = fig.add_subplot(2, 1, 1)
    graph.plot(prepay, monthCal, "bo", prepay, monthAct, "ro")

    graph.xaxis.set_major_locator(dates.MonthLocator())
    graph.xaxis.set_major_formatter(dates.DateFormatter("%Y/%m"))
    
    plt.xticks(rotation="vertical")
    plt.ylabel("payment per month (USD)")
    plt.grid(True)

    # yearly graph
    graph = fig.add_subplot(2, 1, 2)
    graph.plot(prepay, year, "go")

    graph.xaxis.set_major_locator(dates.MonthLocator())
    graph.xaxis.set_major_formatter(dates.DateFormatter("%Y/%m"))
    
    plt.ylabel("payment per year (USD)")
    plt.xticks(rotation="vertical")
    plt.grid(True)

    plt.show()
    fig.savefig(raw_data.replace("csv", "png"))

back up files

Here is another approach for creating back up files onto dropbox on regular basis by cron. A quite simple script though, here are some remark.

import datetime
import hashlib
import os
import os.path
import shutil
import socket
import tarfile

if __name__ == "__main__":

    # base directory, which is one level upper than target directory
    base      = "<BASE_DIRECTORY_OF_YOUR_CHOICE>"
    # backup target directory under base directory
    target    = "<BACK_UP_TARGET_DIRECTORY>"
    # directories to exclude from backup file
    exclude   = ["<DIRECTORY_TO_EXCLUDE>"]

    # backup file name / HOSTNAME_DAY.tar.bz2
    backup    = socket.gethostname() + "_" +\
                datetime.date.today().strftime("%A") +\
                ".tar.bz2"
    # temporary directory & temporary file
    temp_dir  = "/tmp"
    temp_file = os.path.join(temp_dir, backup)
    # destination directory
    dest_dir  = os.path.join(base, "Dropbox/<TARGET_DIRECTORY>")
    dest_file = os.path.join(dest_dir, backup)

    # let's start backup

    # first create a backup file under temporary file
    tar = tarfile.open(temp_file, "w:bz2")

    os.chdir(base)
    for root, dirs, files in os.walk(target):
        for name in files:
            for d in root.split('/'):  
                if d in exclude:
                    break
            else:
                tar.add(os.path.join(root, name))
    
    tar.close()

    # backup file creation has finished
    # now determine if file replacement is required or not
    if os.path.exists(dest_file):
        # dictionary to store md5 checksum of each file
        md5 = {}
        for f in temp_file, dest_file:
            with open(f) as file_to_check:
                data = file_to_check.read()
                md5[f] = hashlib.md5(data).hexdigest()
        # compare checksum value of each file
        # copy temporary file under destination directory
        if md5[temp_file] != md5[dest_file]:
            shutil.copy(temp_file, dest_dir)
    else:
        # somehow backup of same name does not exist, do copy
        shutil.copy(temp_file, dest_dir)

$ crontab -l
0 * * * * python <PATH_TO_SCRIPT>/backup.py

x86 or x64?

I have to confess that I did not know CPU (AMD E2-1800) of my tiny PC (ThinkPad Edge E135) is x64 capable...

You can check whether your CPU is x64 capable or not by referencing existence of lm flag (long mode).

$ grep ^flags /proc/cpuinfo
flags               : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc extd_apicid aperfmperf pni monitor ssse3 cx16 popcnt lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch ibs skinit wdt arat hw_pstate npt lbrv svm_lock nrip_save pausefilter

If there is lm in flags line, then its CPU can be x86_64 architecture. After reinstalling OS, now I have x64 OS at hand!

$ arch
x86_64

So easy... Ok, let’s create an lxc instance on this re-born platform.

Timezone of instance seems to be in EDT, which is not convenient for us. Let us change timezone to JST (Asia/tokyo). You can do it by configuring /etc/localtime file to reference timezone data of Japan.

$ file /usr/share/zoneinfo/Asia/Tokyo
/usr/share/zoneinfo/Asia/Tokyo: timezone data, version 2, 3 gmt time flags, 3 std time flags, no leap seconds, 9 transition times, 3 abbreviation chars

$ sudo ln -sf /usr/share/zoneinfo/Asia/Tokyo /etc/localtime

static IP address assignment

Let us configure network related properties of our DNS Server host.

auto eth0
iface eth0 inet dhcp

auto eth0
iface eth0 inet static
address 10.0.3.10
network 10.0.3.0
netmask 255.255.255.0
broadcast 10.0.3.255
gateway 10.0.3.1

# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
#     DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN

dns-search example.org
dns-nameservers 10.0.3.10 10.0.3.1 8.8.8.8

nameserver 10.0.3.10
nameserver 10.0.3.1
nameserver 8.8.8.8
search example.org

change IP address range

As described in previous posts, we have some servers in lxc network now.

I would like that servers shall have static IP addresses. Say, in the range from 10.0.3.2 until 10.0.3.99.

And client shall have dynamically assigned address, which should not conflict with server’s ones. Let’s limit range of IP addresses, which DHCP Server, dnsmasq assigns for.

You can configure IP address range by modifying lxc file under /etc/default directory.

$ diff lxc lxc.org
27,28c27,28
< LXC_DHCP_RANGE="10.0.3.100,10.0.3.199"
< LXC_DHCP_MAX="100"
---
> LXC_DHCP_RANGE="10.0.3.2,10.0.3.254"
> LXC_DHCP_MAX="253"

After restart, you will see that dnsmasq will set IP address range as follows:

dnsmasq -u lxc-dnsmasq --strict-order --bind-interfaces --pid-file=/var/run/lxc/dnsmasq.pid --conf-file= --listen-address 10.0.3.1 --dhcp-range 10.0.3.100,10.0.3.199 --dhcp-lease-max=100 --dhcp-no-override --except-interface=lo --interface=lxcbr0 --dhcp-leasefile=/var/lib/misc/dnsmasq.lxcbr0.leases --dhcp-authoritative

Name Server

Now let’s move on to Name Server. Sometimes you would like to have your own name services to manipulate protocol like SMTP.

I hereby demonstrate procedure to set up BIND (version 9). In this example, we use fictitious domain, example.org.

We disregard redundancy and focus on primary server. That is, we omit secondary server :-).

By the way, following articles are good reference for you.

• Redhat Doc
• Ubuntu Doc

In this sample, we Ubuntu as platform. And goal is to setup instance as follows.

$ sudo apt-get install bind9 dnsutils

├── bind.keys
├── db.0
├── db.127
├── db.255
├── db.empty
├── db.local
├── db.root
├── named.conf
├── named.conf.default-zones
├── named.conf.local
├── named.conf.options
├── rndc.key
└── zones.rfc1918

zone "example.org" {
        type master;
    file "/etc/bind/db.example.org";
};

;
; BIND data file for example.org
;
$TTL    604800
@       IN      SOA     example.org. root.example.org. (
                       20130928         ; Serial
                         604800         ; Refresh
                          86400         ; Retry
                        2419200         ; Expire
                         604800 )       ; Negative Cache TTL
        IN      A       10.0.3.10
;
@       IN      NS      dns.example.org.
@       IN      A       10.0.3.10
@       IN      AAAA    ::1
dns     IN      A       10.0.3.10
;MTA
        IN      MX 10   mta.example.org.
mta     IN      A       10.0.3.20

zone "3.0.10.in-addr.arpa" {
        type master;
        file "/etc/bind/db.10";
};

;
; BIND reverse data file for network 10.0.3.0
;
$TTL    604800
@       IN      SOA     example.org. root.example.org. (
                       20130928         ; Serial
                         604800         ; Refresh
                          86400         ; Retry
                        2419200         ; Expire
                         604800 )       ; Negative Cache TTL
;
@       IN      NS      dns.
10      IN      PTR     dns.example.org.
20      IN      PTR     mta.example.org.

logging {
    channel query.log {
        file "/var/log/query.log";
        severity debug 3;
    };
    category queries { query.log; };
};

$ sudo touch /var/log/query.log
$ sudo chown bind /var/log/query.log

client 127.0.0.1#34060 (mta.example.org): query: mta.example.org IN A +E (127.0.0.1)

$ sudo service bind9 restart

Directory Server

Sometimes you may (or may not) want to have a Directory Server at your hand for, say, storing your addresses. Let us try if we can have Directory Server in virtual instance.

In this scenario, we use x86(i686) machine as host (not x64).

Download x86 version of Directory Server installer (V19710-01.zip) from OTN. And deploy it onto target virtual machine.

$ scp V19710-01.zip root@10.0.3.xxx:~/

As virtual machine have very limited number of tools, therefore you need to install basic commands like tar/unzip. Following sample will install Directory Server under /home/dsee7 directory.

$ sudo yum install unzip tar
$ unzip V19710-01.zip
$ tar xfv DSEE.7.0.Linux-X86-zip.tar.gz
$ DSEE_ZIP_Distribution/
$ unzip -d /home sun-dsee7.zip

Ok, installation finished. Now let’s move on to configuration.

Let’s create instance.

$ ./dsadm create /home/dsee7/instance
Choose the Directory Manager password:
Confirm the Directory Manager password:
Use 'dsadm start '/home/dsee7/instance'' to start the instance

Now you may encounter problem that instance start fails.

$ ./dsadm start ../instance/
ERROR<4167> - Startup  - conn=-1 op=-1 msgId=-1 - System error  Load library /home/dsee7/lib/pwdstorage-plugin.so: error /home/dsee7/lib/../lib/private/libfreebl3.so: version 'NSSRAWHASH_3.12.3' not found (required by /lib/libcrypt.so.1)

libcrypt.so is dependent upon NSS.

$ ldd pwdstorage-plugin.so
./pwdstorage-plugin.so: /home/dsee7/lib/./../lib/private/libfreebl3.so: version 'NSSRAWHASH_3.12.3' not found (required by /lib/libcrypt.so.1)

And libfreebl3.so seems not to have it.

$ find / -name libfreebl3.so -ls
27270228  320 -rwxr-xr-x   1 root     root       325256 Aug  7 16:17 /lib/libfreebl3.so
27660408  364 -rwxr-xr-x   1 root     root       372385 Aug 27  2009 /home/dsee7/lib/private/libfreebl3.so
27791820    0 lrwxrwxrwx   1 root     root           23 Sep 22 20:03 /usr/lib/libfreebl3.so -> ../../lib/libfreebl3.so

$ objdump -x /lib/libfreebl3.so |grep NSSRAWHASH_3.12.3
3 0x00 0x04ceacd3 NSSRAWHASH_3.12.3
$ objdump -x  /home/dsee7/lib/private/libfreebl3.so |grep NSSRAWHASH_3.12.3
$ 

As temporary workaround, configure libfreebl3.so to reference one, which OS provides.

$ ls -l libfreebl3.so*
lrwxrwxrwx 1 root root     18 Sep 24 04:37 libfreebl3.so -> /lib/libfreebl3.so
-rwxr-xr-x 1 root root 372385 Aug 27  2009 libfreebl3.so.org

Now you can start daemon.

$ ./dsadm start ../instance/
Directory Server instance '/home/dsee7/instance' started: pid=523

Let’s create a suffix to store entires.

$ ./dsconf create-suffix  "dc=lupin, dc=org"

CentOS on ubuntu

Ok, let us create a RedHat clone instance, say, cent OS on ubuntu. So as to manipulate RPM packages, you need to install curl and yum package.

$ sudo apt-get install curl yum

By default, there is no lxc template for Cent OS. You need to deploy lxc template manually. For example,

$ sudo wget -O /usr/share/lxc/templates/lxc-centos https://gist.github.com/hagix9/3514296/raw/7f6bb4e291fad1dad59a49a5c02f78642bb99a45/lxc-centos
$ sudo chmod 755 /usr/share/lxc/templates/lxc-centos

You need some adjustment in case that architecture of you machine is i686.

$ arch
i686
$ diff /usr/share/lxc/templates/lxc-centos /usr/share/lxc/templates/lxc-centos.org
170,171c170
<         #RELEASE_URL="$MIRROR_URL/Packages/centos-release-$release-$releaseminor.el6.centos.10.$arch.rpm"
<         RELEASE_URL="$MIRROR_URL/Packages/centos-release-$release-$releaseminor.el6.centos.10.i686.rpm"
---
>         RELEASE_URL="$MIRROR_URL/Packages/centos-release-$release-$releaseminor.el6.centos.10.$arch.rpm"

Ok, now you are ready to start instance creation.

$ sudo lxc-create -t centos -n centos01

Please be noted that first time execution may take time, as this will download substantial amount of PRM packages.

After successful creation of instance, you can launch it and login via ssh. By the way, default root password is password.

$ sudo lxc-start -n centos01 -d
$ ssh -l root `cut -d " " -f3 /var/lib/misc/dnsmasq.lxcbr0.leases`

lxc continued

Let us create virtual machine of your choice. Suppose that you would like to create an instance of ubuntu. (Please do not ask me reason why I need to run ubuntu on unbutu :-)).

$ sudo lxc-create -t ubuntu -n ubuntu01

As this command execution will download files, therefore first time execution may take time. So please wait for a while, say, by drinking coffee.

After successful execution, you will have an ubuntu instance named “ubuntu01”.

Ok, let us start lxc as daemon.

$ sudo lxc-start -n ubuntu01 -d

According to dhcp server (dnsmasq), 10.0.3.198 may have been leased.

$ cat /var/lib/misc/dnsmasq.lxcbr0.leases
1379835957 00:16:3e:08:50:7f 10.0.3.198 ubuntu01 *

Let us login via ssh. Default uid/password combination is ubuntu/ubuntu.

$ ssh -l ubuntu 10.0.3.198

Ok, network plan would be like this.

lxc introduction

You may be tempted (or may be not) to create lightweight virtual machine on you host. lxc (LinuX Container) must be one of your options.

Installing lxc package will install all dependent packages.

$ sudo apt-get install lxc

virtual machine is manipulated by lxc- commands.

$ lxc-<TAB><TAB>
lxc-aa-custom-profile  lxc-clone              lxc-execute            lxc-list               lxc-restart            lxc-unfreeze
lxc-attach             lxc-console            lxc-freeze             lxc-ls                 lxc-shutdown           lxc-unshare
lxc-cgroup             lxc-create             lxc-halt               lxc-monitor            lxc-start              lxc-version
lxc-checkconfig        lxc-destroy            lxc-info               lxc-netstat            lxc-start-ephemeral    lxc-wait
lxc-checkpoint         lxc-device             lxc-kill               lxc-ps                 lxc-stop

You will see lxc related configuration files under /etc/init directory.

$ ls /etc/init/lxc*
/etc/init/lxc-instance.conf  /etc/init/lxc-net.conf  /etc/init/lxc.conf

You will notice that new virtual interface is added and activated.

$ ifconfig lxcbr0
lxcbr0    Link encap:Ethernet  HWaddr a6:1a:10:32:67:87
          inet addr:10.0.3.1  Bcast:10.0.3.255  Mask:255.255.255.0
          inet6 addr: fe80::a41a:10ff:fe32:6787/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:84 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:0 (0.0 B)  TX bytes:13896 (13.8 KB)

And you will see that DHCP server is already running as well.

dnsmasq -u lxc-dnsmasq --strict-order --bind-interfaces --pid-file=/var/run/lxc/dnsmasq.pid --conf-file= --listen-address 10.0.3.1 --dhcp-range 10.0.3.2,10.0.3.254 --dhcp-lease-max=253 --dhcp-no-override --except-interface=lo --interface=lxcbr0 --dhcp-leasefile=/var/lib/misc/dnsmasq.lxcbr0.leases --dhcp-authoritative

library details

Sometime you feel like (or may not) checking details of certain library on you machine. Suppose that the relevant library may be X related library.

Let us check and search for library name at first. And apt-cache command is your friend.

$ apt-cache search libx11
libx11-6 - X11 client-side library
libx11-6-dbg - X11 client-side library (debug package)

Ok, lib11-6 is what we are looking for. Let us check its content.

$ apt-cache show libx11-6
Package: libx11-6
Priority: standard
Section: libs
Installed-Size: 1489
Maintainer: Ubuntu Developers <ubuntu-devel-discuss@lists.ubuntu.com>
Original-Maintainer: Debian X Strike Force <debian-x@lists.debian.org>
Architecture: i386
Source: libx11
Version: 2:1.5.0-1ubuntu1.1
Depends: libc6 (>= 2.15), libxcb1 (>= 1.2), libx11-data
Pre-Depends: multiarch-support
Filename: pool/main/libx/libx11/libx11-6_1.5.0-1ubuntu1.1_i386.deb
Size: 776896
MD5sum: 3fea2137a989c9cf840c7e0db9b91606
SHA1: 5fcc7ab89b79687fb338e169622a2c57912bcb60
SHA256: 170ce1631c61458216078415ad9d660df98e193c18c4774d45432cb366e64c75
Description-en: X11 client-side library
 This package provides a client interface to the X Window System, otherwise
 known as 'Xlib'.  It provides a complete API for the basic functions of the
 window system.
 .
 More information about X.Org can be found at:
 <URL:http://www.X.org>
 .
 This module can be found at
 git://anongit.freedesktop.org/git/xorg/lib/libX11
Multi-Arch: same
Description-md5: d75c895abf6eca234f7480813aaa95ec
Bugs: https://bugs.launchpad.net/ubuntu/+filebug
Origin: Ubuntu
Supported: 9m
Task: standard, kubuntu-active, kubuntu-active, mythbuntu-frontend, mythbuntu-frontend, mythbuntu-desktop, mythbuntu-backend-slave, mythbuntu-backend-slave, mythbuntu-backend-master, mythbuntu-backend-master

You can download source as described.

$ $ git clone git://anongit.freedesktop.org/git/xorg/lib/libX11

network diagram in document

Sometimes you would like to draw network diagram and incorporate it into your document.

It’s so easy with sphinx and nwdiag. For example, prepare such a text as follows and save it as, say, net.diag.

nwdiag {
    internet [shape = cloud];
    internet -- router;

    network eva{
        address = "192.168.10.0/24";
        router;
        Nerv   [address = "192.168.10.10"];
        Seele  [address = "192.168.10.11"];
        Gehirn [address = "192.168.10.12"];
    }

    network theLupin{
        address = "192.168.20.0/24";
        router;
        lupin   [address = "192.168.20.10"];
        fujiko  [address = "192.168.20.11"];
        jigen   [address = "192.168.20.12"];
    }
}

And merge this file into your sphinx document with as follows.

.. nwdiag:: net.diag

Then you will see beautiful network diagram as follows.

USB devices

Sometimes you would like to reference USB devices, which are attached to your machine. You can make use of lsusb utility for this sake.

For example,

$ lsusb
Bus 002 Device 003: ID 5986:0299 Acer, Inc
Bus 004 Device 002: ID 0a5c:21f4 Broadcom Corp.
Bus 006 Device 008: ID 0bb4:0ffe HTC (High Tech Computer Corp.) Desire HD (modem mode)

In case that you would like to check further details of specific USB device, for example, Desire, then you can specify use verbose option, -v.

$ sudo lsusb -v -s 008

By the way, lsusb command accesses file under /dev/bus/usb directory, which requires root permission. Therefore if you execute command as user other than root without sudo, then you will see open failure.

open("/dev/bus/usb/006/008", O_RDWR)    = -1 EACCES (Permission denied)

copy intermediate directories

Sometimes you would like to cp files under directories, which does not exist on target directory. Current cp command does support this kind of operation.

Suppose that you have a file, which contains list of files with directory. You can copy files in it, for example, as follows.

$ mkdir <TARGER_DIR>
$ while read LINE
> do
> cp -p --parents $LINE <TAGET_DIR>
> done < <INPUT_FILE>

You will see that directory strctures are automatically created under target directory.

So easy...

Please refer to gnu page for details.

password lock of screensaver

By default, gnome-screen runs on ubuntu and password lock is enabled.

$ ps -ef|grep -i screen
lupin   2000     1  0 15:44 ?        00:00:00 /usr/bin/gnome-screensaver --no-daemon

Sometimes you may feel password lock troublesome and would like to disable it. You can achieve it by gsettings command.

Current configuration can be checked,

$ gsettings get org.gnome.desktop.screensaver lock-enabled
true

You can disable password lock by setting lock-enabled value as false,

$ gsettings set org.gnome.desktop.screensaver lock-enabled false

Checking system calls done by this command, screensaver configuration seems to be stored in,

• /run/user/<UID>/dconf/user (may be for running process)
• /<HOME>/.config/dconf/user (for permanent configuration)

You can lock screen by “CTRL” + “ALT” + “L”.

source code reference

Now we have clear precise stack. Here is snippet of stack for relevant thread.

(gdb) where
#0  0xb7705424 in __kernel_vsyscall ()
#1  0xb7504b1f in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
#2  0xb75080b3 in __GI_abort () at abort.c:90
#3  0xb74fd877 in __assert_fail_base (fmt=0xb385fe90 <Address 0xb385fe90 out of bounds>, assertion=assertion@entry=0xb60f8419 "ret != inval_id",
    file=file@entry=0xb60f838a "../../src/xcb_io.c", line=line@entry=529, function=function@entry=0xb60f849e <__PRETTY_FUNCTION__.14075> "_XAllocID")
    at assert.c:92
#4  0xb74fd927 in __GI___assert_fail (assertion=assertion@entry=0xb60f8419 "ret != inval_id", file=file@entry=0xb60f838a "../../src/xcb_io.c",
    line=line@entry=529, function=function@entry=0xb60f849e <__PRETTY_FUNCTION__.14075> "_XAllocID") at assert.c:101
#5  0xb608149f in _XAllocID (dpy=0xaf43e80) at ../../src/xcb_io.c:529

As seen, some assertion may have failed and program aborted by itself. Before applying debug information, which contains symbol table, frame 5 looked like this.

#5  0xb608149f in _XAllocID () from /usr/lib/i386-linux-gnu/libX11.so.6

So let’s download source code of libX11 for reference.

$ apt-get source libx11

It’s so easy... Source files are downloaded under current working directory.

$ ls
libx11-1.5.0  libx11_1.5.0-1ubuntu1.1.diff.gz  libx11_1.5.0-1ubuntu1.1.dsc  libx11_1.5.0.orig.tar.gz

You can configure gdb to reference source code, for example, by “set substitute-path”. But for this case ”../../src” does not work for me...

So quite primitive way though,

$ cd libx11-1.5.0/
$ mkdir -p hoge/hoge
$ cd hoge/hoge/
$ ls ../../src/xcb_io.c
../../src/xcb_io.c
$ gdb <HOME>/.dropbox-dist/dropbox /var/cores/core.dropbox.2147.*

Now you can see relevant section of code and variables.

(gdb) frame 5
#5  0xb608149f in _XAllocID (dpy=0xaf43e80) at ../../src/xcb_io.c:529
529         dpy->xcb->next_xid = inval_id;
(gdb) list
524 /* _XAllocID - resource ID allocation routine. */
525 XID _XAllocID(Display *dpy)
526 {
527         XID ret = dpy->xcb->next_xid;
528         assert (ret != inval_id);
529         dpy->xcb->next_xid = inval_id;
530         _XSetPrivSyncFunction(dpy);
531         return ret;
532 }
533
(gdb) print inval_id
$1 = 4294967295
(gdb) print dpy->xcb->next_xid
$2 = 4294967295

symbol table reference

Now the time has come that we can debug core file. In this case, dropbox seems to have yielded core file.

$ file /var/cores/core.dropbox.2172.1378900464
/var/cores/core.dropbox.2172.1378900464: ELF 32-bit LSB core file Intel 80386, version 1 (SYSV), SVR4-style, from '<HOME>/.dropbox-dist/dropbox'

But some dependent libraries (e.g. libc.so) are stripped and they do not have symbol table.

$ file /lib/i386-linux-gnu/libc-2.17.so
/lib/i386-linux-gnu/libc-2.17.so: ELF 32-bit LSB shared object, Intel 80386, version 1 (SYSV), dynamically linked (uses shared libs), BuildID[sha1]=0x81a55e819c61f581e6a9179eaf59726dd80aea31, for GNU/Linux 2.6.24, stripped
$ objdump -t /lib/i386-linux-gnu/libc-2.17.so

/lib/i386-linux-gnu/libc-2.17.so:     file format elf32-i386

SYMBOL TABLE:
no symbols

On linux we do not replace stripped library with non-stripped one. Instead we install separate debug information file and configure debugger to reference it.

$ apt-cache search libc-
libc6-dbg - Embedded GNU C Library: detached debugging symbols
$ sudo apt-get install libc6-dbg

Packages for debug information has suffix of -dbg (it seems to be so). Installation will deploy files under /usr/lib/debug directory.

$ file /usr/lib/debug/lib/i386-linux-gnu/libc-2.17.so
/usr/lib/debug/lib/i386-linux-gnu/libc-2.17.so: ELF 32-bit LSB shared object, Intel 80386, version 1 (SYSV), dynamically linked (uses shared libs), BuildID[sha1]=0x81a55e819c61f581e6a9179eaf59726dd80aea31, for GNU/Linux 2.6.24, not stripped
$ objdump -t /usr/lib/debug/lib/i386-linux-gnu/libc-2.17.so

/usr/lib/debug/lib/i386-linux-gnu/libc-2.17.so:     file format elf32-i386

SYMBOL TABLE:
00000174 l    d  .note.gnu.build-id 00000000 .note.gnu.build-id
00000198 l    d  .note.ABI-tag      00000000 .note.ABI-tag

gdb is wise enough to automatically detect debug information (as long as libc6 and libc6-dbg match) and applies it!

(gdb) set verbose on
(gdb) run
...
Reading symbols from /lib/i386-linux-gnu/libc.so.6...Reading symbols from /usr/lib/debug/lib/i386-linux-gnu/libc-2.17.so...done.
done.

This entry is quite of help for debug information.

Now we can see precise stack!

spell check in vim

If my memory does not deceive me, I started using vi as my editor on HP-UX 10. (Yes, it is quite long time ago) I have not noticed it until quite recently though, vim (vi improved) has evolved so much more than I had expected.

I write almost all of text on vim, which includes this blog post. As seen, my vocabulary is quite limited and incorrect. And phrase does not go beyond routine. That is, I need spell checking mechanism to correct many typo I make and suggestions to meet what I want to express.

:set spell

z=

zg

$ cat en.utf-8.add
vim

zw

$ cat en.utf-8.add
vim/!

how to notify core file creation

Now you are able to configure system to yield core files for designated directory.

But there is no way to notice creation of core file (as apport has been disabled). Core files shall pile up silently behind the door. Repetition of core dump can easily consume disk space. You have to be careful and pay attention for such situation.

Let’s find out effective method to notify creation of core file.

$ apt-cache search inotify-tools
inotify-tools - command-line programs providing a simple interface to inotify
$ sudo apt-get install inotify-tools
$ inotifywa<TAB>
inotifywait   inotifywatch

#!/usr/bin/env bash

while inotifywait -e create,modify /tmp; do
    echo "Lupin, something happened under /tmp directory"
done

$ ./watch.sh
Setting up watches.
Watches established.
/tmp/ CREATE hi
Lupin, something happened under /tmp directory

$ apt-cache search libnotify-bin
libnotify-bin - sends desktop notifications to a notification daemon (Utilities)
$ apt-get install libnotify-bin

$ for i in `seq 10 -1 0`
> do
> echo $i|festival --tts
> done && notify-send 'Ten Count! Knockout!'

#!/usr/bin/env bash

while inotifywait -e create /var/cores; do
    notify-send -u critical "a new core file is created. Please check /var/cores directory"
done

#!/usr/bin/env bash

nohup <PATH>/watchCore.sh 1> /dev/null 2>&1 &

package management on ubuntu

Why did you choose ubuntu as your platform? Every one must have reason(s) for that.

For me, one of main reasons is easy access of software packages. (Remember the days of Solaris 8 when one had to install all the necessary packages manually by pkgadd...)

How are software packages repositories maintained on ubuntu? Take google-chrome as example and see how it goes.

$ wget -q -O - https://dl-ssl.google.com/linux/linux_signing_key.pub | sudo apt-key add -

$ sudo apt-key list
/etc/apt/trusted.gpg
--------------------

pub   1024D/7FAC5991 2007-03-08
uid                  Google, Inc. Linux Package Signing Key <linux-packages-keymaster@google.com>
sub   2048g/C07CB649 2007-03-08

$ sudo sh -c 'echo "deb http://dl.google.com/linux/chrome/deb/ stable main" >> /etc/apt/sources.list.d/google.list'

$ sudo apt-get update
$ sudo apt-get install google-chrome-stable

vmplayer abnormal exit on ubuntu

These days on my ubuntu box I encounter abnormal exit of vmplayer upon start-up. Specifically speaking vmplayer asks for me to update kernel modules and it fails.

sigh.

$ vmplayer
Logging to /tmp/vmware-ayamada/vmware-modconfig-12999.log
... "VMWare Kernel Module Updater" windows comes up ...
$ echo $?
1

$ uname -r
3.8.0-30-generic
$ sudo ln -s /usr/src/linux-headers-`uname -r`/include/generated/uapi/linux/version.h /usr/src/linux-headers-`uname -r`/include/linux/version.h
$ sudo vmware-modconfig --console --install-all

$ ls -ld /usr/src/linux-headers-3.8.0-*-generic
drwxr-xr-x 7 root root 4096 May 18 14:34 /usr/src/linux-headers-3.8.0-21-generic
drwxr-xr-x 7 root root 4096 May 25 08:59 /usr/src/linux-headers-3.8.0-22-generic
drwxr-xr-x 7 root root 4096 May 31 09:59 /usr/src/linux-headers-3.8.0-23-generic
drwxr-xr-x 7 root root 4096 Jun 19 10:01 /usr/src/linux-headers-3.8.0-25-generic
drwxr-xr-x 7 root root 4096 Jul  5 09:58 /usr/src/linux-headers-3.8.0-26-generic
drwxr-xr-x 7 root root 4096 Jul 31 09:22 /usr/src/linux-headers-3.8.0-27-generic
drwxr-xr-x 7 root root 4096 Aug 21 10:07 /usr/src/linux-headers-3.8.0-29-generic
drwxr-xr-x 7 root root 4096 Sep  6 09:52 /usr/src/linux-headers-3.8.0-30-generic

python tools invocation

There are many tools (e.g. yum/hg), which are written in python, on Linux platform. For example, on my ubuntu 13.04 box there are 45 python scripts under /usr/local/bin directory.

$ file /usr/local/bin/*|grep -ic python
45

So I should say that one can not live without them.

How are these python tools invoked?

Take tikerer as example.

$ which tinker
/usr/local/bin/tinker
$ file /usr/local/bin/tinker
/usr/local/bin/tinker: Python script, ASCII text executable

So executing tinker command will hook python package.

#!/usr/bin/python
# EASY-INSTALL-ENTRY-SCRIPT: 'Tinkerer==1.2.1','console_scripts','tinker'
__requires__ = 'Tinkerer==1.2.1'
import sys
from pkg_resources import load_entry_point

if __name__ == '__main__':
    sys.exit(
        load_entry_point('Tinkerer==1.2.1', 'console_scripts', 'tinker')()
    )

Where does this package come from? Let’s search for its location with primitive way.

$ strace -o test.txt -e open tinker -d "test"
$ grep tinker test.txt |grep -iv enoent
...
open("/usr/local/lib/python2.7/dist-packages/tinkerer/__init__.py", O_RDONLY|O_LARGEFILE) = 3
open("/usr/local/lib/python2.7/dist-packages/tinkerer/__init__.pyc", O_RDONLY|O_LARGEFILE) = 4

Ok, here is the location where relevant python package resides.

$ ls /usr/local/lib/python2.7/dist-packages/tinkerer
__init__.py   __templates  cmdline.pyc      draft.pyc  master.py   page.py   paths.py   post.py   static  utils.py   writer.py
__init__.pyc  cmdline.py   draft.py ext        master.pyc  page.pyc  paths.pyc  post.pyc  themes  utils.pyc  writer.pyc

So when one installs python package, say, with pip command, packages will be deployed under dist-packages directory. And at the same time, a wrapper script will be deployed on search path so that it invokes relevant package by making use of pkg_resources.

compare files recursively

I have to confess that I’m so ignorant...

You can compare all the files under 2 directories recursively by diff command with -r option.

For example,

$ diff -r <DIR_A> <DIR_B>

So easy. sigh...

distribute requests by URL on apache

Alias /blog <TINKER_ROOT>/blog/html/
<LocationMatch "/blog/.*">
    AuthType Basic
    AuthName "Open Sesame!"
    AuthUserFile <PATH_TO_PASS>/htpasswd
    Require user lupin
</LocationMatch>

$ htpasswd -c htpasswd lupin

Require user lupin fujiko jigen

$ sudo service httpd restart

switching from HTTPS to SSH

Communication protocol for github can either be SSH or HTTPS. You can configure protocol to SSH so that ssh key shall be used for authentication. And you do not have to enter password every time you communicate with github.

URL, which includes protocol, is specified in .git/config file.

[remote "origin"]
    url = https://github.com/<USERNAME>/<REPO>.git

So as to change URL, use “git remote set-url” command.

$ git remote set-url origin git@github.com:<USERNAME>/<REPO>.git

URL in config file shall be updated as specified in argument abvoe.

[remote "origin"]
    url = git@github.com:<USERNAME>/<REPO>.git

Therefore you may be able to configure URL directly by modifying value in config file (not tested though :-)).

google anlytics in sphinx

If you would like to analyze statics of access to your site built with sphinx, then you can embed google analytics code in it.

Prepare layout.html file under source/_templates directory. Insert javascript code obtained at google analytics site.

{% extends "!layout.html" %}

{% block footer %}
{{ super() }}
<div class="footer">
<script>
(function(i,s,o,g,r,a,m){
    i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
        (i[r].q=i[r].q||[]).push(arguments)
    },i[r].l=1*new Date();
    a=s.createElement(o),m=s.getElementsByTagName(o)[0];
    a.async=1;a.src=g;
    m.parentNode.insertBefore(a,m)
 })(window,document,'script','//www.google-analytics.com/analytics.js','ga');
ga('create', 'UA-xxxxxxxxx-x', '<site_name>');
ga('send', 'pageview');
</script>
</div>
{% endblock %}

And rebuild html and deploy it onto site.

For details, please refer to sphinx FAQ.

back up files onto dropbox

All machine, which includes Hard Disks, are to break some day. Let’s back up your precious content onto dropbox with encrypted on regular basis.

create backup directory in dropbox.

$ mkdir ~/Dropbox/backup

Prepare backup script kicked by cron on regular interval. Create directory where backup script resides:

$ mkdir ~/Documents/cron

Deploy following backup script under cron directory:

#!/usr/bin/env bash

# we use day of week as file name
LANG=C
NAME=`date "+%A"`
# please specify password for encryption in case that you encrypt backup file
PASSWD=xxxxxx
# we create backup file under temporary directory on local host
# so as to avoid latency
TEMP=/tmp
BACKUP_DIR=~/Dropbox/backup

cd ~/Documents
if true
then
    find . -exec zip -P $PASSWD $TEMP/$NAME.zip {} \; 1>/dev/null 2>&1
else
    find . -exec zip $TEMP/$NAME.zip {} \; 1>/dev/null 2>&1
fi
mv $TEMP/$NAME.zip $BACKUP_DIR

Please be noted that you have to add execution permission for this script. And test it if it works or not.

$ bash -xv backup.sh

Register cron job to start script on regular basis by “crontab -e”. With following example, backup script is executed every one hour at *:00.

0 * * * * ~/Documents/cron/backup.sh

On ubuntu, cron activity is recorded in /var/log/syslog. Monitor syslog if cron works as expected.

try sphinx better theme

Found a new theme for sphinx.

Demo site looks nice & simple.

Let’s apply and see what kind of effect will have on my documentation.

Application procedure is quite simple. Install sphinx-better-theme package.

$ sudo pip install sphinx-better-theme

Apply theme by modifying conf.py.

from better import better_theme_path
html_theme_path = [better_theme_path]
html_theme = 'better'

Ok, stay with this theme.

memo on bitbucket

I would like to have place where I can put some static content created by sphinx. I hereby demonstrate procedure to deploy html files onto bitbucket.

Create a repository under the name, <username>.bitbucket.org. This repository could either be public or private. And you can select mercurial or git as your tool. In this scenario, I use hg as my tool.

Create sphinx project to host your content.

$ mkdir wiki && cd wiki
$ sphinx-quickstart

Write up your content and build it as usual.

$ make html

Copy content from build/html directory.

$ mkdir publish
$ cd publish/
$ rsync -av ../build/html/ .

Now the content is ready for upload, so prepare for publication.

Install mercurial if you still do not have one at your hand.

$ sudo apt-get install python-dev
$ sudo pip install mercurial
$ which hg
/usr/local/bin/hg

OK, now we have hg.

Create .hgrc file under home directory.

[ui]
username = Fist LAST <your_mail_address>
verbose = True

Create ssh key for bitbucket.

$ ssh-keygen

~/.ssh$ ls id_rsa.bitbucket*
id_rsa.bitbucket  id_rsa.bitbucket.pub

Configure ~/.ssh/config file so that newly created ssh key be referenced for bitbucket access.

Host bitbucket.org
    HostName        bitbucket.org
    IdentityFile    ~/.ssh/id_rsa.bitbucket.org
    User            hg

And deploy your public key onto bitbucket.

$ xclip -sel clip < id_rsa.bitbucket.pub

And place hgrc file under .hg directory of publish directory so that push target shall be specified.

[paths]
default=ssh://hg@bitbucket.org/<userid>/<userid>.bitbucket.org

And push your content onto bitbucket.

$ rsync -av ../build/html .
$ hg add .
$ hg commit -m "my first post of mine"
$ hg push

Now you can access your content by http://<userid>.bitbucket.org.

chart in blog post

Same as sphinx, you can make use of blockdiag as extension in tinkerer. That is, you can draw chart in your blog by blockdiag.`

Here is a quick setup procedure.

First install sphinxcontrib-blockdiag package.

$ sudo pip install sphinxcontrib-blockdiag

Then add ‘sphinxcontrib.blockdiag’ in the list of extensions in conf.py.

extensions = ['tinkerer.ext.blog', 'tinkerer.ext.disqus', 'sphinxcontrib.blockdiag']

You can specify font of your choice used in blockdiag. Search for path of your font.

$ fc-list

And specify font paht as value of blockdiag_fontpath.

blockdiag_fontpath = '/usr/share/fonts/truetype/ttf-dejavu/DejaVuSansMono.ttf'

Now it’s ready for usage.

For example,

prevent broken pipe during ssh session

From time to time, you may encounter ssh session lost with following error.

Write failed: Broken pipe

It means that write() system call failed against File Descriptor for socket of ssh session.

$ man -s2 write
...
EPIPE  fd is connected to a pipe or socket whose reading end is closed.  When this  happens  the  writing
       process  will  also  receive  a SIGPIPE signal.  (Thus, the write return value is seen only if the
       program catches, blocks or ignores this signal.)

You may be able to avoid this connection lost by keeping connection alive.

ServerAliveInterval 120

ClientAliveInterval 120

filter packet based upon URI

Wireshark is your good friend. You can find fun time of examining packets with wireshark.

Sometimes you would like to filter packets and extract packets only for specific URL, say, “www.google.com”. Wireshark provides content filter for HTTP as well.

Specify filter as follows.

http contains "www.google.com"

filter manual will help you guide through details of filter configuration.

automatic yum update

Sometime you would like to configure Scientific Linux to update packages automatically without manual intervention. You can configure system as you wish and here is a short memo to achieve such.

$ sudo yum install yum-cron

# Don't install, just check (valid: yes|no)
CHECK_ONLY=no

# Don't install, just check and download (valid: yes|no)
DOWNLOAD_ONLY=no

$ /etc/init.d/yum-cron start
$ sudo chkconfig yum-cron on

Aug 26 03:16:01 <hostname> run-parts(/etc/cron.daily)[24221]: starting 0yum.cron
Aug 26 04:07:46 <hostname> run-parts(/etc/cron.daily)[24421]: finished 0yum.cron

Aug 09 04:09:11 Updated: nss-softokn-devel-3.14.3-3.el6_4.i686
Aug 09 04:09:12 Updated: nss-devel-3.14.3-4.el6_4.i686
Aug 09 04:09:13 Updated: nss-tools-3.14.3-4.el6_4.i686
Aug 14 03:36:46 Updated: httpd-tools-2.2.15-29.sl6.i686
Aug 14 03:36:48 Updated: httpd-2.2.15-29.sl6.i686

[main]
debuglevel=2
logfile=/var/log/yum.log

$ sudo yum history
Loaded plugins: auto-update-debuginfo, downloadonly, fastestmirror, refresh-packagekit, security
ID     | Login user               | Date and time    | Action(s)      | Altered
-------------------------------------------------------------------------------
   138 | root <root>              | 2013-08-14 03:36 | Update         |    2
   137 | root <root>              | 2013-08-09 04:09 | Update         |   12
...

ENABLED="false"

how to play minecraft on ubuntu

My daughter would like to play her favorite game, minecraft on her ubuntu box. Here is a short memo to summarize procedure to setup environment to play minecraft on ubuntu.

$ sudo add-apt-repository ppa:webupd8team/java
$ sudo apt-get update
$ sudo apt-get install oracle-java7-installer

LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/lib/jvm/java-7-oracle/jre/lib/i386/

XMODIFIERS=@im=none

#!/usr/bin/env bash

LANG=C
LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/lib/jvm/java-7-oracle/jre/lib/i386/
XMODIFIERS=@im=none
export LANG LD_LIBRARY_PATH XMODIFIERS

MINE_HOME=~/minecraft
cd $MINE_HOME

START_MSG="start minecraft"
echo $START_MSG|festival --tts

LOG=play.`date "+%Y%m%d"`.log

echo "START: " `date` >> $LOG
java -Xmx1024M -Xms512M -cp minecraft.jar net.minecraft.LauncherFrame >> $LOG 2>&1
echo "FINISH: " `date` >> $LOG

[Desktop Entry]
Type=Application
Terminal=false
Name=minecraft
Icon=/PATH_TO_ICON_DIR/minecraft.ico
Exec=/PATH_TO_SCRIPT_DIR/minecraft.sh

add favicon

You can configure favicon for your blog.

Prepare your favorite icon and place it under _static directory.

$ file _static/skyhigh71.ico
_static/skyhigh71.ico: MS Windows icon resource - 1 icon

And point newly deployed icon file in conf.py.

# Change your favicon (new favicon goes in _static directory)
html_favicon = 'skyhigh71.ico'

customize tinkerer page continued

$ echo $LANG
ja_JP.UTF-8

$ LANG=C tinker -b

language = "en"

language = "de"

$ LANG=de_DE.utf8 tinker -b

# Number of blog posts per page
posts_per_page = 3

how to yield core file

By default, gdb is given on ubuntu desktop, so you can enjoy debugging applications (if you like :-)).

$ which gdb
/usr/bin/gdb
$ gdb -v
GNU gdb (GDB) 7.5.91.20130417-cvs-ubuntu

But it seems that core file of application is not yielded (quite natural). Here is a memo to describe procedure how to configure system to yield core file.

$ ulimit -a
core file size          (blocks, -c) 0

*               soft    core            unlimited

$ cat /proc/sys/kernel/core_pattern
\|/usr/share/apport/apport %p %s %c

$ sudo mkdir /var/cores
$ sudo chmod a+w /var/cores
$ sudo bash -c "echo /var/cores/core.%e.%p.%t > /proc/sys/kernel/core_pattern"

kernel.core_pattern=/var/cores/core.%e.%p.%t

echo "\|/usr/share/apport/apport %p %s %c" > /proc/sys/kernel/core_pattern

#enabled=1
enabled=0

customize tinkerer page

# Add templates to be rendered in sidebar here
html_sidebars = {
    "**": ["recent.html", "searchbox.html", "tags.html"]
}

<!-- Twitter button -->
<a href="https://twitter.com/share" class="twitter-share-button">Tweet</a>
<script>!function(d,s,id){var js,fjs=d.getElementsByTagName(s)[0],p=/^http:/.test(d.location)?'http':'https';if(!d.getElementById(id)){js=d.createElement(s);js.id=id;js.src=p+'://platform.twitter.com/widgets.js';fjs.parentNode.insertBefore(js,fjs);}}(document, 'script', 'twitter-wjs');</script>

(function(i,s,o,g,r,a,m){
    i['GoogleAnalyticsObject']=r;
    i[r]=i[r]||function(){
        (i[r].q=i[r].q||[]).push(arguments)
    },i[r].l=1*new Date();
    a=s.createElement(o),m=s.getElementsByTagName(o)[0];
    a.async=1;
    a.src=g;
    m.parentNode.insertBefore(a,m)
})(window,document,'script','//www.google-analytics.com/analytics.js','ga');

ga('create', 'UA-xxxxxxxx-1', '<your_site>');
ga('send', 'pageview');

{% extends "!page.html" %}

{% set script_files = script_files + ["_static/google_analytics.js"] %}

{% extends "!page.html" %}

{% set script_files = script_files + ["_static/google_analytics.js"] %}

{%- block body %}
    <div class="section_head">
    <div class="timestamp_layout">
      {{ timestamp(metadata.formatted_date) }}
    </div>
    {% block buttons %}
    <div class="buttons">
      <!-- Twitter button -->
      <a href="https://twitter.com/share" class="twitter-share-button">Tweet</a>
      <script>!function(d,s,id){var js,fjs=d.getElementsByTagName(s)[0],p=/^http:/.test(d.location)?'http':'https';if(!d.getElementById(id)){js=d.createElement(s);js.id=id;js.src=p+'://platform.twitter.com/widgets.js';fjs.parentNode.insertBefore(js,fjs);}}(document, 'script', 'twitter-wjs');</script>
    </div>
    {% endblock %}
    </div>
    {{ link }}
    {{ body }}
    {{ post_meta(metadata) }}
    {{ comments }}
{% endblock %}

enable sshd

By default, sshd is disabled on ubuntu desktop. Therefore you need to install & enable sshd if you would like to access host remotely via ssh.

Here is short memo for this sake.

First, search for sshd package via apt-cache command.

$ sudo apt-cache search sshd
openssh-server - secure shell (SSH) server, for secure access from remote machines

$ sudo apt-cache show openssh-server
Package: openssh-server
...
Description-en: secure shell (SSH) server, for secure access from remote machines
...
This package provides the sshd server.

OK, openssh-server seems to be what we are looking for.

$ sudo apt-get install openssh-server

Installation will automatically start sshd daemon.

$ sudo lsof -nPi:22
COMMAND  PID    USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
sshd    3557    root    3u  IPv4  35564      0t0  TCP *:22 (LISTEN)
sshd    3557    root    4u  IPv6  35566      0t0  TCP *:22 (LISTEN)

$ sudo service ssh status
ssh start/running, process 3557

Please be noted, sshd daemon is registered not as sshd but as ssh.

sshd configuration can be set in /etc/ssh/sshd_config. In case that you would like to disable remote root access, then change parameter PermitRootLogin value from yes to no.

PermitRootLogin no

swap Caps Lock key with Ctrl key

I have been using Happy Hacking Keyboard (HHK) for about decade. It’s really nice one, but it becomes lame at last and key touch is quite noisy. So I bought a cheap English layout keyboard at amazon. (good-by HHK)

The problem here is that “caps lock” resides in the left-middle. This is quite inconvenient. So I replaced “caps lock” key with “control” by configuring keyboard mapping.

Here is its procedure on ubuntu 13.04 (LXDE) to do so.

Edit keyboard file and applying argument “ctrl:swapcaps” for XKBOPTIONS option.

$ cd /etc/default
$ diff keyboard keyboard.org
11,12c11
< #XKBOPTIONS=""
< XKBOPTIONS="ctrl:swapcaps"
---
> XKBOPTIONS=""

Or you can specify “ctrl:nocaps” in case that you assign control key for both.

Then execute dpkg-reconfigure command so that above change will take effect. Please be noted that rebooting system has no effect for this keyboard configuration change and it remains as they were.

$ sudo dpkg-reconfigure keyboard-configuration

Answer to questions given by command and at last stage you will be prompted if you save above configuration change.

Following emacs page has quite comprehensive explanation for this sake, which helps must help you.

And following debian page has explanation as to keyboard configuration.

Please be noted, above dpkg-reconfigure command updates initrd.img-X.X.X-XX-generic file under /boot directory. Therefore in case that OS updates image file, then you have to re-execute dpkg-reconfigure command to reflect change onto image file.

Please be noted that there seems to be more work to be done, as change of configuration is lost after rebooting system. sorry...

It seems that following thread shall be similar issue of this.

the first post of mine

A short memo to describe procedure how to host blog on github.

$ ssh-keygen
Generating public/private rsa key pair.
...

$ file *
id_rsa.github:     PEM RSA private key
id_rsa.github.pub: OpenSSH RSA public key

~/.ssh$ more config
Host github.com
    HostName      github.com
    IdentityFile  ~/.ssh/id_rsa.github
    User          git

$ xclip -sel clip < ~/.ssh/id_rsa.github.pub

$ ssh -v -T git@github.com

$ sudo apt-get install git

$ git config --global user.name <USERNAME>
$ git config --global user.email <YOUR_EMAIL_ADDRESS>

[user]
        name = <USERNAME>
        email = <YOUR_EMAIL_ADDRESS>

$ git config --global credential.helper cache
$ git config --global credential.helper 'cache --timeout=3600'

[credential]
        helper = cache --timeout=3600

$ sudo pip install tinkerer

$ tinker -v
Tinkerer version 1.2.1

$ mkdir blog

$ tinker --setup
Your new blog is almost ready!
You just need to edit a couple of lines in conf.py
$ tree .
    .
    ├── _static
    ├── _templates
    │   ├── page.rst
    │   └── post.rst
    ├── conf.py
    ├── drafts
    ├── index.html
    └── master.rst

$ diff conf.py.org conf.py
11c11
< project = 'My blog'
---
> project = "sakana"
14c14
< tagline = 'Add intelligent tagline here'
---
> tagline = 'short memo by SkyHigh71'
20c20
< author = 'Winston Smith'
---
> author = 'SkyHigh71'
23c23
< copyright = '1984, ' + author
---
> copyright = '2013-, ' + author
26c26
< website = 'http://127.0.0.1/blog/html/'
---
> website = 'http://skyhigh71.github.com'

$ mkdir publish
$ cd publish
$ git init
$ git remote add origin https://github.com/SkyHigh71/SkyHigh71.github.io.git

[remote "origin"]
        url = https://github.com/SkyHigh71/SkyHigh71.github.io.git

$ touch .nojekyll

$ tinker -d "the First Post of mine"
$ ls drafts
the_first_post_of_mine.rst

$ tinker -p drafts/the_first_post_of_mine.rst

$ LANG=C tinker -b

$ cd publish
$ rsync -av ../blog/html/ .

$ git add .
$ git commit -m "first post"

$ git push origin master

Hello World

How are you doing?