Monday, 6 December 2010

Converting LVM virtual machine storage to image

To convert the LVM disk to qcow2 formatted disk image,

Use lvdisplay to get the Logical volume name
$ sudo lvdisplay

Use qemu-img to convert to the required image format
# qemu-img convert -O qcow2 /dev/mapper/lv_name <destination_file>.qcow2

eg:
# qemu-img convert -O qcow2 /dev/mapper/disk1 disk1.qcow2

This will be useful to replicate the virtual machines to other hardware.

./arun

Friday, 1 October 2010

Install CyanogenMod 6 (Android 2.2 Froyo alternative) on HTC Dream (ADP1)



Atlast managed to get the 2.2 flavour on ADP1. (this installation is only applicable for root'd Android developer phone HTC dream)

  • Requirements:


Tools: fastboot


images:
Amon_Ra 1.7.0 ( http://files.androidspin.com/downloads.php?dir=amon_ra/RECOVERY/&file=recovery-RA-dream-v1.7.0.img )
radio image for adp1 (http://sapphire-port-dream.googlecode.com/files/ota-radio-2_22_19_26I.zip)
update-cm-6.0.0-DS-signed.zip (http://forum.cyanogenmod.com/files/file/95-update-cm-600-ds-signedzip/)
gapps-mdpi-tiny-20100926-signed.zip (http://forum.cyanogenmod.com/files/file/124-google-addon-mdpi-tiny-20101020/)
DangerSPL (http://sapphire-port-dream.googlecode.com/files/spl-signed.zip)

  • Procedure:


Ref : http://wiki.cyanogenmod.com/index.php?title=Full_Update_Guide_-_Android_Dev_Phone_1#Installing_a_Custom_Recovery_Image

- Download and copy all these images to the root of your sdcard except the recovery image

  • Install custom recovery image:


- boot the phone in fastboot mode by pressing camera button while rebooting

- From your machine make sure the device is visible
# ./fastboot-mac devices
HT845Gxxxxx fastboot

- flash the cyanogen recovery image and reboot
# ./fastboot-mac flash recovery recovery-RA-dream-v1.7.0.img
sending 'recovery' (4594 KB)... OKAY
writing 'recovery'... OKAY

# ./fastboot-mac reboot

- Once the phone is rebooted make sure  you have the 1.7 recovery image

  • Flash the radio image


Now flash the radio once the phone rebooted to recovery RA 1.7.0 by pressing the home button while rebooting
- select install zip from sdcard and choose the radio image , once the installation is done reboot the phone and make sure that you have the 2_22_19_26I base band version





  • Flash danger spl now


Ref: http://wiki.cyanogenmod.com/index.php?title=DangerSPL_%26_CyanogenMod_5/6#Install_DangerSPL

boot the phone in recovery mode, and select install zip from sdcard and select install the danger spl image, reboot

This will update the Hboot to 1.33.2005,  this is required to increase the ROM size to hold the cm6 image.



  • flash the cyanogen mode 6 images


Ref : http://wiki.cyanogenmod.com/index.php?title=DangerSPL_%26_CyanogenMod_5/6#Installing_CyanogenMod_ROM

boot the phone again in recovery mode, select and install the cm 6 image, reboot the phone and make sure that you can boot in to the new cyanogen 6.



  • Install the google addons


The important thing is dont install the full version of google apps on ADP1 just use the MDPI tiny version. otherwise the phone cannot boot properly.

Ref: http://wiki.cyanogenmod.com/index.php?title=Latest_Version#Google_Apps



Now clear wipe the cache and reboot the phone, That is it. Now you have ADP1 installed with Cyanogenmod 6, Enjoy..



./arun

Thursday, 23 September 2010

Install *.apk to android phone

It was disappointing that the android devices selling in most of the gulf countries doesnt have market place installed :( - One of the core and major feature of androis is missing.

Anyway the softwares available in Android Package file (apk) can be installed by downloading it to the pc using adb

eg: to install slide me market place, an alternative for google market place.
download the latest apk from http://slideme.org/.
$ ./adb install ~/sam2.apk
1686 KB/s (506724 bytes in 0.293s)
pkg: /data/local/tmp/sam2.apk
Success


./arun

Tuesday, 10 August 2010

Exporting display over ssh

To export the display from a remote server over ssh:

ssh -X user@host

Just made sure that, X11 forwarding is enabled on the sshd_config .

Once the connection is made, you can make sure the display is exported using:

# echo $DISPLAY
localhost:10.0


if the value is empty, make sure you have the necessary package (mkxauth) installed to create .XAuthority file.

./arun

Wednesday, 28 July 2010

IPv6 configuration for KVM guests

It is simple and straight forward to enable IPv6 on KVM guests

Configure the host machine with IPv6 Address on the bridge interface

cat ifcfg-br0



IPV6INIT=yes
IPV6ADDR=xxxx.xx::10
IPV6_DEFAULTGW=xxxx.xx::1
IPV6_AUTOCONF=no




Configure the interface on virutal machines with ipv6 address

cat ifcfg-eth0



IPV6INIT=yes
IPV6ADDR=xxxx.xx::11
IPV6_DEFAULTGW=xxxx.xx::1
IPV6_AUTOCONF=no




Add the the necessary firewall rules to ip6tables on the host machine


-A FORWARD -m physdev --physdev-is-bridged -j ACCEPT.



./arun

Tuesday, 20 July 2010

NAT with port forwarding on Virtual Box

You can use the host-only-adapter networking, if you require the virtual machine to be accessible only from the host machine. In this case your virtual machine will not have access to anywhere outside the host. Read more about virtual box networking at http://www.virtualbox.org/manual/ch06.html


On the other hand NAT enabled interface can communicate with clients outside the host, but the host cannot access the services on the virtual machine directly. We need to enabled port forwarding with NAT interface to achieve this.


On Linux:
If you need to have ssh accessible from host machine to virtual machine,
$ VBoxManage modifyvm "VM Name" --natpf1 "openssh,tcp,127.0.0.1,2222,,22"

Where --natpf1 is for adapter1, openssh is just a anme, and you can also input the ip address of virtual machine like
$ VBoxManage modifyvm "VM Name" --natpf1 "openssh,tcp,127.0.0.1,2222,10.0.2.20,22"

(assume the virtual machine ip is 10.0.2.20)

Now you can make ssh connection from host like, $ ssh localhost -p 2222

We can use same port number for port number about 1024 , say for a service running on port 8080 we can forward it with
VBoxManage modifyvm "VM Name" --natpf1 "proxy,tcp,127.0.0.1,8080,10.0.2.20,8080"

These rules will be added to the .VirtualBox/Machines/machine_name/machine_name.xml file like:
< Forwarding name="openssh" proto="1" hostip="127.0.0.1" hostport="2222" guestip=10.0.2.20 guestport="2222"/>

You can forward connection to any port on virtual host like this.

Make sure that the virtual machine interface is closed and the vm is not running while you change it, otherwise the changes will not take effect.

On Windows:
VBoxManage setextradata "VM Name" "VBoxInternal/Devices/pcnet/0/LUN#0/Config/guestssh/Protocol" TCP
VBoxManage setextradata "VM Name" "VBoxInternal/Devices/pcnet/0/LUN#0/Config/guestssh/GuestPort" 22
VBoxManage setextradata "VM Name" "VBoxInternal/Devices/pcnet/0/LUN#0/Config/guestssh/HostPort" 2222

* Replace VM Name with your virtual instance name

./arun

Convert KVM images to Virtual Box (VDI)

It took a while to get the KVM image working with Sun virtual box.


The advantages of a virtual box image is, you can run it on any platform (linux, mac or windows), works without virtualization enabled processor and will work on a 32bit machine
Here are the steps to create an image that works with virtual box:


From the KVM installed server
$ qemu-img convert kvm-os.img -O raw kvm-os-raw.img

Copy the image (kvm-os-raw.img) to virtual box machine
$ VBoxManage convertfromrow --format VDI kvm-os-raw.img vbox.vdi

Converting from raw image file="kvm-os-raw.img" to file="vbox.vdi"...
Creating dynamic image with size ....

This will create a virtual box compatible image
Incase required you can compact the image to actual size
$ VBoxManage modifyvdi /home/user/vbox.vdi compact

0%...10%...20%...30%...40%...50%...60%...70%
Here the path to vdi image must be absolute.

Now you can create a new virtual machine from virtual box console/command line, with the vdi image as storage.
Boot the machine and hope for the best :)
But it wasn't easy for me even after this beautiful vdi image, boot hangs with a kernel panic, file system not found.


To fix this issue, we need to recreate the initrd image in the virtual machine:
instructions to do it for redhat:
- Boot the virtual machine in rescue mode with Redhat CD
> linux rescue

# chroot /mnt/sysimage

take a backup of existing initrd
# cp /boot/initrd-2.6.xxx.img initrd-2.6-old

create new initrd image
# mkinitrd -v /boot/initrd-new.img kernel-version

// eg: mkinitrd -v /boot/initrd-new.img 2.6.18-194.8.1.el5

edit the grub configuration and replace the initrd image name with new one
# cat /boot/grub/menu.lst

Reboot the machine and see if it boots :)

Hope this will be helpful for someone, I spent hours to get it working :) .
./arun

Monday, 5 July 2010

Netboot KVM guest

To install the KVM guest operating system (eg: RHEL) from the network
- Create the bridge interface on the KVM host machine (http://arunnsblog.com/2010/04/09/virtualization-with-kvm-under-redhat-linux-migrate-vmware-virtual-images-to-kvm/)
- Make sure that the gateway is configured in the bridge interface (GATEWAY=).
- Make sure that you have the required rules added to the iptables:
-A FORWARD -m physdev --physdev-is-bridged -j ACCEPT
- Create virtual machine with supported network interface type (pcnet, rtl8139 used to work)
- Add the mac address of kvm guest to the dhcp server

Start the virtual machine and see if it can kick start from the network.

You can trouble shoot with a tcpdump on the KVM host machine:
tcpdump -i br0 port bootps -vvv -s 1500

./arun

Monday, 28 June 2010

Install Zope Object Database (ZODB) for python

ZODB is a way to store persistent data, ZODB comes with ZOPE.
To import ZODB directly to python.

$ easy_install ZODB3==VERSION

eg: $ easy_install ZODB3==3.8.3


$python
>>> import ZODB


./arun

Sunday, 23 May 2010

Compile and install python with mysql for users

To run custom python version for a useraccount:

download the latest version of python

$ wget http://www.python.org/ftp/python/2.6.5/Python-2.6.5.tgz
$ tar xvzf Python-2.6.5.tgz
$ cd Python-2.6.5
$ ./configure --prefix=/home/username/python-2.6.5
$ make
$ make install


Install setuptools
as root:

# ln -s /home/username/python-2.6.5/bin/python2.6 /usr/bin/ *this is required for setuptools

as normal user:

$ wget http://pypi.python.org/packages/source/s/setuptools/setuptools-0.6c11.tar.gz#md5=7df2a529a074f613b509fb44feefe74e
$ tar xvzf setuptools-0.6c11.tar.gz
$ sh setuptools-0.6c11-py2.6.egg --prefix=~/python-2.6.5/


download mysql-python

$ wget http://downloads.sourceforge.net/project/mysql-python/mysql-python-test/1.2.3c1/MySQL-python-1.2.3c1.tar.gz?use_mirror=citylan
$ tar xvzf MySQL-python-1.2.3c1.tar.gz
$ cd MySQL-python-1.2.3c1
$ /home/username/python-2.6.5/bin/python setup.py build
$ /home/username/python-2.6.5/bin/python setup.py install

Done:

sh-3.00$ python2.6
Python 2.6.5 (r265:79063, May 23 2010, 14:40:28)
>>> import MySQLdb
>>>

./arun

Friday, 14 May 2010

KVM image on LVM

Convert qcow2/raw images to LVM logical volume to use with KVM:

- Convert the qcow2 image to raw format (if it is in qcow2)
$ qemu-img convert image.qcow2 -O raw image.raw

- Create the physical volume for LVM
# pvcreate /dev/sdb
(replace the device with correspond to the system)

- Create the volume group
# vgcreate pool1 /dev/sdb
(replace pool1 with the name as required)

- Create Logical volume with same size as the image
# lvcreate -n justaname --size 50G pool1
(replace justaname and size as per the requirements)
Use lvresize incase you required the change the volume size

- dd the raw image to lvm logical volume
# dd if=image.raw of=/dev/pool1/justaname bs=8M
(Change the block size according to the requirements.

Edit the kvm xml configuration for the corresponding virutal machine to use the logical volume

< disk type='block' device='disk' >
< source dev='/dev/pool1/justaname'/ >
< /code >


./arun

Friday, 9 April 2010

Virtualization with KVM under Redhat Linux, Migrate VMware virtual images to KVM

KVM (Kernel Based Virtual Machine) - http://www.linux-kvm.org/ , is one of the best choice to do virtualization under linux, and especially without extra licensing cost.

Install KVM
To install KVM on redhat enterprise linux:
- Install the machine with 64 bit version of EL5
- Register the machine with redhat (rhn_register)
- enable virtualization entitlement for the system in RHN
- Install KVM package:
# yum install kvm
# yum install virt-manager libvirt libvirt-python python-virtinst

Migration VMware virtual machines to KVM:
- Login to the vmware server
- make single vmdk image with vmware-diskmanager
eg:
# vmware-vdiskmanager -r path_to_vmware_virtualmachine.vmdk -t 0 destination_file_vmware.vmdk
Creating disk 'destination_file_vmware.vmdk'
Convert: 100% done.
Virtual disk conversion successful.

- Copy the image to KVM server
- Convert the image to KVM supported format with qemu-img
# qemu-img convert destination_file_vmware.vmdk -O qcow2 kvm_supported.img

Create bridge interface to to share the network card.
* This section assumes that you have two nic in your server and would need to have bonding along with bridging and you have static ip required for virtual machines. incase you using dhcp and single network interface create the bridge interface accordingly.

- Create bridge interface:
$ cat /etc/sysconfig/network-scripts/ifcfg-br0

DEVICE=br0
ONBOOT=yes
TYPE=Bridge
IPADDR=11.11.11.11
NETMASK=255.0.0.0
GATEWAY=1.1.1.1


- Configure the bond interface:
$ cat /etc/sysconfig/network-scripts/ifcfg-bond0

DEVICE=bond0
BRIDGE=br0
ONBOOT=yes


- Configure eth0 and eth1
$ cat /etc/sysconfig/network-scripts/ifcfg-eth0

DEVICE=eth0
MASTER=bond0
SLAVE=yes
ONBOOT=yes


- Change bonding to active-backup , i have faced some issues with xor - might be silly to fix
# cat /etc/modprobe.conf

options bond0 miimon=100 mode=active-backup

- Restart network interface and check the bridge status
# brctl show , it will show bond0 as an enabled interface.

Create KVM virtual machine:
- it can be done from the command line or with virt-manager
- open virt-manager application
- click create new, and select qemu hypervisor
- during disk selection, choose the converted vmware image path
- done, just start it.

Register the virtual machine with Redhat, save some license ;)


- enabled network tools entitlement in RHN
- install the package rhn-virtualization-host on the core machine
- # yum install rhn-virtualization-host
- enable virtualization under the properties of host in RHN
- execute the following commands on host machine
# rhn_check
# rhn-profile-sync
- login to virtual machine and use rhn_register, now it will be registered as a virtual machine under the core license.

./arun

Saturday, 20 March 2010

Enable Full virtualization in HP DL servers (Intel)

You need to enable hardware virtualization in BIOS if you want to create Fully virtualized instances.

Enter BIOS (F9) --> Advanced Options --> Processor Options --> Enable intel Virtualization Technology

Now you should be able to create Fully virtualized virtual machines from XEN or similar virtualization packages without OS modifications.

./arun

Thursday, 18 March 2010

Issues with zone transfer in Dual stack IPv4 / IPv6

You might face issues with zone transfer to ipv4 secondaries on a dual stack server where the bind listening on IPv4 and IPv6 address,

client ::ffff:11.11.11.11#43253: zone transfer 'example.com/IN' denied

this happens because , once the v6 is enabled on bind it just try to make ipv4 address looks like v6 address.

Solution : just add the v6 formatted v4 address to the allowed list

allow transfer { ::ffff:11.11.11.11; };


./arun

Configure Apache over IPv6

Once your network interface is configured with IPv6, it is easy to configure the webserver. No real difference with IPv4 configuration.

Configure Apache to listen the IPv6 address:

Listen ipv6_address:port
NameVirtualHost ipv6_address:port


If the apache virtual host is configured with domain name , eg < VirtualHost arunns.com:80 >, just add AAAA record for arunns.com in dns and the website will work without any extra configurations other than the previous two lines.

Also we can specifically configure it :

< VirtualHost ipv4_address:80 ipv6_address:80 >


It is possible to have different contents for ipv4 and ipv6 sites, just create two different virtual hosts with different document root one for IPv4 and other for IPv6.

< VirtualHost ipv4_address:80 >
DocumentRoot /home/123/
< /VirtualHost >
< VirtualHost ipv6_address:80 >
DocumentRoot /home/456/
< /VirtualHost >

./arun

Thursday, 11 March 2010

IPV6 Tunnel from MAC/Linux

It is really easy to establish an ipv6 network tunnel from your machine directly. Make your network/system/services IPv6 ready :)

Create a ipv6 regular tunnel from any connection brokers: List of IPV6 tunnel brokers

I have used Hurricane Electric which is free tunnel broker.
Tunnel Broker

With the tunnel broker, you can create a tunnel by specifying your public ipv4 address in their website.

Once the tunnel is created with tunnel broker, Configure your machine with required interfaces , tunnel and routing.

    For MAC OS X:


1) Configure tunnel
$ sudo ifconfig gif0 tunnel host_ip tunnel_broker_ipv4_ip

If you are behind a natd network specify your machine private address as host_ip, otherwise mention the current public ip assigned to your machine. If you are behind a nat'd network make sure that protocol 41 is allowed in the nat'd device.
eg:
$ sudo ifconfig gif0 tunnel 192.168.1.2 216.66.xxx.xxx

2) Setup the tunnel end points

$ sudo ifconfig gif0 inet6 host_ipv6_address tunnel_broker_ipv6_address prefixlen 128

Both these ipv6 addresses are assigned by the tunnel broker.
eg:
$ sudo ifconfig gif0 inet6 2001:470:xxxx:xxxx::2 2001:470:xxxx:xxxx::1 prefixlen 128

3) Add the default route for ipv6 traffic
$ sudo route -n add -inet6 default tunnel_broker_ipv6_address

eg:
$ sudo route -n add -inet6 default 2001:470:xxxx:xxxx::1

Now you should be able to access the ipv6 networks :)

Incase of any issues, just make sure that ipv6 is enabled on the interface using:

$ sudo ip6 -x gif0

Test your ip6 connectivity:


$ ping6 ipv6.google.com
$ telnet ipv6.google.com 80


[gallery]

    For Linux:


The procedure is exactly same on linux as well:

Make sure that the ipv6 module is present in the kernel:

$ sudo modprobe ipv6

Create the tunnel
$ sudo ip tunnel add he-ipv6 mode sit remote 216.66.xx.xx local 192.168.1.2 ttl 255
* use the public ip if it is directly assigned to your machine

Activate the tunnel
$ sudo ip link set he-ipv6 up

Assign ip address to interface:

$ sudo ip addr add 2001:470:xxxx:xxxx::2/64 dev he-ipv6

Add default route for ipv6:

$ sudo ip route add ::/0 dev he-ipv6

Add protocol family identifer:

$ sudo ip -f inet6 addr

./arun

Tuesday, 9 March 2010

IPv6 and Linux

It is straight forward to enable IPv6 on any linux system, since the latest kernel support it very well. This document is more relevant for Redhat linux but the idea is same for all.

Make sure the ipv6 support is not disabled in kernel
Comment out the following line in /etc/modprobe.conf if existing.

#alias ipv6 off
#alias net-pf-10 off


Enable IPv6 networking:
edit /etc/sysconfig/network

NETWORKING_IPV6=yes

Configure the IPv6 address:
edit /etc/sysconfig/network-scripts/ifcfg-eth0 (or bond0 for bond interfaces, ipv6 works as expected with bond interface as well)

IPV6INIT=yes
IPV6ADDR=
IPV6ADDR_SECONDARIES=
IPV6_DEFAULTGW=
IPV6_AUTOCONF=yes/no


Just restart network and you will be able to see the IPv6 address.

Incase if the ipv6 module doesnt exist in kernel, just do a modprobe:
# modprobe -a ipv6

Almost all softwares in linux works with IPv6,

For apache add the listen address to ipv6 address and enable name virtual host for ipv6 address if required.

You can test your ipv6 connectivity by:
$ ping6 ipv6.google.com

./arun

Friday, 12 February 2010

Android ADP1 Firmware to CyanogenMod

This note is more specific to mac os x, more likely the same steps will works with any OS.

CyanogenMod is a customized, aftermarket firmware distribution for the HTC Dream and Magic cell phones, and the Google Nexus One.

Based on the open-source Android operating system, CyanogenMod is designed to increase performance and reliability over Android-based ROMs , such as FLAC Lossless Audio, multi-touch support, the ability to store applications on the microSD card, and support for tethering.

Here are the steps I followed to get my ADP1 installed with Cyanogen mod.

- Backup your data if required, this steps will completely erase your mobile applications and settings.



Step 1:
Create a source directory:
$ mkdir ~/cyanogen
$cd ~/cyanogen





Download fastboot to your computer:
Refer this url to get the correct fastboot.
HTC Developer site






Download the latest compatible version of Cyanogenmod:
$ wget http://cyanogenmod.com/download/recovery/cm-recovery-1.4.img




Step 2:
Boot the android device in fastboot mode.
To enter fastboot mode, reboot the device while holding down the BACK key.

Once it is in to fast boot mode, flash the Cyanogen recovery image:


$ cd ~/cyanogen
$ ./fastboot-mac flash recovery cm-recovery-1.4.img
sending 'recovery' (4002 KB)... OKAY
writing 'recovery'... OKAY


Then place the phone in to flash boot mode


$ ./fastboot-mac boot cm-recovery-1.4.img
downloading 'boot.img'... OKAY
booting... OKAY
$ adb shell mount -a
* daemon not running. starting it now *
* daemon started successfully *
$ adb shell
/ # mv /system/etc/install-recovery.sh /system/etc/install-recovery.sh.disabled


Reboot back to fastboot mode.

$ ./fastboot-mac flash recovery ../cm-recovery-1.4.img
sending 'recovery' (4002 KB)... OKAY
writing 'recovery'... OKAY

$ ./fastboot-mac reboot



Download the Android 1.6 Recovery Image (signed-dream_devphone_userdebug-ota-14721.zip) if dont have:
From HTC (or) android website - This is just to get the adp tool

Download the latest CyanogenMod ROM for your device:
Dream (G1): Latest version (stable)
update-cm-4.2.14.1-signed.zip



Export the CyanogenMod ROM to SD card.
$ adb push update-cm-4.2.14.1-signed.zip /sdcard/update-cm-4.2.14.1-signed.zip



But this was not working properly for me, for some reason the adb tool was not able to detect the device. So i just copied it over.




$ cp update-cm-4.2.14.1-signed.zip /Volumes/SDCARD/

* no need for this step if adb tool worked




Now are all set to install:



- Turn your phone off.
- Get in to recovery mode my holding Home button while booting
- select wipe data/factory reset and press home to confirm.
- select apply any zip from sd
- Pick signed-dream_devphone_userdebug-ota-14721.zip.
- Press home to confirm and wait for installation to complete
- select apply any zip from sd again
- Pick update-cm-4.2.14.1-signed.zip.
- Press home to confirm and wait for installation to complete

- reboot


That is it!

Reference : Full_Update_Guide_-_ADP1_Firmware_to_CyanogenMod

Friday, 5 February 2010

My favorite android applications

Multimedia


From Google
Goggles : Use pictures to search web, just amazing. A visual search application, instead of using words, take a picture and search. Works very well with logo's. one of my top favorites. More info
Google Goggles

Ringtone
Ringdroid : Easy way to make ring tones from songs we have. More info ringdroid

Picture editor
Picsay : Quick and effective way to add effects to pictures from mobile. More info Picsay

Radio
A Online Radio : Excellent way to listen online radio stations aor



Communication



SMS
Handcent SMS : found it better than the native message application. More info handcent

SMS Backup : useful to backup sms messages to gmail with a separate label android-sms


Voice
Sipdroid : SIP/VOIP client, very useful application especially to connect and make calls with your sip provider like action voip. : More info : sipdroid.org



Networking and System tools


SSH
Connectbot An easy way to establish ssh connection, support ssh key generation and public key authentication.connect bot

Monitoring
NagMonDroid Nice application to connect to your nagios instance and display short summary. I found it very useful to get my server status while i am on road :) More info :nagios page

CIDR
CIDR calculator Very useful IP subnet calculator. More info : CIDR Androidlib

Monday, 25 January 2010

upgrade ubuntu to next release from command line

It is better to update the current installation with latest packages first:

$ sudo apt-get update

Install update manager core if not:

sudo apt-get install update-manager-core

Start the command line upgrade tool

$ sudo do-release-upgrade

Wednesday, 20 January 2010

svn+ssh with custom port number and public key authentication

To make custom configurations for svn+ssh:

Edit ~/.subversion/config

- add the ssh configuration details under [tunnels]
like:
foobar = /usr/bin/ssh -i /home/foo/.ssh/foobar.private -p 12345

Now use:
svn co svn+foobar://user@svn.test.com/home/test/repos/foobar

create svn repository and initial check in

To create svn repository login to the svn server:

$ sudo -u svnuser svnadmin create --fs-type fsfs /path/to/repository
* we can use bdb as well as db format

To make all the group members privilege to write access the repository:

$chmod g+w /path/to/repository

and add the user to svn group.

To create initial contents:
either you can check out the repository and create the file structure like:

[local_machine]$ svn co svn+ssh://user@svnhost/path/to/repos localdirectory
[local_machine]$ mkdir -p localdirectory/trunk localdirectory/tags localdirectory/branches
[local_machine]$ cd localdirectory; svn commit -m "initial repository structure"


or you can do the same from the svn server itself using file:///

Sunday, 10 January 2010

Mysql one way DB replication

One way replication of mysql database:

Mysql replication help us in keeping the data replicated to one or more sites reliably with binary logs. Apart from good amount of advantages Mysql replication doesn't help with data corruption, since the corrupted data is replicated in all slaves. It is good to have periodic backup of database apart from replication.

Replication Steps

- Create database with same name on all servers
> mysql -u db_user -p -e "CREATE DATABASE db_name"

- Create database user with replication privilege on master
> GRANT REPLICATION SLAVE ON *.* TO 'replication_user'@'replication_clients" IDENTIFIED BY 'replication_password'
This can be also supplied with particular database name with ;db_name.*' instead of *.*

- Edit Mysql master configuration (my.cnf) to allow replication
[mysqld]
server-id = 1 # Important with replication
log-slave-updates
log-bin = /var/lib/mysql/mysql-bin
log-bin-index = /var/lib/mysql/mysql-bin.index
replicate-do-db = db_name # specify the dbs to replicate
log-warnings
innodb_flush_log_at_trx_commit=1
sync-binlog=1
innodb_safe_binlog


- Take dump of master db and put them on all replicas

use db_name;
FLUSH TABLES WITH READ LOCK;


$ mysqldump -u dbuser -p db_name > db_dump.sql
install on slaves
$ mysql -u dbuser -p db_name > db_dump.sql


use db_name;
UNLOCK TABLES;


- Edit mysql configuration on replicas with master credentials

[mysqld]
old_passwords=1
server-id=2
innodb_file_per_table
log-slave-updates
master-host = master_hostname
master-port = master_port
master-user = master_user
master-password = master_password
log-bin = /var/lib/mysql/mysql-bin
log-bin-index = /var/lib/mysql/mysql-bin.index


Restart the Mysql daemon on all servers and check the replication status:
Master: > show master status;
Replicas: > show slave status;

Wednesday, 6 January 2010

Share internet with MAC OS X

There are situations we need to share internet connection from one machine, MAC support to share internet
From: Firewire, Ethernet, Airport, Bluetooth To: Firewire or Airport or Bluetooth PAN or Ethernet.

Simply get your mac connected to internet, say with ethernet

Enabled Internet sharing:
System Preference --> Sharing --> Internet Sharing

Select source of connection and mode of sharing
If you share via airport you have options to set encryption to avoid anonymous to use your connection.

Also to make your internet kinda nat'd,
Assign different network ip to the airport interface.

Say :
You are connected to internet with a dhcp ip 192.168.10.6 on ethernet port
Assign a different network ip 10.1.1.1 to the Airport interface and set gateway as 192.168.10.6
Configure 10.1.1.2 with gateway 10.1.1.1 on the client machine.


This way all the connection will be shown as single ip to the provider ;)

Hardening Apache webserver

Tips to harden apache webserver:

Disable weak SSLV2 siphers
edit ssl.conf and add
SSLCipherSuite ALL:!ADH:!EXPORT56:RC4+RSA:+HIGH:+MEDIUM:!LOW:!SSLv2:+EXP

Restrict apache to giveout minimum informations
Edit httpd.conf and change
ServerTokens ProductOnly

Disable track and trace in every virtual hosts
LoadModule rewrite_module modules/mod_rewrite.so
RewriteEngine On
RewriteCond %{REQUEST_METHOD} ^TRACE
RewriteRule .* - [F]


Always better to configure apache to Loan minimal/required modules and include only necessary config files.

./arun

Tuesday, 5 January 2010

Mysql backup script

To backup mysql on a daily/hourly basis with time stamp and compress it after backup also it will remove the files older than x days.

#!/bin/bash
# Arun N S
# variables
DATE="$(date +"%d-%m-%Y")"
TIME="$(date +"%d-%m-%Y-%H%M")"
USER=username
PASSWORD=password
DATABASE=dbname



# Directories and dump
/bin/mkdir -p /backup/Mysql/$DATE
/usr/bin/mysqldump -l -F -u $USER --password=$PASSWORD $DATABASE > /backup/Mysql/$DATE/backup_$TIME.sql



# Compressing
/usr/bin/bzip2 /backup/Mysql/*/*.sql



#Removing files older than x days eg: 90 days
for i in `/usr/bin/find /backup/Mysql/ -maxdepth 1 -type d -mtime +90 -print`; do
/bin/echo -e "Deleting old directories $i"; /bin/rm -rf $i; done

Saturday, 2 January 2010

Fix for - - mixing * ports and non-* ports with a NameVirtualHost address is not supported

If you happend to see this error in apache logs,

- mixing * ports and non-* ports with a NameVirtualHost address is not supported, proceeding with undefined results

Check your apache configuration, to make sure that there is no VirtualHost defined without port numbers like : <VirtualHost 127.0.0.1>, if existing fix it with port number.

Friday, 1 January 2010

Mysql Queries and Tips

Some useful database queries:

login to mysql database:

$ mysql $database_name -u $user_name -h $host_name -p

Take a dump by locking transactions

$ mysqldump -l -F $database_name -u $user_name -h $host_name -p > file.sql
*need lock table and read privilege from the host you trying

Take backup of only some tables
$ mysqldump -l -F $database_name -u $user_name -h $host_name --tables $tables_name -p > file.sql

Take backup of only database structure , without data
$ mysqldump -l -F -d $database_name> -u $username -h $hostname -p > file.sql

List the permissions assigned for a user
login to mysql:
> show grants for 'user'@'hostname';

sed tips

Remove trailing space from a file using sed

$cat | sed 's/[ \t]*$//' >

Find Tips

Remove files older than certain days (using find/mtime)
find -name "" -mtime +N -exec rm -r {} \;

Eg : find /var/log/ -name "*.log" -mtime +5 -exec rm -r {} \;
This will remove the *.log files older than 5 days in directory /var/log/

Find with file type

directories : find / -type d -print0
files: find / -type f -print0

Remove held messages from mailman queue

If you have too many mails/spams tend to pending moderator requests in mailman queue it can be removed by:

$ cd ~mailman
$ bin/discard /var/lib/mailman/data/heldmsg--*

And if the no. of argument list exceeds you can try:

$ find /var/lib/mailman/data -name heldmsg--\* -print | xargs bin/discard

* the path of mailman depends on your implementation anyway.