Migrating Amazon Instance
Contents |
Intro To The Topic
I have covered this topic as we moving large Amazon instances to small one's, but this not the only case that comes in to mind.
Migration can be as a response to
1) Shifting from 32 bit system to 64 bit or vice-versa. 2) Shifting to smaller or larger instance. 3) Process of replicating an instance. 4) etc.
Basic steps
1. start a small instance of any linux distro. 2. install gentoo in a chroot 3. rebundle that chroot into an AMI. 4. replicate packages from issues large to issues small. 5. replicate config. 6. testing. 7. announcements. 8. change the elastic ip. announce again.
Building Gentoo system in "chroot" environment
Preparation for chroot
Make a separate directory
mkdir /mnt/gentoo
Download the required base system needed to boot.
cd /mnt/gentoo wget http://funtoo.org/linux/gentoo/i686/gentoo-i686-2009.06.23/stage3-i686-2009.06.23.tar.bz2
Untar gentoo stage3 tarball and mount other required drives.
tar jxpvf stage3-i686-2009.06.23.tar.bz2 cp -r /lib/modules/`uname -r` /mnt/gentoo/lib/modules/`uname -r` mount -o rbind /proc /mnt/gentoo/proc mount -o rbind /dev /mnt/gentoo/dev mount -o rbind /sys /mnt/gentoo/sys cat /proc/mounts > /mnt/gentoo/etc/mtab
chroot into the directory
chroot /mnt/gentoo /bin/bash
Building Packages
Setting up the environment
Update the environment with these commands.
env-update source /etc/profile export PS1="(<hostname>) $PS1" depmod -a modprobe loop echo 'loop' >>/etc/modules.autoload.d/kernel-2.6
Setup base system
Set password for root user
passwd root
Set Timezone and other configurations
cp /usr/share/zoneinfo/Europe/Madrid /etc/localtime
Edit /etc/conf.d/clock to look like this
CLOCK="UTC" TIMEZONE="Europe/Madrid" CLOCK_SYSTOHC="yes"
Set locale
Edit /etc/locale.gen to look like this
en_US.UTF-8 UTF-8
Edit /etc/env.d/02locale to look like this
LANG=en_US.UTF-8 LC_CTYPE="en_US.UTF-8" LC_NUMERIC="en_US.UTF-8" LC_TIME="en_US.UTF-8" LC_COLLATE="en_US.UTF-8" LC_MONETARY="en_US.UTF-8" LC_MESSAGES="en_US.UTF-8" LC_PAPER="en_US.UTF-8" LC_NAME="en_US.UTF-8" LC_ADDRESS="en_US.UTF-8" LC_TELEPHONE="en_US.UTF-8" LC_MEASUREMENT="en_US.UTF-8" LC_IDENTIFICATION="en_US.UTF-8" LC_ALL=en_US.UTF-8
Editing other system configuration files
Edit /etc/make.conf to look like this
CFLAGS="-O2 -pipe -march=i686 -mno-tls-direct-seg-refs"
CXXFLAGS="${CFLAGS}"
CHOST="i686-pc-linux-gnu"
MAKEOPTS="-j2"
USE="-ipv6 -gdbm -acl apache2 ldap vhosts"
GENTOO_MIRRORS="http://ftp.heanet.ie/pub/gentoo/ "
SYNC="rsync://rsync.europe.gentoo.org/gentoo-portage"
Edit /etc/fstab to look like this
/dev/sda1 / ext3 user_xattr 0 1 /dev/sda2 /mnt ext3 user_xattr 0 2 /dev/sda3 swap swap sw 0 0 shm /dev/shm tmpfs nodev,nosuid,noexec 0 0
Edit /etc/inittab to disable all the terminals except the 1st one (c1). They'll never be used.
c1:12345:respawn:/sbin/agetty 38400 tty1 linux #c2:2345:respawn:/sbin/agetty 38400 tty2 linux #c3:2345:respawn:/sbin/agetty 38400 tty3 linux #c4:2345:respawn:/sbin/agetty 38400 tty4 linux #c5:2345:respawn:/sbin/agetty 38400 tty5 linux #c6:2345:respawn:/sbin/agetty 38400 tty6 linux * Note: This is only a part of the full file.
Rebuild packages
emerge --sync emerge -auDvNe world
As a measure of security we will restrict ssh to public key login only and not root ssh.
useradd <user> passwd <user> mkdir -p /home/<user>/.ssh * Add user to wheel group to give "su" access.
Copy public key of your system to /home/<user>/.ssh/authorized_keys, so that you can login to this system later. And set the permissions of the home directory for error free login.
chown <user>:<user> /home/<user> -R chmod 700 /home/<user>/.ssh chmod 400 /home/<user>/.ssh/authorized_keys
Edit /etc/ssh/sshd_config to deny root access.
Change PermitRootLogin yes TO PermitRootLogin no
Building new packages
Build necessary packages and make them start at boot up.
emerge -av dhcpcd emerge -av syslog-ng emerge -av vixie-cron
rc-update add sshd default rc-update add vixie-cron default rc-update add syslog-ng default
Edit /etc/dispatch-conf.conf to ease configuration file updates.
diff="diff -Nu '%s' '%s' | less --no-init --QUIT-AT-EOF" TO diff="colordiff -Nu '%s' '%s' | less --no-init --QUIT-AT-EOF" * Note: This is only a part of the full file.
Run dispatch-conf
dispatch-conf * Note: This will as few questions, you can select some options as "u" to take the new one. "z" to ignore it, but be carefull and read what it means.
Copy configuration files from old server
Few files need to be either copied or to be referred from the old server for the smooth functioning of the new server.
Update /etc/conf.d/hostname Update /etc/hosts Copy /var/lib/portage/world Copy /etc/portage/package.use Copy /etc/portage/package.keywords
Building packages present in old server
After coping /var/lib/portage/world from old to new server to install all the packages present in old server just run these commands.
emerge -auDvN world emerge -a --depclean revdep-rebuild -- -a
Starting new AMI
Configuring ec2-ami-tools
cd /usr/local/ wget wget http://s3.amazonaws.com/ec2-downloads/ec2-ami-tools.zip unzip ec2-ami-tools.zip rm ec2-ami-tools.zip ln -s ec2-ami-tools* ec2-ami-tools * Copy /usr/local/sbin/image from the old to the new server. * Edit /usr/local/sbin/image and replace x86_64 with i386 (if changing from 64 to 32 bit system). * Copy /root/.ec2 too from one to the other.
Running Image script
Image script creates an image (bundle) of the system installed in the chroot environment.
export HOSTNAME=<hostname> /usr/local/sbin/image
Then this image in un-bundled with the help of Amazon commands to create a new AMI. Basically these commands are used to register the xml manifest which in turn returns the AMI.
issues.openbravo.com Specific Configurations
As for issues server we need services like apache, mysql, qmail to be configured properly.
Making apache and mysql to start at boot
rc-update add apache2 default rc-update add mysql default
Copy configuration
Copy /etc/apache * Note: Make required changes (eg. change ServerRoot "/usr/lib/apache2" from lib64 in /etc/apache2/httpd.conf)
Copy /var/qmail/control/servercert.cnf
Restart services
/etc/init.d/apache2 restart emerge --config netqmail
Setting up qmail
useradd openbravo * Note: openbravo user is added to receive system mails.
cd /var/qmail/alias echo openbravo > .qmail-root echo openbravo > .qmail-postmaster echo openbravo > .qmail-mailer-daemon
rc-update add svscan default /etc/init.d/svscan start * Note; This is required to monitor/supervise qmail service.
ln -s /var/qmail/supervise/qmail-send qmail-send
Test Qmail
Test qmail by sending a mail. A sample is here to help you as it helped me ;-)
cat << EOF > /tmp/mail From: testing@openbravo.com To: some@mail.com Subject: testing from issues This is a test EOF
cat /tmp/mail | /usr/sbin/sendmail -t -f $FROMADDR
For the final step of shifting to new issues we would need access from old server to new through ssh. So enable root access via ssh from new issues to old issues.
* Note: Generate a key for root in new issues, add it to authorized_keys in old and set permitrootlogin to yes in sshd_config. then reload ssh.
Copy apache/mantis data
rsync -avHx --delete root@issues.openbravo.com:/var/www/ /var/www/ /etc/ini.d/apache2 restart
Test Apache
* Note: Check new IP in your browser for Apache to respond.
Setup to enable password less access from code.openravo.com
useradd hgbot passwd hgbot Copy /home/hgbot/.ssh/ from old to new * Note: This copying provides password less access, as it includes authorized_keys file.
Few checks
To ensure your system to be exact copy of old system, match the output of few commands on both the servers.
rc-update show default crontab -l
System Backup
There are two types of backup.
1) entire machine. 2) mantis db, sources, etc.
For 1) copy the old issues /usr/local/sbin/image to the new one and copy the crontab of root from the old to the new.
For 2) (i.e. copy the database and mantis to amazon S3) we have a self-made script in /usr/local/sbin/s3backup.sh
Copy /usr/local/s3sync from old server to the new one. Copy /usr/local/sbin/s3backup.sh from old server to the new one.
End up activities (issues.openbravo.com specific)
- you send an e-mail to openbravo-development@lists.sourceforge.net - we shut down apache in old server. - we rsync the mysql database (adding --delete this time). - Change the elastic IP, assigning it to the new instance. - Announce again daying that it's finished.