Petr Bena's Bl0g

Based on https://discussions.citrix.com/topic/378478-xenserver-7-raid1-mdadm-after-install-running-system/

# 1. Install Xenserver 7 with normal single disk configuration, don't create SR storage
# 2. copy partition talbe from sda to sdb
 
# !!! important don't write the order wrongly, from sda to sdb is like the following
sgdisk /dev/sda -R /dev/sdb


# Important! The partition layout may differ with XenServer version, basically there are 2 partitions with same size for OS and backup
# and at least 1 for GRUB and 1 for swap

parted /dev/sdb
# print
# quit

# You should see a list of partitions, one of them will be flagged as legacy_boot, grub, let's call it BOOT

## flag them as raid disks for sdb partitions
sgdisk --typecode=1:fd00 /dev/sdb # OS
sgdisk --typecode=2:fd00 /dev/sdb # Backup OS
sgdisk --typecode=3:fd00 /dev/sdb # ??
sgdisk --typecode=4:ef02 /dev/sdb # BOOT
sgdisk --typecode=5:fd00 /dev/sdb # LOGS
sgdisk --typecode=6:fd00 /dev/sdb # SWAP

## note that BOOT partition is not the same like the others. Because in the new disk configuration they changed the boot partition.

# 5. create the software raid partitions

mdadm --create /dev/md0 --level=1 --raid-devices=2 --metadata=0.90 /dev/sdb1 missing
mdadm --create /dev/md1 --level=1 --raid-devices=2 --metadata=0.90 /dev/sdb2 missing
mdadm --create /dev/md2 --level=1 --raid-devices=2 --metadata=0.90 /dev/sdb3 missing
mdadm --create /dev/md3 --level=1 --raid-devices=2 --metadata=0.90 /dev/sdb4 missing
mdadm --create /dev/md4 --level=1 --raid-devices=2 --metadata=0.90 /dev/sdb5 missing
mdadm --create /dev/md5 --level=1 --raid-devices=2 --metadata=0.90 /dev/sdb6 missing
mkswap /dev/md5

# 6. copy the contents of / and /var/log directories to the new partitions

mkfs.ext3 /dev/md0
mkfs.ext3 /dev/md4

# 7. mount newly created/formatted partitions
mount /dev/md0 /mnt
mkdir -p /mnt/var/log
mount /dev/md4 /mnt/var/log

# 8. copy contents to the newly mounted directory

cp -xR --preserve=all / /mnt

# 9. create a mdadm file for boot process (!!!if you forget the file the MD devices will have different names)

### the head of the file should include these lines
echo "MAILADDR root" > /mnt/etc/mdadm.conf
echo "auto +imsm +1.x -all" >> /mnt/etc/mdadm.conf
echo "DEVICE /dev/sd*[a-z][1-9]" >> /mnt/etc/mdadm.conf
mdadm --detail --scan >> /mnt/etc/mdadm.conf

# 10. copy the contents to the root folder
cp /mnt/etc/mdadm.conf /etc

# 11. configure mount points
sed -i 's/LABEL=root-[a-zA-Z\-]*/\/dev\/md0/' /mnt/etc/fstab
sed -i 's/LABEL=swap-[a-zA-Z\-]*/\/dev\/md5/' /mnt/etc/fstab
sed -i 's/LABEL=logs-[a-zA-Z\-]*/\/dev\/md4/' /mnt/etc/fstab
sed -i '/md5/ a\/dev/md5          swap      swap   defaults   0  0 ' /mnt/etc/fstab
cp /mnt/etc/fstab /etc

# 12. change the label name for /dev/sdb1 partition
e2label /dev/sda1 |xargs -t e2label /dev/sdb1

# 13. bind mount dev sys proc to the mnt folder
mount --bind /dev /mnt/dev
mount --bind /sys /mnt/sys
mount --bind /proc /mnt/proc
chroot /mnt  /bin/bash

# 14. install grub on /dev/sdb
grub-install /dev/sdb

# 15. backup initrd
cp /boot/initrd-$(uname -r).img /boot/initrd-$(uname -r).img.bck

# 16. create new initrd for raid
dracut --mdadmconf --fstab --add="mdraid" --filesystems "ext3 tmpfs devpts sysfs proc" --add-drivers="raid1 raid456 mdraid1x mdraid09" --force /boot/initrd-$(uname -r).img $(uname -r) -M

###never change the boot configuration via grub-mkconfig.. it will kill xenserver.. change the GRUB configuation, by hand inside the files

# 17. change grub configuration
sed -i 's/quiet/rd.auto rd.auto=1 rhgb quiet/' /boot/grub/grub.cfg
sed -i 's/LABEL=root-[a-zA-Z\-]*/\/dev\/md0/' /boot/grub/grub.cfg
sed -i '/search/ i\   insmod gzio' /boot/grub/grub.cfg
sed -i '/search/ i\   insmod part_msdos' /boot/grub/grub.cfg
sed -i '/search/ i\   insmod diskfilter mdraid09' /boot/grub/grub.cfg
sed -i '/search/ c\   set root=(hd0,gpt1)' /boot/grub/grub.cfg

# 18. exit from chroot
exit

# 19. change the same things in sda1 partition so after reboot you don't need to boot from second disk
cp /mnt/boot/initrd-3.10.0+10.img /boot/

sed -i 's/quiet/rd.auto rd.auto=1 rhgb quiet/' /boot/grub/grub.cfg
sed -i 's/LABEL=root-[a-zA-Z\-]*/\/dev\/md0/' /boot/grub/grub.cfg
sed -i '/search/ i\   insmod gzio' /boot/grub/grub.cfg
sed -i '/search/ i\   insmod part_msdos' /boot/grub/grub.cfg
sed -i '/search/ i\   insmod diskfilter mdraid09' /boot/grub/grub.cfg
sed -i '/search/ c\   set root=(hd0,gpt1)' /boot/grub/grub.cfg

# 20. reboot

# !!!!!!reboot the server the server will boot from software raid..!!!!!
# 21. After the reboot add the /dev/sda to the new MD disks.

sgdisk /dev/sdb -R /dev/sda

mdadm -a /dev/md0 /dev/sda1
mdadm -a /dev/md1 /dev/sda2
mdadm -a /dev/md2 /dev/sda3
mdadm -a /dev/md3 /dev/sda4
mdadm -a /dev/md4 /dev/sda5

# This will take a while for resync of all disks

grub-install /dev/sda

# Create SR

xe sr-create content-type=user device-config:device=/dev/md2 host-uuid=<host-uuid> name-label=”SRRaid1-Local” shared=false type=lvm

This post is basically just a backup of that forum post in case it become dead link

For some reason citrix doesn’t like mdadm so they make everything possible to stop it from working on their xen server.

Here is a guide that would make it work there, but it may not survive system patching

Setup

Connect at least 2 disks to your box. Install a xen server without local storage on first disk.

Installing mdadm

The default install contains mdadm but it doesn’t load raid modules to kernel. In order to enable it, following needs to be done:

echo "modprobe raid1" > /etc/sysconfig/modules/raid.modules
modprobe raid1
chmod a+x /etc/sysconfig/modules/raid.modules

Partitioning the disks

Now we create a final schema we want to use on our server on disk /dev/sdb, xen needs to have at least 3 partitions, 1 is for boot loader, second is for OS, I recommend 20gb or more, because this disk is pretty much impossible to extend, although citrix defaults it to 4GB, last partition is for local storage and it should take all remaining space on disk.

Note: Citrix by default creates 3 partitions, 1 for OS, second is empty, same sized as first one and probably used for system upgrade. Third is used for local storage LVM. You don’t have to create second partition for it to work, but system upgrades may not be available if you don’t create it. On other hand system upgrades will likely not work anyway as citrix doesn’t support mdadm installations.

In this guide I will use old MS-DOS partition table because although it’s old, it’s much better supported and it just works. You can also use GPT partitions if you want, but I had some issues getting them work with mdadm and syslinux.

We will have a separate /boot partition for boot loader, because syslinux shipped with xen is having troubles booting from raid device for some reason.

So this is how the layout of sdb should look after we finish the partitioning:

• /dev/sdb1 (2 GB) for bootloader
• /dev/sdb2 (20 GB) for OS
• /dev/sdb3 (rest) for LVM

# Configure disk
sgdisk --zap-all /dev/sdb
# Now run fdisk and create a new dos partition table
# make 2 partitions, one is for boot (/dev/sdb1 2gb) and one for Dom0 (/dev/sdb2 20gb)
#### after it's done create md device
mdadm --create /dev/md0 --level=1 --raid-devices=2 missing /dev/sdb2
mkfs.ext3 /dev/md0
mkfs.ext3 /dev/sdb1
mount /dev/md0 /mnt
cp -vxpR / /mnt
cd /mnt
mv boot /tmp/old_boot
mkdir boot
mount /dev/sdb1 /mnt/boot
mv /tmp/old_boot/* boot
# Fix /mnt/etc/fstab - replace LABEL with /dev/md0 and insert a record for /boot
# EXAMPLE:
head -2 /etc/fstab
/dev/md0    /         ext3     defaults   1  1 /dev/sda1   /boot     ext3     defaults   1  1  
# Update boot loader 
# You need to open /boot/extlinux.conf and replace all references to old disk with root=/dev/md0
mkdir /mnt/root/initrd-raid mkinitrd -v --fstab=/mnt/etc/fstab /mnt/root/initrd-raid/initrd-`uname -r`-raid.img `uname -r` cd /mnt/root/initrd-raid zcat initrd-`uname -r`-raid.img | cpio -i
mdadm --detail --scan >> etc/mdadm.conf
find . -print | cpio -o -Hnewc | gzip -c > /mnt/boot/initrd-`uname -r`-raid.img rm /mnt/boot/initrd-3.10-xen.img cd /mnt/boot ln -s initrd-`uname -r`-raid.img initrd-3.10-xen.img
extlinux -i boot/
cat /usr/share/syslinux/mbr.bin > /dev/sdb
# Open /mnt/boot/extlinux.conf
# remove absolute path to xen.gz to relative, /boot will be root device for bootloader (/xen.gz), replace LABEL with /dev/md0

###### example conf file for syslinux that works ######
# location mbr ui vesamenu.c32 serial 0 115200 default xe prompt 1 timeout 50

label xe
menu label XenServer
kernel mboot.c32
append xen.gz dom0_mem=752M,max:752M watchdog dom0_max_vcpus=2 crashkernel=128M@32M cpuid_mask_xsave_eax=0 console=vga vga=mode-0x0311 --- /boot/vmlinuz-3.10-xen root=/dev/md0 ro hpet=disable xencons=hvc console=hvc0 console=tty0 --- initrd-3.10-xen.img 
#######################EOF###################### 

reboot

Now you should be able to boot from /dev/sdb if you are not there is something wrong with the setup, you need to figure out if your problem is with

• MBR (No bootable device)
• Boot loader (Missing operating system.)
• /boot (Linux will start booting but die in progress – try removing quiet and splash from parameters)

 Syncing the disks

Now if you were able to boot up you need to setup the sda disk

Create the same 3 partitions as you did on sdb on sda and then

dd if=/dev/sdb1 of=/dev/sda1
# mbr
cat /usr/share/syslinux/mbr.bin > /dev/sda
mdadm --add /dev/md0 /dev/sda2

Wait for disks to sync, meanwhile you can create a new local storage

mdadm --create /dev/md1 --level=1 --raid-devices=2 /dev/sda3 /dev/sdb3
 xe sr-create content-type=user device-config:device=/dev/md1 name-label="Local Storage" shared=false type=lvm