May 6, 2013
Redhat Enterprise Linux patching: Rollback strategies
Soon i will patch a lot of systems ( RHEL 5 – RHEL 6 ) and i’m planning every rollback strategy.
Studying and looking for the web I found that people usually use RPM rollback but this is NOT RECOMMENDED!!
People on RHEL 5 add RPM macro , on RHEL 6 use yum history but both struments can be used for small rollbacks, not for minor release rollbacks!
I read this from Redhat:
Note: Downgrading a system to minor version (ex: RHEL-6.1 to RHEL-6.0) is not recommended as this might leave the system in broken state where libgcc and other libraries won’t rollback as expected. Use the history option for small update rollbacks.
Same is for RHEL 5.
So what can I think about rollback?
1) LVM snapshot ( usually estimated as ~ 20% lvol source space )
2) Files copy from / and /boot partitions with dd. This is useful with soft/hard links and it’s bit a bit copy. I just need a filesystem from SAN or NFS. If snapshot will not work I just need to boot the system in rescue mode , configure the network , mount the filesystems and use dd for rollback!
April 2, 2013
Oracle 11g installation on ESXi/SLES 11 packages issue
In these days i’m working on a ESXi/SLES 11 Cluster/SAP infrastructure.
When we were installing Oracle , we saw that some package was missing … I expected this.
Looking for these packages I did not found any of them and I was really surprised about that.
I looked on the web and I found that this is a known issue.
When Oracle installation begins , on a physical system it correctly determines the operating system ( in that case SLES 11 ). In ESXi/virtual environment something goes wrong and it treats the system as Oracle Enterprise Linux ( bug ? ).
That’s why i was not able to look for that packages! They are for OEL , not for SLES 
We solved this issue changing a parameter in a configuration file.
In our case we were installing database , so we had:
/oracle/stage/112_64/database/stage/cvu/cv/admin/cvu_config
we changed CV_ASSUME_DISTID=OEL4 in CV_ASSUME_DISTID=SUSE11
March 20, 2013
SLES 11 SP2: Ext4 readonly mount
Yesterday i was doing some test on my SLES 11 for SAP.
I was trying to mount an ext4 filesystem on /mnt.
During the mount i saw the message:
mount: warning: /mnt seems to be mounted read-only
I was surprised to see that message , so i tought about some problem with lvm or iSCSi ( i maked a volume group on a disk via iSCSi ).
Looking the system messages i saw this:
Mar 19 15:59:19 suse1 kernel: [ 1147.951164] EXT4-fs (dm-2): ext4 is supported in read-only mode only
Now it was clear! I understand too why SLESS 11 had the root filesystem on ext3
February 26, 2013
Linux grub Error 1: Cannot mount selected partition
In these days i’m playing a lot with mdadm and grub on Linux. I need to learn how make a root disks mirror starting from /dev/sd* booting devices.
During the tasks i did, I encoutered a problem with grub. I had:
- hd0 (sda)
- hd1 (sdb)
/dev/md1 -> /dev/sdb1 ( /boot )
/dev/md3 -> /dev/sdb3 ( / )
/dev/md2 -> /dev/sdb2 ( swap )
I edited /boot/grub/grub.conf as follow:
# grub.conf generated by anaconda
#
# Note that you do not have to rerun grub after making changes to this file
# NOTICE: You have a /boot partition. This means that
# all kernel and initrd paths are relative to /boot/, eg.
# root (hd0,0)
# kernel /vmlinuz-version ro root=/dev/sda3
# initrd /initrd-[generic-]version.img
#boot=/dev/sda
default=0
fallback=1
timeout=5
splashimage=(hd0,0)/grub/splash.xpm.gz
hiddenmenu
title CentOS MD (2.6.32-279.el6.x86_64)
root (hd1,0)
kernel /vmlinuz-2.6.32-279.el6.x86_64 ro root=/dev/md3 rd_NO_LUKS LANG=en_US.UTF-8 KEYBOARDTYPE=pc KEYTABLE=it rd_NO_MD SYSFONT=latarcyrheb-sun16 crashkernel=auto rd_NO_LVM rd_NO_DM rhgb quiet
initrd /initramfs-2.6.32-279.el6.x86_64.img
title CentOS (2.6.32-279.el6.x86_64)
root (hd0,0)
kernel /vmlinuz-2.6.32-279.el6.x86_64 ro root=UUID=7c1ba3b1-756c-4124-9079-d11a32ceea78 rd_NO_LUKS LANG=en_US.UTF-8 KEYBOARDTYPE=pc KEYTABLE=it rd_NO_MD SYSFONT=latarcyrheb-sun16 crashkernel=auto rd_NO_LVM rd_NO_DM rhgb quiet
initrd /initramfs-2.6.32-279.el6.x86_64.img
After i tried to run:
[root@localhost grub]# grub
Probing devices to guess BIOS drives. This may take a long time.
GNU GRUB version 0.97 (640K lower / 3072K upper memory)
[ Minimal BASH-like line editing is supported. For the first word, TAB
lists possible command completions. Anywhere else TAB lists the possible
completions of a device/filename.]
grub> root (hd0,0)
root (hd0,0)
Filesystem type is ext2fs, partition type 0×83
grub> setup (hd0)
setup (hd0)
Checking if “/boot/grub/stage1″ exists… no
Checking if “/grub/stage1″ exists… yes
Checking if “/grub/stage2″ exists… yes
Checking if “/grub/e2fs_stage1_5″ exists… yes
Running “embed /grub/e2fs_stage1_5 (hd0)”… 27 sectors are embedded.
succeeded
Running “install /grub/stage1 (hd0) (hd0)1+27 p (hd0,0)/grub/stage2 /grub/grub.conf”… succeeded
Done.
grub> root (hd1,0)
root (hd1,0)
Filesystem type unknown, partition type 0xfd
grub> setup (hd1)
setup (hd1)
Error 1: Cannot mount selected partition
grub>
Looking on the web i found that the problem is with md devices metadata.
I maked them with:
mdadm –create /dev/md1 –level=1 –raid-disks=2 –metadata=default missing /dev/sdb1
default metadata is 1.2 , and it’s incompatible with GRUB!
So for boot disks the solution is make them with:
mdadm –create /dev/md1 –level=1 –raid-disks=2 –metadata=0.90 missing /dev/sdb1
February 18, 2013
Linux CentOS 6 e Windows 7 dual boot – BOOTMGR is missing
Today I installed CentOS6 on my notebook for KVM purposes.
After the installation I tried to test Windows boot and i got this error:
BOOTMGR is missing
Press CTRL + ALT + DEL to restart
I immediately tought “damn i’m using my work-notebook this is bad”.
From Linux I saw /etc/grub entries and for Windows I saw:
root (hd0,7)
Watching fdisk -l output i saw the first partition with little size and I tried to map it as follow:
root (hd0,1)
After this I was able to boot Windows
February 8, 2013
Linux CentOS 6: strange behaviour with multiple devices on single mount point !!!
I’m doing a lot of tests , experiments on my CentOS.
Yesterday i was experimenti a LVM mirror recovery and i found two devices mounted on same mount point: /mnt
I tought “damn due to my experiments i erased the system again…”
Later i was curious about to try it again. IT WORKED , how is that possible?
I can do this even now:
[root@centos6 ~]# cat /etc/issue
CentOS release 6.3 (Final)
Kernel \r on an \m
[root@centos6 ~]# lvs
LV VG Attr LSize Pool Origin Data% Move Log Copy% Convert
ale rootvg -wi-a— 1.00g
ale2 rootvg -wi-a— 104.00m
mau rootvg -wi-a— 1.00g
rootvol rootvg -wi-ao– 13.46g
swap2vol rootvg -wi-a— 2.00g
swapvol rootvg -wi-ao– 2.05g
test rootvg -wi-ao– 200.00m
mirrvol testvg mwi-a-m- 2.00g mirrvol_mlog 100.00
[root@centos6 ~]# vgs
VG #PV #LV #SN Attr VSize VFree
rootvg 1 7 0 wz–n- 24.51g 4.70g
testvg 2 1 0 wz–n- 9.98g 5.98g
[root@centos6 ~]# mount /dev/mapper/rootvg-test /mnt
[root@centos6 ~]# mount /dev/mapper/testvg-mirrvol /mnt
[root@centos6 ~]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/rootvg-rootvol
14G 4.4G 8.2G 35% /
tmpfs 498M 188K 498M 1% /dev/shm
/dev/sda1 485M 33M 427M 8% /boot
/dev/mapper/rootvg-test
2.0G 67M 1.9G 4% /mnt
/dev/mapper/testvg-mirrvol
2.0G 67M 1.9G 4% /mnt
I need to understand. NOW !!
January 31, 2013
Redhat 6: how to make a snapshot of root logical volume and rollback it
RHEL6 introduces new features about LVM. One of these is the snapshots.
I wanted to test it for rollback purpose. I will think about it for a patching task 
For this test I maked a screenshot of root logical volume , I deleted /etc directory and restored it from the sreenshot. All worked !
[root@localhost ~]# df -h Filesystem Size Used Avail Use% Mounted on /dev/mapper/rootvg-rootvol 23G 3.1G 18G 15% / tmpfs 498M 0 498M 0% /dev/shm /dev/sda1 485M 33M 427M 8% /boot
[root@localhost ~]# vgs
VG #PV #LV #SN Attr VSize VFree
rootvg 1 2 0 wz–n- 24.51g 0
I saw no free space on rootvg so i added a new disk
Disk /dev/sdb: 652 cylinders, 255 heads, 63 sectors/track
Units = cylinders of 8225280 bytes, blocks of 1024 bytes, counting from 0
Device Boot Start End #cyls #blocks Id System
/dev/sdb1 0+ 651 652- 5237158+ 8e Linux LVM
/dev/sdb2 0 - 0 0 0 Empty
/dev/sdb3 0 - 0 0 0 Empty
/dev/sdb4 0 - 0 0 0 Empty
Writing physical volume data to disk “/dev/sdb1″
Physical volume “/dev/sdb1″ successfully created
Volume group “rootvg” successfully extended
[root@localhost ~]# vgs
VG #PV #LV #SN Attr VSize VFree
rootvg 2 2 0 wz–n- 29.50g 4.99g
Nice. After i maked a screenshot with all free space on the vg.
Logical volume “rootsnap” created
LV VG Attr LSize Pool Origin Data% Move Log Copy% Convert
rootsnap rootvg swi-a-s- 4.99g rootvol 0.00
rootvol rootvg owi-aos- 22.46g
swapvol rootvg -wi-ao– 2.05g
s attribute said to me that rootsnap is a snapshot !
— Logical volume —
LV Path /dev/rootvg/rootvol
LV Name rootvol
VG Name rootvg
LV UUID MgWa8V-yUnK-hqC0-Kv6X-strT-eccc-Kiwt8S
LV Write Access read/write
LV Creation host, time localhost.localdomain, 2013-01-29 19:04:19 +0100
LV snapshot status source of
rootsnap [active]
LV Status available
# open 1
LV Size 22.46 GiB
Current LE 5750
Segments 1
Allocation inherit
Read ahead sectors auto
- currently set to 256
Block device 253:1
First test I did:
1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB) copied, 80.6583 s, 13.0 MB/s
THE SPACE ON THE SNAPSHOT MUST BE CHECKED !!!
LV VG Attr LSize Pool Origin Data% Move Log Copy% Convert
rootsnap rootvg swi-a-s- 4.99g rootvol 18.21
rootvol rootvg owi-aos- 22.46g
swapvol rootvg -wi-ao– 2.05g
[root@localhost ~]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/rootvg-rootvol
23G 4.2G 17G 20% /
tmpfs 498M 0 498M 0% /dev/shm
/dev/sda1 485M 33M 427M 8% /boot
After i erased /etc directory!
I did a reboot and as expected the system was not able to boot up.
So , I used rescue mode from CD and I ran on the shell:
At next boot the system came up without any problem/warning!
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/rootvg-rootvol
23G 3.1G 18G 15% /
tmpfs 498M 0 498M 0% /dev/shm
/dev/sda1 485M 33M 427M 8% /boot
[root@localhost ~]# cd /root
[root@localhost ~]# ls
anaconda-ks.cfg install.log install.log.syslog
( The 1 gb file does not exist anymore )
[root@localhost ~]# lvs
LV VG Attr LSize Pool Origin Data% Move Log Copy% Convert
rootvol rootvg -wi-ao– 22.46g
swapvol rootvg -wi-ao– 2.05g
January 22, 2013
Redhat 6: how to know if a logical volume is striped, linear or mirrored
My company said me that I will work soon with Linux systems , so i will do tasks on Solari/Linux
I left HP-UX studying for Linux Redhat 6.
For this purpose i’m studying a lot of LVM. Today i played with linear and striped logical volumes. After I had a question: “How i can understand if a volume is linear , striped or mirrored?”
I initially looked for lvdisplay man page , without success. Discussing this with my friend Maurizio Pagani he said me the best command:
[root@centos ~]# lvs -a -o segtype,devices,lv_name,vg_name
Type Devices LV VG
linear /dev/sda2(0) rootvol rootvg
linear /dev/sda2(5825) swapvol rootvg
linear /dev/sdc1(128) alevol testvg
striped /dev/sdc1(0),/dev/sdb2(0) testvol testvg
Cool!
December 13, 2012
Sun Cluster 3.2: resources migration between two clusters
Last week i was envolved in this task.
Instead of classic scsnapshot method i used xml configuration files for the migration!
I wrote a new document in my library, you can see it here
I suggest to use xml for every migration
December 3, 2012
Redhat 5.5 and Redhat Cluster Suite task!
The last week i was envolved in a nice task!
I had to install , and configure :
- four DELL servers ( 2x DELL Poweredge R720 and 2x DELL Poweredge R320 ) with Redhat Enterprise 5.5
- two DELL FC Storage ( MD3600f 8GB )
After long time i worked with Linux , it was funny
I installed and used Redhat Cluster Suite too. I encountered many issues but i was funny when i was plying with clustered vgs ( never saw them before )
Following some command i did for clustered vgs
[root@system1 ~]# for i in mpath2p1 mpath9p1 mpath8p1 mpath7p1 mpath6p1;
> do
> pvcreate /dev/mapper/$i
> done
Physical volume “/dev/mapper/mpath2p1″ successfully created
Physical volume “/dev/mapper/mpath9p1″ successfully created
Physical volume “/dev/mapper/mpath8p1″ successfully created
Physical volume “/dev/mapper/mpath7p1″ successfully created
Physical volume “/dev/mapper/mpath6p1″ successfully created
[root@system1 ~]# vgcreate -c y tibcovg /dev/mapper/mpath2p1 /dev/mapper/mpath9p1 /dev/mapper/mpath8p1 /dev/mapper/mpath7p1 /dev/mapper/mpath6p1
Clustered volume group “tibcovg” successfully created
[root@system2]# vgscan
Reading all physical volumes. This may take a while…
Found volume group “tibcovg” using metadata type lvm2
Found volume group “VolGroup00″ using metadata type lvm2
[root@system1 ~]# lvcreate -L 1.2T -n tibcovol tibcovg
Rounding up size to full physical extent 1.20 TB
Logical volume “tibcovol” created
( on second node i ran lvscan )
[root@system1 ~]# vgs
VG #PV #LV #SN Attr VSize VFree
VolGroup00 1 2 0 wz–n- 1.09T 0
tibcovg 5 2 0 wz–nc 1.46T 21.16G
[root@system2 ]# vgs
VG #PV #LV #SN Attr VSize VFree
VolGroup00 1 2 0 wz–n- 1.09T 0
tibcovg 5 2 0 wz–nc 1.46T 21.16G
[root@system1 ~]# mkdir /tibco
[root@system2 mapper]# mkdir /tibco
[root@system1 ~]# mkfs.gfs2 -p lock_dlm -t soa-cluster:tibco -j2 /dev/tibcovg/tibcovol
This will destroy any data on /dev/tibcovg/tibcovol.
It appears to contain a gfs2 filesystem.
Are you sure you want to proceed? [y/n] y
Device: /dev/tibcovg/tibcovol
Blocksize: 4096
Device Size 1228.80 GB (322122752 blocks)
Filesystem Size: 1228.80 GB (322122752 blocks)
Journals: 2
Resource Groups: 4916
Locking Protocol: “lock_dlm”
Lock Table: “soa-cluster:/tibco”
UUID: 5D884F81-39A4-FF56-326B-AC4735620175
On both nodes i did:
[root@system1 ~]# mount /dev/tibcovg/tibcovol /tibco
After i encountered hard problems with device fencing… due to it , during a reboot of a first node , the second node had problems on /tibco fs.
I called some friend for help anyway i learned a lot of things about Redhat Cluster , this was the first time i configured it !
