Veritas Cluster and leap second bug

On 30th June we’ll see a new leap second bug, because the day will have one second more.
The customer asked us to open a SYMANTEC service request because we have more than 2.000+ veritas cluster suites active and running on Solaris,Redhat and HP-UX hosts.
The support told us that Symantec products are unaffected, nice!!!

Veritas Cluster: UX:vxfs mkfs: ERROR: V-3-20003

Today I was installing and configuring VCS CFS-HA 6.0.5 on two nodes running HP-UX 11.31.
I made a couple of shared diskgroups, I made some volumes and I tried to run mkfs obtaining the following error:

/opt/VRTS/bin/mkfs -F vxfs /dev/vx/rdsk/testdg/stage
UX:vxfs mkfs: ERROR: V-3-20003: Cannot open /dev/vx/rdsk/testdg/stage: No such device or address
UX:vxfs mkfs: ERROR: V-3-20275: cannot open /dev/vx/rdsk/testdg/stage

This is error is due to missed activation of the volume access!
What i’m talkin about?
Let’s read this link, the option in my case was

shared-write (sw) – The node has write access to the disk group.
So the correct procedure is:

– import the shared diskgroup
– start all volumes on the diskgroup
– set the activation with the command: vxdg -g testdg set activation=sw
– run mkfs

all worked great 😀

Solaris Error: 126: Cannot assign requested address

Today I’m configuring under Veritas Cluster Server a couple of oracle instances ( database + listener )
When I was trying to switch the service groups from the first node to the second I got this error from the listener resources:

(system02) Netlsnr:lst_fmm:online:lsnrctl returned the following output
LD_LIBRARY_PATH – /usr/lib:

LSNRCTL for Solaris: Version – Production on 09-APR-2015 14:37:23

Copyright (c) 1991, 2013, Oracle. All rights reserved.

Starting /xxx/oraengine/app/oracle/ please wait…

TNSLSNR for Solaris: Version – Production
System parameter file is /xxx/oraengine/app/oracle/
Log messages written to /xxx/oraengine/app/oracle/diag/tnslsnr/systems02/fmmlist/alert/log.xml
Error listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=x.x.x.x)(PORT=1525)))
TNS-12545: Connect failed because target host or object does not exist
TNS-12560: TNS:protocol adapter error
TNS-00515: Connect failed because target host or object does not exist
Solaris Error: 126: Cannot assign requested address

Listener failed to start. See the error message(s) above…

I was a bit curious because on the first node I did not have any problem about the listener.
With a collegue we saw the file listener.ora on the second node had wrong ip mapped.

Veritas Cluster Server: how to get a snapshot of a running cluster

Since the new year one of my new duties is to work as Symantec consultant, so I’m working a lot with VxVM/VCS ( 4.*, 5.*, 6.* ) under RHEL,SUSE,HP-UX,Solaris.
Today I learned how to get a snapshot of a running cluster. This procedure is very useful because I can get the complete list of vcs commands for clone,copy or create a new cluster on a new systems ( adapting hostnames,network interfaces, etc.etc. ). I immediatelly tought about scsnapshot on Sun Cluster/Oracle Cluster but with VCS there are a lot of details. This new cluster was composited by just two nodes without resources yet but I obtained more than 3k lines of commands!!
How to get the snapshot? It’s easy 🙂

Go to /etc/VRTSvcs/conf/config and run: hacf -verify .
You will have a main.cmd file, containing the commands list. I was curious to see the option description from the hacf man page:

-verify conf_dir [-display]
The -verify option checks for syntax errors in the
configuration file (which usually includes before starting the VCS engine. It returns
after the first syntax error is detected. VCS does not
execute if there are errors in the configuration. The
hacf command looks for the file in the
conf_dir. The conf_dir is the directory in which the

SunOS 5.11 Last change: 2013-11-07 1

VCS 6.1 HACF(1M) file resides. No error message and a return
value of zero means the syntax is legal. If the
-display option is specified, then the output is also
displayed on the stdout.

This is a part of main.cmd file generated today.

Got Certified!!!

Today I passed the Symantec exam ST0-248 with 85% score, so I got certified as “Symantec Certified Specialist in Storage Foundation 6.1 for UNIX”

Wow it looks serious!! ahaha 🙂

Veritas Cluster Server 5 : How to see reservation keys on a disk

I came back to work in Telecom for a NETAPP storage virtualization project.
I’m working on a mixed Solaris systems ( ZFS , SVM , VxVM , CVM , CFS , MPXIO , DMP ) with Veritas Cluster or Sun Cluster.
On a VCS nodes I usually have reservation conflict problem and I must check the keys on the disks.
For this purpose i can use the vxfenadm command:

# vxfenadm -g /dev/vx/rdmp/hp_xp24k0_01c6
VXFEN vxfenadm WARNING V-11-2-2414 This option is deprecated and would be removed with the next release.
Please use the -s option.

Reading SCSI Registration Keys…

Device Name: /dev/vx/rdmp/hp_xp24k0_01c6
Total Number Of Keys: 2
Key Value [Numeric Format]: 66,86,67,83,0,0,0,0
Key Value [Character Format]: BVCS
Key Value [Numeric Format]: 66,86,67,83,0,0,0,0
Key Value [Character Format]: BVCS

Well, -g is deprecated so I ran vxfenadm again with -s option:

# vxfenadm -s /dev/vx/rdmp/hp_xp24k0_01c6

Reading SCSI Registration Keys…

Device Name: /dev/vx/rdmp/hp_xp24k0_01c6
Total Number Of Keys: 2
[Numeric Format]: 66,86,67,83,0,0,0,0
[Character Format]: BVCS
[Node Format]: Cluster ID: unknown Node ID: 1 Node Name: ssb00005
[Numeric Format]: 66,86,67,83,0,0,0,0
[Character Format]: BVCS
[Node Format]: Cluster ID: unknown Node ID: 1 Node Name: ssb00005

Cool! 😉

Veritas Volume Manager: how to add multiple disks to diskgroup in 1 line

Today i’m working with VxVM , doing basiscs activities. One of them is to add multiple disks to a diskgroup called system1-dg.

For this purpose i was funny making 1line command.

These are disks before initialization ( just visible due to vxdctl enable command ):

Disk_177 auto – – error
Disk_178 auto – – error
Disk_179 auto – – error
Disk_180 auto – – error
Disk_181 auto – – error
Disk_182 auto – – error
Disk_183 auto – – error
Disk_184 auto – – error
Disk_185 auto – – error
Disk_186 auto – – error
Disk_187 auto – – error
Disk_188 auto – – error
Disk_189 auto – – error
Disk_190 auto – – error
Disk_191 auto – – error
Disk_192 auto – – error
Disk_193 auto – – error
Disk_194 auto – – error
Disk_195 auto – – error

I ran this 1line and i obtained:

Disk_177 auto Disk_177 system-dg online
Disk_178 auto Disk_178 system-dg online
Disk_179 auto Disk_179 system-dg online
Disk_180 auto Disk_180 system-dg online
Disk_181 auto Disk_181 system-dg online
Disk_182 auto Disk_182 system-dg online
Disk_183 auto Disk_183 system-dg online
Disk_184 auto Disk_184 system-dg online
Disk_185 auto Disk_185 system-dg online
Disk_186 auto Disk_186 system-dg online
Disk_187 auto Disk_187 system-dg online
Disk_188 auto Disk_188 system-dg online
Disk_189 auto Disk_189 system-dg online
Disk_190 auto Disk_190 system-dg online
Disk_191 auto Disk_191 system-dg online
Disk_192 auto Disk_192 system-dg online
Disk_193 auto Disk_193 system-dg online
Disk_194 auto Disk_194 system-dg online
Disk_195 auto Disk_195 system-dg online

Cool and quick 🙂