Sunday, May 24, 2009

How To Recover Veritas Volume Manager RAID0 Volume

How To Recover Veritas Volume Manager RAID0 Volume


Veritas Volume Manager(VxVM) RAID0 volume has either stripe or
concatenation layout. Suppose a subdisk has being removed from RAID0
volume e.g vxdiskadm #4, the data in the volume will be corrupted. An
attempt to recover the volume data by putting back the original subdisk
into RAID0 volume e.g vxdiskadm #5 will give error message " appears to be
owned by disk group ".

The following procedure show how to recover the Raid0 volume in this
situation with the following assumptions:
(1) no physical disk or Lun has being replaced. All the disks are in good
& original position
(2) no destructive command has being run which affect the integrity of
VxVM configuration e.g initialize the removed disk.

If you are unsure on above assumption, stop proceeding & seek help from Sun.


Steps to Follow

These are the following steps to take:


1) record down relevent disk vtoc table
privlen=tag 15 slice sector count
privoffset=tag 15 slice first sector
publen=tag 14 slice sector count
puboffset=tag14 slice first sector

2) uninitialize disk
# vxdiskunsetup -C

3) re-initialize disk back to original private & public region with input
from step(1)
# vxdisksetup -i privlen=<> privoffset=<> publen=<> puboffset=<>

4) prtvtoc to confirm the newly rebuilded vtoc table
# prtvtoc /dev/dsk/cXtXdXs2
# vxdisk list
** vxdisk output privlen may be smaller that vtoc due to roundup, but
publen MUST match on both outputs

5) put disk back to dg
# vxdg -g -k adddisk =

6) recover plex/volume
# vxmend -g -o force off
** plex under OFFLINE state & volume DISABLED/ACTIVE
# vxmend -g on
** plex under STALE state & volume DISABLED/ACTIVE
# vxmend -g fix active
** plex under DISABLED/ACTIVE & volume DISABLED/NEEDSYNC
# vxvol -g start
** plex & volume are under ENABLE/ACTIVE

7) fsck & mount the volume

VxVM : CLI version of vxdiskadm option 4 & 5

Option 4
# vxdg -g datadg -k rmdisk datadg01

Option 5
# ./vxdisksetup -if TagmaStore-USP0_0
# vxdg -g datadg -k adddisk datadg01=TagmaStore-USP0_0
# vxrecover -g datadg -b datadg01

Real Life example
------------------
# vxmend -g datadg -o force off data01vx-01 (do this if volume is DISABLED RECOVER.
If DISABLED STALE , proceed with vxmend -g datadg on data01vx-01)
# vxmend -g datadg on data01vx-01
# vxvol -g datadg start data01vx

# vxdisk list

DEVICE TYPE DISK GROUP STATUS

TagmaStore-USP0_0 auto:cdsdisk - - online
c1t0d0s2 auto:sliced odg01 odg online
c1t1d0s2 auto:cdsdisk datadg02_scsi datadg online
c1t2d0s2 auto:sliced odg02 odg online
c1t3d0s2 auto:cdsdisk datadg01_scsi datadg online
emcpower1s2 auto:cdsdisk datadg02 datadg online nohotuse
emcpower2s2 auto:cdsdisk datadg10 datadg online
emcpower3s2 auto:cdsdisk datadg06 datadg online nohotuse
emcpower4s2 auto:cdsdisk datadg07 datadg online nohotuse
emcpower5s2 auto:cdsdisk datadg05 datadg online nohotuse
emcpower6s2 auto:cdsdisk datadg11 datadg online
emcpower7s2 auto:cdsdisk datadg04 datadg online nohotuse
emcpower8s2 auto:cdsdisk datadg08 datadg online nohotuse
emcpower9s2 auto:cdsdisk datadg03 datadg online nohotuse
emcpower10s2 auto:cdsdisk datadg09 datadg online nohotuse
- - datadg01 datadg failed was:TagmaStore-USP0_0

# vxdg -g datadg -k rmdisk datadg01
# vxdisk list

DEVICE TYPE DISK GROUP STATUS

TagmaStore-USP0_0 auto:cdsdisk - - online
c1t0d0s2 auto:sliced odg01 odg online
c1t1d0s2 auto:cdsdisk datadg02_scsi datadg online
c1t2d0s2 auto:sliced odg02 odg online
c1t3d0s2 auto:cdsdisk datadg01_scsi datadg online
emcpower1s2 auto:cdsdisk datadg02 datadg online nohotuse
emcpower2s2 auto:cdsdisk datadg10 datadg online
emcpower3s2 auto:cdsdisk datadg06 datadg online nohotuse
emcpower4s2 auto:cdsdisk datadg07 datadg online nohotuse
emcpower5s2 auto:cdsdisk datadg05 datadg online nohotuse
emcpower6s2 auto:cdsdisk datadg11 datadg online
emcpower7s2 auto:cdsdisk datadg04 datadg online nohotuse
emcpower8s2 auto:cdsdisk datadg08 datadg online nohotuse
emcpower9s2 auto:cdsdisk datadg03 datadg online nohotuse
emcpower10s2 auto:cdsdisk datadg09 datadg online nohotuse
- - datadg01 datadg removed was:TagmaStore-USP0_0

root@S21TDI234 # ./vxdisksetup -if TagmaStore-USP0_0
root@S21TDI234 # vxdisk list

DEVICE TYPE DISK GROUP STATUS

TagmaStore-USP0_0 auto:cdsdisk - - online
c1t0d0s2 auto:sliced odg01 odg online
c1t1d0s2 auto:cdsdisk datadg02_scsi datadg online
c1t2d0s2 auto:sliced odg02 odg online
c1t3d0s2 auto:cdsdisk datadg01_scsi datadg online
emcpower1s2 auto:cdsdisk datadg02 datadg online nohotuse
emcpower2s2 auto:cdsdisk datadg10 datadg online
emcpower3s2 auto:cdsdisk datadg06 datadg online nohotuse
emcpower4s2 auto:cdsdisk datadg07 datadg online nohotuse
emcpower5s2 auto:cdsdisk datadg05 datadg online nohotuse
emcpower6s2 auto:cdsdisk datadg11 datadg online
emcpower7s2 auto:cdsdisk datadg04 datadg online nohotuse
emcpower8s2 auto:cdsdisk datadg08 datadg online nohotuse
emcpower9s2 auto:cdsdisk datadg03 datadg online nohotuse
emcpower10s2 auto:cdsdisk datadg09 datadg online nohotuse
- - datadg01 datadg removed was:TagmaStore-USP0_0

# vxdg -g datadg -k adddisk datadg01=TagmaStore-USP0_0
# vxdisk list

DEVICE TYPE DISK GROUP STATUS

TagmaStore-USP0_0 auto:cdsdisk datadg01 datadg online
c1t0d0s2 auto:sliced odg01 odg online
c1t1d0s2 auto:cdsdisk datadg02_scsi datadg online
c1t2d0s2 auto:sliced odg02 odg online
c1t3d0s2 auto:cdsdisk datadg01_scsi datadg online
emcpower1s2 auto:cdsdisk datadg02 datadg online nohotuse
emcpower2s2 auto:cdsdisk datadg10 datadg online
emcpower3s2 auto:cdsdisk datadg06 datadg online nohotuse
emcpower4s2 auto:cdsdisk datadg07 datadg online nohotuse
emcpower5s2 auto:cdsdisk datadg05 datadg online nohotuse
emcpower6s2 auto:cdsdisk datadg11 datadg online
emcpower7s2 auto:cdsdisk datadg04 datadg online nohotuse
emcpower8s2 auto:cdsdisk datadg08 datadg online nohotuse
emcpower9s2 auto:cdsdisk datadg03 datadg online nohotuse
emcpower10s2 auto:cdsdisk datadg09 datadg online nohotuse

# vxrecover -g datadg -b datadg01
# vxprint -g datadg -ht

v data01vx - DISABLED ACTIVE 1909251376 SELECT -
fsgen
pl data01vx-01 data01vx DISABLED RECOVER 1909251376 CONCAT - RW
sd datadg01-01 data01vx-01 datadg01 0 240596992 0 TagmaStore-USP0_0 ENA
sd datadg01_scsi-05 data01vx-01 datadg01_scsi 201326592 85306240 240596992 c1t3d0 ENA
sd datadg02_scsi-04 data01vx-01 datadg02_scsi 201326592 50048 325903232 c1t1d0 ENA
sd datadg02-01 data01vx-01 datadg02 0 159332608 325953280 emcpower1 ENA
sd datadg02_scsi-05 data01vx-01 datadg02_scsi 201376640 85256192 485285888 c1t1d0 ENA
sd datadg03-01 data01vx-01 datadg03 0 159332608 570542080 emcpower9 ENA
sd datadg04-01 data01vx-01 datadg04 0 159332608 729874688 emcpower7 ENA
sd datadg05-01 data01vx-01 datadg05 0 159332608 889207296 emcpower5 ENA
sd datadg06-01 data01vx-01 datadg06 0 159332608 1048539904 emcpower3 ENA
sd datadg07-01 data01vx-01 datadg07 0 159332608 1207872512 emcpower4 ENA
sd datadg08-01 data01vx-01 datadg08 0 159332608 1367205120 emcpower8 ENA
sd datadg09-01 data01vx-01 datadg09 0 159332608 1526537728 emcpower10 ENA
sd datadg10-02 data01vx-01 datadg10 4194304 155138304 1685870336 emcpower2 ENA
sd datadg11-02 data01vx-01 datadg11 4194304 68242736 1841008640 emcpower6 ENA

# vxmend -g datadg -o force off data01vx-01 (do this if volume is DISABLED RECOVER. If DISABLED STALE , proceed with vxmend -g datadg on data01vx-01)
# vxmend -g datadg on data01vx-01
# vxvol -g datadg start data01vx

Solution 215981 : Cold Backup of Oracle using Volume Manager Snapshot

Here is the procedure to create a snapshot of a volume containing an Oracle
database. This assumes the database is built on a file system :

1) vxassist -g snapstart

2) When the snapstart operation is complete and the mirror is in the SNAPDONE
state for the newly created plex (this can be determined through vxprint -ht
), find a time to complete the snapshot operation. This is the
point where you would bring down the oracle db and kick the users out. When
Oracle is down, execute this command:

# vxassist -g snapshot

3) When this operation is finished, you can bring Oracle online and then perform
an fsck of the snapshot:

# fsck -y /dev/vx/rdsk//

4) Next, backup the snapshot volume using your data backup utility and then the
snapshot can be removed using:

# vxedit -g -rf rm

Look at the following example:

bigwillie# vxprint -g bigdg -ht oracle
V NAME USETYPE KSTATE STATE LENGTH READPOL PREFPLEX
PL NAME VOLUME KSTATE STATE LENGTH LAYOUT NCOL/WID MODE
SD NAME PLEX DISK DISKOFFS LENGTH [COL/]OFF DEVICE MODE

v oracle fsgen ENABLED ACTIVE 102400 SELECT -
pl oracle-01 oracle ENABLED ACTIVE 104139 CONCAT - RW
sd bigdg13-01 oracle-01 bigdg13 0 104139 0 c2t26d0 ENA
pl oracle-02 oracle ENABLED ACTIVE 104139 CONCAT - RW
sd bigdg11-02 oracle-02 bigdg11 64638 104139 0 c2t24d0 ENA
bigwillie# vxassist -g bigdg snapstart oracle
bigwillie# df -k /oraclemnt
Filesystem kbytes used avail capacity Mounted on
/dev/vx/dsk/bigdg/oracle
51200 1117 46960 3% /oraclemnt
bigwillie# umount /oraclemnt
bigwillie# vxprint -g bigdg -ht oracle
V NAME USETYPE KSTATE STATE LENGTH READPOL PREFPLEX
PL NAME VOLUME KSTATE STATE LENGTH LAYOUT NCOL/WID MODE
SD NAME PLEX DISK DISKOFFS LENGTH [COL/]OFF DEVICE MODE

v oracle fsgen ENABLED ACTIVE 102400 SELECT -
pl oracle-01 oracle ENABLED ACTIVE 104139 CONCAT - RW
sd bigdg13-01 oracle-01 bigdg13 0 104139 0 c2t26d0 ENA
pl oracle-02 oracle ENABLED ACTIVE 104139 CONCAT - RW
sd bigdg11-02 oracle-02 bigdg11 64638 104139 0 c2t24d0 ENA
pl oracle-03 oracle ENABLED SNAPDONE 104139 CONCAT - WO
sd bigdg05-02 oracle-03 bigdg05 208278 104139 0 c2t19d0 ENA
bigwillie# vxassist -g bigdg snapshot oracle oraclesnap
bigwillie# vxprint -g bigdg -ht oracle oraclesnap
V NAME USETYPE KSTATE STATE LENGTH READPOL PREFPLEX
PL NAME VOLUME KSTATE STATE LENGTH LAYOUT NCOL/WID MODE
SD NAME PLEX DISK DISKOFFS LENGTH [COL/]OFF DEVICE MODE

v oracle fsgen ENABLED ACTIVE 102400 SELECT -
pl oracle-01 oracle ENABLED ACTIVE 104139 CONCAT - RW
sd bigdg13-01 oracle-01 bigdg13 0 104139 0 c2t26d0 ENA
pl oracle-02 oracle ENABLED ACTIVE 104139 CONCAT - RW
sd bigdg11-02 oracle-02 bigdg11 64638 104139 0 c2t24d0 ENA

v oraclesnap fsgen ENABLED ACTIVE 102400 ROUND -
pl oraclesnap-01 oraclesnap ENABLED ACTIVE 104139 CONCAT - RW
sd bigdg05-02 oraclesnap-01 bigdg05 208278 104139 0 c2t19d0 ENA
bigwillie# mount /dev/vx/dsk/bigdg/oracle /oraclemnt
bigwillie# fsck -y /dev/vx/rdsk/bigdg/oraclesnap
file system is clean - log replay is not required
bigwillie#

-Backup up the /dev/vx/rdsk/bigdg/oracle device-

bigwillie# vxedit -g bigdg -rf rm oraclesnap

Solution 217667 : VxVM - Unable to mirror a striped volume using 'vxassist' command

When attempting to grow a volume with a striped plex to a single disk with ample free space, vxassist complains that there is not enough space to perform the operation.

Here are the details of a generic case.

Start with a volume containing at least one striped plex:

  # vxprint -ht
V NAME RVG KSTATE STATE LENGTH READPOL PREFPLEX UTYPE
PL NAME VOLUME KSTATE STATE LENGTH LAYOUT NCOL/WID MODE
SD NAME PLEX DISK DISKOFFS LENGTH [COL/]OFF DEVICE MODE
  v  vol01     -        ENABLED  ACTIVE   105906176 SELECT    -        fsgen
pl vol01-01 vol01 ENABLED ACTIVE 105907104 STRIPE 3/128 RW
sd disk01-01 vol01-01 disk01 0 35302304 0/0 c4t3d0 ENA
sd disk02-01 vol01-01 disk02 0 35302304 1/0 c4t7d0 ENA
sd disk03-01 vol01-01 disk03 0 35302304 2/0 c4t23d0 ENA

and another disk with free space on which to mirror it:

  # vxdg -g ckdg free
DISK DEVICE TAG OFFSET LENGTH FLAGS
disk09 c5t34d0s2 c5t34d0 0 143328960 -

However, when an attempt is made to mirror this volume using this disk, it fails with this error:

  # vxassist -g ckdg mirror vol01 alloc=disk09
vxvm:vxassist: ERROR: Cannot allocate space to mirror 105906176 block volume

Resolution
The reason is that when mirroring a striped plex, the 'vxassist' command wants to create *another* striped plex as the mirror, by default. So, the command is failing not because there was not enough space to mirror it, but because there was not enough disks specified to create another striped plex.


To actually get vxassist to do what you want, that is, to mirror the striped plex to a new concat plex, you must use the 'layout=nostripe' option as well. Therefore, the command

  # vxassist -g ckdg mirror vol01 layout=nostripe alloc=disk09

will actually complete.

VxVM : How to split a mirror in Veritas Volume manager

This document describes how to split a Veritas mirror so that the sub-mirrors can be associated with new volumes, and hence be accessed via Solaris[TM] devices and possibly mounted.

Be aware that you may not be able to mount a filesystem taken from a split mirror, depending on the state of the filesystem at the time of the split. If a filesystem is unmounted cleanly prior to the split, both copies should be mountable afterwards.

So this is our starting point, a single volume with two plexes forming a simple mirror in the disk group 'baddg'.

# vxprint -g baddg
TY NAME ASSOC KSTATE LENGTH PLOFFS STATE TUTIL0 PUTIL0
dg baddg baddg - - - - - -
dm sd01         Disk_1       -        35358848 -        -        -       -
v  vol1         gen          ENABLED  207328   -        ACTIVE   -       -
pl plex1 vol1 ENABLED 207328 - ACTIVE - -
sd sd01-B0 plex1 ENABLED 1 0 - - Block0
sd sd01-02 plex1 ENABLED 207327 1 - - -
pl plex2 vol1 ENABLED 207328 - ACTIVE - -
sd sd01-01 plex2 ENABLED 207328 0 - - -

It has a filesystem which is mounted.

# mount |grep baddg
/a on /dev/vx/dsk/baddg/vol1
read/write/setuid/intr/largefiles/logging/xattr/onerror=panic/dev=3b41f40
on Thu Jul 21 16:05:26 2005

We write a file and calculate the checksum. (Note: This is not a necessary part of the procedure but will be used later in this document to illustrate that the procedure has been a success.)

# cp /kernel/genunix /a
# ls -l /a
total 3600
-rwxr-xr-x 1 root other 1823432 Jul 21 16:08 genunix
drwx------ 2 root root 8192 Jul 21 16:02 lost+found
# cksum /a/genunix
411313929 1823432 /a/genunix

Now unmount the filesystem, so that we know everything has been flushed to the physical disk.

# umount /a

In the following steps we are going to disassociate the plexes from the parent volume using vxplex 'dis'. VxVM also has a vxplex 'det' which is to detatch a plex. Confusingly, detatching a plex with vxplex 'det', stops writes to the plex, but the plex stays associated with the parent volume.

The first step is to detach one of the plexes and break its link to the remaining plex(es) (i.e. dissociate). We'll dissociate 'plex2'

# vxplex -g baddg dis plex2

Let's check how VxVM sees things now - plex2 is no longer associated with the volume.

# vxprint -g baddg
TY NAME ASSOC KSTATE LENGTH PLOFFS STATE TUTIL0 PUTIL0
dg baddg baddg - - - - - -
dm sd01         Disk_1       -        35358848 -        -        -       -
pl plex2        -            DISABLED 207328   -        -        -       -
sd sd01-01 plex2 ENABLED 207328 0 - - -
v  vol1         gen          ENABLED  207328   -        ACTIVE   -       -
pl plex1 vol1 ENABLED 207328 - ACTIVE - -
sd sd01-B0 plex1 ENABLED 1 0 - - Block0
sd sd01-02 plex1 ENABLED 207327 1 - - -

Now create a volume using the disassociated plex (plex2). The volume will be called 'vol2'

# vxmake -g baddg vol vol2 plex=plex2
vxprint shows the new volume, but the volume is disabled.
v vol2 gen DISABLED 207328 - EMPTY - -
pl plex2 vol2 DISABLED 207328 - EMPTY - -
sd sd01-01 plex2 ENABLED 207328 0 - - -

Next, we must start the new volume

# vxvol -g baddg start vol2

The new volume is now available to mount.

v  vol2         gen          ENABLED  207328   -        ACTIVE   -       -
pl plex2 vol2 ENABLED 207328 - ACTIVE - -
sd sd01-01 plex2 ENABLED 207328 0 - - -

We can now mount the two volumes that were previously two halves of a single mirror and see that the same file is there, and has the same checksum - i.e. is a consistent copy.

First we mount the new volume

# mount /dev/vx/dsk/baddg/vol2 /b
# ls -l /b
total 3600
-rwxr-xr-x 1 root other 1823432 Jul 21 16:08 genunix
drwx------ 2 root root 8192 Jul 21 16:02 lost+found
# cksum /b/genunix
411313929 1823432 /b/genunix

Now the original volume (no longer a mirror)

# mount /dev/vx/dsk/baddg/vol1 /a
# ls -l /a
total 3600
-rwxr-xr-x 1 root other 1823432 Jul 21 16:08 genunix
drwx------ 2 root root 8192 Jul 21 16:02 lost+found
# cksum /a/genunix
411313929 1823432 /a/genunix

To recreate the original mirror, follow the above steps to disassociate the plex from the new volume. Then use vxplex 'att' to attach the plex to the original volume. The plex will be synchronised with the plex(es) already attached to the volume.