VXVM : Encapsulate Boot-disk CLI

# vxconfigd -r reset -m disable
# vxdctl init
# vxdctl enable
# rm /etc/vx/reconfig.d/state.d/install-db
# vxencap -g -c rootdisk=
# reboot

How To Recover Veritas Volume Manager RAID0 Volume

How To Recover Veritas Volume Manager RAID0 Volume

Veritas Volume Manager(VxVM) RAID0 volume has either stripe or
concatenation layout. Suppose a subdisk has being removed from RAID0
volume e.g vxdiskadm #4, the data in the volume will be corrupted. An
attempt to recover the volume data by putting back the original subdisk
into RAID0 volume e.g vxdiskadm #5 will give error message " appears to be
owned by disk group ".

The following procedure show how to recover the Raid0 volume in this
situation with the following assumptions:
(1) no physical disk or Lun has being replaced. All the disks are in good
& original position
(2) no destructive command has being run which affect the integrity of
VxVM configuration e.g initialize the removed disk.

If you are unsure on above assumption, stop proceeding & seek help from Sun.

Steps to Follow

These are the following steps to take:

1) record down relevent disk vtoc table
privlen=tag 15 slice sector count
privoffset=tag 15 slice first sector
publen=tag 14 slice sector count
puboffset=tag14 slice first sector

2) uninitialize disk
# vxdiskunsetup -C

3) re-initialize disk back to original private & public region with input
from step(1)
# vxdisksetup -i privlen=<> privoffset=<> publen=<> puboffset=<>

4) prtvtoc to confirm the newly rebuilded vtoc table
# prtvtoc /dev/dsk/cXtXdXs2
# vxdisk list
** vxdisk output privlen may be smaller that vtoc due to roundup, but
publen MUST match on both outputs

5) put disk back to dg
# vxdg -g -k adddisk =

6) recover plex/volume
# vxmend -g -o force off
** plex under OFFLINE state & volume DISABLED/ACTIVE
# vxmend -g on
** plex under STALE state & volume DISABLED/ACTIVE
# vxmend -g fix active
** plex under DISABLED/ACTIVE & volume DISABLED/NEEDSYNC
# vxvol -g start
** plex & volume are under ENABLE/ACTIVE

7) fsck & mount the volume

VxVM : CLI version of vxdiskadm option 4 & 5

Option 4
# vxdg -g datadg -k rmdisk datadg01

Option 5
# ./vxdisksetup -if TagmaStore-USP0_0
# vxdg -g datadg -k adddisk datadg01=TagmaStore-USP0_0
# vxrecover -g datadg -b datadg01

Real Life example
------------------
# vxmend -g datadg -o force off data01vx-01 (do this if volume is DISABLED RECOVER.
If DISABLED STALE , proceed with vxmend -g datadg on data01vx-01)
# vxmend -g datadg on data01vx-01
# vxvol -g datadg start data01vx

# vxdisk list

DEVICE TYPE DISK GROUP STATUS

TagmaStore-USP0_0 auto:cdsdisk - - online
c1t0d0s2 auto:sliced odg01 odg online
c1t1d0s2 auto:cdsdisk datadg02_scsi datadg online
c1t2d0s2 auto:sliced odg02 odg online
c1t3d0s2 auto:cdsdisk datadg01_scsi datadg online
emcpower1s2 auto:cdsdisk datadg02 datadg online nohotuse
emcpower2s2 auto:cdsdisk datadg10 datadg online
emcpower3s2 auto:cdsdisk datadg06 datadg online nohotuse
emcpower4s2 auto:cdsdisk datadg07 datadg online nohotuse
emcpower5s2 auto:cdsdisk datadg05 datadg online nohotuse
emcpower6s2 auto:cdsdisk datadg11 datadg online
emcpower7s2 auto:cdsdisk datadg04 datadg online nohotuse
emcpower8s2 auto:cdsdisk datadg08 datadg online nohotuse
emcpower9s2 auto:cdsdisk datadg03 datadg online nohotuse
emcpower10s2 auto:cdsdisk datadg09 datadg online nohotuse
- - datadg01 datadg failed was:TagmaStore-USP0_0

# vxdg -g datadg -k rmdisk datadg01
# vxdisk list

DEVICE TYPE DISK GROUP STATUS

TagmaStore-USP0_0 auto:cdsdisk - - online
c1t0d0s2 auto:sliced odg01 odg online
c1t1d0s2 auto:cdsdisk datadg02_scsi datadg online
c1t2d0s2 auto:sliced odg02 odg online
c1t3d0s2 auto:cdsdisk datadg01_scsi datadg online
emcpower1s2 auto:cdsdisk datadg02 datadg online nohotuse
emcpower2s2 auto:cdsdisk datadg10 datadg online
emcpower3s2 auto:cdsdisk datadg06 datadg online nohotuse
emcpower4s2 auto:cdsdisk datadg07 datadg online nohotuse
emcpower5s2 auto:cdsdisk datadg05 datadg online nohotuse
emcpower6s2 auto:cdsdisk datadg11 datadg online
emcpower7s2 auto:cdsdisk datadg04 datadg online nohotuse
emcpower8s2 auto:cdsdisk datadg08 datadg online nohotuse
emcpower9s2 auto:cdsdisk datadg03 datadg online nohotuse
emcpower10s2 auto:cdsdisk datadg09 datadg online nohotuse
- - datadg01 datadg removed was:TagmaStore-USP0_0

root@S21TDI234 # ./vxdisksetup -if TagmaStore-USP0_0
root@S21TDI234 # vxdisk list

DEVICE TYPE DISK GROUP STATUS

TagmaStore-USP0_0 auto:cdsdisk - - online
c1t0d0s2 auto:sliced odg01 odg online
c1t1d0s2 auto:cdsdisk datadg02_scsi datadg online
c1t2d0s2 auto:sliced odg02 odg online
c1t3d0s2 auto:cdsdisk datadg01_scsi datadg online
emcpower1s2 auto:cdsdisk datadg02 datadg online nohotuse
emcpower2s2 auto:cdsdisk datadg10 datadg online
emcpower3s2 auto:cdsdisk datadg06 datadg online nohotuse
emcpower4s2 auto:cdsdisk datadg07 datadg online nohotuse
emcpower5s2 auto:cdsdisk datadg05 datadg online nohotuse
emcpower6s2 auto:cdsdisk datadg11 datadg online
emcpower7s2 auto:cdsdisk datadg04 datadg online nohotuse
emcpower8s2 auto:cdsdisk datadg08 datadg online nohotuse
emcpower9s2 auto:cdsdisk datadg03 datadg online nohotuse
emcpower10s2 auto:cdsdisk datadg09 datadg online nohotuse
- - datadg01 datadg removed was:TagmaStore-USP0_0

# vxdg -g datadg -k adddisk datadg01=TagmaStore-USP0_0
# vxdisk list

DEVICE TYPE DISK GROUP STATUS

TagmaStore-USP0_0 auto:cdsdisk datadg01 datadg online
c1t0d0s2 auto:sliced odg01 odg online
c1t1d0s2 auto:cdsdisk datadg02_scsi datadg online
c1t2d0s2 auto:sliced odg02 odg online
c1t3d0s2 auto:cdsdisk datadg01_scsi datadg online
emcpower1s2 auto:cdsdisk datadg02 datadg online nohotuse
emcpower2s2 auto:cdsdisk datadg10 datadg online
emcpower3s2 auto:cdsdisk datadg06 datadg online nohotuse
emcpower4s2 auto:cdsdisk datadg07 datadg online nohotuse
emcpower5s2 auto:cdsdisk datadg05 datadg online nohotuse
emcpower6s2 auto:cdsdisk datadg11 datadg online
emcpower7s2 auto:cdsdisk datadg04 datadg online nohotuse
emcpower8s2 auto:cdsdisk datadg08 datadg online nohotuse
emcpower9s2 auto:cdsdisk datadg03 datadg online nohotuse
emcpower10s2 auto:cdsdisk datadg09 datadg online nohotuse

# vxrecover -g datadg -b datadg01
# vxprint -g datadg -ht

v data01vx - DISABLED ACTIVE 1909251376 SELECT -
fsgen
pl data01vx-01 data01vx DISABLED RECOVER 1909251376 CONCAT - RW
sd datadg01-01 data01vx-01 datadg01 0 240596992 0 TagmaStore-USP0_0 ENA
sd datadg01_scsi-05 data01vx-01 datadg01_scsi 201326592 85306240 240596992 c1t3d0 ENA
sd datadg02_scsi-04 data01vx-01 datadg02_scsi 201326592 50048 325903232 c1t1d0 ENA
sd datadg02-01 data01vx-01 datadg02 0 159332608 325953280 emcpower1 ENA
sd datadg02_scsi-05 data01vx-01 datadg02_scsi 201376640 85256192 485285888 c1t1d0 ENA
sd datadg03-01 data01vx-01 datadg03 0 159332608 570542080 emcpower9 ENA
sd datadg04-01 data01vx-01 datadg04 0 159332608 729874688 emcpower7 ENA
sd datadg05-01 data01vx-01 datadg05 0 159332608 889207296 emcpower5 ENA
sd datadg06-01 data01vx-01 datadg06 0 159332608 1048539904 emcpower3 ENA
sd datadg07-01 data01vx-01 datadg07 0 159332608 1207872512 emcpower4 ENA
sd datadg08-01 data01vx-01 datadg08 0 159332608 1367205120 emcpower8 ENA
sd datadg09-01 data01vx-01 datadg09 0 159332608 1526537728 emcpower10 ENA
sd datadg10-02 data01vx-01 datadg10 4194304 155138304 1685870336 emcpower2 ENA
sd datadg11-02 data01vx-01 datadg11 4194304 68242736 1841008640 emcpower6 ENA

# vxmend -g datadg -o force off data01vx-01 (do this if volume is DISABLED RECOVER. If DISABLED STALE , proceed with vxmend -g datadg on data01vx-01)
# vxmend -g datadg on data01vx-01
# vxvol -g datadg start data01vx

Solution 215981 : Cold Backup of Oracle using Volume Manager Snapshot

Here is the procedure to create a snapshot of a volume containing an Oracle
database. This assumes the database is built on a file system :

1) vxassist -g  snapstart 

2) When the snapstart operation is complete and the mirror is in the SNAPDONE
state for the newly created plex (this can be determined through vxprint -ht
), find a time to complete the snapshot operation. This is the
point where you would bring down the oracle db and kick the users out. When
Oracle is down, execute this command:

# vxassist -g  snapshot  

3) When this operation is finished, you can bring Oracle online and then perform
an fsck of the snapshot:

# fsck -y /dev/vx/rdsk//

4) Next, backup the snapshot volume using your data backup utility and then the
snapshot can be removed using:

# vxedit -g  -rf rm 

Look at the following example:

bigwillie# vxprint -g bigdg -ht oracle
V  NAME         USETYPE      KSTATE   STATE    LENGTH   READPOL   PREFPLEX
PL NAME         VOLUME       KSTATE   STATE    LENGTH   LAYOUT    NCOL/WID MODE
SD NAME         PLEX         DISK     DISKOFFS LENGTH   [COL/]OFF DEVICE   MODE

v  oracle       fsgen        ENABLED  ACTIVE   102400   SELECT    -
pl oracle-01    oracle       ENABLED  ACTIVE   104139   CONCAT    -        RW
sd bigdg13-01   oracle-01    bigdg13  0        104139   0         c2t26d0  ENA
pl oracle-02    oracle       ENABLED  ACTIVE   104139   CONCAT    -        RW
sd bigdg11-02   oracle-02    bigdg11  64638    104139   0         c2t24d0  ENA
bigwillie# vxassist -g bigdg snapstart oracle
bigwillie# df -k /oraclemnt
Filesystem            kbytes    used   avail capacity  Mounted on
/dev/vx/dsk/bigdg/oracle
                      51200    1117   46960     3%    /oraclemnt
bigwillie# umount /oraclemnt
bigwillie# vxprint -g bigdg -ht oracle
V  NAME         USETYPE      KSTATE   STATE    LENGTH   READPOL   PREFPLEX
PL NAME         VOLUME       KSTATE   STATE    LENGTH   LAYOUT    NCOL/WID MODE
SD NAME         PLEX         DISK     DISKOFFS LENGTH   [COL/]OFF DEVICE   MODE

v  oracle       fsgen        ENABLED  ACTIVE   102400   SELECT    -
pl oracle-01    oracle       ENABLED  ACTIVE   104139   CONCAT    -        RW
sd bigdg13-01   oracle-01    bigdg13  0        104139   0         c2t26d0  ENA
pl oracle-02    oracle       ENABLED  ACTIVE   104139   CONCAT    -        RW
sd bigdg11-02   oracle-02    bigdg11  64638    104139   0         c2t24d0  ENA
pl oracle-03    oracle       ENABLED  SNAPDONE 104139   CONCAT    -        WO
sd bigdg05-02   oracle-03    bigdg05  208278   104139   0         c2t19d0  ENA
bigwillie# vxassist -g bigdg snapshot oracle oraclesnap
bigwillie# vxprint -g bigdg -ht oracle oraclesnap
V  NAME         USETYPE      KSTATE   STATE    LENGTH   READPOL   PREFPLEX
PL NAME         VOLUME       KSTATE   STATE    LENGTH   LAYOUT    NCOL/WID MODE
SD NAME         PLEX         DISK     DISKOFFS LENGTH   [COL/]OFF DEVICE   MODE

v  oracle       fsgen        ENABLED  ACTIVE   102400   SELECT    -
pl oracle-01    oracle       ENABLED  ACTIVE   104139   CONCAT    -        RW
sd bigdg13-01   oracle-01    bigdg13  0        104139   0         c2t26d0  ENA
pl oracle-02    oracle       ENABLED  ACTIVE   104139   CONCAT    -        RW
sd bigdg11-02   oracle-02    bigdg11  64638    104139   0         c2t24d0  ENA

v  oraclesnap   fsgen        ENABLED  ACTIVE   102400   ROUND     -
pl oraclesnap-01 oraclesnap  ENABLED  ACTIVE   104139   CONCAT    -        RW
sd bigdg05-02   oraclesnap-01 bigdg05 208278   104139   0         c2t19d0  ENA
bigwillie# mount /dev/vx/dsk/bigdg/oracle /oraclemnt
bigwillie# fsck -y /dev/vx/rdsk/bigdg/oraclesnap
file system is clean - log replay is not required
bigwillie#

-Backup up the /dev/vx/rdsk/bigdg/oracle device-

bigwillie# vxedit -g bigdg -rf rm oraclesnap

Solution 217667 : VxVM - Unable to mirror a striped volume using 'vxassist' command

When attempting to grow a volume with a striped plex to a single disk with ample free space, vxassist complains that there is not enough space to perform the operation.

Here are the details of a generic case.

Start with a volume containing at least one striped plex:

  # vxprint -ht
 V  NAME      RVG      KSTATE   STATE    LENGTH    READPOL   PREFPLEX UTYPE
 PL NAME      VOLUME   KSTATE   STATE    LENGTH    LAYOUT    NCOL/WID MODE
 SD NAME      PLEX     DISK     DISKOFFS LENGTH    [COL/]OFF DEVICE   MODE

  v  vol01     -        ENABLED  ACTIVE   105906176 SELECT    -        fsgen
 pl vol01-01  vol01    ENABLED  ACTIVE   105907104 STRIPE    3/128    RW
 sd disk01-01 vol01-01 disk01   0        35302304  0/0       c4t3d0   ENA
 sd disk02-01 vol01-01 disk02   0        35302304  1/0       c4t7d0   ENA
 sd disk03-01 vol01-01 disk03   0        35302304  2/0       c4t23d0  ENA

and another disk with free space on which to mirror it:

  # vxdg -g ckdg free
 DISK         DEVICE       TAG          OFFSET    LENGTH    FLAGS
 disk09       c5t34d0s2    c5t34d0      0         143328960 -

However, when an attempt is made to mirror this volume using this disk, it fails with this error:

  # vxassist -g ckdg mirror vol01 alloc=disk09
 vxvm:vxassist: ERROR: Cannot allocate space to mirror 105906176 block volume

Resolution

The reason is that when mirroring a striped plex, the 'vxassist' command wants to create *another* striped plex as the mirror, by default. So, the command is failing not because there was not enough space to mirror it, but because there was not enough disks specified to create another striped plex.

To actually get vxassist to do what you want, that is, to mirror the striped plex to a new concat plex, you must use the 'layout=nostripe' option as well. Therefore, the command

  # vxassist -g ckdg mirror vol01 layout=nostripe alloc=disk09

will actually complete.

VxVM : How to split a mirror in Veritas Volume manager

This document describes how to split a Veritas mirror so that the sub-mirrors can be associated with new volumes, and hence be accessed via Solaris[TM] devices and possibly mounted.

Be aware that you may not be able to mount a filesystem taken from a split mirror, depending on the state of the filesystem at the time of the split. If a filesystem is unmounted cleanly prior to the split, both copies should be mountable afterwards.

So this is our starting point, a single volume with two plexes forming a simple mirror in the disk group 'baddg'.

# vxprint -g baddg
TY NAME         ASSOC        KSTATE   LENGTH   PLOFFS   STATE    TUTIL0  PUTIL0
dg baddg        baddg        -        -        -        -        -       -

dm sd01         Disk_1       -        35358848 -        -        -       -

v  vol1         gen          ENABLED  207328   -        ACTIVE   -       -
pl plex1        vol1         ENABLED  207328   -        ACTIVE   -       -
sd sd01-B0      plex1        ENABLED  1        0        -        -       Block0
sd sd01-02      plex1        ENABLED  207327   1        -        -       -
pl plex2        vol1         ENABLED  207328   -        ACTIVE   -       -
sd sd01-01      plex2        ENABLED  207328   0        -        -       -

It has a filesystem which is mounted.

# mount |grep baddg
/a on /dev/vx/dsk/baddg/vol1
 read/write/setuid/intr/largefiles/logging/xattr/onerror=panic/dev=3b41f40
 on Thu Jul 21 16:05:26 2005

We write a file and calculate the checksum. (Note: This is not a necessary part of the procedure but will be used later in this document to illustrate that the procedure has been a success.)

# cp /kernel/genunix /a
# ls -l /a
total 3600
-rwxr-xr-x   1 root     other    1823432 Jul 21 16:08 genunix
drwx------   2 root     root        8192 Jul 21 16:02 lost+found
# cksum /a/genunix
411313929       1823432 /a/genunix

Now unmount the filesystem, so that we know everything has been flushed to the physical disk.

# umount /a

In the following steps we are going to disassociate the plexes from the parent volume using vxplex 'dis'. VxVM also has a vxplex 'det' which is to detatch a plex. Confusingly, detatching a plex with vxplex 'det', stops writes to the plex, but the plex stays associated with the parent volume.

The first step is to detach one of the plexes and break its link to the remaining plex(es) (i.e. dissociate). We'll dissociate 'plex2'

# vxplex -g baddg dis plex2

Let's check how VxVM sees things now - plex2 is no longer associated with the volume.

# vxprint -g baddg
TY NAME         ASSOC        KSTATE   LENGTH   PLOFFS   STATE    TUTIL0  PUTIL0
dg baddg        baddg        -        -        -        -        -       -

dm sd01         Disk_1       -        35358848 -        -        -       -

pl plex2        -            DISABLED 207328   -        -        -       -
sd sd01-01      plex2        ENABLED  207328   0        -        -       -

v  vol1         gen          ENABLED  207328   -        ACTIVE   -       -
pl plex1        vol1         ENABLED  207328   -        ACTIVE   -       -
sd sd01-B0      plex1        ENABLED  1        0        -        -       Block0
sd sd01-02      plex1        ENABLED  207327   1        -        -       -

Now create a volume using the disassociated plex (plex2). The volume will be called 'vol2'

# vxmake -g baddg vol vol2 plex=plex2

vxprint shows the new volume, but the volume is disabled.
v  vol2         gen          DISABLED 207328   -        EMPTY    -       -
pl plex2        vol2         DISABLED 207328   -        EMPTY    -       -
sd sd01-01      plex2        ENABLED  207328   0        -        -       -

Next, we must start the new volume

# vxvol -g baddg start vol2

The new volume is now available to mount.

v  vol2         gen          ENABLED  207328   -        ACTIVE   -       -
pl plex2        vol2         ENABLED  207328   -        ACTIVE   -       -
sd sd01-01      plex2        ENABLED  207328   0        -        -       -

We can now mount the two volumes that were previously two halves of a single mirror and see that the same file is there, and has the same checksum - i.e. is a consistent copy.

First we mount the new volume

# mount /dev/vx/dsk/baddg/vol2 /b
# ls -l /b
total 3600
-rwxr-xr-x   1 root     other    1823432 Jul 21 16:08 genunix
drwx------   2 root     root        8192 Jul 21 16:02 lost+found
# cksum /b/genunix
411313929       1823432 /b/genunix

Now the original volume (no longer a mirror)

# mount /dev/vx/dsk/baddg/vol1 /a
# ls -l /a
total 3600
-rwxr-xr-x   1 root     other    1823432 Jul 21 16:08 genunix
drwx------   2 root     root        8192 Jul 21 16:02 lost+found
# cksum /a/genunix
411313929       1823432 /a/genunix

To recreate the original mirror, follow the above steps to disassociate the plex from the new volume. Then use vxplex 'att' to attach the plex to the original volume. The plex will be synchronised with the plex(es) already attached to the volume.

VxVM 4.0 : Alignment error with device creation commands.

When using device creation commands, such as vxmake, an error is returned:

VxVM vxmake ERROR: Length for subdisk  is not a multiple of the disk group alignment.

VxVM vxmake INFO: Subdisk length is <13422294> while disk group alignment is 16

This error is due to the diskgroup alignment size in vxvm 4.0. The alignment size for 4.0, by default is 16 blocks (8k). This size is set to accommodate Cross-Platform Data Sharing(CDS) capability. Unless specified otherwise, CDS is assumed when a new volume is created, thus the alignment size is set to 16 blocks. In earlier versions, the alignment size was set at 1 block, and was not visible.

This can create problems when using / moving devices within diskgroups.

Resolution

QOALQ6H Internal ID use only.

First, check if the 2 diskgroups have differing alignment sizes:

To compare 2 different diskgroups, use the following command:

vxdg list

Example: (also, notice the flag "cds")

Group:     newdg
dgid:      1106270153.72.lab1
import-id: 1024.71
flags:     cds    <-------------
version:   110
alignment: 8192 (bytes)
ssb:            on
detach-policy: global
copies:    nconfig=default nlog=default
config:    seqno=0.1042 permlen=1280 free=1275 templen=2 loglen=192
config disk Disk_4 copy 1 len=1280 state=clean online
log disk Disk_4 copy 1 len=192

Example:

Group:     newdg1
dgid:      1106276811.60.lab1
import-id: 1024.59
flags:   
version:   110
alignment: 512 (bytes)
ssb:            on
detach-policy: global
copies:    nconfig=default nlog=default
config:    seqno=0.1044 permlen=1280 free=1275 templen=2 loglen=192
config disk Disk_4 copy 1 len=1280 state=clean online
log disk Disk_4 copy 1 len=192

Here, we can see the 2 differing alignment fields, one is set at 8192 bytes, the other at 512 bytes, reflecting the 8 kB -vs- 1 block difference.

If not utilizing the CDS feature, you can set the CDS to off, then change the alignment size of the diskgroup. The CDS feature must first be disabled for the diskgroup, or it will not allow you to change the alignment size.

Example:

#vxdg -g  set cds=off
#vxdg -g  set alignment=1

After these commands, the diskgroups' alignment should now show as 512 in the vxdg list output.

Example:
Group:     newdg
dgid:      1106270153.72.lab1
import-id: 1024.71
flags:    
version:   110
alignment: 512 (bytes)
ssb:            on
detach-policy: global
copies:    nconfig=default nlog=default
config:    seqno=0.1042 permlen=1280 free=1275 templen=2 loglen=192
config disk Disk_4 copy 1 len=1280 state=clean online
log disk Disk_4 copy 1 len=192

Additional Information

QOVLUR4 Internal ID use only.

Example:

Create diskgroup, volumes, etc, in any earlier version of VxVM. (non-4.0)
Mirror volume.
Attach the mirror plex, allow to sync.
Detach one of the mirror plexes.
Upgrade to vxvm 4.0
Create new diskgroup, use pre-existing plex. 

Here is where we would see the error. Once we try to associate the existing plex with a new diskgroup, the alignment mismatch will appear.

This could come up in several different instances. Some customers use their own scripted method of a "snapshot" by adding a 3rd plex to a mirrored volume, syncing the plex, detaching & creating a new diskgroup using this plex. They would then use the plex to create a new volume & diskgroup as means of a backup.

Product

VERITAS Volume Manager 4.1 Software
VERITAS Volume Manager 4.0 Software
VERITAS Storage Foundation 4.1 Software
VERITAS Storage Foundation 4.0 Software