Reattaching to failed devices with Veritas Volume Manager

From : http://prefetch.net/

When Veritas loses contact (e.g., if a fiber cable is removed between a server and a storage array) with an active device, Veritas will place the device in the failed state, which will be reported as “failed was: cXtXdX” in the vxdisk(1m) output:

$ vxdisk list

DEVICE       TYPE            DISK         GROUP        STATUS
c0t0d0s2     auto:none       -            -            online invalid
c0t1d0s2     auto:none       -            -            online invalid
c1t1d0s2     auto:cdsdisk    c1t1d0       oradg        online
c1t2d0s2     auto:cdsdisk    -            -            online
c1t3d0s2     auto:cdsdisk    -            -            online
c1t4d0s2     auto:cdsdisk    -            -            online
c1t5d0s2     auto:cdsdisk    -            -            online
c1t6d0s2     auto:cdsdisk    -            -            online
-            -         c1t2d0       oradg        failed was:c1t2d0s2
-            -         c1t3d0       oradg        failed was:c1t3d0s2
-            -         c1t4d0       oradg        failed was:c1t4d0s2
-            -         c1t5d0       oradg        failed was:c1t5d0s2
-            -         c1t6d0       oradg        failed was:c1t6d0s2

When situations like this arise, the vxreattach(1m) utility can be used to reconnect Veritas to lost devices:

$ vxreattach

$ vxdisk list

DEVICE       TYPE            DISK         GROUP        STATUS
c0t0d0s2     auto:none       -            -            online invalid
c0t1d0s2     auto:none       -            -            online invalid
c1t1d0s2     auto:cdsdisk    c1t1d0       oradg        online
c1t2d0s2     auto:cdsdisk    c1t2d0       oradg        online
c1t3d0s2     auto:cdsdisk    c1t3d0       oradg        online
c1t4d0s2     auto:cdsdisk    c1t4d0       oradg        online
c1t5d0s2     auto:cdsdisk    c1t5d0       oradg        online
c1t6d0s2     auto:cdsdisk    c1t6d0       oradg        online

Prior to running vxreattach(1m), the problem that caused Veritas to lose contact with the devices should be corrected (e.g., the fiber cable should be reconnected to the server), or vxreattach(1m) should be invoked with the “-c” (check if a reattach is possible) option to see if a reattach is possible. vxreattach(1m) is a cool utility, and should go into every storage adminstrators utility belt.

fmthard command

Labeling Multiple Disks for Use With Volume Manager Software

Donald S. Ragbirsingh, April 2007

Introduction

Typically, an admin has to add many disks to production systems. This tip provides a time-saving procedure for labeling multiple disks for use with volume manager software. If you do the procedure appropriately, you can cut down disaster recovery times. The procedure works with all versions of the Solaris Operating System that use VERITAS Volume Manager (VxVM) software as the main disk management utility.

Caution: Be sure you use this procedure appropriately or you might cause undesirable consequences. For example, if your boot disk is c1t2d0 and it has no mirroring, and you mistakenly include the boot disk when performing step 3, the server crashes. You must know which of your disks are expendable disks before performing this procedure.

I usually work with EMC, CLARiiON and STK type disks, but this procedure works with any type of disk. Most EMC/CLARiiON and STK disks need to be labeled before use. (Editor's note: STK is short for StorageTek, now part of Sun Microsystems.)

Step 1

Create a command file (for example, format.cmd) for the format utility, and add the following content to the file:

label
quit

At the command line, write a label on the disk using the format.cmd file, for example:

# format -s -f format.cmd c3t21d14

Repeat this step, as required, using a script if necessary:

for i in 15 16 17 18 19 20
do
  format -s -f format.cmd c3t21d$i
  echo "labeled c3t21d$i ..."
done

Step 2

Use one of the new disks to create a prepared disk for VxVM, that is, zero out all the partitions except slice 2. Note: This means you have to manually do one disk, for example, c3t21d5.

Save the Volume Table of Contents (VTOC) of this disk (in this example, c3t21d5) as a file:

# prtvtoc /dev/rdsk/c3t21d5s2 > master_vtoc

Step 3

Copy the VTOC to a destination disk, for example, c3t21d6.

# fmthard -s /tmp/master_vtoc /dev/rdsk/c3t21d6s2
fmthard:  New volume table of contents now in place.

Repeat this step, as required, using a script if necessary:

for i in 14 15 16 17 18 19 20 21
do
  fmthard -s /tmp/master_vtoc /dev/rdsk/c3t21d$i
  echo "Re-partitioned c3t21d$i ..."
done

Try to avoid recalling backup tapes, that is, do not make mistakes!

As an offshoot of this procedure, you can issue the following devastatingly simple, one-line command instead of using the VTOC file. However, the choice is up to you.

prtvtoc /dev/rdsk/c3t21d5s2 | fmthard -s - /dev/rdsk/c3t21d13s2

Conclusion

The uses of this procedure are myriad. For example, if you wanted to add an SVM mirror, you could take the boot device (c1t0d0 mirrored to c1t1d0) and copy the boot VTOC to the target c1t1d0 as follows:

prtvtoc /dev/rdsk/c1t0d0s2 | fmthard -s - /dev/rdsk/c1t1d0s2

Then it's trivial to start the mirroring process.

You can add also spares, but I'd like to save that process for another tip.

About the Author

Donald Ragbirsingh works for his own company, Opentech Systems Inc. He has been involved in IT in the banking, insurance, and telecommunications industries for over 10 years in Canada.

SunSolve Solution 202396 : How to use fmthard to duplicate the "vtoc" of a hard disk

VXVM : Encapsulate Boot-disk CLI

# vxconfigd -r reset -m disable
# vxdctl init
# vxdctl enable
# rm /etc/vx/reconfig.d/state.d/install-db
# vxencap -g -c rootdisk=
# reboot

How To Recover Veritas Volume Manager RAID0 Volume

How To Recover Veritas Volume Manager RAID0 Volume

Veritas Volume Manager(VxVM) RAID0 volume has either stripe or
concatenation layout. Suppose a subdisk has being removed from RAID0
volume e.g vxdiskadm #4, the data in the volume will be corrupted. An
attempt to recover the volume data by putting back the original subdisk
into RAID0 volume e.g vxdiskadm #5 will give error message " appears to be
owned by disk group ".

The following procedure show how to recover the Raid0 volume in this
situation with the following assumptions:
(1) no physical disk or Lun has being replaced. All the disks are in good
& original position
(2) no destructive command has being run which affect the integrity of
VxVM configuration e.g initialize the removed disk.

If you are unsure on above assumption, stop proceeding & seek help from Sun.

Steps to Follow

These are the following steps to take:

1) record down relevent disk vtoc table
privlen=tag 15 slice sector count
privoffset=tag 15 slice first sector
publen=tag 14 slice sector count
puboffset=tag14 slice first sector

2) uninitialize disk
# vxdiskunsetup -C

3) re-initialize disk back to original private & public region with input
from step(1)
# vxdisksetup -i privlen=<> privoffset=<> publen=<> puboffset=<>

4) prtvtoc to confirm the newly rebuilded vtoc table
# prtvtoc /dev/dsk/cXtXdXs2
# vxdisk list
** vxdisk output privlen may be smaller that vtoc due to roundup, but
publen MUST match on both outputs

5) put disk back to dg
# vxdg -g -k adddisk =

6) recover plex/volume
# vxmend -g -o force off
** plex under OFFLINE state & volume DISABLED/ACTIVE
# vxmend -g on
** plex under STALE state & volume DISABLED/ACTIVE
# vxmend -g fix active
** plex under DISABLED/ACTIVE & volume DISABLED/NEEDSYNC
# vxvol -g start
** plex & volume are under ENABLE/ACTIVE

7) fsck & mount the volume

VxVM : CLI version of vxdiskadm option 4 & 5

Option 4
# vxdg -g datadg -k rmdisk datadg01

Option 5
# ./vxdisksetup -if TagmaStore-USP0_0
# vxdg -g datadg -k adddisk datadg01=TagmaStore-USP0_0
# vxrecover -g datadg -b datadg01

Real Life example
------------------
# vxmend -g datadg -o force off data01vx-01 (do this if volume is DISABLED RECOVER.
If DISABLED STALE , proceed with vxmend -g datadg on data01vx-01)
# vxmend -g datadg on data01vx-01
# vxvol -g datadg start data01vx

# vxdisk list

DEVICE TYPE DISK GROUP STATUS

TagmaStore-USP0_0 auto:cdsdisk - - online
c1t0d0s2 auto:sliced odg01 odg online
c1t1d0s2 auto:cdsdisk datadg02_scsi datadg online
c1t2d0s2 auto:sliced odg02 odg online
c1t3d0s2 auto:cdsdisk datadg01_scsi datadg online
emcpower1s2 auto:cdsdisk datadg02 datadg online nohotuse
emcpower2s2 auto:cdsdisk datadg10 datadg online
emcpower3s2 auto:cdsdisk datadg06 datadg online nohotuse
emcpower4s2 auto:cdsdisk datadg07 datadg online nohotuse
emcpower5s2 auto:cdsdisk datadg05 datadg online nohotuse
emcpower6s2 auto:cdsdisk datadg11 datadg online
emcpower7s2 auto:cdsdisk datadg04 datadg online nohotuse
emcpower8s2 auto:cdsdisk datadg08 datadg online nohotuse
emcpower9s2 auto:cdsdisk datadg03 datadg online nohotuse
emcpower10s2 auto:cdsdisk datadg09 datadg online nohotuse
- - datadg01 datadg failed was:TagmaStore-USP0_0

# vxdg -g datadg -k rmdisk datadg01
# vxdisk list

DEVICE TYPE DISK GROUP STATUS

TagmaStore-USP0_0 auto:cdsdisk - - online
c1t0d0s2 auto:sliced odg01 odg online
c1t1d0s2 auto:cdsdisk datadg02_scsi datadg online
c1t2d0s2 auto:sliced odg02 odg online
c1t3d0s2 auto:cdsdisk datadg01_scsi datadg online
emcpower1s2 auto:cdsdisk datadg02 datadg online nohotuse
emcpower2s2 auto:cdsdisk datadg10 datadg online
emcpower3s2 auto:cdsdisk datadg06 datadg online nohotuse
emcpower4s2 auto:cdsdisk datadg07 datadg online nohotuse
emcpower5s2 auto:cdsdisk datadg05 datadg online nohotuse
emcpower6s2 auto:cdsdisk datadg11 datadg online
emcpower7s2 auto:cdsdisk datadg04 datadg online nohotuse
emcpower8s2 auto:cdsdisk datadg08 datadg online nohotuse
emcpower9s2 auto:cdsdisk datadg03 datadg online nohotuse
emcpower10s2 auto:cdsdisk datadg09 datadg online nohotuse
- - datadg01 datadg removed was:TagmaStore-USP0_0

root@S21TDI234 # ./vxdisksetup -if TagmaStore-USP0_0
root@S21TDI234 # vxdisk list

DEVICE TYPE DISK GROUP STATUS

TagmaStore-USP0_0 auto:cdsdisk - - online
c1t0d0s2 auto:sliced odg01 odg online
c1t1d0s2 auto:cdsdisk datadg02_scsi datadg online
c1t2d0s2 auto:sliced odg02 odg online
c1t3d0s2 auto:cdsdisk datadg01_scsi datadg online
emcpower1s2 auto:cdsdisk datadg02 datadg online nohotuse
emcpower2s2 auto:cdsdisk datadg10 datadg online
emcpower3s2 auto:cdsdisk datadg06 datadg online nohotuse
emcpower4s2 auto:cdsdisk datadg07 datadg online nohotuse
emcpower5s2 auto:cdsdisk datadg05 datadg online nohotuse
emcpower6s2 auto:cdsdisk datadg11 datadg online
emcpower7s2 auto:cdsdisk datadg04 datadg online nohotuse
emcpower8s2 auto:cdsdisk datadg08 datadg online nohotuse
emcpower9s2 auto:cdsdisk datadg03 datadg online nohotuse
emcpower10s2 auto:cdsdisk datadg09 datadg online nohotuse
- - datadg01 datadg removed was:TagmaStore-USP0_0

# vxdg -g datadg -k adddisk datadg01=TagmaStore-USP0_0
# vxdisk list

DEVICE TYPE DISK GROUP STATUS

TagmaStore-USP0_0 auto:cdsdisk datadg01 datadg online
c1t0d0s2 auto:sliced odg01 odg online
c1t1d0s2 auto:cdsdisk datadg02_scsi datadg online
c1t2d0s2 auto:sliced odg02 odg online
c1t3d0s2 auto:cdsdisk datadg01_scsi datadg online
emcpower1s2 auto:cdsdisk datadg02 datadg online nohotuse
emcpower2s2 auto:cdsdisk datadg10 datadg online
emcpower3s2 auto:cdsdisk datadg06 datadg online nohotuse
emcpower4s2 auto:cdsdisk datadg07 datadg online nohotuse
emcpower5s2 auto:cdsdisk datadg05 datadg online nohotuse
emcpower6s2 auto:cdsdisk datadg11 datadg online
emcpower7s2 auto:cdsdisk datadg04 datadg online nohotuse
emcpower8s2 auto:cdsdisk datadg08 datadg online nohotuse
emcpower9s2 auto:cdsdisk datadg03 datadg online nohotuse
emcpower10s2 auto:cdsdisk datadg09 datadg online nohotuse

# vxrecover -g datadg -b datadg01
# vxprint -g datadg -ht

v data01vx - DISABLED ACTIVE 1909251376 SELECT -
fsgen
pl data01vx-01 data01vx DISABLED RECOVER 1909251376 CONCAT - RW
sd datadg01-01 data01vx-01 datadg01 0 240596992 0 TagmaStore-USP0_0 ENA
sd datadg01_scsi-05 data01vx-01 datadg01_scsi 201326592 85306240 240596992 c1t3d0 ENA
sd datadg02_scsi-04 data01vx-01 datadg02_scsi 201326592 50048 325903232 c1t1d0 ENA
sd datadg02-01 data01vx-01 datadg02 0 159332608 325953280 emcpower1 ENA
sd datadg02_scsi-05 data01vx-01 datadg02_scsi 201376640 85256192 485285888 c1t1d0 ENA
sd datadg03-01 data01vx-01 datadg03 0 159332608 570542080 emcpower9 ENA
sd datadg04-01 data01vx-01 datadg04 0 159332608 729874688 emcpower7 ENA
sd datadg05-01 data01vx-01 datadg05 0 159332608 889207296 emcpower5 ENA
sd datadg06-01 data01vx-01 datadg06 0 159332608 1048539904 emcpower3 ENA
sd datadg07-01 data01vx-01 datadg07 0 159332608 1207872512 emcpower4 ENA
sd datadg08-01 data01vx-01 datadg08 0 159332608 1367205120 emcpower8 ENA
sd datadg09-01 data01vx-01 datadg09 0 159332608 1526537728 emcpower10 ENA
sd datadg10-02 data01vx-01 datadg10 4194304 155138304 1685870336 emcpower2 ENA
sd datadg11-02 data01vx-01 datadg11 4194304 68242736 1841008640 emcpower6 ENA

# vxmend -g datadg -o force off data01vx-01 (do this if volume is DISABLED RECOVER. If DISABLED STALE , proceed with vxmend -g datadg on data01vx-01)
# vxmend -g datadg on data01vx-01
# vxvol -g datadg start data01vx

Solution 215981 : Cold Backup of Oracle using Volume Manager Snapshot

Here is the procedure to create a snapshot of a volume containing an Oracle
database. This assumes the database is built on a file system :

1) vxassist -g  snapstart 

2) When the snapstart operation is complete and the mirror is in the SNAPDONE
state for the newly created plex (this can be determined through vxprint -ht
), find a time to complete the snapshot operation. This is the
point where you would bring down the oracle db and kick the users out. When
Oracle is down, execute this command:

# vxassist -g  snapshot  

3) When this operation is finished, you can bring Oracle online and then perform
an fsck of the snapshot:

# fsck -y /dev/vx/rdsk//

4) Next, backup the snapshot volume using your data backup utility and then the
snapshot can be removed using:

# vxedit -g  -rf rm 

Look at the following example:

bigwillie# vxprint -g bigdg -ht oracle
V  NAME         USETYPE      KSTATE   STATE    LENGTH   READPOL   PREFPLEX
PL NAME         VOLUME       KSTATE   STATE    LENGTH   LAYOUT    NCOL/WID MODE
SD NAME         PLEX         DISK     DISKOFFS LENGTH   [COL/]OFF DEVICE   MODE

v  oracle       fsgen        ENABLED  ACTIVE   102400   SELECT    -
pl oracle-01    oracle       ENABLED  ACTIVE   104139   CONCAT    -        RW
sd bigdg13-01   oracle-01    bigdg13  0        104139   0         c2t26d0  ENA
pl oracle-02    oracle       ENABLED  ACTIVE   104139   CONCAT    -        RW
sd bigdg11-02   oracle-02    bigdg11  64638    104139   0         c2t24d0  ENA
bigwillie# vxassist -g bigdg snapstart oracle
bigwillie# df -k /oraclemnt
Filesystem            kbytes    used   avail capacity  Mounted on
/dev/vx/dsk/bigdg/oracle
                      51200    1117   46960     3%    /oraclemnt
bigwillie# umount /oraclemnt
bigwillie# vxprint -g bigdg -ht oracle
V  NAME         USETYPE      KSTATE   STATE    LENGTH   READPOL   PREFPLEX
PL NAME         VOLUME       KSTATE   STATE    LENGTH   LAYOUT    NCOL/WID MODE
SD NAME         PLEX         DISK     DISKOFFS LENGTH   [COL/]OFF DEVICE   MODE

v  oracle       fsgen        ENABLED  ACTIVE   102400   SELECT    -
pl oracle-01    oracle       ENABLED  ACTIVE   104139   CONCAT    -        RW
sd bigdg13-01   oracle-01    bigdg13  0        104139   0         c2t26d0  ENA
pl oracle-02    oracle       ENABLED  ACTIVE   104139   CONCAT    -        RW
sd bigdg11-02   oracle-02    bigdg11  64638    104139   0         c2t24d0  ENA
pl oracle-03    oracle       ENABLED  SNAPDONE 104139   CONCAT    -        WO
sd bigdg05-02   oracle-03    bigdg05  208278   104139   0         c2t19d0  ENA
bigwillie# vxassist -g bigdg snapshot oracle oraclesnap
bigwillie# vxprint -g bigdg -ht oracle oraclesnap
V  NAME         USETYPE      KSTATE   STATE    LENGTH   READPOL   PREFPLEX
PL NAME         VOLUME       KSTATE   STATE    LENGTH   LAYOUT    NCOL/WID MODE
SD NAME         PLEX         DISK     DISKOFFS LENGTH   [COL/]OFF DEVICE   MODE

v  oracle       fsgen        ENABLED  ACTIVE   102400   SELECT    -
pl oracle-01    oracle       ENABLED  ACTIVE   104139   CONCAT    -        RW
sd bigdg13-01   oracle-01    bigdg13  0        104139   0         c2t26d0  ENA
pl oracle-02    oracle       ENABLED  ACTIVE   104139   CONCAT    -        RW
sd bigdg11-02   oracle-02    bigdg11  64638    104139   0         c2t24d0  ENA

v  oraclesnap   fsgen        ENABLED  ACTIVE   102400   ROUND     -
pl oraclesnap-01 oraclesnap  ENABLED  ACTIVE   104139   CONCAT    -        RW
sd bigdg05-02   oraclesnap-01 bigdg05 208278   104139   0         c2t19d0  ENA
bigwillie# mount /dev/vx/dsk/bigdg/oracle /oraclemnt
bigwillie# fsck -y /dev/vx/rdsk/bigdg/oraclesnap
file system is clean - log replay is not required
bigwillie#

-Backup up the /dev/vx/rdsk/bigdg/oracle device-

bigwillie# vxedit -g bigdg -rf rm oraclesnap

Solution 217667 : VxVM - Unable to mirror a striped volume using 'vxassist' command

When attempting to grow a volume with a striped plex to a single disk with ample free space, vxassist complains that there is not enough space to perform the operation.

Here are the details of a generic case.

Start with a volume containing at least one striped plex:

  # vxprint -ht
 V  NAME      RVG      KSTATE   STATE    LENGTH    READPOL   PREFPLEX UTYPE
 PL NAME      VOLUME   KSTATE   STATE    LENGTH    LAYOUT    NCOL/WID MODE
 SD NAME      PLEX     DISK     DISKOFFS LENGTH    [COL/]OFF DEVICE   MODE

  v  vol01     -        ENABLED  ACTIVE   105906176 SELECT    -        fsgen
 pl vol01-01  vol01    ENABLED  ACTIVE   105907104 STRIPE    3/128    RW
 sd disk01-01 vol01-01 disk01   0        35302304  0/0       c4t3d0   ENA
 sd disk02-01 vol01-01 disk02   0        35302304  1/0       c4t7d0   ENA
 sd disk03-01 vol01-01 disk03   0        35302304  2/0       c4t23d0  ENA

and another disk with free space on which to mirror it:

  # vxdg -g ckdg free
 DISK         DEVICE       TAG          OFFSET    LENGTH    FLAGS
 disk09       c5t34d0s2    c5t34d0      0         143328960 -

However, when an attempt is made to mirror this volume using this disk, it fails with this error:

  # vxassist -g ckdg mirror vol01 alloc=disk09
 vxvm:vxassist: ERROR: Cannot allocate space to mirror 105906176 block volume

Resolution

The reason is that when mirroring a striped plex, the 'vxassist' command wants to create *another* striped plex as the mirror, by default. So, the command is failing not because there was not enough space to mirror it, but because there was not enough disks specified to create another striped plex.

To actually get vxassist to do what you want, that is, to mirror the striped plex to a new concat plex, you must use the 'layout=nostripe' option as well. Therefore, the command

  # vxassist -g ckdg mirror vol01 layout=nostripe alloc=disk09

will actually complete.