Vplex Metadata Backup How
Vplex Metadata Backup How
Version: 8 Article Type: Break Fix Audience: Level 30 = Customers Last Published: Thu Feb 28 20:32:30 GMT 2019
Summary: This article talks to what to do to re-create VPLEX meta-data back-ups when either the call home 0x8a4a6006 reports for "The
automated backup of the meta-volume could not be completed" or 0x8a4a6003 reports for "no valid backup meta-volumes exist, or 0x8a4a6005 reports for "Metadata
Backup does not create new backups every day".
Issue:
What is Metadata backup?
The Metadata backups are the backups of the active metadata. It includes all the system configuration settings that are in the active
Metadata. Metadata backup volumes are a system-volume in VPLEX, which are created when a VPLEX cluster is initially configured.
Metadata backups are point-in-time snapshots of the current active meta-volumes. The point-in-time of the meta-volume backup is taken based on the
schedule that was setup when they were initially configured. They can be activated only if the current active meta-volume or one leg of the active
metadata volume fails. They are meant to provide extra protection for major configuration changes, refreshes, or a migration.
Whenever a end user encounters a data unavailable (DU) situation due to back-end array issues, the meta-volume backups play an essential role for
recovering the VPLEX configuration if needed.
For redundancy, VPLEX has two metadata backup volumes that are to be created on two different arrays, same as each leg of the active metadata
residing on two different arrays. The two metadata backups rotate daily as per the schedule. You should always see one one date before the other. If
they show days have passed then there is an issue with the backup script not running or it could not complete due to an issue so the backups did not
run.
For example: "Meta-volume backup ("A")" gets updated today then "Meta-volume backup ("B")" will be updated next day and so on. See the below
output for more details :
VPlexcli:/clusters/cluster-1/system-volumes> ll
Name Volume Type Operational Health Active Ready Geometry Component Block
Block Capacity Slots
---------------------------- ----------- Status State ------ ----- -------- Count Count
Size -------- -----
---------------------------- ----------- ----------- ------ ------ ----- -------- ---------
-------- ----- -------- -----
meta meta-volume ok ok true true raid-1 2
20971264 4K 80G 32000
meta_backup_2016Jul09_040009 (“A”) meta-volume ok ok false true raid-1 1
20971264 4K 80G 64000
meta_backup_2016Jul10_040007 (“B”) meta-volume ok ok false true raid-1 1
20971264 4K 80G 64000
1. <SymptomCode>0x8a4a6006</SymptomCode>
<Category>Status</Category>
<Severity>Error</Severity>
<Status>Failed</Status>
<Component>CLUSTER</Component>
<ComponentID>SMS</ComponentID>
<SubComponent>CLUSTER-1</SubComponent>
<SubComponentID></SubComponentID>
<CallHome>Yes</CallHome>
<FirstTime>2016-07-10T00:00:01.334Z</FirstTime>
<LastTime>2016-07-09T00:00:01.334Z</LastTime>
<Count>1</Count>
<EventData><![CDATA[The automated backup of the meta-volume could not be completed.
[Versions:MS{D10.0.0.196.0, D4_MSB_7, D10.0.0.225}, Director{unknown},
ClusterWitnessServer{D10.0.0.209}] RCA: The automated backup of the meta-volume could not be completed.]]>
</EventData>
<Description>
The automated backup of the meta-volume could not be completed.
2. <SymptomCode>0x8a4a6003</SymptomCode>
<Category>Status</Category>
<Severity>Error</Severity>
<Status>Failed</Status>
<Component>CLUSTER</Component>
<ComponentID>SMS</ComponentID>
<SubComponent>CLUSTER-1</SubComponent>
<SubComponentID></SubComponentID>
<CallHome>Yes</CallHome>
<FirstTime>2018-09-07T03:00:12.191Z</FirstTime>
<LastTime>2018-09-07T03:00:12.191Z</LastTime>
<Count>1</Count>
<EventData><![CDATA[No valid backup meta-volumes exist. [Versions:MS{}, Director{}] RCA: The automated
backup of metadata cannot identify the devices to be used. This is because existing backups cannot be
located. The backups are rotated through being destroyed in order to be re-used.]]>
</EventData>
<Description>No valid backup meta-volumes exist.
<Status>Failed</Status>
3. <SymptomCode>0x8a4a6005</SymptomCode>
<Category>Status</Category>
<Severity>Error</Severity>
<Status>Failed</Status>
<Component>CLUSTER</Component>
<ComponentID>unknown</ComponentID>
<SubComponent>sms</SubComponent>
<SubComponentID></SubComponentID>
<CallHome>Yes</CallHome>
<FirstTime>2017-12-04T00:00:35.420Z</FirstTime>
<LastTime>2018-09-06T23:59:02.813Z</LastTime>
<Count>1</Count>
<EventData><![CDATA[A meta-volume backup could not be destroyed. Reason: The meta-volume backup "<name
of the affected metadata backup>" could not be destroyed: A meta-volume backup "<name of the affected
metadata backup>" is not healthy enough to be destroyed. [Versions:MS{D27.0.0.16.0, D4_MSB_7,
D27.0.0.18}, Director{unknown}, ClusterWitnessServer{unknown}] RCA: Backup meta-volume could not be
destroyed. Remedy: Confirm that the volumes configured to be used for the backup are in a healthy
state. If the volumes are unhealthy, create new automated metavolume backups by: 1. Destroy the
existing backups using the 'meta-volume destroy' command. 2. Unclaim those volumes if they are to be
re-used with the 'storage-volume unclaim' command. 3. Use the 'configuration metadata-backup' command
to reconfigure the backups. If the previous volumes used were not healthy enough to destroy, create
the backups with new healthy devices.
]]></EventData>
<Description><![CDATA[A meta-volume backup could not be destroyed.
The Metadata backup may have failed to capture the point-in-time copy due to the volume used
by the failed metadata backup being unhealthy on the back-end array or if there is a possible
connectivity issue between VPLEX and the back-end array where the backup volume is located.
First list out the system volume details using the ' ll /clusters/cluster-<id>/system-volumes'
command,
VPlexcli:/> ll /clusters/cluster-1/system-volumes/
/clusters/cluster-1/system-volumes:
Name Volume Type Operational Health Active Ready Geometry Component
Block Block Capacity Slots
------------------------------- ----------- Status State ------ ----- -------- Count
Count Size -------- -----
------------------------------- ----------- ----------- ------ ------ ----- -------- ---------
-------- ----- -------- -----
C1_Meta meta-volume ok ok true true raid-1 2
20971264 4K 80G 64000
META_VOLUME_backup_2018Jun11_044501 meta-volume ok ok false true raid-1 1
20971264 4K 80G 64000
META_VOLUME_backup_2018Jun12_044501 meta-volume ok ok false true raid-1 1
20971264 4K 80G 64000
Here is an example of a good Backup Meta-Volume, notice the component is still using the
VPD83T3: ID at the metadata backup volume component context level:
VPlexcli:/clusters/cluster-1/system-volumes/META_VOLUME_backup_2018Jun11_044501/components> ll
Here is an example of a bad Backup Meta-Volume, where the system volume id at the
component context level has been changed from its VPD83T3 system ID to a human-readable
name "C1_MetaBackup_1":
VPlexcli:/clusters/cluster-1/system-volumes/META_VOLUME_backup_2018Jun11_044501/components> ll
When checking the metadata backups you may see the date of backup is not current.
VPlexcli:/> date
Fri Sep 7 13:30:43 UTC YYYY <<<< note the current date in this example is Sep 7th
Next check the dates of when each backup volume had last run under the system-volumes
context, compare the dates listed in the backup volume names with the date just checked above
(date was enlarged for the example):
VPlexcli:/> ll /clusters/cluster-1/system-volumes/
/clusters/cluster-1/system-volumes:
Name Volume Type Operational Health Active Ready Geometry
Block Block Capacity Slots
------------------------------- -------------- Status State ------ ----- --------
Count Size -------- -----
------------------------------- -------------- ----------- ------ ------ ----- --------
-------- ----- -------- -----
c1_meta meta-volume ok ok true true raid-1
20971264 4K 80G 32000
c1_meta_backup_2018Aug01_030002 meta-volume ok ok false true raid-1 20971264
4K 80G 32000
c1_meta_backup_2018Aug02_030003 meta-volume ok ok false true raid-1 20971264
4K 80G 32000
Looking at the storage-array context level of the VPLEX there are LUNs that show 'visibility' as 'none' contained within the storage array that hosts a
backup volume and the 'connectivity-status' shows 'error'.
/clusters/cluster-1/storage-elements/storage-arrays/EMC-CLARiiON-CKM00000000000/logical
units/VPD83T3:6006016099xxxxxxxxxxxxxxxxx1e111:
Name Value
---------------------- --------------------
active-aao-controller [CKM00000000000.SPB]
active-aao-visibility []
alua-support none
connectivity-status error <<<< communication/connectivity issue between VPLEX and array
luns []
passive-aan-controller [CKM00000000000.SPA]
passive-aan-visibility []
storage-volume -
visibility none <<<<< not seeing the the back-end volume
This indicates that there is a connectivity issue between the storage array and VPLEX and this would cause an issue if it occurred during the running of
the automated backup script as it would not be able to see the storage volume from the array due to the connectivity issue.
Change: Renamed Metadata Backup Volume components (using the GUI) or no Metadata backup Volumes were configured. Refer to the Notes section below
for information on meta backup volumes.
Resolution:
A. For SymptomCode's 0x8a4a6003 and 0x8a4a6006:
NOTE: Attempting to rename the component with VPD83T3: ID has problems with the colon " : " and does not work.
Workaround:
If the VPLEX is a Metro configuration ensure you run the workaround on the cluster that reported the issue should you need to delete the
metabackup volumes.
1. List out the system-volumes on the cluster(s) where the issue was reported in the call home message,
(see sample call home details in the Issue section) to see the details using the command
ll /clusters/cluster-<id>/system-volumes,
NOTE: you can type the command as 'll /clusters/*/system-volumes' and this will list the
system-volume details for all clusters in the configuration. If this is a VPLEX-Local then you will
only see the info for cluster-1.
VPlexcli:/> ll /clusters/cluster-1/system-volumes
Name Volume Type Operational Health Active Ready Geometry Component
Block Block Capacity Slots
------------------------------- ----------- Status State ------ ----- -------- Count
Count Size -------- -----
------------------------------- ----------- ----------- ------ ------ ----- -------- ---------
-------- ----- -------- -----
C1_Meta meta-volume ok ok true true raid-1 2
20971264 4K 80G 64000
Meta_backup_2018Sep07_154626 meta-volume ok ok false true raid-1 1
20971264 4K 80G 64000
Meta_backup_2018Sep07_154649 meta-volume ok ok false true raid-1 1
20971264 4K 80G 64000
Example for VPD number: The below command will display only those storage-volumes which
meets the requirement of metadata-backup :
2. Next delete the existing metabackup volumes using the "meta-volume destroy" command.
For example:
VPlexcli:/clusters/cluster-1/system-volumes> meta-volume destroy Meta_backup_2018Sep07_154626
VPlexcli:/clusters/cluster-1/system-volumes> meta-volume destroy Meta_backup_2018Sep07_154649
3. Run the 'schedule list' command from the VPlexcli to display the current "metadata backup local"
schedule and the job number associated with it.
For example:
VPlexcli:/> schedule list
[0] 30 13 * * 3 syrcollect
[2] 23 30 * * * metadata backup local
4. Remove the 'metadata backup local' schedule by running 'schedule remove [job ID]' mentioned in
step (3).
For example:
VPlexcli:/> schedule remove 2
Removed scheduled job 2.
5. Unclaim both of the former metadata backup volumes by using below command.
6. Recreate the metadata backups and set the schedule* for when to run at desired time each day.
*Note: you may see that a scheduled time is already set, if you want to keep that scheduled time
type "Y", if not type "N" and later in the script you will be prompted for the new time you want to
have the metadata backups run.
Would you like to change the volumes on which to backup the metadata? [no]: Yes
Available Volumes for Meta-data Backup
You have chosen to configure the backup of the meta-data. Please note:
All times are UTC and are not based on the local time.
Would you like to run the setup process now? [yes]: yes
Scheduling the backup of metadata...
Performing metadata backup (This will take a few minutes)
Successfully performed the initial backing up of metadata
Successfully scheduled the backing up of metadata
Successfully scheduled the metadata backup
Sample output:
VPlexcli:/> ll /clusters/cluster-1/system-volumes/
/clusters/cluster-1/system-volumes:
Name Volume Type Operational Health Active Ready Geometry Component
Block Block Capacity Slots
------------------------------- ----------- Status State ------ ----- -------- Count
Count Size -------- -----
------------------------------- ----------- ----------- ------ ------ ----- -------- ---------
-------- ----- -------- -----
C1_Meta meta-volume ok ok true true raid-1 2
20971264 4K 80G 64000
C1_Meta_backup_2018Oct07_123208 meta-volume ok ok false true raid-1 1
20971264 4K 80G 64000
C1_Meta_backup_2018Oct07_123208 meta-volume ok ok false true raid-1 1
20971264 4K 80G 64000
8. Then run the 'schedule list' command again to confirm the "metadatabackup local' is listed with
the correct time you had set it to run each day.
9. Now that you have removed the old metadata backup volumes and recreated new ones, monitor
the backups for a couple days to ensure they run as scheduled. The script will alternate between
the two backup volumes each time the backup script runs so you should see one backup dated
the day after the other. The time the backup runs is appended to the backup name and it is okay if
it is not exactly at the time set, it may vary a little, this is normal. You should see at least one
backup volume has a new date if this is the first time running the new backups.
Example:
VPlexcli:/> ll /clusters/*/system-volumes/
/clusters/cluster-1/system-volumes:
Name Volume Type Operational Health Active Ready Geometry
Component Block Block Capacity Slots
------------------------------- -------------- Status State ------ ----- -------- Count
Count Size -------- -----
------------------------------- -------------- ----------- ------ ------ ----- --------
--------- -------- ----- -------- -----
C1Logging_vol logging-volume ok ok - - raid-0 1
2621440 4K 10G -
C1_Meta meta-volume ok ok true true raid-1 2
20971264 4K 80G 64000
C1_Meta_backup_2018Oct08_044532 meta-volume ok ok false true raid-1 1
20971264 4K 80G 64000
C1_Meta_backup_2018Oct07_123208 meta-volume ok ok false true raid-1 1
20971264 4K 80G 64000
Notes: For information on Meta Volumes and Meta Volume Backups, please refer the CLI Guide for the version of GeoSynchrony the VPLEX is running and
search for "configuration metadata-backup" and to see a list of volumes that can be used for meta data backup volumes search for " configuration
show-meta-volume-candidates".
Sample out put for seeing what volumes were available for use to create metadata backup volumes,
Sample output when you want to change the time the backups will run:
A back up of the meta-data is already scheduled to occur everyday at 4:15 (UTC). Do you want change the
existing schedule? (Y/N): y
Would you like to change the time the meta-data is backed up? [no]: yes << [Here is where you're asked again if want to
change the time the backups will run]
What hour of the day (UTC) should the meta-data be backed up? (0..23): 23
What minute of the hour should the meta-data be backed up? (0..59): 00
Would you like to change the time the meta-data is backed up? [no]: <<
[use the default selection this time to keep the newly set time by pressing the Enter/Return key]
You have chosen to configure the backup of the meta-data. Please note:
All times are UTC and are not based on the local time.
Meta-data Backups
Meta-data will be backed up every day at 23:00.
The following volumes will be used for the backup
:VPD83T3:60060160c9c02XXXXXXXXXXXX,VPD83T3:60060160c9c02c005XXXXXXXXXXXX
Would you like to run the setup process now? [yes]: <<
use the default selection, just press the Enter/Return key
Scheduling the backup of metadata...
Performing metadata backup (This will take a few minutes)
Successfully performed the initial backing up of metadata
Successfully scheduled the backing up of metadata
Successfully scheduled the metadata backup
Product: VPLEX VS2, VPLEX Series, VPLEX Metro, VPLEX Local, VPLEX Geo,VPLEX VS6,VPLEX GeoSynchrony 6.0 Service Pack 1 Patch 7,VPLEX
GeoSynchrony 6.0 Service Pack 1 Patch 5,VPLEX GeoSynchrony 6.0 Service Pack 1 Patch 4,VPLEX GeoSynchrony 6.0 Service Pack 1 Patch
2,VPLEX GeoSynchrony 6.0 Service Pack 1 Patch 1,VPLEX GeoSynchrony 6.0 Service Pack 1,VPLEX GeoSynchrony 6.0 Patch 2,VPLEX
GeoSynchrony 6.0 Patch 1,VPLEX GeoSynchrony 6.1,VPLEX GeoSynchrony 5.5 Patch 1,VPLEX GeoSynchrony 5.5 Service Pack 1,VPLEX
GeoSynchrony 5.5 Service Pack 1 Patch 1,VPLEX GeoSynchrony 5.5 Service Pack 1 Patch 2,VPLEX GeoSynchrony 5.5 Service Pack 2,VPLEX
GeoSynchrony 5.5 Service Pack 2 Patch 1,VPLEX GeoSynchrony 5.5 Service Pack 2 Patch 2,VPLEX GeoSynchrony 5.5 Service Pack 2 Patch
3,VPLEX GeoSynchrony 5.5 Service Pack 2 Patch 4,VPLEX GeoSynchrony 5.5 Service Pack 2 Patch 5,VPLEX GeoSynchrony 5.4 Service Pack
1,VPLEX GeoSynchrony 5.4 Service Pack 1 Patch 1,VPLEX GeoSynchrony 5.4 Service Pack 1 Patch 3,VPLEX GeoSynchrony 5.4 Service Pack 1
Patch 4,VPLEX GeoSynchrony 5.4 Service Pack 1 Patch 5,VPLEX GeoSynchrony 6.0 Service Pack 1 Patch 6,VPLEX GeoSynchrony 6.1 Patch 1