Validation Overview


Creation Date: August 24, 2006
Revision Date: September 20, 2012
Product: DS-System

Summary

In medium to large data processing environments, file corruptions may occur from time to time on Large Disk Storage Volumes. These corruptions can be due to hardware failures, software application(s), File System problems, connectivity problems, insufficient resources, unpredicted operation / method / behavior, etc.

The Autonomic Healing Module was implemented to help Service Providers to automatically detect and fix (if possible) such problems. However, even with all the advanced validations implemented in the Autonomic Healing process, some scenarios cannot be predicted. For example, DS-System may not be able to detect data corruptions that lead to "Digital Signature does not match" errors. These are situations when the DS-Client is successful at decrypting and decompressing all individual data blocks, but the final result is not the one that was backed up.

These types of corruptions can only be detected by performing an actual restore, however, this is not convenient for customers. The "Validation" process is designed to help customers to verify restorability of online data, to detect corruptions, and to automatically remove any damaged files (and files that depend on them).

Without the customer / DS-Client encryption keys, DS-System is not able to create the digital signature to validate backed up data for some types of corruption. In order to avoid transmitting large amounts of data from DS-System to DS-Client for validation, DS-Client will send the encryption keys to DS-System based on the customer's ACCEPTANCE / ACKNOWLEDGEMENT / AGREEMENT to this option.

The decision to perform validation must be made by the customer on the DS-Client side. The customer knows better than anyone else what data is more important (i.e. data that needs to be validated more often). When a Validation process is triggered on DS-Client side, DS-Client sends a request to DS-System to perform the validation.

Since the validation process will read all data to check the digital signature, the validation process is Disk I/O intensive on DS-System side.

The whole process is almost the same as an actual restore, except that INSTEAD OF writing to a target location, the decrypted/decompressed data is not kept after generating the signature.

A variety of options are provided to help the customer validate online data in the most flexible and efficient way. These options include: validation of all data, selective validation, scheduled / on-demand validation, excluding data deleted from source, and resuming mechanism, etc.

Validation functionality description

1. The encryption keys must be input by the customer (from DS-User) and DS-Client verifies them. The dialog that asks for encryption key(s) will display a warning with a description of how the encryption key(s) are involved in the Validation process.

2. The Validation process can be triggered on demand, as well as scheduled.

3. The demand Validation has two options:

4. The execution sequence for a backup set's scheduled activities is: 1. Perform Backup, 2. Enforce Retention, 3. Perform Validation, 4. Perform BLM.

5. When setting a schedule to perform Validation, the customer must input the encryption key(s) and DS-Client verifies them. The Validation activity can only be enabled in a schedule if the user provides the correct encryption key(s).

6. The scheduled Validation has options to:

7. The scheduled Validation also has the option to resume from the point of interruption of the previous scheduled Validation process.

8. Validation is not available for statistical backup sets, since there is no data on DS-System to validate.

9. If a file corruption is detected (including "Digital Signature does not match" errors), that file and all files that actually depend on it (occasionally, a delta may formally depend on a file, but does not actually depend on it) will be moved to the trash directory. The corresponding error will be reported on both DS-System and DS-Client sides.

10. If a file's restorability status cannot be determined temporarily (networking problems, etc.), DS-System will skip the validation for this file. The corresponding error will be reported on both DS-System and DS-Client sides.

11. If a file originally did not have a signature that is needed for Validation, a warning will be reported.

12. If any bad files are removed, DS-System will mark the backup set as "out of sync" at the end of the Validation process.

13. Only users with the Administrator Role on DS-Client are allowed to start a Validation process.

Comparison between the Validation Process Features (Core Component) and the Autonomic Healing Tool Features

Major Functionality
Description
Validation Process (Core)
Autonomic Healing (Tool)
Triggering
1
Triggered from DS-Client or DS-System
DS-Client
DS-System
2
Who makes decision when and how to run
Customer
Service Provider
3
Automatic triggering method
Schedule
Configure
4
Can continuously run
No
Yes
Running options
1
On-demand start/stop
Yes
Yes
2
Speed adjustable based on DS-System Load
No
Yes
3
Can automatically resume on file level
Yes
Yes
4
Customized selection (on dir/file/generation level)
Yes
No
Checking Capabilities
1
Check File Headers for damage/inconsistencies
Yes
Yes
2
Check Directory Stream Headers for damage/inconsistencies
No
Yes
3
Check library link for damage/inconsistencies
Yes
Yes
4
Check Delta file for damage/inconsistencies
Yes
Yes
5
Check file name for damage/inconsistencies across generations
No
Yes
6
Check Directory ID/name for damage/inconsistencies
No
Yes
7
Check File ID/name for damage/inconsistencies
No
Yes
8
Check for orphaned recycled generations caused by data damage/corruption
No
Yes
9
Check for session damage/inconsistencies across generations
No
Yes
10
Check for corruptions causing decryption problems
Yes
No
11
Check for corruptions causing decompression problems
Yes
No
12
Check for corruptions causing digital signature does not match
Yes
No
Fixing Capabilities
1
Delete corrupted files (move to trash folder)
Yes
Yes
2
If DS-System is part of a Replication Group, attempt to retrieve the deleted files from a Replication DS-System. If all deleted files are successfully retreived in this manner, skip step 3 (below).
Yes
Yes
3
Mark backup set as out-of-sync after deletion or corrupted files are fixed
Yes
Yes
4
Fix files/directories ID damage/inconsistencies
No
Yes
5
Fix directory location damage/inconsistencies
No
Yes
6
Fix file name damage/inconsistencies within directories
No
Yes
7
Fix file name damage/inconsistencies across generations
No
Yes
8
Fix delta linking/reconstruction damage/inconsistencies
Partially
Yes
9
Fix library link damage/inconsistencies
No
Yes
10
Remove orphaned recycled generations caused by data damage/inconsistencies
No
Yes
11
Clean recycled generations to optimize storage space.
No
Yes
12
Fix session damage/inconsistencies across generations
No
Yes
Reporting
1
Report damage/inconsistencies in event log
Yes
Yes
2
Report damage/inconsistencies in DS-Client event log
Yes
No

Note: Autonomic Healing is a tool that is suggested to run on DS-System all the time to find out and fix corruptions / inconsistencies as soon as possible. Some corruptions / inconsistencies may result in files not being restorable, and the backup set cannot be synchronized if the corruptions / inconsistencies are not fixed. Autonomic Healing is the most powerful tool available to automatically fix such corruptions / inconsistencies.

The Validation Process is meant to verify there are no corruptions / inconsistencies that result in files not being restorable. It has some fixing capability, but it is not primarily designed to do fixing (which is what Autonomic Healing does). Whenever a customer wants to confirm the online storage is restorable, they can run the Validation Process. If the Validation Process finds no problem, they can be sure the online storage is restorable. Running Autonomic Healing constantly can also maximize confidence in the online storage restorability.

See Also



The information provided in this document is provided "AS IS", without warranty of any kind. ASIGRA Inc. (ASIGRA) disclaims all warranties, either express or implied. In no event shall ASIGRA or its business partners be liable for any damages whatsoever, including direct, indirect, incidental, consequential, loss of business profits or special damages, even if ASIGRA or its business partners have been advised of the possibility of such damages. © Asigra Inc. All Rights Reserved. Confidential.


PREVNEXT