Validation Overview
Summary
In medium to large data processing environments, file corruptions may occur from time to time on Large Disk Storage Volumes. These corruptions can be due to hardware failures, software application(s), File System problems, connectivity problems, insufficient resources, unpredicted operation / method / behavior, etc.
The Autonomic Healing Module was implemented to help Service Providers to automatically detect and fix (if possible) such problems. However, even with all the advanced validations implemented in the Autonomic Healing process, some scenarios cannot be predicted. For example, DS-System may not be able to detect data corruptions that lead to "Digital Signature does not match" errors. These are situations when the DS-Client is successful at decrypting and decompressing all individual data blocks, but the final result is not the one that was backed up.
These types of corruptions can only be detected by performing an actual restore, however, this is not convenient for customers. The "Validation" process is designed to help customers to verify restorability of online data, to detect corruptions, and to automatically remove any damaged files (and files that depend on them).
Without the customer / DS-Client encryption keys, DS-System is not able to create the digital signature to validate backed up data for some types of corruption. In order to avoid transmitting large amounts of data from DS-System to DS-Client for validation, DS-Client will send the encryption keys to DS-System based on the customer's ACCEPTANCE / ACKNOWLEDGEMENT / AGREEMENT to this option.
The decision to perform validation must be made by the customer on the DS-Client side. The customer knows better than anyone else what data is more important (i.e. data that needs to be validated more often). When a Validation process is triggered on DS-Client side, DS-Client sends a request to DS-System to perform the validation.
- For each file (generation), DS-System will first check the file header and delta linking/library linking, etc.
- If everything is fine, it will try to validate the data by performing a procedure like a "virtual restore". The data will be decrypted and decompressed to generate the original signature.
- If it fails due to a decryption or decompression problem, the validation fails.
- Finally, it will compare the generated signature with the original one: if it does not match, the validation fails; otherwise the validation is successful.
- For any validation failure, the error will be reported on both DS-Client and DS-System. If the validation fails due to reading or network problems, it is unknown whether or not the file is valid. Validation will skip the file and report the corresponding errors on both DS-System and DS-Client event logs.
- For other failures where DS-System can confirm the file is corrupted, the file will be moved to the trash directory. All files that depend on a corrupted file will also be moved to the trash.
Since the validation process will read all data to check the digital signature, the validation process is Disk I/O intensive on DS-System side.
The whole process is almost the same as an actual restore, except that INSTEAD OF writing to a target location, the decrypted/decompressed data is not kept after generating the signature.
A variety of options are provided to help the customer validate online data in the most flexible and efficient way. These options include: validation of all data, selective validation, scheduled / on-demand validation, excluding data deleted from source, and resuming mechanism, etc.
Validation functionality description
1. The encryption keys must be input by the customer (from DS-User) and DS-Client verifies them. The dialog that asks for encryption key(s) will display a warning with a description of how the encryption key(s) are involved in the Validation process.
- The encryption keys are encrypted when being transmitted from DS-User to DS-Client, and from DS-Client to DS-System.
- Encryption keys are only used to perform the data Validation: DS-System will not keep the keys after the Validation has finished.
2. The Validation process can be triggered on demand, as well as scheduled.
3. The demand Validation has two options:
- Validate all online data (all generations of all files).
- Selective - pop up a selection tree similar to the restore tree to allow selection of directories/files as well as generations to validate.
4. The execution sequence for a backup set's scheduled activities is: 1. Perform Backup, 2. Enforce Retention, 3. Perform Validation, 4. Perform BLM.
5. When setting a schedule to perform Validation, the customer must input the encryption key(s) and DS-Client verifies them. The Validation activity can only be enabled in a schedule if the user provides the correct encryption key(s).
6. The scheduled Validation has options to:
- Validate all online data (all generations of all files) or last generation only (snap shot)
- Include or exclude files already deleted from source.
7. The scheduled Validation also has the option to resume from the point of interruption of the previous scheduled Validation process.
- If a scheduled Validation completely processes all selected files, the next scheduled Validation (for the same backup set) will process files from the beginning.
- To determine the interruption point, the files are processed in sequence based on the order of share ID, directory ID, file ID and generation.
8. Validation is not available for statistical backup sets, since there is no data on DS-System to validate.
9. If a file corruption is detected (including "Digital Signature does not match" errors), that file and all files that actually depend on it (occasionally, a delta may formally depend on a file, but does not actually depend on it) will be moved to the trash directory. The corresponding error will be reported on both DS-System and DS-Client sides.
- If the DS-System is part of a replication group, the Validation Process will try to retrieve the file from one of the other Replication DS-Systems. If the file is successfully recovered, then no action is required from the DS-Client.
- If the file cannot be successfully retrieved from a Replication DS-System, then the backup set is marked "out-of-sync" and DS-Client will have to synchronize it and resend the file (if it still exists on the source machine being backed up).
10. If a file's restorability status cannot be determined temporarily (networking problems, etc.), DS-System will skip the validation for this file. The corresponding error will be reported on both DS-System and DS-Client sides.
11. If a file originally did not have a signature that is needed for Validation, a warning will be reported.
12. If any bad files are removed, DS-System will mark the backup set as "out of sync" at the end of the Validation process.
13. Only users with the Administrator Role on DS-Client are allowed to start a Validation process.
Comparison between the Validation Process Features (Core Component) and the Autonomic Healing Tool Features
- Note: Autonomic Healing is a tool that is suggested to run on DS-System all the time to find out and fix corruptions / inconsistencies as soon as possible. Some corruptions / inconsistencies may result in files not being restorable, and the backup set cannot be synchronized if the corruptions / inconsistencies are not fixed. Autonomic Healing is the most powerful tool available to automatically fix such corruptions / inconsistencies.
The Validation Process is meant to verify there are no corruptions / inconsistencies that result in files not being restorable. It has some fixing capability, but it is not primarily designed to do fixing (which is what Autonomic Healing does). Whenever a customer wants to confirm the online storage is restorable, they can run the Validation Process. If the Validation Process finds no problem, they can be sure the online storage is restorable. Running Autonomic Healing constantly can also maximize confidence in the online storage restorability.
See Also
The information provided in this document is provided "AS IS", without warranty of any kind. ASIGRA Inc. (ASIGRA) disclaims all warranties, either express or implied. In no event shall ASIGRA or its business partners be liable for any damages whatsoever, including direct, indirect, incidental, consequential, loss of business profits or special damages, even if ASIGRA or its business partners have been advised of the possibility of such damages. © Asigra Inc. All Rights Reserved. Confidential.
![]() ![]() |