Storage Scalability (Extensible Storage)
Summary
The "Extensible Data Storage" feature allows a DS-System (standalone or N+1) to make use of multiple storage paths at the same time for storing DS-Client data.
"Extensible Data Storage" allows DS-System to grow its capacity by simply adding a new storage location to the list of available locations. Adding new storage can be done easily and without the need of shutting down the DS-System software.
Extensible data storage
Standard "non-extensible" configuration and drawbacks
A typical hard-drive storage solution features one location where the data is stored. Data is usually added to the storage until the storage space is filled, after which manual intervention is required.
Having one single data storage location has several drawbacks, including the following:
- Once the storage space is nearly full, manual intervention is required and gaining more storage space is not easy: additional space must be added, or mount points must be created, or data must be moved/deleted.
- File systems may have size limitations, either due to internal format (in which case this is an absolute limit) or because of speed/reliability concerns. This means that once a storage location has reached its storage limit, new file systems have to be created and in general it is difficult (and hard manual work) to merge the two file systems by mounting one file system as a sub-path of the other.
- File systems also have limitations on how many files they can hold. Although the number of files is limited only by available storage space on many file systems, there is an approximate limit to the number of file/directory entries a file system can hold because of performance and reliability. (It is recommended that NTFS file systems never exceed 2TB or 10 Million files).
DS-System Solution: Extensible Data Storage
- The "Extensible Data Storage" technology allows the DS-System to dynamically add additional storage space without extensive intervention by the user. The user only needs to define file system parameters (maximum storage space and maximum file/directory entry count) and the DS-System will take care to balance the storage across the available storage.
- If the storage location(s) ever get full (or almost full), new storage locations can be added "on the fly" and the DS-System will make use of them immediately. As soon as a new storage location is plugged into the Extensible Storage, DS-System will create metadata links on the existing ones and save the incoming data to the new location.
How it works
Configuration requirements
The DS-System is configured with a list of available storage locations. You (the Service Provider) define the maximum amount of data that can be stored at each location (file size and file count):
- Set from: DS-Operator > Setup Menu > Storage > Extensible Storage
The DS-System automatically tracks how much data is in each storage location (without user intervention). Based on this information, it will try to balance the new data that is backed up across the different storage locations. The user only needs to define the fill levels (a one-time configuration that applies to all storage locations):
- Set from: DS-Operator > Setup Menu > Storage > Extensible Storage > Update Storage Fill Levels
- Note: The fill levels apply to the Maximum Size you defined for each storage location.
Spreading data across storage locations
The DS-System will use the fill percentages in order to balance the data load across the different available storage locations. If a storage location becomes full (its fill level is higher than some other storage location), all new data will be stored in the storage location(s) that have more space.:
Adding additional Storage locations
If all the storage locations become full (or almost full), more file systems can be created and simply added to the list of available storage locations. The DS-System will take care of the rest. For example, if you have a DS-System that is using four storage locations that have passed an 80% fill level, you can add two (2) new storage locations. Most new data will go to those locations, until the storage in all six locations passes the same fill level (i.e. 80% full).
- Note: When the available storage space becomes low, the DS-System can be configured to send E-Mail and/or SNMP notifications.
Benefits of Extensible Storage
Easy management of storage space
- Disk space usage (both file size and file count) is continuously monitored by the DS-System.
- While storage is available, no human intervention is needed. The DS-System automatically balances its distribution of data across available storage locations.
- The DS-System properly distributes storage load across available storage locations: it takes into account both file system size and maximum number of file entries supported by the storage location.
- The entire solution is fully dynamic: as data is added and removed, the DS-System keeps the online storage balanced. If data is deleted from one storage location (and its storage fill level drops), new data is added to this location.
Easy to increase DS-System storage capacity
- Very little human intervention is needed once storage space becomes low: you only need to add the new storage hardware and instruct the DS-System where the new storage space is located (using the GUI to dynamically add the new storage location without stopping the DS-System).
Storage location disaster recovery support
- The DS-System supports "recovery mode" operation, which allows for partial restore of the DS-System online storage from tape and/or other backup media. While in recovery mode, the DS-System will check its online storage and ensure that all its data is made consistent (even though restored data may be partial/inconsistent).
- When using extensible storage, each storage location can be protected individually. In case of disaster, only the affected storage location(s) need to be recovered from backup media; this reduces the amount of data loss and the backup/restore activities can be resumed faster.
Installation / upgrade notes
- The storage locations can be configured as local drives or UNC paths.
- The operator only needs to select the primary storage location during DS-System installation. Once the installation process is completed and DS-System is started, new storage locations can be added using DS-Operator. DS-System will automatically create the new directory structure; no intervention is required from DS-Operator.
- IMPORTANT NOTE: After DS-System service/daemon starts, the primary storage location ("BackupRoot" parameter from dssys.cfg) should not be changed. The primary storage location is the starting point for all the extensible storage locations. It will contain the /dump and other essential files and folders.
Storage Location Disaster Recovery
Configure DS-System in Disaster Recovery mode
- On a DS-System configured with multiple storage locations, in case of a hardware failure, only the affected storage location(s) need to be restored. However, because of the interdependencies and the links between locations, DS-System must be configured to run in Disaster Recovery mode. In order to do this, you must configure the "DisasterRecoveryMode" value from DS-Operator > Setup > Advanced Configuration. Then you must restart DS-System service.
- When DS-System is started in Disaster Recovery mode, NO activity is allowed until scanning and recovery of all backup set directories from all storage locations is finished. After this step, DS-System must recover the library directories and every single backup set. No other process is allowed to use a backup set that has not been completely recovered. However, DS-System allows new backup sets to be backed up as soon as all libraries are recovered/validated.
- DS-System starts a default of 4 parallel disaster recovery processes. This number can be changed by editing the "RecoveryProcess" value from DS-Operator > Setup > Advanced Configuration and restarting the DS-System service. However, additional processes may be started by DS-System based synchronizations started by DS-Clients.
- Once the Disaster Recovery process finishes processing a set, DS-Client must perform a DS-System based synchronization. Weekly Admin or scheduled backups initiate this synchronization automatically.
- In the case of a crash or hardware replacement, if the storage location is the primary one - holding the online storage root - you must configure the new file system with the same absolute path (name/IP address, share, folder) and as the original one. Preserving the exact absolute path is recommended even when recovering a secondary location, although it is not compulsory. If the replacement storage location has a different path than the original one, all links from the other locations (pointing to the replaced one) must be recreated. However, if you configure it exactly as the original one, the links may still be valid (providing that the files/directories that they are pointing to are still valid after a disaster recovery).
- After a storage location disaster recovery, DS-System should scan it completely in order to update the corresponding statistics.
See Also
- "Retire Storage Locations"
- "Storage Balancing (Extensible Storage)"
- "Retiring the DS-System Primary Storage Location (Backup Root)"
The information provided in this document is provided "AS IS", without warranty of any kind. ASIGRA Inc. (ASIGRA) disclaims all warranties, either express or implied. In no event shall ASIGRA or its business partners be liable for any damages whatsoever, including direct, indirect, incidental, consequential, loss of business profits or special damages, even if ASIGRA or its business partners have been advised of the possibility of such damages. © Asigra Inc. All Rights Reserved. Confidential.
![]() ![]() |