FileStore: Consolidated small regular files
Summary
The FileStore feature enhances the performance of the DS-System by improving the access to the Online Storage and by saving storage space on the Online Storage. Small, regular (non-Master / Delta) files in the same Online Storage directory are consolidated to one large file called "FileStore". This can achieve significant speed improvements for processes involving backup sets with large quantities of small files in the same directory.
This feature is configurable from Setup Menu > Advanced Configuration (see "DS-System Advanced Configuration").
Details
For File System backup sets and E-Mail backup sets, more than 70% of the contents are small files of less than 1KB. The operations required to locate / open / close these small files are expensive in terms of DS-System I/O. Therefore, whenever a backup item contains many small files, it poses a major performance problem due to the high amount of activities that will have to run simultaneously on the disk volume.
The consolidated small files feature works at the directory level on the DS-System online storage. It works on regular files that are smaller than 32KB (by default). Files that are larger than 32KB are usually backed up as Master/Delta and are not consolidated. However, this also depends on the minimum delta size settings (which is 32KB by default, configurable on the DS-Client > Setup Menu > Configuration: Advanced Tab - "MinDeltaSize" value).
Consolidation occurs per directory. It is performed by both the backup process and the System Admin process.
- Backup process: All new small files will automatically be saved to the FileStore.
- System Admin process: Existing small files will be consolidated as their directories are processed by System Admin (if they meet the 'SmallFileCount' threshold).
FileStore Cache
Caching is set up from Setup Menu > Advanced Configuration: FileStoreCache | MLRCache. It keeps the specified maximum number of FileStores open in DS-System to allow fast access with low I/O overhead.
The defaults should be sufficient. Caching will have the most impact on processes where DS-System performs high volume, multi-threaded I/O to DS-System directories (e.g. backup, Disc/Tape write-to-buffer, Autonomic Healing, BLM Archive, and Replication).
Vacuum FileStore
Over time, some small files in a FileStore will become obsolete (e.g. from generation overwriting, delete processes, etc.). Because of the FileStore structure, these files are only marked for delete. These files are removed when the FileStore is 'vacuumed': a process where the whole FileStore is re-written (without files flagged for delete) to the DS-System Online Storage.
A FileStore is vacuumed during a System Admin, depending on the DS-System configuration. This is set up from Setup Menu > Advanced Configuration: FileStoreVacuum. If the percentage of free space that can be reclaimed is at least this amount, System Admin will vacuum the corresponding FileStore.
- Note: Files marked for delete in a FileStore are not counted towards billing or other statistics.
Best Practices (Consolidated Small Files)
The improvements are more noticeable if more small files are involved in the directories that are protected.
- The most noticeable improvement will be for E-Mail backup sets (DS-Message-Level Restore) that have many small files in a few "dedicated" directories. FileStore will dramatically improve the performance for almost all activities (Autonomic Healing, System Admin, validation, delete, retention, synchronization, BLM archiving, replication, backup, restore, etc.).
- Files smaller than the block size of the storage sub-system that are saved individually on the storage are wasting additional storage space (e.g. a 1KB file size may waste 3KB of storage if the sub-system's block size is 4KB).
See Also
The information provided in this document is provided "AS IS", without warranty of any kind. ASIGRA Inc. (ASIGRA) disclaims all warranties, either express or implied. In no event shall ASIGRA or its business partners be liable for any damages whatsoever, including direct, indirect, incidental, consequential, loss of business profits or special damages, even if ASIGRA or its business partners have been advised of the possibility of such damages. © Asigra Inc. All Rights Reserved. Confidential.
![]() ![]() |