FileStore: Consolidated small regular files


Creation Date: July 29, 2008
Revision Date: October 12, 2011
Product: DS-System

Summary

The FileStore feature enhances the performance of the DS-System by improving the access to the Online Storage and by saving storage space on the Online Storage. Small, regular (non-Master / Delta) files in the same Online Storage directory are consolidated to one large file called "FileStore". This can achieve significant speed improvements for processes involving backup sets with large quantities of small files in the same directory.

This feature is configurable from Setup Menu > Advanced Configuration (see "DS-System Advanced Configuration").

Details

For File System backup sets and E-Mail backup sets, more than 70% of the contents are small files of less than 1KB. The operations required to locate / open / close these small files are expensive in terms of DS-System I/O. Therefore, whenever a backup item contains many small files, it poses a major performance problem due to the high amount of activities that will have to run simultaneously on the disk volume.

The consolidated small files feature works at the directory level on the DS-System online storage. It works on regular files that are smaller than 32KB (by default). Files that are larger than 32KB are usually backed up as Master/Delta and are not consolidated. However, this also depends on the minimum delta size settings (which is 32KB by default, configurable on the DS-Client > Setup Menu > Configuration: Advanced Tab - "MinDeltaSize" value).

Consolidation occurs per directory. It is performed by both the backup process and the System Admin process.

FileStore Cache

Caching is set up from Setup Menu > Advanced Configuration: FileStoreCache | MLRCache. It keeps the specified maximum number of FileStores open in DS-System to allow fast access with low I/O overhead.

The defaults should be sufficient. Caching will have the most impact on processes where DS-System performs high volume, multi-threaded I/O to DS-System directories (e.g. backup, Disc/Tape write-to-buffer, Autonomic Healing, BLM Archive, and Replication).

Vacuum FileStore

Over time, some small files in a FileStore will become obsolete (e.g. from generation overwriting, delete processes, etc.). Because of the FileStore structure, these files are only marked for delete. These files are removed when the FileStore is 'vacuumed': a process where the whole FileStore is re-written (without files flagged for delete) to the DS-System Online Storage.

A FileStore is vacuumed during a System Admin, depending on the DS-System configuration. This is set up from Setup Menu > Advanced Configuration: FileStoreVacuum. If the percentage of free space that can be reclaimed is at least this amount, System Admin will vacuum the corresponding FileStore.

Note: Files marked for delete in a FileStore are not counted towards billing or other statistics.

Best Practices (Consolidated Small Files)

The improvements are more noticeable if more small files are involved in the directories that are protected.

See Also



The information provided in this document is provided "AS IS", without warranty of any kind. ASIGRA Inc. (ASIGRA) disclaims all warranties, either express or implied. In no event shall ASIGRA or its business partners be liable for any damages whatsoever, including direct, indirect, incidental, consequential, loss of business profits or special damages, even if ASIGRA or its business partners have been advised of the possibility of such damages. © Asigra Inc. All Rights Reserved. Confidential.


PREVNEXT