Sizing Considerations: DS-System & BLM Archiver storage


Creation Date: February 05, 2009
Revision Date: October 12, 2011
Product: DS-System & BLM Archiver
NOTE: The views expressed in this document are general considerations based on past experience. They do not address any specific hardware vendors.

Summary

This document describes the advantages and disadvantages of selecting a single online storage location versus multiple online storage locations for use with the DS-System & BLM Archiver software.

Single vs. Multiple Storage Locations during normal DS-System Operations

The DS-System experiences an extra overhead when managing multiple file systems because it must locate the corresponding file system where any requested item is stored. DS-System handles multiple storage locations with the help of link files, which need to be opened, read and closed. In addition, each DS-System operation must be redirected to the target storage location specified by the link files.

In an ideal scenario, for storage that provides the same performance/protection with the same number of disks, it is recommended that you configure one single large file system for the DS-System Online Storage (versus multiple storage locations).

Experience has shown that most of the time, a single large file system has its own limitations when it comes to handling hundreds of millions of files consisting of many terabytes of data. Statistics show this occurs at an average of between 10-15 million files per terabyte of data.

Our customers have experienced very poor performance when a single file system stores over 400 million files. This is caused by the majority of the NAS systems being optimized for multiple clients (multiple processes handle parts of the requests at the same time).

DS-System installation (either for a Standalone or an N+1 configuration) provides a small number of clients making requests to the NAS device. NAS devices may not be able to utilize their full hardware resources to handle DS-System requests since the number of NAS clients is low and the NAS device may not be optimized for such a configuration. For such situations, a DS-System or BLM Archiver configured with multiple storage locations may be able to better utilize the NAS resources.

Single vs. Multiple Storage Locations during Disaster Recovery

Another important consideration is the Disaster Recovery of the DS-System Online Storage. The DS-System Online Storage must be separately backed up in case a disaster occurs to the storage subsystem. Using DS-System Replication is strongly recommended.

Historically, Service Providers have backed up the DS-System Online Storage to tape. Some customers are still using this approach, but they usually have difficulties when backing up one large file system as opposed to multiple smaller ones. Although better and faster options to protect data exist today (e.g. snapshots, replication, etc.), this solution may work faster when processing data on multiple smaller file systems in parallel than when processing the data on one single large file system.

If you experience hardware failures or problems when working with one single file system for the DS-System Online Storage, the entire file system may be compromised. Using multiple locations may contain the damage and recovery time to a smaller part of the data.

NAS Disks Considerations

When selecting a NAS solution consider the following:

Increasing the disk size in a NAS architecture usually decreases the performance of the storage. For example, the performance can almost be doubled when using two 500 GB disks vs. one 1 TB disk (assuming that the disk speed is the same). Hardware vendors usually publish the performance of their best configuration, which means as many disks as possible, as fast as possible.

When selecting a NAS configuration, you can balance the price/performance by deciding if you want to maximize the total size of the storage (for better price) or the number of disks (for better performance).

Conclusion: Single vs. Multiple Storage Locations

The DS-System consumes extra overhead when using multiple storage locations, while one large storage location may provide decreased performance and extended backup/recovery time in case of disasters.

Given the above considerations, the hardware vendor is the party who must recommend whether to use a single storage location (one large file system) vs. multiple storage locations (multiple smaller file systems). Parameters to consider are:

Selecting the Size of the Online Storage when using Multiple Storage Locations

When configuring the DS-System & BLM Archiver software to work with multiple storage locations, you must consider the size of the storage locations. The size is highly dependent on the items described in the other sections of this article.

As a rule of thumb, each storage location size can be approximately 4 TB. A larger storage location size can be configured as long as the I/O reliability and the expected Disaster Recovery results can be achieved.

The largest file size expected to be stored on the Online Storage at any one time can provide an indication of the best size for the file system (these values must be confirmed with your hardware vendor). For example:

If the file system configuration can be dynamically modified, without data loss, DS-System & BLM Archiver can begin by using one single location. As soon as you experience a decrease in the performance of the storage subsystem, or if the storage location cannot be backed up in the expected time window, you can add an additional storage location (as opposed to growing the existing one).

Storage Testing Tools

The "ASIGRA Storage I/O and Data Validation Tool" is available for customers to test and compare multiple NAS vendors, their solution, reliability and performance before you purchase a storage solution.

This tool simulates DS-System activities when interacting with the storage. It can perform writes, reads, data validation and can measure the storage performance. This tool is available on the installation DVD or online, on the ASIGRA Technical Support web site (Service Packs section).

Considerations for v.9.0 Release (and higher)

The number of files per Terabyte estimations provided in this document are based on v.8.0 (or older) of the DS-System software. Starting with release v.9.0, the consideration of the number of files per terabyte may decrease significantly, since DS-System will consolidate small files in the same parent directory to one large file for each separate directory on the storage.



The information provided in this document is provided "AS IS", without warranty of any kind. ASIGRA Inc. (ASIGRA) disclaims all warranties, either express or implied. In no event shall ASIGRA or its business partners be liable for any damages whatsoever, including direct, indirect, incidental, consequential, loss of business profits or special damages, even if ASIGRA or its business partners have been advised of the possibility of such damages. © Asigra Inc. All Rights Reserved. Confidential.


PREVNEXT