SMILA/Project Concepts/Binary Storage

From Eclipsepedia

Jump to: navigation, search



Design a service to easy store / access binary data documents.




Storage Mechanism Internal Structure.


  • A. File System
    • I. Local hard drive
      • 1. Flat structure
      • 2. Hierarchical structure (configurable tree)
      • 3. Hierarchical structure (deterministically calculation)
      • Analyze of hierarchical structure
    • II. Distributed file system (SFTP, FTP)
      • 1. Flat structure
      • 2. Hierarchical structure (configurable tree)
      • 3. Hierarchical structure (deterministically calculation)
  • B. RDBMS
  • C. Object DataBase (ODBMS)



A. File System


I. Local hard drive

The service saves data in the local drive using a predefined persistence storage location - Under this root path Binary Storage will create its files system structures flat or hierarchical, depending on the configuration.

1. Flat structure





2. Hierarchical structure (configurable tree)


  • q - Maximum number of subfolders per folder
  • r - Maximum number of persisted attachments per folder


Hierarchical structure nomenclature. Test scenario





  • hierarchy level - n=4 (logarithmical function)





t (number of attachments) q (max number of subfolders in folder) r (max number of folder) TF (number of folders) n (hierarchical level) Sn (full mode hierarchy) M (missing subfolders in deepest level)
360 3 10 36 4 40 4
1000000 25 200 5000 4 16276 11276
1000000 50 200 5000 4 127551 122551
1000000 75 300 3333 3 5701 2368
1000000 100 300 3333 3 10101 6768
1000000 150 300 3333 3 22651 19318
1000000 200 350 2857 3 40201 37344
1000000 250 500 2000 3 62751 60751
1000000 300 750 1333 3 90301 88968
1000000 100 400 2500 3 10101 7601
1000000 200 500 2000 3 40201 38201
1000000 250 600 1666 3 62751 61085
1000000 300 750 1333 3 90301 88968
1000000 300 1000 1000 3 90301 89301

3. Hierarchical structure (deterministically calculation)

Another solution for building the hierarchical structure uses deterministically calculation based on the hash id, passed by the blackboard service to the Binary Storage Service. The solution comes to avoid storing data into a single path location and to offer a hierarchic structure. The structure will be located under the and it will be build by using parts from the passed hashed id.

i.e. for a passed binary data, having the ID = "1d41cd87e735ceb23fdfafd814366b3ca9816e3dfce6fecc30560f8c242de", the deterministically calculation will assign fallowing path, where the record will be stored:\1d\41\cd\87\1d41cd87e735ceb23fdfafd814366b3ca9816e3dfce6fecc30560f8c242de

When using the deterministically calculation all data will be stored at the same hierarchic level.

There are two parameters that could be configured in order to define the deterministically calculation:

  • hierarchy level (in our case n = 4)
  • sub-tree (folder) name length (l = 2, these are the substring-parts starting from the left side of the hashed id, used to build the structure like : "1d", "41", "cd" and "87")

Because of the starting part of hashed id of different records (which define the hierarchic structure - deterministically calculation) may be different enough, it is very possible that in many cases there will be only single record stored into a deterministically calculated location (since there will be no match, or at least not enough, related to the left part of the hashed IDs - which defines the location where the record is going to be persisted). As a matter of fact, it is possible that a hierarchic structure to contains many subfolders that contain only one subfolder(not deepest level) or single record(deepest level). This my be an issue since most of the file system APIs creates listeners for all these sub-folders in order to provide cache (refresh) functionality, so for single record there will be 5 listeners (4 sub-folders + 1 record) created and memory consumption increases significantly. For example, for storing 540 records, a total number of 977 subfolders will be created.

As long as the records number to be stored gets higher, the matching coincidence (the first string part of the hashed IDs that is used by the deterministically calculation) increases and the system becomes more efficient - from the number of records to number of sub-folders report.

Following image, represents a simple sample of hierarchical structure obtained with deterministically calculation. Since accessing (storing and fetching) of records is done based on the hashed ID, there is no necessary to save any mapping between the record and calculated path location.


The sample, also outlines hashed ID part matching on first hierarchic level which means common usage of some first subfloders level.

Analyze of hierarchical structure

Advantages Disadvantages
Configurable Tree
  • configurable structure; splits node only when required
  • less memory consumption
  • better structure control
  • mapping between hashed ID and persistence path needs to be stored
  • possibility of assync. situations between the existing storage persistence and stored mapping (in case of some external factors alters the persistence storage)
  • implementation effort
Deterministically Calculation

II. Distributed file system (SFTP, FTP)


1. Flat structure


2. Hierarchical structure (configurable tree)

It has the same characteristics like in the local hierarchic (configurable tree) configuration, but it is applied on the persistence location from the distributed system.

3. Hierarchical structure (deterministically calculation)

It has the same characteristics like in the local hierarchic (deterministically calculation) configuration, but it is applied on the persistence location from the distributed system.



For performance considerations, database schema shall contain multiple tables (since large BLOBs will be stored). Binary Storage Service shall deterministically distribute records to database tables.

C. Object DataBase


Open Source Database Engines :

  • Oracle Berkeley DB
  • Berkeley DB Java Edition

Cluster configuration

Binary Storage Service must offer fallowing cluster configurations:




Concurrent access


Technical aspects for designing the Binary Storage Service

  • Binary data compressing and encryption shall be available via configuration
  • Binary storage shall internally manage its persistence hierarchy.
  • Resources synchronization shall be done at the lowest possible level

2.Sequence Diagram NewBinaryStorae.jpg

  • The Binary Storage Service API shall stay as simple as possible
void store(String id, InputStream stream);
void store(String id, byte[] blob)
byte[] fetchAsByte(String id)
InputStream fetchAsStream(String id)
void remove(String id)
int fetchSize(String id)

UML Class Diagram