Clone wiki

saturnalia / DB_Design

Database Overview

The Saturnalia database is optimized for aggregating, storing, and displaying a large number of scalar samples for a large number of different metrics (ie: sources of periodic measurement).

The database (like other components) is written with a Crash-Only Design. The design includes optimizations for storage size and read/write latency from disk.

FIXME: Make this overview more comprehensive.


Here is a quick overview of some existing optimizations:

Minimized Disk Seeks

FIXME: Define chunks and tiles.

Chunk index:

List of chunks and there time ranges. Chunks are 100M in size so on a 1T disk we will only have 10,000 chuncks before the disk is full. If each chunk index record is a 100 byte file name + two int64s 116 bytes per entry or 1.6M max for the hole thing on a full 1T disk.

Chunks, finding tiles:

  1. Hash of metrics to tile index. Assume two seeks to find the right entry in the has table.
  2. entry index is a array of time with one entry for each tile in the time range covered by this chunk. See to find tiles is one.
  3. One more seek to get the tiles.

In this way we have four seeks to get to a tile.