Files changed (11)
+ " [maxinitmem size] [maxtotalmem size] [minchainentries start-end] [minmaxblocksperhead start-end]"
+ totalscorewidth = int(0.9 + log2(totalblocks)) + int(0.9 + round(abs(log2(1.0/collisioninterval)), 1))
+ if minmaxblocksperhead != None and (blocksperhead < minmaxblocksperhead or blocksperhead > minmaxblocksperhead):
+is a small venti daemon that runs on unix. A venti daemon allows for storing and retrieving data (in blocks of up to 56 KB) with a numeric type and its SHA1 hash (called score) as address, all with a simple protocol on top of TCP. Once written, data cannot be removed. If data needs to be stored and has a score that is already present, the data is not written again. Authentication is not needed, anyone can read any block. The idea is that the address, the score, is just a shorter version of the "content": only if you know the content, you can determine the score and if you know the score, you know the content. A more advanced venti is used by Plan 9 for filesystem block storage and backup storage.
+by default. This file is opened append-only, therefore previously written data cannot be modified or removed by memventi. Each block has a block header prepended to it, the header contains the following information: score of the block and the type of the block (both part of the address) and the length of the data.
+by default, a part of the score, the type and the offset of the header of the corresponding data block in the data file, is stored. This file is loaded into main memory (and verified or fixed to match the data file) at startup and remains in memory throughout operation for fast score lookups. This contrasts with the Plan 9 venti which reads scores from a big hashtable on index disks (and with a cache) to do lookups of scores. The scheme used by
+cannot store as many data blocks, because main memory is limited. It is however, easier to setup and maintain.
+Main memory usage is reduced as much as possible. For each block, the numeric type is kept in main memory and only just enough of the score, which is 160 bits in total. For reading, a "hit" of the in-memory score-part often means the data block is found with one disk read (the header in the data file is checked to see whether the entire score matches). Sometimes, a hit was false and the block in the data file was different (thus, the score might not be present at all). At other times multiple "hits" are found when reading a score. In this case, the data blocks for the hits have to be read from data file until the full score is encountered.
+can be specified such that a double hit only occurs once every 1000 scores (when the venti is full), triple hits will occur much less often.
+For writes, if the in-memory index has one or more hits, the disk has to be checked. If the data is already present, it is not written again. Otherwise, it is simply appended to the data and index file and an entry put in memory.
+is the number of bits of the score used for the number of buckets in the lookup table. For example, 9 bits means there will be 512 buckets (heads) in the lookup table.
+is the number of bits to use for addressing in the datafile. Fewer bits results in less memory used, but also reduces the maximum memventi storage capacity. Appropriate values for these variables can be determined using the program
+It returns reasonable values when given a maximum data file size, average block size and collisioninterval (1000 means one of every 1000 scores may have a collision). More parameters may be specified to further narrow down the right values.
+Listen on the specified TCP port, on the specified host. The port-part (including the exclamation mark) is optional and defaults to 17034. The connection does not allow writes, only reads. This can be used to prevent public memventi's to be filled up.
+When a SIGUSR1 is received, a histogram of the lengths of the heads (the number of blocks in the heads) is printed to standard out. When a SIGUSR2 is received, a histogram of the number of disk accesses needed to fulfill operations is printed to standard out. The only other statistics are written to syslog at startup, they contain the number of bytes read, whether the index was synchronized and how long startup up took.
+Whenever a problem is encountered, such as an error when writing a block to a file, memventi puts itself in degraded mode. In this mode, only read operations are handled. When it goes in degraded mode, a message is written to syslog clearly explaining the problem.
+The memory used for the lookup table buckets and entries is mlock-ed so lookups are always fast. Some systems, notably OpenBSD/i386 do not allow non-root users to mlock memory.
+There are no tools to aid in data recovery, e.g. checking the data file for consistency and fixing problems. For now,
+Calc.py should be explained better. Some important things: start-end ranges can be specified as numbers with a prefix such as k, m, g, etc. Multiple ranges can be specified, each separated by a comma.
+disklookup(uvlong *addr, int naddr, uchar *score, uchar type, int readdata, uchar *data, DHeader *dh, char **errmsg)
+ syslog(LOG_WARNING, "disklookup: error reading header for block at offset=%llu, score=%s type=%d: %s",
+ syslog(LOG_WARNING, "disklookup: short read for header for block at offset=%llu, have=%d, score=%s type=%d",
+ syslog(LOG_WARNING, "disklookup: unpacking header for block at offset=%llu, score=%s type=%d: %s",
+ syslog(LOG_ALERT, "disklookup: datafile %s has wrong score (has %s, claims %s) in block at offset=%llu size=%d type=%d",
+ "indexstore: writing header to indexfile %s at offset=%llu for datafile block at offset=%llu: %s",
+ "indexstore: short write for header to indexfile %s at offset=%llu for datafile block at offset=%llu, "
+ syslog(LOG_ALERT, "store: short write for header, %d dangling bytes at end of datafile %s, block at offset=%llu, %s",
+ errxsyslog(1, "last header at offset=%llu in index point past end of datafile at block at offset=%llu",
+ errxsyslog(1, "error reading disk block at offset=%llu that indexfile at offset=%llu claims is the last: %s",
+ errxsyslog(1, "score in indexfile at offset=%llu does not match score in datafile at block at offset=%llu",
+ errxsyslog(1, "type in indexfile at offset=%llu does not match score in datafile at block at offset=%llu",
+ errxsyslog(1, "invalid score for block at offset=%llu in datafile, has %s, claims %s (for adding to index)",
+ errxsyslog(1, "could not store newly read datafile block at offset=%llu to indexfile at offset=%llu, %s",
+ syslog(LOG_NOTICE, "added %llu entries from datafile (%llu bytes in datafile) to indexfile, in %.3fs",
+ syslog(LOG_NOTICE, "init done, %llu bytes for heads, read %llu bytes in %.3fs from index, entire startup in %.3fs",
+ fprintf(stderr, "usage: memventi [-fvD] [-r host!port] [-w host!port] [-i indexfile] [-d datafile] headscorewidth entryscorewidth addrwidth\n");
+ snprintf(errmsg, sizeof errmsg, "invalid data, magic wrong (have 0x%x, want 0x%x)", magic, Headermagic);
+- tool to find last valid lump and possibly invalid remainder (for half write during crash/power outage)
+- to find duplicates: readdata | sed 's/.* score=\([^ ]*\).*/\1/' | sort | uniq -d (need to make readdata.c again)
+ snprintf(p, sizeof buf, "score=%s type=%d size=%d", scorestr(dh->score), (int)dh->type, (int)dh->size);