'''This is a work in progress and the API is not complete'''

New Storage API

Problems with the existing storage utility

• The API is messy and inconsistent
• We often want our records to have IDs that we control, independent of its ID in RMS; we currently have no way to store and look up records based on our own IDs, and if we need to, we have to create our own indexing mechanism or do a brute-force search
• Each storage utility is tied to a single RMS, and once that RMS fills up, you’re out of luck, even if the phone itself is not yet full
• Managing meta-data through separate objects that you have to define, create, and manage is messy and cumbersome
• When reading an object from RMS, you have to create and pass the skeleton object yourself; this stinks of the old serialization framework
• Searching/indexing capabilities are primitive
• Object-specific RMSUtility subclasses have to duplicate lots of boilerplate functionality that should be handled by the RMSUtility itself
• It is not robust against errors such as RMS full

A vision for the new storage utility

• Amount of data that can be stored is not restricted by the size limits of a single RMS
• Separation between application-level record IDs and RMS record IDs. Ability to store and retrieve records based on its application-level ID alone (this also frees us from having to worry about write vs. update). (This is especially important now that the record may be stored in any one of several RMSes).
• Still support storage/retrieval of records that don’t have internal IDs, meaning the StorageUtility will manage and allocate IDs.
• Good performance with a set of approximately 1,000 2-3KB records
• New StorageUtility removes as much burden as possible from the API user (this includes not having to create skeleton objects or meta-data objects)
• Support for meta-data: easy access to a few key fields w/o having to deserialize the entire record (important for large records)
• Support for indexing/filtering/searching records
• Support for enumerating through all records or a filtered set of records
• Handle errors gracefully

Details of proposed API

Externalizable and Persistable

The Externalizable interface will remain unchanged. The StorageUtility will only be concerned with reading and writing Externalizables to RMS (however it will support retrieving the raw bytes for a record, for debugging). To store any other type of data, write a small wrapper for it that implements Externalizable.

The IDRecordable interface will be removed, but its idea will carry on in a new interface, Persistable, that extends Externalizable. Persistable is defined as follows:

public interface Persistable extends Externalizable {
  void setID (int ID);
  int getID ();
}

Persistable is for objects that have internal IDs which we want to use to look up records in the StorageUtility. When storing a Persistable object, it is filed under the ID retrieved from getID(), and can be retrieved with the same ID. setID() will never be used by the StorageUtility.

Basic reading/writing

StorageUtility will have the following methods for basic storing/retrieving of records:

• Externalizable read (int id) -- return the record filed under the given ID. return null if no record exists for that ID.

• void write (Persistable p) -- store the record p. ID is taken from the record itself.

• int add (Externalizable e) -- store the record e. Since it is not Persistable, an ID will be allocated by the StorageUtility and returned

• void update (int id, Externalizable e) -- update the record stored under id with the new object e

• boolean remove (int id) -- remove the record under ID. return true if the record existed and was removed, false if no record existed for that ID

• boolean remove (Persistable p) -- same as remove(int), but pull the ID from p

Ideally, a given StorageUtility will only deal with a single type of object. If it stores Persistables, you'd only use the read/write/remove functions. If it stores Externalizables, you'd only use the read/add/update/remove functions, which are similar to the J2ME RMS API.

Metadata

Meta-data will no longer be provided to the RMSUtility as a separate object. Instead, the StorageUtility will pull it from the main object using a new interface, MetaData.

An object that you wish to store metadata about in the StorageUtility will implement this interface, which provides a list of the available metadata fields, and their values.

• Vector getMetaDataFields() -- return a list of String keys that identify the metadata fields available for this object
• Vector getMetaDataTypes() -- return a list of Class objects that identify the type of the corresponding metadata field. Any Externalizable or primitive type (Integer, Double, String, Boolean, Date) may be used.
• Hashtable getMetaData() -- return a hashtable of the object's metadata, where the key for that field maps to the object's value for that field.

This same interface provides a similar set of functions to provide information about which fields should be indexed:

• Vector getIndexFields()
• Vector getIndexTypes() -- only primitive types may be indexed, not general Externalizables
• Hashtable getIndexData()

Implementation details for J2ME StorageUtility

Every storage utility will consist of one 'meta' RMS, and one or more data RMSes. Records themselves will be stored in a data RMS, until there is insufficient space in any data RMS. At this point, another data RMS will be created for spillover.

The meta RMS will have the following structure:

Record 1: Information about the storage utility:

• base type of the records in this StorageUtility
• number of active data RMSes
• list of index and meta-data fields, and their types

Record 2: ID index -- a hashtable mapping record IDs to a tuple (RMS #, RMS record ID) where the record may be found

Record 3: Meta-data -- a hashtable mapping record IDs to the set of meta-data for that record

Records 4-n: Value index (one record per each indexable field) -- a hashtable mapping [value] to list of record IDs whose field has that value

One potential problem is that the meta RMS doesn't spillover, so the total amount of records that can be stored is limited by how much information this meta RMS can hold. For records with lots of meta-data or indexing, this RMS may fill up even though there is still space on the phone for more spillover RMSes to store the actual records. Spillover provisions for the meta RMS seem unduly complicated and like over-engineering at this point.