Recent changes to improvements

Improve accounting of used disk space

2009-04-16T14:20:17Z

Currently, each node keeps track of how much data it *generates*, and merges local updates when that amount passes some threshold. What we really want is for each node to keep track of how much data is actually stored locally. This is somewhat complicated due to the use of remote writes. Probably, the best way to do this is to write a wrapper function for all methods that write or delete data on disk (e.g. Roomy_fwrite, Roomy_unlink, etc.). These wrappers can then keep track of used disk space.

Option to eliminate hidden syncs

2009-04-14T23:03:11Z

Currently, a Roomy data structure is synced in two ways: 1) when a user call sync on it; 2) when disk space limits are reached and updates need to be synced to free space. So, for example, if user is mapping over an array and sending delayed updates to the same array, some updates may occur in the middle of the map (causing problems in some applications). Users can prevent this by having a secondary array to hold the updates temporarily, but this doubles the cost. If hidden syncs are eliminated, this secondary array can be avoided (but an error will occur when disk space is exhausted).

RoomyArray<->RoomyList Conversions

2009-04-13T23:13:47Z

Make methods to convert a RoomyList to a RoomyArray, and a RoomyArray to a RoomyList. The second can already be done by mapping over the array and adding to a list. The first can not currently be done by users.

Multi-threaded operations

2009-04-10T01:33:31Z

Spawning additional "compute threads", to complete map functions (etc.) more quickly. Useful for CPU-bound computations.

RAM-only verion

2009-04-10T01:32:34Z

Make a RAM-only version or Roomy. Possibly include distributed RAM. Helpful for testing. Can significantly improve performance, especially for small Roomy data structures that don't require disk.

Smaller representation of arrray indices

2009-04-10T01:30:53Z

Don't save all Roomy*Array indices as 8-byte ints. We can just use enough bits to represent the largest index of the array.

Compression

2009-04-10T01:29:38Z

Compress/decompress disk-based data structures and related updates on the fly. Requires determining if CPU or I/O is the bottleneck, and only using compression in the latter case.

Seek vs. scan for updates

2009-04-10T01:28:27Z

If there are few updates to an array, do them by direct modifications instead of complete scans. Requires determining the tipping point between the performance of the two methods (based on disk latency/bandwidth, number of updates, and size of updates).

Lazy creation of arrays

2009-04-10T01:26:21Z

No need to write zeroed chunks immediately upon creation. This would allow large savings for sparse arrays, and for arrays that are significantly "over-allocated".

Better distribution of RoomyList elements

2009-04-10T01:25:45Z

Improvement in evenly distributing RoomyList elements among compute nodes (basically needs to use a hash function for elements instead of a simple modulo-based method).