BerkeleyDB Backend Storage Engine for DURUS 20070503

BerkeleyDB Backend Storage Engine for DURUS project provides a storage engine for DURUS, a persistence system for the Python programming language.
Some advantages compared to Durus standard FileStorage:
- Startup time is negligible.
- You dont need an in-memory index, so your repository size is only limited by storage space, not RAM.
- If you change existing objects, your storage size doesnt increase.
- If you delete objects, those objects are garbage collected in background, slowly, without performance degradation.
- You can still do a full fast collection, if you need it. While this collection is in progress, Durus still serves objects. From time to time, nevertheless, it can be unresponsible for 2 or 3 seconds, while is doing a checkpointing to be able to free database logging diskspace.
- Garbage collection doesnt increase storage size. Neither RAM usage.
- Garbage collection deletes objects using nondurable transactions, very efficiently. If the collection is aborted abruptly (program or machine crashes), the collection will start again from the beginning. If the GC finishes without problems, that state is durable.
- Any object store in the storage will commit a durable transaction, including all objects released in the background garbage collector, along the way.
- Garbage collection time is proportional to garbage, not repository size.
There are some disadvantages, nevertheless:
- IMPORTANT: This backend uses reference counting to decide when an object is garbage and can be collected. So, if you have cycles in your data structures, you **MUST*BREAK** them before releasing the objects.
- Failing to do that will leak diskspace. It is possible that in a future release we can collect cycles, but try to avoid that pattern.
- Leaking objects will grow the diskspace, but **NO** corruption or malfunction will happen. No other secondary effect.
- Although this code could work on Windows, I havent checked it. Absolutely no garantee!.
- Sometimes this backend can become irresponsible for a couple of seconds. It is busy doing a checkpoint to recycle database logging space. The pause should be sort, nevertheless.
- Dont use this storage backend over NFS, at least you know what is going on.
- Since we are using BerkeleyDB as the backend:
- You should be experienced with BerkeleyDB deployments.
- Beware when updating Python or BerkeleyDB. In particular, BerkeleyDB is known by breaking (but they DOCUMENT!) binary compatibility between versions. In this case, they ALWAYS document the procedure to do a controlled upgrade, so dont worry. But take note of the risk.
- To do a trustable backup, you should follow instructions in BerkeleyDB documentation:
- http://www.sleepycat.com/docs/ref/transapp/reclimit.html
- http://www.sleepycat.com/docs/ref/transapp/archival.html
- http://www.sleepycat.com/docs/utility/db_hotbackup.html
- In Python you can use the standard "bsddb" or the up-to-date "bsddb3" bindings (which will be included in new python versions). This product will try to use always the more recent BerkeleyDB bindings. Be careful about BerkeleyDB version changes when you update the bindings.
- Since BerkeleyDB files are binary structures, a corrupt database can be unrecoverable. Be diligent and careful with your backups.
You can use this product both as a normal (local) filestorage, or a server (remote) storage system, just like the usual Durus FileStorage.
Enhancements:
- Compatibility with Durus 3.7 was added.