Creating an XML Backed DataServer to speed up writes.
The built-in .net XmlSerializer works great for small DataSets, up to 5/6Mb XML files. Beyond this there's a lagg when flushing to disk.
This lagg is caused by the fact the whole XML content is re-generated and written to disk.
I want to retain these large XML files, for ease of life mainly; it will allow us to easy edit the content of the data w/o having to use bespoke tools.
I want to create a DataServer that writes the Data into another persistent data store, that can be used to back searches and the loading of the data.
Indexes will also be required so that only the minimum about of data is loaded into Memory.
UPDATE 17/12/2013 : There's a new Fork, MemMapping, that contains 95% of the this functionality built using MemoryMappedFiles and Special Serialized B-Tree indexs.
The only outstanding aspect to be built prior to use is logic to dynamically add additional DataFiles as the Data within the Server expands.
This is required as there's is a fundamental restriction to using MemoryMapped Files; the files can not be resized.
Thus the data will span a number of files as the need arises, the file capacity will inc. exponentially from 128Mb to a maximum of 2Gb.