Subj : Committing file changes
To : David Noon
From : Coridon Henshaw
Date : Mon Aug 14 2000 04:33 pm
On Monday August 14 2028 at 20:23, David Noon wrote to Coridon Henshaw:
DN> How low do you want to go?
I'm building an open-source databasing offline Usenet news system, basically
along the lines of standard Fidonet message tossers and readers, except
designed from the ground up for Usenet news. As I intend the system to be
portable, I'd like to keep the number of platform-specific API calls to an
absolute minimum.
Incidentally, if anyone would like to take a look at an alpha copy of NewsDB,
or would like to contribute to the project, drop me a message.
DN> Firstly, you should not be doing buffered I/O if your updates must be
DN> committed immediately, so you should not use fopen() and fwrite() without
a
DN> setbuf() call to suppress buffer allocation. Better yet, you should
DN> consider using open() and write() instead, and use the UNIX-like
DN> unbuffered I/O routines.
I'm concerned that disabling buffering entirely is going to hurt performance
very badly as my application does lots of short (4 to 256 byte) IO calls.
Relying on the disk cache to handle this kind of load seems a bit wasteful.
DN> Moreover, if your data resources are critically important then you should
DN> be handling any traps that occur in your program and cleaning up the
DN> critical data resources in an orderly manner. This is far and away the
most
DN> professional approach to the situation. About the only things you can't
DN> handle are kernel level traps and power outages.
The problem I ran into was that the kernel trapped (for reasons unrelated to
this project) a few hours after I wrote an article into the article database.
Since database was still in open (I leave the article reader running 24x7), the
file system structures were inconsistant enough that CHKDSK truncated the
database well before its proper end point. As you say, catching exceptions
wouldn't help much here.
My database format and engine implementations are robust enough to cope with
applications dying unexpectedly without finishing write operations; they're not
robust enough to handle boot-up CHKDSK removing 80Kb of data from the end of a
100Kb file.
DN> In your situation, I would have used the second facility before
DN> considering any intermediate commits.
It's not intermediate commits I need: what I need is some way to flush out
write operations made to files which might be open for days or weeks at a time.
--- GoldED/2 3.0.1
* Origin: Life sucks and then you croak. (1:250/820)
