defaults for pagesize/ptree_pagesize okay?

Issue #8 wontfix
Christoph Anton Mitterer
created an issue

Hi.

I'm just in the process of migrating my keyserver to another host and during that I was trying out new config values.

Previously I had Debian's defaults pagesize: 16 ptree_pagesize: 16

now I tried with SKS' defaults. But then every time during the full build of the DB, the process just freezed. Could be killed only with SIGKILL.

Now I use. pagesize: 32 ptree_pagesize: 32 which seems to work as well.

The machine is really rather powerfull 4GB RAM, 2 core cpu... are SKS' defaults sane?

Cheers, Chris.

Comments (6)

  1. John Clizbe

    My first question is why did you trash a working database? No rebuild is needed to upgrade SKS or BDB versions. Even distros with only a single version of the db_* tools can be handled by proper sysadmin work.

    The SKS defaults were derived from the output of Berkeley DB's db_tuner utility and then tested as well as sanity checked against 20-some server-years of experience.

    Both the debian defaults you mention and the 32/32 values you cited working may have built databases but they did so by ignoring the base problem and will likely have trouble in production. The build problem is due to the default number of mutexes in BDB being too small. This may be naively avoided by increasing the pagesizes to require less locks. pagesize values of 8k and 16k will just require a much greater number of locks when updating keys. sks db can run out of resources and die. ptree_pagesizes greater than 8K showed a propensity for deadlocking during large recon sessions. BDB locks the full page even for a row update.

    The proper solution to your locking during build issue would have been to place a suitable DB_CONFIG file of BDB tunable settings in the parent directory of KDB and PTree /before/ running sks build and sks pbuild. This will then initialize the bdb environment with suitable resources both to successfully build the database and run in production. DB_CONFIG.KDB and DB_CONFIG.PTree names take precedence. If not found, DB_CONFIG will be used if present. Sample files are in the source. You may also wish to read sksconf.typical, also in the sampleConfig directory for guidance ohe various pagesize settings

  2. Christoph Anton Mitterer reporter

    Hi John.

    Well … "working"… as I mentioned on sks-devel, I'm currently moving my SKS to a new much better server with IPv6 and stuff and I'll set up Apache proxying for it..etc.pp.

    When starting this, I've noticed that my old server was already for some days out of the pool, and it seems reconciliation has stopped... and some weired errors showed up in the log. So I decided that it shouly cannot harm too much, to rebuild the DB... and actually that seems to work pretty fast nowadays (less than an hour I guess).

    Regarding the defaults and BDB / DB_CONFIG... I must admit that I never ever looked really deeply into that, nor how BDB works internally (in terms of pages, mutexes, etc.) So right now, that's just black magic for me ;-)

    But AFAIU, at least for pagesize the default is much higher than Debian's, wasn't it 128? And the Debian maintainer already mentiones in the README that he found the upstream defaults to be problematic and therefore chose 16/16... and I thought it cannot harm to report that here, maybe the issue is just unknown and things can be improved.

    Those pagesizes and similar settings... do they go directly into DB? I mean are these just runtime settings, i.e. how much resources BDB uses, or has the DB now really a different format and I have to stay with 32/32?

    I'll try to have a read through sampleConfig/*

    Thanks, Chris.

  3. Christoph Anton Mitterer reporter

    I've tried that again now with the 1.1.4 package from Daniel's most recent Debian upload... using the default values. Everything being in pristine state again.

    This time the full built worked without freezing...

    Well not sure.... perhaps we can close that now... anyway if you have answers for my question above, I'd be still interested.

    Perhaps we should ship a default DB_CONFIG? I never set anything in there.

  4. John Clizbe

    "Those pagesizes and similar settings... do they go directly into DB? I mean are these just runtime settings, i.e. how much resources BDB uses, or has the DB now really a different format and I have to stay with 32/32?"

    The pagesizes are set at file creation time. It cannot easily be changed later without much more BDB black magic that no one wants to support. The values in DB_CONFIG are known as tunables. Usually the only DB maintenance that needs to be done is to run db_recover to remove the old environment so that a new one is created using the new values.

    "Perhaps we should ship a default DB_CONFIG? "

    We do. It was in the sample config directory at least one release before I added to code to use it to initialize DB_ENV at DB creation time.

    Was there ever a bug here or was this something that would have been more beneficial to the community on the mailing list?

  5. Christoph Anton Mitterer reporter

    Hi John.

    (*) I think we should warn people which settings permanently affect the DB at creation time.

    Ah,... sorry I missed that example file O:-)

    Well, I guess in the end it turned out that there was no bug here... I had problems at first with the default settings but that seemed to have gone when moving to 1.1.4.

    So either we close the ticket, or reuse it perhaps to do (*) from above, if that seems reasonable to you.

    Apart from that, thanks for your help and feel free to close the ticket :)

    Cheers, Chris.

  6. Log in to comment