cumulative indexing gets slower

[Shinichiro Abe]

it happens when using indexholder with small documents batch. because of realtime merging?

[Shinichiro Abe]

hold #147. add benchmark. deletion bug

→ <<cset 8e94b4176619>>

[Shinichiro Abe]

fix a deletion bug:

before: SmallBatches 140 ms 1940:1950 SmallBatches 141 ms 1950:1960 SmallBatches 136 ms 1960:1970 SmallBatches 147 ms 1970:1980 SmallBatches 153 ms 1980:1990 ... SmallBatches 653 ms 9980:9990 SmallBatches 645 ms 9990:10000 indexed 332813 ms

after: SmallBatches 68 ms 1960:1970 SmallBatches 71 ms 1970:1980 SmallBatches 61 ms 1980:1990 SmallBatches 62 ms 1990:2000 SmallBatches 81 ms 2000:2010 ... SmallBatches 318 ms 9980:9990 SmallBatches 291 ms 9990:10000 indexed 160401 ms

[Shinichiro Abe]

hold #147. call forceMerge() when segments > 10

→ <<cset e49d6b24648d>>

[Shinichiro Abe]

call forceMerge() when segments > 10

after:

SmallBatches 25 ms 1960:1970

SmallBatches 17 ms 1970:1980

SmallBatches 18 ms 1980:1990

SmallBatches 19 ms 1990:2000

SmallBatches 17 ms 2000:2010

...

SmallBatches 130 ms 9980:9990

SmallBatches 129 ms 9990:10000

indexed 103877 ms

[Shinichiro Abe]

hold #147. temp prints

→ <<cset ebdb96b0dde1>>

[Shinichiro Abe]

merge 6 ms

resize 16 ms

SmallBatches 25 ms 1960:1970

merge 7 ms

resize 9 ms

SmallBatches 19 ms 1970:1980

merge 6 ms

resize 8 ms

SmallBatches 17 ms 1980:1990

merge 6 ms

resize 8 ms

SmallBatches 17 ms 1990:2000

[Shinichiro Abe]

hold #147. use RAMIndexHolderDirectory instead of realtime merging

→ <<cset c76f7c9a2b0f>>

[Shinichiro Abe]

SmallBatches 2 ms 1960:1970

SmallBatches 3 ms 1970:1980

SmallBatches 3 ms 1980:1990

SmallBatches 3 ms 1990:2000

SmallBatches 5 ms 2000:2010

...

SmallBatches 5 ms 9980:9990

SmallBatches 4 ms 9990:10000

indexed 80449 ms

[Shinichiro Abe]

TODO:

• index "a"

• assert _openRealTimeIndexReader() has "a"

• index "b"

• assert _openRealTimeIndexReader() has "a" and "b"

• reinit

• index "c"

• assert _openRealTimeIndexReader() has "a" and "b" and "c"?

[Shinichiro Abe]

fix #147. re-init

→ <<cset 5a035c4ee857>>