[Neo4j] Performance problem inserting nodes with many short string properties
jklann at iupui.edu
Thu Jul 22 21:53:15 CEST 2010
And in the meantime I'm rewriting some code to use the batch inserter, but
the LuceneIndexBatchInserterImpl is not reading an index that already exists
in the db! (I'm trying to use a pre-existing index to find parent nodes for
the nodes I'm inserting.) The shell verifies the index is in fact there.
- Jeff Klann
On Thu, Jul 22, 2010 at 11:27 AM, Jeff Klann <jklann at iupui.edu> wrote:
> I'm stumped on this one.
> I'm getting the "fast write performance at first that slows to a crawl"
> issue described in the performance guide, so I increased the Linux
> dirty_page ratio (all the way up to 80%), turned of auto log rotation, and
> increased the size of the memory mapped cache. This issue is still happening
> exactly as before.
> I've narrowed my problem to this:
> *If I insert a lot of nodes with about 50 short string properties each,
> the performance slows to a crawl at about 40,000 inserts (and it stays
> slow)* ... however if I don't insert the properties the performance is fine.
> What am I doing wrong? The machine currently has a small amount of RAM, but
> I don't understand why that would impact pure insertion, and only after
> thousands of inserts. (I don't read the properties back after adding them.)
> I have not used BatchInserter because it is nice to have normal database
> access for some parts of this database builder program I'm writing, but if
> that's the only way I could refactor. Also all these inserts are within one
> transaction (about 100k nodes per transaction) - do I need to split this
> into smaller transactions?
> Jeff Klann
More information about the User