[Neo4j] Performance problem inserting nodes with many short string properties
mattias at neotechnology.com
Sat Jul 24 18:03:28 CEST 2010
2010/7/22 Jeff Klann <jklann at iupui.edu>
> And in the meantime I'm rewriting some code to use the batch inserter, but
> the LuceneIndexBatchInserterImpl is not reading an index that already
> in the db! (I'm trying to use a pre-existing index to find parent nodes for
> the nodes I'm inserting.) The shell verifies the index is in fact there.
Yeah... about that: that's an issue in neo4j-kernel version 1.0 (although
it's fixed in 1.1-SNAPSHOT).
> - Jeff Klann
> On Thu, Jul 22, 2010 at 11:27 AM, Jeff Klann <jklann at iupui.edu> wrote:
> > I'm stumped on this one.
> > I'm getting the "fast write performance at first that slows to a crawl"
> > issue described in the performance guide, so I increased the Linux
> > dirty_page ratio (all the way up to 80%), turned of auto log rotation,
> > increased the size of the memory mapped cache. This issue is still
> > exactly as before.
> > I've narrowed my problem to this:
> > *If I insert a lot of nodes with about 50 short string properties
> > the performance slows to a crawl at about 40,000 inserts (and it stays
> > slow)* ... however if I don't insert the properties the performance is
> > What am I doing wrong? The machine currently has a small amount of RAM,
> > I don't understand why that would impact pure insertion, and only after
> > thousands of inserts. (I don't read the properties back after adding
> > I have not used BatchInserter because it is nice to have normal database
> > access for some parts of this database builder program I'm writing, but
> > that's the only way I could refactor. Also all these inserts are within
> > transaction (about 100k nodes per transaction) - do I need to split this
> > into smaller transactions?
> > Thanks,
> > Jeff Klann
> Neo4j mailing list
> User at lists.neo4j.org
Mattias Persson, [mattias at neotechnology.com]
Hacker, Neo Technology
More information about the User