[Neo4j] Performance problem inserting nodes with many short string properties

Mattias Persson mattias at neotechnology.com
Sat Jul 24 18:03:28 CEST 2010


2010/7/22 Jeff Klann <jklann at iupui.edu>

> And in the meantime I'm rewriting some code to use the batch inserter, but
> the LuceneIndexBatchInserterImpl is not reading an index that already
> exists
> in the db! (I'm trying to use a pre-existing index to find parent nodes for
> the nodes I'm inserting.) The shell verifies the index is in fact there.
> What?
>
Yeah... about that: that's an issue in neo4j-kernel version 1.0 (although
it's fixed in 1.1-SNAPSHOT).


>
> - Jeff Klann
>
> On Thu, Jul 22, 2010 at 11:27 AM, Jeff Klann <jklann at iupui.edu> wrote:
>
> > I'm stumped on this one.
> >
> > I'm getting the "fast write performance at first that slows to a crawl"
> > issue described in the performance guide, so I increased the Linux
> > dirty_page ratio (all the way up to 80%), turned of auto log rotation,
> and
> > increased the size of the memory mapped cache. This issue is still
> happening
> > exactly as before.
> >
> > I've narrowed my problem to this:
> >    *If I insert a lot of nodes with about 50 short string properties
> each,
> > the performance slows to a crawl at about 40,000 inserts (and it stays
> > slow)* ... however if I don't insert the properties the performance is
> fine.
> >
> > What am I doing wrong? The machine currently has a small amount of RAM,
> but
> > I don't understand why that would impact pure insertion, and only after
> > thousands of inserts. (I don't read the properties back after adding
> them.)
> > I have not used BatchInserter because it is nice to have normal database
> > access for some parts of this database builder program I'm writing, but
> if
> > that's the only way I could refactor. Also all these inserts are within
> one
> > transaction (about 100k nodes per transaction) - do I need to split this
> > into smaller transactions?
> >
> > Thanks,
> > Jeff Klann
> >
> _______________________________________________
> Neo4j mailing list
> User at lists.neo4j.org
> https://lists.neo4j.org/mailman/listinfo/user
>



-- 
Mattias Persson, [mattias at neotechnology.com]
Hacker, Neo Technology
www.neotechnology.com


More information about the User mailing list