[Neo4j] Node creation limit

Biren Gandhi biren.gandhi at gmail.com
Fri Jun 4 01:15:04 CEST 2010


In that case, what are the ways to estimate storage capacity numbers? Basic
formula of nodes*9 + edges*33 doesn't seem like a practical one.

On Wed, Jun 2, 2010 at 11:26 PM, Mattias Persson
<mattias at neotechnology.com>wrote:

> String properties are stored in blocks so even if you have tiny string
> values each property value will occupy a full block (30 or 60 bytes,
> can someone correct me here?). That's what taking most of your space
> IMHO
>
> 2010/6/3, Biren Gandhi <biren.gandhi at gmail.com>:
> > Here is some content from neostore.propertystore.db.strings - another
> huge
> > file. What are the max number of nodes/relationships that people have
> tried
> > with Neo4j so far? Can someone share disk space usage characteristics?
> >
> > od -N 1000 -x -c neostore.propertystore.db.strings
> >
> > 0000000 0000 8500 0000 0000 0000 0000 0000 0000
> >          \0  \0  \0 205  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0
> > 0000020 0000 0000 0000 0000 0000 0000 0000 0000
> >          \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0
> > *
> > 0000200 0000 0000 0100 ffff ffff 0000 0c00 ffff
> >          \0  \0  \0  \0  \0 001 377 377 377 377  \0  \0  \0  \f 377 377
> > 0000220 ffff 4e00 6f00 6400 6500 2d00 3000 0000
> >         377 377  \0   N  \0   o  \0   d  \0   e  \0   -  \0   0  \0  \0
> > 0000240 0000 0000 0000 0000 0000 0000 0000 0000
> >          \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0
> > *
> > 0000400 0000 0000 0000 0000 0000 ff01 ffff 00ff
> >          \0  \0  \0  \0  \0  \0  \0  \0  \0  \0 001 377 377 377 377  \0
> > 0000420 0000 ff0c ffff 00ff 004e 006f 0064 0065
> >          \0  \0  \f 377 377 377 377  \0   N  \0   o  \0   d  \0   e  \0
> > 0000440 002d 0031 0000 0000 0000 0000 0000 0000
> >           -  \0   1  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0
> > 0000460 0000 0000 0000 0000 0000 0000 0000 0000
> >          \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0
> > *
> > 0000600 0000 0000 0000 0000 0000 0000 0000 0100
> >          \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0 001
> > 0000620 ffff ffff 0000 0c00 ffff ffff 4e00 6f00
> >         377 377 377 377  \0  \0  \0  \f 377 377 377 377  \0   N  \0   o
> > 0000640 6400 6500 2d00 3200 0000 0000 0000 0000
> >          \0   d  \0   e  \0   -  \0   2  \0  \0  \0  \0  \0  \0  \0  \0
> > 0000660 0000 0000 0000 0000 0000 0000 0000 0000
> >          \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0
> > *
> > 0001020 0000 0000 ff01 ffff 00ff 0000 ff0c ffff
> >          \0  \0  \0  \0 001 377 377 377 377  \0  \0  \0  \f 377 377 377
> > 0001040 00ff 004e 006f 0064 0065 002d 0033 0000
> >         377  \0   N  \0   o  \0   d  \0   e  \0   -  \0   3  \0  \0  \0
> > 0001060 0000 0000 0000 0000 0000 0000 0000 0000
> >          \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0
> > *
> > 0001220 0000 0000 0000 0000 0100 ffff ffff 0000
> >          \0  \0  \0  \0  \0  \0  \0  \0  \0 001 377 377 377 377  \0  \0
> > 0001240 0c00 ffff ffff 4e00 6f00 6400 6500 2d00
> >          \0  \f 377 377 377 377  \0   N  \0   o  \0   d  \0   e  \0   -
> > 0001260 3400 0000 0000 0000 0000 0000 0000 0000
> >          \0   4  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0
> > 0001300 0000 0000 0000 0000 0000 0000 0000 0000
> >          \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0
> > *
> > 0001420 0000 0000 0000 0000 0000 0000 0000 ff01
> >          \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0 001 377
> > 0001440 ffff 00ff 0000 ff0c ffff 00ff 004e 006f
> >         377 377 377  \0  \0  \0  \f 377 377 377 377  \0   N  \0   o  \0
> > 0001460 0064 0065 002d 0035 0000 0000 0000 0000
> >           d  \0   e  \0   -  \0   5  \0  \0  \0  \0  \0  \0  \0  \0  \0
> > 0001500 0000 0000 0000 0000 0000 0000 0000 0000
> >          \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0
> > *
> >
> >
> > On Wed, Jun 2, 2010 at 3:50 PM, Biren Gandhi <biren.gandhi at gmail.com>
> wrote:
> >
> >> There is only 1 property - "n" (to store name of the node) - used as
> >> follows:
> >>
> >>         Node node = graphDb.createNode();
> >>         node.setProperty( NAME_KEY, username );
> >>
> >> And the values of username are "Node-1", "Node-2" etc.
> >>
> >> On Wed, Jun 2, 2010 at 3:14 PM, Mattias Persson <
> mattias at neotechnology.com
> >> > wrote:
> >>
> >>> Only 4,4mb out of those 80 is consumed by nodes so you must be storing
> >>> some properties somewhere. Would you mind sharing your code so that it
> >>> would be easier to get a better insight into your problem?
> >>>
> >>> 2010/6/2, Biren Gandhi <biren.gandhi at gmail.com>:
> >>> > Thanks. Big transactions were indeed problematic. Splitting them down
> >>> into
> >>> > smaller chunks did the trick.
> >>> >
> >>> > I'm still disappointed by the on-disk size of a minimal node without
> >>> > any
> >>> > relationships or attributes. For 500K nodes, it is taking 80MB space
> >>> (160
> >>> > byes/node) and for 1M objects it is consuming 160MB (again 160
> >>> byes/node).
> >>> > Is this normal?
> >>> >
> >>> > 4.0K    active_tx_log
> >>> > 12K     lucene
> >>> > 12K     lucene-fulltext
> >>> > 4.0K    neostore
> >>> > 4.0K    neostore.id
> >>> > 4.4M    neostore.nodestore.db
> >>> > 4.0K    neostore.nodestore.db.id
> >>> > 12M     neostore.propertystore.db
> >>> > 4.0K    neostore.propertystore.db.arrays
> >>> > 4.0K    neostore.propertystore.db.arrays.id
> >>> > 4.0K    neostore.propertystore.db.id
> >>> > 4.0K    neostore.propertystore.db.index
> >>> > 4.0K    neostore.propertystore.db.index.id
> >>> > 4.0K    neostore.propertystore.db.index.keys
> >>> > 4.0K    neostore.propertystore.db.index.keys.id
> >>> > 64M     neostore.propertystore.db.strings
> >>> > 4.0K    neostore.propertystore.db.strings.id
> >>> > 4.0K    neostore.relationshipstore.db
> >>> > 4.0K    neostore.relationshipstore.db.id
> >>> > 4.0K    neostore.relationshiptypestore.db
> >>> > 4.0K    neostore.relationshiptypestore.db.id
> >>> > 4.0K    neostore.relationshiptypestore.db.names
> >>> > 4.0K    neostore.relationshiptypestore.db.names.id
> >>> > 4.0K    nioneo_logical.log.active
> >>> > 4.0K    tm_tx_log.1
> >>> > 80M     total
> >>> >
> >>> >
> >>> > On Wed, Jun 2, 2010 at 12:17 AM, Mattias Persson
> >>> > <mattias at neotechnology.com>wrote:
> >>> >
> >>> >> Exactly, the problem is most likely that you try to insert all your
> >>> >> stuff in one transaction. All data for a transaction is kept in
> memory
> >>> >> until committed so for really big transactions it can fill your
> entire
> >>> >> heap. Try to group 10k operations or so for big insertions or use
> the
> >>> >> batch inserter.
> >>> >>
> >>> >> Links:
> >>> >> http://wiki.neo4j.org/content/Transactions#Big_transactions
> >>> >> http://wiki.neo4j.org/content/Batch_Insert
> >>> >>
> >>> >> 2010/6/2, Laurent Laborde <kerdezixe at gmail.com>:
> >>> >> > On Wed, Jun 2, 2010 at 3:50 AM, Biren Gandhi <
> biren.gandhi at gmail.com
> >>> >
> >>> >> wrote:
> >>> >> >>
> >>> >> >> Is there any limit on number of nodes that can be created in a
> >>> >> >> neo4j
> >>> >> >> instance? Any other tips?
> >>> >> >
> >>> >> > I created hundreds of millions of nodes without problems, but it
> was
> >>> >> > splitted into many transaction.
> >>> >> >
> >>> >> > --
> >>> >> > Laurent "ker2x" Laborde
> >>> >> > Sysadmin & DBA at http://www.over-blog.com/
> >>> >> > _______________________________________________
> >>> >> > Neo4j mailing list
> >>> >> > User at lists.neo4j.org
> >>> >> > https://lists.neo4j.org/mailman/listinfo/user
> >>> >> >
> >>> >>
> >>> >>
> >>> >> --
> >>> >> Mattias Persson, [mattias at neotechnology.com]
> >>> >> Hacker, Neo Technology
> >>> >> www.neotechnology.com
> >>> >> _______________________________________________
> >>> >> Neo4j mailing list
> >>> >> User at lists.neo4j.org
> >>> >> https://lists.neo4j.org/mailman/listinfo/user
> >>> >>
> >>> > _______________________________________________
> >>> > Neo4j mailing list
> >>> > User at lists.neo4j.org
> >>> > https://lists.neo4j.org/mailman/listinfo/user
> >>> >
> >>>
> >>>
> >>> --
> >>> Mattias Persson, [mattias at neotechnology.com]
> >>> Hacker, Neo Technology
> >>> www.neotechnology.com
> >>> _______________________________________________
> >>> Neo4j mailing list
> >>> User at lists.neo4j.org
> >>> https://lists.neo4j.org/mailman/listinfo/user
> >>>
> >>
> >>
> > _______________________________________________
> > Neo4j mailing list
> > User at lists.neo4j.org
> > https://lists.neo4j.org/mailman/listinfo/user
> >
>
>
> --
> Mattias Persson, [mattias at neotechnology.com]
> Hacker, Neo Technology
> www.neotechnology.com
> _______________________________________________
> Neo4j mailing list
> User at lists.neo4j.org
> https://lists.neo4j.org/mailman/listinfo/user
>


More information about the User mailing list