[Neo4j] Node creation limit

Mattias Persson mattias at neotechnology.com
Fri Jun 4 09:15:56 CEST 2010


That formula is correct regarding nodes and relationships, yes. When
properties comes into play another formula would, of course, have to
be applied. Depending on property types and length of keys/string
values it is different. It could be good though with a formula/tool to
calculate that.

2010/6/4, Biren Gandhi <biren.gandhi at gmail.com>:
> In that case, what are the ways to estimate storage capacity numbers? Basic
> formula of nodes*9 + edges*33 doesn't seem like a practical one.
>
> On Wed, Jun 2, 2010 at 11:26 PM, Mattias Persson
> <mattias at neotechnology.com>wrote:
>
>> String properties are stored in blocks so even if you have tiny string
>> values each property value will occupy a full block (30 or 60 bytes,
>> can someone correct me here?). That's what taking most of your space
>> IMHO
>>
>> 2010/6/3, Biren Gandhi <biren.gandhi at gmail.com>:
>> > Here is some content from neostore.propertystore.db.strings - another
>> huge
>> > file. What are the max number of nodes/relationships that people have
>> tried
>> > with Neo4j so far? Can someone share disk space usage characteristics?
>> >
>> > od -N 1000 -x -c neostore.propertystore.db.strings
>> >
>> > 0000000 0000 8500 0000 0000 0000 0000 0000 0000
>> >          \0  \0  \0 205  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0
>> > 0000020 0000 0000 0000 0000 0000 0000 0000 0000
>> >          \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0
>> > *
>> > 0000200 0000 0000 0100 ffff ffff 0000 0c00 ffff
>> >          \0  \0  \0  \0  \0 001 377 377 377 377  \0  \0  \0  \f 377 377
>> > 0000220 ffff 4e00 6f00 6400 6500 2d00 3000 0000
>> >         377 377  \0   N  \0   o  \0   d  \0   e  \0   -  \0   0  \0  \0
>> > 0000240 0000 0000 0000 0000 0000 0000 0000 0000
>> >          \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0
>> > *
>> > 0000400 0000 0000 0000 0000 0000 ff01 ffff 00ff
>> >          \0  \0  \0  \0  \0  \0  \0  \0  \0  \0 001 377 377 377 377  \0
>> > 0000420 0000 ff0c ffff 00ff 004e 006f 0064 0065
>> >          \0  \0  \f 377 377 377 377  \0   N  \0   o  \0   d  \0   e  \0
>> > 0000440 002d 0031 0000 0000 0000 0000 0000 0000
>> >           -  \0   1  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0
>> > 0000460 0000 0000 0000 0000 0000 0000 0000 0000
>> >          \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0
>> > *
>> > 0000600 0000 0000 0000 0000 0000 0000 0000 0100
>> >          \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0 001
>> > 0000620 ffff ffff 0000 0c00 ffff ffff 4e00 6f00
>> >         377 377 377 377  \0  \0  \0  \f 377 377 377 377  \0   N  \0   o
>> > 0000640 6400 6500 2d00 3200 0000 0000 0000 0000
>> >          \0   d  \0   e  \0   -  \0   2  \0  \0  \0  \0  \0  \0  \0  \0
>> > 0000660 0000 0000 0000 0000 0000 0000 0000 0000
>> >          \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0
>> > *
>> > 0001020 0000 0000 ff01 ffff 00ff 0000 ff0c ffff
>> >          \0  \0  \0  \0 001 377 377 377 377  \0  \0  \0  \f 377 377 377
>> > 0001040 00ff 004e 006f 0064 0065 002d 0033 0000
>> >         377  \0   N  \0   o  \0   d  \0   e  \0   -  \0   3  \0  \0  \0
>> > 0001060 0000 0000 0000 0000 0000 0000 0000 0000
>> >          \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0
>> > *
>> > 0001220 0000 0000 0000 0000 0100 ffff ffff 0000
>> >          \0  \0  \0  \0  \0  \0  \0  \0  \0 001 377 377 377 377  \0  \0
>> > 0001240 0c00 ffff ffff 4e00 6f00 6400 6500 2d00
>> >          \0  \f 377 377 377 377  \0   N  \0   o  \0   d  \0   e  \0   -
>> > 0001260 3400 0000 0000 0000 0000 0000 0000 0000
>> >          \0   4  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0
>> > 0001300 0000 0000 0000 0000 0000 0000 0000 0000
>> >          \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0
>> > *
>> > 0001420 0000 0000 0000 0000 0000 0000 0000 ff01
>> >          \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0 001 377
>> > 0001440 ffff 00ff 0000 ff0c ffff 00ff 004e 006f
>> >         377 377 377  \0  \0  \0  \f 377 377 377 377  \0   N  \0   o  \0
>> > 0001460 0064 0065 002d 0035 0000 0000 0000 0000
>> >           d  \0   e  \0   -  \0   5  \0  \0  \0  \0  \0  \0  \0  \0  \0
>> > 0001500 0000 0000 0000 0000 0000 0000 0000 0000
>> >          \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0
>> > *
>> >
>> >
>> > On Wed, Jun 2, 2010 at 3:50 PM, Biren Gandhi <biren.gandhi at gmail.com>
>> wrote:
>> >
>> >> There is only 1 property - "n" (to store name of the node) - used as
>> >> follows:
>> >>
>> >>         Node node = graphDb.createNode();
>> >>         node.setProperty( NAME_KEY, username );
>> >>
>> >> And the values of username are "Node-1", "Node-2" etc.
>> >>
>> >> On Wed, Jun 2, 2010 at 3:14 PM, Mattias Persson <
>> mattias at neotechnology.com
>> >> > wrote:
>> >>
>> >>> Only 4,4mb out of those 80 is consumed by nodes so you must be storing
>> >>> some properties somewhere. Would you mind sharing your code so that it
>> >>> would be easier to get a better insight into your problem?
>> >>>
>> >>> 2010/6/2, Biren Gandhi <biren.gandhi at gmail.com>:
>> >>> > Thanks. Big transactions were indeed problematic. Splitting them
>> >>> > down
>> >>> into
>> >>> > smaller chunks did the trick.
>> >>> >
>> >>> > I'm still disappointed by the on-disk size of a minimal node without
>> >>> > any
>> >>> > relationships or attributes. For 500K nodes, it is taking 80MB space
>> >>> (160
>> >>> > byes/node) and for 1M objects it is consuming 160MB (again 160
>> >>> byes/node).
>> >>> > Is this normal?
>> >>> >
>> >>> > 4.0K    active_tx_log
>> >>> > 12K     lucene
>> >>> > 12K     lucene-fulltext
>> >>> > 4.0K    neostore
>> >>> > 4.0K    neostore.id
>> >>> > 4.4M    neostore.nodestore.db
>> >>> > 4.0K    neostore.nodestore.db.id
>> >>> > 12M     neostore.propertystore.db
>> >>> > 4.0K    neostore.propertystore.db.arrays
>> >>> > 4.0K    neostore.propertystore.db.arrays.id
>> >>> > 4.0K    neostore.propertystore.db.id
>> >>> > 4.0K    neostore.propertystore.db.index
>> >>> > 4.0K    neostore.propertystore.db.index.id
>> >>> > 4.0K    neostore.propertystore.db.index.keys
>> >>> > 4.0K    neostore.propertystore.db.index.keys.id
>> >>> > 64M     neostore.propertystore.db.strings
>> >>> > 4.0K    neostore.propertystore.db.strings.id
>> >>> > 4.0K    neostore.relationshipstore.db
>> >>> > 4.0K    neostore.relationshipstore.db.id
>> >>> > 4.0K    neostore.relationshiptypestore.db
>> >>> > 4.0K    neostore.relationshiptypestore.db.id
>> >>> > 4.0K    neostore.relationshiptypestore.db.names
>> >>> > 4.0K    neostore.relationshiptypestore.db.names.id
>> >>> > 4.0K    nioneo_logical.log.active
>> >>> > 4.0K    tm_tx_log.1
>> >>> > 80M     total
>> >>> >
>> >>> >
>> >>> > On Wed, Jun 2, 2010 at 12:17 AM, Mattias Persson
>> >>> > <mattias at neotechnology.com>wrote:
>> >>> >
>> >>> >> Exactly, the problem is most likely that you try to insert all your
>> >>> >> stuff in one transaction. All data for a transaction is kept in
>> memory
>> >>> >> until committed so for really big transactions it can fill your
>> entire
>> >>> >> heap. Try to group 10k operations or so for big insertions or use
>> the
>> >>> >> batch inserter.
>> >>> >>
>> >>> >> Links:
>> >>> >> http://wiki.neo4j.org/content/Transactions#Big_transactions
>> >>> >> http://wiki.neo4j.org/content/Batch_Insert
>> >>> >>
>> >>> >> 2010/6/2, Laurent Laborde <kerdezixe at gmail.com>:
>> >>> >> > On Wed, Jun 2, 2010 at 3:50 AM, Biren Gandhi <
>> biren.gandhi at gmail.com
>> >>> >
>> >>> >> wrote:
>> >>> >> >>
>> >>> >> >> Is there any limit on number of nodes that can be created in a
>> >>> >> >> neo4j
>> >>> >> >> instance? Any other tips?
>> >>> >> >
>> >>> >> > I created hundreds of millions of nodes without problems, but it
>> was
>> >>> >> > splitted into many transaction.
>> >>> >> >
>> >>> >> > --
>> >>> >> > Laurent "ker2x" Laborde
>> >>> >> > Sysadmin & DBA at http://www.over-blog.com/
>> >>> >> > _______________________________________________
>> >>> >> > Neo4j mailing list
>> >>> >> > User at lists.neo4j.org
>> >>> >> > https://lists.neo4j.org/mailman/listinfo/user
>> >>> >> >
>> >>> >>
>> >>> >>
>> >>> >> --
>> >>> >> Mattias Persson, [mattias at neotechnology.com]
>> >>> >> Hacker, Neo Technology
>> >>> >> www.neotechnology.com
>> >>> >> _______________________________________________
>> >>> >> Neo4j mailing list
>> >>> >> User at lists.neo4j.org
>> >>> >> https://lists.neo4j.org/mailman/listinfo/user
>> >>> >>
>> >>> > _______________________________________________
>> >>> > Neo4j mailing list
>> >>> > User at lists.neo4j.org
>> >>> > https://lists.neo4j.org/mailman/listinfo/user
>> >>> >
>> >>>
>> >>>
>> >>> --
>> >>> Mattias Persson, [mattias at neotechnology.com]
>> >>> Hacker, Neo Technology
>> >>> www.neotechnology.com
>> >>> _______________________________________________
>> >>> Neo4j mailing list
>> >>> User at lists.neo4j.org
>> >>> https://lists.neo4j.org/mailman/listinfo/user
>> >>>
>> >>
>> >>
>> > _______________________________________________
>> > Neo4j mailing list
>> > User at lists.neo4j.org
>> > https://lists.neo4j.org/mailman/listinfo/user
>> >
>>
>>
>> --
>> Mattias Persson, [mattias at neotechnology.com]
>> Hacker, Neo Technology
>> www.neotechnology.com
>> _______________________________________________
>> Neo4j mailing list
>> User at lists.neo4j.org
>> https://lists.neo4j.org/mailman/listinfo/user
>>
> _______________________________________________
> Neo4j mailing list
> User at lists.neo4j.org
> https://lists.neo4j.org/mailman/listinfo/user
>


-- 
Mattias Persson, [mattias at neotechnology.com]
Hacker, Neo Technology
www.neotechnology.com


More information about the User mailing list