[Neo] Memory management in neo4j?

Mattias Persson mattias at neotechnology.com
Sun Jan 17 11:11:56 CET 2010


Yes, you're right Raul.

See http://wiki.neo4j.org/content/Transactions#Big_transactions for
more information


2010/1/15, Raul Raja Martinez <raulraja at gmail.com>:
> I think the problem is that everything is in a big transaction. Try
> splitting in smaller transactions
>
> On Jan 14, 2010 6:33 PM, "Defenestrator" <defenestrationism at gmail.com>
> wrote:
>
> How is memory managed in neo4j, does it page data out to disk like all
> database systems?
>
> Here's a sample program that consumes about 2.1gb of memory on my system
> (8gb of total memory) before failing with an OutOfMemoryError.
>
> Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
> at java.util.AbstractList.iterator(AbstractList.java:273)
>  at
> org.neo4j.impl.transaction.TransactionImpl.doBeforeCompletion(TransactionImpl.java:336)
> at org.neo4j.impl.transaction.TxManager.rollback(TxManager.java:725)
>  at
> org.neo4j.impl.transaction.TransactionImpl.rollback(TransactionImpl.java:114)
> at
> org.neo4j.api.core.EmbeddedNeoImpl$TransactionImpl.finish(EmbeddedNeoImpl.java:390)
>  at test.GraphTester.main(GraphTester.java:61)
>
> Below is the program, which tries to create 100K user nodes and 5.5M
> relationships between those user nodes based on a Zipian distribution of
> number of relationships per node limited to 1000 per user node.  It seems
> like this should work because this is well under billions of
> nodes/relationships.
>
> package test;
>
> import java.io.BufferedReader;
> import java.io.FileReader;
>
> import org.neo4j.api.core.EmbeddedNeo;
> import org.neo4j.api.core.NeoService;
> import org.neo4j.api.core.Node;
> import org.neo4j.api.core.Relationship;
> import org.neo4j.api.core.RelationshipType;
> import org.neo4j.api.core.Transaction;
>
> public final class GraphTester
> {
> public static void main(String[] args) throws Exception
> {
> int num_users = 100000;
>  NeoService neo = new EmbeddedNeo("var/neo");
> Transaction tx = neo.beginTx();
>  Node[] nodes = new Node[num_users];
>  try
>  {
> for (int i = 0, n = nodes.length; i < n; i++)
> {
>  nodes[i] = neo.createNode();
> nodes[i].setProperty("name", String.format("user%d", i));
>  if (i == 99999)
> System.out.println(nodes[i].getId());
>  }
> FileReader fr = new
> FileReader("/home/defenestrator/dev/python/zipf/100kusers.csv");
>  BufferedReader br = new BufferedReader(fr);
>  int node_idx = 0;
>  String s = null;
> while ((s = br.readLine()) != null)
> {
>  int num_neighbors = Integer.parseInt(s);
>  for(int i = 1; i <= num_neighbors; i++)
>  {
> int next_neighbor_idx = node_idx + (i * 7);
>  // handle wrap-around
> if (next_neighbor_idx > (num_users - 1))
>  next_neighbor_idx = next_neighbor_idx % num_users;
>  nodes[node_idx].createRelationshipTo(nodes[next_neighbor_idx],
>  MyRelationshipTypes.KNOWS);
> }
>  node_idx++;
> }
>  tx.success();
> }
> finally
>  {
> tx.finish();
> }
>  Node node1 = nodes[99999];
> System.out.println(node1.getProperty("name"));
>  System.out.println(node1.getId());
> Iterable<Relationship> knows = node1.getRelationships();
>  for (Relationship know : knows)
> {
> Node friend_node = know.getOtherNode(node1);
>  System.out.println("\t" + friend_node.getProperty("name"));
> }
>  neo.shutdown();
> }
>
> private static enum MyRelationshipTypes implements RelationshipType
> {
>  KNOWS
> }
> }
>
> _______________________________________________
> Neo mailing list
> User at lists.neo4j.org
> https://lists.neo4j.org/mailman/listinfo/user
> _______________________________________________
> Neo mailing list
> User at lists.neo4j.org
> https://lists.neo4j.org/mailman/listinfo/user
>


-- 
Mattias Persson, [mattias at neotechnology.com]
Neo Technology, www.neotechnology.com


More information about the User mailing list