[Neo] Memory management in neo4j?

Raul Raja Martinez raulraja at gmail.com
Fri Jan 15 04:17:16 CET 2010


I think the problem is that everything is in a big transaction. Try
splitting in smaller transactions

On Jan 14, 2010 6:33 PM, "Defenestrator" <defenestrationism at gmail.com>
wrote:

How is memory managed in neo4j, does it page data out to disk like all
database systems?

Here's a sample program that consumes about 2.1gb of memory on my system
(8gb of total memory) before failing with an OutOfMemoryError.

Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
at java.util.AbstractList.iterator(AbstractList.java:273)
 at
org.neo4j.impl.transaction.TransactionImpl.doBeforeCompletion(TransactionImpl.java:336)
at org.neo4j.impl.transaction.TxManager.rollback(TxManager.java:725)
 at
org.neo4j.impl.transaction.TransactionImpl.rollback(TransactionImpl.java:114)
at
org.neo4j.api.core.EmbeddedNeoImpl$TransactionImpl.finish(EmbeddedNeoImpl.java:390)
 at test.GraphTester.main(GraphTester.java:61)

Below is the program, which tries to create 100K user nodes and 5.5M
relationships between those user nodes based on a Zipian distribution of
number of relationships per node limited to 1000 per user node.  It seems
like this should work because this is well under billions of
nodes/relationships.

package test;

import java.io.BufferedReader;
import java.io.FileReader;

import org.neo4j.api.core.EmbeddedNeo;
import org.neo4j.api.core.NeoService;
import org.neo4j.api.core.Node;
import org.neo4j.api.core.Relationship;
import org.neo4j.api.core.RelationshipType;
import org.neo4j.api.core.Transaction;

public final class GraphTester
{
public static void main(String[] args) throws Exception
{
int num_users = 100000;
 NeoService neo = new EmbeddedNeo("var/neo");
Transaction tx = neo.beginTx();
 Node[] nodes = new Node[num_users];
 try
 {
for (int i = 0, n = nodes.length; i < n; i++)
{
 nodes[i] = neo.createNode();
nodes[i].setProperty("name", String.format("user%d", i));
 if (i == 99999)
System.out.println(nodes[i].getId());
 }
FileReader fr = new
FileReader("/home/defenestrator/dev/python/zipf/100kusers.csv");
 BufferedReader br = new BufferedReader(fr);
 int node_idx = 0;
 String s = null;
while ((s = br.readLine()) != null)
{
 int num_neighbors = Integer.parseInt(s);
 for(int i = 1; i <= num_neighbors; i++)
 {
int next_neighbor_idx = node_idx + (i * 7);
 // handle wrap-around
if (next_neighbor_idx > (num_users - 1))
 next_neighbor_idx = next_neighbor_idx % num_users;
 nodes[node_idx].createRelationshipTo(nodes[next_neighbor_idx],
 MyRelationshipTypes.KNOWS);
}
 node_idx++;
}
 tx.success();
}
finally
 {
tx.finish();
}
 Node node1 = nodes[99999];
System.out.println(node1.getProperty("name"));
 System.out.println(node1.getId());
Iterable<Relationship> knows = node1.getRelationships();
 for (Relationship know : knows)
{
Node friend_node = know.getOtherNode(node1);
 System.out.println("\t" + friend_node.getProperty("name"));
}
 neo.shutdown();
}

private static enum MyRelationshipTypes implements RelationshipType
{
 KNOWS
}
}

_______________________________________________
Neo mailing list
User at lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user


More information about the User mailing list