[Neo] Java outof 64 GB ram

Rick Bullotta rick.bullotta at burningskysoftware.com
Wed Feb 24 12:19:39 CET 2010


Hello, Miguel. 

Approximately how many nodes and relationships are in your graph database? 

Rick 


 

-----Original Message-----
From: Miguel Ángel Águila Lorente <maguila at ac.upc.edu>
Date: Wed, 24 Feb 2010 11:06:35 
To: Neo user discussions<user at lists.neo4j.org>
Subject: Re: [Neo] Java outof 64 GB ram

Hi,

firstly I used the numbers of 9 bytes * number_of_nodes and 33 bytes *
number_of_relations for the neo.props configure, but it doesn't work.

After that I tried with my actual configuration:

neostore.nodestore.db.mapped_memory=180G
neostore.relationshipstore.db.mapped_memory=6000G
neostore.propertystore.db.mapped_memory=50M
neostore.propertystore.db.strings.mapped_memory=100M
neostore.propertystore.db.arrays.mapped_memory=0M

This are the numbers of my database:

488.2M neostore.nodestore.db
12.6G neostore.propertystore.db
67G neostore.propertystore.db.strings
9.9G neostore.relationshipstore.db

In the page configuartion seetings the only thing (in my view) that
could help me is the garbage collector but I've ever thought that
garbage collector works when I need space and I have things to remove,
therefore I'm not sure if this information can help me.

El mié, 24-02-2010 a las 10:38 +0100, Johan Svensson escribió:
> Hi,
> 
> What is the error/problem you get when executing the program?
> 
> It is not a good idea to use all available RAM for the Java heap. Have
> a look at http://wiki.neo4j.org/content/Configuration_Settings for
> more information on how to configure Neo4j to make good use of
> available RAM.
> 
> Regards,
> -Johan
> 
> On Tue, Feb 23, 2010 at 6:11 PM, Miguel Angel Aguila <maguila at ac.upc.edu> wrote:
> > Yes I give 64 GB and only 60 GB in diferent executions.
> >
> > 2010/2/23 Mattias Persson <mattias at neotechnology.com>
> >
> >> One more thing... how much heap have you given the JVM? You control
> >> how much RAM your JVM gets with the -Xmx option to the java command.
> >> F.ex:
> >>
> >>  java -Xmx10G -cp .....
> >>
> >> Please see more information regarding performance and tweaking at
> >> http://wiki.neo4j.org/content/Neo4j_Performance_Guide
> >>
> >> 2010/2/23 Mattias Persson <mattias at neotechnology.com>:
> >> > I see that you're using an older version of Neo4j. Have you tried using
> >> 1.0?
> >> >
> >> > http://components.neo4j.org/neo4j-kernel
> >> > http://neo4j.org/download
> >> >
> >> > Maven
> >> > groupId: org.neo4j
> >> > artifactId: neo4j-kernel
> >> > version: 1.0
> >> >
> >> > ...and at first glance I don't see any direct problem with your code.
> >> >
> >> > 2010/2/23 Miguel Ángel Águila <maguila at ac.upc.edu>:
> >> >> Hello,
> >> >>
> >> >> I'm doing a code to walk around all the nodes and get the node with the
> >> >> maximum out degree. The main problem is that there are around 57 million
> >> >> of nodes and 322 million of edges in the neo database, and using a 64 GB
> >> >> RAM is insufficient to execute the program. It is the code:
> >> >> Can anyone see what I'm doing wrong? I don't understand.
> >> >>
> >> >> Thanks.
> >> >> Mike
> >> >>
> >> >>
> >> >>
> >> >> public static void getNodeMaxOutDegreeNeo(final NeoService neo,
> >> >>            final IndexService indexService) {
> >> >>        Node idNode;
> >> >>        long degree;
> >> >>        long maxDegree = 0;
> >> >>        IndexHits<Node> nodes;
> >> >>        long idDegree;
> >> >>        long idMaxDegree;
> >> >>        Transaction tx = neo.beginTx();
> >> >>        try {
> >> >>            nodes = indexService.getNodes("TYPE", "titles");
> >> >>            Iterator<Node> it = nodes.iterator();
> >> >>            Node idNodeMaxDegree=it.next();
> >> >>            Iterator<Relationship> relIterator =
> >> >> idNodeMaxDegree.getRelationships(
> >> >>                           NeoDataBase.MyRelationshipTypes.REF,
> >> >> Direction.OUTGOING).iterator();
> >> >>
> >> >>            while (relIterator.hasNext()) {
> >> >>                relIterator.next();
> >> >>                maxDegree++;
> >> >>            }
> >> >>            long counter = 0;
> >> >>            while (it.hasNext()) {
> >> >>                if ( ++counter % 50000 == 0 ) {
> >> >>                    tx.success();
> >> >>                    tx.finish();
> >> >>                    tx = neo.beginTx();
> >> >>                }
> >> >>                idNode = it.next();
> >> >>                relIterator =
> >> >> idNode.getRelationships(NeoDataBase.MyRelationshipTypes.REF,
> >> >> Direction.OUTGOING).iterator();
> >> >>                degree = 0;
> >> >>                while (relIterator.hasNext()) {
> >> >>                    relIterator.next();
> >> >>                    degree++;
> >> >>                }
> >> >>                if (degree >= maxDegree) {
> >> >>                    if(degree==maxDegree) {
> >> >>                        idDegree=
> >> >> Long.getLong((String)idNode.getProperty("ID_TITLE"));
> >> >>                        idMaxDegree=
> >> >> Long.getLong((String)idNodeMaxDegree.getProperty("ID_TITLE"));
> >> >>                        if(idDegree>idMaxDegree) {
> >> >>                            idNodeMaxDegree = idNode;
> >> >>                        }
> >> >>                    }
> >> >>                    else {
> >> >>                        maxDegree = degree;
> >> >>                        idNodeMaxDegree = idNode;
> >> >>                    }
> >> >>                }
> >> >>            }
> >> >>
> >> >>            System.out.println("OId   =
> >> >> "+(String)idNodeMaxDegree.getProperty("ID_TITLE"));
> >> >>            System.out.println("Title =
> >> >> "+(String)idNodeMaxDegree.getProperty("NAME"));
> >> >>            System.out.println("#refs = "+maxDegree);
> >> >>            tx.success();
> >> >>        }
> >> >>        finally
> >> >>        {
> >> >>            tx.finish();
> >> >>        }
> >> >> }
> _______________________________________________
> Neo mailing list
> User at lists.neo4j.org
> https://lists.neo4j.org/mailman/listinfo/user


_______________________________________________
Neo mailing list
User at lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user


More information about the User mailing list