[Neo] Java outof 64 GB ram

Miguel Ángel Águila Lorente maguila at ac.upc.edu
Wed Feb 24 12:29:15 CET 2010


Around 57.000.000 nodes and 322.000.000 relationships. Now I'm trying
with the following configuration:

neostore.nodestore.db.mapped_memory=913M
neostore.relationshipstore.db.mapped_memory=11G
neostore.propertystore.db.mapped_memory=50M
neostore.propertystore.db.strings.mapped_memory=100M
neostore.propertystore.db.arrays.mapped_memory=0M

I think that I had understood wrong the neo.props file. Now I'm
expecting what happen.

El mié, 24-02-2010 a las 11:19 +0000, Rick Bullotta escribió:
> Hello, Miguel. 
> 
> Approximately how many nodes and relationships are in your graph database? 
> 
> Rick 
> 
> 
>  
> 
> -----Original Message-----
> From: Miguel Ángel Águila Lorente <maguila at ac.upc.edu>
> Date: Wed, 24 Feb 2010 11:06:35 
> To: Neo user discussions<user at lists.neo4j.org>
> Subject: Re: [Neo] Java outof 64 GB ram
> 
> Hi,
> 
> firstly I used the numbers of 9 bytes * number_of_nodes and 33 bytes *
> number_of_relations for the neo.props configure, but it doesn't work.
> 
> After that I tried with my actual configuration:
> 
> neostore.nodestore.db.mapped_memory=180G
> neostore.relationshipstore.db.mapped_memory=6000G
> neostore.propertystore.db.mapped_memory=50M
> neostore.propertystore.db.strings.mapped_memory=100M
> neostore.propertystore.db.arrays.mapped_memory=0M
> 
> This are the numbers of my database:
> 
> 488.2M neostore.nodestore.db
> 12.6G neostore.propertystore.db
> 67G neostore.propertystore.db.strings
> 9.9G neostore.relationshipstore.db
> 
> In the page configuartion seetings the only thing (in my view) that
> could help me is the garbage collector but I've ever thought that
> garbage collector works when I need space and I have things to remove,
> therefore I'm not sure if this information can help me.
> 
> El mié, 24-02-2010 a las 10:38 +0100, Johan Svensson escribió:
> > Hi,
> > 
> > What is the error/problem you get when executing the program?
> > 
> > It is not a good idea to use all available RAM for the Java heap. Have
> > a look at http://wiki.neo4j.org/content/Configuration_Settings for
> > more information on how to configure Neo4j to make good use of
> > available RAM.
> > 
> > Regards,
> > -Johan
> > 
> > On Tue, Feb 23, 2010 at 6:11 PM, Miguel Angel Aguila <maguila at ac.upc.edu> wrote:
> > > Yes I give 64 GB and only 60 GB in diferent executions.
> > >
> > > 2010/2/23 Mattias Persson <mattias at neotechnology.com>
> > >
> > >> One more thing... how much heap have you given the JVM? You control
> > >> how much RAM your JVM gets with the -Xmx option to the java command.
> > >> F.ex:
> > >>
> > >>  java -Xmx10G -cp .....
> > >>
> > >> Please see more information regarding performance and tweaking at
> > >> http://wiki.neo4j.org/content/Neo4j_Performance_Guide
> > >>
> > >> 2010/2/23 Mattias Persson <mattias at neotechnology.com>:
> > >> > I see that you're using an older version of Neo4j. Have you tried using
> > >> 1.0?
> > >> >
> > >> > http://components.neo4j.org/neo4j-kernel
> > >> > http://neo4j.org/download
> > >> >
> > >> > Maven
> > >> > groupId: org.neo4j
> > >> > artifactId: neo4j-kernel
> > >> > version: 1.0
> > >> >
> > >> > ...and at first glance I don't see any direct problem with your code.
> > >> >
> > >> > 2010/2/23 Miguel Ángel Águila <maguila at ac.upc.edu>:
> > >> >> Hello,
> > >> >>
> > >> >> I'm doing a code to walk around all the nodes and get the node with the
> > >> >> maximum out degree. The main problem is that there are around 57 million
> > >> >> of nodes and 322 million of edges in the neo database, and using a 64 GB
> > >> >> RAM is insufficient to execute the program. It is the code:
> > >> >> Can anyone see what I'm doing wrong? I don't understand.
> > >> >>
> > >> >> Thanks.
> > >> >> Mike
> > >> >>
> > >> >>
> > >> >>
> > >> >> public static void getNodeMaxOutDegreeNeo(final NeoService neo,
> > >> >>            final IndexService indexService) {
> > >> >>        Node idNode;
> > >> >>        long degree;
> > >> >>        long maxDegree = 0;
> > >> >>        IndexHits<Node> nodes;
> > >> >>        long idDegree;
> > >> >>        long idMaxDegree;
> > >> >>        Transaction tx = neo.beginTx();
> > >> >>        try {
> > >> >>            nodes = indexService.getNodes("TYPE", "titles");
> > >> >>            Iterator<Node> it = nodes.iterator();
> > >> >>            Node idNodeMaxDegree=it.next();
> > >> >>            Iterator<Relationship> relIterator =
> > >> >> idNodeMaxDegree.getRelationships(
> > >> >>                           NeoDataBase.MyRelationshipTypes.REF,
> > >> >> Direction.OUTGOING).iterator();
> > >> >>
> > >> >>            while (relIterator.hasNext()) {
> > >> >>                relIterator.next();
> > >> >>                maxDegree++;
> > >> >>            }
> > >> >>            long counter = 0;
> > >> >>            while (it.hasNext()) {
> > >> >>                if ( ++counter % 50000 == 0 ) {
> > >> >>                    tx.success();
> > >> >>                    tx.finish();
> > >> >>                    tx = neo.beginTx();
> > >> >>                }
> > >> >>                idNode = it.next();
> > >> >>                relIterator =
> > >> >> idNode.getRelationships(NeoDataBase.MyRelationshipTypes.REF,
> > >> >> Direction.OUTGOING).iterator();
> > >> >>                degree = 0;
> > >> >>                while (relIterator.hasNext()) {
> > >> >>                    relIterator.next();
> > >> >>                    degree++;
> > >> >>                }
> > >> >>                if (degree >= maxDegree) {
> > >> >>                    if(degree==maxDegree) {
> > >> >>                        idDegree=
> > >> >> Long.getLong((String)idNode.getProperty("ID_TITLE"));
> > >> >>                        idMaxDegree=
> > >> >> Long.getLong((String)idNodeMaxDegree.getProperty("ID_TITLE"));
> > >> >>                        if(idDegree>idMaxDegree) {
> > >> >>                            idNodeMaxDegree = idNode;
> > >> >>                        }
> > >> >>                    }
> > >> >>                    else {
> > >> >>                        maxDegree = degree;
> > >> >>                        idNodeMaxDegree = idNode;
> > >> >>                    }
> > >> >>                }
> > >> >>            }
> > >> >>
> > >> >>            System.out.println("OId   =
> > >> >> "+(String)idNodeMaxDegree.getProperty("ID_TITLE"));
> > >> >>            System.out.println("Title =
> > >> >> "+(String)idNodeMaxDegree.getProperty("NAME"));
> > >> >>            System.out.println("#refs = "+maxDegree);
> > >> >>            tx.success();
> > >> >>        }
> > >> >>        finally
> > >> >>        {
> > >> >>            tx.finish();
> > >> >>        }
> > >> >> }
> > _______________________________________________
> > Neo mailing list
> > User at lists.neo4j.org
> > https://lists.neo4j.org/mailman/listinfo/user
> 
> 
> _______________________________________________
> Neo mailing list
> User at lists.neo4j.org
> https://lists.neo4j.org/mailman/listinfo/user
> _______________________________________________
> Neo mailing list
> User at lists.neo4j.org
> https://lists.neo4j.org/mailman/listinfo/user




More information about the User mailing list