[Neo] Java outof 64 GB ram
Miguel Ángel Águila Lorente
maguila at ac.upc.edu
Wed Feb 24 12:29:15 CET 2010
Around 57.000.000 nodes and 322.000.000 relationships. Now I'm trying
with the following configuration:
neostore.nodestore.db.mapped_memory=913M
neostore.relationshipstore.db.mapped_memory=11G
neostore.propertystore.db.mapped_memory=50M
neostore.propertystore.db.strings.mapped_memory=100M
neostore.propertystore.db.arrays.mapped_memory=0M
I think that I had understood wrong the neo.props file. Now I'm
expecting what happen.
El mié, 24-02-2010 a las 11:19 +0000, Rick Bullotta escribió:
> Hello, Miguel.
>
> Approximately how many nodes and relationships are in your graph database?
>
> Rick
>
>
>
>
> -----Original Message-----
> From: Miguel Ángel Águila Lorente <maguila at ac.upc.edu>
> Date: Wed, 24 Feb 2010 11:06:35
> To: Neo user discussions<user at lists.neo4j.org>
> Subject: Re: [Neo] Java outof 64 GB ram
>
> Hi,
>
> firstly I used the numbers of 9 bytes * number_of_nodes and 33 bytes *
> number_of_relations for the neo.props configure, but it doesn't work.
>
> After that I tried with my actual configuration:
>
> neostore.nodestore.db.mapped_memory=180G
> neostore.relationshipstore.db.mapped_memory=6000G
> neostore.propertystore.db.mapped_memory=50M
> neostore.propertystore.db.strings.mapped_memory=100M
> neostore.propertystore.db.arrays.mapped_memory=0M
>
> This are the numbers of my database:
>
> 488.2M neostore.nodestore.db
> 12.6G neostore.propertystore.db
> 67G neostore.propertystore.db.strings
> 9.9G neostore.relationshipstore.db
>
> In the page configuartion seetings the only thing (in my view) that
> could help me is the garbage collector but I've ever thought that
> garbage collector works when I need space and I have things to remove,
> therefore I'm not sure if this information can help me.
>
> El mié, 24-02-2010 a las 10:38 +0100, Johan Svensson escribió:
> > Hi,
> >
> > What is the error/problem you get when executing the program?
> >
> > It is not a good idea to use all available RAM for the Java heap. Have
> > a look at http://wiki.neo4j.org/content/Configuration_Settings for
> > more information on how to configure Neo4j to make good use of
> > available RAM.
> >
> > Regards,
> > -Johan
> >
> > On Tue, Feb 23, 2010 at 6:11 PM, Miguel Angel Aguila <maguila at ac.upc.edu> wrote:
> > > Yes I give 64 GB and only 60 GB in diferent executions.
> > >
> > > 2010/2/23 Mattias Persson <mattias at neotechnology.com>
> > >
> > >> One more thing... how much heap have you given the JVM? You control
> > >> how much RAM your JVM gets with the -Xmx option to the java command.
> > >> F.ex:
> > >>
> > >> java -Xmx10G -cp .....
> > >>
> > >> Please see more information regarding performance and tweaking at
> > >> http://wiki.neo4j.org/content/Neo4j_Performance_Guide
> > >>
> > >> 2010/2/23 Mattias Persson <mattias at neotechnology.com>:
> > >> > I see that you're using an older version of Neo4j. Have you tried using
> > >> 1.0?
> > >> >
> > >> > http://components.neo4j.org/neo4j-kernel
> > >> > http://neo4j.org/download
> > >> >
> > >> > Maven
> > >> > groupId: org.neo4j
> > >> > artifactId: neo4j-kernel
> > >> > version: 1.0
> > >> >
> > >> > ...and at first glance I don't see any direct problem with your code.
> > >> >
> > >> > 2010/2/23 Miguel Ángel Águila <maguila at ac.upc.edu>:
> > >> >> Hello,
> > >> >>
> > >> >> I'm doing a code to walk around all the nodes and get the node with the
> > >> >> maximum out degree. The main problem is that there are around 57 million
> > >> >> of nodes and 322 million of edges in the neo database, and using a 64 GB
> > >> >> RAM is insufficient to execute the program. It is the code:
> > >> >> Can anyone see what I'm doing wrong? I don't understand.
> > >> >>
> > >> >> Thanks.
> > >> >> Mike
> > >> >>
> > >> >>
> > >> >>
> > >> >> public static void getNodeMaxOutDegreeNeo(final NeoService neo,
> > >> >> final IndexService indexService) {
> > >> >> Node idNode;
> > >> >> long degree;
> > >> >> long maxDegree = 0;
> > >> >> IndexHits<Node> nodes;
> > >> >> long idDegree;
> > >> >> long idMaxDegree;
> > >> >> Transaction tx = neo.beginTx();
> > >> >> try {
> > >> >> nodes = indexService.getNodes("TYPE", "titles");
> > >> >> Iterator<Node> it = nodes.iterator();
> > >> >> Node idNodeMaxDegree=it.next();
> > >> >> Iterator<Relationship> relIterator =
> > >> >> idNodeMaxDegree.getRelationships(
> > >> >> NeoDataBase.MyRelationshipTypes.REF,
> > >> >> Direction.OUTGOING).iterator();
> > >> >>
> > >> >> while (relIterator.hasNext()) {
> > >> >> relIterator.next();
> > >> >> maxDegree++;
> > >> >> }
> > >> >> long counter = 0;
> > >> >> while (it.hasNext()) {
> > >> >> if ( ++counter % 50000 == 0 ) {
> > >> >> tx.success();
> > >> >> tx.finish();
> > >> >> tx = neo.beginTx();
> > >> >> }
> > >> >> idNode = it.next();
> > >> >> relIterator =
> > >> >> idNode.getRelationships(NeoDataBase.MyRelationshipTypes.REF,
> > >> >> Direction.OUTGOING).iterator();
> > >> >> degree = 0;
> > >> >> while (relIterator.hasNext()) {
> > >> >> relIterator.next();
> > >> >> degree++;
> > >> >> }
> > >> >> if (degree >= maxDegree) {
> > >> >> if(degree==maxDegree) {
> > >> >> idDegree=
> > >> >> Long.getLong((String)idNode.getProperty("ID_TITLE"));
> > >> >> idMaxDegree=
> > >> >> Long.getLong((String)idNodeMaxDegree.getProperty("ID_TITLE"));
> > >> >> if(idDegree>idMaxDegree) {
> > >> >> idNodeMaxDegree = idNode;
> > >> >> }
> > >> >> }
> > >> >> else {
> > >> >> maxDegree = degree;
> > >> >> idNodeMaxDegree = idNode;
> > >> >> }
> > >> >> }
> > >> >> }
> > >> >>
> > >> >> System.out.println("OId =
> > >> >> "+(String)idNodeMaxDegree.getProperty("ID_TITLE"));
> > >> >> System.out.println("Title =
> > >> >> "+(String)idNodeMaxDegree.getProperty("NAME"));
> > >> >> System.out.println("#refs = "+maxDegree);
> > >> >> tx.success();
> > >> >> }
> > >> >> finally
> > >> >> {
> > >> >> tx.finish();
> > >> >> }
> > >> >> }
> > _______________________________________________
> > Neo mailing list
> > User at lists.neo4j.org
> > https://lists.neo4j.org/mailman/listinfo/user
>
>
> _______________________________________________
> Neo mailing list
> User at lists.neo4j.org
> https://lists.neo4j.org/mailman/listinfo/user
> _______________________________________________
> Neo mailing list
> User at lists.neo4j.org
> https://lists.neo4j.org/mailman/listinfo/user
More information about the User
mailing list