[Neo] Java outof 64 GB ram

rick.bullotta at burningskysoftware.com rick.bullotta at burningskysoftware.com
Thu Feb 25 14:55:50 CET 2010


   The string propertystore file seems really, really large to me, based
   on the # of nodes/relationships/properties.  The reason I ask is that I
   saw similar behavior in some of our early testing.  Extremely large
   string propertystores file.  It would be helpful to understand why the
   file gets that large, and whether it is related to some of the memory
   stress issues.





   -------- Original Message --------
   Subject: Re: [Neo] Java outof 64 GB ram
   From: Miguel Ãngel_Ãguila Lorente <maguila at ac.upc.edu>
   Date: Thu, February 25, 2010 1:57 am
   To: Neo user discussions <user at lists.neo4j.org>
   There are 322 million relationships, no 57 million relationships. The
   only propierty I save in the relationship is a String. In this 322
   million relationships there are 3 diferents types of relations.
   Do you think that 13 GB is a strange number? Why?
   Mike
   El mié, 24-02-2010 a las 08:36 -0500, Rick Bullotta escribió:
   > I guess the bigger question to me is why 57 million relationships
   would require 13 GB for the propertystore file and 67 GB for the string
   propertystore.
   >
   > Something doesn't seem right. Can you describe what types of
   properties you're storing on each node/relationship?
   >
   > Rick
   >
   > -----Original Message-----
   > From: user-bounces at lists.neo4j.org
   [[1]mailto:user-bounces at lists.neo4j.org] On Behalf Of Miguel Ãngel
   Ãguila Lorente
   > Sent: Wednesday, February 24, 2010 6:29 AM
   > To: Neo user discussions
   > Subject: Re: [Neo] Java outof 64 GB ram
   >
   > Around 57.000.000 nodes and 322.000.000 relationships. Now I'm trying
   > with the following configuration:
   >
   > neostore.nodestore.db.mapped_memory=913M
   > neostore.relationshipstore.db.mapped_memory=11G
   > neostore.propertystore.db.mapped_memory=50M
   > neostore.propertystore.db.strings.mapped_memory=100M
   > neostore.propertystore.db.arrays.mapped_memory=0M
   >
   > I think that I had understood wrong the neo.props file. Now I'm
   > expecting what happen.
   >
   > El mié, 24-02-2010 a las 11:19 +0000, Rick Bullotta escribió:
   > > Hello, Miguel.
   > >
   > > Approximately how many nodes and relationships are in your graph
   database?
   > >
   > > Rick
   > >
   > >
   > >
   > >
   > > -----Original Message-----
   > > From: Miguel Ãngel Ãguila Lorente <maguila at ac.upc.edu>
   > > Date: Wed, 24 Feb 2010 11:06:35
   > > To: Neo user discussions<user at lists.neo4j.org>
   > > Subject: Re: [Neo] Java outof 64 GB ram
   > >
   > > Hi,
   > >
   > > firstly I used the numbers of 9 bytes * number_of_nodes and 33
   bytes *
   > > number_of_relations for the neo.props configure, but it doesn't
   work.
   > >
   > > After that I tried with my actual configuration:
   > >
   > > neostore.nodestore.db.mapped_memory=180G
   > > neostore.relationshipstore.db.mapped_memory=6000G
   > > neostore.propertystore.db.mapped_memory=50M
   > > neostore.propertystore.db.strings.mapped_memory=100M
   > > neostore.propertystore.db.arrays.mapped_memory=0M
   > >
   > > This are the numbers of my database:
   > >
   > > 488.2M neostore.nodestore.db
   > > 12.6G neostore.propertystore.db
   > > 67G neostore.propertystore.db.strings
   > > 9.9G neostore.relationshipstore.db
   > >
   > > In the page configuartion seetings the only thing (in my view) that
   > > could help me is the garbage collector but I've ever thought that
   > > garbage collector works when I need space and I have things to
   remove,
   > > therefore I'm not sure if this information can help me.
   > >
   > > El mié, 24-02-2010 a las 10:38 +0100, Johan Svensson escribió:
   > > > Hi,
   > > >
   > > > What is the error/problem you get when executing the program?
   > > >
   > > > It is not a good idea to use all available RAM for the Java heap.
   Have
   > > > a look at [2]http://wiki.neo4j.org/content/Configuration_Settings
   for
   > > > more information on how to configure Neo4j to make good use of
   > > > available RAM.
   > > >
   > > > Regards,
   > > > -Johan
   > > >
   > > > On Tue, Feb 23, 2010 at 6:11 PM, Miguel Angel Aguila
   <maguila at ac.upc.edu> wrote:
   > > > > Yes I give 64 GB and only 60 GB in diferent executions.
   > > > >
   > > > > 2010/2/23 Mattias Persson <mattias at neotechnology.com>
   > > > >
   > > > >> One more thing... how much heap have you given the JVM? You
   control
   > > > >> how much RAM your JVM gets with the -Xmx option to the java
   command.
   > > > >> F.ex:
   > > > >>
   > > > >> java -Xmx10G -cp .....
   > > > >>
   > > > >> Please see more information regarding performance and tweaking
   at
   > > > >> [3]http://wiki.neo4j.org/content/Neo4j_Performance_Guide
   > > > >>
   > > > >> 2010/2/23 Mattias Persson <mattias at neotechnology.com>:
   > > > >> > I see that you're using an older version of Neo4j. Have you
   tried using
   > > > >> 1.0?
   > > > >> >
   > > > >> > [4]http://components.neo4j.org/neo4j-kernel
   > > > >> > [5]http://neo4j.org/download
   > > > >> >
   > > > >> > Maven
   > > > >> > groupId: org.neo4j
   > > > >> > artifactId: neo4j-kernel
   > > > >> > version: 1.0
   > > > >> >
   > > > >> > ...and at first glance I don't see any direct problem with
   your code.
   > > > >> >
   > > > >> > 2010/2/23 Miguel Ãngel Ãguila <maguila at ac.upc.edu>:
   > > > >> >> Hello,
   > > > >> >>
   > > > >> >> I'm doing a code to walk around all the nodes and get the
   node with the
   > > > >> >> maximum out degree. The main problem is that there are
   around 57 million
   > > > >> >> of nodes and 322 million of edges in the neo database, and
   using a 64 GB
   > > > >> >> RAM is insufficient to execute the program. It is the code:
   > > > >> >> Can anyone see what I'm doing wrong? I don't understand.
   > > > >> >>
   > > > >> >> Thanks.
   > > > >> >> Mike
   > > > >> >>
   > > > >> >>
   > > > >> >>
   > > > >> >> public static void getNodeMaxOutDegreeNeo(final NeoService
   neo,
   > > > >> >> final IndexService indexService) {
   > > > >> >> Node idNode;
   > > > >> >> long degree;
   > > > >> >> long maxDegree = 0;
   > > > >> >> IndexHits<Node> nodes;
   > > > >> >> long idDegree;
   > > > >> >> long idMaxDegree;
   > > > >> >> Transaction tx = neo.beginTx();
   > > > >> >> try {
   > > > >> >> nodes = indexService.getNodes("TYPE", "titles");
   > > > >> >> Iterator<Node> it = nodes.iterator();
   > > > >> >> Node idNodeMaxDegree=it.next();
   > > > >> >> Iterator<Relationship> relIterator =
   > > > >> >> idNodeMaxDegree.getRelationships(
   > > > >> >> NeoDataBase.MyRelationshipTypes.REF,
   > > > >> >> Direction.OUTGOING).iterator();
   > > > >> >>
   > > > >> >> while (relIterator.hasNext()) {
   > > > >> >> relIterator.next();
   > > > >> >> maxDegree++;
   > > > >> >> }
   > > > >> >> long counter = 0;
   > > > >> >> while (it.hasNext()) {
   > > > >> >> if ( ++counter % 50000 == 0 ) {
   > > > >> >> tx.success();
   > > > >> >> tx.finish();
   > > > >> >> tx = neo.beginTx();
   > > > >> >> }
   > > > >> >> idNode = it.next();
   > > > >> >> relIterator =
   > > > >> >>
   idNode.getRelationships(NeoDataBase.MyRelationshipTypes.REF,
   > > > >> >> Direction.OUTGOING).iterator();
   > > > >> >> degree = 0;
   > > > >> >> while (relIterator.hasNext()) {
   > > > >> >> relIterator.next();
   > > > >> >> degree++;
   > > > >> >> }
   > > > >> >> if (degree >= maxDegree) {
   > > > >> >> if(degree==maxDegree) {
   > > > >> >> idDegree=
   > > > >> >> Long.getLong((String)idNode.getProperty("ID_TITLE"));
   > > > >> >> idMaxDegree=
   > > > >> >>
   Long.getLong((String)idNodeMaxDegree.getProperty("ID_TITLE"));
   > > > >> >> if(idDegree>idMaxDegree) {
   > > > >> >> idNodeMaxDegree = idNode;
   > > > >> >> }
   > > > >> >> }
   > > > >> >> else {
   > > > >> >> maxDegree = degree;
   > > > >> >> idNodeMaxDegree = idNode;
   > > > >> >> }
   > > > >> >> }
   > > > >> >> }
   > > > >> >>
   > > > >> >> System.out.println("OId =
   > > > >> >> "+(String)idNodeMaxDegree.getProperty("ID_TITLE"));
   > > > >> >> System.out.println("Title =
   > > > >> >> "+(String)idNodeMaxDegree.getProperty("NAME"));
   > > > >> >> System.out.println("#refs = "+maxDegree);
   > > > >> >> tx.success();
   > > > >> >> }
   > > > >> >> finally
   > > > >> >> {
   > > > >> >> tx.finish();
   > > > >> >> }
   > > > >> >> }
   > > > _______________________________________________
   > > > Neo mailing list
   > > > User at lists.neo4j.org
   > > > [6]https://lists.neo4j.org/mailman/listinfo/user
   > >
   > >
   > > _______________________________________________
   > > Neo mailing list
   > > User at lists.neo4j.org
   > > [7]https://lists.neo4j.org/mailman/listinfo/user
   > > _______________________________________________
   > > Neo mailing list
   > > User at lists.neo4j.org
   > > [8]https://lists.neo4j.org/mailman/listinfo/user
   >
   >
   > _______________________________________________
   > Neo mailing list
   > User at lists.neo4j.org
   > [9]https://lists.neo4j.org/mailman/listinfo/user
   >
   > _______________________________________________
   > Neo mailing list
   > User at lists.neo4j.org
   > [10]https://lists.neo4j.org/mailman/listinfo/user
   _______________________________________________
   Neo mailing list
   User at lists.neo4j.org
   [11]https://lists.neo4j.org/mailman/listinfo/user

References

   1. http://email02.secureserver.net/#Compose
   2. http://wiki.neo4j.org/content/Configuration_Settings
   3. http://wiki.neo4j.org/content/Neo4j_Performance_Guide
   4. http://components.neo4j.org/neo4j-kernel
   5. http://neo4j.org/download
   6. https://lists.neo4j.org/mailman/listinfo/user
   7. https://lists.neo4j.org/mailman/listinfo/user
   8. https://lists.neo4j.org/mailman/listinfo/user
   9. https://lists.neo4j.org/mailman/listinfo/user
  10. https://lists.neo4j.org/mailman/listinfo/user
  11. https://lists.neo4j.org/mailman/listinfo/user


More information about the User mailing list