[Neo4j] Configuring Neo4J via Python bindings

Jacob Hansson jacob.hansson at neotechnology.com
Tue Oct 18 09:35:29 CEST 2011


It looks like our build pipeline is clogged up right now. We're looking into
it, but it might take a while..

/jake

On Mon, Oct 17, 2011 at 9:56 PM, Chris Diehl <diehl at alumni.cmu.edu> wrote:

> Hey Jake,
>
> Awesome! Thanks so much for making this happen! On my end, it doesn't look
> like the docs are up yet. I'll check back...
>
> Thanks again, Chris
>
> On Mon, Oct 17, 2011 at 4:12 AM, Jacob Hansson <
> jacob.hansson at neotechnology.com> wrote:
>
> > Hey Chris,
> >
> > I've just added support for passing in extra JVM args, it's available as
> of
> > this commit:
> >
> >
> https://github.com/neo4j/python-embedded/commit/d41950f8c7e6c65d67f60868976e1864a496a2ea
> >
> > Documentation will pop up in an hour or so here:
> >
> http://docs.neo4j.org/chunked/snapshot/python-embedded-reference-core.html
> >
> > This feature will be available as of the 1.5 release, due later this
> week.
> > If you want to get access to it right away, you can build neo4j-embedded
> > directly from source, the code is available here:
> > https://github.com/neo4j/python-embedded
> >
> > See the readme for how to build from source.
> >
> >
> > Thanks for bringing this up!
> > Jake
> >
> > On Fri, Oct 14, 2011 at 7:47 PM, Chris Diehl <diehl at alumni.cmu.edu>
> wrote:
> >
> > > Hi Michael,
> > >
> > > I added a print statement to my code to output how many nodes have been
> > > inserted. The script zips along until it gets up around 18.6K nodes.
> Then
> > > the program halts when a Java exception is thrown, as shown below.
> > >
> > > Inserting user 18629
> > > Inserting user 18630
> > > Inserting user 18631
> > > Inserting user 18632
> > > Inserting user 18633
> > > Inserting user 18634
> > > Inserting user 18635
> > > Traceback (most recent call last):
> > >  File "jive_app_db_graph_construction.py", line 53, in <module>
> > >    node_idx['userid'][user_node['userid']] = user_node
> > >  File
> > >
> > >
> >
> "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/contextlib.py",
> > > line 34, in __exit__
> > >    self.gen.throw(type, value, traceback)
> > >  File "<string>", line 11, in transaction
> > > RuntimeError: An unknown error occured while handling a Java Exception
> > >
> > > Strangely enough, the script returned a more explicit Out of Heap
> Memory
> > > error once when I reran the script. It's bizarre that the failure
> output
> > is
> > > not consistent, given I start with a clean DB each time. Increasing the
> > > amount of available heap memory seems like the reasonable place to
> start.
> > > If
> > > you can let me know how to specify the amount of heap memory for the
> JVM
> > > instantiated by the bindings, that'd be great.
> > >
> > > I'm running on a MacBook Pro with 8 gigs of memory running OSX 10.6.8.
> > >
> > > Here's the contents of the messages.log file.
> > >
> > > 2011-10-14 10:34:17.336-0700: Creating new db @
> > /Users/chris.diehl/Graphs/
> > > JiveCommunityV5.0.ng/neostore
> > > 2011-10-14 10:34:17.419-0700: Opened [/Users/chris.diehl/Graphs/
> > > JiveCommunityV5.0.ng/nioneo_logical.log.1] clean empty log, version=0,
> > > lastTxId=1
> > > 2011-10-14 10:34:17.441-0700: Opened [/Users/chris.diehl/Graphs/
> > > JiveCommunityV5.0.ng/index/lucene.log.1] clean empty log, version=0,
> > > lastTxId=1
> > > 2011-10-14 10:34:17.443-0700: TM new log: tm_tx_log.1
> > > 2011-10-14 10:34:17.451-0700: --- CONFIGURATION START ---
> > > 2011-10-14 10:34:17.451-0700: Physical mem: 8192MB, Heap size: 123MB
> > > 2011-10-14 10:34:17.452-0700: Kernel version: Neo4j - Graph Database
> > Kernel
> > > 1.5-SNAPSHOT (revision: 1.5.M02-4-g3d1e9d8)
> > > 2011-10-14 10:34:17.453-0700: Neo4j - Graph Database Kernel
> 1.5-SNAPSHOT
> > > (revision: 1.5.M02-4-g3d1e9d8)
> > > 2011-10-14 10:34:17.456-0700: Operating System: Mac OS X; version:
> > 10.6.8;
> > > arch: x86_64; cpus: 8
> > > 2011-10-14 10:34:17.456-0700: VM Name: Java HotSpot(TM) 64-Bit Server
> VM
> > > 2011-10-14 10:34:17.456-0700: VM Vendor: Apple Inc.
> > > 2011-10-14 10:34:17.456-0700: VM Version: 20.1-b02-384
> > > 2011-10-14 10:34:17.456-0700: Boot Class Path:
> > >
> > >
> >
> /System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Classes/jsfd.jar:/System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Classes/classes.jar:/System/Library/Frameworks/JavaVM.framework/Frameworks/JavaRuntimeSupport.framework/Resources/Java/JavaRuntimeSupport.jar:/System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Classes/ui.jar:/System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Classes/laf.jar:/System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Classes/sunrsasign.jar:/System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Classes/jsse.jar:/System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Classes/jce.jar:/System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Classes/charsets.jar
> > > 2011-10-14 10:34:17.457-0700: Class Path:
> > >
> > >
> >
> /Users/chris.diehl/.virtualenvs/py26-neo4j-embedded/lib/python2.6/site-packages/neo4j/javalib/antlr-2.7.7.jar:/Users/chris.diehl/.virtualenvs/py26-neo4j-embedded/lib/python2.6/site-packages/neo4j/javalib/asm-3.2.jar:/Users/chris.diehl/.virtualenvs/py26-neo4j-embedded/lib/python2.6/site-packages/neo4j/javalib/asm-analysis-3.2.jar:/Users/chris.diehl/.virtualenvs/py26-neo4j-embedded/lib/python2.6/site-packages/neo4j/javalib/asm-commons-3.2.jar:/Users/chris.diehl/.virtualenvs/py26-neo4j-embedded/lib/python2.6/site-packages/neo4j/javalib/asm-tree-3.2.jar:/Users/chris.diehl/.virtualenvs/py26-neo4j-embedded/lib/python2.6/site-packages/neo4j/javalib/asm-util-3.2.jar:/Users/chris.diehl/.virtualenvs/py26-neo4j-embedded/lib/python2.6/site-packages/neo4j/javalib/blueprints-core-0.9.jar:/Users/chris.diehl/.virtualenvs/py26-neo4j-embedded/lib/python2.6/site-packages/neo4j/javalib/geronimo-jta_1.1_spec-1.1.1.jar:/Users/chris.diehl/.virtualenvs/py26-neo4j-embedded/lib/python2.6/site-packag
> >  es
> > >
> > >
> >
>  /neo4j/javalib/gremlin-1.2.jar:/Users/chris.diehl/.virtualenvs/py26-neo4j-embedded/lib/python2.6/site-packages/neo4j/javalib/groovy-1.8.1.jar:/Users/chris.diehl/.virtualenvs/py26-neo4j-embedded/lib/python2.6/site-packages/neo4j/javalib/jansi-1.5.jar:/Users/chris.diehl/.virtualenvs/py26-neo4j-embedded/lib/python2.6/site-packages/neo4j/javalib/jettison-1.3.jar:/Users/chris.diehl/.virtualenvs/py26-neo4j-embedded/lib/python2.6/site-packages/neo4j/javalib/lucene-core-3.1.0.jar:/Users/chris.diehl/.virtualenvs/py26-neo4j-embedded/lib/python2.6/site-packages/neo4j/javalib/neo4j-1.5-SNAPSHOT.jar:/Users/chris.diehl/.virtualenvs/py26-neo4j-embedded/lib/python2.6/site-packages/neo4j/javalib/neo4j-cypher-1.5-SNAPSHOT.jar:/Users/chris.diehl/.virtualenvs/py26-neo4j-embedded/lib/python2.6/site-packages/neo4j/javalib/neo4j-graph-algo-1.5-SNAPSHOT.jar:/Users/chris.diehl/.virtualenvs/py26-neo4j-embedded/lib/python2.6/site-packages/neo4j/javalib/neo4j-graph-matching-1.5-SNAPSHOT.jar:/Users/ch
> >  ri
> > >
> > >
> >
>  s.diehl/.virtualenvs/py26-neo4j-embedded/lib/python2.6/site-packages/neo4j/javalib/neo4j-jmx-1.5-SNAPSHOT.jar:/Users/chris.diehl/.virtualenvs/py26-neo4j-embedded/lib/python2.6/site-packages/neo4j/javalib/neo4j-kernel-1.5-SNAPSHOT.jar:/Users/chris.diehl/.virtualenvs/py26-neo4j-embedded/lib/python2.6/site-packages/neo4j/javalib/neo4j-lucene-index-1.5-SNAPSHOT.jar:/Users/chris.diehl/.virtualenvs/py26-neo4j-embedded/lib/python2.6/site-packages/neo4j/javalib/neo4j-udc-1.5-SNAPSHOT.jar:/Users/chris.diehl/.virtualenvs/py26-neo4j-embedded/lib/python2.6/site-packages/neo4j/javalib/pipes-0.7.jar:/Users/chris.diehl/.virtualenvs/py26-neo4j-embedded/lib/python2.6/site-packages/neo4j/javalib/scala-library-2.9.0-1.jar:/Users/chris.diehl/.virtualenvs/py26-neo4j-embedded/lib/python2.6/site-packages/neo4j/javalib/stax-api-1.0.1.jar
> > > 2011-10-14 10:34:17.457-0700: Library Path:
> > >
> > >
> >
> .:/Users/chris.diehl/Library/Java/Extensions:/Library/Java/Extensions:/System/Library/Java/Extensions:/usr/lib/java
> > > 2011-10-14 10:34:17.459-0700: Garbage Collector: ParNew: [Par Eden
> Space,
> > > Par Survivor Space]
> > > 2011-10-14 10:34:17.459-0700: Garbage Collector: ConcurrentMarkSweep:
> > [Par
> > > Eden Space, Par Survivor Space, CMS Old Gen, CMS Perm Gen]
> > > 2011-10-14 10:34:17.459-0700: VM Arguments: []
> > > 2011-10-14 10:34:17.459-0700:
> > > 2011-10-14 10:34:17.460-0700: create=true
> > > 2011-10-14 10:34:17.460-0700: logical_log=/Users/chris.diehl/Graphs/
> > > JiveCommunityV5.0.ng/nioneo_logical.log
> > > 2011-10-14 10:34:17.460-0700: neo_store=/Users/chris.diehl/Graphs/
> > > JiveCommunityV5.0.ng/neostore
> > > 2011-10-14 10:34:17.460-0700: neostore.nodestore.db.mapped_memory=20M
> > > 2011-10-14 10:34:17.460-0700:
> > > neostore.propertystore.db.arrays.mapped_memory=130M
> > > 2011-10-14 10:34:17.460-0700:
> > > neostore.propertystore.db.index.keys.mapped_memory=1M
> > > 2011-10-14 10:34:17.460-0700:
> > > neostore.propertystore.db.index.mapped_memory=1M
> > > 2011-10-14 10:34:17.460-0700:
> neostore.propertystore.db.mapped_memory=90M
> > > 2011-10-14 10:34:17.460-0700:
> > > neostore.propertystore.db.strings.mapped_memory=130M
> > > 2011-10-14 10:34:17.461-0700:
> > > neostore.relationshipstore.db.mapped_memory=100M
> > > 2011-10-14 10:34:17.461-0700: node_auto_indexing=false
> > > 2011-10-14 10:34:17.461-0700: rebuild_idgenerators_fast=true
> > > 2011-10-14 10:34:17.461-0700: relationship_auto_indexing=false
> > > 2011-10-14 10:34:17.461-0700: store_dir=/Users/chris.diehl/Graphs/
> > > JiveCommunityV5.0.ng
> > > 2011-10-14 10:34:17.461-0700: use_memory_mapped_buffers=true
> > > 2011-10-14 10:34:17.461-0700: --- CONFIGURATION END ---
> > > 2011-10-14 10:34:17.518-0700: Extension
> > > org.neo4j.kernel.KernelExtension[kernel jmx] loaded ok
> > > 2011-10-14 10:34:17.520-0700: Extension
> > > org.neo4j.kernel.KernelExtension[kernel udc] loaded ok
> > >
> > > Chris
> > >
> > > On Thu, Oct 13, 2011 at 8:37 PM, Michael Hunger <
> > > michael.hunger at neotechnology.com> wrote:
> > >
> > > > Hi Chris,
> > > >
> > > > What do you mean by "ground to a halt" ? Is it using all available
> CPU
> > or
> > > > heavy IO-waits (top, iostat, vmstat)?
> > > >
> > > > A thread dump (via jps + jstack ) would be great.
> > > >
> > > > You can probably pass in the config map into the  Graph database
> call.
> > > > Current config is logged to messages.log, please share that too as
> well
> > > as
> > > > your system parameters (OS, CPU, RAM, Type of disk).
> > > >
> > > > Michael
> > > >
> > > > Am 13.10.2011 um 23:52 schrieb Chris Diehl:
> > > >
> > > > > Hey All,
> > > > >
> > > > > I just tested a Python script I've been working on to construct a
> > > > > multi-relational graph in Neo4j. I am using the new Python bindings
> > for
> > > > > embedded Neo4j. After creating about 18K nodes, the database ground
> > to
> > > a
> > > > > halt.
> > > > >
> > > > > In the past, when using Gremlin, I would up the heap size for the
> JVM
> > > > when
> > > > > encountering problems. I'm not sure how to tweak settings for the
> JVM
> > > now
> > > > > that I'm using the Python bindings to fire up the database. Can I
> do
> > > that
> > > > > from the GraphDatabase() call? Or do I need to look elsewhere to
> > tweak
> > > > the
> > > > > JVM parameters?
> > > > >
> > > > > If you have any other suggestions about parameters I should change,
> > I'm
> > > > all
> > > > > ears.
> > > > >
> > > > > Thanks!
> > > > >
> > > > > Chris
> > > > >
> > > > > --
> > > > > View this message in context:
> > > >
> > >
> >
> http://neo4j-community-discussions.438527.n3.nabble.com/Configuring-Neo4J-via-Python-bindings-tp3419961p3419961.html
> > > > > Sent from the Neo4j Community Discussions mailing list archive at
> > > > Nabble.com.
> > > > > _______________________________________________
> > > > > Neo4j mailing list
> > > > > User at lists.neo4j.org
> > > > > https://lists.neo4j.org/mailman/listinfo/user
> > > >
> > > > _______________________________________________
> > > > Neo4j mailing list
> > > > User at lists.neo4j.org
> > > > https://lists.neo4j.org/mailman/listinfo/user
> > > >
> > > _______________________________________________
> > > Neo4j mailing list
> > > User at lists.neo4j.org
> > > https://lists.neo4j.org/mailman/listinfo/user
> > >
> >
> >
> >
> > --
> > Jacob Hansson
> > Phone: +46 (0) 763503395
> > Twitter: @jakewins
> > _______________________________________________
> > Neo4j mailing list
> > User at lists.neo4j.org
> > https://lists.neo4j.org/mailman/listinfo/user
> >
> _______________________________________________
> Neo4j mailing list
> User at lists.neo4j.org
> https://lists.neo4j.org/mailman/listinfo/user
>



-- 
Jacob Hansson
Phone: +46 (0) 763503395
Twitter: @jakewins


More information about the User mailing list