[Neo4j] Neo4j.py OS X issues

[Dan Gould]

Thanks Tobias.

I'm using Thrift with a TThreadedServer.  I wrote my own connection 
pooling/persistence.  It should be using just one connection which 
should be bound to the same thread--Thread 21.  (At least in this test!  
I plan to have lots of threads hitting Neo).  0 was just used to load 
things.

Though, if there was an exception that caused it to disconnect, it could 
reconnect on a new thread.  But I don't think that's what's happening in 
at least some of my segfaults.

Is there a function I can call to safely register a thread?  (Ideally 
one that can be re-called by the transaction logic.)  I could make my 
service.

Also, on startup, I load some data from all the nodes.  I didn't see a 
way to get all nodes from the Python API.  Could this code be unsafe in 
a way that causes thread issues down-the-line?

     def load_node_ids(self):
         logger.debug('Loading ids...')
         with self._graphdb.transaction:
             refid = self._graphdb.reference_node.id

             for n in 
self._graphdb._GraphDatabase__neo.getAllNodes().iterator():
                 if n.id!=refid:
                     node = self._graphdb.node[n.id]
                     # do some stuff

                     # Load corresponding relationships ids
                     for rel in node.relationships():
                         # do some stuff

(I'd also like to be able to clear all nodes in a similar way)

Would it be safer via JCC?



On 8/19/10 11:47 PM, Tobias Ivarsson wrote:
> For some strange reason I am unable to get the body of this message
> through to the mailing list. I'm now giving up and including Dan
> directly as a recipient so that at least he gets a response.
>
> On Fri, Aug 20, 2010 at 8:45 AM, Tobias Ivarsson
> <tobias.ivarsson at neotechnology.com>  wrote:
>    
>> Hi Dan,
>>
>>  From the segfault thread dump it looks like you are running at least 2
>> Python threads, is that correct? (Thread 0, and Thread 21)
>>
>>  From the numbering of the threads I'm guessing Thread 0 is the thread
>> that started the JVM (and Neo4j), and Thread 21 is the one that is
>> shutting down Neo4j. Has Thread 21 opened any transactions with Neo4j
>> before attempting to shut down Neo4j?
>>
>> When embedding a JVM through JNI (which is what JPype does) all
>> operating system threads that are not started by the JVM needs to
>> register with the JVM in order to be able to invoke any JVM methods.
>> JPype does not do this automatically, instead it provides functions
>> for the developer (me or you) to do this. I've added a check for new
>> threads, and registrations thereof, in the function that opens a new
>> transaction. This means that after opening a transaction (possibly
>> closing it right away) a new thread should be able to utilize the
>> Neo4j objects. If trying to interact with Neo4j without ever opening a
>> transaction before doing so, any new thread could cause severe
>> problems (of an undocumented kind. This usually means segfault).
>>
>> This is a bit of a wart, and I can't say with a 100% confidence that
>> this is what has happened to you, but it's a theory, and now that it's
>> presented to you, my hope is that you can be confident whether or not
>> this applies to your code.
>> Let me know if this was any help.
>>
>> Cheers,
>> Tobias
>>
>> On Wed, Aug 18, 2010 at 10:55 PM, Dan Gould<dlg at dangould.com>  wrote:
>>      
>>> Hi,
>>>
>>> I'm experimenting with Neo4j.py.  I want to completely clear my DB
>>> between tests in my test framework, which I do with shutdown(), remove
>>> the db dir, and restart (I've had problems removing all nodes; I'll
>>> discuss in a separate thread on the list once I put together a simple test).
>>>
>>> When I call shutdown() on the Graph Database, I get a Segmentation Fault.
>>>
>>> I want to be able to shut down safely, so I was wondering if there is a fix.
>>>
>>> I'm running with:
>>> - Neo4j.py (tried a few versions, including last night's svn HEAD; I
>>> also tried changing pom.xml to use Neo4j 1.1)
>>> - CPython: Python 2.6.5 (r265:79063, May 17 2010, 14:26:12)  [GCC 4.2.1
>>> (Apple Inc. build 5646)] on darwin
>>> - I have both JPype-0.5.4.1 and JCC-2.6 installed in my virtualenv (I
>>> also tried with only JPype installed)
>>> - OS X 10.6.3
>>> - Java Version "1.6.0_20"; Java(TM) SE Runtime Environment (build
>>> 1.6.0_20-b02-279-10M3065); Java HotSpot(TM) 64-Bit Server VM (build
>>> 16.3-b01-279, mixed mode)
>>>
>>> (I can also get a segfault if writing outside a transaction.)
>>>
>>>        
>> ---8<---<cut out long stack trace/>  ---8<---
>> --
>> Tobias Ivarsson<tobias.ivarsson at neotechnology.com>
>> Hacker, Neo Technology
>> www.neotechnology.com
>> Cellphone: +46 706 534857
>>
>>      
>
>
>