[Neo] Some feedback on ZooKeeper use in Neo4j "zha".

[Patrick Hunt]

Hi, I'm Patrick (http://twitter.com/phunt) from the ZooKeeper team. 
Peter Neubauer brought to my attention today that you are considering 
use of ZooKeeper in Neo4j, that's great! I took a quick look at the code 
you currently have in SVN and wanted to provide a bit of feedback.

I don't know your domain requirements but in general the mechanics of 
ZooClient use look fine.

The use of 5second timeout is fine. This allows you to detect a client 
(zk client) failure after just 5 seconds. So if the node/process crashes 
you'd identify this after 5 seconds, same if a network connection fails, 
etc... One thing you may not have considered though, is that anything 
that causes the client to not be able to heartbeat to the server would 
also cause the session to be expired (sessions are expired when the zk 
cluster fails to hear from the client w/in the "timeout" time) - so long 
GC pauses could trigger this as well. In 1.6.x jvms we've seen that the 
GC can pause all threads for very long periods (in some cases with hbase 
we saw 4 minute pauses for gc). HBase was the first to see this, we 
worked with Solr early on to help them understand this issue as well. 
The problem can be alleviated somewhat by using the CMS/incremental GC 
options in the JVM, however it cannot be eliminated entirely (in some 
cases the Gc will still drop back to parallel). You need to consider the 
impact of GC on your domain and how to best handle it.

See this JIRA for details on our discussion with Solr, you might gain 
some good insight:http://bit.ly/d7OSQ1 
https://issues.apache.org/jira/browse/SOLR-1277

I did notice that ZooClient is not handling disconnection and expiration 
of the session in the "process" method. At the very least you need to 
handle the expiration, you may need to do something for disconnection, 
but this depends on whether you have active or passive actors (masters). 
Here's a good link on session lifecycle:
http://hadoop.apache.org/zookeeper/docs/current/zookeeperProgrammers.html#ch_zkSessions


You might also want to setup a wiki page similar to these at some point, 
it would help us with future discussion, feedback and provide insight 
for devs/users:
http://wiki.apache.org/hadoop/ZooKeeper/HBaseAndZooKeeper
http://wiki.apache.org/solr/ZooKeeperIntegration

Regards and good luck,

Patrick