[Neo] Some feedback on ZooKeeper use in Neo4j "zha".

Patrick Hunt phunt at apache.org
Thu Apr 8 18:17:08 CEST 2010

On 04/08/2010 03:20 AM, Johan Svensson wrote:
> Hi Patrick,
> Thanks for the feedback. I will have a look at this and implement
> handling for disconnection and expiration of sessions.

No problem. We'll be psyched to see you roll this out.

> Regarding the GC issues we are well aware of these (hopefully the new
> "garbage first" or G1 GC will solve these problems). As you say the
> concurrent mark sweep GC helps a lot but more important to avoid "GC
> trashing" is to make sure there is more (10-15%) available heap than
> the application ever consumes at any given moment.

Agree. HBase has tested G1 in 1.6.x but so far it is not stable enough 
for production use.

> I do have a question regarding ZooKeeper. Is there a reason why there
> is no embedded version?
> Now I have to start two JVMs on each machine when I really just want to:
>     // start a zookeeper server on this machine
>     ZooKeeperServer server = new ZooKeeperServer( 2181 );
>     // start a client and pass in some zookeeper servers
>     ZooKeeper zoo = new ZooKeeper( "localhost:2181, otherhost:2181, ...", ... );

I'm not sure what you mean by "embedded". In production you typically 
want to have dedicated hosts for the servers. See this page for some insight

HBase wraps the zk server for their "quickstart" type use cases (in 
their startup scripts). But in large online production serving 
environments you typically run a ZK cluster separately from the client 


> Regards,
> -Johan
> On Wed, Apr 7, 2010 at 12:08 AM, Patrick Hunt<phunt at apache.org>  wrote:
>> Hi, I'm Patrick (http://twitter.com/phunt) from the ZooKeeper team.
>> Peter Neubauer brought to my attention today that you are considering
>> use of ZooKeeper in Neo4j, that's great! I took a quick look at the code
>> you currently have in SVN and wanted to provide a bit of feedback.
>> I don't know your domain requirements but in general the mechanics of
>> ZooClient use look fine.
>> The use of 5second timeout is fine. This allows you to detect a client
>> (zk client) failure after just 5 seconds. So if the node/process crashes
>> you'd identify this after 5 seconds, same if a network connection fails,
>> etc... One thing you may not have considered though, is that anything
>> that causes the client to not be able to heartbeat to the server would
>> also cause the session to be expired (sessions are expired when the zk
>> cluster fails to hear from the client w/in the "timeout" time) - so long
>> GC pauses could trigger this as well. In 1.6.x jvms we've seen that the
>> GC can pause all threads for very long periods (in some cases with hbase
>> we saw 4 minute pauses for gc). HBase was the first to see this, we
>> worked with Solr early on to help them understand this issue as well.
>> The problem can be alleviated somewhat by using the CMS/incremental GC
>> options in the JVM, however it cannot be eliminated entirely (in some
>> cases the Gc will still drop back to parallel). You need to consider the
>> impact of GC on your domain and how to best handle it.
>> See this JIRA for details on our discussion with Solr, you might gain
>> some good insight:http://bit.ly/d7OSQ1
>> https://issues.apache.org/jira/browse/SOLR-1277
>> I did notice that ZooClient is not handling disconnection and expiration
>> of the session in the "process" method. At the very least you need to
>> handle the expiration, you may need to do something for disconnection,
>> but this depends on whether you have active or passive actors (masters).
>> Here's a good link on session lifecycle:
>> http://hadoop.apache.org/zookeeper/docs/current/zookeeperProgrammers.html#ch_zkSessions
>> You might also want to setup a wiki page similar to these at some point,
>> it would help us with future discussion, feedback and provide insight
>> for devs/users:
>> http://wiki.apache.org/hadoop/ZooKeeper/HBaseAndZooKeeper
>> http://wiki.apache.org/solr/ZooKeeperIntegration
>> Regards and good luck,
>> Patrick
> _______________________________________________
> Neo mailing list
> User at lists.neo4j.org
> https://lists.neo4j.org/mailman/listinfo/user

More information about the User mailing list