[Neo] Some feedback on ZooKeeper use in Neo4j "zha".

Patrick Hunt phunt at apache.org
Thu Apr 8 18:17:08 CEST 2010


On 04/08/2010 03:20 AM, Johan Svensson wrote:
> Hi Patrick,
>
> Thanks for the feedback. I will have a look at this and implement
> handling for disconnection and expiration of sessions.
>

No problem. We'll be psyched to see you roll this out.

> Regarding the GC issues we are well aware of these (hopefully the new
> "garbage first" or G1 GC will solve these problems). As you say the
> concurrent mark sweep GC helps a lot but more important to avoid "GC
> trashing" is to make sure there is more (10-15%) available heap than
> the application ever consumes at any given moment.
>

Agree. HBase has tested G1 in 1.6.x but so far it is not stable enough 
for production use.

> I do have a question regarding ZooKeeper. Is there a reason why there
> is no embedded version?
>
> Now I have to start two JVMs on each machine when I really just want to:
>
>     // start a zookeeper server on this machine
>     ZooKeeperServer server = new ZooKeeperServer( 2181 );
>     // start a client and pass in some zookeeper servers
>     ZooKeeper zoo = new ZooKeeper( "localhost:2181, otherhost:2181, ...", ... );

I'm not sure what you mean by "embedded". In production you typically 
want to have dedicated hosts for the servers. See this page for some insight
http://wiki.apache.org/hadoop/ZooKeeper/Troubleshooting

HBase wraps the zk server for their "quickstart" type use cases (in 
their startup scripts). But in large online production serving 
environments you typically run a ZK cluster separately from the client 
application.

Patrick

> Regards,
> -Johan
>
> On Wed, Apr 7, 2010 at 12:08 AM, Patrick Hunt<phunt at apache.org>  wrote:
>> Hi, I'm Patrick (http://twitter.com/phunt) from the ZooKeeper team.
>> Peter Neubauer brought to my attention today that you are considering
>> use of ZooKeeper in Neo4j, that's great! I took a quick look at the code
>> you currently have in SVN and wanted to provide a bit of feedback.
>>
>> I don't know your domain requirements but in general the mechanics of
>> ZooClient use look fine.
>>
>> The use of 5second timeout is fine. This allows you to detect a client
>> (zk client) failure after just 5 seconds. So if the node/process crashes
>> you'd identify this after 5 seconds, same if a network connection fails,
>> etc... One thing you may not have considered though, is that anything
>> that causes the client to not be able to heartbeat to the server would
>> also cause the session to be expired (sessions are expired when the zk
>> cluster fails to hear from the client w/in the "timeout" time) - so long
>> GC pauses could trigger this as well. In 1.6.x jvms we've seen that the
>> GC can pause all threads for very long periods (in some cases with hbase
>> we saw 4 minute pauses for gc). HBase was the first to see this, we
>> worked with Solr early on to help them understand this issue as well.
>> The problem can be alleviated somewhat by using the CMS/incremental GC
>> options in the JVM, however it cannot be eliminated entirely (in some
>> cases the Gc will still drop back to parallel). You need to consider the
>> impact of GC on your domain and how to best handle it.
>>
>> See this JIRA for details on our discussion with Solr, you might gain
>> some good insight:http://bit.ly/d7OSQ1
>> https://issues.apache.org/jira/browse/SOLR-1277
>>
>> I did notice that ZooClient is not handling disconnection and expiration
>> of the session in the "process" method. At the very least you need to
>> handle the expiration, you may need to do something for disconnection,
>> but this depends on whether you have active or passive actors (masters).
>> Here's a good link on session lifecycle:
>> http://hadoop.apache.org/zookeeper/docs/current/zookeeperProgrammers.html#ch_zkSessions
>>
>>
>> You might also want to setup a wiki page similar to these at some point,
>> it would help us with future discussion, feedback and provide insight
>> for devs/users:
>> http://wiki.apache.org/hadoop/ZooKeeper/HBaseAndZooKeeper
>> http://wiki.apache.org/solr/ZooKeeperIntegration
>>
>> Regards and good luck,
>>
>> Patrick
> _______________________________________________
> Neo mailing list
> User at lists.neo4j.org
> https://lists.neo4j.org/mailman/listinfo/user


More information about the User mailing list