[Neo] User Digest, Vol 25, Issue 15

Peter Neubauer neubauer.peter at gmail.com
Fri Apr 24 19:53:31 CEST 2009


Mohamed,
I am not the best to answer questions on the storage part of Neo, but
IMHO alternative backends for Neo4j do not make sense until they are
fast than file access. Currently, since Neo4j is transactionally safe,
the data in a transaction has to be saved to the backing storage
mechanism. For performance reasons, the fastest such mechanism is
currently NIO Neo, the specially designed backend for Neo4j.

There might be others, and actually we are thinking of some, but for
the time being a filesystem backed by a SSD disk is the fastest
storage backend for Neo4j. Even Hadoop has to get down to disk at some
point, so everything in between would more or less just slow us down.
OTOH, as Storage Area Networks and others are increasingly adding an
indirection layer between disk and file system API, even that can
become an option at some point of time, but still it is about raw IO
speed to get the bytes onto the storage safely.

Anyone knows of faster ways to store data safely?

HTH, please let us know if you have any thoughts that can add to the discussion!

/peter

GTalk:      neubauer.peter
Skype       peter.neubauer
Phone       +46 704 106975
LinkedIn   http://www.linkedin.com/in/neubauer
Twitter      http://twitter.com/peterneubauer

http://www.neo4j.org     - New Energy for Data - The Graph Database.
http://www.ops4j.org     - New Energy for OSS Communities - Open
Participation Software.
http://www.qi4j.org        - New Energy for Java - Domain Driven Development.



On Fri, Apr 24, 2009 at 3:48 PM, Chenini, Mohamed <MChenini at geico.com> wrote:
> Hi,
>
> My next question is the following:
>
> Since Neo4j is database system (based on graph theory), can Hadoop be
> used for the computation and Neo4j for the storage? Can they co-exist in
> the same application?
>
> I am not sure if my question makes sense, because I am just starting
> reading about these two products.
>
> Regards,
> Mohamed
>
> -----Original Message-----
> From: user-bounces at lists.neo4j.org [mailto:user-bounces at lists.neo4j.org]
> On Behalf Of user-request at lists.neo4j.org
> Sent: Friday, April 24, 2009 6:00 AM
> To: user at lists.neo4j.org
> Subject: User Digest, Vol 25, Issue 15
>
> Send User mailing list submissions to
>        user at lists.neo4j.org
>
> To subscribe or unsubscribe via the World Wide Web, visit
>        https://lists.neo4j.org/mailman/listinfo/user
> or, via email, send a message with subject or body 'help' to
>        user-request at lists.neo4j.org
>
> You can reach the person managing the list at
>        user-owner at lists.neo4j.org
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of User digest..."
>
>
> Today's Topics:
>
>   1. Neo4j vs Hadoop (Chenini, Mohamed )
>   2. Re: Neo4j vs Hadoop (Peter Neubauer)
>
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Thu, 23 Apr 2009 19:04:06 -0400
> From: "Chenini, Mohamed " <MChenini at geico.com>
> Subject: [Neo] Neo4j vs Hadoop
> To: <user at lists.neo4j.org>
> Message-ID:
>
> <E0A3C299B996D840B5E96ECD9EEDC5B515E410 at GP2K0084V3.GEICO.corp.net>
> Content-Type: text/plain; charset="us-ascii"
>
> Hi,
>
>
>
> I just heard about Neo4j and I would like to know how it compares to
> Hadoop.
>
>
>
> Thanks,
>
> Mohamed
>
> ====================
> This email/fax message is for the sole use of the intended
> recipient(s) and may contain confidential and privileged information.
> Any unauthorized review, use, disclosure or distribution of this
> email/fax is prohibited. If you are not the intended recipient, please
> destroy all paper and electronic copies of the original message.
>
>
> ------------------------------
>
> Message: 2
> Date: Fri, 24 Apr 2009 11:06:52 +0200
> From: Peter Neubauer <peter.neubauer at jayway.se>
> Subject: Re: [Neo] Neo4j vs Hadoop
> To: Neo user discussions <user at lists.neo4j.org>
> Message-ID:
>        <308579740904240206y63e04a7eqd4e3943d1a0dd545 at mail.gmail.com>
> Content-Type: text/plain; charset=ISO-8859-1
>
> Hi Mohamed,
> in principle, Hadoop and other Key-Value stores are mostly concerned
> with relatively flat data structures. That is, they are extremely fast
> and scalable regarding retrieval of simple objects, like values,
> documents or even objects.
>
> However, if you want to do deeper traversal of e.g. a graph, you will
> have to retrieve the nodes for every traversal step (very fast) and
> then match them yourself in some manner (e.g. in Java or so) - slow.
>
> Neo4j in contrast is build around the concept of "deep" data
> structures. This gives you almost unlimited flexibility regarding the
> layout of your data and domain object graph and very fast deep
> traversals (hops over several nodes) since they are handled natively
> by the Neo4j engine down to the storage layer and not your client
> code. The drawback is that for huge data amounts (>1Billion nodes) the
> clustering and partitioning of the graph becomes non-trivial, which is
> one of the areas we are working on.
>
> Then of course there are differences in the transaction models,
> consistency and others, but I hope this gives you a very short
> philosophical answer :)
>
> /peter
>
> GTalk:      neubauer.peter
> Skype       peter.neubauer
> Phone       +46 704 106975
> LinkedIn   http://www.linkedin.com/in/neubauer
> Twitter      http://twitter.com/peterneubauer
>
> http://www.neo4j.org     - New Energy for Data - The Graph Database.
> http://www.ops4j.org     - New Energy for OSS Communities - Open
> Participation Software.
> http://www.qi4j.org        - New Energy for Java - Domain Driven
> Development.
>
>
>
> On Fri, Apr 24, 2009 at 1:04 AM, Chenini, Mohamed <MChenini at geico.com>
> wrote:
>> Hi,
>>
>>
>>
>> I just heard about Neo4j and I would like to know how it compares to
>> Hadoop.
>>
>>
>>
>> Thanks,
>>
>> Mohamed
>>
>> ====================
>> This email/fax message is for the sole use of the intended
>> recipient(s) and may contain confidential and privileged information.
>> Any unauthorized review, use, disclosure or distribution of this
>> email/fax is prohibited. If you are not the intended recipient, please
>> destroy all paper and electronic copies of the original message.
>> _______________________________________________
>> Neo mailing list
>> User at lists.neo4j.org
>> https://lists.neo4j.org/mailman/listinfo/user
>>
>>
>
>
> ------------------------------
>
> _______________________________________________
> User mailing list
> User at lists.neo4j.org
> https://lists.neo4j.org/mailman/listinfo/user
>
>
> End of User Digest, Vol 25, Issue 15
> ************************************
> ====================
> This email/fax message is for the sole use of the intended
> recipient(s) and may contain confidential and privileged information.
> Any unauthorized review, use, disclosure or distribution of this
> email/fax is prohibited. If you are not the intended recipient, please
> destroy all paper and electronic copies of the original message.
>
> _______________________________________________
> Neo mailing list
> User at lists.neo4j.org
> https://lists.neo4j.org/mailman/listinfo/user
>


More information about the User mailing list