[Neo] Traversers in the REST API

Fri Apr 9 13:59:15 CEST 2010

   Since in manycases the results of a query will need to be reformed into
   their associated domain objects, we've chosen to do our sorting at that
   point (and on the server).  We do our (primary) filtering within the
   traversal/DB->domain object processes.  That seems to work well.

   Pagination is kinda tricky if the data changes between subsequent
   requests for "pages".  Since pagination is generally used for UIs, a
   common approach is to place the entire dataset (or a cursor, depending
   on where the data is coming from) in a session object.  Regardless of
   where it is kept, if you want to deal with data changes, you either
   have to a) invalidate the "cached" dataset if data changes or b) keep a
   copy of the whole dataset around in its "as queried" state so that
   subsequent paging requests are consistent.  Either case involves
   keeping a fairly big duplicate data structure on the server or middle
   tier and violates one of the objectives of REST-ful APIs, which is that
   of statelessness.  For that reason, I personally think the REST-ful API
   shouldn't deal with paging.  It should probably be done at some
   intermediate level as needed by applications.  We can certainly build a
   separate API that we can all leverage if needed, but I don't think it
   should be in the core REST-ful layer.

   Just my $0.02, after taxes.

   -------- Original Message --------
   Subject: Re: [Neo] Traversers in the REST API
   From: Tobias Ivarsson <tobias.ivarsson at neotechnology.com>
   Date: Fri, April 09, 2010 4:00 am
   To: Neo user discussions <user at lists.neo4j.org>
   I definitely agree that limiting or paging a set of results is probably
   not
   very useful without some sort of sorting. The (only) benefit of pushing
   sorting to the client is that the client might be able to filter the
   result
   further before sorting it. Since sorting is generally the most
   expensive
   operation it should be done as late as possible. However the idea of
   semi-sorting, to get only one page of sorted results at each request,
   that
   was mentioned in some thread yesterday sounds quite compelling.
   I agree that an equivalent of LIMIT, OFFSET and ORDER BY is a good
   target.
   As to indexing: the structure of the graph IS the index to a large
   extent.
   This means that a well designed graph would often not need paging if
   the
   traversal is done right. There are however some cases where this is
   hard to
   accomplish and we need to work on supporting those cases better.
   Remember that a Graph Database is NOT a Relational Database. A lot of
   the
   ideas people have about databases are based on their knowledge of
   Relational
   Databases. I understand that it can be hard, but if that baggage could
   be
   left at the door it would make things a lot easier. Nobody is saying
   that
   Relational Databases are dead (except for some publicity stunts) far
   from
   it! What we (and a lot of other people) are saying is that the age of
   "one
   database to rule them all" is over. Different problems are best solved
   with
   different kinds of databases, RDBMSes are great at some, K/V stores
   some,
   and Graph Databases are great for some. Then there are some problems
   that
   are best solved with a combination of two or more (kinds of) databases,
   where each database brings its own strengths to the table, and is used
   only
   for the things it is good at.
   That's enough deviation from the topic, my conclusions remain the same
   as
   they were before this discussion started, I will state them in as few
   words
   as possible and in bullet point form to convey them as clearly as I
   can:
   * The REST API will probably need result set limiting or pagination.
   * Limiting and pagination will require (server side) sorting
   * Sorting can be better implemented if it's implemented in the core of
   the
   traversal framework
   * Limiting / Pagination can be deferred for a while until we know what
   it
   needs to look like (from looking at actual uses)
   * (Server side) Sorting can be deferred until we need it for limiting /
   pagination
   Peace,
   Tobias
   On Thu, Apr 8, 2010 at 10:17 PM, Michael Ludwig <milu71 at gmx.de> wrote:
   > Tobias Ivarsson schrieb am 08.04.2010 um 18:23:27 (+0200)
   > [Re: [Neo] Traversers in the REST API]:
   >
   > > On Wed, Apr 7, 2010 at 3:05 PM, Alastair James <al.james at gmail.com>
   > > wrote:
   >
   > > > when we start talking about returning 1000s of nodes in JSON over
   > > > HTTP just to get the first 10 this is clearly sub-optimal (as I
   > > > build websites this is a very common use case). So, as you say,
   > > > sorting and limiting can wait, but I suspect the HTTP API would
   > > > benefit from offering it. Limiting need not require changes to
   the
   > > > core API, it could be implemented as a second stage in the HTTP
   API
   > > > code prior to output encoding.
   > >
   > > For paging / limiting: yes, you are absolutely right, this would
   not
   > > effect the core API at all, only the REST API. Limiting/paging is
   > > something we would probably add to the REST API before sorting.
   >
   > Limiting and paging usually go hand in hand with sorting, in my
   > experience. Why would anyone want to page through an unsorted
   > collection?
   >
   > > Sorting might be a similar case, but I still think the client would
   be
   > > better fitted to do sorting well.
   >
   > The server has indexes to support the sorting. (If it doesn't, it has
   a
   > problem anyway.) What does the client have to support sorting? So how
   > would it be better fitted to do sorting well?
   >
   > > But once paging / limiting is added it would be quite natural /
   useful
   > > to add sorting as well. What I want to avoid is keeping state on
   the
   > > server while waiting for the client to request the next page.
   >
   > If you ensure a binary tree index is used to do the sorting, you
   should
   > be fine.
   >
   > --
   > Michael Ludwig
   > _______________________________________________
   > Neo mailing list
   > User at lists.neo4j.org
   > [1]https://lists.neo4j.org/mailman/listinfo/user
   >
   --
   Tobias Ivarsson <tobias.ivarsson at neotechnology.com>
   Hacker, Neo Technology
   [2]www.neotechnology.com
   Cellphone: +46 706 534857
   _______________________________________________
   Neo mailing list
   User at lists.neo4j.org
   [3]https://lists.neo4j.org/mailman/listinfo/user

References

   1. https://lists.neo4j.org/mailman/listinfo/user
   2. http://www.neotechnology.com/
   3. https://lists.neo4j.org/mailman/listinfo/user