[Neo] More complex comparisons for IndexService

Mattias Persson mattias at neotechnology.com
Tue Jul 7 01:17:00 CEST 2009


Ok then, let me give you example code which will solve all of your
problems... using LuceneFulltextIndexService! Let's say you have one
NeoService instance and one (extended) LuceneFulltextIndexService,
like this:


   NeoService neo = new EmbeddedNeo( "neo" );
   IndexService indexService =
      new LuceneFulltextIndexService( neo )
   {
      @Override
      protected Query formQuery( String key, Object value )
      {
         try
         {
            return new QueryParser( DOC_INDEX_KEY,
               new WhitespaceAnalyzer() ).parse( value.toString() );
         }
         catch ( ParseException e )
         {
            throw new RuntimeException( e );
         }
      }
   };
   ...
   // For equals
   indexService.getNodes( "lastName", "smith" );
   // For fuzzy match
   indexService.getNodes( "lastName", "smish~" );
   // For greater than (lucene query syntax may be incorrect)
   indexService.getNodes( "modifiedTime", "[2000 TO >]" );


This was what I was talking about (with the lucene query syntax
parser) and with this you can do more complex things as well.

Again note that these things (nodes added/deleted in the tx) are
handled... see the implementation which uses an in-memory lucene index
per tx to be able to handle it. I just ran this example locally and it
worked just fine (after a minor commit I just made :) ) and I'm
positive that it will for you too. Note that this is in the trunk
(0.7-SNAPSHOT) version of index-util.


2009/7/6 Adam Rabung <adamrabung at gmail.com>:
> I'm starting to suspect I'm using IndexService incorrectly :)
>
> I had been assuming I had one IndexService instance per neo instance.
> In the future, I was hoping to query that instance in many ways:
>
> indexService.getNodes("lastName", "smith"); //default is equals
> indexService.getNodes("lastName", ComparisonType.EQUALS, "smith");
> indexService.getNodes("lastName", ComparisonType.FUZZY_MATCH, "smith");
> indexService.getNodes("modifiedTime", ComparisonType.GREATER_THAN, 2000);
>
> With just key/value passed in the IndexService/LuceneFullText - it's
> unclear to me how to expose that comparison through Neo API, unless we
> have an IndexService instance/comparison:
> fuzzyMatchIndexService.getNodes("lastName", "smith");
> greaterThanIndexService.getNodes("modifiedTime", 2000);
>
> With both approaches, I don't see how you can avoid implementing these
> same comparisons in-memory so that you can ensure you consider tx
> deleted/added/modified nodes in the search.  Now let me strongly
> restate I think I must be using IndexService incorrectly :)
>
>
> On Sun, Jul 5, 2009 at 6:45 PM, Mattias
> Persson<mattias at neotechnology.com> wrote:
>> 2009/7/1 Adam Rabung <adamrabung at gmail.com>:
>>> Hi, thanks for the quick update.  I'd also need some kind of
>>> "comparator" (greaterThan, contains, etc) passed in to that method as
>>> well, in order to form the proper Lucene Query.  The other trick is
>>> considering nodes which have been modified/added/deleted in the
>>> current Lucene transaction.
>>
>> Regarding the "comparator", could you f.ex. use the lucene query
>> syntax http://lucene.apache.org/java/2_4_1/queryparsersyntax.html and
>> have the formQuery method parse that query (with the lucene
>> QueryParser) to accomplish that?
>>
>> The modified/added/deleted nodes are working automatically in the
>> LuceneFulltext* implementation so no worries there.
>>
>>>
>>> I think exposing LuceneIndexService's xaDs to subclasses, as well
>>> txIndexed and txDeleted in LuceneTransaction would allow third parties
>>> to create their own LuceneIndexServices.  OTOH, this seems like it
>>> might be worth offering in the core class.
>> Yep, it'd be great to have it be more customizable-friendly.
>>>
>>> I'm working on the CLA stuff - thanks again everybody!
>>
>> By your answer here I can't really see if you decided to have a look
>> at and try out the LuceneFulltext* classes instead of your own... what
>> have you tried so far? I'd be really happy to hear from you if you're
>> using it and would like to invite you to improve it furthermore if
>> necessary!
>>
>>>
>>>
>>> On Wed, Jul 1, 2009 at 3:47 AM, Mattias
>>> Persson<mattias at neotechnology.com> wrote:
>>>> Great initiative, btw have you looked at the
>>>> LuceneFulltextIndexService in the same package? It might be able to do
>>>> what you're asking for.
>>>>
>>>> By @Overriding the formQuery method you can construct whatever queries
>>>> you like based on key and search value (I just altered the signature
>>>> to also take the key in addition to value for greater flexibility).
>>>> Please have a look at that first and see if that helps you!
>>>>
>>>> Also do an 'svn up' on the index-util component since I committed it just now :)
>>>>
>>>> 2009/6/30 Peter Neubauer <neubauer.peter at gmail.com>:
>>>>> Adam,
>>>>> we would love to get your contribution into the codebase, just contact
>>>>> us regarding the CLA etc and we will get you through it!
>>>>>
>>>>> /peter
>>>>>
>>>>> GTalk:      neubauer.peter
>>>>> Skype       peter.neubauer
>>>>> Phone       +46 704 106975
>>>>> LinkedIn   http://www.linkedin.com/in/neubauer
>>>>> Twitter      http://twitter.com/peterneubauer
>>>>>
>>>>> http://www.neo4j.org     - New Energy for Data - The Graph Database.
>>>>> http://www.ops4j.org     - New Energy for OSS Communities - Open
>>>>> Participation Software.
>>>>> http://www.oredev.org   - Where Good Geeks Grok.
>>>>>
>>>>>
>>>>>
>>>>> On Tue, Jun 30, 2009 at 10:13 PM, Anders
>>>>> Nawroth<anders at neotechnology.com> wrote:
>>>>>> Hi and welcome!
>>>>>>
>>>>>> I can't answer your questions, but regarding the formatting we use an
>>>>>> Eclipse code formatter which you can download from this page:
>>>>>> http://wiki.neo4j.org/content/Coding_Standard
>>>>>>
>>>>>> This page could be of some interest as well:
>>>>>> http://wiki.neo4j.org/content/Code_Contributor%27s_Guide
>>>>>>
>>>>>> /anders
>>>>>>
>>>>>>
>>>>>> Adam Rabung skrev:
>>>>>>> Hi,
>>>>>>> I was wondering if anyone's interested in more complex queries against
>>>>>>> Neo's Lucene (and others) indexes, like so:
>>>>>>>
>>>>>>> long twoHoursAgo = System.currentTimeMillis() - (2 * 60 * 60 * 1000);
>>>>>>> indexService.getNodes(new Comparison("modifiedTime",
>>>>>>> ComparisonType.GREATER_THAN, twoHoursAgo);
>>>>>>>
>>>>>>> I hacked this into my local
>>>>>>> IndexService/LuceneIndexService/LuceneTransaction, and it seems to
>>>>>>> work.  For less than/greater than at least, the Lucene code itself is
>>>>>>> very straightforward (use a different Query class).  A little trickier
>>>>>>> when considering the added/updated/deleted nodes in the current tx,
>>>>>>> but not that bad.  There's _plenty_ of problems with my implementation
>>>>>>> - I applied it only to LuceneIndexService, type conversions, not too
>>>>>>> confident w/ some of the Neo-specific changes I made - but I was very
>>>>>>> encouraged by how easy it was to get a draft version running.
>>>>>>>
>>>>>>> Is there any chance at all this kind of thing could make a near-term
>>>>>>> Neo4j release?  If not, would it be possible to expose internals of
>>>>>>> some the classes so that third parties could easily add custom
>>>>>>> searches?
>>>>>>>
>>>>>>> Proof of concept "patch" attached.  Not really a patch - just 3 files
>>>>>>> - LuceneIndexService, LuceneTransaction and a new class called
>>>>>>> Comparison.  Eclipse formatting ruined my ability to quickly make you
>>>>>>> a nice patch.
>>>>>>>
>>>>>>> Thanks for a great product!
>>>>>>>
>>>>>>> ------------------------------------------------------------------------
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> Neo mailing list
>>>>>>> User at lists.neo4j.org
>>>>>>> https://lists.neo4j.org/mailman/listinfo/user
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Anders Nawroth [anders at neotechnology.com]
>>>>>> GTalk, Skype: anders.nawroth
>>>>>> Phone: +46 737 894 163
>>>>>> http://twitter.com/nawroth
>>>>>> http://blog.nawroth.com/
>>>>>>
>>>>>> _______________________________________________
>>>>>> Neo mailing list
>>>>>> User at lists.neo4j.org
>>>>>> https://lists.neo4j.org/mailman/listinfo/user
>>>>>>
>>>>> _______________________________________________
>>>>> Neo mailing list
>>>>> User at lists.neo4j.org
>>>>> https://lists.neo4j.org/mailman/listinfo/user
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Mattias Persson, [mattias at neotechnology.com]
>>>> Neo Technology, www.neotechnology.com
>>>> _______________________________________________
>>>> Neo mailing list
>>>> User at lists.neo4j.org
>>>> https://lists.neo4j.org/mailman/listinfo/user
>>>>
>>> _______________________________________________
>>> Neo mailing list
>>> User at lists.neo4j.org
>>> https://lists.neo4j.org/mailman/listinfo/user
>>>
>>
>>
>>
>> --
>> Mattias Persson, [mattias at neotechnology.com]
>> Neo Technology, www.neotechnology.com
>> _______________________________________________
>> Neo mailing list
>> User at lists.neo4j.org
>> https://lists.neo4j.org/mailman/listinfo/user
>>
> _______________________________________________
> Neo mailing list
> User at lists.neo4j.org
> https://lists.neo4j.org/mailman/listinfo/user
>



-- 
Mattias Persson, [mattias at neotechnology.com]
Neo Technology, www.neotechnology.com


More information about the User mailing list