[Neo] Possible causes for: org.apache.lucene.index.MergePolicy$MergeException

Symeon (Akis) Papadopoulos papadop at iti.gr
Wed Jan 27 12:26:08 CET 2010


>>> See more information at
>>> http://components.neo4j.org/neo4j-index/apidocs/index.html
>>>
>>> Are you using the "index" component btw (artifactId: neo4j-index)?
>>>
>>>       
>> Yes.
>>
>> Is it possible that the problem is caused due to the fact that we
>> perform several writes in both the graph and the index and we don't
>> commit and that we read in between the transaction commits?
>>     
> No that shouldn't cause any problems. This looks like a bug then.
> Could you give us more information about your code, some snippets
> perhaps or a test case which can trigger it again?
>   
In fact I've tried increasing the ulimit -n parameter of the linux 
machine where I was running the experiment (it was 1024 and I set it to 
10000). It appears to work so far (still running the test). What I 
understand is that there are many open files from lucene due to the fact 
that no optimize() is called on time. Since optimize() is not exposed 
from the LuceneIndexService, should I expect that it is handled internally?

The project I'm running is rather large, but at some point I will try to 
prepare a script in order to replicate the error.



>>     
>>> 2010/1/26 Symeon (Akis) Papadopoulos <papadop at iti.gr>:
>>>
>>>       
>>>> Hi all
>>>>
>>>> While populating a Neo graph, I got the attached exception
>>>> (org.apache.lucene.index.MergePolicy$MergeException. See file for details: I
>>>> replaced a local path with [some-local-path]).
>>>> My setup is like this: I want to benchmark Neo4j for some operations,
>>>> amongst which is graph loading.
>>>> So, I try to load graphs of various sizes to Neo4j. Up to size 1M edges the
>>>> graphs are loaded without any problem, but then this exception is thrown. I
>>>> suspect this has to do with the transaction management in my program, which
>>>> is handled by a BatchTxManager (a class written by me, thus highly likely to
>>>> be the source of trouble). Does the exception ring any bells? What could I
>>>> try out in order to identify the problem?
>>>>
>>>> Best regards,
>>>> Akis
>>>>
>>>>
>>>>
>>>> Exception in thread "Lucene Merge Thread #0"
>>>> org.apache.lucene.index.MergePolicy$MergeException:
>>>> java.io.FileNotFoundException: /[some-local-path]/_4k.cfs (Too many open
>>>> files)
>>>>        at
>>>> org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:351)
>>>>        at
>>>> org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:315)
>>>> Caused by: java.io.FileNotFoundException: /[some-local-path]/_4k.cfs (Too
>>>> many open files)
>>>>        at java.io.RandomAccessFile.open(Native Method)
>>>>        at java.io.RandomAccessFile.<init>(RandomAccessFile.java:233)
>>>>        at
>>>> org.apache.lucene.store.SimpleFSDirectory$SimpleFSIndexInput$Descriptor.<init>(SimpleFSDirectory.java:78)
>>>>        at
>>>> org.apache.lucene.store.SimpleFSDirectory$SimpleFSIndexInput.<init>(SimpleFSDirectory.java:108)
>>>>        at
>>>> org.apache.lucene.store.NIOFSDirectory$NIOFSIndexInput.<init>(NIOFSDirectory.java:94)
>>>>        at
>>>> org.apache.lucene.store.NIOFSDirectory.openInput(NIOFSDirectory.java:70)
>>>>        at
>>>> org.apache.lucene.index.CompoundFileReader.<init>(CompoundFileReader.java:70)
>>>>        at
>>>> org.apache.lucene.index.SegmentReader$CoreReaders.<init>(SegmentReader.java:111)
>>>>        at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:638)
>>>>        at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:608)
>>>>        at
>>>> org.apache.lucene.index.IndexWriter$ReaderPool.get(IndexWriter.java:686)
>>>>        at
>>>> org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4979)
>>>>        at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4614)
>>>>        at
>>>> org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:235)
>>>>        at
>>>> org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:291)
>>>> Exception in thread "main"
>>>> org.neo4j.kernel.impl.transaction.TransactionFailureException: Unable to
>>>> commit transaction
>>>>        at
>>>> org.neo4j.kernel.EmbeddedGraphDbImpl$TransactionImpl.finish(EmbeddedGraphDbImpl.java:331)
>>>>        at
>>>> org.neo4j.util.SimpleBatchTxManager.beginReadOperation(SimpleBatchTxManager.java:94)
>>>>        at
>>>> graph.implementation.neo.STSGraphNeoImpl.getUserNeo(STSGraphNeoImpl.java:463)
>>>>        at
>>>> graph.implementation.neo.STSGraphNeoImpl.increaseUserTagFreq(STSGraphNeoImpl.java:824)
>>>>        at
>>>> graph.benchmark.LoadGraphBenchmark.runRealBenchmark(LoadGraphBenchmark.java:433)
>>>>        at
>>>> graph.benchmark.LoadGraphBenchmark.main(LoadGraphBenchmark.java:489)
>>>> Caused by: java.lang.RuntimeException: Unable to close lucene writer
>>>> org.apache.lucene.index.IndexWriter at 32a4eb93
>>>>        at
>>>> org.neo4j.index.lucene.LuceneDataSource.removeWriter(LuceneDataSource.java:413)
>>>>        at
>>>> org.neo4j.index.lucene.LuceneTransaction.doCommit(LuceneTransaction.java:197)
>>>>        at
>>>> org.neo4j.kernel.impl.transaction.xaframework.XaTransaction.commit(XaTransaction.java:316)
>>>>        at
>>>> org.neo4j.kernel.impl.transaction.xaframework.XaResourceManager.commit(XaResourceManager.java:399)
>>>>        at
>>>> org.neo4j.kernel.impl.transaction.xaframework.XaResourceHelpImpl.commit(XaResourceHelpImpl.java:64)
>>>>        at
>>>> org.neo4j.kernel.impl.transaction.TransactionImpl.doCommit(TransactionImpl.java:514)
>>>>        at
>>>> org.neo4j.kernel.impl.transaction.TxManager.commit(TxManager.java:573)
>>>>        at
>>>> org.neo4j.kernel.impl.transaction.TxManager.commit(TxManager.java:543)
>>>>        at
>>>> org.neo4j.kernel.impl.transaction.TransactionImpl.commit(TransactionImpl.java:102)
>>>>        at
>>>> org.neo4j.kernel.EmbeddedGraphDbImpl$TransactionImpl.finish(EmbeddedGraphDbImpl.java:316)
>>>>        ... 5 more
>>>> Caused by: java.io.IOException: directory '/[some-local-path]/resource_id'
>>>> exists and is a directory, but cannot be listed: list() returned null
>>>>        at org.apache.lucene.store.FSDirectory.listAll(FSDirectory.java:516)
>>>>        at org.apache.lucene.store.FSDirectory.listAll(FSDirectory.java:531)
>>>>        at
>>>> org.apache.lucene.index.IndexFileDeleter.refresh(IndexFileDeleter.java:307)
>>>>        at
>>>> org.apache.lucene.index.IndexWriter.doFlushInternal(IndexWriter.java:4300)
>>>>        at org.apache.lucene.index.IndexWriter.doFlush(IndexWriter.java:4192)
>>>>        at org.apache.lucene.index.IndexWriter.flush(IndexWriter.java:4183)
>>>>        at
>>>> org.apache.lucene.index.IndexWriter.closeInternal(IndexWriter.java:2190)
>>>>        at org.apache.lucene.index.IndexWriter.close(IndexWriter.java:2153)
>>>>        at org.apache.lucene.index.IndexWriter.close(IndexWriter.java:2117)
>>>>        at
>>>> org.neo4j.index.lucene.LuceneDataSource.removeWriter(LuceneDataSource.java:409)
>>>>        ... 14 more
>>>>
>>>>
>>>> _______________________________________________
>>>> Neo mailing list
>>>> User at lists.neo4j.org
>>>> https://lists.neo4j.org/mailman/listinfo/user
>>>>
>>>>
>>>>
>>>>         
>>>
>>>
>>>       
>> _______________________________________________
>> Neo mailing list
>> User at lists.neo4j.org
>> https://lists.neo4j.org/mailman/listinfo/user
>>
>>     
>
>
>
>   



More information about the User mailing list