[Neo] Possible causes for: org.apache.lucene.index.MergePolicy$MergeException

Symeon (Akis) Papadopoulos papadop at iti.gr
Tue Jan 26 20:34:24 CET 2010


Mattias Persson wrote:
> Ah, do you make sure you call IndexHits#close() on your results from
> getNodes(), alternatively iterating over the whole result (in which
> case it's automatically closed)?
>   
Actually I'm calling the getSingleNode, thus I have no access on the 
IndexHits object.


> See more information at
> http://components.neo4j.org/neo4j-index/apidocs/index.html
>
> Are you using the "index" component btw (artifactId: neo4j-index)?
>   
Yes.

Is it possible that the problem is caused due to the fact that we 
perform several writes in both the graph and the index and we don't 
commit and that we read in between the transaction commits?


> 2010/1/26 Symeon (Akis) Papadopoulos <papadop at iti.gr>:
>   
>> Hi all
>>
>> While populating a Neo graph, I got the attached exception
>> (org.apache.lucene.index.MergePolicy$MergeException. See file for details: I
>> replaced a local path with [some-local-path]).
>> My setup is like this: I want to benchmark Neo4j for some operations,
>> amongst which is graph loading.
>> So, I try to load graphs of various sizes to Neo4j. Up to size 1M edges the
>> graphs are loaded without any problem, but then this exception is thrown. I
>> suspect this has to do with the transaction management in my program, which
>> is handled by a BatchTxManager (a class written by me, thus highly likely to
>> be the source of trouble). Does the exception ring any bells? What could I
>> try out in order to identify the problem?
>>
>> Best regards,
>> Akis
>>
>>
>>
>> Exception in thread "Lucene Merge Thread #0"
>> org.apache.lucene.index.MergePolicy$MergeException:
>> java.io.FileNotFoundException: /[some-local-path]/_4k.cfs (Too many open
>> files)
>>        at
>> org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:351)
>>        at
>> org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:315)
>> Caused by: java.io.FileNotFoundException: /[some-local-path]/_4k.cfs (Too
>> many open files)
>>        at java.io.RandomAccessFile.open(Native Method)
>>        at java.io.RandomAccessFile.<init>(RandomAccessFile.java:233)
>>        at
>> org.apache.lucene.store.SimpleFSDirectory$SimpleFSIndexInput$Descriptor.<init>(SimpleFSDirectory.java:78)
>>        at
>> org.apache.lucene.store.SimpleFSDirectory$SimpleFSIndexInput.<init>(SimpleFSDirectory.java:108)
>>        at
>> org.apache.lucene.store.NIOFSDirectory$NIOFSIndexInput.<init>(NIOFSDirectory.java:94)
>>        at
>> org.apache.lucene.store.NIOFSDirectory.openInput(NIOFSDirectory.java:70)
>>        at
>> org.apache.lucene.index.CompoundFileReader.<init>(CompoundFileReader.java:70)
>>        at
>> org.apache.lucene.index.SegmentReader$CoreReaders.<init>(SegmentReader.java:111)
>>        at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:638)
>>        at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:608)
>>        at
>> org.apache.lucene.index.IndexWriter$ReaderPool.get(IndexWriter.java:686)
>>        at
>> org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4979)
>>        at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4614)
>>        at
>> org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:235)
>>        at
>> org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:291)
>> Exception in thread "main"
>> org.neo4j.kernel.impl.transaction.TransactionFailureException: Unable to
>> commit transaction
>>        at
>> org.neo4j.kernel.EmbeddedGraphDbImpl$TransactionImpl.finish(EmbeddedGraphDbImpl.java:331)
>>        at
>> org.neo4j.util.SimpleBatchTxManager.beginReadOperation(SimpleBatchTxManager.java:94)
>>        at
>> graph.implementation.neo.STSGraphNeoImpl.getUserNeo(STSGraphNeoImpl.java:463)
>>        at
>> graph.implementation.neo.STSGraphNeoImpl.increaseUserTagFreq(STSGraphNeoImpl.java:824)
>>        at
>> graph.benchmark.LoadGraphBenchmark.runRealBenchmark(LoadGraphBenchmark.java:433)
>>        at
>> graph.benchmark.LoadGraphBenchmark.main(LoadGraphBenchmark.java:489)
>> Caused by: java.lang.RuntimeException: Unable to close lucene writer
>> org.apache.lucene.index.IndexWriter at 32a4eb93
>>        at
>> org.neo4j.index.lucene.LuceneDataSource.removeWriter(LuceneDataSource.java:413)
>>        at
>> org.neo4j.index.lucene.LuceneTransaction.doCommit(LuceneTransaction.java:197)
>>        at
>> org.neo4j.kernel.impl.transaction.xaframework.XaTransaction.commit(XaTransaction.java:316)
>>        at
>> org.neo4j.kernel.impl.transaction.xaframework.XaResourceManager.commit(XaResourceManager.java:399)
>>        at
>> org.neo4j.kernel.impl.transaction.xaframework.XaResourceHelpImpl.commit(XaResourceHelpImpl.java:64)
>>        at
>> org.neo4j.kernel.impl.transaction.TransactionImpl.doCommit(TransactionImpl.java:514)
>>        at
>> org.neo4j.kernel.impl.transaction.TxManager.commit(TxManager.java:573)
>>        at
>> org.neo4j.kernel.impl.transaction.TxManager.commit(TxManager.java:543)
>>        at
>> org.neo4j.kernel.impl.transaction.TransactionImpl.commit(TransactionImpl.java:102)
>>        at
>> org.neo4j.kernel.EmbeddedGraphDbImpl$TransactionImpl.finish(EmbeddedGraphDbImpl.java:316)
>>        ... 5 more
>> Caused by: java.io.IOException: directory '/[some-local-path]/resource_id'
>> exists and is a directory, but cannot be listed: list() returned null
>>        at org.apache.lucene.store.FSDirectory.listAll(FSDirectory.java:516)
>>        at org.apache.lucene.store.FSDirectory.listAll(FSDirectory.java:531)
>>        at
>> org.apache.lucene.index.IndexFileDeleter.refresh(IndexFileDeleter.java:307)
>>        at
>> org.apache.lucene.index.IndexWriter.doFlushInternal(IndexWriter.java:4300)
>>        at org.apache.lucene.index.IndexWriter.doFlush(IndexWriter.java:4192)
>>        at org.apache.lucene.index.IndexWriter.flush(IndexWriter.java:4183)
>>        at
>> org.apache.lucene.index.IndexWriter.closeInternal(IndexWriter.java:2190)
>>        at org.apache.lucene.index.IndexWriter.close(IndexWriter.java:2153)
>>        at org.apache.lucene.index.IndexWriter.close(IndexWriter.java:2117)
>>        at
>> org.neo4j.index.lucene.LuceneDataSource.removeWriter(LuceneDataSource.java:409)
>>        ... 14 more
>>
>>
>> _______________________________________________
>> Neo mailing list
>> User at lists.neo4j.org
>> https://lists.neo4j.org/mailman/listinfo/user
>>
>>
>>     
>
>
>
>   



More information about the User mailing list