Neo4j Won't start

[ranlandau]

My Neo4j Crushed and won't restart!!

help!

ubuntu@ip-10-0-32-44:/var/log/neo4j$ ulimit -n
1045590

2018-12-23 10:30:26.025+0000 INFO [o.n.k.i.i.s.f.FusionIndexProvider] Schema index cleanup job closed: descriptor=Index( GENERAL, :label[784](property[0]) ), indexFile=/var/lib/neo4j/data/databases/graph.db/schema/index/lucene_native-2.0/9369/string-1.0/index-9369
2018-12-23 10:30:26.025+0000 INFO [o.n.k.i.i.s.f.FusionIndexProvider] Schema index cleanup job closed: descriptor=Index( GENERAL, :label[784](property[0]) ), indexFile=/var/lib/neo4j/data/databases/graph.db/schema/index/lucene_native-2.0/9369/native-1.0/index-9369
2018-12-23 10:30:26.025+0000 INFO [o.n.k.i.i.s.f.FusionIndexProvider] Schema index cleanup job closed: descriptor=Index( GENERAL, :label[784](property[0]) ), indexFile=/var/lib/neo4j/data/databases/graph.db/schema/index/lucene_native-2.0/9369/spatial-1.0/1-4326
2018-12-23 10:30:26.025+0000 INFO [o.n.k.i.i.s.f.FusionIndexProvider] Schema index cleanup job closed: descriptor=Index( GENERAL, :label[784](property[0]) ), indexFile=/var/lib/neo4j/data/databases/graph.db/schema/index/lucene_native-2.0/9369/temporal-1.0/date
2018-12-23 10:30:26.025+0000 INFO [o.n.k.i.i.s.f.FusionIndexProvider] Schema index cleanup job closed: descriptor=Index( GENERAL, :label[784](property[1]) ), indexFile=/var/lib/neo4j/data/databases/graph.db/schema/index/lucene_native-2.0/9379/string-1.0/index-9379
2018-12-23 10:30:26.025+0000 INFO [o.n.k.i.i.s.f.FusionIndexProvider] Schema index cleanup job closed: descriptor=Index( GENERAL, :label[784](property[1]) ), indexFile=/var/lib/neo4j/data/databases/graph.db/schema/index/lucene_native-2.0/9379/native-1.0/index-9379
2018-12-23 10:30:26.025+0000 INFO [o.n.k.i.i.s.f.FusionIndexProvider] Schema index cleanup job closed: descriptor=Index( GENERAL, :label[784](property[1]) ), indexFile=/var/lib/neo4j/data/databases/graph.db/schema/index/lucene_native-2.0/9379/spatial-1.0/1-4326
2018-12-23 10:30:26.025+0000 INFO [o.n.k.i.i.s.f.FusionIndexProvider] Schema index cleanup job closed: descriptor=Index( GENERAL, :label[784](property[1]) ), indexFile=/var/lib/neo4j/data/databases/graph.db/schema/index/lucene_native-2.0/9379/temporal-1.0/date
2018-12-23 10:30:26.025+0000 INFO [o.n.k.i.i.s.f.FusionIndexProvider] Schema index cleanup job closed: descriptor=Index( GENERAL, :label[785](property[2]) ), indexFile=/var/lib/neo4j/data/databases/graph.db/schema/index/lucene_native-2.0/9381/string-1.0/index-9381
2018-12-23 10:30:26.025+0000 WARN [o.n.k.NeoStoreDataSource] Exception occurred while starting the datasource. Attempting to close things down. Component 'org.neo4j.kernel.impl.storageengine.impl.recordstorage.RecordStorageEngine@5e6ef21a' failed to initialize. Please see the attached cause exception "/var/lib/neo4j/data/databases/graph.db/schema/index/lucene_native-2.0/9383/lucene-1.0/failure-message (Too many open files)".
org.neo4j.kernel.lifecycle.LifecycleException: Component 'org.neo4j.kernel.impl.storageengine.impl.recordstorage.RecordStorageEngine@5e6ef21a' failed to initialize. Please see the attached cause exception "/var/lib/neo4j/data/databases/graph.db/schema/index/lucene_native-2.0/9383/lucene-1.0/failure-message (Too many open files)".
        at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.init(LifeSupport.java:427)
        at org.neo4j.kernel.lifecycle.LifeSupport.init(LifeSupport.java:62)
        at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:98)
        at org.neo4j.kernel.NeoStoreDataSource.start(NeoStoreDataSource.java:515)
        at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:445)
        at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:107)
        at org.neo4j.kernel.impl.transaction.state.DataSourceManager.start(DataSourceManager.java:100)
        at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:445)
        at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:107)
        at org.neo4j.kernel.impl.factory.GraphDatabaseFacadeFactory.initFacade(GraphDatabaseFacadeFactory.java:208)
        at org.neo4j.kernel.enterprise.EnterpriseGraphDatabase.<init>(EnterpriseGraphDatabase.java:39)
        at org.neo4j.server.enterprise.OpenEnterpriseNeoServer.lambda$static$1(OpenEnterpriseNeoServer.java:78)
        at org.neo4j.server.database.LifecycleManagingDatabase.start(LifecycleManagingDatabase.java:88)
        at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:445)
        at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:107)
        at org.neo4j.server.AbstractNeoServer.start(AbstractNeoServer.java:212)
        at org.neo4j.server.ServerBootstrapper.start(ServerBootstrapper.java:111)
        at org.neo4j.server.ServerBootstrapper.start(ServerBootstrapper.java:79)
        at com.neo4j.server.enterprise.CommercialEntryPoint.main(CommercialEntryPoint.java:22)
Caused by: java.lang.RuntimeException: java.io.FileNotFoundException: /var/lib/neo4j/data/databases/graph.db/schema/index/lucene_native-2.0/9383/lucene-1.0/failure-message (Too many open files)
        at org.neo4j.kernel.api.impl.index.storage.FailureStorage.loadIndexFailure(FailureStorage.java:104)
        at org.neo4j.kernel.api.impl.index.storage.PartitionedIndexStorage.getStoredIndexFailure(PartitionedIndexStorage.java:130)
        at org.neo4j.kernel.api.impl.schema.LuceneIndexProvider.getInitialState(LuceneIndexProvider.java:129)
        at org.neo4j.kernel.impl.index.schema.fusion.FusionIndexProvider.lambda$getInitialState$3(FusionIndexProvider.java:147)
        at org.neo4j.kernel.impl.index.schema.fusion.InstanceSelector.instancesAs(InstanceSelector.java:73)
        at org.neo4j.kernel.impl.index.schema.fusion.FusionIndexProvider.getInitialState(FusionIndexProvider.java:147)
        at org.neo4j.kernel.impl.api.index.IndexingService.lambda$init$1(IndexingService.java:207)
        at org.neo4j.kernel.impl.api.index.IndexMapReference.modify(IndexMapReference.java:52)
        at org.neo4j.kernel.impl.api.index.IndexingService.init(IndexingService.java:196)
        at org.neo4j.kernel.impl.storageengine.impl.recordstorage.RecordStorageEngine.init(RecordStorageEngine.java:407)
        at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.init(LifeSupport.java:406)
        ... 18 more
Caused by: java.io.FileNotFoundException: /var/lib/neo4j/data/databases/graph.db/schema/index/lucene_native-2.0/9383/lucene-1.0/failure-message (Too many open files)
        at java.io.RandomAccessFile.open0(Native Method)
        at java.io.RandomAccessFile.open(RandomAccessFile.java:316)
        at java.io.RandomAccessFile.<init>(RandomAccessFile.java:243)
        at org.neo4j.io.fs.DefaultFileSystemAbstraction.open(DefaultFileSystemAbstraction.java:69)
        at org.neo4j.io.fs.DefaultFileSystemAbstraction.open(DefaultFileSystemAbstraction.java:54)
        at org.neo4j.kernel.api.impl.index.storage.FailureStorage.isFailed(FailureStorage.java:168)
        at org.neo4j.kernel.api.impl.index.storage.FailureStorage.loadIndexFailure(FailureStorage.java:96)
        ... 28 more
2018-12-23 10:30:26.026+0000 INFO [o.n.k.i.DiagnosticsManager] --- STOPPING diagnostics START ---
2018-12-23 10:30:26.026+0000 INFO [o.n.k.i.DiagnosticsManager] --- STOPPING diagnostics END ---
2018-12-23 10:30:26.026+0000 INFO [o.n.k.i.f.GraphDatabaseFacadeFactory] Shutdown started

[Bert]

Hi,

looking at the exception log, it appears that your index has grown to a size (and number of files) that neo4j can't open them.

The problem and solution is described here: https://neo4j.com/docs/operations-manual/3.5/installation/linux/tarball/#linux-open-files

Hope that helps.
Bert

[stefan_armbrust]

On Ubuntu you need to set the number of open files in /etc/security/limits.d/neo4j.conf.