Gremlin with neo4j graph only responding at node count after "No space left on device" error

[dmtougaw]

Our server (docker image) currently crashed after running out of space ("No space left on device") after I tried to export the graph in GraphSON. File took up all space on the container. So deleted the file and restarted the server which had errors related to "gremlin-groovy is not an available GremlinScriptEngine" so wouldn't start. Looked around for solutions and found this to restore the graph from "No space left on device" https://neo4j.com/developer/kb/how-do-recover-from-no-space-left-on-device/.

I tried to restore the graph from moving the neostore.transactions* files and restarting the server. Graph start without any errors but no query responses from there on (from console I get the prompt back with no response, no error codes). Only query working is the node count

gremlin> g.V().count()
==>1357600

Then decided to use the neo4j-admin check consistency tool to see if there was a way to fix this. Took a while.

At this point count() is still the only think that works. Other queries though return an error now

gremlin> g.V().label().dedup()
Token for id 3

• Exception with stack trace

[WARN] AbstractEvalOpProcessor - Exception processing a script on request [RequestMessage{, requestId=f032ca5f-6ba1-4f47-b06c-2cdd7639a33d, op='eval', processor='', args={gremlin=g.V().label().dedup(), batchSize=64}}].
org.neo4j.kernel.impl.core.TokenNotFoundException: Token for id 3
        at org.neo4j.kernel.impl.core.DelegatingTokenHolder.getTokenById(DelegatingTokenHolder.java:121)
        at org.neo4j.kernel.impl.api.store.StorageLayer.labelGetName(StorageLayer.java:169)
        at org.neo4j.kernel.impl.api.StateHandlingStatementOperations.labelGetName(StateHandlingStatementOperations.java:1343)
        at org.neo4j.kernel.impl.api.OperationsFacade.labelGetName(OperationsFacade.java:666)
        at org.neo4j.kernel.impl.core.NodeProxy.convertToLabel(NodeProxy.java:652)
        at org.neo4j.kernel.impl.core.NodeProxy.lambda$getLabels$2(NodeProxy.java:640)
        at org.neo4j.collection.primitive.PrimitiveIntCollections$14.next(PrimitiveIntCollections.java:734)
        at org.neo4j.helpers.collection.Iterators.addToCollection(Iterators.java:277)
        at org.neo4j.helpers.collection.Iterators.asList(Iterators.java:393)
        at org.neo4j.kernel.impl.core.NodeProxy.getLabels(NodeProxy.java:640)
        at org.neo4j.tinkerpop.api.impl.Neo4jNodeImpl.labels(Neo4jNodeImpl.java:43)
        at org.apache.tinkerpop.gremlin.neo4j.structure.Neo4jVertex.labels(Neo4jVertex.java:166)
        at org.apache.tinkerpop.gremlin.neo4j.structure.Neo4jVertex.label(Neo4jVertex.java:108)
        at org.apache.tinkerpop.gremlin.process.traversal.step.map.LabelStep.map(LabelStep.java:40)
        at org.apache.tinkerpop.gremlin.process.traversal.step.map.LabelStep.map(LabelStep.java:32)
        at org.apache.tinkerpop.gremlin.process.traversal.step.map.MapStep.processNextStart(MapStep.java:37)
        at org.apache.tinkerpop.gremlin.process.traversal.step.util.AbstractStep.hasNext(AbstractStep.java:143)
        at org.apache.tinkerpop.gremlin.process.traversal.step.util.ExpandableStepIterator.next(ExpandableStepIterator.java:50)
        at org.apache.tinkerpop.gremlin.process.traversal.step.filter.FilterStep.processNextStart(FilterStep.java:37)
        at org.apache.tinkerpop.gremlin.process.traversal.step.filter.DedupGlobalStep.processNextStart(DedupGlobalStep.java:105)
        at org.apache.tinkerpop.gremlin.process.traversal.step.util.AbstractStep.hasNext(AbstractStep.java:143)
        at org.apache.tinkerpop.gremlin.process.traversal.step.util.ExpandableStepIterator.next(ExpandableStepIterator.java:50)
        at org.apache.tinkerpop.gremlin.process.traversal.step.map.MapStep.processNextStart(MapStep.java:36)
        at org.apache.tinkerpop.gremlin.process.traversal.step.util.AbstractStep.hasNext(AbstractStep.java:143)
        at org.apache.tinkerpop.gremlin.process.traversal.util.DefaultTraversal.hasNext(DefaultTraversal.java:197)
        at org.apache.tinkerpop.gremlin.server.op.AbstractOpProcessor.handleIterator(AbstractOpProcessor.java:146)
        at org.apache.tinkerpop.gremlin.server.op.AbstractEvalOpProcessor.lambda$evalOpInternal$5(AbstractEvalOpProcessor.java:264)
        at org.apache.tinkerpop.gremlin.groovy.engine.GremlinExecutor.lambda$eval$0(GremlinExecutor.java:278)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

Any suggestions to restore the graph?

• neo4j version = 3.5.20
• apache tinkerpop gremlin server = 3.4.7

[Joel]

It seems like the database is likely corrupt, personally I suggest the best approach would be to do the import with enough disk space to finish successfully.

If you have available disk space on the system, it is possible to map volumes to the docker container so that the container doesn't store everything to the default location, including the neo4j db... I always map data storage volumes because (on linux) docker defaults to storing things down in /var/lib/docker, filling up my boot disk in a (sort of) hidden location, and it is also helpful if you want the db to persist beyond the life of the container.