Impossible to export from embedded database

[ontologiae]

neo4j 3.5 (The version 4 never worked in embedded context, it does not creates links between nodes... Tried everything....)

Hi,
I have a huge database with lots of differents type of nodes and relationships.
I'm trying desperately to export my database to an importable file, whatever it is, csv, cypher, I don't care, I just want it is importable by a standalone server.

I tried a LOT of things :

• Impossible to registers apoc procedure, so I directly used the method inside APOC package and did this :

 def testExport(db: GraphDatabaseService) = {

    val myMap    = Map("format" -> "plain".asInstanceOf[Object])
    val exporter = new ExportCypher(db)
    //val exporterCSV = new ExportCSV(db) // other experiment
    //val s        = exporter.query("match (n) return n", "test.cypher", myMap.asJava) // other experiment
    val s        = exporter.all( "test.cypher", myMap.asJava)
    println(s.iterator.asScala.toList)
  }

I ALWAYS had

[error] (run-main-0) org.neo4j.graphdb.NotFoundException: org.neo4j.graphdb.NotFoundException: org.neo4j.internal.kernel.api.exceptions.EntityNotFoundException: Unable to load NODE with id 0.
[error] org.neo4j.graphdb.NotFoundException: org.neo4j.graphdb.NotFoundException: org.neo4j.internal.kernel.api.exceptions.EntityNotFoundException: Unable to load NODE with id 0.
[error]         at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
[error]         at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
[error]         at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
[error]         at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
[error]         at java.util.concurrent.ForkJoinTask.getThrowableException(ForkJoinTask.java:593)
[error]         at java.util.concurrent.ForkJoinTask.reportException(ForkJoinTask.java:677)
[error]         at java.util.concurrent.ForkJoinTask.invoke(ForkJoinTask.java:735)
[error]         at java.util.stream.ForEachOps$ForEachOp.evaluateParallel(ForEachOps.java:159)
[error]         at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateParallel(ForEachOps.java:173)
[error]         at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:233)
[error]         at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:485)
[error]         at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:650)
[error]         at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:563)
[error]         at apoc.export.cypher.formatter.AbstractCypherFormatter.buildStatementForNodes(AbstractCypherFormatter.java:134)
[error]         at apoc.export.cypher.formatter.CreateCypherFormatter.buildStatementForNodes(CreateCypherFormatter.java:18)
[error]         at apoc.export.cypher.formatter.CreateCypherFormatter.statementForNodes(CreateCypherFormatter.java:57)
[error]         at apoc.export.cypher.MultiStatementCypherSubGraphExporter.exportNodesUnwindBatch(MultiStatementCypherSubGraphExporter.java:131)
[error]         at apoc.export.cypher.MultiStatementCypherSubGraphExporter.export(MultiStatementCypherSubGraphExporter.java:106)
[error]         at apoc.export.cypher.ExportCypher.doExport(ExportCypher.java:127)
[error]         at apoc.export.cypher.ExportCypher.exportCypher(ExportCypher.java:116)
[error]         at apoc.export.cypher.ExportCypher.all(ExportCypher.java:51)
[error]         at org.deepanalyzer.GraphUtil$.testExport(GraphUtil.scala:252)
[error]         at org.deepanalyzer.GraphBuilder$.calculHotSpotWalkInGraph(GraphBuilder.scala:194)
[error]         at org.deepanalyzer.GraphBuilder$.genereGraph(GraphBuilder.scala:541)
[error]         at org.deepanalyzer.Main$.main(Main.scala:128)
[error]         at org.deepanalyzer.Main.main(Main.scala)
[error]         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[error]         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
[error]         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[error]         at java.lang.reflect.Method.invoke(Method.java:498)
[error] Caused by: org.neo4j.graphdb.NotFoundException: org.neo4j.internal.kernel.api.exceptions.EntityNotFoundException: Unable to load NODE with id 0.
[error]         at org.neo4j.kernel.impl.core.NodeProxy.singleNode(NodeProxy.java:836)
[error]         at org.neo4j.kernel.impl.core.NodeProxy.getLabels(NodeProxy.java:657)
[error]         at apoc.export.util.FormatUtils.getLabelsAsStream(FormatUtils.java:102)
[error]         at apoc.export.util.FormatUtils.getLabelsSorted(FormatUtils.java:98)
[error]         at apoc.export.cypher.formatter.CypherFormatterUtils.getNodeIdProperties(CypherFormatterUtils.java:58)
[error]         at apoc.export.cypher.formatter.AbstractCypherFormatter.lambda$buildStatementForNodes$5(AbstractCypherFormatter.java:127)
[error]         at java.util.stream.Collectors.lambda$groupingByConcurrent$51(Collectors.java:1070)
[error]         at java.util.stream.ReferencePipeline.lambda$collect$1(ReferencePipeline.java:563)
[error]         at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183)
[error]         at java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:948)
[error]         at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
[error]         at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290)
[error]         at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731)
[error]         at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
[error]         at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
[error]         at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
[error]         at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175)
[error] Caused by: org.neo4j.internal.kernel.api.exceptions.EntityNotFoundException: Unable to load NODE with id 0.
[error]         at org.neo4j.kernel.impl.core.NodeProxy.singleNode(NodeProxy.java:836)
[error]         at org.neo4j.kernel.impl.core.NodeProxy.getLabels(NodeProxy.java:657)
[error]         at apoc.export.util.FormatUtils.getLabelsAsStream(FormatUtils.java:102)
[error]         at apoc.export.util.FormatUtils.getLabelsSorted(FormatUtils.java:98)
[error]         at apoc.export.cypher.formatter.CypherFormatterUtils.getNodeIdProperties(CypherFormatterUtils.java:58)
[error]         at apoc.export.cypher.formatter.AbstractCypherFormatter.lambda$buildStatementForNodes$5(AbstractCypherFormatter.java:127)
[error]         at java.util.stream.Collectors.lambda$groupingByConcurrent$51(Collectors.java:1070)
[error]         at java.util.stream.ReferencePipeline.lambda$collect$1(ReferencePipeline.java:563)
[error]         at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183)
[error]         at java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:948)
[error]         at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
[error]         at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290)
[error]         at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731)
[error]         at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
[error]         at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
[error]         at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
[error]         at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175)

Precision : my code only add node and never remove them...
Precision2 : All the transactions are closed and another is launch to make the export

• When I close the database and reopen it with newEmbeddedDatabase (I'm desperate, I try...) I get
  throw java.lang.RuntimeException: Export to files not enabled, please set apoc.export.file.enabled=true in your neo4j.conf because with newEmbeddedDatabase I can't define config parameter

How can I give it the right config items ?
I tried a CALL dbms.setConfigValue('apoc.export.file.enabled', 'true') inside but I get
throw org.neo4j.graphdb.QueryExecutionException: There is no procedure with the name dbms.setConfigValue registered for this database instance. Please ensure you've spelled the procedure name correctly and that the procedure is properly deployed.

• The exportCSV work (why ??) but how I can import a single CSV file, I can do nothing with this

So, do you have, please, any solutions ?

In advance Thank you

[michael_hunger]

The easiest thing to do is the following.

1. shut down your application
2. take the database folder and copy it into a server install
3. start up the server to make sure that the db is up and running, potentially need to configure active database
4. shut down server
5. do neo4j-admin dump --database --to file.dump

then you can use that dump in other contexts e.g. importing it into a 4.x database, uploading to aura, use in desktop, seed a docker instance etc.

[ontologiae]

Hi Michael,
It's precisely the process I'm doing for 2 years now and I would like to avoid that, because I have 80 neo4j different database to build..
What I don't understand is why I lost data when I shut down the database. When I reopen it, there's only 5% of the nodes that remaining. I carefully verify all the transactions

[ontologiae]

So I tried the old good method you describe, and guess what, 95% of the data disapeared...

[michael_hunger]

But you copy the whole database folder which includes all neostore.transaction.* files?

And you shut your embedded application cleanly down before doing that?

[ontologiae]

It's exactly what I did : I copy the whole directory, I even tried to define the directory as neo4j's server directory, and I always get 5% of the data remaining.
I shutdown programmatically the database before

[ontologiae]

Michael, sorry, I missed a tx.close somewhere, I looked at everywhere but forgot just one.
And now, it works !
Thank you !

For the record the code needed to makes it work :

new GraphDatabaseFactory()
      .newEmbeddedDatabaseBuilder(new java.io.File(path))
      .setConfig(GraphDatabaseSettings.read_only, "true")
      .setConfig("apoc.export.file.enabled", "true")
      .setConfig("dbms.directories.import", "/home/path/")
      .setConfig("dbms.security.procedures.whitelist", "apoc.*")
      .newGraphDatabase()

val proceduresService = db.asInstanceOf[GraphDatabaseAPI].getDependencyResolver().resolveDependency(classOf[Procedures]);
    proceduresService.registerProcedure(classOf[apoc.export.cypher.ExportCypher])