Upgrade fails from Enterprise 4.0.4 to 4.1.0

[roger_gordon]

Hi,

Hope this is the right topic - I searched and this seemed the best fit.

I am trying to upgrade from Enterprise 4.0.4 to 4.1.0 via the Desktop app on a Mac. I get this error: Failed to upgrade database. Please check log files, make adjustments and try again.

The database was recreated in version 4 after I was unable to move it from 3.5, but has upgraded successfully from 4.02 through to 4.0.4
However, my 4.0.4 logs are silent. There was a previous error pointing to the import directory, and I moved the import directory but still no luck.

The Dev console reports:

Action[DATABASE_UPGRADING]

1. {id: "database-802c51a5-15e4-4701-93aa-657b4ce439f0", type: "DATABASE_UPGRADING", version: "4.1.0"}

Desktop.474c039126a2…35dd0.bundle.js:113 20:16:42.506 › Function

Desktop.474c039126a2…35dd0.bundle.js:113 20:17:42.282 › Action[DATABASE_UPGRADE_FAILED]

1. {id: "database-802c51a5-15e4-4701-93aa-657b4ce439f0", type: "DATABASE_UPGRADE_FAILED"}

2. id: "database-802c51a5-15e4-4701-93aa-657b4ce439f0"

3. type: "DATABASE_UPGRADE_FAILED"

4. proto:

1. constructor: ƒ Object()
2. hasOwnProperty: ƒ hasOwnProperty()
3. isPrototypeOf: ƒ isPrototypeOf()
4. propertyIsEnumerable: ƒ propertyIsEnumerable()
5. toLocaleString: ƒ toLocaleString()
6. toString: ƒ toString()
7. valueOf: ƒ valueOf()
8. __defineGetter__: ƒ __defineGetter__()
9. __defineSetter__: ƒ __defineSetter__()
10. __lookupGetter__: ƒ __lookupGetter__()
11. __lookupSetter__: ƒ __lookupSetter__()
12. get __proto__: ƒ __proto__()
13. set __proto__: ƒ __proto__()

Two other possible clues:

1. Cloning also fails for this DB with this error (it seems fussed about a backup in this directory?):

Failed to clone a graph: Error: Unknown system error -210742476: Unknown system error -210742476, copyfile '/Users/roger/Library/Application Support/Neo4j Desktop/Application/neo4jDatabases/database-802c51a5-15e4-4701-93aa-657b4ce439f0/installation-4.0.4/mnt/backups/neo4j/schema/index/native-btree-1.0/3/index-3' -> '/Users/roger/Library/Application Support/Neo4j Desktop/Application/neo4jDatabases/database-33939c36-b116-4fae-b6fc-3475349d6165/installation-4.0.4/mnt/backups/neo4j/schema/index/native-btree-1.0/3/index-3'

2. Trying to startup a newly created (empty, no plugins) v4.1.0 database yields an error:
   Database failed to start: DB [database-7275a8c1-cef7-4692-925f-d9e060a163fe] 'v4.1.0' exited with status 'KILLED'. Check the logs

The error looks like a listening setting; here's that part of the config:

Bolt connector

dbms.connector.bolt.enabled=true
dbms.connector.bolt.tls_level=DISABLED
dbms.connector.bolt.listen_address=:7687

HTTP Connector. There can be zero or one HTTP connectors.

dbms.connector.http.enabled=true
dbms.connector.http.listen_address=:7474

HTTPS Connector. There can be zero or one HTTPS connectors.

dbms.connector.https.enabled=false
#dbms.connector.https.listen_address=:7473

and here's the error in the 4.1.0 logs:

2020-07-01 20:15:27.406+0000 INFO ======== Neo4j 4.1.0 ========
2020-07-01 20:15:33.716+0000 ERROR Failed to start Neo4j on dbms.connector.http.listen_address, a socket address. If missing port or hostname it is acquired from dbms.default_listen_address. Error starting Neo4j database server at /Users/roger/Library/Application Support/Neo4j Desktop/Application/neo4jDatabases/database-7275a8c1-cef7-4692-925f-d9e060a163fe/installation-4.1.0/data/databases
java.lang.RuntimeException: Error starting Neo4j database server at /Users/roger/Library/Application Support/Neo4j Desktop/Application/neo4jDatabases/database-7275a8c1-cef7-4692-925f-d9e060a163fe/installation-4.1.0/data/databases
at org.neo4j.graphdb.facade.DatabaseManagementServiceFactory.startDatabaseServer(DatabaseManagementServiceFactory.java:198)
at org.neo4j.graphdb.facade.DatabaseManagementServiceFactory.build(DatabaseManagementServiceFactory.java:158)
at com.neo4j.server.enterprise.EnterpriseManagementServiceFactory.createManagementService(EnterpriseManagementServiceFactory.java:38)
at com.neo4j.server.enterprise.EnterpriseBootstrapper.createNeo(EnterpriseBootstrapper.java:20)
at org.neo4j.server.NeoBootstrapper.start(NeoBootstrapper.java:117)
at org.neo4j.server.NeoBootstrapper.start(NeoBootstrapper.java:87)
at com.neo4j.server.enterprise.EnterpriseEntryPoint.main(EnterpriseEntryPoint.java:25)
Caused by: org.neo4j.kernel.lifecycle.LifecycleException: Component 'com.neo4j.dbms.StandaloneDbmsReconcilerModule@2a8f6e6' was successfully initialized, but failed to start. Please see the attached cause exception "Transaction logs are missing and recovery is not possible.".
at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:463)
at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:110)
at org.neo4j.graphdb.facade.DatabaseManagementServiceFactory.startDatabaseServer(DatabaseManagementServiceFactory.java:189)
... 6 more
Caused by: org.neo4j.dbms.api.DatabaseManagementException: A triggered DbmsReconciler job failed with the following cause
at com.neo4j.dbms.ReconcilerResult.join(ReconcilerResult.java:57)
at com.neo4j.dbms.StandaloneDbmsReconcilerModule.startInitialDatabases(StandaloneDbmsReconcilerModule.java:95)
at com.neo4j.dbms.StandaloneDbmsReconcilerModule.start(StandaloneDbmsReconcilerModule.java:85)
at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:442)
... 8 more
Caused by: org.neo4j.dbms.api.DatabaseManagementException: An error occurred! Unable to start database with name system.
at org.neo4j.dbms.database.AbstractDatabaseManager.startDatabase(AbstractDatabaseManager.java:191)
at com.neo4j.dbms.database.MultiDatabaseManager.forSingleDatabase(MultiDatabaseManager.java:134)
at com.neo4j.dbms.database.MultiDatabaseManager.startDatabase(MultiDatabaseManager.java:119)
at com.neo4j.dbms.Transition$Prepared.doTransitionAction(Transition.java:101)
at com.neo4j.dbms.Transition$Prepared.doTransition(Transition.java:88)
at com.neo4j.dbms.DbmsReconciler.doTransitionStep(DbmsReconciler.java:346)
at com.neo4j.dbms.DbmsReconciler.doTransitionStep(DbmsReconciler.java:347)
at com.neo4j.dbms.DbmsReconciler.doTransitionStep(DbmsReconciler.java:347)
at com.neo4j.dbms.DbmsReconciler.lambda$doTransitions$11(DbmsReconciler.java:315)
at com.neo4j.dbms.DbmsReconciler.namedJob(DbmsReconciler.java:326)
at com.neo4j.dbms.DbmsReconciler.doTransitions(DbmsReconciler.java:316)
at com.neo4j.dbms.DbmsReconciler.lambda$doTransitions$9(DbmsReconciler.java:307)
at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1700)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.lang.RuntimeException: org.neo4j.kernel.lifecycle.LifecycleException: Component 'org.neo4j.kernel.recovery.Recovery$MissingTransactionLogsCheck@3dc457d1' failed to initialize. Please see the attached cause exception "Transaction logs are missing and recovery is not possible.".
at org.neo4j.kernel.database.Database.start(Database.java:497)
at org.neo4j.dbms.database.AbstractDatabaseManager.startDatabase(AbstractDatabaseManager.java:187)
... 17 more
Caused by: org.neo4j.kernel.lifecycle.LifecycleException: Component 'org.neo4j.kernel.recovery.Recovery$MissingTransactionLogsCheck@3dc457d1' failed to initialize. Please see the attached cause exception "Transaction logs are missing and recovery is not possible.".
at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.init(LifeSupport.java:424)
at org.neo4j.kernel.lifecycle.LifeSupport.init(LifeSupport.java:65)
at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:101)
at org.neo4j.kernel.recovery.Recovery.performRecovery(Recovery.java:384)
at org.neo4j.kernel.database.Database.start(Database.java:389)
... 18 more
Caused by: java.lang.RuntimeException: Transaction logs are missing and recovery is not possible.
at org.neo4j.kernel.recovery.Recovery$MissingTransactionLogsCheck.checkForMissingLogFiles(Recovery.java:549)
at org.neo4j.kernel.recovery.Recovery$MissingTransactionLogsCheck.init(Recovery.java:522)
at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.init(LifeSupport.java:403)
... 22 more
2020-07-01 20:15:33.716+0000 INFO Neo4j Server shutdown initiated by request

Sorry this is so long,
Roger

[folterj]

We have the same issue on neo4j Desktop.
After removing Apoc (including manually removing archive file still located in Plugins folder), we got a Java error instead (see error messages below).

Here is what we tried (trying to upgrade our db to 4.1.0 after each step):

• Uninstall APOC
• Manually remove apoc archive file from Plugin folder
  (from this point onward instead of the initial error message we got the Java exception)
• Upgrade Neo4j Desktop to v1.3.1
• Upgrade db from 4.0.4 to 4.0.6 (works)
• Uninstall neo4j (remove all files)
• Install latest version of neo4j Desktop from website (v1.2.9)
• Upgrade Neo4j Desktop to v1.3.1
• Upgrade movie or blank db from 4.0.4 to 4.1.0 (works)
• Create blank db 4.1.0, import our db dump (fails to run)
• Create blank db 4.0.4, import our db dump (works)
• Upgrade db to 4.1.0 (fails to upgrade)

Initial Apoc related error message:

2020-07-08 12:40:25.710+0000 INFO Starting...
2020-07-08 12:40:28.892+0000 INFO ======== Neo4j 4.1.0 ========
2020-07-08 12:40:28.927+0000 ERROR Failed to start Neo4j on dbms.connector.http.listen_address, a socket address. If missing port or hostname it is acquired from dbms.default_listen_address. Error starting Neo4j database server at C:\Users\Y.Neo4jDesktop\neo4jDatabases\database-9f7e3ce9-0d72-4c2e-a59c-2002ef1a1073\installation-4.1.0\data\databases
java.lang.RuntimeException: Error starting Neo4j database server at C:\Users\Y.Neo4jDesktop\neo4jDatabases\database-9f7e3ce9-0d72-4c2e-a59c-2002ef1a1073\installation-4.1.0\data\databases
at org.neo4j.graphdb.facade.DatabaseManagementServiceFactory.startDatabaseServer(DatabaseManagementServiceFactory.java:198)
at org.neo4j.graphdb.facade.DatabaseManagementServiceFactory.build(DatabaseManagementServiceFactory.java:158)
at com.neo4j.server.enterprise.EnterpriseManagementServiceFactory.createManagementService(EnterpriseManagementServiceFactory.java:38)
at com.neo4j.server.enterprise.EnterpriseBootstrapper.createNeo(EnterpriseBootstrapper.java:20)
at org.neo4j.server.NeoBootstrapper.start(NeoBootstrapper.java:117)
at org.neo4j.server.NeoBootstrapper.start(NeoBootstrapper.java:87)
at com.neo4j.server.enterprise.EnterpriseEntryPoint.main(EnterpriseEntryPoint.java:25)
Caused by: org.neo4j.kernel.lifecycle.LifecycleException: Component 'org.neo4j.kernel.extension.GlobalExtensions@75f5fd58' failed to initialize. Please see the attached cause exception "logical_logs_location".
at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.init(LifeSupport.java:424)
at org.neo4j.kernel.lifecycle.LifeSupport.init(LifeSupport.java:65)
at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:101)
at org.neo4j.graphdb.facade.DatabaseManagementServiceFactory.startDatabaseServer(DatabaseManagementServiceFactory.java:189)
... 6 more
Caused by: org.neo4j.kernel.extension.FailedToBuildExtensionException: Failed to build kernel extension Extension:ApocConfigExtensionFactory[ApocConfig] because it is compiled with a reference to a class, method, or field, that is not in the class path: 'logical_logs_location'. The most common cause of this problem, is that Neo4j has been upgraded without also upgrading allinstalled extensions, such as APOC. Make sure that all of your extensions are build against your specific version of Neo4j.
at org.neo4j.kernel.extension.ExtensionFailureStrategies.wrap(ExtensionFailureStrategies.java:60)
at org.neo4j.kernel.extension.ExtensionFailureStrategies$1.handle(ExtensionFailureStrategies.java:76)
at org.neo4j.kernel.extension.AbstractExtensions.init(AbstractExtensions.java:72)
at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.init(LifeSupport.java:403)
... 9 more
Caused by: java.lang.NoSuchFieldError: logical_logs_location
at apoc.ApocConfig.(ApocConfig.java:62)
at apoc.ApocConfigExtensionFactory.newInstance(ApocConfigExtensionFactory.java:30)
at apoc.ApocConfigExtensionFactory.newInstance(ApocConfigExtensionFactory.java:15)
at org.neo4j.kernel.extension.AbstractExtensions.newInstance(AbstractExtensions.java:130)
at org.neo4j.kernel.extension.AbstractExtensions.init(AbstractExtensions.java:62)
... 10 more
2020-07-08 12:40:28.928+0000 INFO Neo4j Server shutdown initiated by request

Java exception error:

2020-07-08 12:56:27.214+0000 INFO Starting...
2020-07-08 12:56:29.822+0000 INFO ======== Neo4j 4.1.0 ========
2020-07-08 12:56:34.565+0000 INFO Sending metrics to CSV file at C:\Users\Y.Neo4jDesktop\neo4jDatabases\database-9f7e3ce9-0d72-4c2e-a59c-2002ef1a1073\installation-4.1.0\metrics
2020-07-08 12:56:34.607+0000 INFO Bolt enabled on localhost:7687.

A fatal error has been detected by the Java Runtime Environment:

EXCEPTION_ACCESS_VIOLATION (0xc0000005) at pc=0x00007ff8199a4323, pid=16752, tid=6852

JRE version: OpenJDK Runtime Environment (Zulu11.35+13-CA) (11.0.5+10) (build 11.0.5+10-LTS)
Java VM: OpenJDK 64-Bit Server VM (11.0.5+10-LTS, mixed mode, tiered, compressed oops, g1 gc, windows-amd64)
Problematic frame:
V [jvm.dll+0x6a4323]

No core dump will be written. Minidumps are not enabled by default on client versions of Windows

An error report file with more information is saved as:
C:\Users\Y\AppData\Local\Temp\hs_err_pid16752.log

If you would like to submit a bug report, please visit:
http://www.azulsystems.com/support/

Extract from mentioned Java log file:

...
Internal exceptions (20 events):
Event: 11.007 Thread 0x0000020a6ea90800 Exception <a 'java/lang/NoSuchMethodError'{0x00000000de8cde08}: 'java.lang.Object java.lang.invoke.DirectMethodHandle$Holder.invokeInterface(java.lang.Object, java.lang.Object, java.lang.Object, java.lang.Object)'> (0x00000000de8cde08) thrown at [c:/jenki
Event: 11.044 Thread 0x0000020a0faf0000 Exception <a 'java/lang/NullPointerException'{0x00000000de16d6b0}> (0x00000000de16d6b0) thrown at [c:/jenkins/workspace/zulu11-build-win64/zulu-src/src/hotspot/share/interpreter/linkResolver.cpp, line 1358]
Event: 11.044 Thread 0x0000020a0faf0000 Exception <a 'java/lang/reflect/InvocationTargetException'{0x00000000de16dc28}> (0x00000000de16dc28) thrown at [c:/jenkins/workspace/zulu11-build-win64/zulu-src/src/hotspot/share/runtime/reflection.cpp, line 1246]
Event: 11.045 Thread 0x0000020a0faf0000 Exception <a 'java/lang/reflect/InvocationTargetException'{0x00000000de1738f8}> (0x00000000de1738f8) thrown at [c:/jenkins/workspace/zulu11-build-win64/zulu-src/src/hotspot/share/runtime/reflection.cpp, line 1246]
Event: 11.144 Thread 0x0000020a0faef800 Exception <a 'sun/nio/fs/WindowsException'{0x00000000dd2e0218}> (0x00000000dd2e0218) thrown at [c:/jenkins/workspace/zulu11-build-win64/zulu-src/src/hotspot/share/prims/jni.cpp, line 615]
...

We have found that issues in neo4j Desktop are often resolved in the neo4j linux enterprise version, however in this case we hesitate upgrading our server to 4.1.0. Any help appreciated.

[roger_gordon]

Hi folterj,

I had no plug-ins installed on 4.10; it was a newly created instance that won't start up at all.

V 4.0.4 has APOC 4.0.0.11, but removing and rebooting still doesn't allow an upgrade.

I am unclear if I'm having multiple problems or it's all one big unified problem. Previous upgrades within v4 worked until trying to upgrade from 4.0.4 Enterprise.

Desktop at Version 1.3.1 (1.3.1.20).

My 4.10 also shows out the dbms.connector.http.listen_address error, a problem I previously "fixed" by rebooting or killing java processes

Thanks for replying, it was starting to feel lonely ;^>

Any help would also be really great.

R.

[folterj]

Hi @roger.gordon, we're familiar with that feeling.

It seems our issues may not be related, I certainly don't know what's going on in your case so hope neo4j staff will be able to shed light on things.

It does seem version 4.1.0 introduces a lot of new functionality, so maybe some issues will disappear in version 4.1.1.
In general, we often end up resorting to work-arounds or costly trial/error approaches unfortunately. A very common solution is using the normal server (Linux) installation instead of the Desktop version, in particular in case of database inconsistencies etc. Although I've not found any mention, I imagine the database format may have changed as well between 4.0 to 4.1.

In case your issue is a connection type issue (doing an implicit neo4j Desktop restart used to result in port conflicts as they weren't closed down), perhaps it's possible to try 1. make 'dump' backup, 2. create an empty 4.1.0 db, 3. 'load' old db into it - backing up your database of course and being able to recreate it without loosing anything.

In our case we would simply update regular neo4j Linux server to upgrade, then backup that database for use in neo4j Desktop. However in this case we can't risk loosing our database server and needing to reinstall an older version.

[roger_gordon]

Hi folterj & thanks again.

Following up on your suggestion, I did create a blank 4.0.4 instance, then upgraded successfully step by step to 4.0.5, 4.0.6, & then 4.1.0 so that works. Interestingly, the port / connector config was commented out in the config when I looked at the 4.1.0 install (this is all default behaviour as I didn't edit):

#Bolt connector
dbms.connector.bolt.enabled=true
#dbms.connector.bolt.tls_level=DISABLED
#dbms.connector.bolt.listen_address=:7687

#HTTP Connector. There can be zero or one HTTP connectors.
dbms.connector.http.enabled=true
#dbms.connector.http.listen_address=:7474

#HTTPS Connector. There can be zero or one HTTPS connectors.
dbms.connector.https.enabled=false
#dbms.connector.https.listen_address=:7473

Will try to load a dump file - because of a general concern about being stuck in a specific version unable to upgrade, and because of the allure of the unsolved problem.

Agree with you in general and specific about .0 versions, although I strongly suspect this might be my instance configuration / machine problem.

BTW, at about 70m node labels and over 100m relationships.

[folterj]

An update on this:

• Using neo4j Desktop v1.3.3
• Removing all constraints and indexes allows the update (4.0.4 -> 4.1.0)
• Regenerating constraints and indexes works fine as well
• Subsequently (re)starting the db after this reverts to the original Java EXCEPTION_ACCESS_VIOLATION error

It would be great to receive any feedback

[folterj]

Another update on this:

• Using neo4j Desktop v1.3.4
• The upgrade appeared as successful (4.0.4 -> 4.1.1)
• Subsequently (re)starting the db after this reverts to the original Java EXCEPTION_ACCESS_VIOLATION error

Any kind of insight on the issues with neo4j Desktop or support would be appreciated

[roger_gordon]

Sorry that you are having such troubles.

I found that creating a fresh instance of Enterprise 4.1.1 with the default configuration, and then loading a dump file from the 4.0.4 - indices and all - was successful.

The difference is that I accepted the default settings and then added specific config lines one at a time. Right now that is working. I never did get the upgrade working from 4.0.4 -> 4.anything using the upgrade process in desktop Version 1.3.4 (1.3.4.27) or desktop versions previous to that.

[folterj]

Hi,

Glad to hear your issue is solved @roger.gordon.

We decided to try to work with neo4j server (linux) directly to see if this would resolve the issue:

• New db created in neo4j enterprise server (linux) 4.1.1
• Imported all our data using exported cyphers - this runs fine, consistency check fine as well
• New dump file created
• neo4j Desktop: Imported dump file in blank DB 4.1.1 - the same Java exception once again
• neo4j Desktop: Imported dump file in blank DB 4.0.4 - this works fine

We've spent a lot of time on this and still have no idea why it's not working - to find out what's wrong would require debugging using neo4j dev setup? The log files don't seem to give meaningful information.

So at the moment we can't use neo4j Desktop (Windows) version 4.1.x for development unfortunately.

[andrew_bowman]

Hello,

If you have not already, please open an issue with details on our Neo4j Desktop github:

GitHub

neo4j-apps/neo4j-desktop

The Neo4j Graph Platform, on your desktop OS. . Contribute to neo4j-apps/neo4j-desktop development by creating an account on GitHub.

[folterj]

Hi @andrew.bowman,

Thanks for the tip, I've done that as well now:

github.com/neo4j-apps/neo4j-desktop

neo4j Desktop 4.1.x database failure (on Windows)

opened 09:01PM - 26 Aug 20 UTC

folterj

Using neo4j Desktop on Windows 10 64-bit, unable to upgrade from version 4.0.4 to version 4.1.0 or 4.1.1. This applies to upgrading...

[folterj]

Update on this issue - thanks to Angelo @nglgzz

It appears this is now solved.

Things we tested:

• upgrade v4.0 to v4.1 small graphs: ok
• upgrade v4.0 to v4.1: fail
• import dump or cypher into v4.0: ok
• import dump or cypher into v4.1: fail
• import cypher into 4.1 linux/server: ok
• export 4.1 dump and import into v4.0: ok

Memory:

Using neo4j, Docker, and our own Java core code, optimising memory allocation gets quite complex, so we're relying on neo4j's auto configuration, which has worked better than trying to manually configure it. On a desktop with 16GB RAM, the idle spun up usage is about 2GB.

However it seems that neo4j Desktop (in contrast to neo4j server) automatically sets low limits by default:
dbms.memory.heap.max_size=1G
dbms.memory.heap.initial_size=512m
dbms.memory.pagecache.size=512m

Whereas allowing neo4j to auto-configure - which is what we want and were expecting - sets:
dbms.memory.heap.max_size 4278190080 bytes (4.3GB)
dbms.memory.heap.initial_size 268435456 bytes (0.2GB)
dbms.memory.pagecache.size 6416367616 bytes (6.4GB)

Some unresolved mysteries:

• why does load linux/server 4.1 dump into neo4j Desktop 4.1 fail as this does not require any upgrade conversion
• why does the error message say nothing about memory
• why does neo4j desktop not use default auto-allocation like neo4j server, instead of setting arbitrary low default values

Thank you for your help everyone.