Neo4j-admin check-consistency potentially broken

[aaron]

Hey there everybody!

Hoping someone might have some insight on this...

I've been running into issues with the consistency checking tool in the form of using

neo4j-admin backup

and

neo4j check-consistency

What happens is when neo4j-admin backup is run with the consistency check, the consistency check makes it to about 90% and then presents the following error message:

Consistency checking failed. Full consistency check did not complete

This is occurring on a causal cluster.

Now, what I've done is run the backup command without the consistency check, and I've taken that backup data and loaded it into a local enterprise edition neo4j server.

The data loads just fine when running the 'restore' tool from neo4j-admin, but things get a little weird when attempting a backup with the local version.

The local version takes a very long time, and outputs only a series of .'s, never making it to the point where it shows a completion percentage, e.g. ".................. 10%" etc.

It produces the same error message:

Consistency checking failed. Full consistency check did not complete

Now, when running the consistency-check tool from neo4j-admin, what I have found is that the tool will make it through consistency checking if I set check-indexes=false. So there seems to be a problem with the index check.

Note, if I detach and delete all nodes from the local db (can't do this to the prod db), and then run the backup tool, it runs just fine.

Final note, I have tried running jstack and it shows many threads 'parked', which, I think is normal. I've also tried adjusting pagecache and heap size settings to no avail. (Increased them to very large sizes, that were appropriate for the local machine).

So, I'm not sure if this is a bug, but because there is no specific error message output, and because no inconsistency report is generated, it is a bit odd...

Really hoping to get this one solved, as a backup isn't really useful unless its consistency can be fully validated.

Any insight would be greatly appreciated, I know this was a long read!

[mrksph]

Hi

We're experiencing the same problem. We also have a causal cluster with 3 nodes and when backing up the database we get an error code 2 Back up completed but consistency check failed

Any updates on the topic?

Thanks

[dana_canzano]

@mrksph @aaron

any details on what Neo4j version is in play here?

[mrksph]

v4.2.11

Although the DB guys just told us they solved the problem, it seems the binaries used to make the backups were a different version that the clusters'

Thanks Dana

[krutagn]

I am having the same issue, if I run consistency check on backup or the whole database it hangs on 70% and take forever to complete. configured heap size 5000M and pagecache = 6700M
If I do check-index=false then it runs perfectly fine. So having issue with index check.
I am using neo4j enterprise edition (startup) 3.5.22.

Any solutions?

Thanks,
Krutagn