Can't stop neo4j in Ubuntu

[yyakovson613]

I'm running Ubuntu 16.04.4 on our server, with the Neo4j Enterprise Edition, 3.4.0.

We just lost a lot of data, so I'm attempting a restore, following the instructions at https://neo4j.com/docs/operations-manual/current/backup/restoring/:

1. If the database is running, shut it down.
2. Run neo4j-admin restore .
3. Start up the database.

The problem is that I can't figure out how to shut it down. The example provided uses

neo4j-home> bin/neo4j stop

Running systemctl cat neo4j indicates that NEO4J_HOME=/var/lib/neo4j, but it contains no bin directory.

When I run sudo systemctl stop neo4j, it does add a line

Dec 04 11:40:27 ip-172-31-59-200 systemd[1]: Stopped Neo4j Graph Database.
to the output of systemctl status neo4j.service. But every time I try to run restore I get:
org.neo4j.commandline.admin.CommandFailed: the database is in use -- stop Neo4j and try again

Any timely information greatly appreciated.

[ben_butler-cole]

Hello. I'm sorry that you're having problems with this.

A couple of questions:

1. Did you install Neo4j using the Debian or RPM packages?
2. Did you originally start Neo4j using sudo systemctl start neo4j?

[yyakovson613]

Hi @ben.butler-cole , I actually created an AWS EC2 image from @david.allen at Neo4j. (I don't know any more details that that. We communicated on our private slack channel on https://neo4j-users.slack.com/. The conversation since disappeared because of the Slack limit of 10000 messages, and I don't even know which image it was anymore.)

I do not know what I did originally to start the db. Is there a way to find that out?

[yyakovson613]

@ben.butler-cole:

For what it's worth, sudo systemctl stop neo4j is indeed stopping the database, because when I run it our browser immediately loses it's connection until I run sudo systemctl start neo4j.

I'm running this restore command:

neo4j-admin restore --from /home/ubuntu/backups/2018-12-04-graph.db-backup --database=graph.db --force

I get this result:

org.neo4j.commandline.admin.CommandFailed: the database is in use -- stop Neo4j and try again
	at org.neo4j.commandline.Util.checkLock(Util.java:90)
	at org.neo4j.restore.RestoreDatabaseCommand.execute(RestoreDatabaseCommand.java:84)
	at org.neo4j.restore.RestoreDatabaseCli.execute(RestoreDatabaseCli.java:90)
	at org.neo4j.commandline.admin.AdminTool.execute(AdminTool.java:127)
	at org.neo4j.commandline.admin.AdminTool.main(AdminTool.java:51)
Caused by: org.neo4j.kernel.StoreLockException: Unable to obtain lock on store lock file: /var/lib/neo4j/data/databases/graph.db/store_lock. Please ensure no other process is using this database, and that the directory is writable (required even for read-only access)
	at org.neo4j.kernel.internal.locker.StoreLocker.storeLockException(StoreLocker.java:117)
	at org.neo4j.kernel.internal.locker.StoreLocker.unableToObtainLockException(StoreLocker.java:110)
	at org.neo4j.kernel.internal.locker.StoreLocker.checkLock(StoreLocker.java:98)
	at org.neo4j.kernel.internal.locker.GlobalStoreLocker.checkLock(GlobalStoreLocker.java:60)
	at org.neo4j.commandline.Util.checkLock(Util.java:86)
	... 4 more
command failed: the database is in use -- stop Neo4j and try again

[ben_butler-cole]

Ah, that's good to know.

Did you modify the SystemD unit that came in the image yourself? I'll assume not unless you say otherwise. If you did then I'll need to know what modifications you made.

We need to find the Neo4j data directory. If you're running it via SystemD then it is probably in /var/lib/neo4j. Can you ensure that all files underneath that directory are owned by the neo4j user?

[yyakovson613]

Further update: I've run fuser on the data directory and even the home directory and am getting nothing.

When I try neo4j-admin report I get

Finding running instance of neo4j
No running instance of neo4j was found. Online reports will be omitted.```

Interestingly, it's failing to write a report due an exception.  Could that indicate a problem in my configuration?

[yyakovson613]

@ben.butler-cole I think so... Here are the contents of that directory:

ubuntu@ip-172-31-59-200:/var/lib/neo4j/data$ ls -l
total 8
drwxr-xr-x 3 neo4j adm   4096 Jun 14 19:15 databases
drwxr-xr-x 2 neo4j neo4j 4096 Jun 25 14:02 dbms

Running a recursive listing shows that neo4j owns all of the files in subdirectories.

[ben_butler-cole]

Processes that write to the data directory (or other locations owned by neo4j) need to run as the neo4j user. You could try running the command with sudo -u neo4j ....

[yyakovson613]

Ben: I'm sorry, could you please clarify which command you mean? The restore command, the stop command or the report command?

[n_shirmohamadi]

Tnx a lot. Use sudo -u neo4j neo4j-admin solved my problem.

[ben_butler-cole]

The SystemD unit takes care of this kind of thing itself. So you should not set the user when calling systemctl.

neo4j-admin commands may need write access. Certainly neo4j-admin restore does and should always be invoked as the neo4j user.

I'm surprised if neo4j-admin report needs to be the neo4j user, but that's what the error you reported suggested to me.

[yyakovson613]

@ben.butler-cole I can't thank you enough--that was the problem! I didn't realize that I needed to run network-admin as -u neo4j.