Few questions regarding the usage control

[liang_yeh]

We are working on the monitor solution to display the server health, memory usage, cpu usage, disk storage usage and top SQL in Cypher. Can Neo4j represent those data? If Neo4j is not able to provide those information, is there any 3rd party solution that can use or can we use the system logs to analyze the data with any API?

Another question. If the database we set with the limit that is about 100% usage, how do we deal with this situation? Can we stop the user to continue growing the database usage? Please share any suggestions or ideas on how to prevent this happen.

Liang

[Joel]

Sounds like you may want something like the neo4j desktop Halin plugin for Neo4j monitoring...

If you want to duplicate some of that functionality in another form, the source code is available at

github.com

moxious/halin

Neo4j monitoring tool. Contribute to moxious/halin development by creating an account on GitHub.

[liang_yeh]

Hi Joel,

Thank you for getting back with the solution.
For the second question, do you have experience or ideas on how to deal the database been used 100%?

Liang

[Joel]

If you mean resources, like cpu, memory and disk? Try out Halin, it shows a lot of useful information about the database, and some information on the host as well. If that doesn't give you what you want, I think you may need to use other tools (e.g. the AWS console) or via access to the host(s).

If you didn't mean resources, please define "used 100%".

[liang_yeh]

Hi Joel,

That helps for our question 1 on the monitoring.
My second question is more for the usage situation if we set a limit to a database that uses about 100% storage. Are there any scenarios how to deal with that? ex. Stop that database to grow the usage or let it continue to grow but send system notes to the users to delete some data? I am not sure what is the best way to handle the usage situations in Neo4j.

Liang

[Joel]

how to deal with storage issues, really is up to you, that is a personal choice. In my opinion, Neo4j will grow as it needs to and you'll have to provide the space required.

I should mention that you should always make sure there is some available space because both the database and the operating system can not operate without free disk space available. You will find varying advice on this topic to justify almost any decision, and a percentage recommendation can depend on how much total disk space you have, but personally as a general rule of thumb I don't like to go above 80% utilized on any volume, if at all possible. Bad things start to happen as you get near 100% disk used, and I want to a have a safety buffer. (have a chance to notice a run away disk consumer before the issue becomes critical)

[liang_yeh]

Hi Joel,

Thank you for your great feedback and solutions that help a lot.
One more question regarding the database usage, I know my question was based on the overall disk volume. How about the single database usage if we tried to set a limit for it, but it used near 100%? Is there a way that we can stop the database to grow more spaces with any setup? Or you will still suggest that we sent the notification email to the user when it hit the target of around 70~80%.

Please let me know your thoughts and I am really appreciated your feedback.

Liang

[Joel]

significant growth of the database is often most often due to changed and/or added data, and log files you could manage if they were growing quickly (e.g. change verbosity setting, rotate/cycle, etc)

Agree having an alert is very helpful, be sure to examine how fast disk space is growing, to ensure that an alert occurs, AND the disk is expanded before you run out.

[liang_yeh]

Hi Joel,

Thank you for your feedback.
Let's been said that the Neo4j doesn't have any solution to block the database from growing. Is that correct? I am wondering besides the alert email if there are some functions or tools can block the usage for database growing.

Liang

[Joel]

The challenge is not specific to Neo4j, this is generally true for databases (if online and in use).
When data is added, the database must grow for the data to be added.
When data is modified normally the change is tracked and logged (therefore changing data in place, also increases disk use)
Even if read only, expect a little growth.

For your specific situation, I think you need to understand how the database will be used (read only? new data periodically, how much?), then estimate resource usage over time, monitor growth for a period of time to help confirm estimates, and then prepare a strategy plan. For example, for one project I estimated the growth curve by monitoring actual growth for several months, then installed enough space to accommodate estimated usage over 2 years. I still continued monitoring/alarms in case the usage pattern changed without notice, as it often does...

[liang_yeh]

Hi Joel,

Thank you for the feedback and suggestions.
We will continue to set the monitor workflow to alarm the users as the standard process.
I am really appreciated your help.

Best Regards,
Liang