Documenting a data model

[ocole]

We have a complex application that uses Neo4J, and we would like to improve the documentation regarding our datamodel. While it is possible to extract the properties in use on each Label and relationship type, does anybody have any suggestions how best to record/document their use (e.g. potentientially within the Neo4J database itself).

Is there any tool that would help us perform this task, and ensure we are up to date with our documentation (for example flagging up Labels/properties that are now in use but which have not been documented.

A simple example the resutl of what we're looking for is:
Label MyThing
Used to store a record of MyThings.
Properties
myThingId: Number : A unique identifer for this thing
name: String: A human readable name for the thing

Relationships and Labels
Is_related_to : [ MyOtherThing, MyOldThing]

etc
Any hints, tips, tools, suggestions on how to generate/persist the metadata for the document etc gratefully received.

[abk]

Hi @ocole ,

Well, there's good news ! And.. less good news.

Good news: many people use https://arrows.app for drawing graph diagrams. The results look really nice and can be exported as svg, png, cypher or json.

The export is the less good news. At the moment, the Neo4j DBMS itself does not offer a good way to define or store a schema or model. There is ongoing work headed in that direction, so stay tuned.

Anybody else have suggestions for other tools they use with Neo4j?

Cheers,
ABK

ps. Oh, and welcome to the Neo4j Community!

[dominicvivek06]

Hi @ocole and @abk ,

1. I use https://hackolade.com/. They usually forward engineer as JSON schemas. (it is priced though).
2. We build match all for nodes as API docs (API Swagger).
3. Self -servicing Data Dictionary using React and Nodejs.

[ocole]

Thank you both for your prompt replies.

The arrows tool looks interesting, we spend some time generating proposed models, and this could help make that more efficient. I've pushed that to our architects. The hackolade tool looks interesting too, I'll push this towards whoever does our docs.

It would be nice if Neo4J itself allowed you to comment/mark up properties too natively. Something for the future roadmap perhaps.

[dominicvivek06]

Hi @ocole , the reason i will not take the HTML docs is that, whenever we perform forward engineering , we extract as a JSON schema and upload to github for versioning. This versioning helps the enterprise and the architects throughout their project lifecycle. Also the tool supports to add descriptions, and they are very helpful for the developers.

If you are recommending for an enterprise, please, choose a tool that has forward and reverse engineering features. But end of the day, it is pricey.

[clem]

Kudos for thinking about this!

I diid a project like this years ago for a Ruby on Rails (ROB) and web app and its "Relational" SQL DB. This was because being a newbie to the company, I didn't understand the DB and there was no documentation of our DB. Some obsolete tables were still left in the DB (with zero rows), because we were undergoing rapid development and schema changes.

I walked the meta data by using SQL commands, so that I got a list of Tables and Column names. Because of the naming convention used with ROR, I could connect the primary key to a referring Table. I gather all the columns and calculated things like number of nulls, types, averages, SD, count of zeroes, number of row, etc.

With this data, I created a CSV file that matched the names of the tables with a Table names and their column names with a text description.

Then I wrote a program that combined this CSV and the meta data scraped from the SQL DB and dumped the result into a HTML table with hyperlinks connecting the tables.

I don't think it would be such a hard thing to create a similar thing in Neo4J. I suppose it would be trickier since a node an have multiple (or even zero) Labels, and nodes can have a many to many typed relationships with other nodes. And relationships can have properties.

I would rerun the schema extractor daily and the HTML table would get loaded in our local web server. The schema extractor would alert me whenever it couldn't find a documentation in my CSV file for a Table and/or column, which is how I kept track of the need to update the CSV file of documentation.

My boss was grateful I did this, as my schema tool/table uncovered some discrepancies in the data for him.

There are various APOC functions to start a similar tool but for Neo4J:

Neo4j Graph Database Platform

Viewing schema data with APOC Procedures - Knowledge Base

APOC Procedures offers meta procedures to view information about your database schema and the data it stores. The procedure apoc.meta.schema() uses a sampling of the graph data to produce a…

and lots of APOC schema related functions

Neo4j Graph Database Platform

Schema Information - APOC Documentation

This section describes procedures that can be used to query and create schema indexes.

https://neo4j-contrib.github.io/neo4j-apoc-procedures/3.5/schema/meta-graph/

The potentially tricky part is marrying the text documentation to the elements. Of course, keeping the documentation up to date is a chore!

I might give this a go sometime and recreate what I had done earlier. It was a fun project.

[ocole]

Hi Clem,
Go for it... looks like there's a market for your proposed tool!
I wanted to avoid us having to create something. But we don't have much budget for external tools either.... tricky.
Anyway, thank you all for your discussion. I appreciate your time.