Neo4j Etl tool does not load relationship for mssql connections

[anuragtillu]

I have set up a MS sql db with all primary foreign key relationship. When I try to import data into net4j it only creates nodes but no relationship.

please advse.

[michael_hunger]

Please share the log files from generating the mapping.

[byron]

Same problem, but with MySQL. The mapping creates nodes, but not the relationships. It's a small DB, 20 tables, no complex relationships, basically, all data relates to the core table "person." Everything else is simple 1:1 relationship, IE: A person 'has a' family etc. and the family node (table) contains the properties relevant to 'family.'

This is a basic, normalized schema with no join/union/relationship tables that carries some null data at the field level - null data that are mySQL (NULL) values.

Log extract (relevant section):

• Crawling routines
• Retrieved 0 routines
• Not retrieving synonyms, since this was not requested
• Not retrieving sequences, since this was not requested
• Foreign key nepaldata.Person.ID <-- NepalData.Cardiovascular.FK_ID skipped as one of the reference is excluded.
• Foreign key nepaldata.Person.ID <-- NepalData.Castes.FK_ID skipped as one of the reference is excluded.
• Foreign key nepaldata.Person.ID <-- NepalData.Children.FK_ID skipped as one of the reference is excluded.
• Foreign key nepaldata.Person.ID <-- NepalData.Chronic_Conditions.FK_ID skipped as one of the reference is excluded.
• Foreign key nepaldata.Person.ID <-- NepalData.Cooking_Stoves.FK_ID skipped as one of the reference is excluded.
• Foreign key nepaldata.Person.ID <-- NepalData.Electricity.FK_ID skipped as one of the reference is excluded.
• Foreign key nepaldata.Person.ID <-- NepalData.ENT.FK_ID skipped as one of the reference is excluded.
• Foreign key nepaldata.Person.ID <-- NepalData.Family.FK_ID skipped as one of the reference is excluded.
• Foreign key nepaldata.Person.ID <-- NepalData.Family_Planning.FK_ID skipped as one of the reference is excluded.
• Foreign key nepaldata.Person.ID <-- NepalData.Gastrointestinal.FK_ID skipped as one of the reference is excluded.
• Foreign key nepaldata.Person.ID <-- NepalData.General_Health.FK_ID skipped as one of the reference is excluded.
• Foreign key nepaldata.Person.ID <-- NepalData.General_Resources.FK_ID skipped as one of the reference is excluded.
• Foreign key nepaldata.Person.ID <-- NepalData.Hand_Washing.FK_ID skipped as one of the reference is excluded.
• Foreign key nepaldata.Person.ID <-- NepalData.Health_Risk_Behaviours.FK_ID skipped as one of the reference is excluded.
• Foreign key nepaldata.Person.ID <-- NepalData.Infectious_Diseases.FK_ID skipped as one of the reference is excluded.
• Foreign key nepaldata.Person.ID <-- NepalData.Other_Conditions.FK_ID skipped as one of the reference is excluded.
• Foreign key nepaldata.Person.ID <-- NepalData.Social_and_Mental_Health.FK_ID skipped as one of the reference is excluded.
• Foreign key nepaldata.Person.ID <-- NepalData.Survey_Metadata.FK_ID skipped as one of the reference is excluded.
• Foreign key nepaldata.Person.ID <-- NepalData.Waste_Management.FK_ID skipped as one of the reference is excluded.
• Foreign key nepaldata.Person.ID <-- NepalData.Water_Resources.FK_ID skipped as one of the reference is excluded.
• Foreign key nepaldata.Person.ID <-- NepalData.Water_Treatment.FK_ID skipped as one of the reference is excluded.
• Foreign key nepaldata.Person.ID <-- NepalData.Womens_Health.FK_ID skipped as one of the reference is excluded.

[andrea_santurba]

@byron is the dataset that you're using public?

[byron]

The data are from my sociology research from my MA. I'm planning on anonymizing the data (removing all personally identifying information) and making it available through suitable academic channels.

It's perfectly accessible in MySQL and up until now I've been using Excel and tools like Power BI etc. for analysis and accessing the data but I felt that a Graph DB might open up the data for further use in the future by myself as well as others working in similar research fields.

Best,

Byron

[andrea_santurba]

can you share the mapping json files?

[byron]

Hi, would you like the entire log? The mapping fails seemingly because of null data in the tables - there is one 'blank' row (a non-participant) which is data that can't be removed from the data for reasons of accountability and integrity. There are also null data at the field level throughout the data, which is to be expected in survey data as some blocks of answers are for particular demographics.

This is the log from the ETL mapping tool:
mapping.txt (242.4 KB)

Best,

Byron

[andrea_santurba]

byron:

skipped as one of the reference is excluded.

Did you exclude columns from the mapping section?

[byron]

Hi Andrea,

No actually, and for auditing & accountability purposes I can't skip rows as the data are survey data where participants have opted in or out. There is only one blank record in the person table in the database which then echoes across the rest of the tables based on the relationships. Futher to that, there are fields in tables that carry null data since they may be gender, age or specific to a demographic feature.

Byron

[andrea_santurba]

can you share the DDL of your database?

[byron]

Hi Andrea,

Sure. The original data were in CSV, which is the export format from the digital survey tool I use. As a result, the datatypes are simply left as strings for simplicity; correcting datatypes didn't seem necessary or a good use of time & resources.

Byron

Attached: DDL (No data)

I can't release data at this point as it has not been scrubbed of PII. I can't violate the participant agreements.

This is the schema only.
nepaldata.txt (46.5 KB)