Super frustrated SDN deleting existing relationships

[joram_ngwenya]

I have seen threads on which SDN 6.0 staff try to explain the deletion of existing relationships when saving new connections to existing nodes and I fail to understand what good is supposed to come from SDN boldly doing what it is not asked to do. What is the message to the developer? Redesign the database? Is CRUD still possible with SDN 6? Could you point me to a CRUD example on a graph of a depth of at least three? The movie database with just two entities: Person and Movie is not too helpful. But even on the movie graph if you can show a query involving the Roles relationship and adding and saving new nodes and relationships that would be wonderful.

Also, if I am having problems adding new data to an existing database or retrieving data from the database do you discourage going back to SDN 5?

"2. If the entity is not new all relationships of the first found type at the domain model will get removed from the database.

( MATCH (startNode)-[rel:Has]→(:Hobby) WHERE id(startNode) = $fromId DELETE rel )"

I think step 2 of your save process as cited above is the problem I am running into. Why do you have to delete existing relationships in order to save?

[florent_biville]

Can you share what you are specifically trying to do and see how we can resolve it?

[joram_ngwenya]

I can but that is an approach I would rather avoid because workarounds for what to me appears to be a bug are not going to be helpful in the long term. For instance, I use Spring Security and have users with roles. I am just realizing now that when I add users to departments, the existing relationship between a user and a role is deleted. And if I add a vehicle to a department, the link from department to division the department is in is deleted. This all appears to be from that step 2 in the saving process. So can you first explain how to preserve existing relationships during save.

MATCH (startNode)-[rel:Has]→(:Hobby) WHERE id(startNode) = $fromId DELETE rel )

I am trying to wrap my head around why the relationship between startNode and Hobby has to be deleted. If I create a team of bikers (people with a relationship to Biker node, the team node and the links between team and each biker will be saved but the links to the Biker node will be deleted. Is that not a bug? or how could I preserve the link between a person and a hobby to avoid creating a team of bikers who end up with no relationship to the Biker node?

[florent_biville]

Can you share a sample project where this happens?

This is going to be much easier for the Spring Data Neo4j team to decide or not whether what you describe is a bug or not, and what needs to change in your model if needs be.
If this is a bug or a deficiency of Spring Data Neo4j, rest assured that the team will work on it.

[joram_ngwenya]

I will, I am watching your video where you demonstrate how to use SDN 6, maybe by the end I will have my questions answered. But I will send you the demo.

OK I am done with the video but it ended while you were still setting up testing. So here is what is going on.

Step 1. I start with users and associated roles for Spring security.

Step 2:
Create BusinessAccount node and add user "transporter"

As you can see BusinessAccount node is successfully saved and points to user transporter and and the home city Cape Town. However user transporter no longer has the role, the ROLE link was deleted.

Step 3:
This is an example with a different business where a vehicle is added, doing so deleted both the AUTHORIZED_USER and HOME_CITY links.

I am now checking if abandoning the SDN save method and using a @Query MERGE method will work.

[joram_ngwenya]

Please stop considering this. I have changed the program to only save new nodes without any relationship properties and then add the relationship properties through query methods. I am therefore not running into step 2 of the saving process at all.

[florent_biville]

Any chance you can share a GitHub project, with a simple Spring Data Neo4j/Spring Security setup and small data set?
Screenshot are helpful, but the most efficient way to help us help you is with a reproducible experiment. Do you think you could do that?

[clavaud]

Dear,
I'm facing same issues and feeling frustrated by migrating to SDN 6.0
How did you achieve that ? do you have an example ?
Thanks

[gerrit_meier]

There were some improvement regarding sliced loaded data lately in Spring Data Neo4j 6.1.x. You can now define a projection class for persistence. This means you describe a subset of visible properties and pass it with the entity to the Neo4jTemplate save method.
There is a bit of documentation missing for this yet but this test should give you an impression how it is used. spring-data-neo4j/Neo4jTemplateIT.java at 4f5f5fd4e6a2c629365628a12fdc7319dc52e452 · spring-projects...

[clavaud]

In fact, the ClosedProjection should contains all attributes without relationships ?
Then using Spring Data repository default method like 'save' will not work until you put back all relationship in you entity ?

[gerrit_meier]

If I understand you correctly, yes. Just to be clear:
Either don't use projections and you need to have everything loaded that you want to persist (and not get dropped later).
Or use projections and just persist the slice of data you have already loaded in your application.

[clavaud]

In fact, as I'm migrating my spring project to SDN 6.x and after your explanations, the best way will be to :

• remove all relationships from all java nodes
• if a relation has to be added, changed or removed, then use a custom method
• do not use ClosedProjection (not needed)

-> Why ? because i have big trees with a lot of relations between nodes and, for performance reason, I don't want to load all trees each time i need to load a node, add or update a relation....

The problem now, is to get relationship (still from spring) :
As :

neo4jTemplate.findAll("MATCH(u:User)-[r:HAS_USER_ACTIVITIES]->(ro:Activity) WHERE id(u) = $userId RETURN r", Collections.singletonMap("userId", userId), UserActivityRelationship.class);`

Do not work (because it return a relatioship), I have to do :

(List<UserActivityRelationship>) neo4jClient.query("MATCH(u:User)-[r:HAS_USER_ACTIVITIES]->(ro:Activity) WHERE id(u) = $userId RETURN r").bindAll( Collections.singletonMap("userId", userId)).fetchAs(UserActivityRelationship.class).
        mappedBy((typeSystem, record) -> mapUserActivityRelationship( record )).all();

with :

protected UserActivityRelationship mapUserActivityRelationship( Record record ) {
    UserActivityRelationship userActivityRelationship = new UserActivityRelationship();
    Relationship relationship = record.get(0).asRelationship();
    userActivityRelationship.setId(relationship.id());
    userActivityRelationship.setActivity(activityRepository.getById(relationship.endNodeId()));
    if ( !relationship.get("userActivityType").isNull())
      userActivityRelationship.setUserActivityType(UserActivityTypeEnum.valueOf(relationship.get("userActivityType").asString()));
    if ( ! relationship.get("center").isNull())
      userActivityRelationship.setCenter( new Point(relationship.get("center").asPoint().x(), relationship.get("center").asPoint().y()));
    if ( ! relationship.get("level").isNull())
      userActivityRelationship.setLevel(UserLevelEnum.valueOf(relationship.get("level").asString()));
    if ( ! relationship.get("rangeType").isNull())
      userActivityRelationship.setRangeType(RangeTypeEnum.valueOf(relationship.get("rangeType").asString()));
    if ( ! relationship.get("range").isNull())
      userActivityRelationship.setRange(relationship.get("range").asDouble());
    return userActivityRelationship;
  }

And to create relationship.. :

 @Query(
      "MATCH (u:User) OPTIONAL MATCH (a:Activity) WHERE id(u) = $userId AND id(a) = $activityId "
          + "MERGE (u)-[r:HAS_USER_ACTIVITIES]->(a) "
          + "SET r = {userActivityType: $userActivityType, center: $center, level: $level, rangeType: $rangeType, range: $range} RETURN id(r)")
  Long addUserActivity(@Param("userId") Long userId,
      @Param("activityId") Long activityId,
      @Param("userActivityType") UserActivityTypeEnum userActivityType,
      @Param("center") Point center, @Param("level") UserLevelEnum level,
      @Param("rangeType") RangeTypeEnum rangeType, @Param("range") Double range);

Is it the right way ? do you have any improvement ?

Migrating to SDN 6 is a huge work... it generate much more code than before.

Thanks for help

[fneumann]

I struggeled with the same problems and found some workarounds for now that might be useful for others.
To get a relationship as a POJO (Rel.class) I use

Collection<Rel> votings = neo4jClient.query("MATCH (a:NodeA) - [r:REL] -> (b:NodeB) WHERE ... RETURN r")
            .fetchAs(Rel.class)
            .mappedBy(RelationshipMapper.forClass(Rel.class))
            .all();

public class RelationshipMapper {
   
   private final static ObjectMapper mapper = new ObjectMapper();
   
   
   public static <T> BiFunction<TypeSystem, Record, T> forClass(Class<T> clazz) {
      return (TypeSystem typeSystem, Record record) -> {
         
         Value value = record.get(0);
         Relationship relationship = value.asRelationship();
         Map<String, Object> originalMap = relationship.asMap(); // immutable
         Map<String, Object> resultMap = new HashMap<>();
         resultMap.putAll(originalMap);
         resultMap.put("id", Long.valueOf(relationship.id()));
         
         return mapper.convertValue(resultMap, clazz);
         
      };
   }
   
}

and to create a relationship I use a custom query on the repository (apperently that works on a repository, but returning the relationship does not)

public interface NodeARepository extends Neo4jRepository<NodeAObj, Long>{

   @Query("MATCH (a:NodeA), (b:NodeB) WHERE ID(a) = $0.__id__ AND ID(b) = $1.__id__ CREATE (a)-[r:REL]->(b) SET r = $2.__properties__")
   void addRel(NodeAObj obj1, NodeBObj obj2, Rel relObj);
}

[joram_ngwenya]

I only use "save" once with no relationship properties, to generate the node id then I use @Query methods to add the relationship properties. To retrieve I use @Query methods to retrieve the main node with its relationship properties as null or empty sets. I then add @query methods or findById queries to add the missing relationship properties.

[gerrit_meier]

A general remark on this topic:
It seems that the transport user where not fetched via e.g. a UserRepository or similar. This is the reason the roles are not existing once you persist the ABC Moving.
SDN only creates the relationships it knows about at the moment of saving. But on the other hand also removes the non-existing ones. This it what happens here (a little bit simplified):

I. Save ABC Moving
II. Look at the relationships for modifications
II a. Save Cape Town -> no further relationships here
II b. Save User -> process ROLE relationship
III c. Save all existing roles: there are non, so the relationships will be dropped.

As I said, the above is a little bit simplified and the deletions, if necessary happen upfront the creation.

[joram_ngwenya]

As I mentioned earlier you can drop this one, I found a way of saving without deleting existing relationships. But I think we have a terminology problem. "SDN only creates the relationships it knows about at the moment of saving. But on the other hand also removes the non-existing ones" I don't understand that sentence. How do you delete non-existing relationships? But I see you are consistent with that terminology like when you say;
II b. Save User -> process ROLE relationship
III c. Save all existing roles: there are non, so the relationships will be dropped.
clearly, my understanding of "existing" is clearly different of what is described here. User has a role "transporter", To register a business they must have a role, so when you say they are non, your definition of existing is not what I think you mean. Otherwise when you say "there are non, so the relationships will be dropped" I start wondering which relationships are dropped if there are non. By "existing roles " what exactly are you referring to? Where must the roles exist? The user was supplied by Spring Security and comes with roles defined.

[joram_ngwenya]

User is the logged-in user fetched by Spring Security through a UserRepository that is why the user is able to register the company ABC. But immediately after registering a company all the user's role-based access is disabled because of course the "ROLE" relationship has been removed. But does a successful delete not already say something is there to delete. Why do you successfully locate a relationship and delete it? Wouldn't it be better to throw an exception and leave the data model intact than delete what you find? Once you delete, the model is broken continuing is of no consequence.

[rlegorreta]

Same problem. I have a running app (Security micro.service with user-profiles-roles). When I migrated from SDN 5 to SDN 6 I had to change all "save" methods to @Query . A lot of work and spent a lot of time.

Any work around willl be very helpful. I had another micro.service in production but for now we leave it in SDN 5

[lorenzo_valente]

Hi, I also wonder how can you load an entity without its relationships (for performance purposes for .e.g. load the Spring Security principal on every request but without loading all of its relationships), modify its base properties and save it without having Spring Data necessarily delete all of its relationships. Can someone please share an example of a custom @Query-annotated save() method?

[devuser1212]

Same problem for me too. I have outlined my question in a stack overflow question in detail here. This behaviour is very odd and from what I can tell, has meant I cannot use neo4j over another database. Can someone please assist me? 

https://stackoverflow.com/questions/73947463/neo4j-ogm-unexpectedly-deleting-relationship