Cyclic references can't be saved (with auditing)

[chris3]

Using SDN 6 I have classes like
abstract base class with auditing:

public abstract class Node {
  @Id @GeneratedValue @Getter private Long id;
  @Version @Getter private Long version;
  @CreatedDate @Getter private LocalDateTime created;
  @CreatedBy @Getter private Long createdBy;
  @LastModifiedDate @Getter  private LocalDateTime modified;
  @LastModifiedBy @Getter private Long modifiedBy;
  ...
}

all other @Node classes (Order, User, Coupon) are derived from that like

@Node
@Getter
@Setter
public class Order extends Node {
  ...
  @Relationship("BY") private User user;
  @Relationship("COUPON") private Coupon coupon;
  ...
}

and this sort of existing dataset:

(o:Order)-[:BY]->(:User)<-[:AFFILIATE]-(c:Coupon)

i.e. I already have an Order by a User which also is the affiliate associated with a Coupon.

Now when that same user redeems such a coupon like

o.setCoupon(c);
orderRepo.save(o);

which should create an additional relationship like

(o:Order)-[:BY]->(:User)<-[:AFFILIATE]-(c:Coupon)
      \--------[:COUPON]---------------->/

all I ever get is an exception with the mesage

An entity with the required version does not exist.

It seems like saving the Order traverses to the User attached to the order (via BY) and increments the version nr AND also to the newly attached coupon which is in turn attached to the same User (via AFFILIATE) and that causes version conflicts .... or something like that. Or maybe the problem actually lies in the version of the Order since through the coupon there is now a connection back onto itself?

Someone please fix this or tell me what I am doing wrong because I guess for now I will have to remove auditing completely 😞

[chris3]

Also: SND6 is NOT USABLE at all grrrmmml

Loading a single N3PMVOrder from the following (very simple!) graph

constructs this crazy Cypher query:

MATCH (n:`N3PMVOrder`) WHERE id(n) = $__id__ WITH n, id(n) AS __internalNeo4jId__
RETURN n{.status, .uuid,  .........., 
  __nodeLabels__: labels(n), 
  __internalNeo4jId__: id(n), 
  __paths__: [p = (n)-[:`ORDER`|`COUPON`|`BY`]-()-[:`REFERRER`*0..1]->()-[:`REFERRER`*0..]-()-[:`ORDER`*0..1]->()-[:`ORDER`|`COUPON`|`AFFILIATE`|`BY`|`REFERRER`*0..]-()-[:`ORDER`*0..1]->()-[:`ORDER`|`COUPON`|`AFFILIATE`|`BY`|`REFERRER`*0..]-()-[:`AFFILIATE`*0..1]->()-[:`REFERRER`*0..]-() | p]}

The __paths__ part is obviously insane.

I only have 11 orders in my test graph, each with ~10 files and ~4 texts attached, a single User and one Coupon and loading a single Order takes 7 seconds !!!

Executing the above query in neo4j Browser doesn't finish.
Exeuting it without the __paths__ takes 1ms.
So I guess it is safe to assume that the queries SDN6 generates are NOT USABLE.

In addition to the auditing / version issues (also related to @Relationships) described in my post above I have to reconsider using this entirely

[gerrit_meier]

We are very aware of the path problem in certain domain-model scenarios and try to mitigate the problematic query creation by substituting it with cascading (data-driven) matches.
Please also understand that the query might seems "insane" but this is exactly what we can derive from the metadata provided by the annotation in the domain model. Loading the N3PMVOrder will follow all reachable relationships by your definition. When SDN detects that there will be a circle because of the relationship definition in the domain-model, it can (in the current release version) only fallback into the path mode becaue it does not know where/when to break the infinite loop.
As I said, we are working hard on fixing this problem and our feeling it that it seems to work AND improve the situation a lot. As a reference model we also created a scenario where the DB will run OOM and now it just takes less than one second to load all data.

[chris3]

Hi Gerrit,
thank you for your reply but...

1st of all, the current version should be published as Alpha so that people (like me) don't try and build projects on it - because things really don't work well.

2nd, in any real-world graph almost every node is somehow connected to almost every other node somehow. Using the "old" OGM for many years now I have come to learn the ups and downs of that "old" solution and I realize how difficult it is to map an infinitely large graph to objects of the OO world. However, when I need to load a single node into a mapped object for some data manipulation, I NEVER want to load the entire graph that is attached to that at infinite depth. Usually the node itself is sufficient, sometimes maybe the directly connected nodes and very rarely maybe 1 hop further. "it just takes less than one second to load all data" makes me wonder if your goal really is to load ALL data one can traverse to from a node?

I keep thinking that whilst inside a session/transaction related objects should lazily be loaded if needed and only that tiny little sub-graph that is now in memory needs to be saved when one of the objects is saved. Or maybe as an additional possibility, let the developer define which relations and related nodes should be loaded (similar to GraphQL). After all in 99% of all cases I know which data I will need for a certain business-case.

Anyway, either I basically remove almost all @Relations now from my code (and do it all manually) because everything crashes with just a few nodes as shown above, or ... ???

[knuxith]

I have the same issues, I am currently rewriting my Spring Boot project with Spring Data 6.0.3 and using database version 4.2.3 for future-proofing. Previously the project has been developed for Neo4j 3.5 and was working fine but I can't make my project work anymore.

I have two concerns:

1. an infinite loop bug
2. scalability

1. I experience the same never-ending query / infinite loop / OutOfMemoryError as chris3 with basic repository methods like for example findByUuid that seems to be caused by relationship loop (no infinite loop if I remove some @Relationship but these relationships are needed elsewhere)

2. It is very common that I need to get just one object with properties without connected nodes or with just a few connected nodes at a depth of 1. For this kind of query that retrieves very little data I expect a few milliseconds response time for good user experience (almost 1 second feels sluggish). It seems very suboptimal to get almost all graph data just to retrieve one node, and it seems it will not scale in production.

• Is there a way to make default Repository methods work ?
• Is there a way to decide the depth of retrieval (I can't find @Depth equivalent)
• As a workaround is there a way to write a custom @Query repository method to create a java object with attributes including connected sub-objects (depth of 1) ?
• Are SDN 6.0.3 and Neo4j 4.2.3 production ready ? If not which versions to choose ?

[gerrit_meier]

I experience the same never-ending query / infinite loop / OutOfMemoryError as chris3 with basic repository methods like for example findByUuid that seems to be caused by relationship loop (no infinite loop if I remove some @Relationship but these relationships are needed elsewhere)

This is fixed with the new 6.0.4 service release.
But this does not mean that it won't fetch the whole graph if your model tells SDN to load the whole graph.

It is very common that I need to get just one object with properties without connected nodes or with just a few connected nodes at a depth of 1. For this kind of query that retrieves very little data I expect a few milliseconds response time for good user experience (almost 1 second feels sluggish). It seems very sub-optimal to get almost all graph data just to retrieve one node, and it seems it will not scale in production.

To have a more broader model than you need in most cases but also getting the advantages of just querying slices of your domain, we have support for interface and DTO projections. Spring Data Neo4j
This makes it possible to let SDN only fetch the data you are really interested in

Is there a way to make default Repository methods work ?

???

Is there a way to decide the depth of retrieval (I can't find @Depth equivalent)

No and we did this on purpose. From a domain perspective this is nothing we want to encourage the users to do because you would end up with a not complete hydrated entity (and relationships).

As a workaround is there a way to write a custom @Query repository method to create a java object with attributes including connected sub-objects (depth of 1) ?

Of course you can do this on your own. We have a small paragraph and hints in the documentation Spring Data Neo4j

Are SDN 6.0.3 and Neo4j 4.2.3 production ready ? If not which versions to choose ?

Short answer: Definitely.
Longer answer: We had last year a very long run with alphas, betas and RCs and got a lot of input where and how to improve. We took all this and created a quite solid framework. As a consequence we have taken the decision to bring out SDN 6 last autumn.
To be clear: What we haven't expected is that our assumption (also driven by the near to 0 feedback) regarding self-references will not satisfy the user's needs. We created Spring Data Neo4j 6 with the clear target that we align more with the domain driven design aspect of the other Spring Data modules.

One example that might be helpful regarding custom queries:

github.com

spring-projects/spring-data-neo4j/blob/a741d3b184cef4f053077a7c8113307b509d8309/src/test/java/org/sp...

/*
 * Copyright 2011-2021 the original author or authors.
 *
 * Licensed under the Apache License, Version 2.0 (the "License");
 * you may not use this file except in compliance with the License.
 * You may obtain a copy of the License at
 *
 *      https://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */
package org.springframework.data.neo4j.documentation.repositories.custom_queries;

import static org.assertj.core.api.Assertions.assertThat;

import java.io.BufferedReader;

This file has been truncated. show original

and spring-data-neo4j/MovieRepository.java at a741d3b184cef4f053077a7c8113307b509d8309 · spring-projects...

Update:
The documentation with more information about custom queries is now online:
https://docs.spring.io/spring-data/neo4j/docs/6.0.5/reference/html/#faq.custom-queries-and-custom-ma...