Jackson infinite recursion error after clone operation

[Jiropole]

Neo4j Enterprise 3.4.5, using a custom procedure to clone a subgraph.

MATCH (ir_i:Insight { uuid: "2bba4e25-2bd1-11e9-8948-2ab0e8a6a76e" })
CALL uvs.cloneTree(ir_i, ["Point"], []) YIELD out
WITH out AS ir_i
OPTIONAL MATCH (ir_i)-[ir_T:TITLED]->(ir_mdT:MessageDefinition)
OPTIONAL MATCH (ir_i)-[ir_S:STATES]->(ir_mdS:MessageDefinition)
OPTIONAL MATCH (ir_i)-[ir_AT:AT]->(pt_p:Point)
OPTIONAL MATCH (pt_p)-[pt_A:ALONG]->(pt_a:Axis)
RETURN *

This works just fine in the Neo4j browser, and I see the properly cloned tree. But in my SDN-driven API, this results in:

Error is org.springframework.core.codec.EncodingException: JSON encoding error: Infinite recursion (StackOverflowError); nested exception is com.fasterxml.jackson.databind.JsonMappingException:
Infinite recursion (StackOverflowError) (through reference chain:
social.unveil.api.core.domain.Point["along"]->java.util.ArrayList[0]->
social.unveil.api.core.domain.PointAlong["point"]->social.unveil.api.core.domain.Point["along"]->
java.util.ArrayList[0]->social.unveil.api.core.domain.PointAlong["point"]->
social.unveil.api.core.domain.Point["along"]->java.util.ArrayList[0]->
social.unveil.api.core.domain.PointAlong["point"]->social.unveil.api.core.domain.Point["along"]
...

Now, the out value is just a node; the head of the cloned tree. If I request the cloned tree in a different GET request, it serializes fine. Nor does it help to simply re-MATCH the cloned root node before returning it. Other than the clone operation, this Cypher is identical to that used in any other place this entity is fetched.

This is not the only case where I use a node that's been modified by a procedure, but it's the only case where it's returning a graph that was created in the current transaction. Could that be part of the issue?

Sorry, this isn't a lot to go on, but any thoughts are appreciated.

[Jiropole]

After debugging through OGM a bit, it appears the trouble starts in GraphEntityMapper:136, the map() function. mappingContext.getNodeEntity(id) throws an exception

java.lang.StackOverflowError` ... Cannot evaluate ...<entity>.toString()

When I look up the entity with id via the Browser, it's always a node, and looks completely normal. It looks likes it's the associated @RelationshipEntity that's the problem here, where it is incorrectly recursing the @StartNode until it overflows (?).

I've also realized that if I simply return the output from the procedure without doing additional OPTIONAL matches on its subgraph, there are no issues. So there is something about how the OGM is handling a subgraph that was just created via a stored procedure (?). I've made sure that @Id @GeneratedValue field on all OGM mapped entities are set at the same time as the entity is cloned. As a side-note, the OPTIONALly queried subgraph is a superset of the cloned graph, i.e. the cloned subgraph ties in to non-cloned nodes at the top and bottom of the "tree", and I'm trying to return the cloned parts along with those tie-in nodes. I'm at a loss as to what is tripping up the mapper.

If nobody has ideas, my next try is to return the required subgraph from the procedure itself (instead of just the root node), with hopes the mapper likes this better (?).

[michael_simons]

Would you mind sharing your @NodeEntityOGM model?

[Jiropole]

Thanks @gerrit.meier, I think I've seen that info before, but never needed to use it. Maybe dumb luck or serialization order (?) but I have lots of RelationshipEntities and not needed JsonIgnore*. I guess I'm not sure what's different this time, as it only occurs if I just cloned most of subgraph I'm returning. If I instead returned the one I matched (remove lines 2+3 in my query example), or if I match/return the cloned graph on a subsequent query, no recursion.

However, the ignore annotation hint was useful, as it revealed it will recurse even without the RelationshipEntity, as the Insight and Point entities do refer to each other; this resulted in a new cycle:

social.unveil.api.core.domain.Insight[\"at\"]->java.util.ArrayList[0]->
social.unveil.api.core.domain.Point[\"at\"]->java.util.ArrayList[0]->
social.unveil.api.core.domain.Insight[\"at\"]-> { ... }

@michael.simons, Here the likely suspects (Kotlin), starting at the top of the tree I'm trying to return:

@NodeEntity
data class Insight(@Id @GeneratedValue(strategy = UuidStrategy::class) var uuid: String? = null,
                   @Property(name = "name") var name: String? = null,
                   @Relationship(type = "TITLED", direction = Relationship.OUTGOING)
                   var title: MessageDefinition? = null,
                   @Relationship(type = "STATES", direction = Relationship.OUTGOING)
                   var statement: MessageDefinition? = null,
                   @Relationship(type = "AT", direction = Relationship.OUTGOING)
                   var at: List<Point>? = null) { ... }

@NodeEntity
data class Point(@Id @GeneratedValue(strategy = UuidStrategy::class) var uuid: String? = null,
                 @Property(name = "name") var name: String? = null,
                 @Relationship(type = "ALONG", direction = Relationship.OUTGOING)
                 var along: List<PointAlong>? = null,
                 @Relationship(type = "AT", direction = Relationship.INCOMING)
                 var at: List<Insight>? = null) { ... }

@RelationshipEntity(type = "ALONG")
data class PointAlong(@Id @GeneratedValue var id: Long? = null,
                      @Property(name = "position") var position: Float? = null,
                      @JsonIgnore @StartNode var point: Point? = null,
                      @EndNode var axis: Axis? = null)

[michael_simons]

Two possiblities here:
(Simple and my first guess): The fact that you have mapped the reference from Insight to Point in both directions will lead to the endless recursion inside Jackson.

The other option is a similar circle from Point to PointAlong to Point.

In either way, Jackson cannot deal with those self referential structures niely.

If you really need this, maybe this can help you: http://keenformatics.blogspot.com/2013/08/how-to-solve-json-infinite-recursion.html
There are two annotations explicitly dealing with this (@JsonManagedReference and JsonBackReference)

[Jiropole]

While I can mitigate the issues with your kind suggestions (thank you!), there is still something unusual happening here. Instead of serializing according to my entity maps, it appears to be performing a "raw" serialization in the clone case, where it includes node properties not referenced in my NodeEntities, and graph-native structures versus those imposed by @Relationship or @RelationshipEntity.

It seems, in this case, the OGM is failing to map the entities and relationships returned from the query to the *Entity classes, as it normally does. Although, obviously it is, since adding @JsonIgnore solves the recursion. Still digging...

[Jiropole · Accepted solution]

Argh, the entire issue was due to an oversight, where I was failing to convert to a Mono/Flux output on this particular call. Note to self: when in doubt, re-read every related line of code!

Why I magically do not see recursion without using @JsonIgnore et al, I cannot say, but I'll continue to take that win.

Sorry to waste everyone's time!

[michael_simons]

No worries, glad you found the issue.

Take note please that our repositories are not yet reactive ready, please.

[Jiropole]

Yes, thank you. I found this out from an actual java dev, after we'd already done everything this way. So far, no issues, and this is not a mission critical app. If the stack is not Reactive-ready by the time this goes public, we'll refactor.

[gerrit_meier]

We have a short section about this Jackson / serialization pitfall in our documentation. https://neo4j.com/docs/ogm-manual/current/reference/#_a_note_on_json_serialization
You have to define for yourself where to break this infinite loop because it is related more to your business logic than the technical level.

[Jiropole]

BTW, this is my simplest clone case; in which only one node is cloned, and new in & out relationship to connect it.

For reference, the graph in question before the clone:

And after (with cloned root node selected, the one I'm trying to return):