Explanation of error "Record id 65536 is out of range [0, 65535]"

[dana_canzano]

When running a Cypher statement that creates a new relationship type, for example

MERGE (n1:Person {id:1})-[r:knows]->(n2:Person {id:2})

one may encounter an error which is logged in the $NEO4J_HOME/logs/debug.log as

2017-10-30 17:08:29.741+0000 ERROR [o.n.b.v.r.ErrorReporter] Client triggered an unexpected error [UnknownError]: Could not create token, reference 63c2e7ef-6f5b-4834-b2a8-fe74cac3a50a. Could not create token
org.neo4j.graphdb.TransactionFailureException: Could not create token
        at org.neo4j.kernel.impl.core.DelegatingTokenHolder.getOrCreateId(DelegatingTokenHolder.java:85)
        at org.neo4j.kernel.impl.api.store.StorageLayer.relationshipTypeGetOrCreateForName(StorageLayer.java:376)
        at org.neo4j.kernel.impl.api.StateHandlingStatementOperations.relationshipTypeGetOrCreateForName(StateHandlingStatementOperations.java:1384)
        at org.neo4j.kernel.impl.api.DataIntegrityValidatingStatementOperations.relationshipTypeGetOrCreateForName(DataIntegrityValidatingStatementOperations.java:86)
        at org.neo4j.kernel.impl.api.OperationsFacade.relationshipTypeGetOrCreateForName(OperationsFacade.java:774)
        at org.neo4j.cypher.internal.spi.v3_3.TransactionBoundQueryContext.getOrCreateRelTypeId(TransactionBoundQueryContext.scala:114)
        at org.neo4j.cypher.internal.compatibility.v3_3.ExceptionTranslatingQueryContext$$anonfun$getOrCreateRelTypeId$1.apply$mcI$sp(ExceptionTranslatingQueryContext.scala:203)
#### ..
#### ....
#### .......
        at org.neo4j.bolt.v1.runtime.concurrent.RunnableBoltWorker.run(RunnableBoltWorker.java:96)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
        at org.neo4j.helpers.NamedThreadFactory$2.run(NamedThreadFactory.java:109)
Caused by: org.neo4j.kernel.impl.store.id.validation.IdCapacityExceededException: Record id 65536 is out of range [0, 65535]
        at org.neo4j.kernel.impl.store.id.validation.IdValidator.assertIdWithinCapacity(IdValidator.java:88)
        at org.neo4j.kernel.impl.store.id.validation.IdValidator.assertValidId(IdValidator.java:67)
        at org.neo4j.kernel.impl.store.id.IdGeneratorImpl.nextId(IdGeneratorImpl.java:143)
        at org.neo4j.kernel.impl.core.DefaultRelationshipTypeCreator.createKey(DefaultRelationshipTypeCreator.java:40)
        at org.neo4j.kernel.impl.core.IsolatedTransactionTokenCreator.getOrCreate(IsolatedTransactionTokenCreator.java:59)
        at org.neo4j.kernel.impl.core.DelegatingTokenHolder.createToken(DelegatingTokenHolder.java:103)
        at org.neo4j.kernel.impl.core.DelegatingTokenHolder.getOrCreateId(DelegatingTokenHolder.java:76)
        ... 49 more

and they key part from above is the reference to Record id 65536 is out of range [0, 65535]

This error is caused as a result of hitting the maximum number of relationship types for a graph.db. The current limit is 65536.
Note this is specifically for the 'relationship type', i.e. the identifier used to name the relationship. This limit does not
apply to number of relationships between nodes or the total number of relationships in the graph.

When this error is encountered running

call db.relationshipTypes() yield relationshipType return count(relationshipType) as numRelTypes;

will return

+-------------+
| numRelTypes |
+-------------+
| 65536       |
+-------------+

To resolve this error would require removal of relationship types which are no longer associated with any nodes. As there is currently
no Cypher command to do this, one would need to run copy-store.sh. This command will read a
offline graph.db and prepare a new graph.db but exlcuding any relationship types/properties which are no longer in use.

[petercheng]

@dana.canzano is there any plans to increase this limit in the near future? This seems to be a possible blocker, from us incorporating Neo4j into our products.

Thanks

[dana_canzano]

What version of Neo4J are you running and is this community or enterprise

[andrew_bowman]

Just making sure to emphasize:

Note this is specifically for the 'relationship type', i.e. the identifier used to name the relationship.
This limit does not apply to number of relationships between nodes or the total number of relationships in the graph.

[petercheng]

@dana.canzano Currently we are investigating with Neo4j version 4 Community edition. Ultimately we will need to test with Neo4j Enterprise as well too.

@andrew.bowman yes I'm clear on the relationship types vs total number of relationships. We need it to support over 65,536 relationship types. Thanks!

[dana_canzano]

for most all Neo4j customers they do not encounter this limit and thus do not have the failure.
Is your model such that relationship types are unique and numerous, i.e. rather than

:Peter - [:FOLOWS]->:Dana
:Peter- [:FOLOWS]->:Andrew 

you have similar to

:Peter - [:FOLOWS_DANA]->:Dana
:Peter- [:FOLOWS_ANDREW]->:Andrew 

[petercheng]

No our relationships are not unique, but we have customers have huge amounts of data that have hundreds of thousands of relationship types and growing constantly. For Neo4j to be a viable backend, it would need to support this use case.

[dana_canzano]

hundreds of thousands of relationship types and growing constantly ???

I'm not aware of where this limit has been increased. And even if this was increased, lets say it was increased to 2x (i.e. 130k) wouldnt this just buy more headroom but given 100's of thousands of relationship types result in you sooner or later encountering the limit again. For all but a very few Neo4j installs has this not been encountered and when encountered it is usually resolved by reworking the relationship model

[petercheng]

@dana.canzano @michael.hunger assuming I cannot get away from the number of unique relationship types and this limit. Is there a good pattern to use that would not result in degrading performance by reworking the relationship type say for example: instead of traversing with the relationship type --storing it as a relationship property, or storing it as a node property on a node which the value would then point to another node say an indexed identifier on the other nodes?

[currywangtianxi]

why!!! The open source data set is used for learning, but the data set is too large to load into neo4j, which is not a logical repeated error. It really has too many relationships. In order to process data, only 16g memory can be written, and complex loops can only be written. It takes 2 hours to clean the data. Because it is only simple logic, I don't want to use Hadoop spark or hive for processing, But why do I only support 65536 relationships after working hard for a long time... If I am a programmer, I will open the configuration and let the programmer adjust freely. Just like C language, it can be very efficient and there can be many bugs, but how to choose is up to the programmer to choose. Help the children, programmers from China

[currywangtianxi]

help me please, i need help . why ?!!!! I don't want say : wtf c艹 .buy i 'm die. I want to die