Issues with edge creation performance, possibly due to write locks?

[josh_southerlan]

Hi Neo4j Community,

We are seeing performance issues with concurrent profile updates from multiple users.. In this scenario we are connecting a user node to other nodes such as skills,interests, disciplines, etc.

I have re-created this scenario in a more minimal way, where I am connecting a user to a random subset (size 10) of possible "skill" nodes (variable number of options) for some number of concurrent requests.

A single update is taking 90ms including session overhead( ~60ms using neo4j driver). Running 50 of those concurrently is taking ~3.5 seconds. That means that they are basically happening sequentially and the last person to get a request in may have to wait 3.5 seconds. In our larger scale (production) system we have more going on in those queries and other validation before hand and have seen much more dramatic slowdowns.

In the test setup, I can increase the number possible skills so that choosing ten random skill nodes to connect for each of the 50 users leads to less chance of users connecting to the same skill nodes. I did this with the hypothesis that this slowdown is write lock-related. If I increase that pool from 12 to 1000 skill nodes, all 50 updates complete in 270ms (vs ~3500ms before) , which means I know that the concurrent requests in my test setup are working and it seems like it might be a matter of write-locks slowing things down. It isn't possible to solve this problem in production in the same way (by creating sparsity in user selections) since we want users to be generating multi-hop connections to each other through common skills/interests/etc.

Here is the query for the benchmark times mentioned above, in our simplified test setup. data.add is a list of uuids that can be used to find the skill nodes we are linking to:

    WITH $props as data
    UNWIND data.add AS dst_uuid
    MATCH (a:Node {uuid: data.userRecord.uuid}) 
    MATCH (b:Node {uuid: dst_uuid}) 
    CREATE (a)-[r:test]->(b)
    RETURN COUNT(r);

I have an index on :Node(uuid)

The data being passed in looks like:

{ 
  userRecord: { 
    properties: { 
      name: 'Jeremy Turner', 
      email: 'neov@woturoz.ua' 
    },
    city: 'b8adfcf0-ecd6-4b05-8daf-e3b65179aa6c',
    concepts: [ 'b8adfcf0-ecd6-4b05-8daf-e3b65179aa6c-person' ],
    uuid: '08d83849-63be-4699-960b-47927128e513',
    connections: [ 
      '692ac9e0-fa2e-4bfa-8382-ecdbb9b7c410',
      '70beaecf-11c8-44f5-a79f-e4aa3d4c3e17' 
    ]
  },
  remove: [ 'b540056c-a18d-4fa6-b87c-9c09f172e45e' ],
  add: [ 
    '0868d1db-d735-4d4f-a079-4f4e3ab5dfdb',
    'ce07ca0a-0452-4b8e-a4b0-ef347009b5c9',
    'd8d173a4-27ea-4f87-8d23-a9cee7d531d6',
    '80c5c73c-951a-4336-a7d6-2bec45108696',
    'f096ef5c-06cf-4223-afd1-9358389de7fa',
    '2eadd487-de71-49f2-a055-bfdf55f39130',
    '02a029ff-87db-446d-9154-21f1650d6a90',
    'b3db320c-2888-4008-91e1-e47d49b9c0e9',
    '71a6e29b-2ee5-4b04-8ba8-6cbcbf3b2dd1',
    '1d7d21cb-18f0-49dc-b179-8433a13c201f' 
  ]
}

^^^ that isn't real data, it was dummy data generated with chance.js

Does it seem like this might be a write-lock issue and that the concurrent query performance issue should then be expected?

If so, is there a standard way of dealing with this issue?

If not, any other suggestions?

Thanks