Hi,
Thank you for your reply 
What I would like to remove are the nodes/edges in the red area.
Regards,
I reduce the amount of data so it will probably be more clear.
Below an example of what I would like to do :
In the green circle the Nodes/Edges I would like to keep.
I would like to remove the rest because they are smaller than 6 nodes.
Attached the result of CALL db.schema.visualization()
Schema.txt (1.1 KB)
And The csv files that I used as data source
Edges.txt (1.3 KB) Nodes.txt (869 Bytes)
Thanks in advance !
This query should delete for example, connected components that have 10 nodes or less
You will need the APOC plugin installed on the database.
MATCH (a)-[*]-(b)
WITH id(a) AS id, apoc.coll.sortText(apoc.coll.toSet(collect(DISTINCT b.id) + [a.id])) AS nodes_list
WITH DISTINCT nodes_list, size(nodes_list) AS size
WHERE size <= 10
WITH nodes_list
CALL apoc.periodic.iterate('
MATCH (n)
WHERE n.id IN $nodes_list
RETURN n
', '
DETACH DELETE n
', {batchSize:1000, iterateList:true, params:{nodes_list:nodes_list}}) YIELD batch, operations
RETURN 1
Thank you for your reply.
That looks great. I managed to installed apoc, run the query, 1 is returned, so the query seems to be executed with success. But the nodes and edges are still there :
I feel like this vodoo spell needs to be optimized 😉
It works like charm.
I will try to understand this query.
Thank you very much !
Hi,
Thank you for your feedback. I launch the query yesterday on a large database (5 million nodes, and 10 millions Edges) and the process is still running. So, I am not sure this way will fit my need.
What I do with Gephi : There's the possibility to run stats for a specific set of data. By running the stat based on network components, you obtain a component ID for each network. Then, you can filter out the networks that are smaller than a certain size.The pb with Gephi is that he can't manage big data.
Would it be possible to do more or less the same thing with Neo4j : First, obtaining some stats on the data and then filtering out unintersting data ?
Another approach would be to obtain these stats an rather than deleting small networks, make a query to obtain the list of a specific Nodes for all networks greater than a certain size.
I'm not suer I 'm very clear...
Base on your knoledge what would be the best option to that with a large database ?
Best regards,
Hi Cobra,
I didn't use unique constraints to create the nodes.
I will check the doc to determine how to do that.
Looking forward to get your feedback
Best regards,
@1113 😉
The UNIQUE CONSTRAINT should speed up the query, it's something to have when you work with Neo4j
Can you tell me which property is unique? Like this we can use this one in the query I gave you
Thanks, you too!
Ok, so we will have to create a community for each size and tag each node with his community in order to delete them but I don't know if it will be faster. In your case, the problem is you don't have a unique property, that's why everything takes time I think
Hi @1113
I'm confused because it should not be so long 
Can you give me the configuration of your database?
How many nodes and relations do you have?
Did you use Hardware Sizing Calculator to choose your database?
With this query, you can get the percentage:
MATCH (a) RETURN toFloat(count(a.community_id)) / toFloat(count(a)) * 100
Regards,
Cobra
I increased :
dbms.memory.heap.max_size=4G
and :
dbms.memory.pagecache.size=2G
Makes sense ?
Just relaunched the query with these new parameters
Let's try !
Ok, nice
You should try to add a unique property, for example create an id equal to the Neo4j id and put a unique constraint on it. After we could use it in the query and it should be faster.
I reimported the data structured as :
Entity:ID,UniqEntity,description:LABEL
e53628fb3f714cbc9eb2546cecc7064c,e53628fb3f714cbc9eb2546cecc7064c,Item1
34c075e8781244bdb933c3539cdf167c,34c075e8781244bdb933c3539cdf167c,Item3
I add added a constraint on UniqEntity :
CREATE CONSTRAINT ON(l:UniqEntity) ASSERT l.id IS UNIQUE
(I hpe the syntax is ok (it seems to be))
And here's the screenshot :
Have a nice evening
I feel like my new import is not correct because I have the folowing message :
Invalid input ')': expected whitespace, '.', node labels, '[', '^', '*', '/', '%', '+', '-', "=~", IN, STARTS, ENDS, CONTAINS, IS, '=', '~', "<>", "!=", '<', '>', "<=", ">=", AND, XOR, OR, AS, ',', ORDER, SKIP, LIMIT, WHERE, FROM GRAPH, USE GRAPH, CONSTRUCT, LOAD CSV, START, MATCH, UNWIND, MERGE, CREATE UNIQUE, CREATE, SET, DELETE, REMOVE, FOREACH, WITH, CALL, RETURN, UNION, ';' or end of input (line 2, column 83 (offset: 100))
"WITH id(a) AS id, apoc.coll.sort(apoc.coll.toSet(collect(DISTINCT b.id)) + [a.id])) AS nodes_list"
What is strange is that I don't have any special charatere the csv file. (except ',' as separator)
So, I'm a bit confused
Yeah, good start!
If you want to do everything in one time (maybe you have to change the nodeProjection and the relationship Projection). In my query, it will delete communities which have less than 6 nodes.
CALL gds.wcc.stream({
nodeProjection: "Item",
relationshipProjection: "BELONGS_TO"
})
YIELD nodeId, componentId
WITH componentId, collect(gds.util.asNode(nodeId).id) AS libraries
WITH size(libraries) AS size, libraries
WHERE size < 6
WITH apoc.coll.flatten(collect(libraries)) AS nodes_list
CALL apoc.periodic.iterate('
MATCH (n)
WHERE n.id IN $nodes_list
RETURN n
', '
DETACH DELETE n
', {batchSize:1000, params:{nodes_list:nodes_list}}) YIELD batch, operations
RETURN 1
If you want to do it in two times:
- Save the size of the community in a property:
CALL gds.wcc.stream({
nodeProjection: "Item",
relationshipProjection: "BELONGS_TO"
})
YIELD nodeId, componentId
WITH componentId, collect(gds.util.asNode(nodeId).id) AS libraries
WITH size(libraries) AS size, libraries
WITH size, apoc.coll.flatten(collect(libraries)) AS nodes_list
CALL apoc.periodic.iterate('
MATCH (n)
WHERE n.id IN $nodes_list
RETURN n
', '
SET n.community_id = $community_id
', {batchSize:1000, params:{nodes_list:nodes_list, community_id:size}}) YIELD batch, operations
RETURN 1
- Next, to delete, for example connected components that have less than 6 nodes:
CALL apoc.periodic.iterate('MATCH (n) WHERE n.community_id < $community_id RETURN n', 'DETACH DELETE n', {batchSize:1000, params:{community_id:6}})
Regards,
Cobra
Hi Cobra,
The Query is sucesseful but it seems no nodes are deleted. The query terminates very fast too :
Query used :
CALL gds.wcc.stream({
nodeProjection: "Entity",
relationshipProjection: "DEPENDS"
})
YIELD nodeId, componentId
WITH componentId, collect(gds.util.asNode(nodeId).id) AS libraries
WITH size(libraries) AS size, libraries
WHERE size < 16
WITH apoc.coll.flatten(collect(libraries)) AS nodes_list
CALL apoc.periodic.iterate('
MATCH (n)
WHERE n.id IN $nodes_list
RETURN n
', '
DETACH DELETE n
', {batchSize:1000, params:{nodes_list:nodes_list}}) YIELD batch, operations
RETURN 1
Sure. Here's the result :
Can you show me your properties on the right please? And tell me the one which is unique please
Sure. Here you go :
Here's the headers of my csv file :
Entity:ID,description:LABEL
Entity and ID is the same data . I added a unique constrainte on "Entity" even if I guess it is done automatically because Entity is used as ID.
Best regards
Same result :
I have no idea...
You made it wotk before and now it's not working anymore...
Is GDS still installed?
Anyway, these queries in my message should solve the main issue but maybe you will have to change the property depending of the one which is unique
Regards,
Maxime
I'm sorry but I already put you the two ways on a previous message and both requests are working on my local database and I use the same labels and properties as yours
I don't know what to try anymore, maybe create a completely new database and retry. It was working on your database and now it doesn't...
Yeah, the query is good
The only option I see now is to increase the power (RAM, CPU) of the database
But at least, you have two queries that works on smaller database
Regards,
Cobra
Hi Cobra, I have reduced the number of nodes, increased the ressources on the machine and I got the query achieved in less than 2 hours
That's really cool !
All the best!
Hello @1113
You could just use the same query but with a little modification:
CALL gds.wcc.stream({
nodeProjection: "Item",
relationshipProjection: "BELONGS_TO"
})
YIELD nodeId, componentId
WITH componentId, collect(gds.util.asNode(nodeId).id) AS libraries
WITH componentId, libraries, apoc.create.uuid() AS uuid
CALL apoc.periodic.iterate('
MATCH (n)
WHERE n.id IN $nodes_list
RETURN n
', '
SET n.uuid = $uuid
', {batchSize:1000, params:{nodes_list:libraries, uuid:uuid}}) YIELD batch, operations
RETURN 1
Regards,
Cobra
Brilliant ! It works like a charm ! Many thanks !
NODES 2022, Neo4j Online Education Summit
All the sessions of the conference are now available online
