Hi,
We are using Neo4j driver for python(version 4.2.1).
We have a live system that writes/read from the neo4j cluster all the time, And once in a while we are getting this exception:
Traceback (most recent call last):
File "main.py", line 85, in on_new_snapshot
response = customers_list[customer_id]['customer_model_handler_obj']\
File "/ms_topology_center/../ms_topology_center/customer_model_handler.py", line 130, in on_new_snapshot
return self.load_last_snapshot_to_graphdb(timestamp=timestamp,
File "/ms_topology_center/../ms_topology_center/customer_model_handler.py", line 119, in load_last_snapshot_to_graphdb
self.modeler.create_services('agnostic', region_snapshot_data['created'], account_id, region, timestamp)
File "/ms_topology_center/../ms_topology_center/modeler.py", line 115, in create_services
self.graph_db.commit_transaction()
File "/ms_topology_center/../shared/graph_db.py", line 135, in commit_transaction
self._transaction.commit()
File "/usr/local/lib/python3.9/site-packages/neo4j/work/transaction.py", line 134, in commit
self._connection.fetch_all()
File "/usr/local/lib/python3.9/site-packages/neo4j/io/_bolt4.py", line 429, in fetch_all
detail_delta, summary_delta = self.fetch_message()
File "/usr/local/lib/python3.9/site-packages/neo4j/io/_bolt4.py", line 334, in fetch_message
details, summary_signature, summary_metadata = next(self.inbox)
File "/usr/local/lib/python3.9/site-packages/neo4j/io/_common.py", line 79, in __next__
tag, fields = self.pop()
File "/usr/local/lib/python3.9/site-packages/neo4j/io/_common.py", line 73, in pop
return next(self._messages)
ValueError: generator already executing
After this exception appers we are getting several other very strange exceptions:
-
Failed to read from defunct connection IPv4Address(('10.0.201.154', 7687)) (IPv4Address(('10.0.201.154', 7687)))
-
No data
-
Internal server error (Failed to write to closed connection IPv4Address(('10.0.201.154', 7687)) (IPv4Address(('10.0.201.154', 7687))))
File "/ms_topology_center/../shared/graph_db.py", line 135, in commit_transaction
self._transaction.commit()
File "/usr/local/lib/python3.9/site-packages/neo4j/work/transaction.py", line 133, in commit
self._connection.send_all()
File "/usr/local/lib/python3.9/site-packages/neo4j/io/_bolt4.py", line 296, in send_all
raise ServiceUnavailable("Failed to write to closed connection {!r} ({!r})".format(
neo4j.exceptions.ServiceUnavailable: Failed to write to closed connection IPv4Address(('10.0.201.154', 7687)) (IPv4Address(('10.0.201.154', 7687)))
- [Errno 9] Bad file descriptor
After I restart the pod everything back to normal.
Any idea why it's happened to us and how can we avoid this kind of errors?
Thanks ,
Dan Yazerski
