Head's Up! These forums are read-only. All users and content have migrated. Please join us at community.neo4j.com.
08-29-2018 03:21 AM
Following up on a problem I posted on the Slack spatial channel on 24th of August. I’m trying to remove some old legacy spatial indexes that I’m no longer using. I’ve tried using the neo4j-shell, but they don’t remove. @craig.taverner suggested that I try deleting them from java using the index manager.
I’ve tried that, but they still don’t delete. The lucine indexes are definitely created by neo4j-spatial, not SDN, as was suggested before. The lucene indexes delete, but the spatial indexes don’t.
def uplift_remove_unused_indexes() {
def expected = i(GenericGraphService).fetchAll(FeatureSet).spatialIdx
def manager = i(GraphDatabaseService).index()
def i = manager.nodeIndexNames()
i.each { idx ->
if (!expected.find { idx.startsWith(it) }) {
println "Deleting index $idx"
def index = manager.forNodes(idx)
index.delete()
}
}
}
Running it a few times you can see that the indexes are still registered. The lucene indexes are deleted ok, but the spatial indexes backing them are not. (After the second attempt to delete them, the lucine indexes are automatically recreated by spatial)
DB UPLIFT 106: TDB-716/2018-08-20: Remove old indexes
Deleting index wpd
Deleting index locations
Deleting index spatial__neo4j-spatial__LayerNodeIndex__internal__spatialNodeLookup__
Deleting index landreg__neo4j-spatial__LayerNodeIndex__internal__spatialNodeLookup__
Deleting index blpu__neo4j-spatial__LayerNodeIndex__internal__spatialNodeLookup__
Deleting index locations__neo4j-spatial__LayerNodeIndex__internal__spatialNodeLookup__
Deleting index test
Deleting index junctionLocations__neo4j-spatial__LayerNodeIndex__internal__spatialNodeLookup__
Deleting index property__neo4j-spatial__LayerNodeIndex__internal__spatialNodeLookup__
Deleting index blpu2
Deleting index features__neo4j-spatial__LayerNodeIndex__internal__spatialNodeLookup__
Deleting index routes__neo4j-spatial__LayerNodeIndex__internal__spatialNodeLookup__
Deleting index junctionLocations
Deleting index spatial
Deleting index wpd__neo4j-spatial__LayerNodeIndex__internal__spatialNodeLookup__
Deleting index blpu
Deleting index spatial2__neo4j-spatial__LayerNodeIndex__internal__spatialNodeLookup__
Deleting index routeSpatial__neo4j-spatial__LayerNodeIndex__internal__spatialNodeLookup__
Deleting index test__neo4j-spatial__LayerNodeIndex__internal__spatialNodeLookup__
Deleting index network
Deleting index spatial2
Deleting index addressbase
Deleting index features
Deleting index property
Deleting index routes
Deleting index gps
Deleting index postcode__neo4j-spatial__LayerNodeIndex__internal__spatialNodeLookup__
Deleting index nga__neo4j-spatial__LayerNodeIndex__internal__spatialNodeLookup__
Deleting index topography__neo4j-spatial__LayerNodeIndex__internal__spatialNodeLookup__
Deleting index nga
Deleting index postcode
Deleting index routeSpatial
Deleting index network__neo4j-spatial__LayerNodeIndex__internal__spatialNodeLookup__
Deleting index landreg
Deleting index topography
Deleting index blpu2__neo4j-spatial__LayerNodeIndex__internal__spatialNodeLookup__
Deleting index addressbase__neo4j-spatial__LayerNodeIndex__internal__spatialNodeLookup__
Deleting index gps__neo4j-spatial__LayerNodeIndex__internal__spatialNodeLookup__
DB UPLIFT 107: TDB-716/2018-08-20: Remove old indexes
Deleting index wpd
Deleting index locations
Deleting index test
Deleting index blpu2
Deleting index junctionLocations
Deleting index spatial
Deleting index blpu
Deleting index network
Deleting index spatial2
Deleting index addressbase
Deleting index features
Deleting index property
Deleting index routes
Deleting index gps
Deleting index nga
Deleting index postcode
Deleting index routeSpatial
Deleting index landreg
Deleting index topography
DB UPLIFT 108: TDB-716/2018-08-20: Remove old indexes
Deleting index wpd
Deleting index locations
Deleting index spatial__neo4j-spatial__LayerNodeIndex__internal__spatialNodeLookup__
Deleting index landreg__neo4j-spatial__LayerNodeIndex__internal__spatialNodeLookup__
Deleting index blpu__neo4j-spatial__LayerNodeIndex__internal__spatialNodeLookup__
Deleting index locations__neo4j-spatial__LayerNodeIndex__internal__spatialNodeLookup__
Deleting index test
Deleting index property__neo4j-spatial__LayerNodeIndex__internal__spatialNodeLookup__
Deleting index junctionLocations__neo4j-spatial__LayerNodeIndex__internal__spatialNodeLookup__
Deleting index blpu2
Deleting index features__neo4j-spatial__LayerNodeIndex__internal__spatialNodeLookup__
Deleting index routes__neo4j-spatial__LayerNodeIndex__internal__spatialNodeLookup__
Deleting index junctionLocations
Deleting index spatial
Deleting index wpd__neo4j-spatial__LayerNodeIndex__internal__spatialNodeLookup__
Deleting index blpu
Deleting index routeSpatial__neo4j-spatial__LayerNodeIndex__internal__spatialNodeLookup__
Deleting index spatial2__neo4j-spatial__LayerNodeIndex__internal__spatialNodeLookup__
Deleting index test__neo4j-spatial__LayerNodeIndex__internal__spatialNodeLookup__
Deleting index network
Deleting index spatial2
Deleting index addressbase
Deleting index features
Deleting index property
Deleting index routes
Deleting index gps
Deleting index postcode__neo4j-spatial__LayerNodeIndex__internal__spatialNodeLookup__
Deleting index nga__neo4j-spatial__LayerNodeIndex__internal__spatialNodeLookup__
Deleting index topography__neo4j-spatial__LayerNodeIndex__internal__spatialNodeLookup__
Deleting index nga
Deleting index postcode
Deleting index routeSpatial
Deleting index network__neo4j-spatial__LayerNodeIndex__internal__spatialNodeLookup__
Deleting index blpu2__neo4j-spatial__LayerNodeIndex__internal__spatialNodeLookup__
Deleting index landreg
Deleting index topography
Deleting index addressbase__neo4j-spatial__LayerNodeIndex__internal__spatialNodeLookup__
Deleting index gps__neo4j-spatial__LayerNodeIndex__internal__spatialNodeLookup__
This is with neo4j-spatial (0.15-neo4j-2.3.4)
Perhaps there was a bug regarding the deletion of indexes in that version?
neo4j-sh (?)$ index --indexes
Node indexes:
wpd
locations
bng_landreg__neo4j-spatial__LayerNodeIndex__internal__spatialNodeLookup__
bng_spatial__neo4j-spatial__LayerNodeIndex__internal__spatialNodeLookup__
test
bng_mastermap
bng_postcode
blpu2
junctionLocations
spatial
bng_blpu__neo4j-spatial__LayerNodeIndex__internal__spatialNodeLookup__
blpu
bng_postcode__neo4j-spatial__LayerNodeIndex__internal__spatialNodeLookup__
bng_network__neo4j-spatial__LayerNodeIndex__internal__spatialNodeLookup__
bng_spatial
bng_blpu
network
bng_nga__neo4j-spatial__LayerNodeIndex__internal__spatialNodeLookup__
spatial2
addressbase
bng_mastermap__neo4j-spatial__LayerNodeIndex__internal__spatialNodeLookup__
features
property
routes
gps
nga__neo4j-spatial__LayerNodeIndex__internal__spatialNodeLookup__
geotile__neo4j-spatial__LayerNodeIndex__internal__spatialNodeLookup__
nga
bng_nga
postcode
routeSpatial
bng_wpd
landreg
topography
bng_landreg
bng_network
geotile
bng_wpd__neo4j-spatial__LayerNodeIndex__internal__spatialNodeLookup__
Relationship indexes:
__rel_types__
neo4j-sh (?)$ index --get-config wpd
{
"provider": "spatial",
"wkt": "wkt"
}
neo4j-sh (?)$ index --delete wpd
neo4j-sh (?)$ index --indexes
Node indexes:
wpd
locations
bng_landreg__neo4j-spatial__LayerNodeIndex__internal__spatialNodeLookup__
bng_spatial__neo4j-spatial__LayerNodeIndex__internal__spatialNodeLookup__
test
bng_mastermap
bng_postcode
blpu2
junctionLocations
spatial
wpd__neo4j-spatial__LayerNodeIndex__internal__spatialNodeLookup__
bng_blpu__neo4j-spatial__LayerNodeIndex__internal__spatialNodeLookup__
blpu
bng_postcode__neo4j-spatial__LayerNodeIndex__internal__spatialNodeLookup__
bng_network__neo4j-spatial__LayerNodeIndex__internal__spatialNodeLookup__
bng_spatial
bng_blpu
network
bng_nga__neo4j-spatial__LayerNodeIndex__internal__spatialNodeLookup__
spatial2
addressbase
bng_mastermap__neo4j-spatial__LayerNodeIndex__internal__spatialNodeLookup__
features
property
routes
gps
nga__neo4j-spatial__LayerNodeIndex__internal__spatialNodeLookup__
geotile__neo4j-spatial__LayerNodeIndex__internal__spatialNodeLookup__
nga
bng_nga
postcode
routeSpatial
bng_wpd
landreg
topography
bng_landreg
bng_network
geotile
bng_wpd__neo4j-spatial__LayerNodeIndex__internal__spatialNodeLookup__
Relationship indexes:
__rel_types__
neo4j-sh (?)$
The spatial plugin definitely appears to have code to delete indexes:
SpatialDatabaseService:
public class LayerNodeIndex implements Index<Node> {
[cut]
public void delete() {
this.spatialDB.deleteLayer(this.layer.getName(), new NullListener());
}
}
Maybe it’s buggy.
Forgetting the lucine indexes for a second, the legacy indexes are just standard neo4j indexes with a spatial provider. As far as I know spatial doesn’t have any special state files, it just used layer nodes attached to the ReferenceNode (in this version). So it should be possible to just delete the indexes through Neo4j and cleanup the nodes by hand later.
So, I’m unsure as to why deleting the indexes doesn’t work.
If you’ve got any thoughts they would be gratefully received .
08-29-2018 05:21 AM
At the time that you're running this code, can you please inspect your debug.log
file with neo4j and paste the contents?
08-29-2018 05:22 AM
Do you get any error messages?
It seems that the SpatialDB only deletes the layer (i.e. the nodes) but not the actual lucene index.
There was also a different implementation originally which we removed in between that used an implict mapping between spatial and nodes. Perhaps with that removal the remove lucene index functionality was also gone?
It seems that the delete function doesn't delete the Lucene bit, which is created on create.
Also you can see that instantiation of this class it recreates the lucene index with its name.
So make sure you also delete the spatial index.
Which version are you using?
08-29-2018 05:31 AM
The version is 0.15-neo4j-2.3.4, against Neo4j-2.latest.
When I delete the lucene indexes by name they get removed.
The spatial indexes (etc, 'gps') don't want to be deleted though.
In the output above, I have three goes at deleting the indexes, and that removes the lucine, but leaves the short named indexes. Nothing appears to remove those.
09-19-2018 04:30 PM
Apologies for not coming back sooner - I've just taken a look and it's not clear to me how to configure neo4j to log.
I've naively set logging for org.neo4j to DEBUG in my application logback.xml - it's an embedded neo4j instance, so I presume that the global logback would affect it.
However, there isn't any noise in my log, and there isn't any noise in the messages.log file in the db/ directory either.
What am I missing?
None that I can see.
I'm happy to remove the lucine files by hand; that's ok.
That's the problem. If you look at my original post, I'm not able to remove the spatial index. The code appears to try to, but when I look they are still there. I don't understand why neo4j won't allow me to delete an index.
This is quite a significant problem for us now, as we currently have quite a number of orphaned spatial indexes that we want to get rid of. Unfortunately the SDN code to delete a node just iterates over all of the spatial indexes, just in case it's registered there. This can take several seconds which makes my users's work flow really frustrating.
Joe
09-20-2018 02:03 PM
I’m wondering how to get the indexes removed — they must be registered in the neo4j data structures somewhere. I don’t understand why the standard tools aren’t removing it. Is it possible somehow for the spatial plugin to prevent the removal of the neo4j meta data about the index?
10-11-2018 05:39 PM
Perhaps you can migrate your data with the store-utils tools which shouldn't take along the indexes?
At least you could give it a try.
All the sessions of the conference are now available online