Neo4j

joe · ‎08-29-2018

Following up on a problem I posted on the Slack spatial channel on 24th of August. I’m trying to remove some old legacy spatial indexes that I’m no longer using. I’ve tried using the neo4j-shell, but they don’t remove. @craig.taverner suggested that I try deleting them from java using the index manager.
I’ve tried that, but they still don’t delete. The lucine indexes are definitely created by neo4j-spatial, not SDN, as was suggested before. The lucene indexes delete, but the spatial indexes don’t.

  def uplift_remove_unused_indexes() {
        def expected = i(GenericGraphService).fetchAll(FeatureSet).spatialIdx
        def manager = i(GraphDatabaseService).index()
        def i = manager.nodeIndexNames()
        i.each { idx ->
            if (!expected.find { idx.startsWith(it) }) {
                println "Deleting index $idx"
                def index = manager.forNodes(idx)
                index.delete()
            }
        }
    }

Running it a few times you can see that the indexes are still registered. The lucene indexes are deleted ok, but the spatial indexes backing them are not. (After the second attempt to delete them, the lucine indexes are automatically recreated by spatial)

DB UPLIFT 106: TDB-716/2018-08-20: Remove old indexes
Deleting index wpd
Deleting index locations
Deleting index spatial__neo4j-spatial__LayerNodeIndex__internal__spatialNodeLookup__
Deleting index landreg__neo4j-spatial__LayerNodeIndex__internal__spatialNodeLookup__
Deleting index blpu__neo4j-spatial__LayerNodeIndex__internal__spatialNodeLookup__
Deleting index locations__neo4j-spatial__LayerNodeIndex__internal__spatialNodeLookup__
Deleting index test
Deleting index junctionLocations__neo4j-spatial__LayerNodeIndex__internal__spatialNodeLookup__
Deleting index property__neo4j-spatial__LayerNodeIndex__internal__spatialNodeLookup__
Deleting index blpu2
Deleting index features__neo4j-spatial__LayerNodeIndex__internal__spatialNodeLookup__
Deleting index routes__neo4j-spatial__LayerNodeIndex__internal__spatialNodeLookup__
Deleting index junctionLocations
Deleting index spatial
Deleting index wpd__neo4j-spatial__LayerNodeIndex__internal__spatialNodeLookup__
Deleting index blpu
Deleting index spatial2__neo4j-spatial__LayerNodeIndex__internal__spatialNodeLookup__
Deleting index routeSpatial__neo4j-spatial__LayerNodeIndex__internal__spatialNodeLookup__
Deleting index test__neo4j-spatial__LayerNodeIndex__internal__spatialNodeLookup__
Deleting index network
Deleting index spatial2
Deleting index addressbase
Deleting index features
Deleting index property
Deleting index routes
Deleting index gps
Deleting index postcode__neo4j-spatial__LayerNodeIndex__internal__spatialNodeLookup__
Deleting index nga__neo4j-spatial__LayerNodeIndex__internal__spatialNodeLookup__
Deleting index topography__neo4j-spatial__LayerNodeIndex__internal__spatialNodeLookup__
Deleting index nga
Deleting index postcode
Deleting index routeSpatial
Deleting index network__neo4j-spatial__LayerNodeIndex__internal__spatialNodeLookup__
Deleting index landreg
Deleting index topography
Deleting index blpu2__neo4j-spatial__LayerNodeIndex__internal__spatialNodeLookup__
Deleting index addressbase__neo4j-spatial__LayerNodeIndex__internal__spatialNodeLookup__
Deleting index gps__neo4j-spatial__LayerNodeIndex__internal__spatialNodeLookup__


DB UPLIFT 107: TDB-716/2018-08-20: Remove old indexes
Deleting index wpd
Deleting index locations
Deleting index test
Deleting index blpu2
Deleting index junctionLocations
Deleting index spatial
Deleting index blpu
Deleting index network
Deleting index spatial2
Deleting index addressbase
Deleting index features
Deleting index property
Deleting index routes
Deleting index gps
Deleting index nga
Deleting index postcode
Deleting index routeSpatial
Deleting index landreg
Deleting index topography


DB UPLIFT 108: TDB-716/2018-08-20: Remove old indexes
Deleting index wpd
Deleting index locations
Deleting index spatial__neo4j-spatial__LayerNodeIndex__internal__spatialNodeLookup__
Deleting index landreg__neo4j-spatial__LayerNodeIndex__internal__spatialNodeLookup__
Deleting index blpu__neo4j-spatial__LayerNodeIndex__internal__spatialNodeLookup__
Deleting index locations__neo4j-spatial__LayerNodeIndex__internal__spatialNodeLookup__
Deleting index test
Deleting index property__neo4j-spatial__LayerNodeIndex__internal__spatialNodeLookup__
Deleting index junctionLocations__neo4j-spatial__LayerNodeIndex__internal__spatialNodeLookup__
Deleting index blpu2
Deleting index features__neo4j-spatial__LayerNodeIndex__internal__spatialNodeLookup__
Deleting index routes__neo4j-spatial__LayerNodeIndex__internal__spatialNodeLookup__
Deleting index junctionLocations
Deleting index spatial
Deleting index wpd__neo4j-spatial__LayerNodeIndex__internal__spatialNodeLookup__
Deleting index blpu
Deleting index routeSpatial__neo4j-spatial__LayerNodeIndex__internal__spatialNodeLookup__
Deleting index spatial2__neo4j-spatial__LayerNodeIndex__internal__spatialNodeLookup__
Deleting index test__neo4j-spatial__LayerNodeIndex__internal__spatialNodeLookup__
Deleting index network
Deleting index spatial2
Deleting index addressbase
Deleting index features
Deleting index property
Deleting index routes
Deleting index gps
Deleting index postcode__neo4j-spatial__LayerNodeIndex__internal__spatialNodeLookup__
Deleting index nga__neo4j-spatial__LayerNodeIndex__internal__spatialNodeLookup__
Deleting index topography__neo4j-spatial__LayerNodeIndex__internal__spatialNodeLookup__
Deleting index nga
Deleting index postcode
Deleting index routeSpatial
Deleting index network__neo4j-spatial__LayerNodeIndex__internal__spatialNodeLookup__
Deleting index blpu2__neo4j-spatial__LayerNodeIndex__internal__spatialNodeLookup__
Deleting index landreg
Deleting index topography
Deleting index addressbase__neo4j-spatial__LayerNodeIndex__internal__spatialNodeLookup__
Deleting index gps__neo4j-spatial__LayerNodeIndex__internal__spatialNodeLookup__

This is with neo4j-spatial (0.15-neo4j-2.3.4)
Perhaps there was a bug regarding the deletion of indexes in that version?

neo4j-sh (?)$ index --indexes
Node indexes:
  wpd
  locations
  bng_landreg__neo4j-spatial__LayerNodeIndex__internal__spatialNodeLookup__
  bng_spatial__neo4j-spatial__LayerNodeIndex__internal__spatialNodeLookup__
  test
  bng_mastermap
  bng_postcode
  blpu2
  junctionLocations
  spatial
  bng_blpu__neo4j-spatial__LayerNodeIndex__internal__spatialNodeLookup__
  blpu
  bng_postcode__neo4j-spatial__LayerNodeIndex__internal__spatialNodeLookup__
  bng_network__neo4j-spatial__LayerNodeIndex__internal__spatialNodeLookup__
  bng_spatial
  bng_blpu
  network
  bng_nga__neo4j-spatial__LayerNodeIndex__internal__spatialNodeLookup__
  spatial2
  addressbase
  bng_mastermap__neo4j-spatial__LayerNodeIndex__internal__spatialNodeLookup__
  features
  property
  routes
  gps
  nga__neo4j-spatial__LayerNodeIndex__internal__spatialNodeLookup__
  geotile__neo4j-spatial__LayerNodeIndex__internal__spatialNodeLookup__
  nga
  bng_nga
  postcode
  routeSpatial
  bng_wpd
  landreg
  topography
  bng_landreg
  bng_network
  geotile
  bng_wpd__neo4j-spatial__LayerNodeIndex__internal__spatialNodeLookup__

Relationship indexes:
  __rel_types__
neo4j-sh (?)$ index --get-config wpd
{
    "provider": "spatial",
    "wkt": "wkt"
}
neo4j-sh (?)$ index --delete wpd
neo4j-sh (?)$ index --indexes
Node indexes:
  wpd
  locations
  bng_landreg__neo4j-spatial__LayerNodeIndex__internal__spatialNodeLookup__
  bng_spatial__neo4j-spatial__LayerNodeIndex__internal__spatialNodeLookup__
  test
  bng_mastermap
  bng_postcode
  blpu2
  junctionLocations
  spatial
  wpd__neo4j-spatial__LayerNodeIndex__internal__spatialNodeLookup__
  bng_blpu__neo4j-spatial__LayerNodeIndex__internal__spatialNodeLookup__
  blpu
  bng_postcode__neo4j-spatial__LayerNodeIndex__internal__spatialNodeLookup__
  bng_network__neo4j-spatial__LayerNodeIndex__internal__spatialNodeLookup__
  bng_spatial
  bng_blpu
  network
  bng_nga__neo4j-spatial__LayerNodeIndex__internal__spatialNodeLookup__
  spatial2
  addressbase
  bng_mastermap__neo4j-spatial__LayerNodeIndex__internal__spatialNodeLookup__
  features
  property
  routes
  gps
  nga__neo4j-spatial__LayerNodeIndex__internal__spatialNodeLookup__
  geotile__neo4j-spatial__LayerNodeIndex__internal__spatialNodeLookup__
  nga
  bng_nga
  postcode
  routeSpatial
  bng_wpd
  landreg
  topography
  bng_landreg
  bng_network
  geotile
  bng_wpd__neo4j-spatial__LayerNodeIndex__internal__spatialNodeLookup__

Relationship indexes:
  __rel_types__
neo4j-sh (?)$

The spatial plugin definitely appears to have code to delete indexes:

SpatialDatabaseService:

public class LayerNodeIndex implements Index<Node> {
    [cut]
    public void delete() {
        this.spatialDB.deleteLayer(this.layer.getName(), new NullListener());
    }
}

Maybe it’s buggy.
Forgetting the lucine indexes for a second, the legacy indexes are just standard neo4j indexes with a spatial provider. As far as I know spatial doesn’t have any special state files, it just used layer nodes attached to the ReferenceNode (in this version). So it should be possible to just delete the indexes through Neo4j and cleanup the nodes by hand later.
So, I’m unsure as to why deleting the indexes doesn’t work.
If you’ve got any thoughts they would be gratefully received .

david_allen · ‎08-29-2018

At the time that you're running this code, can you please inspect your debug.log file with neo4j and paste the contents?

michael_hunger · ‎08-29-2018

Do you get any error messages?

It seems that the SpatialDB only deletes the layer (i.e. the nodes) but not the actual lucene index.

There was also a different implementation originally which we removed in between that used an implict mapping between spatial and nodes. Perhaps with that removal the remove lucene index functionality was also gone?

It seems that the delete function doesn't delete the Lucene bit, which is created on create.
Also you can see that instantiation of this class it recreates the lucene index with its name.
So make sure you also delete the spatial index.

Which version are you using?

joe · ‎08-29-2018

The version is 0.15-neo4j-2.3.4, against Neo4j-2.latest.

When I delete the lucene indexes by name they get removed.

The spatial indexes (etc, 'gps') don't want to be deleted though.

In the output above, I have three goes at deleting the indexes, and that removes the lucine, but leaves the short named indexes. Nothing appears to remove those.

joe · ‎09-19-2018

Apologies for not coming back sooner - I've just taken a look and it's not clear to me how to configure neo4j to log.

I've naively set logging for org.neo4j to DEBUG in my application logback.xml - it's an embedded neo4j instance, so I presume that the global logback would affect it.

However, there isn't any noise in my log, and there isn't any noise in the messages.log file in the db/ directory either.

What am I missing?

None that I can see.

I'm happy to remove the lucine files by hand; that's ok.

That's the problem. If you look at my original post, I'm not able to remove the spatial index. The code appears to try to, but when I look they are still there. I don't understand why neo4j won't allow me to delete an index.

This is quite a significant problem for us now, as we currently have quite a number of orphaned spatial indexes that we want to get rid of. Unfortunately the SDN code to delete a node just iterates over all of the spatial indexes, just in case it's registered there. This can take several seconds which makes my users's work flow really frustrating.

Joe

joe · ‎09-20-2018

I’m wondering how to get the indexes removed — they must be registered in the neo4j data structures somewhere. I don’t understand why the standard tools aren’t removing it. Is it possible somehow for the spatial plugin to prevent the removal of the neo4j meta data about the index?

michael_hunger · ‎10-11-2018

Perhaps you can migrate your data with the store-utils tools which shouldn't take along the indexes?

At least you could give it a try.

Neo4j

Problems deleting unwanted spatial indexes. (0.15-neo4j-2.3.4)