How to insert ObjectId's from MongoDB with apoc?

[hedefalk]

I'm trying to seed a neo4j with data from a mongodb using apoc. The idea is to first create a kindof 1-1 mirror of the mongo db and then replace the key-based mapping with relations. I'm having troubles with ObjectId's which are extensively used as the main means for relations (PK/FK):

{
	"_id" : ObjectId("50db1341c2e638a3ebef2404"),
	"parent" : ObjectId("50db12c2c2e638a3ebef2401"),
	"referralDensityInfo" : ObjectId("50db12c2c2e638a3ebef2401"),
	"unitInfo" : [ ],
	"plural" : "Röda paprikor",
	"singular" : "Röd paprika",
	"referralUnitInfo" : [
		{
			"reference" : ObjectId("50db12c2c2e638a3ebef2401"),
			"unit" : ObjectId("508e86f6c2e60ff9d423e293")
		}
	],
	"dataSourceNutritionInfo" : ObjectId("507aa464c2e631891c0d9046"),
	"referralNutritionInfo" : ObjectId("50db12c2c2e638a3ebef2401"),
	"preferredName" : "Röd paprika",
	"searchWords" : [
		""
	]
}

First try was just using:

CALL apoc.mongodb.get('mongodb://host.docker.internal:5700','kostbevakningen','ingredients', {})

Cannot convert org.bson.types.ObjectId to AnyValue

Then I found this little clue:

So now calling:

CALL apoc.mongodb.get('mongodb://host.docker.internal:5700','kostbevakningen','ingredients', {}, true)

I get a bunch of data and it seems to work so far. However, when trying to use the values:

CALL apoc.mongodb.get('mongodb://host.docker.internal:5700','kostbevakningen','ingredients', {}, true) YIELD value as mongo_i
MERGE (i: Ingredient {_id: mongo_i._id})
SET i += {
  parent: mongo_i.parent,
  dataSourceNutritionInfo: mongo_i.dataSourceNutritionInfo
};

Property values can only be of primitive types or arrays thereof

So I guess there is something I'm missing in the puzzle still to be able to use these ObjectId's.

I have the jar's I'm supposed to have I think:

# ls -l plugins
total 15840
-rwxrwxrwx 1 neo4j neo4j     2217 Aug 22 11:27 README.txt
-rw------- 1 neo4j neo4j 13749813 May 31 11:12 apoc-3.5.0.4-all.jar
-rw-r--r-- 1 root  root    494021 Aug 13 11:06 bson-3.11.0.jar
-rw-r--r-- 1 root  root    374817 Aug 13 11:43 mongodb-driver-3.11.0.jar
-rw-r--r-- 1 root  root   1435219 Aug 13 11:25 mongodb-driver-core-3.11.0.jar
-rw-r--r-- 1 root  root    152721 Aug 13 11:34 mongodb-driver-sync-3.11.0.jar

There seems to be some effort to handle Id's here:

github.com

neo4j-contrib/neo4j-apoc-procedures/blob/9f6cf4fc256eb7505fdfa48c6a8f749d8bf352f1/src/main/java/apoc...

        }
    }


    /**
     * A document in MongoDB has a special field "_id" of type ObjectId
     * This object is not "packable" by Neo4jPacker so it must be converted to a value that Neo4j can deal with
     *
     * If the document is not null we simply override the ObjectId instance with its string representation
     */


    if (document != null) {
        Object objectId = document.get("_id");
        if (objectId != null) {
            document.put("_id", objectId.toString());
        }


    }


    return document;
}

but this seems to just handle the object primary key "_id", not any foreign key references to such a primary key which I guess is why I'm having problems.

Can I hook my own serializer in there somewhere? I would like to stick a toString() there on my keys and I would be fine I guess, but now it's just choking…

[hedefalk]

I noticed that the jar's permissions probably wasn't good. I figured maybe some automagic conversion wasn't flying because of missing bson.jar so I fixed that in my docker build:

COPY --from=sbt-build --chown=neo4j:neo4j /root/target/pack/lib/bson*.jar /var/lib/neo4j/plugins/
COPY --from=sbt-build --chown=neo4j:neo4j /root/target/pack/lib/mongodb-driver*.jar /var/lib/neo4j/plugins/

# ls -l plugins
total 15840
-rwxrwxrwx 1 neo4j neo4j     2217 Aug 22 11:27 README.txt
-rw------- 1 neo4j neo4j 13749813 May 31 11:12 apoc-3.5.0.4-all.jar
-rw-r--r-- 1 neo4j neo4j   494021 Aug 13 11:06 bson-3.11.0.jar
-rw-r--r-- 1 neo4j neo4j   374817 Aug 13 11:43 mongodb-driver-3.11.0.jar
-rw-r--r-- 1 neo4j neo4j  1435219 Aug 13 11:25 mongodb-driver-core-3.11.0.jar
-rw-r--r-- 1 neo4j neo4j   152721 Aug 13 11:34 mongodb-driver-sync-3.11.0.jar

but unfortunately that didn't help.

[hedefalk]

Ooh, ok, found the apoc.convert.toString

CALL apoc.mongodb.get('mongodb://host.docker.internal:5700','kostbevakningen','ingredients', {}, true) YIELD value as mongo_i
MERGE (i: Ingredient {_id: apoc.convert.toString(mongo_i._id)})
SET i += {
  parent: apoc.convert.toString(mongo_i.parent),
  dataSourceNutritionInfo: apoc.convert.toString(mongo_i.dataSourceNutritionInfo)
};

seems to work 🙂

[hedefalk]

Nope, that did not work well. That toString seems to do some object traversal which is definitely not what I want:

Any tips highly appreciated!

[mike_r_black]

I can't offer any help but I hope you're able to figure it out because I too will be doing a similar task soon so I'll be following this post for any wisdom I can glean.

[michael_hunger]

Can you create an APOC GH issue, so that we can handle those types of objects recursively both by themselves but also as part of arrays/subdocuments.

and link this thread.

[hedefalk]

Done. I can't link that issue because:

Probably should allow even newcomers to post links to github at least. But it's #1288.

I went another path so no rush for my sake

@mike.r.black I think it makes sense for anything but the most simple cases to probably make an app in your language of choice for the migration. I made a Scala app now that pulls data from Mongo and sticks them into neo4j, just using the appropriate drivers, in my case mongo-scala-driver and neotypes.

It seems to me that the apoc plugin for mongo is very very basic to say the least. I can't imagine anyone has ever used it for anything real, but maybe I'm missing something.

[michael_hunger]

we fixed the issue in the current apoc release

[deep_deepak_389]

Hello Michael,
I am doing a project in which I have to use mongoDB collection with which I want to use mongo ObjectId as neo4j collection's properties. I used apoc.convert.toString, it worked but I only want the exact value part of the objectId. Please suggest me some solution so that I can achieve my goal.

I am attaching screenshot for you to understand my issue.

Thanks.!
Screenshot (88)|690x388