Lord of the Wiki Ring: Importing Wikidata into Neo4j and analyzing family trees

[bratanic_tomaz]

Lord of the Wiki Ring: Importing Wikidata into Neo4j and analyzing family trees

#Neo4j #graphs #sparql #wikidata #DataScience

Medium – 8 Jun 20

Lord of the Wiki Ring: Importing Wikidata into Neo4j and analyzing family trees

Learn how to scrape the LOTR world from WikiData and analyze it with the Neo4j for Graph Data Science toolbox

Reading time: 19 min read

[andreas_kuczera]

Hi Tomaz,
thanks for that great Blogpost. Can you tell me what to change to get it running on neo4j 4.3 ?
I think there is a problem with apoc.
Bests
Andreas

[bratanic_tomaz]

Where and what is the error?

[andreas_kuczera]

Here is the code for testing:

// import_races_query
// Prepare a SPARQL query 
WITH 'SELECT ?item ?itemLabel 
      WHERE{ ?item wdt:P31 wd:Q989255 . 
      SERVICE wikibase:label { bd:serviceParam wikibase:language 
      "[AUTO_LANGUAGE],en" } }' AS sparql 
// make a request to Wikidata
CALL apoc.load.jsonParams('https://query.wikidata.org/sparql?query=' + 
                           apoc.text.urlencode(sparql), 
                         { Accept: "application/sparql-results+json"}, null) 
YIELD value 
// Unwind results to row 
UNWIND value['results']['bindings'] as row 
// Prepare data 
WITH row['itemLabel']['value'] as race, 
     row['item']['value'] as url, 
     split(row['item']['value'],'/')[-1] as id 
// Store to Neo4j 
CREATE (r:Race) SET r.race = race, 
                    r.url = url, 
                    r.id = id;

This is the error i get:

Neo.ClientError.Procedure.ProcedureCallFailed

Failed to invoke procedure apoc.load.jsonParams: Caused by: java.lang.NoClassDefFoundError: Could not initialize class apoc.util.JsonUtil

[andreas_kuczera]

Same query some hours ago i got

Failed to invoke procedure `apoc.load.jsonParams`: Caused by: java.lang.VerifyError: Stack map does not match the one at exception handler 77
Exception Details:
  Location:
    com/fasterxml/jackson/databind/deser/std/StdDeserializer._parseDate(Lcom/fasterxml/jackson/core/JsonParser;Lcom/fasterxml/jackson/databind/DeserializationContext;)Ljava/util/Date; @77: astore
  Reason:
    Type 'com/fasterxml/jackson/core/JsonParseException' (current frame, stack[0]) is not assignable to 'com/fasterxml/jackson/core/exc/StreamReadException' (stack map, stack[0])
  Current Frame:
    bci: @69
    flags: { }
    locals: { 'com/fasterxml/jackson/databind/deser/std/StdDeserializer', 'com/fasterxml/jackson/core/JsonParser', 'com/fasterxml/jackson/databind/DeserializationContext' }
    stack: { 'com/fasterxml/jackson/core/JsonParseException' }
  Stackmap Frame:
    bci: @77
    flags: { }
    locals: { 'com/fasterxml/jackson/databind/deser/std/StdDeserializer', 'com/fasterxml/jackson/core/JsonParser', 'com/fasterxml/jackson/databind/DeserializationContext' }
    stack: { 'com/fasterxml/jackson/core/exc/StreamReadException' }
  Bytecode:
    0000000: 2bb6 0035 aa00 0000 0000 0081 0000 0003
    0000010: 0000 000b 0000 007a 0000 0081 0000 0081
    0000020: 0000 0034 0000 0041 0000 0081 0000 0081
    0000030: 0000 0081 0000 0071 2a2b b600 11b6 0012
    0000040: 2cb6 006b b02b b600 4742 a700 223a 052c
    0000050: 2ab4 0002 2bb6 006e 126f 03bd 0004 b600
    0000060: 70c0 002d 3a06 1906 b600 4c42 bb00 7159
    0000070: 21b7 0072 b02a 2cb6 0073 c000 71b0 2a2b
    0000080: 2cb6 0074 b02c 2ab4 0002 2bb6 0025 c000
    0000090: 71b0                                   
  Exception Handler Table:
    bci [69, 74] => handler: 77
    bci [69, 74] => handler: 77
  Stackmap Table:
    same_frame(@56)
    same_frame(@69)
    same_locals_1_stack_item_frame(@77,Object[#367])
    append_frame(@108,Long)
    chop_frame(@117,1)
    same_frame(@126)
    same_frame(@133)

[bratanic_tomaz]

Best to report this directly to APOC repository. This query works on 4.2 for me, but doesn't work on 4.3 either. I get different error though.

[andreas_kuczera]

Oh thanks, i will use 4.2 then 😉

[bratanic_tomaz]

I've tested it on sandbox, which uses Neo4j 4.3.3, and it works. Perhaps try updating neo4j to the latest version. It's strange behaviour nonetheless.

[andreas_kuczera]

On my Neo4j Desktop it does not work on 4.2.11 either.

[andreas_kuczera]

Got this info from Github but it does not work for me:

github.com/neo4j-contrib/neo4j-apoc-procedures

apoc.load.jsonParams does not work on neo4j 4.3.3

opened 02:58PM - 23 Sep 21 UTC

kuczera

When i try this query: `WITH 'SELECT ?item ?itemLabel WHERE{ ?item wd…t:P31 wd:Q989255 . SERVICE wikibase:label { bd:serviceParam wikibase:language "[AUTO_LANGUAGE],en" } }' AS sparql // make a request to Wikidata CALL apoc.load.jsonParams('http://query.wikidata.org/sparql?query=' + apoc.text.urlencode(sparql), { Accept: "application/sparql-results+json"}, null) YIELD value // Unwind results to row UNWIND value['results']['bindings'] as row // Prepare data WITH row['itemLabel']['value'] as race, row['item']['value'] as url, split(row['item']['value'],'/')[-1] as id // Store to Neo4j CREATE (r:Race) SET r.race = race, r.url = url, r.id = id; ` i get this error. `Neo.ClientError.Procedure.ProcedureCallFailed: Failed to invoke procedure `apoc.load.jsonParams`: Caused by: java.lang.RuntimeException: Can't read url or key http://query.wikidata.org/sparql?query=SELECT+%253Fitem+%253FitemLabel+%250A++++++WHERE%257B+%253Fitem+wdt%253AP31+wd%253AQ989255+.+%250A++++++SERVICE+wikibase%253Alabel+%257B+bd%253AserviceParam+wikibase%253Alanguage+%250A++++++%2522%255BAUTO_LANGUAGE%255D%252Cen%2522+%257D+%257D as json: Server returned HTTP response code: 400 for URL: https://query.wikidata.org/sparql?query=SELECT+%253Fitem+%253FitemLabel+%250A++++++WHERE%257B+%253Fitem+wdt%253AP31+wd%253AQ989255+.+%250A++++++SERVICE+wikibase%253Alabel+%257B+bd%253AserviceParam+wikibase%253Alanguage+%250A++++++%2522%255BAUTO_LANGUAGE%255D%252Cen%2522+%257D+%257D` On 3.5.29 it works.

[andreas_kuczera]

The problem are the newlines in the sparql query part which is not urlencoded in the right way.