cancel
Showing results for 
Search instead for 
Did you mean: 

Head's Up! These forums are read-only. All users and content have migrated. Please join us at community.neo4j.com.

Poc.periodic.commit to loop through REST API pagination

I am querying web API from TheMovieDatabase (tmdb) for list of movies by year for supplied parameters.
The response is 20 movies per page and metadata for currentpage/total results/total pages.. Sample response below for 7th page.

{
	"page": 7,
	"total_results": 122,
	"total_pages": 7,
	"results": [
		{
			"popularity": 0.6,
			"vote_count": 1,
			"video": false,
			"poster_path": null,
			"id": 406847,
			"adult": false,
			"backdrop_path": null,
			"original_language": "hi",
			"original_title": "Gullu Dada 4",
			"genre_ids": [
				35
			],
			"title": "Gullu Dada 4",
			"vote_average": 4,
			"overview": "Miya Bhai (Adnan Sajid Khan), a small-time gangster who terrorizes a whole colony of people finds his heart taken hostage by the charms of aspiring singer Tarannum (Khushbu), but she already has a suitor in the male vocalist (Aziz Nasser) of the band. Meanwhile, another gangsta is harbouring dreams of getting the girl.",
			"release_date": "2013-12-27"
		},
		{
			"popularity": 0.996,
			"vote_count": 1,
			"video": false,
			"poster_path": "/9gUrGqt1EvfuBsUoTcixU2nJSnj.jpg",
			"id": 503928,
			"adult": false,
			"backdrop_path": "/wity5ST8pRLewrsHRk6vP0R7fIe.jpg",
			"original_language": "hi",
			"original_title": "Mahabharat",
			"genre_ids": [
				28,
				12,
				16,
				36,
				10751
			],
			"title": "Mahabharat",
			"vote_average": 6,
			"overview": "Mahabharat is considered as the greatest and the longest epic in world literature It has all the possible elements that a story could have - conflict, duty, sacrifice, heroism, truth, justice, romance, adventure, fantasy and relationships - making it truly an all-encompassing fascinating saga.",
			"release_date": "2013-12-27"
		}
	]
}

I have made below attempts using samples in cypher manual, Blog from Mark and Zendesk example etc.

1- Simply loading first page - Works for upto 20 entries

WITH 'https://api.themoviedb.org/3/discover/movie?api_key=??&language=en&sort_by=release_date.asc&include_adult=false&include_video=false&with_original_language=hi&year=2013&page=' + import.page AS uri, import.page AS initialPage, import
CALL apoc.load.json(uri) YIELD value
	UNWIND value.results as movie
	
	MERGE (mNode:Movie {id:movie.id})
	on create 
	set mNode += {  original_language:movie.original_language,
			title:movie.title,
			release_date:movie.release_date	}

2 - When I want to try looping through multiple pages using apoc.periodic.commit. Set up import node

{
  "identity": 256,
  "labels": [
    "Import"
  ],
  "properties": {
    "page": 1
  }
}
  • I seem to be going wrong in case statements or in periodic commit construct of limit.
    Any help here is highly appreciated.
call apoc.periodic.commit("
MATCH (import:Import)
WITH 'https://api.themoviedb.org/3/discover/movie?api_key=??&language=en&sort_by=release_date.asc&include_adult=false&include_video=false&with_original_language=hi&year=2013&page=' + import.page AS uri, import.page AS initialPage, import
CALL apoc.load.json(uri) YIELD value
	UNWIND value.results as movie
	
	MERGE (mNode:Movie {id:movie.id})
	on create 
	set mNode += {  original_language:movie.original_language,
			title:movie.title,
			release_date:movie.release_date	}
  WITH initialPage, import, value
  CASE 
	WHEN count(movie) < 20 
	   THEN 0 
	ELSE 
		count(movie) END AS count
  FOREACH(ignoreMe in CASE WHEN count = 0 THEN [] ELSE [1] END |
    MERGE (import)
    SET import.page = initialPage+1
  )
  RETURN count", {limit:20});
0 REPLIES 0