How to get URL Parameters from an HTTP REST Call Response?

[bryant]

Does anyone know of a method for getting the URL parameters returned from an http call? I'm thinking some combination of apoc.data.url would give me this, but I'm not sure how to get cypher / apoc to make a call to a url and return the results url instead of the content.

Here's the situation. I have an API that I make a call to. Something like this: https://auth.something.com/oauth?private_id=xxxxxx&Respons_Type=code
The result is that the API returns a url with a parameter containing the code I need.

I'm trying to do this all in Cypher so I can make the web call and get the resulting url provided from the web call and parse out the parameters to get the generated "code" contained in the URL.

The resulting url looks something like this: http://localhost/?code=yyyyyy

So the question is how can I make a web call, get the resulting url back and then parse out the code parm.

I know I can parse out the web query parms using apoc.data.url, but how do I get the return resutls from the original http web call to parse out?

To clarify, the Web API does an http redirect. I want to capture the redirect URL so I can parse out the web query params. Is this even possible with any of the apoc.load or other procedures direclty in Cypher?

Any ideas / thoughts would be appreciated.

[david_allen · Accepted solution]

So under the covers when you do something like apoc.load.json or LOAD CSV, there's a web library/client that Neo4j is using. Let's say you are loading data from https://foo.com/?y=2 and it redirects to http://bar.com/?x=1

Most of that is invisible to the Neo4j procedures, because it's delegated to the HTTP client that's responsible to get your data. So "intermediate steps" in that redirect chain can't be captured. Neo4j is just going to know that the request was for https://foo.com/?y=2 and it got some data back.

In the data that comes back -- you're right you can use the APOC methods to parse URLs and get anything out of that you need....but if your question is about the redirect step (https://foo.com/?y=2) I'm not aware of a way you can do that as-is.

An option for you would be to have a custom stored procedure. This would take the input URL and would use an HTTP client or library itself, so that you could observe the entire "internal state" of any redirection mechanism that's happening. In your situation though, I wouldn't do this. This would be trying to turn the database / cypher into some kind of an http client. You could do it, but it wouldn't be easy or extensible.

On a more practical suggestion, you can use tools like curl to print out the total redirect stack for a particular request. I would be tempted to use curl & log parsing there to "generate extra data" that went into a vanilla LOAD CSV or LOAD JSON into Neo4j. Effectively, you'd be loading URLs, parsing the redirect logs -> transform to CSV, and then it's just any other vanilla CSV or JSON you'd load to be able to capture this info you want.

[bryant]

Thanks so much, for the detailed explanation, David! It makes perfect sense, and as I poked and hacked around I couldn't see any way to access the intermediate request objects being exposed in any of the procedures.

It's pretty easy to get the redirect through other means, I was just hoping to be able to do it all in Cypher. One of the credential tokens I'm using in an API has to be generated every 90 days, and I thought I could avoid having to use another tool to generate the the 90 day credential.

This definitely answers my question and again, I appreciate your thoughtful explanation and reply.