Self-written function is working locally but not on AWS EC2

[phi_lundgren]

Hello everyone,

I am facing a problem with a function that I wrote in Python to calculate the connections between two companies. The code works well on my local PC. I tested and there are many connections found. However, when I bring the code to AWS EC2 instance for faster computation, no connection is found. I checked the log file and found no error there. Do you have any hint if it is the problem with my code or with the Neo4j database?

Thank you very much in advance.

[Joel]

Just to verify, you are able to connect to the database and run a query with no errors, right? (you mention AWS EC2 so my first thought was firewall port issue)

If yes, then did you double check the contents of the neo4j on AWS EC2 matches what you have locally? e.g. check meta-model is the same, and the node/rel counts match, etc...

[phi_lundgren]

Hello Joel,

many thanks for your response. I dumped the database from my local PC and imported the dump file to AWS server. Hence the database is supposed to be identical.

Actually my code could access to the database, but it cannot count connections between companies. Basically my python script is not running properly there, although it's similar to the local one. I tried to find the mistake but unfortunately I could not detect any...

[Joel]

I still would ask that you compare the original and the new database, to make sure? Run this on the original, and then the new database....

This lists the node types and count

match (n)
return labels(n), count(n)

next check the relationship types and count

match ()-[r]-() 
RETURN type(r), count(r)

and/or the metagraph, I prefer to use apoc using

call apoc.meta.graph()

If these all look good try the cypher query in question directly on the database, either from the command line or using the neo4j desktop client. If that also works then you've narrowed down the issue to the python environment.

[phi_lundgren]

I had a look and they are identical. This is the function that could run locally but not on the cloud:

def get_shortest_path(tx, director_id1, director_id2, max_depth, connection_filter, role_type, last_date):
    role = connection_filter['Role']
    comp_type = connection_filter['Company Type']
    conditions = [
        f'all(rel IN relationships(path) WHERE rel.role in $role)' if len(role) else None,
        f'all(rel IN relationships(path) WHERE rel.role_type in $role_type AND any(start in rel.starts where start <= $last_date))' if len(
            role_type) else None,
        f'all(n1 IN [n in nodes(path) WHERE (n:Org)] WHERE n1.type in $comp_type)' if len(comp_type) else None,
        f'all(n1 IN [n in nodes(path) WHERE (n:Org)] WHERE NOT (n1.id in $avoid_org_ids))' if len(
            avoid_org_ids) else None,
    ]
    conditions = [line for line in conditions if line is not None]
    conditions = ' AND '.join(conditions)

    query_string = f'MATCH (a:Director {{id: $director_id1}}),(b:Director {{id: $director_id2}}), ' \
                   f'path = shortestPath((a)-[:ASSOCIATED_WITH*..{max_depth}]-(b)) ' \
                   f'{f"WHERE {conditions} " if conditions != "" else ""}' \
                   f'RETURN path'
    print(query_string)
    result = tx.run(
        query_string,
        director_id1=director_id1, director_id2=director_id2, role=role,
        role_type=role_type, comp_type=comp_type, avoid_org_ids=avoid_org_ids,
        last_date=last_date
    )
    spath = result.single()
    try:
        return spath['path'].nodes, spath['path'].relationships
    except TypeError:
        return None

There is no error, yet I still haven't figured out why this function produces only 0 in aws, but many positive values in my local PC.

[Joel]

I know it isn't fun (and sometimes not easy), but it really sounds like you'll have to dig into one known working example, and dig down deeper. One (working) example would hopefully reveal a difference, either in data or code. For example, the cypher match in the middle, run it by hand on the cloud host, using known data and expected results from the local to compare with. Yes this could be painful to do.

The only other thought I have based on the little I know is, have you checked your try/catch code, is it possible you are consuming/ignoring important throws somewhere? No errors you have seen, may not mean no errors have occurred. Maybe also check the server logs if you haven't.

This code snippet has a try/catch around a return statement, but not the code that I would have thought is most likely to throw on an error, which means those throws would I assume be caught at a higher level outside this function, right?

[phi_lundgren]

Hi Joel, thank you so much for your suggestion. I will try to inspect more log files. So far I have not found any from the Linux instance. However, I created a Windows instance in AWS EC2. The code ran perfectly. I assume that the error arises from an OS mismatch, yet thinking about debugging and building the projects from scratch in Linux gives me a nightmare.