How do you launch PySpark REPL using following command?

[hkim22]

This is regarding the book Graph Algorithms Ch. 8, Using Graph Algos to Enhance ML

I've downloaded spark and has been able to initiate the pyspark instance by running ./bin/pyspark,
in the terminal.

i am supposed to run the following command within the pyspark REPL:

export SPARK_VERSION="spark-2.4.0-bin-hadoop2.7"
./${SPARK_VERSION}/bin/pyspark
--driver-memory 2g
--executor-memory 6g
--packages julioasotodv:spark-tree-plotting:0.2

however, i cannot proceed from here. once i run the code above, i get a 'SyntaxError: invalid syntax' message. Has anyone successfully run that the piece of code?

Can you please assist with this?

[krisgeus]

hkim22:

pyspark
--driver-memory 2g
--executor-memory 6g
--packages julioasotodv:spark-tree-plotting:0.2

You need to make sure it is a single command. In it;s current form it launches pyspark and uses the rest as a spark program. Hence the syntax error. If you make it a one liner or use backslashes to tell your shell that the command continues on the next line it will startup pyspark correclty

pyspark \
--driver-memory 2g \
--executor-memory 6g \
--packages julioasotodv:spark-tree-plotting:0.2