DevOps | Cloud | Analytics | Open Source | Programming

How To Fix - Spark Error “A master URL must be set" ?

Lets try to identify and fix the issue as regards to a common Spark error. Typically Spark application runs fine is Local Mode . But when submitted to the Spark cluster sometimes it errors out with below message -

org.apache.spark.SparkException: A master URL must be set in your configuration


Case 1:

Lets say your spark-submit command is as follows -

spark-submit --class ExmpleApp --master spark://cluster-node1-appnn1:7077 --jars $apppath app.jar

You might try the below fix -

  • Define the sparkContext object inside the main function and inside the class.

Case 2:

Please note - This is not the Right Approach for Production cluster though. But you can also define the Spark Master in the code when you define the SparkSession Object

SparkSession spark = SparkSession
    .appName("Basic App")
    .config("spark.master", "local")

Hope this helps To Fix - Spark Error “A master URL must be set".   Additional Read -