How To Fix - Spark Error “A master URL must be set" ?
Lets try to identify and fix the issue as regards to a common Spark error. Typically Spark application runs fine is Local Mode . But when submitted to the Spark cluster sometimes it errors out with below message -
org.apache.spark.SparkException: A master URL must be set in your configuration
Case 1:
spark-submit --class ExmpleApp --master spark://cluster-node1-appnn1:7077 --jars $apppath app.jar
- Define the sparkContext object inside the main function and inside the class.
Case 2:
Please note - This is not the Right Approach for Production cluster though. But you can also define the Spark Master in the code when you define the SparkSession Object
SparkSession spark = SparkSession
.builder()
.appName("Basic App")
.config("spark.master", "local")
.getOrCreate();