Understand Spark Execution Modes - Local, Client & Cluster Modes
It is important to understand Spark Execution Modes - Local, Client & Cluster Modes . This is a Simple yet very crucial aspect to understand from a Big Data system point of view. In this post , I will try to explain these in very simple & easy terms.
1. Local Mode:
- This is like running a program on someone's laptop or desktop using a single JVM.
- It can be a program of any language - java, scala , python etc.
- But in these programs , you should have defined & used spark context object, imported spark libraries and processed data from your local system files.
- This is exactly the Local Mode - since everything runs LOCALLY and there is no concept of NODE involved and nothing runs in DISTRIBUTED mode.
- The driver & the executor is created inside a single JVM process.
2. Client Mode :
- In Client mode, Driver is started in the Local machine\laptop\Desktop i.e. Driver is outside of the Cluster.
- But the Executors will be running inside the Cluster.
- Hence Layman terms , Driver is a like a Client to the Cluster.
- Please note in this case your entire application is dependent in the Local machine since the Driver resides in here. In case of any issue in the local machine , the driver will go off . Subsequently entire application will go off. Hence this mode is not suitable for Production use cases.
- However it is good for Debugging or Testing since you can throw the outputs on the Driver Terminal which is a Local machine.
3. Cluster Mode :
Consider a Spark Cluster with 5 Executors.- In Cluster Mode, the Driver & Executor both runs inside the Cluster.
- You submit the spark job from your local machine to a Cluster machine inside the Cluster (Such machines are usually called Edge Node).
- You can shut down the local machine . It will not impact anything since now the Driver is running inside the Cluster.
- This is the approach used in Production use cases.