Apache Spark is a fast and general-purpose cluster computing system. It provides high-level APIs in Java, Scala, Python and R, and an optimized engine that supports general execution graphs. It also supports a rich set of higher-level tools including Spark SQL for SQL and structured data processing, MLlib for machine learning, GraphX for graph processing, and Spark Streaming.
module spider spark to find out what environment modules are available for this application.
- HPC_SPARK_DIR - installation directory
- HPC_SPARK_BIN - executable directory
- HPC_SPARK_SLURM - SLURM job script examples
- SPARK_HOME - examples directory
Running Spark on HiperGator
To run your Spark jobs on HiperGator, first, a Spark cluster should be created on HiperGator via SLURM. This section shows a simple example how to create a Spark cluster on HiperGator and how to submit your Spark jobs into the cluster. For details about running Spark jobs on HiperGaotr, please refer to Spark Workshop. For Spark parameters used in this section, please refer to Spark's homepage.
Spark cluster on HiperGator
Spark interactive job
Spark batch job
Job Script Examples
See the Spark_Job_Scripts page for spark Job script examples.