Package eu.dnetlib.dhp.utils

Class SparkSessionSupport

java.lang.Object

eu.dnetlib.dhp.utils.SparkSessionSupport

public class SparkSessionSupport
extends Object

SparkSession utility methods.

Method Summary

Modifier and Type	Method	Description
static void	runWithSparkHiveSession(org.apache.spark.SparkConf conf, Boolean isSparkSessionManaged, FunctionalInterfaceSupport.ThrowingConsumer<org.apache.spark.sql.SparkSession,Exception> fn)	Runs a given function using SparkSession created with hive support and using default builder and supplied SparkConf.
static void	runWithSparkSession(Function<org.apache.spark.SparkConf,org.apache.spark.sql.SparkSession> sparkSessionBuilder, org.apache.spark.SparkConf conf, Boolean isSparkSessionManaged, FunctionalInterfaceSupport.ThrowingConsumer<org.apache.spark.sql.SparkSession,Exception> fn)	Runs a given function using SparkSession created using supplied builder and supplied SparkConf.
static void	runWithSparkSession(org.apache.spark.SparkConf conf, Boolean isSparkSessionManaged, FunctionalInterfaceSupport.ThrowingConsumer<org.apache.spark.sql.SparkSession,Exception> fn)	Runs a given function using SparkSession created using default builder and supplied SparkConf.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Details

runWithSparkSession

public static void runWithSparkSession(org.apache.spark.SparkConf conf,
 Boolean isSparkSessionManaged,
 FunctionalInterfaceSupport.ThrowingConsumer<org.apache.spark.sql.SparkSession,Exception> fn)

Runs a given function using SparkSession created using default builder and supplied SparkConf. Stops SparkSession when SparkSession is managed. Allows to reuse SparkSession created externally.

Parameters:

conf - SparkConf instance

isSparkSessionManaged - When true will stop SparkSession

fn - Consumer to be applied to constructed SparkSession

runWithSparkHiveSession

public static void runWithSparkHiveSession(org.apache.spark.SparkConf conf,
 Boolean isSparkSessionManaged,
 FunctionalInterfaceSupport.ThrowingConsumer<org.apache.spark.sql.SparkSession,Exception> fn)

Runs a given function using SparkSession created with hive support and using default builder and supplied SparkConf. Stops SparkSession when SparkSession is managed. Allows to reuse SparkSession created externally.

Parameters:

conf - SparkConf instance

isSparkSessionManaged - When true will stop SparkSession

fn - Consumer to be applied to constructed SparkSession

runWithSparkSession

public static void runWithSparkSession(Function<org.apache.spark.SparkConf,org.apache.spark.sql.SparkSession> sparkSessionBuilder,
 org.apache.spark.SparkConf conf,
 Boolean isSparkSessionManaged,
 FunctionalInterfaceSupport.ThrowingConsumer<org.apache.spark.sql.SparkSession,Exception> fn)

Runs a given function using SparkSession created using supplied builder and supplied SparkConf. Stops SparkSession when SparkSession is managed. Allows to reuse SparkSession created externally.

Parameters:

sparkSessionBuilder - Builder of SparkSession

conf - SparkConf instance

isSparkSessionManaged - When true will stop SparkSession

fn - Consumer to be applied to constructed SparkSession