To generate an artifact for a different Spark version and/or Scala version, Hadoop version, or HBase version, pass command-line options as follows (changing version numbers appropriately):
$ mvn -Dspark.version=3.1.2 -Dscala.version=2.12.10 -Dhadoop-three.version=3.2.0 -Dscala.binary.version=2.12 -Dhbase.version=2.4.8 clean install
Note: to build the connector with Spark 2.x, compile it with -Dscala.binary.version=2.11
and use the profile -Dhadoop.profile=2.0
Client-side (Spark) configuration:
- The HBase configuration file
hbase-site.xml
should be made available to Spark, it can be copied to$SPARK_CONF_DIR
(default is $SPARK_HOME/conf`)
Server-side (HBase region servers) configuration:
- The following jars need to be in the CLASSPATH of the HBase region servers:
- scala-library, hbase-spark, and hbase-spark-protocol-shaded.
- The server-side configuration is needed for column filter pushdown
- if you cannot perform the server-side configuration, consider using
.option("hbase.spark.pushdown.columnfilter", false)
- if you cannot perform the server-side configuration, consider using
- The Scala library version must match the Scala version (2.11 or 2.12) used for compiling the connector.