java.lang.NoSuchMethodError: 'void com.google.common.base.Preconditions.checkState(boolean, java.lang.String, long)

[RaghadAlkhudhair]

Description

Hello,

I am creating a pyspark job that writes streaming data to BigQuery and I needed this Library: com.google.cloud.bigdataoss:gcs-connector:hadoop3-2.2.10 which has guava as a dependency.
However, when I run the code that writes the streaming dataframe to BigQuery I keep getting an error with the class Preconditions from guava, here is the error:

Caused by: java.lang.NoSuchMethodError: 'void com.google.common.base.Preconditions.checkState(boolean, java.lang.String, long)'
	at com.google.cloud.hadoop.gcsio.GoogleCloudStorageReadOptions$Builder.build(GoogleCloudStorageReadOptions.java:236)
	at com.google.cloud.hadoop.gcsio.GoogleCloudStorageReadOptions.<clinit>(GoogleCloudStorageReadOptions.java:56)
	at com.google.cloud.hadoop.gcsio.GoogleCloudStorageOptions.builder(GoogleCloudStorageOptions.java:134)
	at com.google.cloud.hadoop.gcsio.GoogleCloudStorageOptions.<clinit>(GoogleCloudStorageOptions.java:114)
	at com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystemConfiguration.<clinit>(GoogleHadoopFileSystemConfiguration.java:356)
	at com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystemBase.<init>(GoogleHadoopFileSystemBase.java:248)
	at com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystem.<init>(GoogleHadoopFileSystem.java:58)
	at com.google.cloud.hadoop.fs.gcs.GoogleHadoopFS.<init>(GoogleHadoopFS.java:54)
	at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:76)
	at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:53)
	at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:499)
	at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:483)
	at org.apache.hadoop.fs.AbstractFileSystem.newInstance(AbstractFileSystem.java:143)
	... 75 more

I have doubts that the reason is my Java version which is: 17.0.9
Python Version: 3.12.1
guava release that is used as a dependency: 32.1.2

I checked the source code and the function CheckState exists with same argument, so what is the problem? is it Java version?

Example

Here is the code that generates the error:



import os
import time
import datetime
import warnings
import tempfile
from pyspark.sql.functions import col, lit
from pyspark.sql.avro.functions import from_avro, to_avro

warnings.simplefilter('ignore')

os.environ['PYSPARK_SUBMIT_ARGS'] = "--repositories https://packages.confluent.io/maven/,https://repo1.maven.org/maven2/  --packages com.google.cloud.bigdataoss:gcs-connector:hadoop3-2.2.21,com.google.cloud.spark:spark-bigquery-with-dependencies_2.12:0.36.1,za.co.absa:abris_2.12:6.4.0,org.apache.spark:spark-sql-kafka-0-10_2.12:3.5.0,org.apache.spark:spark-avro_2.12:3.5.0 pyspark-shell"


from pyspark.sql import SparkSession

spark= SparkSession \
    .builder \
    .appName(f"playground_{datetime.datetime.now().time()}") \
    .config("spark.driver.host", "localhost")\
    .config("credentialsFile", "application_default_credentials.json")\
    .config("spark.log.level","DEBUG")\
    .getOrCreate()

sc=spark.sparkContext

print(sc.version)
print(sc)
print("Spark App Name : "+ sc.appName)


streaming_df = spark \
        .readStream \
        .format("kafka") \
        .option("kafka.bootstrap.servers", "")\
        .option("subscribe", topic) \
        .option("startingOffsets", "earliest") \
        .option("partition", "0") \
        .option("startingOffsets", "earliest") \
        .option("maxOffsetsPerTrigger", 10) \
        .load()


extracted_df.writeStream \
  .format("bigquery") \
  .option("credentialsFile", "application_default_credentials.json")\
  .option("table", "")\
  .option("checkpointLocation", "") \
  .option("writeMethod","direct")\
  .option("project", "")\
  .outputMode("update") \
  .option("truncate", "false") \
  .start() \
  .awaitTermination()

Expected Behavior

To be able to use the dependency correctly

Actual Behavior

Error:

Caused by: java.lang.NoSuchMethodError: 'void com.google.common.base.Preconditions.checkState(boolean, java.lang.String, long)'
	at com.google.cloud.hadoop.gcsio.GoogleCloudStorageReadOptions$Builder.build(GoogleCloudStorageReadOptions.java:236)
	at com.google.cloud.hadoop.gcsio.GoogleCloudStorageReadOptions.<clinit>(GoogleCloudStorageReadOptions.java:56)
	at com.google.cloud.hadoop.gcsio.GoogleCloudStorageOptions.builder(GoogleCloudStorageOptions.java:134)
	at com.google.cloud.hadoop.gcsio.GoogleCloudStorageOptions.<clinit>(GoogleCloudStorageOptions.java:114)
	at com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystemConfiguration.<clinit>(GoogleHadoopFileSystemConfiguration.java:356)
	at com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystemBase.<init>(GoogleHadoopFileSystemBase.java:248)
	at com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystem.<init>(GoogleHadoopFileSystem.java:58)
	at com.google.cloud.hadoop.fs.gcs.GoogleHadoopFS.<init>(GoogleHadoopFS.java:54)
	at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:76)
	at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:53)
	at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:499)
	at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:483)
	at org.apache.hadoop.fs.AbstractFileSystem.newInstance(AbstractFileSystem.java:143)
	... 75 more

Packages

com.google.common.base

Platforms

Java 17

Checklist

• I agree to follow the code of conduct.

[kluever]

This question looks better suited for http://stackoverflow.com/questions/tagged/guava