Question

我目前有一个脚本，该脚本从S3存储桶中获取数据并生成一个数据帧。我想将数据帧发送到雪花。目前，我正在这样做

df_write = df \
          .write\
          .format(SNOWFLAKE_SOURCE_NAME)\
          .options(**sfOptions)\
          .option("dbtable","test_table")\
          .mode("append")\
          .save()

产生的错误是

An error occurred while calling o72.save.
: java.lang.NoSuchMethodError: scala.Product.$init$(Lscala/Product;)V
    at net.snowflake.spark.snowflake.Parameters$MergedParameters.<init>(Parameters.scala:294)
    at net.snowflake.spark.snowflake.Parameters$.mergeParameters(Parameters.scala:288)
    at net.snowflake.spark.snowflake.DefaultSource.createRelation(DefaultSource.scala:97)
    at org.apache.spark.sql.execution.datasources.SaveIntoDataSourceCommand.run(SaveIntoDataSourceCommand.scala:45)
    at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:70)

我认为这可能是版本不匹配的错误。当前我正在使用hadoop-aws-2.7.3.jar，spark-snowflake_2.12-2.8.1-spark_3.0.jar，雪花-jdbc-3.12.8.jar

PySpark Streaming，写入时产生错误

0 个答案: