Question

我正在尝试使用 apache spark（版本 3）读取 CSV 文件。我面临两个问题

csv 文件中的一个字段包含换行符，因此该行被分成两行
其中一列包含（也包含换行符）逗号，因此记录被拆分为多列

下面是我得到的代码和输出

import org.apache.spark.rdd.RDD;
import org.apache.spark.sql.Dataset;
import org.apache.spark.sql.Row;
import org.apache.spark.sql.SparkSession;

import scala.Tuple2;

public class MainApp {

    public static void main(String[] args) {
        // TODO Auto-generated method stub
        
        SparkSession session = SparkSession.builder().appName("Sample App").master("local[*]").getOrCreate();
        
        session.sparkContext().setLogLevel("ERROR");
        
        
        

        
        Dataset<Row> df = session.read().format("csv").
                option("quote","*" )
                .option("multiLine", "true")
                 .option("ignoreLeadingWhiteSpace", "true")
                 .option("ignoreTrailingWhiteSpace", "true")
                .load("/home/deepak/sample_dataset/*.csv");
        df.printSchema();
        df.show(false);
        
        session.stop();
        

    }

}



root
 |-- _c0: string (nullable = true)
 |-- _c1: string (nullable = true)
 |-- _c2: string (nullable = true)
 |-- _c3: string (nullable = true)

+--------------------------------------------+---------------+------+----------------------------+
|_c0                                         |_c1            |_c2   |_c3                         |
+--------------------------------------------+---------------+------+----------------------------+
|15169                                       |Sample data I  |RST1  |"*Insurance data conversion.|
|Sample (Sample/Sample)Sample Sample't Sample|Sample Box 9999|Sample|Sample 888888-7014*"        |
+--------------------------------------------+---------------+------+----------------------------+

下面是输入，用MS excel打开是一行。

*15169*,Sample data I,RST1,"*Insurance data conversion.
Sample (Sample/Sample)Sample Sample't Sample, Sample Box 9999, Sample, Sample 888888-7014*"

我不知道如何将 spark 读取为一条记录。

Answer 1

您不需要指定 quote，因为您的引号是默认字符 "。

Dataset<Row> df = session.read().format("csv")
                 .option("multiLine", "true")
                 .option("ignoreLeadingWhiteSpace", "true")
                 .option("ignoreTrailingWhiteSpace", "true")
                 .load("/home/deepak/sample_dataset/*.csv");

df.show()
+-------+-------------+----+----------------------------------------------------------------------------------------------------------------------+
|_c0    |_c1          |_c2 |_c3                                                                                                                   |
+-------+-------------+----+----------------------------------------------------------------------------------------------------------------------+
|*15169*|Sample data I|RST1|*Insurance data conversion.
Sample (Sample/Sample)Sample Sample't Sample, Sample Box 9999, Sample, Sample 888888-7014*|
+-------+-------------+----+----------------------------------------------------------------------------------------------------------------------+

csv 文件中的换行符 spark

1 个答案: