Question

如何按照下面的代码片段查找匹配的字符串的出现，我能够将过滤后的字符串作为输出获取，但不能出现

import org.apache.spark._
import org.apache.spark.SparkConf
import org.apache.spark.SparkContext
import org.apache.spark.SparkContext._

object WordCount {
  def main(args: Array[String]) {

 val conf = new SparkConf().setAppName("wordCount")


 val sc = new SparkContext(conf)

  // Load our input data.
 val input =  sc.textFile("file:///tmp/ganesh/*")

 val matched_pattern = input.filter(line => line.contains("Title"))

 // Split it up into words.
val words = matched_pattern.flatMap(line => line.split(" "))

// Transform into pairs and count.
 val counts = words.map(word => (word, 1)).reduceByKey{case (x, y) => x + y}

 // Save the word count back out to a text file, causing evaluation.
 counts.saveAsTextFile("file:///tmp/sparkout")

  }
}

Answer 1

这里是一个示例-具有广播变量用法。 stopWords实际上是包含单词。

val dfsFilename = "/FileStore/tables/7dxa9btd1477497663691/Text_File_01-880f5.txt"
val readFileRDD = spark.sparkContext.textFile(dfsFilename)

// res4: Array[String] = Array(The the is Is a A to To OK ok I) //stopWords
val stopWordsInput = spark.sparkContext.textFile("/FileStore/tables/filter_words.txt")
val stopWords = stopWordsInput.flatMap(x => x.split(" ")).map(_.trim).collect.toSet
val broadcasted = sc.broadcast(stopWords)

val wcounts1 = readFileRDD.map(x => (x.replaceAll("[^A-Za-z0-9]", " ")                                     
                      .trim.toLowerCase))
                      .flatMap(line=>line.split(" "))
                      .filter(broadcasted.value.contains(_))
                      .map(word=>(word, 1))
                      .reduceByKey(_ + _)

wcounts1.collect

返回：

res2: Array[(String, Int)] = Array((The,1), (I,3), (to,1), (the,1))

您可以在stopWords上播放广播，这就是我所做的。

我看到了您的XML输入和一个replaceAll。您可以按照自己的喜好来摆弄它。我还添加了一个子句以将其全部转换为小写。

Spark查找匹配字符串的出现

1 个答案: