Question

我有一个名为y的命名字符向量，看起来与此类似 -

D1        D2        D3        D4        D5 
 "X D X "  "G U V "   "F Q  " "A C U E"  "H I T "

我想用这个向量做的是创建字母的频率计数以及IDF加权。我试着运行这段代码：

dd <- Corpus(VectorSource(y)) #Make a corpus object from a text vector
dtm <- DocumentTermMatrix(dd, control = list(weighting = weightTfIdf))

然而，当我运行此代码时，我收到错误：

Warning message:
In weighting(x) : empty document(s): 1 2 3 4 5.

所有文件都有信件或至少有一个空格（我也想包括在计数中）。我不确定我做错了什么 - 我能够让这个例子有效 - Different tf-idf values in R and hand calculation。

使用我上面的例子我想要的是这样的：

A C D E F G H I Q T U V X
0 0 1 0 0 0 0 0 0 0 0 0 2 - D1
0 0 0 0 0 1 0 0 0 0 1 1 0 - D2
...

任何帮助将不胜感激！

Answer 1

你可以在R：

中做到

y <- c("X D X ",  "G U V ",   "F Q  ", "A C U E",  "H I T ")
names(y) <- paste0("D", 1:5)
# named vector of strings
y
D1        D2        D3        D4        D5 
"X D X "  "G U V "   "F Q  " "A C U E"  "H I T " 

# get rid of spaces, then split every letter
let <- sapply(y, function(x) strsplit(gsub(" ", "", x), ""))
# all possible letters
let_all <- unique(unlist(let))
# uses table on factored x with all possible levels
let_tab <- sapply(let, function(x) table(factor(x, levels=let_all)))
# with some cosmetics
t(let_tab[order(rownames(let_tab)), ])

   A C D E F G H I Q T U V X
D1 0 0 1 0 0 0 0 0 0 0 0 0 2
D2 0 0 0 0 0 1 0 0 0 0 1 1 0
D3 0 0 0 0 1 0 0 0 1 0 0 0 0
D4 1 1 0 1 0 0 0 0 0 0 1 0 0
D5 0 0 0 0 0 0 1 1 0 1 0 0 0

这是你想要的吗？如果是的话，这是一个完成所有这一切的功能：

tabulate_letters <- function(y){
  let <- sapply(y, function(x) strsplit(gsub(" ", "", x), ""))
  # all possible letters
  let_all <- unique(unlist(let))
  # uses table on factored x with all possible levels
  let_tab <- sapply(let, function(x) table(factor(x, levels=let_all)))
  # with some cosmetics
  t(let_tab[order(rownames(let_tab)), ])
}

tabulate_letters(y)

   A C D E F G H I Q T U V X
D1 0 0 1 0 0 0 0 0 0 0 0 0 2
D2 0 0 0 0 0 1 0 0 0 0 1 1 0
D3 0 0 0 0 1 0 0 0 1 0 0 0 0
D4 1 1 0 1 0 0 0 0 0 0 1 0 0
D5 0 0 0 0 0 0 1 1 0 1 0 0 0

Answer 2

我们也可以使用mtabulate

中的qdapTools来执行此操作

library(qdapTools)
mtabulate(strsplit(y, ' '))[-1]
#   A C D E F G H I Q T U V X
#D1 0 0 1 0 0 0 0 0 0 0 0 0 2
#D2 0 0 0 0 0 1 0 0 0 0 1 1 0
#D3 0 0 0 0 1 0 0 0 1 0 0 0 0
#D4 1 1 0 1 0 0 0 0 0 0 1 0 0
#D5 0 0 0 0 0 0 1 1 0 1 0 0 0

在执行trimws之前，我们可以使用strsplit删除前导/滞后空格。

mtabulate(strsplit(trimws(y), " "))

数据

y <- c("X D X ",  "G U V ",   "F Q  ", "A C U E",  "H I T ")
names(y) <- paste0("D", 1:5)

Answer 3

有一个应用程序： quanteda 包。

require(quanteda)
y <- c("X D X ",  "G U V ",   "F Q  ", "A C U E",  "H I T ")

dtm <- dfm(y, toLower = FALSE, verbose = FALSE)
# sort by letter, if that's important
dtm <- dtm[, sort(features(dtm))]
dtm
## Document-feature matrix of: 5 documents, 13 features.
## 5 x 13 sparse Matrix of class "dfmSparse"
##        features
## docs    A C D E F G H I Q T U V X
##   text1 0 0 1 0 0 0 0 0 0 0 0 0 2
##   text2 0 0 0 0 0 1 0 0 0 0 1 1 0
##   text3 0 0 0 0 1 0 0 0 1 0 0 0 0
##   text4 1 1 0 1 0 0 0 0 0 0 1 0 0
##   text5 0 0 0 0 0 0 1 1 0 1 0 0 0

如果你想要 tf-idf ，那也很容易：

tfidf(dtm)
## Document-feature matrix of: 5 documents, 13 features.
## 5 x 13 sparse Matrix of class "dfmSparse"
##        features
## docs          A       C       D       E       F       G       H       I       Q       T       U       V       X
##   text1 0       0       0.69897 0       0       0       0       0       0       0       0       0       1.39794
##   text2 0       0       0       0       0       0.69897 0       0       0       0       0.39794 0.69897 0      
##   text3 0       0       0       0       0.69897 0       0       0       0.69897 0       0       0       0      
##   text4 0.69897 0.69897 0       0.69897 0       0       0       0       0       0       0.39794 0       0      
##   text5 0       0       0       0       0       0       0.69897 0.69897 0       0.69897 0       0       0

计算字符串和TF-IDF中的字母频率

3 个答案:

数据