Question

我正在尝试在非货币化数据集上创建自己的标记语料库，数据集大约有6250条推文。代码如下，尽管它给出了一个大小为200个条目的小数据集的结果。

df = pd.read_csv('Demonetization_data29th2.csv',encoding = "ISO-8859-1")

text = df['CONTENT']
sentiment = df['sentiment']
a =[]

tagged = [[nltk.word_tokenize(sent)] for sent in df['CONTENT']]
tagged = [nltk.pos_tag(sent) for sent in tagged]
print tagged[0]

print "---------"

brown_tagged_sents = tagged
print brown_tagged_sents[0]
size = int(len(brown_tagged_sents) * 0.7)
tags = [tag for (word, tag) in brown.tagged_words()]
defaultTag = nltk.FreqDist(tags).max()
print defaultTag
train_sents = brown_tagged_sents[:size]
test_sents = brown_tagged_sents[size:]
tagger = ngramTagger(train_sents, 2, defaultTag)
print tagger.evaluate(test_sents)

我收到了如下错误：

File "C:/Users/HP/pos-2.py", line 41, in <module>
    tagged = [[nltk.word_tokenize(sent)] for sent in df['CONTENT']]

  File "C:\ProgramData\Anaconda2\lib\site-packages\nltk\tokenize\__init__.py", line 130, in word_tokenize
    sentences = [text] if preserve_line else sent_tokenize(text, language)

  File "C:\ProgramData\Anaconda2\lib\site-packages\nltk\tokenize\__init__.py", line 97, in sent_tokenize
    return tokenizer.tokenize(text)

  File "C:\ProgramData\Anaconda2\lib\site-packages\nltk\tokenize\punkt.py", line 1235, in tokenize
    return list(self.sentences_from_text(text, realign_boundaries))

  File "C:\ProgramData\Anaconda2\lib\site-packages\nltk\tokenize\punkt.py", line 1283, in sentences_from_text
    return [text[s:e] for s, e in self.span_tokenize(text, realign_boundaries)]

  File "C:\ProgramData\Anaconda2\lib\site-packages\nltk\tokenize\punkt.py", line 1274, in span_tokenize
    return [(sl.start, sl.stop) for sl in slices]

  File "C:\ProgramData\Anaconda2\lib\site-packages\nltk\tokenize\punkt.py", line 1314, in _realign_boundaries
    for sl1, sl2 in _pair_iter(slices):

  File "C:\ProgramData\Anaconda2\lib\site-packages\nltk\tokenize\punkt.py", line 312, in _pair_iter
    prev = next(it)

  File "C:\ProgramData\Anaconda2\lib\site-packages\nltk\tokenize\punkt.py", line 1287, in _slices_from_text
    for match in self._lang_vars.period_context_re().finditer(text):

TypeError: expected string or buffer

请指导我出错的地方

Answer 1

你可能在该列的一行或多行中得到null。

用空字符串填充，如：

df['CONTENT'] = df['CONTENT'].fillna('')

我们如何创建标记语料库

1 个答案: