我正在使用scikit-learn并正在构建管道。构建管道后,我使用GridSearchCV来查找最佳模型。我正在处理文本数据,所以我正在试验不同的词干分析器。我创建了一个名为Preprocessor的类,它接受一个stemmer和vectorizer类,然后尝试覆盖vectorizer的方法build_analyzer以合并给定的stemmer。但是,我看到GridSearchCV的set_params只是直接访问实例变量 - 也就是说它不会用新的分析器重新实例化一个矢量器,就像我一直在做的那样:
class Preprocessor(object):
# hard code the stopwords for now
stopwords = nltk.corpus.stopwords.words()
def __init__(self, stemmer_cls, vectorizer_cls):
self.stemmer = stemmer_cls()
analyzer = self._build_analyzer(self.stemmer, vectorizer_cls)
self.vectorizer = vectorizer_cls(stopwords=stopwords,
analyzer=analyzer,
decode_error='ignore')
def _build_analyzer(self, stemmer, vectorizer_cls):
# analyzer tokenizes and lowercases
analyzer = super(vectorizer_cls, self).build_analyzer()
return lambda doc: (stemmer.stem(w) for w in analyzer(doc))
def fit(self, **kwargs):
return self.vectorizer.fit(kwargs)
def transform(self, **kwargs):
return self.vectorizer.transform(kwargs)
def fit_transform(self, **kwargs):
return self.vectorizer.fit_transform(kwargs)
所以问题是:我如何覆盖传入的所有矢量化器类的build_analyzer?
答案 0 :(得分:0)
是的,GridSearchCV直接设置实例字段,然后调用适合更改字段的分类器。
scikit-learn中的每个分类器都是以这样的方式构建的,__init__
只设置参数字段,而进一步工作所需的所有依赖对象(如在你的情况下调用_build_analyzer)只在fit方法中构造。你必须添加存储vectorizer_cls的附加字段,然后你必须在fit方法中构造依赖于vectorized_cls和stemmer_cls对象。
类似的东西:
class Preprocessor(object):
# hard code the stopwords for now
stopwords = nltk.corpus.stopwords.words()
def __init__(self, stemmer_cls, vectorizer_cls):
self.stemmer_cls = stemmer_cls
self.vectorizer_cls = vectorizer_cls
def _build_analyzer(self, stemmer, vectorizer_cls):
# analyzer tokenizes and lowercases
analyzer = super(vectorizer_cls, self).build_analyzer()
return lambda doc: (stemmer.stem(w) for w in analyzer(doc))
def fit(self, **kwargs):
analyzer = self._build_analyzer(self.stemmer_cls(), vectorizer_cls)
self.vectorizer_cls = vectorizer_cls(stopwords=stopwords,
analyzer=analyzer,
decode_error='ignore')
return self.vectorizer_cls.fit(kwargs)
def transform(self, **kwargs):
return self.vectorizer_cls.transform(kwargs)
def fit_transform(self, **kwargs):
return self.vectorizer_cls.fit_transform(kwargs)