使用Java中的Weka进行基本文本分类

时间:2012-03-14 18:22:53

标签: java classification weka document-classification

我正在尝试使用Weka在JAVA中构建文本分类器。 我已经阅读了一些教程,并且我正在尝试构建自己的分类器。

我有以下类别:

    computer,sport,unknown 

以及以下已经过培训的数据

 cs belongs to computer
 java -> computer
 soccer -> sport
 snowboard -> sport

例如,如果用户想要对单词java进行分类,它应该返回类别计算机(毫无疑问,java只存在于该类别中!)。

它确实编译,但会产生奇怪的输出。

输出结果为:

      ====== RESULT ======  CLASSIFIED AS:  [0.5769230769230769, 0.2884615384615385, 0.1346153846153846]
      ====== RESULT ======  CLASSIFIED AS:  [0.42857142857142855, 0.42857142857142855, 0.14285714285714285]

但是第一个要分类的文本是java,它只出现在类别计算机中,因此它应该是

      [1.0 0.0 0.0] 

而对于另一个它根本找不到,所以应该归类为未知

      [0.0 0.0 1.0].

以下是代码:

    import java.io.FileNotFoundException;
    import java.io.Serializable;
    import java.util.Arrays;

    import weka.classifiers.Classifier;
    import weka.classifiers.bayes.NaiveBayesMultinomialUpdateable;
    import weka.core.Attribute;
    import weka.core.FastVector;
    import weka.core.Instance;
    import weka.core.Instances;
    import weka.filters.Filter;
    import weka.filters.unsupervised.attribute.StringToWordVector;

    public class TextClassifier implements Serializable {

        private static final long serialVersionUID = -1397598966481635120L;
        public static void main(String[] args) {
            try {
                TextClassifier cl = new TextClassifier(new NaiveBayesMultinomialUpdateable());
                cl.addCategory("computer");
                cl.addCategory("sport");
                cl.addCategory("unknown");
                cl.setupAfterCategorysAdded();

                //
                cl.addData("cs", "computer");
                cl.addData("java", "computer");
                cl.addData("soccer", "sport");
                cl.addData("snowboard", "sport");

                double[] result = cl.classifyMessage("java");
                System.out.println("====== RESULT ====== \tCLASSIFIED AS:\t" + Arrays.toString(result));

                result = cl.classifyMessage("asdasdasd");
                System.out.println("====== RESULT ======\tCLASSIFIED AS:\t" + Arrays.toString(result));
            } catch (Exception e) {
                e.printStackTrace();
            }
        }
        private Instances trainingData;
        private StringToWordVector filter;
        private Classifier classifier;
        private boolean upToDate;
        private FastVector classValues;
        private FastVector attributes;
        private boolean setup;

        private Instances filteredData;

        public TextClassifier(Classifier classifier) throws FileNotFoundException {
            this(classifier, 10);
        }

        public TextClassifier(Classifier classifier, int startSize) throws FileNotFoundException {
            this.filter = new StringToWordVector();
            this.classifier = classifier;
            // Create vector of attributes.
            this.attributes = new FastVector(2);
            // Add attribute for holding texts.
            this.attributes.addElement(new Attribute("text", (FastVector) null));
            // Add class attribute.
            this.classValues = new FastVector(startSize);
            this.setup = false;

        }

        public void addCategory(String category) {
            category = category.toLowerCase();
            // if required, double the capacity.
            int capacity = classValues.capacity();
            if (classValues.size() > (capacity - 5)) {
                classValues.setCapacity(capacity * 2);
            }
            classValues.addElement(category);
        }

        public void addData(String message, String classValue) throws IllegalStateException {
            if (!setup) {
                throw new IllegalStateException("Must use setup first");
            }
            message = message.toLowerCase();
            classValue = classValue.toLowerCase();
            // Make message into instance.
            Instance instance = makeInstance(message, trainingData);
            // Set class value for instance.
            instance.setClassValue(classValue);
            // Add instance to training data.
            trainingData.add(instance);
            upToDate = false;
        }

        /**
         * Check whether classifier and filter are up to date. Build i necessary.
         * @throws Exception
         */
        private void buildIfNeeded() throws Exception {
            if (!upToDate) {
                // Initialize filter and tell it about the input format.
                filter.setInputFormat(trainingData);
                // Generate word counts from the training data.
                filteredData = Filter.useFilter(trainingData, filter);
                // Rebuild classifier.
                classifier.buildClassifier(filteredData);
                upToDate = true;
            }
        }

        public double[] classifyMessage(String message) throws Exception {
            message = message.toLowerCase();
            if (!setup) {
                throw new Exception("Must use setup first");
            }
            // Check whether classifier has been built.
            if (trainingData.numInstances() == 0) {
                throw new Exception("No classifier available.");
            }
            buildIfNeeded();
            Instances testset = trainingData.stringFreeStructure();
            Instance testInstance = makeInstance(message, testset);

            // Filter instance.
            filter.input(testInstance);
            Instance filteredInstance = filter.output();
            return classifier.distributionForInstance(filteredInstance);

        }

        private Instance makeInstance(String text, Instances data) {
            // Create instance of length two.
            Instance instance = new Instance(2);
            // Set value for message attribute
            Attribute messageAtt = data.attribute("text");
            instance.setValue(messageAtt, messageAtt.addStringValue(text));
            // Give instance access to attribute information from the dataset.
            instance.setDataset(data);
            return instance;
        }

        public void setupAfterCategorysAdded() {
            attributes.addElement(new Attribute("class", classValues));
            // Create dataset with initial capacity of 100, and set index of class.
            trainingData = new Instances("MessageClassificationProblem", attributes, 100);
            trainingData.setClassIndex(trainingData.numAttributes() - 1);
            setup = true;
        }

    }
顺便说一句,发现了一个很好的页面:

http://www.hakank.org/weka/TextClassifierApplet3.html

3 个答案:

答案 0 :(得分:4)

贝叶斯分类器为您提供单词属于某个类别的(加权)概率。这几乎永远不会完全是0或1.您可以设置硬截止(例如0.5)并根据此确定类的成员资格,或检查计算的概率并根据该决定(即最高的地图为1,最低为0)。

答案 1 :(得分:1)

我想我只是提出你可以通过下载和使用http://lightsidelabs.com中的LightSIDE来完成大部分此类文本分类工作而无需编码。这个开源Java包包含WEKA,可用于Windows和Mac上的发行版 - 可以非常灵活地处理大多数WEKA友好数据集,允许您遍历各种模型,设置和参数,并为快照提供良好支持。在任何时候保存您的数据和模型以及分类结果,直到您构建了您满意的模型。该产品去年在Kaggle.com的ASAP竞赛中证明了这一点,并且获得了很大的关注。当然,人们总是需要或需要“自己动手”,但也许甚至可以作为支票,如果您正在编程WEKA解决方案,了解和使用LightSIDE可能非常方便。

答案 2 :(得分:0)

如果您尝试获取确定的类而不是分布,请尝试切换

return classifier.distributionForInstance(filteredInstance);

return classifier.classifyInstance(filteredInstance);