Question

我正在尝试在以下链接上运行提到的矩阵乘法示例（带源代码）：

http://www.norstad.org/matrix-multiply/index.html

我在伪分布模式下设置了hadoop，我使用本教程配置它：

http://hadoop-tutorial.blogspot.com/2010/11/running-hadoop-in-pseudo-distributed.html?showComment=1321528406255#c3661776111033973764

当我运行我的jar文件时，我收到以下错误：

Identity test
11/11/30 10:37:34 INFO input.FileInputFormat: Total input paths to process : 2
11/11/30 10:37:34 INFO mapred.JobClient: Running job: job_201111291041_0010
11/11/30 10:37:35 INFO mapred.JobClient:  map 0% reduce 0%
11/11/30 10:37:44 INFO mapred.JobClient:  map 100% reduce 0%
11/11/30 10:37:56 INFO mapred.JobClient:  map 100% reduce 100%
11/11/30 10:37:58 INFO mapred.JobClient: Job complete: job_201111291041_0010
11/11/30 10:37:58 INFO mapred.JobClient: Counters: 17
11/11/30 10:37:58 INFO mapred.JobClient:   Job Counters
11/11/30 10:37:58 INFO mapred.JobClient:     Launched reduce tasks=1
11/11/30 10:37:58 INFO mapred.JobClient:     Launched map tasks=2
11/11/30 10:37:58 INFO mapred.JobClient:     Data-local map tasks=2
11/11/30 10:37:58 INFO mapred.JobClient:   FileSystemCounters
11/11/30 10:37:58 INFO mapred.JobClient:     FILE_BYTES_READ=114
11/11/30 10:37:58 INFO mapred.JobClient:     HDFS_BYTES_READ=248
11/11/30 10:37:58 INFO mapred.JobClient:     FILE_BYTES_WRITTEN=298
11/11/30 10:37:58 INFO mapred.JobClient:     HDFS_BYTES_WRITTEN=124
11/11/30 10:37:58 INFO mapred.JobClient:   Map-Reduce Framework
11/11/30 10:37:58 INFO mapred.JobClient:     Reduce input groups=2
11/11/30 10:37:58 INFO mapred.JobClient:     Combine output records=0
11/11/30 10:37:58 INFO mapred.JobClient:     Map input records=4
11/11/30 10:37:58 INFO mapred.JobClient:     Reduce shuffle bytes=60
11/11/30 10:37:58 INFO mapred.JobClient:     Reduce output records=2
11/11/30 10:37:58 INFO mapred.JobClient:     Spilled Records=8
11/11/30 10:37:58 INFO mapred.JobClient:     Map output bytes=100
11/11/30 10:37:58 INFO mapred.JobClient:     Combine input records=0
11/11/30 10:37:58 INFO mapred.JobClient:     Map output records=4
11/11/30 10:37:58 INFO mapred.JobClient:     Reduce input records=4
11/11/30 10:37:58 INFO input.FileInputFormat: Total input paths to process : 1
11/11/30 10:37:59 INFO mapred.JobClient: Running job: job_201111291041_0011
11/11/30 10:38:00 INFO mapred.JobClient:  map 0% reduce 0%
11/11/30 10:38:09 INFO mapred.JobClient:  map 100% reduce 0%
11/11/30 10:38:21 INFO mapred.JobClient:  map 100% reduce 100%
11/11/30 10:38:23 INFO mapred.JobClient: Job complete: job_201111291041_0011
11/11/30 10:38:23 INFO mapred.JobClient: Counters: 17
11/11/30 10:38:23 INFO mapred.JobClient:   Job Counters
11/11/30 10:38:23 INFO mapred.JobClient:     Launched reduce tasks=1
11/11/30 10:38:23 INFO mapred.JobClient:     Launched map tasks=1
11/11/30 10:38:23 INFO mapred.JobClient:     Data-local map tasks=1
11/11/30 10:38:23 INFO mapred.JobClient:   FileSystemCounters
11/11/30 10:38:23 INFO mapred.JobClient:     FILE_BYTES_READ=34
11/11/30 10:38:23 INFO mapred.JobClient:     HDFS_BYTES_READ=124
11/11/30 10:38:23 INFO mapred.JobClient:     FILE_BYTES_WRITTEN=100
11/11/30 10:38:23 INFO mapred.JobClient:     HDFS_BYTES_WRITTEN=124
11/11/30 10:38:23 INFO mapred.JobClient:   Map-Reduce Framework
11/11/30 10:38:23 INFO mapred.JobClient:     Reduce input groups=2
11/11/30 10:38:23 INFO mapred.JobClient:     Combine output records=2
11/11/30 10:38:23 INFO mapred.JobClient:     Map input records=2
11/11/30 10:38:23 INFO mapred.JobClient:     Reduce shuffle bytes=0
11/11/30 10:38:23 INFO mapred.JobClient:     Reduce output records=2
11/11/30 10:38:23 INFO mapred.JobClient:     Spilled Records=4
11/11/30 10:38:23 INFO mapred.JobClient:     Map output bytes=24
11/11/30 10:38:23 INFO mapred.JobClient:     Combine input records=2
11/11/30 10:38:23 INFO mapred.JobClient:     Map output records=2
11/11/30 10:38:23 INFO mapred.JobClient:     Reduce input records=2
Exception in thread "main" java.io.IOException: Cannot open filename /tmp/Matrix Multiply/out/_logs
        at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.openInfo(DFSClient.ja va:1497)
        at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.<init>(DFSClient.java :1488)
        at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:376)
        at org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSyst em.java:178)
        at org.apache.hadoop.io.SequenceFile$Reader.openFile(SequenceFile.java:1 437)
        at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:142 4)
        at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:141 7)
        at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:141 2)
        at TestMatrixMultiply.fillMatrix(TestMatrixMultiply.java:62)
        at TestMatrixMultiply.readMatrix(TestMatrixMultiply.java:84)
        at TestMatrixMultiply.checkAnswer(TestMatrixMultiply.java:108)
        at TestMatrixMultiply.runOneTest(TestMatrixMultiply.java:144)
        at TestMatrixMultiply.testIdentity(TestMatrixMultiply.java:156)
        at TestMatrixMultiply.main(TestMatrixMultiply.java:258)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl. java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces sorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:601)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:156)

有人可以告诉我，我做错了吗？谢谢

Answer 1

尝试读取作业输出。当您将其提交到群集时，它将添加此_log目录。由于目录不是序列文件，因此无法读取它们。

您必须更改读取此内容的代码。

我编写了相同的脚本：

FileStatus[] stati = fs.listStatus(output);
for (FileStatus status : stati) {
    if (!status.isDir()) {
        Path path = status.getPath();
        // HERE IS THE READ CODE FROM YOUR EXAMPLE
    }
}

http://code.google.com/p/hama-shortest-paths/source/browse/trunk/hama-gsoc/src/de/jungblut/clustering/mapreduce/KMeansClusteringJob.java#127

Answer 2

这可能是一个原始的建议，但是，您可能需要更改日志文件名 / tmp / Matrix \ Multiply / out / _logs。目录名中的空格可能无法自动处理，我认为您正在使用Linux。

Answer 3

TestMatrixMultiply.java中存在两个问题：

正如Thomas Jungblut所说，应该在readMatrix（）方法中排除_logs。我已经改变了这样的代码：

if (fs.isFile(path)) {
        fillMatrix(result, path);
    } else {
        FileStatus[] fileStatusArray = fs.listStatus(path);
        for (FileStatus fileStatus : fileStatusArray) {
            if ( !fileStatus.isDir() )  // this line is added by me
                fillMatrix(result, fileStatus.getPath());
        }
    }

在main（）方法的末尾，应该注释fs.delete，或者每次mapreduce作业完成后立即删除输出目录。
```
finally {
        //fs.delete(new Path(DATA_DIR_PATH), true);
    }
```

mapreduce矩阵乘法与hadoop

3 个答案: