Question

我创建了自己的Mapreduce jar文件，并在Cosmos＆＃39;中进行了测试。 old Hadoop cluster成功使用HDFS shell命令。下一步是在新集群中测试相同的jar，所以我将其上传到了 new cluster's HDFS，到我的主文件夹（user / my.username）。

当我尝试使用下面的curl帖子启动Mapreduce作业时，

curl -X POST "http://computing.cosmos.lab.fiware.org:12000/tidoop/v1/user/my.username/jobs" -d '{"jar":"dt.jar","class_name":"DistanceTest","lib_jars":"dt.jar","input":"input","output":"output"}' -H "Content-Type: application/json" -H "X-Auth-Token: xxxxxxxxxxxxxxxxxxx"

我明白了：

{＆＃34;成功＆＃34;：＆＃34;假＆＃34;＆＃34;错误＆＃34;：255}

我为jar尝试了不同的路径值，我得到了相同的结果。我是否必须将我的罐子上传到其他地方，或者我错过了一些必要的步骤？

Answer 1

代码中存在一个错误，已在FIWARE Lab的全局实例中修复。

我测试了这个：

$ curl -X POST "http://computing.cosmos.lab.fiware.org:12000/tidoop/v1/user/myuser/jobs" -d '{"jar":"mrjars/hadoop-mapreduce-examples.jar","class_name":"wordcount","lib_jars":"mrjars/hadoop-mapreduce-examples.jar","input":"testdir","output":"outputdir"}' -H "Content-Type: application/json" -H "X-Auth-Token: mytoken"
{"success":"true","job_id": "job_1460639183882_0016"}

请注意，在这种情况下，mrjars/hadoop-mapreduce-examples.jar是hdfs:///user/myuser下的相对路径。始终使用相对路径与此操作。

$ curl -X GET "http://storage.cosmos.lab.fiware.org:14000/webhdfs/v1/user/myuser/mrjars?op=liststatus&user.name=myuser" -H "X-Auth-Token: mytoken"
{"FileStatuses":{"FileStatus":[{"pathSuffix":"hadoop-mapreduce-examples.jar","type":"FILE","length":270283,"owner":"myuser","group":"myuser","permission":"644","accessTime":1464702215233,"modificationTime":1464702215479,"blockSize":134217728,"replication":3}]}}

结果：

$ curl -X GET "http://storage.cosmos.lab.fiware.org:14000/webhdfs/v1/user/myuser/outputdir?op=liststatus&user.name=myuser" -H "X-Auth-Token: mytoken"
{"FileStatuses":{"FileStatus":[{"pathSuffix":"_SUCCESS","type":"FILE","length":0,"owner":"myuser","group":"myuser","permission":"644","accessTime":1464706333691,"modificationTime":1464706333706,"blockSize":134217728,"replication":3},{"pathSuffix":"part-r-00000","type":"FILE","length":128,"owner":"myuser","group":"myuser","permission":"644","accessTime":1464706333264,"modificationTime":1464706333460,"blockSize":134217728,"replication":3}]}}

我会尽快将修复程序上传到GitHub中的Cosmos回购。

无法在Cosmos中使用自定义Mapreduce jar文件

1 个答案: