Question

我目前正尝试做类似Tracking Hadoop job status via web interface? (Exposing Hadoop to internal clients in the company)的事情。不同之处在于我的目标作业是本地作业（这是在单元测试中运行）。这就是我正在做的事情（我对mapProgress感兴趣）：

        final String job = this.config.get("mapreduce.job.id");
        final JobClient client = new JobClient(this.config);
        final RunningJob running = client.getJob(JobID.forName(job));
        if (running == null) {
            Logger.warn(
                this, String.format("Job id does not exist: %s", job)
            );
            throw new IllegalStateException(
                String.format("No running job found: %s", job)
            );
        }
        while (running.mapProgress() < 1.0) {
            TimeUnit.SECONDS.sleep(1);
        }

现在这实际上并不起作用 - 我在运行时遇到异常：

[←[0;33mWARN←[m] Thread-6 org.apache.hadoop.mapred.LocalJobRunner: job_local204744704_0002
java.lang.Exception: java.lang.IllegalStateException: No running job found: job_local204744704_0002
        at org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:462)
        at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:529)
Caused by: java.lang.IllegalStateException: No running job found: job_local204744704_0002
        at com.xockets.hadoop.XsInput.run(XsInput.java:150)
        at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:376)
        at org.apache.hadoop.mapred.LocalJobRunner$Job$ReduceTaskRunnable.run(LocalJobRunner.java:319)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:744)

我认为JobClient并非旨在获取本地工作的信息。有没有办法为本地运行的工作做类似的事情？

Hadoop：跟踪本地工作的进度

0 个答案: