Question

我正在尝试运行以下代码，但状态变量始终为“PENDING”。你能告诉我我做错了什么吗？

Job execute = bigquery.jobs().insert(PROJECT_ID, runJob).execute();

String status;
while(status.equalsIgnoreCase("PENDING")) {
  status = execute.getStatus().getState();
  System.out.println("Status: " + status);
  Thread.wait(1000);
}

Answer 1

您的代码没有向BigQuery请求获取更新状态，它只是检查插入调用返回的Job的状态。

相反，您应该通过发出jobs.get请求来轮询作业状态，并检查该状态，例如：

Job job = bigquery.jobs().insert(PROJECT_ID, runJob).execute();
String status = job.getStatus().getState();
while(!status.equalsIgnoreCase("DONE")) {
  status = bigquery.jobs().get(PROJECT_ID, job.getId()).execute().getStatus().getState();
  System.out.println("Status: " + status);
  Thread.wait(1000);
}

*根据Jordan Tigani的评论编辑。

Answer 2

我已经意识到检查状态不是＆＃34;完成＆＃34;可能不会一直产生错误。有时，错误可以在作业进入＆＃34;完成＆＃34;之后被捕获。州。即，Job来自＆＃34;待定＆＃34;完成＆＃34;完成＆＃34;在一些错误中，跳过＆＃34;运行＆＃34;阶段。因此，即使作业是＆＃34;完成＆＃34;也可以检查作业[＆＃39; status＆＃39;]中的错误字段。

Answer 3

我没有使用繁忙的等待循环来同步阻塞运行插入的线程，而是使用了一个维护作业ID队列的预定线程。它遍历作业并检查其状态，在发现时记录错误。

这里的关键部分是，

安排线程以监控作业


    jobPollScheduler.scheduleAtFixedRate(new JobPoll(), SCHEDULE_SECONDS, SCHEDULE_SECONDS, TimeUnit.SECONDS);

遍历作业队列并检查其进度。重新排队任何不完整的内容


    while ((job = jobs.poll()) != null) {
      final Job statusJob = bigQuery.jobs().get(projectId, job.jobId).execute();
      if ("DONE".equals(statusJob.getStatus().getState())) {
        final ErrorProto errorResult = statusJob.getStatus().getErrorResult();
        if (errorResult == null || errorResult.toString() == null) {
          logger.debug("status={}, job={}", statusJob.getStatus().getState(), job);
        } else {
          logger.error("status={}, errorResult={}, job={}", statusJob.getStatus().getState(), errorResult, job);
        }
      } else {
        // job isn't done, yet. Add it back to queue.
        add(job.jobId);
        logger.debug("will check again, status={}, job={}", statusJob.getStatus().getState(), job);
      }
    }

完整的工作类


import com.google.api.services.bigquery.Bigquery;
import com.google.api.services.bigquery.model.ErrorProto;
import com.google.api.services.bigquery.model.Job;
import com.google.common.primitives.Longs;
import com.google.common.util.concurrent.ThreadFactoryBuilder;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

import java.util.Objects;
import java.util.Queue;
import java.util.concurrent.DelayQueue;
import java.util.concurrent.Delayed;
import java.util.concurrent.Executors;
import java.util.concurrent.ScheduledExecutorService;
import java.util.concurrent.TimeUnit;
import java.util.function.Supplier;
import javax.annotation.Nonnull;

/**
 * Monitor BigQuery inserts
 */
public class BigQueryMonitorSo21064586 {

  private static final Logger logger = LoggerFactory.getLogger(BigQueryMonitorSo21064586.class);
  private static final int SCHEDULE_SECONDS = 5;

  private final ScheduledExecutorService jobPollScheduler =
      Executors.newSingleThreadScheduledExecutor(new ThreadFactoryBuilder().setNameFormat("big-query-monitory-%d").build());
  private final Queue jobs = new DelayQueue();
  private final Supplier connectionSupplier;
  private final String projectId;

  /**
   * @param connectionSupplier gives us a connection to BigQuery
   * @param projectId Google cloud project
   */
  public BigQueryMonitorSo21064586(@Nonnull final Supplier connectionSupplier, @Nonnull final String projectId) {
    this.connectionSupplier = connectionSupplier;
    this.projectId = projectId;
  }

  public BigQueryMonitorSo21064586 start() {
    jobPollScheduler.scheduleAtFixedRate(new JobPoll(), SCHEDULE_SECONDS, SCHEDULE_SECONDS, TimeUnit.SECONDS);
    return this;
  }

  /**
   * @param jobId insert query job id
   */
  public void add(final String jobId) {
    final DelayedJobCheck job = new DelayedJobCheck(jobId);
    try {
      if (!jobs.offer(job)) {
        logger.error("could not enqueue BigQuery job, job={}", job);
      }
    } catch (final Exception e) {
      logger.error("failed to add job to queue, job={}", job, e);
    }
  }

  public void shutdown() {
    jobPollScheduler.shutdown();
  }

  private class JobPoll implements Runnable {

    /**
     * go through the queue and remove anything that is done
     */
    @Override
    public void run() {
      try {
        final Bigquery bigQuery = connectionSupplier.get();
        DelayedJobCheck job;
        while ((job = jobs.poll()) != null) {
          final Job statusJob = bigQuery.jobs().get(projectId, job.jobId).execute();
          if ("DONE".equals(statusJob.getStatus().getState())) {
            final ErrorProto errorResult = statusJob.getStatus().getErrorResult();
            if (errorResult == null || errorResult.toString() == null) {
              logger.debug("status={}, job={}", statusJob.getStatus().getState(), job);
            } else {
              logger.error("status={}, errorResult={}, job={}", statusJob.getStatus().getState(), errorResult, job);
            }
          } else {
            // job isn't done, yet. Add it back to queue.
            add(job.jobId);
            logger.debug("will check again, status={}, job={}", statusJob.getStatus().getState(), job);
          }
        }
      } catch (final Exception e) {
        logger.error("exception monitoring big query status, size={}", jobs.size(), e);
      }
    }
  }

  private static class DelayedJobCheck extends DelayedImpl {

    private final String jobId;

    DelayedJobCheck(final String jobId) {
      super(SCHEDULE_SECONDS, TimeUnit.SECONDS);
      this.jobId = jobId;
    }

    @Override
    public boolean equals(final Object obj) {
      if (this == obj) {
        return true;
      }
      if (obj == null || getClass() != obj.getClass()) {
        return false;
      }
      if (!super.equals(obj)) {
        return false;
      }
      final DelayedJobCheck other = (DelayedJobCheck) obj;
      return Objects.equals(jobId, other.jobId);
    }

    @Override
    public int hashCode() {
      return Objects.hash(super.hashCode(), jobId);
    }
  }

  private static class DelayedImpl implements Delayed {

    /**
     * timestamp when delay expires
     */
    private final long expiry;

    /**
     * @param amount how long the delay should be
     * @param timeUnit units of the delay
     */
    DelayedImpl(final long amount, final TimeUnit timeUnit) {
      final long more = TimeUnit.MILLISECONDS.convert(amount, timeUnit);
      expiry = System.currentTimeMillis() + more;
    }

    @Override
    public long getDelay(@Nonnull final TimeUnit unit) {
      final long diff = expiry - System.currentTimeMillis();
      return unit.convert(diff, TimeUnit.MILLISECONDS);
    }

    @Override
    public int compareTo(@Nonnull final Delayed o) {
      return Longs.compare(expiry, ((DelayedImpl) o).expiry);
    }

    @Override
    public boolean equals(final Object obj) {
      if (this == obj) {
        return true;
      }
      if (!(obj instanceof DelayedImpl)) {
        return false;
      }
      final DelayedImpl delayed = (DelayedImpl) obj;
      return expiry == delayed.expiry;
    }

    @Override
    public int hashCode() {
      return Objects.hash(expiry);
    }
  }
}

BigQuery工作状态始终是“待命”

3 个答案: