使用DirectPipelineRunner
运行Dataflow时 DatastoreV1.Query q = DatastoreV1.Query.newBuilder()
.addKind(DatastoreV1.KindExpression.newBuilder().setName("KIND").build())
.setFilter(DatastoreHelper.makeFilter(
DatastoreHelper.makeFilter(
"date",
DatastoreV1.PropertyFilter.Operator.GREATER_THAN_OR_EQUAL,
DatastoreHelper.makeValue(date)).build(),
DatastoreHelper.makeFilter(
"date",
DatastoreV1.PropertyFilter.Operator.LESS_THAN,
DatastoreHelper.makeValue(next_date)).build()
))
.build()
按预期工作。将作业提交到Dataflow时,出现以下错误:
(4e1bbdfd880a21c1): java.lang.IllegalArgumentException: Query cannot have any inequality filters. at
com.google.api.services.datastore.client.QuerySplitterImpl.validateFilter(QuerySplitterImpl.java:109) at
com.google.api.services.datastore.client.QuerySplitterImpl.validateFilter(QuerySplitterImpl.java:105) at
com.google.api.services.datastore.client.QuerySplitterImpl.validateQuery(QuerySplitterImpl.java:128) at
com.google.api.services.datastore.client.QuerySplitterImpl.getSplits(QuerySplitterImpl.java:71) at com.google.cloud.dataflow.sdk.io.DatastoreIO$Source.getSplitQueries(DatastoreIO.java:426) at com.google.cloud.dataflow.sdk.io.DatastoreIO$Source.splitIntoBundles(DatastoreIO.java:305) at
com.google.cloud.dataflow.sdk.runners.dataflow.CustomSources.performSplit(CustomSources.java:305) at
com.google.cloud.dataflow.sdk.runners.dataflow.CustomSources.performSourceOperation(CustomSources.java:151) at
com.google.cloud.dataflow.sdk.runners.worker.SourceOperationExecutor.execute(SourceOperationExecutor.java:62) at
com.google.cloud.dataflow.sdk.runners.worker.DataflowWorker.executeWork(DataflowWorker.java:254) at
com.google.cloud.dataflow.sdk.runners.worker.DataflowWorker.doWork(DataflowWorker.java:191) at
com.google.cloud.dataflow.sdk.runners.worker.DataflowWorker.getAndPerformWork(DataflowWorker.java:144) at
com.google.cloud.dataflow.sdk.runners.worker.DataflowWorkerHarness$WorkerThread.doWork(DataflowWorkerHarness.java:180) at
com.google.cloud.dataflow.sdk.runners.worker.DataflowWorkerHarness$WorkerThread.call(DataflowWorkerHarness.java:161) at
com.google.cloud.dataflow.sdk.runners.worker.DataflowWorkerHarness$WorkerThread.call(DataflowWorkerHarness.java:148) at
java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745)
我在本地拥有Dataflow SDK版本:1.3.0。
答案 0 :(得分:2)
这是Dataflow SDK for Java 1.4.0及更早版本中的代码问题。我们会将其跟踪为Issue #101 in the GitHub's repository issue tracker。我们会尽快解决这个问题 - 请点击此处进行更新。
我现在想不出任何微不足道的解决方法。对不起!
解决方案是更新到更新版本的Dataflow SDK。