Question

您好我正在执行Hive查询：

CREATE TABLE temp_session_orgid as
SELECT
    sorgid.property_num, sorgid.visitid, sorgid.fullvisitorid, sorgid.adate, sorgid.hits_customvariables_customvarvalue as orgid
FROM
    (
        SELECT
            *,
            row_number() over (partition by property_num, visitid, fullvisitorid, adate order by hitsid) as rn
        FROM bt_hits_custom_vars
        WHERE hits_customvariables_customvarname = 'orgId'
    ) sorgid
WHERE
    sorgid.rn = 1
;

Hive:2.1.1
EMR:5.3.1

我收到以下错误：

引起：org.apache.hadoop.hive.ql.metadata.HiveException： java.nio.channels.ClosedChannelException 在org.apache.hadoop.hive.ql.exec.FileSinkOperator.process（FileSinkOperator.java:785）在org.apache.hadoop.hive.ql.exec.Operator.forward（Operator.java:879）在org.apache.hadoop.hive.ql.exec.SelectOperator.process（SelectOperator.java:95）在org.apache.hadoop.hive.ql.exec.Operator.forward（Operator.java:879）在org.apache.hadoop.hive.ql.exec.FilterOperator.process（FilterOperator.java:126）在org.apache.hadoop.hive.ql.exec.Operator.forward（Operator.java:879）在org.apache.hadoop.hive.ql.exec.PTFOperator $ PTFInvocation.finishPartition（PTFOperator.java:373）在org.apache.hadoop.hive.ql.exec.PTFOperator.process（PTFOperator.java:122）在org.apache.hadoop.hive.ql.exec.Operator.forward（Operator.java:879）在org.apache.hadoop.hive.ql.exec.SelectOperator.process（SelectOperator.java:95）在org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource $ GroupIterator.next（ReduceRecordSource.java:348） ......还有17个引起：java.nio.channels.ClosedChannelException 在org.apache.hadoop.hdfs.DFSOutputStream.checkClosed（DFSOutputStream.java:1546）在org.apache.hadoop.fs.FSOutputSummer.write（FSOutputSummer.java:104） at org.apache.hadoop.fs.FSDataOutputStream $ PositionCache.write（FSDataOutputStream.java:60）在java.io.DataOutputStream.write（DataOutputStream.java:107） at org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat $ 1.write（HiveIgnoreKeyTextOutputFormat.java:87）在org.apache.hadoop.hive.ql.exec.FileSinkOperator.process（FileSinkOperator.java:751） ......还有27个 ]，TaskAttempt 3失败，info = [错误：运行任务时出错（失败）： attempt_1501196537592_0020_2_01_000000_3：了java.lang.RuntimeException：了java.lang.RuntimeException： org.apache.hadoop.hive.ql.metadata.HiveException：Hive运行时错误处理行时（tag = 0） {＆＃34;键＆＃34; {＆＃34; reducesinkkey0＆＃34;：＆＃34; 89634781＆＃34;＆＃34; reducesinkkey1＆＃34;：＆＃34; 1442844353＆＃34;＆＃34 ; reducesinkkey2＆＃34;：＆＃34; 5186210141339993001＆＃34;＆＃34; reducesinkkey3＆＃34;：＆＃34; 20150921＆＃34;＆＃34; reducesinkkey4＆＃34;：＆＃34; 1＆＃34; }，＆＃34;值＆＃34; {＆＃34; _col1＆＃34;：＆＃34; CUSTOMER＆＃34;}} 在org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:211）在org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run（TezProcessor.java:168）在org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run（LogicalIOProcessorRuntimeTask.java:370） at org.apache.tez.runtime.task.TaskRunner2Callable $ 1.run（TaskRunner2Callable.java:73） at org.apache.tez.runtime.task.TaskRunner2Callable $ 1.run（TaskRunner2Callable.java:61） at java.security.AccessController.doPrivileged（Native Method）在javax.security.auth.Subject.doAs（Subject.java:422）在org.apache.hadoop.security.UserGroupInformation.doAs（UserGroupInformation.java:1698） at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal（TaskRunner2Callable.java:61） at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal（TaskRunner2Callable.java:37）在org.apache.tez.common.CallableWithNdc.call（CallableWithNdc.java:36） at java.util.concurrent.FutureTask.run（FutureTask.java:266）在java.util.concurrent.ThreadPoolExecutor.runWorker（ThreadPoolExecutor.java:1142） at java.util.concurrent.ThreadPoolExecutor $ Worker.run（ThreadPoolExecutor.java:617）在java.lang.Thread.run（Thread.java:745）

此错误的原因是什么？解决此错误的解决方案是什么？

Answer 1

使用以下配置单元中的设置。

set hive.auto.convert.join=false;
set hive.vectorized.execution.enabled=false;
set hive.vectorized.execution.reduce.enabled=false;
set hive.merge.cardinality.check=false;
set hive.exec.dynamic.partition=true;
set hive.exec.dynamic.partition.mode=nonstrict;
set hive.exec.max.dynamic.partitions.pernode=500;
set hive.auto.convert.join.noconditionaltask = true;
set hive.auto.convert.join.noconditionaltask.size = 10000000;
set hive.auto.convert.sortmerge.join=true;

Hive查询错误java.lang.RuntimeException：org.apache.hadoop.hive.ql.metadata.HiveException

1 个答案: