运行scalding作业时出现NoSuchMethodError

时间：2017-06-01 14:01:31

标签： json scala maven jackson scalding

我正在尝试将在CDH 4.5上运行的烫印作业升级到CDH 5.5.1。 该作业使用json4s来解析json数据。我在群集上运行作业时收到以下错误。它在我的本地机器上运行良好。

我正在使用scala 2.10＆amp;烫伤0.15.0版本。

ERROR [main] cascading.flow.stream.SourceStage: caught throwable
cascading.pipe.OperatorException: [JsonLine(/data/raw/gon...][read() @ com.xxx.xxx.GJob.<init>(GonzalesJob.scala:15)] operator Each failed executing operation
at cascading.flow.stream.FunctionEachStage.receive(FunctionEachStage.java:107)
at cascading.flow.stream.FunctionEachStage.receive(FunctionEachStage.java:39)
at cascading.flow.stream.SourceStage.map(SourceStage.java:102)
at cascading.flow.stream.SourceStage.run(SourceStage.java:58)
at cascading.flow.hadoop.FlowMapper.run(FlowMapper.java:130)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Caused by: java.lang.NoSuchMethodError: com.fasterxml.jackson.databind.deser.std.ContainerDeserializerBase: method <init>(Lcom/fasterxml/jackson/databind/JavaType;)V not found
at com.fasterxml.jackson.module.scala.deser.UnsortedMapDeserializer.<init>(UnsortedMapDeserializerModule.scala:39)
at com.fasterxml.jackson.module.scala.deser.UnsortedMapDeserializerResolver$.findMapLikeDeserializer(UnsortedMapDeserializerModule.scala:97)
at com.fasterxml.jackson.databind.deser.BasicDeserializerFactory._findCustomMapLikeDeserializer(BasicDeserializerFactory.java:1030)
at com.fasterxml.jackson.databind.deser.BasicDeserializerFactory.createMapLikeDeserializer(BasicDeserializerFactory.java:994)
at com.fasterxml.jackson.databind.deser.DeserializerCache._createDeserializer2(DeserializerCache.java:384)
at com.fasterxml.jackson.databind.deser.DeserializerCache._createDeserializer(DeserializerCache.java:354)
at com.fasterxml.jackson.databind.deser.DeserializerCache._createAndCache2(DeserializerCache.java:267)
at com.fasterxml.jackson.databind.deser.DeserializerCache._createAndCacheValueDeserializer(DeserializerCache.java:247)
at com.fasterxml.jackson.databind.deser.DeserializerCache.findValueDeserializer(DeserializerCache.java:146)
at com.fasterxml.jackson.databind.DeserializationContext.findRootValueDeserializer(DeserializationContext.java:322)
at com.fasterxml.jackson.databind.ObjectMapper._findRootDeserializer(ObjectMapper.java:2990)
at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:2884)
at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:2041)
at com.twitter.scalding.JsonLine$$anonfun$transformForRead$1.applyOrElse(JsonLine.scala:70)
at com.twitter.scalding.JsonLine$$anonfun$transformForRead$1.applyOrElse(JsonLine.scala:68)
at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:33)
at com.twitter.scalding.CollectFunction.operate(Operations.scala:94)
at cascading.flow.stream.FunctionEachStage.receive(FunctionEachStage.java:99)
... 11 more

这是我的依赖项中的杰克逊的grep 

mvn dependency:tree 

--- maven-dependency-plugin:2.8:tree (default-cli) @  ---
[INFO] bskyb.bigdata:gonzales-etl:jar:1.3
[INFO] +- org.scala-lang:scala-library:jar:2.10.5:compile
[INFO] +- org.scalatest:scalatest_2.10:jar:2.2.2:test
[INFO] |  \- org.scala-lang:scala-reflect:jar:2.10.4:compile
[INFO] +- org.apache.hadoop:hadoop-yarn-client:jar:2.6.0-cdh5.5.1:compile
[INFO] |  +- com.google.guava:guava:jar:11.0.2:compile
[INFO] |  |  \- com.google.code.findbugs:jsr305:jar:1.3.9:compile
[INFO] |  +- commons-logging:commons-logging:jar:1.1.3:compile
[INFO] |  +- commons-lang:commons-lang:jar:2.6:compile
[INFO] |  +- commons-cli:commons-cli:jar:1.2:compile
[INFO] |  +- log4j:log4j:jar:1.2.17:compile
[INFO] |  +- org.apache.hadoop:hadoop-annotations:jar:2.6.0-cdh5.5.1:compile
[INFO] |  +- org.apache.hadoop:hadoop-yarn-api:jar:2.6.0-cdh5.5.1:compile
[INFO] |  |  \- com.google.protobuf:protobuf-java:jar:2.5.0:compile
[INFO] |  \- org.apache.hadoop:hadoop-yarn-common:jar:2.6.0-cdh5.5.1:compile
[INFO] |     +- javax.xml.bind:jaxb-api:jar:2.2.2:compile
[INFO] |     |  +- javax.xml.stream:stax-api:jar:1.0-2:compile
[INFO] |     |  \- javax.activation:activation:jar:1.1:compile
[INFO] |     +- org.apache.commons:commons-compress:jar:1.4.1:compile
[INFO] |     |  \- org.tukaani:xz:jar:1.0:compile
[INFO] |     +- commons-codec:commons-codec:jar:1.4:compile
[INFO] |     +- org.mortbay.jetty:jetty-util:jar:6.1.26.cloudera.4:compile
[INFO] |     +- com.sun.jersey:jersey-core:jar:1.9:compile
[INFO] |     +- com.sun.jersey:jersey-client:jar:1.9:compile
[INFO] |     +- com.google.inject.extensions:guice-servlet:jar:3.0:compile
[INFO] |     +- commons-io:commons-io:jar:2.4:compile
[INFO] |     +- com.google.inject:guice:jar:3.0:compile
[INFO] |     |  +- javax.inject:javax.inject:jar:1:compile
[INFO] |     |  \- aopalliance:aopalliance:jar:1.0:compile
[INFO] |     +- com.sun.jersey:jersey-server:jar:1.9:compile
[INFO] |     |  \- asm:asm:jar:3.1:compile
[INFO] |     +- com.sun.jersey:jersey-json:jar:1.9:compile
[INFO] |     |  +- org.codehaus.jettison:jettison:jar:1.1:compile
[INFO] |     |  \- com.sun.xml.bind:jaxb-impl:jar:2.2.3-1:compile
[INFO] |     \- com.sun.jersey.contribs:jersey-guice:jar:1.9:compile
[INFO] +- com.twitter:scalding-commons_2.10:jar:0.11.0:compile
[INFO] |  +- com.twitter:scalding-args_2.10:jar:0.11.0:compile
[INFO] |  +- com.backtype:dfs-datastores-cascading:jar:1.3.4:compile
[INFO] |  +- com.backtype:dfs-datastores:jar:1.3.4:compile
[INFO] |  |  \- jvyaml:jvyaml:jar:1.0.0:compile
[INFO] |  +- com.twitter:bijection-core_2.10:jar:0.6.3:compile
[INFO] |  +- com.twitter:algebird-core_2.10:jar:0.7.0:compile
[INFO] |  |  \- com.googlecode.javaewah:JavaEWAH:jar:0.6.6:compile
[INFO] |  +- com.twitter:chill_2.10:jar:0.4.0:compile
[INFO] |  |  \- com.esotericsoftware.kryo:kryo:jar:2.21:compile
[INFO] |  |     +-     com.esotericsoftware.reflectasm:reflectasm:jar:shaded:1.07:compile
[INFO] |  |     |  \- org.ow2.asm:asm:jar:4.0:compile
[INFO] |  |     +- com.esotericsoftware.minlog:minlog:jar:1.2:compile
[INFO] |  |     \- org.objenesis:objenesis:jar:1.2:compile
[INFO] |  +- com.twitter.elephantbird:elephant-bird-cascading2:jar:4.4:compile
[INFO] |  +- com.hadoop.gplcompression:hadoop-lzo:jar:0.4.16:compile
[INFO] |  +- org.apache.thrift:libthrift:jar:0.5.0:compile
[INFO] |  \- org.slf4j:slf4j-api:jar:1.6.6:compile
[INFO] +- com.twitter:scalding-core_2.10:jar:0.15.0:compile
[INFO] |  +- com.twitter:scalding-serialization_2.10:jar:0.15.0:compile
[INFO] |  +- com.twitter:maple:jar:0.15.0:compile
[INFO] |  +- cascading:cascading-core:jar:2.6.1:compile
[INFO] |  |  +- riffle:riffle:jar:0.1-dev:compile
[INFO] |  |  +- thirdparty:jgrapht-jdk1.6:jar:0.8.1:compile
[INFO] |  |  \- org.codehaus.janino:janino:jar:2.7.5:compile
[INFO] |  |     \- org.codehaus.janino:commons-compiler:jar:2.7.5:compile
[INFO] |  +- cascading:cascading-local:jar:2.6.1:compile
[INFO] |  +- cascading:cascading-hadoop:jar:2.6.1:compile
[INFO] |  +- com.twitter:chill-hadoop:jar:0.6.0:compile
[INFO] |  +- com.twitter:chill-algebird_2.10:jar:0.6.0:compile
[INFO] |  \- com.twitter:chill-java:jar:0.6.0:compile
[INFO] +- com.twitter:scalding-date_2.10:jar:0.15.0:compile
[INFO] +- com.twitter:scalding-json_2.10:jar:0.15.0:compile
**[INFO] |  +- com.fasterxml.jackson.module:jackson-module-scala_2.10:jar:2.4.2:compile**
[INFO] |  |  \- com.thoughtworks.paranamer:paranamer:jar:2.6:compile
[INFO] |  \- org.json4s:json4s-native_2.10:jar:3.2.11:compile
[INFO] +- parallelai:parallelai.spyglass:jar:2.10_0.10_CDH5.2.1_4.4:compile
[INFO] |  +- org.apache.hadoop:hadoop-client:jar:2.5.0-cdh5.2.1:compile
[INFO] |  |  +- org.apache.hadoop:hadoop-common:jar:2.5.0-cdh5.2.1:compile
[INFO] |  |  |  +- org.apache.commons:commons-math3:jar:3.1.1:compile
[INFO] |  |  |  +- xmlenc:xmlenc:jar:0.52:compile
[INFO] |  |  |  +- commons-net:commons-net:jar:3.1:compile
[INFO] |  |  |  +- commons-configuration:commons-configuration:jar:1.6:compile
[INFO] |  |  |  |  +- commons-digester:commons-digester:jar:1.8:compile
[INFO] |  |  |  |  |  \- commons-beanutils:commons-beanutils:jar:1.7.0:compile
[INFO] |  |  |  |  \- commons-beanutils:commons-beanutils-core:jar:1.8.0:compile
[INFO] |  |  |  +- org.slf4j:slf4j-log4j12:jar:1.7.5:compile
[INFO] |  |  |  +- org.apache.avro:avro:jar:1.7.6-cdh5.2.1:compile
[INFO] |  |  |  |  \- org.xerial.snappy:snappy-java:jar:1.0.5:compile
[INFO] |  |  |  +- com.google.code.gson:gson:jar:2.2.4:compile
[INFO] |  |  |  +- org.apache.curator:curator-client:jar:2.6.0:compile
[INFO] |  |  |  \- org.apache.curator:curator-recipes:jar:2.6.0:compile
[INFO] |  |  +- org.apache.hadoop:hadoop-hdfs:jar:2.5.0-cdh5.2.1:compile
[INFO] |  |  |  \- io.netty:netty:jar:3.6.2.Final:compile
[INFO] |  |  +- org.apache.hadoop:hadoop-mapreduce-client-app:jar:2.5.0-cdh5.2.1:compile
[INFO] |  |  |  +- org.apache.hadoop:hadoop-mapreduce-client-common:jar:2.5.0-cdh5.2.1:compile
[INFO] |  |  |  |  \- org.apache.hadoop:hadoop-yarn-server-common:jar:2.5.0-cdh5.2.1:compile
[INFO] |  |  |  \- org.apache.hadoop:hadoop-mapreduce-client-shuffle:jar:2.5.0-cdh5.2.1:compile
[INFO] |  |  |     \- org.fusesource.leveldbjni:leveldbjni-all:jar:1.8:compile
[INFO] |  |  +- org.apache.hadoop:hadoop-mapreduce-client-core:jar:2.5.0-cdh5.2.1:compile
[INFO] |  |  \- org.apache.hadoop:hadoop-mapreduce-client-jobclient:jar:2.5.0-cdh5.2.1:compile
[INFO] |  +- org.apache.hbase:hbase-server:jar:0.98.6-cdh5.2.1:compile
[INFO] |  |  +- org.apache.hbase:hbase-common:jar:0.98.6-cdh5.2.1:compile
[INFO] |  |  +- org.apache.hbase:hbase-protocol:jar:0.98.6-cdh5.2.1:compile
[INFO] |  |  +- org.apache.hbase:hbase-client:jar:0.98.6-cdh5.2.1:compile
[INFO] |  |  +- org.apache.hbase:hbase-prefix-tree:jar:0.98.6-cdh5.2.1:runtime
[INFO] |  |  |  \- org.apache.hbase:hbase-common:jar:tests:0.98.6-cdh5.2.1:runtime
[INFO] |  |  +- commons-httpclient:commons-httpclient:jar:3.1:compile
[INFO] |  |  +- commons-collections:commons-collections:jar:3.2.1:compile
[INFO] |  |  +- org.apache.hbase:hbase-hadoop-compat:jar:0.98.6-cdh5.2.1:compile
[INFO] |  |  +- org.apache.hbase:hbase-hadoop2-compat:jar:0.98.6-cdh5.2.1:compile
[INFO] |  |  +- com.yammer.metrics:metrics-core:jar:2.2.0:compile
[INFO] |  |  +- com.github.stephenc.high-scale-lib:high-scale-lib:jar:1.1.1:compile
[INFO] |  |  +- org.apache.commons:commons-math:jar:2.1:compile
[INFO] |  |  +- org.apache.zookeeper:zookeeper:jar:3.4.5-cdh5.2.1:compile
[INFO] |  |  +- org.mortbay.jetty:jetty:jar:6.1.26.cloudera.4:compile
[INFO] |  |  +- org.mortbay.jetty:jetty-sslengine:jar:6.1.26.cloudera.4:compile
[INFO] |  |  +- org.mortbay.jetty:jsp-2.1:jar:6.1.14:compile
[INFO] |  |  +- org.mortbay.jetty:jsp-api-2.1:jar:6.1.14:compile
[INFO] |  |  +- tomcat:jasper-compiler:jar:5.5.23:compile
[INFO] |  |  +- tomcat:jasper-runtime:jar:5.5.23:compile
[INFO] |  |  |  \- commons-el:commons-el:jar:1.0:compile
[INFO] |  |  +- org.jamon:jamon-runtime:jar:2.3.1:compile
[INFO] |  |  +- org.cloudera.htrace:htrace-core:jar:2.04:compile
[INFO] |  |  +- org.apache.hadoop:hadoop-auth:jar:2.5.0-cdh5.2.1:compile
[INFO] |  |  |  +- org.apache.httpcomponents:httpclient:jar:4.2.5:compile
[INFO] |  |  |  |  \- org.apache.httpcomponents:httpcore:jar:4.2.4:compile
[INFO] |  |  |  +- org.apache.directory.server:apacheds-kerberos-codec:jar:2.0.0-M15:compile
[INFO] |  |  |  |  +- org.apache.directory.server:apacheds-i18n:jar:2.0.0-M15:compile
[INFO] |  |  |  |  +- org.apache.directory.api:api-asn1-api:jar:1.0.0-M20:compile
[INFO] |  |  |  |  \- org.apache.directory.api:api-util:jar:1.0.0-M20:compile
[INFO] |  |  |  \- org.apache.curator:curator-framework:jar:2.6.0:compile
[INFO] |  |  +- org.apache.hadoop:hadoop-core:jar:2.5.0-mr1-cdh5.2.1:compile
[INFO] |  |  |  +- javax.servlet.jsp:jsp-api:jar:2.1:compile
[INFO] |  |  |  +- net.java.dev.jets3t:jets3t:jar:0.6.1:compile
[INFO] |  |  |  +- hsqldb:hsqldb:jar:1.8.0.10:compile
[INFO] |  |  |  \- org.eclipse.jdt:core:jar:3.1.1:compile
[INFO] |  |  +- org.apache.hadoop:hadoop-hdfs:test-jar:tests:2.5.0-cdh5.2.1:compile
[INFO] |  |  |  \- commons-daemon:commons-daemon:jar:1.0.13:compile
[INFO] |  |  +- com.github.stephenc.findbugs:findbugs-annotations:jar:1.3.9-1:compile
[INFO] |  |  \- junit:junit:jar:4.11:compile
[INFO] |  |     \- org.hamcrest:hamcrest-core:jar:1.3:compile
[INFO] |  +- com.jcraft:jsch:jar:0.1.51:compile
[INFO] |  +- com.twitter.elephantbird:elephant-bird-core:jar:4.1:compile
[INFO] |  |  \- com.googlecode.json-simple:json-simple:jar:1.1:compile
[INFO] |  \- com.twitter.elephantbird:elephant-bird-hadoop-compat:jar:4.1:compile
[INFO] +- com.typesafe:config:jar:1.2.1:compile
[INFO] +- org.json4s:json4s-jackson_2.10:jar:3.3.0.RC1:compile
[INFO] |  \- org.json4s:json4s-core_2.10:jar:3.3.0.RC1:compile
[INFO] |     +- org.json4s:json4s-ast_2.10:jar:3.3.0.RC1:compile
[INFO] |     \- org.json4s:json4s-scalap_2.10:jar:3.3.0.RC1:compile
**[INFO] \- com.fasterxml.jackson.core:jackson-databind:jar:2.4.2:compile
[INFO]    +- com.fasterxml.jackson.core:jackson-annotations:jar:2.4.0:compile
[INFO]    \- com.fasterxml.jackson.core:jackson-core:jar:2.4.2:compile**

2 个答案:

答案 0 :(得分：1)

NoSuchMethodError始终是不兼容的jar文件的标志。在您的情况下，我会说它介于jackson.module.scala和jackson.databind

之间

请切换到两个文件的相同版本。

答案 1 :(得分：0)

Hadoop通常在类路径上有许多jars的版本。您可以尝试设置类路径，通过将mapreduce.job.user.classpath.first设置为true来覆盖此类路径。或者，您可以降级项目以使用您的hadoop部署使用的相同版本的jackson。

非孤立的类路径可能是最大的hadoop痛点。

相关问题
最新问题