从Web应用程序(servlet)在远程Hadoop集群上提交Hadoop作业

时间:2015-02-18 11:24:01

标签: servlets hadoop

我已经配置了一个带有2个Linux机器节点的Hadoop集群,我在第三个Linux机器上运行了Tomcat上的servlet应用程序。我的目标是从Web应用程序提交Hadoop。当我从Eclipse提交作业时,我可以远程提交Hadoop上的作业。在servlet的情况下我得到以下异常。请建议

'java.net.ConnectException:拒绝连接:没有进一步的信息;有关详细信息,请参阅:http://wiki.apache.org/hadoop/ConnectionRefused      at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)      at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)      at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)      at java.lang.reflect.Constructor.newInstance(Constructor.java:513)      在org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:783)      在org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:730)      在org.apache.hadoop.ipc.Client.call(Client.java:1414)      在org.apache.hadoop.ipc.Client.call(Client.java:1363)      在org.apache.hadoop.ipc.WritableRpcEngine $ Invoker.invoke(WritableRpcEngine.java:231)      在$ Proxy11.getSystemDir(未知来源)      在org.apache.hadoop.mapreduce.Cluster $ 1.run(Cluster.java:141)      在org.apache.hadoop.mapreduce.Cluster $ 1.run(Cluster.java:139)      at java.security.AccessController.doPrivileged(Native Method)      在javax.security.auth.Subject.doAs(Subject.java:396)      在org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)      在org.apache.hadoop.mapreduce.Cluster.getFileSystem(Cluster.java:139)      在org.apache.hadoop.mapreduce.Job.submit(Job.java:1040)      在org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1062)      在servlets.JobDriver.doProcess(JobDriver.java:112)      在servlets.JobDriver.doGet(JobDriver.java:42)      在javax.servlet.http.HttpServlet.service(HttpServlet.java:620)      在javax.servlet.http.HttpServlet.service(HttpServlet.java:727)      在org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)      在org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)      在org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220)      在org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122)      在org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:504)      在org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:170)      在org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)      在org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:950)      在org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)      在org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:421)      在org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1074)      at org.apache.coyote.AbstractProtocol $ AbstractConnectionHandler.process(AbstractProtocol.java:611)      at org.apache.tomcat.util.net.JIoEndpoint $ SocketProcessor.run(JIoEndpoint.java:314)      at java.util.concurrent.ThreadPoolExecutor $ Worker.runTask(ThreadPoolExecutor.java:886)      at java.util.concurrent.ThreadPoolExecutor $ Worker.run(ThreadPoolExecutor.java:908)      at org.apache.tomcat.util.threads.TaskThread $ WrappingRunnable.run(TaskThread.java:61)      在java.lang.Thread.run(Thread.java:662)'

1 个答案:

答案 0 :(得分:0)

从日志中看来,提交作业的servlet中的用户无权将作业提交到集群:

$Proxy11.getSystemDir(Unknown Source) at 
org.apache.hadoop.mapreduce.Cluster$1.run(Cluster.java:141) at 
org.apache.hadoop.mapreduce.Cluster$1.run(Cluster.java:139) at 
java.security.AccessController.doPrivileged(Native Method) at 
javax.security.auth.Subject.doAs(Subject.java:396) at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548) at 
org.apache.hadoop.mapreduce.Cluster.getFileSystem(Cluster.java:139) at  
org.apache.hadoop.mapreduce.Job.submit(Job.java:1040) at  

通过检查servlet会话确保只有admin组从servlet提交。