Question

我正在使用Horton Works Cluster（2节点集群）来运行火花和水槽，所以当我使用--master＆＃34; local [*]＆＃34; ，Flume能够发送事件，Spark也可以在localhost接收和检查：4040我可以看到从水槽收到的事件。（我们使用flume-ng-sql源从水槽中抽取100个事件/秒，每个源大小约为1KB）

当我用--master＆＃34; yarn-client＆＃34;运行相同的例子时，我在水槽中得到以下错误，火花也没有发生任何事件。

2015-08-13 18:24:24,927 (SinkRunner-PollingRunner-DefaultSinkProcessor) [ERROR - org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:160)] Unable to deliver event. Exception follows.
org.apache.flume.EventDeliveryException: Failed to send events
        at org.apache.flume.sink.AbstractRpcSink.process(AbstractRpcSink.java:403)
        at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
        at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
        at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.flume.FlumeException: NettyAvroRpcClient { host: localhost, port: 55555 }: RPC connection error
        at org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:182)
        at org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:121)
        at org.apache.flume.api.NettyAvroRpcClient.configure(NettyAvroRpcClient.java:638)
        at org.apache.flume.api.RpcClientFactory.getInstance(RpcClientFactory.java:88)
        at org.apache.flume.sink.AvroSink.initializeRpcClient(AvroSink.java:127)
        at org.apache.flume.sink.AbstractRpcSink.createConnection(AbstractRpcSink.java:222)
        at org.apache.flume.sink.AbstractRpcSink.verifyConnection(AbstractRpcSink.java:283)
        at org.apache.flume.sink.AbstractRpcSink.process(AbstractRpcSink.java:360)
        ... 3 more
Caused by: java.io.IOException: Error connecting to localhost/127.0.0.1:55555
        at org.apache.avro.ipc.NettyTransceiver.getChannel(NettyTransceiver.java:261)
        at org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:203)
        at org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:152)
        at org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:168)
        ... 10 more
Caused by: java.net.ConnectException: Connection refused
        at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
        at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:744)
        at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.connect(NioClientSocketPipelineSink.java:496)
        at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.processSelectedKeys(NioClientSocketPipelineSink.java:452)
        at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:365)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        ... 1 more
^

在群集中也观察到以下观察： - 使用纱线的内存消耗量远高于本地使用的内存消耗量。

- 当我每30秒抽100次活动时，Flume和spark能够使用纱线客户端和本地连接和处理相同的事件。

以下是我用于水槽和火花的命令。

水槽：

sudo -u hdfs flume-ng agent --conf conf/ -f conf/flume_mysql_spark.conf -n agent1 -Dflume.root.logger=INFO,console > flumelog.txt

火花：

sudo -u hdfs spark-submit --master "yarn-client" --class "org.paladion.atm.FlumeEventCount" target/atm-1.1-jar-with-dependencies.jar > sparklog.txt

sudo -u hdfs spark-submit --master "local[*]" --class "org.paladion.atm.FlumeEventCount" target/atm-1.1-jar-with-dependencies.jar > sparklog.txt

请...我知道这可能有什么问题吗？

Answer 1

它解决了如下：

1 - 如果以本地方式运行在Flume中提供本地机器的IP以及火花。

2 - 如果作为集群（yarn-client或yarn-cluster）运行，请在集群中提供要发送事件的机器的IP（除了执行程序的那个之外，因此可能会给出节点的IP）它不是Flume中的主节点机器以及火花。

让我知道如果我错了，这可能会因其他原因而有效，任何更好的解决方案也是如此。

使用yarn-client

1 个答案: