我尝试在HDFS中播放Flume的推文,我没有收到来自Twitter的推文。这个过程在这一点上停止了...... !!没有超越这一步。
INFO instrumentation.MonitoredCounterGroup: Monitoried counter group for type: SINK, name: HDFS, registered successfully.
INFO instrumentation.MonitoredCounterGroup: Component type: SINK, name: HDFS started
我的flume.conf文件:
TwitterAgent.sources = Twitter
TwitterAgent.channels = MemChannel
TwitterAgent.sinks = HDFS
TwitterAgent.sources.Twitter.type = org.apache.flume.source.twitter.TwitterSource
TwitterAgent.sources.Twitter.channels = MemChannel
TwitterAgent.sources.Twitter.consumerKey = **************
TwitterAgent.sources.Twitter.consumerSecret = *********
TwitterAgent.sources.Twitter.accessToken =************
TwitterAgent.sources.Twitter.accessTokenSecret = *********
TwitterAgent.sources.Twitter.keywords = big data,cloud,hadoop,data analysis,hive,pig,hbase,flume,datawarehouse
TwitterAgent.sinks.HDFS.channel = MemChannel
TwitterAgent.sinks.HDFS.type = hdfs
TwitterAgent.sinks.HDFS.hdfs.path = hdfs://localhost:9000/Big
TwitterAgent.sinks.HDFS.hdfs.fileType = DataStream
TwitterAgent.sinks.HDFS.hdfs.writeFormat = Text
TwitterAgent.sinks.HDFS.hdfs.batchSize = 1000
TwitterAgent.sinks.HDFS.hdfs.rollSize = 0
TwitterAgent.sinks.HDFS.hdfs.rollCount = 10000
TwitterAgent.channels.MemChannel.type = memory
TwitterAgent.channels.MemChannel.capacity = 10000
TwitterAgent.channels.MemChannel.transactionCapacity = 100
答案 0 :(得分:1)
试试这段代码..
TwitterAgent.sources = Twitter
TwitterAgent.channels = MemChannel
TwitterAgent.sinks = HDFS
TwitterAgent.sources.Twitter.type = com.cloudera.flume.source.TwitterSource
TwitterAgent.sources.Twitter.channels = MemChannel
TwitterAgent.sources.Twitter.consumerKey = xxxxxxxxxxxxxxx
TwitterAgent.sources.Twitter.consumerSecret = xxxxxxxxxxxx
TwitterAgent.sources.Twitter.accessToken = xxxxxxxxxxxxxxx
TwitterAgent.sources.Twitter.accessTokenSecret = xxxxxxxxx
TwitterAgent.sources.Twitter.keywords = big data,cloud,hadoop,data analysis,hive,pig,hbase,flume,datawarehouse
TwitterAgent.sinks.HDFS.channel = MemChannel
TwitterAgent.sinks.HDFS.type = hdfs
TwitterAgent.sinks.HDFS.hdfs.path = hdfs://localhost:9000/Big
TwitterAgent.sinks.HDFS.hdfs.fileType = DataStream
TwitterAgent.sinks.HDFS.hdfs.writeFormat = Text
TwitterAgent.sinks.HDFS.hdfs.batchSize = 1000
TwitterAgent.sinks.HDFS.hdfs.rollSize = 0
TwitterAgent.sinks.HDFS.hdfs.rollCount = 10000
TwitterAgent.sinks.HDFS.hdfs.filePrefix = twitter_India-
TwitterAgent.sinks.HDFS.hdfs.round = true
TwitterAgent.sinks.HDFS.hdfs.roundValue = 2
TwitterAgent.sinks.HDFS.hdfs.roundUnit = minute
TwitterAgent.channels.MemChannel.type = memory
TwitterAgent.channels.MemChannel.capacity = 10000
TwitterAgent.channels.MemChannel.transactionCapacity = 100

答案 1 :(得分:0)
在flume.conf
档案
TwitterAgent.sinks.HDFS.hdfs.path = hdfs://localhost:**9000**/Big
将您自己的Active localhost instade设置为9000。
转到http://localhost:50070/,您可以看到您的活动本地主机。就像在我的情况下它是
TwitterAgent.sinks.HDFS.hdfs.path=hdfs://localhost:54310/user/flume/tweets