Flume Configuration¶
Naming the Components¶
agent_name.sources = source_name
agent_name.sinks = sink_name
agent_name.channels = channel_name
Sources
- Avro Source
- Thrift Source
- Exec Source
- JMS Source
- Spooling Directory Source
- Twitter 1% firehose Source
- Kafka Source
- NetCat Source
- Sequence Generator Source
- Syslog Sources
- Syslog TCP Source
- Multiport Syslog TCP Source
- Syslog UDP Source
- HTTP Source
- Stress Source
- Legacy Sources
- Thrift Legacy Source
- Custom Source
- Scribe Source
Channels
- Memory Channel
- JDBC Channel
- Kafka Channel
- File Channel
- Spillable Memory Channel
- Pseudo Transaction Channel
Sinks
- HDFS Sink
- Hive Sink
- Logger Sink
- Avro Sink
- Thrift Sink
- IRC Sink
- File Roll Sink
- Null Sink
- HBaseSink
- AsyncHBaseSink
- MorphlineSolrSink
- ElasticSearchSink
- Kite Dataset Sink
- Kafka Sink
TwitterAgent.sources = Twitter
TwitterAgent.channels = MemChannel
TwitterAgent.sinks = HDFS
Describing the Source¶
agent_name.sources.source_name.type = value
agent_name.sources.source_name.property2 = value
agent_name.sources.source_name.property3 = value
TwitterAgent.sources.Twitter.type = Twitter (type name)
TwitterAgent.sources.Twitter.consumerKey =
TwitterAgent.sources.Twitter.consumerSecret =
TwitterAgent.sources.Twitter.accessToken =
TwitterAgent.sources.Twitter.accessTokenSecret =
Describing the Sink¶
agent_name.sinks.sink_name.type = value
agent_name.sinks.sink_name.property2 = value
agent_name.sinks.sink_name.property3 = value
TwitterAgent.sinks.HDFS.type = hdfs (type name)
TwitterAgent.sinks.HDFS.hdfs.path = HDFS directory’s Path to store the data
Describing the Channel¶
agent_name.channels.channel_name.type = value
agent_name.channels.channel_name.property2 = value
agent_name.channels.channel_name.property3 = value
TwitterAgent.channels.MemChannel.type = memory (type name)
Binding the Source and the Sink to the Channel¶
agent_name.sources.source_name.channels = channel_name
agent_name.sinks.sink_name.channels = channel_name
TwitterAgent.sources.Twitter.channels = MemChannel
TwitterAgent.sinks.HDFS.channels = MemChannel
Starting a Flume Agent¶
$ bin/flume-ng agent --conf ./conf/ -f conf/twitter.conf
Dflume.root.logger=DEBUG,console -n TwitterAgent
- agent − Command to start the Flume agent
- –conf ,-c
− Use configuration file in the conf directory - -f
− Specifies a config file path, if missing - –name, -n
− Name of the twitter agent - -D property =value − Sets a Java system property value.