Flume Configuration

Naming the Components

agent_name.sources = source_name
agent_name.sinks = sink_name
agent_name.channels = channel_name

Sources

  • Avro Source

  • Thrift Source

  • Exec Source

  • JMS Source

  • Spooling Directory Source

  • Twitter 1% firehose Source

  • Kafka Source

  • NetCat Source

  • Sequence Generator Source

  • Syslog Sources

  • Syslog TCP Source

  • Multiport Syslog TCP Source

  • Syslog UDP Source

  • HTTP Source

  • Stress Source

  • Legacy Sources

  • Thrift Legacy Source

  • Custom Source

  • Scribe Source

Channels

  • Memory Channel

  • JDBC Channel

  • Kafka Channel

  • File Channel

  • Spillable Memory Channel

  • Pseudo Transaction Channel

Sinks

  • HDFS Sink

  • Hive Sink

  • Logger Sink

  • Avro Sink

  • Thrift Sink

  • IRC Sink

  • File Roll Sink

  • Null Sink

  • HBaseSink

  • AsyncHBaseSink

  • MorphlineSolrSink

  • ElasticSearchSink

  • Kite Dataset Sink

  • Kafka Sink

TwitterAgent.sources = Twitter
TwitterAgent.channels = MemChannel
TwitterAgent.sinks = HDFS

Describing the Source

agent_name.sources.source_name.type = value
agent_name.sources.source_name.property2 = value
agent_name.sources.source_name.property3 = value

TwitterAgent.sources.Twitter.type = Twitter (type name)
TwitterAgent.sources.Twitter.consumerKey =
TwitterAgent.sources.Twitter.consumerSecret =
TwitterAgent.sources.Twitter.accessToken =
TwitterAgent.sources.Twitter.accessTokenSecret =

Describing the Sink

agent_name.sinks.sink_name.type = value
agent_name.sinks.sink_name.property2 = value
agent_name.sinks.sink_name.property3 = value

TwitterAgent.sinks.HDFS.type = hdfs (type name)
TwitterAgent.sinks.HDFS.hdfs.path = HDFS directory’s Path to store the data

Describing the Channel

agent_name.channels.channel_name.type = value
agent_name.channels.channel_name.property2 = value
agent_name.channels.channel_name.property3 = value

TwitterAgent.channels.MemChannel.type = memory (type name)

Binding the Source and the Sink to the Channel

agent_name.sources.source_name.channels = channel_name
agent_name.sinks.sink_name.channels = channel_name

TwitterAgent.sources.Twitter.channels = MemChannel
TwitterAgent.sinks.HDFS.channels = MemChannel

Starting a Flume Agent

$ bin/flume-ng agent --conf ./conf/ -f conf/twitter.conf
Dflume.root.logger=DEBUG,console -n TwitterAgent

  • agent − Command to start the Flume agent

  • –conf ,-c − Use configuration file in the conf directory

  • -f − Specifies a config file path, if missing

  • –name, -n − Name of the twitter agent

  • -D property =value − Sets a Java system property value.

References