Flume Configuration¶

Naming the Components¶

agent_name.sources = source_name
agent_name.sinks = sink_name
agent_name.channels = channel_name

Sources

  • Avro Source
  • Thrift Source
  • Exec Source
  • JMS Source
  • Spooling Directory Source
  • Twitter 1% firehose Source
  • Kafka Source
  • NetCat Source
  • Sequence Generator Source
  • Syslog Sources
  • Syslog TCP Source
  • Multiport Syslog TCP Source
  • Syslog UDP Source
  • HTTP Source
  • Stress Source
  • Legacy Sources
  • Thrift Legacy Source
  • Custom Source
  • Scribe Source

Channels

  • Memory Channel
  • JDBC Channel
  • Kafka Channel
  • File Channel
  • Spillable Memory Channel
  • Pseudo Transaction Channel

Sinks

  • HDFS Sink
  • Hive Sink
  • Logger Sink
  • Avro Sink
  • Thrift Sink
  • IRC Sink
  • File Roll Sink
  • Null Sink
  • HBaseSink
  • AsyncHBaseSink
  • MorphlineSolrSink
  • ElasticSearchSink
  • Kite Dataset Sink
  • Kafka Sink
TwitterAgent.sources = Twitter
TwitterAgent.channels = MemChannel
TwitterAgent.sinks = HDFS

Describing the Source¶

agent_name.sources.source_name.type = value
agent_name.sources.source_name.property2 = value
agent_name.sources.source_name.property3 = value

TwitterAgent.sources.Twitter.type = Twitter (type name)
TwitterAgent.sources.Twitter.consumerKey =
TwitterAgent.sources.Twitter.consumerSecret =
TwitterAgent.sources.Twitter.accessToken =
TwitterAgent.sources.Twitter.accessTokenSecret =

Describing the Sink¶

agent_name.sinks.sink_name.type = value
agent_name.sinks.sink_name.property2 = value
agent_name.sinks.sink_name.property3 = value

TwitterAgent.sinks.HDFS.type = hdfs (type name)
TwitterAgent.sinks.HDFS.hdfs.path = HDFS directory’s Path to store the data

Describing the Channel¶

agent_name.channels.channel_name.type = value
agent_name.channels.channel_name.property2 = value
agent_name.channels.channel_name.property3 = value

TwitterAgent.channels.MemChannel.type = memory (type name)

Binding the Source and the Sink to the Channel¶

agent_name.sources.source_name.channels = channel_name
agent_name.sinks.sink_name.channels = channel_name

TwitterAgent.sources.Twitter.channels = MemChannel
TwitterAgent.sinks.HDFS.channels = MemChannel

Starting a Flume Agent¶

$ bin/flume-ng agent --conf ./conf/ -f conf/twitter.conf
Dflume.root.logger=DEBUG,console -n TwitterAgent
  • agent − Command to start the Flume agent
  • –conf ,-c − Use configuration file in the conf directory
  • -f − Specifies a config file path, if missing
  • –name, -n − Name of the twitter agent
  • -D property =value − Sets a Java system property value.