Hadoop

• Enviornment Setup
  • Creating a User
  • SSH Setup and Key Generation
  • Downloading Hadoop
  • Installing Hadoop in Standalone Mode
    • Setting Up Hadoop
    • Example
  • Installing Hadoop in Pseudo Distributed Mode
  • Verifying Hadoop Installation
  • References
• HDFS
  • Features of HDFS
  • Where to use HDFS
  • Where not to use HDFS
  • HDFS Architecture
  • Starting HDFS
  • Listing Files in HDFS
  • Inserting Data into HDFS
  • Retrieving Data from HDFS
  • Shutting Down the HDFS
  • HDFS Basic File Operations
  • HDFS Other commands
  • References
• Command Reference
  • References
• MapReduce
  • The Algorithm
  • Inputs and Outputs (Java Perspective)
  • Terminology
  • Example Scenario
    • Input Data
    • Example Program
  • Compilation and Execution of Process Units Program
  • Important Commands
  • How to Interact with MapReduce Jobs
    • To see the status of job
    • To see the history of job output-dir
    • To kill the job
  • Word Count Example
    • Steps to execute MapReduce word count example
  • MapReduce Char Count Example
    • Steps to execute MapReduce char count example
  • References
• Streaming
  • Hadoop Streaming
  • Streaming Command Options
    • Specifying a Java Class as the Mapper/Reducer
  • References
• Multi-Node Cluster
  • Installing Java
  • Creating User Account
  • Mapping the nodes
  • Configuring Key Based Login
  • Installing Hadoop
  • Configuring Hadoop
    • core-site.xml
    • hdfs-site.xml
    • mapred-site.xml
    • yarn-site.xml
    • hadoop-env.sh
  • Installing Hadoop on Slave Servers
  • Configuring Hadoop on Master Server
    • Configuring Master Node
    • Configuring Slave Node
    • Format Name Node on Hadoop Master
  • Starting Hadoop Services
  • Results
    • Master
    • Slave
  • Adding a New DataNode in the Hadoop Cluster
    • Networking
  • Adding User and SSH Access
    • Add a User
    • Execute the following on the master
    • Execute the following on the slaves
  • Set Hostname of New Node
  • Start the DataNode on New Node
    • Login to new node
    • Start HDFS on a newly added slave node by using the following command
    • Check the output of jps command on a new node. It looks as follows.
  • Removing a DataNode from the Hadoop Cluster
    • Results
  • References
• YARN
  • Architecture
  • Components Of YARN
  • Benefits of YARN
  • References
• Azkaban Docker
  • Build
  • Installing the Solo Server
  • Installing Azkaban Executor Server
  • Installing Azkaban Web Server
  • Docker Compose
  • Screenshots
  • References