Hadoop Tutorial for Beginners

Introduction

Struggling in your career or finding it hard to get well-paying Big Data positions? The market needs experts badly to manage enormous amounts of data! Hadoop is the core, sought-after platform that opens up such possibilities. Learn distributed storage and processing to become a much-sought-after Big Data professional through this Hadoop tutorial for beginners! Ready to increase your skills and income? Download our complete Hadoop Course Syllabus today!

Why Students or Freshers Learn Hadoop?

Hadoop is learned mainly by students and freshers to gain entry into the Big Data arena because of the high demand in the market.

• Building Blocks for Big Data: Hadoop is a fundamental platform for processing and storing enormous volumes of data, and thus, it is a must-have gateway to the Big Data environment (Spark, Hive, etc.).
• Job Market Demand: There is high and increasing demand for Hadoop/Big Data professionals with skills across industries such as finance, retail, and technology.
• Varied Career Opportunities: It leads to well-paying positions such as Data Scientist, Big Data Engineer, and Hadoop Developer.
• Scalability and Cost: Having knowledge of Hadoop’s distributed, fault-tolerant, and cost-effective design is a very sought-after technical capability.

Now ready to begin your Big Data career? Look at our Hadoop Interview Questions and Answers for freshers to crack your first interview!

Step-by-Step Hadoop Tutorial for Beginners

Apache Hadoop is an open-source distributed storage and processing framework for huge data sets on a cluster of commodity hardware. It supports the cost-effective, scalable, and fault-tolerant processing of Big Data. This step-by-step tutorial is for a single-node pseudo-distributed setup mode, which is suitable for learning and testing purposes by beginners.

Step 1. Hadoop Installation and Setup

Hadoop needs Java Development Kit (JDK) and Secure Shell (SSH) to run. The steps below take a Linux environment (such as Ubuntu) for the installation.

1.1 Prerequisites Installation

Install Java: Hadoop can generally work fine with Java 8.

# Update package list

sudo apt update

# Install OpenJDK 8

sudo apt install openjdk-8-jdk -y

# Verify installation

java -version

Install SSH and configure Passwordless SSH: SSH will be required to control nodes in the cluster (even a single node for pseudo-distributed mode).

# Install OpenSSH server

sudo apt install openssh-server -y

# Generate SSH keypair (use default location and no passphrase)

ssh-keygen -t rsa -P ” -f ~/.ssh/id_rsa

# Add the public key to authorized keys

cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys

# Test SSH to localhost (first time will ask for confirmation, type ‘yes’)

ssh localhost

1.2 Hadoop Download and Environment Configuration

Download and Unzip Hadoop: Download a reliable version of Hadoop (e.g., 3.3.6) and unzip it.

# Download Hadoop (replace URL and version as needed)

wget https://dlcdn.apache.org/hadoop/common/hadoop-3.3.6/hadoop-3.3.6.tar.gz

# Extract the file

tar -xzf hadoop-3.3.6.tar.gz

# Optionally rename for simplicity

mv hadoop-3.3.6 hadoop

Note: Rename the hadoop folder to a handy place, such as /usr/local/ (needs sudo and ownership modification) or your home directory (~/). For this tutorial, let’s say it’s in your home directory (~/hadoop).

Configure Environment Variables: Modify your shell setup file (e.g., ~/.bashrc) to define JAVA_HOME and HADOOP_HOME.

nano ~/.bashrc

Add the following lines (modify paths if needed, especially for JAVA_HOME):

# Java Home

export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64

# Hadoop Home

export HADOOP_HOME=~/hadoop

export HADOOP_INSTALL=$HADOOP_HOME

# Add Hadoop binary and sbin directories to PATH

export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin

# For HDFS/YARN specific components

export HADOOP_MAPRED_HOME=$HADOOP_HOME

export HADOOP_COMMON_HOME=$HADOOP_HOME

export HADOOP_HDFS_HOME=$HADOOP_HOME

export YARN_HOME=$HADOOP_HOME

export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native

export HADOOP_OPTS=”-Djava.library.path=$HADOOP_HOME/lib/native”

Save and close the file, then save changes:

source ~/.bashrc

1.3 Hadoop Configuration (Pseudo-Distributed Mode)

Define core properties for running Hadoop in pseudo-distributed mode. Configuration files are found in $HADOOP_HOME/etc/hadoop/.

Set JAVA_HOME in hadoop-env.sh:

nano $HADOOP_HOME/etc/hadoop/hadoop-env.sh

Locate the line beginning with export JAVA_HOME= and modify it to reference your Java install:

export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64

Configure core-site.xml: 

Specifies the NameNode URI of the Hadoop cluster.

nano $HADOOP_HOME/etc/hadoop/core-site.xml

Incude the following configuration between the <configuration> tags.

<configuration>

    <property>

        <name>fs.defaultFS</name>

        <value>hdfs://localhost:9000</value>

    </property>

</configuration>

Configure hdfs-site.xml: 

Specifies the NameNode and DataNode directories, and the replication factor (set to 1 for single-node).

nano $HADOOP_HOME/etc/hadoop/hdfs-site.xml

Add the following configuration:

<configuration>

    <property>

        <name>dfs.replication</name>

        <value>1</value>

    </property>

    <property>

        <name>dfs.namenode.name.dir</name>

        <value>file:///home/hadoop/hadoopdata/namenode</value>

    </property>

    <property>

        <name>dfs.datanode.data.dir</name>

        <value>file:///home/hadoop/hadoopdata/datanode</value>

    </property>

</configuration>

Note: Substitute /home/hadoop/ with your own home directory path if necessary.

Create the Data Directories: 

Create the directories defined in hdfs-site.xml.

mkdir -p ~/hadoopdata/namenode

mkdir -p ~/hadoopdata/datanode

Configure mapred-site.xml: 

Specifies the MapReduce framework to use YARN. First, copy the template file:

cp$HADOOP_HOME/etc/hadoop/mapred-site.xml.template $HADOOP_HOME/etc/hadoop/mapred-site.xml

nano $HADOOP_HOME/etc/hadoop/mapred-site.xml

Add the following configuration:

<configuration>

    <property>

        <name>mapreduce.framework.name</name>

        <value>yarn</value>

    </property>

</configuration>

Configure yarn-site.xml: 

Specifies the YARN resource manager and node manager.

nano $HADOOP_HOME/etc/hadoop/yarn-site.xml

Add the following configuration:

<configuration>

    <property>

        <name>yarn.nodemanager.aux-services</name>

        <value>mapreduce_shuffle</value>

    </property>

    <property>

        <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>

        <value>org.apache.hadoop.mapred.ShuffleHandler</value>

    </property>

    <property>

        <name>yarn.resourcemanager.hostname</name>

        <value>localhost</value>

    </property>

</configuration>

1.4 Start Hadoop

Format the NameNode: This initializes the HDFS. Only do this once for a fresh install.

hdfs namenode -format

Start daemons of HDFS and YARN:

start-dfs.sh  # Starts NameNode and DataNode

start-yarn.sh # Starts ResourceManager and NodeManager

Check Services: Use the jps command to verify if the necessary Hadoop daemons are running. NameNode, DataNode, ResourceManager, and NodeManager should be present.

jps

Step 2. Basic HDFS Operations

Hadoop Distributed File System (HDFS) is the main storage system. You access it with the hdfs dfs command.

Command	Description	Example
mkdir	Creates a directory in HDFS.	hdfs dfs -mkdir /input_data
put	Copies files from the local filesystem to HDFS.	hdfs dfs -put local_file.txt /input_data
ls	Lists contents of a directory in HDFS.	hdfs dfs -ls /input_data
cat	Displays the content of a file in HDFS.	hdfs dfs -cat /input_data/local_file.txt
get	Copies files from HDFS to the local filesystem.	hdfs dfs -get /output_data/part-r-00000 local_output
rm	Deletes a file in HDFS.	hdfs dfs -rm /input_data/local_file.txt
rm -r	Recursively deletes a directory and its contents.	hdfs dfs -rm -r /output_data

Step 3. Running a MapReduce Job (WordCount Example)

MapReduce is the parallel data processing programming model. Hadoop provides a few example jobs. We will utilize the traditional WordCount task.

3.1 Get Ready with Input Data

Create a sample text file in your local machine (e.g., sample.txt).

echo “Hadoop is great” > sample.txt

echo “Great big data” >> sample.txt

echo “Hadoop Big Data” >> sample.txt

Create an input directory in HDFS and transfer the file into it.

hdfs dfs -mkdir /wordcount_input

hdfs dfs -put sample.txt /wordcount_input

hdfs dfs -ls /wordcount_input

3.2 Run the MapReduce Job

Hadoop comes with a JAR file containing examples bundled in it. You’ll execute the wordcount job from this JAR. The command takes the examples JAR path, the name of the job (wordcount), the HDFS input path, and the HDFS output path.

# Find the Hadoop examples JAR file (path may vary slightly)

HADOOP_EXAMPLES_JAR=$(find $HADOOP_HOME -name “hadoop-mapreduce-examples-*.jar”)

# Run the WordCount job

# The output directory (/wordcount_output) must NOT exist before running the job

hadoop jar $HADOOP_EXAMPLES_JAR wordcount /wordcount_input /wordcount_output

3.3 Verify the Output

List the output directory in HDFS. There should be a _SUCCESS file and possibly one or more part-r-xxxxx files (the output).

hdfs dfs -ls /wordcount_output

Check the result file (e.g., part-r-00000).

hdfs dfs -cat /wordcount_output/part-r-00000

The result should display the word and the count:

Big 2 

Data 2 

Great 2 

Hadoop 2 is 1

3.4 Stop Hadoop Services

Once you are finished, shut down the daemons:

stop-dfs.sh

stop-yarn.sh

Ready to go further into the realm of Big Data? To proceed in your learning and master Hadoop, you will need to encounter and overcome real-world challenges. The actual learning process starts when things don’t work as intended! Explore our Hadoop challenges and solutions for further learning.

Real Time Examples for Hadoop Tutorial for Learners

You have to experience Hadoop to realize the extent of its capabilities. Although the fundamental MapReduce framework is ideal for batch processing, the ecosystem around Hadoop (with tools such as HBase, Spark, and Kafka) makes real-time Big Data applications possible. Here are some real-world examples for Hadoop tutorial for learners:

Fraud Detection in Financial Services

Challenge: Banks must analyze millions of transactions instantly to identify fraudulent activity. Traditional databases are too slow to handle this high-velocity data stream.

Hadoop Ecosystem Solution: In real-time, data streams (transactions) are consumed with Apache Kafka and processed instantly by Apache Spark (that operates on YARN/Hadoop resources). Spark scans the transaction against historical data stored in an HBase (NoSQL database over HDFS) to identify anomalies and block malicious transactions before they settle. This is close to real-time processing.

Recommendation Engines for E-commerce

Challenge: E-commerce websites must suggest products in real time based on a user’s latest clickstream information, search history, and past buys. These need to be fast lookups and sophisticated analysis.

Hadoop Ecosystem Solution: HDFS holds huge quantities of historical customer information and site logs. When a user clicks, the action is serviced by an application written atop Spark or Hive. The application makes use of stored machine learning models (trained in batch against HDFS) to compute and return personalized product recommendations to the user’s browser within milliseconds.

Network Monitoring and Log Analysis

Challenge: Large corporations generate terabytes of log data daily from servers, routers, and applications. Network administrators need to monitor this data in real-time to detect security breaches or system failures.

Hadoop Ecosystem Solution: Apache Flume or Kafka gather and consolidate the log files from numerous sources and pump them into Spark Streaming. Spark quickly scans the logs for key words, error codes, or suspect access patterns and notifies administrators in real time of the problem. The logs are also stored in HDFS in parallel for subsequent batch processing (e.g., historical trend reporting).

Discover our Hadoop Project Ideas! With this knowledge of the components, test your skills on a practical project.

FAQs About Hadoop Tutorial for Beginners

Conclusion

You’ve now understood that Hadoop is the underlying, scalable platform for addressing Big Data, supporting distributed storage (HDFS) and computation (through MapReduce or the quicker Spark). As the landscape changes, its basic principles remain fundamental to contemporary data platforms. Demand for individuals skilled at handling and analyzing huge datasets continues to surge.Ready to start your Big Data journey? Enroll in a comprehensive Hadoop Course in Chennai to turn these powerful concepts into job-ready skills!