How do I Install Hadoop and HBase on Ubuntu 18.04?. HBase is an open-source distributed non-relational database developed under the Apache Software Foundation. It is written in Java and runs on top of Hadoop File Systems (HDFS). HBase is one of the dominant databases when working with big data since it is designed for a quick read and write access to huge amounts of structured data.
For CentOS 7, refer to How to Install Apache Hadoop / HBase on CentOS 7
This is our first guide on the installation of Hadoop and HBase on Ubuntu 18.04. It is HBase installation on a Single Node Hadoop Cluster. It is done on a barebone Ubuntu 18.04 virtual machine with 8GB Ram and 4vCPU.
Install Hadoop on Ubuntu 18.04
Here are the steps used to install a Single node Hadoop cluster on Ubuntu 18.04 LTS.
Step 1: Update System
Update your Ubuntu system before starting deployment of Hadoop and HBase.
sudo apt update
sudo apt -y upgrade
sudo reboot
Step 2: Install Java
Install Java if it is missing on your Ubuntu 18.04 system.
sudo add-apt-repository ppa:webupd8team/java
sudo apt update
sudo apt -y install oracle-java8-installer oracle-java8-set-default
Validate is Java has been installed successfully.
$ java -version
java version "1.8.0_201"
Java(TM) SE Runtime Environment (build 1.8.0_201-b09)
Java HotSpot(TM) 64-Bit Server VM (build 25.201-b09, mixed mode)
Set JAVA_HOME variable.
cat <<EOF | sudo tee /etc/profile.d/hadoop_java.sh
export JAVA_HOME=$(dirname $(dirname $(readlink $(readlink $(which javac)))))
export PATH=\$PATH:\$JAVA_HOME/bin
EOF
Update your $PATH and setting.
source /etc/profile.d/hadoop_java.sh
Then test:
$ echo $JAVA_HOME
/usr/lib/jvm/java-8-oracle
Ref:
How to set default Java version on Ubuntu / Debian
How to Install Java 8 on Ubuntu 18.04/16.04
Step 3: Create a User Account for Hadoop
Let’s create a separate user for Hadoop so we have isolation between the Hadoop file system and the Unix file system.
sudo adduser hadoop
sudo usermod -aG sudo hadoop
Once the user is added, generate SS key pair for the user.
$ sudo su - hadoop
$ ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/home/hadoop/.ssh/id_rsa):
Created directory '/home/hadoop/.ssh'.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/hadoop/.ssh/id_rsa.
Your public key has been saved in /home/hadoop/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:mA1b0nzdKcwv/LPktvlA5R9LyNe9UWt+z1z0AjzySt4 [email protected]
The key's randomart image is:
+---[RSA 2048]----+
| |
| o + . . |
| o + . = o o|
| O . o.o.o=|
| + S . *ooB=|
| o *=.B|
| . . *+=|
| o o o.O+|
| o E.=o=|
+----[SHA256]-----+
Add this user’s key to list of Authorized ssh keys.
cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
chmod 0600 ~/.ssh/authorized_keys
Verify that you can ssh using added key.
[email protected]:~$ ssh localhost
The authenticity of host 'localhost (127.0.0.1)' can't be established.
ECDSA key fingerprint is SHA256:CWrh3rLJRpu5dUGI278L1JnO29waYxJ0LzdUKV1t1/U.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'localhost' (ECDSA) to the list of known hosts.
Welcome to Ubuntu 18.04.2 LTS (GNU/Linux 4.15.0-47-generic x86_64)
Documentation: https://help.ubuntu.com
Management: https://landscape.canonical.com
Support: https://ubuntu.com/advantage
System information as of Sat Apr 6 17:46:37 CEST 2019
System load: 0.01 Processes: 105
Usage of /: 2.2% of 75.06GB Users logged in: 1
Memory usage: 1% IP address for eth0: 192.168.20.11
......
[email protected]:~$ exit
logout
Connection to localhost closed.
Step 4: Download and Install Hadoop
Check for the most recent version of Hadoop before downloading version specified here. As of this writing, this is version 3.2.1.
Save the recent version to a variable.
RELEASE="3.2.1"
Then download Hadoop archive to your local system.
wget https://www-eu.apache.org/dist/hadoop/common/hadoop-$RELEASE/hadoop-$RELEASE.tar.gz
Extract the file.
tar -xzvf hadoop-$RELEASE.tar.gz
Move resulting directory to /usr/local/hadoop.
rm hadoop-$RELEASE.tar.gz
sudo mv hadoop-$RELEASE/ /usr/local/hadoop
Set HADOOP_HOME and add directory with Hadoop binaries to your $PATH.
cat <<EOF | sudo tee /etc/profile.d/hadoop_java.sh
export JAVA_HOME=$(dirname $(dirname $(readlink $(readlink $(which javac)))))
export HADOOP_HOME=/usr/local/hadoop
export HADOOP_HDFS_HOME=$HADOOP_HOME
export HADOOP_MAPRED_HOME=$HADOOP_HOME
export YARN_HOME=$HADOOP_HOME
export HADOOP_COMMON_HOME=$HADOOP_HOME
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
export PATH=\$PATH:\$JAVA_HOME/bin:\$HADOOP_HOME/bin:\$HADOOP_HOME/sbin
EOF
Source file.
source /etc/profile.d/hadoop_java.sh
Confirm your Hadoop version.
$ hadoop version Hadoop 3.2.1 Source code repository https://github.com/apache/hadoop.git -r 1019dde65bcf12e05ef48ac71e84550d589e5d9a Compiled by sunilg on 2019-01-29T01:39Z Compiled with protoc 2.5.0 From source with checksum 64b8bdd4ca6e77cce75a93eb09ab2a9 This command was run using /usr/local/hadoop/share/hadoop/common/hadoop-common-3.2.1.jar
Step 5: Configure Hadoop
All your Hadoop configurations are located /usr/local/hadoop/etc/hadoop/
A number of configuration files need to be modified to complete Hadoop installation on Ubuntu 18.04.
First edit JAVA_HOME in shell script hadoop-env.sh:
$ sudo vim /usr/local/hadoop/etc/hadoop/hadoop-env.sh
# Set JAVA_HOME - Line 54
export JAVA_HOME=/usr/lib/jvm/java-8-oracle
Then configure:
1. core-site.xml
The core-site.xml file contains Hadoop cluster information used when starting up. These properties include:
- The port number used for Hadoop instance
- The memory allocated for
file system - The memory limit for data storage
- The size of
Read / Write buffers.
Open core-site.xml
sudo vim /usr/local/hadoop/etc/hadoop/core-site.xml
Add the following properties in between the <configuration> and </configuration> tags.
<configuration>
<property>
<name>fs.default.name</name>
<value>hdfs://localhost:9000</value>
<description>The default file system URI</description>
</property>
</configuration>
See screenshot below.
2. hdfs-site.xml
This file needs to be configured for each host to be used in the cluster. This file holds information such as:
- The namenode and datanode paths on the local filesystem.
- The value of replication data
In this setup, I want to store Hadoop infrastructure in a secondary disk – /dev/sdb.
$ lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 76.3G 0 disk
└─sda1 8:1 0 76.3G 0 part /
sdb 8:16 0 50G 0 disk
sr0 11:0 1 1024M 0 rom
I’ll partition and mount this disk to /hadoop directory.
sudo parted -s -- /dev/sdb mklabel gpt
sudo parted -s -a optimal -- /dev/sdb mkpart primary 0% 100%
sudo parted -s -- /dev/sdb align-check optimal 1
sudo mkfs.xfs /dev/sdb1
sudo mkdir /hadoop
echo "/dev/sdb1 /hadoop xfs defaults 0 0" | sudo tee -a /etc/fstab
sudo mount -a
Confirm:
$ df -hT | grep /dev/sdb1
/dev/sdb1 xfs 50G 84M 50G 1% /hadoop
Create directories for namenode and datanode.
sudo mkdir -p /hadoop/hdfs/{namenode,datanode}
Set ownership to hadoop user and group.
sudo chown -R hadoop:hadoop /hadoop
Now open the file:
sudo vim /usr/local/hadoop/etc/hadoop/hdfs-site.xml
Then add the following properties in between the <configuration> and </configuration> tags.
<configuration>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
<property>
<name>dfs.name.dir</name>
<value>file:///hadoop/hdfs/namenode</value>
</property>
<property>
<name>dfs.data.dir</name>
<value>file:///hadoop/hdfs/datanode</value>
</property>
</configuration>See screenshot below.
3. mapred-site.xml
This is where you set the MapReduce framework to use.
sudo vim /usr/local/hadoop/etc/hadoop/mapred-site.xml
Set like below.
<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
</configuration>4. yarn-site.xml
Settings in this file will overwrite the configurations for Hadoop yarn. It defines resource management and job scheduling logic.
sudo vim /usr/local/hadoop/etc/hadoop/yarn-site.xml
Add:
<configuration>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
</configuration>Here is the screenshot of my configuration.
Step 6: Validate Hadoop Configurations
Initialize Hadoop Infrastructure store.
sudo su - hadoop
hdfs namenode -format
See output below:
Test HDFS configurations.
$ start-dfs.sh
Starting namenodes on [localhost]
Starting datanodes
Starting secondary namenodes [hbase]
hbase: Warning: Permanently added 'hbase' (ECDSA) to the list of known hosts.
Lastly verify YARN configurations:
$ start-yarn.sh
Starting resourcemanager
Starting nodemanagers
Hadoop 3.x defult Web UI ports:
- NameNode – Default HTTP port is 9870.
- ResourceManager – Default HTTP port is 8088.
- MapReduce JobHistory Server – Default HTTP port is 19888.
You can check ports used by hadoop using:
$ ss -tunelp
Sample output is shown below.
Access Hadoop Web Dashboard on http://ServerIP:9870.
Check Hadoop Cluster Overview at http://ServerIP:8088.
Test to see if you can create directory.
$ hadoop fs -mkdir /test
$ hadoop fs -ls /
Found 1 items
drwxr-xr-x - hadoop supergroup 0 2019-04-07 10:23 /test
Stopping Hadoop Services
Use the commands:
$ stop-dfs.sh
$ stop-yarn.sh
Install HBase on Ubuntu 18.04
You can choose to install HBase in Standalone Mode or Pseudo-Distributed Mode. The setup process is similar to our Hadoop installation.
Step 1: Download and Install HBase
Check latest
VER="1.4.10" wget http://apache.mirror.gtcomm.net/hbase/stable/hbase-$VER-bin.tar.gz
Extract Hbase archive downloaded.
tar xvf hbase-$VER-bin.tar.gz
sudo mv hbase-$VER/ /usr/local/HBase/
Update your $PATH values.
cat <<EOF | sudo tee /etc/profile.d/hadoop_java.sh
export JAVA_HOME=$(dirname $(dirname $(readlink $(readlink $(which javac)))))
export HADOOP_HOME=/usr/local/hadoop
export HADOOP_HDFS_HOME=$HADOOP_HOME
export HADOOP_MAPRED_HOME=$HADOOP_HOME
export YARN_HOME=$HADOOP_HOME
export HADOOP_COMMON_HOME=$HADOOP_HOME
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
export HBASE_HOME=/usr/local/HBase
export PATH=\$PATH:\$JAVA_HOME/bin:\$HADOOP_HOME/bin:\$HADOOP_HOME/sbin:\$HBASE_HOME/bin
EOF
Update your shell environment values.
$ source /etc/profile.d/hadoop_java.sh
$ echo $HBASE_HOME
/usr/local/HBase
Edit JAVA_HOME in shell script hbase-env.sh:
$ sudo vim /usr/local/HBase/conf/hbase-env.sh
# Set JAVA_HOME - Line 27
export JAVA_HOME=/usr/lib/jvm/java-8-oracle
Step 2: Configure HBase
We will do configurations like we did for Hadoop. All configuration files for HBase are located on /usr/local/HBase/conf/ directory.
hbase-site.xml
Set data directory to an appropriate location on this file.
Option 1: Install HBase in Standalone Mode (Not recommended)
In standalone mode all daemons (HMaster, HRegionServer, and ZooKeeper) ran in one jvm process/instance
Create HBase root directory.
sudo mkdir -p /hadoop/HBase/HFiles
sudo mkdir -p /hadoop/zookeeper
sudo chown -R hadoop:hadoop /hadoop/
Open the file for editing.
sudo vim /usr/local/HBase/conf/hbase-site.xml
Now add the following configurations between the <configuration> and </configuration> tags to look like below.
<configuration>
<property>
<name>hbase.rootdir</name>
<value>file:/hadoop/HBase/HFiles</value>
</property>
<property>
<name>hbase.zookeeper.property.dataDir</name>
<value>/hadoop/zookeeper</value>
</property>
</configuration>By default, unless you configure hbase.rootdir property, your data is still stored in /tmp/.
Now start HBase by using start-hbase.sh script in HBase bin directory.
$ sudo su - hadoop
$ start-hbase.sh
running master, logging to /usr/local/HBase/logs/hbase-hadoop-master-hbase.out
Option 2: Install HBase in Pseudo-Distributed Mode (Recommended)
Our value hbase.rootdir
To install HBase in Pseudo-Distributed Mode, set its values to:
<configuration>
<property>
<name>hbase.rootdir</name>
<value>hdfs://localhost:8030/hbase</value>
</property>
<property>
<name>hbase.zookeeper.property.dataDir</name>
<value>/hadoop/zookeeper</value>
</property>
<property>
<name>hbase.cluster.distributed</name>
<value>true</value>
</property>
</configuration>In this setup, Data is stored your data in HDFS instead.
Ensure Zookeeper directory is created.
sudo mkdir -p /hadoop/zookeeper
sudo chown -R hadoop:hadoop /hadoop/
Now start HBase by using start-hbase.sh script in HBase bin directory.
$ sudo su - hadoop
$ start-hbase.sh
localhost: running zookeeper, logging to /usr/local/HBase/bin/../logs/hbase-hadoop-zookeeper-hbase.out
running master, logging to /usr/local/HBase/logs/hbase-hadoop-master-hbase.out
: running regionserver, logging to /usr/local/HBase/logs/hbase-hadoop-regionserver-hbase.out
Check the HBase Directory in HDFS:
$ hadoop fs -ls /hbase
Found 9 items
drwxr-xr-x - hadoop supergroup 0 2019-04-07 09:19 /hbase/.tmp
drwxr-xr-x - hadoop supergroup 0 2019-04-07 09:19 /hbase/MasterProcWALs
drwxr-xr-x - hadoop supergroup 0 2019-04-07 09:19 /hbase/WALs
drwxr-xr-x - hadoop supergroup 0 2019-04-07 09:17 /hbase/corrupt
drwxr-xr-x - hadoop supergroup 0 2019-04-07 09:16 /hbase/data
drwxr-xr-x - hadoop supergroup 0 2019-04-07 09:16 /hbase/hbase
-rw-r--r-- 1 hadoop supergroup 42 2019-04-07 09:16 /hbase/hbase.id
-rw-r--r-- 1 hadoop supergroup 7 2019-04-07 09:16 /hbase/hbase.version
drwxr-xr-x - hadoop supergroup 0 2019-04-07 09:17 /hbase/oldWALs
Step 3: Managing HMaster & HRegionServer
The HMaster server controls the HBase cluster. You can start up to 9 backup HMaster servers, which makes 10 total HMasters, counting the primary.
The HRegionServer manages the data in its StoreFiles as directed by the HMaster. Generally, one HRegionServer runs per node in the cluster. Running multiple HRegionServers on the same system can be useful for testing in pseudo-distributed mode.
Master and Region Servers can be started and stopped using the local-master-backup.shlocal-regionservers.sh
$ local-master-backup.sh start 2 # Start backup HMaster
$ local-regionservers.sh start 3 # Start multiple RegionServers
- Each HMaster uses two ports (16000 and 16010 by default). The port offset is added to these ports, so using an offset of 2, the backup HMaster would use ports 16002 and 16012
The following command starts 3 backup servers using ports 16002/16012, 16003/16013, and 16005/16015.
$ local-master-backup.sh start 2 3 5
- Each RegionServer requires two ports, and the default ports are 16020 and 16030
The following command starts four additional RegionServers, running on sequential ports starting at 16022/16032 (base ports 16020/16030 plus 2).
$ local-regionservers.sh start 2 3 4 5
To stop, replace start parameter stop
$ local-regionservers.sh stop 5
Starting HBase Shell
Hadoop and Hbase should be running before you can use HBase shell. Here the correct order of starting services.
$ start-all.sh
$ start-hbase.sh
Then use HBase shell.
[email protected]:~$ hbase shell
2019-04-07 10:44:43,821 WARN [main] util.NativeCodeLoader: Unable to load native-hadoop library for your platform… using builtin-java classes where applicable
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/local/HBase/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/local/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
HBase Shell
Use "help" to get list of supported commands.
Use "exit" to quit this interactive shell.
Version 1.4.9, rd625b212e46d01cb17db9ac2e9e927fdb201afa1, Wed Dec 5 11:54:10 PST 2018
hbase(main):001:0>
Stopping HBase.
stop-hbase.sh
Conclusion
You have successfully installed Hadoop and HBase on Ubuntu 18.04. Refer to Apache Hadoop Documentation and Apache HBase book to learn more.
Other interesting guides:
Install Zipkin distributed tracing system on Ubuntu / Debian / CentOS
Install WildFly (JBoss) Application Server on Ubuntu 18.04 / Debian 10/9
