Changes

Version 2.0.0 introduces uses wait_for_it script for the cluster startup

Hadoop Docker

Supported Hadoop Versions

See repository branches for supported hadoop versions

Quick Start

To deploy an example HDFS cluster, run:

  docker-compose up

Run example wordcount job:

  make wordcount

Or deploy in swarm:

docker stack deploy -c docker-compose-v3.yml hadoop

docker-compose creates a docker network that can be found by running docker network list, e.g. dockerhadoop_default.

Run docker network inspect on the network (e.g. dockerhadoop_default) to find the IP the hadoop interfaces are published on. Access these interfaces with the following URLs:

• Namenode: http://<dockerhadoop_IP_address>:9870/dfshealth.html#tab-overview
• History server: http://<dockerhadoop_IP_address>:8188/applicationhistory
• Datanode: http://<dockerhadoop_IP_address>:9864/
• Nodemanager: http://<dockerhadoop_IP_address>:8042/node
• Resource manager: http://<dockerhadoop_IP_address>:8088/

Configure Environment Variables

The configuration parameters can be specified in the hadoop.env file or as environmental variables for specific services (e.g. namenode, datanode etc.):

  CORE_CONF_fs_defaultFS=hdfs://namenode:8020

CORE_CONF corresponds to core-site.xml. fs_defaultFS=hdfs://namenode:8020 will be transformed into:

  <property><name>fs.defaultFS</name><value>hdfs://namenode:8020</value></property>

To define dash inside a configuration parameter, use triple underscore, such as YARN_CONF_yarn_log___aggregation___enable=true (yarn-site.xml):

  <property><name>yarn.log-aggregation-enable</name><value>true</value></property>

The available configurations are:

• /etc/hadoop/core-site.xml CORE_CONF
• /etc/hadoop/hdfs-site.xml HDFS_CONF
• /etc/hadoop/yarn-site.xml YARN_CONF
• /etc/hadoop/httpfs-site.xml HTTPFS_CONF
• /etc/hadoop/kms-site.xml KMS_CONF
• /etc/hadoop/mapred-site.xml MAPRED_CONF
• /etc/hive/hive-site.xml HIVE_CONF

If you need to extend some other configuration file, refer to base/entrypoint.sh bash script.

FOR Hive with MariaDB Metastore-DB:

mysql -u root
mysql> flush privileges;
mysql> alter user 'root'@'localhost' identified by 'root';

Restart MariaDB Container

hadoop dfs -mkdir /tmp
hadoop dfs -mkdir -p /user/hive/warehouse
hadoop dfs -chmod g+w /tmp
hadoop dfs -chmod -R g+w /user

mysql -u root -p
mysql> create database metastore default character set utf8;
mysql> create user 'hive'@'%' identified by 'hive';
mysql> grant all privileges on metastore.* to 'hive'@'%';

schematool -dbType mysql -initSchema

Check with "hive" or "beeline"