Webserver setup : to create webserver
sudo yum install httpd -y
sudo chkconfig httpd on
sudo service httpd status
sudo service httpd start
Hit on browser with ip address for centos ip
cd /var/www/html/
ls ---> blank
cd
this is for CM
wget http://archive.cloudera.com/cm5/repo-as-tarball/5.4.0/cm5.4.0-centos6.tar.gz
---> downlod to home folder
wget http://archive.cloudera.com/cm5/repo-as-tarball/5.4.0/cm5.4.0-centos6.tar.gz
sudo tar -zxvf cm5.4.0-centos6.tar.gz -C /var/www/html/
sudo tar -zxvf cm5.4.0-centos6.tar.gz -C /var/www/html/ --> move to this folder
Hit on browser with ip address for centos ip like
10.0.0./cm in browser --> its means it will make this untar & place it on his
server
Click on cloudera-manager-repo -->> this is repo of CM & jenkins comply dont want
this
cd /var/www/html/
ls
cm
cd cm
ls
cat cm.repo
cd /var/www/html/
===================================================================================
=====
===================================================================================
=====
For CDH -> both cm & CDH folder are as same place - > cd /var/www/html/
===================================================================================
==
create folder for CDH
cd /var/www/html/
sudo mkdir cdh5.4.0
ls
cd cdh5.4.0
For CDH download this in this cd cdh5.4.0
sudo wget http://archive.cloudera.com/cdh5/parcels/5.4.1.6/CDH-5.4.1-
1.cdh5.4.1.p0.6-el6.parcel
sudo wget http://archive.cloudera.com/cdh5/parcels/5.4.1.6/CDH-5.4.1-
1.cdh5.4.1.p0.6-el6.parcel
ls
du -h cdh version --> Download all the repos in this floders
sudo wget http://archive.cloudera.com/cdh5/parcels/5.4.1.6/manifest.json
changing the repo address
cd /etc/yum.repos.d/
ls --->there is no cm repo available for CM
sudo nano cloudera-manager.repo --> Create the file for CM with change the
ip/fqdn address as your CM machine
[cloudera-manager]
name=cloudera manager
baseurl=http://ip-10-0-1-135.ec2.internal/cm/5.4.0/
gpgcheck = 0
gpgkey =http://ip-10-0-1-135.ec2.internal/cm/RPM-GPG-KEY-cloudera
Refest the repolist & check for cm in list
sudo yum clean all --> clean all the data
sudo yum makecache --> to make cache for repo & tell server to our repo as local
server
sudo yum repolist all -->. update & available all updated repo list
-->> Ensure the list has the cloudera manager repo <<--
cd
clear
Setup MYSQL SERVER ON local machine -->>MYSQL SERVER ON SAME LOCAL MACHINE
sudo yum install mysql-server -y --> yum will take mysql server from his own
repo
sudo service mysqld status
sudo service mysqld start
sudo service mysqld status Status as OK
sudo yum install -y mysql-connector-java -->jdbc driver installed for onw machine
INSTALLATION OF SQL:
mysql_secure_installation --> To make sceure TO DATABASE
Enter current password for root (enter for none):[ENTER]
Set root password? [Y/n] y
New password: 123
Re-enter new password: 123
Remove anonymous users? [Y/n] y
Disallow root login remotely? [Y/n] n
Remove test database and access to it? [Y/n] n
Reload privilege tables now? [Y/n] y
===============================================
login to mysql :-
mysql -u root -p
password- 123
select host, user, password from mysql.user;
create database hive DEFAULT CHARACTER SET utf8;
grant all on hive.* TO 'hive'@'%' IDENTIFIED BY 'P@ssw0rd';
create database hue DEFAULT CHARACTER SET utf8;
grant all on hue.* TO 'hue'@'%' IDENTIFIED BY 'P@ssw0rd';
create database rman DEFAULT CHARACTER SET utf8;
grant all on rman.* TO 'rman'@'%' IDENTIFIED BY 'P@ssw0rd';
create database navs DEFAULT CHARACTER SET utf8;
grant all on navs.* TO 'navs'@'%' IDENTIFIED BY 'P@ssw0rd';
create database navms DEFAULT CHARACTER SET utf8;
grant all on navms.* TO 'navms'@'%' IDENTIFIED BY 'P@ssw0rd';
create database oozie DEFAULT CHARACTER SET utf8;
grant all on oozie.* TO 'oozie'@'%' IDENTIFIED BY 'P@ssw0rd';
create database actmo DEFAULT CHARACTER SET utf8;
grant all on actmo.* TO 'actmo'@'%' IDENTIFIED BY 'P@ssw0rd';
create database sentry DEFAULT CHARACTER SET utf8;
grant all on sentry.* TO 'sentry'@'%' IDENTIFIED BY 'P@ssw0rd';
grant all on *.* to 'temp'@'%' identified by 'P@ssw0rd' with grant option;
show databases;
select host, user, password from mysql.user;
Exit
INSTALL CM SERVER AGENT DEAMONS:
sudo yum install cloudera-manager-server cloudera-manager-daemons
sudo yum install cloudera-manager-agent cloudera-manager-daemons
-----------------------------------------------------------------
sudo yum install cloudera-manager-server cloudera-manager-daemons
sudo yum install cloudera-manager-agent cloudera-manager-daemons
--------------------------------------------------------------------
tail -f /var/log/cloudera-scm-server/cloudera-scm-server.logs
sudo /usr/share/cmf/schema/scm_prepare_database.sh mysql -h ip-10-0-
1.122.ec2.internal -p 3306 -u temp -p --scm-host ip-10-0-0.152.ec2.internal scmdb
scmuser
password - p@ssw0rd
login to databases
mysql -u root -p password - 123
select host, user, password from mysql.user;
drop user 'temp'@'%';
select host, user, password from mysql.user;
exit;
================================================================
================================================================
INSTALL CM SERVER AND AGENT DAEMONS (Run both)
sudo yum install cloudera-manager-server cloudera-manager-daemons
sudo yum install cloudera-manager-agent cloudera-manager-daemons
sudo less /etc/cloudera-scm-server/db.properties ---> to know all the downloads
to be from local repo
-->> to check logs
FOR LOCAL MYSQL SERVER
tail -f /var/log/cloudera-scm-server/cloudera-scm-server.log
sudo /usr/share/cmf/schema/scm_prepare_database.sh mysql -h ip-10-0-0-
119.ec2.internal -P 3306 -u temp -p --scm-host ip-10-0-0-119.ec2.internal scmdb
scmuser
Password should be - P@ssw0rd P@ssw0rd
mysql -u root -p
select host, user, password from mysql.user;
drop user 'temp'@'%';
select host, user, password from mysql.user;
exit
=============================
in another duplicate shall
sudo less /etc/cloudera-scm-server/db.properties --> to verify DB is create or
not.
netstat -nap | grep 7180
9. START CM AGENT AND CM SERVER
sudo service cloudera-scm-agent start
sudo service cloudera-scm-agent status
sudo service cloudera-scm-server start
sudo service cloudera-scm-server status
After 2 min goto yourcmip:7180 Log in paste the host IPS, Your CM host will appear
in Managed hosts and won’t be
included in the cluster.
Take more machines same like this as prerequisite is completed as 4 machine 1 for
SG & 3 for datanodes
On the Cluster INSTALLATION page for CDH version go to more options and remove all
genkins parcel address and at the local parcel repo box add
and continue
sudo cp /etc/yum.repos.d/cloudera-manager.repo --> To see the CM version from
local repo
cd /var/www/html/cdh -->put cdh version here
pwd
http://10.0.1.135/cdh5.4.0/ only change ip address & paste this in to config
for CM
http://10.0.1.135/cm/5 put proper ip in custome repo adresss & paste
for key - > http://10.0.1.135/cm/RPM-GPG-KEY-cloudera
sh clustercmd.sh sudo yum install -y mysql-connector-java
Set up sql with user name & password
test conection
Use powerWindow
cd /Downloads
.\pscp.exe -i .\vtnew.PPK key .\clustercmd.sh [email protected]:/home/centos
cat clustercmd.sh
to put jdbc connector set to all machines us script with create file as cluster
with all ip addresses
send .pem file on cluster
.\pscp.exe -i .\.PPK key .\.pem centos@ip adresss:/home/centos
chmod 400 .pem
sh clustercmd.sh sudo yum install -y mysql-connector-java
sh clustercmd.sh sudo yum install -y mysql-connector-java
====================================================
====================================================
No kernel versions that are known to be bad are running.
Cloudera recommends setting /proc/sys/vm/swappiness to at most 10. Current setting
is 60. Use the sysctl command
to change this setting at runtime and edit /etc/sysctl.conf for this setting to be
saved after a reboot.
You may continue with installation, but you may run into issues with Cloudera
Manager reporting that
your hosts are unhealthy because they are swapping. The following hosts are
affected:
No performance concerns with Transparent Huge Pages settings.
