Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Commit c2b11b6

Browse files
analyticallyanalytically
committed
Remove network_interface, it's now a fact per host. Upgrade elasticsearch and presto. README changes.
1 parent 088b3c7 commit c2b11b6

File tree

14 files changed

+32
-27
lines changed

14 files changed

+32
-27
lines changed

‎.travis.yml

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -12,8 +12,8 @@ before_script:
1212
- ansible-playbook -i hosts --extra-vars "api_key_password=$DO_API_KEY client_id=$DO_CLIENT_ID" do_cluster.yml
1313
- "export DISPLAY=:99.0"
1414
- "sh -e /etc/init.d/xvfb start"
15-
- "wget http://download.slimerjs.org/v0.9/0.9.0/slimerjs-0.9.0.zip"
16-
- "unzip slimerjs-0.9.0.zip && mv slimerjs-0.9.0 ./slimerjs"
15+
- "wget http://download.slimerjs.org/v0.9/0.9.1/slimerjs-0.9.1.zip"
16+
- "unzip slimerjs-0.9.1.zip && mv slimerjs-0.9.1 ./slimerjs"
1717
- "gem install --version 0.8.9 faraday"
1818
- "gem install travis-artifacts"
1919
- eval `ssh-agent`

‎README.md

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
Hadoop Ansible Playbook [![Build Status](https://travis-ci.org/analytically/hadoop-ansible.png)](https://travis-ci.org/analytically/hadoop-ansible)
1+
Hadoop Ansible Playbook [![Build Status](https://travis-ci.org/analytically/hadoop-ansible.svg?branch=master)](https://travis-ci.org/analytically/hadoop-ansible)
22
=======================
33

44
[Ansible](http://www.ansibleworks.com/) playbook that installs a CDH 4.6.0 [Hadoop](http://hadoop.apache.org/)
@@ -7,7 +7,7 @@ with [HBase](http://hbase.apache.org/), Hive, [Presto](http://prestodb.io/) for
77
[Smokeping](http://oss.oetiker.ch/smokeping/), [Fluentd](http://fluentd.org/), [Elasticsearch](http://www.elasticsearch.org/)
88
and [Kibana](http://www.elasticsearch.org/overview/kibana/) for monitoring and centralized log indexing.
99

10-
Hire/Follow [@analytically](http://twitter.com/analytically). Browse the CI [build screenshots](http://hadoop-ansible.s3-website-us-east-1.amazonaws.com/#artifacts/).
10+
Follow [@analytically](http://twitter.com/analytically). Browse the CI [build screenshots](http://hadoop-ansible.s3-website-us-east-1.amazonaws.com/#artifacts/).
1111

1212
### Requirements
1313

@@ -57,6 +57,9 @@ Required:
5757

5858
Optional:
5959

60+
- Network interface: if you'd like to use a different IP address per host (eg. internal interface), change `site.yml` and
61+
change `set_fact: ipv4_address=...` to determine the correct IP address to use per host. If this fact is not set,
62+
`ansible_default_ipv4.address` will be used.
6063
- Email notification: `notify_email`, `postfix_domain`, `mandrill_username`, `mandrill_api_key`
6164
- [`roles/common`](roles/common/defaults/main.yml): `kernel_swappiness`(0), `nofile` limits, ntp servers and `rsyslog_polling_interval_secs`(10)
6265
- [`roles/2_aggregated_links`](roles/2_aggregated_links/defaults/main.yml): `bond_mode` (balance-alb) and `mtu` (9216)

‎group_vars/all

Lines changed: 1 addition & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -11,5 +11,4 @@ mtu: 9216
1111
# mandrill_api_key: your_api_key
1212

1313
# Upgrade kernel to 3.13, much improved epoll performance
14-
upgrade_kernel: no
15-
network_interface: eth0
14+
upgrade_kernel: no

‎roles/cdh_hadoop_config/templates/hadoop-metrics2.properties

Lines changed: 7 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -35,22 +35,22 @@
3535
*.sink.ganglia.dmax=jvm.metrics.threadsBlocked=70,jvm.metrics.memHeapUsedM=40
3636

3737
# journalnodes
38-
journalnode.sink.ganglia.servers={% for host in groups['namenodes'] %}{{ hostvars[host]["ansible_" + network_interface]["ipv4"]['address'] }}:8649{% if not loop.last %},{% endif %}{% endfor %}
38+
journalnode.sink.ganglia.servers={% for host in groups['namenodes'] %}{{ hostvars[host].ipv4_address|default(hostvars[host].ansible_default_ipv4.address) }}:8649{% if not loop.last %},{% endif %}{% endfor %}
3939

4040
# namenodes
41-
namenode.sink.ganglia.servers={% for host in groups['namenodes'] %}{{ hostvars[host]["ansible_" + network_interface]["ipv4"]['address'] }}:8649{% if not loop.last %},{% endif %}{% endfor %}
41+
namenode.sink.ganglia.servers={% for host in groups['namenodes'] %}{{ hostvars[host].ipv4_address|default(hostvars[host].ansible_default_ipv4.address) }}:8649{% if not loop.last %},{% endif %}{% endfor %}
4242

4343
# datanodes
44-
datanode.sink.ganglia.servers={% for host in groups['namenodes'] %}{{ hostvars[host]["ansible_" + network_interface]["ipv4"]['address'] }}:8649{% if not loop.last %},{% endif %}{% endfor %}
44+
datanode.sink.ganglia.servers={% for host in groups['namenodes'] %}{{ hostvars[host].ipv4_address|default(hostvars[host].ansible_default_ipv4.address) }}:8649{% if not loop.last %},{% endif %}{% endfor %}
4545

4646
# jobtrackers
47-
jobtracker.sink.ganglia.servers={% for host in groups['namenodes'] %}{{ hostvars[host]["ansible_" + network_interface]["ipv4"]['address'] }}:8649{% if not loop.last %},{% endif %}{% endfor %}
47+
jobtracker.sink.ganglia.servers={% for host in groups['namenodes'] %}{{ hostvars[host].ipv4_address|default(hostvars[host].ansible_default_ipv4.address) }}:8649{% if not loop.last %},{% endif %}{% endfor %}
4848

4949
# tasktrackers
50-
tasktracker.sink.ganglia.servers={% for host in groups['namenodes'] %}{{ hostvars[host]["ansible_" + network_interface]["ipv4"]['address'] }}:8649{% if not loop.last %},{% endif %}{% endfor %}
50+
tasktracker.sink.ganglia.servers={% for host in groups['namenodes'] %}{{ hostvars[host].ipv4_address|default(hostvars[host].ansible_default_ipv4.address) }}:8649{% if not loop.last %},{% endif %}{% endfor %}
5151

5252
# maptasks
53-
maptask.sink.ganglia.servers={% for host in groups['namenodes'] %}{{ hostvars[host]["ansible_" + network_interface]["ipv4"]['address'] }}:8649{% if not loop.last %},{% endif %}{% endfor %}
53+
maptask.sink.ganglia.servers={% for host in groups['namenodes'] %}{{ hostvars[host].ipv4_address|default(hostvars[host].ansible_default_ipv4.address) }}:8649{% if not loop.last %},{% endif %}{% endfor %}
5454

5555
# reducetasks
56-
reducetask.sink.ganglia.servers={% for host in groups['namenodes'] %}{{ hostvars[host]["ansible_" + network_interface]["ipv4"]['address'] }}:8649{% if not loop.last %},{% endif %}{% endfor %}
56+
reducetask.sink.ganglia.servers={% for host in groups['namenodes'] %}{{ hostvars[host].ipv4_address|default(hostvars[host].ansible_default_ipv4.address) }}:8649{% if not loop.last %},{% endif %}{% endfor %}

‎roles/cdh_hadoop_config/templates/slaves

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,2 +1,2 @@
1-
{% for host in groups['datanodes'] %}{{ hostvars[host]["ansible_" + network_interface]["ipv4"]['address'] }}
1+
{% for host in groups['datanodes'] %}{{ hostvars[host].ipv4_address|default(hostvars[host].ansible_default_ipv4.address) }}
22
{% endfor %}

‎roles/cdh_hbase_config/templates/regionservers

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,2 +1,2 @@
1-
{% for host in groups['regionservers'] %}{{ hostvars[host]["ansible_" + network_interface]["ipv4"]['address'] }}
1+
{% for host in groups['regionservers'] %}{{ hostvars[host].ipv4_address|default(hostvars[host].ansible_default_ipv4.address) }}
22
{% endfor %}

‎roles/common/templates/hosts

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -3,5 +3,5 @@
33
127.0.0.1 localhost
44

55
{% for host in groups['all'] %}
6-
{{ hostvars[host]["ansible_" + network_interface]["ipv4"]["address"] }} {{ hostvars[host]["ansible_fqdn"] }} {{ hostvars[host]["ansible_hostname"] }}
6+
{{ hostvars[host].ipv4_address|default(hostvars[host].ansible_default_ipv4.address) }} {{ hostvars[host]["ansible_fqdn"] }} {{ hostvars[host]["ansible_hostname"] }}
77
{% endfor %}

‎roles/elasticsearch/tasks/main.yml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@
22
# file: roles/elasticsearch/tasks/main.yml
33

44
- name: install Elasticsearch
5-
shell: creates=/usr/share/elasticsearch/bin/elasticsearch chdir=/tmp wget https://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-1.0.1.deb && dpkg -i elasticsearch-1.0.1.deb
5+
shell: creates=/usr/share/elasticsearch/bin/elasticsearch chdir=/tmp wget https://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-1.1.0.deb && dpkg -i elasticsearch-1.1.0.deb
66
tags: elasticsearch
77

88
- name: make sure Elasticsearch is enabled and started

‎roles/ganglia_metad/templates/gmetad.conf

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -40,7 +40,7 @@
4040
# data_source "my grid" 50 1.3.4.7:8655 grid.org:8651 grid-backup.org:8651
4141
# data_source "another source" 1.3.4.7:8655 1.3.4.8
4242

43-
data_source "Hadoop" {{ hostvars[groups["masters"][0]]["ansible_" + network_interface]["ipv4"]["address"] }} {{ hostvars[groups["masters"][1]]["ansible_" + network_interface]["ipv4"]["address"] }}
43+
data_source "Hadoop" {{ hostvars[groups["masters"][0]].ipv4_address|default(hostvars[groups["masters"][0]].ansible_default_ipv4.address) }} {{ hostvars[groups["masters"][1]].ipv4_address|default(hostvars[groups["masters"][1]].ansible_default_ipv4.address) }}
4444

4545
#
4646
# Round-Robin Archives

‎roles/ganglia_monitor/templates/gmond.conf

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -36,12 +36,12 @@ host {
3636
also have each node send the same information to more than one place for redundancy. */
3737

3838
udp_send_channel {
39-
host = {{ hostvars[groups["masters"][0]]["ansible_" + network_interface]["ipv4"]["address"] }}
39+
host = {{ hostvars[groups["masters"][0]].ipv4_address|default(hostvars[groups["masters"][0]].ansible_default_ipv4.address) }}
4040
port = {{ ganglia_udp_send|default('8649') }}
4141
}
4242

4343
udp_send_channel {
44-
host = {{ hostvars[groups["masters"][1]]["ansible_" + network_interface]["ipv4"]["address"] }}
44+
host = {{ hostvars[groups["masters"][1]].ipv4_address|default(hostvars[groups["masters"][1]].ansible_default_ipv4.address) }}
4545
port = {{ ganglia_udp_accept|default('8649') }}
4646
}
4747

‎roles/presto_common/tasks/main.yml

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -14,7 +14,7 @@
1414
tags: presto
1515

1616
- name: install Presto at /usr/lib/presto
17-
shell: creates=/usr/lib/presto/bin/launcher chdir=/tmp curl http://central.maven.org/maven2/com/facebook/presto/presto-server/0.61/presto-server-0.61.tar.gz | tar xz && mv presto-server-0.61/* /usr/lib/presto
17+
shell: creates=/usr/lib/presto/bin/launcher chdir=/tmp curl http://central.maven.org/maven2/com/facebook/presto/presto-server/0.65/presto-server-0.65.tar.gz | tar xz && mv presto-server-0.65/* /usr/lib/presto
1818
tags: presto
1919

2020
- name: configure presto in /usr/lib/presto/etc
@@ -37,7 +37,7 @@
3737
- presto
3838

3939
- name: install presto command line client
40-
shell: chdir=/usr/bin curl -o presto http://central.maven.org/maven2/com/facebook/presto/presto-cli/0.61/presto-cli-0.61-executable.jar && chmod +x presto
40+
shell: chdir=/usr/bin curl -o presto http://central.maven.org/maven2/com/facebook/presto/presto-cli/0.65/presto-cli-0.65-executable.jar && chmod +x presto
4141
tags: presto
4242

4343
- name: install upstart config for presto

‎roles/smokeping/templates/Targets

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -21,7 +21,7 @@ title = {{ group|replace("_", " ")|title }}
2121
{% for host in groups[group] %}
2222
+++ {{ hostvars[host]['ansible_hostname'] }}
2323
menu = {{ hostvars[host]['ansible_hostname'] }}
24-
title = {{ group|replace("_", " ")|title }} {{ hostvars[host]['ansible_hostname'] }} at {{ hostvars[host]["ansible_" + network_interface]["ipv4"]['address'] }}
24+
title = {{ group|replace("_", " ")|title }} {{ hostvars[host]['ansible_hostname'] }} at {{ hostvars[host].ipv4_address|default(hostvars[host].ansible_default_ipv4.address) }}
2525
host = {{ hostvars[host]['ansible_fqdn'] }}
2626
alerts = bigloss,someloss,startloss,rttdetect,hostdown
2727
{% endfor %}

‎roles/td_agent/templates/td-agent.conf

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -21,10 +21,10 @@
2121
type elasticsearch
2222
logstash_format true
2323
{% if groups["elasticsearch"]|length == 1 %}
24-
host {{ hostvars[groups["elasticsearch"][0]]["ansible_" + network_interface]["ipv4"]["address"] }}
24+
host {{ hostvars[groups["elasticsearch"][0]].ipv4_address|default(hostvars[groups["elasticsearch"][0]].ansible_default_ipv4.address) }}
2525
port {{ elasticsearch_port }}
2626
{% else %}
27-
hosts {% for host in groups['elasticsearch'] %}{{ hostvars[host]["ansible_" + network_interface]["ipv4"]["address"] }}:{{ elasticsearch_port }}{% if not loop.last %},{% endif %}{% endfor %}
27+
hosts {% for host in groups['elasticsearch'] %}{{ hostvars[host].ipv4_address|default(hostvars[host].ansible_default_ipv4.address) }}:{{ elasticsearch_port }}{% if not loop.last %},{% endif %}{% endfor %}
2828
{% endif %}
2929

3030
</match>

‎site.yml

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,10 @@
44

55
- hosts: all
66
user: ansibler
7-
tasks: []
7+
tasks:
8+
- name: determine interface
9+
set_fact: ipv4_address="{{ hostvars[inventory_hostname].ansible_default_ipv4.address }}"
10+
# eg. to use a eth1: ipv4_address="{{ hostvars[inventory_hostname].ansible_eth1.ipv4.address }}"
811

912
- hosts: all
1013
user: ansibler
@@ -97,7 +100,7 @@
97100
sudo_user: hdfs
98101
tasks:
99102
- name: wait for the first namenode to come online
100-
wait_for: host={{ hostvars[groups['namenodes'][0]]["ansible_" + network_interface]["ipv4"]['address'] }} port=50070
103+
wait_for: host={{ hostvars[groups['namenodes'][0]].ipv4_address|default(hostvars[groups['namenodes'][0]].ansible_default_ipv4.address) }} port=50070
101104
tags:
102105
- hadoop
103106
- hbase

0 commit comments

Comments
 (0)