From 31409f1f6059a9ca76b1408f1de113c935f541e7 Mon Sep 17 00:00:00 2001 From: Chris Olstrom Date: Wed, 6 May 2015 11:55:21 -0700 Subject: [PATCH 01/17] Update README.md --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index ebdc3d1..e06a070 100644 --- a/README.md +++ b/README.md @@ -46,7 +46,7 @@ How to Use (Hardcore Mode) -------------------------- If you're cool with allowing arbitrary code from the internet to run with root privileges with no human oversight, you can do this: -```curl https://raw.github.com/colstrom/forge/master/bootstrap.py | python``` +```curl https://raw.github.com/telusdigital/forge/master/bootstrap.py | python``` How to Use (Recommended) ------------------------ From 82252c93c73d96433ea4b2ffcbb5a3fb501f14cd Mon Sep 17 00:00:00 2001 From: Steven Harradine Date: Wed, 24 Jun 2015 12:22:36 -0400 Subject: [PATCH 02/17] Update github raw --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index e06a070..7291837 100644 --- a/README.md +++ b/README.md @@ -46,7 +46,7 @@ How to Use (Hardcore Mode) -------------------------- If you're cool with allowing arbitrary code from the internet to run with root privileges with no human oversight, you can do this: -```curl https://raw.github.com/telusdigital/forge/master/bootstrap.py | python``` +```curl https://raw.githubusercontent.com/telusdigital/forge/master/bootstrap.py | python``` How to Use (Recommended) ------------------------ From 11c5a235bb1803df06b95038f3deb63097385bcc Mon Sep 17 00:00:00 2001 From: Steven Harradine Date: Wed, 24 Jun 2015 17:23:45 -0400 Subject: [PATCH 03/17] run python in hardcore mode with sudo --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 7291837..488090e 100644 --- a/README.md +++ b/README.md @@ -46,7 +46,7 @@ How to Use (Hardcore Mode) -------------------------- If you're cool with allowing arbitrary code from the internet to run with root privileges with no human oversight, you can do this: -```curl https://raw.githubusercontent.com/telusdigital/forge/master/bootstrap.py | python``` +```curl https://raw.githubusercontent.com/telusdigital/forge/master/bootstrap.py | sudo python``` How to Use (Recommended) ------------------------ From 9890e0a258245ccb9f5f3e787103995c2da4006c Mon Sep 17 00:00:00 2001 From: Steven Harradine Date: Tue, 7 Jul 2015 13:10:17 -0400 Subject: [PATCH 04/17] added SETTING UP YOUR ENVIRONMENT FOR PLAYBOOK DEV --- README.md | 27 +++++++++++++++++++++++++++ 1 file changed, 27 insertions(+) diff --git a/README.md b/README.md index 488090e..9f1e1fc 100644 --- a/README.md +++ b/README.md @@ -54,6 +54,33 @@ If you'd prefer a more sane approach, upload ```bootstrap.py``` to somewhere you ```curl https://YOUR_URL_HERE/bootstrap.py | python``` +SETTING UP YOUR ENVIRONMENT FOR PLAYBOOK DEV +-------------------------------------------- +CREATING THE PLAYBOOK FROM SCRATCH +---------------------------------- +* Create a new repository +* Populate it with the Playbook Skeleton scripts +* Make sure any vault files have your master vault password in the forge bucket + +TESTING CHANGES TO ROLES +------------------------ +* Make all your changes locally +* Get them to a next server (test branches, rsync, etc...) in the `/etc/ansible/roles/username.rolename` path +* Edit `/tmp/playbook-*.yaml` on the server to use your test role instead of telusdigital's +* Use `ansible-playbook` to rerun the playbook manually + +SYNCING PLAYBOOKS TO FORGE +-------------------------- +```pip install s3cmd``` +```s3cmd --configure``` +```s3cmd sync playbook-foo/ s3://telusdigital-forge/foo/``` + +RERUN FORGE WITHOUT REPROVISIONING +---------------------------------- +Check https://github.com/telusdigital/forge +CHECKING THAT EVERYTHING RAN PROPERLY +```cat /var/log/cloud-init-output.log``` + License ------- [MIT](https://tldrlegal.com/license/mit-license) From df39d2367ebe0dabaef4cc60fd076859f8125bf0 Mon Sep 17 00:00:00 2001 From: Steven Harradine Date: Tue, 7 Jul 2015 13:14:43 -0400 Subject: [PATCH 05/17] fix case, fix headers --- README.md | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) diff --git a/README.md b/README.md index 9f1e1fc..f29f897 100644 --- a/README.md +++ b/README.md @@ -54,31 +54,31 @@ If you'd prefer a more sane approach, upload ```bootstrap.py``` to somewhere you ```curl https://YOUR_URL_HERE/bootstrap.py | python``` -SETTING UP YOUR ENVIRONMENT FOR PLAYBOOK DEV --------------------------------------------- -CREATING THE PLAYBOOK FROM SCRATCH ----------------------------------- +## Setting up your environment for playbook dev + +### Creating the playbook from scratch * Create a new repository * Populate it with the Playbook Skeleton scripts * Make sure any vault files have your master vault password in the forge bucket -TESTING CHANGES TO ROLES ------------------------- +### Testing changes to roles * Make all your changes locally * Get them to a next server (test branches, rsync, etc...) in the `/etc/ansible/roles/username.rolename` path * Edit `/tmp/playbook-*.yaml` on the server to use your test role instead of telusdigital's * Use `ansible-playbook` to rerun the playbook manually -SYNCING PLAYBOOKS TO FORGE --------------------------- +### Syncing playbooks to forge ```pip install s3cmd``` + ```s3cmd --configure``` + ```s3cmd sync playbook-foo/ s3://telusdigital-forge/foo/``` -RERUN FORGE WITHOUT REPROVISIONING ----------------------------------- +### Rerun forge without reprovisioning Check https://github.com/telusdigital/forge + CHECKING THAT EVERYTHING RAN PROPERLY + ```cat /var/log/cloud-init-output.log``` License From 532af686f9e5b729387963f8f2230e511e223deb Mon Sep 17 00:00:00 2001 From: Steven Harradine Date: Tue, 7 Jul 2015 13:15:30 -0400 Subject: [PATCH 06/17] guessing sudo needed here --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index f29f897..9348467 100644 --- a/README.md +++ b/README.md @@ -52,7 +52,7 @@ How to Use (Recommended) ------------------------ If you'd prefer a more sane approach, upload ```bootstrap.py``` to somewhere you control. -```curl https://YOUR_URL_HERE/bootstrap.py | python``` +```curl https://YOUR_URL_HERE/bootstrap.py | sudo python``` ## Setting up your environment for playbook dev From 28559327c279f1a7a74e359df7530f487238aaeb Mon Sep 17 00:00:00 2001 From: Steven Harradine Date: Tue, 7 Jul 2015 13:18:51 -0400 Subject: [PATCH 07/17] added skeleton link --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 9348467..1c7bfc0 100644 --- a/README.md +++ b/README.md @@ -58,7 +58,7 @@ If you'd prefer a more sane approach, upload ```bootstrap.py``` to somewhere you ### Creating the playbook from scratch * Create a new repository -* Populate it with the Playbook Skeleton scripts +* Populate it with the [Playbook Skeleton](https://github.com/telusdigital/playbook-skeleton) scripts * Make sure any vault files have your master vault password in the forge bucket ### Testing changes to roles From 9b712b9af5bfdadd5bd5072531b6464998c76df8 Mon Sep 17 00:00:00 2001 From: Steven Harradine Date: Tue, 7 Jul 2015 18:45:13 -0400 Subject: [PATCH 08/17] add trouble shooting, update rerun forge --- README.md | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index 1c7bfc0..3d105d3 100644 --- a/README.md +++ b/README.md @@ -75,12 +75,17 @@ If you'd prefer a more sane approach, upload ```bootstrap.py``` to somewhere you ```s3cmd sync playbook-foo/ s3://telusdigital-forge/foo/``` ### Rerun forge without reprovisioning -Check https://github.com/telusdigital/forge - -CHECKING THAT EVERYTHING RAN PROPERLY +* SSH into the server +* Run `sudo reforge` +### Checking that everything ran properly ```cat /var/log/cloud-init-output.log``` +### Troubleshooting +Q: fatal: [localhost] => One or more undefined variables: 'domain' is undefined + +A: Add domain: teluswebteam.com to your infra playbook until we get it sorted out how this can be done better globally. + License ------- [MIT](https://tldrlegal.com/license/mit-license) From bc34471007ac61fa83972925a0835ad7af17ff6f Mon Sep 17 00:00:00 2001 From: AsIf Date: Wed, 26 Aug 2015 16:52:06 -0700 Subject: [PATCH 09/17] added function to record exit status of playbooks --- bootstrap.py | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/bootstrap.py b/bootstrap.py index c3082be..7bb018e 100644 --- a/bootstrap.py +++ b/bootstrap.py @@ -154,7 +154,6 @@ def get_vault(playbook): with open('/etc/ansible/hosts', 'a') as stream: stream.writelines(["\n[" + vault_name + "]\n", 'localhost\n']) - def configure_environment(): """ Exposes information from Resource Tags in Ansible vars """ get_vault('') @@ -164,11 +163,17 @@ def configure_environment(): stream.write("\nsystem_role: " + resource_tags()['Role']) +def record_exit(playbook, status): + """ Saves exit status of playbook for notfication purposes""" + with open('/tmp/'+ playbook + 'playbook.status', 'w+') as stream: + stream.write(status) + + def execute(playbook): """ Downloads and executes a playbook. """ path = '/tmp/' + flat_path(playbook) download_from_s3(playbook + 'playbook.yml', path + 'playbook.yml') - call('ansible-playbook ' + path + 'playbook.yml', shell=True) + record_exit(playbook, call('ansible-playbook ' + path + 'playbook.yml', shell=True)) def ssh_keyscan(host): From c4cd93236864c4ca8b82599df87237ab0b3e55cf Mon Sep 17 00:00:00 2001 From: AsIf Date: Thu, 27 Aug 2015 10:14:13 -0700 Subject: [PATCH 10/17] fix spacing --- bootstrap.py | 1 - 1 file changed, 1 deletion(-) diff --git a/bootstrap.py b/bootstrap.py index 4d69b5e..9e7d88c 100644 --- a/bootstrap.py +++ b/bootstrap.py @@ -178,7 +178,6 @@ def execute(playbook): record_exit(playbook, call('ansible-playbook ' + path + filename, shell=True)) - def ssh_keyscan(host): """ Get the SSH host key from a remote server by connecting to it """ from paramiko import transport From 5c84c699b2115a9b385607ec588ab993603997cb Mon Sep 17 00:00:00 2001 From: AsIf Date: Thu, 27 Aug 2015 17:13:27 -0700 Subject: [PATCH 11/17] assigning exit_status to var to try and deal with futures --- bootstrap.py | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/bootstrap.py b/bootstrap.py index 9e7d88c..298d8d3 100644 --- a/bootstrap.py +++ b/bootstrap.py @@ -163,10 +163,10 @@ def configure_environment(): stream.write("\nsystem_role: " + resource_tags()['Role']) -def record_exit(playbook, status): +def record_exit(playbook, exit_status): """ Saves exit status of playbook for notfication purposes""" with open('/tmp/'+ playbook + 'playbook.status', 'w+') as stream: - stream.write(status) + stream.write(exit_status) def execute(playbook): @@ -175,8 +175,8 @@ def execute(playbook): for hook in ['pre-', '', 'post-']: filename = hook + 'playbook.yml' download_from_s3(playbook + filename, path + filename) - record_exit(playbook, call('ansible-playbook ' + path + filename, shell=True)) - + exit_status = call('ansible-playbook ' + path + filename, shell=True) + record_exit(playbook, exit_status) def ssh_keyscan(host): """ Get the SSH host key from a remote server by connecting to it """ From 8fb658006dd2ef3cfaab198dd9f24e9aeaa12cc6 Mon Sep 17 00:00:00 2001 From: AsIf Date: Thu, 27 Aug 2015 17:49:53 -0700 Subject: [PATCH 12/17] fix error not found. --- bootstrap.py | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/bootstrap.py b/bootstrap.py index 298d8d3..9ca7912 100644 --- a/bootstrap.py +++ b/bootstrap.py @@ -165,8 +165,9 @@ def configure_environment(): def record_exit(playbook, exit_status): """ Saves exit status of playbook for notfication purposes""" - with open('/tmp/'+ playbook + 'playbook.status', 'w+') as stream: - stream.write(exit_status) + playbook_name = flat_path('/tmp/' + playbook + '.status') + with open(playbook_name, 'w+') as stream: + stream.write(str(exit_status)) def execute(playbook): From a1f6df4079a5caf59ed54d78930be9897a4f4725 Mon Sep 17 00:00:00 2001 From: AsIf Date: Thu, 27 Aug 2015 18:07:16 -0700 Subject: [PATCH 13/17] IT WORKS --- bootstrap.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/bootstrap.py b/bootstrap.py index 9ca7912..1fccfdb 100644 --- a/bootstrap.py +++ b/bootstrap.py @@ -165,7 +165,7 @@ def configure_environment(): def record_exit(playbook, exit_status): """ Saves exit status of playbook for notfication purposes""" - playbook_name = flat_path('/tmp/' + playbook + '.status') + playbook_name = flat_path(playbook + 'playbook' + '.status') with open(playbook_name, 'w+') as stream: stream.write(str(exit_status)) From dc1beee16a83bc0afda931bcc930211fca38f700 Mon Sep 17 00:00:00 2001 From: Steven Harradine Date: Thu, 29 Oct 2015 12:07:38 -0400 Subject: [PATCH 14/17] write status to temp not to current working dir --- bootstrap.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/bootstrap.py b/bootstrap.py index 1fccfdb..b444741 100644 --- a/bootstrap.py +++ b/bootstrap.py @@ -165,7 +165,7 @@ def configure_environment(): def record_exit(playbook, exit_status): """ Saves exit status of playbook for notfication purposes""" - playbook_name = flat_path(playbook + 'playbook' + '.status') + playbook_name = '/tmp/' + flat_path(playbook + 'playbook' + '.status') with open(playbook_name, 'w+') as stream: stream.write(str(exit_status)) From 797a90a47335194aadfd48845308019615d2e217 Mon Sep 17 00:00:00 2001 From: Ben Visser Date: Thu, 7 Jan 2016 15:19:20 -0800 Subject: [PATCH 15/17] back to 0s infront of chmod permissions --- bootstrap.py | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/bootstrap.py b/bootstrap.py index 8290f44..b444741 100644 --- a/bootstrap.py +++ b/bootstrap.py @@ -224,7 +224,7 @@ def configure_ansible(): download_from_s3('ansible.cfg', '/etc/ansible/ansible.cfg') download_from_s3('vault.key', '/etc/ansible/vault.key') files = ['/etc/ansible/ansible.cfg', '/etc/ansible/vault.key'] - set_permissions(files, 400) + set_permissions(files, 0400) add_to_known_hosts(ssh_host_key('github.com')) add_to_known_hosts(ssh_host_key('bitbucket.org')) @@ -243,7 +243,7 @@ def get_credentials(): """ Fetches credentials needed for private repositories """ download_from_s3('ssh.ed25519', '/root/.ssh/id_ed25519') download_from_s3('ssh.rsa', '/root/.ssh/id_rsa') - set_permissions(['/root/.ssh/id_ed25519', '/root/.ssh/id_rsa'], 400) + set_permissions(['/root/.ssh/id_ed25519', '/root/.ssh/id_rsa'], 0400) def preconfigure(): @@ -253,7 +253,7 @@ def preconfigure(): configure_environment() get_credentials() download_from_s3('bin/reforge', '/usr/local/sbin/reforge') - set_permissions(['/usr/local/sbin/reforge'], 500) + set_permissions(['/usr/local/sbin/reforge'], 0500) def self_provision(): From c144e0352e0b0e256bc2afadf1cd1d84e63107f8 Mon Sep 17 00:00:00 2001 From: Aaron Pederson Date: Tue, 12 Jan 2016 11:03:55 -0800 Subject: [PATCH 16/17] Temp Fix; version pinned ansible installation to pre2.0 to allow functionality on forge. Will be amended next week. --- bootstrap.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/bootstrap.py b/bootstrap.py index b444741..5604269 100644 --- a/bootstrap.py +++ b/bootstrap.py @@ -248,7 +248,7 @@ def get_credentials(): def preconfigure(): """ Configure everything needed to configure everything else. """ - install_with_pip(['ansible', 'awscli', 'boto']) + install_with_pip(['ansible<2', 'awscli', 'boto']) configure_ansible() configure_environment() get_credentials() From bcbf73388e75ce8fd4a794ad6440274639207642 Mon Sep 17 00:00:00 2001 From: Chris Olstrom Date: Tue, 12 Jan 2016 12:24:54 -0800 Subject: [PATCH 17/17] Fix version pinning Because this uses `subprocess.call()`, the `<2` is read as "read STDIN from file `2`, which doesn't exist. This fixes that. --- bootstrap.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/bootstrap.py b/bootstrap.py index 5604269..697efe1 100644 --- a/bootstrap.py +++ b/bootstrap.py @@ -248,7 +248,7 @@ def get_credentials(): def preconfigure(): """ Configure everything needed to configure everything else. """ - install_with_pip(['ansible<2', 'awscli', 'boto']) + install_with_pip(['"ansible<2"', 'awscli', 'boto']) configure_ansible() configure_environment() get_credentials()