Send Amazon RDS Enhanced Monitoring process metrics to Google BigQuery

• Enhanced Monitoring must be enabled at the target RDS instance
• AWS credentials
  • which is authorized to invoke logs:getLogEvents
• GCP credential file
  • which is authorized to create/edit BigQuery table

bundle install --without development test

docker pull quay.io/dtan4/rdm2bq

bundle exec ./rdm2bq.rb \
  -l db-THISISANEXAMPLEDATABASEID0 \
  -p foobar-1234 \
  -c credentials.json \
  -d database_metrics \
  -t db_process_metrics

will send the latest process metrics...

• from: CloudWatch Log Event db-THISISANEXAMPLEDATABASEID0
• to: BigQuery table foobar-1234:database_metrics.db_process_metrics20180110
  • 20180110 is the YYYYmmdd-formatted current date in local timezone

⚠️ Default timezone in Docker container is UTC. If you want to change the timezone of BigQuery table suffix, environment variable TZ must be set.

docker run --rm -e TZ=Asia/Tokyo --name rdm2bq quay.io/dtan4/rdm2bq:latest \
  -l db-THISISANEXAMPLEDATABASEID0 \
  -p foobar-1234 \
  -c credentials.json \
  -d database_metrics \
  -t db_process_metrics

Sample CronJob manifest is here.

• Invoke every minutes
• Environment variables are read from the Secret dotenv
• GCP credential file is read from the Secret gcp-credentials/credentials.json
  • kubectl create secret generic gcp-credentials --from-file=credentials.json [-n NAMESPACE]

kubectl create -f kubernetes/cronjob.yaml [-n NAMESPACE]

Daisuke Fujita (@dtan4)