Deploy Live

Building Runner Dock in Public With Paperclip

Sat, 02 May 2026 21:00:00 +0000

tl;dr;

I had an itch to run GitHub Actions on top of Google Cloud infrastructure with minimal overhead. I’ve decided to build a solution in public and document my journey. I plan to build this product exclusively through AI and document what is working and what doesn’t. I gave it access to my Cloudflare account with my credit card on file. So far it picked and bought me a domain https://runnerdock.com/, created an interest form, and configured emails so people receive confirmation of their interest.

Running Local LLMs Offline on a Ten-Hour Flight

Wed, 22 Apr 2026 00:00:00 +0000

Update, 29 April 2026

This post was picked up on Hacker News and crossed 100 comments. I love that this sparks emotions. As always, there is a fair mix of reactions, which is exactly what you would expect from HN. Scroll to the bottom for extra responses to some of the comments.

…

Cloud Next was intense. I wrote the first version of this post at 9am my time while dozing off, and I did not expect much interest in it. Given the attention it is now getting, I gave the post the attention it deserved: corrected the model details, added more context about the setup, and responded to the most common points.

gcpiam.com: Instant Search for Google Cloud Permissions

Sat, 31 Jan 2026 15:00:00 +0000

This post extends on the Project honeypots.work kick-off and describes the first utility site instrumented to feed the honeypot data pipeline.

The Itch

Google Cloud Platform has thousands of IAM permissions. Finding the right one is tedious.

Google’s documentation works, but it requires multiple clicks to navigate. They have a permission reference with search, but it still feels slow when you need to quickly check which roles grant compute.instances.setMetadata or find all permissions containing bigquery.tables.

One-shot CRC16 in Helm. Here's Why It Matters.

Wed, 28 Jan 2026 12:00:00 +0000

I needed a CRC16 checksum implementation in Helm. This is not code I would have written before - it felt frivolous, not a good use of my time. But Claude Code gave me a working implementation with tests in one shot.

You might think this is another waste of valuable compute on a frivolous activity. Far from it - this implementation is driven by necessity and it simplifies our stack.

Project honeypots.work: kick-off

Tue, 06 Jan 2026 12:00:00 +0000

As part of my day job, I look after a top 5000 website in the world. We serve hundreds of terabytes of content monthly - and unsurprisingly, that traffic attracts plenty of bad actors, scrapers, and bots. They make engineers’ lives harder.

There are commercial solutions for bots and scrapers. But I want to build my own - specifically, a system that ranks inbound requests by their probability of being malicious.

Run, 2025, Run: A Year in Review

Wed, 31 Dec 2025 12:00:00 +0000

With 2025 wrapped up I’d like to look back and reflect on the year and more specifically, a set of personal goals I’ve set.

2025 was fast-paced, really challenging at times, but overall, a really great year to reflect back on.

Key takeaways:

I am the fittest version of myself, both physically and mentally
AI assisted coding has allowed me to make a lot of direct contributions across my teams, something that wouldn’t be feasible without AI given demands of the role

2025 Goals

At the start of the year, I’ve set myself a rather ambitious set of goals:

Claude in Chrome: First Wins

Tue, 23 Dec 2025 10:00:00 +0000

tl;dr;

Claude for Chrome saved me close to an hour. Three wins: Grafana panel overrides, ArgoCD finalizer patching, and Cloudflare Pages setup. Browser-integrated AI assistance is here and it works.

The Trigger

Just yesterday I read about Simon Willison’s success with Claude for Chrome. He was stuck with Cloudflare UI and Claude did a great job navigating it. Coincidentally, I had my own troubles with Cloudflare UI on the same day - but solved them the old school way. Clicking the same pages and navigation bars repeatedly until I arrived at insights that helped me configure custom DNS for my Pages.

GCP FinOps #003: Boring Predictability With BigQuery Slots

Tue, 23 Feb 2021 08:14:33 +0000

tl;dr;

BigQuery Slots offers predictable BigQuery analysis costs for those exceeding $2000 a month in on-demand queries.

GCP FinOps #002: Chaos Engineer Your Way to a Lower Cost

Wed, 17 Feb 2021 09:16:06 +0000

tl;dr;

Save money and build more reliable systems by running preemptible instances. Frequent failure generated by preemptible instances forces best practices around resilience.

GCP FinOps #001: Google Cloud Storage - Location, Location, Location

Thu, 11 Feb 2021 09:10:49 +0000

This post marks a start of a series about Google Cloud Platform and FinOps.

Visual Guide to Node Selectors, Taints and Tolerations

Mon, 28 Dec 2020 19:16:10 +0000

By default, Kubernetes schedules workloads (pods) onto nodes mainly driven by resource constraints (memory, CPU, ephemeral storage). This works fine for the vast majority of users where neither nodes nor pods have anything custom about them.

Now, imagine that subset of your nodes has a specific type of hardware, TPU, necessary for machine learning workloads. Only some of your workloads need access to TPU enabled nodes.

How can we schedule pods on the nodes that have specialized hardware?

While one might make TPUs ubiquitous and have them available on every node in the cluster, it will likely be wasteful and inefficient as not all nodes may be running applications that require TPUs.

CKS Certified Kubernetes Security Specialist Exam Preparation Guide

Sun, 11 Oct 2020 20:52:09 +0100

Certified Kubernetes Security Specialist (CKS) Preparation based on CNCF CKS curriculum.

Approach to this preparation guide

This guide is being authored before actual certification has went live. The best approach to be most prepared for this certification is to know all of the required domains and competences really well. I’ve based this preparation guide on the available curriculum. I do not aim to author 100% of the content for the CKS preparation, instead, I will aim to rely on existing resources where possible and only fill-in remaining gaps myself.

Serverless Compression of GCS Data With Streaming Golang Google Cloud Function

Sat, 03 Oct 2020 21:26:56 +0100

TL;DR

Compress data and use cold storage tiers to achieve absolute minimum cost for long-term logs archiving at as little as $1.20 per TB per year.

Problem

Imagine you need to capture and store 1TiB of logs monthly and retain them for 10 years - what is the most cost-effective way of doing so (infrastructure cost and long-term maintenance overheads)? Google Cloud produces a lot of audit logging and it is stored in Cloud Logging for free. They give you generous 400 days retention for Admin Activity and System Event logging and 30 days for Data Access. As of recently, they also support Custom retention which allows you to store logs upwards 10 years at a charge of $0.01 per GiB per month (from March 31, 2021). However, pricing stops being favourable when you ingest and retain regular, non-audit logs. Beyond the first 50GiB per project, the charge is hefty $0.50/GiB. We can use Cloud Logging export sinks to route logs directly into GCS, bypassing Cloud Logging’s $0.50 charge. GCS pricing can be found here where standard storage is twice as expensive as Cloud Logging retention, however, Coldline or colder storage classes provide price parity or even considerable savings.

Google Cloud Professional Machine Learning Engineer Certification Preparation Guide

Sun, 16 Aug 2020 20:04:04 +0100

Last updated on 11th of October, 2020. Certification is expected to go into GA on 15th of October.

Authors

Dmitri - knew GCP, didn’t know any of the ML stuff

Steven - knew ML, didn’t know any of the GCP stuff

Perfect reason to collaborate!

New certification

Google announced a new Machine Learninng Engineer beta certification in July with certifications taking place from 15th of July to 21st of August. Dmitri has attempted it on 16th of August. Most of the material below was written before the attempt and are a true reflection of a study guide, not an exam dump. Dmitri is a strong advocate of fair certifications therefore this guide is to motivate a wider audience to learn more about ML and GCP. Steven

Today I Learned: Adding labels to Prometheus queries

Sat, 16 May 2020 20:16:58 +0100

Problem

Recently, I’ve faced an interesting challenge around labels and Prometheus Recording Rules. I was writing a new Recording Rules and felt that labels available in the metrics composing recording rules were insufficient for my purposes of simplifying SLI, SLO and Error budget management with Prometheus. I’ve decided to find a way to add arbitrary labels to the prometheus queries and recording rules. I could not find any resource answering my question directly, so I hope my findings will be useful to others.

Monitoring Dataflow Memory

Fri, 17 Apr 2020 21:44:12 +0100

Hey Dmitri, I have explored options of looking into memory monitoring of dataflow. The thing is that we create dataflow jobs using terraform, and that does not have the –experiments option. Not sure what the option is there.

This is the message I’ve recently received from one of my colleagues who was looking into opportunities to scale down our Dataflow jobs. When you don’t know how much memory your Dataflow job is consuming, you will either overprovision and waste resources or underprovision and cause unnecessary job crashes. Dataflow documentation seems to avoid the topic of memory monitoring altogether, where the most relevant information is --experiments=enable_stackdriver_agent_metrics flag which allows to “monitor persistent disk, CPU, network, and process metrics from your Dataflow worker VM instances”.

Kubernetes Networking Problems Due to the Conntrack

Sat, 15 Feb 2020 20:27:39 +0000

*Photo by Lianhao Qu on Unsplash*

TL;DR

Kubernetes nodes set conntrack_max value proportionally to the size of the RAM on the node. High load applications (especially on small nodes) can easily exceed conntrack_max and result in connection resets and timeouts. There is several options to deal with this issue.

Theory

conntrack is a feature built on top of Netlifier framework. It is essential for performant complex networking of Kubernetes where nodes need to track connection information between thousands of pods and services.

Google Container Registry (GCR) per image permissions

Mon, 10 Feb 2020 21:13:51 +0000

*Photo by [Ricardo Gomez Angel](https://unsplash.com/@ripato) on Unsplash*

As of now, Google Container Registry does not allow for granular access control to the images stored in the registry. By default, you can only grant access to users, groups of users for the entire registry. In addition, you only get one registry per project. Therefore creating more registries to provide unique access permissions is also rather time-consuming.

For situations where you want to grant user or group an access to a specific docker image in GCR you can do the following:

When GKE ran out of IP addresses

Mon, 03 Feb 2020 21:49:01 +0000

*Photo by JOSHUA COLEMAN on Unsplash* ## Theory [RFC 1918](https://tools.ietf.org/html/rfc1918) defines 3 core ranges of IPs for private networking usage: 10.0.0.0/8, 172.16.0.0/12 and 192.168.0.0/16 yielding 16'777'216, 1'048'576 and 65'536 IPs each.

Setup

In our GKE environments, we use Shared VPC functionality to have a separation of concerns between networking and compute projects. This also allows for clearer networking architecture when common networking services need to be shared amongst a number of compute projects, for example, VPN tunnels, Network Peering and Cloud Interconnects. Additionally, GKE offers VPC-native clusters which come with several of scalability and security benefits. This feature allocates pod IPs from a subnet’s secondary range.

Today I Learned: Prometheus Recording Rules

Sun, 14 Jul 2019 14:08:13 +0100

Similarly to Reddit's [todayilearned](https://www.reddit.com/r/todayilearned/), I will be sharing my recent learnings about all things DevOps, cloud and distributed systems.

I’ve been using Prometheus for about six months, and it has been an instant success. However, over a time, the number of metrics stored in Prometheus has grown, and the frequency of querying has also increased. With more dashboards being added to the Grafana, I’ve started to experience situations where Grafana would not render a graph on time and Prometheus query will timeout. This can be very annoying, and refresh of the dashboard is an optimistic fix which will work sporadically, if at all. I needed a better way to fix Prometheus query timeout, especially when aggregating a vast number of metrics over long durations.

A Year With Gitops in Production

Sat, 06 Jul 2019 14:33:45 +0100

Do you know what is running in your production?

At this moment I know exactly what is running in my production, without even accessing it. Would you ask me the same question before the move to the GitOps - I’ll have to delete and reinstall all the VMs, redeploy application and finally answer with a similar level of confidence. This confidence would drift away with every lapsed minute as nearly all non-immutable, CIOps controlled environments are bound to deviate from the desired state. I attribute this drift between desired an actual state to the limitations of the Continuous Delivery tools of the past, where Continuous Integration tool is also a centre-piece to all the deployments. Changes are only captured within the CI tool. Your environment’s desired state of specific applications is expressed via button presses in the CI tool, but never actually verified as a whole. The entire environment is never checked against the desired state as the desired state is not defined in one single place. With these drawbacks of CIOps, I encourage you to move on to the CD tools made for the cloud-native era - GitOps.

42 Reasons to Fall in Love With GCP

Sat, 20 Apr 2019 10:55:28 +0000

By Kleuske - Own work, CC BY-SA 4.0, Link

Last year was a year of GCP for me. I have maintained a list of its unique, productive and performant features and products. While it is easy to find or produce computational benchmarks (e.g. CPU benchmark points per instance type or network throughput between two VMs) - real performance of any cloud is not in compute, but productivity. Those seeking global maximums for CPU speeds, network latency or storage IOPs will probably find it in their datacentres. However productivity of individuals, teams and organisations as a whole is something much harder to quantify and significantly more critical and valuable. Hardware is a commodity, therefore in the long run most cloud vendors will offer no significant advantage in CPU performance or amount of available RAM between them. Storage and networking are more open to interpretation. Therefore vendors will differ more in these domains.

Google Cloud Certified Professional Data Engineer - 2019 Updated exam

Fri, 05 Apr 2019 21:10:59 +0100

Starting 29th of March Google Cloud Professionals attempting Data Engineer certification will be subject to the latest version of the certification exam. The most recognisable difference is the removal of case studies. I was lucky enough to be challenged with and pass the updated version of the Data Engineer certification on the 2nd of April. In my preparation I’ve used materials and exam guide aimed at the v1 of the exam, hence being underprepared for the certification is an understatement. While Data Engineering is not something I’d consider to be a large part of my day-to-day toolkit, I do use BigQuery, Dataproc, Data Studio, GCS, PubSub, Datastore, Cloud SQL and related IAM. Therefore topics of Big Data and Machine Learning are fascinating, novel and challenging to me.

The shipwreck of GKE Cluster Upgrade

Mon, 25 Mar 2019 22:02:20 +0000

GKE Rocks!

I am a huge fan of the Google Cloud Platform and especially GKE. It was fundamental in our migration of Loveholidays on-prem applications to Kubernetes. GKE’s rich integrations with other GCP products allowed us to shift a lot of configuration from Ansible into Kubernetes yaml definitions which are now managed by GitOps. We are running GKE in Prod for 6+ months and will struggle to imagine any other way to run and deploy our applications now. The story I am about to tell you is a precaution on things that may go wrong during cluster/node upgrades. Its aim is to save you stress and worry next time you are in the same situation.

GKE Shared VPC - Add Node-Pool Subnet permissions

Fri, 22 Feb 2019 20:55:28 +0000

Problem

Recently when creating a Node Pool for GKE cluster running on Service project of the Shared VPC I’ve encountered the following problem:

gcloud container node-pools create new-node-pool \
--cluster=shared-vpc-cluster \
--disk-size=50GB \
--enable-autorepair \
--no-enable-autoupgrade \
--image-type=COS \
--enable-autoscaling  \
--local-ssd-count=0 \
--machine-type=n1-standard-1 \
--num-nodes=1 \
--min-nodes=0 \
--max-nodes=1 \
--region=europe-west2

Creating node pool new-node-pool...done.
ERROR: (gcloud.container.node-pools.create) Operation [<Operation
 clusterConditions: []
 detail: u"Google Compute Engine: Required 'compute.subnetworks.use' permission for
 'projects/222222222222/regions/europe-west2/subnetworks/service-project-subnet'."
 endTime: u'2019-02-22T16:23:15.489471514Z'
 name: u'operation-1550852586047-80520a63'
 nodepoolConditions: []
 operationType: OperationTypeValueValuesEnum(CREATE_NODE_POOL, 7)
 selfLink: u'https://container.googleapis.com/v1/projects/222222222222/locations/europe-west2/operations/operation-222222222222-xxxxxxx'
 startTime: u'2019-02-22T16:23:06.047764319Z'
 status: StatusValueValuesEnum(DONE, 3)
 statusMessage: u"Google Compute Engine: Required 'compute.subnetworks.use' permission for
  'projects/222222222222/regions/europe-west2/subnetworks/service-project-subnet'."
 targetLink: u'https://container.googleapis.com/v1/projects/222222222222/locations/europe-west2/clusters/shared-vpc-cluster/nodePools/new-node-pool'
 zone: u'europe-west2'>] finished with error: Google Compute Engine: Required
 'compute.subnetworks.use' permission for
 'projects/222222222222/regions/europe-west2/subnetworks/service-project-subnet'.

 _

Core issue is with permissions on the Service-Account in the Service Project.

finished with error: Google Compute Engine: Required
'compute.subnetworks.use' permission for
'projects/222222222222/regions/europe-west2/subnetworks/service-project-subnet'.

Redis Cluster Corrupted Cluster Config File

Wed, 20 Feb 2019 22:41:45 +0000

Problem

Recently I’ve been setting up Redis Cluster to run in Kubernetes on GKE. During the setup, I’ve been experimenting with various configurations of Redis Nodes and clustering. Unfortunately, at some point, I’ve rendered my cluster unusable. Logs showed the following:

1:C 20 Feb 2019 16:57:45.189 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
1:C 20 Feb 2019 16:57:45.189 # Redis version=5.0.3, bits=64, commit=00000000, modified=0, pid=1, just started
1:C 20 Feb 2019 16:57:45.189 # Configuration loaded
1:M 20 Feb 2019 16:57:45.192 # Unrecoverable error: corrupted cluster config file.

Google Professional Cloud Security Engineer Beta Exam walkthrough

Sat, 09 Feb 2019 19:15:39 +0000

On Thursday 24th of January, I woke up to great news. Firstly, I’ve become GCP certified “Professional Cloud Developer” by passing my 3rd certification and 2nd beta exam. Secondly, Google announced two new beta certifications. On the same day I’ve registered for both Google Cloud Certified - Professional Cloud Network Engineer and Google Cloud Certified - Professional Cloud Security Engineer exams.

I had an opportunity to sit Professional Cloud Security Engineer BETA on 9th of February and here is an outline of my journey.

Google Professional Cloud Network Engineer Beta Exam walkthrough

Wed, 30 Jan 2019 19:15:39 +0000

Edit: I’ve successfully passed this certification, landing #20 certification globally.

Solving Sensu's No Keepalive Sent

Sat, 01 Apr 2017 21:33:55 +0000

No keepalive sent from client for 120 seconds

Those using Sensu for monitoring and alerting have seen this dreadful message many times. Almost always it indicates that node your are monitoring (sensu client) has failed to send keepalive for preconfigured duration of time. In other words - there is no communication between sensu client and sensu server. This could indicate network or host issues on the client side or problems with network route between client and the server.

AWS S3 hosted blog using Jekyll and Ansible

Sat, 25 Mar 2017 19:15:39 +0000

I have been running a Gradle & Groovy related blog for an about a year using Wordpress. Wordpress is a great platform, but comes with its own risks and drawbacks:

More maintenance (needs a server to run)
Prone to hacking
Slower than static blog
Higher operational costs

I’ve decided to host a simple, static blog using S3 static site feature. I’ve also have chosen to use Jekyll as an engine to help me keep my static website maintainable.