Channel.io moved CI from CircleCI to GitHub Actions, and they run the runners themselves. Jaehong Jung (he also goes by Jetty), a DevOps engineer there, published the English write-up on 18 and 19 March 2025. He says the original posts are on the Channel.io tech blog.
Part 1 is the cluster. Part 2 is what broke once real jobs ran. CI spend fell to about a third of the CircleCI level. He credits the self-hosted runners, Karpenter, and the cache work.
The CircleCI → GitHub Actions playbook is the config translation. Channel.io’s posts are the cluster you still have to operate after the YAML moves.
Why they left CircleCI
Repos were already on GitHub. They were looking at GitHub Enterprise and asked whether CI could live in the same place. Actions felt like a GitHub extension, with docs and Marketplace actions product engineers already know.
CircleCI outages were the other push. When CircleCI was down, Channel.io’s whole pipeline stopped. Jung does not claim Actions never goes down. He says a GitHub-wide incident would be bigger than CI, and the response priority would be higher.
They also wanted the bill down, and they wanted builds next to their own registries and caches inside AWS and Kubernetes. A SaaS CI vendor leaves you waiting when it fails. Runners they operate can be scaled the same day, and they can apply their own security rules.
The proof of concept started in Q2 of the year before those posts, so Q2 2024.
ARC, then Docker inside the pod
They did not plan a line-by-line CircleCI conversion and stop there. DevOps wanted a self-hosted runner cluster on Kubernetes. They used Actions Runner Controller, the operator from the GitHub Actions team.
Helm install and basic scale-out worked. minRunners and maxRunners did what they expected. A normal shell step ran. docker version failed with docker: command not found, because the runner is a container without a Docker client.
ARC can set containerMode to dind. That runs a Docker daemon as a sidecar and marks the pod privileged. Jung’s team already knew Dockerfiles and Docker caches. Kaniko would have avoided the privileged daemon, and it would have meant rewriting those Dockerfiles and proving the builds again. They stayed with DinD. Cluster-level access control was their reason for accepting the privileged pods for now.
He says part 2 would cover custom Docker daemon templates. The production pain landed elsewhere.
What broke in production
Runner pods disappeared in the middle of jobs, several times a week. The Actions log said it had received a shutdown signal.
The pods were BestEffort. Kubernetes evicts those first when a node is tight. Karpenter made it worse. Channel.io uses Karpenter to buy cheaper nodes and delete idle ones. It would replace a node while a job was still on it. The fix they used is Karpenter’s annotation karpenter.sh/do-not-disrupt: true on the runner pod, so that node stays until the job is done.
A cold start was slow if minRunners was 0. Jung measured about 1 minute 45 seconds for Karpenter to add a node, plus about 15 seconds to init the runner. They keep a minimum number of runners warm for the jobs that should not wait those two minutes.
They also split runner pools. A general pool covers ordinary builds. A desktop pool has more CPU and is amd64, for Electron and web builds that need that architecture. The Dropwizard API pool is memory-heavy and light on CPU. He says that one typically uses m6g.2xlarge.
Docker Hub was the outage that hurt during the main backend move. CircleCI had not hit this. The new runners sit in private subnets and leave through one NAT gateway. Docker Hub rate-limits anonymous pulls per IP, so every runner shared one bucket. Runner images, the DinD image, Postgres, Redis, and LocalStack all pulled through it.
A new NAT gateway only bought a new IP. Pointing every pipeline at ECR or GHCR would have meant editing pipelines, and third-party actions still pull Docker Hub on their own. They skipped ECR pull-through cache for those reasons. They also skipped a full Harbor install when all they needed was a mirror.
They run a registry that caches upstream images, and they point the DinD daemon at that mirror. He publishes two timings. Postgres, 435 MB, went from 19 seconds direct to 12.6 seconds through the mirror. LocalStack, 1.27 GB, went from 9.3 seconds to 4.75 seconds.
actions/cache was the last squeeze. Jung writes that GitHub caps that cache at 10 GB per repository. Some of their builds cache 700 MB to more than 1 GB, and a monorepo evicts those entries fast. Self-hosted runners also pay public-internet transfer to GitHub’s cache. They replaced it with a composite action that stores one compressed archive in S3, over a VPC endpoint. On their runners he saw peak downloads around 105 MB/s for actions/cache and around 320 MB/s for the S3 action.
What they gained, and what is still open
The bill is the number he leads with in the wrap-up. Overall CI cost is about one third of the old level. He ties that to usage-based runners, Karpenter, and the S3 cache.
The organizational change is not done. On CircleCI, DevOps wrote most pipelines. The point of Actions, for them, is that product engineers can write their own. That needs stable runners, written practices, and composite actions those engineers can call. He says that work is still ahead.
They talked about the same material at CHANNEL DEVTALK on 26 June 2025.
What I would copy
Budget an operator, not only a YAML port. Channel.io’s CircleCI outages were the reason to leave. The new failures were eviction, a two-minute cold start, a shared NAT IP, and a 10 GB cache.
If builds need Docker, decide DinD or a rootless builder before the first backend repo moves. They kept DinD because the rewrite cost was higher than the privileged pod, given the access control they already had.
Pin a do-not-disrupt rule if an autoscaler can delete the node under a job. Keep one warm runner where a two-minute wait is too slow.
Put a mirror in front of Docker Hub before every repo shares one NAT address.
The CircleCI → GitHub Actions playbook still applies to orbs, contexts, and requires becoming needs. Channel.io’s posts start after that, on the cluster.