Scaling GitHub Actions: Implementing On-Demand Runners with KEDA

Running self-hosted GitHub Actions runners can significantly reduce CI/CD costs and provide better control over your build environment. However, maintaining a fixed pool of runners often means either over-provisioning resources or facing pipeline delays. Here's how we implemented auto-scaling runners using KEDA (Kubernetes Event-driven Autoscaling) in our cluster.

Why KEDA?

Unlike the Actions Runner Controller (ARC), which only handles GitHub workloads, KEDA provides a unified scaling solution that can manage GitHub runners alongside other event-driven systems through a single control plane. This reduces operational complexity while enabling future integrations.

Benefits

• Zero-Cost Idle: No resources consumed when workflows aren't running
• Automatic Scaling: Runners scale from 0-N based on workflow demand
• Enhanced Security: GitHub App auth, and ephemeral runners reduce attack surface
• Simplified Management: No overhead from managing static runner pools

Prerequisites

• Kubernetes cluster
• KEDA installed (v2.10 or higher)

Implementation Steps

1. Setting up the GitHub App

We'll use a GitHub App for authentication, providing better security and higher rate limits.

1. Create a new GitHub App:
   • Go to 'Developer Settings' → 'GitHub Apps' → 'New GitHub App'
   • Disable Webhooks
   • Set any Homepage URL
   • Permissions needed:
     • Actions: Read-only
     • Metadata: Read-only
     • Self-hosted Runners: Read & write
   • Installation: Only on this account
2. Post-creation steps:
   • Generate and download the private key
   • Note the App ID
   • Install the app in your organization
   • Grant access to all repositories
   • Note the Installation ID from the URL (format: /installations/YOUR-ID)

2. Deploy Runner Resources

To schedule our runners, we need to create three resources in our cluster.

These are:

• Secret: Stores GitHub App credentials (ID, installation ID, private key) for secure authentication
• Trigger Authentication: Links KEDA to the GitHub App credentials for API authentication
• Scaled Job: Single-run, ephemeral pod template that KEDA creates in response to GitHub workflow triggers

These three resources are all we need to scale our runners with KEDA when an event is ingested from GitHub.

A. Create the secret

First, create the secret with the following command:

kubectl create secret github-auth --from-literal=appID=YOUR_APP_ID --from-literal --from-file=appKey=DOWNLOADED_CERTIFICATE.pem

Ensure you replace and insert the app ID and app key you retrieved from the earlier steps.

B. Create the Trigger Authentication

We can create our trigger authentication by applying the following YAML to our namespace:

apiVersion: keda.sh/v1alpha1
kind: TriggerAuthentication
metadata:
  name: github-trigger-auth
spec:
  secretTargetRef:
    - parameter: appKey
      name: github-auth
      key: appKey

Apply with:

kubectl apply -f triggerauthentication.yaml

C. Create the Scaled Job

The scaled job is what defines our github runner. This is the most critical resource of the three and has lots of configuration options. For the full breakdown, I recommend you read the Keda documentation on the GitHub scaler: https://keda.sh/docs/2.16/scalers/github-runner/

apiVersion: keda.sh/v1alpha1
kind: ScaledJob
metadata:
  name: scaledjob-github-runner
spec:
  jobTargetRef:
    template:
      metadata:
        labels:
          app: scaledjob-github-runner
      spec:
        containers:
        - name: scaledjob-github-runner
          image: myoung34/github-runner:2.321.0-ubuntu-focal # Runner image
          imagePullPolicy: Always
          env:
          - name: EPHEMERAL
            value: "true"
          - name: DISABLE_AUTO_UPDATE
            value: "true"
          - name: RUNNER_SCOPE
            value: "org"
          - name: ORG_NAME
            value: "YOUR-ORG-NAME"
          - name: LABELS
            value: "k8s-runner"
          - name: APP_ID
            valueFrom:
              secretKeyRef:
                name: github-auth # References our secret
                key: appID
          - name: APP_PRIVATE_KEY
            valueFrom:
              secretKeyRef: 
                name: github-auth
                key: appKey
        restartPolicy: Never
  minReplicaCount: 0
  maxReplicaCount: 5 # Set the max number of active runners
  pollingInterval: 20
  triggers:
  - type: github-runner
    metadata:
      owner: "YOUR-ORG-NAME"
      labelsFromEnv: "LABELS"
      runnerScopeFromEnv: "RUNNER_SCOPE"
      applicationID: "YOUR-APP-ID"
      installationID: "YOUR-INSTALLATION-ID"
    authenticationRef:
     name: github-trigger-auth # This is the trigger authentication

Change the following from the above YAML:

• YOUR-ORG-NAME: This should be the name of your GitHub organisation
• YOUR-APP-ID: The app ID we collected in the GitHub app installation steps.
• YOUR-INSTALLATION-ID: The installation ID we retrieved earlier from the GitHub URL.
• Optional: Customise the 'k8s-runner' label if desired.

Please note that values in the trigger metadata with keys like 'fromEnv' refer to environment variables and, therefore, do not need to be adjusted.

3. Configure a workflow

Now we have this configured, and we should be able to configure a GitHub Actions workflow to use our runners.

To do this, we need to configure the jobs in our pipeline with the label we set in the scaled job (This is set to k8s-runner unless you adjusted it in the previous step).

Our github actions YAML should look something like this:

jobs:
  terraform:
    runs-on:
      labels: [k8s-runner] 

This tells GitHub to run this job only on runners with the corresponding label. We can now trigger our workflow and see it initialising on our scaled runner:

We can also watch the pods in our namespace to see the job scale-up:

kubectl get pods -n RUNNER-NAMESPACE -w

Conclusion

KEDA-based GitHub runners provide an efficient, secure, and scalable solution for CI/CD workflows. By automatically scaling runners based on demand, we've eliminated the traditional trade-off between resource costs and pipeline performance. The runners' ephemeral nature ensures consistent environments while maintaining security, making this approach ideal for organizations looking to optimize their CI/CD workflow.