# Security architecture

> How Latchkey isolates every CI job: dedicated single-use VMs, no inbound network, ephemeral credentials, encryption at rest and in transit, and the constraints the self-healing agent runs under.

Source: https://latchkey.dev/documentation/security-architecture

## Summary

- Every job gets its **own EC2 virtual machine** that runs exactly one job and is then destroyed, along with its encrypted disk.
- Runners sit in **private subnets with no public IP and zero inbound rules** - nothing can connect to them.
- Runner credentials are **just-in-time and single-use**, deleted from Parameter Store the moment the runner reads them.
- The self-healing agent runs **as a non-root user** behind a one-entry sudo allowlist, and its AI tools execute in a **default-deny sandbox**.

This page is written for a security review. It describes how the runner platform is actually built - the isolation boundary, what leaves the machine, how credentials are handled, and what the self-healing agent is and is not allowed to do. For what the GitHub App can read and write in your repositories, see [Security and GitHub permissions](/documentation/security-and-permissions).

## Isolation: one job, one machine

Latchkey does not share a machine between jobs. Each job runs on a dedicated EC2 virtual machine (m6a-class, Ubuntu 24.04 LTS) launched for that job. Your job is not a container sharing a kernel with another tenant, and it is not a process on a long-lived shared host.

Single-use is enforced at the GitHub registration layer, not just by convention. Cold-start runners register with a **just-in-time configuration**, which GitHub accepts for exactly one job. Warm-pool runners register with the `--ephemeral` flag, which does the same. In both cases the instance is configured to terminate on shutdown, so it cannot be reused even if something goes wrong.

| Boundary | How it is enforced |
| --- | --- |
| One job per machine | GitHub just-in-time config (cold start) or `--ephemeral` registration (warm pool) |
| Machine cannot be reused | Instance-initiated shutdown behaviour is set to terminate, on both the launch template and the direct launch path |
| Disk does not survive | Root EBS volume is encrypted and marked delete-on-termination |
| Runaway jobs are reaped | A cleanup process terminates instances past the 4-hour job limit (8-hour hard cap) |
| Runners are bound to one workspace | Instances are tagged with the organization ID and registered to a single org runner group |

Runner groups are created with `visibility: selected` and an explicit repository list, so a runner registered for your workspace is only offered jobs from the repositories you enabled.

> ****
> Your isolation boundary is the virtual machine: a dedicated EC2 instance per job, with its own kernel, its own encrypted disk, and no path to any other tenant. If your compliance regime has specific infrastructure requirements, talk to us before you onboard and we will work through them with you.

## What runs as root, and why that is contained

Inside its own VM, your job has full control: the `runner` user has passwordless sudo and Docker access. That is normal for CI - builds install packages and run containers - and it is safe precisely because the blast radius is one short-lived machine with no inbound network that is destroyed at the end of the job. We would rather state this plainly than imply a sandbox that does not exist.

## Network

## Credentials and secrets

1. **Runner credentials are minted per job** A just-in-time runner configuration is generated for a single job and written to AWS Systems Manager Parameter Store as an encrypted SecureString, scoped to that installation and job.
2. **The runner reads it once, then deletes it** On boot the instance fetches the parameter and immediately deletes it, because it contains key material. The same pattern applies to warm-pool registration tokens, which are cached only for their 55-minute usable life.
3. **GitHub access is short-lived** The GitHub App signs a JWT with its private key and exchanges it for a short-lived installation access token per operation. There is no long-lived GitHub token sitting on a runner.
4. **Platform secrets are scoped away from runners** The GitHub App private key lives in Parameter Store as a SecureString. The secret a runner can read is deliberately minimal and separate from the platform configuration secret, so a compromised instance role cannot read platform credentials.

Deploys authenticate to AWS through GitHub OIDC role assumption. There are no long-lived AWS access keys in CI. API keys for the MCP integration are stored as a SHA-256 hash only - the plaintext key is shown once at creation and never persisted.

## Encryption

| Data | At rest | In transit |
| --- | --- | --- |
| Runner root disk | Encrypted EBS (gp3), deleted on termination | n/a |
| Build cache (S3) | SSE-S3 (AES-256), 14-day expiry | TLS |
| Application database | Encrypted storage, not publicly accessible | TLS |
| Cache / queue layer | Encryption at rest and in transit enabled | TLS |
| Runner configs and app keys | Parameter Store SecureString | TLS |
| API traffic | n/a | TLS 1.2 minimum on the API domain |

## Build cache isolation

Cache objects are stored in S3 under a key prefix derived from your organization ID, and the tenant ID is injected by the control plane from the job request - a job cannot claim to be another tenant by setting an environment variable. Objects expire after 14 days.

> ****
> Every cache read and write passes through the cache proxy, which resolves the tenant prefix from the workspace the control plane assigned to that runner - not from anything the job supplies. Cache objects are namespaced per workspace and expire after 14 days.

## What leaves the runner

Two kinds of data leave the machine, and it is worth being precise about the second one.

- **Run and job metadata** - names, timings, statuses, labels, and the workflow YAML - which powers the analytics and optimization features.
- **Failing step output.** When a step fails, the self-healing pipeline captures that step's standard output and error (capped at 1 MiB each) and stores it so the failure can be diagnosed and, where relevant, turned into a proposed fix. Retention is 90 days.

> ****
> Only the failing step's output is captured, and only when a step fails. GitHub Actions masks your registered secrets in log output before Latchkey ever sees it, and secrets are redacted again when logs are served into an escalation bundle. If you want capture scoped differently for your workspace, tell us and we will set that up.

## How the self-healing agent is constrained

Self-healing is the part a reviewer should push hardest on, because it is the one component that changes a running job. Here is exactly what it can do.

- It runs as the **`runner` user, not root**, as a systemd service.
- It listens on a **Unix domain socket** with group-restricted permissions. It has no TCP listener and no network surface.
- It is invoked through a shim that **only** intercepts the runner agent's own step scripts; every other command passes straight through untouched.
- It **fails open**: if the agent times out, refuses, or errors, the command runs exactly as it would have without Latchkey. A broken healer cannot break your build.

The privileged operations it can perform are allowlisted, not filtered:

| Control | What it means |
| --- | --- |
| Sudo allowlist | Exactly one anchored pattern is permitted - a non-interactive `apt-get install` of a single package. Anything else beginning with `sudo` is rejected before the process is spawned. |
| Package allowlist | Installable packages come from a compile-time list, enforced independently at two stages because one stage builds commands from untrusted build output. |
| Environment safety floor | Setting `LD_PRELOAD`, `LD_LIBRARY_PATH`, `PATH`, `BASH_ENV`, or `ENV` is denied, as are shell metacharacters and command substitution. |
| Sensitive path denial | Reads under `/etc`, `/root`, `/proc`, `/sys`, `/var/log`, `~/.ssh`, `~/.aws`, and `~/.config/gh` are denied. |
| AI tool sandbox | Stage-3 tools run inside a bubblewrap-based sandbox that starts default-deny - no network, no write paths - and is widened per tool to the narrowest set needed. |
| No resident AI key | The reasoning step authenticates to AWS Bedrock through the instance role, so there is no long-lived model API key sitting on a machine that runs your code. |
| Kill switch | Self-healing can be turned off per workspace. |

> ****
> The simple heal actions - set an environment variable, install an allowlisted package, free disk, retry - run through the executor, where the allowlists above are the enforcement boundary. The AI reasoning stage runs additionally inside the default-deny sandbox.

## What the GitHub App can access

Every permission below exists to power a specific feature. Grouped by what it is for, rather than as a flat list, so you can see the reason next to the grant.

| Purpose | Access | Permissions |
| --- | --- | --- |
| Run your jobs on Latchkey runners | Read and write | Organization self-hosted runners, administration, organization administration |
| See what your pipelines did | Read | Actions, artifact metadata, metadata |
| React to jobs as they happen | Read and write | Organization hooks, repository hooks |
| Read workflows and propose fixes as PRs | Read and write | Workflows, code, pull requests |
| Model your GitHub costs accurately | Read | Organization plan |
| Check who is in your organization | Read | Members |
| Surface security findings in your dashboard | Read | Dependabot alerts, secret scanning alerts, security events, Dependabot secrets |
| Build runner images matched to your stack | Read | Organization runner custom images |

> ****
> Two of these deserve a plain explanation. **Code** write access exists so a fix can be opened as a pull request on a branch - it is never used to push to your default branch, and the write path is gated three ways (below). **Code** read access lets the diagnosis step look at build manifests such as `package.json`, `go.mod`, or your Dockerfile when working out why a step failed. Latchkey does not request access to your Actions secret values, and GitHub does not expose them to apps.

## Changes to your repositories

Latchkey never pushes to your default branch. Every change is a pull request you review, and the write path is gated three ways:

1. The file path must resolve as editable against a per-repository manifest. Unmatched paths default to **not editable** - the check fails closed.
2. Pull requests originating from forks are rejected, and a missing fork signal is also rejected rather than assumed safe.
3. A proposed fix must match its failure signature in the captured logs, or it is suppressed.

## Operational security

- Runner images are **rebuilt weekly** so OS security patches land on the fleet on a schedule.
- The runner instance role is least-privilege, including an explicit deny that narrows an AWS-managed policy's account-wide Parameter Store access down to Latchkey's own runner paths.
- Every mutating administrative action is written to an **audit log** with operator, action, target, parameters, and result.
- Access to the dashboard and API is authenticated by a JWT authorizer at the API gateway. Workspace roles are Owner, Admin, and Member, with owners protected from removal or demotion.

## Data retention

What Latchkey stores, and for how long.

| Data | Retention |
| --- | --- |
| Your source code on the runner | Destroyed with the instance at the end of the job |
| Build cache objects | 14 days from last write |
| Failing-step output captured for diagnosis | 90 days |
| Self-heal attempt records | 90 days |
| Proposed fix records | 365 days |
| Run and job metadata powering your dashboards | Retained while the repository stays enabled |

Disabling a repository stops collection for it. Deleting your workspace removes its data. If you need a specific retention window for your organization, ask and we will talk it through.

## Reporting a vulnerability

Email **security@latchkey.dev**. Please do not open a public GitHub issue for a security report.

### Does my source code stay on the runner after the job?

No. The instance and its encrypted root volume are destroyed when the job finishes. Nothing you checked out survives, other than build cache entries you explicitly opted into, which expire after 14 days.

### Can another customer's job see mine?

No. Each job runs on its own virtual machine, registered for a single workspace and destroyed after that one job. Runners have no inbound network access.

### Can the self-healing agent run arbitrary commands on my build?

No. Privileged actions are allowlisted rather than filtered: a single permitted sudo pattern, a fixed package list, denial of environment variables that enable code injection, and a default-deny sandbox for the AI tool stage. If the agent fails for any reason, your command runs exactly as it would have without it.

### Are my build secrets visible to Latchkey?

No. Your GitHub Actions secrets are injected by GitHub directly into your job at runtime, exactly as on GitHub-hosted runners, and Latchkey never requests or stores their values. GitHub masks registered secrets in log output before any of it reaches us.

---

Latchkey runs CI/CD that repairs its own failures. Agent entry points: https://latchkey.dev/agent.txt, https://latchkey.dev/openapi.json, https://latchkey.dev/llms.txt