Skip to content
Latchkey home
Documentation menu

Security architecture

How Latchkey isolates every CI job: dedicated single-use VMs, no inbound network, ephemeral credentials, encryption at rest and in transit, and the constraints the self-healing agent runs under.

job per virtual machine
1
destroyed after it finishes
inbound network rules
0
runners accept no connections
permitted sudo pattern
1
for the self-healing agent
build cache retention
14d
90 days for self-heal records

In short

  1. Every job gets its own EC2 virtual machine that runs exactly one job and is then destroyed, along with its encrypted disk.
  2. Runners reach the internet through their own public IPv4 address, behind a security group with zero inbound rules, so a runner accepts no inbound connection.
  3. Runner credentials are just-in-time and single-use (a GitHub Actions runner fetches its own once, through a short-lived link only that machine is given, then deletes it), and no runner carries an AWS instance role.
  4. The self-healing agent runs as a non-root user behind a one-entry sudo allowlist, and its AI tools execute in a default-deny sandbox.

This page is written for a security review. It describes how the runner platform is actually built - the isolation boundary, what leaves the machine, how credentials are handled, and what the self-healing agent is and is not allowed to do. For what the GitHub App can read and write in your repositories, see Security and GitHub permissions.

security · one job across three boundaries

One job crosses three trust boundaries. In GitHub, the job is queued and a signed webhook is sent. The Latchkey control plane verifies the webhook signature, launches a machine for this one job, mints a just-in-time config valid for one job, and stores it encrypted, readable only through a short-lived link given to that machine alone. The runner VM, which accepts no inbound connection, boots, reads the config once, deletes it, registers for exactly one job, runs the job alone, and is terminated with its encrypted disk deleted.

FIG. 7.3-AEvery crossing is authenticated and scoped to one job: the credential is read once and deleted, and the machine that used it is gone when the job ends.

Isolation: one job, one machine#

Latchkey does not share a machine between jobs. Each job runs on a dedicated EC2 virtual machine (m6a-class, Ubuntu 24.04 LTS) launched for that job. Your job is not a container sharing a kernel with another tenant, and it is not a process on a long-lived shared host.

Single-use is enforced at the GitHub registration layer, not just by convention. Cold-start runners register with a just-in-time configuration, which GitHub accepts for exactly one job. Warm-pool runners register with the --ephemeral flag, which does the same. In both cases the instance is configured to terminate on shutdown, so it cannot be reused even if something goes wrong.

BoundaryHow it is enforced
One job per machineGitHub just-in-time config (cold start) or --ephemeral registration (warm pool)
Machine cannot be reusedInstance-initiated shutdown behaviour is set to terminate, on both the launch template and the direct launch path
Disk does not surviveRoot EBS volume is encrypted and marked delete-on-termination
Runaway jobs are reapedA cleanup process terminates instances past the 4-hour runner lifetime (8-hour hard cap) on Latchkey on GH; a Latchkey CI or CLI job ends at its own time limit at the latest
Runners are bound to one workspaceInstances are tagged with the organization ID and registered to a single org runner group
01
Launch
A dedicated EC2 instance boots for one specific job
02
Register
Just-in-time config, valid for exactly one job
03
Read once
The runner fetches its credential and deletes it immediately
04
Run
Your job executes alone on the machine
05
Terminate
The instance shuts down; shutdown behaviour is terminate
06
Destroy
The encrypted root volume is deleted with the instance

Runner groups are created with visibility: selected and an explicit repository list, so a runner registered for your workspace is only offered jobs from the repositories you enabled.

What runs as root, and why that is contained#

Inside its own VM, your job has full control: the runner user has passwordless sudo and Docker access. That is normal for CI - builds install packages and run containers - and it is safe precisely because the blast radius is one short-lived machine with no inbound network that is destroyed at the end of the job. We would rather state this plainly than imply a sandbox that does not exist. The one place a sandbox does exist is a pull request from a fork on Latchkey CI: its steps run with no sudo, no Docker and a fixed network, as the Network section says. On GitHub Actions, a job of a pull request from a fork has the same full control as any other job, and it restores your dependency cache without saving to it and does not use your Docker layer cache.

Network#

What is locked down
  • A runner reaches the internet through its own public IPv4 address, used only for connections the runner opens and different for every machine, so there is no fixed address to allowlist
  • The runner security group has zero ingress rules and is the boundary - nothing on the internet can open a connection to a runner
  • IMDSv2 is required, with the hop limit set to 1 so a bridge-networked container cannot read the instance metadata through the host
  • No runner carries an AWS instance role, so the instance metadata service holds no AWS credentials for a job to take. Every credential a Latchkey CI runner holds is short-lived and minted by the control plane, either for the machine itself, naming no workspace, or for the one job it runs.
  • Traffic to S3 and DynamoDB in the runner's own region stays on VPC gateway endpoints; other AWS API calls, including S3 and DynamoDB in other regions, go from the runner's public address to each service's public endpoint
  • A WAF (rate limiting, AWS Common Rule Set, Known Bad Inputs) fronts the web applications
What your build can still reach
  • Outbound traffic is open, because CI has to reach npm, PyPI, Docker Hub, your cloud, and your own services
  • Treat any secret you expose to a job as reachable by that job - the same model as GitHub-hosted runners
  • The one exception is a pull request from a fork on Latchkey CI: its steps reach GitHub and a fixed set of public package registries, and nothing else

A runner created after our network change on November 3, 2026 does not send its traffic through our NAT gateways, and no flow log records its connections. What a step prints, including any host or address it names, can still be kept in the job's log and in the heal record. The NAT gateways carry our control plane's own traffic, and the traffic of a runner created before that change until that runner ends; they log the remote addresses those outbound connections reach and the bytes they carry, and the connection attempts made to the gateways from the internet, in totals of up to ten minutes with no packet content and no ports. The log is kept for 14 days and deleted within three days after that. Only our operators read it, to investigate network cost, and a copy an operator makes is deleted within 14 days.

Credentials and secrets#

01
Runner credentials are minted per job

A just-in-time runner configuration is generated for a single job and stored, encrypted, as a private object scoped to that installation and job. Only the machine launched for the job is given the signed link that reads it, and the link expires after 30 minutes.

02
The runner reads it once, then deletes it

On boot the instance fetches it once through that link and deletes it at once through a second one, because it contains key material; anything a failed delete leaves expires after a day. Warm-pool registration tokens are handed over and deleted the same way, and each is minted for its own machine and never cached.

03
GitHub access is short-lived

The GitHub App signs a JWT with its private key and exchanges it for a short-lived installation access token per operation. There is no long-lived GitHub token sitting on a runner.

04
Platform secrets are scoped away from runners

The GitHub App private key lives in Parameter Store as a SecureString. No runner carries an instance role, and no runner reads a platform secret from Secrets Manager or Parameter Store.

On Latchkey CI, a secret a step reads is hidden in the job's log wherever its value appears whole: as the value itself, as its standard base64 (padded, with or without a trailing newline, and as base64 wraps it at 76 columns), and as the body of its JSON string. The masking runs where the log is written, before any chunk is cut or uploaded, from the moment the first step that names the secret fetches it, and the same masking covers everything self-heal reads and sends off the runner. A value shorter than 28 bytes is never delivered, because it could not be hidden. What can still appear: one line of a multi-line value printed on its own, a part of a value, or a value in another encoding.

Deploys run on Latchkey CI and authenticate to AWS through OIDC role assumption with the job's Latchkey identity token, and the workflows that still run on GitHub Actions authenticate the same way with GitHub's token. There are no long-lived AWS access keys in CI. API keys for the MCP integration are stored as a SHA-256 hash only - the plaintext key is shown once at creation and never persisted.

Encryption#

DataAt restIn transit
Runner root diskEncrypted EBS (gp3), deleted on terminationn/a
Build cache (S3)SSE-S3 (AES-256), 14-day expiryTLS
Artifacts (S3)SSE-S3 (AES-256), 7-day expiryTLS
Application databaseEncrypted storage, not publicly accessibleTLS
Cache / queue layerEncryption at rest and in transit enabledTLS
App keysParameter Store SecureStringTLS
Runner configs and registration tokens (S3)SSE-S3 (AES-256), deleted on read, 1-day expiryTLS
API trafficn/aTLS 1.2 minimum on the API domain

Build cache isolation#

Cache objects are stored in S3 under a key prefix derived from your organization ID, and the tenant ID is injected by the control plane from the job request - a job cannot claim to be another tenant by setting an environment variable. Objects expire after 14 days. Within your organization, fewer jobs can write to the cache than can read it. On GitHub Actions, only a job of a run started by push, pull_request, schedule, workflow_dispatch or merge_group saves to your caches, and never one of a pull request from a fork or of a run Dependabot started or re-ran; every other job restores from them and cannot save. On Latchkey CI, a job saves dependency caches only to its own branch, pull request or tag, other branches read what the default branch saved, and a job of a pull request from a fork is served no cache.

What leaves the runner#

Two kinds of data leave the machine, and it is worth being precise about the second one.

  • Run and job metadata: names, timings, statuses, labels and the workflow file, which the dashboard and AI Insight are built from.
  • Failed step output. When a step fails, self-heal captures its standard output and error and keeps up to 1 MiB of each in the heal record for 90 days, so the failure can be diagnosed and, where relevant, turned into a proposed fix. When its AI stage runs, the end of that output goes to our AI model provider, with any of the job's files it reads. From a failed test step, the runner also sends the names and files of failing tests and the step's test counts, after the removals below, and Latchkey keeps them for 90 days. On Latchkey CI the names and files are kept to tell a flaky test from a broken one.

What the runner removes before self-heal sends or keeps output#

  • Values a step registers with ::add-mask:: (or the older ##[add-mask]) while it runs in bash, the default shell for a run: step, and, on GitHub Actions, values a JavaScript action registers with core.setSecret or ::add-mask:: before the failing step, shown as ***. Their base64 forms and their escaped forms (quoted, JSON, percent-encoded and XML) are removed too, and so is a piece of one cut off at the edge of the captured output, once that piece is 8 characters long.
  • Secrets in a known format, shown as [REDACTED:<type>]: private keys; GitHub, GitLab, Anthropic, OpenAI, Google, Stripe, AWS, Slack, SendGrid, npm, RubyGems, Square, Docker Hub, Shopify, DigitalOcean, NuGet, Mailgun and Vault tokens and keys; JWTs; Bearer, Token and Basic authorization values; the password in a connection string; an AWS secret access key on a command line; signed-URL signatures; Azure storage account keys; and Sentry DSN keys. Text that only looks like one of these, such as a test password in a connection string or a JWT-shaped test fixture, is removed too.
  • On Latchkey CI, the values of the secrets your job uses.
  • On GitHub Actions, the value of each secret the workflow file maps into the failing step's environment as one ${{ secrets.NAME }} reference in an env: entry, at the workflow, job or step level, with only fixed text around it, shown as ***. Its base64 and escaped forms are removed too.
  • If a step's output holds a registered value split across two fields of a JSON document, the whole document is removed.
  • A heal PR's content is never altered. Latchkey does not open the pull request when the proposed change contains one of the job's Latchkey CI secrets, or a value of 8 or more characters it hid with ::add-mask:: or that the workflow file maps from a secret into the step's env:, as written or in its base64 or escaped forms. The one exception is a value of fewer than 24 characters with no digit, such as production: as written, it blocks the pull request only where no ASCII letter, digit or underscore touches either end of it, and its encoded forms block it anywhere. Latchkey also does not open the pull request when a line the change writes holds one of these placeholders, unless that line holds more than placeholders and appears whole in a line of your workflow file.
  • If a job hides more than 10,000 values with ::add-mask::, or values too long for the runner to hold, or its output grows past the 64 MiB the runner hands self-heal, or the runner finds it lost a value a step hid, or masking the output fails, self-heal does not diagnose it: nothing is sent to our AI model provider or kept. On Latchkey CI, self-heal does not run on a step that starts once its job holds more than 64 values for the runner's heal service to hide (its secrets, its service settings of 28 bytes or more and the credentials Latchkey hands its steps), so nothing of that step is sent to our AI model provider or kept by self-heal; the job's log still hides every one of those values. On GitHub Actions it does not diagnose a failed step either when its workflow file, at its top level or in the failing job, uses a secret anywhere but as a plain env: mapping (in a run: script, a with: input, any other key but if:, or inside a larger expression), when a mapped value does not match the file, when the file calls a reusable workflow, when the failing step runs inside a composite action and the file maps a secret other than the GITHUB_TOKEN into env:, when a step listed before it in the job, or any step of the composite action it runs inside, is a container action, runs in a shell other than bash (shell: python, sh, pwsh or a custom shell such as node {0}), calls bash by path such as /usr/bin/bash, or uses an action whose file self-heal cannot read (a step listed before it counts even when its if: skipped it, a container action that runs a pre-entrypoint as the job starts counts wherever it is listed, and self-heal judges an action from the repository by its file at the commit the workflow ran from and an action from another repository by its file as it stands on the runner when the step fails), or when the runner cannot read the file whole as it ran: in a job with no checkout of the repository, or when the file is over 1 MiB, is not valid YAML or does not hold the job.
  • Not removed: a secret with no known format that a step prints without hiding it with ::add-mask::, when the step did not get it through the workflow file's env:, such as one an earlier step wrote to $GITHUB_ENV, a step output or a file; a value hidden only in a step that runs another shell (shell: python, sh, pwsh or a custom shell such as node {0}), calls /usr/bin/bash by path or is a container action (one that runs in its own Docker container), which is why self-heal does not diagnose a step that fails after a step it reads as one; a value hidden only by a JavaScript action on a runner that updated GitHub's runner software before running its job, which a runner does whenever GitHub has released a version newer than the one in the Latchkey runner image it started from (GitHub's own log still hides such a value); the GITHUB_TOKEN in an encoded form, unless the workflow file maps it into the step's env:; a value the runner can no longer find because a later step that runs another shell, calls /usr/bin/bash by path or is an action deleted the files it keeps hidden values in under the job's temporary directory, or deleted its notes of a value a full disk or a disk error lost; and a value printed only in another encoding, such as hex or URL-safe base64.

How the self-healing agent is constrained#

Self-healing is the part a reviewer should push hardest on, because it is the one component that changes a running job. Here is exactly what it can do.

  • It runs as the runner user, not root, as a systemd service.
  • It listens on a Unix domain socket with group-restricted permissions. It has no TCP listener and no network surface.
  • It is invoked only through a shim on the runner agent's own step scripts, and every other command passes straight through untouched, with two exceptions. On GitHub Actions, the shim also stands in for the Node.js the runner carries for JavaScript actions, where it keeps the ::add-mask:: commands an action prints and passes the action's output on byte for byte, without invoking the agent. And on every runner, before self-heal runs a step again, the shim copies the test reports and test output files the step left in the job's working directory, such as Playwright's screenshots and traces, into the job's temporary directory, where only the job's user can read them; when the agent runs the step again itself, it calls the shim for this as the job's user.
  • It fails open: if the agent times out, refuses, or errors, a failed command still fails. A broken healer cannot break your build.

The privileged operations it can perform are allowlisted, not filtered:

ControlWhat it means
Sudo allowlistExactly one anchored pattern is permitted - a non-interactive apt-get install of a single package. Anything else beginning with sudo is rejected before the process is spawned.
Package allowlistInstallable packages come from a compile-time list, enforced independently at two stages because one stage builds commands from untrusted build output.
Environment safety floorSetting LD_PRELOAD, LD_LIBRARY_PATH, PATH, BASH_ENV, or ENV is denied, as are shell metacharacters and command substitution.
Sensitive path denialReads under /etc, /root, /proc, /sys, /var/log, ~/.ssh, ~/.aws, and ~/.config/gh are denied.
AI tool sandboxStage-3 tools run inside a bubblewrap-based sandbox that starts default-deny - no network, no write paths - and is widened per tool to the narrowest set needed.
No resident AI keyOn Latchkey CI and GitHub Actions runners the reasoning step uses a model credential the control plane mints for that one heal, so no long-lived model API key sits on a machine that runs your code.
Kill switchSelf-healing can be turned off per workspace.

What the GitHub App can access#

The permissions below are grouped by what they are for, rather than listed flat, so you can see the reason next to each grant. The last two rows are permissions the app holds that no Latchkey feature uses today.

PurposeAccessPermissions
Run your jobs on Latchkey runnersRead and writeOrganization self-hosted runners
Read basic repository detailsReadMetadata
Read your runs and logs, dispatch a workflow when you or your coding agent asks, and cancel a run waiting on a repository you disabledRead and writeActions
Remove webhooks Latchkey created on a repositoryRead and writeRepository hooks
Read workflows, docs and files, read a pull request and the list of files it changes, and propose fixes as PRsRead and writeWorkflows, code, pull requests
Write Latchkey CI checks on your commitsRead and writeChecks
See each group your merge queue makes, so Latchkey CI can build the ones your workflow files run onReadMerge queues
Record your organization's GitHub plan and Actions minutes usedReadOrganization plan, organization administration
Check who is in your organizationReadMembers
Not used by any Latchkey feature todayRead and writeAdministration, organization hooks
Not used by any Latchkey feature todayReadArtifact metadata

Changes to your repositories#

Latchkey never pushes to your default branch. Every change is a pull request you review, and the write path is gated three ways:

  1. The file path must resolve as editable against a per-repository manifest. Unmatched paths default to not editable - the check fails closed.
  2. Pull requests originating from forks are rejected, and a missing fork signal is also rejected rather than assumed safe.
  3. A proposed fix must match its failure signature in the captured logs, or it is suppressed.

Operational security#

  • Runner images are rebuilt from the latest Ubuntu 24.04 image, with every pending OS update applied, when a Latchkey release changes what they hold. There is no fixed schedule, and a runner takes no OS update on its own while it runs.
  • No runner carries an AWS instance role. The only AWS session Latchkey issues to a runner is short-lived and covers the build cache and container images alone, so it cannot read any instance's tags or reach where runner credentials are stored, and no runner can read another runner's credentials.
  • Access to the dashboard and API is authenticated by a JWT authorizer at the API gateway. Workspace roles are Owner, Admin, and Member, with owners protected from removal or demotion.

Data retention#

What Latchkey stores, and for how long.

DataRetention
Your source code on the runnerDestroyed with the instance at the end of the job
Build cache objects14 days from last write
Artifacts a job uploads with latchkey artifact7 days from upload
Values a Latchkey CI job records under outputs:, for the jobs that need it in the run and in any re-run of itRetained until you delete the workspace
Job logs of Latchkey CI and CLI jobs7 days
A working tree latchkey run uploads to start a jobDeleted when the job ends, or by a 2-day expiry if that deletion fails. What self-heal reads of it stays in the heal record for 90 days
Failing-step output captured for diagnosis90 days
The first line of a failed step's command and up to its last 20 lines of output, when a workflow fails on Latchkey CI but passes on GitHub Actions at the same commit180 days
Self-heal attempt records90 days
Proposed fix records365 days
The names and files of failing tests in a failed test step, and that step's test counts90 days
How often a test that failed on Latchkey CI proved flaky, with its name and file90 days after the test last failed
The paths of the files a pull request changes90 days
Which of a pull request's jobs no change reaches90 days
A repository's docs and CI config, indexed for the knowledge baseKept while the repository is enabled, and removed when you disable it. If it stops being available to Latchkey another way, removed in the first weekly cleanup at least 90 days later, except in the cases set out below the table
Documents you upload to the knowledge baseRetained until you delete the document or the workspace
Run and job metadata powering your dashboardsRetained until you delete the workspace

Knowledge base documents indexed from a repository. A repository stops being available to Latchkey when it is removed from the GitHub App's access (which is also how Latchkey sees a repository deleted on GitHub) or when the GitHub App is uninstalled. Its 90 days run from the first such stop, and only giving the repository back to the GitHub App's access starts them again: a repository given back but not enabled again counts its 90 days from when it was given back, or from the last change Latchkey recorded for it, such as a rename or the GitHub App being uninstalled, if that is later. A suspension of the GitHub App does not by itself start the 90 days. A file deleted from the repository is removed in the first weekly cleanup at least 90 days after Latchkey next indexes the repository and finds it gone, or with the repository if that comes first. A document of a stopped repository, other than a file Latchkey found gone, is kept past its 90 days while an enabled repository in the same workspace has the name it was indexed under, letter case included, until Latchkey holds that enabled repository's own copy of the file or indexes that repository in full after the stopped one's 90 days began. Where Latchkey's records cannot show that a repository stopped, its documents are kept until you disable it or delete the workspace. Documents indexed before Latchkey recorded which repository each document came from are kept until you delete them or the workspace, or until you disable a repository while it has the name they were indexed under, letter case included. A stop that lands while Latchkey is indexing the repository can remove some of its documents at once, and leave a few written in those minutes to count their 90 days from when they were written.

Disabling a repository stops collection for it. Deleting your workspace removes its data. If you need a specific retention window for your organization, ask and we will talk it through.

Reporting a vulnerability#

Email security@latchkey.dev. Please do not open a public GitHub issue for a security report.

Q.01Does my source code stay on the runner after the job?

No. The instance and its encrypted root volume are destroyed when the job finishes. Nothing you checked out survives, other than build cache entries you explicitly opted into, which expire after 14 days, artifacts a job uploads with latchkey artifact, which expire 7 days after upload, the outputs a job records when it succeeds, which are kept until you delete the workspace, what self-heal read of your files, which the heal record keeps for 90 days, and the file contents of a fix it proposes, kept 365 days.

Q.02Can another customer's job see mine?

No. Each job runs on its own virtual machine, registered for a single workspace and destroyed after that one job. Runners have no inbound network access.

Q.03Can the self-healing agent run arbitrary commands on my build?

No. Privileged actions are allowlisted rather than filtered: a single permitted sudo pattern, a fixed package list, denial of environment variables that enable code injection, and a default-deny sandbox for the AI tool stage. If the agent fails for any reason, a failed command still fails.

Q.04Are my build secrets visible to Latchkey?

They can be. Like any runner, a Latchkey runner receives the GitHub Actions secrets your job references, injected by GitHub at runtime, and it is destroyed when the job ends. Latchkey never calls GitHub's API for their values. When self-heal diagnoses a failed step, the step's output is captured before GitHub masks it: the end of it goes to our AI model provider and up to 1 MiB of it is kept for 90 days. The runner first removes values a bash step hid with ::add-mask::, any secret in a known format and every secret the workflow file maps into the step's env:, and self-heal does not diagnose a step whose workflow file uses a secret any other way. So a secret with no known format is in both only when it reached the step some other way and the step printed it without hiding it. Known secret patterns are removed again when that output is served.