Skip to content
Latchkey home
Documentation menu

Troubleshooting

Symptom-by-symptom fixes: jobs stuck in queued, an empty dashboard, missing insights, blocked sign-in, and more.

In short

  1. A job stuck in queued = label, monitoring, config, billing, image build, or concurrency; check in that order.
  2. An empty dashboard is almost always filters, or no run yet, not lost data.
  3. Every blocked state has an explicit tell: a banner, the notification bell, or the activation screen.
app.latchkey.dev / acme / overview

The sidebar head of the acme workspace, the notification bell showing 2 unread. On the bell's next poll the count turns to 3 and the bell opens its panel: Notifications, 3 unread, with Mark all read. The newest notice is on top, Your latchkey runner minutes are used up, just now; below it Pipeline failed on main in acme/api, 3h ago, and Managed runner blocked, repository not enabled, for acme/docs, 6h ago, all three unread, then two read notices, latchkey runner minutes at 80%, 1d ago, and 5 new insights found in your CI, 2d ago. The foot reads Showing the last 7 days, Manage preferences in Settings.

FIG. 7.1-AA blocked runner, a used-up allowance or a failing pipeline lands in the bell, which checks for new notices every 30 seconds. A notice keeps its lime edge until you open it, and the panel keeps the last 7 days.

Each section below is one symptom, with its causes in the order they are most worth checking. Most problems announce themselves somewhere: a banner in the dashboard, a notification in the bell, or a status in a modal. This table tells you where to look first.

SymptomFirst checkWhere
Job targeting a runner stays queuedThe exact label stringThe "Your runners" table on the Runners page
Dashboard shows no dataA repository is selectedThe Filters on Command & Control
No AI insightsThe nightly scan and the analysis branchThe AI Insight page and Settings, Code hosts, Enrol repositories
Runner launches blockedTrial, subscription, and free-tier stateThe Billing page and the "Managed runner blocked" or "Free tier running out" notification
Access suddenly blockedSubscription status (trial ended, Past Due, Canceled)The activation screen or the Billing page

A job targeting a Latchkey runner stays queued#

This is the universal symptom for anything that stops a runner from being provisioned. GitHub keeps a job in "queued" until a runner with a matching label picks it up, and when Latchkey blocks or cannot route a launch there is no error on GitHub's side; the job just waits. That means the diagnosis always happens on the Latchkey side. For how provisioning decides between a warm pickup and a cold start when nothing is wrong, see How provisioning works. Triage in this order:

01
Label right?
Exact preset or custom label, no typos
02
Repo monitored?
Check Settings, Code hosts, Enrol repositories
03
Config enabled?
A disabled configuration stops accepting jobs
04
Billing OK?
Trial, subscription, and free tier state
05
Image ready?
Custom runner builds take time
06
At the limit?
Every workspace starts at 40 concurrent jobs, the Launch number
01
Rule out a label typo

The label must be exactly latchkey-small, latchkey-medium, latchkey-large, latchkey-xlarge, or one of your custom labels. How to confirm: the "Your runners" table on the Runners page shows the exact strings; compare against your workflow file character by character (a stray space or a wrong hyphen is enough to miss). Both label forms route identically, so runs-on: latchkey-small and runs-on: [self-hosted, latchkey-small] behave the same; the array form is not the problem.

02
Confirm the repository is monitored

Managed runners only serve monitored repositories, so a job from an unmonitored repo waits forever. How to confirm: open Enrol repositories on the Code hosts tab in Settings (owners and admins) and check the repository's toggle. If you are at your plan's repository limit, further toggles are blocked and the modal points you to upgrade; see Managing repositories. For a Latchkey CI job, see what each failure_reason means.

03
Check that the runner configuration is enabled

A disabled configuration stops accepting new jobs, and jobs targeting its label queue. How to confirm: the configuration's row on the Runners page shows its enable/disable state, for the presets and for custom runners alike.

04
Check trial, subscription, and free-tier state

An expired trial, a lapsed (past due or canceled) subscription, or an exhausted free tier on a card-less trial all block new runner launches; an exhausted free tier also stops running jobs and holds new ones until a plan starts. How to confirm: look in the notification bell for a "Managed runner blocked" notification (an expired trial or a lapsed subscription) or the month's "Free tier running out" notification (an exhausted free tier), then open the Billing page: the trial notice and the free-tier meter, or the activation screen a past due or canceled subscription lands on, tell you which case you are in. Subscribing, or fixing the payment method in the Stripe portal, restores launches at the next dispatch, usually within a minute; see Runner usage and free minutes. For a Latchkey CI job, see what each failure_reason means.

05
Check whether a custom image is still building

Jobs targeting a new custom configuration wait until its image finishes building. How to confirm: the configuration row on the Runners page shows image build status, and you are notified when the runner is ready ("AI scan runner is ready") or if the build failed. See Custom runners.

06
Check the concurrency ceiling

If your workspace's concurrent-job limit is already in use (every workspace starts at 40, the Launch number; idle warm and parked runners hold no slot), further jobs wait for a slot. How to confirm: this shows up when many workflows fire at once across your repositories, and the queued jobs start on their own as running jobs finish. Higher limits are available; see Limits and concurrency.

Find the cause, one check at a time
Answer each check in order. Answering No shows that cause and its fix.
  1. 01
    Does the job's label exactly match one in the "Your runners" table? Yes
    Exact preset or custom label, no typos
    If not: Rule out a label typo
  2. 02
    Is the repository monitored? Yes
    Check Settings, Code hosts, Enrol repositories
    If not: Confirm the repository is monitored
  3. 03
    Is the runner configuration enabled? Yes
    A disabled configuration stops accepting jobs
    If not: Check that the runner configuration is enabled
  4. 04
    Is the bell free of "Managed runner blocked" and "Free tier running out" notifications? Yes
    Trial, subscription, and free tier state
    If not: Check trial, subscription, and free-tier state
  5. 05
    If the job targets a custom runner, has its image finished building? Yes
    Custom runner builds take time
    If not: Check whether a custom image is still building
  6. 06
    Is a concurrent-job slot free? Yes
    Every workspace starts at 40 concurrent jobs, the Launch number
    If not: Check the concurrency ceiling

The dashboard shows no data#

An empty dashboard almost always means filters or timing, not lost data. Check in this order:

  1. No repository selected. Command & Control draws the repositories selected in its Filters. How to confirm: with no repository selected, the workflows select is disabled and reads "Select a repository first". Reset filters (Reset in Easy) selects every repository and workflow again and sets the Date range window back to the last 30 days.
  2. No runs in the window. Pages draw only what Latchkey ran, and the Date range window must overlap it. How to confirm: the Runs page with a wider window shows the last run Latchkey served.
  3. App installed on the wrong org. If the GitHub App is not installed on the organization that owns the repositories, no events arrive at all. How to confirm: on GitHub, open the organization's Settings, then GitHub Apps, and check that Latchkey is listed there. Fix by installing on the right organization; see Install the GitHub App.

No AI insights are appearing#

Insight generation is automatic; there is no button to press. Deterministic detectors run nightly over the data Latchkey already stores, and once shortly after you enable a repository; a few cost detectors also run hourly. So "no insights" usually means the nightly scan has not run yet, or the scan is looking at a different branch than you are:

  • Analysis still running or not yet triggered. Insights generate automatically after you enable a repository. How to confirm: the AI Insight page reads "Refreshed nightly", and with nothing open it reads "Nothing needs your yes. Latchkey checks nightly."; give a new repository a few minutes for findings read from its workflow files, since cost and reliability findings follow the runs Latchkey serves; the nightly scan is the backstop.
  • Wrong analysis branch. Analysis follows the analysis branch chosen in Settings, Code hosts, Enrol repositories (the repository default branch by default); workflows that exist only on other branches are not analyzed. How to confirm: open Settings, Code hosts, Enrol repositories (owners and admins) and check Analysis branch under the repository; the dialog draws it for each repository already monitored. See Managing repositories.
  • Legitimately little to say. Small or rarely run workflows can produce few recommendations; that is a finding, not a failure. Recommendations come from the nightly scan, so check back after the nightly scan that follows your next workflow change.

Applying recommendations failed or dropped changes#

Recommendations are generated against the workflow file as it existed at analysis time. If the file has changed since (your edits, or a previously merged Latchkey PR), some recommendations no longer apply cleanly:

  • Latchkey opens the PR with what still fits and lists the recommendations that need manual placement. The PR arrives titled "Latchkey insight: <what it does>" (for example "Latchkey insight: pin GitHub Actions to commit SHAs") on a latchkey/insight-<detector>-<hex> branch, and the card's PR #N opened link, with a toast carrying View PR, takes you straight to it.
  • If all selected recommendations were dropped, do not retry the stale set. The next nightly scan reads the edited workflow; wait for it (the AI Insight page refreshes nightly), then apply from the fresh set. See AI Insight.

A payment failed and access is blocked#

A Past Due status means a payment failed: dashboard and API access are blocked and runner launches stop until the payment method is fixed in the Stripe portal, where payment methods, invoices (with PDFs), and subscription details are managed. If the subscription was Canceled instead, access is blocked but billing remains reachable so you can re-subscribe. Your data is retained in both cases. Statuses and their effects are listed in Managing your subscription.

I cannot use the dashboard on my phone#

That is by design for now: the dashboard needs a tablet, laptop or desktop screen, and a screen narrower than 768 px shows a notice after sign-in that points to a larger screen. Notifications by email, Slack, and push, plus the weekly digest, keep you covered away from your desk.

I installed the app on my personal account#

That installation is connected, and the account runs Latchkey CI on the repositories you enrol from the dashboard. A GitHub Actions job from an enrolled repository that targets a latchkey-* label does not run on a Latchkey runner, and Latchkey says why: "Latchkey on GH needs a GitHub organization. This personal account runs Latchkey CI: commit a workflow file under .latchkey/workflows/ and push." Workflow files covers the file. To use Latchkey on GH (Latchkey's managed runners in GitHub Actions) with a repository in a personal account, move it into an organization as Install the GitHub App shows.

Connecting GitHub did not finish#

When the GitHub step of setup does not finish, Latchkey brings you back to the Get started card and says why in a note above the Connect GitHub button, which keeps its label. Only a failed installation is fixed by connecting again; the other cases need an invite, more seats, or the login linked to that GitHub account.

app.latchkey.dev / acme / overview

The Get started card on Command & Control shows a note above the Connect GitHub button, which keeps its label, and the note changes with the error the dashboard's address names. onboarding_error=install_failed: Latchkey could not finish connecting GitHub. Try connecting again. onboarding_error=org_already_connected: This GitHub organization is already connected to a Latchkey workspace. Ask an owner or admin of that workspace to invite you. onboarding_error=github_account_already_connected: This GitHub account is already connected to another Latchkey workspace. Ask an owner or admin of that workspace to invite you. onboarding_error=seat_limit_reached: This workspace has reached its seat limit. Ask an owner or admin to add more seats. onboarding_error=github_account_not_linked_here: This personal GitHub account can be connected only by the login linked to it. Link it to this login from Settings, or sign in to Latchkey with that GitHub account, then connect it again.

FIG. 7.1-BThe note changes with the case the dashboard's address names in onboarding_error, so you can quote that word when you ask for help.

My trial ended and everything is blocked#

Your data is intact; the block is on access, not on storage. The dashboard is replaced by a "Your free trial has ended" screen, API access is blocked with the same message, and managed runner launches are blocked (jobs targeting latchkey-* labels stay queued on GitHub rather than erroring). Subscribe from the activation screen (or the Billing page if you can reach it) and access, including runner launches, is restored immediately. There is one free trial per person, and subscribing or changing plans does not restart it. See Plans and the free trial.

What a job's failure_reason means#

When Latchkey ends a Latchkey CI job itself, rather than your command deciding it, the job's failure_reason is one of the values below. Each value is the exact word Latchkey records, so it is not translated.

Latchkey's own faults#

ValueWhat it meansRetryable
provisioning_deadline_exceededLatchkey did not give this job a runner in time, so it never started.yes
provisioning_claim_abandonedLatchkey began preparing a runner for this job and did not finish.yes
runner_lost_before_startThe runner assigned to this job disappeared before the job started.yes
runner_lost_heartbeat_deadThe runner stopped responding before the job finished.yes
runner_lost_timeout_exceededLatchkey lost contact with the runner and stopped waiting. NOT the same as the agent's timeout_exceeded, which is the customer's own command running too long.yes
runner_reaped_max_age_exceededLatchkey retired the runner while the job was on it.yes
runner_reaped_timeout_exceededLatchkey retired the runner after it ran past its limit.yes
runner_reaped_heartbeat_deadLatchkey retired the runner after it stopped responding.yes
pending_deadline_exceededThe job waited longer than Latchkey allows without starting.yes
no_capacityLatchkey had no runner of that size for this job and stopped trying.yes, and a smaller size is likelier to place
requeue_cap_exceededLatchkey could not place this job on a runner within its attempt limit.yes
not_queued_at_final_attemptLatchkey made its last placement attempt and the job was no longer waiting.yes
claim_contended_at_final_attemptTwo Latchkey components tried to claim this job at once and neither finished it.yes
unbound_pipelineLatchkey could not bind this job to a pipeline. Every job carries a pipeline identity, so this is a defect on our side rather than a problem with your command.no, a retry reaches the same point; report it
blocked: token_life_below_launch_floorLatchkey could not mint a credential that would outlive the job, and could not find the job's place in its queue to try again later, so it ended the job.yes
blocked: concurrency_limitThis workspace was at its concurrency limit, and Latchkey could not find the job's place in its queue to hold it there, so it ended the job.yes, once a running job finishes
forge_provider_unsupportedLatchkey does not support this code host yet.no; it is a gap on our side
refused: runner_cannot_read_secretsThe runner this job reached runs a Latchkey agent too old to handle this job's secrets safely, so Latchkey ended the job before any step ran and did not bill it.yes
refused: runner_cannot_record_outputsThe runner this job reached runs a Latchkey agent too old to record this job's outputs, so Latchkey ended the job before any step ran and did not bill it.yes
refused: runner_cannot_prepare_servicesLatchkey checked each runner this job reached before the job began, and moved the job on each time a runner could not run the job's services. It ended the job when no attempt was left, before any step ran, and did not bill it.yes
context_fetch_failedLatchkey could not stage the context it was holding for this job, so your command never ran. Latchkey attests this fault itself; the same word reported by the job is the job's own words, described below the tables.yes, a new run uploads the context again

Your cancellations#

ValueWhat it meansRetryable
cancel_requested_before_runningYou cancelled this job before it started.yes, by submitting again
cancelled_before_provisioningYou cancelled this job while Latchkey was still preparing a runner.yes, by submitting again

Approvals#

ValueWhat it meansRetryable
environment_rejectedAn owner or admin rejected this run for the environment this job names, so Latchkey ended the job without starting it and did not bill it.yes, by running it again, which asks for approval again
environment_approval_expiredNo owner or admin approved this run for the environment this job names within 24 hours, so Latchkey ended the job without starting it and did not bill it.yes, by running it again, which asks for approval again
environment_ref_not_allowedThe environment this job names accepts jobs only from the branches an owner or admin set for it, and this run is not on one of them, so Latchkey ended the job without starting it and did not bill it.yes, by running it again after an owner or admin changes the environment's branches to accept this run

Connection and enrolment#

ValueWhat it meansWhat fixes it
not_enrolledThis repository is not enrolled in Latchkey CI.enrol it from the dashboard
no_connectionLatchkey has no connection to the code host for this repository.reconnect it from Settings, Code hosts
connection_deletedThe connection this repository ran under was deleted.reconnect it from Settings, Code hosts
connection_suspendedThe Latchkey app is suspended on this account, so Latchkey cannot read the repository.unsuspend it in your code host's settings
connection_credential_unusableLatchkey's stored credential for this connection no longer works.uninstall the Latchkey app from the account on your code host, then reconnect it from Settings, Code hosts
repo_grant_revokedThe Latchkey app no longer has access to this repository.grant it again in your code host's settings
tenant_mismatchThis repository is connected to a different Latchkey workspace.check the workspace; report it if you believe it is right
forge_provider_mismatchThis repository belongs to a different code host than the connection Latchkey holds for it.reconnect the repository to the right host
repository_not_identifiedLatchkey could not tell which repository this job belongs to.re-enrol the repository
repository_not_foundLatchkey could not find this repository on the code host.check it still exists and is still granted
repository_identity_unusableLatchkey holds an identity for this repository it can no longer use.re-enrol the repository

Billing and eligibility#

ValueWhat it meansWhat fixes it
refused: no_subscriptionThis workspace has no plan, so Latchkey did not start the job.choose a plan; if you already have one, contact support
blocked: no_subscriptionSame cause as refused: no_subscription, recorded at launch rather than at the offer.choose a plan; if you already have one, contact support
refused: trial_expiredThe trial has ended, so Latchkey did not start the job.choose a plan; if your trial is still active, contact support
blocked: trial_expiredSame cause as refused: trial_expired, recorded at launch.choose a plan; if your trial is still active, contact support
refused: subscription_inactiveThe subscription is not active, so Latchkey did not start the job.reactivate billing; if it is already active, contact support
blocked: subscription_inactiveSame cause as refused: subscription_inactive, recorded at launch.reactivate billing; if it is already active, contact support
refused: eligibility_lapsed_at_offerThe workspace stopped being eligible to run jobs between submitting this job and starting it.check billing; if it looks correct, contact support
blocked: spend_cap_reachedLatchkey refused or stopped the job because the workspace reached the spend limit an owner or admin set.raise the limit or wait for the reset
stopped: spend_cap_reachedLatchkey refused or stopped the job because the workspace reached the spend limit an owner or admin set.raise the limit or wait for the reset
stopped: free_allowance_reachedLatchkey stopped the job because the workspace has no plan and too few of its included minutes were left for the job's next minute, which counts by its size.choose a plan or wait for the reset
budget_reachedLatchkey stopped the job when it reached the budget it was started with (budget_usd on the job, or --budget on latchkey run). You are not charged past that budget.start it again with a larger budget if the work needs more

Your own input#

ValueWhat it meansRetryable
command_spawn_failedLatchkey could not start your command because one of the environment values you set contains a NUL byte. The command never ran.no, not until the value is changed. Fix: remove the NUL byte from the named environment value.
blocked: secret_scope_refusedA secret one of this job's steps names no longer exists at this repository or its workspace: it was deleted after the run was queued, so Latchkey ended the job before it ran and did not bill it.no, not until the secret is set again. Fix: set it for this repository or its workspace, or remove the reference from the workflow file.
need_output_not_writtenThis job reads an output that the job it needs did not write, so Latchkey ended it before it started and did not bill it. Latchkey never reads a missing output as empty.no, not on a re-run, which reads the same recorded values. Fix: have the job it needs write that output to $LATCHKEY_CI_OUTPUT, or stop reading it, and push the change.
need_output_not_recordedThis job reads an output that Latchkey CI did not record from the job it needs, so Latchkey ended it before it started and did not bill it. The log of the job it needs says why.no, not on a re-run, which reads the same recorded values. Fix: change what the job it needs writes, as its log says, and push the change.
need_output_over_run_boundThis job reads an output that Latchkey CI did not record, because the outputs of the job it needs would have taken the run attempt past 50 MiB of outputs, so Latchkey ended it before it started and did not bill it.no, not on a re-run, which reads the same recorded values. Fix: have the run's jobs record fewer or smaller outputs, and push the change.
need_outputs_over_env_limitWith the outputs it reads, this job's environment would pass what one job can hold, 64 variables and 32 KiB in all, so Latchkey ended it before it started and did not bill it.no, not on a re-run, which reads the same recorded values. Fix: read fewer or smaller outputs, or set fewer variables in this job, and push the change.
need_outputs_over_message_limitWith the outputs it reads, this job would be larger than Latchkey CI can serve one job, so Latchkey ended it before it started and did not bill it.no, not on a re-run, which reads the same recorded values. Fix: read fewer or smaller outputs, and push the change.
need_output_outside_input_ruleA value one of this job's steps reads from an output into a with: input does not fit that input's rule, so Latchkey ended the job before it started and did not bill it. The job's page quotes the rule.no, not on a re-run, which reads the same recorded values. Fix: have the job it needs write a value the input accepts, and push the change.

Two kinds of failure_reason are open, so no table can list them. The first is a reason your job reported itself: Latchkey shows it as the job's own words, and the Latchkey check is red. One of those, context_fetch_failed, means the job could not fetch the context Latchkey was holding for it. When Latchkey confirms the fault as its own (the context it was holding is missing), the check is neutral and the row and the CLI say so; when it cannot, the check stays red with the job's own words. Another, service <label> could not start, is a Latchkey fault when the runner itself failed the start (its host-name write, its container engine not answering or refusing it, or its engine client not prepared) and your job's when its label is already a host name on the runner, the image could not be pulled, stopped, or was not ready in time; the job log says which, with the registry's own words for a refused pull, and in Latchkey's case the check is red anyway until Latchkey can confirm that fault itself. Latchkey checks the runner before a job with services begins, and where that check proves the runner unable (its engine client cannot be set up, its engine refuses a network, it lacks the privilege to write host names, or, on a runner cold-started for the job, its engine does not answer), the job moves to another runner instead. A job that runs out of attempts that way ends with refused: runner_cannot_prepare_services from the first table, and the check is not red. The second is launch_failed: followed by an error name: Latchkey could not launch a runner for the job, which is a Latchkey fault like those in the first table, and it is safe to retry.

Still stuck?#

The FAQ covers the quick answers. For anything else, Support and Send feedback in the Account and help menu both reach the team directly.

Q.01My job is stuck in queued. What do I check first?

Six things, in this order: the runner label, whether the repository is monitored, the runner configuration, billing state, an image build still in progress, and the concurrency ceiling. Each has an explicit tell, so working down the list finds it faster than guessing.

Q.02My dashboard is empty. Where did my data go?

Almost always the filters rather than lost data. Reset filters selects every repository and workflow again and returns the Date range window to the last 30 days; if the workspace is new, pages stay empty until Latchkey runs its first job.

Q.03I installed the GitHub App on my personal account. What now?

The installation is connected, and that account runs Latchkey CI: enrol the repository from the dashboard, then commit a workflow file under .latchkey/workflows/ and push. Latchkey on GH needs a GitHub organization; Install the GitHub App shows how to move a repository into one.