Cycles Top-Up Procedure

Last Updated: 2026-05-12 Alert: LowCyclesBalance, CriticalCyclesBalance Severity: Warning / Critical Response Time: < 15 minutes for critical

Overview

This runbook covers topping up a canister's cycle balance when it falls below operational thresholds. The procedure was rewritten after BL-472 to use icp-cli and direct wallet_send calls; the legacy dfx wallet send / dfx ledger top-up flow is broken for most cases and must not be used (see Legacy commands to avoid below).

Alert Thresholds

Level	Threshold	Response Time
Warning	< 1T cycles	< 1 hour
Critical	< 500B cycles	< 15 minutes

The monitor-metrics.yml workflow on ops-infra runs every 6 hours and alerts when any canister crosses these thresholds (when alerting is wired — see BL-522).

Symptoms

Alert: LowCyclesBalance or CriticalCyclesBalance
Canister calls failing with "out of cycles" error
502/503 errors on frontend (canister frozen)
monitor-metrics.yml workflow shows cycles below threshold (check GitHub Actions — ops-infra)

Diagnosis

Step 1: Verify current canister balance

Use icp canister status (NOT the canister's own balance — the IC management canister reports the canonical cycle balance):

bash

icp canister status <canister-id> --network ic --identity github-ci | grep Cycles:

Example output:

Cycles: 38_832_809_601
Idle cycles burned per day: 91_560_831

Divide cycles by burn-per-day to estimate runway. 38B at 91M/day ≈ 424 days idle — but real traffic will drain that much faster.

Step 2: Check available funding sources

There are three places cycles can come from:

bash

# A) Legacy wdhec wallet — still holds ~10.5+ TC accumulated from years of
#    dfx ledger top-up calls that silently routed through it
dfx wallet --network ic balance --identity github-ci

# B) Identity's principal-direct cycles ledger (the post-BL-472 model)
icp cycles balance --network ic --identity github-ci

# C) ICP balance available to mint fresh cycles
dfx ledger --network ic balance --identity github-ci

Prefer draining wdhec (Path A in Resolution) while it still has cycles — free, just redirects already-purchased cycles.

Step 3: Identify affected canisters

Check the monitor-metrics.yml workflow output in GitHub Actions for a fleet snapshot, or sweep manually:

bash

for f in ~/git/*/canister_ids.json; do
  repo=$(basename $(dirname $f))
  id=$(jq -r '.. | objects | select(has("ic")) | .ic' $f 2>/dev/null | head -1)
  [ -n "$id" ] && echo "[$repo] $id"
done | while read line; do
  repo=$(echo $line | awk '{print $1}')
  id=$(echo $line | awk '{print $2}')
  cycles=$(icp canister status $id --network ic --identity github-ci 2>/dev/null | grep Cycles: | awk '{print $2}')
  echo "$repo $id $cycles"
done

See Canister Cycle Monitoring for the canonical inventory.

Resolution — two valid paths

There are exactly two correct ways to top up a canister's cycle balance. Both deposit cycles directly into the target canister's cycle balance via the management canister. Pick based on whether wdhec still has cycles.

Path A — drain wdhec via wallet_send (preferred while it has cycles)

The legacy wallet wdhec-2yaaa-aaaao-a6dgq-cai currently holds 10.5+ TC accumulated from years of dfx ledger top-up calls that silently routed through it (BL-466 — IC0504 bug). Draining it is FREE — the cycles are already minted and attributed to the org's wallet.

bash

export DFX_WARNING=-mainnet_plaintext_identity
dfx canister --network ic --identity github-ci \
  call wdhec-2yaaa-aaaao-a6dgq-cai wallet_send \
  '(record { canister = principal "<target-canister-id>"; amount = 300_000_000_000 : nat64 })'
# Expect: (variant { 17_724 })  — Ok variant

Suffix conventions on amount: 300_000_000_000 = 300B cycles ≈ 0.3 TC. Use _ separators for readability; nat64 max is 18_446_744_073_709_551_615 so amounts up to several TC fit comfortably.

Path B — fresh mint via `icp canister top-up`

When wdhec is empty (eventually) OR when you want to bypass it (e.g., during the BL-472 decommission phase):

bash

# Convert ICP from identity's ledger account → cycles in identity's principal balance
icp cycles mint --icp 0.1 --identity github-ci --network ic

# Top up the canister directly — bypasses the wallet
icp canister top-up --amount 300b <target-canister-id> --network ic --identity github-ci

Suffix shortcuts: k (thousand), m (million), b (billion), t (trillion). 300b = 300,000,000,000 cycles ≈ 0.3 TC.

Verify after top-up

Always confirm cycles landed in the canister's balance (not the canister's principal-cycles ledger account — see anti-pattern below):

bash

icp canister status <target-canister-id> --network ic --identity github-ci | grep Cycles:
# Cycles: should have increased by approximately the amount topped up

Legacy commands to avoid

Anti-pattern — `icp cycles transfer` for canister top-ups

bash

# WRONG — this is an ICRC-1 principal-to-principal transfer.
# When <canister-id> is the receiver, cycles land in the canister's
# principal-cycles-ledger account, NOT in the canister's cycle balance.
# The canister can't access them without a custom withdraw method.
icp cycles transfer 300b <canister-id> --identity github-ci --network ic

If you've already run this mistakenly, the cycles are stuck in the canister's principal-cycles-ledger account (um5iw-rqaaa-aaaaq-qaaba-cai on mainnet) keyed by the canister's principal. Recovery requires either (a) the canister exposes a claim_cycles / withdraw_cycles method that pulls from its own principal ledger account, or (b) a controller-side bridge. user-service does not expose either (as of 2026-05-12), so cycles transferred this way are effectively lost.

Anti-pattern — `dfx ledger top-up`

bash

# WRONG (mostly) — silently routes through wdhec wallet, then wallet_send to
# the target. The wallet's WASM has an unfixed IC0504 bug that bricks
# wallet_send to frozen targets. CI ran this for years without realizing the
# forward was failing — cycles accumulated in wdhec while target canisters
# slowly starved.
dfx ledger --network ic top-up <canister-id> --amount 0.1 --identity github-ci

This is the command the old version of this runbook recommended. Do not use. Use Path A (drain wdhec directly via wallet_send) or Path B (icp canister top-up) instead.

ICP Funding

If the ICP wallet is empty (verified via dfx ledger --network ic balance):

Identify the dfx identity account ID: dfx ledger --network ic account-id --identity github-ci
Transfer ICP from a team-owned exchange account or wallet to that account ID
Verify balance received: dfx ledger --network ic balance --identity github-ci

For sustained operations, maintain a minimum 1 ICP reserve in the identity's ledger account.

Post-Resolution

Step 1 — Verify resolution

bash

icp canister status <canister-id> --network ic --identity github-ci | grep -E "(Status|Cycles)"
# Expect: Status: Running + Cycles: <new higher value>

# Smoke-test a real query against the canister
dfx canister --network ic call <canister-id> <test-method> --identity github-ci

Step 2 — Monitor

Watch the next monitor-metrics.yml run (every 6 hours, or trigger manually: gh workflow run monitor-metrics.yml --repo Hello-World-Co-Op/ops-infra). Verify no new alerts fire and cycles balance is stable.

Step 3 — Document

If this was a critical incident (canister was frozen + dropping requests):

File an incident report under hello-world-workspace
Update the cycles consumption baseline if traffic patterns changed
Consider adjusting monitor-metrics.yml thresholds for high-traffic canisters

Prevention

Cycle consumption monitoring

Review weekly cycle consumption trends via monitor-metrics.yml history
Maintain at least 7-day runway before warning threshold for each canister
For canisters approaching prod traffic, raise their warning threshold to 5T (5x normal)

ICP reserve

Maintain at least 1 ICP in the github-ci identity's ledger account
Set up alerts for ICP balance below 0.5 ICP

Automation

monitor-metrics.yml runs every 6 hours; auto-top-up is currently blocked on BL-522 (alerting + auto-topup wallet-send migration). Verify the cron is firing via the Actions tab.
Until BL-522 lands, expect to top up manually on receiving an alert.

Escalation

Condition	Action
ICP wallet empty	Contact team lead (Coby) for ICP transfer
`icp` or `dfx` CLI not installed	Run `npm install -g @dfinity/icp-cli` + `dfx --version` to verify
Identity import / principal mismatch	Check BL-472 migration notes — re-import the PEM with explicit `--assert-key-type ed25519`
`wallet_send` returns non-Ok variant	wdhec target may be frozen (IC0504 trigger) — use Path B (`icp canister top-up`) instead
Top-up succeeds but `canister status` cycles unchanged	You probably ran `icp cycles transfer` (anti-pattern above) — cycles landed in principal-ledger account, not the canister. Re-run with Path A or B.
Canister still failing after successful top-up	Check Canister Unresponsive runbook

Canister Production Activation — includes cycles topup as Step 3 of activating a prod canister
Monitoring Setup
Incident Response Runbook
Cycles Top-Up runbook (workspace, icp-cli specifics) — deeper detail on icp-cli identity import + legacy wallet drain paths
DFINITY Cycles Documentation

Cycles Top-Up Procedure ​

Overview ​

Alert Thresholds ​

Symptoms ​

Diagnosis ​

Step 1: Verify current canister balance ​

Step 2: Check available funding sources ​

Step 3: Identify affected canisters ​

Resolution — two valid paths ​

Path A — drain wdhec via wallet_send (preferred while it has cycles) ​

Path B — fresh mint via icp canister top-up ​

Verify after top-up ​

Legacy commands to avoid ​

Anti-pattern — icp cycles transfer for canister top-ups ​

Anti-pattern — dfx ledger top-up ​

ICP Funding ​

Post-Resolution ​

Step 1 — Verify resolution ​

Step 2 — Monitor ​

Step 3 — Document ​

Prevention ​

Cycle consumption monitoring ​

ICP reserve ​

Automation ​

Escalation ​

Related Documentation ​

Cycles Top-Up Procedure

Overview

Alert Thresholds

Symptoms

Diagnosis

Step 1: Verify current canister balance

Step 2: Check available funding sources

Step 3: Identify affected canisters

Resolution — two valid paths

Path A — drain wdhec via wallet_send (preferred while it has cycles)

Path B — fresh mint via `icp canister top-up`

Verify after top-up

Legacy commands to avoid

Anti-pattern — `icp cycles transfer` for canister top-ups

Anti-pattern — `dfx ledger top-up`

ICP Funding

Post-Resolution

Step 1 — Verify resolution

Step 2 — Monitor

Step 3 — Document

Prevention

Cycle consumption monitoring

ICP reserve

Automation

Escalation

Related Documentation