Monitoring & automation

Use the same project metrics in Metalhost, your own Prometheus, or Grafana. Give integrations only the access they need.

Start with the task guides: VM monitoring and guest installation, alerts and notifications, scoped credentials, or GitHub Actions with a complete workflow. Managed GitHub runners are planned separately, not supplied by this release.

Choose an identity

Request and response schemas are in the API reference, generated from SDK v1.1.2.

• Scoped personal key: an integration acting within your own current access, restricted to one project and explicit capabilities.
• Project service account: unattended automation that should survive a teammate leaving. Its keys cannot exceed the account’s current permissions.
• GitHub Actions: a verified workflow receives a 15-minute credential without saving a long-lived Metalhost key in GitHub.

In Developers, choose an existing service account or create one from a permission template. Metrics scraper grants monitoring.read, not VM mutation, console access or IAM administration. Review the project, permissions and expiry before creating a key.

New scoped keys default to 90 days and can last up to one year. Copy the secret once. It cannot be retrieved later. Rotation supports a deliberate overlap of up to 24 hours; immediate revocation stops access without waiting for expiry. Disabling an identity or removing a grant affects subsequent requests.

After an uncertain create/rotate response, retry the exact request with the original request_id. Recovery returns metadata, not the lost secret. Revoke that credential before issuing a replacement. Do not generate a new request ID merely because a request timed out.

Know what the measurements mean

Measurement	Meaning and limitation
CPU utilization	Average utilization of observed vCPUs, normalized to capacity. Counter resets are retained; a resize must not divide historical activity by today’s vCPU count.
Guest memory	Guest-visible capacity minus free memory, including cache. Not application working set or memory pressure.
Disk and network rates	Observed virtual-device traffic. These are not invoice-grade transfer measurements.
Filesystem usage	Guest-agent-reported capacity and usage where available. A mount is not necessarily one attached disk, and a missing reading is not 0% full.
Data quality	Source observation time, stale data and query errors are distinct from VM health. Buffered samples keep their original times.

Baseline monitoring uses the hypervisor and the QEMU guest agent when the guest reports it. It does not give staff a shell in your VM. Expanded monitoring is a separate opt-in: on a new Ubuntu or Debian VM, choose it at create time, or install it later from the VM’s monitoring page. Opening Monitoring does not install the agent or restart the VM. Expanded metrics are available in DC-1.

Hosted retention is seven days, with 30-second source sampling. The query gateway has a ten-second upstream timeout, a minimum 30-second range step, at most 10,000 points per series, a 16 KiB expression limit and a 4 MiB response cap. Concurrency is limited per project. Back off on 429 and honor Retry-After when supplied. Narrow requests rejected for size; never treat errors or absent samples as zero.

Example: one hour of VM metrics

Set METALHOST_ENDPOINT to https://api.metalhost.net with no trailing slash, load METALHOST_API_KEY from your secret manager, and set METALHOST_VM to the full VM name returned by the API. This read-only example requires monitoring.read, bash, curl and jq. It is included in CLI v1.1.2:

#!/usr/bin/env bash
# Read-only example. Requires bash, curl, jq and a monitoring.read credential.
set -euo pipefail
set +x
: "${METALHOST_ENDPOINT:?Set the API origin for your enabled environment}"
: "${METALHOST_API_KEY:?Load a credential from your secret manager}"
: "${METALHOST_VM:?Set the full VM resource name copied from Metalhost}"
end=$(date +%s)
start=$((end - 3600))
jq -nc --arg name "$METALHOST_VM" --argjson start "$start" --argjson end "$end" \
  '{name:$name,metricIds:["cpu_utilization_pct","memory_used_bytes"],
    startTimeUnix:($start|tostring),endTimeUnix:($end|tostring),stepSeconds:60}' |
  curl --fail --silent --show-error --max-time 30 \
    -H "Authorization: Bearer $METALHOST_API_KEY" \
    -H 'Content-Type: application/json' -H 'Connect-Protocol-Version: 1' \
    --data-binary @- "$METALHOST_ENDPOINT/aes.monitoring.v1.MonitoringService/QueryVMMonitoring" |
  jq '{startTimeUnix,endTimeUnix,stepSeconds,quality,series}'

Download query script. Inspect quality alongside series. Use the response's actual bounds and step. For a seven-day query use a coarser step, such as 1800 seconds; 60 seconds exceeds the point budget.

Connect Grafana directly

1. Open the project’s Monitoring → Integrations → Connect Grafana.
2. Create or choose a project-scoped monitoring.read credential.
3. Use the displayed Prometheus query base URL, not the scrape URL, in a Grafana Prometheus data source.
4. Configure Authorization: Bearer … in Grafana’s secure server-side header settings. Never put a key in a URL, browser variable or dashboard JSON.
5. Run Grafana’s Save & test, then import the downloadable Metalhost VM dashboard.

Grafana runs in your environment; Metalhost hosts the project’s metric history. Read-only PromQL, dashboard variables and supported metadata endpoints remain tenant-scoped. Write, remote-write, runtime-configuration and rule-management endpoints are not exposed through this query URL.

# Set PROJECT to the full canonical name, for example projects/PROJECT_ID.
curl --fail --silent --show-error --get \
  -H "Authorization: Bearer $METALHOST_API_KEY" \
  --data-urlencode 'query=sum(rate(metalhost_vm_cpu_seconds_total[2m]))' \
  "$METALHOST_ENDPOINT/v1/monitoring/$PROJECT/prometheus/api/v1/query"

The CPU example returns consumed CPU cores, not a normalized percentage. The imported dashboard uses the normalized expression. Configuration output contains credential placeholders, not your saved key.

Scrape into your own Prometheus

Choose Scrape into your Prometheus for a separate copy of current samples in your own storage. Use authenticated HTTP discovery so every stable export shard is included; do not assume one shard covers a large project.

# Replace the API host and canonical project path with values from Integrations.
# Mount a monitoring.read credential at this path; readable by Prometheus only.
scrape_configs:
  - job_name: metalhost
    scrape_interval: 60s
    scrape_timeout: 10s
    honor_timestamps: true
    scheme: https
    authorization:
      type: Bearer
      credentials_file: /etc/prometheus/secrets/metalhost-key
    http_sd_configs:
      - url: https://API_HOST/v1/monitoring/projects/PROJECT_ID/scrape-targets
        refresh_interval: 60s
        authorization:
          type: Bearer
          credentials_file: /etc/prometheus/secrets/metalhost-key

Download scrape configuration. Replace API_HOST and the entire canonical project path with the values from Integrations. Protect the credential file so only Prometheus can read it; do not commit it. Discovery supplies each shard's host, path and query parameters. See the Prometheus HTTP discovery reference.

Save the credential in the protected file named in the generated configuration. Both discovery and scrape requests need it. Keep honor_timestamps: true; otherwise old source observations could appear newly observed. A 60-second external scrape interval is a useful starting point.

Keyless GitHub Actions

In Developers → GitHub Actions, select an existing service account and start workflow verification. The temporary pairing flow reads GitHub-signed repository, owner, workflow, event, and branch/environment identity. Review those observed details before approving trust. A typed repository name alone does not establish trust.

The job needs id-token: write. Follow the complete HTTP exchange workflow, which requires only monitoring.read and keeps the token in a single step. No CLI authentication helper is required. The September CLI branch alternatively provides auth github and monitoring commands; older CLI releases do not.

Do not echo either token or persist it through artifacts or GITHUB_ENV. Jobs longer than 15 minutes need another authorized exchange. Pull-request contexts are rejected. Editing trust permissions invalidates outstanding tokens; changing repository/workflow identity requires new verification. Disable the trust for emergency revocation.

Configure alerts without raw PromQL

In Monitoring → Alert rules, choose a template, existing VMs or all current/future project VMs, a threshold, sustained duration and destinations. Initial templates cover CPU, guest memory, supported filesystem usage, unexpected VM stops and unavailable telemetry. These are infrastructure observations, not external HTTP uptime checks.

Review capability coverage and preview results before saving. An accepted rule is not active until its desired revision reaches the evaluator. No rules means nothing has been configured to watch conditions; it does not mean everything is healthy.

• Acknowledge records human attention; it does not recover an incident.
• Snooze temporarily suppresses notifications while evaluation continues.
• Intentional stops and maintenance suppress inappropriate down alerts.
• Missing data and evaluator errors do not prove recovery.
• Pause, edit or delete retires affected instances explicitly rather than manufacturing a recovery.

In-app incidents work without external destinations. Email recipients must complete verification and retain project access. Choose an existing webhook or create an HTTPS endpoint, then use Send test. “Sent” means the provider/endpoint accepted the request, not that a person read it. Default project limits are 20 rules, five destinations and 10,000 evaluated instances.

Preview this CPU rule without saving it or sending notifications. Replace the canonical project name in the JSON, save it as alert-preview.json, and use a credential with monitoring.read:

{
  "projectName": "projects/PROJECT_ID",
  "spec": {
    "displayName": "Sustained high CPU",
    "metricId": "cpu_utilization_pct",
    "operator": "GREATER_THAN",
    "threshold": 90,
    "sustainedSeconds": 300,
    "severity": "WARNING",
    "allVms": true,
    "enabled": true,
    "repeatIntervalSeconds": 3600,
    "destinationNames": []
  },
  "historySeconds": 3600
}

curl --fail --silent --show-error \
  -H "Authorization: Bearer $METALHOST_API_KEY" \
  -H 'Content-Type: application/json' -H 'Connect-Protocol-Version: 1' \
  --data-binary @alert-preview.json \
  "$METALHOST_ENDPOINT/aes.monitoring.v1.AlertService/PreviewAlertRule"

Download preview body. Check queryStatus, historyStatus and evaluated/expected sample counts. Preview is a bounded estimate, not a replay of notifications. Saving is a separate SaveAlertRule mutation: provide this spec plus distinct client-generated UUIDs for id and requestId, with expectedVersion: "0" for creation. Retain that exact request on ambiguous retry; for edits use the latest version. Verify appliedVersion and syncStatus afterward.

Chat destinations support Slack, Discord and Teams in addition to email and signed webhooks. See destination setup and testing.

Saving rules requires monitoring.write. Capabilities apply to explicitly allowed RPCs, not every method in a service. Destination verification, test sends and incident acknowledgement/snooze are human dashboard operations in the current scoped-automation policy.

Verify and deduplicate webhook deliveries

Read the raw request body before JSON decoding. The SDK’s metalhost.VerifyWebhook validates the timestamped V2 signature, signed delivery ID and attempt ID with a five-minute clock window. Bound the body to 1 MiB and durably deduplicate DeliveryID; an accepted retry may have a different attempt ID. Verification alone does not prevent a replay within the clock window.

During secret overlap, V2 signatures support both keys. The compatible legacy raw-body signature uses the previous key until overlap ends. Migrate receivers to V2 before relying on timestamp/replay protection. Immediate rotation discards older keys; never log or retrieve sealed secrets.

Delivery is at least once, with bounded retries for up to 24 hours. Expired or no-longer-relevant firing messages are not sent later. Recipient removal, subscription pause, recovery and snooze are rechecked before sending. Initial incident/delivery history retention is 90 days; billing and security audit retention are separate.

Troubleshooting

• Permission denied: check project boundary, credential capabilities, identity state and current grants. Do not broaden a key just to hide an error.
• Grafana connects but no samples: inspect VM source freshness and metric capability. A connection probe is not proof that a guest produces every metric.
• Stale DC: history may still be readable, but old samples must not be interpreted as live.
• Rule pending application: inspect the desired/applied revision and evaluator status before assuming the rule is watching the VM.
• Notification failed: inspect the retained delivery/test status, endpoint reachability and recipient eligibility. A timed-out delivery can already have been accepted.