Platform Observability

9.1 Architecture Principles

Observability is centralised; collection is distributed. Each managed cluster runs its own collector, forwarding directly to the central aggregation backend. Observability data does not transit the Portainer Control Plane — the collector on each cluster forwards directly to the aggregation endpoint over TLS. A Control Plane failure does not disrupt collection; the collector buffers locally until the connection is re-established.

Portainer provides real-time baseline metrics and container logs. Portainer surfaces real-time metrics and logs for cluster, nodes and pods, covering day-to-day operational needs: spotting a resource-pressured node, identifying a container approaching its memory limit, verifying that a scaled deployment is distributing load, or tailing a problematic container. They are not stored, not retained, and not queryable historically.

OneUptime is the unified observability backend for new deployments. One deployment, one Observability Policy, one consistent access model — logs, metrics, uptime, alerting, incidents, and on-call in a single platform. Assembling equivalent capability from separate CNCF components is a significant engineering project that the organisation must then own and operate indefinitely. OneUptime eliminates that assembly cost.

Portainer is the operational interface and RBAC enforcement layer. Portainer surfaces logs and metrics from the central OneUptime backend within its management interface, applying its Kubernetes-aligned RBAC model to every query. A developer with access to a specific environment sees only their namespace's data; a platform engineer with fleet-wide access queries across all clusters. No additional proxy infrastructure is required for standard operator access. Standard operators interact with observability data through Portainer; platform engineers and SREs use the OneUptime interface directly for dashboards, incident management, and deep investigation.

Out-of-band alerting is mandatory. Alerts must not rely exclusively on the Portainer management UI. Operators must receive alerts through external channels — OneUptime, PagerDuty, email, Slack — that remain functional when the Portainer interface or the management cluster itself is degraded.

OneUptime back-end, Portainer front-end

Log taxonomy: operational vs. security audit. Operational logs (container stdout/stderr, Kubernetes events, node system logs) and metrics are the subject of this chapter. Security audit logs (Portainer Control Plane events, Kubernetes API server audit log) are covered in Chapter 10 and must be forwarded to the SIEM independently of the operational observability pipeline.

9.2 OneUptime Deployment

OneUptime is an open-source (Apache 2.0), self-hosted observability and incident management platform covering the complete SRE toolchain: log management, metrics, distributed traces, uptime monitoring, incident management, on-call scheduling, and status pages.

Backend deployment: OneUptime is deployed centrally on the management cluster or on dedicated infrastructure via Portainer's Helm repository integration; the chart is published at helm-chart.oneuptime.com . A single-click backend install from the Portainer UI is on its roadmap. OneUptime's Helm chart deploys all platform components into a dedicated namespace. For production deployments, three data stores are required: PostgreSQL (configuration and state), Redis (caching, queues, and sessions), and ClickHouse (all telemetry — logs, metrics, and traces). Size the management cluster to accommodate both Portainer and OneUptime, accounting for the expected telemetry volume.

OneUptime back-end deployment

Fleet-wide collector deployment — the Observability Policy: Once the central OneUptime instance is running, create an Observability Policy in Portainer, configure the OneUptime connection (URL, API key, Project ID), select an operational mode, and attach it to one or more environment groups. All current and newly added Kubernetes environments in those groups automatically receive the correct collectors and connection configuration. The same policy covers both logs and metrics — no separate policy is needed. These policies applies to Kubernetes environments only; Docker environments are out of scope.

Two operational modes:

Deploy-and-connect: Portainer deploys the OneUptime Kubernetes Agent ( oneuptime/kubernetes-agent from helm-chart.oneuptime.com ) into each attached environment via Helm charts embedded in Portainer. The agent collects cluster metrics, events, and pod logs, forwarding telemetry to the central OneUptime instance via OTLP. Portainer configures itself to query OneUptime for UI enrichment. No separate collector deployment is required.

Connect-only: For environments where OneUptime collectors are already deployed. Portainer validates their presence, configures the connection for UI enrichment, and marks the attachment healthy once prerequisites are confirmed.

Log collection mode: The agent supports two modes. DaemonSet mode (default — self-managed clusters, EKS on EC2, GKE Standard, AKS) runs one collector pod per node reading from node-level log files via a hostPath volume — lowest overhead. API mode (GKE Autopilot, EKS Fargate) reads logs via the Kubernetes API without hostPath volumes — required where pod security standards block host volume access. The agent supports namespace-level filtering ( namespaceFilters.include / namespaceFilters.exclude ), enabling collection scope to align with the namespace-as-tenancy model (Chapter 4).

Portainer UI enrichment: Once the Observability Policy is active, resource views — Helm chart releases, application workloads, and namespaces — gain an Observability tab showing logs and metrics scoped to the current resource. Operators investigate workloads without leaving Portainer. A link-out to the OneUptime interface is provided for deeper investigation, incident workflows, and dashboard access.

Sizing reference

OneUptime stores telemetry across three independent datastores that must be sized separately. ClickHouse absorbs all logs, metrics, and traces and is the dominant cost driver — approximately 95% of storage. PostgreSQL stores configuration, incident records, and platform state; it grows slowly and stays small relative to ClickHouse. Redis handles caches, queues, and sessions; it is memory-bound and modest at any fleet size.

ClickHouse storage formula:

disk ≈ (daily raw telemetry GB ÷ compression ratio) × retention days × replicas × 1.3

Compression ratios per signal type: logs ~5:1 , metrics ~2:1 (high label cardinality reduces compression significantly and inflates RAM disproportionately — enforce low-cardinality label conventions at the collection layer), traces fall between the two.

A reference fleet of 10 clusters, ~10 nodes and ~100 pods per cluster at INFO-level verbosity produces approximately 50–150 GB of raw logs per cluster over 30 days (~1.7–5 GB/day per cluster). Across the full fleet, combining logs, metrics, and traces after compression, budget approximately 5–15 GB/day of compressed ClickHouse data :

Storage scales linearly with retention — a 90-day window costs approximately 3× a 30-day window. ClickHouse requires NVMe or fast SSD; it is write-heavy with bursty aggregation reads and will not perform adequately on spinning disk.

Starting tiers (from OneUptime's sizing documentation ):

Two deployment notes before sizing:

• OneUptime's default telemetry retention is 15 days (hardcoded in the default Helm chart). Set retention explicitly per project and per signal type before going to production — unset retention will silently drop data earlier than expected.
• The default Helm chart ships with 25Gi volumes and no resource limits on any component. These are not production values. Override them explicitly using the starting tiers above.

For high availability , enable the bundled CloudNativePG operator (3 PostgreSQL instances with automatic failover) and the Altinity ClickHouse operator (≥2 replicas per shard + 3 Keeper nodes for distributed coordination). The default chart does not include Redis replication — use an external managed Redis for HA deployments.

Actual consumption depends heavily on application log verbosity, number of namespaces, scrape interval, metric cardinality, and whether DEBUG logging is enabled on any workloads. Instrument your environment for at least four weeks before committing to a retention-based storage decision. See OneUptime sizing documentation for the authoritative reference.

9.3 Logs

Portainer Native Log Viewer

Portainer provides built-in container and pod log access directly within its management interface. Platform engineers and developers view real-time and recent container logs from the Portainer UI, for any container running in any managed environment they have access to, without additional logging infrastructure. Log visibility is scoped by Portainer's RBAC model: a user with Namespace Operator access sees logs only for containers in their assigned namespaces; a platform engineer with Environment Administrator access sees all containers in that environment.

💡

These logs are not stored long-term. Only real-time, current logs are shown.

This is appropriate when the team's operational needs are met by per-container log inspection rather than cross-cluster aggregation; compliance requirements do not mandate centralised long-term log retention; and the operational overhead of deploying and maintaining a logging backend is not yet justified. This capability remains available even when an external logging backend is configured.

Log Collection — Observability Policy with OneUptime

For OneUptime deployments, the Portainer Observability Policy in deploy-and-connect mode deploys and manages an OpenTelemetry Collector for log collection on each attached Kubernetes environment. No separate collector deployment is needed. See section 9.2 for Observability Policy configuration.

Log Collection — Loki-based deployments

Teams running an existing logs stack (based on Loki, Alloy and Grafana) can integrate Loki into Portainer’s UI, so that users can query Loki’s logs natively from the Portainer UI — with full support for Portainer’s RBAC.

9.4 Metrics

Portainer Baseline Metrics

Portainer surfaces real-time CPU and memory usage at the cluster, node, and container level, powered by metrics-server . These graphs cover immediate operational awareness — spotting a resource-pressured node, verifying load distribution. They are not retained beyond the current session and are available regardless of whether a metrics backend is deployed.

Metrics Collection — Observability Policy with OneUptime

The Portainer Observability Policy in deploy-and-connect mode deploys an OpenTelemetry collector for metrics on each attached Kubernetes environment alongside the log collector. The collector gathers Kubernetes system component metrics and pod-level telemetry, forwarding to the central OneUptime instance via OTLP. This is the recommended path for new deployments.

Metrics Collection — Prometheus (existing investments)

Teams running an existing metrics stack (based on Prometheus, Alloy/FluentBit and Grafana) can integrate Prometheus into Portainer’s UI, so that users can query metrics natively from the Portainer UI — with full support for Portainer’s RBAC.

Instrumentation

Kubernetes native metrics layer: kube-state-metrics (object-level state: Pod phase, Deployment replica counts, resource limits), cAdvisor embedded in the kubelet (container resource usage per running container), and node-exporter DaemonSet (host-level metrics: CPU, memory, disk I/O, network) are included in kube-prometheus-stack and scraped by Prometheus.

Application metrics: New applications use the OpenTelemetry SDK — a single SDK covers metrics, traces, and logs, is vendor-neutral, and exports to OneUptime or any OTLP-compatible backend without code changes. Existing applications using Prometheus client libraries do not require migration.

9.5 Alerting and Incident Management

Alert Sources

Portainer built-in alerting: Portainer Business Edition includes eight built-in alert rules: environment availability (environment down), resource pressure (high CPU, memory, network usage), authentication anomalies (brute-force attack, high single-user authentication failures), backup failure, and TLS certificate expiry. Notification channels — Slack, Microsoft Teams, email, and webhook — are configurable; alert rules are fixed. Configure Portainer's webhook channel to forward alerts to OneUptime's incoming alert endpoint so Portainer-originated events enter the same incident pipeline as metrics-originated alerts. Alert scoping is per-environment; notifications include direct Portainer links to the affected environment.

OneUptime native monitor rules: Alert rules defined directly in OneUptime against ingested metrics and logs, without a separate Prometheus pipeline. OneUptime evaluates these rules and creates incidents directly — the recommended approach for teams deploying via the Observability Policy.

Prometheus alerting rules: Prometheus evaluates PrometheusRule CRDs on each evaluation interval. When a rule's condition persists beyond the for duration, the alert fires to AlertManager. Rules are version-controlled in Git (Chapter 7). In OneUptime deployments, AlertManager forwards fired alerts to OneUptime via its webhook receiver.

Log-based alerting: OneUptime supports alert rules evaluated against ingested log data — detecting error pattern spikes, specific error messages, or security event patterns that cannot be represented as metric thresholds. A typical example: alert when any container emits more than 20 log lines containing level=error within a 5-minute window, creating an incident before the issue surfaces as a metrics anomaly.

Synthetic monitoring: OneUptime provides built-in uptime monitoring — HTTP, TCP, and ping probes targeting cluster ingress endpoints, the Portainer API, and external DNS. Probes must target endpoints from outside the cluster boundary to provide meaningful availability coverage.

OneUptime as the Alerting and Incident Hub

OneUptime consolidates alert ingestion, incident creation, on-call management, escalation, and status pages in a single platform.

Alert ingestion: OneUptime receives alerts via its incoming webhook endpoint: Portainer's built-in webhook, AlertManager's webhook receiver, and its own native monitor rules. All arrive as structured JSON payloads and are mapped to OneUptime incidents.

Incident management: OneUptime deduplicates repeated firings into a single open incident. Incident metadata includes alert source, severity, affected environment, and a link to the relevant Portainer environment page. Operators access incidents scoped to their project.

On-call scheduling: On-call schedules per project/team with rotation definitions (daily, weekly, follow-the-sun) and escalation layers. When an incident is created, OneUptime notifies the current on-call responder via configured channels (email, Slack, Teams, SMS, or phone). If not acknowledged within the configured window, it escalates to the next layer. Align schedules to Portainer's team structure (Chapter 4).

Escalation policies: A reference policy for critical platform incidents: notify primary on-call immediately → secondary after 5 minutes → team lead after 15 minutes. Map production and staging environments to separate OneUptime projects with independent escalation policies.

Status pages: Status page components reflect Portainer environment groups; pages update automatically when incidents are created or resolved.

OneUptime as the Alerting and Incident hub

AlertManager in OneUptime deployments

In OneUptime deployments, AlertManager narrows its role from full notification routing to forwarding fired Prometheus alerts to OneUptime via webhook receiver. Alert grouping, inhibition rules, and silences remain in AlertManager. Group by cluster and alertname ; define inhibition rules to suppress pod-level alerts when a node-down alert is firing for the same node. Align silences with Portainer's change window model (Chapter 7).

Alert Design

Symptom-based over cause-based: Alert on user-facing or operator-facing impact, not internal state changes. Alert on symptoms; use dashboards to diagnose causes during response.

Severity taxonomy: Three tiers applied uniformly across all alert sources — Prometheus rules, OneUptime monitor rules, and Portainer webhook events. Critical: immediate impact, pages on-call. Warning: potential impact if unaddressed, posts to Slack or Teams. Informational: awareness only, no notification. Encode severity consistently so all downstream routing derives from the same field.

Ownership and runbooks: Every alert must have a defined owner encoded in alert labels ( team=platform-eng ). Every critical and warning alert must carry a runbook_url annotation. Store runbooks in Git alongside the alerting rules that reference them.

SLO-based alerting: For services with defined SLOs, multi-window multi-burn-rate error budget burn rate alerts provide earlier and more accurate signal than threshold-based alerts and are the recommended pattern for production services with formal SLOs.

9.6 Loki, Prometheus, Grafana

Teams with existing investments in Loki, Prometheus and Grafana do not need to migrate to OneUptime. Portainer’s query integration supports Loki and Prometheus in addition to OneUptime. Logs and Metrics from Loki and Prometheus can be surfaced to Portainer’s UI, just like OneUptime, with RBAC enforced natively, with references to Grafana for more advanced dashboards.

For integration guidance of your existing stack into Portainer — architecture Helm values, Alloy collector configuration, namespace setup — contact Portainer’s managed services team.

If your existing on-call workflow runs on PagerDuty or OpsGenie, Portainer's webhook delivers alerts into that pipeline without requiring OneUptime. Per-cluster AlertManager instances route fired alerts to PagerDuty (Events API v2 receiver) or OpsGenie. Portainer's built-in webhook delivers to AlertManager's webhook receiver. Team membership in PagerDuty and OpsGenie must be maintained separately — the shared-OIDC model does not extend to these external SaaS platforms.

9.7 Non-Functional Considerations

Log retention: Operational retention (14–90 days) and compliance retention (1–7 years) require separate configurations. In OneUptime, use project-level settings connected to durable object storage for compliance-grade retention. For Loki, use the compactor's per-tenant retention configuration.

Multi-tenancy via projects: OneUptime’s project model provides the tenant isolation boundary. Each team’s metrics are stored within their designated project. Project membership is governed by the IdP-federated SSO model — users access their scoped project metrics in OneUptime using the same identity as their Portainer session (Chapter 4). The IdP group definition is the single source of truth for access across both systems, allowing users to use the Portainer UI for basic OneUptime usage, but dive into the OneUptime UI for advanced use cases while respecting RBAC.

TLS in transit: All forwarding paths crossing network boundaries must be encrypted. OTel Collector to OneUptime: TLS on the OTLP/HTTP ingest endpoint. Alloy to Loki: TLS on the output plugin. Portainer syslog forwarding: mTLS via --syslog-cert , --syslog-key , --syslog-ca-cert flags.

Alerting pipeline resilience: OneUptime and AlertManager must run on the management cluster independently of the workload clusters they monitor. Configure multiple notification channels in OneUptime so that failure of any single channel does not result in a missed notification.

Cardinality management: Enforce low-cardinality label conventions at the collection layer. Monitor active series counts and alert on unexpected growth before it reaches the backend.

Air-gapped environments: OTel collectors use filesystem buffering; Prometheus uses WAL-based durability — records are retained locally and forwarded when connectivity is restored. For fully air-gapped clusters, deploy a local OneUptime instance; pre-stage OTel collector and OneUptime images in the mirror registry (Chapter 6, Scenario E).

Scenarios and Maturity Levels

The scenarios for each chapter are based on Portainer's maturity levels. See the Appendix for more details about the maturity levels and a link to the assessment. Scenarios build on each other and are relative to each other. Find the scenario closest to your current environment; implement relevant improvements from there.