LibreSense

LibreSense is a lightweight, self-hosted infrastructure sensing and incident-intelligence engine. It accepts normalized observations, detects explainable metric behaviors, evaluates declared objectives, and retains bounded behavioral memory: temporal event associations, event-to-metric responses, deployment epochs, and epoch-to-epoch response drift.

Run the daemon

go run ./cmd/libresense -db libresense.db -listen :8080 -config config.json

-config is optional. The daemon restores compact metric baseline/window state and unresolved incidents from SQLite. Raw observations are not loaded into analyzer memory.

Production is anonymous read-only until OIDC is configured. Mutation endpoints require membership in the configured admin group:

OIDC_ISSUER_URL=https://identity.example \
OIDC_CLIENT_ID=libresense \
OIDC_CLIENT_SECRET=replace-me \
OIDC_REDIRECT_URL=https://libresense.example/auth/callback \
OIDC_GROUP_CLAIM=groups \
OIDC_ADMIN_GROUP=libresense-admin \
go run ./cmd/libresense -db libresense.db -listen :8080

For local development only, LIBRESENSE_MODE=dev disables OIDC and supplies a synthetic dev admin through the same authorization middleware. No other mode value enables this behavior.

Entity, source, and collector boundary

The persistent registry stores exact (kind,id) entity identities, operator metadata, enablement, origin (configured, discovered, or synthetic), and public-status visibility. First-class source instances hold connection identity, configuration, health, capabilities, queue depth, retry/drop counters, and attached-entity counts. Source bindings associate any number of exact entities with a source instance. Observed normalized telemetry incrementally discovers entities, bindings, and capabilities without replacing operator configuration. An explicit configured update claims a previously discovered identity; later discovery cannot demote it. Disabling a binding marks its capabilities stale, while externally supplied observations remain durable facts. Disabling an entity removes it from active operator configuration but does not reject incoming reality.

Source-specific adapters implement the collector contract and emit only normalized entities, source bindings, capabilities, topology, Observations, and Events. Analytical engines consume these normalized semantics and have no source-adapter-specific branches. Collector batches are validated before any write, applied in one SQLite transaction, and keyed by a stable ID and payload hash. Exact retries are no-ops. Analytics are prepared on isolated engine state and published only after the transaction commits.

Kubernetes collector

The first production collector uses Kubernetes Go clients and shared informers; it never shells out to kubectl. Enable it in a pod with the supplied ServiceAccount and RBAC:

go run ./cmd/libresense -db /data/libresense.db -kubernetes \
  -kubernetes-instance production -kubernetes-queue-size 1024

Use deploy/kubernetes.yaml as a starting manifest after replacing the example image and selecting suitable persistent storage. Its RBAC grants only get, list, and watch for the resources consumed. Initial informer cache population discovers entities and topology without emitting operational Events. A bounded queue backpressures watch handlers; failed batches retry exponentially and expose degraded source health and counters through GET /api/v1/source-instances and the Sources UI.

Unannotated objects receive cluster-qualified stable IDs. To merge a Kubernetes object into an entity supplied by another source, configure both exact mapping annotations (partial mappings are ignored):

metadata:
  annotations:
    libresense.io/entity-kind: service
    libresense.io/entity-id: photos

Kubernetes supplies availability, restart/OOM, deployment, placement, and topology evidence. It does not claim latency or request-ratio capability.

Capabilities are evidence-backed or explicitly declared. Metric identity is the metric name plus source type and source instance; catalog responses also include display label, unit, metric type, SLO category, sample count, and last observation. Capability provenance and source-binding identity are separate fields rather than encoded strings. This prevents same-name metrics from different sources being combined for SLO evaluation.

Example configuration:

{
  "metrics": {
    "window_size": 12,
    "trend_samples": 5,
    "trend_relative_change": 0.20,
    "trend_min_duration": "15m",
    "trend_max_duration": "2h",
    "spike_relative_change": 0.50,
    "baseline_warmup": 20,
    "anomaly_z_score": 4.0,
    "anomaly_min_relative_change": 0.20,
    "event_cooldown": "30m",
    "max_streams": 1000,
    "dimensions": ["route"]
  },
  "objectives": [{
    "name": "API latency p95",
    "entity_kind": "service",
    "entity_id": "api",
    "metric": "process.http.latency.p95",
    "operator": "<",
    "threshold": 0.1,
    "effect": "degrading",
    "violation_samples": 2,
    "recovery_samples": 3
  }],
  "incidents": {
    "recovery_window": "10m",
    "max_open": 1000,
    "max_evidence": 200
  },
  "behavior": {
    "relationship_horizon": "1h",
    "response_horizon": "30m",
    "max_recent_events": 500,
    "max_pending_responses": 500,
    "max_relationships": 5000,
    "max_evidence": 20,
    "min_association_exposures": 5,
    "min_association_support": 3,
    "min_association_lift": 1.5,
    "min_drift_samples": 5,
    "drift_relative_threshold": 0.5
  },
  "predictions": {
    "horizon": "1h",
    "max_sequence_length": 3,
    "max_recent_events": 32,
    "max_patterns": 10000,
    "max_pending": 500,
    "max_timing_samples": 128,
    "max_evidence": 20,
    "min_exposures": 5,
    "min_support": 3,
    "min_probability": 0.5,
    "min_lift": 1.5,
    "incident_comparables": 200,
    "min_incident_samples": 3,
    "recurrence_window": "10m"
  }
}

An objective operator describes the desired condition. For example, < 0.1 is violated by values greater than or equal to 0.1.

Accelerated simulation

Generate almost three virtual days without real sleeps:

go run ./cmd/libresense-sim -db ./simulation.db -scenario multi-day -seed 42
go run ./cmd/libresense -db ./simulation.db -listen :8080

Use a new database path for each simulator run because observation IDs are deterministic. Inspect the result with:

curl 'http://localhost:8080/api/v1/stats'
curl 'http://localhost:8080/api/v1/events?limit=100'
curl 'http://localhost:8080/api/v1/incidents?limit=100'
curl 'http://localhost:8080/api/v1/incidents?state=open&limit=100'
curl 'http://localhost:8080/api/v1/observations?limit=25'
curl 'http://localhost:8080/api/v1/relationships?min_support=3&limit=100'
curl 'http://localhost:8080/api/v1/epochs?entity_kind=service&entity_id=service-0&limit=100'
curl 'http://localhost:8080/api/v1/metric-responses?source_event=activity.completed&entity_kind=service&entity_id=service-0&limit=100'
curl 'http://localhost:8080/api/v1/drifts?entity_kind=service&entity_id=service-0&limit=100'
curl 'http://localhost:8080/api/v1/predictions?status=pending&limit=100'
curl 'http://localhost:8080/api/v1/entities/service/service-0/predictions?limit=20'
curl 'http://localhost:8080/api/v1/incidents/INCIDENT_ID/prediction'
curl 'http://localhost:8080/api/v1/incidents/INCIDENT_ID/similar'
curl 'http://localhost:8080/api/v1/slos'
curl 'http://localhost:8080/api/v1/slos/SLO_ID/report'
curl 'http://localhost:8080/api/v1/entities/service/service-0/slos'
curl 'http://localhost:8080/api/v1/maintenance'
curl 'http://localhost:8080/api/v1/status'
curl 'http://localhost:8080/api/v1/status/service/service-0'

Lists accept limit (default 50, maximum 200) and offset. Incidents may be filtered with state=active, recovering, resolved, or open. A single incident is available at /api/v1/incidents/{id}.

Behavioral memory semantics

Every meaningful derived or normalized discrete Event participates, including healthy and health-neutral Events; learning is not limited to incidents. For each source Event, LibreSense counts an exposure. A distinct target Event within the configured horizon contributes at most one support occurrence to a contextual source/target/entity relationship. counterexamples = exposures - support. Same-entity and cross-entity relationships remain distinguishable, and cross-entity targets remain contextualized by entity.

The API reports Laplace-smoothed frequencies:

conditional = (support + 1) / (exposures + 2)
background  = (all occurrences of target Event + 1) / (all Events + 2)
lift        = conditional / background

A relationship is labeled established only when exposures, support, and lift meet all configured minima. This is temporal association, not causation or a future-outcome prediction. Same-name Event pairs are currently ignored, observations older than the last accepted observation are ignored by behavioral learning, and evidence samples are capped while aggregate counts remain durable.

An Event also opens a bounded metric-response window using the entity's latest metric values as baselines. For every metric observed in the window, LibreSense aggregates the signed peak change, relative change, and time to peak. A zero or small response is evidence too, provided a post-Event sample arrives. No goroutine is created per window.

When relationship capacity is full, existing knowledge is retained and deterministic new candidates are declined rather than failing ingestion. /api/v1/stats exposes the cumulative declined_relationship_candidates count.

deployment.changed creates an entity epoch only when it carries a string revision attribute and closes the previous epoch. Once two neighboring epochs each have at least min_drift_samples for the same Event→metric response, drift is recorded when:

abs((current mean signed change - previous mean signed change) /
    max(abs(previous mean signed change), 1e-12)) >= drift_relative_threshold

Drift is interesting but health-neutral; separate objective or failure evidence determines health.

Sequence prediction

LibreSense learns bounded two- and three-Event suffixes. Tokens contain Event name, health effect, and same/cross-entity relation; actual entity IDs remain in prediction evidence rather than becoming product-specific pattern constants. Each possible next Event records matching exposures, support, counterexamples, bounded evidence, and the latest 128 timing samples.

Prediction eligibility defaults to at least five exposures, three supporting outcomes, Laplace-smoothed conditional probability of at least 0.5, and lift of at least 1.5 over the outcome's background frequency. Timing fields are empirical lower (approximately p10), median, and upper (approximately p90) estimates. Small probability differences are not converted into an opaque confidence score.

Predictions persist as pending, occurred, stale, or superseded. They are evaluated by subsequent Events and virtual/real time. Capacity exhaustion declines new sequence patterns or supersedes the oldest pending prediction without rejecting ingestion. Predictions describe historical association, never causality.

Incident prediction endpoints rank at most 200 historical resolved incidents using entity identity/kind and Event-type overlap. With at least three comparables they expose duration quantiles, remaining-duration ranges, unusually-long status, observed automatic versus intervention-associated recovery, and recurrence within ten minutes. Otherwise they explicitly return evidence_quality: "insufficient".

SLOs, maintenance, topology, and canonical status

SLO definitions are persisted per exact entity identity and can only be created for an observed or trusted capability. Availability is explicitly at_least a target ratio. Latency is an explicitly at_most source-aware metric percentile and threshold in the metric's unit; it has no percentage target. Success ratio is explicitly at_least a threshold and error ratio is explicitly at_most a threshold. Metric objectives return insufficient until their configured minimum evidence is present, including when a previously configured metric has become stale.

Availability error budget is (eligible window seconds) × (1-target). Qualifying maintenance overlap is removed from eligible time and reported separately; counted downtime consumes the budget. Ratio budget is calculated in the explicitly configured success/error direction. Latency compliance compares the configured empirical percentile directly with its maximum threshold; it is not represented as a percentage error budget.

Topology is persistent, event-time factual data, separate from statistical behavioral relationships. Supported authoritative edges include placement, ownership, service backends, membership, and explicitly declared dependencies. Edges have validity intervals, so a maintenance decision uses topology valid when the incident opened. Learned behavioral correlations never propagate maintenance.

Maintenance never suppresses Observations, Events, or Incidents. Availability exclusion requires an enabled SLO policy, sufficient announcement lead time, an outage start inside the declared window, declared unavailable impact, and either direct scope or an authoritative event-time topology path. Scope is reported as direct, live_placement, declared_dependency, or other_authoritative_topology, with the reason and path. Only scheduled-window overlap is excluded; overruns and impact beyond the declaration count conservatively. API updates cannot rewrite the original announcement timestamp, and scope, schedule, reason, and expected impact become immutable when the window starts. Completed maintenance remains available to event-time lookup so late-arriving facts from inside its actual window retain the same attribution.

Events occurring in maintenance receive structured maintenance context. Behavioral relationships, metric responses, and sequence tokens retain maintenance-conditioned history in a separate maintenance context; ordinary learned patterns use ordinary, preventing planned drains from teaching the spontaneous-failure model. Status reports flag impact beyond the declared level, window overruns, and durations beyond the empirical p95 of at least three comparable completed maintenance incidents.

Create or replace definitions with POST /api/v1/slos, PUT /api/v1/slos/{id}, POST /api/v1/maintenance, and PUT /api/v1/maintenance/{id}. Delete an SLO with DELETE /api/v1/slos/{id}. New IDs and maintenance announcement timestamps are generated by the service when omitted. Registry, capability, source, topology, and source-aware metric discovery are available under /api/v1/entities, /api/v1/topology, and GET /api/v1/metrics?entity_kind=…&entity_id=…. Close, rather than erase, a topology fact with POST /api/v1/topology/{id}/close. Maintenance transitions use POST /api/v1/maintenance/{id}/start, /complete, or /cancel. JSON bodies are size-limited, reject trailing or unknown fields, and are domain-validated. GET /api/public/v1/status returns only explicitly public entities and a minimal safe status DTO; source settings and observation-source attributes are never returned by API reads.

Web interface

The LibreSense daemon serves its embedded web interface at /. It is a progressively rendered browser application with no Node or frontend build requirement in production. Every operational value is fetched through /api/v1; the UI never opens SQLite or calls storage internals.

Primary routes are /, /entities, /entities/{kind}/{id}, /sources, /incidents/{id}, /slos, /maintenance, /relationships, and /predictions. The UI exposes registry metadata and enablement, first-class source health and counters, source bindings, observed capabilities, simple dependency management, typed capability-gated SLOs, auth state, maintenance scope reasoning, and read-only anonymous views. Server-side authorization remains authoritative.

Runtime branding is embedded from the application-owned ui/static/brand/ directory. The ignored design/ directory remains local source guidance and is never required to build or run LibreSense. Theme choice persists under libresense-theme, respects the operating-system preference until explicitly changed, and is initialized before CSS to avoid a theme flash.

Inspect the deterministic sequence-prediction fixture with:

go run ./cmd/libresense-sim -db ./sequence-prediction.db -scenario sequence-prediction
go run ./cmd/libresense -db ./sequence-prediction.db -listen :8080

Generate a UI review database whose final observation is anchored near the current time:

go run ./cmd/libresense-sim -db ./demo.db -scenario ui-demo -seed 42
LIBRESENSE_MODE=dev go run ./cmd/libresense -db ./demo.db -listen :8080

Then open http://localhost:8080/. The demo includes configured and discovered entities, source-aware capabilities, services with and without latency and request-ratio telemetry, node→pod→service placement, an explicit database dependency, placement- and dependency-derived maintenance, unrelated impact, typed SLOs, historical incidents, recovery history, learned associations, sequences, and evaluated predictions.

Long generated worlds

Fixed scenarios remain available. A separately seeded generator varies service baselines, activity timing, deployment revisions, motif phase, response noise, and failure placement:

go run ./cmd/libresense-sim -db ./world.db -scenario generated -seed 42 -duration 90d -services 10
go run ./cmd/libresense -db ./world.db -listen :8080

Use a fresh database path for a new run because generated observation IDs are deterministic for a seed. Durations accept Go duration syntax or whole-day forms such as 365d. Calendar-scale runs use -duration-years, which advances the end date with time.AddDate and therefore includes Gregorian leap years without converting years to a fixed-hour duration. Generated worlds are produced in bounded chunks (-chunk-days, default 7) with one chunk of lookahead; chunk size does not change world contents or learned results. The simulator advances virtual time without sleeping and reports simulated duration, wall time, observations, events, incidents, relationships, epochs, metric responses, and drift findings. The dedicated epoch-drift scenario demonstrates a health-neutral response change from revision A to B.

Exact sensing semantics

A metric stream is the entity kind and ID, metric name, and configured dimension values. Only numeric metric observations participate.

• sustained_increase / sustained_decrease: the latest trend_samples values are strictly monotonic, their absolute end-to-start relative change is at least trend_relative_change, and elapsed time from first to last sample is within trend_min_duration and trend_max_duration, inclusively. Defaults are five samples, 20%, 15 minutes, and two hours.
• sudden_spike / sudden_drop: the change from the immediately previous value meets spike_relative_change.
• Events for the same stream and pattern are rate-limited by event_cooldown.
• Trend and spike events describe behavior and carry no failing health interpretation.

The baseline uses Welford's incremental mean and variance. Anomaly evaluation starts only after baseline_warmup previous samples. A value is anomalous only when both its absolute z-score meets anomaly_z_score and its relative difference from the mean meets anomaly_min_relative_change. Anomalies have suspect health based on learned_normality; they do not open incidents.

Objectives require consecutive violating samples before emitting objective.violated, and consecutive satisfying samples before objective.restored. This debounce is separate from the incident recovery window. Objective health uses the declared_objective basis.

Intrinsic source facts such as OOMKilled use the intrinsic basis. Event meaning, health effect, and health basis remain separate fields.

Meaningful discrete source facts are normalized without turning every observation into an event. deployment.changed becomes a neutral event retaining its attributes and source-observation evidence. Explicit OOMKilled and service.healthy facts retain their intrinsic health semantics.

Incident lifecycle

A degrading or failing event opens an incident. Further health events for the same entity join its unresolved incident. A positive or recovering event moves it to recovering; another degrading event returns it to active. Resolution occurs when the configured recovery deadline is due. Simulation advances this evaluation with virtual time, while production uses one bounded periodic evaluator. No same-entity observation, per-incident timer, or real sleep is required, and the recorded end remains the calculated deadline.

Incidents are grouped only by exact entity identity. They do not assert causality. Evidence is bounded to the latest max_evidence references and reports how many older references were truncated.

Resource and restart behavior

Per-stream sample windows, resident metric streams, open incidents, incident evidence, behavioral counters, latest-metric keys, response models, sequence outcomes, topology pages, and SLO report details are bounded. Capacity declines are counted in stats. There is no goroutine or queue per sample. Compact Welford statistics, recent windows, objective debounce state, cooldown state, and unresolved incidents are persisted atomically with each observation and its derived events. Restart loads configured maxima rather than raw telemetry history.

Intentionally not implemented

There is no causal inference, root-cause engine, semantic embedding model, topology-aware incident grouping, production collector, OTLP receiver, LLM, message broker, vector database, or external machine-learning framework. The collector contract, registry, and event-time topology foundation are present; real source adapters remain future work. The sequence-prediction fixture uses deterministic, bounded statistical sequence matching, not a general AI or a guarantee of future outcomes.