Machine learning, score, and forecast

SPOT learns the expected behavior of each SAP target, metric, parameter set, and series. adaptive-robust-v4 is precision-first: a high score requires a change to be rare, material, and harmful for that metric.

ML complements thresholds. It does not confirm a root cause or replace Events, Jobs, Alerts, or SAP context.

Global controls

The global Analysis menu starts disabled and can display:

• Static thresholds: warning and critical lines.
• Expected range: calibrated band and typical value.
• Forecast: prediction for the next 24 hours.
• ML score points: points on the actual series and a 0–100 score selector.

The setting applies to all compatible time charts. An expanded chart can enable Forecast or ML scores for that modal only.

What the model learns

• It uses up to 42 days of observations in internal 5-minute buckets.
• It uses the source timezone to preserve daily, weekly, and daylight-saving patterns.
• It compares level/trend, daily, weekly, combined-seasonality, and intermittent-signal models. It keeps the simplest model with equivalent accuracy in temporal validation.
• During daily training, it selects the model for each forecast horizon's central value separately, based on past forecast errors. Each historical forecast is evaluated using only data available before it was issued, so the same series may use different models at one hour and 24 hours.
• For level signals, it compares the last closed bucket with the median of the last hour and the median of 24 hours. Daily causal validation selects each horizon; if the recent option does not improve by more than 5%, it keeps the 24-hour option.
• It calibrates the expected band from previous prediction errors that did not use future data. The one-hour and 24-hour forecasts calibrate their intervals from errors at their own horizon. It uses the last 14 days of evaluable errors, giving errors from 7 days ago half the weight of recent ones.
• The historical band used by the score and alerts retains a conservative 99% interval. Forecast separately presents a 95% nominal coverage target.
• It remains warming_up until at least 21 days, three complete cycles, and 70% coverage are available.

Coverage follows each metric's canonical collection interval. For example, ICM error rates are collected every 15 minutes even though the engine keeps internal five-minute buckets; those regular samples are not counted as missing data.

SPOT queries the 42 days in seven-day chunks and, when a query is too large or times out, splits it down to six hours. It publishes a model only when the whole window is complete; a failed chunk leaves the variant in training_error and never creates a partial model.

Every view uses the exact SAP target, metricId, parameters, and canonical series key. A visible name is never used to join models. An observation score belongs only to that timestamp; SPOT does not reuse the last score for a newer value.

On-prem models, series, and caches are separated by readScopeId + targetId, and searches filter the exact source_id. Two SAP systems sharing a namespace never train one combined model. Neither the scope ID nor the visible alias is part of SAP identity.

External Elasticsearch on-prem

In an external suite, the connector publishes telemetry only. The SPOT backend discovers targets from its read scopes and, after an administrator enables ML, runs automatically:

• inference every five minutes;
• forecast updates hourly;
• daily training.

Before the first enablement, Configuration > Elasticsearch read scopes > Run ML preflight requires the exact metrics and logs names to be native Data Streams, and checks the IANA time zone per target and permissions of a separate writer credential. It also queries the same canonical series the worker will train: at least one must reach 21 observed days, three weekly cycles, and 70% coverage inside the 42-day window. Other schema v2 documents do not replace that evidence. It is a read-only test and does not modify external Elasticsearch. If it fails, SPOT does not save the enabled state or start workers.

The four destinations for derived models, baselines, forecasts, and anomalies are managed by SPOT. They are not telemetry Data Streams and do not authorize SPOT to create or change metrics, logs, templates, or ILM. An operational error appears as preflight, readiness, writer, storage, or runtime without exposing endpoints, credentials, or raw errors.

Reading the score

The 0–100 score combines:

• Rarity: the calibrated probability of an equal or larger deviation.
• Impact: how far the change exceeds a material minimum for its unit and normal scale.
• Direction: increases normally matter for latency, utilization, errors, and age; decreases matter for free space, availability, and healthy ratios.
• Persistence: windows of 3, 6, and 12 distinct observations complement the current point.

An improvement cannot open an ML incident. For example, lower latency or more free space may leave the expected band without representing degradation.

The calibrated scale makes 75 approximately a rare deviation and 90 a far more exceptional one, provided its impact is also material. Effective thresholds remain configurable with 0 ≤ warning < critical ≤ 100.

Spikes, persistence, and alerts

An isolated point can remain visible evidence without opening an incident. The default precision profile requires two distinct anomalous observations over at least five minutes. Only an exceptional score-100 event with extreme impact can open from one point.

Re-evaluating the same bucket does not increase persistence or create another incident. Two distinct healthy observations resolve the ML condition. warming_up, no_recent_data, insufficient_history, insufficient_coverage, training_error, no_matching_series, and unavailable do not trigger ML.

When adaptive-robust-v4 is activated, each series keeps its v3 historical evidence, scores, and anomalies strictly before its own cutover time. V4 evidence starts at that time: one timestamp never combines both generations. Past alerts are not recalculated. Check each series' model status and generation when reading points before and after the change.

Forecast and explanation

Forecast covers 24 hours using the selected model and a calibrated band. The Now line separates history from the shaded future. New observations update the hourly forecast, but model selection for each horizon changes only after the next daily training. If a closed bucket arrives late or its value is corrected, the next hourly forecast includes that observation. This update does not change daily model selection or past alerts. It can therefore follow a recent level when evidence supports it; a flat line can also be valid. An improvement for memory does not imply the same improvement for every metric. Central value accuracy and interval coverage are separate measures: improving one does not guarantee improving the other.

Point to a timestamp to compare all visible series in a compact card. In history it shows each observed value; in the future it labels predictions as Forecast, rather than measurements. A missing sample remains explicit. Point to an ML marker or forecast to see context for that series beside the card: center, range, and score or calibration status. The card moves to the side with room or stacks on a narrow screen. Select View details to open the full point explanation and close it with Escape; you can also activate an ML marker by click or keyboard. Details include bounds, model type and generation, status and available evidence, including Model confidence index (for example, 91 / 100), which is not a probability. The same flow works in the expanded chart. Long lists scroll within the viewport. A gap never inherits another series' value or an earlier timestamp's value. Each future sample appears only near its own timestamp according to that series' observed cadence; an isolated forecast does not extend across a gap. In history, Sampling coverage (Data coverage) is the share of expected samples received. Validation coverage is the share of validation observations inside the calibrated band. They are different measurements and always belong to the pointed series. A measured ML score 0 or Validation coverage 0% is valid evidence for that exact series and timestamp; if the field is absent, no evidence is available for it. Do not read a missing field as zero.

At each future point, Nominal coverage is the forecast target (95%), not an observed result or the coverage of the historical alert interval (99%). Validation coverage measures the share of actually issued v4 forecast intervals that contained the observed value after their horizon matured. It appears only when issued, evaluable evidence exists; measured 0% is valid, while an absent field does not mean zero. Calibration samples counts evaluable past forecast errors used to estimate the band; it is not the number of issued intervals used to calculate Validation coverage. Calibration status is ready when evidence is sufficient, limited when it is sparse, or insufficient when it cannot yet support a reliable estimate. Do not read the nominal target as achieved coverage. The 21-day training minimum also does not guarantee enough matured forecast errors at every horizon; check each horizon's sample count and calibration status. The 95% target is not guaranteed coverage. Model confidence index is a 0 to 100 scale, not a calibrated probability or a guarantee that a future value will fall inside the band.

To calibrate the operational band, SPOT compares the central value of forecasts actually issued with later closed observations for the same series and horizon. An older interval without an issued central value supplies no measurable error; SPOT does not reconstruct it. A forecast may therefore appear while Calibration status remains insufficient: a long history does not replace issued forecasts that have matured and can be evaluated.

Baseline and forecast expose independent states. An available forecast cannot hide a baseline that is still warming up or lacks sufficient coverage. Messages distinguish missing recent data, insufficient history or coverage, training errors, unmatched series, and unavailability. Ready forecasts refresh once per hour.

If warming_up remains unchanged for more than one training cycle while the chart still receives recent data, check Alerts → ML coverage. The state should identify insufficient coverage or history; a persistent processing error requires review by the SPOT administrator.

Charts and Alert Center explain:

• observed value, typical value, and expected band;
• absolute and relative change;
• point score and multi-bucket score;
• rarity, impact, direction, and persistence;
• model, effective samples, coverage, and score reasons.

Check Alerts → ML coverage for version, generation, timezone, state, coverage, calibration, and last inference before enabling an ML rule. Forecast remains visual evidence and does not open predictive alerts.

Evidence tooltips stay inside the window and can scroll when an explanation is long; point to a chart point again after resizing or scrolling. In Analysis, both ML range controls remain keyboard-operable even when the limits match.