Operational configuration

SPOT 2.0 separates producer and reader: the agent publishes schema v2 SAP telemetry to metrics and logs, and the backend reads those same Data Streams. There are no separate Jobs, Events, or WPInfo index fields.

SAP targets and storage

Throughout SPOT, target means one SAP instance. sapconnection[].id is published as source_id and is the same value used as targetId by the API, global selector, alerts, thresholds, ML, and AI. It is mandatory, compared exactly after trimming spaces, and is never resolved through an alias.

A read scope combines an Elasticsearch URL, namespace, and read-only credentials. It never appears as a target in the top bar. One on-prem backend may use several scopes, and one namespace may contain several SAP systems as long as each document has a distinct source_id.

Target and read-scope IDs use ^[A-Za-z0-9][A-Za-z0-9._-]{0,127}$ after trimming and are case-sensitive. Configuration saves read scopes deterministically, one at a time. If one save fails, later scopes remain as drafts and the UI reports how many are pending, preventing lost concurrent updates.

setup:
  elastic_mode: external
  connector_mode: hybrid
  local_connector_read_scope_id: elastic-pro
target_catalog_path: /var/lib/spot/targets/catalog.json
elasticsearch_read_scopes:
  - id: elastic-pro
    alias: Production
    url: https://elastic.example.local:9200
    namespace: PRO
    enabled: true

One connector process has one Elasticsearch writer and N explicit sapconnection IDs. They share spot-integration-PRO-metrics and spot-integration-PRO-logs; every read still filters tenant_id=onprem + source_id=targetId. SPOT never fails over to another scope.

The secret-free catalog is stored at /var/lib/spot/targets/catalog.json and is preserved by backup, restore, and upgrade. The same source_id in two scopes becomes target_binding_conflict; inconsistent SID/client identity becomes target_identity_conflict. Conflicting targets remain visible but cannot be queried until the connector is corrected.

A target without recent observations becomes stale: it remains visible and queryable through its known binding. If its read scope does not respond, it becomes unreachable; that target's APIs return 503 target_storage_unavailable, while targets on other scopes keep working. SPOT preserves the last valid catalog and never rebinds or deletes an SAP because of a temporary outage.

Legacy elasticsearch_targets, default/implicit targets, generated sap-N IDs, alias lookup, and automatic topology migration are not supported. Rebuild an old configuration through setup.

In Configuration, Elasticsearch read cards add, edit, test, and remove scopes with dependency checks. Local connector shows its mode and writer; remote mode explains that standalone connectors are configured on their hosts. SAP targets groups by scope when useful and shows alias, ID, SID, client, origin, last reception, and state. Only alias and enabled are editable; ID and binding are read-only. The IANA time zone is also editable and is validated before external ML is enabled for that target.

Embedded setup enables manage_data_streams_ilm. External targets leave it off, so bootstrap checks connectivity without creating resources. When explicitly enabled, SPOT manages metrics and logs only.

TLS trust and destination-bound secrets

Import corporate CAs in Configuration > Trust store. SPOT displays name, SHA-256 fingerprint, subject, issuer, expiration, and assigned destinations. System packages store them below /etc/spot/tls; portable installs use $SPOT_HOME/config/tls. The UI accepts no arbitrary path and does not delete an assigned CA.

External Elastic, LDAP/AD, and HTTPS AI providers can use system CAs, an imported CA, or explicit insecure_skip_verify. The exception never changes HTTPS into HTTP, permits redirects, or disables DNS/IP controls. A persistent warning remains visible and Save/Test are audited while it is active.

A masked credential belongs to the exact destination identity. If protocol, host, port, base path, authentication, user/bind DN, SNI, CA, or TLS policy changes, enter the full secret again. Test and Save apply the same rule and never test an old secret against a new destination.

LDAP and Active Directory

LDAPS and StartTLS are supported, including RFC1918 destinations. Choose strict validation, an imported CA, or disabled certificate verification. The bind password is bound to protocol, host, port, bind DN, SNI, and TLS policy. SPOT pins the validated IP for the connection to prevent DNS rebinding.

Local users and AD

Open Configuration > Local users with an administrator account. You can create, rename, enable or disable, change the role, reset the password, and delete local accounts. New accounts receive exactly one of these roles:

• admin: complete configuration and administration;
• viewer: dashboards and evidence only, with no mutations.

The Configuration view for a viewer does not mount navigation, forms, tests, or save buttons. If configuration is already loaded, it presents only a read-only status summary; administration remains restricted to admin.

SPOT does not allow the last enabled local administrator to be deleted, disabled, or downgraded. Changing a password, role, or state revokes the account's previous sessions. Identities that have already signed in through Active Directory appear in the same table with provider AD, but are read-only: their credentials and lifecycle are managed in the directory, and their permissions come from group mapping.

The same username can exist once as Local and once as AD without mixing identities. On-premise does not use SAML; SAML remains a SaaS integration.

AI providers

Providers with an API key require HTTPS. They can use a valid CA or disabled certificate verification, with redirects blocked. Loopback HTTP is allowed only without an API key. Private LAN HTTP also requires no API key and an explicit risk acknowledgement; the UI presents that risk separately from unverified HTTPS.

Enabling ML with external Elasticsearch

The presence of data does not enable ML automatically. For every external read scope:

1. assign a valid IANA time zone to every enabled SAP target;
2. configure a writer API key limited to the four displayed ML destinations;
3. select Run ML preflight in Elasticsearch read scopes;
4. enable ML and save only when the result is ML ready.

The preflight is read-only: it requires both spot-integration-<namespace>-metrics and spot-integration-<namespace>-logs to exist as exact native Data Streams; an index, alias, or only one family is insufficient. It queries the canonical trainable series for that target and requires at least one to reach 21 observed days, three weekly cycles, and 70% coverage inside a 42-day window. Unrelated schema v2 documents cannot declare ML ready. It also checks the IANA time zone, target isolation, and effective permissions. It does not create indices, templates, pipelines, Data Streams, or ILM.

After enablement, the SPOT backend runs inference every five minutes, updates forecasts hourly, and trains daily. The external connector remains only the telemetry producer. SPOT writes only derived models, baselines, forecasts, and anomalies, always with the exact source_id; it never modifies metrics or logs streams.

Sources and collections

Each SAP source has a stable ID, IANA time zone, and eight collections:

Family	Default	Content
Metrics	300 s	CCMS and Operations metrics
CCMS events	300 s	Normalized events
Jobs	300 s	Runs and log lines
Program statistics	300 s	Standard ABAP program executions and dependencies
Work processes	300 s	Process snapshots
ABAP dumps	300 s	Sanitized ST22 evidence
RFC failures	300 s	tRFC/aRFC/qRFC/bgRFC without payload
Certificates	86400 s	Public inventory and transitions

Every interval is a wall-clock-aligned fixed frequency. At 300 seconds, the families start at deterministic phases within the first minute of each five-minute block. Every source/family runs independently, so one slow call does not shift the others. Startup and configuration reload perform a staggered bootstrap extraction and coalesce missed cycles instead of queuing late runs.

Each capability reports supported, partial, unsupported, unauthorized, unavailable, or error, plus last success and a sanitized error. For Program statistics, partial retains all main measurements for the acknowledged interval and declares batchstep, rfc, http, and epp unavailable. Their absence does not claim that those steps, calls, or contexts did not occur. The RFC user needs minimum S_RFC access to the standard SWNC_GET_STATRECS_FRAME function in SCSM_NW_WORKLOAD; no transport or custom ABAP artifact is installed. The release must expose the remote READ_START_DATE/TIME, READ_END_DATE/TIME, READ_TIME_ZONE, MAX_RECORDS, and READ_ONLY_MAINRECORDS window. SPOT requires the same invocation descriptor to expose optional IMPORT INT MAX_RECORDS with default -1, READ_ONLY_MAINRECORDS as IMPORT CHAR, and both outputs as EXPORT TABLE; it sends READ_ONLY_MAINRECORDS="X", omits MAX_RECORDS, and sends no artificial limit. Every cycle rereads 15 minutes and attempts to advance up to 30, in frames no wider than 60 seconds. A complete leaf with more than 400 main records retains every execution and reduces only that source's future maximum; exactly 400 does not. Count does not indicate truncation. The protocol may be empty or contain only standard completion rows with a decimal-zero RC; a missing, ambiguous, differently shaped, or non-zero protocol leaves the leaf pending, and its error text is not logged. Explicit truncation also leaves the leaf pending and may split down to one second, with at most 32 Programs RFC reads in any rolling 60 seconds across all sources. The budget survives restarts; when it is exhausted, the continuation waits without holding a SAP connection or starting another worker. Each leaf's acknowledgement confirms only its read boundary. Each horizon advances at most 30 minutes; if lag exceeds the collection interval plus 120 seconds, the next starts immediately. Check the last confirmed SAP time, not only arrival. Minimum-frame saturation and other read/validation errors leave no exportable documents or checkpoint progress. Cancellation or a failed acknowledgement during export leaves the cursor uncommitted even though the destination may retain already received, deduplicable documents. No unread tail is declared covered. Each RFC result has a 20,000-row and 64 MiB decode budget. The cycle retains at most 50,000 documents and 32 MiB of serialized normalized content, including the manifest. Exceeding a limit or repeating an identity with incompatible content discards the entire cycle without export or cursor progress. These limits do not constrain memory inside the native RFC invoke. Every valid leaf retains all main records and a manifest with the exact count and window; only its acknowledgement advances the checkpoint. Before the child starts, source admission has an independent 30-second limit. Only after admission succeeds does it process one subwindow with a fresh 240-second budget and finite termination and reap reserves; it never inherits residual time. An admission timeout or cancellation does not run the worker, bisect a subwindow, export, acknowledge, or move cursors; the next queued source can proceed. The parent retains sanitized descriptor lookup, parameter preparation, output selection, invoke, and decode boundaries even without a terminal record: zero duration marks entry and a positive duration marks completion. They contain only phase and duration, are diagnostic only, and never replace export, acknowledgement, or checkpoint progress. If the child reaches its exact 240-second deadline and the active subwindow is still splittable, the coordinator exports nothing and keeps the cursor and prior acknowledgements unchanged. It durably replaces only that subwindow with two contiguous halves, preserves the rest of the queue and the backoff, and queues the continuation immediately with fair scheduling. A reload cancellation takes precedence, never authorizes this split, and remains a collection cancellation rather than a handoff failure. Missing sub-capabilities do not stop the remaining collections.

A complete read that takes at least 60 seconds or returns more than 400 main records halves the durable maximum for that source's next subwindows, from 60 seconds down to one second. The value survives restarts, does not change acknowledged boundaries, and is not shared with another source. Eight consecutive full-size, acknowledged leaves under ten seconds, with at most 100 main records and 8 MiB normalized each, may double it up to 60 seconds for the next horizon only. Failures reset the streak; timeout or saturation prevents growth for 15 minutes. Smaller leaves can use all 32 admissions in a minute sooner; the durable continuation waits for capacity. It does not guarantee a universal memory ceiling.

The sanitized error separates call-budget exhaustion, minimum-frame saturation, an invalid window, aggregate-budget exhaustion, and an identity/system conflict. Only a version that reports one of those classes supports that attribution; an earlier generic error does not identify the cause.

These limits reduce source and decoder work. Under sustained pressure, preserve the prior configuration, temporarily disable only program_statistics, and confirm that the remaining families keep advancing. Install a corrected, supported connector version; measure the whole process under real SAP load against host headroom before re-enabling the collection. Do not raise resource limits as the first step.

When manage_data_streams_ilm=false, bootstrap only preflights mappings in every backing index. Missing Programs fields produce program_template_update_required: update the external template before ingesting Programs. This action is separate from Job Logs' template_update_required and does not disable Job Logs whose mappings are already valid.

SAP passwords are write-only. SPOT retains a saved password during unrelated edits only while the connection ID, ASHOST, system number, client, and RFC user remain unchanged. If any of those values changes, re-enter the complete password; a mask or empty field does not reuse the previous credential for the new destination.

The same destination controls block cloud metadata, link-local, multicast, and unspecified addresses. RFC1918 is allowed on-prem; insecure_skip_verify does not bypass these limits.

Incremental state and safety

Watermarks/capabilities are stored under /var/lib/spot-agent-sap (or $SPOT_HOME/data). Full pagination, overlap, and deterministic event_id protect restarts; the first Jobs, dump, and RFC backfill covers 24 hours.

Dumps exclude memory/business data, RFC records exclude call payloads, and certificate records never include private keys. Viewer/admin reads remain source-scoped; only admins change configuration. ML uses sparse dump/RFC counts, while certificates are deterministic and non-forecastable. AI tools are read-only and source-scoped.

The backend filters only schema_version=2 within the assigned read scope and for the exact source_id. Documents from an older producer are ignored; SPOT does not migrate them or enable a fallback. Without a genuine v2 signal, the UI shows No data or UNKNOWN according to the view semantics, never a synthetic zero. In remote and hybrid modes, local connector journal state does not participate in standalone-target queries.