Skip to main content

On-premise troubleshooting

License required or connector restart loop​

Check the validator and sanitized status first:

systemctl status spot-license.service --no-pager
sudo -u spot /opt/spot/bin/spot-agent-sap license-status --json
sudo -u spot /opt/spot/bin/spot-agent-sap license-check
journalctl -u spot-license.service -u spot-agent-sap.service -n 120 --no-pager

If the license is missing, activate it or install the offline file by following Licenses and renewal. expired, revoked, and invalid block the product; going above user or instance limits only produces a warning and must not cause restarts. Never paste keys or raw responses into a ticket or logs.

If an online license was valid and the remote service becomes temporarily unavailable, SPOT may show up to seven days of grace from the last successful validation, never beyond expiry. A local validator failure does not enable that grace; fix spot-license.service first.

UI does not open​

systemctl status spot-server --no-pager
ss -ltnp | grep 8080
journalctl -u spot-server -n 120 --no-pager

If the port is busy, change it from setup or runtime configuration.

Setup does not save HTTPS​

Validate that certificate and private key exist, belong together, and are readable by the process user. HTTP remains valid when you do not enable TLS yet.

Embedded Elastic does not start​

Check memory, permissions, and logs:

systemctl status spot-elasticsearch --no-pager
journalctl -u spot-elasticsearch -n 120 --no-pager
curl -sS http://127.0.0.1:9200

The service must listen only on 127.0.0.1:9200. Embedded mode must not have a user, API key, CA, or insecure_skip_verify; save embedded again if values remain from an earlier external configuration.

External Elastic HTTP is blocked​

External profiles require HTTPS. A legacy HTTPS profile without transport_profile is safely classified as external; a legacy HTTP profile blocks the new runtime with elastic_transport_migration_required until it is migrated. Select system CAs, an imported CA, or Disable certificate validation. The last choice does not permit HTTP. Only credential-free http://127.0.0.1:9200 can be classified as embedded, and only inside spot-suite.

Elasticsearch pending​

This is valid for spot-suite-external after deferral. Health, users, and configuration remain available. Configure an HTTPS destination, run Test, and save. Do not start an embedded service: this package does not include one.

Permission denied in code or SDK​

Do not give /opt/spot to spot or spot-elastic. System payload must remain root-owned; only data and logs are mutable. Reinstall to repair an insecure old tree. For portable, keep $SPOT_HOME private to its dedicated runtime user.

A private certificate fails​

  1. Check hostname, expiration, and chain.
  2. Import the corporate CA under Configuration > Trust store and assign it.
  3. To temporarily accept the risk, explicitly enable Disable certificate validation and save.

There is no automatic fallback. The exception keeps encryption but does not authenticate the server; the warning remains after reload and restart. Changing TLS policy requires the password or API key again.

A CA cannot be deleted​

SPOT blocks deletion while Elastic, LDAP, or AI uses the CA. Move every affected destination to another CA or system trust and save before deletion. Do not manually edit a fingerprint or enter a file path in configuration.

LDAPS or StartTLS cannot connect​

Check scheme/mode, host, port, bind DN, SNI, and CA. Re-enter the bind password after any identity change; a mask is not reused. Link-local, metadata, multicast, and unspecified addresses remain blocked with certificate validation disabled.

SAP data does not arrive​

Check:

  • RFC SDK availability in the service environment.
  • SAP user and secret.
  • RFC ports.
  • SAP connection marked as enabled.
  • spot-agent-sap logs.

If only CCMS events fails, distinguish an RFC failure from an alert without text. An alert with a valid identity, timestamp, and value is preserved even when SAP omits the message; SPOT displays a stable fallback text. Authorization errors, corrupt responses, or a missing VALUE still stop the cycle and retain the previous watermark. Diagnostics separate eligible nodes, candidates, emitted events, fallbacks, and real failures.

An ABAP dump has no description​

Open the dump from 03 Work Process and check Description status. SPOT shows the generic runtime error description that SAP stores in SNAPT; it does not collect the long text or instance-specific detail shown by ST22.

  • complete: SAP returned a valid description.
  • empty: the read was valid but returned no text.
  • unauthorized: the RFC user could not read the description.
  • unavailable: SAP or the read was temporarily unavailable.
  • unsupported: SPOT could not prove the response boundaries and rejected it.
  • legacy_missing: the document came from an earlier connector that did not collect a usable description.

After upgrading, allow one ABAP dumps cycle and the historical repair to finish. In connector status, confirm that current collection advances and that the abap_dump_description_repair subcapability reaches supported. The repair checks one 30-day window per source/client, appends new observations without overwriting earlier ones, and retries idempotently after interruption. A dump that is older, removed by SAP retention, or has no SNAPT text cannot be reconstructed as ST22 detail.

For unauthorized, check RFC_READ_TABLE access to SNAPT. For unsupported or unavailable, retain the sanitized code and check connectivity and DDIC metadata. Do not attach rows, runtime error identifiers, programs, users, or credentials. Also confirm that Metrics, Work Processes, Jobs, RFC, and Certificates continue to advance; a historical repair failure must not block those families. Open the corresponding dump directly in ST22 when you need its complete instance detail.

Jobs does not advance or shows unsupported_rfc_read_table_shape​

The connector orders requested fields from DDIC metadata before reading RFC_READ_TABLE. This supports both SAP systems that return columns in the requested order and systems that serialize them in physical DDIC order. An ambiguous format or inconsistent metadata is rejected with unsupported_rfc_read_table_shape: SPOT does not export that read or advance its cursor. This contract does not claim compatibility with arbitrary formats whose column boundaries cannot be proven.

After an upgrade, validate one recent execution:

  1. Select the same SAP source and time range in Jobs and SM37. Confirm that start, end, and JOBCOUNT belong to the same execution.
  2. Inspect at least two complete Jobs cycles in spot-agent-sap: the last successful cycle and its cursor must advance without date or time errors.
  3. Open a terminal execution and confirm that its Job Log is complete, empty, or has an explicit status; a failed table read must not present a partial log as complete.
  4. Confirm that the other collections and heartbeat continue to advance.

If the error persists, check RFC_READ_TABLE authorization and retain only the sanitized diagnostic. Do not attach SAP rows, Job names, users, messages, or credentials.

Programs is empty or its capability is incomplete​

Check Configuration > SAP connections > Program statistics and its last check. unauthorized requires minimum S_RFC access to SWNC_GET_STATRECS_FRAME; unsupported means that the release does not expose the remote READ_START_DATE/TIME, READ_END_DATE/TIME, READ_TIME_ZONE, MAX_RECORDS, and READ_ONLY_MAINRECORDS window; partial keeps all main timings for the acknowledged interval while batchstep, rfc, http, and epp remain unavailable. Do not install custom Z code or read an absent subrecord as “no calls occurred”. SPOT requires optional IMPORT INT MAX_RECORDS with native default -1 and READ_ONLY_MAINRECORDS as IMPORT CHAR in that same invocation descriptor, sends READ_ONLY_MAINRECORDS="X", omits MAX_RECORDS, and does not send 500. It uses frames no wider than 60 seconds. A leaf with more than 400 main records retains every execution and reduces only that source's future maximum; exactly 400 does not. Count does not indicate truncation. The protocol may be empty or contain only standard completion rows with a decimal-zero RC; a missing, ambiguous, differently shaped, or non-zero protocol leaves the leaf pending, and its error text is not logged. Explicit truncation also leaves the leaf pending. Explicit truncation may split down to one second, with at most 32 Programs RFC reads in any rolling 60 seconds across all sources. The budget survives restarts; when exhausted, the continuation waits without holding a SAP connection or starting another worker. Each leaf confirms only its read boundary after acknowledgement. Each horizon advances at most 30 minutes; if lag exceeds the collection interval plus 120 seconds, the next starts immediately. Check the last confirmed SAP time, not only arrival. If error still appears, the sanitized diagnosis distinguishes call-budget exhaustion, minimum-frame saturation, an invalid window, the aggregate budget (50,000 documents/32 MiB of normalized content, including the manifest), or an identity/system conflict. Each RFC response retains its 20,000-row/64 MiB limit. Read or validation errors leave no exportable documents and preserve the checkpoint. A valid leaf includes every main record and a manifest with the exact count and window; only its acknowledgement advances the checkpoint. Cancellation or a failed acknowledgement during export also leaves the cursor uncommitted even though the destination may retain already received, deduplicable documents. An unread tail is never published as covered. Widen the range if SAP retention is short and confirm that heartbeat and all other collections advance.

The class is conclusive only when the installed version reports it; an earlier generic error does not prove which class occurred. Before the child starts, source admission has an independent 30-second limit. Only after admission succeeds does it process one subwindow with a fresh 240-second budget and finite termination and reap reserves. It never consumes time left over from an earlier read. An admission timeout or cancellation does not run the worker, bisect a subwindow, export, acknowledge, or move cursors; the next queued source can proceed. When the worker budget reaches its limit, sanitized descriptor lookup, parameter preparation, output selection, invoke, and decode boundaries can survive a missing terminal record. Zero duration marks entry and a positive duration marks completion. Use them only to identify the last observed stage: they contain no RFC values, do not prove that SAP completed the call, and cannot permit export, acknowledgement, or checkpoint movement. If the child reaches its exact 240-second deadline and the active subwindow is still splittable, the coordinator exports nothing and keeps the cursor and prior acknowledgements unchanged. It durably replaces only that subwindow with two contiguous halves, preserves the rest of the queue and the backoff, and queues the continuation immediately with fair scheduling. A rollout or reload cancellation takes precedence, never authorizes this split, and is reported as collection cancellation rather than a handoff failure; a truly missing report remains an error.

A complete read that takes at least 60 seconds or returns more than 400 main records halves the durable maximum for the source's next subwindows, from 60 seconds down to one second. Eight consecutive full-size, acknowledged leaves under ten seconds, with at most 100 main records and 8 MiB normalized each, may double it up to 60 seconds for the next horizon only. Failures reset the streak; timeout or saturation prevents growth for 15 minutes. Smaller leaves can use all 32 admissions in a minute sooner; the durable continuation waits for capacity without changing acknowledged coverage. This reduces the expected working set but does not guarantee a universal memory ceiling. If a small frame still exhausts 240 seconds, the stall remains unresolved.

Under memory pressure, inspect whole-process RSS/PSS, memory.current, memory.high events, PSI, Program statistics duration, and progress of the other families. Reclaim without an OOM does not by itself prove a Go heap leak or native spin, and one heartbeat does not prove continuity. Temporarily disable only program_statistics when necessary to recover the other collections and preserve the prior configuration. Install a corrected, supported connector version; measure the whole process under real SAP load against host headroom, and confirm progress of every family before re-enabling it. Do not raise resource limits as the first step. Disabling Programs is not a permanent solution. Also confirm that ABAP dumps keeps a recent last check even when there are no new dumps, and correlate document times with Programs deadlines. Verify Metrics, Work Processes, Jobs, Job Logs, RFC, and Certificates; a heartbeat alone does not prove continuity.

On unmanaged Elasticsearch, run bootstrap and inspect program_template_update_required. If present, add Programs mappings to the external template. Do not confuse it with Job Logs' template_update_required: the Programs check is independent and must not disable a valid Job Log reader. See Programs for timing and dependency semantics.

Dashboards have no data​

Check that the selector contains the expected SAP source_id, not a read-scope ID. Under Configuration > Elasticsearch read, validate URL+namespace and Test, then confirm documents with schema_version=2, tenant_id=onprem, and a non-empty source_id in the exact metrics and logs streams. A failed scope affects only its own targets.

In remote and hybrid modes, these reads do not depend on the local connector capability journal. If only legacy documents exist, SPOT ignores them and the view remains at No data or UNKNOWN; correct or upgrade the producer. There is no automatic migration or fallback to another identity or read scope.

target_binding_conflict means the same source_id appeared in two scopes; target_identity_conflict means incompatible SID/client values. Correct the producing connector—renaming it or choosing another Elasticsearch does not fix the conflict. If Events has data but Operations does not, validate one metric with the same target and time range.

stale means that the SAP has not sent a recent observation. The target is preserved and remains bound to the same scope; inspect the connector and its schedule. unreachable means that the assigned read scope does not respond: test that card in Configuration and fix its network, TLS, or read credentials. The API returns target_storage_unavailable only for that scope's targets; do not expect or configure failover to another Elasticsearch.

DB alternates between No data and OK​

Oracle health uses only Oracle / CCMS_ORA_health_check_run_pfcl / Database health check / Last succ. database check. Confirm that exact identity arrives for the selected source_id. SPOT never infers health from latency, backups, or tablespaces. Without a genuine signal for 90 minutes, the correct stable result is UNKNOWN; refreshes or another SAP in the namespace must not make it alternate.

Agent cadence or resource use​

An observation can appear under the previous bucket-start label; Observed still represents SAP time. Inspect scheduled_at, start_delay_ms, duration_ms, skipped_cycles, @timestamp, and ingested_at in the logs. Do not widen freshness to hide scheduler lag.

Derive CPU from the difference between two process_cpu_time_ns samples. Also inspect RSS/PSS, heap/GC/cgo, threads, queue, RFC, rows, and bytes. The runtime must coalesce cycles, stay below two cores/768 MiB, and return near baseline after two cycles. Otherwise stop the upgrade and retain only sanitized evidence.

Bootstrap returns management_disabled or preflight fails​

management_disabled means the target permits reads/writes but does not authorize SPOT to manage storage. Enable SPOT manages Data Streams and ILM only with approval from the Elastic owner. If preflight fails, correct the version, template/ILM/index privileges, or disk space before retrying; SPOT does not mutate the cluster when preflight fails.

With management disabled, management_status: unmanaged is expected. detached means SPOT managed the target previously and has now stopped reconciliation without deleting existing resources.

AI does not respond​

Review provider origin/base path, TLS policy, CA, key, and outbound connectivity. Test uses the runtime policy and records a sanitized result. Providers with an API key require HTTPS; loopback HTTP works only without a key, and private LAN HTTP without a key requires its separate acknowledgement.