APM troubleshooting

Nothing appears in APM

1. Agent mode: run sudo logcore-agent status and look at the APM (OpenTelemetry) section.
   • Disabled means the apm: block in logcore.yaml is still commented out. See Enable APM.
   • If the request counters stay at 0, the app isn't sending to the agent. Check OTEL_EXPORTER_OTLP_ENDPOINT.
   • A Last error line shows why forwarding to Logcore failed.
2. Direct mode: look in the app's own output for exporter errors.
3. Wait up to a minute. Traces are batched by the SDK, and metrics are exported every 60 seconds by default.

Common problems

Symptom	Fix
401 / UNAUTHENTICATED	The API key is wrong or revoked, or the header name is wrong. Use OTEL_EXPORTER_OTLP_HEADERS=x-logcore-api-key=lc_…. In agent mode, check the agent's api_key.
Connection refused	Agent mode: APM isn't enabled, the agent isn't running, or the port differs. Direct mode: use https://logcore.lsofito.com without a port (OTLP/HTTP over 443).
404 on /v1/traces	The endpoint contains a path. Set it to the base URL only: https://logcore.lsofito.com.
gRPC errors through port 443	gRPC uses port 4317. Over 443, use http/protobuf.
Service shows as unknown_service	Set OTEL_SERVICE_NAME.
Traces but no metrics	Set OTEL_METRICS_EXPORTER=otlp and wait for the export interval (OTEL_METRIC_EXPORT_INTERVAL, in milliseconds).
A counter chart is empty	Rates need two points per series. Wait for the second export.
No logs	Set OTEL_LOGS_EXPORTER=otlp, and make sure your logging library is bridged to OpenTelemetry. For Python, set OTEL_PYTHON_LOGGING_AUTO_INSTRUMENTATION_ENABLED=true.
Requests counted twice, or not at all	Requests are entry spans (server / consumer). Check that your framework is instrumented, not just your HTTP client.
Environment facet is empty	Set OTEL_RESOURCE_ATTRIBUTES=deployment.environment=production, or add an env: tag to the agent.
Agent fails to start after enabling APM	Another program uses port 4317 or 4318. Change otlp_grpc_addr or otlp_http_addr, or set one to "off".
Container can't reach the agent	The agent listens on 127.0.0.1 by default. Set otlp_http_addr: "0.0.0.0:4318" and use host.docker.internal. See Apps in Docker.

Test the endpoint by hand

Send one test span over OTLP/HTTP with JSON:

NOW=$(date +%s%N)
curl -sS -X POST https://logcore.lsofito.com/v1/traces \
  -H "Content-Type: application/json" \
  -H "x-logcore-api-key: lc_xxxxxxxx" \
  -d '{"resourceSpans":[{"resource":{"attributes":[{"key":"service.name","value":{"stringValue":"curl-test"}}]},
       "scopeSpans":[{"spans":[{"traceId":"5b8efff798038103d269b633813fc60c","spanId":"eee19b7ec3c1b174",
       "name":"GET /test","kind":2,"startTimeUnixNano":"'"$((NOW-50000000))"'","endTimeUnixNano":"'"$NOW"'"}]}]}]}'

A 200 response means the key and endpoint work, and a curl-test service appears under APM → Services. For agent mode, send the same request to http://localhost:4318/v1/traces without the key header.