Viewing Service Logs

Using fleet.sh (Single-node)

./scripts/fleet.sh logs -f <service-name>

Common service names: orchestrator, identity-service, ag-ui-server, ai-infra-rs, delivery, etc. (see Deployment Components for the full list).

Using watch-logs.sh (Single-node)

# Tail after fleet has written the log file
LOG_FILES=./fleet.log ./scripts/fleet.sh up -d
./scripts/watch-logs.sh -- ./fleet.log

# Focus on specific containers only
./scripts/watch-logs.sh chatkit-identity chatkit-llm-adapter -- ./fleet.log

K8s Log Viewing

# Real-time tail
kubectl -n <ns> logs -f <pod> -c <container>

# Batch by label
kubectl -n <ns> logs -l app=<svc> --tail=200

# Previous container logs (CrashLoopBackOff troubleshooting)
kubectl -n <ns> logs <pod> --previous

For Loki queries, see Logging System.

Structured Log Fields

Key fields in JSON logs:

Field	Description
timestamp	Timestamp
level	Log level
service	Service name
tenant_id	Tenant ID
trace_id	Cross-service trace ID
request_id	Request ID
user_id	User ID
flow_id	Flow ID

jq Filtering

docker logs <svc> 2>&1 | jq 'select(.level=="error")'
docker logs <svc> 2>&1 | jq 'select(.trace_id=="<trace_id>")'

Common Error Log Samples and Troubleshooting Approaches

Error Log Sample	Troubleshooting Approach
LLM 连通性检查未通过 (HTTP 401)	API Key is incorrect or expired; verify LLM_API_KEY; inside the container run curl -H "Authorization: Bearer $KEY" $LLM_BASE_URL/v1/models
LLM 连通性检查未通过 (HTTP timeout)	The container uses localhost to access the host LLM; or the intranet proxy is not configured; use the host IP or Docker gateway; inject HTTP_PROXY
contract_version=unknown	The contract version is baked into the image at build time; unknown indicates a defective image - contact Finogeeks support
AI_INFRA_RS_HOST_PROFILE not set, degrading to Tier A	The profile is not explicitly injected in production; add AI_INFRA_RS_HOST_PROFILE=linux-cloud to .env or ConfigMap, then restart
DEPLOYMENT_LICENSE_KEY missing	The license is not configured; init-env.sh prompts for required values; contact Finogeeks to obtain and write it into .env
Deployment license loaded (already expired...)	The license has expired; login and other APIs return 403 (DEPLOYMENT_LICENSE_EXPIRED); check the deployment_license field of http://<host>:26101/health to confirm the expiry time; contact Finogeeks to renew (see "Update License")
Restate cluster unreachable	Restate nodes are not clustered; run curl http://<node>:26181/health on each node; confirm v1.6.2+; ensure the RKE2 token is consistent
PostgreSQL replication lag > 60s	Primary-replica sync lag; run SELECT * FROM pg_stat_replication; check WAL, network, and slave IO
finsafe execution denied	The sandbox policy denied execution; check the finsafe-authority (8090) policy; verify FINSAFE_SHELL_POLICY_ID is correct
Channel deliver_failed: feishu 401	Feishu token expired; re-authorize in Admin -> Tenant -> Channels; verify FEISHU_APP_SECRET
Channel deliver_failed: wecom 40014	WeCom corpid/secret mismatch; same as above; verify WECOM_CORP_ID / WECOM_SECRET
OOMKilled	Container memory limit reached; run kubectl describe pod; adjust resources.limits.memory or reduce concurrency via AI_INFRA_RS_GLOBAL_CONCURRENCY
CrashLoopBackOff	Startup failure loop; run kubectl logs <pod> --previous to see the exit cause; common causes are missing env vars or volume mount failures

Log Aggregation Troubleshooting

Cross-service traces are linked by trace_id across api-gateway / orchestrator / ai-infra-rs / delivery.

For Loki query examples, see Logging System.

References

• Service Health Check
• Access Path Check
• Deployment Components