Application Layer Deployment (Helm)

The application layer is deployed by the chatkit Helm chart, which brings up all FinClaw business microservices (api-gateway / identity-service / orchestrator / ai-infra-rs / claw-fleet-manager, etc.). The base layer (RKE2 + stateful services) must already be ready.

Chart Structure

Chart.yaml metadata: apiVersion: v2, name: chatkit, version: 0.1.0, appVersion: 1.0.0, described as "ChatKit Middleware on Kubernetes (dual-track with Docker Compose / fleet.sh)".

templates/ directory listing:

Template	Render target	Purpose
`platform-service.yaml`	platform namespace	Deployment + Service for all platform microservices (api-gateway / identity-service / orchestrator / ag-ui-server / claw-fleet-manager, etc.); nodePort optional
`shared-claw.yaml`	platform namespace	Profile A (B0 PoC) shared ai-infra-rs Deployment + Service
`tenant-claw.yaml`	claw namespace	Profile B static gitops tenant claw StatefulSet + headless Service (dynamic tenants get the same manifest applied by claw-fleet-manager via the K8s API)
`restate-bundled.yaml`	platform namespace	Optional in-cluster Restate stub, PoC only; production uses externalServices to point at the base layer
`restate-claw-namespace-alias.yaml`	claw namespace	ExternalName alias letting tenant claw resolve Restate cross-namespace via `ext-restate-svc`
`ingress.yaml`	cluster	Entry Ingress; supports splitStreamingPaths to route SSE paths directly to ag-ui-server
`storage.yaml`	platform namespace	Platform PVCs (workspaces / aiInfraData / skillsData / docToolkitCache)
`storage-nfs.yaml`	cluster	NFS CSI static PV (profile=csi-nfs)
`storage-claw-namespace.yaml`	claw namespace	NFS PV and PVC under the tenant claw namespace
`storage-extra.yaml`	platform namespace	Extra volumes
`fleet-manager-rbac.yaml`	platform + claw namespace	Namespace + ServiceAccount + Role/RoleBinding, allowing claw-fleet-manager to manage tenant claw StatefulSets
`identity-keys.yaml`	platform namespace	JWT signing key Secret for identity-service (existingSecret or chart files sources)
`secrets.yaml`	platform namespace	internal-service-token / cron-internal-token / llm-api-key; the chart refuses empty / `CHANGE_ME` / legacy placeholders at render time
`infra-configmap.yaml`	platform namespace	Documents the infra.* endpoints into a ConfigMap
`app-config.yaml`	platform namespace	Packs `chart-config/*` files (services.yaml / ai-infra.yaml / identity-service.yaml / skill-service.yaml / mcp-servers.yaml / persona-templates.json / nginx.conf / inbound.yaml) into a ConfigMap mounted via subPath
`external-services.yaml`	cluster	Exposes out-of-cluster services (base-layer Restate / TEI) as Headless Service + Endpoints for in-cluster DNS
`searxng.yaml`	platform namespace	SearXNG Deployment + Service used by Claw `web_search`
`_helpers.tpl`	-	Helpers like `chatkit.image` / `chatkit.storageProfile` / `chatkit.storageClaimName`

Service images default to the global.registry + global.tag combination; a single-service hotfix can be fully overridden via images.overrides.<svc>.full.

values Layers

Helm multi-file overlay, merged with -f in the following order (later overrides earlier):

File	Purpose	When to use
`values.generated.yaml`	SSOT: service ports / replicas / peer URLs / `servicesEnv`, auto-generated from `config/services.yaml`	Always
`values.yaml`	Global registry / tag / secrets defaults / claw / storage / ingress / infra.restate / clawFleet defaults	Always
`values-basic.yaml`	Default stack overlay (`stack: default`; filename `basic` is a legacy path key)	Always
`values-poc-basic.yaml`	Profile A (B0 PoC shared single claw)	PoC / shared claw scenarios
`values-k8s-fleet.yaml`	Profile B (per-tenant claw via claw-fleet-manager K8s backend)	Production multi-tenant
`values-k8s-ingress.yaml`	Enable external Ingress + SSE sticky session	When an external domain is needed
`values-k8s-nfs-example.yaml`	NFS CSI static PV example	Copy to a site overlay when using NFS storage
`values-k8s-external-example.yaml`	Example for out-of-cluster TEI / Restate via Headless Service	Reference when TEI is outside the cluster
`values-k8s-tenant-examples.yaml`	Static gitops tenant claw items example	Reference when using static claw
`values-site.yaml` (self-built)	Site-specific: image registry, secrets, infra.* host IPs, NodePort, license	Always
`deploy/sites/<customer>/*` (optional)	Multi-customer split: `site.yaml` / `infra.yaml` / `images.yaml` / `secrets.yaml` / `services.env.yaml`	Multi-customer environments

values.generated.yaml is the SSOT, auto-generated by the release pipeline; do not edit it manually. To change service ports or infra endpoints, contact Finogeeks support.

Internal registry image pull

The internal registry deployed by the base-layer ansible uses the domain registry.finogeeks.internal:5000. Set in the site values-site.yaml:

yaml

global:
  registry: registry.finogeeks.internal:5000/finclaw
  tag: "1.1.11"                    # Match the offline package version
  imagePullPolicy: IfNotPresent
  imagePullSecrets: []             # Fill in [{ name: harbor-pull }] if the private registry requires credentials

Private registry credentials:

If the registry requires authentication, first create an imagePullSecret in the cluster, then reference it in global.imagePullSecrets
A single-service hotfix does not require a full package upgrade; use images.overrides.<svc>.full to override a single image

Ingress and SSE long connections

ingress.yaml is disabled by default (ingress.enabled: false). When external domain access is needed, merge values-k8s-ingress.yaml and set ingress.host:

yaml

ingress:
  enabled: true
  host: chatkit.example.com
  className: nginx
  splitStreamingPaths: true
  agUiServer:
    stickySessions:
      enabled: true
services:
  ag-ui-server:
    replicas: 2

Key design:

api-gateway catch-all: the / path hits api-gateway, handling static assets, REST, and auth_request subrequests; proxy-read-timeout defaults to 1200s
SSE long-connection paths (splitStreamingPaths: true): /agent (Prefix) and /events (Exact) bypass api-gateway and hit ag-ui-server directly; proxy-read-timeout is raised to 86400s, proxy-buffering is disabled, proxy_http_version 1.1, and X-Accel-Buffering: no
Sticky session: when ag-ui-server has multiple replicas, ingress cookie affinity pins a client to a specific Pod, because ag-ui-server holds a runtime registry in-process; cookie name chatkit-ag-ui-affinity, max-age 86400
clientMaxBodySize: defaults to 100m, consistent with api-gateway

External Restate / NFS storage / external-services

External Restate (production default)

infra.restate.mode: external (default) - the chart does not deploy Restate; instead it uses externalServices with a Headless Service + Endpoints to expose the base-layer Restate as the in-cluster DNS ext-restate-svc:

yaml

externalServices:
  enabled: true
  items:
    restate:
      name: ext-restate-svc
      ips: ["10.0.1.187", "10.0.1.188", "10.0.1.189"]   # All node IPs in the base-layer restate group
      ports: [26180, 26182]
infra:
  restate:
    ingressUrl: http://ext-restate-svc:26180
    adminUrl: http://ext-restate-svc:26182
    clawNamespaceAlias:
      enabled: true
      serviceName: ext-restate-svc

restate-claw-namespace-alias.yaml then creates an ExternalName in the chatkit-claw namespace so tenant claw Pods can resolve Restate cross-namespace.

For PoC scenarios where the chart should bundle a Restate stub, set infra.restate.bundled.enabled: true to render restate-bundled.yaml (StatefulSet, single replica, docker.restate.dev/restatedev/restate:1.3.2, 10Gi storage). Disabled in production.

NFS storage

When storage.profile: csi-nfs, the chart renders storage-nfs.yaml and storage-claw-namespace.yaml to create a set of static PVs (reclaimPolicy: Retain, storageClassName: "", nfs.csi.k8s.io driver). The NFS server must export the subdirectory for each PV in advance:

shell

mkdir -p /nfs/finclaw/{workspaces-pvc,ai-infra-data-pvc,doc-toolkit-cache-pv,skills-data-pvc}
echo '/nfs/finclaw/workspaces-pvc *(rw,sync,no_subtree_check,no_root_squash)' >> /etc/exports
# Repeat for the other three
exportfs -ra

Profile B also needs a dynamic StorageClass for each tenant claw's volumeClaimTemplates (do not use the unexported /nfs root; the CSI will report "read-only file system").

External-services

external-services.yaml is a general mechanism, not just for Restate. An out-of-cluster TEI can be wired in the same way:

yaml

externalServices:
  items:
    tei:
      name: chatkit-tei
      ips: ["10.0.6.11", "10.0.6.3"]
      ports: [26117]
infra:
  memory:
    enabled: true
    embeddingEndpoint: "http://chatkit-tei:26117/embed"

Installation Steps

Step 1: Generate / reuse Identity JWT keys

bash

./deploy/scripts/generate-identity-keys-for-k8s.sh \
  --output-dir ./identity-keys \
  --create-secret --namespace chatkit \
  --secret-name chatkit-identity-keys

Reference in the site overlay:

yaml

identityKeys:
  enabled: true
  existingSecret: chatkit-identity-keys

Step 2: Prepare values-site.yaml

Copy from values-site-example.yaml and fill in at minimum:

global.registry / global.tag: must match the REGISTRY used by push-images-to-registry.sh
secrets.internalServiceToken: a strong random string; the chart refuses empty / CHANGE_ME / legacy placeholders at render time
secrets.llmApiKey: LLM credential
infra.postgres.host / infra.timescaledb.host: the master IP of the base-layer PG/Timescale; do not use the ext-*-svc Headless DNS for writes
infra.redis.url / infra.nats.url / infra.restate.ingressUrl / infra.restate.adminUrl
externalServices.items.restate.ips: all node IPs in the base-layer restate group

Step 3: Push images to the internal registry

bash

./deploy/scripts/load-images.sh                              # Load into local docker
./deploy/scripts/push-images-to-registry.sh registry.finogeeks.internal:5000/finclaw

Step 4: helm install

Profile B (production recommended):

bash

helm upgrade --install chatkit deploy/k8s/helm/chatkit \
  -f deploy/k8s/helm/chatkit/values.generated.yaml \
  -f deploy/k8s/helm/chatkit/values.yaml \
  -f deploy/k8s/helm/chatkit/values-basic.yaml \
  -f deploy/k8s/helm/chatkit/values-k8s-fleet.yaml \
  -f deploy/k8s/helm/chatkit/values-k8s-ingress.yaml \
  -f values-site.yaml \
  --namespace chatkit --create-namespace

Profile A (PoC shared claw):

bash

helm upgrade --install chatkit deploy/k8s/helm/chatkit \
  -f deploy/k8s/helm/chatkit/values.generated.yaml \
  -f deploy/k8s/helm/chatkit/values.yaml \
  -f deploy/k8s/helm/chatkit/values-basic.yaml \
  -f deploy/k8s/helm/chatkit/values-poc-basic.yaml \
  -f values-site.yaml \
  --namespace chatkit --create-namespace

Before the formal install, run --dry-run=server to validate template rendering and K8s API server-side reference integrity.

License hard gate

identity-service validates DEPLOYMENT_LICENSE_KEY + DEPLOYMENT_LICENSE_CIPHER at startup and fail-fasts if they are missing or decryption fails. The two are injected via servicesEnv (or services.env.yaml):

DEPLOYMENT_LICENSE_KEY: base64url(no padding) 32-byte random key
DEPLOYMENT_LICENSE_CIPHER: 0x01 || IV(12) || tag(16) || AES-256-GCM(plaintext), AAD is ck-deployment-license-v1, plaintext is {"expiresAt","customer","issuedAt"} JSON

The KEY/CIPHER are issued and provided by Finogeeks (the offline package ships with a default 30-day validity). Contact Finogeeks when the license expires or needs renewal; see Update License for the renewal flow.

Post-install Verification

bash

./scripts/verify-k8s-install.sh --gateway http://<node-ip>:30001
# Full e2e
./scripts/verify-k8s-install.sh --gateway http://10.0.0.5:30001 --profile b --e2e
# HTTP-only check (on a machine without kubectl access)
./scripts/verify-k8s-install.sh --gateway http://10.0.0.5:30001 --skip-kubectl

verify-k8s-install.sh checks:

Kubernetes cluster access and namespace permissions
Platform Pod status: api-gateway / identity-service / orchestrator / ag-ui-server / claw-fleet-manager Deployment ready; no Pending / CrashLoop
All platform PVCs Bound
claw-fleet-manager /health: status: ok, runtime_backend matches the Profile (Profile B should be kubernetes)
Profile B tenant claw Pods ready, PVCs Bound, cron Restate registered, cross-namespace access to identity, INTERNAL_SERVICE_TOKEN injected
Option --e2e: register / login / agent SSE smoke test

Supplementary manual checks:

bash

kubectl get all --all-namespaces | grep -E "chatkit|claw"
kubectl -n chatkit get deploy,svc,ingress,pvc,cm,secret
kubectl -n chatkit-claw get pods,svc,pvc
helm -n chatkit history chatkit

For more troubleshooting commands see Troubleshooting and Inspection.

Application Layer Deployment (Helm) ​

Chart Structure ​

values Layers ​

Internal registry image pull ​

Ingress and SSE long connections ​

External Restate / NFS storage / external-services ​

External Restate (production default) ​

NFS storage ​

External-services ​

Installation Steps ​

Step 1: Generate / reuse Identity JWT keys ​

Step 2: Prepare values-site.yaml ​

Step 3: Push images to the internal registry ​

Step 4: helm install ​

License hard gate ​

Post-install Verification ​

Application Layer Deployment (Helm)

Chart Structure

values Layers

Internal registry image pull

Ingress and SSE long connections

External Restate / NFS storage / external-services

External Restate (production default)

NFS storage

External-services

Installation Steps

Step 1: Generate / reuse Identity JWT keys

Step 2: Prepare values-site.yaml

Step 3: Push images to the internal registry

Step 4: helm install

License hard gate

Post-install Verification