Skip to main content

Forward audit logs to your SIEM

AI Gateway v2 emits structured JSON audit events to pod standard output. It writes one event for each model interaction, one for each rejected authentication attempt, and one for each model list request that model access filters down to nothing.

Events record decisions and metadata only. They never carry prompts, completions, credentials, request or response bodies, token claims, or email addresses. Every event carries a unique audit ID, a UTC timestamp, the outcome, the component, the source, the subjects, an optional target, and an extra block for correlation.

Enable audit events​

Audit events are off by default. Turn them on for the gateway:

values.yaml
ai-gateway-v2:
  audit:
    enabled: true

Setting global.stacklok.audit.enabled: true also turns them on. Either value enables audit events, so audit.enabled: false doesn't turn them off while the global value is true. The change applies when the gateway pods roll out.

Enabled audit events cover every interaction without sampling, and there are no per-event toggles. Filter and route events in your log pipeline.

What gets captured​

Event typeFires whenOutcome
aigw.v2.interaction.completedAn authenticated generation request reaches its final outcome, including an early refusalsuccess, denied, failure, or error
aigw.v2.authentication.failedThe gateway rejects a credential on a protected route, or can't reach a service it needs to validate itfailure, or error when unreachable
aigw.v2.models.no_grantModel access filters a non-empty model list to nothing, and the caller receives a 200denied

Interaction events cover the OpenAI Chat Completions, OpenAI Responses, Anthropic Messages, and Gemini generate and stream-generate endpoints. A streaming interaction emits one event when it finishes, never one per frame. Health endpoints emit no events, and a model list request that returns at least one model emits none.

Event schema​

{
  "time": "2026-10-16T14:22:31.418Z",
  "level": "INFO",
  "msg": "audit_event",
  "logger": "audit",
  "audit_id": "b3d4f7a2-8c11-4f6e-9b0a-1c2d3e4f5a6b",
  "type": "aigw.v2.interaction.completed",
  "logged_at": "2026-10-16T14:22:31.418Z",
  "outcome": "success",
  "component": "v2",
  "source": { "type": "local", "value": "ai-gateway-v2-6d8f9c7b5-x2k4q" },
  "subjects": { "user": "00u1a2b3c4d5e6f7g8h9", "department": "engineering" },
  "target": {
    "model": "claude-sonnet-4-6",
    "provider": "anthropic",
    "route": "/v1/messages"
  },
  "metadata": {
    "extra": {
      "schema_version": 1,
      "request_id": "c0ffee01-2a3b-4c5d-6e7f-8091a2b3c4d5",
      "interaction_id": "7e1d5390-41e5-4f71-b3c2-9a0815793c4a",
      "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736",
      "span_id": "00f067aa0ba902b7",
      "http_method": "POST",
      "http_status": 200,
      "dialect": "anthropic-messages",
      "streaming": true,
      "duration_ms": 8421,
      "terminal_stage": "delivery",
      "reason_code": "delivered",
      "usage_present": true,
      "input_tokens": 1834,
      "output_tokens": 412
    }
  }
}

Subjects appear only after authentication succeeds. user is the caller's canonical subject, and virtual_key_id identifies the virtual key on virtual-key requests. Directory group labels named in directory.groupLabels, such as department, are added alongside them. A rejected authentication event carries no subjects.

Targets carry model, provider, and route, each only once the gateway has resolved it. provider is the provider kind from the gateway configuration: openai, openrouter, anthropic, bedrock-anthropic, gemini-aistudio, or vertex. route is the endpoint path template.

Source is the gateway pod that wrote the event, with type set to local. Events record no client IP address or user agent.

Common extras are schema_version, request_id, http_status, and, when tracing is on, trace_id and span_id for OpenTelemetry correlation. The gateway generates request_id itself, ignores any inbound X-Request-ID, and returns the ID to the client in the X-Request-ID response header. Authentication events add auth_method, which is jwt, virtual_key, or unknown.

Interaction events add interaction_id, http_method, dialect, streaming, duration_ms, terminal_stage, reason_code, and usage_present, plus input_tokens and output_tokens when the provider reported usage. When a governance hook denies or fails a request, hook names the hook entry. When a required hook reports audit annotations, hook_annotations maps the hook entry name to them, on refused and delivered requests alike. The budgets entries use annotations to report the charged subject and amount. The scanning entry reports detection_types for a block, redact_types for a redaction, log_only_types and shadow_types for log-only matches, and unscanned_reason and unscanned_nodes when a scan failure let the request through unscanned.

schema_version is 1. New fields can appear within version 1. A removed field, or a change to a field's meaning, type, or enumerated values, comes with a new version.

Outcomes and reason codes​

outcome is coarse. reason_code says what ended the interaction, and terminal_stage says where: validation, routing, governance, provider, metering, or delivery.

Reason codeOutcomeMeaning
deliveredsuccessThe response reached the client
request_invalidfailureThe request body or parameters were invalid
dialect_unsupportedfailureNo provider serves the Responses API or Gemini streaming
route_not_foundfailureNo provider is configured for the model
governance_denieddeniedA governance hook, such as budgets, refused the request
governance_hook_failederrorA required governance hook gave no usable answer
governance_projection_failederrorThe gateway couldn't read the request for governance hooks
provider_request_failederrorThe call to the provider failed
provider_response_failederrorThe provider's response failed
usage_missingerrorThe provider response carried no usage
metering_failederrorThe usage report to the budgets service failed
client_disconnectedfailureThe client disconnected before the interaction finished
internal_failederrorAny other internal failure

Rejected authentication events use these reason codes:

Reason codeOutcomeMeaning
auth_missing_credentialsfailureThe request carried no credential
auth_invalid_jwtfailureThe token failed validation
auth_invalid_virtual_keyfailureThe virtual key failed validation
auth_insufficient_scopefailureThe token lacks the scope the issuer configuration requires
auth_dependency_unavailableerrorA service needed to validate the credential, such as the directory, was unavailable

Hook-supplied text and provider error messages never appear in these fields.

Separate audit records from application logs​

Each event is one line of JSON on the pod's standard output. Every audit record carries "logger":"audit" and "msg":"audit_event", so shippers should filter on that pair to separate audit records from ordinary application logs on the same stream.

Ship them​

Each example below filters on those same two keys. Adapt hostnames, credentials, and <NAMESPACE> for your environment.

Fluent Bit to Splunk​

[INPUT]
    Name              tail
    Path              /var/log/containers/*<NAMESPACE>*.log
    multiline.parser  docker, cri
    Tag               kube.audit.*
    Refresh_Interval  5

[FILTER]
    Name    grep
    Match   kube.audit.*
    Regex   log "logger":"audit"
    Regex   log "msg":"audit_event"

[FILTER]
    Name          parser
    Match         kube.audit.*
    Key_Name      log
    Parser        json
    Reserve_Data  On

[OUTPUT]
    Name          splunk
    Match         kube.audit.*
    Host          <SPLUNK_HEC_HOST>
    Port          8088
    Splunk_Token  ${SPLUNK_HEC_TOKEN}
    TLS           On
    event_source  aigw-audit
    event_index   aigw_audit

A record passes the grep filter only when both Regex rules match, so the filter works whatever order the two keys appear in.

Fluent Bit to Elastic​

Keep the input and both filters from the Splunk example above, which tag records kube.audit.*, and swap the output. The Match value has to be that same tag, or the output receives nothing:

[OUTPUT]
    Name            es
    Match           kube.audit.*
    Host            <ELASTICSEARCH_HOST>
    Port            9200
    HTTP_User       ${ES_USER}
    HTTP_Passwd     ${ES_PASSWORD}
    tls             On
    Logstash_Format On
    Logstash_Prefix aigw-audit
    Replace_Dots    On

Vector to Kafka​

sources:
  kubernetes_logs:
    type: kubernetes_logs
    extra_namespace_label_selector: 'kubernetes.io/metadata.name=<NAMESPACE>'

transforms:
  parse_json:
    type: remap
    inputs: [kubernetes_logs]
    source: |
      . = parse_json!(.message)
      if .logger != "audit" || .msg != "audit_event" { abort }

sinks:
  kafka:
    type: kafka
    inputs: [parse_json]
    bootstrap_servers: '<KAFKA_BROKERS>'
    topic: aigw-audit
    encoding:
      codec: json
    compression: snappy

Promtail to Loki​

Promtail's match stage selects on labels, so this pipeline promotes logger and msg to labels, drops every line that isn't an audit record, then removes those two labels.

scrape_configs:
  - job_name: aigw-audit
    kubernetes_sd_configs:
      - role: pod
    relabel_configs:
      - source_labels: [__meta_kubernetes_namespace]
        regex: <NAMESPACE>
        action: keep
      - source_labels:
          - __meta_kubernetes_pod_uid
          - __meta_kubernetes_pod_container_name
        separator: /
        target_label: __path__
        replacement: /var/log/pods/*$1/*.log
    pipeline_stages:
      - cri: {}
      - json:
          expressions:
            logger: logger
            msg: msg
            type: type
            outcome: outcome
            component: component
      - labels:
          logger:
          msg:
      - match:
          selector: '{logger!="audit"}'
          action: drop
      - match:
          selector: '{msg!="audit_event"}'
          action: drop
      - labeldrop:
          - logger
          - msg
      - labels:
          type:
          outcome:
          component:

On nodes that use the Docker runtime, replace cri: {} with docker: {}. Promtail has reached end of life. For new installs, use Grafana Alloy, which converts this configuration with alloy convert --source-format=promtail.

CloudWatch and rsyslog​

The CloudWatch agent cannot filter on a JSON key natively. Either ingest everything and apply a log-group metric filter:

{ $.logger = "audit" && $.msg = "audit_event" }

or run Fluent Bit alongside it with the cloudwatch_logs output, using the filters above so only audit lines reach the dedicated log group.

For rsyslog, match both markers before forwarding:

if $msg contains '"logger":"audit"' and $msg contains '"msg":"audit_event"' then {
    action(type="omfwd"
           target="<SIEM_HOST>"
           port="514"
           protocol="tcp"
           template="RSYSLOG_SyslogProtocol23Format")
    stop
}

Retention, tamper evidence, and GDPR​

Configure retention, tamper evidence, and privacy controls in your ingest and storage systems. Use lifecycle policies and write-once storage where your compliance requirements call for them.

Apply field-level redaction before events reach long-term storage. The gateway does not retain or replay emitted audit events.

Operational failure records go to the pod's standard error stream, one per failed request, including a stream that fails after the gateway already sent a 200. Each record carries "event":"operational_failure", the request_id, http_status, outcome, stage, and reason, plus interaction_id, route, model, and provider when known. For a verified directory user, the record includes the user's email.

These records also include a bounded message parsed from the provider's error response. Restrict access to standard error logs, and keep retention short. To log metadata only, turn the message off:

values.yaml
ai-gateway-v2:
  failureLogs:
    includeMessage: false

failureLogs.level sets the records' severity to warn (the default), error, or debug. Operational failure records are separate from audit events, so audit shippers that filter on "logger":"audit" don't collect them.

Conversation content reaches no audit event or log. Screening sends prompt text to Presidio and AWS Bedrock for analysis, but two settings persist prompt content:

  • Conversation capture to the budgets service, configured with budgets.capture. See Budgets and pricing.
  • Snapshot capture, configured with the top-level capture value, which records full request and response bodies for testing. It's unset by default; leave it unset in production.

Next steps​