Architecture Decisions
Architecture Decision Records (ADRs) for every significant design choice in log0 - why Kafka, why ClickHouse, why deterministic fingerprinting, why manual ACK, and more.
About These Records
Architecture Decision Records (ADRs) document the why behind design choices. Code shows you what was built. ADRs show you why it was built that way - including the alternatives that were considered and rejected.
Each record follows this format: Context (what problem forced a decision), Decision (what was chosen), Alternatives considered (what was rejected), and Consequences (trade-offs accepted).
ADR-001: Kafka-API Event Bus (Redpanda)
Status: Accepted · updated 2026-06 (Apache Kafka → Redpanda)
Context
log0 needs to move log events through a multi-stage processing pipeline: ingestion → normalization → clustering → incident creation → notification. The pipeline stages have different throughput requirements, can fail independently, and need to scale independently. The ingestion gateway must return immediately to the client - it cannot wait for downstream processing to complete.
A message queue or event bus is required to decouple the stages.
Decision
Use a Kafka-API event bus between all services. The deployed broker is Redpanda v24.2.7 - a single Kafka-compatible binary with no Zookeeper. The original local stack used Apache Kafka (Confluent 7.5.0); the migration to Redpanda changed only the bootstrap-servers address. The Kafka wire protocol, topics, consumer groups, and all producer/consumer code are identical across both, so everything below applies unchanged.
Alternatives Considered
| Alternative | Why Rejected |
|---|---|
| RabbitMQ | AMQP-based push model makes it harder to replay events. No native consumer group semantics. Log replay (for debugging or DLQ recovery) requires additional tooling. |
| AWS SQS/SNS | Vendor lock-in. No consumer group semantics - SQS requires one queue per consumer. Fan-out to multiple consumers requires SNS → multiple SQS queues. Replay is not native. |
| Redis Streams | Suitable for low-throughput use cases. At 10,000 logs/second, Redis becomes a bottleneck. No partition-level ordering guarantee keyed by tenantId. |
| Direct HTTP | Synchronous coupling. If the normalization service is down, ingestion fails. Does not support the at-least-once reliability model. |
Consequences
Accepted trade-offs:
- An event bus is still infrastructure to run locally (Docker Compose) - though Redpanda makes this a single container
- Message ordering is guaranteed per partition (per tenantId), not globally
Benefits realized:
- Each service can scale independently based on its consumer lag
- Consumer group semantics allow multiple normalization instances to share load automatically
- Message replay is native - reprocessing from a given offset requires no extra tooling
- The DLQ pattern is a natural extension of the topic model
- Producer idempotence (
enable.idempotence=true) prevents duplicate messages under retries - Redpanda specifically: no Zookeeper, one binary, lower memory (~1 GB saved vs a JVM broker + Zookeeper), and faster startup - ideal for the single-host deployment. See Deployment.
ADR-002: ClickHouse for Log Event Storage
Status: Accepted
Context
log0 stores every normalized log event for historical queries and AI summary generation. The access patterns are:
- High-volume writes: up to 10,000 events/second
- Aggregation queries:
GROUP BY fingerprint ORDER BY COUNT DESCover millions of rows - Time-range scans: "show all ERROR logs for tenant X in the last 24 hours"
- Filtered reads: "show logs matching fingerprint Y for incident Z"
These are analytical access patterns, not transactional ones.
Decision
Use ClickHouse as the log event store.
Alternatives Considered
| Alternative | Why Rejected |
|---|---|
| Elasticsearch | Excellent full-text search, but expensive to operate at scale. Storage is 3–5x ClickHouse for the same data. Aggregation queries (GROUP BY, COUNT) are significantly slower on Elasticsearch than ClickHouse. |
| PostgreSQL | Already used for incident state. Not suitable for log storage - OLTP database, poor compression for time-series data, slow on large aggregations. GROUP BY fingerprint over millions of rows requires full table scans. |
| DynamoDB | No native aggregation support. Running GROUP BY queries requires exporting to Athena. Operational complexity disproportionate to the use case. |
| Loki (Grafana) | Opinionated log label model. Limited query language. Better suited for infrastructure logs than application event analytics. |
Consequences
Accepted trade-offs:
- ClickHouse has a different query dialect (not standard SQL in all aspects)
- ACID transactions are not supported - not appropriate for transactional writes
- Learning curve for engineers familiar with PostgreSQL but not ClickHouse
Benefits realized:
- Columnar storage means
GROUP BY fingerprintreads only the fingerprint column - dramatically faster than row-based alternatives - Native compression reduces storage cost 5–10x compared to Elasticsearch for log data
DateTime64type provides nanosecond precision for accurate log correlationMap(String, String)type forattributesallows schema-free structured data without migrations
ADR-003: Deterministic SHA-256 Fingerprinting
Status: Accepted
Context
Incident deduplication requires grouping "similar" log events together. Two errors that differ only in dynamic values (user ID, IP address, timeout duration) should be treated as the same error pattern. The question is: how do you define "similar"?
Decision
Use deterministic SHA-256 fingerprinting based on a normalized message template.
The fingerprint formula:
fingerprint = SHA-256(
serviceName + "|" +
messageTemplate + "|" + ← dynamic values stripped
exceptionType + "|" + ← nullable
firstStackFrame ← line numbers stripped
)Message templating strips dynamic values before hashing:
- Numbers →
<number> - IP addresses →
<ip> - UUIDs →
<uuid>
Alternatives Considered
| Alternative | Why Rejected |
|---|---|
| ML-based semantic clustering | Requires training data, model maintenance, and introduces non-determinism. The same error may be clustered differently after a model update, breaking existing incidents. No reproducibility guarantee. |
| Edit distance / Levenshtein | O(n²) complexity. At 10,000 events/second, pairwise comparison is computationally infeasible. |
| Exact message match | Too strict. Two instances of timeout after 30000ms and timeout after 28514ms would create separate incidents. |
| Exception class only | Too loose. NullPointerException appears everywhere. Grouping by exception type only would aggregate unrelated errors. |
Consequences
Accepted trade-offs:
- Novel log formats with irregular structure may not template cleanly, producing noisy fingerprints
- The regex patterns for stripping dynamic values must be maintained as log formats evolve
- Edge cases: logs with no stack trace, logs in non-English languages, logs with custom dynamic patterns
Benefits realized:
- Fully deterministic - the same log always produces the same fingerprint, regardless of time or deployment
- No infrastructure required beyond a SHA-256 hash function
- Fingerprints are stable across deployments - an incident created today and a reoccurrence next week will share the same fingerprint and be correctly linked
- Zero false negatives for structurally identical errors (unlike fuzzy matching)
ADR-004: Always-ACK DLQ Pattern
Status: Accepted
Context
Kafka consumers in log0 use manual offset acknowledgment. When a consumer fails to process a message (normalization error, serialization failure, downstream Kafka publish failure), a decision is needed: what should happen to the unprocessed message?
The naive answer is to not ACK - let Kafka redeliver the message. But if the message itself is the problem (malformed payload, unexpected schema), it will fail on every redelivery, permanently stalling the partition.
Decision
On any processing failure:
- Wrap the original event in a
DlqEvent(capturingoriginalEvent,errorMessage,failedAt,failedAtTs) - Publish the
DlqEventtoraw-logs-dlq - Always ACK the original offset - even if the DLQ publish fails
Alternatives Considered
| Alternative | Why Rejected |
|---|---|
| Retry without DLQ | A bad message is retried indefinitely. One malformed event from a misconfigured service stalls the entire partition for all tenants sharing that partition. |
| Discard on failure | Data loss. No ability to replay or investigate failures. |
| DLQ only if retries exhausted | Adds retry delay before DLQ capture. For a known-bad message, retries add latency without benefit. |
| Block partition on failure | Guarantees ordering but makes the system brittle. One bad actor (a service sending malformed logs) can halt the entire ingestion pipeline. |
Consequences
Accepted trade-offs:
- Messages that fail DLQ publish are lost (the ACK has already been sent). This is an acknowledged edge case - a failure to write to the DLQ is treated as fatal and triggers an alert.
- The DLQ must be monitored. A growing DLQ is a silent signal that something is broken upstream.
Benefits realized:
- The ingestion and normalization pipelines are never stalled by individual bad messages
- Failed events are preserved with full context (
originalEvent, error, timestamp, service) for post-mortem analysis - Replay is possible: fix the bug, re-publish
originalEventtoraw-logs, the pipeline processes it correctly
ADR-005: Manual Kafka Offset Acknowledgment
Status: Accepted
Context
Kafka consumers can manage offset commits in two ways: automatically (Kafka commits periodically, regardless of processing state) or manually (the application commits only when it decides to).
Decision
Use manual offset acknowledgment for all Kafka consumers in log0.
Configuration:
kafka:
consumer:
enable-auto-commit: false
listener:
ack-mode: manualConsumer code:
@KafkaListener(topics = "raw-logs")
public void consume(RawLogEvent event, Acknowledgment ack) {
try {
// process...
producer.publish(normalized);
ack.acknowledge(); // commit only on success
} catch (Exception e) {
dlqProducer.publish(dlqEvent);
ack.acknowledge(); // commit after DLQ (ADR-004)
}
}Alternatives Considered
| Alternative | Why Rejected |
|---|---|
| Auto-commit | Kafka commits the offset after a configurable interval, regardless of whether processing succeeded. If the service crashes after committing but before completing processing, messages are silently lost. At-most-once semantics - not acceptable for incident data. |
Consequences
Accepted trade-offs:
- If the service crashes after processing but before ACKing, the message is reprocessed. Consumers must be idempotent.
- Slightly more complex consumer code
Benefits realized:
- At-least-once delivery guarantee - no message is ever silently lost
- Combined with the DLQ pattern, every message either succeeds or is captured for replay
ADR-006: Custom Kafka Serializers
Status: Accepted
Context
Spring Kafka provides a JsonSerializer and JsonDeserializer that use Jackson to convert Java objects to/from bytes. They work automatically but rely on Spring's managed Jackson ObjectMapper, which introduces coupling to Spring's auto-configuration and Jackson version management.
Decision
Write custom serializers and deserializers for every Kafka event type.
Each serializer is a straightforward wrapper:
public class RawLogEventSerializer implements Serializer<RawLogEvent> {
private final ObjectMapper objectMapper = new ObjectMapper()
.registerModule(new JavaTimeModule());
@Override
public byte[] serialize(String topic, RawLogEvent data) {
return objectMapper.writeValueAsBytes(data);
}
}Consequences
Accepted trade-offs:
- More classes to maintain (one serializer + one deserializer per event type)
- Jackson version must be explicitly managed per serializer
Benefits realized:
- Full control over serialization behavior - no Spring magic
- Custom serializers can be unit tested independently
- Jackson version can be chosen per serializer (tools.jackson 3.x for ingestion, com.fasterxml.jackson 2.x for normalization - intentional due to Spring Boot 4 transitional period)
- Schema evolution is explicit: changing a serializer is a deliberate code change, not an implicit configuration change
ADR-007: tenantId as the Kafka Partition Key
Status: Accepted
Context
Kafka assigns messages to partitions based on the message key. The partition a message lands on determines which consumer instance processes it (within a consumer group). The key design determines two things: ordering guarantees and load distribution.
Decision
Use tenantId as the Kafka message key for all topics in the data pipeline (raw-logs, normalized-logs). Use incidentId as the key for notification-events.
Reasoning
All messages with the same key land on the same partition, and Kafka guarantees ordering within a partition. Setting tenantId as the key means:
- All logs from the same tenant are processed in order. This is critical for the Clustering Service - if two logs from the same tenant are processed out of order, the clustering time window may be computed incorrectly.
- Tenant data is isolated to specific partitions. While this is not a security boundary (all consumers read all partitions), it simplifies reasoning about which consumer instance is handling which tenant's data.
notification-events uses incidentId as the key because multiple notifications about the same incident (created, assigned, resolved) must be processed in order by the Notification Service to correctly update the Slack message.
Consequences
Accepted trade-offs:
- Uneven partition distribution if tenant log volume is highly skewed (one large tenant, many small ones). Mitigated at scale by sub-tenant keying (e.g.,
tenantId:serviceId). - A tenant cannot be distributed across multiple consumer instances within the same group - all their logs are on one partition. Horizontal scaling applies across tenants, not within a single tenant.
Benefits realized:
- Per-tenant ordering guarantee - essential for correct clustering
- Simple, predictable routing - given a tenantId, you know which partition its events are on
- Natural foundation for per-tenant rate limiting at the partition level
ADR-008: Strategy Pattern for LLM Provider
Status: Accepted (Java implementation, March 2026). Updated September 2026: ai-service is Python; provider selection uses LangChain and LLM_* / API-key env vars. The goal-swap models without changing incident-service- is unchanged.
Context
The AI Summary Service must call an external LLM to generate incident summaries. The LLM landscape is fragmented: OpenAI, Azure OpenAI, Google Gemini, Anthropic, Groq, and self-hosted models all have different base URLs, authentication schemes, and request/response shapes. Hardcoding any single provider creates lock-in and makes development difficult (not everyone has a paid OpenAI key).
Decision
Use the Strategy Pattern to abstract LLM provider selection behind a single interface.
public interface LlmProvider {
String generateSummary(String prompt);
}Each provider implements this interface independently. The active implementation is selected at startup via @ConditionalOnProperty:
# application.yml
ai:
provider: groq # swap to: openai | gemini | anthropic | azureDefault provider: Groq. Groq offers a free API tier with an OpenAI-compatible endpoint (/openai/v1/chat/completions), making GroqProvider nearly identical in code to OpenAiProvider. This means the system works out-of-the-box with a free API key during development.
Alternatives Considered
| Alternative | Why Rejected |
|---|---|
| Hardcode OpenAI | Requires a paid API key. Breaks local dev for contributors without one. Any provider change requires code surgery across the service. |
| LangChain4j or Spring AI | Rejected for the original Java service. The Python rewrite uses LangChain for summaries and planned LangGraph investigations. |
| Kafka event for summary request | Adds a new topic and consumer just to trigger an LLM call. The AI Service is already a dedicated service - a synchronous REST call from Incident Service is simpler and sufficient. |
Consequences
Accepted trade-offs:
- Each
LlmProviderimplementation must handle its own HTTP client, error handling, and retry logic. Some duplication across providers. - Adding a new provider requires a new class and a new
@ConditionalOnPropertyconfig block.
Benefits realized:
- Swap providers by changing one config line - no code changes, no redeployment of other services
GroqProviderworks with a free API key, unblocking local development- Each provider is independently unit-testable
- Same pattern already used in the codebase:
OccurrenceStoreinterface →InMemoryOccurrenceStorein the Clustering Service
Implementation update (September 2026)
The Java ai-service was replaced by a FastAPI service under services/ai-service/:
- HTTP contract unchanged:
POST /api/v1/summaries(202, background work), callbackPATCH /api/v1/incidents/{id}/ai-summarywith{ "aiSummary": "..." }. - Providers:
app/llm/resolvesLLM_SUMMARY,LLM_TRIAGE, etc. (provider:model) via LangChain; platform keys in.env(GROQ_API_KEY, …). - Observability:
configure_tracing()mirrors LangSmith settings from pydantic-settings intoos.environ. - Incident caller:
AiSummarizerin incident-service usesRestTemplate+ JSON body.
The Java LlmProvider / ai.provider YAML described above is historical reference for the first implementation.
ADR-009: Stateless JWT Validation in Each Service
Status: Accepted
Context
log0 has a dedicated auth-service (port 8086) that issues JWT access tokens on login. When a protected service (e.g. incident-service) receives a request carrying a JWT, it must verify the token is valid. There are two architectural options:
- Remote validation - each service calls auth-service on every request:
GET /api/v1/auth/validate?token=... - Local validation - each service validates the JWT locally using the shared
JWT_SECRETsigning key
Decision
Use local (stateless) JWT validation in every protected service via a JwtAuthFilter (OncePerRequestFilter). Each service holds the JWT_SECRET as an environment variable and verifies the HS256 signature and exp claim directly - no network call to auth-service.
API key validation (ingestion path) remains remote - the Ingestion Gateway calls POST /api/v1/auth/validate-key on auth-service because API keys require a database lookup (hash comparison against the API_KEY table) that cannot be done statelessly.
Alternatives Considered
| Alternative | Why Rejected |
|---|---|
| Remote JWT validation on every request | Adds network latency on every authenticated API call. Makes auth-service a synchronous dependency - if it is slow or down, all API calls fail. Unnecessary given JWT is cryptographically self-verifiable. |
| API Gateway handling all auth | Would require introducing Spring Cloud Gateway or a similar proxy. Adds infrastructure complexity that is not justified for the current number of services. Can be added later without changing the JWT design. |
| Opaque tokens (session IDs) | Would require a stateful session store (Redis) shared across services. Adds infrastructure dependency and per-request DB lookup. JWT's stateless verifiability is a key advantage for a distributed system. |
Consequences
Accepted trade-offs:
JWT_SECRETmust be shared across all services that validate tokens. Secret rotation requires redeployment of all protected services simultaneously.- Issued tokens cannot be revoked before expiry (short 1-hour TTL mitigates this). Logout only revokes the refresh token - the access token remains valid until it expires.
- Each protected service must include
JwtAuthFilterandJwtUtil- small duplication, mitigated by a shared library (Phase 0.4).
Benefits:
- Zero latency overhead on the hot API path - no auth-service call per request
- auth-service is not a runtime dependency of incident-service for reads
- Horizontally scalable - each service instance validates independently with no shared state
ADR-010: Python and LangGraph for the AI Service
Supersedes the "no framework" part of ADR-008. The summary path keeps its HTTP contract.
Context
ADR-008 gave the AI service a Strategy-pattern provider layer in Java, enough for one prompt in and one summary out. Investigating an incident is a different shape of work: gather evidence from several places in parallel, loop while the model wants more, stop for a human to review, resume after a restart, and never lose the run. That needs a stateful graph with checkpoints and a way to pause, which the Java ecosystem did not offer at the time.
Decision
Rewrite the AI service in Python 3.13 with FastAPI and LangGraph 1.2, with a Postgres checkpointer. The POST /api/v1/summaries contract is byte-compatible, so incident-service changed nothing. Nodes stay thin: the decisions between them live in plain, tested functions (graph/routing.py), and the graph builder only wires them.
Alternatives Considered
| Alternative | Why Rejected |
|---|---|
| Stay in Java (Spring AI, LangChain4j) | No equivalent of interrupt-and-resume with durable checkpoints; weaker tooling for evaluation. |
| A hand-written state machine | Re-implements checkpointing, parallel fan-out and interrupts, the parts that are hardest to get right. |
| A multi-agent "swarm" | Hard to bound in cost and time, and hard to evaluate. One graph with explicit budgets is easier to reason about. |
Consequences
- The repository now has two languages; the AI service has its own Docker image and CI job (uv, ruff, pytest).
- Every run has hard limits (tool calls, rounds, time) enforced in code, not requested from the model.
- A run survives a restart: it resumes from its last checkpoint instead of starting again.
ADR-011: Structured Changes, Deterministic Diff
Context
When asked for a unified diff, language models often produce one that looks right and does not apply: wrong line numbers, mangled context, a missing newline marker. A fix the engineer cannot apply is worse than no fix.
Decision
The model proposes edits: file, the exact old text, the new text. Code does the rest. It checks that the old text occurs exactly once (overlapping matches count), refuses a match that differs only in whitespace, builds the unified diff itself with difflib, and runs git apply --check in a private worktree of the pinned commit. Only a diff that passes is stored. A failed validation goes back to the model once or twice with the exact reasons, then the run gives up honestly.
Alternatives Considered
| Alternative | Why Rejected |
|---|---|
| Trust the model's unified diff | Fails often enough to make the feature unusable. |
| Ask for whole files | Costly, and silently rewrites code the model did not mean to touch. |
Consequences
- A stored diff always applies cleanly to the commit the investigation read; a bad proposal never reaches the user.
- Whitespace mismatches cost a retry. Diffs over 500 KB are refused.
- The demo repository's own tests can be run against the diff in the evaluation (see ADR-017).
ADR-012: A Local Clone and ripgrep, Not Code-Search APIs or Embeddings
Context
The investigation has to read a customer's private code: find the file named in a stack trace, the commits that touched it, the line that fails.
Decision
Keep a disposable partial clone per repository, pin the commit at the start of the investigation, and use git and ripgrep through bounded, read-only tools (list, search, read, recent commits, blame, show commit). The token is injected for the duration of one git command and never written to .git/config. Secret-looking files (.env, keys) and paths that climb out of the checkout are refused.
Alternatives Considered
| Alternative | Why Rejected |
|---|---|
| GitHub code-search API | Default branch only, rate limited, no blame or history. |
| Embeddings over the repository | Stale after every push, costly to build, and worse than exact search for debugging: a symbol name is a better key than a vector. |
Consequences
- Deterministic, citable results: every cited line was really returned by a tool.
- Disk use per connected repository; the cache is safe to delete.
- Only GitHub is supported, with a personal access token (read-only is enough).
ADR-013: pgvector in the Existing Postgres, with Plain SQL
Context
Runbook search and long-term memory need nearest-neighbour search over embeddings, per tenant, on a host with no room for another service.
Decision
Enable pgvector in the Postgres log0 already runs (the pgvector/pgvector:pg16 image keeps the same data volume). Use plain SQL: a vector(384) column, an HNSW index with cosine distance, and the tenant filter written out in the query. MMR selection, relevance grading and query rewriting are code on top. The default embedder is a built-in hashing embedder that needs no model or network; a Hugging Face sentence-transformer is opt-in because runbook text then leaves the service.
Alternatives Considered
| Alternative | Why Rejected |
|---|---|
| A vector database (Chroma, Qdrant, Pinecone) | Another service to run and secure, with its own tenancy model. |
langchain-postgres or LangGraph PostgresStore | Hides the parts that matter most here (the tenant filter, the model filter, the transaction that replaces a runbook) and adds table layouts we do not control. |
Consequences
- One database to back up; tenant isolation is a
WHEREclause with tests, and a mutation check proves the tests notice its removal. - The embedding size is part of the column type; another model size needs a migration.
- With the default embedder, matching is by shared words, not meaning.
ADR-014: A Persisted Event Log Behind the Stream
Context
An investigation takes minutes. The console must show progress, survive a dropped connection, and show the same history to someone who opens the page later.
Decision
Every step writes an event (sequence number, type, small payload) to ai.investigation_event. The stream endpoint reads that log and sends Server-Sent Events with the sequence number as the event id; a reconnecting client sends Last-Event-ID and receives exactly what it missed. The console reads the stream with fetch, not EventSource (which cannot send an Authorization header), and falls back to polling /events?after= when streaming does not work. Events carry facts and one-line summaries, never prompts or raw log content.
Alternatives Considered
| Alternative | Why Rejected |
|---|---|
| WebSockets | Bidirectional is not needed, and it is harder through the Vercel proxy. |
| In-memory progress | Lost on restart, invisible to a second viewer. |
Consequences
- The history of a finished run is the same log the live view used.
- Waiting for a human review is not an end of stream: the run stays open across disconnects.
- Streaming through a proxy that buffers degrades to polling instead of failing.
ADR-015: The Model Proposes, Code Decides
Context
Language models are fluent and sometimes wrong, and an incident report that cites evidence that does not exist, or states a confidence nobody earned, is worse than none.
Decision
Every claim is checked by code before it counts. A root cause may cite only evidence ids that exist in this run. A hypothesis is "supported" only when the cited evidence exists, and only a supported one can raise the confidence band, which comes from a deterministic rule (for example, two independent kinds of evidence), not from the model's own number. A file or commit may be cited only if a tool returned it. Every language-model step has a deterministic fallback: if triage fails the plan searches every allowed source; if a step cannot conclude, the run records a gap ("what we could not find out") instead of guessing. The model's own confidence is logged for calibration and never shown.
Alternatives Considered
| Alternative | Why Rejected |
|---|---|
| Show the model's confidence | Not calibrated, and it reads as authority. |
| Let the model self-verify only | The same failure mode checks itself. |
Consequences
- Some correct answers are downgraded or dropped when the evidence behind them cannot be shown. That is intended.
- Gaps are part of the product: the report says what was not examined.
ADR-016: Redact at the Door, and Trace Only With Consent
Context
Logs, stack traces, code and runbooks contain things nobody meant to send to a model vendor: keys, passwords in connection strings, customers' e-mail addresses. A trace sent to a monitoring service is a second copy.
Decision
Redaction happens where text enters the investigation, not where it is printed. Every piece of evidence is redacted when it is created (one validator on the Evidence type, so no node or tool can skip it); the incident's messages are redacted where they are read; tool output a model reads is redacted while the exact result is kept for grounding; the question, stored memories and everything returned over MCP are redacted too. Matches become [REDACTED:kind], and an event says how many values of which kinds were hidden, never what they were. Tracing (LangSmith) follows consent: tenants that brought their own LLM key are traced only if they opted in; tenants on the platform's keys are traced when tracing is on.
Alternatives Considered
| Alternative | Why Rejected |
|---|---|
| Redact in the prompt builders | One forgotten builder leaks. |
| Trust the provider's retention terms | Not a control we own. |
Consequences
- The model sees markers, so it knows something was there.
- Redaction is pattern based: a password written in prose is not caught. It is one layer beside tenant scoping, read-only tools and refusing secret files, not a guarantee.
- A secret already in the repository can still appear in a diff's context lines; the evaluation judges only what the system adds.
ADR-017: Evaluate on Recorded Incidents, Planted Bugs and a Scripted Self-Test
Context
"It seems to work" is not evidence, and an evaluation that needs a running stack cannot gate a pull request. Real customer incidents cannot be shared or reproduced.
Decision
Build the evidence ourselves. A demo shop in three languages (Node, Go, Rust) has eight planted bugs, one commit each, plus an isolated branch per bug so a case has exactly one cause; its answer key (bugs.yml) is checked against git by a script. Each case's incident and the exact rows the tools read are recorded from a real run of log0's own pipeline, then replayed through fixture backends, so the production nodes run unchanged and offline. Evaluators are deterministic where possible (localization, grounding, fix pattern, diff validity, safety, tests passing with the fix applied) with an optional model judge for the root cause. A scripted model answers perfectly, wrongly or hostilely so the harness itself can be tested: it proves the evaluators score a perfect run as perfect and a bad one as bad. A committed baseline gates regressions; real-model runs are manual and nightly, never on pull requests.
Alternatives Considered
| Alternative | Why Rejected |
|---|---|
| Hand-written fixtures | They encode what we expect rather than what the pipeline produces. |
| Judge only with a model | Slow, costly, and itself unmeasured. |
| Evaluate only on a live stack | Cannot run in CI. |
Consequences
- The harness found real defects before any model was involved (Rust dependency frames counted as application code, a redaction hole for tokens with underscores, no fix ever proposed by default). Those are fixed and tested.
- Numbers from the scripted model test the harness, not log0's quality. Quality numbers need a real-model run, which sends the synthetic logs and the demo code to the provider.
- Eight bugs and one canary is a small set: it is a regression gate and a comparison tool, not a benchmark.
ADR-018: MCP Is an Adapter Over the Same Tools
Context
Engineers already work in Claude Desktop and Cursor. log0's tools are useful there, but a second implementation would drift from the one the investigation uses.
Decision
An MCP server exposes ten read-only tools that call the same functions as the graph. The tenant is never a tool argument: it comes from the verified token (per request over HTTP, once for a stdio session, with the token's expiry checked on every call). The HTTP endpoint is stateless, so one client's identity cannot leak into another's, and is off unless enabled. Expected failures become short safe messages; unexpected ones become a generic message with the real cause only in the server log. Results are redacted like everything else.
Alternatives Considered
| Alternative | Why Rejected |
|---|---|
| MCP as the internal architecture | Couples the graph to a protocol it does not need. |
| Tenant as an argument | A client could ask about another tenant. |
Consequences
- A fix to a tool benefits the investigation and MCP at once.
- A stdio token is a secret with the reach of its tenant; it expires and is not refreshed.
ADR-019: With the Review Gate Off, a Root Cause Is Approved
Found by the evaluation harness; recorded because it changed a default.
Context
The review gate pauses an investigation after the root cause so an engineer can approve, refine or stop. It is off by default. With the gate off the graph ended at the root cause: the review decision is what routes to the step that proposes a fix, so a default deployment never produced a diff, and the demo guide said it did.
Decision
Gate off means "no human in the loop", which means approved: a root cause is approved automatically when the server can read code, and a fix is proposed. A server with no code access still ends at the root cause, silently, as before. Turning the gate on restores the pause.
Alternatives Considered
| Alternative | Why Rejected |
|---|---|
| Turn the gate on by default | A waiting investigation blocks a new one for the same incident until someone answers; not a safe default. |
| Document that diffs need the gate | The feature would look broken out of the box. |
Consequences
- Every investigation with a mapped repository now spends the extra tool calls and model turns of a fix proposal.
- A proposal is never applied: it is a validated diff to read, copy or download.
How is this guide?
Kafka Event Contracts (AsyncAPI)
AsyncAPI specification for all Kafka topics in the log0 pipeline - raw-logs, normalized-logs, incident-events, notification-events, and raw-logs-dlq. Every message schema, producer, and consumer documented in one place.
AI Investigations
How log0 investigates an incident with AI - evidence from logs, history, code and runbooks, a root cause that cites its sources, a validated fix you can apply, and how all of it is tested.