Building Intelligence Hub: Redis, RabbitMQ, and Agentic AI

The challenge: document-to-test-cases at scale 

Intelligence Hub does one thing that looks simple from the outside: you upload a spec and get test cases back in three minutes. Inside, it’s coordinating five moving pieces in parallel: storing a user’s session across requests, dispatching long-running document analysis as an async job, extracting requirements from a PDF or Word doc, streaming progress back to the browser in real time, and persisting test cases with bidirectional links to the source document. 

Build it the naive way synchronous endpoint, wait for the doc processor, return a response and a 40MB PDF times out before the analysis finishes. Add naive concurrency and you lose track of which result belongs to which user’s session. The architecture we built solves both problems and handles the cases nobody plans for: browser disconnects mid-stream, document processor crashes, test case schema evolving while queries are running. 

The architecture at a glance 

Intelligence Hub runs four layers: 

• Session layer: Redis-backed session management (TypeScript). Who are you, what’s your project, which upload is this? 

• Job dispatch layer: RabbitMQ queues and workers. Queue the doc analysis task, dispatch to available processors. 

• Processing layer: Go document processor. Extract requirements, identify test scenarios, generate test case structures. 

• Persistence & streaming layer: MongoDB for test case storage with traceability links, Redis Pub/Sub for real-time browser updates. 

Data flows downward: client submits doc → session stores metadata in Redis → job queued in RabbitMQ → Go processor reads doc → test cases written to MongoDB → streaming events via Redis Pub/Sub notify the browser. Each layer can fail independently and be retried. 

Layer 1: Redis-based session management 

When a user uploads a document, we don’t just start processing it. We first create a session: a scoped container for one specific document processing request. The session tracks project ID, user ID, document metadata, processing status, and error state. 

Sessions live in Redis (via  

// Pseudocode: session lifecycle 

session = { 

  id: UUID, 

  userId, projectId, documentId, 

  status: "uploading" | "queued" | "processing" | "complete" | "failed", 

  progress: 0-100, 

  createdAt, expiresAt: +24h 

} 

// Stored as: 

SETEX session:{sessionId} 86400 <JSON> 

The session is the single source of truth for what’s happening to a specific upload. It’s also the scope for Redis Pub/Sub subscriptions: the browser subscribes to  

Layer 2: RabbitMQ job dispatch 

Once the session exists and the document is uploaded to cloud storage, we don’t process it immediately. We queue a job in RabbitMQ. Why? Because documents can be huge (40MB PDFs aren’t rare), and document processors are stateful and slow. A single LLM call to extract test scenarios from a complex spec takes 30–45 seconds. The API request would time out if we tried it synchronously. 

RabbitMQ gives us durable, AMQP-compliant job queueing. If a processor crashes mid-document, the message stays in the queue until a healthy worker picks it up. If we add a new processor instance, the queue automatically distributes jobs across it. If demand spikes, jobs back up safely instead of dropping silently. 

The job payload is minimal: session ID, document URI, processor hints (test framework preference, scenario types to generate). The processor downloads the doc on demand rather than the API pushing it, decoupling storage from job queueing. 

Layer 3: Go document processor 

The heavy lifting happens in a Go microservice. Why Go and not Python? Speed. Python is great for data science, but extracting structure from a 40MB PDF requires tight loops over byte arrays and concurrent I/O. Go’s standard library pdf library, concurrent goroutines, fast JSON marshalling handles this workload with less memory and no GIL contention. 

The processor does four things in sequence: 

• Parse the document: extract text, handle images embedded in PDFs, detect structure. 

• Chunk the document: break it into semantic sections so traceability links point to specific paragraphs, not the whole spec. 

• Extract requirements: use an LLM to identify user stories, features, and acceptance criteria from each chunk. 

• Generate test scenarios: for each requirement, generate test cases (happy path, negatives, edges, security) and output them as structured JSON. 

Each of these steps emits progress events to Redis: “finished parsing” (10%), “generated 12 test cases from section 3.2” (50%), etc. Those events are pub/sub-published, so the browser sees real-time progress bars. 

Layer 4: Redis Pub/Sub streaming and MongoDB persistence 

As the Go processor generates test cases, it publishes events to Redis using Pub/Sub. Each event includes the test case, the section of the spec it came from, and a severity/priority score. Subscribers (the user’s browser, any logged analytics, background indexing) get the event in real time. 

This design makes the processor stateless. It doesn’t need to open a database connection, wait for writes to finish, or handle connection pools. It just publishes events. MongoDB persistence happens asynchronously: a separate MongoDB writer service subscribes to Pub/Sub, batches incoming test cases, and writes them to Mongo with traceability links. 

The traceability link is the key structure. Each test case record includes: 

{ 

  _id: ObjectId, 

  sessionId, projectId, 

  testCaseId, testName, steps: [...], 

  tracedTo: { 

    documentId, sectionId, sectionTitle, 

    sourceText: "the exact paragraph this came from", 

    confidence: 0.92  // LLM confidence in the extraction 

  }, 

  createdAt, updatedAt 

} 

That link is queryable. A PM can later ask “which test cases came from section 4.3 of the spec?” and get them instantly. If the spec changes, the PM knows exactly which test cases need review because they’re tracked by source. 

Why RabbitMQ and Redis, not Kafka or Redis Streams? 

Common question from architects: why dual messaging (RabbitMQ + Redis)? 

RabbitMQ is for commands. “Process this document” is a command: it must be executed exactly once, reliably, even if the processor crashes. RabbitMQ’s AMQP semantics and ack-based delivery guarantee that. 

Redis Pub/Sub is for signals. “Hey, the processor just finished section 5” is a signal: if the browser isn’t listening right now, that signal doesn’t need to be stored. If 10 other sessions are processing documents, they don’t care about this user’s progress. Pub/Sub is the right primitive and it’s orders of magnitude faster than durable message brokers. 

We considered Redis Streams for both layers. Streams give you durable event logs, which sounds good in theory. In practice: Streams add latency to progress updates (each publish is a disk write), and we don’t need durability for progress signals, only for job dispatch. The combination of RabbitMQ (jobs) + Pub/Sub (signals) is the right split of concerns. 

Handling scale and reliability 

The design handles several failure modes: 

• Processor crash: RabbitMQ keeps the job in the queue. The next healthy worker picks it up. 

• Browser disconnect: The Pub/Sub connection closes, but the processor keeps running. When the user refreshes, they re-subscribe to the session and get a final progress update (100% complete). 

• MongoDB write fails: The Pub/Sub subscriber retries with exponential backoff. Test case data is resynthesized from the processor on demand if needed. 

• Redis session expires: Sessions live for 24 hours. If a user steps away for a day, they lose the real-time progress, but they can re-query their test cases from MongoDB by project/session. 

Scaling is horizontal. Add more Go processors, and RabbitMQ distributes jobs across them. Add more Pub/Sub subscribers (e.g., for analytics, indexing), and Redis fan-out broadcasts to all of them. MongoDB scales separately with standard replica sets. 

Monitoring and traceability in the system itself 

We apply the same traceability principle internally: every test case knows which spec section it came from, and every processing job has a full execution log in MongoDB. That means when something goes wrong (a processor generated test cases with 60% confidence instead of 85%), we can replay the exact document, the exact processor state, and the exact decision points that led to that output. 

Metrics flow to a time-series database: how long does document parsing take, how many test cases per document on average, what’s the processor success rate by document type. That telemetry drove several optimizations for instance, we found that large PDFs with scanned images were 3x slower than born-digital PDFs, so we optimized the image-to-text pipeline. 

Lessons for builders: the stack and the trade-offs 

• Use the right tool for each job. Redis for sessions and signals, RabbitMQ for commands, MongoDB for queryable state, Go for CPU-bound processing. 

• Separate concerns: job dispatch (durability matters) vs. progress signaling (speed matters). Treating them the same leads to either slow signals or lost jobs. 

• Traceability from day one. Build systems that remember where data came from. It costs almost nothing in design, and it’s invaluable when debugging or when compliance requires you to explain a decision. 

• Stateless processors. Let them emit events. Let other services persist. This makes the processors easy to test, debug, and replace. 

What’s next 

We’re exploring GPU-accelerated document parsing for even faster extraction on large specs, and event sourcing on top of our current MongoDB setup to create an immutable audit log of every test case change. The foundation is solid; the layers are decoupled enough that we can upgrade each one independently. 

See Intelligence Hub in action. Start a free trial.