BullMQ

Work sits between uploading a document and having it searchable: text is extracted, split into chunks, an embedding call is made per chunk, and vectors are written. On large files this can take minutes.

Doing that synchronously inside an HTTP request would block the user and risk timeouts. A queue moves the work to the background; the user completes the upload and document status advances as processing proceeds.

KobiGPT’s document statuses reflect that progress: pending, processing, embedding, indexing, completed, and failed. The user can see which stage a file is in.

The queue proves its value during bulk upload. When hundreds of documents arrive at once the system does not stall; jobs are enqueued and workers consume them at their own capacity.

This also respects the embedding provider’s rate limits. Calls are paced, and when the provider returns a transient error the job is retried, so a document can still complete rather than failing.

Retry logic is critical here. Without a queue, a momentary network error leaves a document permanently failed; with one, the same error becomes an invisible delay.

The queue is a component that needs observation. A document stuck in processing or embedding may mean the queue is backed up, a worker has died, or the provider is unreachable.

Document status is therefore not only user-facing information but an operational signal. Requeuing stuck jobs is part of normal maintenance.

In a self-hosted deployment Redis and the worker processes run in your infrastructure, making capacity, persistence settings, and monitoring your responsibility.

A useful way to evaluate bullmq is to follow one real question from the source document to the final answer. Record which file was selected, what context reached the model, and what a reviewer would need to verify. This turns a definition into an operational check and makes the result comparable across teams.

The same check should include ownership and change management. Decide who updates the relevant documents, how an outdated result is reported, and which access boundary applies. KobiGPT can provide the assistant and the cited document context, but the organisation still owns the source material, permissions, and the decision made from the answer.