A Cloud-Native Architecture for Human-in-Control LLM-Assisted OpenSearch in Investigative Settings

In a recent comprehensive analysis, [Skipanes.2025] identify the processing of unstructured text as a critical bottleneck in contemporary criminal investigations. They highlight that, while computational opportunities exist, current methods largely fail to bridge the gap between technical retrieval logic and the qualitative reasoning required by investigators. Our work directly addresses this architectural gap by operationalizing these opportunities within a secure on-premises environment.

The integration of Large Language Models into Information Retrieval systems has evolved rapidly from simple re-ranking tasks to complex query generation [Zhu.2023]. Current approaches often focus on Text-to-SQL paradigms, in which Large Language Models translate natural language into structured SQL queries for relational databases [Shi.2024]. However, these methods are inherently constrained by the unstructured and fuzzy nature of forensic text data. Conversely, Retrieval-Augmented Generation grounds Large Language Model responses in retrieved search results but may introduce hallucinations or lack the deterministic precision required for rigorous investigative filtering [Gao.2024].

Our work builds upon the findings of [Liu.2023] concerning the “Lost-in-the-Middle” phenomenon, which posits that LLMs often fail to identify relevant information in long contexts. We address this limitation by segmenting documents into semantic units rather than processing entire texts. Furthermore, we adopt the Chain-of-Thought prompting strategy proposed by [Wei.2022] to improve the logical consistency of generated OpenSearch queries. Unlike autonomous agents, our architecture enforces a “Human-in-Control” design that prioritizes the investigator’s control over search logic to ensure procedural accountability within legal domains.

The system follows a microservice architectural pattern composed of four distinct layers that communicate via RESTful APIs and asynchronous message queues (see Figure˜1):

Presentation Layer: A Single Page Application built with Next.js provides the investigative user interface. It uses Server-Side Rendering to optimize initial load performance and communicates asynchronously with the backend to ensure a non-blocking user experience, which is essential for reviewing large document sets.

Orchestration Layer (API Gateway): The core application logic is handled by a FastAPI backend. Unlike monolithic frameworks, FastAPI implements the ASGI standard, enabling native asynchronous request handling. This capability is critical for maintaining high throughput when coordinating I/O-intensive operations across the database, search engine, and Large Language Model services. State management is offloaded to persistent stores - PostgreSQL for case management and Redis for job queues. Crucially, this design transforms the system from a stateless search engine into a case-management workspace. The backend tracks the “reviewed status” of each retrieved document, supporting a coverage-oriented workflow that enables investigators to systematically examine evidence by relevance.

Asynchronous Processing Layer (Worker): To handle large-scale data ingestion, we implemented a producer-consumer pattern using Redis. Python-based workers perform CPU-intensive heuristic parsing and disentanglement. However, computationally expensive vectorization is offloaded to the OpenSearch cluster thorugh a dedicated ingest pipeline that runs on specialized machine learning nodes. This design separates the cleaning logic from the inference workload, enabling independent scaling of ingestion workers and neural inference resources.

Generative Logic Layer: Instead of hardcoding prompt logic, the system integrates Langflow [Langflow] as a visual low-code environment to manage interaction chains. To enforce data sovereignty, the architecture deliberately avoids reliance on public APIs. Instead, it uses locally hosted open-weight models (e.g., Llama 3 or Mixtral) running on on-premise inference servers via vLLM. This design ensures that no sensitive investigative intent leaves the secure private-cloud perimeter.

A major challenge in processing large-scale forensic data is the “Lost-in-the-Middle” phenomenon, in which Large Language Models fail to retrieve relevant information embedded in long, unstructured contexts [Liu.2023]. Indexing a typical investigative document (e.g., a 50-page witness statement or an extended email thread) as a monolithic block degrades vector search performance because of the architectural token limitations of the underlying transformer models [Reimers.2019].

To address this limitation, we developed a configurable, modular adapter pattern within the asynchronous worker nodes. Unlike generic chunking strategies (e.g., fixed-size splitting), our system supports custom parsing profiles explicitly designed for each input type to preserve semantic boundaries in forensic data:

Heuristic Chunking (Legacy Documents): For unstandardized text documents, we implemented a custom regex-based heuristic to detect semantic shifts (e.g., speaker changes or timestamps). This approach allows the system to generate atomic segments even in the absence of structured separators.

Disentanglement (Communication Data): For the Enron dataset, the ingestion logic was tailored to disentangle forwarded message chains. By selectively stripping technical headers (e.g., X-UID) from the semantic payload (Level 2) while preserving them for structured filtering (Level 1), we minimize noise in the vector space.

This pre-processing step ensures that embeddings represent specific statements rather than diluted document-level averages.

A core architectural decision was to implement a hybrid search index within OpenSearch. The current approach focuses exclusively on text-based data. Image and audio-based data would need to be converted into text. The hybrid search index addresses the fundamental linguistic limitations of purely lexical retrieval, specifically synonymy and polysemy [Manning.2008]. By integrating vector-based semantic retrieval, our schema combines the strengths of both paradigms, a pattern applicable to both interview protocols and digital communication:

Unstructured Baseline (Level 1): The document’s full text and metadata are indexed using standard BM25 lexical search [Robertson.1995]. This configuration ensures high recall for specific keywords, such as names, case numbers, or senders/receivers.

Structured Nested Embeddings (Level 2): To mitigate context dilution, we avoid embedding long documents as a single vectors. Instead, the atomic semantic units identified in Section˜III-B (e.g., specific paragraphs, message bodies, or individual statements) are stored as nested objects containing the segment text and a 384-dimensional vector embedding. We use the HNSW algorithm [Malkov.2020] with Cosine Similarity, adhering to the optimization objective of the underlying paraphrase-multilingual-MiniLM-L12-v2 model [Reimers.2019]. To minimize architectural complexity, this model is provisioned through the OpenSearch ML Commons framework and executed directly on the cluster’s internal ML nodes. This configuration ensures that ranking is determined by semantic alignment (vector orientation) rather than by vector magnitude. Additionally, a search-time synonym graph expands queries, e.g., mapping “detention” to “imprisonment”, without increasing the physical index size.

This nested structure is designed to prevent “cross-object matching” errors in which a query matches unrelated parts of a document (e.g., matching a suspect’s name from page 1 with an action described on page 10), and to enable the Large Language Model to generate queries that target specific semantic segments.

Hybrid Fusion for Prioritized Review: During the exploratory evidence-review phase, minimizing False Negatives is paramount. Investigators require a ranking mechanism that surfaces the most relevant documents first to support the coverage-oriented workflow described in Section˜III-A. Our system employs a score normalization approach to fuse unbounded lexical scores (BM25) with normalized semantic scores (Cosine Similarity). This design ensures that a document describing “off-balance sheet debt” (semantic match) is ranked competitively with documents containing the specific project code “Raptor” (lexical match), even when their vocabularies do not overlap.

The translation of natural language into the complex OpenSearch Query Domain-Specific Language is handled by a multi-agent Large Language Model pipeline orchestrated via Langflow. To ensure domain agility without code modifications, the system employs schema-aware prompting: the current index definition is injected into the prompt context at runtime. Crucially, this design enforces a strict separation of concerns: the Large Language Model operates exclusively on the abstract index schema, never on the actual evidentiary content. Unlike standard Retrieval-Augmented Generation workflows [Lewis.2020], which require feeding retrieved text into the model’s context window, our architecture ensures that sensitive document payloads remain confined within the OpenSearch cluster and are never exposed to the inference context. Rather than relying on opaque “black box” logic, we implement a Chain-of-Thought workflow:

Reasoning & Generation: The first agent acts as a “Query Architect”. It analyzes the user’s intent and the injected schema structure, generating an intermediate reflection before constructing the JSON query.

Auditing (Quality Assurance): The second agent, the “Auditor”, validates each generated query against known error patterns. For instance, it detects when the model attempts to search for structured entities (e.g., specific dates or person names) within the semantic vector field, which typically yields lower precision. The Auditor enforces correct mapping to structured fields (e.g., moving a name search to the sender or people field) before execution.

Transparent Execution & Deterministic Retrieval: The validated query is then executed to provide immediate feedback. However, unlike fully autonomous agents, the system enforces transparency: the generated search logic (the “translation”) is displayed alongside the results. Since all presented results are deterministic database retrievals rather than Large Language Model-generated text, the risk of evidence hallucination is eliminated. The investigator retains full authority to evaluate the relevance of retrieved documents and iteratively refine the generated query logic, ensuring that the final assessment of evidence remains a human decision.

This pipeline ensures that the resulting Domain-Specific Language query is syntactically valid and semantically aligned with the investigator’s intent prior to execution.