Building QuarterLens: hybrid retrieval for financial reports

Ask a general-purpose chatbot about a company's quarterly results and you'll usually get a fluent, confident answer. Whether it's correct is another matter. The figures may be outdated, from the wrong quarter, or simply made up. In finance, a plausible wrong number is worse than no answer at all.

QuarterLens is our prototype for a narrower and more useful job: answering questions about Infosys financial reports, using only what the reports say, and showing exactly where each answer came from. Every response includes the document and page it was drawn from, so a reader can check it in seconds.

This post walks through how it works and, more importantly, why it's designed the way it is. Financial documents turn out to be an excellent stress test for retrieval-augmented generation (RAG), and the lessons carry over to contracts, policies, technical manuals, and most other serious document collections.

A quick refresher on RAG

A RAG system answers questions in two steps:

1. Retrieve. Search a collection of documents for the passages most relevant to the question.
2. Generate. Give those passages to a language model and ask it to answer using only that evidence.

The language model gets the headlines, but retrieval decides the outcome. If the right passage isn't retrieved, the model either says it doesn't know (the good case) or fills the gap with something plausible (the bad case). Most of the engineering effort in QuarterLens goes into getting retrieval right.

Why financial reports are hard

Financial reports break several assumptions that simpler document search relies on.

• They repeat themselves. Revenue, operating margin, and headcount appear in the highlights, the management commentary, the segment breakdowns, the notes, and the tables, each time with slightly different framing.
• Small tokens carry the meaning. The difference between the right and wrong answer is often a period ("Q2" vs "Q3", "FY2025" vs "FY2026"), a unit ("crore" vs "million"), or a qualifier ("constant currency" vs "reported").
• A lot of the content is tables. Tables don't survive conversion to plain text well, and a number without its row and column labels is meaningless.
• Questions use different words than the documents. A user asks "how much did they make?" The report says "revenues from operations."

A good retrieval system has to handle both kinds of difficulty: exact matching on precise tokens and flexible matching on meaning.

Step 1: Ingestion and chunking

Before anything can be searched, reports are converted to text and split into chunks: short passages that can be retrieved independently. Chunking sounds mundane, but it quietly decides what's even possible later.

• Chunks that are too large blend several topics together, which dilutes their relevance to any single question and wastes the model's limited context.
• Chunks that are too small lose context. A sentence like "this represents growth of 4.2%" means nothing without knowing what "this" refers to.

Every chunk also keeps its source metadata: which document it came from and which page. That metadata is what makes citations possible later, so it is preserved through every step of the pipeline.

Step 2: Hybrid search

QuarterLens runs two different retrievers over the same chunks and combines their results.

Keyword search

Keyword search scores passages by how well their exact terms match the question, giving more weight to rarer terms. It is excellent at what financial questions need most: exact fiscal periods, metric names used word for word, segment names, and specific figures. If a user asks about "Q2 FY2026," keyword search strongly prefers passages that literally contain those tokens.

Its weakness is the opposite case. If the question and the passage use different words for the same idea, keyword search can miss the passage entirely.

Semantic search

Semantic search converts both the question and each chunk into embeddings: vectors that capture meaning, so related texts end up close together. QuarterLens stores these in ChromaDB. Semantic search handles paraphrase well. "How much money did they make" lands near passages about revenue even though they share no words.

Its weakness mirrors keyword search's strength. Embeddings blur fine distinctions. Passages about Q2 and Q3 revenue can look almost identical in vector space, because they are almost identical in meaning. Only one token differs.

Combining them

Each retriever covers the other's blind spot, so running both greatly improves the chance that the right passage is somewhere in the candidate set. The two result lists then have to be merged. A common and robust way to do this is reciprocal rank fusion, which combines results by their rank positions rather than their raw scores. That matters because keyword and semantic scores are on completely different scales and can't be compared directly:

def reciprocal_rank_fusion(result_lists, k=60):
    scores = {}
    for results in result_lists:
        for rank, chunk_id in enumerate(results, start=1):
            scores[chunk_id] = scores.get(chunk_id, 0) + 1 / (k + rank)
    return sorted(scores, key=scores.get, reverse=True)

A passage that ranks well in both lists rises to the top. A passage that ranks very highly in only one list still gets a fair chance.

Step 3: Reranking

Hybrid search answers "is the right passage among the candidates?" Reranking answers a different question: "is it near the top?"

That distinction matters because the language model only sees the top handful of passages. A correct passage in position fifteen is as good as missing.

First-stage retrievers are built for speed. They compare a question with millions of passages by turning each into a single vector or a set of term counts in advance. A reranker is slower but far more precise: it reads the question and each candidate passage together and scores how well that passage answers that question. Because it only runs on a short list of candidates, it can afford to be thorough.

In practice, reranking is often the step that moves a system from "usually finds something related" to "usually puts the right evidence first."

Step 4: Grounded generation with citations

The top-ranked passages go to a language model running locally through Ollama, with clear instructions:

• Answer only from the provided passages.
• Cite the document and page behind each fact.
• If the passages don't contain the answer, say so instead of guessing.

The answer and its evidence are shown together in a Streamlit interface, so the reader can compare the claim against the source passage straight away.

Running the model locally has a practical benefit beyond cost: documents never leave the machine. For many organisations handling sensitive or non-public documents, that's a requirement rather than a preference.

Citations are not decoration. They change how people use the system. Without citations, users have to either trust the answer completely or redo the research themselves. With citations, checking an answer takes seconds, and wrong answers are caught instead of spreading into reports and decisions.

Where it still struggles

QuarterLens is a prototype, and there are clear limits we expect to work on.

• Tables. Numbers taken out of tables can lose their row and column labels during text extraction, which makes them hard to retrieve and easy to misread.
• Questions across periods. "How did margin change from last year?" needs evidence from two places. If only one of them is retrieved, the answer will be incomplete or wrong.
• Ambiguous questions. "What was growth?" has several valid answers in one report: reported or constant currency, year on year or quarter on quarter, company-wide or by segment. A good system should ask which one you mean, or answer all of them clearly.
• Arithmetic. Language models are unreliable at calculations. When an answer needs computing, the calculation should be done explicitly and shown, not left to the model.

What carries over to other domains

Financial reports are one example, but the same design principles apply to any document collection where precision matters:

1. Use hybrid retrieval. Exact terms and meaning are both important, and no single retriever handles both well.
2. Rerank before generating. The order of evidence matters as much as whether it was found.
3. Keep source metadata through every step, so citations are always possible.
4. Make refusing to answer a valid outcome, and evaluate how often the system does it correctly.
5. Measure retrieval separately from generation. We cover how to do that in a separate post.