Hybrid Retrieval-Augmented Generation with Hallucination Mitigation for Indonesian MSME Financial Advisory

Abstract

Micro, small, and medium enterprises (MSMEs) are the backbone of Indonesia's economy, yet many owners face persistent gaps in financial literacy and difficulty navigating complex financial and tax regulations. General-purpose large language models (LLMs) can make such knowledge more accessible, but they often produce fluent yet unsourced or hallucinated answers, which is unacceptable in a financial and regulatory context. This study designs and evaluates IndoRAG-FA, a hybrid Retrieval-Augmented Generation (RAG) framework with explicit hallucination mitigation for Bahasa Indonesia MSME financial advisory. The framework integrates coherence-based semantic chunking; hybrid retrieval combining lexical BM25 and dense multilingual embeddings fused through reciprocal rank fusion; cross-encoder re-ranking; and grounded generation that produces source-cited answers with a confidence-based abstention mechanism and a groundedness verifier. A curated knowledge base was built from authoritative Indonesian financial and tax sources, and an expert-validated benchmark of representative MSME questions, including out-of-scope items, was developed. The framework was compared with an LLM-only baseline and a vanilla RAG baseline using retrieval metrics (Recall@k, MRR, nDCG), faithfulness, answer relevance and correctness, hallucination rate, abstention precision and recall, and a human-expert rubric. Findings indicate that a retrieval-grounded, abstention-aware design offers a practical and trustworthy path toward deployable financial advisory for Indonesian MSMEs and provides a reusable evaluation protocol for domain-specific LLM systems.