Margin- and Vocabulary-Aware Adaptive Retrieval for Industrial Technical Question Answering: A Controlled Pilot
DOI:
https://doi.org/10.54097/bhr7y923Keywords:
Technical question answering, Adaptive retrieval, BM25, Lexical coverage, Character n-grams, Robustness, Evidence retrievalAbstract
Efficient industrial question answering requires reliable evidence retrieval despite document-specific terminology and spelling variation. We study confidence-gated dual retrieval (CGDR), which uses the normalized BM25 top-two margin to decide whether to invoke a complementary character n-gram index. On 80 held-out synthetic questions generated from a published review of domain-adapted industrial language models, the margin-only selector obtains 92.5% Recall@1 with 17.5% complementary-index calls. Yet additional spelling edits reveal substantial brittleness: Recall@1 falls to 38.75% for character deletion or transposition. We therefore introduce an out-of-vocabulary (OOV)-aware gate, combining rank ambiguity with lexical coverage of the indexed corpus. Its rule is fitted on separate development pages and their deterministic perturbations under a 55% development-stage call budget. On held-out deletion and transposition edits, the revised gate increases Recall@1 to 60.0% in each condition, compared with 38.75% for the margin-only gate, while requiring the character index on 60% of these requests. On unmodified questions, it returns 91.25% Recall@1 at a 21.25% call rate. Because this gate was designed after the margin-only analysis, its held-out gains are reported as exploratory. The improvements are measured on 80 underlying held-out queries with dependent synthetic transformations, not on naturally authored questions or a production system. Passage-level evaluation, sentence-extraction checks, matched-budget controls, and source-auditable code characterize the robustness–cost trade-off; no generative language model is evaluated.
Downloads
References
[1] Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W.-T., Rocktäschel, T., Riedel, S., & Kiela, D. (2020). Retrieval-augmented generation for knowledge-intensive NLP tasks. In Advances in Neural Information Processing Systems 33 (NeurIPS) (pp. 9459-9474).
[2] Guu, K., Lee, K., Tung, Z., Pasupat, P., & Chang, M.-W. (2020). Retrieval augmented language model pre-training. In Proceedings of the 37th International Conference on Machine Learning (ICML), PMLR 119 (pp. 3929-3938). PMLR.
[3] Karpukhin, V., Oguz, B., Min, S., Lewis, P., Wu, L., Edunov, S., Chen, D., & Yih, W.-T. (2020). Dense passage retrieval for open-domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) (pp. 6769-6781). Association for Computational Linguistics. https://doi.org/10.18653/v1/2020.emnlp-main.550
[4] Castelli, V., Chakravarti, R., Dana, S., Ferritto, A., Florian, R., Franz, M., Garg, D., Khandelwal, D., McCarley, S., McCawley, M., Nagesh, A., Pan, L., Pendus, C., Pitrelli, J., Pujar, S., Roukos, S., Sakrajda, A., Sil, A., Uceda-Sosa, R., … & Tymoshenko, K. (2020). The TechQA dataset. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL) (pp. 1269-1278). Association for Computational Linguistics. https://doi.org/10.18653/v1/2020.acl-main.117
[5] Zhou, F. (2026). Moment-constrained side-force allocation for wind-assisted merchant ships: Hydrodynamic performance of retractable anti-leeway fins and flap rudders. Innovation and Technology Studies, 3(1), 27-41.
[6] Robertson, S., & Zaragoza, H. (2009). The probabilistic relevance framework: BM25 and beyond. Foundations and Trends in Information Retrieval, 4(1-2), 1-174. https://doi.org/10.1561/1500000019
[7] Cormack, G. V., Clarke, C. L. A., & Büttcher, S. (2009). Reciprocal rank fusion outperforms Condorcet and individual rank learning methods. In Proceedings of the 32nd International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) (pp. 758-759). ACM. https://doi.org/10.1145/1571941.1572114
[8] Reimers, N., & Gurevych, I. (2019). Sentence-BERT: Sentence embeddings using Siamese BERT-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) (pp. 3982-3992). Association for Computational Linguistics. https://doi.org/10.18653/v1/D19-1410
[9] Santhanam, K., Khattab, O., Saad-Falcon, J., Potts, C., & Zaharia, M. (2022). ColBERTv2: Effective and efficient retrieval via lightweight late interaction. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL) (pp. 3715-3734). Association for Computational Linguistics. https://doi.org/10.18653/v1/2022.naacl-main.272
[10] Izacard, G., & Grave, E. (2021). Leveraging passage retrieval with generative models for open domain question answering. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics (EACL) (pp. 874-880). Association for Computational Linguistics. https://doi.org/10.18653/v1/2021.eacl-main.74
[11] Liu, N. F., Lin, K., Hewitt, J., Paranjape, A., Bevilacqua, M., Petroni, F., & Liang, P. (2024). Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics, 12, 157-173. https://doi.org/10.1162/tacl_a_00638
[12] Jeong, S., Baek, J., Cho, S., Hwang, S. J., & Park, J. (2024). Adaptive-RAG: Learning to adapt retrieval-augmented large language models through question complexity. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL) (pp. 7036-7050). Association for Computational Linguistics. https://doi.org/10.18653/v1/2024.naacl-long.389
[13] Wang, J., Liu, J., Ge, Y., & Jiang, B. (2023). Risk-aware ensemble learning for credit default prediction under economic uncertainty. World Journal of Economics and Business Research, 1(1), 18-32.
[14] Jiang, Z., Xu, F. F., Gao, L., Sun, Z., Liu, Q., Dwivedi-Yu, J., Yang, Y., Callan, J., & Neubig, G. (2023). Active retrieval augmented generation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP) (pp. 7969-7992). Association for Computational Linguistics. https://doi.org/10.18653/v1/2023.emnlp-main.495
[15] Ma, X., Gong, Y., He, P., Zhao, H., & Duan, N. (2023). Query rewriting in retrieval-augmented large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP) (pp. 5303-5315). Association for Computational Linguistics. https://doi.org/10.18653/v1/2023.emnlp-main.322
[16] Ding, D., Mallick, A., Wang, C., Gan, Y., Sun, S., Zhang, Y., & Zhang, Y. (2024). Hybrid LLM: Cost-efficient and quality-aware query routing. In Proceedings of the International Conference on Learning Representations (ICLR).
[17] Chen, J., Zhang, X., & Han, X. (2023). Explainable identification of tax-related financial reporting anomalies in U.S. public companies: Cross-fitted tax-consistency residuals and Owen-value Shapley propagation. World Journal of Economics and Business Research, 1(1), 33-45.
[18] Es, S., James, J., Espinosa Anke, L., & Schockaert, S. (2024). RAGAs: Automated evaluation of retrieval augmented generation. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations (EACL) (pp. 150-158). Association for Computational Linguistics. https://doi.org/10.18653/v1/2024.eacl-demo.16
[19] Saad-Falcon, J., Khattab, O., Potts, C., & Zaharia, M. (2024). ARES: An automated evaluation framework for retrieval-augmented generation systems. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL) (pp. 338-354). Association for Computational Linguistics. https://doi.org/10.18653/v1/2024.naacl-long.20
[20] Yang, J.-S., Shen, Z., Zeng, Z., & Chen, Z. (2025). Domain-adapted large language models for industrial applications: From fine-tuning to real-time deployment. Computer Science Bulletin, 8(1), 272-289. https://doi.org/10.71465/csb162
[21] Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., & Chen, W. (2022). LoRA: Low-rank adaptation of large language models. In Proceedings of the International Conference on Learning Representations (ICLR).
[22] Wang, J., Ge, Y., Liu, J., & Jiang, B. (2024). Early warning of liquidity stress in U.S. banks with seasonally aligned frailty hazards and delayed-feedback conformal alarms. Journal of Trends in Finance and Economics, 1(1), 85-100.
[23] Zhou, Y. (2024, December). PANDAX for industrial question answering with explanation (QAE). Zenodo. https://doi.org/10.5281/zenodo.14510798
[24] Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., & Zettlemoyer, L. (2020). BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL) (pp. 7871-7880). Association for Computational Linguistics. https://doi.org/10.18653/v1/2020.acl-main.703
[25] Pope, R., Douglas, S., Chowdhery, A., Devlin, J., Bradbury, J., Heek, J., Xiao, K., Agrawal, S., & Dean, J. (2023). Efficiently scaling transformer inference. In Proceedings of Machine Learning and Systems (MLSys) (Vol. 5). MLSys.
[26] Yang, J.-S., Zeng, Z., & Shen, Z. (2025). Neural-symbolic dual-indexing architectures for scalable retrieval-augmented generation. IEEE Access, 13, 1-15. https://doi.org/10.1109/ACCESS.2025.3638761
[27] Rhee, M., Zhang, Y., Li, H., Wang, J., & Chen, X. (2026). Enhancing web search agents with self-play contrastive fine-tuning. IEEE Access, 14, 1-14. https://doi.org/10.1109/ACCESS.2026.3717594
Downloads
Published
Issue
Section
License
Copyright (c) 2026 Journal of Computing and Electronic Information Management
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
