LLMs for Suptech: Knowledge Extraction from Unstructured Regulatory Reports

Abstract

The heterogeneous formats and narrative complexity of Pillar 3 reports under the Basel Accord framework significantly hinder automated analysis of criti-cal financial metrics such as the Common Equity Tier 1 (CET1) ratio, creat-ing substantial challenges for financial supervision. This study investigates the effectiveness of Large Language Models (LLMs) as Supervisory Tech-nology (Suptech) tools for automated knowledge extraction from unstruc-tured regulatory documents. A proof of concept was developed using GPT-4o and GPT-4o mini models with a simple pipeline architecture incorporating Retrieval-Augmented Generation (RAG). The models were tested on 36 quarterly Pillar 3 reports from four Global Systemically Important Banks (G-SIBs) to extract CET1 ratios across six different regulatory configurations. GPT-4o achieved 83.8% accuracy in CET1 extraction while GPT-4o mini reached 73.5% accuracy. Both models demonstrated capability in interpret-ing diverse report structures and inferring contextual information, though challenges emerged with inconsistent outputs and complex reasoning re-quirements. While LLMs show significant potential for regulatory document analysis, current limitations necessitate hybrid approaches combining auto-mated extraction with human-in-the-loop supervision to ensure reliability in financial supervision contexts.