Why financial answers need provenance
A number without its window and its source is not a weaker answer. It is an answer nobody can check afterwards.
The failure mode that matters is not a wrong number. It is a right-looking number whose window nobody recorded.
FinanceBench measured a strong model with retrieval answering incorrectly or declining on 81% of a sampled subset of questions its authors call clear-cut 1. Those two outcomes are not the same failure. Declining costs a minute. Answering fluently from the wrong page costs a decision, and it costs it silently.
#The four things a figure has to carry
- The value.
- The period it covers — FY2025 is not the trailing twelve months.
- The document it came from, identified once so that two facts from the same filing point at the same filing.
- The date we read it. It is the only date we observed ourselves.
Drop any one and the sentence still parses. That is precisely the problem: nothing downstream can detect the loss.
#Why the table alone is not enough
TAT-QA exists because in real reports the figure sits in the table and the qualification sits in the paragraph beside it 2. A system that reads only the table produces a number that is arithmetically correct and materially wrong. Provenance that points at the table without the surrounding prose reproduces the same error one layer down.
#Sources
- FinanceBench: A New Benchmark for Financial Question AnsweringPranab Islam, Anand Kannappan, Douwe Kiela, Rebecca Qian, Nino Scherrer, Bertie Vidgenhttps://arxiv.org/abs/2311.11944
- TAT-QA: A Question Answering Benchmark on a Hybrid of Tabular and Textual Content in FinanceFengbin Zhu, Wenqiang Lei, Youcheng Huang, Chao Wang, Shuo Zhang, Jiancheng Lv, Fuli Feng, Tat-Seng Chuahttps://aclanthology.org/2021.acl-long.254/
Read next