Conformal Gating Of Extracted Factor Vectors In LLM Signal Pipelines Under Delayed Labels And Regime Shift: A Simulation Study
Read Abstract
Language model pipelines extract multi-dimensional factor vectors from financial text, but a trading desk needs to know when an extracted vector can be trusted. Audit labels arrive late, errors cluster in stress regimes, and error scale shifts after model or prompt updates. We evaluate a conformal gate on the extraction layer in a seeded simulation (300 firms, 60 quarters, 200 replications) with heavy-tailed errors, regime stress, a structural break and a two-quarter audit lag. Per-dimension 90% intervals cover the full eight-dimension vector only 40 to 45% of the time, while a max-score construction reaches 89.7% before the break with intervals 18 to 26% narrower than Bonferroni. Calibration frozen at go-live falls to 78.2% joint coverage after the break. Rolling, weighted and delayed-feedback adaptive calibration recover 88.0 to 89.9% on average, but no label-driven method protects the quarters just after a shift. As a risk control, the adaptive gates reject most signals after the break, accepting only 9 to 14%, yet the error rate among accepted signals is still about 20%, against 28% without a gate, and gating lowers risk-adjusted return in this design. A label-free drift monitor flags 78.5% of abrupt breaks in the quarter they occur, against 3.0% for label-based monitoring, but only 3.5% under gradual drift, so its advantage is specific to abrupt breaks. To test whether these mechanisms appear with a real extractor, we run a pilot that extracts seven financial fields from the earnings press releases of 492 large US companies from an S&P 500 earnings-call dataset with an open-weight model and scores each figure against the SEC XBRL-reported value. Extraction is accurate for most figures, 96.3% within 1% once the reference labels are corrected and verified. Fixing our choice of revenue tag and checking every large-error label against the press release show that 38 of the 81 figures first scored as large errors rested on wrong labels rather than misreadings. The one self-reported signal collected, the model's GAAP flag, is constant and uninformative, while disagreement across three models detects 59% of the remaining large errors, and 74% of those confirmed as extraction errors, at the cost of flagging 8.6% of correct figures, which is consistent with the view that a usable confidence signal must come from a check outside a single model's output, although richer self-reported signals were not tested. A Monte Carlo resampling calibrated to the pilot, with the detector's operating point held fixed, gives sampling intervals for the accuracy and large-error rates. The results suggest three design rules: report joint reliability with a max-score rectangle, gate on the composite the operator consumes, and size positions by a smoothed scale rather than by calibrated width. We place the gate in a deployment framework and state its limits.