When 98% complete is incomplete
One rate-limited request out of sixty put the wrong day's move in the header of every page. Here is why a percentage was the wrong test.
What went wrong
Our catch-up loop decided a trading session was loaded once the table held at least 80% of the rows of the fullest recent day. On 7 October the data provider rate-limited exactly one request out of sixty. The one it refused was NIFTY 50.
Fifty-nine of sixty is 98%, comfortably over the threshold, so the loop reported the table current and moved on. The header of every page then quoted the 6th's +0.98% through a session that actually closed −0.76% down. It was the only green tile on a red board, and it was green because the data was missing, not because the index had risen.
Why a fraction is the wrong test
A row-count fraction answers “is most of this day here?” The question that matters is “is every instrument here?” Those come apart precisely when one important instrument is the one that failed, which is the case the check exists to catch.
Coverage is now measured per instrument. We take the fullest session in a recent window as the expected instrument set, and report the earliest date that every one of them has reached. One laggard holds the whole watermark back.
It found a second one immediately
Turned on, the rule found a case the fraction had been passing for weeks: ADANIPORTS missing from the daily candle table. 499 of 500 symbols is 99.8%.
And it was too strict in one place
It also flagged a futures and options table, which was a false positive and is the clearest statement of the rule's limit. F&O contracts expire weekly, so a contract that was in the expected set on Monday can never have a row on Friday. That table went back to the row fraction. The rule applies only where the universe is genuinely the same every session.
A later lesson sharpened it again: an index reconstitution dropped 28 names, and because the old session was still inside the window, two tables reported nine sessions behind for over a week while every session in them was complete. Any token with no row in the table's last three sessions is now dropped from the expected set.
The general lesson
Ask a table the question it can answer. We now have three freshness rules and every source declares which one applies to it: per instrument for a fixed universe, “did it run for the last session” for tables whose contents are an output, and a row fraction for the rest. The admin console and the nightly email call the same function, so “the console says current” and “the email stayed silent” are the same claim.