The manual says 15 months. It hasn't been 15 months since 2006.

A citation proves a claim came from the document. It proves nothing about whether the document is still true. Three findings from a retrieval system over CMS manuals, and one problem retrieval cannot solve at all.

I built a retrieval system over CMS documentation to answer questions about durable medical equipment and telehealth. Hybrid retrieval, a cross-encoder reranker, a 7B model generating structured answers with citations, all running locally on a laptop. Here is a question I asked it and the answer it gave:

That is about as confident as the reranker gets. Ask a related question and it will tell you the beneficiary receives a purchase option in the tenth month. Both answers are quoted accurately from the current published chapter. Neither has been correct since January 1, 2006.

Where the wrong answer comes from

The passage is §30.5:

Section 5101(a) of the Deficit Reduction Act of 2005 amended §1834(a)(7)(A) of the Social Security Act to cap capped rental at 13 months, and to require that on the first day after the thirteenth month, the supplier transfer title to the beneficiary. The tenth-month election disappeared, and with it the branch where the supplier keeps the equipment. This took effect for items whose first rental month fell on or after January 1, 2006, so §30.5 was live guidance for real claims into 2007 and dead thereafter.

One thing did survive: complex rehabilitative power wheelchairs still carry a purchase option, offered at the time the supplier furnishes the item, under §1834(a)(7)(A)(iii). The last sentence of the passage above, on electric wheelchairs, is the one part of the section that isn't entirely stale.

Look at the revision stamp. Nothing has amended that section since 2003.

The chapter is unmaintained in three different ways

That stamp is easy to misread, so it's worth being precise about what it does and doesn't mean. Transmittal numbers run sequentially across the whole CMS Manual System rather than counting revisions to any one chapter. You can see that in my own corpus: Claims Processing Chapter 20 is Rev. 12557, issued 03-28-24, while Benefit Policy Chapter 15 is Rev. 13774, issued two years later. Two different publications drawing on one counter. So the number doesn't tell you Chapter 20 changed 12,557 times. It tells you the chapter was reissued in 2024 with §30.5 untouched inside it since 2003.

§30.5 is not the only one. §100.2, "Certificates of Medical Necessity (CMN)," also stamped Rev. 1, 10-01-03, says in the present tense that "the supplier must receive a signed Certificate of Medical Necessity (CMN) from the treating physician." CMS retired those forms for dates of service on or after January 1, 2023, and says so elsewhere in the same chapter.

And §40.2, which is properly dated at Rev. 3196 and correctly distinguishes items furnished before and after January 1, 2006, opens by describing pre-2006 items as having "reached the 13-month rental cap." Pre-2006 items were capped at 15 months. That's the section that gets the regime boundary right and the number wrong.

So the chapter contains stale undated text, retired requirements stated in the present tense, and an arithmetic error inside the passage that handles the transition correctly. A retriever ranking by relevance to "capped rental" prefers §30.5 over all of it, because §30.5 is the section actually titled "Capped Rental Items."

The form question

Another question I tested asks what form documents medical necessity for certain DMEPOS items. The system answers "Certificate of Medical Necessity," citing Chapter 20, and my evaluation scored that correct.

It is correct, and it is missing the thing that matters. From the same chapter:

The CMN is still the right name for the right form. It still governs dates of service before 2023, which are still in play for appeals, reopenings, and audits, and the DME MAC supplier manuals still describe it in the present tense today. What the system omitted is the date qualifier that decides which of those worlds you're in.

Retrieval is not what failed here. Three of the four passages the model cited contain the sentence saying providers and suppliers no longer need to submit these forms for services rendered after January 1, 2023. The instruction was in the context window, in the passages the model itself chose to cite, and the answer came out undated anyway. That is worse than a retrieval miss, because there is no ranking change that fixes it.

An answer that is correct and undated is not a small defect in a domain where every rule is scoped to a date of service.

The one a human review would pass

I keep a set of questions the corpus can't answer, to check that the system declines instead of inventing. One asks whether Medicare Part D covers insulin. The system answered "Medicare Part D covers insulin," citing §140.1.1 of Chapter 20. Here is that section's title:

The heading says Part B. Insulin used with a covered external insulin pump is a Part B supply drug, furnished with the equipment. Insulin dispensed at a pharmacy is Part D. It is the same molecule reaching the patient by two routes that Medicare treats as separate benefits, with different suppliers, different claim paths, and different first-level appeals, which is why the Inflation Reduction Act capped cost-sharing for them in two separate sections with two different effective dates.

What makes this the worst of the three is that the answer is true. Part D does cover insulin. Checking outputs against your own knowledge marks it correct and moves on. The error only surfaces if you check the answer against the passage it cited rather than against the world.

The problem retrieval cannot solve

I also asked whether a distant site provider can use audio-only technology. The system answered "No," citing a telehealth booklet passage scoring 0.9991.

On the face of the booklet, "No" is reasonable. It describes audio-only as available when the patient is at home, and elsewhere restricts home as an originating site to behavioral health, substance use disorder, and monthly ESRD assessments. Two passages, four pages apart, that together imply a narrow answer.

"No" is still wrong, and I got it wrong the same way when I first read this. The Consolidated Appropriations Act of 2026 (Pub. L. 119-75, §6209), signed February 3, 2026, extended the geographic waiver and home-as-originating-site for all telehealth services through December 31, 2027. The booklet describes the statutory baseline. Congress has been overriding that baseline with short-term extensions since 2020, and the December 2025 edition captures a moment between them.

No reranker fixes this. No larger model fixes it. The correct answer is not in the corpus, and nothing inside the corpus indicates that the corpus is incomplete, except a line on the same page telling the reader to check the CMS website for current information.

What a relevance score measures

The clearest case is bariatric surgery, which appears nowhere in my documents and scores 0.916. Three of my sixteen genuinely answerable questions score lower. The reranker is doing what it was trained to do, since bariatric surgery is a Medicare coverage topic and this is a corpus of Medicare coverage documents. Topical relevance is not answerability. That distinction is old, and score-based confidence gates ignore it.

What I'd build differently

None of this is Medicare-specific.

A citation is not a currency check. Grounding a claim in a retrieved passage proves the claim came from the document, and proves nothing about whether that passage is still operative. Every attribution metric I know of scores the first thing and is silent on the second.

Regulatory answers are functions of a date, and nothing in my system carried one. Every claim in that corpus has an effective date, many have an end date, and the index has neither. Better ranking will not help. The fix is a metadata layer: parse the revision stamp out of each section header at ingest, carry it, and refuse to serve a section whose stamp predates a statute that touches it.

That last part is where it stops being an engineering problem. Knowing that §30.5 was overridden requires knowing that the Deficit Reduction Act of 2005 exists and which sections it amended. Nobody has published that mapping. It is why the metadata layer is still unbuilt, and why I would not put a system like this in front of a supplier today.

What was actually in the index

Two things I should state plainly, because I found both while checking this piece rather than while building.

The index contains two synthetic files, _sample_dme.txt and _sample_rpm.txt, left over from pipeline smoke tests. Each opens by announcing it is not real CMS guidance. One of them is the top-ranked citation for a question in my evaluation set, which means one of my scored answers came from a fixture I wrote rather than from CMS. So the corpus is not "three CMS documents," and I described it that way for months.

The system also has a field for a verbatim supporting quote. Fifteen of the 23 answers filled it, and 13 of those 15 quotes really are present, word for word, in a passage the answer cited. The other two are not. Asked whether durable medical equipment must serve a medical purpose, the model put this in the quote field:

It paraphrased and presented the paraphrase as a quotation. The citation is correct, the passage is correct, the claim is supported, and the quotation marks are lying. A fabricated quote passes every check this system performs, which makes it the one failure here that a stricter grounding metric would not have caught either.

What the corpus is not

The three real documents are Benefit Policy Manual Chapter 15 (Rev. 13774, issued 05-08-26), Claims Processing Manual Chapter 20 (Rev. 12557, issued 03-28-24), and MLN901705, the December 2025 telehealth booklet. Chapter 15 is titled "Covered Medical and Other Health Services" and covers far more than equipment. Chapter 20 is claims processing. The booklet is billing guidance.

Item-level coverage criteria for a given patient live in National and Local Coverage Determinations and Policy Articles, none of which are in the index, though Chapter 15 references some of them by number. The system cannot tell you whether an item is covered for a patient, which is what I had been calling it for months.

It is also fee-for-service only. Fifty-five percent of the eligible Medicare population is enrolled in Medicare Advantage plans that apply their own criteria and prior authorization rules, and a tool built on these manuals answers confidently and silently wrong for all of them.

All three documents are public CMS publications. No patient data was involved. The evaluation harness, both question sets, and the raw outputs behind every number here are at github.com/Dnakitare/medicare-rag-eval, including the two synthetic fixtures. The retrieval pipeline itself is not published.

The evaluation is small: sixteen answerable questions and seven unanswerable ones, one run. That is enough to show these failures exist and nowhere near enough to say how often they happen.

Corrections, 24 August 2026

This essay went up on 23 August with two claims in it that were wrong. Both came from bugs in the script I wrote to gather the evidence, which is the thing the essay is about, so they belong in it rather than in a quiet edit.

The first version said the supporting-quote field was empty on all 23 items and that the system had never quoted anything. My dump script read a.quote. The attribute is supporting_quote, and getattr with a default swallowed the difference twenty-three times. The field was never empty. Fifteen answers filled it and thirteen of those are genuinely verbatim, which is the finding now in the section above. I published a confession manufactured by reading the wrong attribute name, in an essay arguing that people mistake artifacts of their measurement for properties of the world.

The second said none of the passages the model cited contained the CMN retirement note. My dump truncated every stored passage at 900 characters against a 1,200-character chunk, so the evidence I checked was three quarters of what the model actually saw. Three of the four cited passages contain it. The corrected version of that finding is stronger than the one I published, because a retrieval miss has an obvious fix and this does not.

Smaller fixes in the same pass: the insulin section is §140.1.1, not §140.1; the §30.5 excerpt now runs to the end of its paragraph instead of stopping where it suited me; the telehealth extension carries its public law number; and the two telehealth passages are two pages apart, not four. The repository history has all of it.