Reveal file. Decision frame: close of . Later records published to .

The launch eve call, tested against what later official findings established

How to read this

On the public record at the close of September 30, 2013, the prior file called consumer enrollment Not demonstrably ready on the public record, with 0 independently verified passes. That call stays as it was: a decision on the record then available, not a prophecy. Later findings below test its requirements and its unknowns. They do not change what was knowable before opening, and nothing here judges any person's honesty or intent.

Each source sits on one side of the date wall by its actual publication date. GAO-15-238 is placed by its publication date of , not by its report number.

Two dated columns

Knowable before opening

Public record through .

  • CGI release, published , 669 days before the decision. Contemporaneous contractor award announcement; not a final federal obligation.
  • GAO-13-601, published , 103 days before the decision. Published June 19; findings mostly based on CMS records through early May.
  • CMS release, published , 98 days before the decision. Press release with CMS launch plans.
  • House E&C staff memo, published , 24 days before the decision. Majority staff position; not GAO finding.
  • Three exact planned deadlines had passed with no public record of completion: Certify qualified health plans (, 61 days before); Navigator awards anticipated (, 46 days before); Finalize and load plan data into the website (, 15 days before). GAO-13-601
  • GAO in June: timely and smooth implementation "cannot yet be determined." GAO-13-601
  • Money on record: CGI announced an estimated $93,700,000 total with a $55,700,000 base (CGI release, ); GAO reported $87,997,938 obligated on one CGI line through (GAO-13-601).
  • Verified passes on record for any core function: 0.

Established later

Published after .

  • House Oversight release, published , 31 days after the decision.
  • GAO-14-694, published , 303 days after the decision.
  • GAO-15-238, published , 520 days after the decision.
  • HHS OIG OEI-06-14-00350, published , 875 days after the decision.
  • Provisional war room tallies in notes dated and : 6, about 100, and 248 enrollments at separate meetings. Not a certified final count for October 1. House Oversight release
  • GAO: launched without verification against performance requirements; not fully tested; capacity not adequately planned; code errors not adequately corrected. GAO-14-694 GAO-15-238
  • FFM obligations rose from $56 million (September 2011) to more than $209 million (February 2014); data hub obligations from $30 million to nearly $85 million. GAO-14-694
  • HHS inspector general: absence of clear leadership; contract not properly managed; recovery for high consumer use within 2 months. HHS OIG OEI-06-14-00350

Earlier requirement, later finding

Each card quotes a requirement or status from the public record before opening, then the later official finding that tests it. Press a card's button to reveal the finding; press again to hide it. Without scripting, every finding is shown.

R2 Testing finished by October 1?

Before opening,

"CMS expects to complete development and testing of the information technology systems necessary for FFEs to determine eligibility for enrollment into a QHP and to enroll individuals by October 1, 2013, when enrollment is scheduled to begin for the 2014 plan year."

GAO-13-601,

Later finding, published

"However, Healthcare.gov and its supporting systems were not fully tested prior to launch, and test documentation was missing key elements such as criteria for determining whether a system passed a test. Thus, CMS's assurance that these systems would perform as intended was limited."

GAO-15-238, . What it establishes: Testing was incomplete at launch and test records lacked pass criteria. Not shown: A pass rate or defect count for any single function.

R6 Performance and response time?

Before opening,

"Service Level Agreements (SLA) between CMS and the states, which define characteristics of the system once it is operational, such as transaction response time and days and hours of availability, are planned to be completed in July 2013;"

GAO-13-601,

Later finding, published

"CMS launched Healthcare.gov without verification that it met performance requirements."

GAO-14-694, . What it establishes: GAO found that launch went ahead without verification against performance requirements. Not shown: Which individual decided, what each person knew, or that this alone produced any specific outcome.

Also later, published

"The Centers for Medicare & Medicaid Services (CMS) did not plan for adequate capacity to support Healthcare.gov and its supporting systems."

GAO-15-238, . What it establishes: GAO found capacity planning inadequate. Not shown: A measured capacity figure or a peak demand figure.

Also later, published

"CMS recovered the website for high consumer use within 2 months, and adopted more effective organizational practices."

HHS OIG OEI-06-14-00350, . What it establishes: Recovery for high consumer use took up to about 2 months after launch. Not shown: An exact recovery date; the excerpt gives a month span, so no day count is computed.

R1 Promised functions delivered?

Before opening,

"The website will add functionality over the summer so that, by October, consumers will be able to create accounts, complete an online application, and shop for qualified health plans."

CMS release,

Later finding, published

"However, CMS did not always ensure that requirements were approved and were linked to source and lower-level requirements. As a result, CMS was hindered in ensuring that expected functionality for the system was delivered."

GAO-15-238, . What it establishes: Requirements management weakened assurance that promised functions were delivered. Not shown: Which promised function, if any, was absent on October 1, 2013.

R3 Final testing and defects?

Before opening,

"While CMS has met project schedules, several critical tasks, such as final testing with federal and state partners, remain to be completed."

GAO-13-601,

Later finding, published

"CMS and its contractors identified errors in the software code for Healthcare.gov and its supporting systems, but did not adequately correct them prior to launch."

GAO-15-238, . What it establishes: Known code errors were not adequately corrected before launch. Not shown: A count of errors or which function each affected.

R7 Contract managed?

Before opening,

"For one contract, with CGI Federal, CMS obligated about $88 million for activities to support establishment of the FFEs, such as information technology and technical assistance."

GAO-13-601,

Later finding, published

"Additional missteps included devoting too much time to developing policy, which left too little time for developing the website, and failing to properly manage its key website development contract."

HHS OIG OEI-06-14-00350, . What it establishes: Contract management of the key website development contract was found deficient. Not shown: Contract dollar figures; the excerpt contains none.

Also later, published

"In September 2013, CMS program officials became so concerned about the contractor's performance that they moved operations to the FFM contractor's offices to provide on-site direction."

GAO-14-694, . What it establishes: An internal concern about contractor performance existed in September 2013; it was not in the public record at the close of September 30, 2013. Not shown: The exact day in September, or that the concern was disclosed to the public before opening.

R8 Readiness determined?

Before opening,

"Whether these efforts will assure the timely and smooth implementation of the exchanges by October 2013 cannot yet be determined."

GAO-13-601,

Later finding, published

"CMS continued on a failing path despite signs of trouble, making rushed corrections that proved insufficient."

HHS OIG OEI-06-14-00350, . What it establishes: Signs of trouble existed and corrections were insufficient, per the inspector general. Not shown: That the signs were public on September 30, 2013.

Also later, published

"Most critical was the absence of clear leadership, which caused delays in decisionmaking and a lack of clarity in project tasks."

HHS OIG OEI-06-14-00350, . What it establishes: The inspector general judged leadership absence the most critical management problem. Not shown: Any finding about an individual's honesty or intent.

R1 Enrollment opened on October 1?

Before opening,

"HHS is on target for open enrollment in the Marketplace, which begins Oct. 1, 2013, and other key milestones approaching in the months ahead. Coverage will begin Jan. 1, 2014."

CMS release,

Provisional tallies, published

  • "6 enrollments have occurred so far with 5 different issuers" (War room meeting, notes dated October 2 to 3, 2013; time of day not stated in the excerpt; count is 'so far' at that meeting)
  • "[a]pproximately 100 enrollments have happened as of this meeting." (War room meeting, notes dated October 2 to 3, 2013; 'as of this meeting'; time of day not stated in the excerpt)
  • "[as] of yesterday, there were 248 enrollments." (War room meeting, notes dated October 2 to 3, 2013; refers to 'yesterday' relative to that meeting, which the excerpt does not tie to one calendar day)

House Oversight release, . What it establishes: enrollments were being recorded in the first days. Not shown: a certified final October 1 total, or the reason any tally was low.

Money: three measures kept apart

The chart shows only like for like snapshots of one measure, FFM obligations, from one later source. The CGI award announcement, the March 2013 CGI obligation line and data hub obligations are different measures and appear only in the table.

FFM obligations, two GAO-14-694 snapshots, US dollarsSeptember 2011: $56 million. February 2014: more than $209 million, drawn to $209 million with a dashed open end because the true value is higher. Zero baseline. Source HR1.$0$50M$100M$150M$200M$250M$56 millionSep 2011> $209 millionFeb 2014US dollars obligated, FFMSource: HR1, GAO-14-694
FFM obligations as reported in GAO-14-694 (published ). The February 2014 bar is a lower bound: GAO wrote "more than $209 million", so the dashed open end marks that the true value is higher. Zero baseline; units are US dollars.
Separate money measures; none is added to or divided by another
MeasureKindDateAmountSource
Contractor award, estimated totalAward estimate$93,700,000CGI release
Contractor award, baseAward estimate$55,700,000CGI release
One CGI contract line, FFE information technology and healthcare.govObligationthrough $87,997,938GAO-13-601
FFM obligationsObligationSeptember 2011$56,000,000 (GAO: $56 million)GAO-14-694
FFM obligationsObligation, lower boundFebruary 2014more than $209,000,000GAO-14-694
Data hub obligationsObligationSeptember 2011$30,000,000GAO-14-694
Data hub obligationsObligation, upper boundFebruary 2014nearly $85,000,000GAO-14-694
FFM increase, September 2011 to February 2014Arithmetic, same measurecomputedmore than $153,000,000; more than 3.73 timesGAO-14-694
Data hub increase, September 2011 to February 2014Arithmetic, same measurecomputednearly $55,000,000; under 2.83 timesGAO-14-694

Early enrollment tallies: provisional meeting counts

The House Oversight committee release of quotes CCIIO war room notes dated and . Each figure is a count as of a meeting, with the scope stated in the note. None is a certified final total for October 1, 2013, and none shows why a count was what it was.

War room tallies by meeting time and scope
CountQuoteMeeting timeScopeStatusSource
6"6 enrollments have occurred so far with 5 different issuers"War room meeting, notes dated October 2 to 3, 2013; time of day not stated in the excerpt; count is 'so far' at that meetingEnrollments recorded in the CCIIO war room notes up to that meeting, across 5 issuersProvisional meeting tally; not a certified final first day totalHouse Oversight release
100 (approximately)"[a]pproximately 100 enrollments have happened as of this meeting."War room meeting, notes dated October 2 to 3, 2013; 'as of this meeting'; time of day not stated in the excerptEnrollments recorded as of that meetingProvisional meeting tally; not a certified final first day totalHouse Oversight release
248"[as] of yesterday, there were 248 enrollments."War room meeting, notes dated October 2 to 3, 2013; refers to 'yesterday' relative to that meeting, which the excerpt does not tie to one calendar dayEnrollments recorded as of the prior day, per the notesProvisional meeting tally; not a certified final first day totalHouse Oversight release

Evidence table: decisive dates and numbers

Every decisive date and number, by side of the date wall ()
SideDateWhatValueSource
Before openingCGI award announcedEstimate $93,700,000; base $55,700,000; 670 days before openingCGI release
Before openingCGI contract line obligations through this date$87,997,938 (about $88 million)GAO-13-601
Before openingGAO-13-601 published103 days before the decisionGAO-13-601
Before openingCMS release: open enrollment begins October 1, 201398 days before the decisionCMS release
Before openingPlan certification plannedPassed 61 days before, no public completionGAO-13-601
Before openingNavigator awards anticipatedPassed 46 days before, no public completionGAO-13-601
Before openingHouse majority staff memo24 days before the decisionHouse E&C staff memo
Before openingPlan data load plannedPassed 15 days before, no public completionGAO-13-601
Before openingDecision point0 verified passes on recordGAO-13-601
LaterCommittee release quoting war room notes dated October 2 and 3, 2013Provisional tallies 6, about 100, 248; published 31 days afterHouse Oversight release
LaterGAO-14-694 published; describes September 2013 move to on site direction303 days after; FFM obligations $56 million to more than $209 million; data hub $30 million to nearly $85 millionGAO-14-694
LaterGAO-15-238 published (March, not February)520 days after; not fully tested; capacity not adequately plannedGAO-15-238
LaterHHS OIG case study published875 days after; recovery for high consumer use within 2 monthsHHS OIG OEI-06-14-00350

What a prudent official could have demanded, and what later records establish

The left column is the demand that the public record of September 30, 2013 already justified. The later column says what official findings published afterward establish about the same requirement, and what they still leave open. This compares records, not people.

Requirement by requirement
RequirementDemand thenLater record establishesStill unknown
R1 Public promiseA dated end-to-end transaction test record for account, application, eligibility result and plan selection, with pass criteria and open defect counts.Some enrollments were recorded in the first days, per provisional meeting tallies; GAO later found requirements and code defects weakened assurance that functions were delivered. GAO-15-238 House Oversight releaseA certified count of enrollments completed on October 1, 2013.
R2 Eligibility and enrollmentA signed test completion report for eligibility and enrollment plus an independent verification of the pass, dated before opening.GAO later found the systems were not fully tested before launch and test records lacked pass criteria. GAO-15-238 GAO-14-694A pass rate or defect count for eligibility and enrollment specifically.
R3 Data exchange (data hub)Signed SLA list by partner and state; external test results per partner; a September readiness sign-off.Supporting systems were not fully tested and code errors were not adequately corrected; data hub obligations rose from $30 million to nearly $85 million by February 2014. GAO-15-238 GAO-14-694Whether the July 2013 SLAs were signed, and per partner external test results.
R4 Qualified health plansCertified plan count by state and a load verification extract dated on or after September 15, 2013.No later excerpt in this folder addresses plan certification or the September 15, 2013 plan data load. Certified plan counts and whether plan data loaded by September 15, 2013.
R5 Consumer assistanceAward notices, call center staffing and application-help scripts tested against the live system, and an in-person system test.No later excerpt in this folder addresses navigator awards or call center readiness. Navigator award date and call center application help readiness.
R6 Capacity and performanceLoad and performance test results against a stated peak volume, the measured ceiling, and a throttling plan if the ceiling is below expected demand.GAO later found capacity planning inadequate and launch without verification against performance requirements; the OIG says the site recovered for high consumer use within 2 months. GAO-14-694 GAO-15-238 HHS OIG OEI-06-14-00350Measured capacity, peak demand and response times on October 1, 2013.
R7 Contractor scope and moneyCurrent obligated total, contract modifications and deliverable acceptance records.FFM obligations rose from $56 million to more than $209 million by February 2014; officials moved operations on site in September 2013; the OIG found the key contract was not properly managed. GAO-14-694 HHS OIG OEI-06-14-00350Outlays, contract modification history and deliverable acceptance records.
R8 Oversight signalsAn independent readiness review dated in September 2013.The OIG found an absence of clear leadership and a failing path continued despite signs of trouble. HHS OIG OEI-06-14-00350What any individual knew on September 30, 2013; the record makes no finding on personal honesty or intent.

Still unknown after the later records