|
In recent weeks, both the Senate and House Budget Committees have held oversight hearings on the Congressional Budget Office (CBO). More hearings are planned. Members have been rigorous in their questioning, and they appear focused on strengthening the institution to make the CBO the best version of itself. This is essential given the central role that Congressional scorekeepers at CBO and the Joint Committee in Taxation (JCT) play in the policymaking process.
But one aspect of CBO Director Hall’s testimony, which he has repeated several times, struck me as inaccurate. The most succinct expression of this broken logic came during a Senate hearing last month when Hall responded to a question about how the CBO can become more transparent and responsive by saying, “You have identified the two biggest problems we have, the responsiveness and the transparency. Unfortunately, they sometimes collide with fixed resources. “
I addressed this matter directly in a recent article on scorekeeper transparency for National Affairs. The excerpt below is taken from a section headed, “False Constraints.”
When presented with the case for replicability [which would accommodate full transparency], the scorekeepers sometimes respond that severe resource constraints — like having too little funding, too few staff, and generally not enough time given their legal obligations to Congress — stand in their way. But these arguments are misleading at best. There may need to be some consideration for the transition to replicability, and for how to carry it out while still doing all the work Congress demands of the scorekeepers. But beyond the transition, the scorekeepers’ budgets need not be higher to accommodate full replicability. In fact, there is a path to ensuring replicability that should increase the scorekeepers’ efficiency, make their models cheaper and easier to maintain, and make their staffing and training requirements easier to satisfy. And that is not even accounting for the free technical contributions from independent experts that the scorekeepers are likely to receive.
To understand this point, we need to delve into a few basics of software development. Remember, the scorekeepers’ models, both spreadsheets and complex computational suites, are fundamentally software products. There are four basic techniques that are widely recommended to ensure the quality of any software project. These four techniques are somewhat technical, but understanding them is essential to understanding why the scorekeepers are wrong to suggest that resource constraints would prevent replicability. They are version control, automated testing, documentation, and internal code review.
Implementing version control means adopting a system for carefully tracking changes to software and data over time. Implementing automated testing means building the capability to automatically run your software in order to see how your results change over time, among other objectives. Documentation means writing down descriptions of the structure and details, and instructions for how to use your software and data. Practicing internal code review means ensuring that every substantive change is reviewed by someone beyond the modeler proposing the change. These four techniques provide immeasurable efficiency gains for development teams. Most important, these techniques are essential for limiting the bugs in software and generally ensuring its quality.
If these basic standards for software quality are followed in a modeling project, then the additional cost of providing external replicability (in terms of time, staff resources, and money) should be negligible. Every one of the requirements for external replication is satisfied simply by following these standard software-development practices internally. With version control you have an easy way to track the versions of your model that contributed to any particular analysis. With automated testing, you have concrete examples — in the automatic tests themselves — of exactly how to run your model under a wide variety of scenarios, including, ideally, the scenarios that you publish in your reports. With documentation, you have pre-written English descriptions of your code and data. With internal code review, you gain experience making your project accessible to reviewers. Beyond these basic quality assurances, the marginal cost of providing external replicability, above and beyond writing high-quality models, is essentially zero.
Given the importance of their work and the need for accuracy, it may be surprising to learn that the scorekeepers do not consistently follow these standards (a fact I have learned through conversations with scorekeeping staff and others close to their work). Failing to follow these practices leads to something called “technical debt.” And the term is awfully apt. It describes a situation in which you have sacrificed quality for some expediency and are now paying a long-term carrying cost in the form of lower-quality software that is more expensive to maintain.
This is not to say that the scorekeepers must necessarily adopt these four techniques before they begin practicing replication. But it does show that resource constraints are no excuse because, if it is not free or close to free to make their models replicable, then there must be some other mismanagement or inefficiency going on that needs to be addressed regardless of any new efforts to achieve replicability.
The scorekeepers themselves should want to pay off their technical debt and adopt these techniques as their path to replicability, but they should also be encouraged to do so by Congress. This is both because they need to know that Congress prioritizes this work and will allow them to devote some energy and resources to the required transition, and because they need to know that Congress will not penalize or embarrass them for their past practices. It is likely that part of the reason CBO and JCT do not already practice replicability is that they are embarrassed by the software quality of their models. Given that trust in the scorekeeping institutions should be carefully protected, Congress should be sensitive to their plight, and, should the scorekeepers request a transition period and temporary funding, Congress should consider those requests seriously.
To read more about how JCT and CBO can achieve transparency and greater efficiency, please check out the rest of my article on the National Affairs website.
There is a path to ensuring replicability that should increase the scorekeepers’ efficiency, make their models cheaper and easier to maintain, and make their staffing and training requirements easier to satisfy.
|
|
