Many government agencies with enforcement powers face a common problem. This means you have to be selective because you only have the resources to visit or audit a fraction of the possibilities. How should they make that choice?
Think of the Occupational Safety Administration as a health department responsible for monitoring and enacting workplace safety regulations. OSHA has jurisdiction over approximately 8 million workplaces, but (in collaboration with state-level agencies) has the resources to actually visit less than 1% of that number.Which one to choose? Matthew S. Johnson, David I. Levine, and Michael W. Toffel, “Making Workplaces Safer Through Machine Learning,” Regulatory Review, Penn Regulatory Programs, February 26, 2024, Foundational Research Paper For more information, see “Improving regulatory effectiveness through better targeting: Evidence from OSHA.” Published in American Economic Journal: Applied Economics, October 2023, 15:4, pp. 30-67. For an ungated preprint version, see . here).
One insight is that if there is some randomness in the inspection process, it can serve regulatory purposes because companies need to pay just a little bit more attention. As it turns out, the largest Her OSHA inspection program by random His OSHA process also allows researchers to examine his post-OSHA inspection workplace safety record from 1999 to his 2014, It was called site-specific targeting. The idea was to create a list of businesses that had the highest injury rates two years ago and then randomly select a group to visit. That way, we could compare what happened to companies that received a visit from an OSHA regulator (randomly) and were regulated versus those that were not regulated (remember, injury rates were similarly high). becomes possible. The authors write: „Randomly assigned OSHA inspections reduce the number of serious injuries at inspected facilities by an average of 9 percent, which equates to 2.4 fewer injuries.“
Five years of injuries.Therefore, each test has a social benefit
The cost is approximately $125,000, approximately 35 times the cost of enforcement by OSHA.
inspection. „
But is it possible to achieve better results while keeping OSHA's limited resources fixed? For example, instead of looking at injury rates from two years ago, we can look at average injury rates from the past four years and What if we identified companies with the highest rates of workplace injuries? But can we do better? Use machine learning models to identify which companies are most likely to be injured, or which companies What if we predicted which companies could most improve safety and focused on those companies? The authors write:
We found that many more injuries could have been avoided if OSHA had targeted inspections using one of these alternative standards. If OSHA had assigned the same number of inspections to establishments that have historically caused the most injuries as it did in the SST program, it would have avoided 1.9 times the injuries that the SST program actually caused. It would have been. If OSHA had instead allocated the same number of inspections to the establishments with the most predicted injuries or the highest estimated treatment effects, they would have avoided 2.1 and 2.2 times as many injuries as the SST program, respectively. It could have been done.
Here are some thoughts.
1) I was surprised to see such a significant improvement by a simple rule of looking back at injury rates over four years instead of just looking at injury rates from two years ago. The reason is that injury rates can vary widely from year to year. For example, imagine a company in which a bad event occurs once in 20 years, but immediately rectifies the situation. In this bad year, it may be on OSHA's priority list, but it won't do much for OSHA inspections. Companies that rank low for accidents over a four-year period are more likely to have real problems.
2) Modifying inspection rules by simply looking at 4-year injury rates, as well as using more sophisticated and difficult to explain machine learning approaches, yields only modest benefits. Perhaps machine learning analysis could help show whether better regulatory targeting could yield greater benefits. If so, regulators may want to figure out how to capture most of those benefits using simple rules that can be explained. Rather than black box machine learning rules that can't be easily explained.
3) One concern is that these new targeting methods eliminate the randomization element. This means companies can now predict that they are more likely to receive a visit from her OSHA. It's not clear whether this is a terrible thing. Companies with a poor workplace safety record over several years should be concerned about a visit from regulators. However, it may be wise to leave an element of randomness in who visits.
Finally, it seems to me that regulators, who are constantly under political pressure, sometimes think of their role as akin to law enforcement. So they have an incentive to show that they're going after people who are clearly in the wrong. But as this OSHA example shows, going after an employer who had a very bad workplace incident two years ago may not be as effective in the workplace as going after an employer with a worse record over a longer period of time. Safety may not be improved.
I wrote last year about similar problems that arise with IRS audits. It turns out that when deciding who to audit, the IRS weighs whether fraud can be easily proven. So they tend to do more audits of low-income people who receive the Earned Income Tax Credit, and computers have shown it's easier to prove fraud. But, of course, auditing low-income earners doesn't yield much benefit. Consider a situation where the IRS audits 10 people, all of whom had income over $10 million last year. Presumably, nine of her audits found nothing wrong, but her tenth audit would result in an additional $500,000 being levied on him. If an IRS auditor values high conviction rates, they will choose one. If they are focused on strategies that will bring them the most returns, they will go after the bigger fish.
My point is not that we should leave the choice of regulatory priorities to machine learning. Rather, the point is that machine learning tools can help you assess whether your existing rules are well-configured and how well those rules perform compared to alternative rules.
8 million, visits
So how do you choose? Would another selection method have greater benefits?