Abstract
Understanding how best to estimate state-level policy effects is important, and several unanswered questions remain, particularly about the ability of statistical models to disentangle the effects of concurrently enacted policies. In practice, many policy evaluation studies do not attempt to control for effects of co-occurring policies, and this issue has not received extensive attention in the methodological literature to date. In this study, we utilized Monte Carlo simulations to assess the impact of co-occurring policies on the performance of commonly-used statistical models in state policy evaluations. Simulation conditions varied effect sizes of the co-occurring policies and length of time between policy enactment dates, among other factors. Outcome data (annual state-specific opioid mortality rate per 100,000) were obtained from 1999 to 2016 National Vital Statistics System (NVSS) Multiple Cause of Death mortality files, thus yielding longitudinal annual state-level data over 18 years from 50 states. When co-occurring policies are ignored (i.e., omitted from the analytic model), our results demonstrated that high relative bias (> 82%) arises, particularly when policies are enacted in rapid succession. Moreover, as expected, controlling for all co-occurring policies will effectively mitigate the threat of confounding bias; however, effect estimates may be relatively imprecise (i.e., larger variance) when policies are enacted in near succession. Our findings highlight several key methodological issues regarding co-occurring policies in the context of opioid-policy research yet also generalize more broadly to evaluation of other state-level policies, such as policies related to firearms or COVID-19, showcasing the need to think critically about co-occurring policies that are likely to influence the outcome when specifying analytic models.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Data availability
The data that support the findings of this study are available from National Vital Statistics System (NVSS) Multiple Cause of Death mortality files (1989 through 2018) but restrictions apply to the availability of these data, which were used under license for the current study, and so are not publicly available. The data can be request under a similar license (data use agreement) from the NVSS.
References
Abouk, R., Pacula, R.L., Powell, D.: Association between state laws facilitating pharmacy distribution of naloxone and risk of fatal overdose. JAMA Int. Med. 179(6), 805–811 (2019). https://doi.org/10.1001/jamainternmed.2019.0272
Ali, M.M., Dowd, W.N., Classen, T., Mutter, R., Novak, S.P.: Prescription drug monitoring programs, nonmedical use of prescription drugs, and heroin use: evidence from the national survey of drug use and health. Addict. Behav. 69, 65–77 (2017). https://doi.org/10.1016/j.addbeh.2017.01.011
Angrist, J.D., Pischke, J.R.-S.: Mostly Harmless Econometrics: An Empiricist’s Companion. Princeton University Press, Princeton (2009)
Blanchard, J., Weiss, A.J., Barrett, M.L., McDermott, K.W., Heslin, K.C.: State variation in opioid treatment policies and opioid-related hospital readmissions. BMC Health Serv. Res. 18(1), 971 (2018). https://doi.org/10.1186/s12913-018-3703-8
Buchmueller, T.C., Carey, C.: The effect of prescription drug monitoring programs on opioid utilization in Medicare. Am. Econ. J. Econ. Policy 10(1), 77–112 (2018). https://doi.org/10.1257/pol.20160094
Callaway, B., Sant’Anna, P.: Difference-in-Differences with multiple time periods. J. Econmet. 225(2), 200–230 (2021)
Castillo-Carniglia, A., Gonzalez-Santa Cruz, A., Cerda, M., Delcher, C., Shev, A.B., Wintemute, G.J., Henry, S.G.: Changes in opioid prescribing after implementation of mandatory registration and proactive reports within California’s prescription drug monitoring program. Drug Alcohol Depend. 218, 108405 (2021). https://doi.org/10.1016/j.drugalcdep.2020.108405
De Chaisemartin, C., d'Haultfoeuille, X.: Two-way fixed effects and differences-in-differences estimators with several treatments. National Bureau of Economic Research Working Paper. No. 29734. (2022)
Chan, N.W., Burkhardt, J., Flyr, M.: The effects of recreational marijuana legalization and dispensing on opioid mortality. Econ. Inq. 58(2), 589–606 (2020). https://doi.org/10.1111/ecin.12819
Cochrane, D., Orcutt, G.H.: Application of least squares regression to relationships containing auto-correlated error terms. JASA 44(245), 32–61 (1949). https://doi.org/10.1080/01621459.1949.10483290
Dave, D., Deza, M., Horn, B.P.: Prescription Drug Monitoring Programs, Opioid Abuse, and Crime (Working Paper 24975). Retrieved from Cambridge, MA (2018). https://www.nber.org/papers/w24975
Encinosa, W., Bernard, D., Selden, T.M.: Opioid and non-opioid analgesic prescribing before and after the CDC’s 2016 opioid guideline. Int. J. Health Econ. Manag. (2021). https://doi.org/10.1007/s10754-021-09307-4
Goodman-Bacon, A.: Difference-in-differences with variation in treatment timing. J. Econmet. 225(2), 254–277 (2021)
Griffin, B.A., Schuler, M.S., Patrick, S.W., Schell, T., Morral, A., Smart, R., Stuart, E.A.: Moving beyond the classic difference-in-differences model: a simulation study comparing statistical methods for estimating effectiveness of state-level policies. BMC Med. Res. Methodol. 21(279) (2021)
Ji, X., Haight, S.C., Ko, J.Y., Cox, S., Barfield, W.D., Zhang, K., Li, R.: Association between state policies on improving opioid prescribing in 2 states and opioid overdose rates among reproductive-aged women. Med. Care 59(2), 185–192 (2021). https://doi.org/10.1097/MLR.0000000000001475
Kennedy-Hendricks, A., Richey, M., McGinty, E.E., Stuart, E.A., Barry, C.L., Webster, D.W.: Opioid overdose deaths and Florida’s crackdown on pill mills. Am. J. Public Health 106(2), 291–297 (2016). https://doi.org/10.2105/AJPH.2015.302953
Kilby, A.: Opioids for the Masses: Welfare Tradeoffs in the Regulation of Narcotic Pain Medications. Massachusetts Institute of Technology, Cambridge (2015)
Kuo, Y.F., Raji, M.A., Chen, N.W., Hasan, H., Goodwin, J.S.: Trends in opioid prescriptions among Part D Medicare recipients from 2007 to 2012. Am. J. Med. 129(2), 221 e221–230 (2016). https://doi.org/10.1016/j.amjmed.2015.10.002
Lee, B., Zhao, W., Yang, K.C., Ahn, Y.Y., Perry, B.L.: Systematic evaluation of state policy interventions targeting the US opioid epidemic, 2007–2018. JAMA Netw. Open 4(2), e2036687 (2021). https://doi.org/10.1001/jamanetworkopen.2020.36687
Li, Y., Li, L.: Propensity score analysis methods with balancing constraints: a Monte Carlo study. Stat. Methods Med. Res. 30(4), 1119–1142 (2021)
Mallatt, J.: The effect of prescription drug monitoring programs on opioid prescriptions and heroin crime rates (Working Paper 1292) (2018). Retrieved from https://ssrn.com/abstract=3050692
Matthay, E.C., Gottlieb, L.M., Rehkopf, D., Tan, M.L., Vlahov, D., Maria Glymour, M.: What to do when everything happens at once: analytic approaches to estimate the health effects of co-occurring social policies. Epidemiol. Rev. 43(1), 33–47 (2021a). https://doi.org/10.1101/2020.10.05.20205963
Matthay, E.C., Hagan, E., Joshi, S., Tan, M.L., Vlahov, D., Adler, N., Maria Glymour, M.: The revolution will be hard to evaluate: how co-occurring policy changes affect research on the health effects of social policies. Epidemiol. Rev. 43(1), 19–32 (2021b)
Mauri, A.I., Townsend, T.N., Haffajee, R.L.: The association of state opioid misuse prevention policies with patient- and provider-related outcomes: a scoping review. Milbank Q. 98(1), 57–105 (2020). https://doi.org/10.1111/1468-0009.12436
McClellan, C., Lambdin, B.H., Ali, M.M., Mutter, R., Davis, C.S., Wheeler, E., Kral, A.H.: Opioid-overdose laws association with opioid use and overdose mortality. Addict. Behav. 86, 90–95 (2018). https://doi.org/10.1016/j.addbeh.2018.03.014
McInerney, M.: The Affordable Care Act, Public Insurance Expansion and Opioid Overdose Mortality. University of Connecticut, Department of Economics, Working papers: 2017–23 (2017). Retrieved from http://web2.uconn.edu/economics/working/2017-23.pdf
Pacula, R.L., Smart, R.: Medical Marijuana and Marijuana Legalization. Annu. Rev. Clin. Psychol. 13, 397–419 (2017). https://doi.org/10.1146/annurev-clinpsy-032816-045128
Paulozzi, L.J., Kilbourne, E.M., Desai, H.A.: Prescription drug monitoring programs and death rates from drug overdose. Pain Med. 12(5), 747–754 (2011). https://doi.org/10.1111/j.1526-4637.2011.01062.x
Pauly, N.J., Slavova, S., Delcher, C., Freeman, P.R., Talbert, J.: Features of prescription drug monitoring programs associated with reduced rates of prescription opioid-related poisonings. Drug Alcohol Depend. 184, 26–32 (2018). https://doi.org/10.1016/j.drugalcdep.2017.12.002
Phillips, E., Gazmararian, J.: Implications of prescription drug monitoring and medical cannabis legislation on opioid overdose mortality. J. Opioid Manag. 13(4), 229–239 (2017). https://doi.org/10.5055/jom.2017.0391
Puac-Polanco, V., Chihuri, S., Fink, D.S., Cerda, M., Keyes, K.M., Li, G.: Prescription drug monitoring programs and prescription opioid-related outcomes in the United States. Epidemiol. Rev. 42(1), 134–153 (2020). https://doi.org/10.1093/epirev/mxaa002
Rutkow, L., Chang, H.Y., Daubresse, M., Webster, D.W., Stuart, E.A., Alexander, G.C.: Effect of Florida’s prescription drug monitoring program and pill mill laws on opioid prescribing and use. JAMA Int. Med. 175(10), 1642–1649 (2015). https://doi.org/10.1001/jamainternmed.2015.3931
Schell, T., Cefalu, M., Griffin, B.A., Smart, R., Morral, A.: Changes in firearm mortality following the implementation of state laws regulating firearm access and use. Proc. Natl. Acad. Sci. u. s. a. 117(26), 14906–14910 (2020)
Schell, T., Griffin, B., Morral, A. Evaluating Methods to Estimate the Effect of State Laws on Firearm Deaths: A Simulation Study. RR-2685-RC. Santa Monica, CA: RAND Corporation (2019)
Schuler, M.S., Heins, S.E., Smart, R., Griffin, B.A., Powell, D., Stuart, E.A., Stein, B.D.: The state of the science in opioid policy research. Drug Alcohol Depend. 214, 108137 (2020). https://doi.org/10.1016/j.drugalcdep.2020.108137
Snell-Rood, C., Willging, C., Showalter, D., Peters, H., Pollini, R.A.: System-level factors shaping the implementation of "hub and spoke" systems to expand MOUD in rural areas. Subst. Abus., 1–17 (2020). https://doi.org/10.1080/08897077.2020.1846149
Sun, L., Abraham, S.: Estimating dynamic treatment effects in event studies with heterogeneous treatment effects. J. Economet. 225(2), 175–199 (2021)
U.S. Department of Labor. Bureau of Labor Statistics (2019). Retrieved from https://www.bls.gov/2019
Wing, C., Simon, K., Bello-Gomez, R.A.: Designing difference in difference studies: best practices for public health policy research. Annu. Rev. Public Health 39, 453–469 (2018). https://doi.org/10.1146/annurev-publhealth-040617-013507
Yarbrough, C.R.: Prescription drug monitoring programs produce a limited impact on painkiller prescribing in Medicare Part D. Health Serv. Res. 53(2), 671–689 (2018). https://doi.org/10.1111/1475-6773.12652
Acknowledgements
The authors would like to thank fellow OPTIC team members Rosalie Liccardo Pacula and David Powell as well as members of the OPTIC advisory board, Erin Bagalman, Collen Barry, Richard Frank, Adam Gordon, Karmen Hanson, Keith Humphreys, Christopher Jones, Jeffrey Locke, and Harold Pollack, for their feedback on the initial design and findings from this work. Finally, the authors want to thank Hilary Peterson for her assistance with manuscript preparation and submission.
Funding
This research was financially supported through a National Institutes of Health (NIH) grant (P50DA046351, PI: Stein). NIH had no role in the design of the study, analysis, and interpretation of data nor in writing the manuscript.
Author information
Authors and Affiliations
Contributions
The authors jointly conceived the idea for the study. BG and RS derived the first iteration of the design of the simulation. BG and GG designed, developed and implemented the original simulation code; JP lead all adaptions and runs under consultation with all authors. GG and JP developed needed graphics and the associated Shiny app for this work. BG, MS, and ES drafted the manuscript with input from all authors. All authors extensively edited and provided input on all phases of the study and all authors read and approved the final manuscript.
Corresponding author
Ethics declarations
Conflict of interest
The authors have no relevant financial or non-financial interests to disclose.
Ethical approval
All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki Declaration and its later amendments or comparable ethical standards. The corresponding author’s Institutional Review Board deemed this study exempt (Human Subjects Assurance Number 00003425 (6/22/2023).
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Griffin, B.A., Schuler, M.S., Pane, J. et al. Methodological considerations for estimating policy effects in the context of co-occurring policies. Health Serv Outcomes Res Method 23, 149–165 (2023). https://doi.org/10.1007/s10742-022-00284-w
Received:
Revised:
Accepted:
Published:
Issue date:
DOI: https://doi.org/10.1007/s10742-022-00284-w