In public health, community health assessments and strategic plans are more than just paperwork, they’re roadmaps for change. These documents guide how health departments and community organizations target issues like food insecurity, tobacco exposure, or access to care. But we are not always able to use these important tools to their full potential because monitoring and updating these documents are labor-intensive and time-consuming. To address this problem, we asked the question, how might we extract meaningful insights from community health assessments and public health strategic plans more efficiently?

Why is this important? If we don’t have the time or capacity to systematically analyze and act on what’s in these plans, we risk missing opportunities to improve population health in a manner that aligns with communities’ priorities.

So, we asked a bold question: Can artificial intelligence (AI) help?

Read the Article in JPHMP

In our recent study, we tested GPT-4o, a generative AI model which is a type of artificial intelligence that can create new content, to see if it could replicate human-led content analysis. We tested GPT-4o on four publicly available public health strategic plans, Social Determinants of Health (SDOH) Accelerator Plans. With support from the CDC, multisector community partnerships developed these systematic action plans aimed at improving five SDOH domains related to chronic disease: food security, tobacco policy, connections to clinical care, social connection, and built environment. These plans are complex and filled with detailed strategies and goals. Manually reviewing the four plans took our team about 40 hours. We wanted to see if GPT-4o could do the same work faster and with reasonable accuracy.

The answer? Sort of.

GPT-4o got some things right, especially basic elements like identifying SDOH focus areas or locating evaluation methods in the plans. For abstracting data from the four plans individually, GPT-4o demonstrated abstraction accuracy of 79% (n = 17 errors) compared to 94% accuracy by the study team. But when it came to synthesizing information across multiple documents or interpreting nuanced content like leadership team composition and priority populations, it stumbled. It made up information (“fabricated” data), missed key details, and at times showed inconsistent outputs when given the same prompts for different plans. When abstracting synthesis data elements across the four plans, GPT-4o demonstrated an accuracy of 50%. Although GPT-4o required fewer hours compared to the study team abstraction, the resource savings were reduced when considering the time needed to develop prompts and correct errors.

That’s not to say this experiment was a failure, far from it. One key takeaway is that AI has potential to support routine, high-level data extraction tasks. Public health professionals pressed for time could use GPT-4o as a first-pass tool, if the results are carefully reviewed and validated by human eyes. We also learned that prompt engineering (how you ask GPT-4o questions) is just as important as the data itself. Refining those prompts took 10 hours, more than we anticipated, and even then, some errors persisted.

Who can use our findings? Local and state health departments, public health researchers, community organizations, and funders can all benefit. Anyone who needs to make sense of large volumes of qualitative data can adapt our approach. The prompts and methods we used are publicly documented, meaning others can replicate and refine them in their own work. Especially for agencies with limited staff or tight deadlines, this hybrid AI-human method could save time and increase consistency across evaluations.

Here are the key implications for public health professionals:

• GPT-4o shows promise for extracting well-defined data from complex public health plans focused on social determinants of health (SDOH).
• Insights from this study may apply to broader public health planning efforts, but using GPT-4o effectively requires well-crafted prompts and thorough quality checks.
• While GPT-4o can save time and resources, users must weigh potential errors and approach the tool with caution, given its experimental nature.
• AI cannot replace public health practitioners’ expertise, but its prudent use can amplify our capacity, not replace our judgment.

Want to dig deeper? You can read the full study in the July/August 2025 issue of the Journal of Public Health Management and Practice or check out our appendices with the exact prompts we used. Email me at kdepriest@rti.org if you’d like to connect or collaborate.

We would like to acknowledge our co-authors Kailen Gore, Robert Chew, and Dr. Karen Hacker, whose expertise and contributions were essential to the research behind this blog post.

Kelli DePriest, PhD, RN, is a nurse researcher at RTI International. She works to advance public health through innovative mixed-methods research, leadership in program evaluations, and involvement in shaping policies that prioritize well-being and address the social determinants of health. Dr. DePriest received her PhD from Johns Hopkins University.

LaShawn Glasgow, DrPH, MPH is a researcher with over 20 years of public health experience. She worked for the CDC and Pennsylvania Department of Health before joining RTI International in 2008. She received her DrPH from University of Pittsburgh’s Graduate School of Public Health and MPH from Rutgers School of Public Health.

John Feher III is a researcher at RTI international. He provides mixed-methods research and project management support to a variety of projects aimed at advancing public health. Mr. Feher III received his BA from American University.

Clint Grant, MSPH, is the director of healthy community design at the Association of State and Territorial Health Officials (ASTHO), where he leads efforts to advance community design strategies that enhance the public’s health. He holds a MSPH degree from the University of North Carolina at Charlotte.

Peter L. Holtgrave, MA, MPH, is senior director of public health infrastructure and systems at the National Association of County and City Health Officials (NACCHO), where he oversees the organization’s Performance Improvement, Workforce and Leadership Development, and Health Equity and Social Justice portfolios.