I Asked ChatGPT About Public Health Service and Resource Sharing and the Output Was Not Actually Terrible

Posted on March 2, 2023 by Jason Orr

This entry is part 26 of 43 in the series Wide World of Public Health Systems

New technologies such as ChatGPT, an artificial intelligence natural language processing tool, have the potential to benefit public health but must be approached with caution.

The world is changing, and new (sometimes terrifying) technology is now at our fingertips. New technologies have the potential to benefit public health, perhaps humanity, but must be approached with caution. One new tool, “ChatGPT”—an artificial intelligence natural language processing tool—can, among other functions, respond to questions or prompts posed to it.¹ Basically, a more reasoned Google search. This new software may hold promise for those wishing to expand their minds and learn about new topics. Though, as this blog may illustrate, and acknowledged by the developers,^1,2 ChatGPT can also produce reasonable-sounding responses that are completely inaccurate.

For this blog post, I constructed a narrow prompt on a specific public health topic to the ChatGPT system and critiqued its response. The chosen topic, “cross-jurisdictional sharing” (CJS), refers to government-to-government services and resource sharing arrangements to support delivery of services. See Notes at the end of this post for a detailed rationale for the prompt.

Prompt to ChatGPT

Create a brief analytic essay that describes the frequencies and types of cross-jurisdictional service and resource sharing arrangements among local health departments and state health departments in the United States, as of 2021. Cite any sources. ~ Submitted on February 26, 2023

Response from ChatGPT

As of 2021, cross-jurisdictional service and resource sharing arrangements among local health departments and state health departments in the United States are common but vary in frequency and type.

According to a study published in the Journal of Public Health Management and Practice in 2021, cross-jurisdictional service and resource sharing arrangements are common among local and state health departments in the United States. The study found that over 90% of local health departments and state health departments reported engaging in at least one type of cross-jurisdictional sharing, such as shared staffing, shared services, or shared programs. The most common types of sharing were program-specific sharing and shared services, with about 70% of local health departments and 80% of state health departments reporting engagement in these types of sharing.

The study also found that the frequency and type of sharing varied by geographic region and population size. For example, local health departments in the Northeast region were more likely to engage in program-specific sharing, while local health departments in the South were more likely to engage in shared services. In addition, larger local health departments were more likely to engage in all types of sharing, while smaller local health departments were more likely to engage in program-specific sharing.

Overall, cross-jurisdictional service and resource sharing arrangements have become an important strategy for improving the efficiency and effectiveness of public health services in the United States. However, there is a need for ongoing evaluation and monitoring of these arrangements to ensure that they are meeting the needs of communities and improving public health outcomes.

Sources: Leider, J. P., Blakely, C., & Castrucci, B. C. (2021). Cross-Jurisdictional Sharing of Public Health Services and Resources Among Local and State Health Departments in the United States. Journal of Public Health Management and Practice, 27(2), 148-155. https://doi.org/10.1097/PHH.0000000000001196

Critique
At first blush, the response appears adequate and accurate. To OpenAI’s credit, the response was logical and seemed to be delivered from a position of knowledge. It was only after considering specific claims and percentages in the response that the review began to go sideways.

1. The central findings described by the article appear reasonable…

From my prior work in this area, the general content and conclusions were reasonable: CJS arrangements vary widely (eg, can represent any formal or informal arrangement between governments) and additional evaluation of service delivery arrangements is warranted. The percentages reported were feasible, given that there are so many different ways that local health departments (LHDs) and state health agencies (SHAs) work with one another to deliver services and CJS arrangements are likely underreported. Also, different states and regions (both regions of states and regions within states) have differing paradigms and mores for sharing resources and services, such as if the Northeast region LHDs may share programs whereas Southern region LHDs may share staff or resources.

2. …however, the “study” described by ChatGPT does not appear to exist…

The central argument, while compelling, does not appear to be clearly evidenced. After quite a bit of searching, I was unable to find the study described in the response (see the next item related to the source cited) and could not verify the specific findings. While the claims that 90% of local and state health departments engaged in at least one form of sharing and 70% of LHDs and 80% of SHAs engaged in program-specific sharing and shared services seem accurate, these percentages are fairly high and could not be confirmed. For reference, the 2019 NACCHO Profile study found that only 55% of LHD respondents (n = 369 LHDs) reported sharing services or resources3 and the 2016 ASTHO Profile study found that 27% of SHA respondents (n = 49 SHAs) shared resources with other states, 71% of SHA respondents (n = 49 SHAs) facilitated local resource sharing, and 88% of SHA respondents (n = 49 SHAs) worked with LHDs on projects. The last example appears closest to the claimed percentages and, as noted in the critique above, CJS arrangements are likely underreported. Overall, however, the core arguments from the ChatGPT response could not be confirmed.

3. …and the source cited in ChatGPT’s response is completely inaccurate.

There really is no discernable aspect of the citation that is factual and no such article exists that a) includes the specific authors listed, b) includes the specific title listed, or c) was published in that specific volume and issue of the Journal of Public Health Management and Practice (JPHMP). This was confirmed by one of the source’s listed “authors” (Dr. Leider). The URL and DOI of the article link to a completely unrelated article. Further, multiple searches using the listed authors, title, and journal did not lead to a more appropriate reference to the study described in the response.

In summary, ChatGPT—in this particular instance—offered a clear and comprehensive response to the submitted prompt that was only partially rooted in reality. While mostly correct in general claims and conclusions, this example shows that such outputs must be taken with a sizeable grain of salt before internalizing or sharing learnings.

References

1. Staudacher N. ChatGPT General FAQ. OpenAI Help Center. https://help.openai.com/en/articles/6783457-chatgpt-general-faq. Published February 1, 2023. Accessed February 26, 2023.
2. OpenAI. ChatGPT: Optimizing language models for dialogue. OpenAI. https://openai.com/blog/chatgpt/. Published February 2, 2023. Accessed February 26, 2023.

Notes

The topic of “cross-jurisdictional service and resource sharing arrangements” was specific and objective enough such that outputs may be critically critiqued; I chose CJS as a topic based on my knowledge and experience in this area (see Center for Sharing Public Health Services).
Requesting a “brief analytic essay” enabled more of a narrative response than a list of items.
Requesting description of “frequencies and types” was to obtain quantitative outputs for critique.
The prompt was narrowed to “local health departments and state health departments in the United States” to ensure that outputs were bounded for specific scenarios.
The timeframe “as of 2021” was used for audience knowledge; OpenAI trained ChatGPT on data through 2021 and ChatGPT is not connected to the internet.

Author Profile

Jason Orr: Mr. Orr is a Researcher with the Center for Public Health Systems. He is experienced in policy analysis and mixed-methods research as well as systems design, systems analysis, and engineering project and risk management. He holds a BS in Chemical Engineering and an MPH from Kansas State University and is a doctoral candidate in Systems Engineering at Colorado State University. He has academic interests in topics related to public health services frameworks (e.g., Foundational Public Health Services); collaborative service delivery (i.e., cross-jurisdictional or cross-sectoral collaboration); and other public health systems transformation and innovation initiatives.