“There are no ordinary people. You have never talked to a mere mortal. Nations, cultures, arts, civilizations, these are mortal, and their life is to ours as the life of a gnat. But it is immortals whom we joke with, work with, marry, snub, and exploit.”
-C.S. Lewis[1]
The physician-patient encounter is intimate. Like other important relationships in our lives, this relationship shows the human desire to be known and seen. Medicine has historically best addressed this desire through an attentive, personalized physician experience. Add to this an exponential level of importance: Our health is arguably the most consequential consideration to us, and yet it is something most of us know little about and have little control over. This makes us, in the role of patients, quite vulnerable.
Enter technology.
Recent developments in AI-driven data analysis have provided hopeful advancements in medicine. Two that really jump out, recently, are the early detection of sepsis[2] and better mammography screening.[3] If you are familiar with my writing,[4] you know that I hold the use of technology in health data analysis to be among the few reasons to feel positive about how AI is impacting humanity. Well-trained machines are detecting and disrupting disease in ways that are saving lives. That’s good news.
However, as humanity races to use the power of technology now at our fingertips, the healthcare industry has fallen victim to the same dilemma as many other sectors: Technology development is outpacing ethical and regulatory boundary-setting.
This conversation is young because the advancements are recent, but enough of us are concerned about the nuances to draw conclusions that deserve our attention. Recent studies give us a reasonable look at the issue. These studies indicate that technology has negatively changed the physician-patient encounter in two ways: diagnosis (risk assessment and care allocation) and data sharing/consent.
There is a theological problem with this: The biblical truth of imago Dei demands that we see and acknowledge individuals as unique creations who bear the very image of God. Yet some of the recent changes in how medicine is practiced disregard our biblical anthropology by inaccurately sorting and commodifying us. The outcomes show that medicine was never meant to be dehumanized.
This is a matter of trust. One poll shows individual doctors are still the most trusted source of health information when compared with institutions like the CDC and FDA.[5] Roughly translated, this might equate to a statement like “I trust my congressperson, but I don’t trust Congress.” So, individual doctors remain the most trusted source of health information, although the share who trust their own doctor “a great deal” or “a fair amount” to make the right health recommendations declined from 93% to 85% over the study period (June 2023 to January 2025; this 18-month window was selected to mitigate the massive trust decline during and immediately after COVID).
What do we trust our providers with? Patients are comfortable sharing data with their hospital but not with commercial tech companies. Researchers surveyed a large sample of patients to identify current awareness regarding health data research and obtain their opinions and views on data sharing for AI research purposes. Respondents had generally low levels of prior knowledge about AI, but most were comfortable with sharing health data with the National Health Service (77.9%) or universities (65.7%). Far fewer were at ease with sharing data with commercial organizations such as technology companies (26.4%). The majority of respondents endorsed AI research on healthcare data (87.4%) and healthcare imaging (86.4%) in a university setting, provided that concerns about privacy and consent were addressed.[6]
The respondents in this study made wise distinctions between trusting people and trusting machines. In current practice, these distinctions are not being honored.
Data shows predictive AI is now quite embedded in most US hospitals’ clinical decision-making: 7 out of 10 hospitals reported using predictive AI in 2024. The most common use of predictive AI was to forecast health trajectories or risks for inpatients.[7] However, patients are likely unaware of the AI element in their care. As of right now, there’s no federal requirement to disclose AI use in patient care, and only a handful of states have begun dealing with this issue. The TRAIGA law in Texas requires healthcare providers to disclose AI use in diagnosis or treatment specifically.[8] California’s AB 3030 requires disclosure on AI communications, but not AI use in diagnosis or treatment.[9] Colorado and Utah seem to be making efforts, but with little headway as of this publication date.
So what’s the problem with all of this? In two words: bias and consent.
First, bias. Models trained on narrow data cannot equally serve all populations. Three recent studies expose the issues here. A 2019 study by Obermeyer et al. found that
a widely used algorithm, typical of the industry-wide approach and affecting millions of patients, exhibits significant racial bias: At a given risk score, Black patients are considerably sicker than White patients, as evidenced by signs of uncontrolled illnesses. Remedying this disparity would increase the percentage of Black patients receiving additional help from 17.7 to 46.5% [emphasis mine]. The bias arises because the algorithm predicts health care costs rather than illness, but unequal access to care means that we spend less money caring for Black patients than for White patients. Thus, despite health care costs appearing to be an effective proxy for health by some measures of predictive accuracy, large racial biases arise.[10]
A 2026 study found MRI-based dementia tools trained mostly on non-Hispanic White data generalized poorly to Hispanic patients. This study brings similar evidence to the above study, showing that the same biased sorting problem is recurring in different clinical tools and a different population, which suggests that this is a pattern in how these AI diagnostic systems get built rather than a single bad algorithm or product.[11]
A 2025 study found that GPT-4 and GPT-4o, which are “increasingly used to generate clinical teaching cases and support diagnostic reasoning,” had biases in their training data that could “skew the portrayal and interpretation of cardiovascular symptoms in women, potentially leading to delayed or inaccurate diagnoses.”[12] You may already be familiar with this issue: When women and men present with the same cardiac symptoms, women are more likely than men to be considered for anxiety or mental health issues as the cause rather than cardiovascular disease or an acute cardiac episode. This study concluded that the training models used biased data, and “these biases risk reinforcing historical disparities in cardiovascular care,” and recommended that “future efforts should focus on bias detection and mitigation.”[13]
These examples are not isolated, but don’t take my word for it. The title of a systematic review doesn’t bury the lede: “Gender and Racial Bias Unveiled: Clinical Artificial Intelligence (AI) and Machine Learning (ML) Algorithms Are Fanning the Flames of Inequity,” published in 2025, says explicitly that this is a structural pattern across clinical AI, not a set of isolated incidents.[14] I have linked that study below, and it’s worth a skim even by a layperson not in healthcare. Sure, it contains a lot of jargon, but the authors chose that provocative title for a reason that we should all pay attention to.
Worth noting: After the authors of the Obermeyer study shared their results with the algorithm’s manufacturer, the manufacturer independently reran the analysis on its own national dataset of 3,695,943 patients and confirmed the bias, finding Black patients had 48,772 more active chronic conditions than white patients at the same risk score. Kudos to this company for paying attention to the study and displaying a desire to actually improve their product by corroborating the finding on a dataset roughly 75 times larger than the original study. This corroboration overcomes any objection that the original findings were some fluke of a single hospital’s data. The authors of the study then worked with the manufacturer to fix the problem: changing the training label from pure cost to a blended health-and-cost index to cut the racial gap by 84%.
This interaction is proof that by becoming aware of and working together to correct these problems, we can gain the benefits of AI capacity to do incredible work in the medical field. But that takes money, intentionality, time, and an ethical mandate we can all agree upon.
Now let’s talk about consent, because it is a parallel crisis: Patient data escapes protections that patients assume exist.
HIPAA covers only a narrow set of entities and leaves most of the digital health ecosystem unregulated. There’s no general federal law covering non-HIPAA health data, but patients don’t know that. In 2023, ClearDATA commissioned a study on this very topic and found that
81% of Americans assume that all protected health data collected by digital health apps is protected under HIPAA. . .
in reality, HIPAA does not safeguard health information in the context of digital apps or other health companies in the same way it does for covered health entities like hospitals and providers. This means it’s perfectly legal for app makers to sell their users’ health data, and most Americans are utterly unaware of this potential invasion of privacy.[15]
Not only can these companies legally sell that data, they already are. A Duke investigation found data brokers were selling mental health data cheaply and with little oversight. This team contacted 37 data brokers and found eleven of them were willing to sell bulk mental health data on Americans, with some offering full names and addresses and even offering data-sorting services by conditions like depression and PTSD alongside age, ethnicity, credit score, and other personal details. Sadly, the Duke research team reported that there was minimal vetting of buyers and no patient consent statements included. Here again, we find industry outpacing regulation. HIPAA does not cover data brokers, and so the commodification of these image bearers’ private information is not illegal. The study concludes that “the lack of clear consumer privacy protections in the U.S. necessitates a comprehensive federal privacy law or, at the very least, an expansion of HIPAA’s privacy protections alongside bans on the sale of mental health data on the open market.”[16]
Finally, I briefly but emphatically submit a study done by electronic health records company EPIC to prove that this troublesome data economy extends past medical settings: “Browsing, purchasing, and location data that people create in the process of seeking care are all sold by data brokers to advertisers, scammers, and more.”[17]
What is the takeaway? Systematic bias and a wholesale disregard for consent, taken together, turn patients into products.
At the beginning of this article, I called us, in our roles as patients, a vulnerable population. We depend on the healthcare system to support us in a sensitive time. For some of us, this is a short ER visit. For others, it is a long season of struggle with disease or injury. I sourced a lot of studies to show that exploitation is inarguably happening to us. Sadly, exploitation of the powerless is theologically familiar.
We need only look to the Hebrew prophets to see that there’s nothing new here. We are not the first people to live inside systems or institutions that treat human beings as objects and not people. In particular, Amos 5:11–12 and 8:4–6 call out those who oppress the poor and innocent. Isaiah 3:14–15 promises judgment on oppressors. Technology may add scale and speed, but the core sinfulness is the same: Fallen people build fallen institutions, and those institutions often exploit the most vulnerable people. At one time or another, we will all be a patient who needs to be seen as a unique individual—we are all potentially the person with the least leverage in this system.
Both the Old and New Testaments of Scripture support this view of the situation. For example, Psalm 139:14 tells us we are “fearfully and wonderfully made” (ESV). 1 Corinthians 13:12 says we are known: “For now we see in a mirror dimly, but then face to face. Now I know in part; then I shall know fully, even as I have been fully known.” Exodus 3:7 tells us God sees us in our most difficult seasons: “I have surely seen the affliction of my people . . . . I know their sufferings.” And Luke 12:6–7 reminds us of our value: “Are not five sparrows sold for two pennies? And not one of them is forgotten before God. Why, even the hairs of your head are all numbered.”
Church, we know who we are.
As a body, we have three obligations in this environment.
Presence: Show up for the patients who have the least power and trust in the system. Women, lower-income individuals, several racial-minority groups, and rural patients report the least confidence in physicians and hospitals. How can we pastorally support these individuals to help them get the care they need? Locally, that might look like hospital/bedside visits, but it might also look like asking healthcare providers in your congregation to develop a practical checklist for patient care and advocacy in this new technologically infused healthcare system.
Communal advocacy: Some congregations have advantages in engaging civil society through well-placed congregants or even geographic proximity (are you close to the capital in your state?). But all of us can write to our lawmakers to exert some pressure about data privacy regulation, algorithmic transparency, and HIPAA’s scope. Use this link to quickly find and write to your representatives (Congress and Senate).
Personal accounting: That checklist above that your congregants made? Use it. Have the Q&A experience with providers so that you can share your personal experience and wisdom with your congregation.
AI has the potential to transform medicine. Some of the advances are truly exciting, and we should be hopeful about them. But with technology that moves this quickly, there are dangers too. The imago Dei stands as a bulwark against the fast-moving but sometimes poorly considered progressions of technology in healthcare.
References
[1] C.S. Lewis, “The Weight of Glory,” in The Weight of Glory and Other Addresses (Macmillan, 1949), 15.
[2] “FDA OKs AI Early Warning System for Hospital Sepsis Detection,” 24x7 Magazine, May 12, 2026, https://24x7mag.com/standards/fda-updates/clearances/fda-approves-ai-early-warning-system-hospital-sepsis-detection/. This technology, developed by Johns Hopkins University researchers, identifies sepsis cases up to 48 hours earlier than traditional methods and has reduced mortality rates by 18%.
[3] Jessie Gommers, Veronica Hernström, Viktoria Josefsson et al., “Interval Cancer, Sensitivity, and Specificity Comparing AI-Supported Mammography Screening with Standard Double Reading Without AI in the MASAI Study,” The Lancet 407, no. 10527 (2026): 505–14, https://doi.org/10.1016/S0140-6736(25)02464-X. AI-supported screening significantly improved sensitivity, catching about 81 of 100 actual cancers versus about 74 of 100 with standard double reading, without any increase in false positives. The trial also found numerically fewer interval cancers and fewer invasive, large, and aggressive-subtype cancers in the AI group, though these secondary results were not statistically significant on their own.
[4] More of the author’s broad views on biblical tech ethics can be found on her Substack Formed, Not Fed, https://formednotfed.substack.com/.
[5] Audrey Kearney, Grace Sparks, Liz Hamel et al., “KFF Tracking Poll on Health Information and Trust: January 2025,” KFF, January 28, 2025, https://www.kff.org/health-information-trust/kff-tracking-poll-on-health-information-and-trust-january-2025/.
[6] Ravi Aggarwal, Soma Farag, Guy Martin et al., “Patient Perceptions on Data Sharing and Applying Artificial Intelligence to Health Care Data: Cross-Sectional Survey,” Journal of Medical Internet Research 23, no. 8 (2021): e26162, https://doi.org/10.2196/26162. For full text, please see PubMed Central: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8430862/.
[7] Wei Chang, Priscilla Owusu-Mensah, Jordan Everson, and Chelsea Richwine, “Hospital Trends in the Use, Evaluation, and Governance of Predictive AI, 2023–2024,” Data Brief: 80 (Office of the Assistant Secretary for Technology Policy, September 2025), https://healthit.gov/data/data-briefs/hospital-trends-use-evaluation-and-governance-predictive-ai-2023-2024/.
[8] Texas Responsible Artificial Intelligence Governance Act, H.B. 149, 89th Leg., Reg. Sess. (Tex. 2025), codified at Tex. Bus. & Com. Code Ann. § 552.051 (effective January 1, 2026), https://capitol.texas.gov/tlodocs/89R/billtext/html/HB00149F.htm.
[9] Cal. Assemb. B. 3030, 2023–2024 Leg., Reg. Sess. (Cal. 2024), codified at Cal. Health & Safety Code § 1339.75 (effective January 1, 2025), https://leginfo.legislature.ca.gov/faces/billNavClient.xhtml?bill_id=202320240AB3030.
[10] Ziad Obermeyer, Brian Powers, Christine Vogeli, and Sendhil Mullainathan, “Dissecting Racial Bias in an Algorithm Used to Manage the Health of Populations,” Science 366, no. 6464 (2019): 447, https://doi.org/10.1126/science.aax2342.
[11] Ngoc-Huynh Ho, Sokratis Charisis, Sachintha Ransara Brandigampala et al., “Addressing AI Bias in MRI-Based Dementia Classification Among Hispanic Individuals,” Alzheimer’s and Dementia 22, Suppl. 1 (2026): e70856, https://doi.org/10.1002/alz70856_105856.
[12] Katharina Krieger, Irbaz Hameed, Giorgio Quer et al., “Generative Pre-Trained Transformer Reinforces Historical Gender Bias in Diagnosing Women’s Cardiovascular Symptoms,” European Heart Journal—Digital Health 7, no. 2 (2025): ztaf131, https://doi.org/10.1093/ehjdh/ztaf131.
[13] Krieger et al., “Generative Pre-Trained Transformer.”
[14] Ahmed Umar Otokiti, Huan-ju Shih, and Karmen S. Williams, “Gender and Racial Bias Unveiled: Clinical Artificial Intelligence (AI) and Machine Learning (ML) Algorithms Are Fanning the Flames of Inequity,” Oxford Open Digital Health (2025): oqaf027, https://doi.org/10.1093/oodh/oqaf027.
[15] “ClearDATA Survey Reveals Many Americans Don’t Realize Personal Data Shared with Digital Health Apps Could Be Sold Without Their Consent,” press release, ClearDATA, July 11, 2023, https://www.cleardata.com/news/cleardata-survey-reveals-many-americans-dont-realize-personal-data-shared-with-digital-health-apps-could-be-sold-without-their-consent/. Survey conducted by The Harris Poll, May 24–26, 2023.
[16] Joanne Kim, “Data Brokers and the Sale of Americans’ Mental Health Data: The Exchange of Our Most Sensitive Data and What It Means for Personal Privacy” (Duke Cyber Policy Program, February 2023), 1, https://techpolicy.sanford.duke.edu/data-brokers-and-the-sale-of-americans-mental-health-data/.
[17] Sara Geoghegan and Ben Winters, “A Health Privacy ‘Check-Up’: How Unfair Modern Business Practices Can Leave You Under-Informed and Your Most Sensitive Data Ripe for Collection and Sale,” Electronic Privacy Information Center (EPIC), June 5, 2025, https://epic.org/a-health-privacy-check-up-how-unfair-modern-business-practices-can-leave-you-under-informed-and-your-most-sensitive-data-ripe-for-collection-and-sale/.