Tanzeem Choudhury

This study explores the potential of integrating LLMs into healthcare to enhance empathetic patient interactions, comparing chatbot responses to those of physicians using real-world data and novel empathy evaluation metrics. 1️⃣ ChatGPT's responses were found to be more empathetic than physicians' in 72.85% of cases, according to human evaluations. 2️⃣ Introduced LLaMA-EMRank metrics (zero-shot, one-shot, few-shot, and ensemble methods) showed ChatGPT’s higher empathy levels compared to physicians. 3️⃣ The ensemble method, which combines results from multiple evaluation approaches, showed the closest alignment with human judgment, validating the automatic empathy metrics. 4️⃣ ChatGPT’s responses were more fluent, indicated by lower perplexity scores in 92% of cases, though fluency did not strongly correlate with empathy. 5️⃣ Using actual patient data ensures findings are applicable to real healthcare settings, highlighting the practical potential of empathetic chatbots. 6️⃣ Addressed the subjective nature of empathy and variability in evaluation methods by using multiple examples and consensus voting. ✍🏻 Luo M., Warren C.J., Cheng L., Abdul-Muhsin H.M., Banerjee I. Assessing Empathy in Large Language Models with Real-World Physician-Patient Interactions. arXiv. 2024. DOI: 10.48550/arXiv.2405.16402v1

86

9 Comments

https://tcrn.ch/3xszfxx Interesting piece in TechCrunch about what #generative AI could mean for healthcare. Some key points: ⭐ The article described the potential of using generative AI in #radiology. ⭐ Worth noting that both examples given in the radiology section aren't actually #generativeAI though (both examples are machine learning). ⭐ Despite enthusiasm from the investor community, patients and clinicians are less sure about its value ⭐ Only 53% of surveyed consumers thought it could add value in healthcare ⭐ A paper in JAMA Pediatrics found ChatGPT made errors diagnosing pediatric diseases 83% of the time Despite reservations, I think there's no doubt that generative AI could be game-changer, especially for routine, mundane tasks. Be interesting to hear what others think ! #radiology #AI

19

8 Comments

🎙️A wide ranging discussion in the The Economist babbage podcast covering a wide range of healthcare benefits and concerns in healthcare artificial intelligence dealing with types of models including neural networks, generative AI, validation, monitoring and medical device regulation led by Alok Jha and Natasha Loder with contributions from myself, Pranav Rajpurkar, Peter Kecskemethy and the inimitable Dr Hugh Harvey I've always wondered how to pronounce Peter's surname, I think that's been nailed here 😊 #AI #healthcare #generativeAI https://lnkd.in/eJ__H_e6

27

Big healthcare AI news yesterday as Tempus AI received the first-ever CPT code for an algorithm-only analysis of previously-sequenced, laboratory developed test transcriptomic data. TLDR: AI molecular phenotyping has a new business model. The PurISTSM test uses RNA profiling to identify molecular subtypes in patients with advanced pancreatic ductal adenocarcinoma, which may inform first-line therapy selection and management. While I'm skeptical that payer balance sheets are going to solve the "who pays for healthcare AI" problem, the reimbursement pipes for clinical AI are maturing. I am surprised to see more progress for usage-based vs. performance-based models given the many challenges with FFS economics. It will be interesting to see how payers assess the "medical necessity" and therefore reimbursability of clinical AI as these models come to market. https://t.co/eLe18SmihV

420

9 Comments

Important talks on #AI hosted at Northeastern University, including a very enlightening one by John Halamka, M.D., M.S. on #AI and what is needed for #trainingdata to counter flawed, missing, incomplete, mislabeled data. He describes the frontline reality of data collection, what is happening in the clinic visit or #emergencymedicine visit, as to why things like #SDoH #data in the #EHR is likely inaccurate. The best people to build #medicalAI and #healthtech have been at the frontlines. Almost all of our #tech and #datascience talent for HER Heard comes from Khoury College of Computer Sciences and Northeastern University. If you want down-to-earth, dedicated, hardworking, smart students or graduates, you can't go wrong looking there. The events held are world-class as well. https://lnkd.in/epXhSCgk

6

1 Comment

News Report Finds Racial Bias in Radiology Reports A recent Yale study found that when asked to summarize test results to be understood by nonprofessionals, the Language Model (LLM) produced results at lower reading levels if told the patient was black vs white. This news report has implications well beyond health, it impacts education and other areas. The findings suggest that if our data is biased, then AI's will be biased. Check out the full report here: https://lnkd.in/edSN_iTa #health #AI #education #racialbias #diversityandinclusion

6

Alexandr Wang, CEO of ScaleAI and an MIT alumnus, recently shared a powerful framework on the future of AI, emphasizing reasoning as the next big milestone. At Equ Healthcare, we believe the path to achieving this in healthcare lies in neuro-symbolic AI, particularly through program synthesis in programming languages, an area where our research at MIT has led to groundbreaking work. Program synthesis allows AI to bridge human intent and machine execution, enabling systems to reason, learn, and integrate expert knowledge—essential for solving complex challenges in healthcare. We’re building systems that combine the best of data-driven AI and symbolic reasoning to create truly intelligent tools for clinicians. The next phase of AI isn’t just bigger models—it’s smarter systems that work alongside experts to deliver real-world impact. We’re excited to be at the forefront of this transformation. How do you see AI reshaping industries like healthcare? #AI #HealthcareInnovation #NeuroSymbolicAI #EquHealthcare #FutureOfAI

39

1 Comment

As the discourse shifts from models to compound AI systems / agents, we need better AI benchmarks to evaluate multi-modal and multi-step task performance, especially in healthcare and life sciences. When we wrote the first paper demonstrating ChatGPT's performance on the USMLE, we chose the US Medical Licensing Exam as a benchmark for accessibility, speed, and ease. This benchmark was never intended to represent AI model performance on real-world clinical tasks. Today, I still see so many research teams and startups using benchmarks (like the USMLE) that are ill-fitted for assessing the true clinical or scientific performance and utility of the models they are developing for real-world contexts. Benchmark development may be seen as a "less sexy" area of research, but it is of paramount importance. Years after the rise of the transformer, we still lack adequate benchmarks for so many single-step tasks in biomedicine. With compound AI systems (i.e., architectures that integrate multiple AI models to perform complex tasks) emerging, we need new benchmarks for agentic behaviors. I'd argue that developing an agent with novel capabilities without at least proposing a companion benchmark (if an industry standard does not yet exist) may hinder the adoption of said agent, especially for high-stakes workflows. Designing more benchmarks that capture/simulate real-world clinical and scientific workflows will help us mitigate the major discrepancies observed between in silico and in vivo performance and better support safe + effective deployment of AI in biomedicine. There are already brilliant people focused here, and we need more. DMs are open if you're researching or working in this area of multi-step/multi-modal benchmarking in healthcare and life sciences! #healthcare #ai #artificialintelligence #generativeai

100

18 Comments

Here’s your recap of last week’s health IT news 🗞️ 👇 🥼AI Clinician Assistant • Suki : The AI scribe announced an SDK and APIs enabling developers to integrate ambient documentation into their applications. • Apricot: The gen AI pre-seed startup focused on reducing documentation overhead for home health nurses launched. • Tampa General Hospital + Palantir Technologies: The health system and the software platform are partnering to develop an AI care coordination and decision support platform. ☎️ AI Contact Center • Mediktor + Sensely: The AI symptom triage solution acquired the conversational member engagement platform. 📋  Patient Administration • Oracle (Cerner) + Loyal: The EHR now offers integration with the patient engagement solution through its Healthcare Marketplace. • HubSpot: The CRM and marketing platform announced it will now offer HIPAA support and sensitive data tools. 🏥 Provider and Practice Administration • LeanTaaS: The AI-enabled facility management solution released updates including: mobile-first live view of OR schedule, predictive staffing automation, and a gen AI solution allowing decision-makers to glean staffing and scheduling insights. ⚕️ Clinical Operations • Keragon: The HIPAA-compliant automation platform launched out of stealth with a $3M funding round. 🧑‍💻 Virtual and At-Home Care • AvaSure + CLEW: The virtual care platform is partnering with the AI-powered remote patient monitoring and analytics company to improve patient outcomes. 🍎 VBC • Evolent + Machinify, Inc.: The VBC org acquired AI Authorization assets from the AI operating system platform.    • ProgenyHealth, LLC + Wildflower Health: The maternity and NICU case management solution is partnering with the women’s health VBC enabler to “help health plans fill gaps, collapse silos and integrate into existing clinical and operational workflows.”    • Health Catalyst + Carevīve: The population health data and analytics company has completed acquisition of the cancer care management solution. 🧠 Food for Thought: • This article provided an interesting look at Nebraska Medicine’s efforts to design a next-generation hospital room, including the four technologies they’ve selected to anchor their work. https://lnkd.in/em-u-a5v    • McKinsey released a survey of 200 global health system executives on their investment priorities. Among the notable findings, 88% listed AI as an area of high potential impact, yet only 40% have implemented any solution thus far and over 20% don’t have plans to invest in the next two years. https://lnkd.in/eK3CeqQ2

38

4 Comments

Enjoyed giving this presentation today for Open Data Science Conference (ODSC) on Developing Equitable #AI Diagnostics. #ODSC2024 My slides can be found here: https://lnkd.in/gAbT4Yve

7

1 Comment

There is no better place than Cambridge to scale a company like Qureight This city (that we call home) gives us access to everything we need to supercharge the growth of the business and compete on a global scale - from world-class talent to specialist investors and a dynamic life science ecosystem. Did you know that: 💸The Cambridge tech ecosystem has a combined value of €176bn, 18% of the entire UK’s tech ecosystem 🌡️ in the first half of this year, Cambridge tech companies have raised a record-breaking $1.6bn 🇬🇧Cambridge companies produce 16.9 times in value for every dollar of VC investment, while the UK and London produce 5.4 times and 4.8 times, respectively. It is a honour to be profiled in Silicon Republic, alongside some of the city’s top startups, in this list of Cambridge’s largest tech funding rounds of 2024 Congratulations to our Cambridge neighbours who also made the list Quantinuum Luminance Mission Therapeutics Sano Genetics PlaqueTec Ltd. Monumo Biofidelity Xampla https://lnkd.in/eaaG7THT

98

4 Comments

The 2024 HealthIMPACT Forum is just around the corner, and one session that we're particularly excited about is "Now That We Know that Patients Want AI – Who Should Use What and When – Establishing an AI Governance Structure." This session will feature a distinguished panel of speakers: 🌟 Chris Gibbons, MD, MPH - Founder & CEO, The Greystone Group 🌟 Mike Mosquito, CHCIO, MBA, PMP, CDH-E - Emerging Technology & Innovation - Special Projects, Northeast Georgia Health System 🌟 Shelly Nash, DO - SVP, Global Chief Medical Information Officer, Fresenius Medical Care 🌟 Shahid Shah - Chairperson, HealthIMPACT Live Forum Healthcare's risk-averse nature necessitates a comprehensive AI governance framework to ensure ethical, equitable, and safe AI deployment. How do we vet AI technologies effectively across ethics, access, equity, and patient safety? What processes can help us evaluate and prioritize AI solutions that align with our strategic goals? We're eager to hear from experts on how to establish an AI governance committee with diverse stakeholders to oversee AI technologies, ensuring transparency and continuous improvement. What are your thoughts on creating a robust AI governance structure in healthcare? 📌 Check out more exciting sessions from great industry leaders here: https://lnkd.in/gJ-Ekady #HealthIMPACTForum #HealthcareInnovation #PatientCareTransformation #NYCEvents #Healthcarechangemakers #HealthIMPACT #Healthtech #AIinhealthcare #AIGovernance

2

Great to see KP on the list of health systems in the US "that are pioneers in demonstrated, outcomes-based AI solutions." While so many companies are tripping over themselves in positioning themselves as "AI-first" the secret for KP's success is that we're decidedly NOT AI-first. We're a people-first organization. Our core mission is delivering high-quality, affordable and equitable care for our 12.5M members and to make KP the best place to work for our 230k+ employees. We deploy technologies to drive those outcomes and we use AI, sparingly and cautiously, when simpler technologies can't get the job done. AI is also a team-sport and KP's success in this area reflects the work of thousands of employees across the organization. People like Jennie Shin who led the national IT team deploying our ambient scribe technology to 25,000 clinicians or Anna Davis and Carol Cain who led our quality assurance testing for our roll-out. And it's particularly gratifying to be recognized alongside my close colleague and friend Vincent Liu. His leadership in AI at KP has been both sustained and truly outcomes-driven. There are many other physician leaders that also deserve to be recognized - Khang Nguyen, Kristine Lee, MD, Brian Hoberman MD MBA, Dr. Ainsley MacLean, Chris Cable, MD, Dinh Nguyen, MD, MSHI and so many more.

177

15 Comments

👉🏼 Pediatric Cancer Communication on Twitter: Natural Language Processing and Qualitative Content Analysis 🤓 Nancy Lau 👇🏻 https://lnkd.in/eMb_6py3 🔍 Focus on data insights: - Sentiment analyses revealed more positive words than negative words. - Common positive words included support, love, amazing, heaven, and happy. - Common negative words included grief, risk, hard, abuse, and miss. 💡 Main outcomes and implications: - Twitter serves as an effective platform for examining pediatric cancer communication and public sentiment globally. - Despite the challenges of the COVID-19 pandemic, sentiments in pediatric cancer-related tweets were predominantly positive. - Tweets focused on health and treatment information, social support, and raising awareness of pediatric cancer. 📚 Field significance: - Understanding sentiment and content of pediatric cancer-related tweets can provide valuable insights for researchers and healthcare professionals. - Social media platforms like Twitter play a crucial role in disseminating health information and fostering support networks for pediatric cancer patients and their caregivers. 🗄️: [#pediatriccancer #Twitter #sentimentanalysis #healthcommunication #socialmedia]

1

Takeaways from week two Stanford Med216: Generative AI and Medicine - “Be part of the wave of innovation, don’t let it crash on you” with Dr. John Hennessy - Alphabet Chairman and former President Stanford University. Pulling out a key thread from Dr. Hennessy’s session was his take on technology coming in waves of innovation: • We are in the AI wave of innovation - we have seen previous waves of innovation with the internet and with mobile. This is the wave to learn about, contribute to, and be a part of. • What’s different with AI innovation? - The speed of adoption and rate of change in this wave is unprecedented. Previously, we have seen technology take years for adoption and for its impact to be seen. We are going to see changes much faster with AI and need to react specifically around impacts to education and jobs.  • Role of education - Undergrad Education – Focusing on creating lifelong learners - Graduate Education - learning enough to be fearless to push forward when a new area comes along. These goals are not different in the AI wave of innovation - now we just have more people studying linear algebra.   • Regulatory environment is very tricky - the technology can be used for good or bad, and just because it can be used for bad does not mean we outright ban it. There are no easy answers to the copyright/usage of content. If we over-regulate we will push innovation elsewhere.  • What’s coming down the pipe?  We aren’t close to AGI. The improvements seen in the past few years have certainly shifted the timeline closer than most thought, but we are still years away. The Transformer architecture was a big step forward, but this isn't the end of innovation, we will continue to improve how we are building AI. What’s exciting in healthcare? • Patient interfacing - explaining medical concepts and treatments in ways that patients can fully understand. • Diagnostic error reduction – AI should be able to reduce errors in diagnosis and reduce the time it takes to reach the correct diagnosis. • Research and drug discovery will be made much more efficient with AI. • Overall with AI patients will take a larger role in their own healthcare journey. Quick hit questions: •Contrarian truth - "Individuals can change the world." •Key advice - "Remain intellectually curious for your entire life." Thanks to Longsha Liu and Bryant Lin my co-instructors for making this course possible. #genai #lifelonglearning #futureofhealth #AI (image created with Dalle)

86

4 Comments