AMIA KDDM Collaborative Workshop on
Open-Source Large Language Models for Health Applications:
Performance, Multimodality, and Real-World Implementations

Introduction

The rapid advancement of Large Language Models (LLMs) has transformed numerous domains, including health informatics, clinical decision support, and biomedical research. While proprietary LLMs such as GPT-4, Claude, and Gemini have demonstrated impressive capabilities, they remain closed-source, limiting transparency, reproducibility, and adaptability for healthcare-specific needs. Open-source LLMs, such as LLaMA 3, DeepSeek, and Mistral, offer a promising alternative by enabling researchers and practitioners to develop, fine-tune, and deploy models that are tailored to domain-specific requirements while maintaining greater control over data privacy, compliance, and bias mitigation. However, the practical adoption of open-source LLMs in healthcare presents unique challenges, including performance trade-offs, multimodal integration, computational efficiency, and regulatory considerations. As healthcare applications increasingly rely on multimodal AI - combining text, imaging, electronic health records (EHRs), and genomics - the need for interpretable, transparent, and efficient models has never been greater. Additionally, ensuring HIPAA-compliant deployment and understanding the limitations of LLM-based solutions remain critical hurdles for widespread adoption.

This collaborative workshop, organized by the AMIA Knowledge Discovery and Data Mining (KDDM), Natural Language Processing (NLP), and Knowledge Representation & Semantics (KRS) Working Groups, provides a deep dive into the state-of-the-art open-source LLMs for healthcare applications. We will explore:

• Comparative performance analysis of open-source vs. closed-source LLMs for medical tasks.
• Multimodal LLMs, incorporating text, medical imaging, and structured EHR data.
• Interpretability, reproducibility, and trustworthiness in LLM-powered healthcare tools.
• Regulatory and ethical considerations, including HIPAA compliance and data privacy.
• Hands-on tutorials featuring LangChain and LangGraph for developing LLM-based AI agents.
• Real-world implementation case studies from academic and industry leaders.

Through keynote talks, panel discussions, technical tutorials, and live demonstrations, this workshop will engage researchers, clinicians, informaticians, and AI practitioners in a collaborative exploration of how open-source LLMs can drive innovation in health applications while addressing challenges of performance, compliance, and scalability.

By the end of the workshop, participants will have a deeper understanding of open-source LLM trade-offs, hands-on experience with implementation frameworks, and insights into future directions for multimodal AI in healthcare.

Objective

Our intended audience will include researchers, clinicians, data scientists, and informaticians interested in leveraging open-source LLMs for healthcare applications, including those working on natural language processing (NLP), multimodal AI, clinical decision support, and AI model compliance. We anticipate attendees to have expertise levels of 25% novice, 50% intermediate, and 25% advanced.

By the end of the workshop, participants will:

• Understand the trade-offs between open-source and closed-source LLMs in health applications.
• Explore multimodal LLMs trained for text, imaging, and structured health data.
• Learn about HIPAA compliance and regulatory considerations when deploying LLMs in clinical environments.
• Gain hands-on experience in building and deploying open-source LLM applications using LangChain and LangGraph.
• Hear from experts about challenges and real-world implementation of LLMs in healthcare research.
• Participate in interactive demos showcasing LLM-powered healthcare tools.
• Engage in discussions about open-source model interpretability, reproducibility, and deployment challenges.

Submissions

Topics of Interest

Submissions should be relevant to the workshop themes, including but not limited to:

• Applications of open-source LLMs (e.g., LLaMA 3, DeepSeek, Mistral) in healthcare
• Multimodal LLMs for integrating text, imaging, EHRs, and genomics
• Comparative performance of open-source vs. closed-source models
• Interpretability, reproducibility, and trustworthiness of LLMs
• HIPAA-compliant and ethical LLM deployment
• Real-world case studies and implementation experiences
• Tutorials and demos using platforms like LangChain and LangGraph

Submission Instructions

Please use the AMIA 2025 Submission Template (2-page maximum).

Indicate your preferred presentation type at the top of your submission:

• Oral Research Talk
• Oral Demo Talk
• Poster Presentation

While we will consider your preference, final presentation format will be determined based on the review and program balance.

Important Dates

• Submission Deadline: September 12, 2025
• Notification of Acceptance: October 10, 2025
• Workshop: 8:30 AM – 12 PM EST, November 15, 2025

To submit your work, please prepare a PDF and use the Submit Now button below to upload it.

Submit Now