Human Healthcare Pipeline

The research aligns with human healthcare pipeline, including the clinical diagnosis, surgery intervention and medical foundation models. With the multimodal medical data from patient examinations, our research can assist clinicians with diagnosis and assist surgeons with surgery. In particular, our research on multimodal medical foundation models can further support the entire healthcare pipeline.

Multimodal Clinical Diagnosis

Our research targets at the multimodal clinical diagnosis through AI training pipeline. The research focuses on addressing significant clinical challenges caused by imperfect medical data, including the low-quality data, modality-missing data, decentralized data, and coarse/noisy annotations. The research has been verified on diverse modalities, such as endoscopy, dermoscopy, pathology, radiology, ophthalmology, genetics, etc.

Selected Publications

• [MedIA'23] Medical Federated Learning with Joint Graph Purification for Noisy Label Learning [Paper] [Code]
• [TMI'22] Personalized Retrogress-Resilient Federated Learning Towards Imbalanced Medical Data [Paper] [Code]
• [MICCAI'22] Discrepancy and Gradient-guided Multi-modal Knowledge Distillation for Pathological Glioma Grading (MICCAI Young Scientist Award, Oral) [Paper] [Code]
• [AAAI'21] Diagnose Like A Pathologist: Weakly-Supervised Pathologist-Tree Network for Slide-Level Immunohistochemical Scoring [Paper] [Code]
• [TMI'21] Super-Resolution Enhanced Medical Image Diagnosis with Sample Affinity Interaction [Paper] [Code]

Surgery Intervention Assistance

We have developed a surgeon-centric Surgery Copilot system to provide comprehensive assistance throughout the preoperative, intraoperative and postoperative processes. By understanding the surgeon's intention through voice commands, our Surgery Copilot system can automatically execute various advanced surgical understanding algorithms, and further support surgical pipeline automation and embodied ultrasound scanning. The research enhances precision, efficiency, and safety in surgical intervention.

Selected Publications

• [MICCAI'24] Transforming Surgical Interventions with Embodied Intelligence for Ultrasound Robotics (Oral)
  [Paper] [Code]
• [IROS'24] ASI-Seg: Audio-Driven Surgical Instrument Segmentation with Surgeon Intention Understanding (Oral) [Paper] [Code]
• [BigData'24] SurgBox: Agent-Driven Operating Room Sandbox with Surgery Copilot [Paper] [Code] [AI Scientist Blog]
• [BIBM'24] SurgFC: Multimodal Surgical Function Calling Framework on the Demand of Surgeons [Paper] [Code]
• [MICCAI'23] Surgical Video Captioning with Mutual-Modal Concept Alignment (Early accepted) [Paper] [Code]

Medical Foundation Models

Beyond the diagnosis and surgery, our research is exploring on medical foundation models to support more stages in human healthcare pipeline. Namely, these stages include triage, imaging, diagnosis, treatment, surgery, and rehabilitation, covering the entire spectrum of patient care. By driving the evolution of medical foundation models across these stages, our goal is to create a more seamless, efficient, and patient-centric healthcare system.

Selected Publications

• [NeurIPS'24] Flaws can be Applause: Unleashing Potential of Segmenting Ambiguous Objects in SAM [Paper] [Code]
• [JBHI'24] Unified Multi-modal Diagnostic Framework with Reconstruction Pre-training and Heterogeneity-combat Tuning [Paper] [Code]
• [MICCAI'24] EndoUIC: Promptable Diffusion Transformer for Unified Illumination Correction in Capsule Endoscopy [Paper] [Code]
• [AAAI'25] U-KAN Makes Strong Backbone for Medical Image Segmentation and Generation [Paper] [Code]
• More research coming soon ...