Draft Status

Draft - team will hold off on page creation

Category

Infrastructure

Key Investigators

• Paul Dumont (University of North Carolina at Chapel Hill, USA)
• Alexandre Buisson (University of North Carolina at Chapel Hill, USA)
• Juan Carlos Prieto (University of North Carolina at Chapel Hill, USA)
• Lucia Cevidanes (University of North Carolina at Chapel Hill, USA)
• Steve Pieper (Isomics, USA)

Project Description

Problem: The Slicer Automated Dental Tools extension provides robust craniofacial analysis, but complex module selection and parameter tuning create a steep learning curve for users.

Solution: This project introduces an AI Agent and chatbot UI integrated into 3D Slicer to streamline the workflow. By allowing drag-and-drop inputs and natural language prompts, users can easily request complex tasks (e.g., segmentation, landmarking, or orientation on CBCT/IOS). The agent autonomously translates these requests into actions, selecting the right tools and automatically configuring the parameters to execute the workflow.

Objective

• Local LLM Integration: Uses Ollama to run a lightweight local model (currently llama3:latest).
• Cross-Encoder Retrieval: the system leverages a Cross-Encoder. It directly scores the semantic relevance between the user's query and tool use cases to retrieve the top 3 most appropriate modules.
• Autonomous Execution: The LLM analyzes the retrieved context, selects the optimal tool, extracts the required parameters from the user's text, and automatically triggers the execution within Slicer.

Approach and Plan

• Build the core backend pipeline integrating the Cross-Encoder retrieval model and the local LLM.
• Deploy the pipeline within 3D Slicer and connect it to the user-facing chatbot UI.
• Implement an autonomous feedback loop. The model will verify if the selected Slicer tools executed successfully or encountered an error, and provide real-time feedback to the user.

Progress and Next Steps

Current Progress
The backend retrieval system is completely operational. The Cross-Encoder model reliably identifies and selects the appropriate Slicer tool from natural language input.

Next Steps

• LLM Integration: Implement the logic for the local LLM to parse the user's prompt, auto-fill the required parameters, and trigger the tool execution.

• Slicer Deployment: Embed the interactive UI and connect the entire AI pipeline directly within the 3D Slicer environment.

Illustrations

What would the Slicer user interface look like?

Slicer Automated Dental Tools Overview :

Background and References

SlicerAutomatedDentalTools: https://github.com/DCBIA-OrthoLab/SlicerAutomatedDentalTools