Evaluating Large Language Models in Medicine

Large language models such as ChatGPT, Claude, and Llama have exploded in popularity over the past year and are being applied to many different fields. After commercial models such as ChatGPT were released, open source models began to emerge such as Meta's Llama model and its derivatives. These open source models have the potential to be useful in privacy-preserving medical applications such as talking with patients directly without sending sensitive patient data to external corporations.

Thomas and Harry are two junior doctors and UWA Computer Science graduates performing research with Professor Britta von Ungern-Sternberg at Perth Children's Hospital. Previously they have published research on using SMS to contact parents after their children have surgery. Now, they are looking at using large language models to perform similar tasks in the medical field. To facilitate this the Pawsey Supercomputing Centre is offering supercomputer time to use towards this purpose.

This project will involve building a web application to evaluate and compare large language models. This may look like a leaderboard combined with some sample prompts and answers for the task of asking parents to describe pain their child is experiencing. The leaderboard may range from simple qualitative evaluation to more complex depending on student interest. ChatGPT API credits will be available to run models. You will also be allocated time on the Pawsey supercomputer to assist with hardware requirements to run large language models. The final website may be used at Perth Children's Hospital in the Paediatric Anaesthesia research group as well as in research publications.

Feel free to contact us for more information.

Client


Contact: Thomas Drake-Brockman, Harry Smallbone
Phone: 0413243360
Email: [email protected] [email protected]
Preferred contact: Phone,Email,SMS
Location: Perth Children's Hospital

IP Exploitation Model


The IP exploitation model requested by the Client is: An agreement to joint exploitation of any IP that is created


Department of Computer Science & Software Engineering
The University of Western Australia
Last modified: 22 July 2023
Modified By: Michael Wise 		UWA