ChatGPT Needs a Chemistry Tutor Too

Abstract

AI and ChatGPT technology have the potential to revolutionize the education sector, and this study aimed to evaluate if prompt formats, response consistency, and reliability of ChatGPT responses could help colleges make the most of this technology. The results of this study can guide future AI and ChatGPT implementations and ensure they are used to their fullest potential. The data does not demonstrate a statistically significant difference between multiple-choice and free response prompt formats. Neither format achieved scores higher than 37%, and testing different locations did not improve scores. Interestingly, ChatGPT's free version provides accurate responses to discipline-specific questions that contain information from unrelated topics, improving its accuracy over the free response questions. However, it's important to remember that while ChatGPT can identify the correct answer within a given context, it may not be able to tell if the answer it chooses is correct computationally or through analysis.

Cite: Leon, A., & Vidhani, D. (2023). ChatGPT Needs a Chemistry Tutor Too. ChemRxiv. https://doi.org/10.26434/chemrxiv-2023-qpzx3