A study conducted by Carmen Atkins, Gina Girgente, Manoochehr Shirzaei & Junghwan Kim published in Communications Earth & Environment evaluated the accuracy and reliability of ChatGPT in identifying climate change-related hazards to enhance climate literacy and inform the responsible use of AI in educational contexts.
The discussion and results of the study focused on evaluating the accuracy of ChatGPT in identifying climate change-related hazards, comparing these results with credible indices from the International Panel on Climate Change (IPCC). This evaluation was centred around three major hazards: floods, droughts, and cyclones.
The study found that ChatGPT, especially the GPT-4 version, showed a relatively high accuracy in identifying floods and cyclones. The accuracy for cyclones was noted to be around 80.6%, and for floods, it was slightly lower at 76.4%. However, the tool performed less effectively in recognising droughts, with an accuracy of only 69.1%. These figures were drawn from confusion matrices, which detailed the counts of true positives, false negatives, and false positives for each hazard.
Further, when assessing the consistency of ChatGPT’s responses across multiple iterations, the study noted minimal variation in the accuracy of the tool's hazard identification, suggesting a reliable performance in the case of floods and cyclones. However, for droughts, the consistency in accuracy was not as stable, indicating potential areas for improvement in the AI model's learning and response generation.
The authors speculate that inaccuracies might stem from several sources, including language biases since the study and the AI's training predominantly involve English. This might limit the AI's effectiveness in regions with non-English languages or diverse dialectical variations. The inherent complexity and variability in defining and understanding droughts compared to more identifiable hazards like cyclones might contribute to lower accuracy rates.
Despite some inaccuracies, the authors posit that ChatGPT can still be a valuable tool for enhancing climate literacy, particularly for more reliably identified hazards like floods and cyclones. However, caution is advised when using the tool for educational purposes regarding droughts, where the information might be less accurate.
The performance difference between GPT-3.5 and GPT-4 also raises ethical questions about accessibility and the digital divide. Higher-performing, more advanced models like GPT-4 may not be as accessible to all users, particularly in less developed regions, potentially exacerbating existing inequalities in digital literacy and access to information.
This underscores the importance of ongoing validation and calibration of AI tools used in educational settings, particularly concerning critical issues like climate change.
communications earth & environment -
https://lnkd.in/eB-XNGnE
#climatecrisis #generativeai #responsibleai