You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Star1 (1)You must be signed in to star a repository
About
Research on evaluating automated scientific claim verification frameworks, focusing on the MultiVerS model's performance against disinformation. Includes dataset creation, evaluation metrics, and analysis of results in detecting scientific misinformation. Published by ACM.
An Assessment of Scientific Claim Verification Frameworks
This research has been published by the Association for Computing Machinery (ACM). Click here to view the published paper.
Click here to access my final, published slide deck that I presented for this project.
Introduction
Objective: To evaluate automated scientific claim verification in response to the growing spread of scientific disinformation.
Key Models: Evaluated the state-of-the-art model, MultiVerS, trained on the SciFact dataset.
Scientific Questions:
How well does the SciFact-trained model detect open-domain scientific disinformation?
How does the model perform when trained on other datasets for the same purpose?
Research Goals
Replicate MultiVerS Results:
Successfully replicated the results of the SciFact leaderboard model, MultiVerS.
Create an Open-Domain Scientific Claims Dataset:
Focused on COVID-19 misinformation.
Dataset used for testing the model's generalizability.
Labeled to assess the model's ability to verify claim accuracy and identify supporting or conflicting sentences.
Analysis of Results:
Utilized precision, recall, and F1 metrics to evaluate the model's performance on different checkpoints.
How MultiVerS Works
Model Components:
Utilized the Long-former model (Beltagy et al., 2020) for encoding claims and context.
Multitasked rationale selection and label prediction using Vert5Erini (Pradeep et al., 2021).
Training Datasets:
Employed datasets such as SciFact, COVIDFact, HealthVer, and Fever for training.
Procedures
New Dataset Creation:
Used claims from Snopes.com.
61 instances, each containing a claim and at least one relevant paper (DOI).
Open domain, with 18 true claims, 40 false claims, and 3 mixture claims.
Preprocessing:
Tokenized sentences in paper abstracts using NLTK.
Generated compatible files in JSON format.
Labeling:
Double independent labeling (with another undergraduate student: Dominik Soos).
Achieved a consensus rate of 75%.
Evaluation Metrics
Precision, Recall, and F1 Scores:
Abst (Abstract) and Sent (Sentence) levels.
Assessed accuracy of predicted labels on both levels.
Results
Performance Overview:
All versions of MultiVerS performed poorly in detecting disinformation within the new dataset.
Checkpoint trained on CovidFact achieved the best abstract_label_only.
Sensitivity to data domains highlighted.
Future Work:
Identified areas for improvement, emphasizing the model's current reliance on text semantics without background knowledge and inference.
About
Research on evaluating automated scientific claim verification frameworks, focusing on the MultiVerS model's performance against disinformation. Includes dataset creation, evaluation metrics, and analysis of results in detecting scientific misinformation. Published by ACM.