Authors:
Maren Bormann, Aylin Hanne, Parya Tavaloki-Tehrani, Katharina Baumgartner
This project began as part of our Capstone Phase at neuefische GmbH. Our idea was to train an NLP Model to detect logical fallacies in sentences and build a web application where users can enter their own sentences.
Logical fallacies are a common phenomenon and are also associated with the spread of disinformation. A logical fallacy is a shortcut that sounds clever but doesn't really make sense when you look closer. It is an error in reasoning that weakens an argument. Being aware of logical fallacies is a good idea, but sometimes tricky. Even for AI! The AI needs to understand the underlying logical structure of the argument.
We created a web application that can detect 5 common logical fallacies in a sentence. It also explains the different fallacies and provides a small game to beat the AI at recognising logical fallacies.
We have merged our dataset from the following datasets:
-
Jin et al. (2022). Dataset: LogicClimate
-
Midhun Kandan. Logical Fallacy Classification. 2004. Dataset: Hugging Face Dataset
-
Yeh et al. (2024). Dataset: CoCoLoFa
-
Chaves et al. (2025). Dataset: FALCON
-
Alhindi et al. (2022). Dataset: Datasets
-
Helwe et al. (2024). Dataset: MAFALDA
-
Goffredo et al. (2023). Dataset: ElecDeb60to20
After cleaning the data, we trained our model on the five most common fallacies present in the dataset, as well as on sentences that do not contain any logical fallacies.
Categories used for training:
| Category | Definition |
|---|---|
| Ad Hominem | This fallacy occurs when the speaker is attacking the other person or some aspect of them rather than addressing the argument itself. |
| Appeal to Authority | This fallacy occurs when an argument relies on the opinion or endorsement of an authority figure who may not have relevant expertise or whose expertise is questionable. |
| Appeal to Emotion | This fallacy occurs when an emotion is used to support an argument, such as pity, fear, anger, etc.. |
| False Dilemma | This fallacy occurs when only two options are presented in an argument, even though more options may exist. A case of "either this or that". |
| None | There are no fallacies in the text. |
| Slippery Slope | This fallacy occurs when someone argues that a event will trigger a series of events that lead to an extreme outcome without demonstrating any causal connections between the events. |
Please note: The definitions of the fallacies were taken from the research papers cited above or adjusted by us as needed.
After training and fine-tuning different models, we got the following results:
| Model | Macro F1 |
|---|---|
| Logistic Regression | 0.65 |
| DeepSeek | 0.53 |
| DistilBERT | 0.71 |
| Deberta | 0.76 |
Our model used in the webapp is Deberta, you can find here.
The Webapp has been build using Flask and Vue.js. It only runs locally at the moment.
- backend
- additional: additional files, which where not used in the final product but may be useful, e. g. build pipelines with different models
- data: folder created to store the datasets, which will be created in the eda notebooks
- eda: cleaning different datasets, combining datasets and creating final datasets to train, includes also notebooks for plots and wordclouds
- modeling: main folder with files for different models: training, testing, evaluating
- model: folder to store models locally
- app.py: main flask file, which handels the incoming requestes, predicts the inputs and send the result to the frontend
- frontend: Vue.js App
- research: Notes about the different models we used
in your backend folder, in different terminal tabs
1. Flask App:
source .venv/bin/activate
python app.py # should run on http://127.0.0.1:50022. for training: MLFlow:
source .venv/bin/activate
mlflow ui --port 5001 #should run on http://127.0.0.1:5001in your frontend folder, in a new terminal tab
1. VueApp
npm run dev # app should run on http://localhost:5173/