XAIR: Explainable Automated Program Repair

Using Deep Learning and Explainable AI

Techniques

Explainable AI (XAI): Explained

Published in: 2023 IEEE Open Conference of Electrical, Electronic and
Information Sciences (eStream)

The abstract provides a detailed summary of the paper's content, including an overview of
XAI, specific techniques like LIME and SHAP, and their applications across various domains
such as healthcare, finance, and law.
The paper addresses a current and pressing issue in AI, which is the need for explainability
in complex AI models. This is particularly relevant given the growing use of AI in high-stakes
domains.
The inclusion of ethical and legal implications highlights the broader impact of XAI and the
necessity for responsible AI deployment.

The use of technical terms like "Local Interpretable Model-Agnostic Explanations" (LIME)
and "SHapley Additive exPlanations" (SHAP) might be difficult for readers unfamiliar with
XAI.
The abstract mentions that few review papers are available but does not clearly state what
new insights or unique contributions this paper offers compared to existing literature.

Current Trends, Challenges and

Techniques in XAI Field; A Tertiary
Study of XAI Research
Published in: 2024 47th MIPRO ICT and Electronics Convention (MIPRO)

Increased Adoption Across Industries: XAI is being integrated into various high-stakes
domains such as healthcare, finance, law, and autonomous vehicles. The need for
transparency and trust in AI decision-making is driving this trend.
Regulatory Push: There is a growing emphasis on AI explainability due to regulatory
requirements. Governments and organizations are pushing for AI systems that can be
audited and understood by non-experts to ensure fairness and accountability.
Trade-off Between Performance and Explainability: High-performing models like deep
neural networks are often complex and less interpretable. Striking a balance between model
accuracy and interpretability remains a major challenge.
Scalability: Many XAI methods are computationally intensive and may not scale well with
large datasets or complex models, limiting their practical applicability.

Bayesian XAI Methods Towards a

Robustness-Centric Approach to Deep
Learning: An ABIDE I Study
The integration of Bayesian Neural Networks (BNNs) with Explainable AI (XAI) methods
represents a novel approach in the diagnosis of Autism Spectrum Disorder (ASD),
showcasing the potential for advancements in both model interpretability and reliability.
By using Layerwise Relevance Propagation (LRP), the study emphasizes the importance of
understanding model predictions, which is crucial in high-stakes fields like healthcare.
The combination of BNNs and LRP provides a robustness-centric deep learning approach,
enhancing the reliability of the model's predictions by quantifying epistemic uncertainty.

The use of BNNs and the repeated inference required for uncertainty estimation can be
computationally intensive, which may limit the practical application of this approach in
real-time scenarios.The reliance on the ABIDE dataset, while comprehensive, may not fully
capture the diversity of ASD presentations, potentially limiting the model's applicability to
broader populations.

Automated Program Repair for

Introductory Programming
Assignments
Novel approach: Proposes CEMR, a new automated program repair tool that combines
learning from existing code modifications with a large language model
(CodeBERT).Comprehensive evaluation: Tested on both open online judge platform (LuoGu)
and real classroom datasets, comparing against multiple baselines.
Strong performance: Achieves higher repair rates compared to baseline methods, especially
for semantic and logical errors.
Efficiency: Repairs incorrect programs in about half the time compared to AlphaRepair.

Limited scope: Only tested on introductory Python programming problems, may not
generalize to more complex programs or other languages.
Inability to fix syntactical errors: Unlike AlphaRepair, CEMR cannot repair programs with
syntax errors due to its reliance on ASTs.
Dependence on existing solutions: As a data-driven approach, CEMR may struggle with
novel or uncommon problem-solving approaches not present in the training data.

Towards JavaScript program repair

with Generative Pre-trained
Transformer (GPT-2)
Novel use of GPT-2 model for automated program repair (APR), which had not been done
before according to the authors.
Focus on JavaScript, which is an extremely popular programming language but
underrepresented in APR research.
Able to generate syntactically correct source code in most attempts.
Achieved an overall accuracy of up to 17.25% in generating correct fixes.
Created and used a large dataset of 16,863 JavaScript code snippets for training.

Failed to learn good bug-fixes in some cases, indicating inconsistent performance.

17.25% accuracy, while promising, still leaves significant room for improvement.
Limited to fixing single-line bugs only, not more complex multi-line issues.
Approach may be computationally intensive, given the size of the GPT-2 model.
Potential for data leakage or overfitting, as the model needs to be trained on project-specific
data to accurately predict variable names.
Lack of comparison to state-of-the-art APR techniques, making it difficult to assess relative
performance.

An Evaluation of the Effectiveness of

OpenAI's ChatGPT for Automated
Python Program Bug Fixing using
QuixBugs
Uses a state-of-the-art language model (GPT-3.5) for automated bug fixing in Python code.

Evaluates the effectiveness using an established benchmark (QuixBugs), allowing for

comparison with other methods.

Demonstrates high accuracy, successfully fixing 30 out of 40 bugs from the QuixBugs
benchmark.
Outperforms other tools like standard program repair and Codex in bug-fixing capability.

Highlights the potential of ChatGPT as a powerful tool for enhancing code quality and
reducing manual bug-fixing efforts.

Limited scope - only tested on 40 Python bugs from a single benchmark suite.

Lack of details on the specific types of bugs that were fixed or not fixed.

No mention of the time or computational resources required for the bug-fixing process.

Doesn't address potential limitations or challenges of using ChatGPT for this task.

Doesn't discuss how the approach might generalize to more complex or real-world coding
scenarios beyond the benchmark.

No information on false positives or potential introduction of new bugs during the fixing
process.

DeepRepair: Style-Guided Repairing

for Deep Neural Networks in the
Real-World Operational Environment
Addresses an important real-world problem - repairing deployed deep neural networks
(DNNs) that fail due to mismatches between training and operational environments.

Proposes a novel approach using style-guided data augmentation to repair DNNs.

Introduces clustering-based failure data generation to improve the effectiveness of the

augmentation.

Conducts large-scale evaluation across 15 different degradation factors/failure patterns.

Demonstrates significant accuracy improvements (62.88% for CNNs, 39.02% for RNNs on
average) compared to state-of-the-art methods.

Shows the repaired DNNs maintain or even improve accuracy on clean data.
Limited to image classification tasks - may not generalize to other types of DNN applications.

Requires collecting some failure examples from the operational environment - may be
challenging in some real-world scenarios.

Focuses only on naturally occurring degradations/noise - does not address adversarial

attacks or malicious perturbations.

Evaluation is limited to CIFAR-10 dataset - more diverse datasets could strengthen the
results.

Does not provide theoretical guarantees on the effectiveness or generalizability of the

approach.

Generative AI for Self-Healing Systems

Clear problem statement: The abstract effectively identifies the risk of component failures in
large-scale system production and the current reliance on human experts for system
monitoring.

Innovative approach: It proposes integrating generative AI technology into self-healing

systems, which is a novel and potentially impactful solution.

Specific focus areas: The abstract outlines clear areas of application for generative AI,
including anomaly detection, code generation, debugging, and auto-generative reporting.

Practical application: The study aims to optimize system functionality and efficiency at scale,
which has real-world implications for large-scale systems.

Comprehensive solution: The proposed approach covers multiple aspects of system

maintenance, from detection to repair and reporting.

Lack of quantitative goals: The abstract doesn't provide specific, measurable objectives for
improvement over current methods.

Limited discussion of challenges: It doesn't address potential challenges or limitations of

integrating generative AI into self-healing systems.
Absence of methodology details: The abstract doesn't provide an overview of the research
methodology or experimental setup.

No mention of comparative analysis: There's no indication of how the proposed solution

compares to existing self-healing systems or other AI-based approaches.

Vague on implementation details: While it mentions using GPT-4 for code completion, it
doesn't provide specifics on how other aspects of the generative AI integration will be
implemented.

Examining Zero-Shot Vulnerability

Repair with Large Language Models

Novel application of large language models (LLMs) to security bug repair, exploring an
important and timely research question.

Comprehensive evaluation using multiple types of scenarios: synthetic, hand-crafted, and

real-world security bugs.

Examination of multiple commercial and open-source LLMs, providing a broad comparison.

Exploration of prompt engineering techniques to improve LLM performance on this task.

Demonstrates some promise, with LLMs collectively able to repair 100% of synthetic and
hand-crafted scenarios.

Limited to zero-shot performance, without fine-tuning LLMs specifically for this task.

Challenges identified in generating functionally correct code for real-world examples.

Focused only on vulnerabilities that can be fixed with localized changes in a single file.

Reliance on existing test suites and security tools to validate fixes, which may miss some
issues.

Does not fully solve the problem of automatic security bug repair, but provides initial
characterization of LLM capabilities in this domain.