AI-generated summaries
Today's ML research,
without the noise.
Daily summaries of the latest machine learning papers from arXiv, processed every 8 hours.
24
Papers today
8h
Update frequency
7
Days of history
DAMP: Decay-Aware Mixed-Precision Recurrent-State Quantization
NLP
Large Language Models
Efficient ML
- DAMP is the first to explore post-training quantization of recurrent states in GDN and KDA models.
- Uniform quantization leads to significant accuracy loss, particularly with lower bit formats.
- DAMP utilizes quantization-error energy and decay-based persistence to optimize channel precision.
- The method achieves a 69.1% reduction in recurrent-state storage and accelerates updates by up to 2.01Γ.
Read more
DAMP: Decay-Aware Mixed-Precision Recurrent-State Quantization
Summary
The paper introduces DAMP, a novel approach for post-training quantization of recurrent states in language models utilizing Gated DeltaNet (GDN) and Kimi Delta Attention (KDA). Traditional methods of storing key and value vectors for each token in softmax attention lead to significant memory consumption and increased latency during inference. DAMP addresses these issues by employing a mixed-precision quantization strategy that leverages the decay characteristics of recurrent states. The authors find that uniform quantization methods, such as INT8 and FP8, result in substantial accuracy degradation, particularly in complex reasoning tasks. By analyzing the quantization-error energy and the decay persistence of channels, DAMP intelligently allocates higher precision to channels that exhibit higher risk of error while storing the rest in INT8 format. Evaluations on Qwen3.6-35B and Kimi-Linear-48B demonstrate that DAMP achieves a storage reduction of 69.1%, accelerates the recurrent-state update process by up to 2.01 times, and maintains accuracy close to the FP32 baseline at an effective state storage of 9.9 bits.
Methodology
DAMP employs a mixed-precision quantization strategy that identifies high-risk channels based on their quantization-error energy and decay persistence. This is achieved through offline calibration, allowing for a static layout that does not require retraining or token-wise selection during inference.
Results
DAMP was evaluated on Qwen3.6-35B and Kimi-Linear-48B, achieving a storage reduction of 69.1% for recurrent states, an acceleration of the recurrent-state update kernel by up to 2.01 times, and a reduction in full-model time per output token (TPOT) by up to 10.9%, while maintaining accuracy close to the FP32 baseline at 9.9 bits per state value.
Implications
The findings suggest that DAMP can significantly enhance the efficiency of language models in terms of memory usage and processing speed, making it suitable for applications requiring complex reasoning and long context handling in real-time scenarios.
REPLICANT: Learning Policies for Evading and Hardening Malware Detectors
Reinforcement Learning
- REPLICANT achieves a mean attack success rate of 78.8%, outperforming existing methods.
- The framework facilitates policy transfer, yielding an 82.0% relative increase in ASR.
- Adversarial training with REPLICANT significantly reduces attacker success rates.
- The study conducts the largest evaluation of Android malware evasion to date, with 379,680 attack evaluations.
Read more
REPLICANT: Learning Policies for Evading and Hardening Malware Detectors
Summary
The paper introduces REPLICANT, a deep reinforcement learning (DRL) framework designed to enhance the evasion of malware detectors under a strict label-only black-box threat model. Unlike previous methods that often rely on privileged information, REPLICANT learns to modify malware samples and determine optimal querying strategies without such access. The framework is evaluated across seven Android malware detectors and three feature spaces, achieving a mean attack success rate (ASR) of 78.8%, representing a significant improvement of 20.9% to 39.2% over existing state-of-the-art techniques. Additionally, when employed for adversarial training, REPLICANT demonstrates superior performance in producing more robust detectors. The study highlights the importance of learning evasion strategies not only for effective attacks but also for enhancing the resilience of malware detection systems. The authors address limitations in current evasion research, such as the lack of transferability and generalization across classifiers and feature spaces, and propose a comprehensive evaluation framework that includes the impact of temporal drift and active learning on evasion strategies.
Methodology
REPLICANT utilizes a deep reinforcement learning approach to model evasion as a sequential decision-making problem. It learns a hierarchical policy that determines how to modify malware samples and when to query detectors. The framework is agnostic to the specific capabilities used for evasion, allowing it to adapt across different malware samples and detection systems.
Results
REPLICANT achieved a mean attack success rate of 78.8% across various Android malware detectors, significantly outperforming previous methods by 20.9% to 39.2%. The policy transfer mechanism demonstrated an 82.0% increase in ASR compared to sample transfer. Adversarial training with REPLICANT reduced attacker success rates to below 17% for both REPLICANT and a competing method, while also addressing the challenges posed by evolving attacker capabilities.
Implications
The findings suggest that learning effective evasion strategies can not only enhance the performance of malware attacks but also provide valuable insights for hardening malware detectors. This dual benefit could lead to more resilient cybersecurity measures and improved defenses against sophisticated malware threats.
Generalized Context in Cross Attention for Transfer Learning of Disjoint Tabular Data
Theory
Efficient ML
- Introduces CATTLE, a method for transfer learning in disjoint tabular datasets without requiring shared features.
- Utilizes transformer projection weights to capture generalized context for cross-domain learning.
- Demonstrates superior performance over nine state-of-the-art methods in terms of AUROC and ranking.
- Addresses the unique challenges posed by heterogeneous feature types in tabular data.
Read more
Generalized Context in Cross Attention for Transfer Learning of Disjoint Tabular Data
Summary
This paper addresses the challenges of applying transfer learning to tabular data, which often lacks shared features across disparate domains. The authors propose a novel approach called Cross-domain Attention Transfer Learning (CATTLE) that utilizes generalized context learning through transformer projection weights for keys, values, and queries. Unlike traditional methods that rely on shared features, CATTLE enables knowledge transfer in a data-agnostic manner, allowing for effective learning from a single source dataset. The methodology captures generalized context rather than domain-specific information, facilitating transfer learning across disjoint tabular datasets. Experimental results demonstrate that CATTLE outperforms nine state-of-the-art baselines, achieving a statistically significant average AUROC gain of 3.7% and the best average rank of 2.9 across ten pairs of disjoint datasets. This approach highlights the potential for effective transfer learning in scenarios where traditional methods struggle due to the heterogeneous nature of tabular data.
Methodology
The proposed methodology, CATTLE, employs transformer projection weights to facilitate cross-domain attention transfer learning. It captures generalized context by engineering the projection weights for keys, values, and queries, enabling data-agnostic learning from a single source dataset. This approach contrasts with traditional methods that depend on attention-weighted representations tied to specific datasets.
Results
CATTLE was evaluated on ten pairs of disjoint source-target datasets, demonstrating a statistically significant improvement over nine baseline methods. The method achieved an average AUROC gain of 3.7% and an average rank of 2.9, indicating its effectiveness in learning generalized context for transfer learning in tabular data.
Implications
The findings suggest that CATTLE can significantly enhance transfer learning capabilities in tabular data applications, particularly in fields where data is sparse or heterogeneous, such as healthcare, finance, and government. This approach could lead to more robust models that can adapt to new domains with limited data, ultimately improving decision-making processes across various industries.
An Enclosed Mode Is a Gauge Choice: Topology Relative to Reach in Certified Code World Models
Theory
Robotics
Reinforcement Learning
- Acceptance-with-certainty by a sampling gate certifies models only up to the reachable query set, leaving errors beyond as gauge.
- The study identifies three regimes of model behavior based on the angular channel width, impacting the model's exploitability and falsifiability.
- Repair mechanisms are limited by the geometry of the omitted structure and cannot recover unreachable regions from outside evidence.
- Mitigation strategies must match the dimensionality and direction of the model's errors to be effective.
Read more
An Enclosed Mode Is a Gauge Choice: Topology Relative to Reach in Certified Code World Models
Summary
This paper explores the limitations of certified code world models (CWMs) in the context of sampling gates, particularly when dealing with topologically nontrivial regions. It introduces the concept of 'gauge' in relation to what a model can know and the errors it can incur when the omitted structure is an unreachable interior. The study establishes that acceptance-with-certainty by a sampling gate only certifies the model up to the reachable query set, leaving everything beyond it as gauge. Through a minimal 2D thrust-and-drag plant model with an annular freeze band, the paper demonstrates how varying a single parameter (an angular channel width) can transition the model through three regimes: unfalsifiable-and-harmless, falsifiable-and-costly, and instantly falsified. The findings reveal that danger is topology relative to reach, and that repair mechanisms are constrained by parameter and sensor bounds. The paper also highlights that mitigation strategies must align with the dimensionality and direction of the model's errors, emphasizing the need for tailored defenses against specific types of errors.
Methodology
The paper employs a minimal 2D thrust-and-drag plant model with an annular freeze band to analyze the effects of varying an angular channel width on the model's behavior. It uses empirical measurements and theoretical propositions to explore the relationships between topology, reachability, and model performance across different regimes.
Results
The research demonstrates that the model's behavior can be categorized into three distinct regimes based on the angular channel width. It finds that the model's exploitation collapses significantly when the channel is opened, while hidden channels maintain performance. The study also reveals that no tested model can recover the ring's topology from outside evidence, and that the effectiveness of mitigation strategies is contingent on matching the error's dimensionality.
Implications
The findings have significant implications for the design and certification of safety-critical hybrid systems, suggesting that certification processes must account for the topology of omitted structures and the limitations of reachability. This could influence future developments in certified code world models and their applications in various fields, including robotics and autonomous systems.
A Method for Layer Bit-Width Allocation in LLM Quantization via Performance Maximization Under a Quality-Degradation Constraint
Large Language Models
Efficient ML
Optimization
- Introduces a selective mixed-precision quantization method based on layer sensitivity profiles.
- Demonstrates significant performance improvements in LLM inference with minimal quality degradation.
- Identifies optimal configurations for different model subsystems (FFN, Attention, lm_head).
- Proposes additional optimizations to further accelerate model execution.
Read more
A Method for Layer Bit-Width Allocation in LLM Quantization via Performance Maximization Under a Quality-Degradation Constraint
Summary
This paper presents a novel method for layer bit-width allocation in the quantization of the Gemma-3-1B language model, focusing on maximizing performance while adhering to a specified quality-degradation constraint. The proposed approach diverges from traditional uniform quantization methods, which are often inefficient and lack performance acceleration evidence. Instead, the author leverages a layer sensitivity profile derived from previous research (SA-PTQ) and implements it using TensorRT LLM's activation pass-through mode. The method allows for individual precision determination of model layers based on their sensitivity, effectively grouping them into configurations (5+5, 10+10, all26) to optimize speedup. Experimental results indicate that the FFN 5+5 configuration with lm_head achieves an 11.0% latency reduction with minimal quality loss, while a more aggressive FFN all26 + lm_head configuration yields a 19.1% performance increase under acceptable quality loss. The paper also suggests several optimizations for further enhancing model performance, including fused INT8 attention kernels and KV-cache quantization.
Methodology
The methodology involves analyzing the sensitivity of different layers in the Gemma-3-1B model to quantization, allowing for tailored bit-width allocation. The author employs TensorRT LLM's activation pass-through mode to implement this selective quantization, measuring performance metrics such as latency and quality degradation across various configurations.
Results
The study found that the FFN 5+5 configuration with lm_head achieved an 11.0% reduction in latency with 98.90% Top-1 agreement and a +0.85% increase in perplexity. The FFN all26 + lm_head configuration provided a 19.1% performance boost under acceptable quality loss conditions. The paper also highlights the performance trade-offs for different layers, particularly noting that the Attention mechanism may incur additional latency under certain conditions.
Implications
The findings suggest that tailored quantization strategies can significantly enhance the efficiency of LLMs, making them more viable for real-time applications such as gaming and other consumer-facing technologies. The proposed optimizations could lead to broader adoption of mixed-precision techniques in various AI applications.
Initialization Is Critical: Advancing Federated Short-Term Load Forecasting under Load Heterogeneity via Model Initialization
Federated Learning
Time Series
- Identifies structured load heterogeneity among clients affecting STLF performance.
- Proposes two innovative model initialization strategies to mitigate client drift.
- Demonstrates compatibility of proposed strategies with existing FL frameworks.
- Experimental results show significant improvements in forecasting accuracy and convergence.
Read more
Initialization Is Critical: Advancing Federated Short-Term Load Forecasting under Load Heterogeneity via Model Initialization
Summary
This paper addresses the challenges of short-term load forecasting (STLF) in the context of federated learning (FL), particularly focusing on the issue of load heterogeneity among clients. The authors identify that different clients exhibit varied responses to external factors and distinct temporal load profiles, which can lead to client drift and degrade forecasting performance. To tackle this, the paper proposes two model initialization strategies: a global initialization approach that utilizes auxiliary public load data for pretraining the global model, and a local initialization strategy called SLIAvg, which allows clients to adapt their models progressively during training. These strategies aim to stabilize the training process and improve the alignment of local updates with the global model. The proposed methods are compatible with existing FL frameworks and enhance privacy preservation. Experimental results demonstrate that the new initialization strategies significantly reduce client drift, improve convergence rates, and lower forecasting errors, showcasing their effectiveness in enhancing federated STLF performance.
Methodology
The authors conducted a systematic analysis of load heterogeneity and developed two model initialization strategies: a global pretraining approach using auxiliary public load data and a sequential local initialization strategy (SLIAvg) that adapts models progressively during training. They validated these strategies through experiments on real smart-meter data using two representative forecasting architectures.
Results
The proposed initialization strategies led to a marked reduction in client drift, improved convergence behavior, and lower forecasting errors compared to traditional methods. The experiments confirmed that the new approaches effectively enhanced the overall performance of federated STLF.
Implications
The findings suggest that careful model initialization can significantly improve the performance of federated learning systems in STLF applications, making it a valuable approach for privacy-preserving forecasting in smart grids and other distributed environments.
VICT: Verifier-Instrumented Credit Tracing for Long-Horizon LLM Agent Reinforcement Learning
Reinforcement Learning
Large Language Models
- VICT introduces a new training-time interface for credit assignment in RL using verifier structures.
- The method shifts credit assignment from rollout-side inference to verifier-side tracing.
- VICT preserves original terminal rewards while enhancing credit assignment accuracy.
- Experiments show substantial performance improvements on long-horizon tasks.
Read more
VICT: Verifier-Instrumented Credit Tracing for Long-Horizon LLM Agent Reinforcement Learning
Summary
The paper addresses the challenge of fine-grained credit assignment in reinforcement learning (RL) for long-horizon large language model (LLM) agents. Traditional methods often assign sparse terminal rewards uniformly across actions in a trajectory, which can obscure the reasons for success or failure. The authors propose a novel approach called Verifier-Instrumented Credit Tracing (VICT), which leverages the internal structure of programmatically verifiable tasks to improve credit assignment. VICT transforms terminal verifiers into actionable insights by exposing executable atoms and tracing them back to actions through dependency-valid proof edges. This method allows for a more accurate redistribution of credit based on the verifier's checks, rather than relying solely on rollout-side inference. The authors demonstrate that VICT maintains the original terminal reward while enhancing the training-time advantage tensor without requiring a learned critic or additional inference-time access. The effectiveness of VICT is validated through experiments on benchmarks such as ALFWorld and WebShop, where it shows significant improvements over traditional outcome-only training and competes well with existing fine-grained credit methods.
Methodology
The authors propose VICT, which instruments terminal verifiers to create executable atoms linked to observable trajectory evidence. It redistributes advantages based on verified action-to-atom proof edges, ensuring that corrections are made only when the verifier's criteria are met. The method focuses on dependency-core attribution and maintains an explicit eligibility invariant.
Results
VICT significantly outperforms traditional outcome-only training methods and shows competitive performance against recent fine-grained credit assignment techniques on benchmarks like ALFWorld and WebShop. Ablation studies confirm that the improvements are not due to dense atom rewards or other common factors.
Implications
The findings suggest that leveraging the internal structures of verifiers can lead to more effective training of LLM agents in RL settings, potentially enhancing their decision-making capabilities in complex, long-horizon tasks. This approach could be applied to various interactive decision-making scenarios where verifiable outcomes are present.
Temporal Memory-Aware Online Test-Time Adaptation on Dynamic Graphs
Graph Learning
Time Series
Optimization
- Introduces DGOTTA, a framework for online test-time adaptation of DGNNs on dynamic graphs.
- Addresses the challenges of temporal and structural distribution shifts in dynamic graph data.
- Incorporates three innovative modules: temporal-aware augmentation, memory-aware prediction, and consistency-guided adaptation.
- Demonstrates significant improvements in generalization performance across diverse datasets and DGNN architectures.
Read more
Temporal Memory-Aware Online Test-Time Adaptation on Dynamic Graphs
Summary
This paper addresses the challenge of test-time adaptation (TTA) for dynamic graph neural networks (DGNNs), which struggle with distribution shifts between training and test graphs. The authors propose a novel framework called DGOTTA, which incorporates three key modules: temporal-aware augmentation to enhance the diversity of test dynamic graphs, memory-aware model prediction to mitigate catastrophic forgetting, and consistency-guided online adaptation to ensure smooth temporal alignment. The framework is designed to adapt DGNNs effectively during test time without requiring labeled data, thus improving generalization in dynamic environments. The experiments conducted on three real-world datasets demonstrate that DGOTTA significantly outperforms existing methods, showcasing its robustness against various distribution shifts and its applicability across multiple DGNN architectures.
Methodology
The proposed DGOTTA framework consists of three modules: (1) Temporal-aware augmentation, which increases the diversity of test dynamic graphs to handle complex shifts; (2) Memory-aware model prediction, which uses an exponential moving average strategy to stabilize predictions and reduce forgetting; (3) Consistency-guided online adaptation, which ensures temporal alignment and smooth transitions in model predictions over time.
Results
Extensive experiments on three real-world datasets (Wikipedia, MOOC, and Reddit) and four different DGNN backbones show that DGOTTA significantly enhances the generalization capabilities of DGNNs under various distribution shifts, outperforming traditional methods that do not account for dynamic changes.
Implications
The findings suggest that DGOTTA can be effectively utilized in real-world applications involving dynamic graphs, such as social networks and recommendation systems, where the underlying data distributions are non-stationary. This approach can improve the performance of models deployed in environments with evolving data characteristics.
There and Back Again: Bidirectional Diffusion Bridges for Multimodality Translation
Generative Models
Multimodal
- BIT provides a bidirectional generative framework for multimodal translation, allowing for both T2I and I2T processes.
- The model starts from text and interpolates to images, enhancing the flexibility of sampling algorithms.
- BIT is derived from stochastic calculus, yielding a simulation-friendly SDE with tractable loss functions.
- Empirical results show BIT's competitive performance against existing models in various tasks, including scientific applications.
Read more
There and Back Again: Bidirectional Diffusion Bridges for Multimodality Translation
Summary
The paper introduces BIT (Bidirectional Image-Text Diffusion Bridges), a novel framework for multimodality translation that addresses limitations in existing generative models, particularly in text-to-image (T2I) translation. Current models often follow unidirectional paths that do not adequately represent the source modality, leading to challenges in flexibility and inversion (image-to-text). BIT proposes a bidirectional approach that starts from text and interpolates into images, allowing for a source-aware generative path and enabling endpoint-conditioned processes that can be traversed in both directions. The authors derive BIT through stochastic calculus, resulting in a simulation-friendly stochastic differential equation (SDE) framework with tractable loss functions. Experiments demonstrate that BIT is competitive with and sometimes outperforms existing denoising-diffusion and deterministic-flow models across various vision-language and natural-science tasks. The paper emphasizes the importance of capturing joint probability distributions and constructing semantically meaningful interpolations to enhance the generative process, ultimately unifying T2I and image-to-text (I2T) translation within a single framework.
Methodology
BIT is developed through stochastic calculus, utilizing Girsanovβs Theorem and Doobβs h-transform to create a stochastic process that transitions between text and image modalities. The model employs a continuous representation of text to unify discrete and continuous data handling within the SDE framework. The forward and reverse SDEs are trained to ensure they describe the same process in opposite directions, allowing for bidirectional translation.
Results
BIT demonstrates competitive performance with existing noise-to-data diffusion models in both T2I and I2T tasks. It outperforms these models in data-variation tasks, indicating better information retention during the generative process. Additionally, BIT achieves the best average rank in scientific applications, particularly in cell-fate modeling.
Implications
The introduction of BIT has significant implications for generative AI, enabling more flexible and unified approaches to multimodal translation. It opens avenues for new sampling algorithms and enhances the capability of models to perform cross-modality inversion, which is crucial for applications in creative fields and scientific research.
Performative Privacy: When Differential Privacy Maximizes Utility
Theory
- Introduces the concept of performative privacy, linking privacy protection to long-term utility.
- Demonstrates that a finite privacy budget can outperform non-private estimation in specific scenarios.
- Establishes a feedback loop between data leakage and user participation, affecting future data contributions.
- Extends the analysis to a d-dimensional average with realistic data leakage definitions.
Read more
Performative Privacy: When Differential Privacy Maximizes Utility
Summary
This paper introduces the concept of performative privacy, which merges the ideas of privacy-preserving learning and performative learning. The authors argue that protecting user data not only preserves trust but can also enhance long-term utility by encouraging user participation. They formalize this relationship through a model where agents contribute data for mean estimation and may withdraw if their data is leaked. The study employs differential privacy mechanisms to create a trade-off between estimation noise and future participation. The authors demonstrate that under certain conditions, a finite privacy budget can lead to better long-term utility compared to non-private estimation, providing evidence that differential privacy can be optimal not just for protection but also for maximizing utility in a feedback loop scenario. The paper contributes to the understanding of how privacy impacts user behavior and the dynamics of data contribution in machine learning systems.
Methodology
The authors develop a theoretical model to analyze repeated mean estimation tasks involving privacy-sensitive data. They study the dynamics of user participation in relation to data leakage and apply differential privacy mechanisms to quantify the trade-offs between privacy and utility. Numerical experiments complement the theoretical findings.
Results
The study finds that a finite privacy budget can maximize long-term utility in scenarios where user participation is affected by data leakage. The results indicate that differential privacy can be advantageous not only as a protective measure but also in enhancing the overall utility of learning systems over time.
Implications
The findings suggest that implementing differential privacy in machine learning systems can lead to increased user trust and participation, ultimately improving the quality of data and model performance. This has significant implications for industries relying on sensitive data, such as healthcare and finance, where user participation is crucial.
More Data Cannot Break a Symmetry: Identifiability by Design
Theory
Optimization
- Identifiability in unsupervised alignment is constrained by the automorphism group of the stimulus geometry.
- A design-time diagnostic can identify structural failures in experimental designs, particularly in color representation.
- Selecting stimulus colors based on this diagnostic can reduce catastrophic alignment failures from 75% to 2%.
- Model discrimination and correspondence recovery are largely uncorrelated objectives.
Read more
More Data Cannot Break a Symmetry: Identifiability by Design
Summary
This paper addresses the challenge of unsupervised representational alignment in recovering stimulus correspondences based solely on geometric properties. The authors highlight that the automorphism group of the stimulus geometry limits the identifiability of such correspondences, regardless of the amount of data collected. They introduce a design-time diagnostic that reveals structural failures in experimental designs, particularly in color representation. By demonstrating that certain designs lead to catastrophic alignment failures, the authors propose a method to select stimulus colors that significantly reduces these failures. Their findings indicate that model discrimination and correspondence recovery are largely uncorrelated objectives, and that careful design can mitigate inherent risks associated with symmetry in stimulus geometry. The paper concludes that the choice of experimental design is crucial for ensuring effective unsupervised alignment, as it can influence the outcomes even before data collection begins.
Methodology
The authors analyze the automorphism group of stimulus geometries and apply a GromovβWasserstein objective to recover correspondences. They conduct experiments varying the restart budget to assess alignment recovery under different designs, particularly focusing on color stimuli. The effectiveness of their design-time diagnostic is evaluated by comparing the performance of different color sets in terms of alignment success rates.
Results
The study reveals that a symmetric design fails to improve alignment recovery even with increased data, while an asymmetric design consistently achieves successful correspondence recovery. The catastrophic cost of alignment failures is quantified, showing significant reductions when using a well-designed stimulus set. The authors demonstrate that their design-time intervention effectively moves model representations away from degenerate points, leading to a drastic decrease in alignment failures.
Implications
The findings suggest that researchers should prioritize experimental design in studies involving unsupervised alignment, as it can fundamentally affect the outcomes. This work has implications for various fields where representational alignment is critical, such as neuroscience, computer vision, and machine learning, by emphasizing the importance of designing experiments that avoid symmetry-related pitfalls.
FedEHR-Agents: Federated Agentic Optimization for Automated EHR Modeling
Federated Learning
- Introduces FedEHR-Agents, a federated optimization framework for EHR modeling.
- Shifts focus from model parameters to clinical modeling experience for improved collaboration.
- Demonstrates superior performance over local and federated baselines in clinical prediction tasks.
- Maintains patient data locality while facilitating knowledge transfer across hospitals.
Read more
FedEHR-Agents: Federated Agentic Optimization for Automated EHR Modeling
Summary
The paper introduces FedEHR-Agents, a novel federated optimization framework aimed at enhancing automated electronic health record (EHR) modeling through experience-centric collaboration among autonomous clinical agents deployed in different hospitals. Traditional federated learning (FL) approaches focus primarily on model parameters, limiting the ability to share valuable modeling experiences that can improve clinical prediction systems. FedEHR-Agents shifts this paradigm by allowing each hospital's EHR agent to accumulate and refine local modeling experiences, such as data preprocessing strategies and feature engineering decisions, while keeping patient data local. The framework employs a federated server to aggregate these experiences into global meta-prompts, which guide local agent execution. Extensive experiments on multi-hospital EHR benchmarks demonstrate that FedEHR-Agents consistently outperforms both local and federated baselines across various clinical prediction tasks, showcasing robustness across different federation scales and large language model (LLM) backbones. This work highlights the potential of federated autonomous clinical intelligence by enabling collaborative learning from diverse clinical environments without compromising patient privacy.
Methodology
The methodology involves deploying autonomous clinical EHR agents at individual hospitals that perform data preprocessing and model development. These agents accumulate local modeling experience through historical memory and task-specific evaluations. A federated server aggregates this experience into global meta-prompts, which are then used to refine local agent execution iteratively.
Results
The experiments conducted on real-world multi-hospital EHR benchmarks show that FedEHR-Agents outperforms both local and federated learning baselines across a range of clinical prediction tasks. The framework demonstrates robustness across various federation scales and different LLM backbones, indicating its effectiveness in improving automated EHR modeling.
Implications
The findings suggest that federated learning can be effectively adapted to enhance clinical modeling by focusing on experience sharing rather than just model parameters. This has significant implications for privacy-preserving collaborations in healthcare, enabling hospitals to improve their predictive capabilities without compromising patient data security.
Actionable CBFI: Integrating Structural Decomposition and Causal Counterfactual Recourse for Tabular Machine Learning
Interpretability
- A-CBFI integrates structural causal models to improve actionable counterfactual recourse.
- The framework significantly reduces the cognitive burden of interventions by focusing on root causes.
- Empirical results demonstrate a 76.9% reduction in active intervention effort while maintaining cost-effectiveness.
- A-CBFI addresses the limitations of existing methods by prioritizing causal bottlenecks over diffuse modifications.
Read more
Actionable CBFI: Integrating Structural Decomposition and Causal Counterfactual Recourse for Tabular Machine Learning
Summary
The paper introduces Actionable Case-Based Feature Importance (A-CBFI), a novel framework aimed at enhancing actionable counterfactual recourse in tabular machine learning. Current methodologies often struggle with causal invalidity and excessive cognitive burden, particularly in complex models like XGBoost. A-CBFI addresses these issues by leveraging structural causal models (SCMs) to isolate synergistic interactions and identify root causes for intervention. The framework mathematically separates the active user intervention space from downstream effects, allowing for targeted interventions that concentrate over 98.3% of effort on diagnosed bottlenecks. Empirical evaluations in financial and healthcare domains show that A-CBFI reduces the active intervention burden by 76.9% while maintaining comparable global recourse costs to traditional exhaustive causal methods. This approach not only enhances causal validity but also achieves full relative convergence across all causally feasible instances, providing users with actionable and effective recourse options.
Methodology
The A-CBFI framework utilizes structural causal models to identify and isolate synergistic interactions among features. It mathematically separates the active intervention space from downstream effects, allowing for targeted interventions that focus on root causes rather than diffuse modifications across multiple attributes.
Results
The empirical evaluations across financial and healthcare domains reveal that A-CBFI reduces the active human intervention burden by 76.9% while maintaining a comparable global recourse cost to exhaustive causal baselines. The framework successfully prioritizes diagnosed causal bottlenecks, ensuring that interventions are both actionable and causally valid.
Implications
The findings suggest that A-CBFI can be applied in high-stakes domains such as finance and healthcare, where actionable recommendations are crucial for decision-making. By providing targeted interventions, the framework enhances the interpretability and usability of machine learning models in real-world applications.
Leveraging a Foundation Model for the EEG-Based Diagnosis of Alzheimer's Disease
Time Series
- Introduces a framework combining LaBraM and Random Forest for AD diagnosis.
- Achieves high diagnostic accuracy with only 8-second EEG segments.
- Surpasses traditional spectral analysis methods in performance.
- Identifies clinically relevant biomarkers linked to cognitive decline.
Read more
Leveraging a Foundation Model for the EEG-Based Diagnosis of Alzheimer's Disease
Summary
This paper addresses the diagnostic challenges of Alzheimer's Disease (AD) by proposing a novel framework that utilizes the Large Brain Model (LaBraM), pretrained on over 2,500 hours of EEG data. Traditional linear methods struggle to capture the non-linear dynamics of neural signals, leading to diagnostic bottlenecks. The authors integrate high-dimensional latent embeddings from LaBraM with a non-linear Random Forest classifier to isolate robust disease markers. The proposed method demonstrates superior performance in distinguishing dementia patients from healthy controls, achieving an ROC-AUC of 89.36%, PR AUC of 81.45%, and Balanced Accuracy of 82.44% using only 8-second EEG segments. This performance outstrips traditional spectral baselines, such as band-power features. The study also confirms that the model captures clinically validated biomarkers, particularly in the alpha and theta rhythms, and shows significant correlations between predicted dementia probabilities and cognitive performance metrics. Overall, the findings highlight the potential of deep latent representations in extracting clinically relevant signatures from noisy EEG signals, facilitating rapid and efficient diagnosis of AD.
Methodology
The authors employed a framework that integrates the Large Brain Model (LaBraM) for high-dimensional EEG data representation with a Random Forest classifier to capture non-linear interactions. The LaBraM model was pretrained on extensive EEG data, and the resulting embeddings were used for downstream classification. The methodology included rigorous 5-fold cross-validation and comparisons against traditional spectral analysis methods.
Results
The proposed method achieved an ROC-AUC of 89.36% Β± 3.49%, PR AUC of 81.45% Β± 4.43%, and Balanced Accuracy of 82.44% Β± 4.34% in distinguishing dementia patients from healthy controls. The model effectively identified clinically validated biomarkers and showed significant correlations between predicted dementia probabilities and cognitive performance.
Implications
This research suggests that deep learning models, particularly foundation models like LaBraM, can significantly enhance the diagnostic capabilities for Alzheimer's Disease by capturing complex neural dynamics. The findings may lead to more accurate and efficient diagnostic tools in clinical settings, improving early detection and treatment strategies for AD.
Exact Risk Ratios for Weighted Data Selection in Linear Regression
Theory
Optimization
- Determines exact values for the worst-case risk ratios in weighted data selection for linear regression.
- Proves Fw(d, 2d - 1) = 1 + 1/d, Fw(3, 4) = 5/3, and Fw(4, 5) = 2.
- Establishes a lower bound for Fw(d, d + k) using harmonic quantities.
- Conjectures that the exact minimax value holds throughout the open regime.
Read more
Exact Risk Ratios for Weighted Data Selection in Linear Regression
Summary
This paper addresses an open problem posed by Hanneke et al. regarding the performance of empirical risk minimizers when trained on a subset of examples from a dataset. Specifically, it focuses on weighted data selection in linear regression, where a selector chooses at most n examples from a dataset D and assigns nonnegative weights to them. The main quantity of interest is the worst-case ratio Fw(d, n) between the loss of the predictor returned by the minimum-norm empirical risk minimization (ERM) and the optimal loss. The author determines the values of Fw(d, n) in several cases, particularly in the open regime where d < n < 2d. Key results include proving that Fw(d, 2d - 1) = 1 + 1/d for all d β₯ 1, and calculating Fw(3, 4) = 5/3 and Fw(4, 5) = 2. The paper also establishes a lower bound for Fw(d, d + k) and conjectures that equality holds throughout the open regime. The methodology involves geometric arguments, rigidity theorems, and structural results that contribute to understanding the behavior of weighted data selection in linear regression.
Methodology
The paper employs geometric arguments and rigidity theorems to analyze the structure of loss gradients and their configurations. It utilizes explicit harmonic quantities to establish lower bounds and explores the properties of positive spanning sets and bases in the context of linear regression.
Results
The main results include the exact values of Fw(d, n) for specific cases, particularly confirming Fw(d, 2d - 1) = 1 + 1/d, Fw(3, 4) = 5/3, and Fw(4, 5) = 2. Additionally, a lower bound for the general case is established, and the conjecture regarding the equality of the lower bound and the exact value is proposed.
Implications
The findings have implications for the design of efficient algorithms in empirical risk minimization, particularly in scenarios where data selection is critical. The results can enhance understanding of data selection strategies in linear regression and potentially inform practices in machine learning applications that rely on weighted data.
Euclidean Fourier Neural Operators
Theory
Efficient ML
- EFNOs provide a domain-independent alternative to FNOs by parameterizing the spectral kernel as a continuous function of physical wavevectors.
- EFNOs can learn operators that generalize across different periodic domains, addressing the limitations of FNOs in domain transfer.
- The methodology was validated through experiments on a heat equation and materials science tasks, showing superior performance in generalization.
- EFNOs maintain consistent performance across varying grid sizes and shapes, unlike FNOs, whose performance degrades with domain changes.
Read more
Euclidean Fourier Neural Operators
Summary
The paper introduces Euclidean Fourier Neural Operators (EFNOs) as a novel approach to address the limitations of Fourier Neural Operators (FNOs) in transferring learned mappings across different periodic domains. While FNOs are effective in learning mappings between function spaces and are independent of grid resolution, they are constrained by their dependence on the periodic domain, which affects their applicability across varying shapes and sizes. The authors propose a continuous parameterization of the spectral kernel in EFNOs, allowing them to learn operators that maintain consistency across different domains. The effectiveness of EFNOs is demonstrated through evaluations on a simple heat equation and a materials science task involving exchange-correlation potentials across diverse crystal structures. The results indicate that EFNOs can generalize to unseen grid sizes and domains, overcoming the limitations faced by traditional FNOs.
Methodology
The authors propose EFNOs by modifying the parameterization of the spectral kernel in Fourier Neural Operators. Instead of using discrete trainable weights indexed by Fourier mode numbers, EFNOs parameterize the spectral kernel as a continuous function of physical wavevectors. This allows the EFNOs to act consistently across different periodic domains. The methodology includes training on uniform grids and utilizing Fast Fourier Transforms (FFTs) for efficient computation.
Results
The evaluation of EFNOs on a heat equation and a materials science task demonstrated that EFNOs could effectively generalize to unseen grid sizes and domains. In contrast to FNOs, which exhibited increasing error with larger supercells, EFNOs maintained a constant error rate, showcasing their robustness and adaptability.
Implications
The development of EFNOs has significant implications for various applications in physics and materials science, where the ability to transfer learned operators across different domains is crucial. This advancement can enhance modeling capabilities in fields such as fluid dynamics, climate forecasting, and materials design.
Beyond Flat Netlist: Hierarchical Graph Representation Learning for Scalable Analysis of Sequential Circuits
Graph Learning
- DeepSeq3 introduces a hierarchical representation for sequential netlists, enhancing scalability and representation richness.
- The framework utilizes two types of graphs: combinational logic subgraphs and super-node graphs, processed hierarchically.
- A state-centric pre-training task is introduced to improve the model's understanding of register-level semantics.
- DeepSeq3 achieves an 18% reduction in BMC solving time on large-scale benchmarks, demonstrating its practical effectiveness.
Read more
Beyond Flat Netlist: Hierarchical Graph Representation Learning for Scalable Analysis of Sequential Circuits
Summary
This paper introduces DeepSeq3, a novel hierarchical framework for Circuit Representation Learning (CRL) aimed at improving the analysis of sequential circuits. Traditional methods struggle with the scalability of large industrial netlists and fail to adequately model register-level temporal dynamics. DeepSeq3 addresses these challenges by creating a two-level representation of circuits: fine-grained combinational subgraphs partitioned by flip-flops (FFs) and a high-level Super-Node Graph (SNG) that captures the register-transfer structure. The framework employs a dual Graph Neural Network (GNN) architecture to learn representations at both levels, effectively capturing local Boolean logic and global state transitions. A key innovation is the state-centric pre-training scheme that predicts the reachability between FF states, enhancing the model's understanding of temporal behavior. Experimental results demonstrate that DeepSeq3 significantly improves scalability and representation richness, achieving an 18% reduction in bounded model checking (BMC) solving time while maintaining correctness across large-scale benchmarks.
Methodology
The methodology involves creating a hierarchical representation of circuits using two types of graphs: fine-grained combinational subgraphs and a high-level Super-Node Graph (SNG). The framework employs two Graph Neural Networks (GNNs) to encode these graphs, with the first GNN focusing on combinational reasoning and the second on the SNG. A state-centric pre-training scheme is implemented to predict reachable states of flip-flops, enhancing the model's temporal understanding.
Results
DeepSeq3 demonstrated superior scalability and richer representations compared to existing methods, achieving an 18% reduction in solving time for bounded model checking tasks on large-scale benchmarks, while ensuring correctness.
Implications
The implications of this work are significant for the field of Electronic Design Automation (EDA), particularly in improving the efficiency of circuit analysis tasks. The hierarchical representation and enhanced understanding of temporal dynamics can lead to better optimization and design processes in large-scale circuit designs.
A Deeper Analysis of Block-Sparse Featurizers
Computer Vision
Theory
Interpretability
- The BSF introduces a block-based approach to feature representation, improving upon traditional sparse autoencoders.
- Architectural modifications, such as the Tournament Top-K selection rule, effectively reduce feature splitting.
- The BSF demonstrates superior performance in recovering features from low-dimensional manifolds compared to classical SAEs.
- The study highlights the importance of seed stability in model training, showing that BSFs maintain consistency across different initializations.
Read more
A Deeper Analysis of Block-Sparse Featurizers
Summary
This paper investigates the block-sparse featurizer (BSF), a novel approach designed to handle features residing on low-dimensional manifolds, particularly in vision tasks. The BSF operates by grouping latent variables into blocks, allowing for a more efficient representation of complex geometries compared to traditional sparse autoencoders (SAEs). The authors identify several limitations of the BSF, such as feature splitting and composition issues, which are reminiscent of classic SAE failure modes. To address these challenges, they propose architectural modifications, including a Tournament Top-K selection rule that mitigates feature splitting. The study employs a synthetic dataset, termed the 'Manifold Zoo,' to systematically evaluate the BSF's performance against classical SAEs, focusing on metrics like seed stability and spurious correlation resistance. The findings reveal that while the BSF still exhibits some tiling behavior on manifolds, it significantly outperforms SAEs in terms of feature recovery, particularly in scenarios with correlated data. Additionally, the authors extend the block paradigm to crosscoders and discuss the potential for BSFs to capture various geometrical structures beyond low-dimensional manifolds.
Methodology
The authors trained BSFs on a synthetic dataset called the 'Manifold Zoo,' which consists of 128 features with varying geometries. They systematically tested the BSF for seed stability, spurious correlation resistance, and compared its performance against classical SAEs using metrics such as R2 scores for feature recovery. The BSF's architecture was modified to include a Tournament Top-K selection rule to enhance its robustness.
Results
The BSF outperformed classical SAEs in recovering features, achieving higher R2 scores in both independent and correlated datasets. Specifically, the BSF achieved a mean recovery R2 of 0.77, while the SAE with post-hoc methods reached a maximum of 0.65. The BSF also demonstrated greater stability across different training seeds, effectively recovering various geometrical features with consistent accuracy.
Implications
The findings suggest that BSFs could be a valuable tool in computer vision tasks where features lie on low-dimensional manifolds. The architectural improvements proposed may lead to more robust models that can better interpret complex data structures, potentially enhancing performance in various machine learning applications.
Efficient Online Continual Foundation Model Fine-Tuning for Predictive Process Monitoring
Efficient ML
- Introduces COMPASS, the first framework for online continual fine-tuning of foundation models in PPM.
- Adapts loss-plateau drift detection for autonomous task boundary identification in event streams.
- Combines residual subspace projection with pre-trained knowledge anchoring to achieve stability and plasticity.
- Demonstrates superior performance over SOTA non-FM competitors in various drift scenarios.
Read more
Efficient Online Continual Foundation Model Fine-Tuning for Predictive Process Monitoring
Summary
This paper introduces COMPASS (Continual Online foundation Model-based PPM with Adaptive SubSpaces), a novel framework for the online continual fine-tuning of foundation models (FMs) specifically for Predictive Process Monitoring (PPM). The authors address the challenges posed by concept drift in dynamic environments, where traditional PPM models struggle due to cold-start problems and catastrophic forgetting. COMPASS employs a loss-plateau drift detection mechanism to autonomously identify task boundaries in event streams, allowing for the maintenance of a unified knowledge subspace that integrates both pre-trained and task-specific knowledge. The framework is evaluated across nine event streams, including both synthetic and real-world scenarios, demonstrating superior performance compared to state-of-the-art (SOTA) non-FM competitors and various update strategy baselines. The results indicate significant improvements in handling recurrent drift and complex, long-running cases, while maintaining acceptable computational overhead.
Methodology
The methodology involves adapting loss-plateau drift detection to identify task boundaries in event streams and utilizing a unified knowledge subspace that incorporates both pre-trained and task-specific knowledge. The framework employs parameter-efficient fine-tuning techniques to ensure that new task adaptations do not interfere with previously learned knowledge.
Results
COMPASS outperformed three state-of-the-art non-FM competitors and two baseline update strategies across nine event streams, particularly excelling in scenarios with recurrent drift and complex, long-running cases. The framework demonstrated effective adaptation to new tasks while preserving prior knowledge, resulting in improved predictive performance.
Implications
The proposed framework has significant implications for organizations utilizing PPM in dynamic environments, enabling more robust predictive capabilities that adapt to changing process distributions. This could enhance operational intelligence and decision-making in various sectors, including healthcare and finance.
Curvature-Conditioned Multiscale Momentum with Sphere Constraints for LLM Pretraining
Large Language Models
Optimization
Efficient ML
- Introduces a novel optimization method for LLM pretraining that enhances training dynamics along flat directions.
- Combines slow-decay and fast-decay momentum components to balance noise reduction and curvature adaptation.
- Employs sphere constraints to prevent parameter inflation and maintain stability during training.
- Demonstrates significant performance improvements over existing optimizers like Muon across various architectures and model sizes.
Read more
Curvature-Conditioned Multiscale Momentum with Sphere Constraints for LLM Pretraining
Summary
This paper addresses the challenges of pretraining large language models (LLMs), particularly the issues of noise-dominant gradients and an ill-conditioned loss landscape. The authors propose a novel optimization method called curvature-conditioned multiscale momentum with sphere constraints, which aims to enhance training dynamics specifically along flat directions in the loss landscape. By combining slow-decay and fast-decay momentum components, the method effectively reduces noise while adapting to curvature changes. The introduction of sphere constraints prevents parameter inflation and ensures stability during training. The proposed method, denoted as MuonM, is empirically evaluated across various architectures and model sizes, demonstrating significant improvements in training efficiency and lower terminal loss compared to existing optimizers. The theoretical analysis supports the design principles of the proposed method, highlighting its potential to accelerate optimization in LLM pretraining.
Methodology
The authors developed a curvature-conditioned multiscale momentum method that activates in flat directions for robust noise reduction while deactivating in sharp directions to maintain stability. Sphere constraints are applied to the momentum components to prevent norm inflation and ensure proper momentum inheritance across varying tangent spaces.
Results
The empirical evaluation shows that the proposed MuonM method consistently outperforms the baseline Muon optimizer, achieving lower terminal loss across diverse architectures (dense and MoE) and model sizes (0.12B to 2.3B parameters). The improvements persist under extended token budgets, indicating scalability for longer training durations.
Implications
The proposed optimization method has the potential to significantly enhance the efficiency of LLM pretraining, making it more feasible to train larger models and improve their performance. This could lead to advancements in various NLP applications that rely on large language models.
Learning to Difference: Adaptive Reversible Differencing (AdaRDiff) for Time Series Forecasting
Time Series
- AdaRDiff is a plug-and-play module that learns differencing weights adaptively, improving forecasting accuracy.
- The method captures trends and seasonalities jointly and allows for autoregressive reconstruction of forecasts.
- AdaRDiff shows significant speed improvements on GPU, achieving up to 33.7Γ speedup over naive recurrence.
- The two-phase training schedule enhances the learning dynamics by separating structure discovery from reconstruction.
Read more
Learning to Difference: Adaptive Reversible Differencing (AdaRDiff) for Time Series Forecasting
Summary
The paper presents Adaptive Reversible Differencing (AdaRDiff), a novel approach for time series forecasting that addresses the challenges posed by trends and seasonality in long-horizon predictions. Traditional differencing methods, which are essential in classical time series analysis, are often rigid and require manual specification of parameters, making them less compatible with modern deep learning architectures. AdaRDiff introduces a learnable differencing scheme that adapts to the data, allowing for the capture of multiple seasonalities and trends simultaneously. The method employs weighted differencing using learnable weights from past observations, simplifying the time series for forecasting. After forecasting, the removed components are autoregressively restored to reconstruct the original forecast, achieving a closed-form convolutional expression that allows for significant speed improvements on GPU. The authors also propose a two-phase training schedule to optimize the learning process, separating the discovery of temporal structures from the reconstruction learning. Extensive experiments demonstrate that AdaRDiff achieves state-of-the-art accuracy across various benchmarks in electricity, weather, traffic, and energy forecasting, while being compatible with multiple forecasting backbones.
Methodology
AdaRDiff employs a learnable differencing approach that uses weighted combinations of past observations to simplify time series data. It integrates with various forecasting models, allowing for end-to-end training. The method includes a two-phase training schedule to optimize the learning process, focusing first on temporal structure discovery and then on reconstruction learning.
Results
AdaRDiff achieves state-of-the-art forecasting accuracy across eight benchmark datasets, improving performance by up to 25.9% with linear models and 18.3% with Transformer-based models. The method also demonstrates significant computational efficiency, with up to 33.7Γ speedup on GPU.
Implications
AdaRDiff can be applied in various real-world scenarios requiring accurate long-term forecasting, such as traffic prediction, energy management, and sales forecasting. Its adaptability and efficiency make it a valuable tool for practitioners in time series analysis.
Beyond Search-Imitation: Prior-Directed Exploration for Searchless Chess
Reinforcement Learning
Theory
- Introduces prior-directed exploration to enhance searchless chess networks.
- Replaces traditional entropy bonuses with a forward KL divergence towards MCTS priors.
- Achieves significant improvements in puzzle accuracy and tactical performance.
- Demonstrates a dissociation between tactical accuracy and overall playing strength.
Read more
Beyond Search-Imitation: Prior-Directed Exploration for Searchless Chess
Summary
This paper presents a novel approach to enhancing searchless chess networks, which achieve human master-level strength through a single forward pass by imitating a stronger teacher. The authors argue that imitating a search is inadequate for effective searchless play. To address this, they propose a fine-tuning method using self-play reinforcement learning (RL) that incorporates prior-directed exploration. Instead of the traditional entropy bonus, they introduce a forward Kullback-Leibler (KL) divergence towards the network's own Monte Carlo Tree Search (MCTS) prior, which guides exploration towards promising moves. This method is paired with an adaptive sampling temperature based on the value head's outcome uncertainty. The results demonstrate that this approach improves puzzle accuracy from 93.9% to 94.9% and mate-in-four accuracy from 77% to 81%, while maintaining searchless strength. The study also reveals a dissociation between tactical accuracy and playing strength, indicating that improvements in puzzle-solving do not necessarily translate to stronger gameplay. The findings suggest that anchoring exploration to the network's prior significantly enhances performance and preserves the quality of deeper tactical solutions.
Methodology
The authors employed self-play reinforcement learning to fine-tune a chess network originally trained through imitation. They replaced the standard entropy bonus with a forward KL divergence towards the network's own MCTS prior, guiding exploration towards more promising moves. The exploration strategy was further refined with an adaptive sampling temperature based on the uncertainty of the value head's outcomes.
Results
The proposed method improved puzzle accuracy from 93.9% to 94.9% and mate-in-four accuracy from 77% to 81% over a suite of 100,000 puzzles. The study found that while tactical accuracy improved, the overall playing strength remained at or slightly above the base level, indicating a dissociation between puzzle-solving ability and actual gameplay performance.
Implications
This research has implications for the development of more effective reinforcement learning strategies in chess and potentially other domains where searchless decision-making is critical. It suggests that leveraging prior knowledge can significantly enhance exploration and performance in complex environments.
Explainable Uncertainty Estimation for Reliable Medical AI
Interpretability
- Introduces egRUE, a method that combines uncertainty estimation with explainability.
- Demonstrates that egRUE provides feature-level insights into prediction uncertainties.
- Validates the method through theoretical analysis and experiments on real-world datasets.
- User studies indicate that egRUE improves trust in AI predictions among medical professionals.
Read more
Explainable Uncertainty Estimation for Reliable Medical AI
Summary
This paper addresses the critical issue of trust in artificial intelligence (AI) applications in healthcare by proposing a novel approach called explainable uncertainty estimation. The authors highlight the limitations of existing methods that separately treat uncertainty estimation and explainable AI (XAI), which fail to provide insights into the reasons behind prediction uncertainties. To bridge this gap, they introduce the Expected Gradients Reconstruction Uncertainty Estimate (egRUE), which integrates uncertainty quantification with feature-level explanations. The method decomposes uncertainty into contributions from individual features, allowing for a clearer understanding of which factors influence model predictions. The authors validate the theoretical properties of egRUE and demonstrate its effectiveness through experiments on four real-world datasets, showing improved reliability and interpretability compared to traditional methods. A user study with medical experts further reveals that egRUE enhances trust in AI predictions by providing meaningful explanations alongside uncertainty scores, ultimately supporting better decision-making in safety-critical healthcare environments.
Methodology
The authors developed the Expected Gradients Reconstruction Uncertainty Estimate (egRUE), which utilizes the Expected Gradients (EG) technique for feature attribution. This method computes uncertainty by averaging gradients of the prediction function over a distribution of baseline inputs, allowing for the decomposition of uncertainty into contributions from individual features.
Results
The experiments conducted on four real-world datasets demonstrated that egRUE outperformed existing uncertainty estimation methods in terms of reliability and interpretability. The user study indicated that medical experts found egRUE's explanations to significantly enhance their trust in the model's predictions, leading to increased confidence in correct predictions and reduced confidence in incorrect ones.
Implications
The findings suggest that integrating uncertainty estimation with explainable AI can significantly improve the trustworthiness of AI systems in healthcare. This approach not only aids clinicians in making informed decisions but also enhances the overall safety and reliability of AI applications in medical settings.
When Muon Meets Task Interference: A Spectral Perspective on Continual Learning and Model Merging
Optimization
Theory
- Unification of catastrophic forgetting in CL and weight-disentanglement error in MM as instances of task interference.
- Derivation of an upper bound on task interference that isolates the spectral norm of weight updates as an optimizer-controllable factor.
- Identification of the Muon optimizer as a mechanism that effectively regulates task interference through spectral norm control.
- Empirical validation showing significant accuracy improvements when using Muon over AdamW in various benchmarks.
Read more
When Muon Meets Task Interference: A Spectral Perspective on Continual Learning and Model Merging
Summary
This paper addresses the challenges of continual learning (CL) and model merging (MM), which aim to create a single model that performs well across multiple tasks while facing issues like catastrophic forgetting and weight-disentanglement error. The authors propose that these challenges stem from a common phenomenon termed 'task interference,' where parameter updates beneficial for one task adversely affect another. They formalize this interference using a layer-wise Frobenius inner product, which allows for a unified analysis of both CL and MM. The study derives an upper bound on task interference that highlights the spectral norm of weight updates as a controllable factor influenced by the optimizer. The Muon optimizer is identified as a mechanism that effectively regulates this spectral norm, thereby reducing task interference. Experimental results demonstrate that replacing the AdamW optimizer with Muon leads to significant improvements in accuracy across various benchmarks for both CL and MM, validating the theoretical findings and positioning Muon as a principled optimizer-centric approach to mitigate task interference.
Methodology
The authors conducted a theoretical analysis to unify the concepts of task interference in CL and MM, deriving an upper bound on task interference based on the spectral norm of weight updates. They performed a per-mode analysis to validate the impact of the spectral norm on task interference. Experimental validation was carried out by replacing the AdamW optimizer with the Muon optimizer in various CL and MM benchmarks to assess performance improvements.
Results
The study found that using the Muon optimizer improved accuracy by up to +5.02 points on the eight-task model-merging benchmark across three CLIP backbones. Additionally, Muon consistently delivered positive gains across ten class-incremental protocols, three task-incremental protocols, and improved metrics on the 11-task MTIL benchmark.
Implications
The findings suggest that optimizing the spectral norm of weight updates can significantly enhance model performance in continual learning and model merging scenarios. This approach may lead to more robust models capable of handling multiple tasks without suffering from performance degradation due to task interference.