AI-generated summaries
Today's ML research,
without the noise.
Daily summaries of the latest machine learning papers from arXiv, processed every 8 hours.
24
Papers today
8h
Update frequency
7
Days of history
Phase Transition Frequency as a Training Time Predictor of Test Accuracy in ResNets
Computer Vision
- Phase transition frequency is a strong negative predictor of test accuracy in ResNets on standard benchmarks.
- The correlation between phase transitions and accuracy diminishes under distributional stress.
- The transition count retains predictive power even when controlling for architecture depth.
- Compared to alternative training-curve signals, phase transition frequency shows competitive predictive capability.
Read more
Phase Transition Frequency as a Training Time Predictor of Test Accuracy in ResNets
Summary
This paper investigates the number of discrete class-separability jumps, termed phase transitions, during the fine-tuning of ResNet architectures as a predictor of final test accuracy. Through 75 experiments across four benchmarks (CIFAR-10, CIFAR-100, TinyImageNet, and CIFAR-10-C) and three ResNet architectures (ResNet-18, ResNet-50, and ResNet-101), the study finds a strong negative correlation between the number of phase transitions and test accuracy on standard i.i.d. benchmarks, with correlation coefficients of r = -0.84 for CIFAR-10 and r = -0.87 for CIFAR-100. However, this correlation weakens under distributional stress, as evidenced by lower correlations on TinyImageNet (r = -0.45) and CIFAR-10-C (r = -0.19). Further analyses indicate that the transition count retains predictive power even when controlling for architecture depth, particularly on CIFAR-100. The study also compares the transition count against six alternative training-curve signals, finding it to be the strongest predictor on CIFAR-100 and competitive on CIFAR-10, although it is outperformed by other signals on stressed benchmarks. The findings suggest that while phase transition frequency is a useful in-distribution training-quality probe, it may not universally predict accuracy under distribution shifts.
Methodology
The methodology involves counting phase transitions based on class separability during training, using an empirical approach across multiple ResNet architectures and datasets. The Fisher criterion is employed to compute class separability, and the study analyzes correlations between transition counts and final test accuracy.
Results
The results indicate a strong negative correlation between phase transition frequency and test accuracy on CIFAR-10 and CIFAR-100, with correlation coefficients of r = -0.84 and r = -0.87, respectively. The correlation weakens significantly on more challenging datasets like TinyImageNet and CIFAR-10-C. The transition count remains a statistically significant predictor when controlling for architecture depth on CIFAR-100.
Implications
The findings suggest that monitoring phase transition frequency could serve as a practical tool for assessing training quality in neural networks, particularly in standard classification tasks. However, its limited predictive power under distributional shifts indicates that additional metrics may be necessary for robust performance evaluation.
Too Rare to Learn: Prescribed Cyclone Tracks Degrade a Bay of Bengal Ocean Emulator
Time Series
- Prescribing cyclone tracks as input to ocean emulators can degrade performance.
- The ocean-only model outperforms the storm-conditioned model in all test runs.
- The failure is attributed to the low frequency of cyclone occurrences in training data.
- Replacing cyclone inputs with no-storm maps during inference improves forecast accuracy.
Read more
Too Rare to Learn: Prescribed Cyclone Tracks Degrade a Bay of Bengal Ocean Emulator
Summary
This paper investigates the impact of incorporating prescribed cyclone tracks as input to neural ocean emulators designed for forecasting in cyclone-prone regions, specifically the Bay of Bengal. The study reveals that conditioning the model on cyclone tracks is detrimental to its performance. By withholding 15 cyclones from the GLORYS12 reanalysis dataset, the author compares two U-Net models: one conditioned on cyclone tracks and the other trained without this conditioning. The results show that the ocean-only model consistently outperforms the storm-conditioned model across multiple seeds, with significant statistical differences in skill. The primary reason for this degradation is identified as the low frequency of cyclone occurrences in the training data, leading to the model's inability to generalize effectively when faced with rare events. The findings suggest that the model's performance improves when the cyclone input is replaced with a no-storm map during inference, indicating that rare conditioning can lead to confidently incorrect predictions.
Methodology
The study employs two U-Net models, one conditioned on cyclone tracks and the other without. It uses a dataset of GLORYS12 reanalysis data, withholding 15 cyclones for testing. The models are trained on historical ocean states, and their performance is evaluated against a persistence baseline using masked mean squared error (MSE). Statistical significance is assessed using the Wilcoxon signed-rank test.
Results
The ocean-only model consistently beats persistence across all test runs, achieving skill scores ranging from +0.048 to +0.161. In contrast, the storm-conditioned model shows negative skill scores, indicating poor performance, with ranges from -0.477 to -0.075. The results are statistically significant, with p-values indicating strong evidence against the effectiveness of the storm conditioning.
Implications
These findings suggest that incorporating rare events as conditioning inputs in machine learning models may lead to overfitting and poor generalization. This has implications for the design of forecasting models in other domains where rare events are critical, highlighting the need for careful consideration of training data characteristics.
Conformity Breaks Conformal Prediction
NLP
Large Language Models
Theory
- Introduction of the score-mechanism shift, highlighting how peer influence can invalidate conformal prediction guarantees.
- Demonstration of a significant drop in coverage rates under peer pressure, particularly affecting low-confidence items.
- Identification of a failure in decision-making processes where models may act confidently on incorrect answers due to peer influence.
- Traditional conformal prediction methods fail to address the issues arising from social conformity in multi-agent systems.
Read more
Conformity Breaks Conformal Prediction
Summary
This paper investigates the impact of social conformity on the effectiveness of conformal prediction in multi-agent large language model (LLM) systems. The authors introduce the concept of a 'score-mechanism shift,' which occurs when an LLM, initially calibrated to provide accurate predictions in isolation, is influenced by peer models that assert incorrect answers. This shift leads to a significant drop in coverage guarantees, from a calibrated 90% to 74% when exposed to unanimous wrong answers from peers. The study highlights a hidden failure in the decision-making process, where the model may become overly confident in incorrect answers due to peer pressure, undermining its ability to escalate uncertainty appropriately. The authors demonstrate that traditional conformal prediction methods do not address this issue, as the underlying distribution of questions remains unchanged, but the scoring behavior of the model does. Through empirical evaluation across various models and tasks, they reveal that an adversary can exploit this vulnerability to reduce coverage further, particularly targeting low-confidence items. The findings challenge existing assumptions in conformal prediction and suggest that peer influence must be considered in multi-agent systems to maintain reliable performance.
Methodology
The authors conducted experiments using multiple open-weight LLMs on various multiple-choice question-answering tasks. They analyzed the models' performance under different peer pressure conditions, including unanimous wrong, mixed, and unanimous correct answers. The study involved calibrating the models on solo data and testing their coverage under peer influence, measuring the impact on scoring behavior and decision-making.
Results
The results indicated that under unanimous-wrong peer conditions, the coverage rate dropped from 90% to 74%. Additionally, for low-confidence items, coverage fell from 87% to 47%, despite the overall average appearing stable. The act-vs-escalate defense mechanism proposed in prior work was shown to fail under the same conditions, with significant misclassifications occurring.
Implications
The findings suggest that conformal prediction methods need to be re-evaluated in the context of multi-agent systems, particularly in applications where LLMs interact with one another. This research highlights the necessity of developing robust mechanisms to mitigate the effects of social conformity on model predictions, which could have implications for safety and reliability in AI systems.
Simulation-free Unbalanced Dynamic Optimal Transport with General Growth Penalty
Optimization
Theory
Efficient ML
- Introduction of SUDO, a simulation-free framework for UDOT with general growth penalties.
- Identification of the non-degenerate regime of growth penalties, emphasizing the importance of convex penalties.
- SUDO achieves accuracy comparable to analytical solvers while being significantly faster than NeuralODE-based methods.
- Flexibility in selecting growth penalties allows integration of diverse biological priors.
Read more
Simulation-free Unbalanced Dynamic Optimal Transport with General Growth Penalty
Summary
This paper addresses the challenge of inferring cellular dynamics from unpaired single-cell snapshots, which requires modeling state transitions alongside population growth or death. The authors focus on Unbalanced Dynamic Optimal Transport (UDOT), which incorporates a growth penalty to account for these biological processes. Existing UDOT solvers are limited by either computationally expensive NeuralODE simulations or reliance on analytical solutions restricted to quadratic penalties, such as Wasserstein-Fisher-Rao (WFR) geodesics. To overcome these limitations, the authors introduce Simulation-free Unbalanced Dynamic Optimal Transport (SUDO), a framework that efficiently handles general non-quadratic convex growth penalties without simulations. The paper demonstrates that concave growth penalties lead to degenerate solutions, motivating the use of convex penalties for meaningful biological inference. SUDO learns conditional paths and transport costs, solves the semi-coupling problem, and utilizes unbalanced flow matching to achieve a simulation-free solution. The results show that SUDO matches the accuracy of analytical solvers on WFR benchmarks while significantly outperforming simulation-based methods in computational speed. Additionally, SUDO supports asymmetric penalties, allowing for more realistic modeling of biological processes, such as proliferation-dominant scenarios.
Methodology
The authors propose SUDO, which learns conditional paths and transport costs to solve UDOT problems without relying on simulations. The framework extends previous flow matching approaches, enabling the handling of general non-quadratic convex growth penalties. The methodology includes theoretical analysis of growth penalties and practical implementation on benchmark datasets.
Results
SUDO matches the accuracy of existing analytical solution-based algorithms on WFR problems while significantly improving computational speed compared to NeuralODE-based methods. The framework also demonstrates the ability to model asymmetric growth penalties, producing more plausible trajectories and growth estimates in biological contexts.
Implications
The development of SUDO has significant implications for single-cell trajectory inference, allowing researchers to model complex biological processes more accurately. The ability to incorporate diverse biological priors through flexible penalty selection enhances the understanding of cellular dynamics and could lead to improved insights in developmental biology and related fields.
Fast Gauss Sums via Flash Attention
Efficient ML
Theory
Optimization
- Introduces a method to compute Gaussian kernel sums using flash attention without custom GPU code.
- Demonstrates significant performance improvements over existing methods like PyTorch and PyKeOps.
- Provides a detailed analysis of numerical stability and pitfalls when using half-precision formats.
- Offers two differentiable reductions for Gaussian kernel sums that integrate seamlessly with PyTorch.
Read more
Fast Gauss Sums via Flash Attention
Summary
This paper presents a novel approach to efficiently compute Gaussian kernel sums using flash attention, a technique originally developed for softmax attention. The authors demonstrate that Gaussian kernel sums, which are central to various kernel methods like maximum mean discrepancies (MMDs) and Stein variational gradient descent (SVGD), can be evaluated without custom GPU code by leveraging flash attention. The proposed method involves two small input augmentations that transform the normalized softmax reduction into the unnormalized Gaussian sum. The authors provide two differentiable reductions to attention calls, analyze potential pitfalls in using half-precision floating-point formats, and benchmark their approach against existing implementations in PyTorch and PyKeOps. The results show that for feature dimensions greater than 8 in fp16, their method significantly outperforms traditional compiled code in terms of speed, memory efficiency, and accuracy, maintaining linear memory scaling.
Methodology
The authors utilize flash attention to compute Gaussian kernel sums by augmenting the input data to transform the softmax attention mechanism into a Gaussian sum evaluation. They propose two algorithms: one that requires the logits from the attention backend and another that avoids this requirement by modifying the query and key inputs. Both methods are implemented using the public PyTorch API, ensuring compatibility with automatic differentiation.
Results
Benchmarking against naive PyTorch implementations and PyKeOps revealed that the proposed flash attention-based methods (reweight and prescale) significantly reduce computation time and memory overhead while achieving comparable or better accuracy. The experiments were conducted on an NVIDIA GeForce RTX 5090, demonstrating the efficiency of the proposed methods in handling large input sizes.
Implications
The findings suggest that flash attention can be effectively repurposed for Gaussian kernel computations, potentially leading to faster and more efficient implementations of various kernel-based machine learning algorithms. This could enhance the scalability of methods like MMD and SVGD in practical applications.
SharedSAE: One Feature Dictionary Across Language Models
NLP
Large Language Models
Interpretability
- Introduction of SharedSAE, which utilizes a single shared sparse dictionary for multiple language models.
- Preservation of activation magnitudes and support for single-model inference through model dropout.
- Demonstrated high cross-model activation alignment and transferability of latent descriptions.
- Efficient adaptation of new models to a frozen shared dictionary without altering existing latents.
Read more
SharedSAE: One Feature Dictionary Across Language Models
Summary
The paper introduces SharedSAE, a novel approach that utilizes a single shared sparse autoencoder (SAE) to interpret activations across multiple language models, rather than training separate SAEs for each model. This method combines a shared dictionary with model-specific encoder-decoder pairs, allowing for efficient inference and adaptation. Unlike previous methods that discard activation magnitudes and require all models during inference, SharedSAE normalizes selection scores while preserving magnitudes and employs model dropout for single-model inference. The authors trained SharedSAE on four 1B-scale base language models from different families and tokenizers. The results indicate that SharedSAE retains 96.6% of the mean explained variance of dedicated SAEs, exhibits significantly higher cross-model correlations, and allows for effective transfer of latent descriptions across models. Additionally, after freezing the dictionary, new models can be adapted efficiently, achieving near-dedicated reconstruction quality. This work presents a significant advancement in the interpretability of language models by reducing redundancy and enhancing cross-model alignment.
Methodology
SharedSAE employs a shared dictionary learned through tokenizer, latent choice, and activation magnitude alignment. It utilizes per-model encoder-decoder pairs to map each model's hidden width into and out of the shared dictionary. During training, selection scores are normalized to ensure consistent latent selection across models, while magnitudes are preserved. The method also incorporates model dropout to facilitate single-model inference.
Results
SharedSAE retains 96.6% of the mean explained variance of dedicated SAEs and achieves cross-model activation correlations that are 1.8 times higher than those of separate SAEs aligned post hoc. The method allows for effective transfer of latent descriptions across models and enables new models to be adapted to the frozen dictionary with near-dedicated reconstruction quality.
Implications
The SharedSAE framework has significant implications for the interpretability of language models, allowing researchers to analyze and understand model activations more efficiently. It reduces redundancy in training separate SAEs for each model and enhances the ability to transfer knowledge across different architectures, potentially leading to more robust and interpretable AI systems.
PRICE: A Systematic Study of LLM Adaptation Choices for Bitcoin Price Forecasting
Large Language Models
Time Series
- PRICE integrates multiple adaptation strategies for improved Bitcoin price forecasting.
- Controlled ablation studies highlight the significant impact of each adaptation choice on forecasting accuracy.
- The framework demonstrates competitive performance against specialized time-series models.
- LoRA and recursive inference are key components that enhance model efficiency and accuracy.
Read more
PRICE: A Systematic Study of LLM Adaptation Choices for Bitcoin Price Forecasting
Summary
The paper presents PRICE, a systematic framework designed for adapting Large Language Models (LLMs) to short-term Bitcoin price forecasting. Recognizing the challenges posed by the extreme volatility and non-stationary dynamics of cryptocurrency markets, the authors explore how various adaptation choices impact forecasting performance. PRICE is built on a 4-bit quantized LLaMA-3 8B model and incorporates five key components: parameter-efficient fine-tuning using Low-Rank Adaptation (LoRA), recursive multi-step inference, integer-rounded numerical representation, Context-Task-Format (CTF) prompting, and exact zero-temperature decoding. Through controlled ablation studies, the authors demonstrate that each adaptation choice significantly contributes to forecasting accuracy and reliability. The findings reveal that LoRA facilitates efficient training on limited hardware, recursive inference enhances predictive accuracy, integer-rounded values reduce forecasting errors, CTF prompting outperforms traditional prompting methods, and zero-temperature decoding stabilizes outputs. Comparative evaluations against eight transformer-based and time-series foundation models show that PRICE achieves the lowest forecasting errors while maintaining robust performance across different evaluation periods. Notably, PRICE performs competitively against specialized time-series models despite being based on a text-pretrained model, underscoring the importance of adaptation choices in enhancing LLMs for numerical time series forecasting tasks.
Methodology
The methodology involves a systematic framework that integrates five components: parameter-efficient fine-tuning with LoRA, recursive multi-step inference, integer-rounded numerical representation, CTF prompting, and exact zero-temperature decoding. Controlled ablation studies are conducted to isolate the effects of each adaptation choice on forecasting performance.
Results
PRICE achieves the lowest forecasting errors on validation and test sets compared to eight transformer-based and time-series foundation models. The framework maintains robust performance across various evaluation periods, with significant improvements in accuracy and reliability attributed to the adaptation choices.
Implications
The findings suggest that LLMs can be effectively adapted for financial time series forecasting, particularly in volatile markets like cryptocurrencies. This work opens avenues for further research into LLM adaptations and their applications in other financial forecasting tasks.
Fractal basins trap latent reasoning
Theory
Optimization
- Reasoning models experience transient chaos, leading to overthinking and extended convergence times on difficult tasks.
- Fractal basins emerge in the dynamical systems of reasoning models, with complexity increasing with task difficulty.
- Basin entropy is introduced as a metric to measure the complexity of convergence basins, correlating with task difficulty.
- The study reveals that reasoning slowdowns are an inherent feature of problem hardness in AI models.
Read more
Fractal basins trap latent reasoning
Summary
This paper explores the dynamics of reasoning models in artificial intelligence, particularly focusing on the phenomenon of 'overthinking' where models take longer to converge on solutions for harder tasks. The authors demonstrate that reasoning models exhibit transient chaos, leading to the emergence of fractal basins in their dynamical systems. These fractal basins become more complex as task difficulty increases, indicating that reasoning slowdowns are a natural consequence of problem hardness. The study employs dynamical systems theory to analyze how initial latent states affect convergence times across various reasoning tasks, including Sudoku and maze solving. The findings reveal that even minor changes in initial conditions can significantly impact convergence times, suggesting a rich structure in the dynamics of reasoning models. The paper introduces basin entropy as a metric to quantify the complexity of these basins, showing a strong correlation between basin entropy and the number of iterations required for convergence. Overall, the research provides a mechanistic understanding of reasoning dynamics and highlights the importance of fractal basins in modern AI reasoning models.
Methodology
The authors utilized dynamical systems theory to analyze reasoning models by varying initial latent states and measuring convergence times across different tasks. They introduced basin entropy as a metric to quantify the complexity of convergence basins and conducted experiments on various reasoning tasks to observe the emergence of fractal basins.
Results
The study found that reasoning models produce fractal basins that become increasingly complex with task difficulty. There is a strong correlation between basin entropy and the number of iterations required for convergence, indicating that reasoning slowdowns are linked to the inherent complexity of the tasks being solved.
Implications
The insights from this research could lead to improved understanding and optimization of reasoning models in AI, potentially enhancing their efficiency and effectiveness in solving complex problems. It also opens avenues for further exploration of dynamical systems in AI reasoning.
Physics-Aware Random Walk Fingerprints for Scalable Power Grid Graph Classification
Graph Learning
- Introduces MC-PA-RWF, which incorporates physical edge states into random-walk dynamics for power grid classification.
- Demonstrates substantial improvements in classification accuracy over traditional topology-only RWF methods.
- Maintains competitive performance against advanced GNN models while being more interpretable and scalable.
- Utilizes multiple edge-weighted channels to capture operational conditions affecting system stability.
Read more
Physics-Aware Random Walk Fingerprints for Scalable Power Grid Graph Classification
Summary
This paper introduces Multi-Channel Physics-Aware Random Walk Fingerprints (MC-PA-RWF), a novel framework for graph-level representation of power grid systems that integrates physical edge states into random-walk dynamics. Traditional graph neural networks (GNNs) have shown strong performance in cascading-failure classification tasks but require extensive training and tuning, making them less interpretable. In contrast, MC-PA-RWF offers a lightweight and interpretable alternative by constructing multiple edge-weighted channels based on domain-relevant attributes, allowing for a more nuanced understanding of how operational conditions affect system stability. The method was evaluated on three benchmark systems from PowerGraph, demonstrating significant improvements over topology-only Random Walk Fingerprints (RWF) and competitive accuracy against various GNN architectures. The results indicate that MC-PA-RWF can effectively capture the propagation behavior influenced by physical states in power systems, making it a promising tool for cyber-physical graph mining.
Methodology
The proposed MC-PA-RWF framework constructs multiple edge-weighted channels from domain-relevant attributes of power systems. Each channel captures specific operational conditions, and channel-specific random-walk fingerprints are extracted and concatenated to form a compact graph-level representation. This method enhances the traditional RWF by integrating physical edge states into the random-walk process, allowing for a more comprehensive analysis of cascading failures in power grids.
Results
The experiments conducted on three PowerGraph benchmark systems revealed that MC-PA-RWF achieved balanced accuracy rates between 98.04% and 99.32%. It also improved failure-class F1 scores over the strongest GNN baseline by 1.60 to 5.84 percentage points, with statistically significant gains across all evaluated systems, demonstrating its effectiveness in capturing the dynamics of power grid operations.
Implications
The MC-PA-RWF framework has significant implications for the analysis and management of power systems, particularly in enhancing the interpretability and scalability of graph-based models for cascading-failure classification. It can be applied in real-time monitoring and risk assessment of power grid stability, aiding in the development of more resilient cyber-physical infrastructures.
Spectral-Target Physical Latent Structuring for JEPA-Style World Models
Robotics
Reinforcement Learning
Efficient ML
- Identification of 'physical representation laziness' as a failure mode in latent world models.
- Introduction of a lightweight Fourier auxiliary head to enforce physical structuring of latent representations.
- Significant improvements in planning success rates in dynamic environments with the proposed method.
- Enhanced latent space correlations with physical properties, indicating better representation.
Read more
Spectral-Target Physical Latent Structuring for JEPA-Style World Models
Summary
This paper addresses a critical limitation in latent world models, particularly in dynamic environments, where existing architectures like LeWorldModel (LeWM) exhibit a failure mode termed 'physical representation laziness.' This phenomenon occurs when the learned latent states do not collapse but fail to capture essential physical properties, leading to planning failures. To mitigate this issue, the authors propose a novel training-time auxiliary supervision technique using a lightweight 'Fourier auxiliary head.' This head enforces a physically-informed structuring of the latent space without incurring additional inference costs. The experimental results demonstrate that this auxiliary head significantly enhances planning success rates in dynamic environments, even when the baseline model does not show representation laziness. Furthermore, the method improves latent space correlations with key physical properties, indicating a more structured representation conducive to effective planning. The findings also highlight the auxiliary head's effectiveness in low-data scenarios, showcasing its potential to enhance data efficiency while avoiding representation laziness in latent world models.
Methodology
The authors propose a Fourier auxiliary head that provides additional supervision during training, guiding the encoder to preserve relevant physical information in the latent space. This head is a small multilayer perceptron trained to recover spectral features of important physical quantities from the encoded latent state. The method is evaluated through experiments comparing planning success rates and latent space correlations in various environments.
Results
The introduction of the Fourier auxiliary head led to substantial improvements in planning success rates, particularly in dynamic environments where the baseline model struggled. The method also resulted in better latent space correlations with key physical properties, indicating a more effective representation for planning tasks. In low-data scenarios, the auxiliary supervision proved especially impactful in enhancing success rates.
Implications
The findings suggest that incorporating physically-informed structuring in latent world models can significantly enhance their effectiveness in planning tasks, especially in dynamic and data-limited environments. This approach could be applied to various applications in robotics and autonomous systems where understanding physical interactions is crucial.
GNN-Guided Graph Coarsening and Adaptive QUBO Penalties for the Capacitated Vehicle Routing Problem with Time Windows on a Quantum Annealer
Optimization
Graph Learning
- Introduces adaptive penalty calibration to improve QUBO formulations for CVRPTW.
- Utilizes a GNN for tuning-free graph coarsening, achieving 100% feasibility on certain instance families.
- Demonstrates significant reductions in constraint violations and QUBO size.
- Validates the approach on quantum hardware, showing improved feasibility rates.
Read more
GNN-Guided Graph Coarsening and Adaptive QUBO Penalties for the Capacitated Vehicle Routing Problem with Time Windows on a Quantum Annealer
Summary
This paper addresses the challenges of solving the Capacitated Vehicle Routing Problem with Time Windows (CVRPTW) using quantum annealing, particularly focusing on the limitations of existing graph coarsening heuristics that require instance-specific tuning. The authors propose a novel approach that combines adaptive penalty calibration and a graph neural network (GNN) for graph coarsening. The adaptive penalty calibration improves the QUBO formulation by selectively managing constraint penalties, significantly reducing raw constraint violations. The GNN-guided coarsening replaces the traditional hand-tuned merge score with a single configuration applicable across all instance families, achieving higher feasibility rates in the solutions. The methodology is validated on the Solomon benchmark using both simulated annealing and a D-Wave Advantage2 quantum processor, demonstrating substantial improvements in solution feasibility and QUBO size. The findings indicate that the proposed techniques enhance the tractability of CVRPTW on quantum hardware, although a classical repair step remains necessary for optimal feasibility.
Methodology
The authors employed a two-pronged approach: first, they implemented adaptive penalty calibration to optimize the QUBO formulation by managing constraint penalties effectively. Second, they developed a GNN-based merge scorer for graph coarsening, which eliminates the need for instance-specific tuning. The methods were evaluated using simulated annealing and on a D-Wave Advantage2 quantum processor, focusing on the Solomon benchmark for CVRPTW.
Results
The adaptive penalty calibration reduced the mean number of raw constraint violations from 33.0 to 0.06, while the GNN-guided coarsening achieved 100% feasibility for N=10 and improved feasibility rates across larger instance sizes. The QUBO size was reduced by approximately 5-6 times, making larger instances solvable. On quantum hardware, the conditioned formulation yielded a 39% feasibility rate compared to 0.02% for the poorly conditioned version.
Implications
The findings suggest that the proposed methods can significantly enhance the performance of quantum annealers for combinatorial optimization problems like CVRPTW. This could lead to more efficient solutions in logistics and transportation, where such routing problems are prevalent. Additionally, the techniques may be applicable to other optimization problems that can be formulated as QUBOs.
Solution-space heterogeneity shapes federated learning dynamics across partial differential equations
Federated Learning
- Introduces solution-space PDE-Dirichlet for federated learning in PDEs.
- Establishes a relationship between population allocation heterogeneity and Dirichlet concentration.
- Demonstrates that lower concentration increases optimization heterogeneity.
- Findings indicate task-dependent degradation in final error.
Read more
Solution-space heterogeneity shapes federated learning dynamics across partial differential equations
Summary
This paper addresses the challenges of federated learning (FL) in the context of partial differential equations (PDEs), particularly focusing on the issue of non-independent and identically distributed (non-IID) data. The authors introduce a novel protocol called solution-space PDE-Dirichlet, which transforms continuous supervised responses into reusable solution bins and quantifies the heterogeneity among clients using optimal transport metrics. They derive a relationship between population allocation heterogeneity and Dirichlet concentration, establishing conditions under which response heterogeneity leads to gradient disagreement and parameter divergence. The study evaluates the proposed method across seven PDE tasks and three neural operator families, demonstrating that lower concentration increases solution distance and optimization heterogeneity. The findings reveal that the degradation in final error is task-dependent, with significant effects observed in specific cases, such as low-viscosity Burgers equations. This work provides a reproducible geometric mechanism for evaluating non-IID federated PDE learning, contributing to the understanding of statistical heterogeneity in federated learning frameworks.
Methodology
The authors developed a protocol that normalizes and discretizes continuous solution fields into pseudo-classes, utilizing a symmetric Dirichlet distribution for client allocation. They employed optimal transport metrics to quantify response-space heterogeneity and analyzed the impact of response heterogeneity on gradient disagreement and parameter divergence across various PDE tasks and neural operator families.
Results
The study found that lower Dirichlet concentration consistently led to increased realized solution distance and optimization heterogeneity. The most significant degradation in final error was observed in low-viscosity Burgers equations, with an increase of 4.157 percentage points under the most heterogeneous conditions. The results highlighted that additional communication or smoother dynamics could mitigate the final error gap despite persistent parameter separation.
Implications
This research has implications for improving federated learning protocols in scientific machine learning, particularly in settings where data cannot be centralized. The findings can enhance the understanding of how statistical heterogeneity affects model training and performance in collaborative environments, potentially leading to more effective federated learning strategies in various scientific and engineering applications.
Fast Surrogate Modeling of Excitable and Oscillatory FitzHugh-Nagumo Dynamics with Parametric Neural Operators
Efficient ML
Theory
Time Series
- Introduction of parameter-conditioned Fourier Neural Operators for modeling the FitzHugh-Nagumo system.
- Achieved sub-0.1% relative L2 error in the oscillatory regime, significantly faster than classical solvers.
- Demonstrated strong generalization across the 5D parameter space and effective extrapolation beyond training bounds.
- Accurately captured dynamics in both oscillatory and excitable regimes, including key neuronal firing characteristics.
Read more
Fast Surrogate Modeling of Excitable and Oscillatory FitzHugh-Nagumo Dynamics with Parametric Neural Operators
Summary
This paper presents a novel approach to modeling the FitzHugh-Nagumo (FHN) system, a simplified representation of neuronal voltage dynamics, using parameter-conditioned Fourier Neural Operators (FNOs). The authors aim to address the computational challenges associated with classical finite-difference solvers, which become prohibitively expensive when exploring the 5D physiological parameter space of the FHN model. By employing feature-wise linear modulation (FiLM) to condition each Fourier layer on the parameter vector, the trained FNO serves as a fast, differentiable surrogate for the FHN voltage and recovery fields in a one-dimensional spatial domain. The study includes a bifurcation analysis that delineates the oscillatory and excitable regimes of the model. The results demonstrate that the surrogate achieves sub-0.1% relative L2 error in the oscillatory regime and operates nearly three orders of magnitude faster than traditional methods. Furthermore, it generalizes well across the parameter space and accurately captures key dynamics in the excitable regime, including the firing threshold and conduction-velocity law. This work lays the groundwork for future applications in inverse parameter inference and closed-loop neuromodulation.
Methodology
The authors trained parameter-conditioned Fourier Neural Operators (FNOs) using feature-wise linear modulation (FiLM) to incorporate the 5D parameter vector into each Fourier layer. They conducted a bifurcation analysis to identify the oscillatory and excitable regimes and characterized the error sensitivity across the parameter space.
Results
The trained FNO achieved sub-0.1% relative L2 error in the oscillatory regime and demonstrated nearly three orders of magnitude speedup compared to finite-difference methods. It also maintained strong generalization capabilities across the parameter space and accurately reproduced critical dynamics in the excitable regime, including the conduction-velocity law.
Implications
This work provides a fast and efficient surrogate modeling approach for exploring complex neuronal dynamics, which can facilitate tasks such as inverse parameter inference and closed-loop neuromodulation in neuroscience applications.
Optimal Rates for Agentic Networked Information Aggregation
Theory
Federated Learning
Optimization
- Improved bounds on excess mean squared error for agentic information aggregation models.
- Establishment of M-covered paths and their significance in reducing prediction error.
- Extension of results from regression to logistic classification with binary cross-entropy loss.
- Demonstration of geometric contraction of excess error along the agent path.
Read more
Optimal Rates for Agentic Networked Information Aggregation
Summary
This paper builds on the work of Kearns, Roth, and Ryu to investigate information aggregation in a networked learning model where agents only see a subset of data and pass on their conclusions. The authors focus on a linear regression problem with mean squared error (MSE) loss, analyzing a directed acyclic graph (DAG) of agents. They introduce the concept of M-covered paths, where every block of M consecutive agents collectively sees all raw features. The authors improve upon previous bounds on excess mean squared error, demonstrating that the correct rate is constant up to depth M^2 and Θ(M^2/D) beyond that. They also provide a sharper analysis of the cyclic instance, improving its lower bound to Ω(√(M/D)) for D < M^2. Additionally, they extend their findings to logistic classification, showing that similar upper and lower bounds apply. The results indicate that the depth of the agent network significantly impacts prediction accuracy, and the authors provide a comprehensive analysis of the bounds on excess error for both regression and classification tasks.
Methodology
The authors analyze a networked learning model where agents are arranged in a directed acyclic graph (DAG). Each agent fits a linear predictor based on its inputs and the predictions of its predecessors. They derive new upper and lower bounds for excess mean squared error through theoretical analysis and construction of specific instances that demonstrate the tightness of these bounds.
Results
The paper presents a refined upper bound of O(M^2/D) for the excess mean squared error of the last agent on an M-covered path of depth D. For depths less than M^2, the authors establish a lower bound of Ω(√(M/D)), and for depths greater than or equal to M^2, they show a lower bound of Ω(M^2/D). These results close the gaps in previous work and confirm the optimal rates for both regression and logistic classification.
Implications
The findings have significant implications for the design of distributed AI systems, particularly in scenarios where agents operate with limited data visibility. Understanding the optimal rates of information aggregation can enhance the performance of decentralized learning systems, making them more efficient and effective in real-world applications such as federated learning and social learning.
Locating and Steering Refusal Beyond Attention
NLP
Large Language Models
Interpretability
- Refusal is a shared representation across different model architectures.
- The refusal direction must be read at the fresh write site for effective control.
- A detector-triggered gate can significantly reduce adversarial attack success.
- Safety tooling can be adapted across architectures by focusing on the refusal direction.
Read more
Locating and Steering Refusal Beyond Attention
Summary
This paper investigates the concept of refusal in language models, particularly focusing on how this behavior is represented across different architectures, such as transformers and state-space models (SSMs). The authors demonstrate that refusal is not architecture-specific but rather a shared representation that can be aligned across different model types through a rigid rotation of their representation spaces. They establish that the refusal direction must be read at the fresh write site of the model rather than the accumulated residual stream. The study further shows that applying a detector-triggered gate can effectively control refusals across various architectures, significantly reducing the success of adversarial attacks aimed at bypassing model refusals. The findings suggest that safety tooling for language models can be adapted to new architectures by re-estimating the refusal direction at the appropriate readout site, rather than needing to be completely rebuilt for each new model type.
Methodology
The authors employed a combination of theoretical analysis and empirical testing across various language model architectures, including transformers and state-space models. They utilized a rigid rotation technique to align representation spaces and conducted experiments to evaluate the effectiveness of a detector-triggered gate in controlling refusals during adversarial attacks.
Results
The study found that a single rigid rotation could align the refusal representation across different architectures, allowing for effective transfer of the refusal direction. The application of a detector-triggered gate reduced the success rate of jailbreak attacks from 15.0% to 1.0% on the SSM model and similarly lowered attack success across other architectures. The automated harm detection mechanism achieved a high agreement rate with human raters, validating its effectiveness.
Implications
The findings have significant implications for the development of safer language models, suggesting that refusal mechanisms can be effectively transferred across architectures. This could streamline the process of enhancing model safety and robustness against adversarial attacks, making it easier to deploy new architectures without starting from scratch in safety tooling.
Mitra-v2 Technical Report
Optimization
Efficient ML
Theory
- Mitra-v2 achieves state-of-the-art performance on diverse real-world tasks.
- The model is trained solely on synthetic data with an expanded pretraining distribution.
- Mitra-v2 utilizes a small 2D Transformer backbone, enabling efficient learning from larger datasets.
- It significantly outperforms larger models like TabFM and EXAONE while being much smaller.
Read more
Mitra-v2 Technical Report
Summary
The Mitra-v2 Technical Report introduces a new tabular foundation model, Mitra-v2, which achieves state-of-the-art performance on various real-world classification and regression tasks, including credit-risk scoring, clinical prediction, equipment-failure detection, and house-price estimation. Unlike its predecessor, Mitra-v1, Mitra-v2 is trained exclusively on synthetic data with a significantly larger and more diverse pretraining distribution. Utilizing a compact 2D Transformer backbone, the model is capable of handling longer contexts and larger feature spaces. Enhanced optimization techniques allow it to learn effectively from a broader task distribution. Evaluated on the TabArena and TALENT benchmarks, Mitra-v2 outperforms existing models, including industry-scale models like TabFM and EXAONE, while being substantially smaller in size (77M parameters compared to 1.6B for TabFM). It excels particularly in multiclass classification tasks, achieving top rankings even when pretrained on tasks with fewer classes. The report concludes by releasing the model weights, inference and fine-tuning code, and evaluation results under the Apache-2.0 license, marking Mitra-v2 as one of the most robust and versatile open tabular foundation models available.
Methodology
Mitra-v2 employs a 2D Transformer architecture and is pretrained on a diverse synthetic dataset. It incorporates advanced optimization techniques to enhance learning from a broader range of tasks and supports longer contexts and larger feature spaces. The model's design allows it to adapt to various prediction tasks without extensive retraining.
Results
Mitra-v2 achieved the highest Elo scores on the TabArena benchmark, surpassing TabFM, EXAONE, and TabPFN-3 in both classification and regression tasks. It ranked first in multiclass classification tasks with more than ten classes, despite being pretrained on tasks with a maximum of ten classes. On the TALENT benchmark, it maintained a leading position, outperforming other models in various categories.
Implications
Mitra-v2's performance indicates its potential for broad applicability in real-world tabular data scenarios, making it a valuable tool for industries such as finance, healthcare, and real estate. Its efficiency and effectiveness could lead to more accessible machine learning solutions for practitioners dealing with tabular data.
A Fairness Audit of the Duckworth-Lewis-Stern Method: Format-Specific and Gender-Differential Bias, with an Interpretable Calibration Layer for Cricket Target Revision
Interpretability
- First systematic audit of DLS prediction bias using extensive match data.
- Quantifies gender-differential bias in DLS, with women's matches miscalibrated by +6.13 runs.
- Introduces DLS-Cal, a calibration layer that significantly reduces prediction bias.
- Presents the Win-Flip Rate, a new metric for assessing fairness in target revision.
Read more
A Fairness Audit of the Duckworth-Lewis-Stern Method: Format-Specific and Gender-Differential Bias, with an Interpretable Calibration Layer for Cricket Target Revision
Summary
This paper presents the first large-scale empirical audit of the Duckworth-Lewis-Stern (DLS) method, which is used to revise target scores in rain-interrupted limited-overs cricket matches. Utilizing a dataset of 8,150 international matches from Cricsheet, the author investigates the prediction bias of DLS by sampling synthetic interruption points and comparing the method's resource-based projections against actual runs scored. The findings reveal that DLS exhibits systematic bias that varies with match state, notably over-predicting runs in the death overs of Twenty20 matches and under-predicting in collapse scenarios during one-day matches. Additionally, the study identifies a significant gender-differential bias, where the same resource table used for both men's and women's cricket miscalibrates women's matches, resulting in an average over-prediction of +6.13 runs. To address these biases, the author introduces DLS-Cal, an interpretable calibration layer that corrects DLS predictions, reducing absolute bias by 31% for ODIs and 19% for T20Is. A gender-aware variant further reduces women's ODI residual bias from +6.19 to +0.65 runs. The paper also introduces the Win-Flip Rate, a new fairness metric for target revision, and provides resources for reproducible research.
Methodology
The study employs a large dataset of ball-by-ball records from international cricket matches to audit the DLS method. It uses synthetic interruption sampling to evaluate DLS's predictive performance across various match states. The reliability-diagram approach is utilized for calibration, and a new fairness metric, the Win-Flip Rate, is introduced to assess bias in target revisions.
Results
The audit reveals significant bias in DLS predictions, with a 137-run range of bias across match states. The introduction of DLS-Cal reduces absolute bias by 31% for ODIs and 19% for T20Is, while the gender-aware variant significantly lowers the bias in women's ODIs from +6.19 to +0.65 runs.
Implications
The findings highlight the need for more equitable calibration methods in sports analytics, particularly in contexts where gender differences exist. The DLS-Cal layer and the Win-Flip Rate metric can be applied to improve fairness in target revision methods, potentially influencing policy changes in cricket and other sports.
Training Large Language Models for Small-Molecule Design with Synthetic Task Scaling
Large Language Models
Reinforcement Learning
Generative Models
- LLMs can learn molecular design strategies from cheaper synthetic tasks that generalize to expensive lead optimization settings.
- Curriculum-based training that gradually incorporates challenging tasks leads to superior performance compared to training solely on high-fidelity tasks.
- A 35-billion-parameter LLM outperformed larger frontier models in generating promising molecular candidates.
Read more
Training Large Language Models for Small-Molecule Design with Synthetic Task Scaling
Summary
This paper explores the use of large language models (LLMs) for small-molecule drug design, focusing on the challenge of training these models to navigate complex chemical spaces efficiently. The authors propose a novel approach that employs synthetic task scaling, where LLMs are initially trained on cheaper, low-fidelity synthetic design tasks before transitioning to more expensive, high-fidelity molecular lead optimization tasks. This curriculum-based training strategy allows the models to learn foundational design principles that can be applied to more complex scenarios, ultimately improving their performance in high-cost experimental settings. The results demonstrate that a smaller LLM, with 35 billion parameters, can outperform larger models that have been extensively trained on small-molecule tasks, indicating the effectiveness of the proposed training recipes in enhancing the models' ability to generate viable drug candidates.
Methodology
The authors developed a spectrum of post-training recipes that combined low-cost, low-fidelity design tasks with high-cost, high-fidelity tasks. They employed a curriculum-based approach to gradually introduce more complex tasks, allowing the LLMs to build upon foundational knowledge gained from simpler tasks.
Results
The study found that LLMs trained with a curriculum-based approach significantly outperformed those trained only on high-fidelity tasks. The smaller LLM achieved better results in lead optimization tasks than larger models, demonstrating the efficacy of synthetic task scaling in training LLMs for drug design.
Implications
This research suggests that synthetic task scaling can be a viable strategy for training LLMs in high-cost experimental scenarios, potentially accelerating the drug discovery process by enabling more efficient exploration of chemical space.
SMILE: Bridging Continuous Optimization and Discrete Symbolic Recovery
Optimization
Interpretability
Theory
- SMILE unifies continuous optimization and discrete symbolic recovery in symbolic regression.
- The framework employs a three-stage pipeline: structural analysis, continuous optimization, and symbolic recovery.
- SMILE achieves high robustness to noise and significantly reduces expression complexity and discovery time.
- The method demonstrates the ability to produce interpretable closed-form expressions directly from trained networks.
Read more
SMILE: Bridging Continuous Optimization and Discrete Symbolic Recovery
Summary
The paper introduces SMILE, a novel framework for symbolic regression (SR) that combines continuous optimization with discrete symbolic recovery. Traditional SR methods face challenges such as slow convergence in combinatorial search spaces and inadequate exploitation of compositional structures in data. SMILE addresses these issues through a three-stage process: (1) structural analysis of data to identify the compositional hierarchy of the target expression, (2) continuous optimization to learn parameters of a neural network with interpretable activations, and (3) symbolic recovery via structured pruning and coefficient optimization. The framework utilizes a feedforward network with fixed symbolic activations (Sine, Multiplication, Identity, Logarithm, Exponential) to efficiently represent complex expressions. SMILE demonstrates its robustness by achieving the highest symbolic solution rates even at high noise levels, while also producing simpler expressions in significantly less time compared to existing methods. The paper validates each component of SMILE through extensive experiments on standard SR benchmarks, showcasing its effectiveness and efficiency in recovering interpretable mathematical expressions from data.
Methodology
SMILE employs a hybrid approach that begins with structural analysis of input data to identify the compositional hierarchy of the target expression. It then uses a feedforward neural network with fixed symbolic activations for continuous optimization, followed by a structured pruning process for symbolic recovery. This methodology allows for efficient exploration of the search space without the need for user-specified structural hyperparameters.
Results
SMILE was evaluated on the SRBench dataset, achieving the highest symbolic solution rates at large noise levels. It consistently produced simpler expressions compared to competing methods and reduced the time required for discovery from hours to minutes. The framework's robustness and efficiency were validated through extensive experiments and ablation studies.
Implications
The SMILE framework has potential applications in various scientific fields where interpretable mathematical models are essential, such as fluid mechanics, molecular systems, and materials science. Its ability to recover closed-form expressions efficiently could enhance the understanding of complex systems and facilitate model development in engineering and scientific research.
Hakken: Predicting future discoveries to fill the gaps in today's knowledge
Graph Learning
Large Language Models
NLP
- Hakken predicts novel scientific relationships using a transformer-based model and knowledge graphs.
- The system establishes a new benchmark for time-aware multi-label relation prediction in the biomedical field.
- Hakken generated 1.5 million hypotheses, with several validated through collaboration with biologists.
- Two significant undocumented interactions were confirmed, enhancing understanding in biomedical science.
Read more
Hakken: Predicting future discoveries to fill the gaps in today's knowledge
Summary
The paper introduces Hakken, a novel AI system designed for knowledge prediction, which aims to enhance scientific discovery by identifying new relationships between scientific concepts that are not currently documented. Hakken employs a transformer-based model that integrates temporal sequences of knowledge graphs derived from extensive research literature, combined with the semantic knowledge of large language models (LLMs). This system is particularly focused on the biomedical domain, where it establishes a new benchmark for time-aware multi-label relation prediction. Hakken's predictive capabilities were validated through the identification of 1.5 million hypotheses related to aging, with qualitative assessments from biologists leading to the empirical validation of three hypotheses. Notably, two significant interactions were confirmed: between TP53 and BAMBI, and between RAF1 and TNF, both of which have implications for drug discovery. The paper emphasizes Hakken's ability to provide not only predictions but also explanations for these predictions, thereby assisting researchers in evaluating and validating new scientific insights.
Methodology
Hakken utilizes a unified temporal graph and language model called THiGERLLM, which predicts future relation labels in a multi-label setting. It models temporal evolution using a temporal Graph Neural Network and integrates LLM semantic knowledge with the graph structure. Additionally, it employs a model-agnostic explanation framework named PHELInE to elucidate the predictions.
Results
Hakken's prediction model achieved a new benchmark in time-aware multi-label relation prediction, demonstrating coherence and informativeness over extended historical data. It successfully identified and validated two significant gene interactions, contributing new insights to the biomedical literature.
Implications
Hakken has the potential to significantly accelerate scientific discovery by providing researchers with novel insights and hypotheses that can lead to empirical validation. Its application in drug discovery could facilitate the identification of new therapeutic targets and enhance the understanding of complex biological interactions.
Beyond Homoscedasticity: Decoupled Uncertainty Optimization for Deep Imbalanced Regression
Computer Vision
Optimization
Theory
- DUO framework explicitly models instance-level predictive uncertainty in DIR tasks.
- Decoupled mean-variance optimization enhances learning for tail samples.
- Distribution-guided contrastive learning alleviates feature entanglement.
- Achieves state-of-the-art results on multiple DIR benchmarks.
Read more
Beyond Homoscedasticity: Decoupled Uncertainty Optimization for Deep Imbalanced Regression
Summary
This paper addresses the challenges of Deep Imbalanced Regression (DIR), which is common in continuous prediction tasks where tail samples are scarce but valuable. Traditional methods often assume uniform uncertainty across samples, leading to suboptimal performance, especially for underrepresented tail samples. The authors introduce DUO, a novel framework that models regression targets as conditional Gaussian distributions, allowing for instance-level predictive uncertainty. This approach employs decoupled mean-variance optimization to enhance learning signals for tail samples and incorporates a distribution-guided contrastive learning mechanism to improve feature representation. The proposed method demonstrates superior performance on various DIR benchmarks, achieving the best few-shot bMAE and GM metrics while maintaining competitiveness in few-shot MAE.
Methodology
The DUO framework models regression targets as conditional Gaussian distributions to capture instance-level uncertainty. It utilizes decoupled mean-variance optimization to enhance the learning signal for tail samples and incorporates a distribution-guided contrastive learning mechanism to construct adaptive positive and negative pairs based on sample distribution overlaps.
Results
DUO outperforms existing methods on DIR benchmarks such as IMDB-WIKI-DIR, AgeDB-DIR, and AAV2-DIR, achieving the best few-shot bMAE and GM metrics while remaining competitive in few-shot MAE, demonstrating its effectiveness in addressing the challenges posed by imbalanced data distributions.
Implications
The proposed DUO framework has significant implications for applications requiring reliable predictions in imbalanced datasets, such as age estimation, depth prediction, and biological data analysis. By improving the handling of uncertainty in predictions, it can enhance the performance of models in real-world scenarios where data is often skewed.
RegionFed: Federated Learning for Personalized Query Understanding in Heterogeneous Retail Environments
Federated Learning
NLP
Large Language Models
- Introduces RegionFed, a gradient-level federated learning framework for personalized query understanding.
- Addresses challenges of data heterogeneity in retail environments by using gradient conflict for personalization.
- Achieves architecture-agnostic deployment across various models without code changes.
- Demonstrates significant accuracy improvements while ensuring differential privacy.
Read more
RegionFed: Federated Learning for Personalized Query Understanding in Heterogeneous Retail Environments
Summary
The paper presents RegionFed, a novel federated learning framework designed to enhance personalized query understanding in heterogeneous retail environments. Traditional federated learning methods struggle with data heterogeneity and often produce global models that fail to capture regional nuances, leading to poor performance. Existing personalized federated learning approaches, which operate at the parameter level, encounter significant challenges when applied to modern transformer architectures due to shared embeddings and LayerNorm interactions. RegionFed addresses these issues by operating at the gradient level, utilizing the â„“2 conflict between regional and global gradients to diagnose heterogeneity, determine the most efficient personalization strategy for each region, and control personalization strength adaptively. This gradient-level approach allows RegionFed to be architecture-agnostic, enabling deployment across various models such as T5 and RoBERTa without code modifications. The framework achieves significant improvements in accuracy while maintaining differential privacy, demonstrating its effectiveness across multiple datasets and model architectures.
Methodology
RegionFed operates at the gradient level rather than the parameter level, using â„“2 gradient conflict as a signal for adaptive personalization. It dynamically routes regions to the most efficient personalization strategy based on the detected gradient conflicts, ensuring that the framework is robust across different model architectures. The method includes differential privacy mechanisms to maintain privacy guarantees during training.
Results
RegionFed-Meta achieved an accuracy of 92.27% across three public datasets, closely approaching the centralized upper bound of 92.04%. The framework demonstrated O(1/√T) convergence and provided (ϵ≈0.60)-differential privacy. It outperformed traditional federated learning methods like FedAvg and FedProx by 12-25 percentage points on T5-Small, while existing parameter-level methods collapsed to below 10% accuracy on transformers.
Implications
RegionFed has significant implications for retail search systems, allowing for effective personalization while adhering to privacy regulations. Its architecture-agnostic nature makes it suitable for deployment in diverse environments, potentially improving user experience in large-scale retail platforms. The framework can also be adapted for other applications requiring personalized models in heterogeneous data settings.
Beyond a Universal Forecasting Selector: Demand-Conditioned Model Selection across Demand Patterns and Horizons
Time Series
- No universal forecasting selector consistently outperforms others across diverse demand conditions.
- CCG-AHSC and CCG-AHSCD are effective for Smooth and Erratic demand patterns.
- OWA and ERA perform better in Intermittent and Lumpy demand scenarios.
- Selector effectiveness varies with historical data availability and forecasting horizons.
Read more
Beyond a Universal Forecasting Selector: Demand-Conditioned Model Selection across Demand Patterns and Horizons
Summary
This paper addresses the challenges of forecasting-model selection in heterogeneous demand environments, where the optimal model may vary based on demand structure, data availability, and forecasting horizon. The authors investigate whether the model-selection mechanism itself should adapt to these contexts. They compare five selection mechanisms—RMSSE, ERA, OWA, CCG-AHSC, and CCG-AHSCD—across 24 optimized forecasting models, nine datasets, and various training-testing partitions over forecasting horizons from 1 to 12 cycles. Performance is evaluated using Global Relative Accuracy (GRA) and statistical tests. The findings reveal that no single selector consistently outperforms others across all conditions. CCG-AHSC and CCG-AHSCD are more effective for Smooth and Erratic demand patterns, while OWA and ERA excel in Intermittent and Lumpy scenarios. The study emphasizes the importance of a context-dependent approach to model selection, suggesting that the effectiveness of selectors varies with historical data availability and forecasting horizons. This research shifts the focus from merely identifying the best forecasting model to understanding which selection mechanism is most suitable under specific conditions.
Methodology
The authors conducted a comparative analysis of five automatic model-selection mechanisms across 24 optimized forecasting models and nine heterogeneous datasets. They utilized Global Relative Accuracy (GRA) as a performance measure and employed statistical tests to evaluate selector performance across different demand classes and experimental setups.
Results
The study found that no single model-selection mechanism consistently outperformed the others across all tested conditions. CCG-AHSC and CCG-AHSCD showed superior performance for Smooth and Erratic demand patterns, while OWA and ERA were more suitable for Intermittent and Lumpy demand. The effectiveness of selectors was influenced by the availability of historical data and the forecasting horizon.
Implications
The findings suggest that organizations should adopt a context-dependent approach to model selection in demand forecasting, tailoring their selection mechanisms to specific demand patterns and operational contexts. This could lead to improved forecasting accuracy and better inventory management.
FedDRAW: Federated Dual Reputation Annealing Weighting for Heterogeneous Multi-Institutional Chest Radiograph Classification
Federated Learning
Computer Vision
- FedDRAW adjusts aggregation weights based on model similarity rather than solely on data size.
- The method employs dual annealing schedules to balance influence from smaller and larger hospitals.
- FedDRAW achieved the highest average rank compared to eight other federated learning methods.
- The approach addresses biases in federated learning, leading to less biased diagnostic models.
Read more
FedDRAW: Federated Dual Reputation Annealing Weighting for Heterogeneous Multi-Institutional Chest Radiograph Classification
Summary
The paper introduces FedDRAW, a novel federated learning approach designed to improve chest radiograph classification across heterogeneous multi-institutional datasets while addressing the limitations of traditional federated averaging methods. FedDRAW innovatively adjusts client aggregation weights based on both data size and model similarity, allowing for a more equitable influence of smaller hospitals during the training process. The method employs two coupled annealing schedules: an inner schedule that transitions client reputation from data size to model similarity, and an outer deferred schedule that normalizes weights to uniformity at convergence. This approach mitigates the bias introduced by larger institutions and enhances the overall model performance. The authors evaluated FedDRAW on two chest radiograph datasets (CheXpert and ChestMNIST) across twelve simulated client-partition scenarios, demonstrating that it outperformed seven state-of-the-art federated learning methods in terms of average rank, AUC, and the geometric mean of sensitivity and specificity, with statistically significant improvements confirmed through Friedman tests and Nemenyi post-hoc analysis.
Methodology
FedDRAW utilizes a federated learning framework where hospitals train a shared model locally and send updates to a central server. The aggregation weights are determined through a combination of data-size prior and cosine similarity between client and global model parameters, adjusted via two annealing schedules to ensure equitable influence during training.
Results
FedDRAW demonstrated superior performance, achieving the highest average rank across all evaluated methods on both AUC and geometric mean metrics. The results were statistically significant, indicating that the proposed method effectively reduces bias and improves classification accuracy in heterogeneous settings.
Implications
The findings suggest that FedDRAW could enhance collaborative medical imaging diagnostics by enabling equitable contributions from institutions of varying sizes, ultimately leading to more robust and generalized AI models in healthcare applications.