AI-generated summaries
Today's ML research,
without the noise.
Daily summaries of the latest machine learning papers from arXiv, processed every 8 hours.
50
Papers today
8h
Update frequency
7
Days of history
An Introduction to Bayesian and Frequentist Simulation-Based Inference with Machine Learning
Theory
- Overview of Bayesian and frequentist frameworks for simulation-based inference.
- Introduction of machine learning techniques for parameter estimation.
- Discussion on the validation of inference results and limitations of SBI.
- Application of SBI methods to Empirical Bayes and unfolding tasks.
Read more
An Introduction to Bayesian and Frequentist Simulation-Based Inference with Machine Learning
Summary
This paper provides a comprehensive overview of simulation-based inference (SBI) using machine learning within both Bayesian and frequentist statistical frameworks. The authors discuss the significance of SBI in addressing inverse problems prevalent in science and engineering, such as parameter estimation and the inversion of detector effects. They introduce key machine learning techniques, including neural posterior estimation and neural likelihood estimation, which facilitate parameter estimation in these frameworks. The paper also explores the application of these methods to Empirical Bayes and unfolding tasks. Furthermore, the authors emphasize the importance of validating inference results and outline the limitations associated with machine learning-based SBI. The article serves as a technical introduction to the methodologies rather than a survey of applications, contributing to the VERaiPHY initiative aimed at establishing verification and validation standards for machine learning in physics.
Methodology
The authors review the statistical frameworks of Bayesian and frequentist inference, detailing how machine learning techniques such as density estimation and likelihood information can be integrated into simulation-based inference. They also discuss validation techniques to ensure the reliability of inference results.
Results
The paper does not present specific experimental results but rather synthesizes existing methodologies and frameworks, highlighting the applicability of machine learning techniques in simulation-based inference and the importance of validation in scientific inference.
Implications
The findings suggest that machine learning can significantly enhance the efficiency and accuracy of simulation-based inference in various scientific fields, particularly in astrophysics and particle physics. The establishment of validation standards could lead to more robust applications of AI in these domains.
Class-Balanced Softmax: A Bayes Theory-Based Method for Long-Tailed Recognition
Computer Vision
Theory
Efficient ML
- Introduction of Class-Balanced Softmax (CBS) as a logit adjustment method for imbalanced classification.
- Characterization of the preference issue in models trained on imbalanced data, highlighting its impact on tail classes.
- Development of a novel metric, model imbalance level (I), to quantify the preference issue.
- Extensive experimental validation showing CBS's superior performance on long-tailed benchmarks.
Read more
Class-Balanced Softmax: A Bayes Theory-Based Method for Long-Tailed Recognition
Summary
This paper addresses the challenges of imbalanced datasets in deep learning, particularly in image classification tasks where some classes have significantly more training samples than others. The authors propose a novel method called Class-Balanced Softmax (CBS), which is based on a theoretical Bayesian framework and a heuristic power-law assumption. CBS aims to improve the performance of models on tail classes, which often suffer from lower accuracy due to their limited training data. The method introduces a simple logit adjustment that can be easily integrated into existing deep learning pipelines without additional computational overhead. The authors also identify a phenomenon termed the 'preference issue,' where models trained on imbalanced data exhibit higher training errors and generalization gaps for tail classes. To quantify this issue, they introduce a new metric called model imbalance level (I). Through extensive experiments on large-scale datasets, including ImageNet-LT and iNaturalist2018, the authors demonstrate that CBS outperforms existing methods, including Balanced Softmax, and effectively mitigates the preference issue, revealing fundamental limitations of softmax-based models in imbalanced settings.
Methodology
The authors propose CBS, which modifies the raw logit outputs of a model by introducing a power-law calibration hyperparameter. This adjustment is computationally inexpensive and integrates seamlessly into existing models. They also conduct a theoretical analysis of gradient impacts due to imbalanced training data and validate their findings through extensive experiments on various datasets.
Results
The results indicate that CBS significantly outperforms existing methods, including Balanced Softmax, across multiple long-tailed benchmarks. The proposed method effectively addresses the preference issue, leading to improved accuracy for tail classes and reduced generalization gaps.
Implications
The findings suggest that CBS can be a valuable tool for improving classification performance in applications with imbalanced datasets, such as medical imaging, object detection, and other domains where certain classes are underrepresented.
A Graph-Based Control Interface for Traffic Signals on Heterogeneous Road Networks
Graph Learning
Reinforcement Learning
Robotics
- Introduces a graph neural network-based control interface for traffic signals.
- Utilizes a deterministic incidence matrix for junction-specific signal phase generation.
- Demonstrates performance retention across unseen synthetic grid geometries.
- Identifies sensitivity to signal coverage distribution shifts.
Read more
A Graph-Based Control Interface for Traffic Signals on Heterogeneous Road Networks
Summary
This paper introduces a novel traffic-signal control interface leveraging a graph neural network (GNN) to assign scores to individual traffic movements at junctions. The approach utilizes a deterministic incidence matrix to convert these scores into a variable-sized set of legal signal phases specific to each junction. The design emphasizes a narrow boundary between learned movement scoring and deterministic phase construction, allowing for flexibility in graph size and action dimensions. The evaluation of the interface is conducted through Proximal Policy Optimization (PPO) experiments across synthetic grid geometries and five heterogeneous city graphs. The results indicate that while the policies maintained performance across unseen geometries within the synthetic grid family, they exhibited sensitivity to changes in signal coverage. Notably, a single trained policy was able to operate across multiple city graphs, yielding heterogeneous outcomes. This work highlights the feasibility of applying a shared GNN architecture for traffic signal control, although it does not claim general transferability to arbitrary road networks.
Methodology
The methodology involves the use of a graph neural network to score individual traffic movements, which are then converted into junction-specific signal phases using a deterministic incidence matrix. The system is evaluated through PPO experiments on synthetic grid geometries and city simulations, focusing on transferability and performance under varying conditions.
Results
The experiments revealed that the trained policies performed well on unseen geometries within the synthetic grid family, but were sensitive to changes in signal coverage. The same trained policy was able to execute across five heterogeneous city graphs, demonstrating variable outcomes, thus providing evidence of the system's feasibility rather than establishing a general transfer to arbitrary road networks.
Implications
The findings suggest that a graph-based approach to traffic signal control can enhance adaptability and efficiency in managing heterogeneous road networks. This could lead to improved traffic flow and reduced congestion in urban environments, paving the way for future research and applications in smart city infrastructure and automated traffic management systems.
Autoregressive EHR Foundation Models with Multimodal Inputs
Multimodal
Generative Models
Time Series
- Introduces a framework for integrating multimodal inputs into autoregressive EHR models.
- Demonstrates the importance of latent compression for efficient and effective model performance.
- Finds that stronger pretrained encoders consistently yield better results within modalities.
- Highlights that adding modalities does not automatically enhance prediction accuracy, necessitating thoughtful fusion architecture.
Read more
Autoregressive EHR Foundation Models with Multimodal Inputs
Summary
This paper presents a novel framework for enhancing autoregressive foundation models trained on electronic health records (EHRs) by incorporating multimodal inputs such as ECG waveforms, chest X-ray images, and clinical notes. The authors address the limitations of existing models that primarily rely on structured event codes and propose a method that utilizes modality-specific latent compression and gated cross-attention mechanisms with temporal alignment. The study investigates two critical design choices: the compression of long modality sequences to reduce computational overhead and the impact of different pretrained encoders on downstream performance. Through controlled experiments on the MIMIC-IV dataset, the authors demonstrate that the best configurations for latent compression outperform traditional methods like uncompressed cross-attention and mean pooling. They also find that simply adding auxiliary modalities does not guarantee improved performance in ICU mortality prediction, highlighting the necessity for careful architectural design and clinical context evaluation.
Methodology
The authors utilize a transformer-based architecture that incorporates modality-specific encoders for ECG, chest X-ray, and clinical notes. They implement a latent compression module to reduce the dimensionality of input sequences before applying cross-attention mechanisms. The model is trained on the MIMIC-IV dataset, with careful attention to temporal alignment and masking to prevent leakage of future information into past predictions.
Results
The experiments reveal that the best latent compression configurations significantly outperform both uncompressed cross-attention and mean pooling approaches. Additionally, the choice of pretrained encoders has a notable impact on performance, with stronger models yielding better results. However, the addition of auxiliary modalities does not consistently lead to improved outcomes in ICU mortality predictions compared to an EHR-only baseline.
Implications
This work suggests that while multimodal data can enhance clinical prediction models, the integration must be carefully designed to ensure that it contributes positively to model performance. The findings underscore the importance of evaluating models in a clinical context to understand the true value of different data modalities.
RIS-Kernel: A Model-Agnostic Architecture for Long-Context LLM Inference via Sparse Attention
Large Language Models
Efficient ML
NLP
- RIS-Kernel reduces self-attention complexity from O(N^2) to O(N log N).
- The architecture is model-agnostic and does not require weight modifications.
- RIS-Stochastic outperforms dense attention at low density levels, achieving higher accuracy.
- The method allows long-context LLM inference on standard CPU hardware without GPU support.
Read more
RIS-Kernel: A Model-Agnostic Architecture for Long-Context LLM Inference via Sparse Attention
Summary
The paper presents the Reduced Interaction Sampling (RIS) inference engine, a model-agnostic architecture designed to address the computational limitations of large language models (LLMs) when processing long-context documents. Traditional self-attention mechanisms scale quadratically with the input size, making it challenging to analyze documents with up to 65,536 tokens without significant hardware resources. RIS reduces the complexity of self-attention from O(N^2) to O(N log N) by employing sparse stochastic geometry, allowing for efficient inference on standard CPU hardware without the need for GPU acceleration. The architecture is validated through experiments on the Qwen2-1.5B-Instruct model, demonstrating that RIS can achieve competitive accuracy levels compared to dense attention methods. The results indicate that RIS-Stochastic at 1% density and 70 ensemble seeds outperforms the dense baseline, achieving 75.00% accuracy, while RIS-Structural at 1% density with 10 seeds recovers 75% of the contextual gap relative to a zero-context floor. The findings suggest that RIS can facilitate long-context LLM inference on commodity hardware, making advanced document analysis more accessible to research groups with limited computational resources.
Methodology
The RIS-Kernel architecture implements two sampling regimes: Stochastic Mode for global coverage and Structural Mode for local community preservation. It generates sparse geometric masks to manage memory efficiently and normalizes token fragments to maintain competitive weights. The architecture is tested on CPU servers, focusing on accuracy and context recovery through controlled evaluations.
Results
In controlled evaluations, RIS-Stochastic at 1% density and 70 seeds achieved 75.00% accuracy, surpassing the dense baseline of 71.88%. RIS-Structural reached 68.75% accuracy at 1% density with just 10 seeds, recovering 75% of the contextual gap. At 65,536 tokens, RIS provided retrieval gains of up to 14.06 percentage points over a zero-context baseline, with results confirmed as marginally significant.
Implications
The RIS-Kernel architecture has the potential to democratize access to long-context LLM inference, enabling researchers with limited computational resources to perform deep document analysis. This could lead to advancements in various fields requiring large-scale text processing, such as bioinformatics, legal document analysis, and academic research.
Adjustment Speed as a Safety Constraint for Nonstationary Reinforcement Learning
Reinforcement Learning
- Introduces adjustment speed as a safety constraint for nonstationary reinforcement learning.
- Formalizes safety in terms of adaptation feasibility, focusing on the relationship between required and achievable adaptation.
- Proposes the ASASC-NS framework, which proactively regulates agent behavior based on predicted environmental changes.
- Empirical results show significant reductions in safety violations during rapid context changes.
Read more
Adjustment Speed as a Safety Constraint for Nonstationary Reinforcement Learning
Summary
This paper addresses the challenge of ensuring safety in reinforcement learning (RL) systems operating in nonstationary environments, where conditions change over time. The author introduces the concept of 'adjustment speed' as a critical safety constraint, emphasizing that the ability of an RL agent to adapt to forecasted environmental changes is essential for maintaining safety. Traditional safe RL methods often assume stationary conditions and do not account for the adaptation speed necessary to remain safe during transitions. The proposed framework, Adjustment Speed as a Safety Constraint in Nonstationary Reinforcement Learning (ASASC-NS), formalizes safety in terms of adaptation feasibility. It compares the required adaptation due to predicted environmental shifts with the agent's calibrated recovery capacity. If the required adaptation exceeds what the agent can achieve safely, the framework proactively tightens safety constraints and activates an action-level shielding mechanism. Empirical evaluations in a nonstationary driving environment demonstrate that this approach effectively reduces safety violations during short-horizon windows aligned with context changes. The findings suggest that proactive safety interventions are necessary when environmental changes occur faster than the agent can adapt, thereby advancing the field of safe continual reinforcement learning by focusing on feasible safe adaptation rather than static safety specifications.
Methodology
The methodology involves developing the ASASC-NS framework, which utilizes representation learning to create a latent context representation of the environment. It predicts context evolution and estimates the adaptation demand induced by these shifts. The framework then compares this demand with the agent's calibrated recovery capacity to determine the necessity of safety interventions.
Results
The empirical evaluation in a nonstationary driving environment indicates that the ASASC-NS framework significantly reduces safety violations, particularly in short-horizon scenarios aligned with context changes. The ablation studies reveal that the shielding mechanism is more conservative for peak- and tail-risk suppression, while optimization-level adjustments further decrease violations.
Implications
The findings imply that reinforcement learning systems deployed in dynamic environments can benefit from proactive safety measures that account for adaptation speed. This approach could enhance the reliability of RL applications in safety-critical areas such as autonomous driving and robotics, where rapid environmental changes are common.
Teaching LLMs to Self-Evolve: Cultivating Core Meta-Skills with Reinforcement Learning
Large Language Models
Reinforcement Learning
Optimization
- Introduces METAEVOLVE, a framework for cultivating self-evolution meta-skills in LLMs.
- Utilizes a data synthesis pipeline to create training samples that reflect iterative self-improvement.
- Demonstrates significant performance improvements on both in-distribution and out-of-distribution coding tasks.
- Highlights the importance of continuous reward signals from program execution for effective training.
Read more
Teaching LLMs to Self-Evolve: Cultivating Core Meta-Skills with Reinforcement Learning
Summary
This paper introduces METAEVOLVE, a framework designed to cultivate core meta-skills in large language models (LLMs) through reinforcement learning (RL). The authors argue that traditional post-training methods overlook the importance of meta-skills such as self-reflection and feedback incorporation, which are crucial for iterative self-evolution. METAEVOLVE is grounded in coding tasks, where program execution provides continuous reward signals based on correctness and efficiency. The framework employs a data synthesis pipeline to create evolution trajectories that mirror the self-evolution process during inference. By training on large-scale coding data, METAEVOLVE aims to develop domain-agnostic meta-skills that can be applied to various open-ended problems. Experimental results demonstrate that METAEVOLVE outperforms existing baselines significantly across multiple coding benchmarks, showcasing its effectiveness in fostering self-evolution capabilities in AI systems.
Methodology
The METAEVOLVE framework consists of a data synthesis pipeline that constructs evolution-trajectory-aware samples and a reinforcement learning training phase that utilizes execution-based rewards. The framework is grounded in competitive coding, leveraging program execution to provide fine-grained feedback for both correctness and efficiency. The training data is synthesized from diverse coding tasks, ensuring a rich foundation for cultivating generalizable meta-skills.
Results
Across seven coding benchmarks, METAEVOLVE achieves a 10.01% absolute improvement on in-distribution tasks and a 24.12% improvement on out-of-distribution tasks compared to the strongest baseline. On open-ended algorithm optimization problems, METAEVOLVE achieves a 46.9% relative improvement. Additionally, the framework produces more structurally diverse solutions and demonstrates emergent self-evolution behaviors.
Implications
The findings suggest that explicitly cultivating self-evolution meta-skills can lead to more capable AI systems that can autonomously improve their performance over time. This has potential applications in various domains requiring adaptive problem-solving and optimization.
κ-LoRA: Condition Numbers Reveal Which LoRA Matrices Worth Updating
Efficient ML
Large Language Models
Optimization
- Identifies that not all LoRA matrices are equally valuable for adaptation, based on their condition numbers.
- Proposes κ-LoRA, which selectively updates matrices with larger condition numbers to improve efficiency.
- Demonstrates a significant reduction in fine-tuning time and memory costs while preserving model accuracy.
- Introduces a lightweight, one-shot selection procedure for matrix adaptation that is independent of training time.
Read more
κ-LoRA: Condition Numbers Reveal Which LoRA Matrices Worth Updating
Summary
The paper introduces κ-LoRA, a novel approach to Low-Rank Adaptation (LoRA) that optimizes the fine-tuning of neural networks by selectively updating LoRA matrices based on their condition numbers. The authors demonstrate that not all LoRA matrices contribute equally to model adaptation; matrices with smaller condition numbers are well-balanced and contribute marginally, while those with larger condition numbers contain underdeveloped directions that significantly enhance performance. By focusing updates on the top 50% of matrices ranked by condition number, κ-LoRA reduces the number of trainable parameters by half, leading to a 16.2% reduction in fine-tuning time and a 4.5% decrease in memory cost, while maintaining accuracy comparable to standard LoRA. The study highlights the spectral asymmetry in pre-trained transformers and provides a lightweight selection mechanism that requires no training-time signal, making κ-LoRA efficient and applicable across various tasks.
Methodology
The authors analyze the condition numbers of LoRA matrices to determine their contribution to model performance. They propose κ-LoRA, which ranks matrices based on their condition numbers and selectively applies updates to the top-performing matrices. This approach is supported by theoretical analysis of spectral properties and empirical observations during training.
Results
κ-LoRA achieves a 16.2% reduction in fine-tuning time and a 4.5% reduction in memory cost, while maintaining accuracy similar to that of standard LoRA. The condition numbers of selected matrices decrease over training, indicating effective spectral rebalancing.
Implications
The findings suggest that κ-LoRA can significantly enhance the efficiency of fine-tuning large-scale models, making it particularly valuable for resource-constrained environments such as edge deployment and on-device applications. This method could lead to broader adoption of efficient fine-tuning techniques in various machine learning applications.
Multi-Horizon Consistency as Geometry: When Latent Dynamics Contract, and When They Do Not
Reinforcement Learning
Theory
Generative Models
- Increasing multi-horizon consistency (λ) significantly reduces prediction errors in Moving-MNIST.
- Action-conditioned environments do not exhibit the same contraction behavior as observed in passive settings.
- A strong correlation exists between λ and the expansion proxy, indicating its diagnostic utility.
- The study introduces a stochastic-forcing law that connects different control domains.
Read more
Multi-Horizon Consistency as Geometry: When Latent Dynamics Contract, and When They Do Not
Summary
This paper investigates the impact of multi-horizon latent consistency on the transition geometry of video predictors and world models. The authors focus on the parameter λ, which controls the weight of multi-step latent agreement, and analyze its effects on an empirical expansion proxy (L20,q95) and horizon-20 prediction error (E20). Using the Moving-MNIST dataset, they demonstrate that increasing λ from 0 to 0.8 significantly reduces both L20 and E20, with four out of six seeds achieving L<1 at λ=0.8. However, similar improvements are not observed in action-conditioned environments like Pendulum-v1 and CartPole-v1, indicating a domain-specific limitation of the multi-horizon consistency approach. The study employs associational mediation analysis, revealing a strong correlation between λ and the expansion proxy. The authors propose a stochastic-forcing law that unifies control domains under a common curve, suggesting that soft consistency can enhance passive video performance but is constrained by the specific domain. The findings highlight the nuanced relationship between multi-horizon consistency and latent dynamics, providing insights for practitioners in model-based reinforcement learning.
Methodology
The authors conducted experiments using the Moving-MNIST dataset and other action-conditioned environments to analyze the effects of varying the multi-horizon consistency parameter λ. They measured the empirical expansion proxy L20,q95 and horizon-20 prediction error E20, employing statistical tests to validate their findings. Additionally, they performed associational mediation analysis and introduced a stochastic-forcing law to unify control domains.
Results
The results indicate that increasing λ leads to a substantial decrease in both L20 and E20 in the Moving-MNIST dataset, with significant statistical support. However, similar reductions in L20 were not observed in action-conditioned environments, highlighting a critical difference in behavior across domains. The stochastic-forcing law established a relationship between different control domains, suggesting a common underlying mechanism.
Implications
The findings of this study have implications for the design and training of video predictors and world models in reinforcement learning. Understanding the effects of multi-horizon consistency can help practitioners optimize model performance and reliability, particularly in passive versus active settings. The insights gained may also inform future research on latent dynamics and their role in model-based reinforcement learning.
A Defense of the Quadratic Model
Large Language Models
Optimization
Theory
- The quadratic model can effectively predict optimization dynamics in LLMs for significant training windows.
- Taylor expansions of the model and loss function provide accurate approximations of training trajectories.
- The Hessian spectrum exhibits structured behavior influenced by batch size and training duration.
- LLM training often occurs at a stochastic edge of stability, highlighting the sensitivity to hyperparameters.
Read more
A Defense of the Quadratic Model
Summary
This paper investigates the efficacy of the quadratic model as a theoretical framework for understanding the optimization dynamics of large language models (LLMs). The authors demonstrate that by Taylor expanding the model and loss function at intermediate training checkpoints, the quadratic model can accurately predict optimization dynamics for up to 10% of the training duration. They analyze the structure of local quadratic optimization problems using the Hessian spectrum and local stability, revealing significant insights into the eigenvalues and eigenvectors influenced by factors such as batch size and training time. The study employs Lanczos quadrature to estimate the Hessian spectrum, uncovering a structured tail in the eigenvalue distribution that is consistent across different batch sizes. Furthermore, the authors empirically validate the local linear stability of checkpoints against theoretical predictions, concluding that LLM training typically occurs at a stochastic edge of stability, which is also contingent on batch size. Overall, the findings suggest that the quadratic model serves as a valuable proxy for understanding pretraining optimization dynamics in LLMs.
Methodology
The authors trained a 150M parameter transformer model on 3 billion tokens, saving intermediate checkpoints every 10% of training. They applied Taylor expansions to the model and loss function at these checkpoints and compared the resulting optimization trajectories to the original. The Hessian spectrum was estimated using Lanczos quadrature, and local stability was analyzed through empirical tests against theoretical predictions.
Results
The study found that the quadratic model's approximations held for up to 10% of the training duration, with improved accuracy near the end of training. The Hessian spectrum revealed a consistent power-law tail across different batch sizes, and the analysis of local stability indicated that training dynamics are sensitive to learning rates and batch sizes, often operating at the edge of stability.
Implications
These findings suggest that the quadratic model can be a useful tool for understanding and optimizing training dynamics in large language models, potentially leading to better design choices in hyperparameter tuning and model training strategies.
On the Identifiability of Controlled World Models
Theory
Robotics
Reinforcement Learning
- Establishes a joint identifiability framework for controlled world models.
- Identifies spectral separation and conditional action variation as key conditions for identifiability.
- Proves that under certain conditions, JEPA can identify latent states and controlled dynamics.
- Derives quantitative bounds on identifiability in the presence of optimization errors.
Read more
On the Identifiability of Controlled World Models
Summary
This paper addresses the identifiability of controlled world models, which are essential for learning environment dynamics from high-dimensional observations and predicting outcomes based on actions. The authors introduce a joint identifiability theory for controlled world models that utilize Gaussian latent states under state-dependent Gaussian behavior policies. They identify two critical conditions for identifiability: spectral separation of the predictable signal for representation identifiability and non-degenerate conditional action variation for transition identifiability. The paper proves that when these conditions are met, every global minimizer of the Joint-Embedding Predictive Architecture (JEPA) objective can identify the latent state and controlled transition up to an orthogonal transformation. Additionally, the authors derive quantitative bounds on representation and transition identifiability in the context of approximate optimization. They also explore the implications of limited action coverage on counterfactual predictions and goal-conditioned latent planning through experiments involving nonlinear observation maps and behavior policies, demonstrating the practical relevance of their theoretical findings.
Methodology
The authors develop a theoretical framework to analyze the identifiability of controlled world models, focusing on Gaussian latent states and state-dependent behavior policies. They derive conditions for representation and transition identifiability, and conduct experiments to validate their theoretical claims across various nonlinear observation scenarios.
Results
The paper proves that under the conditions of spectral separation and sufficient conditional action variation, the JEPA framework can identify the true latent state and controlled dynamics. Quantitative bounds on identifiability are established, and empirical results confirm the theoretical predictions regarding the effects of action coverage on identifiability and planning.
Implications
The findings have significant implications for the development of more robust world models in applications such as autonomous driving, robotics, and other areas requiring effective planning and decision-making based on learned dynamics. The insights into identifiability can guide future research in improving model accuracy and reliability.
Evolution-Aware MSA Reasoning for Subsampling via Factor Graphs
Optimization
Graph Learning
Theory
- Introduces a new optimization framework for MSA subsampling, moving beyond heuristic methods.
- AP-REASONER leverages Affinity Propagation and factor graphs to optimize subsampling objectives.
- Demonstrates improved performance in structure-sensitive tasks compared to baseline methods.
- Incorporates controllable parameters for balancing diversity and query identity in subsampling.
Read more
Evolution-Aware MSA Reasoning for Subsampling via Factor Graphs
Summary
This paper addresses the challenge of subsampling Multiple Sequence Alignments (MSAs) in protein language models (PLMs) due to computational constraints. Traditional subsampling methods, such as random selection and identity-based filtering, lack control over the evolutionary signals retained in the selected subset. The authors propose a novel approach called AP-REASONER, which reformulates MSA subsampling as an explicit optimization problem. This method utilizes Affinity Propagation (AP) within a factor-graph framework, incorporating evolution-aware unary factors and exemplar-consistency factors to balance query identity and diversity as controllable objectives. The AP-REASONER employs message-passing reasoning to infer a fixed-budget MSA subset. Experimental results demonstrate that AP-REASONER outperforms existing subsampling techniques in long-range contact prediction and conformational ensemble prediction tasks, showcasing the effectiveness of factor-graph optimization in enhancing downstream performance and allowing for controlled recovery of alternative protein conformations.
Methodology
The authors recast MSA subsampling as an optimization problem, using Affinity Propagation within a factor-graph structure. They design unary factors to represent evolutionary measures and implement message-passing reasoning to select a subset of sequences that balances diversity and query identity. Two control knobs are introduced to explicitly modulate these aspects during the subsampling process.
Results
AP-REASONER consistently outperformed baseline subsampling methods in tasks related to long-range contact prediction and conformational ensemble prediction. The results indicate that the factor-graph approach effectively captures evolutionary signals, leading to better performance in structure-sensitive applications.
Implications
The findings suggest that modeling MSA subsampling as a controllable optimization problem can significantly enhance the performance of protein language models. This approach may facilitate more accurate predictions in protein structure and function, with potential applications in drug discovery and biological research.
Hyperball May Not Be a Free Lunch
Optimization
Theory
- Introduces an angular effective learning rate that accounts for parameter-update angles.
- Radial updates have a limited effect on the angular effective learning rate.
- Differences in optimizer performance are linked to effective step size evolution rather than update direction.
- Careful learning-rate scheduling is essential for maximizing Hyperball optimizer performance.
Read more
Hyperball May Not Be a Free Lunch
Summary
This paper investigates the performance of Hyperball-style optimizers in scale-invariant deep networks, particularly focusing on their advantages in large-scale training. The authors derive an angular effective learning rate that incorporates the angle between parameter updates, parameter norms, and update norms, challenging the conventional norm-based measures. They analyze the radial and tangential components of optimizer updates and find that radial updates have a limited direct impact on the angular effective learning rate. The study reveals that the differences in performance between Hyperball optimizers, such as MuonH and MuonWD, are not solely due to superior update directions but rather stem from the evolution of effective step sizes. A heuristic experiment is conducted to modify learning-rate schedules, demonstrating that careful scheduling is crucial for optimizing Hyperball's performance. The findings suggest that while Hyperball optimizers can accelerate training, aggressive learning-rate decay may hinder later performance, emphasizing the need for balanced learning-rate strategies.
Methodology
The authors derive an angular effective learning rate by analyzing the angular displacement caused by parameter updates. They decompose optimizer updates into radial and tangential components and conduct numerical analyses to assess the impact of radial updates. A state-dependent learning-rate alignment method is proposed to isolate the differences in dynamics between Hyperball and non-Hyperball optimizers.
Results
The study finds that radial updates have a minimal direct effect on the angular effective learning rate. The experiments show that modifying the learning-rate schedule can reproduce the dynamics of different optimizers, indicating that the evolution of effective step sizes is a key factor in performance differences. Additionally, aggressive learning-rate decay can improve early training performance but may negatively impact later stages.
Implications
The findings suggest that while Hyperball optimizers can enhance training efficiency, their effectiveness is highly dependent on learning-rate scheduling. This insight can inform the design of future optimizers and training protocols in deep learning, particularly for large-scale applications.
On the Depth Scalability of Logic Gate Networks
Theory
Optimization
Efficient ML
- Identifies optimization collapse and topology-induced limitations as barriers to depth scalability in LGNs.
- Proposes Input-Anchored Logic Gate Networks (IALGNs) to enhance depth scalability by conditioning layers on original input.
- Demonstrates that IALGNs achieve consistent depth–accuracy improvements across multiple datasets.
- Establishes a strict path-wise depth hierarchy that separates computational capability from optimization.
Read more
On the Depth Scalability of Logic Gate Networks
Summary
This paper addresses the limitations of Logic Gate Networks (LGNs) in achieving depth scalability, which is crucial for enhancing their computational power. The authors identify two primary issues: optimization collapse in deep relaxed LGNs and a topology-induced limitation that hinders effective computation in deeper layers. To overcome these challenges, they introduce Input-Anchored Logic Gate Networks (IALGNs), where each gate combines an evolving hidden feature with a direct input anchor. This design maintains a computational spine while ensuring that every layer is conditioned on the original input, allowing for richer representations and longer computational paths. The authors demonstrate that IALGNs achieve consistent improvements in depth–accuracy performance across benchmark datasets such as MNIST, CIFAR-10, and CIFAR-100, while alternative LGN architectures either saturate or degrade in performance. The study concludes that both stable optimization and an effective information-access pattern are essential for scalable depth in LGNs.
Methodology
The authors propose the Input-Anchored Logic Gate Network (IALGN) architecture, which integrates a direct input anchor with hidden features at each gate. They conduct experiments on benchmark datasets (MNIST, CIFAR-10, CIFAR-100) to evaluate performance improvements in depth scalability. The methodology includes layer-wise probes, topology ablations, and effective-depth analysis to assess representation quality and computational path longevity.
Results
IALGNs consistently outperform traditional LGN architectures in terms of depth–accuracy performance, achieving significant improvements beyond 100 layers. The experiments reveal that input anchoring is crucial for maintaining informative representations and preserving longer computational paths, while alternative architectures fail to benefit from increased depth.
Implications
The findings suggest that enhancing the architecture of LGNs can lead to more effective models for discrete inference tasks, potentially impacting areas such as logic-based reasoning and efficient computation in AI systems. The insights on depth scalability may also inform future research on neural network architectures.
IFCLoRA: Topology-Aware Rank Allocation for Parameter-Efficient Fine-Tuning
NLP
Large Language Models
Efficient ML
- IFCLoRA introduces a topology-aware, one-shot rank allocation method for parameter-efficient fine-tuning.
- The method leverages Information-Flow Centrality scores to quantify the importance of model components based on task-specific information flow.
- Experimental results show consistent performance improvements over traditional LoRA and adaptive methods across multiple tasks and models.
- The learned rank distributions are task-specific and non-uniform, highlighting the importance of structural information in rank allocation.
Read more
IFCLoRA: Topology-Aware Rank Allocation for Parameter-Efficient Fine-Tuning
Summary
The paper introduces IFCLoRA, a novel approach to parameter-efficient fine-tuning (PEFT) of large language models that enhances the traditional Low-Rank Adaptation (LoRA) method. While LoRA has been effective in reducing the number of trainable parameters, its performance is heavily influenced by how ranks are allocated across Transformer modules. Existing methods often rely on local gradient statistics, which can lead to inefficiencies. IFCLoRA addresses this by employing a topology-aware rank allocation strategy that utilizes a task-conditioned interaction graph constructed from a small calibration dataset. This graph informs the allocation of ranks based on Information-Flow Centrality (IFC) scores, which combine global information flow topology with local gradient sensitivity. The method allows for a one-shot rank allocation before fine-tuning, significantly improving performance across various tasks and models. Experimental results demonstrate that IFCLoRA consistently outperforms standard LoRA and other adaptive methods, achieving notable gains in tasks such as mathematical reasoning and semantic understanding, while maintaining comparable training costs.
Methodology
IFCLoRA employs a pre-fine-tuning rank allocation process that constructs a task-conditioned interaction graph from a frozen pretrained model using a small calibration dataset. It combines global information-flow topology with local gradient sensitivity to compute Information-Flow Centrality scores, which guide the allocation of ranks in a single pass under a global budget constraint.
Results
In experiments, IFCLoRA demonstrated significant performance improvements over LoRA, achieving an average gain of 1.13 percentage points on the GSM8K benchmark and a maximum gain of 1.82 percentage points at specific rank settings. The method also showed favorable results on the SuperGLUE benchmark, with task-dependent rank distributions that concentrated capacity on structurally important modules.
Implications
The findings suggest that incorporating global information-flow topology into rank allocation can enhance the efficiency and effectiveness of parameter-efficient fine-tuning methods. This approach could be beneficial for adapting large language models in various NLP tasks, particularly in scenarios with limited computational resources.
Integrated Order Dispatching and Routing for Last-Mile Pickup via Deep Reinforcement Learning
Reinforcement Learning
Graph Learning
Optimization
- First work to integrate order dispatching and routing using deep reinforcement learning.
- Development of novel mixed-integer linear programming formulations with time-dependent travel times.
- Utilization of a Dynamic-Residual Graph Attention Network for routing decisions.
- Real-time decision-making achieved through a routing-oracle-guided dispatching heuristic.
Read more
Integrated Order Dispatching and Routing for Last-Mile Pickup via Deep Reinforcement Learning
Summary
This paper addresses the complexities of last-mile pickup operations, which require efficient decision-making for order dispatching and routing. The authors propose an integrated optimization framework that combines a learned routing oracle with real-time dispatching heuristics. The routing subproblem is modeled using a Dynamic-Residual Graph Attention Network encoder and a Look-Ahead Courier-Personalized decoder, while the dispatching subproblem utilizes a routing-oracle-guided heuristic with local search capabilities. The framework is evaluated using real-world datasets from Cainiao Logistics, demonstrating superior performance in both solution quality and computational efficiency compared to existing benchmarks. The study emphasizes the importance of integrating dispatching and routing processes to enhance operational efficiency in urban logistics.
Methodology
The authors decompose the last-mile pickup problem into two hierarchical subproblems: order dispatching and routing. The routing problem is formulated as a Markov decision process (MDP) and solved using a policy network trained with a rollout-baseline policy-gradient approach. The dispatching heuristic leverages the routing oracle to evaluate candidate couriers efficiently, employing a position-pool screening mechanism and local search refinement.
Results
The proposed framework outperformed existing benchmarks in terms of both solution quality and solving time in extensive experiments conducted on real-world datasets. The results indicate that the integrated approach effectively addresses the complexities of last-mile pickup operations, providing scalable and efficient solutions.
Implications
This research has significant implications for logistics companies, as it provides a robust framework for real-time decision-making in last-mile delivery scenarios. The integration of dispatching and routing can lead to reduced operational costs and improved service quality, ultimately enhancing customer satisfaction in urban logistics.
A Leakage-Free Stacked Ensemble Method for Multiclass Classification
Theory
- Introduction of LFS-FRAME, a leakage-free stacked ensemble framework for multiclass classification.
- Integration of Kolmogorov-Arnold Networks and XGBoost to combine functional and rule-based learning.
- Strict out-of-fold stacking strategy to prevent data leakage and ensure unbiased meta-features.
- Experimental results show significant performance improvements over traditional single-model approaches.
Read more
A Leakage-Free Stacked Ensemble Method for Multiclass Classification
Summary
This paper presents LFS-FRAME, a novel leakage-free stacked ensemble framework designed for multiclass classification. The authors address common challenges in multiclass settings, such as high inter-class similarity, class imbalance, and data distribution variability. Traditional classifiers like XGBoost excel in structured data but struggle with capturing complex relationships, while neural networks can model nonlinear interactions but often face overfitting. LFS-FRAME integrates Kolmogorov-Arnold Networks (KAN) for functional learning and XGBoost for rule-based learning, ensuring unbiased meta-features through a strict out-of-fold stacking strategy that prevents data leakage. The framework leverages probabilistic outputs from diverse base learners to enhance classification performance. Experimental results demonstrate that LFS-FRAME achieves an overall accuracy of 89.85% in identifying major families and 81.74% in identifying sub-families, outperforming strong single-model baselines. This highlights the framework's effectiveness in providing reliable and generalizable multiclass classification solutions.
Methodology
The proposed methodology involves a stacked ensemble approach that combines functional learning through Kolmogorov-Arnold Networks (KAN) and rule-based learning via XGBoost. The framework employs a strict out-of-fold stacking strategy to create unbiased meta-features, ensuring complete isolation between training and validation datasets. This approach mitigates data leakage and enhances the robustness of the meta-classifier by learning from probabilistic outputs of heterogeneous base learners.
Results
The experimental evaluations on multiclass datasets indicate that LFS-FRAME achieves an overall accuracy of 89.85% for major family identification and 81.74% for sub-family identification, significantly surpassing the performance of strong single-model baselines. These results underscore the effectiveness of the leakage-free functional and rule-based stacking approach.
Implications
The findings suggest that LFS-FRAME can be effectively applied in various domains requiring multiclass classification, such as healthcare analytics, bioinformatics, and cybersecurity. The leakage-free design enhances the reliability of predictions, making it suitable for real-world applications where data integrity is crucial.
An Insight on Evaluation Metrics Under the Imbalanced Case of Anomaly Detection
Time Series
- Evaluation metrics for anomaly detection behave differently under varying class imbalance ratios.
- Metric landscapes provide an intuitive framework for visualizing the relationship between metrics and performance trade-offs.
- Standard metrics like AUROC and AUPR can still be useful if their interpretation considers class imbalance.
- The study offers practical guidance for comparing anomaly detection results across different datasets.
Read more
An Insight on Evaluation Metrics Under the Imbalanced Case of Anomaly Detection
Summary
This paper addresses the challenges of evaluating anomaly detection models in the context of severe class imbalance, which is a common characteristic of such tasks. The authors analyze four widely used evaluation metrics—AUROC, AUPR, F1-score, and MCC—under varying levels of class imbalance. They introduce the concept of 'metric landscapes,' which visualize how these metrics behave in relation to true positive and true negative rates across different anomaly ratios. The analysis reveals that the interpretation of these metrics can vary significantly based on the imbalance ratio, highlighting the importance of understanding how metric values evolve with changes in class distribution. The authors provide practical guidance for interpreting and comparing anomaly detection results, emphasizing that while standard metrics remain useful, their interpretation must account for the underlying class imbalance. The study is grounded in real-world datasets, particularly those relevant to image and time-series anomaly detection, and aims to enhance the understanding of metric performance in realistic scenarios.
Methodology
The authors conducted a systematic analysis of four common anomaly detection metrics by creating metric landscapes that map metric scores against true positive and true negative rates across various levels of class imbalance. This approach allows for a visual understanding of how different metrics emphasize trade-offs between sensitivity and specificity.
Results
The analysis demonstrated that the values of AUROC, AUPR, F1-score, and MCC can convey different meanings depending on the anomaly ratio. The metric landscapes illustrated how these metrics shift in their emphasis on sensitivity versus specificity as class imbalance increases, providing insights into their stability and interpretability.
Implications
The findings of this study have significant implications for practitioners in anomaly detection, as they highlight the need for careful interpretation of evaluation metrics in imbalanced scenarios. The insights can guide model selection and performance assessment in various applications, including fraud detection, network security, and healthcare.
From Score Approximation to Distribution Approximation in Score-Based Diffusion Models
Generative Models
Theory
- Establishes a quantitative link between score approximation and distribution approximation in diffusion models.
- Proves that accurate score function approximation leads to small KL divergence between generated and target distributions.
- Derives an explicit upper bound on distribution approximation error based on various factors.
- Combines results from neural network theory, stochastic analysis, and information theory.
Read more
From Score Approximation to Distribution Approximation in Score-Based Diffusion Models
Summary
This paper addresses the theoretical foundations of score-based diffusion models, which have shown significant success in generative modeling across various domains. The author establishes a quantitative relationship between the approximation of score functions by neural networks and the approximation of the probability distributions generated by reverse diffusion processes. The main contribution is a rigorous proof that if a neural network accurately approximates the true score function, the resulting distribution from the reverse diffusion model will closely match the target data distribution, quantified by Kullback–Leibler (KL) divergence. The analysis combines classical universal approximation theorems, Girsanov’s theorem, and the data processing inequality for relative entropy. The paper provides an explicit upper bound on the distribution approximation error, linking it to the score approximation error, the diffusion noise schedule, and the terminal prior mismatch. This work complements existing research by providing a theoretical framework that connects score approximation with distribution approximation, thereby enhancing our understanding of the capabilities of neural score models.
Methodology
The methodology involves a two-step analysis: first, deriving an identity for KL divergence between path measures using Girsanov’s theorem; second, applying the data processing inequality to relate path-space divergence to distribution approximation. The results are combined with Hornik’s universal approximation theorem to establish the main theorem linking score and distribution approximation.
Results
The paper demonstrates that for any desired level of accuracy (ε), there exists a neural network that can approximate the score function such that the KL divergence between the generated distribution and the target distribution is bounded by a term that includes the prior mismatch and the score approximation error.
Implications
The findings provide a mathematical justification for the effectiveness of neural score models in generative tasks, potentially influencing future research in generative modeling and the development of more robust diffusion models.
Susceptible Reservoir Architectures for Regime-Conditional Volatility Forecasting
Time Series
- Introduction of Susceptible Architectures (SUSA) for volatility forecasting.
- Implementation of complex-valued reservoirs and quantum reservoir computing.
- Competitive performance against traditional GARCH models with significant improvements in QLIKE.
- Demonstration of enhanced forecasting accuracy through regime-conditioned interpretations.
Read more
Susceptible Reservoir Architectures for Regime-Conditional Volatility Forecasting
Summary
This paper addresses the challenge of volatility forecasting, which is often hindered by persistence and measurement noise, limiting the effectiveness of nonlinear models. The author introduces Susceptible Architectures (SUSA), a novel reservoir design principle aimed at enhancing volatility forecasting. SUSA includes two implementations: complex-valued open-chain and periodic reservoirs, alongside regime-conditioned experts to interpret reservoir features across different market states (calm, onset, recovery, and persistent stress). The study also explores quantum reservoir computing by implementing open-system q-qubit counterparts in Qiskit while maintaining a common AR-Ridge anchor and a bounded residual correction trained under QLIKE. The models are evaluated on 16 U.S. equity and ETF series using a structured training, validation, and test approach. The results indicate that the proposed models perform competitively against traditional GARCH models, achieving statistically significant QLIKE improvements for specific assets. Additionally, a stacked ensemble of the models enhances mean QLIKE performance, demonstrating the effectiveness of the SUSA principle in volatility forecasting.
Methodology
The methodology involves designing susceptible reservoirs that exploit structural and regime-conditional susceptibility to enhance volatility forecasting. The study employs complex-valued open-chain and periodic reservoirs, and implements quantum reservoir computing using q-qubit models. The models are evaluated using a structured approach with disjoint chronological training, validation, and test folds, focusing on a 12-observation input window and a five-observation forecast horizon.
Results
The proposed models achieved competitive results compared to GARCH, with statistically significant QLIKE improvements for specific assets such as IWM and XLP. The stacked ensemble of models improved mean QLIKE by 0.0116 over the strongest individual model and succeeded in 75% of test scenarios, indicating the effectiveness of the SUSA principle.
Implications
The findings suggest that susceptible reservoir architectures can significantly enhance volatility forecasting accuracy, providing a new approach for financial modeling. The integration of quantum reservoir computing may open avenues for further research in financial forecasting and complex system analysis.
Toward Goal-Agnostic Joint-Embedding Predictive Control of Partial Differential Equations
Optimization
Robotics
Theory
- First implementation of closed-loop PDE control using a joint-embedding latent world model.
- Introduction of an observable-aligned controller that estimates kinetic energy from predicted latents.
- Significant reduction in late field RMSE by 53% compared to traditional latent-L2 planning.
- Demonstrated flexibility in planning machinery supporting arbitrary control objectives.
Read more
Toward Goal-Agnostic Joint-Embedding Predictive Control of Partial Differential Equations
Summary
This paper introduces a novel framework for controlling partial differential equations (PDEs) using a goal-agnostic approach based on a joint-embedding predictive architecture (JEPA). The proposed method employs a lightweight 2D Vision Transformer (ViT) encoder and action-conditioned latent dynamics, which are trained offline without any specific reward or goal. The frozen model is then utilized by a model-predictive path integral (MPPI) controller. The authors demonstrate that applying control objectives to explicit physical observables yields better results than minimizing Euclidean distance in the latent space. They achieve impressive performance on the PDE Control Gym 2D Navier–Stokes benchmark, showing significant improvements in control accuracy and efficiency. The framework allows for dynamic control without the need for retraining when objectives change, making it a versatile solution for various PDE control tasks.
Methodology
The authors developed a JEPA that learns action-conditioned dynamics from offline state transitions without requiring a reward or specific goal. They trained a 2D ViT encoder and a latent predictor, which were then frozen for use in an MPPI controller. The framework incorporates a linear probe for estimating kinetic energy, allowing the controller to evaluate tracking errors in physical units rather than latent distances.
Results
The proposed method improved the matched 50-episode native reward from -12.08 to -10.90 and reduced last-quarter velocity-field RMSE from 0.0765 to 0.0692. Across three distinct aperiodic targets, the kinetic energy planning approach lowered late field RMSE by 53%, achieving superior performance in all 30 paired episodes. The frozen model also effectively supported stabilization tasks with a mean relative error of 2.7%.
Implications
This research suggests that joint-embedding architectures can significantly enhance control strategies for PDE systems, providing a flexible and efficient framework that adapts to various control objectives without the need for retraining. The findings could have broad applications in fields such as climate modeling, biological processes, and chemical process monitoring, where PDEs are prevalent.
Multi-Agent Debate and Visual Information Extraction for SeePhys Pro: A 1st-Place Technical Report from ICML 2026 AI4Math Track 3 Challenge
Multimodal
NLP
Large Language Models
- Introduced multi-agent debate for solving physics problems, significantly outperforming single-solver approaches.
- Developed a visual information extraction method that translates figures into structured text, addressing modality gaps.
- Demonstrated that the effectiveness of visual aids scales with the complexity of the visual information.
- Achieved first place in the SeePhys Pro Challenge, showcasing the effectiveness of the proposed framework.
Read more
Multi-Agent Debate and Visual Information Extraction for SeePhys Pro: A 1st-Place Technical Report from ICML 2026 AI4Math Track 3 Challenge
Summary
This technical report details the authors' approach to the SeePhys Pro Challenge Track 3 at the AI for Math Workshop during ICML 2026. The challenge involved answering college-level physics questions that included visual components, which posed significant difficulties for large language models (LLMs) when critical information was embedded in images rather than text. To tackle this, the authors developed a two-stage framework comprising a visual information extraction phase and a reasoning phase that utilized multi-agent debate among three heterogeneous solvers. The visual information extraction stage translated figures into solver-readable text, effectively bridging the modality gap. The reasoning stage employed a multi-agent debate mechanism to enhance answer selection reliability. The authors discovered that the benefits of orchestration stemmed more from improved answer selection than from the debate itself, and that the effectiveness of visual aids increased with the complexity of the visual information. Their approach achieved a significant accuracy improvement from a baseline of 0.643 to 0.802 on the public dataset, securing first place on both public and private leaderboards.
Methodology
The authors proposed a two-stage framework consisting of a visual information extraction stage and a reasoning stage. The visual information extraction stage involved converting figures into structured representations that solvers could read, while the reasoning stage orchestrated three different LLMs through a multi-agent debate mechanism, enhancing answer selection reliability.
Results
The proposed system improved accuracy from 0.643 to 0.802 on the public split of the challenge and secured first place on both public and private leaderboards, demonstrating the effectiveness of the multi-agent debate and visual information extraction approach.
Implications
The findings suggest that multi-agent systems can significantly enhance problem-solving capabilities in complex tasks involving visual information. This approach could be applied to other domains requiring multimodal reasoning, such as robotics, interactive learning systems, and advanced AI tutoring systems.
Remedying Coarsening-Based GNN Training under Heterophily via Adaptive Complementary Enhancement
Graph Learning
Efficient ML
Theory
- Identifies the performance challenges of coarsening-based GNN training in heterophilic graphs.
- Introduces Adaptive Complementary Enhancement (ACE) as a solution to reintegrate lost graph information.
- Demonstrates the effectiveness of ACE through extensive empirical evaluations on various benchmarks.
- Maintains competitive performance on homophilic graphs while significantly improving results on heterophilic graphs.
Read more
Remedying Coarsening-Based GNN Training under Heterophily via Adaptive Complementary Enhancement
Summary
This paper addresses the limitations of coarsening-based training methods for Graph Neural Networks (GNNs) when applied to heterophilic graphs, where neighboring nodes often have different labels or features. Previous research primarily focused on homophilic graphs, leading to significant performance degradation in heterophilic settings due to the loss of critical graph information during coarsening. The authors propose a novel framework called Adaptive Complementary Enhancement (ACE), which is model-agnostic and enhances coarsening-based training by reintegrating the discarded information. ACE employs a projector to reconstruct original node features and incorporates anisotropic structural regularization to capture local heterophily. Additionally, it utilizes homoscedastic uncertainty weighting to balance the training objectives of coarsened-graph loss and full-graph auxiliary loss. The extensive experiments conducted demonstrate that ACE consistently improves performance on heterophilic benchmarks while maintaining competitive results on homophilic graphs, all with minimal computational overhead.
Methodology
The methodology involves the development of the ACE framework, which includes learning a projector for reconstructing node features from coarsened graphs, applying anisotropic structural regularization to account for local heterophily, and utilizing homoscedastic uncertainty weighting to adaptively balance the training objectives. This approach is designed to be model-agnostic, allowing it to be integrated into various GNN architectures.
Results
The results indicate that ACE leads to consistent performance improvements on heterophilic benchmarks, with notable gains in accuracy compared to existing coarsening-based training methods. The framework also preserves competitive performance on homophilic graphs, demonstrating its versatility and effectiveness across different graph types.
Implications
The findings suggest that ACE can significantly enhance the scalability and effectiveness of GNNs in real-world applications where heterophily is prevalent, such as social networks, biological networks, and recommendation systems. This work opens avenues for further research into adaptive training strategies for GNNs in complex graph structures.
Cross-Tokenizer On-Policy Distillation via Byte-Prefix Marginalization
NLP
Large Language Models
Efficient ML
- Introduction of Byte-Prefix Marginalization (BPM) for cross-tokenizer distillation.
- BPM satisfies vocabulary completeness, byte-level alignment, and mass preservation.
- Demonstrated superior performance over existing methods on multiple benchmarks.
- Addressed failure modes in code generation tasks related to whitespace.
Read more
Cross-Tokenizer On-Policy Distillation via Byte-Prefix Marginalization
Summary
This paper addresses the challenge of knowledge distillation from open-weight language models with different tokenization schemes. The authors introduce a novel method called Byte-Prefix Marginalization (BPM), which allows for effective on-policy distillation (OPD) by transforming the teacher's next-token distribution into a format compatible with the student's vocabulary. BPM ensures vocabulary completeness, byte-level alignment, and mass preservation, which are critical for accurate distillation. The method aggregates the probability mass of teacher tokens to the longest matching student tokens based on byte prefixes, while handling unmatched mass through a residual category. The authors demonstrate that BPM outperforms existing cross-tokenizer methods across multiple benchmarks, achieving significant improvements in performance metrics. The paper also identifies and addresses a failure mode related to whitespace in code-generation tasks, ensuring robust training. Overall, BPM provides a systematic approach to distilling knowledge from heterogeneous models, enhancing the efficiency and capability of student models.
Methodology
The authors developed BPM to re-express the teacher's next-token distribution over the student vocabulary by mapping teacher tokens to student tokens based on byte prefixes. The method aggregates probabilities for matching tokens and manages unmatched mass through a residual category. The student model is trained using a generalized Jensen–Shannon divergence objective, with a precomputed mask to exclude problematic whitespace positions.
Results
BPM consistently outperformed existing cross-tokenizer methods across six mathematics and programming benchmarks, improving average performance metrics by 3.7–6.6 points over the strongest baselines. The method demonstrated effectiveness in distilling knowledge from three different teacher models, showcasing its robustness and versatility.
Implications
The findings suggest that BPM can significantly enhance the performance of compact language models by effectively consolidating knowledge from diverse teacher models. This has implications for deploying more efficient models in practical applications, particularly in areas requiring advanced reasoning and coding capabilities.
Physiological Signals as a Forensic Modality for Talking-Face Deepfake Detection
Computer Vision
Generative Models
Multimodal
- First rPPG-based detection framework specifically for TF deepfakes.
- Achieved AUC of 0.806, close to the best general-purpose detector.
- Documented performance degradation of existing rPPG detectors on TF data.
- Stable AUC performance across different TF generators indicates interpretable physiological properties.
Read more
Physiological Signals as a Forensic Modality for Talking-Face Deepfake Detection
Summary
This paper addresses the challenge of detecting talking-face (TF) deepfakes, which generate realistic facial videos from static images and audio, making them difficult to identify with traditional image-based methods. The authors propose a novel detection framework that utilizes remote photoplethysmography (rPPG) to extract physiological signals from videos, specifically targeting TF deepfakes where no real video substrate exists. The framework employs RhythmFormer for waveform extraction and lightweight classifiers to differentiate between real and synthesized physiological signals. The evaluation is conducted on the TF subset of the Celeb-DF++ dataset using a strict subject-independent protocol, achieving an AUC of 0.806 with a 1D ResNet classifier. The study also highlights the degradation of performance in existing rPPG detectors when applied to TF data, demonstrating a significant drop in AUC from 0.999 to 0.622. Furthermore, the authors analyze the performance across various TF generators, revealing a stable AUC range that reflects the physiological characteristics of each generator. This work represents a significant advancement in forensic detection methods for TF deepfakes, emphasizing the unique challenges posed by this category of forgeries.
Methodology
The proposed framework extracts rPPG waveforms using RhythmFormer and trains lightweight 1D classifiers to distinguish between real and synthesized physiological signals. The evaluation is conducted under a strict subject-independent protocol to ensure robustness against cross-identity physiological leakage.
Results
The framework achieved an AUC of 0.806 and an EER of 27.8%, placing it within 2.4 points of the best published general-purpose detector while exclusively using physiological data. The study also revealed a significant performance drop in existing rPPG detectors when applied to TF deepfake data, with AUC values ranging from 0.690 to 0.985 across different TF generators.
Implications
The findings suggest that rPPG can serve as a viable modality for detecting TF deepfakes, potentially leading to improved forensic techniques in digital media verification. This work may influence future research in deepfake detection and the development of more robust forensic tools.
Energy Manifold Natural Gradient Descent: Riemannian Optimization for Neural PDE Solvers
Optimization
Theory
- EMNGD extends ENGD to constrained Riemannian manifolds, improving optimization for neural PDE solvers.
- The method provides a best-admissible approximation to the function-space Newton correction under energy metrics.
- Global first-order convergence is achieved with Armijo backtracking, ensuring robustness in optimization.
- Scalable solvers using the Woodbury identity and Nyström approximation enhance computational efficiency.
Read more
Energy Manifold Natural Gradient Descent: Riemannian Optimization for Neural PDE Solvers
Summary
This paper introduces Energy Manifold Natural Gradient Descent (EMNGD), a novel optimization framework designed for neural partial differential equation (PDE) solvers that operate on Riemannian manifolds. Traditional Energy Natural Gradient Descent (ENGD) methods assume an unconstrained Euclidean parameter space, which can be inadequate for neural PDE models that impose parameter constraints. EMNGD addresses this limitation by aligning parameter updates with the curvature of an energy function defined over a constrained manifold. The authors prove that EMNGD provides the best feasible approximation to the function-space Newton vector in the energy metric, ensuring that updates respect parameter constraints while minimizing the energy. The framework exhibits coordinate invariance, reduces to ENGD in Euclidean space, and guarantees global first-order convergence with Armijo backtracking. Additionally, the paper discusses the use of the Woodbury identity and Nyström approximation to facilitate scalable sample-space solves, enhancing computational efficiency. Experimental results demonstrate that EMNGD outperforms existing state-of-the-art methods in terms of accuracy and convergence speed on various neural PDE benchmarks.
Methodology
The authors developed EMNGD by formulating an energy-induced quadratic model directly over feasible tangent directions on a Riemannian manifold. They employed retraction techniques to maintain parameter constraints during optimization. The theoretical framework includes proving the best-admissible Newton correction and establishing convergence properties under specific assumptions. The use of the Woodbury identity allows for efficient computation in sample space, while the Nyström approximation provides scalable solutions with controlled direction errors.
Results
EMNGD was tested against various neural PDE benchmarks, showing higher accuracy and faster convergence compared to state-of-the-art optimization methods. The theoretical results confirmed the method's convergence properties and its ability to maintain parameter constraints effectively.
Implications
The development of EMNGD has significant implications for the training of neural PDE solvers, particularly in fields requiring high accuracy and efficiency in solving complex differential equations. This framework can be applied to various scientific and engineering problems where PDEs are prevalent, potentially improving the performance of simulations and predictive models.
TriGlue: a Biology-Inspired Generative Model for Generating Molecular Glue-Induced Ternary Complex
Generative Models
- TriGlue is a biology-inspired generative framework for designing molecular glue degraders.
- The model decomposes the ternary complex generation into interface estimation and complex generation stages.
- An SE(3)-equivariant module is developed for accurate interface estimation from unbound protein structures.
- The interface-conditioned ternary flow matching network generates ligands and predicts protein docking transformations.
Read more
TriGlue: a Biology-Inspired Generative Model for Generating Molecular Glue-Induced Ternary Complex
Summary
The paper presents TriGlue, a novel generative model designed for the computational design of molecular glue degraders (MGDs), which facilitate targeted protein degradation by forming ternary complexes between an E3 ubiquitin ligase and a target protein. The authors identify the challenges in molecular glue design, which include the need to model the unknown protein-protein interface and the simultaneous generation of ligands and complex assembly. TriGlue addresses these challenges by decomposing the ternary complex generation into two stages: interface estimation and interface-conditioned complex generation. The first stage involves an SE(3)-equivariant interface estimation module that predicts the protein-protein interface from unbound monomer structures. The second stage employs an interface-conditioned ternary flow matching network to generate the molecular glue and predict the necessary rigid-body transformations for complex assembly. The effectiveness of TriGlue is demonstrated through extensive experiments, which show that it can generate chemically valid molecules and plausible ternary complexes, indicating its potential to accelerate molecular glue discovery in drug development.
Methodology
TriGlue employs a two-stage approach for molecular glue design: first, it uses an SE(3)-equivariant interface estimation module to predict the protein-protein interface from unbound monomer structures. Then, it utilizes an interface-conditioned ternary flow matching network to jointly generate the molecular glue and predict the rigid-body transformations necessary for assembling the ternary complex.
Results
The experiments conducted demonstrate that TriGlue successfully generates chemically valid molecules and plausible ternary complexes, showcasing its effectiveness in the context of molecular glue discovery.
Implications
The development of TriGlue has significant implications for accelerating the discovery of molecular glue degraders, which are becoming increasingly important in targeted protein degradation and therapeutic applications. This generative modeling approach could streamline the drug discovery process, reducing reliance on traditional experimental methods.
MissHyper: Restoring Clinical Synchronicity in Missingness-Guided Hypergraph Forecasting
Time Series
Graph Learning
- Identifies a pre-propagation bottleneck in clinical event representations, emphasizing the need for improved event initialization.
- Introduces MissHyper, which combines support-density encoding, timestamp-level context restoration, and missingness-guided gated fusion.
- Demonstrates consistent performance improvements in multi-step forecasting across multiple clinical datasets.
- Highlights the significance of co-timestamp context in enhancing the interpretability of sparse clinical measurements.
Read more
MissHyper: Restoring Clinical Synchronicity in Missingness-Guided Hypergraph Forecasting
Summary
The paper introduces MissHyper, a novel forecasting model designed to address the challenges of clinical irregular multivariate time series (IMTS) data, which are often sparse and asynchronously collected. Traditional event-centric models tend to flatten co-timestamp records too early, leading to a pre-propagation representation bottleneck where local patient-state context is lost. MissHyper aims to restore this context before the message-passing phase by employing a missingness-guided hypergraph approach. The model enhances each event with a local support-density cue and aggregates co-timestamp records to recover patient-state context. A missingness-guided gate is utilized to adaptively fuse this context with node-specific evidence. The authors validate MissHyper on benchmark datasets including PhysioNet 2012, MIMIC-III, and MIMIC-IV, demonstrating significant improvements in multi-step forecasting performance compared to existing hypergraph baselines. The findings suggest that enhancing event initialization can substantially benefit sparse clinical forecasting without necessitating changes to downstream architectures. The study emphasizes the importance of snapshot restoration, adaptive fusion, and support-density encoding in improving clinical forecasting outcomes.
Methodology
MissHyper employs a missingness-guided initialization strategy that modifies event representations before hypergraph propagation. It incorporates a local support-density cue, aggregates co-timestamp records to create a timestamp-level context, and uses a missingness-guided gate for adaptive fusion of this context with individual event nodes.
Results
The experiments reveal that MissHyper achieves significant reductions in mean squared error (MSE) and mean absolute error (MAE) when compared to both irregular time-series baselines and a direct hypergraph baseline. The ablation studies confirm the contributions of snapshot restoration, adaptive fusion, and local support density to the model's performance.
Implications
The findings suggest that improving event initialization can enhance the accuracy of clinical forecasting models, which is crucial for early warning systems and resource planning in intensive care settings. This approach could lead to better patient monitoring and management in clinical environments.
Synthetic Speech, Real Signal: Paralinguistic Preservation and Cross-Lingual Augmentation via Voice Cloning
Audio & Speech
- Voice cloning can effectively augment paralinguistic datasets, particularly in clinical contexts.
- Most evaluated voice cloning models preserve paralinguistic signals with minimal degradation.
- Cloning English clinical speech into Japanese improves depression and anxiety detection in Japanese speech.
- Voice cloning offers a promising approach for addressing data scarcity in low-resource languages.
Read more
Synthetic Speech, Real Signal: Paralinguistic Preservation and Cross-Lingual Augmentation via Voice Cloning
Summary
This paper explores the use of voice cloning as a synthetic data augmentation technique for paralinguistic tasks, particularly in clinical settings where labeled data is scarce. The authors benchmark eight voice cloning models across five paralinguistic tasks, including emotion recognition and clinical biomarker detection, using both public and proprietary datasets. They find that most models preserve paralinguistic signals with only modest degradation. Furthermore, the study demonstrates that training on cloned English clinical speech data can significantly enhance the detection of depression and anxiety in real Japanese speech, outperforming traditional cross-lingual transfer methods. This suggests that voice cloning could be a valuable tool for augmenting clinical speech data in low-resource languages, thereby improving mental health assessments across diverse populations.
Methodology
The authors benchmarked eight open-source voice cloning models across five paralinguistic tasks using a combination of public and proprietary datasets. They evaluated the models based on their ability to preserve paralinguistic information and tested the effectiveness of cloned English clinical speech in detecting mental health conditions in Japanese speech. The methodology included the use of various architectures for voice cloning and a processing pipeline that involved transcription, cloning, and classification using logistic regression.
Results
The results indicate that most voice cloning models maintain paralinguistic signals with only slight degradation. Additionally, training classifiers on cloned data from English to Japanese significantly improved the detection performance for depression and anxiety in Japanese speech, surpassing the performance of models trained on raw cross-lingual data.
Implications
The findings suggest that voice cloning can serve as an effective augmentation strategy for clinical speech data, particularly in low-resource languages. This could lead to improved mental health diagnostics and interventions by enabling better access to diverse linguistic data. Furthermore, the study highlights the need for evaluating voice cloning models beyond traditional metrics to ensure the preservation of critical paralinguistic information.
LatentFlow: Visual Analytics for Latent Space Analysis in Molecular Graph Neural Networks
Graph Learning
- LatentFlow is an interactive visual analytics system for analyzing latent spaces in molecular GNNs.
- It employs a modified Sankey diagram to track changes in clusters of embeddings across layers and model states.
- The system connects model outputs to chemical knowledge, enabling experts to interpret results meaningfully.
- LatentFlow was developed in collaboration with domain experts to ensure practical applicability.
Read more
LatentFlow: Visual Analytics for Latent Space Analysis in Molecular Graph Neural Networks
Summary
The paper presents LatentFlow, a visual analytics system designed to facilitate the exploration of latent spaces in molecular Graph Neural Networks (GNNs). As machine learning models, particularly GNNs, become increasingly prevalent in predicting molecular properties and reaction outcomes, understanding the organization of chemical information within these models is crucial. Existing methods for analyzing latent spaces are limited, particularly in tracking changes across different layers and model states. LatentFlow addresses these challenges by providing an interactive platform that groups embeddings into clusters and visualizes their transitions using a modified Sankey diagram. This system allows scientists to observe how clusters evolve across layers and model states, linking them to representative molecules and their substructures. The development of LatentFlow involved collaboration with domain experts to ensure its relevance and usability. The system supports interactive filtering and summary visualizations to enhance clarity while preserving essential structural patterns. Through two case studies, LatentFlow demonstrated its effectiveness in helping scientists understand latent space evolution, identify meaningful molecular patterns, and interpret model behavior in a chemically relevant context.
Methodology
LatentFlow utilizes a multi-level visual analytics workflow that allows users to explore high-dimensional latent spaces. It groups embeddings into clusters and employs a modified Sankey diagram to visualize transitions between these clusters across different layers and model states. The system also incorporates interactive filtering and summary visualizations to manage complexity and enhance user experience.
Results
The evaluation of LatentFlow through two case studies revealed that it effectively assists scientists in understanding the evolution of latent spaces, identifying significant molecular patterns, and interpreting model behavior in a way that aligns with chemical knowledge.
Implications
LatentFlow has the potential to enhance the interpretability of molecular GNNs, facilitating better understanding and application of machine learning in chemistry and materials science. It allows researchers to leverage their domain knowledge in conjunction with model outputs, potentially leading to improved model design and application in real-world scenarios.
MA-DAR: Manifold-Aligned Dynamic Adaptive Routing for Continual Temporal Knowledge Graph Reasoning
Graph Learning
Time Series
Optimization
- MA-DAR is a lightweight, plug-and-play framework for replay representation fusion in continual TKG reasoning.
- The framework addresses critical representation conflicts: norm domination and semantic blurring.
- Dynamic gating and manifold alignment are key components that enhance the integration of historical and current representations.
- Extensive experiments show that MA-DAR outperforms existing TKG encoders across multiple benchmarks.
Read more
MA-DAR: Manifold-Aligned Dynamic Adaptive Routing for Continual Temporal Knowledge Graph Reasoning
Summary
The paper introduces MA-DAR (Manifold-Aligned Dynamic Adaptive Routing), a novel framework designed to enhance continual temporal knowledge graph (TKG) reasoning. The primary challenge addressed is the integration of newly replayed representations with current ones without losing previously acquired knowledge. Existing methods often neglect how to effectively merge these representations, leading to issues such as norm domination and semantic blurring, which degrade reasoning performance. MA-DAR tackles these issues by aligning replayed and current representations onto a shared manifold, thus mitigating distribution discrepancies. It employs a dynamic gating mechanism that learns dimension-wise fusion weights, allowing for adaptive contributions from both replayed and current representations. Additionally, a polarization regularizer is introduced to encourage decisive routing behaviors, enhancing the stability and effectiveness of knowledge integration. The framework is evaluated across four public continual TKG benchmarks, demonstrating consistent performance improvements over existing TKG encoders under various replay settings. The results are further supported by comprehensive ablation studies and visualization analyses, confirming the effectiveness of the manifold alignment and dynamic adaptive routing in resolving representation conflicts.
Methodology
MA-DAR employs a multi-stage routing protocol that includes Manifold Alignment to normalize representations, Dynamic Gating to compute dimension-wise fusion weights, and a Polarization Penalty to enforce binary-like retention decisions during representation fusion. This decouples the replay representation fusion from the current structural encoding, allowing for more effective integration.
Results
The extensive evaluations on four public continual TKG benchmarks demonstrate that MA-DAR consistently achieves state-of-the-art performance, significantly improving the reasoning capabilities of various TKG encoders while effectively managing different replay settings.
Implications
The MA-DAR framework has potential applications in areas requiring continual learning and reasoning over dynamic data, such as event forecasting, intelligent risk analysis, and other domains that utilize temporal knowledge graphs.
Phylogenetic signal in marine mammal and bird vocalizations captured by audio foundation models: the limited benefit of domain-specific pretraining
Audio & Speech
- Foundation models capture strong phylogenetic signals in marine mammal vocalizations, outperforming traditional MFCC features.
- No significant differences were found among the three foundation models, indicating that domain-specific pretraining may not enhance performance.
- The study extends findings to bird vocalizations, where general-purpose models again perform better than specialized models.
- The results challenge the necessity of domain-specific pretraining in bioacoustics and suggest that general-purpose models can effectively encode evolutionary information.
Read more
Phylogenetic signal in marine mammal and bird vocalizations captured by audio foundation models: the limited benefit of domain-specific pretraining
Summary
This study investigates whether acoustic similarities among marine mammals and birds reflect their evolutionary history, focusing on the phylogenetic signal in vocalizations captured by audio foundation models. The research compares traditional hand-crafted features (MFCC) with three advanced audio models: Audio Spectrogram Transformer (AST), Contrastive Language-Audio Pretraining (CLAP), and a bioacoustics fine-tuned BEATs encoder (BEATs-bio). Using Mantel tests, the study analyzes acoustic distance matrices from 32 marine mammal species and finds that the foundation models recover a strong phylogenetic signal, particularly within the cetacean clade. The correlation is notably high (r = 0.82 for CLAP and BEATs-bio) compared to MFCC, which shows no significant correlation. The study also extends the analysis to 20 bird species, where general-purpose models again outperform domain-specific models like BirdNET. The findings suggest that domain-specific pretraining may not be necessary for recovering phylogenetic signals and that general-purpose models can effectively capture evolutionary information across different taxa. The paper discusses potential reasons for these results, including differences in training objectives and model architecture, and proposes further experiments to explore these factors.
Methodology
The study utilized Mantel tests to compare acoustic distance matrices derived from four audio representations (MFCC, AST, CLAP, BEATs-bio) with a phylogenetic distance matrix across 32 marine mammal species, analyzing 1,754 recordings. The analysis was then repeated for 20 bird species using a similar approach.
Results
The foundation models demonstrated a strong phylogenetic signal within the cetacean clade (CLAP r = 0.82, BEATs-bio r = 0.82, AST r = 0.74), while MFCC showed no significant correlation (r = 0.040). Across all species, the correlation decreased to r ≈0.40 for CLAP. In the bird analysis, general-purpose models again outperformed specialized models, with AST and CLAP showing strong phylogenetic signals (AST r = 0.55, CLAP r = 0.52).
Implications
The findings suggest that general-purpose audio models can effectively capture evolutionary information in vocalizations, potentially influencing future research in comparative bioacoustics and the development of audio analysis tools. This could lead to more efficient methodologies in studying animal communication and evolutionary biology.
Parameter-free Adaptive Sparse Attention via Compression-Based Content Selection
NLP
Large Language Models
Efficient ML
- Introduces compression-guided sparse attention, using gzip compression ratios for adaptive attention mask construction.
- Achieves superior performance to existing learned mask methods without additional parameters or custom kernels.
- Demonstrates significant improvements in long-context scenarios, with performance gaps widening as context length increases.
- Provides a clean decomposition of contributions to performance improvements, highlighting the effectiveness of compression-derived connections.
Read more
Parameter-free Adaptive Sparse Attention via Compression-Based Content Selection
Summary
This paper introduces a novel approach to adaptive sparse attention in Transformers, leveraging classical data compression techniques to create attention masks without requiring additional learnable parameters. The authors propose a method called compression-guided sparse attention, which computes gzip compression ratios for fixed-size blocks of input data. By identifying blocks that are less compressible, the method selectively routes long-range attention to these blocks, which are deemed to contain non-redundant information. This approach significantly outperforms existing methods that rely on learned masks, particularly in long-context scenarios, while maintaining a computational efficiency of O(N) and avoiding the complexities associated with custom kernels or gradient estimators. The results demonstrate that the proposed method achieves a bits-per-byte (BPB) of 1.71 on the PG-19 byte-level language modeling task, outperforming dense attention and other adaptive methods, especially as the sequence length increases.
Methodology
The authors divide the input into fixed-size blocks and compute gzip compression ratios for each block. Attention is routed selectively through blocks with high compression ratios, indicating non-redundant content. This method operates without additional parameters or custom kernels, relying solely on standard library compression techniques.
Results
The proposed method achieves a bits-per-byte (BPB) of 1.71 on the PG-19 dataset, outperforming dense attention (2.89 BPB), BigBird (2.34 BPB), Longformer (3.21 BPB), and a reimplemented SBM-Transformer (3.38 BPB) by up to 1.67 BPB. The performance advantage increases with longer sequences, with the gap over BigBird growing from 0.05 BPB at 4K context to 0.63 BPB at 8K context.
Implications
This approach could lead to more efficient Transformer architectures, particularly for tasks involving long sequences, by reducing the computational burden associated with attention mechanisms. It opens avenues for further research into the intersection of data compression and machine learning, potentially enhancing model performance without increasing complexity.
Pretraining EHR Foundation Models with Patient-Aware Sampling
NLP
Generative Models
Time Series
- Introduction of Patient Sampling as a method to improve EHR model pretraining.
- Demonstration of the bias introduced by traditional Global Stream methods.
- Significant performance improvements in downstream clinical tasks using Patient Sampling.
- Highlighting the importance of sequence construction in autoregressive models for EHR data.
Read more
Pretraining EHR Foundation Models with Patient-Aware Sampling
Summary
This paper addresses the limitations of current autoregressive foundation models for electronic health records (EHRs) that utilize standard pretraining methods derived from language modeling. The authors highlight that these methods often mix patient data, leading to potential biases, particularly when longer patient records dominate the training process. To mitigate this issue, they propose a novel pretraining sequence-construction method called Patient Sampling, which allows for controlled distribution of training signals across patients. The authors conduct a comparative analysis between their Patient Sampling method and the conventional Global Stream approach, demonstrating that Patient Sampling significantly enhances performance on real-world EHR datasets, specifically MIMIC-IV v2.2 and v3.1. Their findings underscore the importance of sequence construction in autoregressive EHR models, marking a significant step in optimizing pretraining strategies for clinical applications.
Methodology
The authors compare three methods for constructing training sequences: Global Stream, Patient Chunks, and Patient Sampling. They focus on how these methods affect the distribution of training signals across patients, using a decoder-only transformer model with a next-token prediction objective. The performance of these methods is evaluated on the MIMIC-IV datasets, measuring Macro AUROC and AUPRC metrics.
Results
Patient Sampling outperformed the Global Stream baseline across various clinical tasks, showing improved Macro AUROC and AUPRC scores. The results indicate that the choice of pretraining sequence construction significantly impacts the model's performance, with Patient Sampling providing a more equitable training signal distribution.
Implications
The findings suggest that optimizing pretraining methods for EHR models can lead to better clinical predictions and outcomes. This research could influence future work in EHR modeling and the design of foundation models in healthcare, emphasizing the need for patient-aware approaches in data handling.
Self-Poisoning in Adaptive Out-of-Distribution Detection: A Sharp-Threshold Theory and Certified Label-Free Calibration
Theory
- Introduces a sharp-threshold theory for self-poisoning in adaptive OOD detection.
- Proposes WARDEN, a certified admission method that prevents feedback loops in memory-bank detectors.
- Develops CDC for label-free calibration under drift, ensuring controlled false positive rates.
- Demonstrates empirical validation of theoretical predictions across 96 experimental settings.
Read more
Self-Poisoning in Adaptive Out-of-Distribution Detection: A Sharp-Threshold Theory and Certified Label-Free Calibration
Summary
This paper addresses the challenges of test-time adaptive out-of-distribution (OOD) detection, particularly focusing on the phenomenon of self-poisoning in memory-bank detectors. The author establishes a theoretical framework that models the impurity of the memory bank as a generalized Pólya urn process, demonstrating that the adaptation process converges to a mean-field equilibrium. The paper identifies a critical threshold for contamination, above which the detector's performance collapses. A novel solution, termed WARDEN, is proposed to sever the feedback loop that leads to self-poisoning by utilizing a certified admission gate that relies on a frozen reserve of data, ensuring that false positives are controlled without the need for labels. Additionally, the paper introduces a method called CDC to address static calibration failures under drift, which also operates without labels. The results indicate that the proposed methods effectively maintain low false positive rates and improve detection performance across various settings, while also establishing a theoretical impossibility result regarding the indistinguishability of drift and contamination in label-free scenarios.
Methodology
The paper employs a theoretical approach based on stochastic processes to model the dynamics of memory bank impurity. It uses generalized Pólya urn models to analyze the convergence behavior of the detection system. Empirical evaluations are conducted across multiple settings to validate the theoretical predictions, and the proposed methods (WARDEN and CDC) are tested for their effectiveness in controlling false positive rates and adapting to drift.
Results
The findings reveal that the predicted contamination threshold aligns with observed performance collapses in ungated detectors, with a significant drop in AUROC. WARDEN successfully maintains a mean realized false positive rate of 0.056 across various configurations, while CDC effectively calibrates the detector under drift, achieving a mean false positive rate of 0.042 in tested scenarios.
Implications
The results have significant implications for the deployment of adaptive OOD detectors in real-world applications, particularly in environments where data drift and contamination are prevalent. The methods proposed can enhance the reliability and robustness of such systems, making them more suitable for production use without the need for labeled data.
Cross-Domain Off-Policy Evaluation and Learning for Contextual Bandits
Reinforcement Learning
Optimization
Theory
- Introduction of Cross-Domain OPE/L to leverage data from multiple domains.
- Development of the COPE estimator for unbiased evaluation under challenging conditions.
- Demonstration of improved performance in OPE/L tasks with few-shot data and new actions.
- Theoretical guarantees of COPE's bias properties.
Read more
Cross-Domain Off-Policy Evaluation and Learning for Contextual Bandits
Summary
This paper addresses the challenges of Off-Policy Evaluation and Learning (OPE/L) in contextual bandits, particularly in scenarios with few-shot data, deterministic logging policies, and new actions. Existing OPE/L methods struggle in these situations due to high variance and limited exploration in logged data. To overcome these limitations, the authors propose a novel problem formulation called Cross-Domain OPE/L, which utilizes historical logged data from both the target domain and other source domains. This approach allows for effective evaluation and optimization of new policies even when the target domain has limited data. The authors introduce a new estimator, Cross-domain Off-Policy Evaluation (COPE), which decomposes the expected reward function into domain-cluster and domain-specific effects. COPE employs multiple importance weighting to estimate the domain-cluster effect and uses reward regression for the domain-specific effect. Theoretical analysis shows that COPE can provide unbiased estimates even in challenging scenarios. Empirical evaluations demonstrate that COPE significantly outperforms existing methods, particularly in cases with few target data and unexplored actions.
Methodology
The authors propose the COPE estimator, which combines multiple importance weighting and reward regression to effectively utilize both target and source domain data. The expected reward function is decomposed into domain-cluster and domain-specific effects to enhance the accuracy of policy evaluation and learning.
Results
Empirical evaluations indicate that COPE outperforms traditional OPE/L methods, particularly in scenarios with limited target data and new actions. The results highlight COPE's ability to provide unbiased estimates and improve policy learning effectiveness.
Implications
The proposed Cross-Domain OPE/L framework can be applied in various fields such as personalized medicine, content recommendations, and advertising, where historical data from different domains can enhance decision-making processes without the need for risky online testing.
Measuring the Dependency Gap: Diagnosing Inter-Column Fidelity in Tabular Generative Models
Generative Models
- Introduces a dependency-aware fidelity diagnostic for tabular generative models.
- Demonstrates that standard metrics are inadequate for measuring inter-column dependencies.
- Identifies a significant dependency gap in a state-of-the-art generator affecting minority-class utility.
- Confirms that the gap is not due to structural limitations of the generative objective.
Read more
Measuring the Dependency Gap: Diagnosing Inter-Column Fidelity in Tabular Generative Models
Summary
This paper addresses the critical issue of inter-column dependency in synthetic tabular data generated by machine learning models. While existing metrics primarily assess marginal distributions, they fail to capture the dependencies between columns, which are essential for accurately representing minority classes in imbalanced datasets. The author introduces a novel dependency-aware fidelity diagnostic, XGB-C2ST, which decomposes the two-sample test into marginal, dependency, and numerical-categorical cross components. This diagnostic reveals a significant dependency gap in a state-of-the-art flow-matching generator, TabbyFlow/EF-VFM, indicating that destroying dependencies severely impacts minority-class utility. The study further explores whether this gap is due to structural limitations of the generative model, concluding that it is not. Instead, the gap persists even with a 16-fold increase in model capacity, suggesting that the lack of direct dependency supervision is the root cause. The findings caution against the assumption that architectural changes or increased capacity can resolve dependency issues in generative models.
Methodology
The paper employs a classifier two-sample test using gradient-boosted trees (XGB-C2ST) to evaluate the fidelity of synthetic tabular data. It decomposes the test score into components that measure marginal fidelity, dependency fidelity, and numerical-categorical interactions, providing a clearer understanding of how well dependencies are preserved in generated data.
Results
The analysis reveals that existing metrics fail to detect a real dependency gap in synthetic data. The XGB-C2ST diagnostic exposes this gap, showing that destroying dependencies leads to a collapse in minority-class performance. Additionally, increasing model capacity by 16 times does not mitigate the gap, indicating that the issue lies in the lack of direct supervision for dependencies rather than model capacity or structure.
Implications
The findings suggest that practitioners should be cautious when evaluating synthetic tabular data, particularly in applications involving minority classes. The proposed diagnostic can help improve the design of generative models by emphasizing the importance of preserving inter-column dependencies, which is crucial for maintaining the utility of synthetic data in sensitive domains like fraud detection and clinical risk assessment.
On the Convergence of Stochastic Low-Rank Adaptation
Theory
Optimization
Efficient ML
- LoRA-GD's full-gradient evaluations complexity is improved to O(ϵ−4).
- LoRA-NSGDM achieves O(ϵ−8) stochastic oracle complexity for finding ϵ-stationary points.
- LoRA-STORM further reduces stochastic oracle complexity to O(ϵ−6) under mean-square smoothness.
- The paper provides a comprehensive analysis of LoRA's convergence in both deterministic and stochastic settings.
Read more
On the Convergence of Stochastic Low-Rank Adaptation
Summary
This paper investigates the convergence properties of Low-Rank Adaptation (LoRA), a technique that optimizes a low-rank update to a frozen pretrained weight matrix in machine learning. The authors refine the analysis of LoRA's deterministic gradient descent (LoRA-GD) and demonstrate that it requires O(ϵ−4) full-gradient evaluations to achieve an ϵ-stationary point, a significant improvement over the previously established exponential complexity of exp{O(ϵ−2)}. Furthermore, they explore the stochastic version of LoRA, introducing LoRA-NSGDM, which utilizes unbiased gradient estimates and achieves an oracle complexity of O(ϵ−8) for finding an ϵ-stationary point. Additionally, under mean-square smoothness conditions, they propose LoRA-STORM, which enhances the stochastic oracle complexity to O(ϵ−6). The paper addresses two critical questions regarding the iteration complexity of LoRA-GD and the convergence of stochastic LoRA methods under finite variance, providing affirmative answers through their contributions.
Methodology
The authors employ a theoretical analysis approach, refining existing convergence proofs for LoRA-GD and developing new algorithms (LoRA-NSGDM and LoRA-STORM) that incorporate techniques like joint normalization and variance reduction to improve convergence rates.
Results
The main results include the establishment of polynomial convergence bounds for LoRA-GD, LoRA-NSGDM, and LoRA-STORM, demonstrating significant reductions in oracle complexity compared to previous methods. Specifically, the paper shows that O(ϵ−4), O(ϵ−8), and O(ϵ−6) evaluations are sufficient for the respective methods to achieve ϵ-stationarity.
Implications
These findings suggest that LoRA can be more efficiently utilized in large-scale machine learning applications, particularly in scenarios requiring low-rank adaptations, such as federated learning and decentralized fine-tuning, where computational resources are limited.
CARNet Cycle-Conditioned Core Aggregation and Redistribution for Multivariate Time Series Forecasting
Time Series
- Introduction of CARNet, a cycle-conditioned architecture for multivariate time series forecasting.
- Development of Multihead Core Aggregation (MHCA) to enhance cross-variate interactions.
- Demonstrated consistent performance improvements over existing transformer and non-attention models.
- Maintains linear complexity in modeling cross-variate dependencies.
Read more
CARNet Cycle-Conditioned Core Aggregation and Redistribution for Multivariate Time Series Forecasting
Summary
The paper presents CARNet, a novel framework for multivariate time series forecasting that addresses the challenge of accurately modeling cross-variate dependencies, particularly in the presence of strong periodic patterns. Traditional attention-based methods often suffer from quadratic complexity, making them less scalable with increasing variates. In contrast, CARNet employs a Cycle-Conditioned Core Aggregation and Redistribution approach, which integrates global recurrent cycle information into efficient core-based interaction modeling. The architecture features a Multihead Core Aggregation (MHCA) mechanism that enhances cross-variate interactions by partitioning variate representations into multiple subspaces, allowing for the extraction of compact representations. The global cycle-conditioned core is then redistributed back into the variate embeddings, facilitating structured interaction modeling while maintaining linear complexity. Extensive experiments across 12 real-world multivariate forecasting benchmarks demonstrate that CARNet consistently outperforms both transformer and non-attention baselines across various prediction horizons, showcasing its efficiency and effectiveness in capturing complex dependencies in multivariate time series data.
Methodology
CARNet utilizes a cycle-conditioned architecture that incorporates learnable recurrent cycles as global conditioning signals. It employs a star-shaped aggregation mechanism and introduces the Multihead Core Aggregation (MHCA) to partition variate features into multiple interaction subspaces, extracting and combining core representations efficiently. The global representation is then redistributed back into the variate embeddings to enable structured cross-variate interactions.
Results
CARNet achieved superior forecasting performance across multiple horizons on 12 real-world multivariate time series datasets, outperforming strong transformer and non-attention baselines while preserving linear complexity in computations.
Implications
The proposed CARNet framework has significant implications for various applications requiring accurate multivariate time series forecasting, such as energy management, medical predictions, climate modeling, and traffic analysis. Its efficient modeling approach can facilitate real-time forecasting in resource-constrained environments.
Interpretable EEG biomarkers with bag-of-waves: Spatial and temporal waveform dictionaries for low-data regimes
Time Series
Interpretability
- Introduces bag-of-waves, an interpretable EEG analysis framework that learns waveform dictionaries without labels.
- Enhances representation with n-grams for temporal structure and regional/cross-channel spatial atoms.
- Demonstrates competitive performance against deep learning models across three diverse EEG datasets.
- Maintains interpretability by allowing clinicians to inspect learned waveforms directly.
Read more
Interpretable EEG biomarkers with bag-of-waves: Spatial and temporal waveform dictionaries for low-data regimes
Summary
This paper presents a novel framework called 'bag-of-waves' for analyzing electroencephalography (EEG) data, which aims to provide interpretable biomarkers while operating effectively in low-data regimes. Traditional EEG analysis often relies on predefined spectral features or complex deep learning models, which can be biased or require large datasets. The bag-of-waves approach learns a small dictionary of recurring EEG waveform templates (atoms) using a shift-invariant k-means clustering method without the need for labels. The continuous EEG signals are then converted into sequences of atom tokens, which can be used in simple classifiers or clustering algorithms. The authors enhance this representation by introducing n-grams to capture temporal dynamics and by developing regional and cross-channel spatial atoms for multichannel EEG data. The method is evaluated on three distinct datasets: a low-data scenario with mouse genotype clustering, a resting-state dementia classification task, and a high-data benchmark for clinical EEG events. Results show that bag-of-waves achieves competitive performance compared to state-of-the-art deep learning models while maintaining interpretability and requiring significantly fewer parameters. This framework allows for the recovery of clinically relevant waveform morphologies, making it a valuable tool for EEG analysis, particularly in clinical settings where data may be limited.
Methodology
The bag-of-waves framework employs shift-invariant k-means clustering to learn a dictionary of EEG waveform templates (atoms) from continuous EEG data. The method tokenizes the EEG signals into sequences of atom tokens, which can then be analyzed using simple classifiers or clustering techniques. The framework is extended to include n-grams for capturing temporal relationships and to accommodate regional and cross-channel spatial atoms for multichannel EEG data.
Results
The bag-of-waves method was tested on three datasets, achieving performance comparable to state-of-the-art deep learning models while using a fraction of the parameters. It effectively classified EEG data in low-data scenarios and demonstrated the ability to recover clinically relevant waveform morphologies, which can be validated by neurophysiologists.
Implications
The bag-of-waves framework has significant implications for EEG analysis in clinical settings, particularly for conditions where data is scarce. Its interpretability and efficiency make it a promising tool for diagnosing neurological conditions and could facilitate the development of personalized treatment strategies based on EEG biomarkers.
Physically Constrained Federated Additive Models for O-RAN SLA-Risk Prediction
Federated Learning
Interpretability
- Unconstrained neural additive models can produce physically implausible predictions, undermining trust in SLA risk assessments.
- Monotone FedNAM enforces monotonicity constraints on KPIs, ensuring predictions adhere to known physical behaviors.
- The model is designed for federated learning, allowing base stations to train on local data without sharing sensitive information.
- Monotone FedNAM improves shape consistency from 0.71 to 1.00 and reduces uplink traffic by 65% while maintaining acceptable AUC performance.
Read more
Physically Constrained Federated Additive Models for O-RAN SLA-Risk Prediction
Summary
This paper addresses the challenge of predicting service-level agreement (SLA) violations in Open Radio Access Networks (O-RAN) while ensuring model interpretability and physical validity. The authors highlight the limitations of traditional neural additive models (NAMs) that, despite being interpretable, can produce physically implausible predictions, such as indicating increased SLA violation risk with improved channel quality. This issue is exacerbated in federated learning settings, where non-independent and identically distributed (non-IID) data can distort model training. To overcome these challenges, the authors propose Monotone FedNAM, a federated additive model that enforces monotonicity constraints on key performance indicators (KPIs) with clear physical directions. This model is designed to maintain interpretability while ensuring that the predictions align with known wireless physics. The evaluation of Monotone FedNAM on the ColO-RAN testbed dataset demonstrates its effectiveness in eliminating monotonicity violations, improving shape consistency, and reducing uplink traffic while maintaining a reasonable area under the curve (AUC) performance. The paper concludes that physically constrained federated additive models can significantly enhance the reliability of SLA risk inference in multi-tenant O-RAN environments.
Methodology
The authors developed Monotone FedNAM, a federated neural additive model that constrains certain KPIs to preserve monotonicity during aggregation. The model is trained on local data from multiple base stations, with only model updates shared, ensuring data privacy. The training process involves windowing raw per-slice KPIs and using them to predict SLA violation probabilities.
Results
Monotone FedNAM successfully eliminated all monotonicity violations, achieving a shape consistency score of 1.00. The model also generalized well to unseen scheduling policies and reduced uplink traffic by 65%, with a minimal decrease in AUC performance (0.04 to 0.07).
Implications
The findings suggest that incorporating physical constraints into federated learning models can enhance the reliability and interpretability of predictions in sensitive environments like O-RAN. This approach can be applied to other domains requiring trust and transparency in machine learning predictions.
Complexity Bounds and Approaches to Learning Projected Gradient Descent Solver Iterates
Optimization
Generative Models
Theory
- Introduction of a k-neighborhood data collection strategy to enhance training datasets for generative models.
- Derivation of a generalization bound using Rademacher complexity that emphasizes the role of intermediate solver iterates.
- Focus on one-sided box-constrained quadratic programs solved by projected gradient descent.
- Connection of the proposed methods to the GLENS global search technique, highlighting their applicability in optimization.
Read more
Complexity Bounds and Approaches to Learning Projected Gradient Descent Solver Iterates
Summary
This paper addresses the challenge of data scarcity in training generative models for parametric optimization problems, particularly in the context of projected gradient descent (PGD) solvers. The authors propose a k-neighborhood data collection strategy that retains not only the final converged solutions from solver runs but also intermediate iterates, thereby augmenting the dataset without incurring additional computational costs. They derive a generalization bound based on Rademacher complexity, which highlights the significance of the k-neighborhoods in improving the efficiency of the data-model-optimization loop. The study focuses on one-sided box-constrained quadratic programs and illustrates the behavior of PGD through examples. The findings suggest that leveraging solver iterates can enhance the dynamic data-driven applications systems (DDDAS) paradigm by facilitating more effective generative learning and optimization processes. The paper also connects its analysis to GLENS, a data-efficient global search method, indicating a broader impact on optimization strategies.
Methodology
The authors formalize the k-neighborhood data collection approach, retaining the last k+1 iterates from each solver run. They analyze the convergence behavior of projected gradient descent for box-constrained quadratic programs and derive a generalization bound based on Rademacher complexity. The methodology includes the use of a denoising diffusion probabilistic model (DDPM) for generative learning, leveraging the retained solver iterates to train the model effectively.
Results
The paper demonstrates that retaining intermediate solver iterates significantly increases the amount of training data available for generative models without additional solver runs. The derived generalization bound indicates that the k-neighborhood approach can effectively capture the local behavior of the optimization landscape, leading to improved performance in generating initial guesses for optimization problems.
Implications
The findings suggest that incorporating intermediate solver iterates into generative models can reduce the data requirements for training, thereby enhancing the efficiency of optimization processes in dynamic data-driven applications. This approach could lead to more effective decision-making and optimization strategies across various fields that rely on parametric optimization.
LunarFM: A Shared Multimodal Representation of the Moon's Surface
Multimodal
- LunarFM integrates data from six instruments across three lunar missions into a shared embedding space.
- The model supports diverse applications including similarity search and geological classification.
- A machine learning-ready dataset (LunarChips) is provided for pretraining and evaluation.
- The pretrained multimodal masked autoencoder generates dense representations for lunar surface properties.
Read more
LunarFM: A Shared Multimodal Representation of the Moon's Surface
Summary
The paper presents LunarFM, a multimodal foundation model designed to create a unified representation of the Moon's surface by integrating diverse orbital remote-sensing data from multiple lunar missions. The model assimilates observations from six different instruments across three lunar missions, resulting in a shared embedding space that supports various downstream applications such as similarity search, few-shot resource mapping, mineral abundance regression, and geological unit classification. The authors provide a comprehensive dataset, LunarChips, which includes co-registered multimodal observations covering latitudes from 70°S to 70°N, and a pretrained multimodal masked autoencoder that generates a 768-dimensional representation of lunar surface properties. The paper highlights the challenges of integrating heterogeneous lunar datasets and demonstrates how LunarFM addresses these issues by reducing the need for bespoke modeling workflows. Evaluation experiments showcase the model's capabilities in reconstructing multimodal inputs and performing geological classifications, thereby facilitating efficient scientific investigations and resource-oriented analyses.
Methodology
LunarFM employs a multimodal architecture that assimilates diverse orbital measurements into a shared embedding space. It utilizes a masked autoencoder for pretraining on a curated dataset of lunar observations, facilitating the generation of dense representations that can be applied to various downstream tasks.
Results
The evaluation of LunarFM demonstrated its effectiveness in several tasks, including multimodal input reconstruction, similarity search, mineral mapping, and geological classification. The model's representations were shown to be robust and versatile, enabling efficient analysis of lunar resources.
Implications
LunarFM has significant implications for lunar exploration and resource utilization, providing a framework for integrating and analyzing large-scale lunar data. It can enhance scientific investigations and support the development of sustainable lunar missions by facilitating resource mapping and geological studies.
Beyond Negative-Ridge Endpoints: Mixed-Sign Spectral Regularization via Negative-Shifted Gradient Descent
Theory
Optimization
- Introduces negative-shifted gradient descent (NS-GD) for mixed-sign spectral regularization.
- NS-GD allows for effective management of weak spectral directions in overparameterized linear regression.
- Identifies a Marchenko–Pastur barrier for optimal shift in implicit penalty cancellation.
- Demonstrates polynomial improvement in risk over traditional ridge regression and early stopping methods.
Read more
Beyond Negative-Ridge Endpoints: Mixed-Sign Spectral Regularization via Negative-Shifted Gradient Descent
Summary
This paper addresses the limitations of traditional ridge regression and early-stopped gradient descent in overparameterized linear regression settings. The author introduces a novel approach called negative-shifted gradient descent (NS-GD), which allows for mixed-sign spectral regularization. The key innovation is that NS-GD can effectively manage weak spectral directions, enabling a more flexible correction that raises signal-rich prefixes while controlling the lower spectrum. The paper demonstrates that the NS-GD method can escape the constraints of the stable negative-ridge endpoint, which is limited by the smallest empirical eigenvalue. Through theoretical analysis, the author identifies a Marchenko–Pastur barrier that indicates the optimal shift for canceling implicit penalties. The results show that NS-GD outperforms traditional methods by a polynomial factor in risk under specific conditions, thus providing a more effective regularization strategy in high-dimensional settings.
Methodology
The paper develops a two-parameter spectral regularization path using NS-GD, which operates under a negative ridge parameter. The method is analyzed both in discrete and continuous-time settings, with a focus on the spectral filter's behavior. The author employs localized Duhamel integrals to manage noncontractive dynamics and establishes a finite-grid hold-out inequality for validation.
Results
The main results indicate that NS-GD significantly improves upon the stable negative-ridge endpoint by allowing for a controlled prefix of spectral directions to be anti-shrunk while maintaining lower directions shrunk or exposure-controlled. The theoretical framework supports the effectiveness of NS-GD in achieving lower risk compared to traditional methods, particularly in scenarios characterized by benign overfitting.
Implications
The findings suggest that NS-GD can be a powerful tool for practitioners dealing with high-dimensional linear regression problems, particularly in fields where overparameterization is common. The ability to effectively manage spectral directions could lead to better model performance and generalization in various applications.
RED-PIM: Reducing Data Movement for Transformers using Processing-in-Memory
NLP
Large Language Models
Efficient ML
- RED-PIM reduces attention operation latency by minimizing inter-bank data movement.
- It reorganizes attention computations to operate on smaller d×d matrices instead of full N×N matrices.
- The approach leads to substantial inference time reductions, particularly for longer sequences.
- RED-PIM maintains or improves model accuracy while significantly reducing memory usage.
Read more
RED-PIM: Reducing Data Movement for Transformers using Processing-in-Memory
Summary
The paper introduces RED-PIM, an innovative approach aimed at enhancing the efficiency of transformer models by minimizing data movement during attention operations. Transformers are widely utilized in various domains, but their performance is often hindered by the substantial data transfer between processing units and memory, particularly during self-attention computations. Traditional Processing-In-Memory (PIM) implementations face challenges due to inter-bank communication overhead and limited memory bank capacity. RED-PIM addresses these issues through an algorithm-architecture co-design that reduces attention latency from O(N^2) to O(N) and compresses intermediate attention matrices from N×N to d×d, where d is significantly smaller than N. The methodology involves reorganizing matrix operations, performing computations locally, and optimizing data transfer strategies. The results demonstrate significant reductions in inference time, ranging from 16.05% to 99.99% (geometric mean of 66.42%), with the most substantial gains observed for longer sequences. On real-world datasets, RED-PIM achieves a 99.60% performance improvement for long documents and a 13.44% improvement for shorter ones, all while maintaining or enhancing accuracy. This work showcases RED-PIM's potential for scalable and efficient transformer inference, making it a significant contribution to the field.
Methodology
The authors employed an algorithmic optimization of the Scaled Dot-Product Attention (SDPA) to enhance data locality and reduce data movement. They integrated architectural enhancements with FIMDRAM to facilitate in-memory processing and introduced a hierarchical aggregation strategy to optimize inter-bank communication. A specialized simulation framework was also developed to evaluate data movement in high-bandwidth memory architectures.
Results
The evaluation of RED-PIM showed inference time reductions between 16.05% and 99.99%, with a geometric mean improvement of 66.42% compared to baseline PIM implementations. For long documents, performance improved by 99.60%, while shorter documents saw a 13.44% enhancement, all without sacrificing accuracy.
Implications
The findings suggest that RED-PIM can significantly enhance the efficiency of transformer models, making them more scalable and suitable for applications requiring processing of long sequences, such as natural language processing and large language models.
Bounding the Causal Impact of ML-assisted Decision-Making via Counterfactual Correctness
Theory
- Introduces a partial-identification approach for evaluating the causal impact of ML models using prior RCT data.
- Incorporates assumptions about counterfactual correctness and subgroup predictive performance.
- Extends existing causal frameworks to allow for data-driven estimation of model performance.
- Demonstrates the method's effectiveness through a simulation study, providing more informative bounds.
Read more
Bounding the Causal Impact of ML-assisted Decision-Making via Counterfactual Correctness
Summary
This paper addresses the challenge of evaluating the causal impact of machine learning (ML) models in high-risk decision-making domains, such as healthcare and criminal justice, where randomized control trials (RCTs) are often impractical due to model updates. The authors propose a novel partial-identification approach that utilizes existing RCT data to construct bounds on the causal effects of new ML models. The key innovation lies in incorporating assumptions about individual-level 'counterfactual correctness'—the idea that correct predictions lead to no worse outcomes than incorrect ones—and the relationship between subgroup predictive performance and outcomes. The authors extend previous causal frameworks by allowing for the estimation of model performance from data rather than requiring prior knowledge. They demonstrate their method through a simulation study, showing that their approach yields more informative bounds than existing methods. This framework is particularly useful in diagnostic tasks where true labels are observable, and it can be adapted to scenarios with missing labels under specific conditions.
Methodology
The authors develop a causal framework that incorporates assumptions about counterfactual correctness and subgroup performance. They provide upper and lower bounds for causal effects based on these assumptions and derive a consistent estimator for these bounds. The methodology is validated through a semi-synthetic simulation study.
Results
The simulation study illustrates that the proposed method results in more informative bounds on the causal impact of ML models compared to previous approaches. The incorporation of individual-level prediction accuracy and subgroup performance leads to better estimates of the causal effects of new models.
Implications
The findings suggest that this framework can enhance the evaluation of ML models in high-stakes environments, allowing decision-makers to better understand the potential impacts of model updates. This could lead to more informed deployment of ML systems in critical areas such as healthcare and criminal justice.
Scaling Laws for Classical Machine Learning on Tabular Data: A Benchmark Study
Theory
Efficient ML
- Power laws fit well to classical ML learning curves, with R2 > 0.8 in 77.7% of cases.
- Tree ensembles dominate performance, especially Boosting and Random Forest models.
- Approximate shared exponents can predict model performance across datasets with minimal predictive loss.
- Implementation variance is significant, influenced by unconstrained aspects of the protocol.
Read more
Scaling Laws for Classical Machine Learning on Tabular Data: A Benchmark Study
Summary
This paper investigates the scaling laws of classical machine learning (ML) models applied to tabular data, addressing a gap in the literature regarding how generalization error scales with training set size. The study involved 127 students who followed a fixed protocol across 18 datasets from various classification and regression tasks, resulting in 11,536 training runs and 1,648 fitted power-law curves. The findings reveal that power laws fit well to the data, with tree ensembles (Boosting and Random Forest) outperforming linear models, particularly in classification tasks. The study also identifies approximate shared exponents within model families, suggesting that a single exponent can predict performance across datasets with minimal loss in accuracy. However, the universality of this approach is questioned, as individual dataset fits are favored. Additionally, the research highlights the variance in implementation outcomes among students, indicating that factors such as preprocessing and encoding significantly influence results. The paper contributes to the empirical understanding of classical ML scaling laws and provides practical data-requirement curves for practitioners aiming to achieve specific error thresholds.
Methodology
The study employed a distributed classroom-scale replication approach, where 127 students executed a fixed protocol on 3 assigned datasets each, leading to a comprehensive analysis of 18 datasets across 6 model families. The models were evaluated based on their performance as a function of training set size, and power-law fits were applied to the resulting learning curves.
Results
The analysis yielded 11,536 model fits and 1,648 successful scaling curves. The results indicated that tree ensembles performed best at full data, with Boosting accounting for 50% of datasets and Random Forest for 33%. The study found that while a single family-level exponent could predict performance across datasets, the universality of this approach was not fully supported, as individual dataset fits were generally preferred.
Implications
The findings have significant implications for practitioners in the field of classical machine learning, particularly in understanding how to allocate data resources effectively to minimize error. The practical data-requirement curves can guide decision-making in various applications, including churn prediction and sales forecasting.
Cloud-Native Evaluation-as-a-Service: A Microservices Architecture for Scalable AI Monitoring with Conformal Guarantees
Theory
- EAAS is the first open-source microservices architecture for AI evaluation that integrates conformal prediction, calibration, drift detection, and fairness monitoring.
- The architecture achieves robust empirical coverage and demonstrates the bounded impact of token imputation on coverage guarantees.
- Random Fourier Features-based drift detection shows 100% detection power for various drift scenarios with controlled Type I error rates.
- Fairness monitoring reveals significant demographic disparities in real datasets, highlighting the need for continuous monitoring.
Read more
Cloud-Native Evaluation-as-a-Service: A Microservices Architecture for Scalable AI Monitoring with Conformal Guarantees
Summary
The paper introduces EAAS, a cloud-native architecture designed for continuous and rigorous evaluation of AI models in production. It addresses the limitations of existing tools, which are either offline or proprietary, by providing a microservices-based solution that operationalizes conformal prediction, calibration assessment, drift detection, and fairness monitoring. EAAS consists of six stateless microservices deployed on Kubernetes, enabling composability and formal statistical guarantees. The methodology includes empirical validation of conformal coverage, drift detection using Random Fourier Features, and fairness monitoring on real datasets. The results demonstrate robust performance across various metrics, including consistent empirical coverage, minimal impact from token imputation, and high detection power for drift, alongside revealing significant demographic disparities in fairness monitoring. This architecture aims to fill the gap in existing evaluation frameworks by providing a comprehensive, open-source solution suitable for production environments.
Methodology
EAAS employs a microservices architecture consisting of six components: conformal prediction with Adaptive Prediction Sets, calibration assessment, drift detection using Random Fourier Features, fairness monitoring with bootstrap confidence intervals, a DAG-based orchestrator, and a result storage API. The methodology includes targeted experiments to validate coverage guarantees, imputation impacts, drift detection power, and fairness monitoring results.
Results
EAAS achieves conformal coverage within 1.4 percentage points of the nominal target across 50 random splits, with minimal impact from imputation. The drift detection method shows 100% detection power for both mild and severe drift scenarios, while fairness monitoring on the UCI ADULT INCOME dataset reveals a demographic parity gap of 0.33. Core services maintain a latency of under 2 ms for batch sizes of 100.
Implications
The EAAS architecture provides a scalable and rigorous framework for continuous AI model evaluation, which is crucial for maintaining trustworthiness in AI deployments. Its open-source nature allows for extensibility and integration into existing production systems, promoting responsible AI practices.
Molt: A Scalable PyTorch-Native Training Framework for Agentic Reinforcement Learning
Reinforcement Learning
Multimodal
Efficient ML
- Molt provides a compact, readable framework for agentic reinforcement learning, reducing complexity for researchers.
- The framework integrates multiple components into a single asynchronous loop, ensuring performance parity with larger systems.
- A token-first agent boundary allows for seamless integration with existing SDKs without additional integration code.
- Molt emphasizes correctness and navigability, making it easier for researchers and AI coding assistants to work with.
Read more
Molt: A Scalable PyTorch-Native Training Framework for Agentic Reinforcement Learning
Summary
Molt is a novel PyTorch-native training framework designed to streamline the process of agentic reinforcement learning (RL) research. Traditional RL frameworks often impose significant complexity on researchers due to their multi-layered architecture, which complicates algorithm modifications and experimentation. Molt addresses this by providing a compact and readable codebase that allows researchers to easily trace and modify the entire algorithm flow. The framework integrates Ray, vLLM, and NeMo AutoModel into a single asynchronous loop, ensuring that every component remains up-to-date without forking. Molt emphasizes human readability and navigability for AI coding assistants, allowing for efficient prototyping and experimentation. It introduces a token-first agent boundary that captures SDK traffic seamlessly, and maintains high performance comparable to existing state-of-the-art systems, such as Megatron. The authors present four key contributions: a readability-first design, a token-first agent boundary, a fast asynchronous training path, and a principled approach to correctness in agentic RL. The results demonstrate that Molt can achieve similar throughput to larger frameworks while being significantly smaller and more accessible for researchers.
Methodology
Molt employs a PyTorch-native architecture that integrates Ray, vLLM, and NeMo AutoModel within a single asynchronous loop. It focuses on human readability and correctness by establishing invariants around token identity, policy-version semantics, and forward consistency. The framework allows for direct weight synchronization and multimodal training without altering the dense-model programming model.
Results
Molt demonstrates statistical performance parity with state-of-the-art systems like Megatron while maintaining a significantly smaller codebase. The framework's design allows for rapid prototyping and experimentation, enabling researchers to modify algorithms with minimal overhead.
Implications
Molt's design could significantly accelerate the pace of research in agentic reinforcement learning by simplifying the experimentation process. Its open-source nature encourages community contributions and could lead to broader adoption and innovation in RL methodologies.
Dysphagia Risk Stratification in Head and Neck Cancer via Two-Stage PRO-Clinical Stacking
Interpretability
- Introduces a two-stage stacking model for dysphagia risk prediction using PROs and clinical data.
- Demonstrates that individual MDADI responses are more informative than composite scores for predicting swallowing impairment.
- Provides a framework that allows for single-visit assessments, addressing the need for timely intervention.
- Highlights the importance of integrating patient-reported symptoms with clinical variables for better risk assessment.
Read more
Dysphagia Risk Stratification in Head and Neck Cancer via Two-Stage PRO-Clinical Stacking
Summary
This study addresses the challenge of timely identification of dysphagia risk in head and neck cancer (HNC) survivors, a significant late effect of treatment. Traditional assessment methods, such as videofluoroscopic imaging, are limited due to their requirement for specialized equipment and trained personnel. Instead, the authors propose a two-stage PRO-clinical prediction framework that utilizes patient-reported outcomes (PROs) and structured clinical variables to predict swallowing impairment risk without the need for imaging. The framework integrates PRO responses, specifically from the MD Anderson Dysphagia Inventory (MDADI), with clinical factors to create a unified risk assessment model. The study demonstrates that individual MDADI responses provide predictive information beyond composite scores, revealing significant symptom patterns and clinical risk factors associated with swallowing impairment. This approach offers a practical, imaging-free method for dysphagia risk stratification, enhancing the potential for early intervention in HNC survivorship care.
Methodology
The authors developed a two-stage stacking model that first generates a risk score from PRO responses and then incorporates structured clinical covariates for further adjustment. The model was evaluated through 50-seed ablation experiments and interpretability analyses to assess the contributions of various symptom items and clinical factors.
Results
The study found that the proposed model effectively predicts swallowing impairment risk using single-visit MDADI responses, revealing that these responses contain valuable predictive information. The interpretability analyses identified specific symptom patterns and clinical factors that correlate with dysphagia, supporting the model's clinical relevance.
Implications
This research has significant implications for improving dysphagia risk stratification in HNC survivorship care, enabling healthcare providers to identify at-risk patients earlier and implement timely interventions without relying on resource-intensive imaging techniques.