AI-generated summaries
Today's ML research,
without the noise.
Daily summaries of the latest machine learning papers from arXiv, processed every 8 hours.
43
Papers today
8h
Update frequency
7
Days of history
Graph-Regularized Deep Learning for EEG-Based Emotion Recognition with Psychologically-Grounded Label Structure
Graph Learning
Time Series
Audio & Speech
- Introduces a graph-based approach to model emotional relationships in EEG emotion recognition.
- Implements three regularization strategies to incorporate psychological proximity into training.
- Demonstrates architecture-agnostic improvements across multiple deep learning models.
- Achieves up to +5.42% accuracy and 39% reduction in implausible misclassifications on benchmark datasets.
Read more
Graph-Regularized Deep Learning for EEG-Based Emotion Recognition with Psychologically-Grounded Label Structure
Summary
This paper addresses the challenge of EEG-based emotion recognition by proposing a novel graph-regularized deep learning framework that incorporates the psychological interdependencies of emotions. Traditional deep learning methods treat emotion classes as isolated labels, which contradicts established psychological theories regarding the relationships between emotions. The authors introduce an emotion graph where nodes represent emotions and edges encode their psychological proximity based on dimensional emotion theories. They implement three complementary regularization strategies: Graph Label Smoothing, Graph Laplacian-based commuting distance, and Sliced Wasserstein Distance, which penalize misclassifications based on the established emotional topology. The framework is evaluated using three different backbone architectures: AudioTransformer, Conformer, and DCGNN, demonstrating its architecture-agnostic benefits. Experiments on the SEED-IV and SEED-V datasets show significant improvements in classification accuracy and a reduction in psychologically implausible misclassifications, thereby enhancing the clinical relevance of emotion recognition systems.
Methodology
The authors construct an emotion graph based on Russell's circumplex model, where emotions are positioned in a two-dimensional valence-arousal space. They employ three regularization strategies: Graph Label Smoothing for local relationships, Graph Laplacian for global connectivity, and Sliced Wasserstein Distance for optimal transport, to penalize misclassifications based on psychological proximity. The framework is tested on EEG datasets SEED-IV and SEED-V using various backbone architectures.
Results
The proposed framework shows consistent improvements in emotion classification accuracy, with the best case achieving an increase of 5.42% in accuracy and a 39% reduction in psychologically implausible misclassifications. The results validate the effectiveness of integrating psychological proximity into the training process.
Implications
This research has significant implications for mental health monitoring and affective computing applications, as it enhances the accuracy and clinical relevance of emotion recognition systems. The framework can be applied in psychiatric care to track emotional trajectories and inform treatment decisions.
Systematic Evaluation of Learning Rate Scheduling Strategies Across Heterogeneous Architectures
Optimization
Computer Vision
Efficient ML
- Systematic evaluation of 25 LR scheduling strategies across 30 neural network architectures.
- Architecture-specific preferences for LR schedulers were identified, with CosineAnnealingWarmRestarts and CyclicLR performing best.
- The study generated 3,938 model variants, contributing to the LEMUR nn-dataset.
- The best configuration achieved a top-1 accuracy of 86.45%, with many variants exceeding 80% accuracy.
Read more
Systematic Evaluation of Learning Rate Scheduling Strategies Across Heterogeneous Architectures
Summary
This paper presents a comprehensive study on the impact of learning rate (LR) scheduling strategies on the performance of neural networks across various architectures. The authors systematically evaluate 30 different architectures from convolutional and transformer families using the LEMUR neural network dataset. They implemented 25 different LR scheduler configurations through automated source-code injection and assessed a total of 3,938 model variants on the CIFAR-10 dataset. The findings reveal that the choice of LR scheduler significantly affects classification accuracy, with architecture-specific preferences observed. Notably, CosineAnnealingWarmRestarts and CyclicLR consistently outperformed basic decay strategies. The best configuration achieved a top-1 accuracy of 86.45%, with 237 variants surpassing 80% accuracy. The study contributes a detailed accuracy landscape to the LEMUR dataset, providing a valuable reference for future scheduler selection in neural network training.
Methodology
The authors employed automated source-code injection to implement 25 different LR scheduler configurations across 30 neural network architectures. They evaluated the performance of these configurations on the CIFAR-10 dataset, focusing on top-1 accuracy over five epochs. The study systematically varied only the scheduler while keeping other hyperparameters fixed to isolate the impact of the scheduler on model performance.
Results
The study found that the choice of LR scheduler significantly influences model accuracy, with the best configuration achieving a top-1 accuracy of 86.45%. A total of 237 out of 3,938 variants exceeded 80% accuracy. The results highlighted that different architectures have distinct preferences for specific LR scheduling strategies, with CosineAnnealingWarmRestarts and CyclicLR showing superior performance.
Implications
The findings suggest that careful selection of learning rate schedulers can lead to substantial improvements in model performance. This research provides a systematic framework for future studies and practical guidance for practitioners in the field of deep learning, particularly in optimizing training recipes for diverse neural network architectures.
MPFlow: Learning Budgeted Max-Flow Optimization on the Lightning Network with Deep Graph Reinforcement Learning
Reinforcement Learning
Graph Learning
Optimization
- MPFlow formulates liquidity placement as a budget-constrained combinatorial optimization problem on graphs.
- The method utilizes a message-passing neural network combined with proximal policy optimization and action masking.
- A hub-exclusion curriculum is employed to enhance the policy's learning of capacity-aware placements.
- Extensive experiments show consistent performance improvements over heuristic baselines on the max-flow objective.
Read more
MPFlow: Learning Budgeted Max-Flow Optimization on the Lightning Network with Deep Graph Reinforcement Learning
Summary
This paper addresses the challenge of liquidity placement in the Bitcoin Lightning Network (LN) by framing it as a budget-constrained combinatorial optimization problem on graphs. The authors propose MPFlow, a deep graph reinforcement learning (RL) approach that selects k edge additions to maximize the source-to-sink max-flow, which serves as a measure of routing capacity. The method employs a lightweight agent that integrates a message-passing policy network with proximal policy optimization (PPO) and action masking. A unique hub-exclusion curriculum is introduced, where the training process removes the top hubs from the subgraphs, encouraging the policy to learn effective liquidity placement independent of these hubs. The authors validate their approach through extensive experiments on real LN snapshots, demonstrating that MPFlow consistently outperforms traditional heuristic methods. The agent has been successfully deployed in production, making 4,640 channel-open decisions that allocated over $16 million across 30 nodes, showcasing its practical relevance and effectiveness.
Methodology
The authors developed a graph reinforcement learning agent that combines a message-passing neural network (MPNN) with proximal policy optimization (PPO) and action masking. The training process involves a hub-exclusion curriculum, where the top 50 hubs are removed from the training subgraphs to promote independent learning of liquidity placement.
Results
MPFlow outperformed strong heuristic baselines in maximizing the max-flow objective across multiple experiments on real Lightning Network snapshots. The agent's deployment in production led to 4,640 channel-open decisions, cumulatively allocating 267.3 BTC (over $16 million) across 30 managed nodes.
Implications
The findings suggest that MPFlow can significantly enhance liquidity management in the Bitcoin Lightning Network, potentially leading to improved routing efficiency and transaction throughput. Its deployment in real-world scenarios indicates a promising avenue for optimizing financial networks using advanced machine learning techniques.
Spectral Stability of Pseudoinverse-Based Extreme Learning Machine
Theory
Efficient ML
Optimization
- The smallest singular value of the hidden-layer matrix governs perturbation amplification in ELM output weights.
- Condition number provides a quantitative measure of hidden-layer instability.
- SVD-based methods are more reliable than iterative methods under ill-conditioning.
- Larger training sample sizes generally improve ELM stability, while excessive hidden width can deteriorate it.
Read more
Spectral Stability of Pseudoinverse-Based Extreme Learning Machine
Summary
This paper investigates the spectral stability of the Extreme Learning Machine (ELM), which computes output weights using the MooreβPenrose pseudoinverse. The authors highlight that the numerical stability of ELM is significantly influenced by the conditioning of the hidden-layer matrix. They establish that the smallest singular value of this matrix is crucial for understanding perturbation amplification in output weights, while the condition number serves as a quantitative measure of hidden-layer instability. The study compares SVD-based pseudoinverse computation with iterative hyperpower methods, revealing that SVD methods are more reliable under ill-conditioning. The authors also provide a random feature interpretation of width-dependent conditioning, suggesting that larger sample sizes improve stability, while excessively large hidden widths may worsen conditioning. Experimental results demonstrate that iterative methods struggle with severely ill-conditioned matrices, while SVD-based methods maintain reliability, underscoring the importance of singular-value structure in ELM stability.
Methodology
The authors conducted a spectral stability analysis of the hidden-layer matrix in ELM using singular value decomposition (SVD). They compared SVD-based pseudoinverse computation with iterative methods like NewtonβSchulz and hyperpower iterations. Experiments were performed on synthetic matrices and benchmark datasets (MNIST, Fashion-MNIST, ISOLET) to evaluate convergence behavior, classification accuracy, and the relationship between singular values and conditioning.
Results
The results indicated that iterative methods are effective for well-conditioned and moderately ill-conditioned matrices but fail in severely ill-conditioned scenarios. In contrast, SVD-based methods consistently succeeded, confirming that the singular-value structure of the matrix strongly influences convergence reliability. The experiments showed that larger hidden widths often led to poorer conditioning, as indicated by a decrease in the smallest singular value.
Implications
The findings suggest that careful consideration of the hidden-layer matrix's spectral properties is essential for ensuring the stability and reliability of ELM. This has implications for the design and implementation of ELM in practical applications, particularly in scenarios where numerical stability is critical.
Eigenvalue Calibration for Semantic Embeddings of Large Language Models
NLP
Large Language Models
Theory
- Introduces a novel calibration framework for eigenvalues of semantic embeddings in LLMs.
- Establishes theoretical foundations linking entropy and risk in the context of eigenvalue calibration.
- Demonstrates that current LLMs are overconfident in their predictions and validates the calibration method empirically.
- Proposes a new approach for visualizing calibration through reliability diagrams.
Read more
Eigenvalue Calibration for Semantic Embeddings of Large Language Models
Summary
This paper addresses the critical issue of uncertainty quantification in large language models (LLMs) by proposing a novel framework for calibrating the eigenvalues of semantic embeddings. The authors highlight that conventional calibration methods for classification probabilities cannot be directly applied to eigenvalues, which represent latent outcome probabilities in a different mathematical space. To bridge this gap, the authors interpret LLMs combined with semantic embeddings as density matrix predictors and introduce a method that applies temperature scaling to their eigenvalues. They establish a theoretical foundation for this calibration approach, demonstrating entropy-risk equivalence and deriving a specific calibration inequality for eigenvalues. Empirical evaluations reveal that current LLMs exhibit systematic overconfidence in their predictions, which can be mitigated through the proposed calibration method. The results contribute to advancing the understanding and practical application of uncertainty quantification in semantic embeddings, making them more reliable for real-world applications.
Methodology
The authors propose a framework that interprets LLMs as density matrix predictors. They apply temperature scaling to the eigenvalues of these matrices to optimize calibration. The methodology includes theoretical derivations of calibration inequalities and empirical validation through experiments on real-world datasets.
Results
The experiments show that the proposed temperature scaling method significantly reduces the overconfidence of LLMs, as evidenced by lower expected calibration errors. The theoretical findings are validated through practical applications, demonstrating the effectiveness of the calibration framework.
Implications
The findings have significant implications for the deployment of LLMs in high-stakes applications where reliable uncertainty estimates are crucial. Improved calibration of eigenvalues can enhance decision-making processes and foster better human-AI collaboration.
Spectral Analysis of Dueling Q-Learning
Reinforcement Learning
Theory
- Introduces a theoretical framework for understanding unregularized Dueling Q-Learning.
- Establishes convergence guarantees for the constant step-size recursion.
- Derives a finite-time error bound for the sampled stochastic version of the algorithm.
- Clarifies the roles of value and advantage updates in the Q-function decomposition.
Read more
Spectral Analysis of Dueling Q-Learning
Summary
This paper presents a theoretical analysis of Dueling Q-Learning, a variant of Q-learning that separates the Q-function into a value function and an advantage function to improve learning efficiency. While previous studies have explored tabular dueling Q-learning, they primarily focused on regularized formulations, leaving the unregularized case less understood. This work strengthens the theoretical foundation by providing a direct interpretation of the centered tabular decomposition and establishing convergence guarantees for the unregularized, constant step-size recursion. The author derives a switching linear system representation for deterministic dueling Q-learning and a finite-time error bound for its sampled stochastic version. The analysis clarifies the distinct roles of value and advantage updates, which act as different gains on the action-common and action-differential components of the Q-function. The results indicate that the algorithm converges to a first-moment neighborhood of the optimal Q-function, with the neighborhood size diminishing as the common scalar gain approaches zero.
Methodology
The paper employs an orthogonal decomposition of the tabular Q-function into action-common and action-differential components. It uses switching linear system (SLS) theory to analyze the deterministic and stochastic versions of the Dueling Q-Learning algorithm, deriving convergence conditions based on the joint spectral radius (JSR).
Results
The analysis shows that the deterministic dueling Q-learning error recursion can be expressed as an SLS, leading to convergence under a JSR condition. For the stochastic version, a finite-time error bound in expectation is established, indicating convergence to a neighborhood of the optimal Q-function that shrinks as the scalar gain decreases.
Implications
The findings enhance the theoretical understanding of Dueling Q-Learning, potentially leading to more efficient reinforcement learning algorithms. This could have applications in various high-dimensional decision-making problems where Q-learning is employed.
Super Weights in LLMs and the Failure of Selective Training
Large Language Models
Efficient ML
Theory
- Super Weights do not universally lead to improved training outcomes when targeted in isolation.
- Training Super Weights results in performance collapse, while random parameter training in the same layers improves accuracy.
- LoRA (Low-Rank Adaptation) successfully fine-tunes models by updating entire layers rather than isolated parameters.
- The study validates the structural consistency of Super Weights across diverse inputs.
Read more
Super Weights in LLMs and the Failure of Selective Training
Summary
This paper investigates the concept of Super Weights in large language models (LLMs), which are parameters whose removal significantly degrades model performance. The authors challenge the assumption that training these Super Weights in isolation would enhance model accuracy. Through experiments on OLMo-1B and OLMo-7B, they demonstrate that training only Super Weights or their neighborhoods leads to drastic performance drops, while randomly selecting parameters from the same layers yields better results. The study reveals that the failure to improve performance is specific to targeting Super Weights rather than a general issue with sparsity. The authors also show that parameter importance does not equate to trainability in isolation, emphasizing the need for structured updates across entire layers. Their findings suggest that effective fine-tuning strategies should focus on layer-wide coordination rather than isolated parameter adjustments.
Methodology
The authors conducted a series of experiments involving pruning, direct training, neighborhood training, and low-rank adaptations (LoRA) on two LLMs (OLMo-1B and OLMo-7B). They validated Super Weight consistency across multiple samples and performed ablation studies to isolate the effects of targeting Super Weights versus randomly chosen parameters.
Results
The experiments revealed that training Super Weights in isolation led to accuracy drops to random-guessing levels, while random training in the same layers improved performance. LoRA achieved a 67% accuracy with only 0.16% of parameters updated, demonstrating the effectiveness of structured updates across layers. The results confirmed that the failure to improve performance was specific to targeting Super Weights and not due to sparsity or module choice.
Implications
These findings suggest that fine-tuning strategies for LLMs should prioritize coordinated updates across entire layers rather than focusing on individual parameters. This could lead to more effective and efficient training methods in the development of large language models.
Frequency-Domain Multi-Modality Transportation Modeling
Time Series
Multimodal
- Introduces a novel framework (FreMo) for multi-modality transportation forecasting.
- Utilizes frequency-domain analysis to address challenges in traditional time-domain methods.
- Employs Modality-Wise Frequency Filter (MFF) for spectral refinement and noise suppression.
- Incorporates Frequency-Guided Synergy Integrator (FSI) for selective information aggregation.
Read more
Frequency-Domain Multi-Modality Transportation Modeling
Summary
The paper addresses the challenges of multi-modality transportation forecasting, where urban systems consist of various transportation modes like traffic flow and public transit. Traditional methods struggle due to distinct spectral characteristics of different modalities and their uneven interactions across frequencies. To overcome these limitations, the authors propose a novel framework called Frequency-Domain Multi-Modality modeling (FreMo). This framework utilizes the frequency domain to enable adaptive and selective cross-modality synergy. FreMo introduces a Modality-Wise Frequency Filter (MFF) that refines spectral components for each modality, enhancing informative frequencies while reducing noise. Additionally, it features a Frequency-Guided Synergy Integrator (FSI) that aggregates information across modalities based on their frequency-specific contributions. The effectiveness of FreMo is demonstrated through extensive experiments on real-world datasets, showing consistent outperformance of state-of-the-art baselines in various forecasting scenarios. The proposed method not only improves predictive performance but also enhances generalization across diverse contexts.
Methodology
The methodology involves a two-part approach: first, the Modality-Wise Frequency Filter (MFF) refines the spectral components of each modality, focusing on informative frequencies while suppressing noise. Second, the Frequency-Guided Synergy Integrator (FSI) selectively aggregates information across modalities based on their contributions at different frequencies, allowing for effective cross-modality knowledge sharing.
Results
The results indicate that FreMo consistently outperforms state-of-the-art baselines in multi-modality transportation forecasting tasks. The framework shows improved predictive accuracy and generalization capabilities across various real-world datasets, validating its effectiveness in handling the complexities of multi-modality data.
Implications
The implications of this research extend to urban computing and transportation systems, where enhanced forecasting can lead to better traffic management, improved public transit scheduling, and more efficient urban planning. The framework can also be adapted for other domains that involve multi-modal time series data.
Image classification via a quantum-inspired strategy involving a mixture of experts
Computer Vision
Efficient ML
Theory
- Introduces a hybrid classical-quantum framework for image classification.
- Utilizes a mixture of experts approach to enhance feature extraction and classification.
- Demonstrates improved performance on MNIST and Fashion-MNIST datasets.
- Reduces failure rates of image classification by approximately 50%.
Read more
Image classification via a quantum-inspired strategy involving a mixture of experts
Summary
This paper presents a novel approach to image classification by integrating quantum-inspired techniques with a mixture of experts framework. Traditional convolutional neural networks (CNNs) are commonly used for feature extraction and classification in image processing. However, these classical methods often involve lossy operations like diffusion-based smearing and pooling. The authors propose a hybrid classical-quantum model that utilizes amplitude encoding of images, local unitary operations for convolution, and multiple experts analyzing the same image from different perspectives. The extracted features are then processed by a fully connected neural network for classification. The proposed method is evaluated using the MNIST and Fashion-MNIST datasets, demonstrating a significant improvement in classification performance and a reduction in failure rates compared to individual expert analyses. The computational overhead of this quantum-inspired strategy is manageable on GPU workstations, making it a practical alternative to existing methods. Additionally, the authors discuss the potential for executing the quantum components on actual quantum processors.
Methodology
The methodology involves several key components: amplitude encoding of images into quantum states, iterative convolutional smearing using local unitary operations, and the introduction of multiple experts that process the same image with varying parameters. The final classification is performed using a standard fully connected neural network that integrates features from the different experts.
Results
The results indicate that the joint analysis of multiple experts significantly outperforms individual expert analyses, achieving a reduction in classification failure rates by around 50%. The proposed quantum-inspired strategy shows practical feasibility with moderate computational demands on GPU workstations.
Implications
This research has potential implications for advancing image classification techniques by leveraging quantum-inspired methods, which could lead to more efficient processing of large datasets in various fields such as surveillance, medical imaging, and autonomous systems. The ability to execute quantum components on actual quantum processors may further enhance performance in the future.
Towards the Explainability of Temporal Graph Networks via Memory Backtracking and Topological Attribution
Graph Learning
Interpretability
Time Series
- Introduces a framework for explainability in Temporal Graph Networks that considers both spatial and temporal influences.
- Utilizes topology attribution and memory backtracking to quantify contributions from neighboring and historical events.
- Implements Layer-wise Relevance Propagation (LRP) to ensure accurate attribution of event contributions.
- Demonstrates improved performance over existing explanation methods on multiple temporal graph datasets.
Read more
Towards the Explainability of Temporal Graph Networks via Memory Backtracking and Topological Attribution
Summary
This paper addresses the challenge of explainability in Temporal Graph Networks (TGNs), which are increasingly used in applications such as fraud detection and healthcare forecasting. While TGNs have demonstrated superior predictive accuracy, their black-box nature limits transparency regarding how historical events influence predictions. The authors propose a novel framework that incorporates memory backtracking and topological attribution to enhance the explainability of TGNs. The framework consists of two main components: the topology attribution tree, which captures the influence of neighboring events and their memory vectors, and the memory backtracking tree, which quantifies how historical events shape node memory vectors. By applying Layer-wise Relevance Propagation (LRP), the authors ensure that the total contributions of events align with the model's logits. They also address potential issues with top-k selection by designing optimization objectives to identify significant events. The proposed method is evaluated on nine temporal graph datasets across various tasks, demonstrating that it provides faithful explanations and outperforms existing state-of-the-art methods.
Methodology
The authors developed a framework that combines topology attribution trees and memory backtracking trees to analyze the contributions of neighboring and historical events in TGNs. They applied Layer-wise Relevance Propagation (LRP) to ensure that the contributions attributed to events sum to the model's output logits. Additionally, they designed optimization objectives to enhance the identification of important events, addressing the limitations of traditional top-k selection methods.
Results
The proposed method was tested on nine temporal graph datasets, covering tasks such as node property prediction, link prediction, and graph classification. The results indicated that the framework provided more faithful explanations compared to existing methods and achieved superior performance on various tasks, validating its effectiveness in enhancing the explainability of TGNs.
Implications
The findings of this research have significant implications for the deployment of TGNs in critical areas like fraud detection and healthcare, where understanding model predictions is essential for trust and safety. The proposed explainability framework can facilitate better decision-making by providing insights into the factors influencing predictions, thereby enhancing user trust in these models.
Principled Analysis of Deep Reinforcement Learning Evaluation and Design Paradigms
Reinforcement Learning
Theory
- Critiques the implicit assumptions in DRL research regarding performance and sample complexity.
- Introduces theoretical foundations on scaling laws in DRL.
- Demonstrates through experiments that performance profiles are non-monotonic with respect to sample complexity.
- Highlights the impact of canonical methodological choices on research conclusions.
Read more
Principled Analysis of Deep Reinforcement Learning Evaluation and Design Paradigms
Summary
This paper provides a comprehensive analysis of the evaluation and design paradigms in deep reinforcement learning (DRL). It highlights the significant progress made in DRL over the past decade, particularly through the use of deep neural networks to approximate state-action value functions. The author critiques the canonical evaluation methodologies that have led to incorrect conclusions in the field, particularly the assumption of a monotonic relationship between performance rankings and data regimes. The paper introduces theoretical foundations regarding scaling laws in DRL and demonstrates through extensive experiments that the performance profiles of DRL algorithms exhibit a non-monotonic relationship with sample complexity. This misdirection in research has implications for future studies and the development of DRL algorithms, as it shapes the understanding of capacity and complexity in the field. The findings suggest that many recent advancements in DRL may be based on flawed evaluations, necessitating a reevaluation of methodologies used in low-data regime studies.
Methodology
The author conducts a theoretical analysis of evaluation paradigms in DRL and performs large-scale experiments on a diverse set of baseline algorithms in both low-data and high-data regimes using the Arcade Learning Environment benchmark. The analysis focuses on the relationship between sample complexity and algorithmic performance, revealing biases in current evaluation practices.
Results
The experiments reveal that many DRL algorithms evaluated in low-data regimes are significantly affected by the implicit assumption of monotonic performance profiles, leading to systematic biases in algorithmic evaluations. The theoretical analysis confirms that the performance profiles of DRL algorithms do not follow a monotonic relationship with sample complexity, challenging the validity of conclusions drawn from existing research.
Implications
The findings suggest that current methodologies in DRL research may misguide future developments and evaluations of algorithms. By addressing these implicit assumptions, researchers can better understand the scaling, capacity, and complexity of DRL, leading to more accurate assessments and advancements in the field.
Architecture Generalization with MetaNCA
Efficient ML
Optimization
Theory
- Introduction of MetaNCA, a framework for self-organizing neural network weights through local rules.
- Utilization of a Weight Transformer architecture for local interactions in weight updates.
- Demonstrated ability to generate diverse neural network architectures without backpropagation.
- Generalization to unseen architectures, with improved performance through architectural diversity in training.
Read more
Architecture Generalization with MetaNCA
Summary
The paper introduces Meta Neural Cellular Automata (MetaNCA), a novel framework that learns local rules to self-organize the weights of artificial neural networks. Inspired by biological neurons and their ability to adapt through local interactions, MetaNCA employs a Weight Transformer architecture that utilizes linear attention to aggregate signals from neighboring weights and hidden states. This approach allows the generation of diverse neural network architectures without the need for backpropagation. The authors demonstrate that MetaNCA can effectively generate weights for various architectures, including feedforward MLPs, CNNs, and ResNets, while scaling to networks with up to 2 million parameters. Notably, the framework exhibits generalization capabilities to unseen architectures, with architectural diversity during training enhancing this generalization. The work addresses limitations of traditional gradient-based optimization methods, such as memory consumption and rigidity in architecture, by proposing a more adaptable and efficient method for neural network training.
Methodology
MetaNCA employs a graph neural cellular automaton approach to iteratively generate neural network parameters using local information from the computational graph. The Weight Transformer architecture aggregates signals from neighboring weights and hidden states to update weights effectively.
Results
MetaNCA successfully generates weights for various architectures, including MLPs, CNNs, and ResNets, achieving scalability to networks with 2 million parameters. The framework demonstrates strong generalization to unseen architectures, with architectural diversity during training contributing to enhanced performance.
Implications
The findings suggest that MetaNCA could lead to more efficient and adaptable neural network training methods, reducing reliance on traditional backpropagation and enabling the development of models that can generalize across different architectures. This could have significant implications for resource-constrained environments and applications requiring flexible model adaptation.
DeepPySR -- A Symbolic Regression Framework with Dynamic Pruning, Pareto Selection, and Hierarchical Composition for Real-World Scientific Discovery
Interpretability
- DeepPySR provides a transparent alternative to black-box models by directly generating interpretable mathematical expressions.
- The framework effectively addresses high-dimensional data challenges through dynamic variable pruning and hierarchical composition.
- DeepPySR outperforms existing symbolic regression methods and traditional machine learning models across various scientific datasets.
- The integration of a principled Pareto selection criterion enhances the selection of optimal models without extensive retraining.
Read more
DeepPySR -- A Symbolic Regression Framework with Dynamic Pruning, Pareto Selection, and Hierarchical Composition for Real-World Scientific Discovery
Summary
DeepPySR is a novel symbolic regression framework designed to address the challenges of high-dimensional inputs, principled selection of Pareto-front formulae, and data irregularities such as multicollinearity and class imbalance. Unlike traditional black-box models that require post-hoc interpretation tools, DeepPySR directly discovers analytical equations from data, providing transparent and interpretable models crucial for fields like clinical medicine and social science. The framework introduces three main innovations: a Dynamic Variable Pruning Schedule (DVPS) that eliminates irrelevant features during the evolutionary search process, an Exponential Pareto Selection (EPS) criterion that selects optimal formulas by balancing accuracy and complexity, and a multi-layer architecture for hierarchical symbolic composition that captures complex relationships in data. Evaluated on four Feynman physics benchmarks and seven real-world datasets, DeepPySR demonstrated superior performance compared to existing methods, recovering interpretable formulas that align with domain-specific risk factors and revealing mechanistic insights that single-layer models could not expose.
Methodology
DeepPySR employs a dynamic variable pruning schedule to eliminate irrelevant features during the evolutionary search, an exponential Pareto selection criterion for optimal formula selection, and a multi-layer architecture to discover hierarchical symbolic functions. This integrated approach allows for effective handling of high-dimensional datasets and complex relationships.
Results
DeepPySR was evaluated against PySR and several machine learning baselines on four Feynman physics benchmarks and seven real-world datasets. It achieved notable improvements in predictive performance, including R2 scores of 0.794 for body fat regression (compared to 0.702 for PySR) and F1 scores of 0.898 for heart disease classification (compared to 0.787 for PySR). The multi-layer architecture also uncovered significant interactions in the Raine Study dataset, highlighting its capability to reveal mechanistic insights.
Implications
The ability of DeepPySR to produce interpretable models directly from data has significant implications for fields requiring transparency, such as clinical medicine and social science. Its approach can facilitate better understanding of complex phenomena and support decision-making processes based on clear, understandable models.
A Practical Investigation of Training-free Relaxed Speculative Decoding
NLP
Large Language Models
Efficient ML
- Relaxed speculative decoding can offer speed-ups but requires careful capability evaluation.
- Many relaxed approaches depend on high-quality drafter models, which may not be suitable for lightweight applications.
- The paper provides a unified framework for understanding various relaxed speculative decoding methods.
- Benchmarking of relaxed approaches reveals significant differences in performance across different settings.
Read more
A Practical Investigation of Training-free Relaxed Speculative Decoding
Summary
This paper investigates training-free relaxed speculative decoding techniques for accelerating sampling from autoregressive large language models (LLMs). Speculative decoding utilizes a faster auxiliary model to draft tokens that are verified by the LLM, traditionally preserving the sampling distribution. The authors explore the potential benefits of relaxing this strict preservation, which can lead to increased speed and capability trade-offs. They unify existing methods under a common framework, benchmark various relaxed approaches, and provide practical insights for practitioners. Key findings indicate that while relaxed methods can speed up generation, they often require careful capability evaluation and may depend on the quality of the drafter model, limiting their applicability for lightweight multi-token predictions. The paper also includes a taxonomy of relaxed speculative decoding methods and empirical evaluations across different inference settings.
Methodology
The authors present a primer on strict speculative decoding, followed by a unified framework for relaxed speculative decoding. They categorize existing methods into a taxonomy and conduct benchmarking experiments on various drafter-verifier pairs, assessing their performance across different inference-time parameters.
Results
The benchmarking results demonstrate that relaxed speculative decoding methods can achieve varying levels of speed-up and capability trade-offs. The findings highlight the importance of the drafter model's quality and the necessity for thorough capability evaluations when implementing these techniques.
Implications
The insights from this investigation can guide practitioners in selecting and implementing relaxed speculative decoding methods in applications requiring low-latency responses from LLMs. The findings may also influence future research directions in optimizing LLM inference processes.
Provably Optimal Learning Algorithms for Assistance Games
Reinforcement Learning
Theory
Optimization
- Introduces the concept of assistance regret for measuring performance in assistance games.
- Presents decentralized algorithms achieving (1 - 1/e)-approximate assistance regret at a rate of ΛO(T^(3/4)).
- Establishes computational intractability for achieving better than (1 - 1/e) regret approximation.
- Demonstrates optimization of algorithms in a pseudo-decentralized setting for improved performance.
Read more
Provably Optimal Learning Algorithms for Assistance Games
Summary
This paper investigates an online variant of assistance games, where an informed agent (human) and an uninformed agent (assistant) interact over T timesteps to optimize a shared reward function. The informed agent observes a latent state while the uninformed agent only sees the human's actions. The authors introduce the concept of assistance regret, which quantifies the difference between the cumulative utility of interactions and that of optimal joint policies. They present decentralized algorithms for both agents that achieve a (1 - 1/e)-approximate assistance regret rate of ΛO(T^(3/4)), with polynomial runtime concerning the action and state spaces. The algorithms are adaptable to any no-regret algorithm for the assistant. The paper also establishes that achieving a better regret approximation than (1 - 1/e) is computationally intractable. Furthermore, the authors demonstrate how to optimize these algorithms in a pseudo-decentralized setting, achieving a rate of ΛO(βT), optimal up to logarithmic factors. The work contributes to the understanding of cooperative multi-agent systems and the challenges of communication and coordination in settings with asymmetric information.
Methodology
The authors develop decentralized learning algorithms for both the human and assistant agents, leveraging a reduction that frames joint policy optimization as online submodular maximization with matroid constraints. They utilize a regret-decomposition lemma to analyze and bound assistance regret, guiding the design towards stability for the human and adaptivity for the assistant.
Results
The proposed algorithms achieve a (1 - 1/e)-approximate assistance regret rate of ΛO(T^(3/4)) with polynomial runtime. With initial coordination using a shared random string, the rate improves to ΛO(βT), which is optimal up to logarithmic factors. The paper proves that no efficient algorithm can achieve sublinear Ξ±-approximate assistance regret for Ξ± > 1 - 1/e unless RP = NP.
Implications
The findings have significant implications for the design of assistive AI systems, particularly in enhancing human-AI collaboration by providing efficient learning mechanisms that adapt to the user's preferences. The results can be applied in various domains, including human-robot interaction, cooperative multi-agent systems, and assistive technologies.
SLORR: Simple and Efficient In-Training Low-Rank Regularization
Efficient ML
Computer Vision
Large Language Models
- SLORR introduces a stateless, architecture-preserving framework for low-rank regularization.
- The method avoids the computational burden of SVDs and cached quantities.
- SLORR improves compressibility of neural networks with less than 8% training overhead.
- The framework is validated across vision and language modeling tasks, showing superior performance retention post-compression.
Read more
SLORR: Simple and Efficient In-Training Low-Rank Regularization
Summary
The paper introduces SLORR, a novel framework for in-training low-rank regularization that addresses the limitations of existing methods. Traditional low-rank regularizers often require singular value decompositions (SVDs) of large weight matrices, modify model architectures, or rely on cached quantities, leading to inefficiencies and increased complexity. SLORR is designed to be stateless and architecture-preserving, directly regularizing the original weight matrices without the need for SVDs. It employs GPU-friendly approximations for the forward and backward passes of the regularizers, specifically utilizing two variants based on the Hoyer sparsity metric and the nuclear norm. The authors evaluate SLORR on various tasks, including short-horizon continued training of ResNet and ViT models on ImageNet-1K, as well as pretraining of large language models (LLMs). The results demonstrate that SLORR significantly enhances post-training compressibility while maintaining low training overhead, making it a practical solution for improving the efficiency of neural networks.
Methodology
SLORR operates directly on the original weight matrices of neural networks, using GPU-efficient approximations for the necessary spectral quantities during training. It implements two variants of regularization: one based on the Hoyer sparsity metric and another on the nuclear norm, both designed to encourage low-rank structures without altering the model architecture or requiring SVDs.
Results
Empirical evaluations on ImageNet-1K show that SLORR enhances the compressibility of models like ResNet-50 and ViT with less than 8% additional training overhead. In large language model pretraining, SLORR-Hoyer-trained models maintain performance significantly better than unregularized models, with an average training overhead of less than 1%.
Implications
The SLORR framework has the potential to make neural networks more efficient by enabling effective low-rank regularization during training, which can lead to reduced computational and memory costs in deployment. This is particularly relevant for large-scale models in both computer vision and natural language processing.
Understanding Layer Patching in Model Size Interpolation
NLP
Large Language Models
Optimization
- Formalization of student patching as an optimization problem over interpolation curves.
- Exhaustive study of patching orders reveals that simple strategies can yield strong performance.
- Introduction of KLPatch, a greedy algorithm that approximates optimal layer patching efficiently.
- Demonstration that patching order is a critical design choice for model size interpolation.
Read more
Understanding Layer Patching in Model Size Interpolation
Summary
This paper investigates the process of zero-shot model size interpolation, which allows for the creation of new models of intermediate sizes by combining existing models without additional training. The authors focus on the layer patching aspect of boomerang distillation, where layers from a larger teacher model are used to replace layers in a smaller student model. They present a systematic study of how to select layers for patching, framing it as an optimization problem that can be represented as a shortest-path problem in an acyclic graph. Through extensive experiments, the authors demonstrate that the choice of patching order significantly influences the performance of the interpolated models. They introduce KLPatch, a greedy algorithm that optimizes the patching process based on KL divergence, which reduces computational complexity and often yields better performance than traditional patching strategies. The findings highlight the importance of patching order and provide practical guidance for constructing effective interpolated models.
Methodology
The authors formalize the layer patching process as an optimization problem and derive a shortest-path problem to approximate optimal interpolations. They conduct exhaustive searches over possible interpolations in small language models and test various patching orders in larger models. KLPatch is introduced as a computationally efficient algorithm that uses KL divergence to guide the patching process.
Results
The experiments reveal that patching from the last layer to the first often yields the best performance for DistilBERT and competitive results for DistilGPT2. KLPatch consistently outperforms traditional patching methods and achieves near-optimal interpolations across multiple language models, significantly reducing computational complexity.
Implications
The insights from this study can inform the design of more efficient and effective model size interpolation techniques, potentially leading to the development of a wider range of language models tailored to specific computational constraints and performance needs.
A Transdiagnostic Space of Disorder Like Phenotypes in Reinforcement Learning Agents
Reinforcement Learning
- Introduces a dose-controllable method for modeling psychological disorders in RL agents.
- Demonstrates emergent properties of disorders, including a two-dimensional affective space.
- Finds distinct recovery patterns for different types of disorders when manipulating parameters.
- Validates the framework across multiple environments, confirming its generalizability.
Read more
A Transdiagnostic Space of Disorder Like Phenotypes in Reinforcement Learning Agents
Summary
This paper presents a novel approach to modeling psychological disorders in reinforcement learning (RL) agents, providing insights into computational psychiatry and affective control failures. Unlike previous methods that induce disorders through hand-tuned reward shaping and post hoc labeling, this work introduces a dose-controllable manipulation of cognitive appraisal signals in an appraisal-guided Proximal Policy Optimization (PPO) agent. The study successfully expresses seven psychological disordersβanxiety, mania, obsessive-compulsive checking, depression, impulsivity, addiction, and post-traumatic stressβusing a single adjustable parameter for each disorder. The authors conducted over a thousand experimental runs, demonstrating that each disorder exhibits a graded, monotone dose-response that is not replicated by control conditions. Notably, the findings reveal emergent properties: the disorders self-organize into a two-dimensional affective space, with mania and anxiety mirroring each other; the removal of certain disorder parameters leads to distinct recovery patterns; and interactions between multiple disorder parameters yield nonadditive effects, suggesting potential comorbidities. The framework is validated across different environments, indicating its robustness beyond specific task settings.
Methodology
The study employs an appraisal-guided Proximal Policy Optimization (AG-PPO) agent, manipulating cognitive appraisal signals to induce and measure psychological disorders. The methodology includes extensive experimental runs with pre-registered assays and control conditions to ensure robust statistical analysis.
Results
The results indicate that the seven psychological disorders can be induced with a graded response to parameter adjustments. The emergent findings include the self-organization of disorders into a two-dimensional space, distinct recovery mechanisms for different disorder types, and nonadditive interactions between multiple disorder parameters, supporting predictions of comorbidity.
Implications
This research has significant implications for computational psychiatry, offering a controlled experimental framework to study psychological disorders and their treatments. It may also inform the design of more reliable RL agents in healthcare and other human-facing applications by enhancing their affective stability.
When Thinking Hurts: Epistemic Signals in the Reasoning Chains of Visual Language Models
Multimodal
Large Language Models
Interpretability
- First empirical characterization of answer entropy behavior in thinking-mode VLMs, identifying three distinct patterns.
- Thinking chain entropy is a more reliable predictor than answer entropy in VLMs, particularly in Qwen and GLM models.
- Structured abstention affects a significant portion of queries, with implications for model reliability.
- A practical abstention gate can significantly improve accuracy without incurring additional inference costs.
Read more
When Thinking Hurts: Epistemic Signals in the Reasoning Chains of Visual Language Models
Summary
This paper investigates the uncertainty quantification in visual language models (VLMs) that utilize a reasoning chain prior to answering questions. The author presents a three-family empirical characterization of answer entropy behavior in thinking-mode VLMs, revealing three distinct patterns: complete collapse, no collapse, and selective thinking. The study shows that thinking chain entropy consistently outperforms answer entropy, indicating that the reasoning chains provide more reliable epistemic signals. Additionally, the paper documents structured abstention in VLMs, where a significant percentage of queries are affected by uncertainty, and proposes a practical abstention gate that enhances accuracy without additional inference costs. The findings suggest that the architectural changes in thinking-mode VLMs impact uncertainty estimation, necessitating new approaches for reliable predictions in practical applications.
Methodology
The study involved running four different VLMs on identical adversarial samples to analyze their answer entropy behavior. Controlled ablation experiments were conducted to compare thinking-mode models against non-thinking counterparts, focusing on the entropy of answer tokens and reasoning chains. The analysis included metrics such as AUROC for hallucination detection and entropy calculations from a single forward pass.
Results
The results revealed that Qwen3-VL-8B-Thinking exhibited a complete collapse in answer entropy (AUROC = 0.492), while GLM-4.1V-9B-Thinking showed no collapse (AUROC = 0.716). InternVL3-8B demonstrated selective thinking with a 50% chain rate. Thinking chain entropy outperformed answer entropy across models, particularly in challenging reasoning tasks, and a practical abstention gate improved accuracy from 71.0% to 93.8% at 62.7% coverage.
Implications
The findings have significant implications for the deployment of VLMs in real-world applications where reliable uncertainty estimates are crucial. The proposed methods for utilizing reasoning chains can enhance model performance and reliability, particularly in complex visual reasoning tasks.
When Does Continual Learning Require Learning
Large Language Models
NLP
Reinforcement Learning
- Continual learning is redefined as increasing competence in changing environments rather than just mitigating forgetting.
- The authors introduce a unified framework for evaluating continual learning methods across different axes of change.
- Different methods (prompt-based, distillation, context compression, reinforcement learning) exhibit unique strengths and weaknesses.
- Understanding the nature of environmental change is crucial for determining the appropriate continual learning strategy.
Read more
When Does Continual Learning Require Learning
Summary
This paper addresses the challenges of continual learning in large language models (LLMs), arguing that the traditional framing of continual learning as merely mitigating forgetting is insufficient. The authors propose a new perspective that emphasizes increasing model competence as the environment changes, characterized by two axes: spatial (new domains) and temporal (data drift). They introduce a unified framework to evaluate various continual learning methods, including prompt-based techniques, supervised learning, reinforcement learning, and context compression, under realistic conditions where tasks evolve over time. The study reveals that different methods exhibit distinct trade-offs in their ability to adapt to new information and maintain performance on previous tasks. For instance, prompt-based methods quickly adapt but suffer from degradation on future tasks, while distillation-based methods accumulate knowledge stably but struggle with outdated facts. The findings suggest that continual learning encompasses a range of capabilities, necessitating different strategies depending on the nature of environmental changes. This work aims to guide the development of more effective continual learning systems by clarifying when adaptation should occur within model weights versus through external mechanisms.
Methodology
The authors recast widely used LLM benchmarks as sequential problems and develop a mechanism-agnostic protocol to compare various continual learning methods. They evaluate eight methods across four families: prompt optimization, offline supervised updates, online reinforcement learning, and context-compression, using a common backbone model.
Results
The results indicate that prompt-based methods achieve strong backward accuracy but degrade on future tasks, while distillation methods accumulate knowledge stably but struggle with rapid updates. Context compression improves efficiency without enhancing learning capabilities, and online reinforcement learning adapts effectively but is sensitive to noisy rewards. The study also finds that agentic interactions can compound experience and improve performance, although success diminishes with increasing task complexity.
Implications
The findings suggest that continual learning systems should be designed with a nuanced understanding of the types of changes they will encounter, allowing for tailored strategies that optimize learning and adaptation. This has implications for the deployment of LLMs in dynamic environments where knowledge and tasks evolve.
ReCoLoRA: Spectrum-Aware Recursive Consolidation for Continual LLM Fine-Tuning
NLP
Large Language Models
Efficient ML
- ReCoLoRA introduces a spectrum-aware framework for continual fine-tuning of LLMs, addressing catastrophic forgetting.
- The framework employs recursive consolidation to preserve knowledge from previous tasks effectively.
- ReCoLoRA-TaskBank serves as an upper bound for performance by isolating task branches.
- Experimental results show ReCoLoRA achieves better performance than several existing methods with fewer trainable parameters.
Read more
ReCoLoRA: Spectrum-Aware Recursive Consolidation for Continual LLM Fine-Tuning
Summary
The paper introduces ReCoLoRA (Recursive Consolidation of Low-Rank Adapters), a novel framework designed for continual fine-tuning of large language models (LLMs) that addresses the challenges of catastrophic forgetting and parameter efficiency. Traditional low-rank adaptation methods, such as LoRA, often lead to overwriting of previous tasks when adapting to new ones. ReCoLoRA mitigates this by employing a spectrum-aware approach that initializes adapters from a randomized singular value decomposition (SVD) of pretrained weights and selects effective ranks using an elbow criterion. The framework operates in two stages: first, it adapts the principal subspace, which retains the dominant pretrained structure, and then it opens up residual capacity as needed. A key innovation is the recursive consolidation mechanism, which re-decomposes the effective weight into a frozen residual, a slowly updated principal component, and a fresh adapter before each new task, thereby preserving knowledge from previous tasks. The paper also presents ReCoLoRA-TaskBank, an oracle-routed variant that isolates branches for each task, serving as an upper bound for performance. Experimental results demonstrate that ReCoLoRA outperforms several baseline methods on a six-task continual GLUE sequence across multiple LLM backbones, achieving superior average scores while training fewer parameters.
Methodology
ReCoLoRA initializes low-rank adapters using randomized SVD of pretrained weights, selects effective ranks per layer with an elbow criterion, and employs a two-stage training process. The recursive consolidation mechanism re-decomposes the effective weight into a frozen residual, a slowly trainable principal component, and a fresh adapter before each new task.
Results
On a six-task continual GLUE sequence, ReCoLoRA achieved the best final average score on three out of four tested LLM backbones, outperforming rank-swept LoRA, PiSSA, AdaLoRA, and DoRA baselines while requiring fewer trainable parameters. The ReCoLoRA-TaskBank variant attained a final average score of 0.8957 Β± 0.0026 with zero average forgetting.
Implications
ReCoLoRA has significant implications for the deployment of LLMs in dynamic environments where continual adaptation to new tasks is necessary. Its ability to retain knowledge from previous tasks while efficiently adapting to new ones can enhance the performance of LLMs in real-world applications.
Federated Deep Learning for Privacy-Preserving Cardiovascular Disease Risk Prediction
Federated Learning
- Federated learning allows for collaborative model development without sharing sensitive patient data.
- The study integrates two diverse cohorts to improve cardiovascular disease risk prediction.
- Deep survival models trained through federated learning show superior performance compared to local models.
- The approach maintains patient privacy while enhancing model generalizability across different populations.
Read more
Federated Deep Learning for Privacy-Preserving Cardiovascular Disease Risk Prediction
Summary
This paper presents a federated deep learning approach aimed at enhancing cardiovascular disease (CVD) risk prediction while preserving patient privacy. Traditional risk prediction models often rely on centralized data, which can be limited by privacy regulations. The authors propose a federated learning framework that integrates two distinct population-based cohorts: Lifelines, with 148,230 participants and self-reported outcomes, and the Rotterdam Study, with 10,155 participants and clinically linked outcomes. The study evaluates the performance of deep survival models trained using federated learning, demonstrating that these models outperform locally trained models. Specifically, the C-statistic for the Rotterdam Study improved from 0.728 to 0.739, while for Lifelines, it increased from 0.783 to 0.787. These results indicate that federated deep learning can effectively enhance CVD risk prediction across heterogeneous datasets without compromising individual data privacy.
Methodology
The authors employed a federated learning framework to train deep survival models on two population-based cohorts. The Lifelines cohort provided a large dataset with self-reported outcomes, while the Rotterdam Study offered a smaller dataset with clinically linked outcomes. The federated learning process involved local model training at each site, followed by the aggregation of model updates to create a global model, ensuring that raw patient data remained secure and private.
Results
The federated deep learning models achieved a C-statistic of 0.739 (95% CI: 0.728β0.749) for the Rotterdam Study, an improvement from 0.728. For the Lifelines cohort, the C-statistic increased from 0.783 (95% CI: 0.775β0.791) to 0.787 (95% CI: 0.780β0.792), indicating enhanced predictive performance across both cohorts.
Implications
This study highlights the potential of federated learning in healthcare, particularly for developing predictive models that respect patient privacy. The findings suggest that federated deep learning can be a viable solution for improving risk prediction models in diverse populations, which could lead to better prevention strategies for cardiovascular diseases.
Steering Neural Network Training through Interpretable Constraints Based on Partial Dependence
Interpretability
- Introduces a novel approach to neural network training that incorporates partial dependence for improved interpretability.
- Demonstrates that models trained with this method perform better and are more data-efficient than unconstrained models.
- Aligns model interpretations with prior domain knowledge, enhancing the reliability of explanations.
- Focuses on regression tasks, particularly in dynamical systems forecasting, expanding the application of explanation-guided learning.
Read more
Steering Neural Network Training through Interpretable Constraints Based on Partial Dependence
Summary
This paper addresses the challenge of enhancing the interpretability of machine learning models, particularly neural networks, by introducing a novel approach that incorporates explanation-guided learning (EGL) based on partial dependence. The authors argue that while many techniques exist for interpreting model outputs, fewer focus on aligning these interpretations with domain knowledge. Their proposed method steers the training of neural networks to ensure that the average response to specific features corresponds with known functional relationships in the data. The authors empirically validate their approach on various regression tasks, including dynamical systems forecasting, demonstrating that models trained with their method outperform unconstrained models in terms of performance and data efficiency. Furthermore, the interpretations derived from the constrained models align more closely with user-provided knowledge, highlighting the effectiveness of their approach in producing faithful explanations. This work contributes to the growing field of explainable artificial intelligence by providing a framework for integrating domain knowledge into the training process, ultimately leading to more interpretable and reliable machine learning models.
Methodology
The authors propose a training algorithm that incorporates partial dependence constraints into the neural network training process. This approach ensures that the model's marginal response to certain inputs aligns with specific functional knowledge about the problem domain. The methodology is validated through empirical experiments on various regression problems.
Results
The results indicate that the constrained models not only outperform their unconstrained counterparts in terms of predictive performance but also require fewer training samples and generalize better to out-of-distribution data. Additionally, the interpretations from the constrained models are more consistent with user-provided domain knowledge.
Implications
This research has significant implications for the development of interpretable machine learning models, particularly in fields where domain knowledge is critical. By integrating prior knowledge into the training process, practitioners can create models that are not only accurate but also provide trustworthy explanations, which is essential for decision-making in sensitive applications such as healthcare and finance.
Open-ended Multi-agent Autocurricula via Visual Inspection of Policies with Multi-modal LLMs
Reinforcement Learning
Computer Vision
Multimodal
- Introduction of Visual Inspection of Policies (VIP) for open-ended RL curricula.
- VIP leverages episode videos processed by a Video Language Model (VLM) for task recommendations.
- Empirical validation on the StarCraft Multi-Agent Challenge (SMAC) shows superior performance compared to traditional methods.
- Demonstrates the importance of visual cues in assessing agent behavior and task difficulty.
Read more
Open-ended Multi-agent Autocurricula via Visual Inspection of Policies with Multi-modal LLMs
Summary
This paper introduces a novel approach to open-ended curricula in Reinforcement Learning (RL) called Visual Inspection of Policies (VIP). The authors address the challenge of assessing task difficulty relative to an agent's learning progress by leveraging recorded episode videos of agent behavior, processed through a Video Language Model (VLM). Unlike traditional methods that rely on scalar task scores or textual summaries, VIP utilizes visual information to recommend tasks that facilitate the learning of increasingly complex skills. The methodology is empirically tested in the StarCraft Multi-Agent Challenge (SMAC), demonstrating that VIP can generate more effective curricula than both text-only approaches and those based on scalar task scores. The results indicate that even a lightweight VLM, such as VideoLLaMa2-7B, can significantly enhance the training process by identifying promising curriculum directions that are often overlooked by conventional methods. This approach not only improves agent performance but also provides insights into the learning process that are more aligned with human-like evaluation methods.
Methodology
The methodology involves recording episode videos of the current policy during training and feeding these videos into a Video Language Model (VLM) alongside a textual summary of the agent's performance. The VLM analyzes the visual content to recommend the next task that balances difficulty and interest, ensuring that the tasks are neither too easy nor too hard for the agent's current capabilities.
Results
The results indicate that VIP outperforms traditional curriculum methods based on scalar task scores and text-only summaries. Specifically, agents trained using VIP achieved an approximately 80% win rate in the SMAC environment, highlighting the effectiveness of visual insights in guiding the learning process.
Implications
The findings suggest that incorporating visual inspection into RL training can lead to more effective learning strategies, potentially transforming how curricula are designed in multi-agent environments. This approach could be applied to various complex RL tasks, enhancing agent adaptability and performance.
Predicting Pseudo-nitzschia harmful algal blooms along the Portuguese Coast using satellite-derived predictors
Time Series
- Developed a machine-learning framework for predicting Pseudo-nitzschia HABs using satellite data.
- Implemented a strict cross-validation strategy to prevent data leakage.
- Achieved a ROCβAUC of 0.77 Β± 0.06 with Extra Trees model using biological predictors.
- Identified key predictors including lagged sea surface temperature and chlorophyll-a.
Read more
Predicting Pseudo-nitzschia harmful algal blooms along the Portuguese Coast using satellite-derived predictors
Summary
This study addresses the prediction of harmful algal blooms (HABs) caused by Pseudo-nitzschia diatoms along the Portuguese Atlantic coast, which pose risks to coastal ecosystems and shellfish harvesting. The authors developed a spatio-temporal machine-learning framework that utilizes satellite-derived predictors to forecast HAB occurrences under realistic operational constraints. They characterized environmental and biological variability across nine shellfish production zones using 5,882 observations, focusing on zones L1 and L2, known hotspots for Pseudo-nitzschia blooms. A decade-long dataset (2013-2023) was employed, comprising 1,440 observations and over 1,000 satellite-based predictors, including sea surface temperature, upwelling index, chlorophyll-a, and plankton functional types. To manage coastal heterogeneity, the study applied a river-aware spatial clustering scheme for sampling locations. A rigorous spatio-temporal cross-validation strategy was implemented to prevent data leakage and simulate real-world forecasting conditions. The results indicated that HAB occurrences were moderately predictable, with ensemble tree-based methods, particularly Extra Trees, achieving the highest performance (ROCβAUC of 0.77 Β± 0.06) when biological predictors were included. The analysis highlighted that seasonal structure, spatial context, and lagged environmental conditions were critical in model predictions, while biological indicators refined bloom likelihood during favorable periods. This framework demonstrates significant potential for operational early-warning systems for HABs along the Iberian upwelling margin.
Methodology
The study utilized a spatio-temporal machine-learning framework that incorporated satellite-derived environmental and biological predictors. A decade-long dataset was analyzed, and a river-aware spatial clustering scheme was employed to partition sampling locations into ecologically meaningful sub-regions. A stringent spatio-temporal cross-validation strategy was used to ensure realistic forecasting conditions.
Results
The predictive models demonstrated moderate accuracy in forecasting HAB occurrences, with ensemble tree-based methods showing the best performance. The Extra Trees model, when enhanced with biological predictors, achieved a ROCβAUC of 0.77 Β± 0.06. Key predictors influencing model decisions included seasonal structure, spatial context, and lagged environmental conditions.
Implications
The findings suggest that the developed framework can enhance operational monitoring and early-warning systems for harmful algal blooms, potentially improving management responses to mitigate risks associated with shellfish harvesting and public health along the Portuguese coast.
Towards Efficient Large Language Model Serving: A Survey on System-Aware KV Cache Optimization
NLP
Large Language Models
Efficient ML
- The survey categorizes KV cache optimization techniques into temporal, spatial, and structural dimensions.
- It highlights the importance of KV cache optimization for efficient LLM serving, particularly as model sizes and input lengths increase.
- The authors propose a novel behavior-oriented perspective for analyzing KV cache optimization methods.
- Future research opportunities are identified, particularly in cross-behavior co-design and behavior-objective links.
Read more
Towards Efficient Large Language Model Serving: A Survey on System-Aware KV Cache Optimization
Summary
This paper presents a comprehensive survey on system-aware key-value (KV) cache optimization for serving large language models (LLMs). As LLMs become increasingly memory-intensive and costly to serve, optimizing the KV cache, which stores intermediate tensors during autoregressive decoding, is essential for achieving low-latency and high-throughput inference. The authors categorize existing optimization techniques into three dimensions: execution and scheduling (temporal), placement and migration (spatial), and representation and retention (structural). They analyze the interplay between these behaviors and their impact on system performance, identifying opportunities for future research. This survey aims to systematize the rapidly evolving field of KV cache optimization, providing a foundation for understanding and innovating in LLM serving infrastructure.
Methodology
The authors conducted a systematic review of recent literature on KV cache optimization techniques, organizing the findings into a taxonomy based on system behaviors. They analyzed existing methods and their interactions, providing insights into the design space for future innovations.
Results
The survey reveals that existing KV cache optimization techniques can be effectively categorized into three main axes of system behavior. It also identifies gaps in current research and suggests potential directions for future work, emphasizing the need for a more integrated approach to KV cache optimization in LLM serving.
Implications
The findings of this survey have significant implications for the design of efficient LLM serving systems. By focusing on KV cache optimization, researchers and practitioners can improve the performance of LLMs in real-world applications, leading to faster and more cost-effective inference solutions.
Vanilla SGD with Momentum Survives Heavy-Tailed Noise: Convergence Analysis without Gradient Clipping or Normalization
Optimization
Theory
- First theoretical guarantee of convergence for vanilla SGD with momentum under heavy-tailed noise.
- Convergence rates established for strongly convex, convex, and nonconvex functions without gradient clipping or normalization.
- Demonstrated that the condition Ξ½ + 1 β€ p is crucial for stable convergence.
- Results indicate that vanilla SGD with momentum can serve as a baseline for future algorithmic improvements.
Read more
Vanilla SGD with Momentum Survives Heavy-Tailed Noise: Convergence Analysis without Gradient Clipping or Normalization
Summary
This paper investigates the performance of vanilla Stochastic Gradient Descent (SGD) with momentum in the presence of heavy-tailed noise, a common issue in modern optimization problems. Unlike previous studies that often rely on gradient clipping or normalization to ensure convergence, the authors provide a comprehensive theoretical analysis of vanilla SGD with momentum across strongly convex, convex, and nonconvex objectives without employing any gradient control mechanisms. The findings reveal that while the convergence rates of vanilla SGD with momentum are inferior to those of modified variants, they still provide a fundamental baseline for understanding the algorithm's behavior under heavy-tailed noise. The paper also emphasizes the importance of the condition relating the HΓΆlder continuity parameter and the tail index for stable convergence, suggesting that HΓΆlder smoothness is a more suitable framework for analyzing such scenarios.
Methodology
The authors refine existing convergence results and provide a theoretical framework for analyzing vanilla SGD with momentum under heavy-tailed noise. They establish convergence rates for different classes of objective functions and conduct experiments on synthetic functions to validate their theoretical claims.
Results
The paper presents convergence rates of O(T^(-p-1)/2p) for nonconvex objectives when the tail index p is known, and O(T^(-p-1)/4) when it is unknown. The results show that vanilla SGD with momentum converges in expectation across all three classes of objective functions, under weaker conditions than previously assumed.
Implications
The findings suggest that vanilla SGD with momentum can be a viable optimization method in scenarios with heavy-tailed noise, providing insights for future research on improving optimization algorithms without relying on gradient clipping or normalization.
Uncertainty-gated selection for block-sparse attention
NLP
Large Language Models
Efficient ML
- Introduces a value-of-information router to enhance block-sparse attention in long-context models.
- Achieves significant improvements in recall rates compared to traditional top-k selection methods.
- The router is agnostic to the scoring backbone and can be combined with existing methods.
- Demonstrates consistent performance across multiple models and architectures.
Read more
Uncertainty-gated selection for block-sparse attention
Summary
This paper introduces a novel approach to enhance block-sparse attention mechanisms in long-context language models by addressing the limitations of traditional top-k selection methods. The proposed method, termed the value-of-information (VoI) router, evaluates the decisiveness of the top-k cutoff for each query and expands the selection set when the cutoff margin is small, thereby allowing the model to attend to additional key blocks that may contain critical evidence. This approach is agnostic to the underlying scoring backbone and can be integrated with existing block-scoring methods such as Quest. Empirical evaluations demonstrate that the router significantly improves recall rates, achieving a paired recall of 0.75 compared to 0.47 for the standard top-k method on the LongBench-v2 medium dataset. The router's effectiveness is consistent across multiple models and architectures, preserving high accuracy while maintaining efficient runtime performance. The paper also presents a fused selection-plus-kernel implementation that streamlines the selection process, further enhancing computational efficiency. Overall, the proposed method offers a promising direction for improving the performance of long-context language models by optimizing the selection of relevant information during inference.
Methodology
The methodology involves treating the top-k cutoff as a decision under uncertainty, where the cutoff margin is computed for each query. If the margin indicates a high-risk decision (i.e., the scores of the k-th and (k+1)-th blocks are close), the router expands the selection set to include more blocks. This is done selectively based on the bottom quantile of tiles per layer, ensuring that the average increase in the attended set is minimal. The approach is validated empirically across various models and architectures, demonstrating its effectiveness in improving recall and maintaining efficiency.
Results
The router-on-Quest method achieved a paired recall of 0.75 on the LongBench-v2 medium dataset, significantly outperforming the top-k method's recall of 0.47. The method preserved 0.81 and 0.89 of dense accuracy on different models while maintaining efficient runtime, with wall-time profiles crossing dense performance thresholds at various context lengths.
Implications
The proposed uncertainty-gated selection method has the potential to enhance the performance of long-context language models in tasks requiring precise information retrieval, such as multi-hop reasoning and query-latent retrieval. By improving the selection of relevant information, this approach could lead to more accurate and efficient language models, benefiting applications in natural language processing and beyond.
Dimensionality Reduction Meets Network Science: Sensemaking on UMAP's kNN Graph
Graph Learning
- UMAP's kNN graph is a rich resource for data analysis, often overlooked in favor of 2D visualizations.
- Standard graph algorithms (PageRank, k-core decomposition, clustering coefficient) can provide valuable insights into data structure.
- Graph-based analyses yield results that are competitive with traditional clustering methods.
- The approach bridges dimensionality reduction techniques and network science, enhancing data sensemaking capabilities.
Read more
Dimensionality Reduction Meets Network Science: Sensemaking on UMAP's kNN Graph
Summary
This paper explores the potential of UMAP's internal k-nearest-neighbor (kNN) graph as a valuable analytical resource for high-dimensional data exploration. While UMAP is primarily used for generating 2D visualizations, the authors argue that the kNN graph, which captures the data manifold before projection distortion, should be utilized for deeper data sensemaking. They demonstrate how standard graph algorithms can be applied to this graph to enhance understanding of the data's structure. Specifically, they employ PageRank to identify representative data points, k-core decomposition to uncover dense core regions, and clustering coefficients to detect tightly-knit neighborhoods. The effectiveness of these graph-based analyses is quantitatively and qualitatively validated using MNIST and Fashion MNIST datasets, showing that they are competitive with or complementary to traditional methods like k-medoids and HDBSCAN.
Methodology
The authors applied standard graph algorithms to UMAP's kNN graph to analyze high-dimensional data. They utilized PageRank for identifying representative points, k-core decomposition for revealing core structures, and clustering coefficients to find cohesive neighborhoods. The methodology was evaluated on MNIST and Fashion MNIST datasets, comparing the results with traditional clustering techniques.
Results
The results indicated that PageRank-selected points achieved superior class balance compared to k-medoids, while also maintaining high representativeness and classification accuracy. The k-core decomposition revealed a hierarchical structure that traditional clustering methods could not capture, and the clustering coefficient identified distinct micro-clusters of similar data points.
Implications
This work suggests that leveraging the kNN graph can significantly enhance the interpretability and analysis of high-dimensional data, providing new avenues for data exploration and understanding. It encourages the integration of network science techniques into dimensionality reduction workflows, potentially improving outcomes in various applications such as image classification and clustering.
Learning $ ext{AC}^0$ under Locally Sampleable Graphical Models
Theory
Graph Learning
- Introduces a quasipolynomial-time learner for AC0 under locally sampleable graphical models.
- Circumvents the polynomial growth requirement from previous works.
- Establishes a new low-degree approximation technique for Gibbs distributions.
- Applies the framework to hard-core and Ising models on bounded-degree graphs.
Read more
Learning $ ext{AC}^0$ under Locally Sampleable Graphical Models
Summary
This paper addresses the problem of learning constant-depth Boolean circuits (AC0) under locally sampleable graphical models, which represent complex joint distributions. The authors build on previous work that established learning guarantees for AC0 under bounded-degree Gibbs distributions with strong spatial mixing and polynomial growth. They introduce a quasipolynomial-time learning algorithm that circumvents the polynomial growth requirement by leveraging a new low-degree approximation for Gibbs distributions, achieved through simulating and truncating classical Glauber dynamics. The key contribution is the establishment of a connection between the existence of efficient local samplers for Gibbs distributions and the ability to approximate AC0 functions with low-degree polynomials. The paper demonstrates this approach through applications to two-spin systems, including the hard-core model and Ising model, on arbitrary bounded-degree graphs, particularly in regimes close to their respective sampling thresholds.
Methodology
The authors utilize a systematic-scan Glauber dynamics approach to simulate Gibbs distributions, allowing for efficient local sampling. They establish a low-degree approximation for AC0 functions under these distributions, which facilitates the learning process. The methodology involves analyzing the mixing properties of the Markov chain and ensuring that local samplers can produce accurate outputs with limited queries.
Results
The paper presents a learning algorithm that, given N samples from a hard-core distribution on a bounded-degree graph, can output a hypothesis that approximates the target AC0 function with high probability. The algorithm runs in quasipolynomial time and achieves learning guarantees under conditions that are less restrictive than those in previous works, specifically not requiring polynomial growth in the underlying graph.
Implications
The findings have significant implications for computational learning theory, particularly in understanding the learning of Boolean functions under complex distributions. The results may enhance the efficiency of learning algorithms in practical applications involving graphical models, such as in statistical physics, machine learning, and network analysis.
Trustworthy Machine Learning through the Lens of Combinatorial Optimization: Survey and Research Perspectives
Optimization
Theory
Interpretability
- Trustworthy ML requires more than just predictive accuracy; it necessitates transparency, interpretability, and fairness.
- Combinatorial optimization offers a framework for addressing trustworthiness in ML, providing global guarantees and formal certificates.
- The Rashomon effect allows for the selection of models that meet trustworthiness criteria without sacrificing performance.
- Recent advances in CO techniques can be applied to various post-training tasks, enhancing the robustness and fairness of ML systems.
Read more
Trustworthy Machine Learning through the Lens of Combinatorial Optimization: Survey and Research Perspectives
Summary
This survey explores the intersection of trustworthy machine learning (ML) and combinatorial optimization (CO), emphasizing the need for transparency, interpretability, robustness, fairness, privacy, and certifiability in ML systems. The authors argue that traditional empirical performance metrics are insufficient for evaluating ML models, as similar-performing models can differ significantly in their trustworthiness attributes. The paper synthesizes recent advancements in CO techniques applied to both training and post-training tasks in ML, such as interpretable model learning, explanation generation, robustness analysis, fairness auditing, model compression, and privacy protection. The authors highlight that CO provides global guarantees and formal certificates, which are often lacking in heuristic approaches. Despite scalability challenges, the integration of CO in trustworthy ML design and deployment is seen as increasingly viable due to advancements in solvers and hybrid algorithms. The survey aims to unify diverse research efforts across operations research, theoretical computer science, and formal methods, advocating for a broader perspective that encompasses both training and auditing tasks in trustworthy ML.
Methodology
The survey adopts a structured, subdomain-driven approach to synthesize literature on the application of combinatorial optimization techniques to trustworthy machine learning, covering both training and post-training tasks across various CO paradigms.
Results
The survey identifies numerous applications of CO in trustworthy ML, including model training, robustness verification, fairness certification, and privacy auditing. It highlights the advantages of CO over heuristic methods, particularly in providing formal guarantees and facilitating trade-off analysis.
Implications
The findings suggest that integrating combinatorial optimization into ML practices can enhance the design and deployment of trustworthy systems, potentially leading to more reliable and fair AI applications across various domains.
MatBind: A Shared Embedding Space for Multimodal Materials Characterization
Multimodal
- MatBind aligns four materials modalities into a unified embedding space, facilitating cross-modal retrieval.
- The framework allows for emergent zero-shot retrieval, outperforming directly trained modality pairs in some cases.
- Materials are organized in the embedding space according to physically meaningful properties without explicit supervision.
- Combining multiple modalities at query time enhances retrieval performance and resolves ambiguities.
Read more
MatBind: A Shared Embedding Space for Multimodal Materials Characterization
Summary
The paper presents MatBind, a novel contrastive learning framework designed to unify and align four distinct materials modalities: crystal structure, powder X-ray diffraction (pXRD), electronic density of states (DOS), and textual descriptions. The challenge in materials characterization arises from the need to integrate heterogeneous data sources, which are often analyzed in isolation. MatBind addresses this fragmentation by creating a shared embedding space where these modalities can be compared and queried across representational boundaries. By using crystal structure as the anchor modality, the framework enables emergent zero-shot cross-modal retrieval, allowing for effective retrieval even between modalities that were not explicitly paired during training. The results indicate that the learned embedding space organizes materials according to meaningful physical properties without requiring explicit supervision. Furthermore, the performance of cross-modal retrieval improves significantly when multiple modalities are combined at query time, demonstrating the framework's potential to enhance data-driven materials discovery.
Methodology
MatBind employs a contrastive learning approach, using crystal structure as the central anchor modality. It trains pairwise contrastive objectives between the crystal structure and each of the auxiliary modalities (pXRD, DOS, and text), resulting in a shared representation that allows for mutual comparability among all modalities, including those not explicitly aligned during training.
Results
The framework demonstrates high recall rates for cross-modal retrieval, particularly between crystal structure and text, and crystal structure and DOS. Notably, the emergent retrieval link between DOS and text surpasses the performance of the directly trained crystal structure and pXRD pair. The embedding space effectively organizes materials based on physical properties without explicit supervision.
Implications
MatBind has significant implications for materials science, enabling more efficient data-driven discovery by allowing researchers to query and analyze materials across different modalities. This unified approach could lead to better understanding and identification of materials, ultimately accelerating advancements in material design and application.
Reinforcing the Generation Order of Multimodal Masked Diffusion Models
Generative Models
Multimodal
Optimization
- Existing confidence-based strategies do not improve image generation quality and multimodal reasoning.
- A learnable control block is proposed to optimize generation order using Group Relative Policy Optimization (GRPO).
- The method shows significant improvements in text-to-image alignment and multimodal understanding benchmarks.
- The research highlights the necessity of advanced order control mechanisms in multimodal tasks.
Read more
Reinforcing the Generation Order of Multimodal Masked Diffusion Models
Summary
This paper investigates the optimization of generation order in Diffusion Language Models (DLMs) for text-to-image synthesis and multimodal understanding. The authors highlight that traditional methods of determining generation order based on model logits are insufficient for multimodal tasks. To address this, they introduce a learnable control module that utilizes Group Relative Policy Optimization (GRPO) to dynamically determine the generation order. The proposed method significantly enhances the model's ability to capture spatial relationships in images and improves performance on multimodal reasoning tasks. The framework is evaluated on two benchmarks: GenEval for text-to-image alignment and VLMEvalKit for multimodal understanding, achieving notable relative improvements of 4.08% and 4.85%, respectively. This work underscores the importance of learning-based control mechanisms in multimodal generation tasks, moving beyond heuristic approaches.
Methodology
The authors propose a control block that learns to determine the generation order in multimodal masked diffusion models. This control block is trained using Group Relative Policy Optimization (GRPO), allowing it to adaptively optimize the generation process based on the specific requirements of image synthesis and multimodal understanding.
Results
The proposed method achieved a 4.08% relative improvement on the GenEval benchmark for text-to-image alignment and a 4.85% relative improvement on the VLMEvalKit benchmark for multimodal understanding, demonstrating the effectiveness of the learning-based control approach.
Implications
The findings suggest that incorporating learnable control mechanisms in multimodal models can lead to significant enhancements in performance, particularly in tasks requiring complex interdependencies among visual tokens. This could have broad applications in areas such as automated content generation, visual question answering, and enhanced human-computer interaction.
Optimal Learning Rate Scaling Depends on Data in Deep Scalar Linear Networks
Theory
Optimization
- Optimal learning rate scaling in deep scalar linear networks is data-dependent.
- Data-agnostic scaling rules fail to transfer effectively across network depths.
- The study provides exact solutions for gradient descent dynamics using special functions.
- Learning dynamics exhibit a constant linear convergence rate across all depths.
Read more
Optimal Learning Rate Scaling Depends on Data in Deep Scalar Linear Networks
Summary
This paper investigates the gradient descent dynamics of deep scalar linear networks, demonstrating that the optimal learning rate scaling is inherently data-dependent. The authors show that traditional data-agnostic scaling rules do not effectively transfer across different depths of the network. By analyzing a simple model of deep scalar linear networks, the study reveals that the learning dynamics can be expressed through exact solutions involving special functions. The results indicate that under a data-dependent optimal scaling, the learning dynamics remain independent of the data and exhibit a weak dependence on depth, leading to a constant linear convergence rate across all depths, including the case of infinite depth. Additionally, the authors extend their findings to deep scalar linear networks with residual connections, confirming similar data-dependent effects. This work refines previous analyses and emphasizes the importance of data in determining optimal learning rates in deep learning models.
Methodology
The authors analyze the gradient descent dynamics of deep scalar linear networks, deriving exact solutions for any integer depth using special functions such as the hypergeometric function and the Lambert W function. They explore the effects of learning rate scaling on convergence and stability, incorporating a balanced initialization scheme and examining the dynamics under a stable learning rate regime.
Results
The findings reveal that the optimal learning rate scaling is not only dependent on the data but also leads to a constant linear convergence rate across all depths of the network. The analysis shows that data-dependent scaling rules facilitate effective hyperparameter transfer across depths, while data-agnostic rules do not. The results are consistent even when extending the analysis to networks with residual connections.
Implications
This research highlights the critical role of data in optimizing learning rates for deep learning models, suggesting that practitioners should consider data characteristics when setting hyperparameters. The findings could lead to improved training efficiency and performance in deep learning applications, particularly in scenarios involving varying data distributions.
Linear Attention Architectures: Mechanisms, Trade-offs, and Cross-Layer Routing
NLP
Large Language Models
Efficient ML
- Linear attention architectures can significantly reduce computational costs compared to traditional softmax attention.
- Kimi Delta Attention with the Muon optimizer achieved the best validation loss among the architectures tested.
- Gated DeltaNet demonstrated the highest training throughput, highlighting the trade-off between efficiency and accuracy.
- Cross-Layer Value Routing (CLVR) provides a lightweight method for improving performance in DeltaNet-style architectures.
Read more
Linear Attention Architectures: Mechanisms, Trade-offs, and Cross-Layer Routing
Summary
This paper investigates the limitations of traditional self-attention mechanisms in transformer models, particularly their quadratic computational cost with respect to sequence length. It presents a comparative analysis of softmax attention and four recent linear attention architectures: DeltaNet, Gated DeltaNet, Kimi Delta Attention, and Gated DeltaNet-2. The authors express these architectures in a unified recurrent-memory framework, highlighting differences in expressivity, memory management, and computational efficiency. The study includes extensive experiments with 350M-parameter models trained on 15B tokens, examining various factors such as optimizer performance, learning rates, and hybrid versus pure stack structures. Notably, Kimi Delta Attention with the Muon optimizer achieved the lowest validation loss, while Gated DeltaNet demonstrated the highest training throughput. The paper also introduces lightweight cross-layer routing mechanisms, specifically Cross-Layer Value Routing (CLVR), which showed modest improvements in validation loss for DeltaNet and Gated DeltaNet architectures. Overall, the findings aim to clarify the design space of linear attention mechanisms, providing insights into their trade-offs and performance characteristics.
Methodology
The authors employed a recurrent-memory notation to express and compare different linear attention architectures. They conducted experiments on 350M-parameter models trained with 15B tokens, analyzing various factors including optimizer performance, learning rates, and hybrid versus pure stack structures. The study also introduced and evaluated cross-layer routing mechanisms.
Results
The experiments revealed that Kimi Delta Attention with the Muon optimizer achieved the lowest final validation loss, while Gated DeltaNet had the highest normalized training throughput. The introduction of CLVR showed a modest reduction in validation loss for both DeltaNet and Gated DeltaNet architectures, indicating potential for improved performance with cross-layer routing.
Implications
The findings suggest that linear attention architectures can provide efficient alternatives to traditional self-attention mechanisms, particularly for applications requiring long context windows. The insights into cross-layer routing may also inform future designs of neural architectures that balance efficiency and expressivity.
Scalable and Trustworthy Earth Observation Foundation Models
Computer Vision
Multimodal
Reinforcement Learning
- Foundation models (FMs) are crucial for adapting to diverse downstream tasks in Earth observation.
- Remote sensing data require specialized models due to their unique physical and operational characteristics.
- Evaluation of RSFMs should consider not just accuracy but also transferability and physical plausibility.
- Case studies demonstrate the practical implications of RSFMs in environmental monitoring.
Read more
Scalable and Trustworthy Earth Observation Foundation Models
Summary
This paper discusses the development and evaluation of Remote Sensing Foundation Models (RSFMs) tailored for Earth Observation (EO) data. It highlights the limitations of existing models that are not optimized for the unique characteristics of EO data, which include multimodality, varying spatial and spectral resolutions, and temporal dynamics. The authors argue that RSFMs must be designed with an understanding of the physical principles governing EO data and the operational constraints faced in real-world applications. The paper reviews the current landscape of RSFMs, including their pretraining objectives and adaptation strategies, and emphasizes the need for models to be evaluated not only on benchmark accuracy but also on their ability to provide physically plausible representations and modality-aware transfers. Two case studies are presented: one focusing on predicting harmful algal blooms using physics-informed spectral masking, and another on adaptive environmental monitoring station selection using reinforcement learning. These examples illustrate the practical application of RSFMs and the importance of domain-specific adaptations for reliable EO decision-making.
Methodology
The authors review existing RSFMs and their design principles, focusing on pretraining objectives, model architectures, and adaptation protocols. They incorporate case studies to illustrate the application of these principles in real-world scenarios, emphasizing the importance of domain-specific adaptations.
Results
The paper reveals that no single geospatial foundation model is universally optimal, and it highlights the challenges of inconsistent evaluation methods. The case studies demonstrate effective applications of RSFMs in predicting environmental phenomena and optimizing monitoring strategies.
Implications
The findings suggest that future RSFMs should be developed with a focus on the unique characteristics of EO data, which could enhance the reliability and trustworthiness of decisions made in environmental monitoring and management.
ArtMine: Discovering and Formalizing Artistic Processes
Generative Models
Interpretability
Multimodal
- ArtMine shifts the focus from modeling completed artworks to reconstructing creative processes from historical evidence.
- The framework integrates deep-research-based evidence construction with Peircean abductive reasoning.
- A case study demonstrates the feasibility of using fragmented documentary evidence to infer structured artistic workflows.
- ArtMine supports the development of interpretable and auditable representations of artistic processes.
Read more
ArtMine: Discovering and Formalizing Artistic Processes
Summary
The paper introduces ArtMine, a novel framework designed to discover and formalize artistic processes from heterogeneous historical evidence. Unlike existing generative AI systems that focus on modeling finished artworks, ArtMine aims to reconstruct the iterative decisions and contextual influences that shape artistic production. The framework synthesizes fragmented documentary evidence, such as archival records and preparatory studies, into a structured repository. A Peircean abductive agent then infers evidence-grounded production steps, which are organized into a compositional graph. This process is optimized through self-reflection, comparing generated outputs with reference artworks. The authors present a proof-of-concept case study that demonstrates how ArtMine can effectively utilize open-domain historical sources to create coherent and interpretable representations of artistic workflows. The work emphasizes the importance of understanding creative processes for enhancing human-AI co-creativity, artistic interpretation, and educational applications.
Methodology
ArtMine employs a multi-step approach that includes gathering heterogeneous historical evidence through a deep-research agent, organizing this evidence into a structured repository, and utilizing a Peircean abductive reasoning agent to infer plausible artistic production steps. The inferred processes are rendered and optimized through self-reflection based on deviations from reference artworks.
Results
The case study provided evidence that ArtMine can successfully reconstruct coherent and auditable artistic workflows from fragmented historical sources, demonstrating the potential for structured inference over creative processes.
Implications
ArtMine has significant implications for the fields of artistic interpretation, creative education, and collaborative human-AI systems. By focusing on the processes of creation rather than just the final artifacts, it opens new avenues for understanding and teaching artistic practices.
Deep Learning for Joint Narrowband Interference Cancellation and Soft Demodulation in OFDM Systems
Theory
Efficient ML
Optimization
- Introduces a deep learning framework for joint NBI cancellation and soft demodulation in OFDM systems.
- NBI-CNet reduces computational complexity by up to 60% compared to state-of-the-art methods.
- LLR-CNet effectively maps non-Gaussian residuals to reliable soft metrics.
- Demonstrates significant performance improvements under severe interference conditions.
Read more
Deep Learning for Joint Narrowband Interference Cancellation and Soft Demodulation in OFDM Systems
Summary
This paper addresses the challenges posed by narrowband interference (NBI) in orthogonal frequency-division multiplexing (OFDM) systems, which can severely degrade performance by corrupting subcarriers. Traditional methods, such as compressed sensing (CS), suffer from high latency and leave residuals that hinder effective demodulation. The authors propose a unified deep learning framework that combines NBI cancellation and robust soft demodulation. The framework includes two main components: NBI-CNet, a convolutional neural network (CNN) that estimates NBI parameters and removes interference efficiently, and LLR-CNet, which transforms non-Gaussian residuals into calibrated soft metrics. The proposed approach significantly reduces computational complexity and enhances performance, achieving a coding gain of over 3 dB in scenarios with severe interference. The architecture is designed to generalize across different FFT sizes without retraining, making it adaptable to varying conditions in real-world applications.
Methodology
The authors developed two neural network architectures: NBI-CNet for estimating NBI parameters and mitigating interference, and LLR-CNet for transforming the output into calibrated soft metrics. The models leverage convolutional layers to process raw data efficiently, avoiding the need for prior knowledge of the number of active interferers.
Results
The proposed framework outperforms traditional methods, eliminating error floors and achieving a block error rate of 10^-4 with minimal SNR margins from optimal baselines. Under severe interference conditions, the system operates effectively, maintaining performance levels close to the best iterative methods.
Implications
This research has significant implications for the design of future wireless communication systems, particularly in 5G and beyond, where interference management is crucial. The deep learning approach can enhance the reliability of OFDM systems in congested environments, making it suitable for applications in dense IoT networks and other scenarios with high interference.
Write-Protected Discrete Bottlenecks for Language-Grounded World Models: A Structural Limitation and Sufficient Fix
Robotics
NLP
Multimodal
- Language gradients entering discrete bottlenecks lead to a structural trade-off that limits performance.
- A three-layer architectural fix is proposed to overcome the identified limitations.
- The proposed solution achieves high grounding accuracy across various architectures and environments.
- The findings challenge the end-to-end scaling paradigm in embodied AI.
Read more
Write-Protected Discrete Bottlenecks for Language-Grounded World Models: A Structural Limitation and Sufficient Fix
Summary
This paper investigates the interaction between language and discrete symbol systems in robot world models, challenging the prevalent end-to-end integration of large language models (LLMs) and vision-language models (VLMs) with robotic systems. The author identifies a critical structural limitation where language gradients entering a discrete symbol bottleneck lead to a trade-off: the Gumbel-softmax estimator collapses to a minimal number of symbols, while alternative strategies maintain diversity but fail to learn semantic labels effectively. To address this issue, a three-layer architectural fix is proposed: (1) cutting the gradient chain to prevent language signals from reaching the symbol bottleneck, (2) introducing a gradient-free semantic channel using a non-parametric Memory Table for co-occurrence counting, and (3) employing DP-Means streaming clustering for collision resolution among symbols. The combination of these layers significantly improves grounding accuracy from 22.2% to 97.2%. The findings are validated across multiple encoder architectures and environments, demonstrating the robustness of the proposed solution. This work emphasizes the need for architectural separation between physical perception and language processing, challenging the assumption that larger LLMs inherently lead to better grounding in embodied AI.
Methodology
The study employs empirical experiments to test the performance of end-to-end language integration in robotic systems, comparing it with the proposed three-layer fix. The methodology includes the use of Gumbel-softmax estimators, anti-collapse strategies, and a non-parametric Memory Table for semantic binding, validated across different encoder architectures and environments.
Results
The results indicate that end-to-end approaches face a structural ceiling, with the proposed three-layer fix achieving grounding accuracy of 97.2% compared to 22.2% without the collision resolution layer. The experiments demonstrated zero symbol collapse across 74 independent runs, confirming the effectiveness of the architectural modifications.
Implications
The findings suggest that future work in embodied AI should focus on separating language processing from physical perception to enhance grounding accuracy. The proposed architecture offers a minimal baseline for further exploration of how language and physical systems can be integrated more effectively.
SHIFT: Survival Prediction from Incomplete and Heterogeneous Genomic Data
Theory
- SHIFT is a transformer-based model that predicts survival from incomplete genomic data without imputation.
- The model employs a variable-rate masking strategy during training to enhance robustness to cross-cohort variability.
- Incorporating patients from incomplete cohorts during model development can improve predictive performance.
- SHIFT shows strong generalization across multiple cohorts, even with severe cross-cohort panel mismatches.
Read more
SHIFT: Survival Prediction from Incomplete and Heterogeneous Genomic Data
Summary
The paper presents SHIFT, a novel transformer-based model designed for survival prediction using incomplete genomic data, addressing the challenges posed by heterogeneous genomic panels across different institutions. Traditional genomic prediction models often struggle with missing data due to varying sequencing panels, leading to the exclusion of patients with incomplete profiles or reliance on imputation methods that can distort biological signals. SHIFT utilizes a masked self-attention mechanism and a feature-availability mask to directly predict outcomes from incomplete genomic inputs without the need for test-time imputation. The model is trained with a variable-rate feature masking strategy to enhance its robustness against diverse missingness patterns. Evaluated on glioblastoma and lung squamous cell carcinoma datasets, SHIFT demonstrates strong generalization capabilities and outperforms standard survival baselines and imputation-based methods. The findings suggest that incorporating patients from incomplete cohorts can improve model performance, advocating for a more inclusive approach in multi-center survival prediction in precision oncology.
Methodology
SHIFT employs a masked self-attention mechanism to focus on available genomic features while handling missing data. It incorporates a feature-availability mask and is trained using a variable-rate feature masking strategy that simulates different patterns of missingness, allowing the model to learn inter-feature relationships effectively.
Results
SHIFT demonstrated superior performance in survival prediction across glioblastoma and lung squamous cell carcinoma datasets, outperforming traditional survival models and imputation-based approaches. The model maintained strong generalization capabilities despite significant structural missingness in the data, indicating its effectiveness in real-world clinical settings.
Implications
The findings suggest that missingness-aware modeling can significantly enhance the robustness of survival predictions in precision oncology, allowing for better integration of multi-center genomic data. This approach could lead to more accurate prognostic tools and improved patient outcomes by utilizing diverse datasets that include incomplete profiles.
Stop Guessing When to Stop Testing: Efficient Model Evaluation with Just Enough Data
Efficient ML
Theory
- Proposes an adaptive evaluation framework to replace fixed-size benchmarks.
- Utilizes sequential testing to balance efficiency and reliability in model evaluation.
- Demonstrates significant cost savings (up to 80%) while maintaining statistical significance.
- Allows users to define stopping criteria based on practical evaluation needs.
Read more
Stop Guessing When to Stop Testing: Efficient Model Evaluation with Just Enough Data
Summary
This paper addresses the inefficiencies of fixed-size benchmarks in model evaluation, which often lead to excessive computational costs or unreliable results. The authors propose an adaptive evaluation framework that utilizes sequential testing to balance the trade-off between efficiency and reliability. This framework allows for stopping evaluations based on user-defined criteria, such as achieving a minimum detectable effect size or detecting diminishing returns. The authors demonstrate the effectiveness of their approach on the Open VLM Leaderboard, achieving up to an 80% reduction in computational costs while maintaining statistical significance. The framework is designed to adaptively manage evaluation needs, ensuring that evaluations are neither underpowered nor excessive, thus providing a transparent and efficient evaluation process.
Methodology
The authors developed an adaptive evaluation framework that integrates sequential testing methods with tailored stopping criteria for various evaluation objectives. This framework allows for early termination of evaluations when sufficient statistical power is achieved, thus optimizing resource usage.
Results
The framework was tested on the Open VLM Leaderboard, showing an 80% reduction in computational costs while maintaining a confidence interval width of Β±2.5 points. It also demonstrated the ability to stop evaluations based on diminishing returns, achieving a 44% cost reduction with minimal loss in precision.
Implications
The proposed framework can significantly enhance model evaluation practices in machine learning, particularly in scenarios with limited computational resources. It allows practitioners to make informed decisions about model performance without unnecessary evaluations, ultimately leading to more efficient development cycles.
Robust Bayesian Decision Making under Adversarial Uncertainty
Theory
Optimization
- Focus on decision stability rather than just nominal optimality in experimental design.
- Introduction of adversarially robust decision-aware experimental design framework.
- Validation through experiments showing improved stability and reliability of decisions.
- Emphasis on the influence of hidden or weakly modeled adversarial variables on decision-making.
Read more
Robust Bayesian Decision Making under Adversarial Uncertainty
Summary
This paper addresses the challenge of designing scientific experiments that support reliable decision-making in the presence of adversarial uncertainty. Traditional experimental design methods often assume well-specified outcome models and can lead to fragile decisions when faced with unexpected variations in outcomes due to hidden or weakly modeled effects. The authors propose a framework for adversarially robust decision-aware experimental design, which builds on Bayesian decision theory. This framework emphasizes the importance of decision stability over nominal optimality, aiming to ensure that decisions remain reliable under plausible worst-case scenarios. The proposed methodology is validated through experiments on both synthetic and real-world datasets, demonstrating that conventional decision-aware designs can yield high confidence but fragile decisions, while the robustness-aware approach results in significantly more stable and reliable outcomes.
Methodology
The authors develop a framework based on Bayesian decision theory that formalizes adversarially robust optimal decision-making. This involves deriving a Bayesian experimental design criterion that explicitly targets decision stability under adversarial variations. The methodology includes theoretical formulations and empirical validation through experiments on various datasets.
Results
The experiments reveal that traditional decision-aware experimental designs can quickly converge to high-confidence decisions that are not robust to adversarial variations. In contrast, the proposed robustness-aware approach yields decisions that are significantly more stable and reliable, demonstrating the effectiveness of the new framework in real-world scenarios.
Implications
This research has significant implications for fields requiring reliable decision-making under uncertainty, such as personalized medicine and other scientific domains. By prioritizing robustness in experimental design, practitioners can make more informed and stable decisions, ultimately leading to better outcomes in complex systems.
Ensemble Diversity Optimization for Subjective Supervision
NLP
Optimization
Theory
- EDO optimizes ensemble structure and diversity to handle annotator disagreement in subjective NLP tasks.
- The framework uses a signed diversity regularizer to control the balance between preserving and suppressing disagreement.
- EDO significantly improves probabilistic calibration and maintains competitive performance on subjective classification tasks.
- The method is model-agnostic and can be integrated into various predictive models.
Read more
Ensemble Diversity Optimization for Subjective Supervision
Summary
This paper introduces Ensemble Diversity Optimization (EDO), a novel framework designed to address the challenges posed by subjective NLP tasks that often exhibit significant annotator disagreement. Traditional supervised learning methods typically collapse this variability into a single target, which can lead to overfitting and loss of valuable information. EDO proposes a prediction-space approach that optimizes ensemble weights, cardinality, and calibration through a unified differentiable objective. It incorporates a signed diversity regularizer that can either preserve or suppress disagreement based on validation data, thus preventing ensemble collapse and allowing for controlled navigation of the utility-calibration trade-off. The framework employs a soft F1 surrogate and class-weighted cross-entropy to manage class imbalance, alongside reliability-weighted diversity to regulate intra-ensemble variability. Experimental results on four subjective text-classification benchmarks demonstrate that EDO significantly enhances probabilistic calibration, achieving reductions in cross-entropy and Brier scores while maintaining competitive F1 scores and better alignment with annotator distributions. This indicates that EDO provides an efficient, model-agnostic method for modeling human subjectivity in supervised learning.
Methodology
EDO employs a prediction-space framework that utilizes GumbelβSoftmax relaxation for end-to-end learning of ensemble composition and size. It incorporates a signed diversity regularizer to manage disagreement based on validation data, and optimizes multiple objectives including predictive utility, calibration, and internal diversity through a unified differentiable objective.
Results
Experiments on four subjective text-classification benchmarks (ArMIS, ConvAbuse, HS-Brexit, MD-Agreement) show that EDO reduces cross-entropy by 40-78% compared to baseline methods, lowers Brier scores, and maintains competitive F1 scores while achieving better alignment with annotator distributions.
Implications
The findings suggest that EDO can be effectively applied to various NLP tasks characterized by subjective interpretations, such as sentiment analysis and content moderation, improving model robustness and interpretability in the presence of annotator disagreement.