AI-generated summaries
Today's ML research,
without the noise.
Daily summaries of the latest machine learning papers from arXiv, processed every 8 hours.
64
Papers today
8h
Update frequency
7
Days of history
Adversarial Attacks on Online Handwriting using Salience-based Temporal Editing
Time Series
- Introduces a new adversarial attack framework for online handwriting recognition based on temporal editing.
- Demonstrates the inadequacy of spatial perturbations for time series data in maintaining handwriting quality.
- Achieves stronger one-shot black-box transferability compared to conventional image-based attacks.
- Preserves the visual structure of handwriting while effectively targeting model vulnerabilities.
Read more
Adversarial Attacks on Online Handwriting using Salience-based Temporal Editing
Summary
This paper addresses the vulnerability of online handwriting recognition systems to adversarial attacks, which have primarily been developed for image-based inputs. The authors propose a novel attack framework called Adversarial Iterative Temporal Editing (AITE) that utilizes salience-guided temporal editing instead of traditional spatial perturbations. By inserting and deleting points in the time series of pen trajectories based on their temporal salience, the method preserves the natural shape and smoothness of handwriting. The salience is estimated using gradient-based activation mapping, targeting critical time steps that influence the model's predictions. The proposed method is evaluated on the Unipen and CASIA-OLHWDB datasets under both white-box and one-shot black-box settings. Results indicate that while conventional image-based attacks perform well in white-box scenarios, they lack transferability across models. In contrast, the AITE method demonstrates improved transferability in black-box settings while maintaining visual fidelity, highlighting its relevance as a threat model for online handwriting recognition.
Methodology
The proposed AITE method generates adversarial examples by performing discrete editing operations (insertion and deletion of points) on the time series data of handwriting. Temporal salience is estimated using Gradient-weighted Class Activation Maps (Grad-CAM) to identify critical time steps for modification, ensuring the overall geometric structure and kinematic smoothness of the handwriting is preserved.
Results
The experimental evaluation shows that AITE achieves better visual similarity to original handwriting compared to traditional spatial perturbation methods. It also demonstrates effective performance in one-shot black-box attack scenarios, indicating its robustness across different models.
Implications
The findings suggest that adversarial attacks on online handwriting recognition systems can be more effectively executed through temporal editing methods, which could have significant implications for the security and reliability of such systems in real-world applications.
From Novice to Expert: Cost-Aware Bandits for Evolving Worker Performance in Crowdsensing
Optimization
Theory
- Introduces a structured bandit model for evolving worker performance in crowdsensing.
- Develops the CATI-UCB algorithm to optimize worker selection under budget constraints.
- Demonstrates the importance of accounting for learning dynamics in worker performance.
- Provides theoretical guarantees for the proposed method's performance.
Read more
From Novice to Expert: Cost-Aware Bandits for Evolving Worker Performance in Crowdsensing
Summary
This paper addresses the challenge of online worker recruitment in mobile crowdsensing (MC) systems, where the performance of workers evolves with experience and costs are uncertain. Traditional models assume fixed worker performance, which neglects the learning dynamics of workers who improve over time. The authors propose a structured bandit model that captures the increasing-then-converging performance of workers, allowing for effective budget-constrained recruitment. They develop a cost-aware online learning framework, CATI-UCB, which learns the evolving reward trajectories and heterogeneous costs while detecting performance saturation. The framework balances exploration and exploitation to maximize long-term sensing utility. Theoretical performance guarantees are provided, and extensive experiments demonstrate that CATI-UCB outperforms baseline methods that overlook experience-driven dynamics or assume known costs.
Methodology
The authors formulate the worker selection problem as a structured bandit setting, where each worker's expected reward follows an unknown increasing-then-converging function. They develop the CATI-UCB algorithm, which incorporates a reward-cost ratio for decision-making, an online linear model for early learning behavior estimation, and change-point detection for identifying performance saturation.
Results
The CATI-UCB algorithm achieves sublinear regret compared to the optimal policy, indicating effective learning and decision-making under uncertainty. Experimental results show that CATI-UCB consistently outperforms baseline methods that do not account for evolving worker performance or assume known costs.
Implications
The findings suggest that crowdsensing platforms can significantly enhance data quality and efficiency by adopting learning-aware recruitment strategies. This approach can be applied to various domains requiring real-time data collection from distributed users, such as urban monitoring and environmental sensing.
Data-Efficient Adaptation of LLMs via Attention Head Reweighting
NLP
Large Language Models
Efficient ML
- Introduces Attention Head Reweighting (AHR) for data-efficient adaptation of LLMs.
- AHR learns only one scalar per attention head, drastically reducing trainable parameters.
- Outperforms standard baselines like LoRA with 200-1000 times fewer parameters.
- Demonstrates significant accuracy improvements in security-related text classification tasks.
Read more
Data-Efficient Adaptation of LLMs via Attention Head Reweighting
Summary
This paper addresses the challenge of adapting large language models (LLMs) to new text-classification tasks with limited labeled data, particularly in high-stakes domains like security. The authors propose a novel method called Attention Head Reweighting (AHR), which simplifies the adaptation process by learning only a single scalar per attention head, significantly reducing the number of parameters that need to be trained. This approach leverages the functional specialization of attention heads, allowing the model to focus on the most relevant heads for specific tasks. Experimental results demonstrate that AHR outperforms established baselines like LoRA, achieving better performance with 200-1000 times fewer trainable parameters. The method shows notable improvements in accuracy, particularly in security-related tasks such as phishing detection and jailbreak detection, even when trained on as few as 10 samples. Additionally, the learned weights are interpretable, providing insights into the attention mechanisms that contribute to in-context learning in LLMs. The paper highlights the potential of AHR to enhance data-efficient learning in scenarios where labeled data is scarce, making it a valuable tool for real-world applications in AI security.
Methodology
The authors developed the AHR method, which adapts LLMs by adjusting the weights of individual attention heads rather than modifying the entire model's parameters. This approach minimizes the risk of overfitting in data-scarce environments by focusing on the most relevant heads for specific tasks. The methodology includes experiments on various text classification datasets, particularly in security contexts, to validate the effectiveness of AHR.
Results
AHR achieved an average accuracy improvement of around 3 percentage points over LoRA when trained on ≤100 examples. In specific tasks like Webpage Phishing Detection and Jailbreak detection, AHR showed 6-7% accuracy gains with only 10 training samples, while utilizing significantly fewer trainable parameters (200-1000 times less). The method also improved accuracy by approximately 10 percentage points when employing in-context finetuning.
Implications
The findings suggest that AHR can be a powerful tool for adapting LLMs in scenarios where labeled data is limited, particularly in critical applications like AI security. The ability to interpret the learned weights also aids in understanding model behavior, which is essential for developing robust AI systems that can respond to evolving threats.
Algebraic Representability as the Limiting Regime of Grokking: An Exactly Solvable Model with Holomorphic Activations
Theory
- Introduces an algebraic characterization of representability in neural networks.
- Demonstrates that representability constraints affect both memorization and generalization.
- Finds that non-representable tasks cannot be fitted even on training data.
- Establishes a binary outcome in performance: instant success or failure, with no grokking.
Read more
Algebraic Representability as the Limiting Regime of Grokking: An Exactly Solvable Model with Holomorphic Activations
Summary
This paper investigates the phenomenon of grokking in neural networks, particularly in the context of modular arithmetic tasks. Grokking refers to the delayed transition from memorization to generalization during training, which is influenced by the model's capacity. The authors focus on a two-layer neural network with holomorphic monomial activation functions, specifically trained on tasks encoded via roots of unity. They establish a complete algebraic characterization of the expressible function class of the network, demonstrating that a task is representable if its discrete Fourier support lies within a specific set of frequencies. This representability criterion imposes constraints not only on generalization but also on memorization itself, as non-representable targets cannot be fitted even on the training set. The authors conduct extensive experiments, confirming that their algebraic predictions align closely with observed outcomes, revealing a binary behavior in performance: either instant success or outright failure, with no intermediate grokking phase. This work highlights the importance of representability in understanding the dynamics of neural network training and the conditions under which grokking occurs.
Methodology
The authors utilize a two-layer complex-valued neural network with holomorphic monomial activations, trained on modular arithmetic tasks. They analyze the expressible function class through algebraic methods and conduct 585 experimental runs to validate their theoretical predictions against observed outcomes.
Results
The study finds that the algebraic predictions match the experimental outcomes with 99.8% accuracy. The results indicate a clear division in performance outcomes, with no instances of grokking observed, reinforcing the idea that representability is a crucial factor in determining the training dynamics of the network.
Implications
This research has implications for the design of neural networks, particularly in understanding the limits of expressibility and the conditions necessary for effective learning. It suggests that architectures with fixed algebraic expressible classes can provide insights into the fundamental constraints of neural network training.
TSSM: Triaxial State Space Model for Global Station Weather Forecasting with Temporal-Variable-Historical Modeling
Time Series
- Introduction of the Triaxial State Space Model (TSSM) for improved weather forecasting.
- Utilizes a history-enhanced Temporal-Variable-Historical paradigm to capture long-term weather patterns.
- Achieves state-of-the-art performance on the Weather-5K dataset with significant accuracy gains.
- Demonstrates robustness in forecasting under missing observations.
Read more
TSSM: Triaxial State Space Model for Global Station Weather Forecasting with Temporal-Variable-Historical Modeling
Summary
The paper introduces a novel Triaxial State Space Model (TSSM) aimed at improving Global Station Weather Forecasting (GSWF) by addressing the limitations of existing methods that rely heavily on short-term temporal patterns. The authors argue that these methods struggle with accuracy, particularly in extreme weather events and long-horizon forecasting due to error accumulation. TSSM incorporates a history-enhanced Temporal-Variable-Historical paradigm, which utilizes period-aligned historical weather data to capture long-term patterns beyond conventional temporal look-back windows. The model organizes historical data into triaxial tensors, allowing for the modeling of temporal dependencies, variable correlations, and historical evolution. The results demonstrate that TSSM achieves state-of-the-art performance on the Weather-5K dataset, showing significant improvements in accuracy and extreme event metrics, particularly in long-horizon and iterative forecasting scenarios. The model also exhibits robustness under conditions of missing observations, making it suitable for real-world applications in global weather forecasting.
Methodology
The TSSM employs a triaxial data reorganization approach, stacking historical weather data into a tensor format that aligns temporal, variable, and historical axes. It utilizes causal forecasting techniques to predict future weather dynamics based on both current and historical observations, enhancing the model's ability to capture long-term dependencies and mitigate error accumulation in iterative forecasting.
Results
TSSM achieved a 10% improvement in accuracy and a 61% increase in extreme event metrics on the Weather-5K dataset. It also showed a 37.5% gain in performance at a 240-hour forecasting horizon and up to 103.5% improvement in a 48-hour iterative setting. The model maintained over 90% performance even with up to 80% missing observations, compared to less than 43% for baseline models.
Implications
The TSSM model has significant implications for enhancing the accuracy and reliability of localized weather forecasts, which are critical for disaster management, transportation, and climate change mitigation. Its robustness to missing data also suggests potential applications in real-world scenarios where data incompleteness is common.
An Agentic AI Scientific Community for Automated Neural Operator Discovery
Theory
Large Language Models
Optimization
- Introduces an agentic AI framework for neural operator discovery using a community of virtual labs.
- Demonstrates the effectiveness of LLM agents in proposing diverse neural architectures.
- Finds that LLM agency is crucial for maintaining architectural diversity in the discovery process.
- Establishes a no-free-lunch theorem for neural operators, indicating no universal winner across problems.
Read more
An Agentic AI Scientific Community for Automated Neural Operator Discovery
Summary
This paper introduces an innovative agentic framework for the autonomous discovery of neural operators through a simulated AI scientific community. The community consists of multiple virtual laboratories that operate under a citation-based economy, where successful labs influence the creation of new labs while underperforming ones are replaced. Each lab is composed of three specialized agents: a large language model (LLM) planner that proposes neural architectures, a numerical worker that trains these architectures, and an LLM reviewer that conducts peer reviews across labs. The framework is evaluated on five different problems, including various partial differential equations (PDEs), demonstrating its ability to discover high-accuracy and low-parameter-count neural operator architectures. The study reveals that the LLM planners predominantly hybridize architectures, achieving a 99.8% rate of hybridization in their decisions. An ablation study indicates that replacing LLM agents with rule-based alternatives leads to a collapse of diversity in architecture, emphasizing the necessity of LLM agency for maintaining a rich variety of solutions. The findings suggest a no-free-lunch theorem for neural operators, indicating that no single architecture universally outperforms others across all problems.
Methodology
The authors employ a decentralized framework of virtual laboratories, each containing three types of agents: an LLM planner for architecture proposal, a numerical worker for training, and an LLM reviewer for peer evaluation. The labs operate under a citation-based economy, allowing successful labs to propagate their research direction while replacing less effective ones. The framework is tested on five different PDE-related problems, with results logged for analysis.
Results
The neural operator AI scientific community successfully discovered high-accuracy architectures with low parameter counts. The LLM planners exhibited a strong tendency to hybridize architectures, achieving a 99.8% hybridization rate. The ablation study showed that replacing LLM agents with rule-based agents led to a reduction in architectural diversity, confirming the importance of LLM agency. The results also indicated that no single neural operator architecture was universally superior across all tested problems.
Implications
This research could significantly impact the field of operator learning and PDE solving by providing a framework for automated architecture discovery. The findings emphasize the importance of diversity in neural architectures and suggest that future research could explore the use of agentic AI in other domains of machine learning.
Scalable Optimal Transport Algorithm for Network Alignment
Graph Learning
Optimization
Efficient ML
- Introduction of FastAlign, a scalable framework for OT-based network alignment.
- Preservation of the original OT formulation while optimizing for sparsity.
- Development of a custom SpMM algorithm tailored for network alignment tasks.
- Significant runtime improvements over existing methods, achieving up to 32.54× speedup on GPU.
Read more
Scalable Optimal Transport Algorithm for Network Alignment
Summary
This paper presents FastAlign, a scalable and sparsity-aware framework for optimal transport (OT)-based network alignment. Network alignment is crucial for various applications such as social network analysis and fraud detection, but existing methods often struggle with scalability due to the need for dense matrix computations. FastAlign addresses this issue by preserving the original OT formulation while reinterpreting its computation as a series of mixed sparse-dense operations. This approach allows for the exploitation of graph sparsity and the optimization of dense operations for memory efficiency. The authors implement several optimizations, including a custom sparse-dense matrix multiplication (SpMM) algorithm and domain-specific kernel fusion, which significantly enhance performance. The results demonstrate that FastAlign achieves alignment quality comparable to state-of-the-art methods while reducing end-to-end runtime by up to 9.45× on CPU and 32.54× on GPU, making it practical for large-scale networks.
Methodology
FastAlign utilizes a computational reinterpretation of OT-based network alignment, focusing on mixed sparse-dense operations. It incorporates sparsity-aware graph computations, a custom SpMM algorithm for efficient matrix multiplication, and domain-specific kernel fusion to minimize memory traffic and enhance performance. The implementation is optimized for both CPU and GPU architectures.
Results
FastAlign achieves alignment quality on par with state-of-the-art OT-based methods while significantly reducing runtime. The paper reports speedups of 3.89×–9.45× on CPU and 2.24×–32.54× on GPU compared to existing algorithms, demonstrating its efficiency and scalability.
Implications
The advancements presented in FastAlign have significant implications for applications requiring network alignment, such as social network analysis, fraud detection, and knowledge graph integration. The ability to handle larger networks efficiently opens new avenues for research and practical applications in data science.
Mechanical Analysis of Parachute Suspension Line Deployment with Binding Tapes Using PINN
Theory
Optimization
- Development of a PINN algorithm for predicting tension in parachute suspension lines.
- Investigation of the impact of binding tape parameters on line dynamic tension.
- Validation of the PINN framework against flight test data and conventional numerical methods.
- Improved computational efficiency and accuracy in tension prediction compared to traditional methods.
Read more
Mechanical Analysis of Parachute Suspension Line Deployment with Binding Tapes Using PINN
Summary
This paper addresses the mechanical analysis of parachute suspension line deployment, focusing on the initial extraction and straightening process, which is critical for successful parachute inflation. Traditional methods for calculating line tension, primarily based on numerical integration of ordinary differential equations, are limited in their computational efficiency and accuracy. To overcome these limitations, the authors propose a physics-informed neural network (PINN) algorithm that predicts tension during the line extraction process. The study also investigates how binding tape parameters influence dynamic tension in suspension lines. The proposed PINN framework demonstrates superior performance compared to conventional methods, validated against flight test data and traditional numerical results. The findings contribute to a better understanding of suspension line dynamics and provide insights for optimal parachute design, enhancing safety and reliability in parachute systems.
Methodology
The authors constructed a force model for suspension line particles, using a multi-particle spring-damper model to analyze the dynamic behavior during line extraction. The PINN model was developed to learn continuous solutions of partial differential equations, allowing for interpolation and prediction of tension at arbitrary spatial and temporal points. The influence of binding tapes on line tension was incorporated into the tension calculation model.
Results
The PINN framework outperformed traditional numerical integration methods in both computational efficiency and accuracy. The study provided a comprehensive analysis of force variations in suspension lines from extraction to full straightening, demonstrating the significant role of binding tapes in regulating dynamic tension.
Implications
The findings have important implications for the design and optimization of parachute suspension lines, potentially leading to enhanced safety and reliability in parachute deployment systems. The innovative use of PINN offers a new approach for solving similar engineering problems in dynamic systems.
How the Hessian-Spectrum of Neural Networks Depends on Data
Theory
Optimization
- The Hessian matrix is crucial for understanding optimization dynamics and generalization in neural networks.
- The sharpness of neural network solutions is related to the class distribution in the training data.
- The authors derive eigenvalues of the Hessian for linear networks, extending previous theoretical frameworks.
- Empirical validation shows that predictions remain robust even when relaxing common assumptions.
Read more
How the Hessian-Spectrum of Neural Networks Depends on Data
Summary
This paper investigates the relationship between the Hessian spectrum of neural networks and the data used for training. The authors derive the eigenvalues of the Hessian for linear networks of arbitrary depth and width, focusing on datasets with varying samples, features, and labels. They establish that the sharpness of the solution in classification tasks with Mean Squared Error (MSE) loss is directly linked to the maximum proportion of samples from any class. The study systematically relaxes common assumptions in previous works and empirically validates their findings, demonstrating robustness across various scenarios. The research contributes to a deeper understanding of the loss landscape in neural networks and offers insights into the optimization dynamics influenced by data geometry.
Methodology
The authors utilize the generalized Gauss-Newton approximation to derive the Hessian spectrum for linear neural networks. They analyze the eigenvalues under various assumptions, including the structure of the networks and the characteristics of the datasets. The methodology includes empirical validation through experiments on standard datasets like MNIST and CIFAR10.
Results
The results indicate that the eigenvalues of the Hessian can be approximated accurately as training progresses, with the sharpness of the solution correlating with the distribution of classes in the dataset. The study also reveals that the sum of eigenvalues provides insights into the dynamics of training, particularly regarding the flattening and sharpening of the loss landscape.
Implications
The findings have significant implications for the design of optimization algorithms and generalization measures in deep learning. Understanding the relationship between the Hessian spectrum and data can lead to improved training strategies and better-performing models, particularly in scenarios with imbalanced datasets.
Gauge-Invariant, Parameter-Insensitive Regularization for Potential Recovery from Flow on Directed Graphs
Graph Learning
Theory
Optimization
- Introduces gauge-invariant regularization that prevents inversion of potential ordering.
- Demonstrates parameter-insensitivity across four orders of magnitude in regularization strength.
- Proves that the new method preserves dynamic range and maintains high rank correlation.
- Applies the gauge invariance concept to improve performance in graph neural networks.
Read more
Gauge-Invariant, Parameter-Insensitive Regularization for Potential Recovery from Flow on Directed Graphs
Summary
This paper addresses the challenge of recovering a latent potential from observed flow on directed graphs, a problem that is inherently ill-posed. Traditional methods, particularly ridge regularization, often lead to incorrect solutions by collapsing the recovered ordering of potentials. The author introduces a gauge-invariant graph Dirichlet energy as a new regularization technique that is parameter-insensitive, maintaining stability across a wide range of regularization parameters (λ). The proposed method, termed graph-Sobolev regularization, penalizes differences rather than amplitudes, thus preserving the dynamic range of the recovered potential. The paper demonstrates that this new approach retains 28-41% of the interior dynamic range in comparison to ridge regularization, which can collapse to as little as 0.2%. Additionally, the gauge invariance principle is shown to be applicable in graph neural networks, preventing oversmoothing in deep directed graph convolutional networks (GCNs). The findings are validated through experiments on three public clickstream datasets, showcasing the robustness and effectiveness of the proposed method in practical applications.
Methodology
The paper employs a mathematical framework based on discrete Poisson equations and graph theory. It replaces the standard ridge regularization with a gauge-invariant graph Dirichlet energy, which is derived from the incidence operator. This new penalty is designed to be insensitive to the choice of gauge, thus stabilizing the recovery of potential across varying regularization parameters. The methodology includes theoretical proofs of the properties of the proposed regularization, as well as empirical validation on clickstream data.
Results
The proposed gauge-invariant regularization method significantly outperforms traditional ridge regularization, maintaining a rank correlation of +0.81 across a wide range of λ values, while ridge regularization shows a drastic drop to approximately -0.42. The method retains 28-41% of the dynamic range in potential recovery, compared to ridge's collapse to as low as 0.2%. The results are consistent across multiple datasets, confirming the robustness of the approach.
Implications
The findings suggest that gauge-invariant regularization can be a powerful tool for recovering latent potentials in various applications, such as traffic networks, supply chains, and navigation logs. Furthermore, the application of this principle in graph neural networks could lead to improved performance in deep learning tasks involving graph-structured data, addressing issues of oversmoothing and enhancing feature extraction.
From Geometric Recovery to Causal Validation: A Reproducible Audit of Sparse Autoencoder Features, from Superposition Geometry to Causal Inertness
Interpretability
- Demonstrates the inadequacy of current evaluation metrics for Sparse Autoencoders, highlighting the need for causal validation.
- Introduces the sae-causal-audit tool for rigorous auditing of feature relevance in machine learning models.
- Finds that a significant percentage of features with high cosine similarity are causally inert, questioning their interpretability.
- Proposes a new framework for reproducibility in machine learning, emphasizing the importance of semantic equality over byte-exactness.
Read more
From Geometric Recovery to Causal Validation: A Reproducible Audit of Sparse Autoencoder Features, from Superposition Geometry to Causal Inertness
Summary
This paper investigates the interpretability of Sparse Autoencoders (SAEs) by addressing the limitations of current evaluation practices that rely on correlational recovery metrics. The author demonstrates that these metrics conflate two distinct claims: the alignment of decoder geometry and the behavior of encoder activations. Through a controlled experimental setup, the paper reproduces the superposition phase diagram and the TopK versus L1 comparison, revealing artifacts and new geometric regimes. The central contribution is a causal validation approach that shows a significant portion of features deemed recovered are causally inert, meaning they do not influence model outputs despite high correlation. The author introduces the sae-causal-audit tool, which provides a structured methodology for auditing features across various models. The findings suggest that many features, even those with high cosine similarity, do not contribute to model behavior, highlighting a gap in interpretability methods. The paper also discusses reproducibility challenges in machine learning and proposes a framework for reporting reproducibility claims. Finally, the author applies the methodology to a real model, GPT-2-small, uncovering unexpected patterns in feature activation across unrelated concepts.
Methodology
The author employs a controlled experimental setup to reproduce existing phase diagrams and comparisons in the literature. A causal validation battery is applied to assess the causal efficacy of features through ablation and steering interventions. The sae-causal-audit tool is developed to facilitate this auditing process, ensuring reproducibility and structured evaluation.
Results
The study reveals that up to 77% of features in a degraded SAE and 9% in a well-trained SAE are causally inert despite high correlation metrics. The reproduction of the superposition phase diagram and the TopK versus L1 comparison provides new insights into feature behavior and geometric regimes. The application of the sae-causal-audit tool to GPT-2-small indicates a 14% causal inertness among recovered features, with some decoder atoms recurring across unrelated concepts.
Implications
The findings challenge existing interpretability methods in machine learning, suggesting that many features identified as important may not contribute to model outputs. This has implications for the design of interpretable models and the evaluation of feature relevance. The proposed reproducibility framework could enhance the reliability of machine learning research.
CARE-LoRA: Compressed Activation REconstruction for Memory-Efficient LoRA
NLP
Large Language Models
Efficient ML
- CARE-LoRA effectively reduces memory usage during LoRA fine-tuning by compressing activations.
- The framework maintains the trainability of LoRA matrices while keeping memory overhead low.
- Empirical results show that CARE-LoRA outperforms LoRA-FA and achieves better performance than standard LoRA when memory is reinvested into higher ranks.
- The method is applicable across diverse tasks, demonstrating its versatility and efficiency.
Read more
CARE-LoRA: Compressed Activation REconstruction for Memory-Efficient LoRA
Summary
The paper introduces CARE-LoRA, a novel framework designed to enhance memory efficiency during the fine-tuning of large pre-trained models using Low-Rank Adaptation (LoRA). As the size of these models increases, fine-tuning them within limited memory constraints becomes challenging, primarily due to the memory required for storing activations during backpropagation. While LoRA reduces the number of trainable parameters by optimizing low-rank adaptation matrices, it does not address the memory bottleneck posed by activations. CARE-LoRA addresses this by leveraging the inherent structure of LoRA to replace full input activations with low-rank compressed activations. It computes a lightweight reconstruction matrix during the forward pass, which is then used during backpropagation to reconstruct the gradient signal, allowing LoRA matrices to remain trainable. The authors demonstrate through extensive experiments that CARE-LoRA significantly reduces memory usage while achieving competitive or superior performance compared to standard LoRA and its variants across various tasks, including natural language understanding and image generation.
Methodology
CARE-LoRA employs a data-aware framework that stores compressed activations alongside a lightweight reconstruction matrix. This allows for the reconstruction of gradient signals during backpropagation without the need for full activation storage, thus optimizing memory usage while keeping LoRA factors trainable.
Results
CARE-LoRA reduces total peak memory by approximately 20% compared to standard LoRA. It shows significant performance improvements over LoRA-FA, with an average score increase of 4.3 points on GLUE and 7.4 points on SuperGLUE. Additionally, it achieves the highest average performance on Mistral-7B-v0.3 fine-tuning tasks, requiring only 1.02 times the per-step training time of LoRA.
Implications
The findings suggest that CARE-LoRA can facilitate the fine-tuning of larger models in memory-constrained environments, making it a valuable tool for researchers and practitioners in NLP and other domains requiring efficient model adaptation.
LIDAR-AD: A Decoder-Free Latent-Interaction Dreamer with Action-Residual Chains for Autonomous Driving
Reinforcement Learning
Robotics
Optimization
- Introduces a decoder-free latent interaction model for autonomous driving.
- Focuses on risk-relevant relationships and reduces redundancy in observations.
- Implements residual action updates for improved continuous control.
- Demonstrates superior performance in simulated and real-world driving scenarios.
Read more
LIDAR-AD: A Decoder-Free Latent-Interaction Dreamer with Action-Residual Chains for Autonomous Driving
Summary
The paper presents LIDAR-AD, a novel approach for autonomous driving that addresses the challenges of long-horizon decision-making in dynamic traffic environments. Traditional latent world models often struggle with redundant observations and suboptimal action modeling, which can hinder effective decision-making. LIDAR-AD innovatively replaces observation reconstruction with redundancy-reduced latent alignment, focusing on risk-relevant relationships in driving data. It models vehicle control through residual action updates and employs residual-action sequence contrastive learning to align multi-step actions with future latent states. This approach enhances risk-aware state abstraction, continuous control modeling, and long-horizon dynamics prediction. The authors demonstrate that LIDAR-AD outperforms existing world-model baselines in various simulated driving scenarios, achieving superior rewards and success rates. Additionally, evaluations on real-world traffic scenarios indicate the model's transferability and robustness in practical applications.
Methodology
LIDAR-AD employs a decoder-free approach that emphasizes latent alignment over observation reconstruction. It utilizes residual action updates to model vehicle control and incorporates contrastive learning techniques to align actions with future latent states. The model is designed to extract decision-relevant structures from complex observations while maintaining smooth and predictive control.
Results
LIDAR-AD consistently outperformed world-model baselines across diverse simulated driving scenarios, achieving the highest reward and success rate among learning-based methods. The model also demonstrated strong transferability in evaluations on nuPlan-derived log-reconstructed scenarios, indicating its effectiveness in real-world traffic layouts.
Implications
The findings suggest that LIDAR-AD can significantly improve the safety and efficiency of autonomous driving systems by enhancing decision-making capabilities in complex environments. Its approach may be applicable to other domains requiring long-horizon planning and decision-making under uncertainty.
OrDA: Orthogonal Disentanglement of Access Habits Framework for Homepage Marketing Block Recommendations
Theory
- Introduces a novel framework (OrDA) to disentangle user interests from habitual access patterns in recommendation systems.
- Identifies and formalizes the concept of 'Pseudo-Positives' caused by habitual clicks, providing a new perspective on bias in recommendations.
- Utilizes a dual-tower architecture with orthogonal regularization to ensure rigorous separation of interest and habit signals.
- Demonstrates superior performance over state-of-the-art methods in large-scale empirical evaluations.
Read more
OrDA: Orthogonal Disentanglement of Access Habits Framework for Homepage Marketing Block Recommendations
Summary
The paper introduces the Orthogonal Disentanglement of Access habits (OrDA) framework, which addresses the issue of biased click data in homepage marketing blocks caused by habitual user access. This bias, termed 'Pseudo-Positives', occurs when users click on content not out of genuine interest but due to their habitual navigation patterns. The OrDA framework employs a dual-tower architecture with a gated allocation layer to effectively separate user interests from access habits. By applying orthogonal regularization, the framework ensures that the latent interest and habit manifolds remain geometrically perpendicular, thus minimizing interference. During inference, OrDA utilizes causal intervention techniques to rank items based solely on purified interest scores. The empirical evaluations demonstrate that OrDA significantly reduces access-habit bias and enhances predictive accuracy, achieving a notable 5.64% improvement in user click-through rates (UCTR) in online A/B tests on the Zhima homepage marketing block.
Methodology
The OrDA framework employs a dual-tower architecture designed to decouple user intent from habitual access patterns. It incorporates a gated allocation layer to adaptively route features and minimize interference. Orthogonal regularization is used to constrain the latent interest and habit manifolds to be geometrically perpendicular, ensuring a clear causal boundary. Causal intervention techniques are applied during inference to rank items based on purified interest scores.
Results
OrDA outperforms existing state-of-the-art methods in predictive accuracy on large-scale datasets. An online A/B test conducted on the Zhima homepage marketing block revealed a 5.64% increase in user click-through rates, demonstrating the effectiveness of the proposed framework in real-world applications.
Implications
The findings suggest that the OrDA framework can significantly enhance the accuracy of recommendation systems by effectively addressing bias caused by habitual user behavior. This has potential applications in various online platforms that rely on personalized content recommendations, improving user engagement and satisfaction.
Local Redundancy: An Information-Theoretic Measure of Plasticity from Synthetic Memorization
Theory
Optimization
Time Series
- Local redundancy is a principled measure of plasticity based on information theory.
- It provides a lower bound on plasticity that can be computed efficiently using gradient norms.
- Local redundancy correlates better with future task performance than traditional plasticity metrics.
- The measure aids in selecting optimal pretraining checkpoints for improved adaptation.
Read more
Local Redundancy: An Information-Theoretic Measure of Plasticity from Synthetic Memorization
Summary
This paper introduces 'local redundancy,' a novel information-theoretic measure of plasticity in neural networks, which is crucial for continual and transfer learning. The author critiques existing plasticity measures, such as effective rank and weight norm, for their lack of theoretical grounding and poor correlation with performance on new tasks. Local redundancy is defined as the worst-case redundancy of a local model family, derived from universal compression theory. Although it is intractable to compute exactly, the author proves that the expected squared gradient norm on a synthetic memorization task provides an efficiently computable lower bound for local redundancy. The paper presents experiments in continual image classification and time series transfer learning, demonstrating that local redundancy predicts downstream performance more accurately than existing measures and facilitates better pretraining checkpoint selection when validation loss plateaus.
Methodology
The author defines local redundancy using concepts from universal compression theory, focusing on the worst-case redundancy of parameters in a local neighborhood along gradient directions. The expected squared gradient norm on synthetic memorization data is used to derive a lower bound for local redundancy, which can be computed in a single backward pass during training.
Results
Experiments show that local redundancy outperforms existing plasticity measures in predicting future task accuracy in continual learning scenarios. In time series transfer learning, selecting pretraining checkpoints based on local redundancy results in better adaptation performance compared to traditional methods that rely on validation loss.
Implications
The introduction of local redundancy as a measure of plasticity could significantly enhance the training and adaptation processes of neural networks in continual and transfer learning settings, leading to more robust and efficient learning systems.
STKAN: Kolmogorov-Arnold Networks for Spatio-Temporal Forecasting
Time Series
Graph Learning
- Introduction of STKAN, a KAN-based architecture for spatio-temporal forecasting.
- Incorporation of Taylor-polynomial KAN mappings into spatial and temporal token-mixing modules.
- Development of a learnable soft node-group assignment mechanism for compact spatial representations.
- Demonstration of competitive performance on five traffic forecasting benchmarks.
Read more
STKAN: Kolmogorov-Arnold Networks for Spatio-Temporal Forecasting
Summary
The paper introduces STKAN, a novel architecture for spatio-temporal forecasting that leverages Taylor-polynomial Kolmogorov-Arnold Network (KAN) modules to enhance the modeling of spatial and temporal dependencies in traffic data. The authors argue that existing forecasting models often rely on fixed nonlinear function approximators, which may limit their performance. STKAN addresses this by incorporating a learnable soft node-group assignment mechanism to create high-level spatial representations and employing group-wise spatial mixing followed by temporal modeling of dependencies. The architecture also integrates spatial and temporal self-attention layers to capture long-range interactions. Experiments conducted on five traffic forecasting benchmarks demonstrate that STKAN achieves competitive forecasting accuracy, outperforming a baseline MLP-based variant. The findings suggest that the design of nonlinear function approximators can significantly influence the effectiveness of spatio-temporal forecasting models.
Methodology
The STKAN architecture employs Taylor-polynomial KAN modules for spatial and temporal token mixing, utilizing a learnable soft node-group assignment mechanism to create spatial representations. It combines group-wise spatial mixing with temporal modeling and incorporates self-attention layers to capture long-range dependencies.
Results
STKAN was evaluated on five traffic forecasting benchmarks, showing competitive performance and improved accuracy compared to an MLP-based variant. Ablation studies indicated that the TaylorKAN token mixers, adaptive spatial grouping, and attention components each contributed positively to the model's performance.
Implications
The findings suggest that enhancing the design of nonlinear function approximators can improve spatio-temporal forecasting accuracy, which has significant implications for applications in intelligent transportation systems, autonomous traffic management, and route optimization.
Disentangling Knowledge States with Ability and Proficiency Modeling for Knowledge Tracing
Theory
Time Series
Interpretability
- Introduction of Phase-Aware Knowledge Tracing (PAKT) framework for modeling knowledge states.
- Decomposition of student interaction sequences into ability and proficiency phases.
- Utilization of a multi-branch Transformer architecture for phase-specific modeling.
- Causal analysis revealing biases in traditional phase-agnostic KT models.
Read more
Disentangling Knowledge States with Ability and Proficiency Modeling for Knowledge Tracing
Summary
This paper addresses the challenge of accurately modeling students' evolving knowledge states in the context of Knowledge Tracing (KT). Traditional KT methods often treat interaction sequences as a unified process, neglecting the distinct phases of learning behavior. The authors propose a novel framework called Phase-Aware Knowledge Tracing (PAKT), which decomposes student interactions into two phases: ability and proficiency. The ability phase focuses on initial knowledge acquisition through practice, while the proficiency phase reflects the stabilization and automation of skills. To implement this, a multi-branch Transformer architecture is introduced, which utilizes a type-aware readout module to integrate phase-specific and holistic knowledge representations. The paper also includes a causal analysis to highlight the biases in existing phase-agnostic KT models. Extensive experiments on six public benchmarks demonstrate that PAKT consistently outperforms existing methods, indicating its effectiveness in capturing the dynamics of student learning.
Methodology
The authors developed a decomposition mechanism that partitions student interaction sequences into ability and proficiency phases based on cumulative correct responses. A multi-branch Transformer model processes these decomposed sequences alongside the complete interaction history, employing separate decoders for each phase and a type-aware readout module to unify the knowledge representations for prediction.
Results
The proposed PAKT framework achieved a maximum AUC gain of 1.33% and an average gain of 0.82% over representative baseline models across six public benchmarks, demonstrating its superior performance in predicting student knowledge states.
Implications
The findings suggest that distinguishing between ability and proficiency phases can lead to more accurate predictions of student performance, which can enhance the effectiveness of Intelligent Tutoring Systems (ITS) and personalized learning environments.
Cluster-Weighted EDMD
Theory
Time Series
- CW-EDMD learns state space partitions and corresponding Koopman operators simultaneously.
- The method utilizes an Expectation-Maximization algorithm to assign responsibilities based on proximity and prediction accuracy.
- CW-EDMD significantly outperforms standard EDMD in multiple dynamical systems, particularly in challenging configurations.
- The approach demonstrates residual-awareness, allowing for better prediction in regions where the model is more accurate.
Read more
Cluster-Weighted EDMD
Summary
This paper introduces Cluster-Weighted Extended Dynamic Mode Decomposition (CW-EDMD), a novel approach that enhances the traditional Extended Dynamic Mode Decomposition (EDMD) by learning the partition of the state space jointly with per-cluster Koopman operators. Unlike previous methods that predefined partitions based on various criteria, CW-EDMD employs an Expectation-Maximization (EM) algorithm to determine cluster responsibilities based on both geometric proximity and prediction accuracy. The method is evaluated on three classical dynamical systems: the Lorenz attractor, a damped pendulum, and a double-well Duffing oscillator, demonstrating significant improvements over standard EDMD. The results indicate that CW-EDMD consistently outperforms EDMD across various configurations, particularly in scenarios where EDMD saturates, showcasing its effectiveness in capturing the dynamics of complex systems.
Methodology
CW-EDMD fits separate Koopman operators for each cluster using an Expectation-Maximization (EM) approach. Responsibilities for each cluster are determined by a combination of geometric proximity to the cluster center and the accuracy of predictions made by the cluster's operator. The method updates each Koopman matrix using responsibility-weighted least squares, extending the standard EDMD solution to a per-cluster framework.
Results
In experiments involving 36 configurations across three classical systems, CW-EDMD achieved 258 wins, 4 losses, and 26 ties against EDMD, demonstrating superior performance. The method showed particularly strong results in the Lorenz attractor and the damped pendulum, with significant reductions in mean ℓ2 test error compared to EDMD.
Implications
The findings suggest that CW-EDMD can be effectively applied to complex dynamical systems where traditional EDMD methods struggle, particularly in scenarios with varying dynamics across the state space. This could have implications for fields such as control systems, robotics, and any domain requiring accurate modeling of nonlinear dynamics.
Maximally Robust Satisficing Bayesian Optimization
Optimization
- Introduces the concept of satisficing solutions in Bayesian optimization.
- Focuses on robustness to input perturbations as a selection criterion.
- Presents a new optimization method (MRSBO) that efficiently finds robust satisficing solutions.
- Demonstrates superior performance of MRSBO compared to existing methods.
Read more
Maximally Robust Satisficing Bayesian Optimization
Summary
This paper introduces a novel approach to Bayesian optimization focused on finding satisficing solutions that are robust to input perturbations. Traditional Bayesian optimization aims to find the global optimum of a function, but many practical applications require only a solution that meets a minimum quality threshold. The authors propose a new problem formulation where the goal is to identify a solution that remains satisfactory even when subjected to the maximum expected perturbations after deployment. They argue that robustness to input noise is a critical criterion for selecting solutions from the superlevel set of satisfactory outcomes. The proposed method, Maximally Robust Satisficing Bayesian Optimization (MRSBO), efficiently explores the function space to identify solutions that maximize robustness while minimizing the number of evaluations. The authors demonstrate that MRSBO outperforms existing methods, particularly in scenarios where the optimization phase is conducted under controlled conditions, but the deployment phase is subject to variability.
Methodology
The authors develop a Bayesian optimization framework that assumes clean inputs during the optimization phase but acknowledges that inputs will be perturbed during deployment. The MRSBO method is designed to maximize the robustness radius of solutions while ensuring they remain above a specified quality threshold. The method strategically evaluates function values around selected parts of the level set, avoiding unnecessary evaluations that do not contribute to robustness.
Results
The experimental results show that MRSBO significantly outperforms the best existing method (AdveRS2) in terms of optimization performance and robustness. MRSBO effectively identifies solutions that maintain their quality under perturbations, achieving better convergence rates and lower regret compared to methods that assume adversarial conditions during optimization.
Implications
The findings suggest that MRSBO can be applied in various fields requiring robust design solutions, such as materials science, chemical synthesis, and robotic control. By focusing on satisficing solutions, practitioners can save resources and time while ensuring that the deployed solutions meet necessary performance criteria despite real-world variability.
What Makes a Representational Prior Work? Feature Families, Label-Free Invariances, and Critical Windows in Grokking
Theory
- Feature-family alignment is critical for enabling generalization in machine learning models.
- Label-free invariance priors can outperform label-supervised methods in terms of generalization speed.
- The timing of prior application is essential, with early exposure providing the most significant benefits.
- Coherent but incorrect feature families can block generalization, behaving similarly to random noise.
Read more
What Makes a Representational Prior Work? Feature Families, Label-Free Invariances, and Critical Windows in Grokking
Summary
This paper investigates the factors that determine the effectiveness of representational priors in machine learning, specifically in the context of 'grokking'—the phenomenon where models transition from memorization to generalization. Building on previous work, the author conducts 188 experiments to analyze the impact of feature families, supervision types, timing of prior application, and the structural dissociation across various tasks. The findings reveal that a coherent prior based on an incorrect feature family can hinder generalization, while a label-free invariance prior significantly enhances performance. The study demonstrates that the timing of prior application is crucial, with early exposure yielding the best results. The results indicate that feature-family alignment is essential for enabling generalization, and that label-free invariance can accelerate learning without the need for explicit labels. The paper concludes that a brief early window for applying the prior captures most benefits, refining the understanding of how representational priors function in neural networks.
Methodology
The author conducts a series of controlled experiments using a one-layer transformer model to evaluate different types of representational priors. These include a band-structure control, a commutativity prior, and various timing experiments to assess the impact of early versus late application of priors. The experiments are designed to isolate the effects of feature-family alignment, supervision type, and timing on the model's ability to generalize from memorization.
Results
The results show that a coherent prior based on the wrong feature family leads to poor generalization (1/15 runs), while a label-free commutativity prior achieves full generalization (15/15 runs) with a median speedup of 2.7x. The early application of the prior during the first 2,000 epochs results in a 10/10 success rate, outperforming continuous application. The study also quantifies the impact of structural priors on the weight-norm delay-law exponent, demonstrating a significant reduction in delay when using the correct prior.
Implications
These findings suggest that designing effective representational priors requires careful consideration of feature-family alignment and the potential for label-free learning. The insights gained could inform the development of more efficient training strategies for neural networks, particularly in tasks where labeled data is scarce or expensive to obtain.
RF Spectrogram Anomaly Detection with Quantum Kitchen Sinks: Architecture, Representation, and Hardware Validation
Theory
Efficient ML
Time Series
- Introduction of a novel QKS architecture for RF spectrogram anomaly detection.
- Validation of the QKS approach on real quantum hardware, bridging theory and practical application.
- Demonstration of superior performance of QKS over classical methods in detecting anomalies.
- Development of a comprehensive dataset combining real and synthetic RF signals for robust testing.
Read more
RF Spectrogram Anomaly Detection with Quantum Kitchen Sinks: Architecture, Representation, and Hardware Validation
Summary
This paper addresses the critical need for anomaly detection in radio-frequency (RF) networks, which are vulnerable to malicious transmissions. The authors propose an innovative approach using Quantum Kitchen Sinks (QKS), a hybrid quantum-classical model, to enhance anomaly detection in RF spectrograms. They extend the standard QKS framework by incorporating multi-depth data re-uploading and ring entanglement, allowing for a more nuanced analysis of structured signal data. The study introduces a five-stage ablation protocol to systematically evaluate the impact of various architectural choices and input representations on the anomaly detection performance. The authors validate their approach using real measured sub-6 GHz cellular signals and demonstrate the effectiveness of QKS on a quantum processing unit, achieving a test Area Under the Receiver Operating Characteristic curve (AUROC) of 0.8778 and a test F1 score of 0.7995. The results indicate that QKS outperforms classical baselines across all tested configurations, providing a practical framework for deploying quantum-based anomaly detection in wireless networks.
Methodology
The authors developed a labeled dataset of RF spectrograms by combining real sub-6 GHz cellular signals with synthetically generated anomalous signals. They extended the QKS framework with multi-depth data re-uploading and ring entanglement, and implemented a five-stage ablation protocol to evaluate the effects of various architectural choices and input representations on anomaly detection performance. The experiments were conducted on a quantum processing unit to validate the approach in a real-world context.
Results
The QKS-based anomaly detection achieved a test AUROC of 0.8778 and a test F1 score of 0.7995, outperforming matched classical direct-readout baselines across all evaluated representation-readout pairs. The study found that Discrete Cosine Transform (DCT) representations consistently outperformed raw and PCA inputs, and moderate-depth entangled QKS configurations provided the best performance.
Implications
The findings suggest that QKS can significantly enhance anomaly detection capabilities in wireless networks, making it a viable option for secure spectrum management. This research opens avenues for further exploration of quantum machine learning applications in real-time signal processing and anomaly detection.
Is the Statistical Advantage Worth the Cost? An Empirical Comparison of KANs and MLPs for Structured Data Classification
Theory
Efficient ML
Interpretability
- KANs statistically outperform MLPs in binary and multiclass classification tasks.
- KANs achieve a significant aggregate performance advantage across diverse datasets.
- The medium effect size indicates a trade-off between performance and computational complexity.
- KANs are recommended for high-precision applications, while MLPs are suitable for resource-limited scenarios.
Read more
Is the Statistical Advantage Worth the Cost? An Empirical Comparison of KANs and MLPs for Structured Data Classification
Summary
This study conducts an empirical benchmarking comparison between Kolmogorov-Arnold Networks (KANs) and Multi-Layer Perceptrons (MLPs) for structured tabular classification tasks. The authors evaluate the performance of both architectures on twelve publicly available datasets, covering binary, multiclass, multilabel, and ordinal classification problems. The models were trained under standardized conditions with fixed hyperparameters, and their performance was assessed using test accuracy and F1-Score, alongside statistical significance tests and effect size analysis. The results indicate that KANs statistically outperform MLPs in binary and multiclass domains, achieving a significant aggregate advantage across all datasets. However, the medium effect size (d = -0.46) suggests a cost-benefit consideration, as KANs exhibit higher parameter and computational complexity compared to MLPs. The findings imply that while KANs are preferable for high-precision applications, MLPs remain a viable option for resource-constrained environments. The study encourages further exploration of KANs across additional data modalities to refine architectural selection criteria.
Methodology
The authors implemented and trained KAN and MLP models with comparable architectures and fixed hyperparameters across various classification task types. They evaluated performance using test accuracy and F1-Score, conducted paired hypothesis tests for statistical significance, and analyzed effect sizes to compare predictive gains against architectural complexity.
Results
KANs demonstrated superior performance compared to MLPs in binary and multiclass classification tasks, with a significant aggregate advantage across all datasets. However, the observed medium effect size (d = -0.46) highlighted the increased computational complexity associated with KANs.
Implications
The findings suggest that KANs may be more suitable for applications requiring high precision, while MLPs could be preferred in scenarios where computational resources are limited. This research provides a framework for future studies to evaluate neural architectures in structured data classification.
Saturation Makes Quantization Error Additive: A Coverage Model with a Certificate
Theory
Efficient ML
Optimization
- 85-93% of quantization loss variance can be explained by per-layer effects.
- A coverage model accurately predicts configuration loss based on saturation effects.
- Traditional sensitivity measures can significantly misestimate quantization loss.
- The proposed models achieve lower KL divergence in layer allocation compared to existing methods.
Read more
Saturation Makes Quantization Error Additive: A Coverage Model with a Certificate
Summary
This paper investigates the effects of mixed-precision quantization on deep learning models, particularly focusing on the quantization of weights and activations to 4 bits. The author challenges the conventional wisdom that the loss incurred from quantizing a set of layers can be accurately reconstructed from per-layer or pairwise sensitivity measurements. Through a comprehensive analysis using classical changes of basis, the study reveals that a significant portion (85-93%) of the variance in quantization loss can be attributed to per-layer effects alone. The findings suggest that the quantization loss is bounded, leading to a saturation effect where each layer's contribution to loss is dependent on the quantization of other layers. This saturation is modeled through a coverage function that captures the relationship between quantized layers and their impact on overall loss. The paper introduces two predictive models for estimating configuration loss, both of which demonstrate comparable performance. The results indicate that traditional sensitivity-based methods may misestimate the loss by a substantial margin (71-376%). The proposed models not only provide a more accurate assessment of quantization impacts but also achieve lower KL divergence in allocation strategies compared to existing methods, particularly in large mixture-of-experts models.
Methodology
The author employs a mathematical analysis using Walsh transforms and M"obius expansions to decompose the variance of quantization loss and assess the contributions of individual layers. The study fits a coverage model to measured losses and evaluates the performance of two predictive models for estimating configuration loss.
Results
The analysis shows that the majority of quantization loss variance is attributable to per-layer effects, with the coverage model reproducing the variance profile of loss to within a few percent. The proposed models demonstrate comparable predictive accuracy and outperform traditional methods in terms of KL divergence in allocation strategies.
Implications
The findings suggest that more accurate models for quantization loss can lead to better decisions in mixed-precision quantization, potentially improving the efficiency and performance of deep learning models deployed in resource-constrained environments.
Beyond Backbone Backpropagation: A Decoupled Strategy for Efficient Transfer Learning
Efficient ML
Computer Vision
- Introduction of a decoupled framework that eliminates backbone backpropagation.
- Normalization tuning is proposed for efficient domain adaptation.
- Margin-based weighted training enhances the classifier's performance.
- Achieves competitive accuracy on medical benchmarks with reduced training time.
Read more
Beyond Backbone Backpropagation: A Decoupled Strategy for Efficient Transfer Learning
Summary
This paper presents a novel decoupled framework for transfer learning that eliminates the need for backbone backpropagation, addressing the computational inefficiencies associated with traditional deep learning training methods. The authors propose a strategy that focuses on adapting normalization layers to new domains while separating feature extraction from classifier optimization. This approach significantly reduces training time and computational overhead by precomputing features only once. The authors introduce normalization tuning for efficient domain adaptation and a margin-based weighted training method for the classifier head, which enhances performance in data-scarce scenarios. The proposed method is evaluated across multiple CNN and Transformer architectures on three medical imaging datasets, demonstrating competitive accuracy while achieving substantial reductions in training time and energy consumption. The findings suggest that the performance degradation often observed in transfer learning is more closely related to misalignment in normalization layers rather than the representational capacity of pretrained models, highlighting the importance of lightweight normalization adaptation.
Methodology
The authors developed a decoupled training strategy that separates feature extraction from classifier optimization. They focused on adapting normalization layers (Batch Normalization for CNNs and Layer Normalization for Transformers) with minimal parameter updates. A margin-based weighted loss was introduced for the classifier head to improve decision boundaries, particularly for ambiguous samples.
Results
The proposed approach significantly reduced training time and computational overhead across various architectures while maintaining competitive accuracy on medical datasets. The method was shown to be viable for training on standard CPU infrastructure, offering a practical solution for resource-constrained environments.
Implications
This work provides a practical and environmentally sustainable solution for deploying deep learning models in clinical settings, where computational resources are limited. It emphasizes the need for energy-efficient machine learning practices and offers insights into improving transfer learning performance through targeted normalization adaptations.
Fisher Rank Inflation: A Spectral Signature of Memorization under Label Noise
Theory
- Fisher Rank Inflation is identified as a spectral phenomenon linked to memorization under label noise.
- Corrupted labels inflate effective rank by spreading spectral mass into low-energy eigendirections.
- The study provides a first-order attribution formula that highlights the contribution of corrupted examples to rank inflation.
- Empirical validation shows consistent inflation-collapse dynamics across multiple datasets and architectures.
Read more
Fisher Rank Inflation: A Spectral Signature of Memorization under Label Noise
Summary
This paper investigates the phenomenon of Fisher Rank Inflation, which occurs in deep networks trained with label noise. The authors demonstrate that during the training process, networks initially learn structured patterns before memorizing corrupted labels, leading to a spectral signature in the last-layer gradients. Specifically, the effective rank of the Fisher-gradient scatter matrix expands during the memorization phase and contracts afterward. The study reveals that corrupted labels can increase effective rank by introducing spectral mass into previously unused eigendirections, thus enhancing the entropy of the gradient spectrum. The authors derive a first-order leave-one-out attribution formula to identify the contributions of training examples to rank inflation, showing that corrupted examples tend to contribute more significantly than clean examples. Empirical tests on CIFAR-10 and CIFAR-100 using various architectures confirm the inflation-collapse trajectory of Fisher effective rank, with a notable enrichment of corrupted examples at peak rank checkpoints. The findings suggest that Fisher Rank Inflation serves as a broader spectral signature of memorization, applicable across different noise regimes and datasets.
Methodology
The authors analyze the effective rank of the centered Fisher-gradient scatter matrix during training under label noise. They derive a first-order leave-one-out attribution formula to quantify the contributions of individual training examples to rank inflation. Empirical tests are conducted on CIFAR-10 and CIFAR-100 datasets using SmallCNN, ResNet18, and Vision Transformers under symmetric label corruption, as well as on CIFAR-10N with naturally occurring annotation errors.
Results
The study finds that the effective rank of the Fisher-gradient scatter matrix exhibits a pronounced inflation during the memorization phase, peaking before collapsing after fitting corrupted labels. Corrupted examples are significantly enriched among the highest rank-contributing samples, with top-100 noisy fractions ranging from 69.2% to 96.2%. Additionally, peak Fisher effective rank increases monotonically with corruption severity, from 28.88 under clean training to 97.09 at 60% corruption.
Implications
The findings provide insights into the dynamics of deep learning under label noise, potentially guiding the development of more robust training methods that can better handle label corruption. Understanding Fisher Rank Inflation may also enhance interpretability in neural networks by elucidating how corrupted data influences learning.
Repairing Shape-Prior Shortcuts in Long-Range Single-Shot Fringe Projection Profilometry
Computer Vision
Robotics
Interpretability
- Introduction of PhiCalNet architecture that outputs wrapped-phase representation instead of direct depth.
- Significant reduction in mean absolute error (MAE) from 14.54 mm to 4.46 mm in depth recovery.
- Demonstration of the ineffectiveness of traditional methods to eliminate shape-prior shortcuts through data or model capacity.
- First application of pixel-wise conformal uncertainty quantification in fringe projection profilometry.
Read more
Repairing Shape-Prior Shortcuts in Long-Range Single-Shot Fringe Projection Profilometry
Summary
This paper addresses the limitations of single-shot fringe projection profilometry (FPP) networks that rely on shape-prior shortcuts for depth recovery, particularly in long-range applications. The authors introduce PhiCalNet, a novel architecture that outputs a wrapped-phase representation and maps it to depth through a fixed differentiable calibration layer, effectively removing the shape-prior solution from the hypothesis space. The study demonstrates that traditional methods fail to eliminate these shortcuts through increased data or model capacity. Instead, PhiCalNet achieves a significant reduction in mean absolute error (MAE) from 14.54 mm to 4.46 mm, while maintaining a low residual error confined to a small percentage of pixels. The paper also introduces pixel-wise conformal uncertainty quantification, which localizes errors effectively, marking a first in FPP. The findings suggest that architectural choices in neural networks can significantly impact performance in optical metrology tasks.
Methodology
The authors developed PhiCalNet, which utilizes a wrapped-phase output and a fixed differentiable calibration layer to map phase to depth. The architecture avoids direct depth regression, thus eliminating shape-prior shortcuts. The fringe order is provided as auxiliary input, and a sensitivity analysis confirms the robustness of this approach. The performance of PhiCalNet is compared against a UNet baseline and a physics-informed neural network (PINN) baseline to isolate the architectural advantages.
Results
PhiCalNet achieved a 3.3× reduction in object MAE, reaching 4.46 mm compared to the UNet baseline of 14.54 mm. The residual error was confined to 0.103% of pixels at the ±π wrap discontinuity. A three-frame extension of the method further improved the MAE to 1.16 mm. Additionally, the application of pixel-wise conformal uncertainty quantification localized errors effectively, reducing root-mean-square error by 64% for the top 5% of pixels with snapshot disagreement.
Implications
The findings suggest that architectural design in neural networks can fundamentally alter performance in optical metrology, paving the way for more accurate and reliable depth reconstruction methods in real-time applications. The introduction of uncertainty quantification techniques also enhances the reliability of measurements in practical scenarios.
Tabular Foundation Models for Discrete Choice Estimation
Theory
Efficient ML
Optimization
- TFMs can be adapted for discrete choice estimation by addressing structural mismatches.
- The proposed reformulation captures choice-set dependence and individual heterogeneity.
- The new approach outperforms traditional hierarchical Bayesian methods in predictive accuracy and speed.
- Fine-tuning on population data enhances performance for consumers with shallow purchase histories.
Read more
Tabular Foundation Models for Discrete Choice Estimation
Summary
This paper explores the application of Tabular Foundation Models (TFMs) to discrete choice estimation, a critical framework in marketing and operations for understanding consumer decision-making. The authors identify a structural mismatch between the assumptions of TFMs, which treat observations as independent, and the inherently relational nature of discrete choice data, where choices are dependent on the composition of the choice set and exhibit consumer preference heterogeneity. To address this, they propose a reformulation that encodes choice-set dependence and individual heterogeneity within a row-based learning framework. Their evaluation on a yogurt scanner panel dataset demonstrates that this reformulation significantly enhances predictive accuracy, outperforming traditional hierarchical Bayesian estimation by 8% in holdout log-likelihood and 3.6% in hit rate, while also being 16 times faster. The findings suggest that fine-tuning on population choice data can further improve performance, particularly for consumers with limited purchase histories. This work establishes a new approach for leveraging foundation models in consumer choice problems.
Methodology
The authors reformulate discrete choice prediction tasks for TFMs by creating a choice-set-to-tabular representation that encodes choice-set dependence and individual heterogeneity. This involves constructing set-aware and pairwise inputs and utilizing respondent identifiers for in-context learning. The performance of the reformulated model is evaluated against traditional hierarchical Bayesian methods using a yogurt scanner panel dataset.
Results
The reformulated TFMs achieved an 8% improvement in holdout log-likelihood and a 3.6% increase in hit rate compared to hierarchical Bayesian estimation, while also being 16 times faster in computation. The advantages were most pronounced in medium-data regimes, highlighting the effectiveness of the proposed approach in handling consumer preference heterogeneity.
Implications
The findings suggest that TFMs can be effectively utilized in marketing and operations for demand estimation, providing a faster and more accurate alternative to traditional methods. This approach could be extended to other consumer choice problems, enhancing the understanding of consumer behavior and decision-making processes.
MetaPerch: Learning from metadata for bioacoustics foundation models
Audio & Speech
Multimodal
- MetaPerch utilizes metadata as auxiliary supervision to improve species identification in bioacoustics.
- The model is trained using a multi-task learning approach, addressing challenges related to metadata availability and spurious correlations.
- Extensive experiments show that metadata significantly enhances model performance across diverse datasets.
- The study provides insights into the importance of different metadata modalities and training design choices.
Read more
MetaPerch: Learning from metadata for bioacoustics foundation models
Summary
The paper introduces MetaPerch, a novel bioacoustic foundation model that leverages metadata from citizen science platforms to enhance species identification in bioacoustics. Traditional models primarily rely on vocalization data, but this research highlights the untapped potential of auxiliary metadata, such as recording location and time, to improve model performance. The authors employ a multi-task learning approach, training the model on both species identification and various metadata prediction tasks. This method addresses challenges such as incomplete metadata coverage and the risk of spurious correlations. The extensive empirical study conducted across 17 bioacoustic datasets demonstrates that incorporating metadata significantly enhances species identification accuracy, particularly in the face of domain shifts. The findings underscore the importance of metadata in developing robust models for real-world passive acoustic monitoring applications.
Methodology
The authors developed MetaPerch using a multi-task learning framework that jointly trains the model on species identification and various metadata prediction tasks. They conducted an empirical study to evaluate the effects of nine diverse metadata sources on 17 bioacoustic datasets, addressing challenges such as missing metadata and the complexity of multi-objective optimization.
Results
The results indicate that learning from metadata improves species identification performance across multiple datasets and tasks. The empirical analysis reveals the varying importance of different metadata modalities and the impact of different loss formulations on training outcomes.
Implications
The findings suggest that incorporating metadata into training can lead to more robust bioacoustic models, enhancing their applicability in real-world passive acoustic monitoring scenarios. This approach may also inform future research in other domains where auxiliary information can improve model performance.
The Spectrum Is Not Enough: When Context Helps Time-Series Forecasting
Time Series
- Distinction between series-level predictability and configuration-level context value.
- Introduction of the 'coverage deficit' as a diagnostic tool for forecasting.
- Demonstration that spectral indices do not capture the benefits of context due to phase randomization.
- Validation of findings across multiple benchmarks, showing varying context value.
Read more
The Spectrum Is Not Enough: When Context Helps Time-Series Forecasting
Summary
This paper addresses the limitations of existing predictability measures in time-series forecasting, particularly those based on the power spectrum. The authors argue that while these measures can indicate how predictable a series is, they fail to capture the value of context, such as longer lookbacks or retrieval mechanisms, which can significantly influence forecasting performance. They introduce the concept of 'coverage deficit,' a diagnostic tool that separates series-level predictability from configuration-level context value. By employing phase-randomized surrogate series, the authors demonstrate that traditional spectral indices are invariant under phase randomization, thus failing to account for the additional structure that context can provide. The study shows that the benefits of context are not inherent to the series but depend on the specific configuration used in forecasting. The findings are validated across seven benchmarks, revealing that while spectral indices remain constant, the value of context can vary dramatically, highlighting the necessity of distinguishing between predictability and context value in practical applications.
Methodology
The authors utilize phase-randomized surrogate series to analyze the limitations of spectral predictability measures. They introduce the coverage deficit diagnostic, which quantifies beyond-spectrum structure and assesses the value of context in forecasting configurations. The methodology involves controlled comparisons across seven standard benchmarks to evaluate the performance of various forecasting mechanisms.
Results
The study finds that the value of context varies significantly across different configurations, with retrieval mechanisms showing a collapse in value when phase-randomized. In contrast, longer linear windows maintain their predictive value. The coverage deficit effectively predicts the sign of beyond-spectrum value, demonstrating its utility in forecasting decisions.
Implications
The findings suggest that practitioners should be cautious when interpreting spectral predictability scores, as they may not reflect the true benefits of context in forecasting. The coverage deficit can serve as a valuable tool for guiding deployment decisions in time-series forecasting, potentially leading to improved model performance in practical applications.
Mono-Z Dark Matter Search with Neural Spline Flows Using CMS Run 2015D Open Data
Generative Models
Theory
- First application of Neural Spline Flows in a mono-Z dark matter search.
- Utilizes CMS Run 2015D open data with a focus on leptonically decaying Z bosons.
- Defines a signal region based on kinematic observables and employs a control region for background modeling.
- Sets upper limits on signal strength parameters for scalar, vector, and axial-vector mediators.
Read more
Mono-Z Dark Matter Search with Neural Spline Flows Using CMS Run 2015D Open Data
Summary
This paper presents a novel approach to search for dark matter (DM) produced in association with a leptonically decaying Z boson using CMS Run 2015D open data. The analysis focuses on the mono-Z → ℓ+ℓ− final state, specifically in the channels Z → µ+µ− and Z → e+e−, with an integrated luminosity of 2.32 fb−1. The authors employ Neural Spline Flows (NSFs) to model the background and signal densities, allowing for a likelihood-ratio scoring method that enhances sensitivity to potential DM signals. The study defines a signal region based on specific kinematic criteria and utilizes a control region for background modeling. The results indicate that while fitted signal strengths are nonzero, they are primarily influenced by background modeling discrepancies rather than evidence for DM. The paper sets upper limits on the signal-strength parameter for various mediator hypotheses, marking the first application of NSF in this context, which could pave the way for more sophisticated analyses in dark matter searches.
Methodology
The authors extract forty kinematic observables from CMS data, which are cleaned and reduced to a 37-dimensional feature vector. They train five Neural Spline Flows: two for the Standard Model background and three for different dark matter mediator hypotheses. The likelihood-ratio score is computed for each event, comparing the DM hypothesis against the SM background model, allowing for a comprehensive analysis of the kinematic phase space.
Results
The analysis sets observed upper limits on the signal-strength parameter for the scalar mediator (µ < 0.0177), vector mediator (µ < 0.0362), and axial-vector mediator (µ < 0.0498). The observed limits are significantly higher than expected limits, attributed to background modeling issues rather than a genuine DM signal.
Implications
This work highlights the potential of using advanced machine learning techniques, such as Neural Spline Flows, in high-energy physics for dark matter searches. The findings suggest that while current models do not indicate a DM signal, the methodology could enhance future searches and improve background modeling in collider experiments.
MxGPS: Multiplex Graph Transformers for a Power Grid Foundation Model
Graph Learning
- Introduces MxGPS to combat topology overfitting in GNNs for power grids.
- Utilizes a multiplex architecture with shared node encoding and task-specific branches.
- Achieves 0% boundary violation rate on unseen topologies.
- Demonstrates significantly reduced degradation under topology shifts compared to traditional models.
Read more
MxGPS: Multiplex Graph Transformers for a Power Grid Foundation Model
Summary
The paper introduces MxGPS, a multiplex graph transformer designed to address the issue of topology overfitting in power grid applications of graph neural networks (GNNs). Traditional single-task GNN models tend to perform well on training topologies but fail to generalize to unseen topologies due to their reliance on topology-specific relational structures. MxGPS mitigates this by employing a shared node encoder across multiple task-specific branches, allowing for joint training on Static State Estimation (SSE) and AC Power Flow (PF) tasks. This architecture incorporates a self-supervised pre-training phase and a multi-task fine-tuning protocol, along with a cross-branch attention mechanism that facilitates mutual regularization among tasks. The results demonstrate that MxGPS achieves a 0% boundary violation rate on four unseen topologies while exhibiting significantly lower degradation under topology shifts compared to models with lower in-distribution error. With only 1.6 million parameters, MxGPS proves to be a parameter-efficient solution for enhancing topology-agnostic generalization in power grid foundation models.
Methodology
MxGPS employs a multiplex graph transformer architecture that runs multiple task-specialized branches over a shared node encoder. It utilizes self-supervised pre-training and multi-task fine-tuning to learn representations that are robust to topology changes. A cross-branch attention mechanism is included to enhance mutual regularization between tasks.
Results
Under a 3-fold sliding-window cross-validation across four unseen topologies, MxGPS achieved a 0% boundary violation rate on all tested zero-shot Power Flow topologies. In contrast, traditional models with lower in-distribution PF error showed degradation rates between 190% and 1400% under topology shifts, while MxGPS only degraded by 39%. This indicates the effectiveness of the proposed architecture in preventing topology overfitting.
Implications
The findings suggest that MxGPS can serve as a robust foundation model for power grid applications, enabling better generalization across varying grid topologies. This has potential implications for improving the reliability and efficiency of power grid operations, particularly in the context of increasing renewable energy integration and dynamic operational conditions.
A VAE-Driven Multi-Task Satellite-Aided Semantic Communication Framework for 6G-Enabled Connected Autonomous Vehicles
Computer Vision
Generative Models
Robotics
- Introduction of a VAE-based framework for semantic communication in CAVs.
- Utilization of probabilistic latent representations to enhance robustness in noisy satellite channels.
- Joint optimization of traffic sign reconstruction and classification tasks.
- Significant bandwidth reduction achieved while maintaining performance.
Read more
A VAE-Driven Multi-Task Satellite-Aided Semantic Communication Framework for 6G-Enabled Connected Autonomous Vehicles
Summary
This paper presents a novel framework for semantic communication in connected autonomous vehicles (CAVs) utilizing a Variational Autoencoder (VAE) to enhance communication efficiency and robustness in satellite-assisted networks. The authors address the limitations of traditional communication systems that transmit raw data, which is inefficient in resource-constrained environments such as satellite channels. The proposed VAE-based framework focuses on transmitting task-relevant semantic information rather than full signal representations, thereby optimizing bandwidth usage. The framework employs a probabilistic latent representation to improve robustness against noise and incorporates a composite perceptual loss that combines Mean Squared Error (MSE), Structural Similarity Index (SSIM), and image-gradient terms for better reconstruction quality. The system is designed to jointly optimize two tasks: traffic sign reconstruction and classification. Experimental results demonstrate that the proposed approach achieves significant bandwidth reductions of up to 98.17% while maintaining stable performance across varying signal-to-noise ratios, showcasing its effectiveness in real-world applications for autonomous driving.
Methodology
The methodology involves a VAE-based architecture that encodes task-relevant features for satellite communication. It employs a two-branch decoder with a residual refinement sub-network to recover fine details lost during transmission. The framework is trained end-to-end, optimizing both reconstruction and classification tasks using a composite perceptual loss function.
Results
The proposed framework demonstrated a bandwidth reduction of 87.23% to 98.17% while maintaining stable performance across different signal-to-noise ratio conditions. The use of a composite perceptual loss resulted in higher quality reconstructions compared to conventional methods, particularly in preserving the details necessary for accurate traffic sign classification.
Implications
The findings suggest that the proposed semantic communication framework can significantly enhance the efficiency of data transmission in connected autonomous vehicles, particularly in bandwidth-constrained environments like satellite networks. This has implications for the development of safer and more reliable autonomous driving systems, especially in rural or disaster-affected areas where satellite communication is critical.
AI-Augmented Adaptive Digital Twin Modeling for Brain Tumor Evolution Prediction and Treatment Scheduling
Optimization
Theory
Interpretability
- Introduces an AI-augmented digital twin framework for brain tumor modeling.
- Combines reaction-diffusion modeling with residual learning for improved accuracy.
- Demonstrates significant reductions in prediction errors and improvements in tumor burden estimates.
- Shows effectiveness of model predictive control in optimizing treatment scheduling.
Read more
AI-Augmented Adaptive Digital Twin Modeling for Brain Tumor Evolution Prediction and Treatment Scheduling
Summary
This paper presents an AI-augmented adaptive digital twin (DT) framework aimed at predicting brain tumor evolution and optimizing treatment scheduling. The framework integrates an interpretable reaction-diffusion (RD) tumor model with a 3D residual learning module to correct model discrepancies and update the DT during recursive rollouts. The study emphasizes the importance of patient-specific dynamics in treatment decisions, as traditional clinical imaging provides limited and irregular data. By employing a hybrid RD-residual modeling approach, the authors demonstrate significant improvements in tumor burden predictions, achieving an 84.3% reduction in masked voxel-wise mean squared error and a 43.5% increase in Dice overlap compared to the baseline RD model. Furthermore, the DT updating process during recursive deployment led to a further 45.9% reduction in mean squared error and a 9.6% increase in Dice overlap. In model predictive control (MPC) simulations, the updated DT controller effectively reduced final tumor burden by 22.4% compared to a fixed-schedule comparator. The findings highlight a comprehensive workflow that connects patient-specific initialization, mechanistic modeling, learned corrections, adaptive updates, and treatment scheduling, paving the way for future clinical validations.
Methodology
The methodology involves a hybrid approach that combines a reaction-diffusion model for tumor growth with a 3D residual learning module to correct for discrepancies in tumor evolution predictions. The digital twin is updated recursively using new patient data, and model predictive control is applied to evaluate treatment strategies.
Results
The hybrid RD-residual model achieved an 84.3% reduction in mean squared error and a 43.5% increase in Dice overlap compared to the baseline RD model. Recursive updates further reduced mean squared error by 45.9% and increased Dice overlap by 9.6%. In treatment scheduling simulations, the updated DT controller reduced final tumor burden by 22.4% compared to a fixed-schedule approach.
Implications
The proposed framework has the potential to enhance personalized treatment strategies for brain tumor patients by providing more accurate predictions of tumor evolution and optimizing treatment schedules based on patient-specific data. This could lead to improved patient outcomes and more effective management of aggressive brain tumors.
From Preimage Search To Source-Grounded Feature Inversion
Interpretability
- Introduces source-grounded feature inversion for improved neural network interpretability.
- Conditions feature inversion on local network geometry, ensuring sample-specific results.
- Utilizes closed-form matrix Wiener maps for correcting backpropagation signals.
- Demonstrates effectiveness across diverse neural network architectures.
Read more
From Preimage Search To Source-Grounded Feature Inversion
Summary
This paper introduces the concept of source-grounded feature inversion, which aims to enhance the interpretability of neural networks by providing a clearer understanding of what internal features extract from specific inputs. Traditional feature inversion methods often rely on iterative preimage searches that can yield multiple valid inputs, making it difficult to ascertain the unique inverse associated with a particular feature. The proposed method conditions the inversion on the local network geometry at the input that generated the feature, thereby ensuring that the inversion is sample-specific. The approach utilizes backpropagation to trace the dependencies of the network while employing a closed-form matrix Wiener map to correct the adjoint signal transported during backpropagation. This method allows for the estimation of upstream states along the computational Directed Acyclic Graph (DAG) of the network, leading to a more accurate representation of the internal features. The results demonstrate that the source-grounded feature inversion can be applied across various architectures and visual distributions, revealing architecture-specific structures without requiring query-specific optimization. This advancement opens new avenues for inspecting the hidden feature hierarchy of neural networks, linking the extracted features to the internal evidence influencing model decisions.
Methodology
The methodology involves formulating feature inversion as a source-grounded problem, where the inversion is conditioned on the local geometry of the network at the input that generated the feature. The approach employs backpropagation to trace dependencies and utilizes matrix Wiener maps to correct the adjoint signals, allowing for the estimation of upstream states in a single reverse pass through the computational DAG.
Results
The results show that the proposed source-grounded feature inversion method successfully reveals architecture-specific spatial and frequency structures across various neural network architectures, including CNNs and Transformers. The inverses produced do not collapse to a generic image template, indicating that the method captures the unique characteristics of the features being analyzed.
Implications
The implications of this work extend to enhancing the interpretability of neural networks, allowing researchers and practitioners to better understand the relationship between input data and model decisions. This could lead to improved model transparency and trustworthiness in applications where interpretability is crucial.
Institutional Equity Holdings Prediction Using Node Affinities of Dynamic Graphs
Graph Learning
Time Series
- Introduces the first benchmark for institutional equity holdings prediction using temporal graph machine learning.
- Frames holdings prediction as a node affinity prediction task on a bipartite graph of managers and securities.
- Achieves state-of-the-art performance with the NAVIS model, significantly outperforming competitors.
- Demonstrates that temporal and structural signals in the ownership graph capture most predictive information.
Read more
Institutional Equity Holdings Prediction Using Node Affinities of Dynamic Graphs
Summary
This paper addresses the challenge of predicting institutional equity holdings based on SEC Form 13F filings, which provide a temporal record of portfolio decisions by large investment managers. The authors introduce a novel approach that frames holdings prediction as a node affinity prediction task within a discrete-time temporal bipartite graph, representing managers and securities. They utilize a dataset of 99 managers and the S&P 500 index, spanning 48 quarters from 2013 to 2025, to benchmark their model, named NAVIS (Node Affinity prediction model using Virtual State). The model achieves a state-of-the-art Normalized Discounted Cumulative Gain (NDCG) of 0.9127, significantly outperforming existing dynamic graph representation learning methods and heuristic approaches. The study highlights the importance of temporal and structural signals in the 13F ownership graph, suggesting that these factors capture most of the predictable information, with domain-specific features providing only marginal improvements. By establishing a reproducible foundation for temporal graph machine learning in holdings prediction, this work opens avenues for further research in institutional demand modeling and portfolio allocation.
Methodology
The authors constructed a discrete-time temporal heterogeneous directed, weighted bipartite graph from cleaned quarterly Form 13F filings. They formulated the prediction task as dynamic node property prediction, specifically focusing on forecasting the expected market value of holdings for each institutional manager over the next quarter. The NAVIS model was benchmarked against various Temporal Graph Benchmark models.
Results
The NAVIS model achieved a test NDCG of 0.9127, outperforming all dynamic graph representation learning competitors and heuristic methods. A simple Exponential Moving Average baseline achieved an NDCG of 0.8882, indicating the persistence of institutional portfolio behavior. Domain-specific features contributed marginally to performance improvements, suggesting that the temporal and structural signals in the data are already highly informative.
Implications
The findings suggest that accurately predicting institutional equity holdings can enhance understanding of market dynamics and institutional behavior, potentially leading to improved asset pricing models and regulatory monitoring of ownership concentration. The methodology provides a foundation for further exploration of institutional demand modeling and portfolio allocation strategies.
Sparse Autoencoders for Interpretable Out-of-Distribution Detection
Computer Vision
Interpretability
- SAID improves OOD detection performance by utilizing sparse autoencoders on intermediate layer activations.
- Intermediate layers contain valuable discriminative information often lost in final-layer representations.
- The method provides interpretable insights into the activation of learned concepts, aiding in understanding distribution shifts.
Read more
Sparse Autoencoders for Interpretable Out-of-Distribution Detection
Summary
This paper addresses the critical challenge of detecting out-of-distribution (OOD) samples in machine learning, particularly for deep neural networks that often yield overconfident predictions when faced with inputs outside their training distribution. The authors propose a novel method called Sparse Autoencoders for Interpretable Detection (SAID), which utilizes sparse autoencoders (SAEs) to extract interpretable features from intermediate activations of neural networks. Unlike traditional OOD detection methods that rely heavily on final-layer outputs, SAID leverages the rich hierarchical information present in earlier layers. The method computes an OOD score based on the cosine similarity between the sparse feature activations of a test sample and the mean activations of in-distribution classes. The authors demonstrate that their approach not only achieves state-of-the-art performance on standard OOD detection benchmarks but also provides valuable insights into how distribution shifts affect learned representations. By analyzing the sparse features, the authors show that in-distribution samples activate more semantically consistent concepts compared to OOD samples, highlighting the potential of SAID for both improved detection and interpretability.
Methodology
The authors trained sparse autoencoders on selected intermediate layers of a frozen image classifier, including both convolutional and Transformer-based models. At test time, the SAE encoding of an input is compared to the mean SAE encoding of the predicted in-distribution class to produce an OOD score based on cosine similarity.
Results
The proposed SAID method achieved state-of-the-art performance on various OOD detection benchmarks, demonstrating improved detection capabilities across both semantic and covariate distribution shifts. Additionally, the analysis of SAE-derived concept labels revealed that in-distribution samples exhibited higher semantic consistency compared to OOD samples.
Implications
The findings suggest that leveraging intermediate layer representations can significantly enhance OOD detection methods, making them more reliable for real-world applications. Furthermore, the interpretability aspect of SAID can aid practitioners in diagnosing model behavior and understanding the nature of distribution shifts.
Reassessing Muon for Matrix Factorization
Optimization
Theory
- Muon optimizer's advantages are context-dependent and sensitive to hyperparameter tuning.
- In low-rank matrix factorization, Muon does not consistently outperform AdamW.
- Muon retains an advantage in nonnegative matrix factorization due to its orthogonalized updates.
- The study highlights the importance of controlled benchmarks for evaluating optimizer performance.
Read more
Reassessing Muon for Matrix Factorization
Summary
This paper investigates the performance of the Muon optimizer, which has shown promise in large-scale deep learning, particularly in training large language models. The authors aim to isolate Muon's advantages by applying it to a simpler, well-defined problem: low-rank matrix factorization. They conduct a systematic comparison of Muon against adaptive optimizers like AdamW, focusing on how hyperparameter choices affect performance. The findings reveal that Muon does not consistently outperform AdamW in low-rank matrix factorization tasks, and its previously reported advantages are sensitive to specific problem structures and tuning. Notably, while Muon shows some benefits in nonnegative matrix factorization (NMF), its overall effectiveness is context-dependent, suggesting that its success in deep learning may stem from the complexities of modern architectures rather than intrinsic properties of the optimizer itself. The authors advocate for evaluating optimizers on controlled problems to better understand their strengths and limitations.
Methodology
The authors conducted a controlled evaluation of the Muon optimizer on low-rank matrix factorization tasks, comparing its performance against adaptive optimizers like AdamW. They systematically tuned hyperparameters to isolate the effects of the optimizer from confounding factors related to architecture and data.
Results
The results indicate that Muon does not consistently outperform AdamW in low-rank matrix factorization settings, with performance gains being sensitive to hyperparameter choices. However, Muon does show a consistent advantage in nonnegative matrix factorization tasks, suggesting that its effectiveness may depend on specific problem structures.
Implications
The findings imply that while Muon may be effective in certain contexts, its advantages are not universal. This underscores the need for careful evaluation of optimizers in controlled settings, which could lead to better understanding and selection of optimization strategies in machine learning.
VAIOM: Continuous-Input, Discrete-Output Decoder-Only Financial Sequence Modeling
Time Series
- VAIOM separates input representation from output likelihood, allowing for continuous financial data processing.
- The model outperforms traditional statistical and tree-based baselines in predicting financial returns.
- Full-sequence autoregressive supervision enhances model performance compared to last-position training.
- The integration of auxiliary objectives and a mixture-structured return head improves return likelihood.
Read more
VAIOM: Continuous-Input, Discrete-Output Decoder-Only Financial Sequence Modeling
Summary
The paper introduces VAIOM (Vector-Input Autoregressive Inference for Ordinal-Return Modeling), a novel decoder-only Transformer model designed for probabilistic next-return modeling in financial sequences, specifically targeting one-hour foreign-exchange (FX) bars. Unlike traditional models that rely on discrete symbolic inputs, VAIOM utilizes continuous multivariate financial-event vectors to maintain the numerical structure of financial data while predicting a categorical distribution over the next volatility-normalized return bucket. The model employs a Hybrid Continuous Input (HybridContIn) approach, integrating continuous event features with categorical asset metadata and a Mixture-of-Market-States (MoMS) return head. It leverages auxiliary objectives and full-sequence supervision to enhance performance. The empirical evaluation demonstrates that VAIOM significantly outperforms various baselines, including Frequency, Markov, and LightGBM models, across multiple test periods, achieving notable gains in predictive accuracy. The findings highlight the effectiveness of continuous input representation and the importance of autoregressive supervision in financial sequence modeling.
Methodology
VAIOM employs a decoder-only Transformer architecture that processes continuous multivariate financial-event vectors as input while predicting a categorical distribution over the next return bucket. The model is trained using cross-entropy loss and incorporates auxiliary objectives for improved performance. It is evaluated on historical FX market data, with model selection based on validation performance and final testing on held-out data.
Results
The selected VAIOM model outperformed train-fitted baselines, achieving approximately 0.029–0.043 bits per event improvement over LightGBM across three independent training seeds. Validation experiments confirmed that continuous input representation and full-sequence supervision significantly enhance return likelihood, while the smallest evaluated architecture achieved the best validation performance.
Implications
The findings suggest that VAIOM can effectively model complex financial sequences, providing a robust framework for predicting market returns. This approach may have applications in algorithmic trading, risk management, and financial forecasting, where accurate return predictions are critical.
NodeImport: Imbalanced Node Classification with Node Importance Assessment
Graph Learning
- Introduces NodeImport, a framework for class-imbalanced node classification.
- Utilizes a balanced meta-set for dynamic assessment of node importance.
- Separates synthetic node generation from filtering, enhancing flexibility.
- Demonstrates superior performance over state-of-the-art baselines on benchmark datasets.
Read more
NodeImport: Imbalanced Node Classification with Node Importance Assessment
Summary
The paper addresses the challenge of class imbalance in node classification tasks within graph data, where traditional Graph Neural Networks (GNNs) often overfit to majority classes, leading to biased performance. The authors propose a novel framework called NodeImport, which utilizes a balanced meta-set for assessing node importance. This approach identifies significant nodes that enhance model performance in an unbiased context, allowing for dynamic node selection during training. The framework separates the synthetic node generation process from the filtering process, ensuring compatibility with various node generation techniques. The authors derive a formula for node importance assessment that reduces computational overhead and provides an intuitive threshold for selection. NodeImport is evaluated across multiple benchmark datasets using popular GNN architectures, demonstrating its effectiveness in improving classification outcomes and mitigating class imbalance issues.
Methodology
The methodology involves creating a balanced meta-set to measure node importance, where nodes are considered significant if they improve model performance in an unbiased setting. The framework filters valuable labeled, unlabeled, and synthetic nodes throughout the training process, allowing for dynamic adjustments based on the model's learning progress. A theoretical formula for node importance assessment is derived to facilitate efficient computation.
Results
The results indicate that NodeImport outperforms existing state-of-the-art methods across various benchmark datasets, effectively addressing the class imbalance issue and leading to improved node classification accuracy.
Implications
The findings suggest that NodeImport can be applied in real-world scenarios where class imbalance is prevalent, such as social network analysis, fraud detection, and biological network classification. The framework's flexibility allows it to integrate with various node generation techniques, making it a versatile tool for researchers and practitioners in graph learning.
ReDiTT: Retrieval Augmented Conditional Diffusion Transformers for Asynchronous Time Series
Time Series
Generative Models
- Introduction of ReDiTT, the first retrieval-based diffusion framework for asynchronous time series prediction.
- Utilization of a token memory bank to retrieve top-k similar latent sequences for conditioning.
- Significant improvements in next-event and long-horizon prediction performance on real-world datasets.
- Demonstration of enhanced sample diversity and stability in long-horizon forecasting.
Read more
ReDiTT: Retrieval Augmented Conditional Diffusion Transformers for Asynchronous Time Series
Summary
The paper introduces ReDiTT, a novel model designed for predicting asynchronous time series by leveraging a retrieval-augmented conditional diffusion transformer. The model aims to predict both the next inter-event time and the event type, addressing the challenges posed by the inherent uncertainty and complex temporal dynamics of asynchronous data. ReDiTT operates in latent space, retrieving structurally similar sequences from a memory bank during both training and inference. This retrieval-based conditioning enhances the model's ability to attend to relevant temporal dynamics and provides global structural guidance, which stabilizes long-horizon forecasting and improves sample diversity. The authors demonstrate the effectiveness of ReDiTT through experiments on seven real-world datasets, achieving state-of-the-art performance in next-event prediction and long-horizon forecasting, thus showcasing its potential in various applications such as healthcare monitoring, finance, and user behavior modeling.
Methodology
ReDiTT employs a retrieval-augmented approach where each sequence retrieves its top-k nearest neighbors from a token memory bank. These retrieved sequences are integrated into the diffusion transformer through cross-attention modules, allowing the model to leverage structurally similar temporal dynamics for both training and inference.
Results
The experiments conducted on seven real-world datasets reveal that ReDiTT achieves state-of-the-art performance in predicting the next event and long-horizon forecasting, outperforming existing models and demonstrating its robustness in handling asynchronous time series data.
Implications
ReDiTT's framework can be applied in various domains that rely on asynchronous time series data, such as healthcare monitoring, finance, and user behavior modeling, potentially leading to more accurate predictions and better decision-making processes.
Transforming Rank: How Architecture Navigates the Spectral Pathologies of Depth
Theory
- Skip connections preserve rank by routing gradients around rank-reducing branches, creating a tradeoff between rank collapse and ensemble-like behavior.
- Normalization placement influences the branch-to-skip ratio, affecting rank preservation across depth.
- The two-matrix structure in feedforward blocks helps maintain the rank of the residual branch Jacobian.
- The effective rank at initialization can predict the training success of networks on tasks like CIFAR-10.
Read more
Transforming Rank: How Architecture Navigates the Spectral Pathologies of Depth
Summary
This paper investigates the impact of Transformer feedforward block architecture on the preservation of rank across depth during initialization. The author reinterprets skip connections and normalization layers, traditionally viewed as mechanisms for controlling magnitude, as crucial for maintaining gradient rank. The study reveals that skip connections can mitigate rank collapse while promoting ensemble-like behavior, influenced by the relative scales of the branch and the skip paths. The placement of normalization layers is shown to control the tradeoff between rank collapse and ensemble behavior, explaining the differences in rank behavior between Post-Norm and Pre-Norm configurations. Additionally, the two-matrix structure within the feedforward block is analyzed, demonstrating how it helps maintain the rank of the residual branch Jacobian. The paper establishes a logical progression of architectural choices that navigate the rank tradeoff, ultimately showing that the initialization rank of the input-output Jacobian can predict training performance on CIFAR-10.
Methodology
The author employs theoretical analysis and empirical measurements to assess the effective rank of the input-output Jacobian, residual stream representation, and residual branch Jacobian. The study examines the effects of architectural choices on rank preservation and training performance.
Results
The findings indicate that skip connections and normalization placements significantly influence the rank of gradients and representations in deep networks. The initialization rank of the input-output Jacobian is shown to be a reliable predictor of network performance on CIFAR-10, highlighting the importance of architectural design in deep learning.
Implications
This research provides insights into the architectural design of deep networks, emphasizing the importance of managing rank preservation to enhance training stability and performance. It suggests that careful consideration of skip connections and normalization placements can lead to more effective deep learning models.
Deconstructing Actor-Critic: A Large-scale Empirical Study of Design Components for Practitioners
Reinforcement Learning
- Actor-critic algorithms are widely used but often lack generality across different problems.
- Common defaults in actor-critic configurations can lead to unreliable performance.
- Bounded distributions with adaptive update schedules are more robust than traditional Gaussian distributions.
- The study provides empirical insights that can guide practitioners in making component-level decisions.
Read more
Deconstructing Actor-Critic: A Large-scale Empirical Study of Design Components for Practitioners
Summary
This paper investigates the design components of actor-critic algorithms in reinforcement learning (RL) through a large-scale empirical study. The authors conducted over 33,000 experiments using a control task derived from a real water treatment plant to analyze how various design choices impact the reliability and performance of these algorithms. The study highlights that common configurations, such as Gaussian action distributions with pathwise gradient estimators, are among the least reliable. In contrast, bounded distributions with adaptive update schedules demonstrate robustness across a wide range of settings. The findings provide empirical guidance for practitioners in scientific and engineering domains, helping them make informed decisions when adapting actor-critic methods to real-world control problems. The paper emphasizes the importance of understanding individual algorithmic components and their interactions, which can significantly affect performance and reliability in practical applications.
Methodology
The authors performed a comprehensive empirical analysis involving over 33,000 experiments on a control task based on a real-world water treatment plant. They systematically varied key design components of actor-critic algorithms, such as policy update methods, action distribution representations, gradient estimation techniques, and update frequencies relative to the value estimator, to assess their impact on algorithm performance and reliability.
Results
The results indicate that configurations commonly used in actor-critic algorithms, particularly those involving Gaussian action distributions and pathwise gradient estimators, are among the least reliable. In contrast, configurations utilizing bounded distributions and adaptive update schedules showed consistent robustness across various settings. These findings underscore the importance of careful design choices in actor-critic implementations.
Implications
The insights from this study can significantly aid practitioners in the fields of reinforcement learning and control systems by providing a clearer understanding of how to effectively adapt actor-critic algorithms to new real-world applications. This can enhance the reliability and performance of RL systems in critical domains such as autonomous vehicles, drug discovery, and process control.
HEDGEHOG: Hierarchical Evaluation of Drug Generators Through Rigorous Filtration
Generative Models
- HEDGEHOG is a six-stage benchmark for evaluating molecular generators in drug discovery.
- Only 0.65% of 230,000 generated molecules passed all evaluation stages, indicating significant limitations in current models.
- The benchmark emphasizes the importance of multi-parameter design constraints in assessing drug candidates.
- HEDGEHOG provides insights into where generative models fail, enabling targeted improvements.
Read more
HEDGEHOG: Hierarchical Evaluation of Drug Generators Through Rigorous Filtration
Summary
The paper introduces HEDGEHOG, a comprehensive six-stage benchmark designed to rigorously evaluate generative molecular models used in drug discovery. Traditional metrics for assessing molecular generators often fail to reflect the practical medicinal chemistry requirements, leading to inefficiencies and false positives. HEDGEHOG addresses this gap by implementing a structured filtration process that mimics industrial hit identification workflows. The six stages include preprocessing, physicochemical screening, structural alerts, synthesis feasibility, docking and binding affinity estimation, and 3D pose checks. The authors evaluated 23 molecular generators across three model classes, generating a total of 230,000 molecules, of which only 0.65% passed all stages. The results highlight the limitations of current molecular generators, revealing that compounds may meet isolated criteria but fail to satisfy the comprehensive demands of medicinal chemistry and synthesis. This benchmark not only provides a more realistic assessment of generative models but also identifies specific failure points, guiding future improvements in molecular generation aligned with drug discovery needs.
Methodology
The HEDGEHOG benchmark consists of six stages: preprocessing, physicochemical descriptor screening, structural alerts and graph-sanity checks, synthesis feasibility assessment, docking and binding affinity estimation, and 3D pose and interaction checks. This structured approach allows for a comprehensive evaluation of generated molecules against practical medicinal chemistry criteria.
Results
Out of 230,000 generated molecules, only 0.65% were deemed viable after passing through all six evaluation stages. The study revealed that many molecules that seemed acceptable based on isolated criteria failed to meet the combined requirements of medicinal chemistry, synthesis, and docking assessments.
Implications
HEDGEHOG serves as a critical tool for improving the evaluation of molecular generators, ensuring that generated compounds are not only chemically valid but also practically useful for drug discovery. This benchmark can guide the development of more effective generative models that align with the complexities of real-world drug development processes.
CDS: Counterfactual Directionality Score for Structured Interventions in Spatial Graphs
Graph Learning
- Introduction of a framework for structured counterfactual interventions in spatial graphs.
- Development of the Counterfactual Directionality Score (CDS) to measure directional influence.
- Theoretical interpretation of CDS as a finite-difference measure of local intervention sensitivity.
- Implementation of a core-level bootstrap procedure for valid uncertainty estimation.
Read more
CDS: Counterfactual Directionality Score for Structured Interventions in Spatial Graphs
Summary
This paper addresses the challenge of quantifying directional influence between node populations in graph-based models, particularly in spatial biological systems. The authors introduce a novel framework for structured counterfactual interventions that allows for the estimation of directional influence between different cell types. The core of their approach is the Neighbor Influence Model (NIM), which predicts a node's state based on its local neighborhood. By applying constrained interventions that modify the neighborhood composition while maintaining essential spatial and structural properties, the authors define the Counterfactual Directionality Score (CDS). This score quantifies the change in predicted node state resulting from targeted perturbations and is interpreted as a finite-difference measure of local intervention sensitivity. To ensure valid uncertainty estimates, a bootstrap procedure is developed to account for dependencies within spatial samples. Experimental results demonstrate that CDS effectively recovers known directional structures in synthetic spatial graphs, remains well-calibrated under null conditions, and shows robustness against confounding signals. Preliminary applications to spatial transcriptomics data indicate that CDS identifies biologically plausible interactions across tissue cores, suggesting its potential utility in understanding complex biological processes.
Methodology
The methodology involves training a Neighbor Influence Model (NIM) to predict node states based on local neighborhoods. The authors apply structured counterfactual interventions that modify neighborhood composition while preserving spatial and structural properties. The CDS is calculated based on the sensitivity of predicted node states to these interventions, and a bootstrap procedure is employed to derive uncertainty estimates.
Results
The results indicate that CDS successfully recovers the ground truth directional structure in synthetic spatial graphs, maintains calibration under null conditions, and is robust to confounding signals. Preliminary results on spatial transcriptomics data reveal consistent and biologically plausible sender-receiver relationships across tissue cores.
Implications
The proposed framework and CDS can significantly enhance the understanding of cell-cell interactions in biological systems, potentially aiding in the study of tumor-immune interactions, signaling pathways, and other complex biological processes. This approach may also inform therapeutic strategies by identifying key cellular influences.
Relevance-Aware Rule: Structural Deletion of Irrelevant Conditions in Decision Trees
Interpretability
Theory
Efficient ML
- Establishes the structural mechanics of IRCs in decision trees.
- Introduces a relevance-aware rule framework for diagnosing and deleting IRCs.
- Utilizes a three-layer analytical approach to assess the relevance of conditions.
- Achieves substantial simplification of decision tree rules while maintaining reliability.
Read more
Relevance-Aware Rule: Structural Deletion of Irrelevant Conditions in Decision Trees
Summary
This paper addresses the issue of irrelevant conditions (IRCs) in decision trees, which arise from the structural mechanisms of tree splitting and persist even in modern sparse tree induction algorithms. The authors propose a novel structural IRC deletion framework that is grounded in theoretical foundations, including the establishment of theorems and propositions related to the IRC mechanism. The key insight is that a binary split alters class proportions in opposite directions, leading to the identification of class-1-increasing (C1) and class-0-increasing (C0) links. The framework diagnoses the relevance of these links relative to specific leaf nodes, allowing for selective deletion of structurally and empirically irrelevant conditions while preserving those essential for maintaining prediction reliability. The methodology involves three analytical layers: local annotation, leaf-relative diagnosis, and path-level effect assessment. Experimental results demonstrate that the proposed framework achieves significant rule simplification without compromising the reliability of the decision tree, thus enhancing interpretability.
Methodology
The authors develop a structural IRC deletion framework that identifies IRCs based on the structural relationships of class proportions in binary splits. The framework employs a three-layer analytical approach: local annotation of links, leaf-relative diagnosis of relevance, and path-level effect assessment to determine the impact of deleting conditions on rule reliability. Two procedures are introduced for candidate generation and acceptance criteria, allowing for both broader and more conservative simplification strategies.
Results
The experimental results indicate that the proposed framework effectively simplifies decision tree rules by removing IRCs without sacrificing the reliability of the original tree. The framework demonstrates significant improvements in interpretability and maintains predictive performance, achieving scalability that is linear with respect to the number of leaf rules.
Implications
The findings of this study have potential applications in enhancing the interpretability of decision trees in various domains, such as healthcare, finance, and any field where decision trees are employed for classification tasks. The relevance-aware rule framework can be integrated into existing decision tree algorithms to improve their clarity and usability.
Mitigating The Effect of Class Imbalance in Data with Hierarchical and Dependable Structure
NLP
- Introduces a Hierarchy-Aware RoBERTa framework for CWE classification.
- Critiques the effectiveness of traditional oversampling techniques in hierarchical contexts.
- Achieves a weighted F1-score of 0.76 without data augmentation.
- Demonstrates significant improvements in minority class performance.
Read more
Mitigating The Effect of Class Imbalance in Data with Hierarchical and Dependable Structure
Summary
This paper addresses the challenge of classifying cybersecurity vulnerabilities using the Common Weakness Enumeration (CWE) taxonomy, which suffers from extreme class imbalance and strong hierarchical dependencies among categories. The authors critique traditional oversampling techniques like SMOTE and ADASYN, noting their limited effectiveness in preserving the hierarchical structure of CWE during classification. To overcome these challenges, they propose a Hierarchy-Aware RoBERTa framework that incorporates learnable parent-class embeddings, ensuring taxonomic consistency in the classification process. Through empirical evaluations on a CWE Research Concept dataset, the proposed model demonstrates superior performance, achieving a weighted F1-score of 0.76 without data augmentation. This approach significantly improves the F1-score for minority classes, particularly the Class category, which sees an increase from 0.49 to 0.60 compared to the BERT baseline. The findings suggest that maintaining hierarchical relationships in representation learning is a more effective strategy than traditional oversampling methods for structured vulnerability classification.
Methodology
The authors empirically evaluate various machine learning models, including classical (Random Forest, SVM), deep learning (CNN, BiGRU), and transformer-based (BERT) models under the influence of SMOTE and ADASYN resampling techniques. They then introduce their Hierarchy-Aware RoBERTa model, which integrates CWE structural information through learnable parent-class embeddings to enhance classification performance while respecting the inherent hierarchical relationships.
Results
The proposed Hierarchy-Aware RoBERTa model outperformed all baseline models, achieving a weighted F1-score of 0.76 on the CWE Research Concept dataset. Notably, the F1-score for the Class category improved from 0.49 to 0.60 over the BERT baseline, indicating enhanced performance on minority classes without the need for data augmentation.
Implications
The findings suggest that incorporating hierarchical structures into machine learning models can lead to more effective classification of imbalanced datasets, particularly in cybersecurity contexts. This approach may be applicable to other domains where hierarchical relationships exist among classes.
Conditional Invertible Neural Networks for Data-Driven UAV Control: A 2-D Proof of Concept
Robotics
- Introduction of a probabilistic inverse-dynamics model for UAV control using cINNs.
- Demonstration of effective uncertainty estimation in motor commands.
- High performance in open-loop and closed-loop evaluations compared to INDI.
- Identification of command bandwidth and data coverage as critical factors for control failures.
Read more
Conditional Invertible Neural Networks for Data-Driven UAV Control: A 2-D Proof of Concept
Summary
This paper explores the use of Conditional Invertible Neural Networks (cINNs) as probabilistic models for controlling multirotors, specifically a planar X8 coaxial multicopter. The authors propose a novel approach to model the conditional distribution of motor commands given the current state and tracking command, thereby addressing the limitations of traditional deterministic control methods. By employing rational-quadratic spline coupling and invertible linear mixing, the cINN architecture allows for uncertainty estimation in motor commands, which is crucial for safety-critical applications. The study demonstrates that the cINN can effectively reproduce the control outputs of an Incremental Nonlinear Dynamic Inversion (INDI) teacher, achieving a high coefficient of determination (R² = 0.944) and matching the INDI baseline in terms of position RMSE across various closed-loop scenarios. The findings highlight the importance of command bandwidth and data coverage as key factors influencing control performance, particularly under aggressive maneuvers and high-frequency references.
Methodology
The authors utilize Conditional Invertible Neural Networks (cINNs) to model the conditional distribution of motor commands based on the current state and tracking command. The architecture incorporates rational-quadratic spline coupling, activation normalization, and learned invertible linear mixing, with sine-cosine encoding for attitude angles. The model is trained using data generated from an INDI teacher, and performance is evaluated through both open-loop and closed-loop simulations on a planar X8 multicopter.
Results
The trained cINN achieved an R² value of 0.944 in open-loop reproduction of motor commands and matched the INDI baseline with a position RMSE of 9.7 m over 15 closed-loop scenarios. The study identified two main failure modes: attitude divergence during aggressive commands and phase lag under high-frequency references, emphasizing the impact of command bandwidth and training data coverage on control performance.
Implications
The findings suggest that cINNs can enhance the adaptability and safety of UAV control systems by providing uncertainty estimates alongside control commands. This approach could be beneficial in safety-critical applications where understanding the confidence of predictions is essential. The methodology may also be applicable to other robotic systems requiring robust control under uncertainty.
Leveraging unlabelled data for generalizable neural population decoding
Multimodal
Interpretability
Time Series
- Introduction of MOJO, a joint SSL-SL framework for neural decoding.
- Demonstrated superior performance over traditional SL models, especially with limited labeled data.
- Improved interpretability of neuronal representations.
- Generalization of MOJO to human ECoG data, achieving competitive results.
Read more
Leveraging unlabelled data for generalizable neural population decoding
Summary
This paper introduces MOJO (Masked autOencoder-based JOint training), a novel training framework designed to enhance neural population decoding by leveraging both self-supervised learning (SSL) and supervised learning (SL) on spike-tokenizing models. Traditional approaches to neural decoding have been limited by their reliance on supervised learning, which requires paired behavioral labels and restricts the use of unlabelled data. MOJO addresses this limitation by integrating masked autoencoding with SL objectives, allowing for the effective use of unlabelled data across various neural datasets. The authors evaluated MOJO on three distinct spiking datasets involving monkey motor cortex and multi-regional mouse recordings, demonstrating that it outperforms models trained solely with SL, particularly in scenarios with limited labeled data. The framework also yields more interpretable neuronal representations and generalizes well to human electrocorticography during speech tasks, achieving performance comparable to specialized neuro-foundation models. Overall, MOJO represents a significant advancement in the flexibility and scalability of neural decoding methodologies, paving the way for broader applications in neurotechnologies.
Methodology
The MOJO framework combines self-supervised learning via masked autoencoding with supervised learning objectives. It tokenizes neural data at the spike level and optimizes both SSL and SL objectives simultaneously, allowing for effective representation learning from both unlabelled and labelled data. The framework was evaluated using various neural datasets, including spiking data from monkeys and mice, as well as human electrocorticography data.
Results
MOJO consistently outperformed purely supervised models across multiple datasets and tasks, particularly excelling in few-shot finetuning scenarios. The integration of SSL led to more interpretable neuronal representations and improved performance in brain region classification and spike-statistics prediction. Additionally, MOJO demonstrated effective generalization to human speech decoding tasks, achieving results comparable to specialized neuro-foundation models.
Implications
The findings suggest that MOJO can significantly enhance the training of neural decoders by utilizing unlabelled data, which is abundant in neuroscience. This approach could lead to more robust brain-computer interfaces and other neurotechnologies, enabling better performance in diverse applications while reducing the reliance on labeled datasets.
Energy-Based Physics-Informed Form Finding for Clustered Tensegrity Structures
Optimization
Robotics
Theory
- Introduces an energy-based learning framework for tensegrity structures.
- Incorporates clustering to reduce the complexity of member forces.
- Enhances physical consistency and robustness through energy-based loss functions.
- Demonstrates scalability and accuracy in form finding via numerical experiments.
Read more
Energy-Based Physics-Informed Form Finding for Clustered Tensegrity Structures
Summary
This paper addresses the challenges of tensegrity form-finding and physical property prediction, which are crucial inverse problems in structural mechanics. Traditional methods struggle with nonlinearity, structural stability, and robustness to noise. The authors propose an innovative energy-based learning framework that integrates total potential energy minimization and constitutive relations into the training process. This approach allows for the simultaneous prediction of equilibrium nodal configurations and internal force distributions while enhancing physical consistency and data efficiency. The framework employs a clustering technique to reduce the number of independent member forces, leveraging similarities among structural members. By embedding energy-based loss functions directly into the learning process, the proposed method demonstrates improved robustness and computational efficiency. Numerical experiments on various tensegrity structures, including prism and lander systems, validate the effectiveness of the framework, showcasing its potential for scalable form finding and accurate structural property prediction.
Methodology
The proposed methodology involves an energy-based physics-informed learning framework that minimizes total potential energy and incorporates constitutive relations. It utilizes a clustering approach to reduce the number of independent member forces, allowing for a more efficient mapping from member rest-length variations to equilibrium configurations. The learning process integrates physics through energy-based loss functions, combined with data-driven loss terms for physical quantities.
Results
Numerical experiments conducted on tensegrity structures, including prism and lander systems, demonstrate the framework's capability for scalable form finding and accurate prediction of structural properties. The results indicate significant improvements in robustness and computational efficiency compared to traditional methods.
Implications
The proposed framework has potential applications in aerospace engineering, robotics, and architectural design, where lightweight and deployable structures are essential. It offers a robust alternative to conventional numerical solvers for tensegrity analysis, paving the way for more efficient design and analysis of complex structural systems.
ExTernD: Expanded-Rank Ternary Decomposition Ternary LLM PTQ with Accuracy Approaching Any Quantization Level
Large Language Models
Efficient ML
Optimization
- ExTernD achieves accuracy close to bf16 precision through expanded-rank ternary decomposition.
- The method allows for continuous scaling of accuracy and resource usage, unlike fixed-plane quantization methods.
- A greedy ALS algorithm is proposed for efficient factorization, with a GPU-optimized batched implementation.
- Empirical results show competitive performance on benchmark datasets, with accuracy surpassing traditional methods.
Read more
ExTernD: Expanded-Rank Ternary Decomposition Ternary LLM PTQ with Accuracy Approaching Any Quantization Level
Summary
The paper introduces ExTernD (Expanded-rank Ternary Decomposition), a novel approach for post-training quantization (PTQ) of large language models (LLMs) that allows for a flexible and accurate representation of weight matrices. The method factors each weight matrix into a product of ternary matrices and a real-valued scale vector, enabling the model to achieve accuracy levels close to bf16 precision. By expanding the inner rank beyond the full rank, ExTernD effectively reduces quantization error, allowing for continuous scaling of memory and computational resources. The authors provide a proof that the residual error decreases monotonically with the inner rank, ensuring that any target accuracy can be reached. The methodology includes a greedy alternating least squares (ALS) algorithm for efficient computation, along with a batched block-ALS GPU implementation that significantly speeds up the process. Empirical results demonstrate that ExTernD achieves competitive accuracy on benchmark datasets, surpassing traditional fixed-plane ternary quantization methods. This work represents a significant advancement in the field of model quantization, offering a flexible and efficient solution for deploying large language models with minimal loss in performance.
Methodology
The methodology involves factorizing each weight matrix into a product of ternary matrices and a scale vector, with an inner rank that can be adjusted to minimize quantization error. A greedy alternating least squares (ALS) algorithm is employed for the factorization, and a batched block-ALS approach is utilized for GPU acceleration. The method also incorporates an importance-weighted variant to account for varying significance across input channels.
Results
ExTernD matches the per-matrix accuracy of existing ternary quantization methods at effective bit-widths of 5.2–5.5 bits per weight. A full conversion of the Qwen3.5-4B model at an inner rank multiplier of 3 achieved a perplexity of 10.10 on the wikitext-2 dataset, outperforming bf16 precision (9.78 perplexity) by 3.2%. The method demonstrates a significant reduction in quantization error and improved accuracy across various benchmarks.
Implications
The findings suggest that ExTernD can be a powerful tool for deploying large language models in resource-constrained environments, enabling high-performance applications in natural language processing without the need for extensive retraining. This approach could facilitate the broader adoption of LLMs in practical applications where computational efficiency is critical.
ShortOPD: Recovering Pruned LLMs with Short-to-Long On-Policy Distillation
Large Language Models
NLP
Efficient ML
- Structured pruning can severely impact the performance of LLMs in generation tasks.
- Useful generation outputs are often demoted rather than erased after pruning.
- ShortOPD optimizes the recovery process by focusing on effective sequence lengths and reducing repetitive suffixes.
- The proposed method achieves significant performance gains with reduced training time and token usage.
Read more
ShortOPD: Recovering Pruned LLMs with Short-to-Long On-Policy Distillation
Summary
The paper addresses the challenges of structured pruning in large language models (LLMs), which often leads to performance degradation in free-form generation tasks despite retaining effectiveness in multiple-choice recognition tasks. The authors identify that while greedy pass@1 scores drop significantly post-pruning, pass@k scores improve with repeated sampling, indicating that useful outputs are still present but demoted. To recover the performance of pruned models, the authors propose ShortOPD, a novel on-policy distillation method that optimizes the training process by focusing on the effective lengths of generated sequences, thus avoiding the pitfalls of repetitive suffixes that dominate early rollouts. By leveraging the pre-compression model as a frozen teacher, ShortOPD enhances the recovery of compressed models significantly, achieving performance improvements across various tasks including math, code, and open-ended generation. The method demonstrates a substantial increase in generation scores, reducing training time and token usage compared to traditional recovery methods.
Methodology
The authors introduce ShortOPD, which employs a short-to-long on-policy distillation schedule. This method detects and avoids repetitive suffixes during training, focusing on the effective lengths of sequences generated by the pruned model. The compressed model samples rollouts from its own distribution while using a frozen pre-compression model as a teacher to provide dense token-level supervision.
Results
ShortOPD improves the compressed model's generation scores to approximately 9 times its unrecovered value and 1.6 to 4.4 times better than standard recovery methods. It matches the performance of a fixed 8192-token rollout horizon within two points while requiring only a quarter of the training time (8.5 hours vs. 35.9 hours) and 71% fewer rollout tokens.
Implications
The findings suggest that ShortOPD can enhance the practical application of structured pruning in LLMs, making them more efficient and effective for deployment in real-world generation tasks. This could lead to broader adoption of pruned models in various applications requiring high-quality text generation.
Understanding Structured Health Data through Interaction-Aware Mixture-of-Experts
Interpretability
- The study explores multi-view learning for structured health data to improve prediction and interpretability.
- Minimal performance gains were observed, indicating that views may not be independent modalities.
- Routing attribution showed systematic differences in importance across different views.
- View construction is essential for interpretability in clinical predictions.
Read more
Understanding Structured Health Data through Interaction-Aware Mixture-of-Experts
Summary
This paper investigates the use of interaction-aware mixture-of-experts (MoE) models for predicting post-stroke rigidity by leveraging multi-level views of structured health records. The authors highlight that traditional modeling approaches often treat structured health data as a single tabular input, which may overlook clinically significant interactions. By employing multi-view learning, the study explores whether transforming structured records into various representations can enhance predictive performance and interpretability. The authors utilize the I2MoE framework, which routes inputs through specialized experts to capture view-specific and synergistic signals. The findings indicate that while the multi-view approach yields only minimal improvements in predictive performance, the routing attribution reveals significant differences in importance allocations across views. This suggests that the construction of views is crucial for interpretability, even when predictive gains are modest, and emphasizes the potential of routing-based attribution for building explanations from structured health records.
Methodology
The authors employed the I2MoE framework, which utilizes a mixture-of-experts approach to route inputs through specialized experts based on multiple views derived from structured health records. They analyzed model-level, data-level, and representation-level views to assess their contributions to prediction and interpretability in the context of post-stroke rigidity prediction.
Results
The introduction of a multi-view approach resulted in minimal improvements in predictive performance, suggesting that the views are alternative decompositions rather than independent modalities. However, the routing attribution analysis revealed that different view designs led to systematic differences in importance allocation, which were consistent across similar patients.
Implications
The findings underscore the importance of view construction in enhancing the interpretability of clinical predictions from structured health data. This approach may inform future research on developing more interpretable models in healthcare settings, where understanding the rationale behind predictions is critical.
EXPLORE: Exploration with Guided Search for Analog Topology Generation using Language Models
Generative Models
Optimization
Large Language Models
- EXPLORE is the first framework to integrate structured test-time search with language model decoding for analog topology generation.
- The framework improves the success rate of topology generation from 12% (one-shot) and 33% (sampling-and-filter) to 65%.
- By employing structured-token filtering, EXPLORE reduces simulation trials by 24-48%, making it feasible to scale to higher-complexity circuits.
- The results show over 20× lower MSE compared to sampling-and-filter methods under the same search budget.
Read more
EXPLORE: Exploration with Guided Search for Analog Topology Generation using Language Models
Summary
The paper introduces EXPLORE, a novel framework for automating analog circuit topology design, addressing the limitations of existing one-shot generation methods that struggle with complex circuits due to vast search spaces and limited datasets. EXPLORE combines simulator-guided Monte Carlo Tree Search (MCTS) with transformer-based decoding to enhance the generation process. By leveraging language model priors and filtering high-confidence structural tokens, the framework efficiently allocates simulation resources, significantly improving the success rate and reducing mean squared error (MSE) in topology generation. The authors demonstrate that EXPLORE achieves a 65% success rate on a 6-component benchmark, a substantial improvement over previous methods. This work not only establishes a new approach for analog topology generation but also sets the stage for scaling LLM-driven design automation to more complex circuits.
Methodology
EXPLORE employs a search-enhanced framework that integrates simulator-guided Monte Carlo Tree Search (MCTS) with transformer-based decoding. It utilizes language model priors to guide the search process and implements structured-token filtering to bypass high-confidence structural tokens, reducing the computational burden during simulation.
Results
The framework achieved a 65% success rate in generating analog topologies at a tight tolerance of 0.01 for 6-component circuits, significantly outperforming previous methods. It also demonstrated over 20× reduction in mean squared error (MSE) compared to the sampling-and-filter baseline under the same search budget.
Implications
The advancements presented in EXPLORE have the potential to revolutionize the design automation of analog circuits, enabling faster and more efficient development cycles. This could lead to more rapid innovation in electronic systems, particularly in applications requiring customized circuit designs.
PUe: Biased Positive-Unlabeled Learning Enhancement by Causal Inference
Theory
- PUe addresses the limitations of existing PU learning methods by relaxing the assumption of uniform sampling of labeled examples.
- The framework employs normalized inverse probability weighting to correct PU risk estimators under biased conditions.
- Regularization techniques are introduced to improve propensity score estimation using deep learning models.
- PUe integrates with modern cost-sensitive PU methods, enhancing their performance in biased scenarios.
Read more
PUe: Biased Positive-Unlabeled Learning Enhancement by Causal Inference
Summary
The paper introduces PUe, a novel framework for Positive-Unlabeled (PU) learning that addresses the challenges posed by biased labeling in real-world scenarios. Traditional PU learning methods often assume that labeled positive examples are selected randomly, which is not the case in many applications. PUe builds upon the SAR-PU propensity-weighted framework and proposes a normalized inverse-probability-weighted PU risk formulation. The authors provide theoretical analyses of normalized sample-weight error and common PU estimators under biased labeling, and they extend propensity-score estimation to deep neural networks with regularization techniques to mitigate overfitting. The framework integrates with existing cost-sensitive PU methods and supports selectively labeled negative classes. Experimental results on datasets such as MNIST, CIFAR-10, and the Alzheimer’s Disease Neuroimaging Initiative (ADNI) demonstrate that PUe outperforms several baseline methods, particularly under non-uniform label distributions, showcasing its effectiveness in enhancing PU learning in biased contexts.
Methodology
The authors developed the PUe framework by formulating a normalized inverse-probability-weighted PU risk estimator. They conducted theoretical analyses of sample-weight errors and common PU estimators under biased labeling. Additionally, they utilized regularized deep learning models for propensity score estimation and integrated the framework with existing cost-sensitive PU methods.
Results
The experiments conducted on MNIST, CIFAR-10, and ADNI datasets showed that PUe significantly outperformed several baseline PU learning methods, particularly in scenarios with non-uniform label distributions, indicating its robustness and effectiveness in real-world applications.
Implications
The proposed PUe framework has potential applications in various domains where positive-unlabeled learning is relevant, such as recommendation systems, medical diagnostics, and spam detection. By addressing selection bias, PUe can improve the accuracy of classifiers trained on limited labeled data.
When Does Reward Teach State? A Hidden-Automaton Instrument and the Group-Language Boundary
Reinforcement Learning
Theory
Optimization
- Introduces a white-box measurement methodology to assess whether an RL agent learns the task state or a reward shortcut.
- Establishes three axes (optimizer strength, task structure, observation informativeness) that influence the coupling of reward success and latent-state learning.
- Demonstrates that high rewards do not necessarily indicate task understanding, as agents may exploit shortcuts.
- Identifies a pre-training structural warning signal for perception gaps linked to group-language structures.
Read more
When Does Reward Teach State? A Hidden-Automaton Instrument and the Group-Language Boundary
Summary
This paper addresses the question of whether a reinforcement learning (RL) agent that achieves high rewards truly understands the latent state of its task or merely exploits a reward-correlated shortcut. The author introduces a novel measurement methodology using a hidden deterministic finite automaton (DFA) framework, allowing for precise evaluation of the agent's learning. By structuring the task as a hidden DFA, the agent can observe a symbol stream and choose the next symbol under partial control, receiving a sparse terminal reward for acceptance. This setup enables the measurement of reward success and latent-state learning as distinct, quantifiable entities. The paper identifies three axes that govern the coupling between these two quantities: optimizer strength, task structure (specifically the presence of group-language characteristics), and the informativeness of observations. The findings reveal that weak RL often leads to decoupled reward and state learning, while stronger optimizers can couple them, except in cases of group-language tasks. The study also highlights the importance of observation informativeness in recovering latent states, providing a clear distinction between perception gaps and planning gaps. Overall, the proposed methodology offers a robust framework for understanding the relationship between reward and state learning in RL agents.
Methodology
The methodology involves expressing the task as a hidden DFA, allowing the agent to observe symbol streams and make decisions under partial control. The framework enables the exact measurement of optimal achievable returns and the latent state at each step, facilitating a clear analysis of reward and state learning.
Results
The study found that weak on-policy RL often results in decoupled reward and state learning, while stronger optimizers like PPO+GAE can recover latent states, albeit with high variance. The presence of group-language structures serves as a predictive indicator of perception gaps, achieving a precision of 0.86 in identifying such gaps across tested automata.
Implications
The findings have significant implications for the design and evaluation of RL agents, particularly in understanding their learning processes and ensuring they do not rely on reward shortcuts. The methodology can be applied to various finite-state tasks, enhancing the interpretability of RL outcomes and guiding future research in reinforcement learning.
Lyapunov Exponent as Physics-Informed Dense Reward: RL Discovery of Stabilization Beyond the Kapitza Pendulum
Reinforcement Learning
Robotics
Theory
- The Lyapunov characteristic exponent (LCE) is proposed as a dense reward signal for RL in stabilizing inverted pendulums.
- Previous methods using sparse rewards failed to stabilize the pendulum, highlighting the importance of reward design.
- The RL agent successfully discovered stabilization strategies for the inverted pendulum with vertical motion using LCE.
- The paper provides a theoretical framework for understanding LCE and its application in dynamic systems.
Read more
Lyapunov Exponent as Physics-Informed Dense Reward: RL Discovery of Stabilization Beyond the Kapitza Pendulum
Summary
This paper introduces the use of the Lyapunov characteristic exponent (LCE) as a dense reward signal in reinforcement learning (RL) for stabilizing an inverted pendulum with vertical motion. The author investigates whether an RL agent can discover the stabilization of the inverted pendulum, particularly in the context of the Kapitza pendulum, which maintains an upright position through oscillatory motion of its pivot. Previous attempts using standard reward signals, such as the squared angle from vertical, failed to achieve stabilization due to reward sparsity. By employing the LCE as a reward, which provides a direct measure of system stability, the agent successfully learned to stabilize the pendulum in an upright position. The paper discusses the theoretical background of LCE, its computation, and the dynamics of the inverted pendulum system, ultimately demonstrating that the LCE can effectively guide the RL agent towards discovering stable control strategies.
Methodology
The methodology involves formulating the inverted pendulum problem with an oscillating pivot and deriving the equations of motion. The LCE is computed numerically to serve as a reward signal for the RL agent. The agent is trained using reinforcement learning techniques to stabilize the pendulum by maximizing the LCE, which indicates system stability.
Results
The results demonstrate that the RL agent, when trained with LCE as a reward, successfully learns to stabilize the inverted pendulum in an upright position, surpassing previous attempts that utilized simpler reward structures. The agent effectively discovers the oscillatory motion characteristic of the Kapitza pendulum and maintains stability through vertical motion of the pivot.
Implications
The implications of this research extend to the design of RL systems for complex dynamic control tasks, suggesting that physics-informed reward structures can significantly improve learning outcomes. This approach could be applied to various robotics and control applications where stability is critical.
TerraZero: Procedural Driving Simulation for Zero-Demonstration Self-Play at Scale
Reinforcement Learning
Robotics
Efficient ML
- TerraZero achieves high simulation throughput of 1.3M agent-steps per second, enabling large-scale RL training.
- The simulator generates diverse driving scenarios through procedural methods, enhancing training coverage of safety-critical situations.
- Policies are trained from scratch using self-play without human demonstrations, showcasing strong generalization across different environments.
- TerraZero outperforms existing methods in benchmark evaluations for safety and collision metrics.
Read more
TerraZero: Procedural Driving Simulation for Zero-Demonstration Self-Play at Scale
Summary
The paper introduces TerraZero, a procedural driving simulator designed to facilitate the training of autonomous driving agents through reinforcement learning (RL) without relying on human demonstrations. TerraZero addresses the limitations of existing simulators by providing a high-throughput, feature-rich environment that can generate diverse and complex driving scenarios. The simulator operates on a configurable C engine, achieving up to 1.3 million agent-steps per second on a single server-grade GPU, significantly faster than current object-level simulators. It utilizes real-world map geometry as a foundation while populating the environment with randomized road users and traffic dynamics, allowing for an effectively infinite variety of training scenarios. The training process employs a compute-efficient self-play strategy across multiple GPUs, enabling policies to be learned from scratch without imitation or fallback planners. The resulting driving policies demonstrate strong generalization capabilities, excelling in various benchmarks, including the InterPlan long-tail benchmark and routine-driving evaluations, while also achieving competitive performance in realism assessments against other methods. Overall, TerraZero represents a significant advancement in the field of autonomous driving simulation and training.
Methodology
TerraZero employs a procedural scenario generation approach, utilizing a configurable C engine for simulation and GPU for policy inference. It randomizes agent dynamics, rewards, and sizes per episode while enforcing traffic rules and populating maps with diverse road users. The training utilizes a distributed Proximal Policy Optimization (PPO) algorithm across multiple GPUs, allowing for efficient self-play training without reliance on logged data or human demonstrations.
Results
The policies trained using TerraZero achieved top performance on the InterPlan long-tail benchmark, outperforming larger learned planners. On routine-driving evaluations, TerraZero ranked among the best approaches, demonstrating the lowest collision rates and optimal time-to-collision scores. The simulator also showed competitive performance in realism assessments against other self-play methods.
Implications
TerraZero's advancements in procedural simulation and self-play training can significantly enhance the development of robust autonomous driving systems. By eliminating the need for human demonstrations, it opens new avenues for scalable training methods in various driving environments, potentially accelerating the deployment of safe autonomous vehicles.
TRACE: Turn-level Reward Assignment via Credit Estimation for Long-Horizon Agents
Reinforcement Learning
Large Language Models
- TRACE provides a dense credit-assignment mechanism for long-horizon reinforcement learning agents.
- The method uses a frozen reference model to evaluate the effectiveness of intermediate actions in a trajectory.
- Significant performance improvements were observed on both closed-web and open-web benchmarks.
- TRACE eliminates the need for cold-start supervised fine-tuning or additional critic models.
Read more
TRACE: Turn-level Reward Assignment via Credit Estimation for Long-Horizon Agents
Summary
The paper introduces TRACE, a novel framework for dense credit assignment in reinforcement learning, specifically designed for long-horizon agents that engage in multi-turn interactions. Traditional outcome-based rewards often fail to provide adequate feedback for intermediate actions, leading to high variance and ineffective learning. TRACE addresses this by assigning turn-level rewards based on credit estimation, utilizing a frozen reference model to evaluate the predictability of the final answer at tool-call boundaries. By transforming log-probabilities into log-ratio state values, TRACE computes per-action rewards through temporal-difference changes, allowing for a more nuanced understanding of which actions contribute positively or negatively to the task. The method is evaluated on complex search tasks, demonstrating significant improvements in tool-use ability without the need for extensive pre-training or additional supervision. The results indicate that TRACE not only enhances performance on closed-web benchmarks but also shows effective transfer to open-web tasks, with faster convergence during training.
Methodology
TRACE employs a critic-free framework that assigns rewards at tool-call boundaries by evaluating the log-probabilities of gold answers from a frozen reference model. It calculates log-ratio state values to measure progress and derives turn-level rewards using temporal-difference changes, effectively distinguishing productive actions from irrelevant ones.
Results
On the closed-web BrowseComp-Plus benchmark, TRACE improved the performance of Qwen3-4B from 7.2 to 35.6 and Qwen3-30B-A3B from 8.4 to 42.6. The learned behaviors also transferred well to open-web benchmarks, with notable scores on GAIA and xbench-DeepSearch. The training curves indicated earlier improvements and faster convergence during reinforcement learning.
Implications
TRACE's approach to dense credit assignment could significantly enhance the training of long-horizon agents in various applications, including web navigation and complex task execution, by providing more informative feedback on intermediate actions. This could lead to more efficient learning processes in reinforcement learning scenarios.
DeepLoop: Depth Scaling for Looped Transformers
NLP
Large Language Models
Optimization
- Introduces the concept of tied-depth aggregation in Looped Transformers.
- Derives a loop-aware first-order perturbation bound with a visit-alignment coefficient.
- Proposes a new scaling rule for residual connections in looped architectures.
- Demonstrates improved performance in language modeling tasks with increased loop counts.
Read more
DeepLoop: Depth Scaling for Looped Transformers
Summary
The paper introduces DeepLoop, a novel framework for scaling the depth of Looped Transformers without increasing the number of stored parameters. Traditional Transformers face a challenge where increasing depth also increases parameter count, but Looped Transformers mitigate this by reusing a compact stack of physical blocks across multiple rounds. The authors formalize the concept of tied-depth, where a shared update aggregates gradients from repeated visits to the same physical blocks, leading to a new residual-scaling problem. They derive a first-order perturbation bound that incorporates a visit-alignment coefficient, κR, which adjusts the scaling rules for residual connections. The proposed scaling rules, α = (2N)1/2 and β = (8N)−1/2, are shown to stabilize training in scenarios where blocks are revisited multiple times. Empirical evaluations on GPT-2 small and medium models demonstrate that DeepLoop improves validation loss and downstream task performance as the loop count increases, confirming that effective depth should be considered in residual scaling.
Methodology
The authors analyze the residual scaling problem in Looped Transformers by introducing a visit-alignment coefficient, κR, which modifies the stability conditions for training. They derive a first-order perturbation bound that accounts for the shared updates across repeated visits to the same physical blocks. The proposed scaling rules are then empirically tested on GPT-2 models to evaluate their effectiveness in improving validation loss and downstream task accuracy.
Results
DeepLoop shows neutral performance at R = 1 (no block revisits) but consistently improves validation loss and downstream task accuracy as the loop count increases. The results validate the hypothesis that residual scaling should depend on how depth is realized, rather than just the nominal layer count.
Implications
The findings suggest that Looped Transformers can be effectively scaled without a proportional increase in parameters, making them more efficient for various NLP tasks. This approach could lead to advancements in the design of deep learning models that require high expressivity without excessive resource consumption.
What Does Goodness Measure? A Likelihood-Ratio Account of Forward-Forward Learning
Theory
- The goodness measure is shown to be a sufficient statistic for a likelihood-ratio test, providing a theoretical basis for its use in FF learning.
- Generalizations to anisotropic and heavy-tailed distributions yield new insights into the behavior of the goodness measure and its implications for network training.
- Normalization between layers is critical for effective learning, and the paper identifies the correct form of normalization to prevent performance collapse.
- Empirical results validate the theoretical predictions, demonstrating the practical relevance of the derived goodness measure.
Read more
What Does Goodness Measure? A Likelihood-Ratio Account of Forward-Forward Learning
Summary
This paper presents a theoretical framework for the Forward-Forward (FF) learning algorithm, which trains neural network layers locally by optimizing a scalar goodness measure based on the sum of squared activations. The author argues that this goodness measure can be understood as a sufficient statistic in a likelihood-ratio test between two populations differing in scale. The paper derives the optimal threshold for this goodness measure under a zero-mean isotropic Gaussian model and extends the analysis to anisotropic and heavy-tailed populations. The normalization process between layers is also examined, revealing that it should preserve per-coordinate energy while removing length to avoid trivial solutions. An empirical study on convolutional FF networks demonstrates the theoretical predictions, showing that the structured readout improves performance and aligns with the expected statistical properties. Overall, the paper transforms the heuristic nature of the FF learning objectives into measurable and testable constructs, providing a solid foundation for future research in local learning methods.
Methodology
The paper employs a generative modeling approach to derive the goodness measure as a sufficient statistic for a likelihood-ratio test. It analyzes the implications of different population distributions (isotropic, anisotropic, heavy-tailed) on the goodness measure and its normalization. An empirical study is conducted using convolutional FF networks to validate the theoretical claims.
Results
The theoretical framework successfully identifies the goodness measure as a sufficient statistic, with empirical results showing that the structured readout improves performance in separating clean images from distorted ones. The study confirms that the predicted statistical properties align with the observed behavior of the networks, particularly in terms of activation distributions and the effects of normalization.
Implications
The findings suggest that FF learning can be a viable alternative to traditional backpropagation methods, particularly in scenarios where local learning is preferred. The theoretical insights into the goodness measure and normalization can inform the design of more effective neural network architectures and training algorithms.
Temperature Scaling Is Not Enough: Calibration Gaps Under Human Label Distributions
Computer Vision
NLP
Theory
- Temperature scaling is ineffective for models trained on soft label distributions.
- A positive soft-label calibration gap exists, with larger models exhibiting greater discrepancies.
- Calibration gaps are more pronounced in language tasks compared to vision tasks.
- The study provides a formal definition of the soft-label calibration gap.
Read more
Temperature Scaling Is Not Enough: Calibration Gaps Under Human Label Distributions
Summary
This paper investigates the effectiveness of temperature scaling, a widely used post-hoc calibration method for deep learning models, under conditions where ground-truth labels are not deterministic but rather soft or distributional. The author highlights that temperature scaling assumes one-hot labels, which is often not the case in practical scenarios involving human annotators. The study utilizes two datasets, CIFAR-10H and ChaosNLI, to evaluate the calibration performance of models across different scales when calibrated on hard versus soft labels. The findings reveal a consistent positive soft-label calibration gap, indicating that models calibrated with hard labels perform worse than those calibrated directly on soft labels. This gap increases with model scale, particularly in the language domain compared to vision. The paper also compares the results with multiclass isotonic regression, confirming similar trends. The results suggest that calibration practices based on majority-vote labels may misrepresent model reliability, especially in applications where label ambiguity is inherent, raising concerns for deployment in safety-critical contexts.
Methodology
The study conducts a controlled measurement analysis using two datasets with soft label distributions. It evaluates models across three scales for both hard one-hot and soft distributional label targets. The Brier Score is used to quantify calibration performance, and multiclass isotonic regression is employed as a comparative calibration method.
Results
The results indicate a significant positive soft-label calibration gap across all configurations, with Brier Score differences ranging from 0.002 to 0.134. The gap increases with model scale, particularly in the language domain (mean gap of 0.079) compared to the vision domain (mean gap of 0.003). The study also identifies a scale-ordering anomaly in one dataset that requires further investigation.
Implications
The findings suggest that reliance on majority-vote labels for calibration can lead to misestimations of model reliability, particularly in safety-critical applications. This highlights the need for calibration methods that account for label ambiguity and human disagreement, which is crucial for deploying models in real-world scenarios.
Contrastive-Collapsed Loss for Flexible and Geometrically Optimal Embeddings and Faster Convergence
Theory
Optimization
Efficient ML
- Introduction of CoCo loss function for improved representation learning.
- Theoretical advantages over traditional loss functions like dot regression and cross-entropy.
- Empirical results show competitive performance with state-of-the-art methods.
- CoCo promotes faster convergence and tighter class clustering.
Read more
Contrastive-Collapsed Loss for Flexible and Geometrically Optimal Embeddings and Faster Convergence
Summary
This paper introduces CoCo, a novel loss function designed to enhance the learning of normalized and well-structured representations in neural networks. The CoCo loss function promotes intra-class collapse and inter-class contrast while allowing flexibility for networks to achieve geometrically optimal embeddings with significant angular separation between classes. The authors provide a theoretical framework that positions CoCo in relation to existing objectives like dot regression and cross-entropy, demonstrating its advantages in terms of initialization proximity to optimal configurations, informative gradients, and incentives for class-wise representation collapse. Extensive experiments conducted on various tabular datasets from the OpenML-CC18 benchmark indicate that CoCo performs competitively against state-of-the-art methods, including kernel SVM, Random Forest, and traditional neural network losses. The findings suggest that CoCo not only fosters tighter class clustering but also accelerates convergence during training, making it a promising objective for learning discriminative representations while maintaining predictive performance.
Methodology
The authors define the CoCo loss function, which aims to collapse intra-class samples into a single vector while maximizing inter-class separation and enforcing normalization. The loss is mathematically formulated to ensure that the transformed data maintains optimal angular distances based on class labels. Theoretical analysis and empirical experiments are conducted to validate the effectiveness of CoCo against established methods.
Results
The experiments reveal that CoCo achieves competitive performance across various datasets, demonstrating faster convergence and tighter clustering of class representations compared to traditional loss functions. The theoretical analysis supports the empirical findings, indicating that CoCo provides more informative gradients and closer initialization to optimal configurations.
Implications
The CoCo loss function has significant implications for the design of neural network architectures, particularly in tasks requiring robust and discriminative embeddings. Its ability to enhance convergence speed and representation quality could lead to advancements in various machine learning applications, including classification tasks in high-dimensional spaces.
Where Should RL Post-Training Compute Go? Model Size, Search, Learning, and Feedback
Reinforcement Learning
Robotics
Efficient ML
- Introduces a FLOP-accounting framework for RL post-training that categorizes compute allocation.
- Demonstrates that optimal compute allocation is contingent on model size, budget, reward systems, and evaluation targets.
- Finds that larger models require more compute per token, affecting the number of rollouts and updates possible.
- Presents RACE as a diagnostic tool for determining effective compute allocation strategies.
Read more
Where Should RL Post-Training Compute Go? Model Size, Search, Learning, and Feedback
Summary
This paper investigates the allocation of computational resources during reinforcement learning (RL) post-training, particularly in the context of adapting foundation models for reasoning and feedback in robot-learning systems. The authors introduce a FLOP-accounting framework that categorizes compute resources into three main areas: rollout/search, policy-update/learning, and reward/feedback evaluation. They explore how the allocation of these resources affects performance under a fixed budget, revealing that the optimal allocation depends on various factors such as model size, compute budget, reward system, and evaluation targets. The study emphasizes that larger models consume more compute per token, which can limit the number of updates or rollouts achievable within the same budget. The authors also present RACE, a diagnostic protocol designed to help identify effective allocation regimes prior to costly validation runs. The findings indicate that the effectiveness of compute allocation strategies varies significantly based on the specific goals of the training process, such as maximizing native rewards or achieving high accuracy in evaluations.
Methodology
The authors conducted empirical studies using GRPO post-training on LoRA-adapted language-model policies, focusing on mathematical reasoning tasks. They analyzed the effects of different compute allocation strategies on performance metrics, comparing the outcomes of varying model sizes, rollout lengths, and reward systems under a fixed computational budget.
Results
The study found that the best allocation of compute resources varies significantly based on the model size and the specific evaluation targets. Different reward systems led to distinct allocation behaviors, with some configurations favoring policy rollouts over updates or vice versa. The introduction of RACE provided a systematic approach to identify promising allocation strategies before engaging in more resource-intensive validation processes.
Implications
The findings suggest that RL post-training strategies should be tailored to specific goals and contexts, as the effectiveness of compute allocation can differ widely. This has implications for the design of robot-learning systems and the development of foundation models, emphasizing the need for careful consideration of how computational resources are distributed during training.
A Hybrid Mamba for Audio-Visual Navigation
Multimodal
Robotics
Audio & Speech
- Introduction of Samba, a hybrid state-space architecture for audio-visual navigation.
- Replacement of traditional GRUs with the Mamba State Encoder for better temporal aggregation.
- Development of the Audio Mamba Encoder to capture global time-frequency dependencies in audio data.
- Significant improvements in navigation success rates compared to existing state-of-the-art models.
Read more
A Hybrid Mamba for Audio-Visual Navigation
Summary
This paper introduces Samba, a novel hybrid state-space architecture designed to enhance audio-visual navigation (AVN) capabilities. Traditional models, primarily based on convolutional neural networks (CNNs) and gated recurrent units (GRUs), have shown limitations in effectively representing dynamic multimodal sequences. Samba addresses these issues by incorporating the Mamba State Encoder (M-SE) to replace conventional GRUs, allowing for adaptive selection and improved temporal aggregation. Additionally, the Audio Mamba Encoder (AME) is introduced to overcome the challenges posed by convolutional operators in capturing global time-frequency dependencies in audio spectrograms. The proposed framework demonstrates significant improvements in navigation success rates, achieving an 11.3% increase on the Matterport3D dataset and even greater gains on the Replica dataset. The architectural innovations not only enhance representation capabilities but also maintain lower computational costs, paving the way for advancements in AVN research.
Methodology
The Samba framework employs a hybrid state-space architecture that integrates the Mamba State Encoder for temporal state tracking and the Audio Mamba Encoder for audio feature extraction. This architecture utilizes a selective scanning mechanism to enhance the representation of multimodal signals, ensuring efficient processing of audio and visual data in complex environments.
Results
Samba achieved an 11.3% improvement in navigation success rate on the Matterport3D dataset compared to existing models. The performance gains were even more pronounced on the Replica dataset, indicating the framework's robustness in handling diverse and complex scene structures.
Implications
The advancements presented in this paper could lead to more effective audio-visual navigation systems in robotics and autonomous agents, enhancing their ability to operate in unstructured environments. The hybrid architecture may also inspire further research into multimodal representation learning and state-space models in various applications.