AI-generated summaries
Today's ML research,
without the noise.
Daily summaries of the latest machine learning papers from arXiv, processed every 8 hours.
63
Papers today
8h
Update frequency
7
Days of history
EXHOLD: Experience-Aware Real-Time Hold Control for Large-Scale Ride-Hailing Matching at DiDi
Optimization
- EXHOLD improves hold control in ride-hailing systems by decoupling assessment and execution.
- The framework uses experience tiers to optimize multiple satisfaction-related signals.
- Constrained optimization ensures that hold times are managed within service guardrails.
- Real-world deployment in Brazil showed significant improvements in trip success and driver welfare.
Read more
EXHOLD: Experience-Aware Real-Time Hold Control for Large-Scale Ride-Hailing Matching at DiDi
Summary
The paper introduces EXHOLD, a two-stage framework designed to enhance hold control in large-scale ride-hailing matching systems, specifically implemented at DiDi. The primary goal of hold control is to improve the passenger-driver experience by selectively delaying certain driver-order pairs to optimize trip success rates. Traditional heuristic strategies often struggle with non-stationary traffic conditions and multi-objective optimization. EXHOLD addresses these challenges by decoupling the assessment of driver-order pairs from the execution of hold times. In Stage I, a decision model categorizes each pair into discrete experience tiers based on multiple satisfaction signals across the matching funnel. Stage II employs constrained optimization to determine a hold-time schedule that adheres to service guardrails, ensuring that promising matches are not unnecessarily held. The framework was validated through online A/B experiments in Brazil, showing significant improvements in trip completion rates, driver income, and reductions in passenger cancellations. The results indicate that both stages of EXHOLD are crucial for achieving these gains, demonstrating its effectiveness in real-world applications.
Methodology
EXHOLD consists of two main stages: Stage I involves learning a decision model that assigns driver-order pairs to experience tiers based on various satisfaction signals. Stage II utilizes constrained optimization to create a hold-time schedule that respects service guardrails, ensuring efficient and effective hold control.
Results
The implementation of EXHOLD in DiDi's production system led to increased trip completion rates, enhanced driver income, and a notable reduction in passenger cancellations. The framework also improved key efficiency metrics, such as faster call-to-acceptance times, demonstrating its effectiveness in optimizing the matching process.
Implications
EXHOLD's approach to hold control can be applied to other large-scale matching systems beyond ride-hailing, potentially improving user experiences in various domains where timely decision-making is critical. Its design principles may inform future developments in multi-objective optimization and real-time decision-making frameworks.
Semantic Pareto-DQN: A Multi-Objective Reinforcement Learning Framework for Financial Anomaly Detection
Reinforcement Learning
Optimization
- Introduces a multi-objective reinforcement learning framework for financial anomaly detection.
- Utilizes large language models to create cohesive state representations from transaction features.
- Decouples multiple objectives into a vectorial reward system to navigate trade-offs effectively.
- Demonstrates superior performance in minority-class recall compared to traditional methods.
Read more
Semantic Pareto-DQN: A Multi-Objective Reinforcement Learning Framework for Financial Anomaly Detection
Summary
The paper addresses the challenges of financial anomaly detection, particularly the issue of extreme class imbalance that leads to the phenomenon known as 'fraud collapse', where traditional single-objective algorithms fail to effectively identify anomalies. To tackle this problem, the authors propose the Semantic Pareto-DQN, a multi-objective reinforcement learning (MORL) framework that synthesizes transaction features into natural-language narratives using large language models. This approach enables the creation of a robust state representation that is scale-invariant. The agent operates on a vectorial reward system that decouples financial efficacy, operational friction, and semantic discovery, allowing it to navigate the trade-offs between missed anomalies and false positives dynamically. Empirical evaluations on E-Commerce fraud and UCI Credit datasets demonstrate that the Semantic Pareto-DQN outperforms traditional scalarized models, achieving higher minority-class recall and effectively avoiding the zero-recall trap. This framework provides a novel alternative for balancing operational friction with the need for effective financial anomaly detection.
Methodology
The authors formalize financial anomaly detection as a Semantic Multi-Objective Markov Decision Process (MOMDP) and implement a Pareto Deep Q-Network (Pareto-DQN). The framework employs hypervolume-based action selection to navigate the continuous Pareto frontier, allowing the agent to balance competing objectives of financial efficacy, operational friction, and semantic anomaly discovery.
Results
Empirical evaluations show that the Semantic Pareto-DQN achieves significantly higher minority-class recall across both E-Commerce fraud and UCI Credit datasets, effectively mapping the Pareto frontier and maintaining high state trajectory variance to prevent global recall collapse.
Implications
The proposed framework has potential applications in various financial sectors, particularly in fraud detection systems where balancing the trade-offs between detecting anomalies and minimizing customer friction is crucial. It offers a more effective approach to handling class imbalance in anomaly detection tasks.
Variable-Length Generative Protein Design via Generalized Poisson Flow
Generative Models
- Introduction of GPFlow, a variable-length generative model for protein design.
- Theoretical guarantees for joint multimodal distribution recovery and KL divergence bounds.
- Superior performance of GPFlow over fixed-length models in various protein design tasks.
- Demonstrated flexibility in generating proteins of varying lengths without prior length specification.
Read more
Variable-Length Generative Protein Design via Generalized Poisson Flow
Summary
This paper introduces Generalized Poisson Flow (GPFlow), a novel framework for variable-length generative protein design that addresses the limitations of existing diffusion and flow-based models which require fixed protein lengths. GPFlow utilizes an inhomogeneous generalized Poisson process to learn the rate function, allowing for flexible protein length generation without prior specification. The authors establish theoretical guarantees for joint multimodal distribution recovery and derive an upper bound on KL divergence between generated and real distributions. The framework is evaluated across five protein design scenarios: unconditional structure design, unconditional sequence design, structure-based motif scaffolding, sequence-based motif scaffolding, and peptide structure-sequence co-design. Results show that GPFlow significantly outperforms fixed-length baselines in terms of designability, sequence fitness, and unique successes in motif scaffolding tasks. The findings suggest that dynamic-length modeling can enhance protein design flexibility and effectiveness, making GPFlow a promising tool for future protein engineering applications.
Methodology
The methodology involves formulating a variable-length generative model using an inhomogeneous generalized Poisson process. The authors derive a tractable objective from the negative log-likelihood of the stochastic process and establish a marginalization principle for multimodal generation. The framework is evaluated through comprehensive experiments across multiple protein design tasks, comparing GPFlow's performance with fixed-length baselines.
Results
GPFlow achieved significant improvements in designability and sequence fitness compared to fixed-length models. In unconditional design tasks, GPFlow showed a 96.1% designability rate and reduced ΔpLDDT from 14.40 to 3.17. In conditional motif scaffolding, it ranked first on 10 out of 16 tasks, with a notable increase in unique successes. In peptide co-design, GPFlow maintained competitive performance even without a native-length oracle, demonstrating its robustness and flexibility.
Implications
The introduction of GPFlow has the potential to revolutionize protein design by allowing for more flexible and efficient exploration of the design space. This could lead to advancements in synthetic biology, drug design, and therapeutic protein engineering, where the optimal protein length is often unknown.
Autoregressive latent diffusion for 3D molecule generation
Generative Models
Graph Learning
- Introduction of KRONOS, a latent autoregressive diffusion framework for 3D molecule generation.
- Combines autoregressive sequence modeling with diffusion-based latent token prediction.
- Enables both unconditional and fragment-conditioned molecular generation within a single model.
- Achieves leading unconditional generation performance on benchmark datasets.
Read more
Autoregressive latent diffusion for 3D molecule generation
Summary
This paper presents KRONOS, a novel framework for generating three-dimensional (3D) molecules using a latent autoregressive diffusion approach. Traditional diffusion models excel in generation quality but require predefined molecular sizes, while autoregressive models offer flexibility in generating variable-length molecules and conditioning on partial structures. KRONOS addresses the challenge of balancing unconditional and context-conditioned generation by modeling molecular graph topology and geometry in the latent space of a pre-trained autoencoder. The framework employs a mixed training strategy inspired by the Fill-in-the-Middle (FIM) paradigm, allowing for both unconditional and fragment-conditioned generation within a single autoregressive model. Experimental results on the QM9 and GEOM-Drugs datasets demonstrate that KRONOS achieves leading performance among autoregressive methods while remaining competitive with diffusion models. Additionally, the framework supports fragment-conditioned generation without compromising unconditional generation quality, showcasing its versatility in molecular design.
Methodology
KRONOS operates in the latent space of a pre-trained Unified AutoEncoder (UAE), modeling the joint distribution over latent molecular tokens autoregressively. It incorporates a diffusion objective for parameterizing conditional distributions and includes a stop-prediction mechanism for variable-length generation. The mixed training strategy allows for simultaneous unconditional and fragment-conditioned generation.
Results
KRONOS outperforms existing autoregressive methods in unconditional generation on the QM9 and GEOM-Drugs datasets, while also maintaining competitive performance against diffusion models. The framework successfully enables fragment-conditioned generation without negatively affecting unconditional generation quality.
Implications
The development of KRONOS has significant implications for drug discovery and molecular design, providing a robust tool for generating diverse molecular structures. Its ability to condition on fragments while maintaining high-quality unconditional generation opens new avenues for designing novel compounds and optimizing existing ones.
Frequency-Domain Multi-Modality Transportation Modeling
Time Series
Multimodal
- Introduces a frequency-domain approach to multi-modality transportation modeling.
- Employs a Modality-Wise Frequency Filter (MFF) for spectral refinement.
- Incorporates a Frequency-Guided Synergy Integrator (FSI) for selective cross-modality information sharing.
- Demonstrates superior performance compared to state-of-the-art methods on real-world datasets.
Read more
Frequency-Domain Multi-Modality Transportation Modeling
Summary
The paper presents a novel approach to multi-modality transportation forecasting through a framework called Frequency-Domain Multi-Modality modeling (FreMo). Traditional methods struggle with the complexities of different transportation modes, which exhibit unique spectral characteristics and interact unevenly across frequencies. FreMo addresses these challenges by operating in the frequency domain, allowing for adaptive and selective synergy between modalities. The framework includes a Modality-Wise Frequency Filter (MFF) that refines spectral components for each modality, enhancing informative frequencies while reducing noise. Additionally, a Frequency-Guided Synergy Integrator (FSI) aggregates information across modalities based on their frequency-specific contributions, promoting effective knowledge sharing while minimizing negative transfer. The authors conducted extensive experiments on real-world datasets, demonstrating that FreMo consistently outperforms existing state-of-the-art methods, showcasing superior performance and generalization across various forecasting scenarios.
Methodology
The methodology involves the development of the FreMo framework, which operates in the frequency domain. It utilizes a Modality-Wise Frequency Filter (MFF) to refine the spectral components of each modality and a Frequency-Guided Synergy Integrator (FSI) to selectively aggregate information based on frequency-dependent contributions. This allows for adaptive cross-modality synergy, enhancing the forecasting accuracy of transportation dynamics.
Results
The experimental results indicate that FreMo significantly outperforms existing forecasting models across various scenarios, demonstrating enhanced predictive accuracy and generalization capabilities. The framework effectively captures the complex interactions and distinct characteristics of different transportation modalities.
Implications
The findings suggest that FreMo can be applied in urban transportation systems for improved forecasting, potentially aiding in traffic management, public transit optimization, and urban planning. The framework's ability to handle multi-modality data in the frequency domain opens avenues for further research in related fields such as smart city development and real-time traffic analysis.
Data-Efficient Deep Learning: Empirical Guidelines for Training Set Size Estimation in Inertial Sensor Classification
Efficient ML
Time Series
Theory
- Learning curve convergence for inertial sensor classification is evaluated.
- Classification accuracy consistently follows a logarithmic growth pattern.
- A new stability point metric is introduced to optimize training data collection.
- Models often reach stability with fewer samples than traditional heuristics suggest.
Read more
Data-Efficient Deep Learning: Empirical Guidelines for Training Set Size Estimation in Inertial Sensor Classification
Summary
This paper addresses the challenge of determining the minimum sample size required for effective classification in inertial sensor-based tasks, such as human activity recognition (HAR) and smartphone location recognition (SLR). The authors conduct a systematic empirical evaluation of learning curve convergence across six diverse datasets, totaling 102.7 hours of inertial data. They discover that classification accuracy follows a logarithmic growth pattern, leading to the introduction of a new metric termed the 'stability point,' which quantifies the sample size needed for the learning curve to stabilize within a specified mean absolute percentage deviation of its maximum accuracy. The findings indicate that models can achieve stability with significantly fewer samples than traditional heuristics suggest, shifting the focus from maximizing data volume to optimizing data efficiency. This research provides a framework for extrapolating total data requirements from small-scale pilot studies, offering practical guidelines for planning data collection in inertial sensing applications.
Methodology
The authors analyze six diverse datasets through empirical evaluations of learning curves, deriving an empirical formula to estimate classification performance relative to dataset size. They introduce a stability point metric to determine the sample size necessary for the learning curve to stabilize.
Results
The analysis reveals that classification accuracy improves logarithmically with dataset size, and models achieve practical stability with fewer samples than previously recommended. This finding allows for more efficient data collection strategies in inertial sensor applications.
Implications
The results have significant implications for data collection strategies in fields relying on inertial sensors, such as healthcare and sports analytics. By optimizing data efficiency, researchers can reduce the time and resources spent on data collection while maintaining model reliability.
A Unified Approach to Interpreting Knowledge Distillation for Large Language Models via Interactions
NLP
Large Language Models
Interpretability
- The paper provides a unified interpretation of knowledge distillation mechanisms in LLMs through interaction analysis.
- Sparsification of interactions is identified as the common mechanism across different KD methods.
- The performance of KD methods is linked to their ability to handle complex interactions effectively.
- The proposed Complex Interaction Penalty (CIP) loss function improves the performance of KD methods.
Read more
A Unified Approach to Interpreting Knowledge Distillation for Large Language Models via Interactions
Summary
This paper investigates the underlying mechanisms of knowledge distillation (KD) in large language models (LLMs) by focusing on interactions among input variables. The authors propose a unified framework that decomposes the output scores of LLMs into a sum of interactions, revealing that the effectiveness of various KD methods is linked to the sparsification of these interactions. Specifically, student models tend to retain fewer, more salient interactions while suppressing others to near-zero effects. The study identifies that the performance variance across different KD methods is primarily due to their ability to manage complex interactions. To leverage these insights, the authors introduce a new loss function called Complex Interaction Penalty (CIP), which encourages the sparsification of complex interactions during the distillation process. Experimental results demonstrate that incorporating CIP enhances the performance of various KD methods across multiple benchmarks, both in-domain and out-of-distribution.
Methodology
The authors decompose the output of LLMs into interactions among input variables, analyzing how different KD methods affect the sparsity of these interactions. They introduce the Complex Interaction Penalty (CIP) to enforce sparsity during the distillation process and conduct experiments to evaluate the performance of various KD methods with and without CIP.
Results
The integration of the Complex Interaction Penalty consistently improves the performance of diverse KD methods, showing significant enhancements on both in-domain and out-of-distribution benchmarks. The findings indicate that methods achieving higher sparsity in complex interactions yield better performance.
Implications
This research provides insights into the interpretability of knowledge distillation in LLMs, guiding future developments in model compression techniques. The proposed CIP could be applied to enhance various KD methods, potentially leading to more efficient and effective language models.
Application of machine learning to monster level prediction in tabletop RPG game design
Theory
Interpretability
- Introduction of the first dataset for TTRPG monster-level prediction, fostering further research.
- Formalization of monster level prediction as a tabular ordinal regression problem.
- Comprehensive benchmarking of 16 models, demonstrating the superiority of tree-based ensembles.
- Use of domain-specific evaluation schemes for realistic model generalization assessment.
Read more
Application of machine learning to monster level prediction in tabletop RPG game design
Summary
This paper addresses the challenge of designing balanced adversaries in tabletop role-playing games (TTRPGs), specifically focusing on monster level prediction in the Pathfinder Second Edition system. The authors introduce a novel dataset specifically created for TTRPG monster-level prediction, derived from publicly available data. They frame the prediction task as a tabular ordinal regression problem and benchmark various models, including classical regression methods, dedicated ordinal regression algorithms, and neural networks with ordinal-aware losses. The evaluation employs realistic design workflows, utilizing chronological and expanding-window protocols alongside complementary metrics. The findings reveal that tree-based ensemble models significantly outperform linear and neural approaches, achieving near-perfect ordinal ranking and high predictive accuracy. Additionally, explainable AI analyses indicate that the models align well with human intuition and game rules, suggesting that machine learning can effectively assist in monster balancing and broader TTRPG system design.
Methodology
The authors developed a dataset for TTRPG monster-level prediction and framed the problem as tabular ordinal regression. They compared various models, including classical regression with rounding, dedicated ordinal regression algorithms, and neural networks with ordinal loss functions. Evaluation was conducted using time-based expanding-window train-test splits and metrics such as macro-averaged MAE.
Results
Tree-based ensemble models achieved near-perfect ordinal ranking and high predictive accuracy, outperforming linear models and neural networks. The models demonstrated robustness and alignment with human intuition, as confirmed by explainable AI analyses.
Implications
The findings suggest that machine learning can serve as an effective tool for game designers, enabling quicker and more accurate monster level estimations, thereby enhancing the game design process and improving player experiences in TTRPGs.
Dimensionality Reduction Meets Network Science: Sensemaking on UMAP's kNN Graph
Graph Learning
- UMAP's kNN graph is a valuable resource for data analysis, often overlooked in favor of 2D projections.
- Standard graph algorithms can be effectively applied to the kNN graph to enhance understanding of data structure.
- PageRank, k-core decomposition, and clustering coefficient provide insights into data representativeness, density, and local cohesion.
- Graph-based analyses on MNIST and Fashion MNIST datasets show competitive performance compared to traditional methods.
Read more
Dimensionality Reduction Meets Network Science: Sensemaking on UMAP's kNN Graph
Summary
This paper explores the underutilized potential of the k-nearest neighbor (kNN) graph constructed by UMAP, a popular dimensionality reduction technique. While UMAP is primarily used for visualizing high-dimensional data in a 2D scatter plot, the authors argue that the internal kNN graph retains significant structural information about the data manifold that is lost during the projection process. The paper presents three standard graph algorithms—PageRank, k-core decomposition, and clustering coefficient—to analyze this kNN graph, enhancing data sensemaking capabilities. The authors demonstrate that these graph-based analyses can identify representative data points, reveal dense core regions, and detect cohesive neighborhoods, offering insights that the 2D scatter plot alone cannot provide. Through quantitative and qualitative evaluations on MNIST and Fashion MNIST datasets, the authors show that their graph-based methods are competitive with or complementary to existing clustering and exemplar selection techniques, highlighting the importance of retaining the kNN graph as a first-class analytical resource.
Methodology
The authors applied three graph algorithms—PageRank for identifying representative points, k-core decomposition for revealing dense regions, and clustering coefficient for detecting cohesive neighborhoods—on UMAP's kNN graph. They conducted quantitative and qualitative evaluations using MNIST and Fashion MNIST datasets to compare the effectiveness of these methods against traditional clustering techniques like k-medoids and HDBSCAN.
Results
The results indicated that PageRank-selected points achieved superior class balance compared to k-medoids and were competitive in representativeness and classification accuracy. The k-core decomposition revealed a hierarchical structure in the data that discrete clustering methods could not capture, while the clustering coefficient identified distinct micro-clusters of highly similar data points.
Implications
The findings suggest that leveraging the kNN graph can significantly enhance data analysis workflows in various domains, particularly in exploratory data analysis and clustering tasks. This approach encourages researchers to reconsider the utility of intermediate representations in dimensionality reduction techniques.
Sticky Routing: Training MoE Models for Memory-Efficient Inference
NLP
Large Language Models
Efficient ML
- Introduction of StickyMoE, a routing consistency loss for MoE models.
- Significant reduction in expert switch rates and cache misses during inference.
- No architectural modifications required, only a single hyperparameter added.
- Empirical results show improved perplexity and efficiency in MoE models.
Read more
Sticky Routing: Training MoE Models for Memory-Efficient Inference
Summary
This paper introduces StickyMoE, a novel approach to improve the efficiency of Mixture-of-Experts (MoE) models during inference by addressing the issue of expert switching. Traditional MoE models activate a sparse subset of experts for each token, but often different experts are activated for consecutive tokens, leading to inefficiencies due to constant weight swapping between memory types. Existing solutions either focus on system-level caching or post-hoc router fine-tuning, which do not address the underlying problem during pretraining. StickyMoE proposes a differentiable routing consistency loss that penalizes abrupt expert switches between adjacent tokens, encouraging the router to maintain consistent expert assignments across semantically coherent spans. This method requires no architectural changes and introduces a single hyperparameter, allowing for co-adaptation of expert representations and routing decisions from the start of training. Experimental results demonstrate that StickyMoE reduces the expert switch rate by up to 59% and improves perplexity on medium-sized models while also decreasing cache misses by up to 3.92 times, outperforming post-hoc fine-tuning methods. The findings suggest that instilling routing temporal locality during training significantly enhances the performance of MoE models on memory-constrained devices.
Methodology
The methodology involves introducing a routing consistency loss, which is a differentiable â„“2 penalty on the gate distributions of consecutive tokens. This loss is combined with standard cross-entropy loss and a load-balancing auxiliary loss to optimize for temporal locality in routing. The approach is architecture-agnostic and can be applied to any standard top-k MoE model without requiring changes to the router structure.
Results
Experiments conducted on small and medium MoE language models show that StickyMoE reduces the expert switch rate by up to 59% and improves perplexity on the medium model. Additionally, it reduces cache misses by up to 3.92 times compared to post-hoc fine-tuning methods, demonstrating a significant enhancement in memory efficiency and overall model performance.
Implications
The implications of this work suggest that by optimizing routing consistency during training, MoE models can achieve better performance on edge devices where memory constraints are critical. This approach can lead to more efficient deployment of large language models in practical applications, enhancing their usability in real-world scenarios.
Unlocking Temporal Generalization in Hamiltonian Video Dynamics Models
Generative Models
Robotics
Time Series
- World models typically lack the ability to generalize across different temporal resolutions due to fixed training step sizes.
- Hamiltonian Generative Networks (HGN) can predict dynamics based on continuous-time energy functions but face challenges in non-conservative settings.
- The authors identify specific failure modes in HGN rollouts and propose targeted solutions to enhance temporal generalization.
- The extended HGN framework can predict stable dynamics at temporal resolutions outside the training regime.
Read more
Unlocking Temporal Generalization in Hamiltonian Video Dynamics Models
Summary
This paper addresses the limitations of world models that predict physical dynamics using a fixed discrete-time step, which restricts their ability to generalize across different temporal resolutions. The authors focus on Hamiltonian Generative Networks (HGN), which are designed to ground predictions in a continuous-time energy function, allowing for more flexible temporal predictions. However, the authors identify that HGN struggles with temporal generalization in non-conservative environments, leading to failures such as latent magnitude growth and truncation errors when extrapolating beyond the training step size. To tackle these issues, the authors propose targeted fixes for each failure mode and extend HGN with a port-Hamiltonian structure to better handle externally forced, dissipative environments. Their analysis includes strategies for enabling temporal generalization in continuous-time video generation, demonstrating stable dynamics predictions at various temporal resolutions beyond the training distribution.
Methodology
The authors analyze the temporal generalization capabilities of HGN by examining interpolation and extrapolation at different step sizes. They identify failure modes in non-conservative environments and propose a port-Hamiltonian structure to address these issues. The methodology includes detailed analysis and recommendations for enabling temporal generalization in continuous-time video generation.
Results
The proposed solutions successfully stabilize dynamics predictions at temporal resolutions beyond the training distribution, addressing the identified failure modes of latent magnitude growth and truncation error accumulation. The extended HGN framework demonstrates improved performance in predicting dynamics in externally forced, dissipative environments.
Implications
The findings have significant implications for hierarchical planning, sim-to-real transfer, and applications in scientific simulations and game engines, where querying dynamics at multiple timescales is essential. The enhanced temporal generalization capabilities can lead to more robust and flexible models in various domains.
Certified Interventional Fidelity: Anytime-Valid, Adaptive Evaluation of Causal Claims in Mechanistic Interpretability
Interpretability
- CIF provides a statistical layer for interventional interpretability evaluations, ensuring claims are backed by uncertainty quantification.
- The framework allows for adaptive evaluation while maintaining the original target estimand through bounded mixture importance weighting.
- CIF introduces anytime-valid confidence sequences, which remain valid under repeated monitoring of results.
- The methodology significantly reduces certification costs by 10-30 times using variance-adaptive betting sequences.
Read more
Certified Interventional Fidelity: Anytime-Valid, Adaptive Evaluation of Causal Claims in Mechanistic Interpretability
Summary
This paper introduces Certified Interventional Fidelity (CIF), a novel statistical framework designed to enhance the evaluation of causal claims in mechanistic interpretability. Traditional methods often summarize interventional evaluations with point estimates, which can obscure the reliability of fidelity claims due to finite sampling and evaluation choices. CIF reformulates interventional evaluations as causal estimands, providing confidence intervals and anytime-valid confidence sequences that adapt to ongoing evaluations. The methodology includes the use of Hoeffding-style sequences and variance-adaptive betting sequences, significantly reducing certification costs. CIF is applied to MNIST abstractions and GPT-2 Small IOI circuits, demonstrating its ability to certify high-fidelity claims, identify unsupported method differences, and clarify sensitivity to intervention distributions. The framework emphasizes the importance of uncertainty quantification in interpretability workflows, allowing for adaptive evaluation without compromising the integrity of causal claims.
Methodology
CIF reformulates interventional evaluations as causal estimands, providing confidence intervals and anytime-valid confidence sequences. It employs Hoeffding-style sequences and variance-adaptive betting sequences for efficient certification. The framework supports adaptive evaluation through bounded mixture importance weighting, allowing evaluators to focus on likely failures or impactful interventions.
Results
CIF was evaluated on MNIST neural abstractions and GPT-2 Small IOI circuits, successfully certifying high-fidelity claims and revealing instances where method differences lacked statistical support. The framework also highlighted the sensitivity of results to the choice of intervention distribution.
Implications
The introduction of CIF has significant implications for the field of mechanistic interpretability, providing a robust framework for evaluating causal claims. It encourages more rigorous statistical practices in interpretability research, potentially leading to more reliable and actionable insights from machine learning models.
A Practical Investigation of Training-free Relaxed Speculative Decoding
NLP
Large Language Models
Efficient ML
- Relaxed speculative decoding can yield speed-ups but requires careful capability evaluation.
- A unified framework for understanding various relaxed speculative decoding methods is presented.
- Benchmarking of relaxed approaches reveals performance trade-offs compared to strict speculative decoding.
- Many relaxed methods depend on the quality of the drafter model, limiting their applicability.
Read more
A Practical Investigation of Training-free Relaxed Speculative Decoding
Summary
This paper investigates training-free relaxed speculative decoding techniques for accelerating sampling from autoregressive large language models (LLMs). Speculative decoding utilizes a faster auxiliary model to draft tokens, which are then verified by the LLM, traditionally preserving the sampling distribution. The authors explore the potential benefits of relaxing this strict guarantee, which can lead to increased speed and capability trade-offs. They unify existing methods into a shared framework, benchmark various approaches, and provide empirical findings for practitioners. Key contributions include a primer on strict speculative decoding, a taxonomy of relaxed speculative decoding methods, and practical takeaways regarding the evaluation of capability and performance in relaxed approaches. The findings indicate that while relaxed methods can enhance speed, they often require careful capability evaluation and may not be suitable for lightweight drafters.
Methodology
The authors present a technical introduction to strict speculative decoding and develop a unified framework for relaxed speculative decoding. They benchmark various relaxed approaches using modern drafter-verifier pairs and reasoning benchmarks, allowing for fair comparisons across methods and their respective narratives.
Results
The benchmarking results show that while relaxed speculative decoding can provide speed advantages, it often necessitates significant capability evaluation. The authors find that many relaxed methods are not well-suited for lightweight drafters, which may limit their practical application.
Implications
The findings suggest that practitioners should carefully evaluate the trade-offs between speed and capability when implementing relaxed speculative decoding methods. The unified framework and empirical insights can guide future research and application in accelerating LLM inference.
Present but Rescaled: Chat-to-Agent Transfer of Additive Activation Steering
NLP
Large Language Models
Generative Models
- Additive activation steering shows real but rescaled transfer from chat to agent contexts.
- The injected direction survives with near-full strength, but behavioral outcomes reset per model and context.
- Amplification and attenuation of steering effects vary significantly across different models.
- Directional ablation does not amplify, highlighting a specific mechanism for additive injection.
Read more
Present but Rescaled: Chat-to-Agent Transfer of Additive Activation Steering
Summary
This paper investigates the effectiveness of additive activation steering when transferring from single-turn chat contexts to multi-turn ReAct agent contexts. The study reveals that while the injected activation direction maintains its strength across different settings, the behavioral outcomes are context-dependent and vary significantly across models. The author employs a matched-information design to systematically analyze the transfer, ensuring that the same items are tested in both chat and ReAct formats. The findings indicate a dissociation between the representational survival of the injected direction and its behavioral coupling, with some models amplifying the intended steering effects while others attenuate them. The results highlight the unpredictability of additive steering in agentic deployments, suggesting that safety cannot be universally assumed across different models. The paper also introduces a novel methodology for analyzing the effects of additive steering, emphasizing the need for continuous re-assertion of steering directions during generation.
Methodology
The study employs a matched-information five-rung ladder design, contrasting plain single-turn chat with multi-turn ReAct episodes. It utilizes a representation read-out at a late layer to measure the strength of the injected direction and a setting-invariant parser-based metric for behavioral measurement. Random-direction controls are applied to ensure specificity of the observed effects.
Results
The results demonstrate that the injected direction maintains a strength ratio of 0.83 to 1.16 across different model families. Behavioral coupling varies, with amplification observed in some models (e.g., Qwen2.5-7B with T = 1.45) and attenuation in others (e.g., Yi-1.5-9B with T = 0.43). A significant gain difference of 20.1 points is noted between additive injection and directional ablation, indicating a unique mechanism at play.
Implications
The findings suggest that additive activation steering can lead to unpredictable outcomes in agentic deployments, raising concerns about safety and reliability. The study emphasizes the need for careful evaluation of steering methods in multi-turn contexts and highlights the importance of understanding model-specific behaviors.
Optimal Learning Rate Scaling Depends on Data in Deep Scalar Linear Networks
Theory
Optimization
- Optimal learning rate scaling in deep scalar linear networks is data-dependent.
- Data-agnostic scaling rules fail to transfer effectively across different depths.
- The proposed data-dependent scaling leads to a constant linear convergence rate.
- The findings extend to deep scalar linear networks with residual connections.
Read more
Optimal Learning Rate Scaling Depends on Data in Deep Scalar Linear Networks
Summary
This paper investigates the gradient descent dynamics of deep scalar linear networks, demonstrating that optimal learning rate scaling is inherently data-dependent. The authors show that traditional data-agnostic scaling rules do not effectively transfer across different depths of the network. Instead, they propose a data-dependent scaling rule that leads to a constant linear convergence rate regardless of depth, including the case of infinite depth. The study also extends the findings to deep scalar linear networks with residual connections, confirming that the optimal learning rate scaling remains data-dependent. By analyzing the learning dynamics using exact solutions expressed through special functions, the authors provide insights into how depth affects learning rates and convergence in deep networks, challenging existing assumptions about hyperparameter transfer across depths.
Methodology
The authors analyze the gradient descent dynamics of deep scalar linear networks by deriving exact solutions for any integer depth. They utilize special functions, including the hypergeometric function and the Lambert W function, to express the learning dynamics. The study incorporates a balanced initialization scheme and examines the impact of data on learning rate scaling.
Results
The research reveals that the optimal learning rate scaling is not only dependent on the data but also leads to a consistent linear convergence rate across all depths of the network. This contrasts with traditional data-agnostic scaling methods, which do not yield the same results. The extension to networks with residual connections further supports the data-dependent nature of optimal learning rates.
Implications
These findings suggest that practitioners should consider data characteristics when selecting learning rates for deep networks, as this can significantly impact convergence rates and overall model performance. The results may influence future research on hyperparameter optimization and the design of deep learning architectures.
Risk-Aware General-Utility Markov Decision Processes
Reinforcement Learning
Robotics
Optimization
- Introduction of risk-aware GUMDPs that allow for a trade-off between expected performance and risk aversion.
- Development of an MCTS-based approach for solving risk-aware GUMDPs with provable accuracy.
- Experimental validation of the proposed method across diverse tasks, highlighting its versatility and effectiveness.
Read more
Risk-Aware General-Utility Markov Decision Processes
Summary
This paper introduces and formalizes the concept of risk-aware General-Utility Markov Decision Processes (GUMDPs), where agents optimize a risk measure of the distribution of objective values based on state visitation frequencies. The authors focus on the entropic risk measure (ERM) as a means to balance expected performance with risk aversion. They propose a Monte Carlo Tree Search (MCTS)-based approach for solving risk-aware GUMDPs, ensuring solutions can be achieved with any desired accuracy. Experimental results demonstrate the effectiveness of their method across various tasks, including standard MDPs, maximum state entropy exploration, imitation learning, and multi-objective MDPs, showcasing the ability to optimize for a range of risk-aware behaviors.
Methodology
The authors propose a framework for risk-aware GUMDPs and utilize an MCTS-based online planning technique to solve these processes. The focus is on optimizing the entropic risk measure (ERM) to account for risk in decision-making.
Results
The experimental results indicate that the proposed MCTS-based approach successfully optimizes for a spectrum of risk-aware behaviors in various contexts, demonstrating its applicability in standard MDPs, exploration tasks, imitation learning, and multi-objective scenarios.
Implications
The findings suggest that incorporating risk measures into GUMDPs can enhance decision-making in uncertain environments, making it relevant for applications in robotics, finance, and other fields where risk management is critical.
A Survey on the Green Development of Large Models: From Resource-Efficient Architectures to Hardware-Software Co-Design
Efficient ML
- The survey emphasizes the need for resource-efficient architectures in large AI models to mitigate environmental impacts.
- It highlights the importance of hardware-software co-design for optimizing energy consumption and computational efficiency.
- The paper reviews various strategies for efficient model construction and deployment, including sparsification and data-efficient learning.
- Applications of large models in sustainability-critical areas are explored, showcasing their potential for real-world impact.
Read more
A Survey on the Green Development of Large Models: From Resource-Efficient Architectures to Hardware-Software Co-Design
Summary
This survey addresses the pressing issues of computational costs, energy consumption, and environmental sustainability associated with the rapid expansion of large-scale AI models. It provides a comprehensive overview of the green development strategies for these models, focusing on resource-efficient architectures and hardware-software co-design. The authors systematically review advancements in efficient model construction, including attention operator optimization, linear-complexity architectures, and model sparsification. They also discuss training and deployment strategies such as data-efficient learning and parameter-efficient fine-tuning. The survey highlights the importance of energy-efficient AI hardware and explores applications of large models in sustainability-critical domains like remote sensing and national-scale infrastructure. Key challenges and future directions are identified, emphasizing the need for continual learning paradigms and standardized evaluation protocols. The paper aims to present a holistic roadmap for the sustainable development of large models, integrating algorithmic improvements with hardware considerations.
Methodology
The authors adopt a top-down perspective to systematically review the green development of large models, organizing their findings into a triangular framework that encompasses resource-efficient architectures, optimized training and deployment strategies, and hardware-software co-design. They analyze recent advancements in model construction and training techniques while considering the implications of hardware capabilities on model design.
Results
The survey identifies significant advancements in efficient model architectures and training strategies that can reduce computational costs and energy consumption. It also highlights the role of AI hardware in supporting these advancements and discusses the environmental implications of large model training. The authors provide a unified view of how efficiency gains can emerge from collaborative efforts across different layers of model development.
Implications
The findings of this survey have important implications for the future of AI development, particularly in promoting sustainable practices in the design and deployment of large models. By integrating efficiency considerations into both algorithmic and hardware design, the research encourages the development of AI systems that are not only powerful but also environmentally responsible.
NFTR: From Provable Mode-Averaging to Geodesic Subgoal Selection in Offline Goal-Conditioned RL
Reinforcement Learning
Robotics
Theory
- NFTR addresses optimistic bias and mode collapse in HIQL by using Normalizing Flows for subgoal selection.
- The triangle-slack score provides a mechanism to downweight unreliable subgoals based on geometric consistency.
- NFTR preserves population-level monotonic improvement and allows for a detailed suboptimality decomposition.
- Empirical evaluations indicate substantial performance improvements over HIQL in various offline GCRL tasks.
Read more
NFTR: From Provable Mode-Averaging to Geodesic Subgoal Selection in Offline Goal-Conditioned RL
Summary
This paper introduces NFTR (Normalizing Flows subgoal policies with Triangle-slack Reweighting), a novel approach to offline goal-conditioned reinforcement learning (GCRL) that addresses the limitations of Hierarchical Implicit Q-Learning (HIQL). HIQL suffers from two main issues: optimistic bias due to stochastic environments and mode collapse from using a unimodal Gaussian policy for subgoal selection. NFTR replaces the Gaussian policy with a conditional Normalizing Flow, enabling the modeling of multi-modal subgoal distributions. Additionally, it introduces a triangle-slack score that corrects the advantage-weighted regression (AWR) weights, downweighting subgoals that are geometrically inconsistent or have high detour costs. The triangle-slack score vanishes on geodesics in deterministic Markov Decision Processes (MDPs) and provides a conservative upper bound on composability violations in stochastic settings. The paper demonstrates that NFTR maintains population-level monotonic improvement and offers a three-term suboptimality decomposition, thus ensuring stable subgoal selection. Empirical results show that NFTR significantly outperforms HIQL across various tasks in OGBench, including stochastic, stitching, and manipulation tasks.
Methodology
The authors propose NFTR, which employs Normalizing Flows to model multi-modal subgoal distributions instead of a unimodal Gaussian. They introduce a triangle-slack score derived from a quasimetric that corrects the AWR weights, ensuring that subgoals with high detour costs are downweighted. The methodology includes theoretical characterizations of triangle-slack in both deterministic and stochastic MDPs, along with a three-term suboptimality decomposition for the RWDR objective.
Results
NFTR demonstrates significant improvements over HIQL in offline GCRL tasks, particularly in environments characterized by stochastic dynamics and complex subgoal structures. The empirical results on OGBench show that NFTR effectively avoids the Gaussian collapse and maintains stability under varying conditions.
Implications
The NFTR framework has potential applications in robotics and autonomous systems, where reliable goal-reaching from static datasets is crucial. By improving subgoal selection and ensuring stability in stochastic environments, NFTR can enhance the performance of agents in real-world scenarios where online data collection is limited or risky.
Action-Factored Multi-Agent Reinforcement Learning for Scalable Quantum Device Tuning
Reinforcement Learning
Optimization
Robotics
- Introduction of QADAPT, a multi-agent reinforcement learning framework for quantum device tuning.
- Adaptive action-space factorization reduces cross-agent interference and improves sample efficiency.
- Zero-shot generalization allows the framework to adapt to unseen quantum device configurations.
- Empirical results demonstrate superior performance compared to existing tuning methods.
Read more
Action-Factored Multi-Agent Reinforcement Learning for Scalable Quantum Device Tuning
Summary
This paper presents QADAPT, a novel multi-agent reinforcement learning (MARL) framework designed for the efficient tuning of electrostatically-defined quantum-dot arrays. The authors address the challenges of strong parameter cross-talk and non-stationary environments that complicate the tuning process. By employing an adaptive action-space factorization approach, QADAPT learns a factored representation of the action space online, allowing agents to operate with reduced interference. This modular architecture supports centralized training with decentralized execution, enabling agents to act independently based on local observations while sharing policies. The framework demonstrates zero-shot generalization across various quantum device sizes and configurations, achieving consistent convergence to target regimes without retraining. The empirical evaluation shows that QADAPT outperforms existing state-of-the-art methods in sample efficiency and tuning accuracy, providing a scalable solution for the rapid calibration of large-scale quantum processors.
Methodology
The authors developed a Kalman-guided pipeline for estimating the local gate-to-dot capacitance matrix, which facilitates the construction of a virtual action basis. They formulated the tuning problem as a cooperative Decentralized Partially Observable Markov Decision Process (Dec-POMDP) and implemented a modular actor-critic architecture with gate-type-specific parameter sharing. This approach allows for independent learning of policies while maintaining efficiency in high-dimensional action spaces.
Results
QADAPT achieved zero-shot generalization across different system sizes and configurations, maintaining a consistent number of convergence steps to reach target regimes. The empirical evaluations showed that QADAPT outperformed various state-of-the-art baselines in terms of tuning accuracy and sample efficiency.
Implications
The proposed framework has significant implications for the rapid calibration of large-scale quantum processors, potentially accelerating advancements in quantum computing technology. It also demonstrates the applicability of multi-agent reinforcement learning in complex control scenarios beyond traditional domains.
SafeExplorer: An Unbiased Policy Gradient for Reinforcement Learning with Recovery Interventions
Reinforcement Learning
Robotics
Theory
- Introduces SafeExplorer, an unbiased policy gradient method for safe reinforcement learning.
- Addresses the bias introduced by mixed-policy rollouts when using a recovery policy.
- Empirically reduces training-time falls by factors of 233×, 48×, and 26× in different environments compared to standard PPO.
- Includes a closed-form value for recovery-triggering states and an imitation loss for successful recovery actions.
Read more
SafeExplorer: An Unbiased Policy Gradient for Reinforcement Learning with Recovery Interventions
Summary
The paper introduces SafeExplorer, a novel reinforcement learning (RL) algorithm designed to minimize falls during training on physical robots, addressing the challenges posed by costly failures in real-world environments. Traditional methods often rely on a recovery policy that intervenes when the agent leaves a designated safe region, but this can introduce bias in policy updates due to mixed-policy rollouts. SafeExplorer proposes an unbiased policy-gradient estimator that operates only at safe timesteps, effectively decoupling the main policy from the recovery policy, even when the latter is deterministic. The algorithm includes additional components to enhance learning efficiency: a closed-form value for states triggering recovery actions and an imitation loss that reinforces successful recovery actions. Empirical results demonstrate that SafeExplorer significantly reduces training-time falls across various environments while achieving or surpassing the performance of standard proximal policy optimization (PPO).
Methodology
SafeExplorer modifies the proximal policy optimization (PPO) algorithm by introducing an unbiased policy-gradient estimator that only uses the score function at safe timesteps. It avoids evaluating the recovery policy's density, making it applicable to both deterministic and stochastic recovery policies. The algorithm also incorporates a closed-form value for recovery-triggering states and an outcome-gated compatibility regularizer to enhance learning efficiency.
Results
In experiments across three environments (HalfCheetah, Ant, and Unitree Go1), SafeExplorer reduced training-time falls by factors of 233×, 48×, and 26× compared to standard PPO, while matching or exceeding PPO's final reward. Notably, on the Ant environment, where the recovery policy was less reliable, SafeExplorer was the only method to achieve 80% of the best final reward.
Implications
SafeExplorer's approach can significantly improve the safety and efficiency of training reinforcement learning agents in real-world applications, particularly in robotics, where minimizing physical failures is crucial. The methods developed could be applied to other domains requiring safe exploration and intervention strategies.
Prompt-Driven Exploration
Reinforcement Learning
Large Language Models
Robotics
- PDE enables global exploration in RL by modifying natural language prompts.
- The method allows RL to escape weak initial policies by refining prompts based on rollout performance.
- PDE improves sample efficiency and success rates in tasks with sparse rewards.
- The approach is validated on manipulation and reasoning tasks, as well as coding tasks.
Read more
Prompt-Driven Exploration
Summary
The paper introduces Prompt-Driven Exploration (PDE), a novel exploration strategy in reinforcement learning (RL) that leverages large language models (LLMs) and vision-language-action (VLA) models to enhance exploration capabilities. Traditional RL methods often rely on stochastic action perturbations, which limit exploration to local changes and fail to escape weak initial policies. PDE addresses this by conditioning the policy on natural language prompts, allowing for global changes in behavior through prompt modifications. The authors propose a method where a vision-language model (VLM) analyzes rollout videos, diagnoses policy responses, and refines prompts to improve future performance. This approach effectively implements posterior sampling at the prompt level, enabling RL to learn successful policies even from zero-reward starts. The authors demonstrate the effectiveness of PDE across various manipulation and reasoning tasks, showing significant improvements in sample efficiency and success rates compared to standard action-noise exploration methods.
Methodology
The authors developed Prompt-Driven Exploration (PDE) by utilizing a vision-language model (VLM) to analyze the outcomes of policy rollouts. The VLM diagnoses the policy's responses and iteratively refines the prompts used to generate rollouts. This process mimics posterior sampling in RL, allowing the VLM to maintain and update a distribution over useful prompts based on observed trajectories, without requiring gradient updates.
Results
PDE was evaluated on the LIBERO and LIBERO-PRO benchmarks, showing that it achieved higher success rates and improved sample efficiency compared to traditional action-noise exploration methods. The results indicated that PDE could successfully learn policies from zero-reward starts and outperform standard RL techniques in various task difficulties.
Implications
The findings suggest that integrating prompt engineering into RL can significantly enhance exploration capabilities, particularly in scenarios with sparse rewards. This approach has potential applications in robotics, manipulation tasks, and other areas where RL is applied, especially when starting from weak initial policies.
LieBN: Batch Normalization over Lie Groups
Theory
Optimization
- Introduction of LieBN, a general framework for Riemannian Batch Normalization over Lie groups.
- Development of a novel right-invariant metric on the SPD manifold, enhancing normalization capabilities.
- Demonstration of LieBN's effectiveness across multiple geometries and tasks.
- Theoretical guarantees for controlling both Riemannian mean and variance.
Read more
LieBN: Batch Normalization over Lie Groups
Summary
The paper introduces LieBN, a novel framework for Riemannian Batch Normalization (RBN) specifically designed for Lie groups, addressing the limitations of existing normalization methods that are either specific to certain manifolds or ineffective in normalizing manifold-valued sample distributions. LieBN utilizes left- and right-invariant metrics inherent to Lie groups, providing theoretical guarantees for controlling both the Riemannian mean and variance. The framework is instantiated across nine distinct geometries, including four on the Symmetric Positive Definite (SPD) manifold, one on rotation matrices, and four on full-rank correlation matrices. A significant contribution is the introduction of a new right-invariant metric on the SPD manifold, termed Cholesky Right Invariant Metric (CRIM), which is the first of its kind. The effectiveness of LieBN is validated through extensive experiments on various manifolds, demonstrating its applicability in tasks such as radar recognition, human action recognition, and electroencephalography (EEG) classification. The authors provide a PyTorch-compatible toolbox for practical implementation.
Methodology
The authors propose a framework for Riemannian Batch Normalization that leverages left- and right-invariant metrics on Lie groups. They instantiate this framework across various geometries, including SPD manifolds, rotation matrices, and correlation matrices. The methodology includes the introduction of a new right-invariant metric and the generalization of existing Lie group structures through matrix power deformation. Extensive experiments are conducted to validate the framework's effectiveness across different tasks.
Results
The experiments demonstrate that LieBN effectively normalizes the distribution of manifold-valued samples, achieving improved performance in tasks such as radar recognition, human action recognition, and EEG classification. The proposed framework outperforms existing methods, providing better control over the Riemannian mean and variance.
Implications
LieBN has significant implications for machine learning tasks involving manifold-valued data, offering a robust normalization technique that can enhance the performance of deep learning models operating on complex geometries. Its availability as a PyTorch module facilitates broader adoption in various applications.
Accelerating GPU Inference of Large Language Models with Moderately Unstructured Sparse Weight Matrices
NLP
Large Language Models
Efficient ML
- Introduces a three-layer matrix storage format for efficient SpMM under moderate sparsity.
- Develops a co-optimized SpMM kernel that utilizes both sparse tensor cores and CUDA cores.
- Achieves up to 1.64× kernel-level speedup over existing methods and outperforms dense matrix multiplication.
- Addresses key challenges in sparse LLM inference, including tensor core compatibility and metadata overhead.
Read more
Accelerating GPU Inference of Large Language Models with Moderately Unstructured Sparse Weight Matrices
Summary
This paper addresses the challenge of high inference costs associated with large language models (LLMs) by proposing an efficient method for GPU inference that leverages moderately unstructured sparse weight matrices. The authors identify that existing GPU kernels for sparse matrix multiplication (SpMM) fail to outperform dense counterparts at moderate sparsity levels, typically around 50%. To overcome this limitation, they introduce a novel three-layer matrix storage format designed to optimize SpMM performance. This format includes a Sparse-TC layer for utilizing sparse tensor cores, a Slot-Filling layer for efficient matrix compression and on-chip decoding, and a lightweight Residual Layer to ensure accurate computation. The proposed SpMM kernel effectively combines the strengths of sparse tensor cores and CUDA cores, allowing for an efficient execution pipeline that overlaps computation with memory access. The results demonstrate that their method achieves significant speedups, outperforming dense matrix multiplication on modern GPUs equipped with high-bandwidth memory (HBM).
Methodology
The authors propose a three-layer matrix storage format that includes a Sparse-TC layer for leveraging sparse tensor cores, a Slot-Filling layer for efficient matrix compression, and a lightweight Residual Layer for accurate computation. They design a SpMM kernel that co-utilizes sparse tensor cores and CUDA cores, optimizing the execution pipeline to maximize memory bandwidth utilization and minimize decoding overhead.
Results
The proposed method achieves up to 1.64× kernel-level speedup over SpInfer and 1.41× end-to-end speedup over FlashLLM, marking the first instance of outperforming dense matrix multiplication on modern GPUs with HBM.
Implications
This work has significant implications for the deployment of large language models in resource-constrained environments, enabling faster inference without sacrificing model quality. It opens avenues for further research in optimizing sparse matrix operations in deep learning applications.
COBS: Cumulant Order Block Sparse Attention
Large Language Models
NLP
Efficient ML
- COBS formalizes block selection as ranking blocks by attention mass, improving selection accuracy.
- The method incorporates a second-order statistic to capture within-block covariance, enhancing performance.
- COBS achieves a mean score of 0.8195 on the 32k RULER benchmark, significantly outperforming the NSA baseline.
- The method uses only 1.21× the KV cache read traffic of the NSA baseline and 15.15× less than dense attention.
Read more
COBS: Cumulant Order Block Sparse Attention
Summary
The paper introduces COBS (Cumulant Order Block Sparse Attention), a novel method designed to enhance block sparse attention mechanisms in large language models (LLMs). The authors identify the limitations of existing block selection strategies, particularly those based on first-order approximations, which fail to capture the full complexity of attention mass. By formalizing block selection as a ranking of blocks based on their attention mass, the authors demonstrate that accurate selection can significantly improve performance. COBS builds on the Native Sparse Attention (NSA) framework, incorporating a second-order statistic for block selection that captures within-block covariance. This approach allows COBS to achieve performance close to dense attention while maintaining a lower key-value (KV) cache read traffic. The evaluation on the 32k RULER long-context retrieval benchmark shows a substantial improvement in mean scores, indicating that COBS effectively bridges the gap between traditional sparse and dense attention methods.
Methodology
The authors analyze the Native Sparse Attention (NSA) method to isolate the block selection process. They formalize block selection as ranking blocks by their attention mass and propose COBS, which utilizes a cumulant expansion to include second-order statistics in its block selector. This involves storing compressed covariance information for each block, allowing for more accurate approximations of attention mass without the need for full key reads.
Results
COBS significantly improves the mean score from 0.2999 (NSA baseline) to 0.8195 on the 32k RULER benchmark, approaching the dense attention score of 0.9040. It achieves this while only increasing KV cache read traffic by 1.21× compared to the NSA baseline and reducing it by 15.15× compared to dense attention.
Implications
The findings suggest that COBS can be effectively utilized in large language models to enhance performance in long-context tasks while maintaining efficiency in memory usage. This could lead to more scalable and effective implementations of LLMs in various applications, including natural language processing and other areas requiring attention mechanisms.
FairSelect: A Systematic Evaluation of Multi-Level and Intersectional Algorithmic Fairness
Theory
- FairSelect provides a systematic framework for evaluating fairness strategies across multiple modeling stages.
- The toolkit supports intersectional subgroup analysis, addressing disparities that arise from multiple demographic characteristics.
- Combining fairness interventions can lead to improved fairness outcomes, though results can vary significantly in practical applications.
- The study emphasizes the importance of context in assessing the effectiveness of fairness strategies.
Read more
FairSelect: A Systematic Evaluation of Multi-Level and Intersectional Algorithmic Fairness
Summary
The paper introduces FairSelect, a toolkit designed to systematically evaluate algorithmic fairness mitigation strategies across multiple stages of the machine learning lifecycle. Unlike traditional approaches that assess fairness in isolation and along single demographic axes, FairSelect enables the evaluation of fairness interventions in combination and supports intersectional subgroup analysis. The framework was validated using synthetic clinical datasets and a real-world clinical prediction task related to stroke risk among patients with atrial fibrillation. Results indicated that targeted fairness methods effectively reduced subgroup disparities, and combining strategies generally yielded greater fairness improvements with minimal utility tradeoffs. However, in clinical applications, the effectiveness of mitigation strategies varied significantly, with some combinations enhancing both fairness and predictive performance, while others were ineffective or detrimental. This highlights the complex, context-dependent interactions between fairness interventions, underscoring the need for a systematic approach to selecting and applying fairness strategies in clinical machine learning.
Methodology
The authors developed FairSelect, a toolkit that evaluates fairness mitigation strategies across pre-processing, in-processing, and post-processing stages. The framework was tested using synthetic datasets designed to simulate specific bias mechanisms and a real-world clinical dataset for stroke risk prediction. The evaluation included comparisons of fairness–utility tradeoffs across various configurations.
Results
Synthetic experiments demonstrated that targeted fairness methods reduced subgroup disparities, while combined strategies resulted in larger average improvements in fairness with modest impacts on utility. In the clinical prediction task, the effectiveness of mitigation strategies varied, with some combinations improving both fairness and predictive performance, while others did not yield positive results.
Implications
FairSelect provides a practical tool for researchers and practitioners in healthcare and other domains to systematically assess and select fairness strategies that enhance equity in machine learning models. Its focus on intersectional analysis may lead to better-informed clinical decision-making and improved patient outcomes.
Secure Decentralized Federated Learning via Gossip and Virtual Voting
Federated Learning
- gspDAG-FL provides a secure framework for decentralized federated learning that enhances resilience against adversarial participants.
- The framework utilizes gossip history for consensus, allowing for efficient model dissemination without central coordination.
- Finality is achieved through unique model-origin tuples, improving provenance tracking and filtering invalid updates.
- Experimental results indicate that gspDAG-FL maintains high learning quality while reducing coordination bottlenecks.
Read more
Secure Decentralized Federated Learning via Gossip and Virtual Voting
Summary
This paper introduces gspDAG-FL, a novel framework for secure decentralized federated learning (DFL) that enhances resilience against Byzantine and lazy participants while maintaining locality and reducing coordination costs. Traditional decentralized federated learning methods often lack provenance finality and can be vulnerable to malicious behaviors. gspDAG-FL addresses these issues by utilizing a gossip-based consensus mechanism that derives finality from the gossip history used for model dissemination. Nodes communicate model updates through one-hop neighbor gossip, while full nodes collect event certificates and accepted gossip proofs to reconstruct a directed acyclic graph (DAG). This structure allows for Hashgraph-style virtual voting, ensuring that finality is based on unique model-origin tuples rather than identical local parameter states. The framework also incorporates mechanisms for payload validation and private semantic audits to enhance resilience. The authors formalize the adversarial setting, proving the safety and conditional liveness of the control plane, and provide convergence guarantees for certified perturbed gossip under varying conditions. Experimental results demonstrate that gspDAG-FL achieves learning quality comparable to validation-based ledger FL while significantly improving throughput and maintaining high detection rates for invalid origins in mixed participation scenarios.
Methodology
The authors propose a gossip-based framework where nodes exchange model updates through one-hop neighbor communication. Full nodes collect event certificates and accepted gossip proofs to reconstruct a topology DAG, which is then used for virtual voting to achieve consensus. The framework incorporates mechanisms for validating payloads and conducting private audits before aggregation.
Results
Experiments conducted on MNIST classification and Penn Treebank language modeling show that gspDAG-FL achieves learning quality comparable to traditional validation-based ledger FL systems. It also demonstrates improved throughput and effective detection of invalid origins in scenarios with mixed Byzantine and lazy participation.
Implications
The gspDAG-FL framework has significant implications for secure decentralized federated learning applications, particularly in environments where data privacy and resilience against malicious participants are critical. It can be applied in various domains such as healthcare, finance, and any distributed systems requiring collaborative learning without centralized control.
TSRouter: Dynamic Modality-Model Selection for Time Series Reasoning
Time Series
Graph Learning
Multimodal
- TSROUTER leverages a graph-based approach to model complex interactions among tasks, queries, modalities, and models.
- The framework allows for dynamic selection of the most suitable modality and model, enhancing performance in time series reasoning.
- TSROUTER achieves significant improvements over baseline methods, with relative performance gains of 16% to 46%.
- It demonstrates strong zero-shot generalization capabilities, effectively handling new tasks and models without retraining.
Read more
TSRouter: Dynamic Modality-Model Selection for Time Series Reasoning
Summary
The paper introduces TSROUTER, a novel framework for dynamic modality-model selection in time series reasoning. Recognizing the complementary strengths of Large Language Models (LLMs) and Vision-Language Models (VLMs), TSROUTER aims to optimize the selection process based on the specific characteristics of tasks and queries. The framework constructs a heterogeneous graph that models interactions among tasks, queries, modalities, and models, allowing for context-aware routing. By framing the routing problem as a candidate scoring task, TSROUTER evaluates modality-model pairs based on user-defined performance and cost preferences. Comprehensive evaluations across four distinct time series reasoning tasks demonstrate that TSROUTER significantly outperforms existing baselines, achieving 16% to 46% relative improvements. Additionally, it exhibits robust zero-shot generalization capabilities to unseen models and tasks while optimizing for computational efficiency.
Methodology
TSROUTER constructs a heterogeneous graph comprising nodes for tasks, queries, modalities, and models. It uses a Graph Neural Network (GNN) to aggregate information across these nodes and scores candidate modality-model pairs based on user-defined performance and cost preferences. The system utilizes an LLM to profile nodes, enabling zero-shot generalization to new tasks and models.
Results
TSROUTER outperformed various baseline models across four time series reasoning tasks, achieving relative improvements ranging from 16% to 46%. The framework also demonstrated robust performance on unseen tasks and models, maintaining high efficiency and low computational costs.
Implications
The dynamic routing capabilities of TSROUTER could enhance applications in high-stakes domains such as education and scientific discovery, where accurate time series reasoning is crucial. Its cost-aware optimization may also lead to more efficient deployment of machine learning models in resource-constrained environments.
Scalable and Trustworthy Earth Observation Foundation Models
Computer Vision
Multimodal
- Foundation models (FMs) can be adapted for multiple downstream tasks in Earth Observation.
- Remote Sensing Foundation Models (RSFMs) require domain-specific adaptations due to unique EO data characteristics.
- Evaluation of RSFMs should consider modality-aware transfer and physical plausibility, not just benchmark accuracy.
- Two case studies demonstrate practical applications of RSFMs in environmental monitoring.
Read more
Scalable and Trustworthy Earth Observation Foundation Models
Summary
This paper discusses the development of Remote Sensing Foundation Models (RSFMs) tailored for Earth Observation (EO) data, emphasizing the need for domain-specific adaptations due to the unique characteristics of EO data. The authors review the principles guiding the design of RSFMs, including pretraining objectives, model architectures, and evaluation metrics. They highlight the challenges posed by the multimodal and geospatial nature of EO data, which differ significantly from conventional computer vision tasks. The paper also presents two case studies: one on predicting harmful algal blooms using physics-informed spectral masking, and another on optimizing the selection of environmental monitoring stations through reinforcement learning. These examples illustrate the application of RSFMs in real-world scenarios, emphasizing the importance of modality-aware transfer and physically plausible representations for reliable EO decision-making. The authors argue that future RSFMs should be evaluated not only on benchmark accuracy but also on their ability to adapt to various modalities and provide trustworthy outputs.
Methodology
The authors conducted a comprehensive review of existing RSFMs, synthesizing design principles, pretraining objectives, and adaptation protocols. They included case studies to demonstrate the practical application of these models in environmental monitoring, employing techniques such as physics-informed spectral masking and reinforcement learning.
Results
The review revealed that no single geospatial foundation model is universally optimal, and the authors provided evidence of the challenges in evaluating RSFMs due to inconsistent metrics. The case studies illustrated the effectiveness of RSFMs in addressing specific EO tasks, highlighting the importance of adapting models to the physical and operational constraints of the domain.
Implications
The findings suggest that RSFMs can significantly enhance the analysis and monitoring of environmental phenomena, but their design and evaluation must align closely with the unique characteristics of EO data. This work paves the way for more reliable and trustworthy applications of AI in Earth observation.
LLT: Local Linear Transformer for PDE Operator Learning
Efficient ML
Theory
Optimization
- LLT combines linear global attention with local spatial mixing to improve efficiency in PDE operator learning.
- The architecture incorporates coordinate and geometry information to enhance performance on PDE problems.
- LLT demonstrates competitive accuracy with lower relative L2 error compared to existing models.
- Significant reductions in training time per iteration make LLT a more efficient alternative for PDE simulations.
Read more
LLT: Local Linear Transformer for PDE Operator Learning
Summary
The paper introduces the Local Linear Transformer (LLT), a novel architecture designed for learning partial differential equation (PDE) operators. Traditional transformer models face significant limitations when applied to PDEs, primarily due to their quadratic scaling with the number of computational nodes and a lack of emphasis on local interactions. LLT addresses these challenges by combining linear global attention with local spatial mixing, while also incorporating coordinate and geometry information. The authors evaluate LLT on various PDE problems, including elasticity, plasticity, airfoil flow, pipe flow, and Darcy flow, using reference data generated from multiple discretization methods. The results demonstrate that LLT achieves competitive or lower relative L2 error compared to existing neural operator and transformer baselines. Additionally, LLT shows a significant reduction in wall-clock time per training iteration, improving efficiency by factors of 1.8 to 2.5 compared to the Transolver model. The architecture is further tested on a complex three-dimensional car aerodynamics dataset, showcasing its scalability and effectiveness across different mesh types and problem settings. Overall, LLT presents a promising solution for accurate and computationally efficient PDE operator learning.
Methodology
The LLT architecture employs kernelized linear attention for global communication and a local mixing path for spatial neighborhoods. It integrates coordinate encodings, distance-to-reference-grid encoding, and a skip-connected decoder to effectively learn PDE solution maps from input fields sampled at finite points.
Results
LLT achieves competitive or lower relative L2 error across multiple PDE problems and demonstrates a reduction in wall-clock time per training iteration by factors of 1.8 to 2.5 compared to the Transolver model. The model also scales effectively to a three-dimensional car aerodynamics dataset with 32,186 unstructured mesh points per sample.
Implications
The LLT architecture has the potential to enhance the efficiency and accuracy of numerical simulations in various scientific computing applications, particularly in fields requiring repeated solutions of PDEs under varying conditions. Its ability to handle both structured and unstructured meshes makes it versatile for real-world applications.
SLORR: Simple and Efficient In-Training Low-Rank Regularization
Efficient ML
Computer Vision
Large Language Models
- SLORR is SVD-free and does not require architectural changes, making it practical for modern neural networks.
- The framework provides GPU-efficient approximations for low-rank regularization, ensuring scalability.
- Empirical results show that SLORR improves post-training compressibility with minimal training overhead.
- SLORR maintains model performance better than traditional methods during compression.
Read more
SLORR: Simple and Efficient In-Training Low-Rank Regularization
Summary
The paper introduces SLORR, a novel framework for in-training low-rank regularization that addresses the limitations of existing methods which often require costly singular value decompositions (SVDs) or modify model architectures. SLORR operates directly on the original weight matrices without altering the architecture, making it stateless and efficient. The framework is instantiated with two variants based on the Hoyer sparsity metric and the nuclear norm, both of which utilize GPU-friendly approximations for the forward and backward passes. Empirical evaluations demonstrate that SLORR enhances the compressibility of models while maintaining performance, with minimal training overhead. Specifically, SLORR was tested on ImageNet-1K with various architectures, showing less than 8% overhead, and on large language models (LLMs) at 135M and 560M scales, where it preserved performance significantly better than unregularized models with less than 1% overhead.
Methodology
SLORR employs GPU-friendly approximations for low-rank regularization, specifically using polar factor approximations to compute the necessary spectral quantities for weight matrices. It includes two main variants: SLORR-Hoyer, based on the squared Hoyer sparsity metric, and SLORR-Nuc, based on the nuclear norm. The method is designed to be stateless and does not maintain cached quantities, allowing for efficient training without additional architectural modifications.
Results
SLORR was evaluated on ImageNet-1K, showing improved compressibility for ResNet-50, ViT-B/16, and ViT-L/16 with less than 8% training overhead. In LLM pretraining at 135M and 560M scales, SLORR-trained models exhibited significantly better performance retention post-compression compared to unregularized models, with less than 1% average training overhead.
Implications
The SLORR framework can be applied to various neural network architectures to enhance their compressibility without sacrificing performance, making it particularly useful for deploying models in resource-constrained environments. Its efficiency and simplicity could lead to broader adoption in both academic research and industry applications.
COAST: Context-Aware Differential Learning for Gene Expression Prediction in Spatial Transcriptomics
Computer Vision
- COAST integrates local and global context features for improved gene expression prediction.
- The framework employs a joint training objective combining absolute and differential regression.
- Experiments show consistent performance improvements over existing methods.
- COAST retains clinically meaningful prognostic information in gene expression representations.
Read more
COAST: Context-Aware Differential Learning for Gene Expression Prediction in Spatial Transcriptomics
Summary
The paper introduces COAST, a novel context-aware differential learning framework aimed at improving gene expression prediction from H&E stained histopathology images in the field of spatial transcriptomics. Traditional methods have focused on absolute expression prediction, often neglecting the importance of relative expression relationships between spatially adjacent spots. COAST addresses this gap by integrating local and global context features through type-specific modulation and employing a Transformer encoder to aggregate these features. This approach allows for the capture of both fine-grained local patterns and broader slide-level structures. The framework is trained using a joint objective that combines absolute expression regression with signed differential regression, which explicitly supervises the spatial relationships between target and context spots. Experimental results across multiple spatial transcriptomics datasets demonstrate that COAST consistently outperforms existing methods, showing improvements in correlation and distribution-based metrics. Additionally, the gene expression representations produced by COAST retain clinically relevant prognostic information, underscoring its practical utility in clinical applications.
Methodology
COAST utilizes a feature extraction process to obtain spot features from histopathology images, followed by context-specific feature modulation to adaptively condition local and global context features. A spatio-relational Transformer aggregates these features, and the model is trained with a joint objective that includes both absolute expression regression and signed differential regression to explicitly supervise spatial relationships.
Results
The experiments conducted on multiple spatial transcriptomics datasets indicate that COAST achieves significant improvements in correlation and distribution-based metrics compared to existing baseline methods. The reconstructed gene expression representations also demonstrate the retention of clinically relevant prognostic information.
Implications
COAST has the potential to enhance the predictive accuracy of gene expression from histopathology images, making spatial transcriptomics more accessible and practical for clinical applications. This could lead to better understanding and treatment of diseases by leveraging spatial gene expression data.
Beyond Backpropagation: Monte Carlo Method Can Train Deep Neural Networks
Optimization
Theory
Efficient ML
- Introduces a gradient-free Monte Carlo method for training deep neural networks.
- Demonstrates effectiveness without traditional techniques like batch normalization.
- Validates the method on various architectures and tasks, including deep networks and Transformers.
- Reveals substantial redundancy in deep networks and supports unconventional transfer functions.
Read more
Beyond Backpropagation: Monte Carlo Method Can Train Deep Neural Networks
Summary
This paper presents a novel approach to training deep neural networks using a simple Monte Carlo algorithm, which operates without the reliance on gradients, thus circumventing issues such as vanishing and exploding gradients commonly associated with backpropagation (BP). The proposed method involves randomly mutating network parameters and accepting changes that reduce the loss, demonstrating its effectiveness in training deep networks without the need for techniques like batch normalization or residual connections. The Monte Carlo method's flexibility allows it to handle various scenarios, including pure pruning training, discrete weights, and unconventional transfer functions like Gaussian. The feasibility of this approach is validated through experiments on deep networks exceeding 20 layers, wide networks with up to 16,384 hidden neurons, and a Transformer architecture, applied to tasks such as image classification (MNIST) and character-level language modeling (Tiny Shakespeare). This gradient-free method not only offers a complementary perspective on neural network training but also opens avenues for developing physically inspired deep learning systems.
Methodology
The methodology involves a Monte Carlo algorithm where network parameters are randomly mutated. If the mutation results in a lower loss, it is accepted; otherwise, it is discarded. This process can be applied to single or multiple parameters simultaneously, allowing for flexible training strategies, including weight pruning and discrete weight networks. The implementation is executed on a GPU, enhancing computational efficiency.
Results
The Monte Carlo method successfully trained deep networks with over 20 layers and wide networks with up to 16,384 hidden neurons. It was also effective in training a simple Transformer architecture on both MNIST for image classification and Tiny Shakespeare for language modeling, demonstrating competitive performance compared to traditional methods.
Implications
This research suggests that gradient-free methods can serve as viable alternatives to backpropagation, potentially leading to new insights into neural network training and self-organization. It may also inspire the development of more robust and physically inspired deep learning systems.
Shortcut Trajectory Planning for Efficient Offline Reinforcement Learning
Reinforcement Learning
Generative Models
Robotics
- Introduction of Shortcut Trajectory Planning (STP) for offline reinforcement learning.
- STP simplifies the training process by using a single-stage training of shortcut models.
- The framework allows for adjustable inference steps, enhancing planning efficiency.
- STP demonstrates competitive performance across various D4RL benchmarks.
Read more
Shortcut Trajectory Planning for Efficient Offline Reinforcement Learning
Summary
This paper introduces Shortcut Trajectory Planning (STP), a novel framework for offline model-based reinforcement learning that utilizes shortcut models as efficient trajectory generators. Traditional diffusion-based trajectory planners, while effective, often suffer from high inference costs due to their iterative denoising processes. Consistency-based planners mitigate this issue but rely on a two-stage teacher-student distillation process, which can increase training costs and introduce instability. STP addresses these challenges by training a conditional shortcut trajectory model in a single stage, allowing for adjustable inference steps and the selection of candidate plans through a critic that incorporates feasibility-aware corrections. The authors evaluate STP on standard D4RL benchmarks across various tasks, demonstrating that it achieves competitive performance while simplifying the training pipeline and reducing both inference costs and training complexity. This positions shortcut models as a practical alternative to existing distillation-based generative planners in offline reinforcement learning.
Methodology
The authors propose a shortcut model that learns step-size-conditioned updates, enabling one-step, few-step, and multi-step trajectory generation within a single network. This model is trained directly without the need for a teacher-student distillation process, thus simplifying the training pipeline. The framework incorporates a critic for selecting candidate plans with feasibility-aware corrections.
Results
STP was evaluated against existing diffusion-based and consistency-based planners on standard offline reinforcement learning benchmarks. The results indicate that STP achieves strong performance while significantly reducing inference costs and training complexity, making it a more efficient option for trajectory planning.
Implications
The findings suggest that shortcut models can enhance the efficiency of offline reinforcement learning applications, particularly in scenarios requiring fast generative planning. This could lead to broader adoption of STP in real-time control systems and robotics, where computational resources are limited.
Write-Protected Discrete Bottlenecks for Language-Grounded World Models: A Structural Limitation and Sufficient Fix
Robotics
NLP
Multimodal
- Language gradients entering discrete bottlenecks create a structural trade-off that limits learning and diversity.
- A three-layer architectural fix is proposed to address the limitations of existing end-to-end approaches.
- The proposed architecture achieves high grounding accuracy while maintaining low computational requirements.
- The findings challenge the assumption that larger LLMs inherently improve physical grounding in robotic systems.
Read more
Write-Protected Discrete Bottlenecks for Language-Grounded World Models: A Structural Limitation and Sufficient Fix
Summary
This paper investigates the interaction between language and discrete symbol systems in robot world models, challenging the prevalent end-to-end integration of large language models (LLMs) and vision-language models (VLMs) into robotic systems. The author identifies a critical structural limitation: language gradients entering a discrete symbol bottleneck lead to a trade-off where either learning collapses or diversity is maintained without semantic accuracy. The study proposes a minimal architectural fix consisting of three layers: (1) cutting the gradient chain to prevent language signals from reaching the symbol bottleneck, (2) introducing a gradient-free semantic channel using a non-parametric Memory Table for co-occurrence counting, and (3) employing DP-Means streaming clustering to resolve symbol collisions. The combination of these layers significantly improves grounding accuracy from 22.2% to 97.2%. The findings are validated across multiple encoder architectures and environments, demonstrating that the proposed architecture can effectively separate physical perception from language processing without the need for extensive parameter tuning or computational overhead.
Methodology
The methodology involves empirical testing of end-to-end language integration in robotic systems, identifying structural limitations through experiments, and proposing a three-layer architectural fix. The effectiveness of the proposed architecture is validated across different encoder architectures and environments through multiple independent runs.
Results
The results show that the proposed three-layer fix achieves grounding accuracy of 97.2%, compared to only 22.2% without the collision resolution layer. The architecture demonstrates zero symbol collapse across all tested conditions, with successful semantic binding rates between 79% and 100%. The study confirms that the structural limitations of existing methods cannot be resolved through optimization alone.
Implications
The findings suggest a need to rethink the integration of language and perception in robotic systems, advocating for architectural designs that separate these functions. This could lead to more robust and effective language-grounded world models in robotics, potentially influencing future research and applications in embodied AI.
Forget Narrowly, Retain Broadly: Unlearning as an Asymmetric Generalization Problem
NLP
Large Language Models
Optimization
- Introduces SUITE, a fine-grained evaluation protocol for unlearning in LLMs that captures the asymmetric generalization problem.
- Identifies and addresses the failures of existing benchmarks in measuring unlearning effectiveness.
- Presents JensUn++, an advanced unlearning algorithm that optimizes the trade-off between forgetting and retaining knowledge.
- Demonstrates the importance of training data quality in achieving effective unlearning outcomes.
Read more
Forget Narrowly, Retain Broadly: Unlearning as an Asymmetric Generalization Problem
Summary
This paper addresses the challenge of machine unlearning in large language models (LLMs), focusing on the need to remove specific knowledge while retaining other capabilities, which is crucial for privacy and safety. The authors identify two main failures in existing benchmarks: 'under-forgetting,' where knowledge resurfaces under paraphrased or indirect queries, and 'over-forgetting,' where unrelated knowledge is lost. They argue that these issues stem from an asymmetric generalization problem, where the forget set is finite but requires extensive generalization across various query formulations, while the retain set is vast and implicitly defined. To tackle this, the authors introduce SUITE, a novel evaluation protocol and training corpus designed to capture the forget-retain structure in factual domains. SUITE includes diverse query types to assess both under-forgetting and over-forgetting. The paper also presents JensUn++, an improved unlearning algorithm that enhances the balance between forgetting and retaining knowledge across multiple LLMs. The findings demonstrate that the choice of training data is as critical as the algorithmic design, with methods trained on SUITE showing significant improvements in performance.
Methodology
The authors developed SUITE, which includes a training corpus and evaluation protocol that probes both under-forgetting and over-forgetting through various query types, including indirect and multi-hop questions. They also introduced JensUn++, which employs an adaptive optimization scheme and a pairing strategy for queries to improve unlearning performance.
Results
The study showed that models trained on SUITE significantly outperformed those trained on existing benchmarks, revealing that effective unlearning is heavily dependent on the quality of training data. JensUn++ demonstrated improved performance in reducing both under-forgetting and over-forgetting compared to previous methods.
Implications
The findings suggest that refining unlearning methodologies and evaluation protocols can enhance the privacy and safety of LLMs, making them more reliable in sensitive applications. The introduction of SUITE can serve as a standard for future research in machine unlearning.
Architecture Generalization with MetaNCA
Graph Learning
Efficient ML
Theory
- MetaNCA enables the generation of diverse neural network architectures through local self-organization.
- The Weight Transformer architecture uses linear attention to facilitate local weight updates.
- MetaNCA demonstrates generalization to unseen architectures, enhancing adaptability.
- The framework scales to large networks with millions of parameters without backpropagation.
Read more
Architecture Generalization with MetaNCA
Summary
The paper introduces Meta Neural Cellular Automata (MetaNCA), a novel framework that enables the generation of diverse neural network architectures through local self-organization of weights. Inspired by biological systems, MetaNCA learns local update rules that iteratively adjust the weights of a task network based on local interactions within the computation graph. The authors propose a Weight Transformer architecture for the local rule network, which utilizes linear attention to aggregate signals from neighboring weights and hidden states. The framework is demonstrated to generate weights for various architectures, including feedforward MLPs, CNNs, and ResNets, on benchmark datasets like MNIST and CIFAR-100, scaling up to networks with 2 million parameters. Importantly, MetaNCA exhibits the ability to generalize to unseen architectures, with architectural diversity during training enhancing this generalization capability. This work addresses the limitations of traditional backpropagation methods by providing a more adaptable and efficient approach to neural network training.
Methodology
MetaNCA employs a neural cellular automaton approach where a local rule network iteratively updates the weights of a task network based on local interactions in the computation graph. The Weight Transformer architecture aggregates information from neighboring weights and hidden states using linear attention mechanisms.
Results
The authors successfully generated weights for various neural network architectures, including MLPs, CNNs, and ResNets, achieving performance on datasets like MNIST and CIFAR-100. MetaNCA was shown to generalize effectively to architectures not encountered during training, with improved performance linked to the diversity of architectures used in the training phase.
Implications
MetaNCA has the potential to revolutionize neural network training by providing a more efficient, adaptable, and scalable approach that mimics biological learning processes. This could lead to advancements in creating more flexible models that require fewer resources and can adapt to new tasks with minimal retraining.
Federated Deep Learning for Privacy-Preserving Cardiovascular Disease Risk Prediction
Federated Learning
- Federated learning allows for collaborative model development without sharing sensitive patient data.
- The study integrates two heterogeneous cohorts to improve cardiovascular disease risk prediction.
- Federated deep learning models achieved higher predictive performance compared to local models.
- The approach preserves patient privacy while enhancing model generalizability.
Read more
Federated Deep Learning for Privacy-Preserving Cardiovascular Disease Risk Prediction
Summary
This paper presents a federated deep learning approach aimed at improving cardiovascular disease (CVD) risk prediction while preserving patient privacy. Traditional models often rely on data from single institutions or pooled datasets, which can be limited by privacy regulations. The authors integrate two distinct population-based cohorts: Lifelines, with 148,230 participants and self-reported outcomes, and the Rotterdam Study, with 10,155 participants and clinically linked outcomes. The study primarily evaluates model performance on the Rotterdam Study due to its complete follow-up data. The results indicate that deep survival models trained using federated learning outperform locally trained models, with the C-statistic for the Rotterdam Study increasing from 0.728 to 0.739, and for Lifelines from 0.783 to 0.787. These findings suggest that federated deep learning can enhance CVD risk prediction across heterogeneous cohorts without compromising individual data privacy.
Methodology
The study employs a federated learning framework to train deep survival models across two distinct cohorts. Each cohort trains a model on its local data and shares only model parameter updates with a central server, which aggregates these updates into a global model. The performance of the models is evaluated using C-statistics to assess predictive accuracy.
Results
The federated deep learning models demonstrated improved predictive performance, with the C-statistic for the Rotterdam Study increasing from 0.728 to 0.739, and for Lifelines from 0.783 to 0.787. These results indicate that federated learning can effectively enhance cardiovascular disease risk prediction.
Implications
The findings suggest that federated deep learning can be a viable solution for developing predictive models in healthcare settings where data privacy is a concern. This approach could facilitate better risk prediction across diverse populations, ultimately aiding in the prevention of cardiovascular diseases.
Distributed Sketching on Data Partitions for OLS Regression
Theory
Optimization
Efficient ML
- Introduces a distributed sketching method for OLS regression using partitioned data subsets.
- Characterizes the exact excess loss of the averaged OLS estimator in this context.
- Shows that the performance of the new method is comparable to traditional sketching when subset covariances are similar.
- Highlights the importance of covariance divergence in determining the effectiveness of the sketching approach.
Read more
Distributed Sketching on Data Partitions for OLS Regression
Summary
This paper investigates a novel approach to distributed sketching for ordinary least squares (OLS) regression, focusing on partitioned subsets of large datasets. The authors propose a method that reduces computational costs by allowing each machine to work with smaller sketches derived from these partitions, rather than the entire dataset. The study characterizes the exact excess loss of the averaged OLS estimator under this new sketching mechanism, demonstrating that the loss is comparable to that of traditional sketching methods when the covariance divergence among subsets is minimal. This approach not only enhances computational efficiency but also maintains the accuracy of OLS estimators in distributed settings, making it particularly relevant for large-scale data analysis.
Methodology
The authors utilize a fixed design setting to analyze the performance of OLS estimators constructed from sketches derived from partitioned subsets of data. They derive the exact excess loss for the averaged estimator and compare it with existing results from sketching on the whole dataset. The study employs theoretical analysis to establish conditions under which the new method performs favorably.
Results
The main results indicate that the excess loss of the averaged OLS estimator from partitioned sketches is comparable to that from sketches of the entire dataset, particularly when the covariance divergence among subsets is low. The findings suggest that the proposed method can outperform traditional sketching methods under certain conditions, particularly when data is i.i.d. sampled.
Implications
The implications of this research are significant for large-scale data analysis, particularly in distributed computing environments. By reducing computational costs while maintaining accuracy, this method can facilitate more efficient data processing in various applications, including machine learning and statistical analysis.
AlphaZero in Sparsely Rewarded Games: Limits and Auxiliary Supervision
Reinforcement Learning
Theory
- Vanilla AlphaZero achieves strong self-play policies but does not consistently recover oracle-consistent play in Connect Four and Chomp.
- The introduction of AZAL significantly improves oracle consistency, particularly in Chomp, suggesting that standard AlphaZero may struggle with exact optimality.
- Multi-frame inputs alone do not resolve the performance gap in Chomp, indicating that more sophisticated learning signals are necessary.
- The study provides empirical evidence that highlights the distinction between superhuman and perfect play in AlphaZero-style agents.
Read more
AlphaZero in Sparsely Rewarded Games: Limits and Auxiliary Supervision
Summary
This paper investigates the limitations of AlphaZero in achieving perfect play in sparsely rewarded games, specifically focusing on Connect Four and Chomp. While AlphaZero has shown superhuman performance through a neural-guided Monte Carlo Tree Search (MCTS), the authors highlight a significant gap between strong play and perfect play. They evaluate vanilla AlphaZero against exact oracles and introduce two variants: a multi-frame representation for Chomp and an AlphaZero Auxiliary Loss (AZAL) that incorporates oracle-derived policy supervision. The study finds that vanilla AlphaZero performs well but fails to maintain optimal trajectories in both games. AZAL, however, significantly enhances oracle consistency, achieving perfect consistency on certain board sizes in Chomp and improving performance in Connect Four. The findings suggest that the standard AlphaZero training signal may be a bottleneck for recovering exact play, emphasizing the need for auxiliary supervision in games with sparse global features.
Methodology
The authors employed a unified self-play and MCTS framework to evaluate the performance of vanilla AlphaZero, a multi-frame variant, and the AZAL variant. They conducted empirical studies in two contrasting game domains, Connect Four and Chomp, using exact oracle evaluations to measure consistency and performance across multiple game states and board configurations.
Results
Vanilla AlphaZero demonstrated strong performance but failed to maintain optimal play trajectories in both games. AZAL achieved perfect full-game oracle consistency on Chomp 10×11 and high but incomplete consistency on 9×10. In Connect Four, AZAL improved the oracle-match rate and delayed the first oracle mistake, but did not reach perfect play.
Implications
The findings suggest that enhancing the learning signal through auxiliary supervision can bridge the gap between strong and perfect play in reinforcement learning agents. This has implications for the design of AI systems in games and other domains where optimal decision-making is critical, particularly in environments characterized by sparse rewards.
Deep Learning Method for Stationary Distribution of Reflected Brownian Motion
Theory
Efficient ML
Optimization
- Develops a deep learning method for estimating the Laplace transform of high-dimensional reflected Brownian motion.
- Combines a tailored loss function, sampling scheme, and neural network architecture to enhance performance.
- Achieves near-perfect prediction of tail probabilities in high-dimensional settings.
- Provides a scalable computational tool for analyzing stochastic systems.
Read more
Deep Learning Method for Stationary Distribution of Reflected Brownian Motion
Summary
This paper presents a novel deep learning approach for estimating the stationary distribution of reflected Brownian motion (RBM), which is crucial for analyzing high-dimensional stochastic systems. The authors highlight the challenges in obtaining closed-form solutions for RBM and the intractability of computing performance metrics like tail probabilities. They propose a framework that leverages the basic adjoint relationship (BAR) to learn the Laplace transform of high-dimensional RBMs. The methodology involves a carefully designed loss function, a targeted training data sampling procedure, and a neural network architecture that maintains efficiency as the dimensionality increases. The authors validate their approach on RBM instances with known tail probabilities, achieving near-perfect predictions in high-dimensional scenarios. This work represents a significant advancement in the application of deep learning to stochastic systems, providing a scalable tool for performance analysis beyond analytically tractable regimes.
Methodology
The authors utilize a deep learning framework that approximates the Laplace transform of the stationary distribution of RBMs. They design a structured loss function based on the squared residual of the basic adjoint relationship (BAR), implement a targeted sampling scheme, and employ feedforward neural networks in the complex domain to ensure robust numerical inversion for tail probability calculations.
Results
The proposed method demonstrates high accuracy in predicting tail probabilities for RBM instances, particularly in high-dimensional settings. The results indicate that the deep learning approach can effectively capture the Laplace transform, which is essential for evaluating performance metrics in stochastic systems.
Implications
This research has significant implications for the analysis of multiclass queueing networks and other high-dimensional stochastic systems. The ability to accurately estimate tail probabilities can enhance performance evaluations and decision-making in various applications, including operations research and network traffic management.
Jet-Long: Efficient Long-Context Extension with Dynamic Bifocal RoPE
Large Language Models
Efficient ML
NLP
- Jet-Long is a tuning-free method for zero-shot context extension in LLMs.
- It dynamically adjusts the rescaling factor for long-range windows based on sequence length.
- The method achieves superior performance on benchmarks compared to existing zero-shot methods.
- Jet-Long incurs minimal latency overhead and is compatible with hybrid attention architectures.
Read more
Jet-Long: Efficient Long-Context Extension with Dynamic Bifocal RoPE
Summary
The paper introduces Jet-Long, a novel zero-shot context extension method designed to enhance the performance of large language models (LLMs) in long-context applications. Traditional methods for extending context in LLMs often struggle with maintaining fidelity in short contexts while adapting to longer sequences. Jet-Long addresses this by employing a bifocal approach that combines a local RoPE-faithful window with a dynamically rescaled long-range window, allowing for effective context management without the need for tuning. The method utilizes an analytic schedule to determine the rescaling factor based on the current sequence length, ensuring that the model performs optimally across varying input lengths. The authors demonstrate that Jet-Long achieves significant improvements in accuracy and perplexity on various benchmarks, including RULER and HELMET-RAG, while maintaining low latency overhead during generation. Additionally, Jet-Long is compatible with hybrid attention architectures, further enhancing its applicability in long-context scenarios.
Methodology
Jet-Long employs a bifocal context-extension strategy that combines a local window with a long-range window. The rescaling factor for the long-range window is dynamically calculated based on the current sequence length, using a parameter-free analytic schedule. This approach allows the model to maintain fidelity in short contexts while effectively extrapolating to longer sequences. The method integrates an inclusion-exclusion attention merge and an on-the-fly RoPE correction rotation, optimizing performance during inference.
Results
Empirical evaluations show that Jet-Long outperforms the strongest baseline methods on the RULER benchmark by +4.79, +2.18, and +2.03 percentage points for models of sizes 1.7B, 4B, and 8B, respectively. It also achieves the best accuracy on the HELMET-RAG benchmark and the lowest perplexity on the PG-19 dataset. The method demonstrates a throughput increase of up to 1.39× compared to FlashAttention-2, with a latency overhead of less than 4% across all tested lengths.
Implications
Jet-Long's ability to efficiently extend context in LLMs without requiring extensive retraining or tuning has significant implications for applications in long-document question answering, coding tasks, and multi-step workflows. Its compatibility with hybrid architectures suggests potential for broader adoption in various AI systems requiring long-context processing.
How are linear representations learned? Exact solutions to the dynamics of abstraction
Theory
Interpretability
- Introduces a framework for studying the dynamics of abstraction in neural networks during training.
- Establishes that data and target geometry jointly determine the final abstraction achieved.
- Demonstrates that deeper networks improve abstraction and that initialization scale affects maximum abstraction.
- Analyzes the impact of nonlinearity on abstraction dynamics, with ReLU networks showing different behaviors compared to erf networks.
Read more
How are linear representations learned? Exact solutions to the dynamics of abstraction
Summary
This paper investigates the dynamics of how linear representations, which are crucial for encoding concepts in neural networks, are learned during training. The authors introduce a framework to analyze the alignment of concept directions, termed 'abstraction', in a minimal linear network setting. They derive exact solutions for the trajectory of abstraction throughout training, revealing that the geometry of data and targets significantly influences the final abstraction achieved. Key findings include that deeper networks enhance abstraction and that the scale of initialization impacts the maximum abstraction reached. The study extends to nonlinear networks, demonstrating that the choice of nonlinearity affects abstraction dynamics, with ReLU networks showing a weaker dependence on target geometry compared to erf networks. The authors also establish a law indicating that nonlinearities diminish abstraction in activations relative to preactivations. Empirical validation is provided through experiments on models like DINOv3 and Gemma 4, where the theory is applied to enhance linear probe generalization. Overall, this work offers a comprehensive dynamical theory of abstraction with implications for interpretability and control in both deep learning and neuroscience.
Methodology
The authors developed a theoretical framework to analyze abstraction in a minimal linear network setting, deriving exact solutions for the trajectory of abstraction during training. They extended their analysis to nonlinear networks, examining how different nonlinearities influence abstraction dynamics. Empirical validation was conducted on various models to demonstrate the practical implications of their findings.
Results
The study found that abstraction is influenced by the geometry of the dataset and targets, improves with network depth, and is affected by the scale of initialization. Nonlinearities were shown to weaken abstraction in activations compared to preactivations, with empirical evidence supporting these theoretical insights in models like DINOv3 and Gemma 4.
Implications
The findings have significant implications for the interpretability and control of neural networks, suggesting that understanding the dynamics of abstraction can enhance model performance and generalization capabilities. This work also bridges concepts from deep learning and neuroscience, providing insights into how abstract representations can be leveraged in AI systems.
Adaptive Bayes exactly tracks information over intrinsic time
Theory
Optimization
- Introduces an exact information-accounting identity for Bayesian updates.
- Establishes two exact adaptive decompositions of cumulative regret.
- Demonstrates the applicability of the framework across various learning paradigms.
- Highlights the role of intrinsic time in understanding learning dynamics.
Read more
Adaptive Bayes exactly tracks information over intrinsic time
Summary
This paper presents a novel framework for understanding Bayesian and multiplicative-weights updates in the context of sequential feedback. The author establishes an exact information-accounting identity that relates the learner's excess loss to a chosen comparator, decomposing it into an immediate payment for uncertainty and a reduction in information distance to the comparator. This leads to two exact adaptive decompositions of cumulative regret, applicable across various learning paradigms, including Bayesian model averaging, online convex optimization, and contextual bandits. The paper emphasizes that the decompositions are exact rather than upper bounds, revealing that favorable learning conditions manifest as self-bounding properties of the realized intrinsic time. The methodology is grounded in a one-step information balance, which holds for any Bayesian update, and introduces two adaptive update strategies based on temperature variations. The findings suggest a unified approach to understanding different learning algorithms through the lens of information theory.
Methodology
The paper employs a theoretical approach, deriving an exact one-step information balance that relates the learner's loss to a comparator. It explores two adaptive update strategies based on the temperature of the updates, allowing for a comprehensive analysis of the learning process across different scenarios.
Results
The author shows that the cumulative regret can be decomposed into an intrinsic-time payment and a drift due to changing temperatures, providing a clear accounting of the learning process. This framework reveals that the performance of learning algorithms can be understood as a function of information dynamics rather than merely worst-case scenarios.
Implications
The findings have significant implications for the design of adaptive learning algorithms, suggesting that understanding the information dynamics can lead to more efficient learning strategies. The framework can be applied to various fields, including reinforcement learning, online optimization, and decision-making processes.
Learning Physics-Informed Surrogate Model of Linear Elastic Displacement Fields from Geometry
Theory
Efficient ML
- Development of a physics-informed DeepONet framework for predicting displacement fields.
- Introduction of a dedicated geometry-encoding strategy that allows direct input of fracture geometry.
- Weak enforcement of traction-free conditions on fracture boundaries through a localized penalty term.
- Demonstration of the model's feasibility using a specific fracture geometry as a proof of concept.
Read more
Learning Physics-Informed Surrogate Model of Linear Elastic Displacement Fields from Geometry
Summary
This paper presents a novel approach to developing a fast and physically consistent surrogate model for real-time structural health monitoring of fractured elastic domains. The authors introduce a physics-informed DeepONet framework that predicts displacement fields based on boundary conditions and fracture geometry, utilizing a unique encoding strategy for the geometry without relying on finite-element-generated training data. The method incorporates a weak penalty term to enforce traction-free conditions on fracture boundaries. The study focuses on a specific fracture geometry to demonstrate the feasibility of the proposed formulation, which lays the groundwork for extending the surrogate modeling to various fracture shapes and configurations. The authors emphasize the importance of accurate and rapid predictions in structural health monitoring, especially in industrial applications where computational efficiency is critical. The proposed model aims to overcome the limitations of traditional finite element methods (FEM), which can be computationally expensive in scenarios requiring repeated evaluations across different geometries and loading conditions.
Methodology
The methodology involves using a physics-informed operator-learning approach to predict displacement fields in elastic solids. The model takes the geometry, physical properties, and boundary conditions as inputs and learns to reconstruct the physical solution while ensuring compliance with governing partial differential equations. The DeepONet framework is employed to map the functional inputs to the full displacement field without relying on finite element method (FEM) data.
Results
The numerical examples presented in the paper demonstrate the effectiveness of the proposed framework in accurately predicting displacement fields for a specific fracture geometry. The results indicate that the model can achieve fast inference times while maintaining physical consistency, showcasing its potential as a lightweight surrogate model for structural health monitoring.
Implications
The implications of this research are significant for structural health management in industrial applications, where rapid and accurate predictions of mechanical responses are essential. The proposed surrogate model could facilitate real-time monitoring, fault detection, and diagnosis in smart industrial assets, potentially leading to improved safety and efficiency in engineering practices.
Group Invariant Spectral Embedding
Graph Learning
Theory
Efficient ML
- Introduces group-invariant spectral embedding to account for symmetries in data.
- Proves that graph Laplacians from invariant kernels converge to differential operators on quotient spaces.
- Demonstrates improved convergence rates and effective dimension reduction.
- Validates the approach on datasets with SO(2) and SO(3) symmetry.
Read more
Group Invariant Spectral Embedding
Summary
This paper addresses the limitations of standard spectral embedding methods in handling datasets with intrinsic symmetries, such as rotations. Traditional methods treat symmetry-related data points as unrelated, which can lead to suboptimal representations. The authors propose a novel approach that incorporates symmetries directly into the affinity kernels used for spectral embedding. They analyze the case of a Riemannian data manifold with symmetries defined by a compact Lie group and demonstrate that graph Laplacians constructed from group-invariant kernels converge to differential operators on the quotient space of the manifold. This results in improved convergence rates and effective dimensionality reduction by leveraging the symmetries present in the data. The paper validates the proposed method on datasets exhibiting SO(2) and SO(3) symmetries, showing that the G-invariant spectral embedding recovers the intrinsic geometry of the data more effectively than standard methods, even with infinite data.
Methodology
The authors study three classes of group invariant affinity kernels: minimization over the group, integration over the group, and G-invariant feature mapping. They analyze the continuous case on a Riemannian manifold and derive theoretical results regarding the convergence of graph Laplacians constructed from these kernels.
Results
The main theoretical result establishes that graph Laplacians from G-invariant kernels converge pointwise to second-order differential operators on the quotient space, leading to improved convergence rates. Specifically, the effective dimension reduces according to the dimension of the symmetry group, enhancing sample efficiency and interpretability in the embedding process.
Implications
This work has significant implications for various applications in machine learning where data exhibits symmetries, such as in computer vision tasks like cryo-electron microscopy and in analyzing set-structured data. The proposed method can lead to more accurate and efficient data representations, improving clustering and dimensionality reduction tasks.
GatedLinear: Adaptive Routing of Complementary Linear Bases for Time Series Forecasting
Time Series
- GatedLinear utilizes three complementary linear bases to address diverse temporal dynamics in time series forecasting.
- The Tri-Factorized Fusion Gate enables adaptive routing of predictions based on variable characteristics and forecast horizons.
- The framework achieves competitive accuracy against state-of-the-art models while being more parameter-efficient.
- GatedLinear provides interpretable routing patterns, enhancing the understanding of the forecasting process.
Read more
GatedLinear: Adaptive Routing of Complementary Linear Bases for Time Series Forecasting
Summary
The paper introduces GatedLinear, a novel framework for time series forecasting that addresses the limitations of existing deep learning models which often rely on a single computational backbone. GatedLinear is designed to capture diverse temporal dynamics by employing three specialized linear bases: a global trend-seasonal basis for smooth projections, a difference-based incremental basis for nonstationary drift, and a phase-aligned recurrence basis for cyclic patterns. The framework utilizes a Tri-Factorized Fusion Gate to dynamically route predictions based on the specific characteristics of the time series data, allowing for granular, point-wise routing decisions. This approach not only enhances forecasting accuracy but also maintains a lightweight model architecture with a smaller parameter footprint. Experiments demonstrate that GatedLinear achieves state-of-the-art performance on standard benchmarks, outperforming complex models while providing interpretable routing patterns.
Methodology
GatedLinear employs a tri-basis approach for time series forecasting, integrating a global trend-seasonal basis, a difference-based incremental basis, and a phase-aligned recurrence basis. The Tri-Factorized Fusion Gate orchestrates the routing of predictions by generating weights that are specific to each forecast point, allowing for flexible and interpretable model behavior without the need for complex neural architectures.
Results
GatedLinear demonstrated state-of-the-art or highly competitive accuracy on various time series forecasting benchmarks, outperforming several recent complex models. The model's lightweight design resulted in a significantly smaller parameter footprint while maintaining high interpretability of the routing decisions.
Implications
The GatedLinear framework has potential applications in various fields requiring time series forecasting, such as finance, energy management, and industrial monitoring. Its ability to adaptively route predictions based on temporal dynamics can lead to more accurate forecasts and better decision-making in real-world scenarios.
Active rejection enables reliable generalization of universal machine-learning interatomic potentials
Theory
Optimization
Efficient ML
- Introduction of the Adaptive Multi-Teacher Routing (ATR) framework for reliable data construction.
- ATR utilizes multiple pretrained uMLIPs to filter and generate high-confidence pseudo-labels.
- The framework successfully distills a large dataset from a minimal amount of high-fidelity labels.
- Models trained on ATR-generated data show improved performance and stability in simulations.
Read more
Active rejection enables reliable generalization of universal machine-learning interatomic potentials
Summary
This paper addresses the challenge of training universal machine learning interatomic potentials (uMLIPs) with high fidelity while managing the limitations of available high-accuracy data. The authors introduce the Adaptive Multi-Teacher Routing (ATR) framework, which utilizes multiple pretrained uMLIPs to generate reliable pseudo-labels for a vast pool of candidate structures. ATR operates by assessing the reliability of predictions from different teacher models based on structural descriptors and inter-teacher disagreement. By employing only 0.2% of real r2SCAN labels, ATR successfully generates 2.89 million pseudo-labels, significantly enhancing the training dataset for uMLIPs. The experiments demonstrate that models trained on ATR-generated data outperform baseline models and exhibit improved dynamical stability in molecular dynamics simulations, effectively addressing the reliability gaps associated with localized predictions. This work establishes active rejection as a viable strategy for constructing high-fidelity datasets, thereby facilitating the generalization of uMLIPs across diverse materials.
Methodology
The ATR framework reformulates the problem of high-fidelity data construction as a structure-wise decision-making process. It calibrates multiple pretrained uMLIPs, evaluates the reliability of their predictions based on structural descriptors and disagreement, and selectively generates pseudo-labels while rejecting unreliable structures.
Results
The ATR framework generated 2.89 million traceable r2SCAN-level pseudo-labels using only 0.2% of real r2SCAN labels. Models trained on this dataset consistently outperformed baseline models and showed improved dynamical robustness in molecular dynamics simulations, maintaining stable trajectories where baseline models failed.
Implications
The findings suggest that ATR can significantly enhance the training of uMLIPs, making them more reliable for a broader range of materials. This approach could lead to more accurate simulations in materials science and other fields requiring atomistic modeling.
Learning $ ext{AC}^0$ under Locally Sampleable Graphical Models
Theory
Graph Learning
- Introduces a quasipolynomial-time learner for AC0 circuits under locally sampleable graphical models.
- Circumvents the polynomial growth requirement of previous work by utilizing a new low-degree approximation method.
- Establishes a connection between efficient local samplers and the approximation of AC0 functions.
- Applies the framework to two-spin systems, including the hard-core and Ising models, on arbitrary bounded-degree graphs.
Read more
Learning $ ext{AC}^0$ under Locally Sampleable Graphical Models
Summary
This paper addresses the challenge of learning constant-depth Boolean circuits (AC0) under locally sampleable graphical models, extending previous work that focused on learning under uniform and certain Gibbs distributions. The authors introduce a quasipolynomial-time learning algorithm that circumvents the polynomial growth requirement of prior methods by leveraging a new low-degree approximation for Gibbs distributions. This approximation is achieved through the simulation and truncation of classical Glauber dynamics, allowing for effective learning in regimes approaching sampling thresholds for two-spin systems, such as the hard-core and Ising models on arbitrary bounded-degree graphs. The paper establishes a connection between the existence of efficient local samplers and the ability to approximate AC0 functions, demonstrating that if a Gibbs distribution has a good local sampler, then every AC0 function can be approximated by a low-degree polynomial under that distribution. This result broadens the scope of learning guarantees for AC0 functions beyond previously established conditions, making significant strides in the understanding of learning in complex, correlated distributions.
Methodology
The authors develop a learning algorithm based on low-degree polynomial approximations for Gibbs distributions, utilizing systematic-scan Glauber dynamics to simulate and truncate the sampling process. This approach allows for the effective approximation of AC0 functions under locally sampleable graphical models.
Results
The main result is a quasipolynomial-time learning algorithm that can learn AC0 functions under the hard-core model with specific conditions on fugacity. The algorithm operates efficiently with a sample complexity of N = n logO(d)(n/ε) and achieves high accuracy with a probability of at least 0.9.
Implications
The findings have significant implications for computational learning theory, particularly in understanding the learning of Boolean functions in complex distributions. The results may influence future research on learning algorithms in various applications, including statistical physics and machine learning on graphical models.
On-Device Adaptive Battery Power Prediction for Electric Vehicles
Time Series
- Introduces on-device learning for adaptive battery power prediction in EVs.
- Demonstrates significant performance improvements through online and offline adaptation strategies.
- Achieves mean absolute error reductions of up to 14.88% in battery power forecasting.
- Highlights the importance of adapting to dynamic driving conditions and user behavior.
Read more
On-Device Adaptive Battery Power Prediction for Electric Vehicles
Summary
This paper addresses the challenge of accurate battery power prediction in Electric Vehicles (EVs) using adaptive power management techniques. The authors highlight the limitations of existing deep learning models, which often degrade in performance when faced with data distributions that differ from their training datasets. To overcome this, they propose a novel approach that enables on-device learning, allowing for continuous adaptation of pretrained battery prediction models to new, unseen data. The study investigates both online and offline adaptation strategies, demonstrating significant improvements in forecasting performance across various models and time horizons. The authors achieve mean absolute error reductions of up to 7.49% with online adaptation and 14.88% with offline adaptation. This research emphasizes the benefits of on-device adaptation, resulting in enhanced battery power predictions in real-world EV scenarios, thereby supporting Battery Management System operations and optimizing vehicle battery power supply.
Methodology
The authors trained state-of-the-art deep learning models for battery power prediction with short forecasting horizons (1-3 seconds). They implemented two adaptation strategies: online learning, which processes incoming data streams in real-time, and offline learning, which analyzes historical trip data to learn power consumption patterns. The models were evaluated on resource-constrained edge devices to mimic the computational limitations of in-vehicle systems.
Results
The study found that the proposed on-device adaptation strategies significantly improved forecasting accuracy, with reductions in mean absolute error of up to 7.49% for online adaptation and 14.88% for offline adaptation. These results demonstrate the effectiveness of the approach in real-world EV scenarios, where dynamic conditions can impact battery power demands.
Implications
The findings suggest that on-device adaptive learning can enhance the reliability and accuracy of battery power predictions in EVs, leading to better energy management and optimization of battery usage. This could ultimately contribute to improved performance and efficiency of electric vehicles in diverse driving conditions.
Trustworthy Machine Learning through the Lens of Combinatorial Optimization: Survey and Research Perspectives
Optimization
Theory
Interpretability
- Trustworthiness in ML requires more than just predictive accuracy; it encompasses transparency, interpretability, robustness, fairness, and privacy.
- The Rashomon effect indicates that multiple models can achieve similar performance, allowing for the selection of models based on trustworthiness criteria.
- Combinatorial optimization offers a robust framework for addressing various trustworthiness challenges in ML, including model training, auditing, and certification.
- CO techniques can provide global optimality and formal certificates, which are essential for ensuring the reliability of ML systems.
Read more
Trustworthy Machine Learning through the Lens of Combinatorial Optimization: Survey and Research Perspectives
Summary
This survey paper explores the intersection of trustworthy machine learning (ML) and combinatorial optimization (CO), emphasizing the need for ML systems to be transparent, interpretable, robust, fair, and privacy-preserving. The authors argue that traditional empirical performance metrics are insufficient for evaluating ML models, as similar-performing models can differ significantly in their trustworthiness attributes. The survey synthesizes recent advancements in CO techniques applied to both training and post-training tasks in ML, including model selection, explanation generation, robustness analysis, fairness auditing, model compression, and privacy protection. The authors highlight that CO provides global guarantees and formal certificates that are often lacking in heuristic approaches, thus enhancing the reliability of ML systems. Despite scalability challenges, advancements in CO solvers and hybrid algorithms suggest a growing role for CO in developing trustworthy ML applications. The paper aims to unify diverse research efforts across operations research, theoretical computer science, and formal methods, providing a comprehensive overview of how CO can address critical trustworthiness issues in ML.
Methodology
The authors conducted a structured, subdomain-driven survey of the literature at the intersection of combinatorial optimization and trustworthy machine learning, reviewing both training and post-training tasks and synthesizing findings from various research domains.
Results
The survey reveals that CO techniques can effectively address trustworthiness issues in ML, offering capabilities such as robustness verification, fairness certification, and model simplification. The authors also note the scalability challenges associated with CO but emphasize the potential for hybrid algorithms to mitigate these issues.
Implications
The findings suggest that integrating CO into the design and deployment of ML systems can enhance their trustworthiness, making them more suitable for high-stakes applications. This integration could lead to better governance and institutional measures for responsible AI use.
Model Agnostic Graph Prompt Learning for Crystal Property Prediction
Graph Learning
- Introduction of a model agnostic soft prompt learning framework for crystal property prediction.
- Combines node-level and graph-level prompts to capture both local and global features.
- Achieves significant performance improvements (3% - 15%) over existing GNN models.
- Lightweight addition of only 0.32% extra parameters to existing architectures.
Read more
Model Agnostic Graph Prompt Learning for Crystal Property Prediction
Summary
This paper presents a novel approach to crystal property prediction using a soft prompt learning framework that is model agnostic and integrates seamlessly with existing Graph Neural Network (GNN) architectures. The authors address the limitations of traditional GNNs, which often require extensive domain knowledge and have large parameter sizes, by introducing a multi-level graph prompt learning framework that captures latent features at both node-level and graph-level. Node-level prompts focus on local chemical semantics of atom types, while graph-level prompts encode global structural symmetry. The proposed framework is lightweight, adding minimal parameters to existing models, and significantly enhances prediction performance across various benchmark datasets. The results demonstrate that the incorporation of prompt learning leads to performance improvements of 3% to 15% over state-of-the-art GNN models, while also enabling cross-property knowledge transfer, particularly beneficial for properties with limited training data.
Methodology
The authors propose a multi-level graph prompt learning framework that includes both node-level and graph-level soft prompts. Node-level prompts consist of independent vectors representing latent chemical features, while graph-level prompts capture structural features of the crystal. The framework is designed to be integrated with any existing GNN encoder, enhancing its predictive capabilities without substantial increases in computational overhead.
Results
Extensive experiments on benchmark datasets show that the proposed prompt learning framework significantly improves the performance of various GNN models. For instance, the simpler CGCNN model saw a performance increase of 15.11%, making it competitive with more complex models like ALIGNN. Advanced models such as ALIGNN, Matformer, and PotNet also benefited from the framework, with performance gains of 5.52%, 6.64%, and 3.15%, respectively. Additionally, the learned soft prompts facilitated cross-property knowledge transfer, improving predictions for properties with limited training data.
Implications
The proposed framework has the potential to accelerate the discovery of new materials by improving the accuracy and efficiency of crystal property predictions. Its lightweight nature allows for easy integration into existing models, making it a valuable tool for researchers in materials science and computational chemistry.
Training, Reading, and Editing Legible Transformers
Interpretability
- Introduces legibility by construction in transformer models, enhancing interpretability.
- Proposes a per-channel variance floor to maintain operator context during training.
- Achieves significant legibility improvements, with a majority of operations being crisp detectors.
- Enhances the model's editability and readability, allowing for more precise modifications.
Read more
Training, Reading, and Editing Legible Transformers
Summary
This paper presents a novel approach to building transformers that are legible by construction, meaning their operators are designed to be interpretable from the outset rather than requiring post-hoc analysis. The author identifies a critical challenge in maintaining legibility during training, particularly due to a 'crispness penalty' that can inadvertently lead to operators becoming constant values, thus losing their contextual information. To address this, the paper introduces a per-channel variance floor as a loss function, which effectively preserves both legibility and model quality. The results demonstrate that the proposed model achieves a significant increase in legibility, with 78% of feed-forward operands and 50% of attention value channels functioning as crisp contextual detectors. The paper also discusses the enhanced readability of the model's outputs, allowing for clearer distinctions between detection and naming processes. Furthermore, the model facilitates more localized edits, improving the efficiency of modifications to the network. A decorrelation pressure is introduced to promote independence among units, enhancing editability without sacrificing quality. Overall, the contributions of this work lie in establishing a framework for legible transformers that can be trained, read, and edited effectively, providing a foundation for future research in interpretable machine learning.
Methodology
The methodology involves designing transformers with bounded, named units that perform fuzzy set operations. A crispness penalty is applied during training to ensure operators remain decisive, while a per-channel variance floor is introduced to prevent collapse into constant values. The model is trained end-to-end, allowing for both reading and editing of the legible operators.
Results
The proposed model shows that 78% of feed-forward operands and 50% of attention value channels are crisp contextual detectors. Legibility improves significantly from shallow to deep layers, with per-head legibility rising from 18% to 78%. The model also allows for localized edits that are 50-184 times more efficient in deep layers, and maintains quality comparable to conventional transformers.
Implications
The findings suggest that legible transformers can enhance interpretability in machine learning applications, making it easier for users to understand model decisions and facilitate targeted modifications. This could have significant implications for fields requiring transparency, such as healthcare, finance, and autonomous systems.
Multi-Modal, Multi-Environment Machine Teaching for Robust Reward Learning
Reinforcement Learning
Robotics
Multimodal
- Analysis of feedback modalities reveals that comparisons impose stronger constraints than demonstrations for reward learning.
- Formal characterization of environment-dependent reward identifiability shows residual ambiguity even with unlimited feedback in a single MDP.
- Introduction of HSCOT, a hierarchical algorithm that selects informative environments and feedback queries for efficient reward learning.
- Empirical validation indicates HSCOT achieves better performance than uniform teaching under identical feedback budgets.
Read more
Multi-Modal, Multi-Environment Machine Teaching for Robust Reward Learning
Summary
This paper addresses the challenge of aligning the behavior of autonomous agents with human intent across diverse operational contexts by developing a robust reward learning framework. The authors critique existing approaches that focus solely on single-environment, demonstration-only settings, which often lead to overfitting and poor generalization when agents are deployed in new environments. They analyze how different feedback modalities, such as comparisons and demonstrations, influence the learning of reward functions, revealing that comparisons provide stronger global constraints in unlimited-data scenarios. To tackle the limitations of current methods, the authors propose a hierarchical machine teaching algorithm called Hierarchical Set Cover Optimal Teaching (HSCOT). This algorithm strategically selects informative environments and queries low-cost feedback to efficiently constrain rewards across multiple Markov Decision Processes (MDPs). Empirical results demonstrate that HSCOT significantly reduces regret and enhances generalization to held-out environments compared to uniform teaching baselines, underscoring the importance of multi-environment and multi-modal teaching for developing robust reward functions that can adapt to varying dynamics.
Methodology
The authors conducted a theoretical analysis of feedback modalities in reward learning and introduced the HSCOT algorithm, which operates across multiple MDPs. The algorithm selects environments that provide complementary reward constraints and queries low-cost feedback strategically to enhance learning efficiency.
Results
HSCOT demonstrated significantly lower regret and improved generalization to held-out environments compared to uniform teaching methods, validating the effectiveness of multi-modal and multi-environment teaching strategies.
Implications
The findings suggest that incorporating diverse feedback modalities and environments can enhance the robustness of reward learning in autonomous agents, making them more adaptable to varying operational contexts. This has potential applications in robotics, service automation, and any domain where agents must operate under different conditions while maintaining alignment with human intent.
Stochastic Linear Bandits with Partially Observed Actions
Theory
Optimization
Reinforcement Learning
- Introduces a novel algorithm, TOFU-POV, for stochastic linear bandits with partially observed actions.
- Demonstrates that sublinear regret is achievable when action vectors lie in a low-dimensional subspace.
- Provides a regret guarantee that scales with the intrinsic dimension rather than the ambient dimension.
- Presents a rank-adaptive version of the algorithm that does not require knowledge of the intrinsic dimension.
Read more
Stochastic Linear Bandits with Partially Observed Actions
Summary
This paper addresses the stochastic linear bandit (SLB) problem under the condition of partial observability, where a learning agent can only observe a random subset of coordinates for each action vector. This scenario is particularly relevant in applications such as recommendation systems and healthcare, where complete action descriptions may be costly or impossible to obtain. The authors propose an algorithm named TOFU-POV, which estimates the latent action subspace from masked actions, freezes the representation within epochs, and applies the Optimistic Linear Bandit (OFUL) algorithm in the resulting low-dimensional space. The theoretical contributions include a regret guarantee that scales with the intrinsic dimension of the action subspace rather than the ambient dimension, demonstrating that the algorithm can achieve sublinear regret despite the challenges posed by partial observability. Additionally, a rank-adaptive variant of TOFU-POV is introduced, which does not require prior knowledge of the intrinsic dimension. The paper also presents a lower bound that distinguishes the uncertainty in reward learning from the inherent costs associated with partial observations. Experimental results on synthetic and real datasets validate the theoretical findings, showing that TOFU-POV significantly outperforms existing baselines.
Methodology
The authors develop the TOFU-POV algorithm, which operates in epochs to estimate the latent action subspace from masked actions. It freezes the representation during each epoch to ensure that the reward learning process is treated as a standard linear bandit problem. The algorithm employs OFUL in the low-dimensional coordinates derived from the estimated subspace. The theoretical analysis includes deriving regret bounds that account for the intrinsic dimension, missingness, and other factors affecting performance.
Results
The theoretical results show that the regret of TOFU-POV scales as eO(m√T + κm√T/(p²√K)), where m is the intrinsic dimension, κ is the conditioning of the action covariance matrix, p is the probability of observing each coordinate, and K is the decision set size. The algorithm achieves sublinear regret, outperforming existing methods in both synthetic and real-world experiments.
Implications
The findings suggest that in scenarios where full observability is not feasible, leveraging low-dimensional structures can significantly enhance decision-making processes in various applications, including recommendation systems and healthcare. The proposed methods can lead to more efficient learning strategies in environments with limited data availability.
A law of robustness for two-layer neural networks with arbitrary weights
Theory
- Proves the conjectured law of robustness for two-layer neural networks with arbitrary weights.
- Establishes that Lipschitz constant must scale with √(n/m) for fitting noisy labels.
- Introduces a function-space covering method to handle unbounded weights.
- Demonstrates the applicability of results to continuous piecewise-linear activations, particularly ReLU.
Read more
A law of robustness for two-layer neural networks with arbitrary weights
Summary
This paper addresses a conjecture by Bubeck, Li, and Nagaraj regarding the Lipschitz constant of two-layer neural networks with arbitrary weights that fit noisy labels. The conjecture posits that for generic data, the Lipschitz constant must be at least of order √(n/m), where n is the number of data points and m is the number of neurons. Previous work established a universal version of this law under polynomial bounds on parameters, but the unbounded-weight case requires a different approach. The author proves the conjectured law for continuous piecewise-linear activations, including ReLU networks, up to a logarithmic factor. The results indicate that fitting data below the noise floor necessitates a Lipschitz constant that scales with the number of data points and the network width. The proof utilizes a function-space covering approach instead of parameter-space covering, which is not feasible for unbounded weights. Key components of the proof include a rigidity lemma that controls the coefficients of canonical kinks based on the Lipschitz constant of the realized function. The paper also discusses the implications of the results for different activation functions and provides insights into the limitations of existing methods in lower dimensions.
Methodology
The methodology involves proving the conjectured law using a function-space covering approach, which is necessary due to the challenges posed by unbounded weights. A central component is a rigidity lemma that links the coefficients of canonical kinks to the Lipschitz constant of the function. The analysis is conducted under two data models: uniform distribution on the sphere and Gaussian distribution.
Results
The main results show that for a fixed-width two-layer network with arbitrary weights, fitting data below the noise floor leads to a Lipschitz constant that is bounded below by a function of the number of data points and the network width, specifically Lip(f) ≥ c ε √(n/¯m log(C ¯mnd/ε)), where ¯m is a function of the number of neurons. The paper also provides a finite-horizon simultaneous-width version and a realized-kink-count version of the law.
Implications
The findings have significant implications for understanding the capacity and robustness of neural networks, particularly in high-dimensional settings. They suggest that robust interpolation requires a substantial number of neurons relative to the data points, which can inform the design of neural network architectures and training strategies.
Multimodal Unlearning Across Vision, Language, Video, and Audio: Survey of Methods, Datasets, and Benchmarks
Multimodal
- Introduces a unified taxonomy for multimodal unlearning across vision, language, video, and audio.
- Addresses the challenges of targeted forgetting in multimodal foundation models.
- Highlights the trade-offs among deletion strength, utility retention, efficiency, and reversibility.
- Identifies open problems and practical considerations for future research in multimodal unlearning.
Read more
Multimodal Unlearning Across Vision, Language, Video, and Audio: Survey of Methods, Datasets, and Benchmarks
Summary
This paper presents a comprehensive survey on multimodal unlearning, addressing the challenges posed by the inadvertent encoding of sensitive or biased information in foundation models such as Vision Language Models (VLMs), Diffusion Models (DMs), Large Language Models (LLMs), and Audio Foundation Models (AFMs). The authors highlight the impracticality of retraining models after deletion requests and the difficulties in targeted forgetting due to the distributed nature of knowledge in shared representations. The survey introduces a unified, system-oriented taxonomy that categorizes multimodal unlearning methods based on intervention stages and control pathways, facilitating systematic comparisons across various model architectures and modalities. Key contributions include a detailed overview of existing methods, datasets, and benchmarks, as well as the identification of open problems and practical considerations for future research. The authors also provide a curated repository to support ongoing work in this area.
Methodology
The survey employs a system-first approach to categorize multimodal unlearning methods based on intervention stages (data-side, training-time, architecture-constrained, training-free, and decoding time) and control pathways. This organization allows for a comprehensive comparison of existing methods and highlights the strengths and weaknesses of different approaches.
Results
The survey synthesizes existing literature on multimodal unlearning, providing a structured overview of methods and their applications across various modalities. It clarifies the landscape of multimodal unlearning, emphasizing the need for targeted forgetting mechanisms that maintain model utility while addressing privacy and ethical concerns.
Implications
The findings of this survey have significant implications for the governance of multimodal foundation models, enabling developers to implement selective data removal and behavior correction without extensive retraining. This capability is crucial for addressing privacy concerns and ensuring compliance with data protection regulations.
Sensitivity-Aware Thresholding and Token Routing for Activation Sparsification in Large Language Models
Large Language Models
Efficient ML
- Introduction of SATS, a sensitivity-aware method for threshold calibration in activation sparsification.
- Demonstrated improvement over percentile-based thresholding methods in terms of model quality at matched sparsity.
- Development of a token routing framework that allows dynamic selection of computation paths for each token.
- Token routing enhances the quality-throughput trade-off compared to static execution methods.
Read more
Sensitivity-Aware Thresholding and Token Routing for Activation Sparsification in Large Language Models
Summary
This paper addresses the challenge of efficient inference in Large Language Models (LLMs) by proposing two novel methods: Sensitivity-Aware Thresholding for Sparsity (SATS) and a lightweight token routing framework. SATS improves the calibration of layerwise gate thresholds by utilizing a local MLP output sensitivity proxy instead of relying on percentile-based methods. This approach allows for better preservation of model quality while achieving desired sparsity levels. The token routing framework dynamically selects between dense and sparse computation paths on a per-token basis, enhancing the quality-throughput trade-off compared to static activation modifications. Evaluations on recent open-weight LLMs, such as llama 3.1 8B and Qwen 3 8B, demonstrate that SATS outperforms traditional thresholding methods at matched sparsity levels, and token routing further optimizes performance, indicating significant potential for improving inference efficiency in LLMs.
Methodology
The authors propose SATS, which replaces percentile-based threshold selection with a sensitivity-aware approach that considers local MLP output distortion. Additionally, a token routing framework is introduced, allowing for dynamic selection of computation paths based on token identity, thereby optimizing inference without significant overhead.
Results
The evaluation results show that SATS provides better quality than traditional percentile-based thresholding at the same sparsity levels. Furthermore, the token routing method improves the quality-throughput trade-off compared to static sparse or dense model execution, demonstrating faster inference while maintaining model performance.
Implications
The findings suggest that implementing sensitivity-aware methods and dynamic token routing can lead to more efficient inference in LLMs, which is crucial for deploying these models in resource-constrained environments or real-time applications.
Mach-Mind-4-Flash Technical Report
Large Language Models
Reinforcement Learning
Efficient ML
- Mach-Mind-4-Flash achieves high performance with only 3B activated parameters.
- The model utilizes a novel training infrastructure that accelerates the training process by 17%.
- Domain-specific RL experts are trained in parallel and fused into a single generalist model.
- The Hybrid Median-length Policy Optimization method significantly reduces token generation length with minimal accuracy loss.
Read more
Mach-Mind-4-Flash Technical Report
Summary
The Mach-Mind-4-Flash is a 35B-parameter Mixture-of-Experts (MoE) model that activates only 3B parameters, achieving performance comparable to or exceeding that of 100B-parameter models through post-training optimization without increasing pre-training compute. The model leverages scalable agentic interaction environments for large-scale reinforcement learning, resulting in significant performance improvements on real-world tasks. The training pipeline consists of three main stages: a unified RL/On-Policy Distillation (OPD) infrastructure that accelerates training by 17%, the parallel training of domain-specific RL experts followed by their fusion into a generalist model using Multi-Teacher On-Policy Distillation (MOPD), and a Hybrid Median-length Policy Optimization (HMPO) method that enhances token efficiency while maintaining accuracy. The model demonstrates superior performance on various benchmarks, scoring 92.70 on AIME’26 and leading or matching larger models across multiple tasks, all while maintaining a fraction of the inference cost.
Methodology
The methodology involves a three-stage training pipeline: (1) a unified RL/OPD training infrastructure with dynamic multi-teacher scheduling, (2) parallel training of domain-specific RL experts followed by their fusion into a generalist model using MOPD, and (3) the application of HMPO for token-efficient reasoning.
Results
Mach-Mind-4-Flash scores 92.70 on AIME’26, 82.82 on IFBench, 80.74 on Behavioral-SafetyBench, and 75.80 on BFCL-v4, demonstrating competitive performance against models with 10-30 times its activated size at a lower inference cost.
Implications
The findings suggest that aggressive post-training strategies can effectively enhance the performance of smaller models, making them viable for latency-sensitive applications while reducing computational costs.
SYNRARE: Synthetic Rare Disease EHR Generation for ML Benchmarking
Generative Models
- SYNRARE enables the generation of synthetic EHRs for rare disease patients, facilitating ML benchmarking.
- The tool provides a no-code interface, making synthetic data generation accessible to a wider range of researchers.
- SYNRARE allows for the modeling of comorbidities based on empirical evidence, enhancing the realism of generated data.
- The framework supports the creation of patient cohorts with controlled dissimilarity from common diseases.
Read more
SYNRARE: Synthetic Rare Disease EHR Generation for ML Benchmarking
Summary
The paper introduces SYNRARE, a graphical user interface (GUI) built on the Synthea framework, designed to generate synthetic Electronic Health Records (EHRs) for patients with rare diseases (RD). The motivation behind SYNRARE stems from the challenges faced in diagnosing rare diseases due to their symptom similarities with common diseases, leading to delays in diagnosis. Machine Learning (ML) algorithms applied to EHRs have shown potential in accelerating these diagnoses, but privacy and ethical concerns limit access to real-world data. SYNRARE addresses this issue by allowing researchers to create synthetic EHRs that mimic real patient data while maintaining control over the degree of dissimilarity between common and rare disease patients. This enables effective benchmarking of ML algorithms in a controlled environment. The tool supports the generation of patient cohorts with definable characteristics, facilitating the evaluation of ML methods for rare disease detection. Additionally, SYNRARE includes a no-code interface to lower barriers for researchers and supports BMI-driven comorbidity modeling based on empirical evidence. The paper emphasizes the importance of synthetic data in overcoming the limitations posed by real-world data access and encourages contributions from clinical experts to enhance the accuracy of disease progression modeling.
Methodology
SYNRARE utilizes the Synthea framework to generate synthetic patient data. It allows users to modify existing disease modules to create EHRs that simulate patients with rare diseases. The GUI facilitates easy customization and generation of data without requiring programming skills, thus enabling rapid testing and evaluation of ML algorithms in a controlled setting.
Results
SYNRARE successfully generates synthetic EHRs that differ in a definable degree from common disease patients, allowing researchers to benchmark their ML algorithms effectively. The tool's no-code interface and support for comorbidity modeling enhance its usability and applicability in research.
Implications
The development of SYNRARE has significant implications for the field of medical informatics and machine learning. It provides a valuable resource for researchers to test and refine ML algorithms for rare disease diagnosis without the ethical and privacy concerns associated with real patient data. This can lead to improved diagnostic tools and faster identification of rare diseases, ultimately benefiting patient care.
TSAI-MetaFraud: A Benchmark Dataset for Financial Fraud Transaction and Behavioral Risk Detection in Metaverse Ecosystems
Multimodal
Graph Learning
- Introduction of TSAI-MetaFraud, a comprehensive dataset for fraud detection in metaverse environments.
- Integration of behavioral, transactional, and graph-structured information to reflect the complexities of virtual economies.
- Definition of benchmark tasks for systematic evaluation of fraud detection methods.
- Provision of baseline results using machine learning and graph neural network approaches.
Read more
TSAI-MetaFraud: A Benchmark Dataset for Financial Fraud Transaction and Behavioral Risk Detection in Metaverse Ecosystems
Summary
The paper presents TSAI-MetaFraud, a novel multimodal benchmark dataset designed to address the challenges of fraud detection in metaverse ecosystems. As virtual economies evolve, they face unique threats such as fraudulent transactions and automated bot activities, which existing datasets fail to adequately capture. TSAI-MetaFraud integrates behavioral, transactional, and graph-structured data, providing a comprehensive framework for analyzing fraud in virtual environments. The dataset includes realistic scenarios of both benign and malicious behaviors, enabling researchers to evaluate multimodal fraud detection methods effectively. The authors define several benchmark tasks, including transaction fraud detection and weakly supervised learning, and provide baseline evaluations using machine learning models and graph neural networks. This work aims to facilitate reproducible research and advance the field of trustworthy AI in metaverse contexts.
Methodology
The authors collected data from a metaverse environment built using OpenSimulator, capturing various aspects of user interactions, financial transactions, and graph-based relationships. They defined benchmark tasks for fraud detection and provided baseline evaluations using machine learning and graph neural networks to characterize the dataset's challenges.
Results
The paper establishes baseline performance levels for various tasks, demonstrating the dataset's utility in evaluating multimodal and graph-based methods for fraud detection. The results highlight the dataset's ability to facilitate research in fraud analytics and trustworthy AI.
Implications
TSAI-MetaFraud has significant implications for advancing research in fraud detection within metaverse ecosystems, enabling the development of more robust and trustworthy AI systems. It provides a foundation for future studies on multimodal learning and graph mining in complex virtual environments.
Super-Tuning: From Activation-Aware Pruning to Sparse Fine-Tuning
NLP
Large Language Models
Efficient ML
- Introduction of Super, a sparse PEFT method based on activation-weighted magnitude scores.
- Development of Supra, a hybrid method combining Super with LoRA under a fixed parameter budget.
- Evaluation of various sparse and low-rank adaptation methods on arithmetic reasoning tasks.
- Demonstration that simple pruning-inspired metrics can effectively guide parameter-efficient tuning.
Read more
Super-Tuning: From Activation-Aware Pruning to Sparse Fine-Tuning
Summary
This paper addresses the challenges of fine-tuning large language models (LLMs), which are resource-intensive due to their full-parameter updates. The authors propose a novel parameter-efficient fine-tuning (PEFT) method called Super, which utilizes saliency signals from pruning techniques to determine which model parameters to adapt. Super employs a Wanda-style activation-weighted magnitude score computed from a calibration pass to select a small set of trainable weights. Additionally, the authors introduce Supra, a hybrid adapter that combines the sparse updates from Super with low-rank adaptation (LoRA), ensuring a matched trainable-parameter budget through a straightforward budget-splitting rule. The paper evaluates these methods on the Math17K dataset using Llama-3.2-1B and Meta-Llama-3-8B models, demonstrating that the best variants of Super and Supra achieve superior accuracy compared to other adapter configurations. The findings suggest that pruning-inspired techniques can effectively inform sparse supports for PEFT, particularly when integrated with low-rank adapters.
Methodology
The authors propose Super, which selects a small subset of model weights for fine-tuning based on a training-free Wanda-style activation-weighted magnitude score. They also introduce Supra, which combines these sparse updates with LoRA, maintaining a transparent parameter-count budget through a budget-splitting rule. The methods are evaluated on the Math17K dataset, focusing on arithmetic reasoning tasks.
Results
In experiments, the best variants of Super and Supra achieved the highest average accuracy compared to other adapter configurations, including LoRA, SIFT, RoSA, and random sparse supports. The results indicate that both low-score supports and Wanda-style selections can be effective for PEFT, depending on the model size and training schedule.
Implications
The proposed methods could significantly reduce the computational resources required for fine-tuning large language models, making them more accessible for real-world applications. The findings also suggest that leveraging pruning techniques can enhance the efficiency of parameter updates in LLMs.
Towards the Explainability of Temporal Graph Networks via Memory Backtracking and Topological Attribution
Graph Learning
Interpretability
Time Series
- Introduces a framework for explaining predictions in Temporal Graph Networks by considering both spatial and temporal factors.
- Utilizes topology attribution and memory backtracking trees to quantify contributions from neighboring and historical events.
- Addresses limitations of existing explanation methods that ignore the memory module's role in TGNs.
- Demonstrates improved performance in providing faithful explanations compared to state-of-the-art methods.
Read more
Towards the Explainability of Temporal Graph Networks via Memory Backtracking and Topological Attribution
Summary
This paper addresses the challenge of explainability in Temporal Graph Networks (TGNs), which are increasingly used in applications like fraud detection and healthcare forecasting. While TGNs have demonstrated superior predictive accuracy, they often operate as black-box models, making it difficult to understand the historical events that influence their predictions. The authors propose a novel framework that incorporates memory backtracking and topological attribution to provide insights into TGN predictions. Specifically, they introduce a topology attribution tree to capture the contributions of neighboring events and a memory backtracking tree to quantify how historical events shape node memory vectors. The proposed method ensures that the total contributions of events align with the model's logits, addressing the limitations of existing explanation methods that overlook the memory module. The authors validate their approach through experiments on nine temporal graph datasets, demonstrating that their method yields faithful explanations and outperforms state-of-the-art baselines in various prediction tasks.
Methodology
The authors developed a framework that constructs a topology attribution tree to assess the influence of neighboring events and a memory backtracking tree to evaluate the impact of historical events on node memory vectors. They applied Layer-wise Relevance Propagation (LRP) to ensure that the contributions of events correspond to the model's logits. Additionally, they designed optimization objectives to identify significant events, addressing the challenges posed by nonlinear mappings from logits to probabilities.
Results
Experiments conducted on nine temporal graph datasets demonstrated that the proposed method provides more accurate and faithful explanations compared to existing state-of-the-art explanation methods. The results indicate that the framework effectively captures the contributions of both neighboring and historical events, leading to enhanced interpretability of TGN predictions.
Implications
The findings of this research have significant implications for the deployment of TGNs in high-stakes applications such as fraud detection and healthcare, where understanding model predictions is crucial for trust and accountability. By improving the explainability of TGNs, the proposed framework can facilitate better decision-making and enhance user trust in automated systems.
Similarity search generalisation in contrastive learning with InfoNCE loss
Theory
- Establishes a new perspective on InfoNCE loss by analyzing similarity search generalisation.
- Introduces a continuity bound for InfoNCE loss that incorporates an inverse temperature parameter.
- Demonstrates that increasing the number of negative samples stabilizes generalisation error for Lipschitz functions.
- Provides a theoretical framework that complements existing interpretations of InfoNCE in terms of mutual information.
Read more
Similarity search generalisation in contrastive learning with InfoNCE loss
Summary
This paper explores the theoretical underpinnings of similarity search generalisation in contrastive learning, specifically focusing on the InfoNCE loss function. The author demonstrates that the population risk associated with k negative samples in InfoNCE is O(1/k) close to an expected cross-entropy, which measures the deviation between a learned embedding's softmax similarity search on unseen data and an idealised search using similarity from the positive sample generator. The work introduces a new continuity bound for the InfoNCE loss, derived through Gˆateaux differentiation, which retains the averaging structure over negative samples and includes a tunable 'inverse temperature' parameter. This continuity bound shows that as k increases, the averaging effect stabilizes the generalisation error for Lipschitz embedding functions. The paper aims to clarify the theoretical understanding of InfoNCE and its implications for similarity search generalisation, contrasting it with previous analyses that primarily focused on downstream classification tasks.
Methodology
The paper employs theoretical analysis, specifically Gˆateaux differentiation, to derive a continuity bound for the InfoNCE loss. It examines the relationship between the population risk of InfoNCE and the expected cross-entropy in the context of similarity search, focusing on the implications of varying the number of negative samples.
Results
The main results indicate that the population risk with k negative samples approaches an expected cross-entropy, and the newly introduced continuity bound shows that the generalisation error stabilizes as k increases for Lipschitz embedding functions. This provides a clearer understanding of the trade-offs involved in using multiple negative samples in contrastive learning.
Implications
The findings have significant implications for the design and evaluation of embedding models in similarity search applications. By clarifying the role of negative samples, the research can inform better practices in training contrastive learning models, potentially leading to improved performance in real-world applications.