AI-generated summaries
Today's ML research,
without the noise.
Daily summaries of the latest machine learning papers from arXiv, processed every 8 hours.
24
Papers today
8h
Update frequency
7
Days of history
One Step, One Lead: Mitigating Higher-Order Interference in Multi-Domain Reinforcement Learning via Cross-Step Control
Reinforcement Learning
Large Language Models
Optimization
- Introduces a cross-step view of higher-order interference in multi-domain RL, shifting focus from same-point gradient relations.
- Proposes OSOL, an online cross-step controller that integrates with standard policy gradient updates.
- Demonstrates that token log-probability footprints can effectively capture interactions between consecutive updates.
- Achieves significant performance improvements on multi-domain tasks without the need for auxiliary models or higher-order derivatives.
Read more
One Step, One Lead: Mitigating Higher-Order Interference in Multi-Domain Reinforcement Learning via Cross-Step Control
Summary
This paper addresses the challenges of reinforcement learning (RL) in multi-domain settings, particularly the degradation of individual-domain performance due to higher-order interference during joint training. Traditional methods focus on first-order gradient alignment or curvature-based proxies, which fail to capture critical sequential interference. The authors introduce a novel perspective by analyzing token log-probability footprints from consecutive updates, revealing how these updates interact in output space. They propose a new method called OSOL (One Step, One Lead), which focuses on a designated domain at each iteration and utilizes the preceding checkpoint's footprint to rank token-level rebound risks. This approach allows for adaptive corrections within the standard policy gradient update, effectively mitigating cross-step output backtracking. Empirical results demonstrate that OSOL significantly improves performance on the Qwen3-30B-A3B model, achieving a domain-macro average of 0.4822, which is a 5.7% improvement over the strongest baseline without requiring explicit higher-order differentiation.
Methodology
The authors analyze the interactions between consecutive updates in multi-domain RL by examining token log-probability footprints. They propose OSOL, which designates a focus domain for each iteration, computes token-level drift from log-probability differences, and applies adaptive corrections to the policy gradient update. This method does not require auxiliary supervision or explicit higher-order derivatives, making it computationally efficient.
Results
OSOL achieves a domain-macro average of 0.4822 on the Qwen3-30B-A3B model, representing a 5.7% improvement over the strongest baseline (MGS) and outperforming other methods like CGPO. The results indicate that the proposed method effectively reduces cross-step output backtracking and enhances overall performance in multi-domain reinforcement learning.
Implications
The findings suggest that understanding and mitigating higher-order interference can lead to more stable and effective training of large language models across multiple domains. This could enhance the reasoning capabilities of these models, making them more versatile in real-world applications.
Leveraging Cardiac Imaging to Improve ECG-Based Detection of Chagas Disease in Resource-Constrained Settings
Multimodal
- Introduces an imaging-supervised ECG pre-training framework for Chagas disease detection.
- Demonstrates successful transfer of structural cardiac knowledge from CMR to ECG despite no prior exposure to Chagas cases.
- Achieves competitive performance on benchmark datasets, indicating the model's generalizability.
- Addresses the disparity in cardiovascular diagnostics in resource-constrained settings.
Read more
Leveraging Cardiac Imaging to Improve ECG-Based Detection of Chagas Disease in Resource-Constrained Settings
Summary
This paper addresses the challenge of diagnosing Chagas disease, a leading cause of cardiomyopathy in Latin America, particularly in resource-constrained settings where access to advanced imaging techniques like cardiac magnetic resonance (CMR) is limited. The authors propose a novel approach that leverages CMR-derived structural knowledge to enhance ECG-based detection of Chagas disease. They utilize a large dataset of 63,193 paired ECG and CMR examinations from the UK Biobank to train an ECG encoder through contrastive pre-training. This method aligns the ECG representations with a clinically grounded CMR embedding space using an asymmetric InfoNCE objective. Despite the absence of Chagas cases in the pre-training data, the resulting ECG representations significantly improve the detection of Chagas disease. The model achieves an AUROC of 0.851 and a Top5%-TPR of 0.427 in five-fold cross-validation, outperforming a baseline model. Furthermore, it demonstrates strong generalization capabilities, achieving the highest AUROC on the PhysioNet/CinC 2025 Challenge test set, indicating that imaging-supervised ECG representations can be effectively applied in diverse populations and settings beyond the original training distribution.
Methodology
The authors employed a contrastive learning approach to align ECG representations with CMR-derived structural information. They utilized an asymmetric InfoNCE objective for training on a dataset of paired ECG and CMR examinations, followed by fine-tuning the model for Chagas disease detection.
Results
The proposed model achieved an AUROC of 0.851 and a Top5%-TPR of 0.427 in five-fold cross-validation, surpassing the performance of an unaligned ECG-FM baseline. On the PhysioNet/CinC 2025 Challenge test set, it obtained the highest AUROC on SaMi-Trop-3 and the best score in the ELSA-Brasil challenge among top-performing methods.
Implications
This research suggests that structural cardiac knowledge from imaging can be effectively transferred to ECG-based diagnostics, potentially improving cardiovascular disease detection in low-resource settings. It paves the way for more equitable healthcare solutions by enabling the use of widely available ECG technology for diagnosing diseases typically assessed through costly imaging techniques.
Tracking the Moving Frontier: Long-Short Term Advantage Estimator
Reinforcement Learning
Large Language Models
Efficient ML
- LSTAE introduces a two-dimensional credit assignment mechanism for more efficient RL training.
- The method reduces the need for multiple trajectory samples, addressing sample inefficiency in traditional group-based RL methods.
- LSTAE maintains a historical experience tracker for long-term advantage estimation and a state-experience buffer for short-term advantage estimation.
- The approach has been validated across agentic and mathematical reasoning benchmarks, showing improved performance and reduced costs.
Read more
Tracking the Moving Frontier: Long-Short Term Advantage Estimator
Summary
The paper introduces the Long-Short Term Advantage Estimator (LSTAE), a novel single-stream reinforcement learning (RL) algorithm designed to enhance the efficiency of training large language model (LLM) agents. Traditional group-based RL with Verifiable Rewards (RLVR) methods require multiple trajectory samples for each prompt, leading to high costs and inefficiencies. LSTAE addresses this by utilizing historical experience for advantage estimation while updating the policy based solely on the current rollout. The method employs a dual framework: a long-term trajectory-level tracker that monitors the moving success frontier of task anchors and a short-term state-experience buffer that estimates localized action advantages. This two-timescale design allows LSTAE to convert accumulated experience into multi-granular credit signals, significantly reducing the number of rollouts needed. Experimental results demonstrate that LSTAE matches or surpasses the performance of existing group-based baselines while substantially lowering rollout costs, thus providing a more efficient approach to training LLM agents in complex environments.
Methodology
LSTAE employs a dual framework for advantage estimation: a long-term trajectory-level tracker that monitors the success frontier of task anchors and a short-term state-experience buffer that groups similar states for localized action advantage estimation. This allows the algorithm to utilize historical data while only requiring a single current rollout for policy updates.
Results
LSTAE demonstrated performance that matches or exceeds that of strong group-based RLVR baselines across various benchmarks, while significantly reducing the rollout costs associated with training LLM agents.
Implications
The findings suggest that LSTAE could lead to more efficient training of LLM agents in diverse applications, including web navigation, embodied control, and multi-turn tool usage, by enabling better utilization of historical experiences and reducing the computational burden of training.
Not Just Oversmoothing: Detecting the Echo Chamber Effect in Graph Neural Networks
Graph Learning
Theory
- Introduction of the Echo Chamber Effect as a failure mode in GNNs.
- Development of the Echo Chamber Index (ECI) to diagnose this effect.
- Characterization of the echo chamber's impact on node classification based on community-label alignment.
- Proposal of Community-Aware Split Propagation (CASP) to optimize message passing in GNNs.
Read more
Not Just Oversmoothing: Detecting the Echo Chamber Effect in Graph Neural Networks
Summary
This paper addresses a critical failure mode in Graph Neural Networks (GNNs) known as the Echo Chamber Effect, which occurs when intra-community representations collapse while inter-community separation remains. Traditional diagnostics for oversmoothing, such as Dirichlet energy and Mean Average Distance (MAD), fail to capture this phenomenon due to their global aggregation nature. The authors introduce the Echo Chamber Index (ECI), a metric that stratifies pairwise distances based on community membership, allowing for the detection of the echo chamber effect. They demonstrate that while Dirichlet energy may indicate alleviation of oversmoothing, ECI can remain elevated, revealing the presence of the echo chamber. The implications of this effect on node classification are explored, showing that it can enhance performance in homophilic settings (where communities align with classes) but degrade it in heterophilic settings (where communities do not align with classes). To address these challenges, the authors propose Community-Aware Split Propagation (CASP), a lightweight plugin that decouples intra- and inter-community aggregation, optimizing the balance based on label structure. CASP shows improvements across various GNN architectures in both homophilic and heterophilic scenarios.
Methodology
The authors developed the Echo Chamber Index (ECI) to stratify pairwise distances by community membership, allowing for the detection of the echo chamber effect. They conducted theoretical analyses to understand the implications of the echo chamber on node classification and proposed the Community-Aware Split Propagation (CASP) plugin, which modifies the aggregation process in GNNs to better balance intra- and inter-community message passing based on label structure.
Results
The introduction of ECI revealed the presence of the echo chamber effect in GNNs, which traditional metrics failed to detect. The analysis showed that the echo chamber can improve classification in homophilic settings while hindering it in heterophilic settings. CASP was shown to enhance the performance of various GNN architectures across different graph structures, effectively managing the trade-off between intra- and inter-community aggregation.
Implications
The findings suggest that understanding and addressing the echo chamber effect can lead to more effective GNN designs, particularly in applications involving community-structured data. The proposed methods can improve classification tasks in social networks, citation networks, and other domains where community structures are prevalent.
TrojanWorld: Backdooring World-Model Agents via Imagination Steering
Reinforcement Learning
Robotics
Theory
- TROJANWORLD introduces a backdoor framework for world-model agents that activates via a physical trigger.
- The framework combines Decision-Reflective Induction, Clean Behavior Anchoring, and Causal Propagation to enhance attack stealth and persistence.
- Experiments show that the framework can achieve a target-action deviation as low as 0.026 while maintaining over 98.8% clean performance.
- The induced malicious behavior can persist even after the trigger is removed, highlighting a significant security risk.
Read more
TrojanWorld: Backdooring World-Model Agents via Imagination Steering
Summary
The paper presents TROJANWORLD, a novel framework designed to implement backdoor attacks on world-model agents in reinforcement learning. World models are crucial for these agents as they simulate future dynamics and guide decision-making. The authors highlight the vulnerability of these models to supply chain threats, particularly through backdoor attacks that can remain dormant during validation but activate under specific conditions. TROJANWORLD utilizes a physical object as a trigger to induce attacker-specified behavior by steering the agent's internal imagination. The framework integrates three key mechanisms: Decision-Reflective Induction, which uses feedback from the agent's decision-making process to guide training; Clean Behavior Anchoring, which maintains performance fidelity in the absence of the trigger; and Causal Propagation, which ensures that the induced behavior persists even after the trigger is removed. The effectiveness of TROJANWORLD is evaluated across multiple world-model systems and benchmarks, demonstrating that it can achieve significant control over agent behavior while retaining high performance. The findings underscore the need for enhanced security measures tailored to interactive world-model agents.
Methodology
The authors developed the TROJANWORLD framework by embedding a backdoor during the training of world-model agents. They employed a physical object as a trigger to steer the agent's internal imagination towards malicious actions. The framework's effectiveness was evaluated using three world-model systems (TD-MPC2, DreamerV3, R2-Dreamer) across four benchmarks (DeepMind Control, MetaWorld, MyoSuite, RoboDesk).
Results
TROJANWORLD demonstrated the ability to induce attacker-specified actions with minimal deviation from normal performance. Under trigger activation, the framework achieved a target-action deviation of 0.026 while retaining at least 98.8% of clean performance. The compromised agents continued to exhibit malicious behavior even after the trigger was removed, indicating the persistence of the backdoor.
Implications
The findings of this research highlight significant security vulnerabilities in the deployment of world-model agents, particularly in scenarios where pretrained models are distributed. The ability to induce persistent malicious behavior raises concerns for applications in robotics and autonomous systems, necessitating the development of robust security measures to safeguard against such attacks.
Suan: Rectifying Direct Preference Safety Alignment in Large Language Models
NLP
Large Language Models
Optimization
- Suan is a new preference optimization algorithm that improves safety alignment in LLMs.
- The method formulates optimization objectives directly at the gradient level, enhancing interpretability.
- Suan addresses common issues in existing methods, such as over-refusal and quality degradation.
- Extensive evaluations show that Suan outperforms existing methods in safety alignment without sacrificing response utility.
Read more
Suan: Rectifying Direct Preference Safety Alignment in Large Language Models
Summary
The paper introduces Suan, a novel preference optimization algorithm aimed at enhancing safety alignment in Large Language Models (LLMs). The authors highlight the challenges faced by existing methods, particularly in open-weight models, which often suffer from over-refusal and degraded response quality. Suan differentiates itself by formulating the optimization objective directly at the gradient level, which leads to more interpretable and robust training dynamics. The authors conduct a systematic investigation into the shortcomings of current safety-focused preference optimization methods and demonstrate that Suan effectively addresses these issues. Through extensive evaluations against various competitive baselines, the results indicate that Suan achieves superior safety alignment while maintaining the utility of responses, thus providing a promising approach for ensuring that LLMs generate helpful and harmless content.
Methodology
The authors propose Suan as a direct alignment algorithm that optimizes preferences at the gradient level, bypassing traditional variational derivations. They conduct a systematic investigation into the challenges of safety-focused preference optimization and evaluate Suan against a diverse set of competitive baselines and benchmarks.
Results
Suan demonstrates state-of-the-art performance in safety alignment, significantly reducing over-refusal rates while preserving the quality of responses. The evaluations indicate that Suan effectively mitigates the risks associated with harmful content generation in LLMs.
Implications
The findings suggest that Suan could be a valuable tool for developing safer LLMs, particularly in applications where harmful content generation poses significant risks. This work may influence future research on safety alignment methodologies in both academic and industrial settings.
CLUES-WEASEL: No additional clues required to choose your time series clustering algorithm
Time Series
- CLUES-WEASEL is a new time series clustering algorithm that balances performance and speed.
- It employs an unsupervised feature extraction method based on WEASEL 2.0 and PCA for dimensionality reduction.
- Extensive experiments show that CLUES-WEASEL outperforms existing time series clustering algorithms significantly.
- The architecture allows compatibility with other feature extraction methods, enhancing its applicability.
Read more
CLUES-WEASEL: No additional clues required to choose your time series clustering algorithm
Summary
The paper introduces CLUES-WEASEL, a novel time series clustering algorithm that aims to improve clustering performance while maintaining faster runtimes compared to existing methods. CLUES-WEASEL utilizes an unsupervised feature extraction method derived from the WEASEL 2.0 algorithm, followed by dimensionality reduction using principal component analysis (PCA), and finally applies the k-means clustering algorithm on the reduced feature set. The author conducts extensive experiments comparing CLUES-WEASEL against state-of-the-art time series clustering algorithms, demonstrating that it significantly outperforms them in clustering quality while being faster. The findings also suggest that the architecture of CLUES-WEASEL is compatible with other time series feature extraction algorithms, indicating its versatility. The paper emphasizes the relevance of the unsupervised transformation step of WEASEL 2.0 for time series clustering, showcasing its effectiveness in extracting meaningful features for clustering tasks.
Methodology
The methodology involves three main steps: feature extraction using the unsupervised version of WEASEL 2.0, dimensionality reduction through PCA, and clustering using the k-means algorithm on the reduced feature set. The performance of CLUES-WEASEL is evaluated against various state-of-the-art time series clustering algorithms using the UCR time series classification archive.
Results
CLUES-WEASEL demonstrates superior clustering performance compared to existing algorithms, achieving better results while also being faster. The experiments confirm that the unsupervised transformation step of WEASEL 2.0 yields the best clustering results, and the algorithm's architecture is effective with other feature extraction methods.
Implications
The findings suggest that CLUES-WEASEL can be a valuable tool for practitioners in various fields that utilize time series data, such as healthcare, finance, and climate science, by providing an efficient and effective means of clustering time series data for analysis and decision-making.
Efficient Exploration Is Enough
Reinforcement Learning
Robotics
Theory
- Efficient exploration is redefined as prioritizing generalizable experiences over mere environment coverage.
- Theoretical results indicate that optimal exploration strategies involve visiting informative regions first.
- Empirical evidence shows that complex behaviors emerge from agents optimized for intrinsic exploration objectives.
- The study challenges the reward-centric view of behavior development in RL, suggesting intrinsic signals can suffice.
Read more
Efficient Exploration Is Enough
Summary
This paper presents a novel perspective on efficient exploration in reinforcement learning (RL), focusing on the intrinsic motivation of agents to generate generalizable experiences rather than maximizing extrinsic rewards. The authors define efficient explorers as agents that prioritize experiences that enhance their ability to predict and adapt to various environmental scenarios. Theoretical analysis shows that these agents can optimally schedule their exploration to visit the most informative regions first, thereby facilitating better learning and generalization. Empirical results demonstrate that optimizing for efficient exploration leads to the emergence of complex behaviors, such as navigating mazes and exploiting environmental symmetries, even in simple environments. The findings suggest that intrinsic exploration objectives can drive sophisticated behavior development without the need for external rewards, challenging traditional views that emphasize reward-driven learning. This work lays the groundwork for understanding how agents can sustain complex behavior generation in the absence of explicit tasks or goals.
Methodology
The authors employ a theoretical framework to analyze efficient exploration through prediction and generalization. They utilize Monte Carlo approximation methods to optimize agents in partially observable settings, allowing for empirical investigation of behavior emergence in neural network-parameterized agents.
Results
The study finds that agents optimized for efficient exploration exhibit increasingly complex behaviors, such as solving mazes and sophisticated navigation strategies. The optimization process leads to a natural curriculum of behavior complexity, demonstrating that intrinsic exploration can drive sophisticated behavior development.
Implications
This research has significant implications for the design of autonomous agents in various applications, including robotics and artificial intelligence, where complex behaviors are required without explicit external rewards. It suggests new avenues for developing agents that can learn and adapt in dynamic environments solely based on intrinsic motivations.
PPIM: Pennes Physics-Informed Mamba for Heat-Source-Conditioned 3D Bioheat Simulation
Theory
Optimization
Time Series
- Introduction of PPIM, a neural PDE solver specifically designed for bioheat simulations.
- Comparison of PPIM with existing models (PINN and PINNMamba) under controlled conditions.
- PPIM incorporates problem-specific features such as conditioned heat-source input and SSM decay initialization.
- Demonstrated superior accuracy in approximating temperature distributions in biological tissues.
Read more
PPIM: Pennes Physics-Informed Mamba for Heat-Source-Conditioned 3D Bioheat Simulation
Summary
This paper presents the Pennes Physics-Informed Mamba (PPIM), a novel neural solver for three-dimensional bioheat simulation based on the Pennes bioheat equation. The study focuses on simulating transient temperature distributions in biological tissues under localized heat-source conditions, specifically inspired by microwave ablation (MWA). The authors compare the performance of three neural PDE solvers: a spatial Fourier-feature physics-informed neural network (PINN), a generic PINNMamba temporal subsequence model, and the proposed PPIM. The PPIM enhances the temporal subsequence model by incorporating conditioned heat-source input and a Pennes-aware state-space model (SSM) decay initialization. All models are trained under identical conditions, with an explicit finite-difference method (FDM) solution serving as the numerical reference. In a representative simulation run of 600 seconds, PPIM demonstrated superior performance, achieving the lowest mean absolute error (MAE), relative L1 error, and relative L2 error among the evaluated models. The error maps indicated that the remaining errors in PPIM were concentrated near the heat-source region, suggesting its effectiveness in approximating the FDM reference temperature field.
Methodology
The authors developed PPIM as a temporal-subsequence-based neural PDE solver, integrating conditioned heat-source input and Pennes-aware SSM decay initialization. The performance of PPIM was evaluated against two other models (PINN and PINNMamba) using the same Pennes residual and a finite-difference method solution as a reference.
Results
PPIM achieved the lowest errors across all evaluated metrics, including MAE, relative L1 error, and relative L2 error, during a 600-second simulation run. Error maps revealed that PPIM's remaining errors were primarily localized near the heat-source region, indicating its effectiveness in capturing the dynamics of the temperature field.
Implications
The findings suggest that PPIM can be effectively utilized in thermal therapy simulations, particularly in applications like microwave ablation. The model's ability to accurately predict temperature distributions in biological tissues could enhance the development of digital twin models for personalized medicine and improve treatment planning in thermal therapies.
When Does Scale-Invariant Optimization Become Unstable? An Exact Schedule Law with Weight Decay
Optimization
Theory
- Derivation of a universal recurrence for effective directional stepsize in scale-invariant optimization.
- Identification of a sharp boundary separating contraction and expansion regimes governed by a single scalar quantity, Bt.
- Demonstration of intrinsic instability in the balance point of optimization dynamics, leading to recurrent behavior.
- Extension of findings to a homogeneous-optimizer framework, explaining varying stability across different optimization methods.
Read more
When Does Scale-Invariant Optimization Become Unstable? An Exact Schedule Law with Weight Decay
Summary
This paper investigates the dynamics of scale-invariant optimization in neural networks, particularly focusing on the interaction between learning-rate schedules and weight decay. The authors derive an exact discrete-time law that captures the effective directional stepsize in optimization, represented by a scalar quantity, Bt. This scalar encapsulates the effects of schedule forcing and geometric self-quenching, establishing a clear boundary between contraction and expansion regimes of the effective learning rate. The analysis reveals that the balance point of these dynamics is intrinsically unstable, indicating that constant learning rates with weight decay cannot maintain a stable equilibrium, leading to recurrent behavior. The authors extend their findings to a unified framework for various optimizers, explaining why adaptive methods tend to be less stable under normalization. Empirical validation across different neural network architectures confirms the predictive power of the derived law, allowing for direct control of training dynamics through the scalar Bt, which peaks performance at the identified boundary. Overall, the paper provides a comprehensive understanding of the training dynamics in scale-invariant optimization, offering actionable insights for optimizer design and schedule formulation.
Methodology
The authors employ a theoretical approach to derive an exact discrete-time law governing the effective stepsize in scale-invariant optimization. They analyze a normalized regression model to understand the dynamics in a simplified two-dimensional space. Additionally, they extend their analysis to various optimizers through a homogeneous-optimizer framework, classifying them based on their scaling behavior with parameter norms. Empirical validation is conducted across different neural network architectures to confirm the theoretical predictions.
Results
The study successfully derives a deterministic exact law for the effective directional stepsize, revealing a clear contraction/expansion boundary. The empirical results demonstrate that the scalar Bt can be controlled to optimize training performance, with accuracy peaking at the predicted boundary. The findings also highlight the instability of the balance point in optimization dynamics, confirming that constant learning rates with weight decay lead to recurrent behaviors.
Implications
The insights from this paper can significantly influence the design of learning-rate schedules and weight decay strategies in neural network training. By understanding the governing dynamics of scale-invariant optimization, practitioners can better control training processes, potentially leading to improved performance in various machine learning tasks.
Second-Order Smooth Planning with Optimal-Transport Bellman Smoothing
Reinforcement Learning
Optimization
Theory
- Introduces SecondOrderSmoothCruiser, achieving eO(ε−3) oracle complexity.
- Establishes a curvature-complexity principle linking Taylor remainder order to planning efficiency.
- Utilizes optimal-transport smoothing to enhance action distribution modeling.
- Develops gap-dependent extensions for improved oracle bounds.
Read more
Second-Order Smooth Planning with Optimal-Transport Bellman Smoothing
Summary
This paper presents a novel approach to planning in reinforcement learning by introducing the SecondOrderSmoothCruiser, which improves upon the existing SmoothCruiser method. The key innovation is the use of an optimal-transport (OT) smoothed Bellman backup over action distributions, which allows for a second-order approximation of the value function. The authors demonstrate that the sample complexity of this new method is eO(ε−3), a significant improvement over the first-order method's eO(ε−4). The paper establishes a curvature-complexity principle that links the local Taylor remainder of the Bellman aggregator to the oracle complexity, showing that higher-order approximations can lead to more efficient planning. The OT smoothing incorporates action geometry, enhancing the robustness of the value estimation while maintaining a closed form and explicit gradient policy. Additionally, the paper discusses gap-dependent extensions that further refine the oracle bounds, making the method applicable to a wider range of scenarios in simulation-based control and decision-making.
Methodology
The methodology involves deriving an optimal-transport-smoothed Bellman backup that utilizes a closed-form solution and a Lipschitz Hessian. The approach estimates the quadratic correction through a variance/cross-product identity, avoiding the need for explicit Hessian computation. The paper also explores the relationship between entropy-regularized and unregularized objectives through regularization-bias bounds.
Results
The main results indicate that the SecondOrderSmoothCruiser method achieves a worst-case oracle complexity of eO(ε−3) for fixed optimal-transport parameters, improving upon the previous eO(ε−4) complexity of SmoothCruiser. The paper also provides theoretical guarantees for the curvature-complexity principle and demonstrates the effectiveness of the OT smoothing in enhancing planning efficiency.
Implications
The findings suggest that incorporating higher-order approximations and optimal-transport techniques can lead to more efficient planning algorithms in reinforcement learning. This has potential applications in various fields such as robotics, game playing, and online decision-making, where efficient state value estimation is crucial.
GraphNOSE: A Graph Transformer in Olfaction
Graph Learning
- GraphNOSE outperforms traditional linear models and baseline GNNs in predicting olfactory qualities.
- The model achieves an AUROC of 84% on out-of-distribution compounds, surpassing the previous state-of-the-art.
- GraphNOSE utilizes fewer parameters while maintaining high predictive performance.
- Explainable AI methods reveal the molecular features driving odor predictions, enhancing interpretability.
Read more
GraphNOSE: A Graph Transformer in Olfaction
Summary
The paper introduces GraphNOSE, an innovative graph transformer framework designed to predict olfactory qualities from molecular structures, specifically using simplified molecular-input line-entry system (SMILES) strings. Traditional linear models struggle with extrapolating to novel chemical structures and complex odor mixtures, which GraphNOSE aims to address. By incorporating positional and structural encodings within a transformer-based architecture, GraphNOSE achieves superior performance with significantly fewer parameters compared to standard graph neural networks (GNNs). The model demonstrates an average improvement of 4.52% in area under the ROC curve (AUROC) over various baseline methods, achieving an AUROC of 84% on out-of-distribution compounds. This performance surpasses the previous state-of-the-art GNN for olfactory prediction, OpenPOM, which achieved 81% AUROC. Additionally, the authors apply explainable AI (XAI) techniques to discern which molecular features and substructures influence odor predictions, providing insights that align with chemical intuition. Overall, GraphNOSE establishes itself as a scalable and interpretable solution for olfactory predictions, particularly for structurally diverse compounds that are underrepresented in existing databases.
Methodology
GraphNOSE employs a graph transformer architecture that integrates positional and structural encodings to predict multi-label odor descriptors from SMILES strings. The model is evaluated against various baseline methods, including linear models and other GNNs, using metrics such as AUROC to assess performance.
Results
GraphNOSE achieved an AUROC of 84% on out-of-distribution compounds, outperforming the previous best model (OpenPOM) by 3%. The model demonstrated an average AUROC improvement of 4.52% over baseline methods, while utilizing six times fewer parameters than standard GNNs.
Implications
The development of GraphNOSE has significant implications for the fields of cheminformatics and olfaction, providing a robust tool for predicting odor qualities from molecular structures. Its ability to generalize to novel compounds could enhance the understanding of olfactory perception and improve applications in the food, flavor, and fragrance industries.
Forecasting the Winner of a Live Tennis Match
Time Series
- The study utilizes a large dataset of Grand Slam matches to analyze live tennis forecasting.
- Trace, a hybrid model, achieved high accuracy rates at various stages of match progress.
- Effective integration of pre-match and live data is crucial for accurate win probability estimates.
- The research highlights the importance of data preprocessing and feature engineering in model performance.
Read more
Forecasting the Winner of a Live Tennis Match
Summary
This paper addresses the challenge of forecasting the winner of live tennis matches, particularly in the context of the growing live sports betting market. The authors explore how to effectively integrate pre-match and live data to produce accurate win probability estimates. Utilizing a dataset of 8,222 Grand Slam matches and over 1.5 million point-level data points, the study evaluates five different models through a chronological split of training, validation, and testing data spanning from 2011 to 2024. The hybrid model, named Trace, demonstrated significant accuracy improvements, achieving 76.06%, 82.15%, and 88.34% accuracy at 25%, 50%, and 75% match progress, respectively. This suggests that hybrid modeling is a viable approach for live tennis forecasting, balancing the incorporation of live data while mitigating the risks of overfitting or underfitting. The study emphasizes the importance of preprocessing and feature engineering in creating robust predictive models for dynamic sports environments.
Methodology
The authors employed a chronological split of data, using matches from 2011-2021 for training, 2022 for validation, and 2023-2024 for testing. They developed and evaluated five models, focusing on the hybrid model Trace, which integrates pre-match Elo ratings and live match statistics. The data was preprocessed to ensure valid match states were used for predictions, and various features were engineered to capture player performance accurately.
Results
The hybrid model Trace achieved accuracies of 76.06%, 82.15%, and 88.34% at 25%, 50%, and 75% match progress, respectively. These results indicate that the model effectively adapts to the dynamic nature of live tennis matches and successfully integrates both pre-match and live data.
Implications
The findings of this study have significant implications for the sports betting industry, particularly in enhancing the accuracy of live betting models. The approach can be applied to other sports and real-time prediction scenarios, potentially improving decision-making processes in dynamic environments.
Prevalence calibration as shortcut mitigation
Computer Vision
Theory
Algorithmic fairness
- Shortcut learning is reframed as a calibration problem, highlighting the need for prevalence-equalized calibration.
- Two new encoder-agnostic methods are proposed: an in-processing regularizer and a post-hoc recalibration step.
- The methods significantly outperform existing baseline approaches in mitigating shortcut learning.
- The study connects shortcut learning to calibration theory and algorithmic fairness, emphasizing the importance of model calibration.
Read more
Prevalence calibration as shortcut mitigation
Summary
This paper addresses the issue of shortcut learning in medical image classification, where classifiers exploit spurious correlations instead of diagnostic features. The authors propose a novel approach that reframes shortcut learning as a calibration problem, suggesting that existing mitigation strategies are limited, especially when dealing with frozen foundation model encoders. They introduce two encoder-agnostic methods for prevalence-equalized calibration: an in-processing regularizer and a post-hoc recalibration step. These methods aim to equalize the calibration across different shortcut groups, thereby improving model confidence and performance. The authors validate their approach through extensive experiments on chest-drain–pneumothorax benchmarks using CheXpert and SIIM-ACR datasets, demonstrating significant improvements over baseline methods. The findings highlight the connection between shortcut learning, calibration theory, and algorithmic fairness, suggesting that addressing calibration can mitigate shortcut reliance effectively.
Methodology
The authors propose two methods for prevalence-equalized calibration: an in-processing regularizer that adjusts the training process to account for shortcut group prevalence, and a post-hoc recalibration step that modifies the classifier's output probabilities after training. Both methods are designed to be encoder-agnostic, allowing them to be applied to models with frozen encoders. Additionally, a hyperparameter optimization scheme is developed to fine-tune the regularization constants for optimal performance.
Results
The proposed methods were tested on chest-drain–pneumothorax classification tasks, achieving a significant increase in misaligned-group AUROC from 0.23 to 0.73 for a standard ERM-trained DenseNet. This indicates that the reliance on shortcuts degrades the classification head rather than the underlying representation. The results demonstrate that the new methods substantially outperform all baseline approaches across various experiments.
Implications
The findings suggest that improving model calibration can effectively mitigate shortcut learning, which is crucial for enhancing the reliability of medical image classifiers. This has broader implications for algorithmic fairness in machine learning, as it emphasizes the need for models to reflect true disease prevalence and avoid overconfidence in spurious correlations.
Stability and Generalization of Straight-Through Estimators for Training Two-Layer Quantized Neural Networks
Theory
Efficient ML
Optimization
- Introduces a convex representation of the samplewise hinge-loss identity-STE recursion.
- Establishes a framework for algorithmic stability analysis of STE training without Gaussian assumptions.
- Derives explicit generalization bounds and excess risk guarantees for STE-trained models.
- Demonstrates the connection between latent vector stability and full first-layer weight matrix stability.
Read more
Stability and Generalization of Straight-Through Estimators for Training Two-Layer Quantized Neural Networks
Summary
This paper investigates the identity straight-through estimator (STE) for training two-layer binary-activation networks with hinge loss, focusing on the relationship between algorithmic stability and statistical generalization. The author derives an exact distance identity for two coupled updates and proves approximate non-expansiveness of the common-example map, leading to explicit â„“2 on-average model-stability and generalization bounds. The study reveals that the zero-initialized samplewise STE recursion corresponds to stochastic subgradient descent on a convex latent loss, enabling a stability analysis without the need for Gaussian covariates or a planted ground-truth network. The findings include an explicit excess induced-risk guarantee and optimal-order expected misclassification error under margin separability, highlighting the potential of STEs in quantized neural network training while addressing theoretical gaps in existing literature.
Methodology
The paper employs a theoretical approach rooted in Statistical Learning Theory (SLT) to analyze the stability and generalization of the identity STE. It constructs a convex latent loss representation and applies stability analysis techniques to derive generalization bounds, focusing on the dynamics of a two-layer binary-activation network.
Results
The author establishes that the identity-STE training leads to an exact rank-one representation of the latent loss, allowing for a stability analysis that yields explicit generalization bounds. The results include an O(n−1/2) rate for excess induced-risk and an optimal-order O(R2/(γ2n)) expected misclassification error under margin separability conditions.
Implications
The findings provide a deeper theoretical understanding of STEs in quantized neural networks, potentially guiding future research and applications in efficient deep learning, particularly in resource-constrained environments. The results may influence the design of training algorithms for quantized models, enhancing their deployment in practical applications.
FedRAW: Preserving Rare-Label Influence in Asynchronous Federated Learning
Federated Learning
- Introduces the concept of silent rarity failure in asynchronous federated learning.
- Proposes FedRAW, which combines update deduplication and rare-label-aware weighting.
- Demonstrates that FedRAW preserves rare-label influence while maintaining global accuracy.
- Shows that existing methods do not adequately address the unique challenges posed by rare-label clients in AFL.
Read more
FedRAW: Preserving Rare-Label Influence in Asynchronous Federated Learning
Summary
The paper introduces FedRAW, a novel approach to address the issue of silent rarity failure in asynchronous federated learning (AFL). Silent rarity failure occurs when clients with rare labels contribute insufficiently to the global model, despite overall accuracy appearing stable. This issue arises from participation bias, where rare-label clients submit updates less frequently, and aggregation bias, where their updates receive inadequate influence during aggregation. FedRAW employs a server-side aggregation method that includes client-level update deduplication to prevent frequently updating clients from dominating the update buffer and rare-label-aware weighting to enhance the influence of clients with low-coverage labels. The authors demonstrate that FedRAW effectively increases the representation of rare-label clients while maintaining global model accuracy and requiring minimal additional server-side computation. The experiments conducted on datasets such as EMNIST Balanced, CIFAR-10, HAM10000, and ISIC-2019 show significant improvements in rare-label accuracy without compromising overall performance.
Methodology
FedRAW employs a two-pronged approach: (1) client-level update deduplication to limit the influence of frequently updating clients in the server-side buffer, and (2) rare-label-aware weighting to increase the aggregation influence of clients with rare labels. This method is implemented without altering local training, client objectives, or communication protocols.
Results
The implementation of FedRAW resulted in improved rare-label accuracy across various datasets, including EMNIST Balanced, CIFAR-10, HAM10000, and ISIC-2019, while preserving comparable global accuracy. The results indicate that rare-label accuracy can increase significantly, addressing the silent rarity failure issue effectively.
Implications
FedRAW has the potential to enhance the reliability of federated learning models in scenarios where certain labels are underrepresented. This approach can be particularly beneficial in applications involving imbalanced datasets, such as medical imaging or anomaly detection, where rare classes are critical for model performance.
Analysis of Respiratory Sinus Arrhythmia with Neural Networks
Time Series
- Introduction of a neural network-based method for estimating respiratory rates from ECG signals.
- Development and evaluation of three neural network architectures for feature extraction.
- Demonstration of improved robustness and scalability compared to traditional ECG processing methods.
- Potential applications in healthcare and wearable technology for non-invasive monitoring.
Read more
Analysis of Respiratory Sinus Arrhythmia with Neural Networks
Summary
This paper presents a novel approach for analyzing ECG signals to estimate respiratory rates by utilizing the phenomenon of Respiratory Sinus Arrhythmia (RSA). The authors developed a deep learning model that predicts respiratory waveforms directly from ECG data, eliminating the need for manual preprocessing. They evaluated three different neural network architectures capable of automatically extracting relevant features from ECG signals. The proposed method demonstrates robustness and scalability for non-invasive respiratory monitoring, with significant implications for healthcare and wearable technology. The study highlights the limitations of traditional ECG signal processing methods, which often require task-specific designs and manual feature extraction, making them less adaptable and sensitive to noise. In contrast, the neural network-based approach shows superior performance, especially in noisy conditions, and allows for easy personalization through individual patient data integration.
Methodology
The authors employed deep learning techniques to analyze ECG signals, focusing on the extraction of features relevant to respiratory monitoring. They trained neural network models to predict respiratory waveforms from ECG input data, comparing their performance against traditional methods.
Results
The neural network-based approach outperformed traditional ECG signal processing methods, particularly in noisy environments. The models demonstrated effective detection of R-wave peaks and accurate predictions of respiratory rates, showcasing the potential for real-time monitoring applications.
Implications
The findings suggest that neural network algorithms can significantly enhance non-invasive respiratory monitoring, making them suitable for integration into healthcare systems and wearable devices. This could lead to improved patient outcomes through better monitoring of respiratory health.
Selective Posterior Margin Regularization for Forward-Corrected Classification
Computer Vision
Robotics
Theory
- Introduction of Selective Posterior Margin Regularization (SPMR) to enhance forward correction in noisy-label learning.
- SPMR utilizes the disagreement between leading posterior classes to inform selective updates to the clean classifier.
- Demonstrated significant accuracy improvements across multiple benchmarks compared to existing methods.
- SPMR maintains the primary objective of forward correction while addressing ambiguous label conflicts.
Read more
Selective Posterior Margin Regularization for Forward-Corrected Classification
Summary
This paper addresses the challenge of learning with class-conditional label noise, particularly in the context of forward correction methods that utilize a transition model to map latent clean classes to observed annotations. The authors introduce Selective Posterior Margin Regularization (SPMR), a novel approach that enhances the forward correction framework by incorporating a selective update mechanism based on the posterior distribution of clean classes. SPMR leverages the disagreement between the leading reverse-posterior class and the observed annotation to inform updates to the clean classifier. The method scales the influence of these updates based on the separation between the leading posterior classes, allowing for more nuanced adjustments in the presence of label noise. The authors demonstrate that SPMR maintains the primary objective of maximizing the corrected likelihood while effectively addressing ambiguous cases. Through extensive experiments on five known-transition benchmarks, SPMR shows significant improvements over existing methods, including a 2.5–7.0 percentage point increase in accuracy compared to full-length Forward correction and outperforms alternatives like Mixup and early stopping. The findings suggest that SPMR can be effectively applied across various scenarios, including estimated transitions and human annotations, highlighting its versatility and robustness in noisy-label learning.
Methodology
The authors formulate the problem of auxiliary supervision in forward correction as an evidence–target–action framework. SPMR uses the gap between the two leading reverse-posterior classes to determine the strength of updates applied to the clean classifier. The method employs a pairwise margin that is adjusted based on the transition model, allowing for selective emphasis on clearer class separations while downplaying ambiguous cases.
Results
SPMR achieves accuracy improvements of 2.5–7.0 percentage points over full-length Forward correction and remains 0.7–2.5 percentage points above Forward with Mixup and early stopping. The method's effectiveness is supported by matched interventions that isolate the contributions of posterior evidence, pairwise direction, and transition-adjusted targets.
Implications
The findings suggest that SPMR can significantly enhance the robustness of classifiers in environments with noisy labels, making it applicable in various domains such as computer vision and robotics where label noise is prevalent. The approach's adaptability to different scenarios indicates its potential for broader applications in machine learning.
Data Efficient Sample Selection for In-Context Learning
NLP
Large Language Models
Efficient ML
- DEARICL models demonstration example selection as a subset ranking problem.
- It employs a non-linear surrogate and a gap-index bandit algorithm for improved sample efficiency.
- The framework achieves significant accuracy improvements over existing linear bandit methods.
- DEARICL is designed to adapt to unseen queries, enhancing the flexibility of ICL.
Read more
Data Efficient Sample Selection for In-Context Learning
Summary
This paper addresses the challenge of selecting optimal demonstration examples for In-Context Learning (ICL) in large language models (LLMs). Traditional methods for sample selection often fail to account for the complex relationships between selected samples and LLM performance, typically relying on static task-level selection that does not generalize well to unseen queries. The authors propose DEARICL (Data Efficient Algorithm for Ranking), a novel framework that reformulates sample selection as a subset ranking problem. DEARICL utilizes a non-linear surrogate model combined with a gap-index bandit algorithm to efficiently distinguish between optimal and borderline example subsets. This approach allows for instance-level subset ranking, significantly improving sample efficiency and performance. The authors demonstrate that DEARICL achieves accuracy gains of 8.08% to 15.9% over strong linear bandit baselines on exemplar selection benchmarks with open-source LLMs, showcasing its effectiveness in adapting to new tasks with fewer examples.
Methodology
The authors introduce DEARICL, which utilizes a gap-index multi-armed bandit framework to focus on top-m selection as an auxiliary objective. It fits a non-linear surrogate model in the inner loop for instance-level subset ranking, ensuring effective differentiation between optimal and borderline subsets based on LLM feedback.
Results
DEARICL demonstrates accuracy improvements of 8.08% to 15.9% compared to strong linear bandit baselines on exemplar selection benchmarks, while maintaining low sample complexity.
Implications
The proposed framework can enhance the performance of LLMs in various applications requiring efficient sample selection, such as few-shot learning and domain adaptation, by enabling better task-specific adaptation without extensive retraining.
I Don't Miss You, but I Do: Self-Explanation Faithfulness of Modality Missingness in Vision-Language Models
Multimodal
- Introduces a novel protocol for evaluating self-explanations in VLMs regarding modality dynamics.
- Models significantly underestimate the impact of restoring missing modalities on predictions.
- Insufficiency claims are rare but accurate when made, indicating a lack of awareness in models.
- Retrospective attributions show models over-credit single-modality sufficiency.
Read more
I Don't Miss You, but I Do: Self-Explanation Faithfulness of Modality Missingness in Vision-Language Models
Summary
This paper investigates the ability of vision-language models (VLMs) to accurately self-explain the impact of missing modalities on their predictions. The authors introduce an interventional evaluation protocol that assesses how well models can articulate the influence of each modality on their outputs, particularly when some modalities are absent. The study evaluates eight open-weight VLMs across four tasks, revealing a consistent trend where models overstate the sufficiency of available evidence and significantly underestimate the impact of restoring missing modalities. Specifically, while models predict a median change rate of only 8.8% upon restoring missing modalities, actual interventions show a change rate of up to 72.1%. Furthermore, claims of insufficiency are rare but accurate when made, with restoration leading to a change in predictions in 78-100% of cases flagged. The findings highlight that VLMs mischaracterize their reliance on available evidence, suggesting that executable interventions provide a more reliable ground truth for evaluating self-explanations in multimodal contexts.
Methodology
The authors developed an interventional protocol that involves both retrospective attribution and prospective counterfactual self-knowledge. They tested models by removing or restoring modalities and comparing the models' stated beliefs about their reliance on inputs with their actual behavior under these conditions. This involved evaluating eight VLMs across four distinct tasks with varying modality relationships.
Results
The results indicate that VLMs systematically overstate the sufficiency of available modalities, with a median predicted change rate of 8.8% compared to an actual change rate of 72.1% when restoring missing modalities. Insufficiency claims were found to be rare but precise, and retrospective evaluations showed that models often over-credit the sufficiency of single modalities.
Implications
These findings suggest that VLMs may not be reliable in communicating their dependence on different modalities, which has implications for their deployment in real-world applications where input modalities may be incomplete. The study advocates for the use of executable interventions as a standard for evaluating the faithfulness of model explanations.
When Retain Constraints Conflict: Mitigating Forget-Retain Interference in Tabular Data
Large Language Models
Optimization
Theory
- Identifies schema-induced forget-retain overlap as a significant challenge in tabular unlearning.
- Proposes Conflict-Aware Unlearning (CAU) to relax preservation constraints on conflicting retain records.
- Demonstrates that CAU outperforms traditional unlearning methods in high-overlap scenarios.
- Shows that reliable unlearning in tabular data requires explicit management of schema-induced overlaps.
Read more
When Retain Constraints Conflict: Mitigating Forget-Retain Interference in Tabular Data
Summary
This paper addresses the challenge of machine unlearning in tabular data, focusing on the conflict between forgetting specific records while retaining others. The authors identify a unique issue in tabular unlearning known as schema-induced forget-retain overlap, where records share fixed attributes and can be closely related in the model's representation space. This overlap complicates the unlearning process, as forgetting one record may inadvertently affect nearby retained records. To tackle this, the authors propose a novel method called Conflict-Aware Unlearning (CAU), which adjusts the preservation constraints on retained records that are in conflict with the forget set. CAU aims to optimize the unlearning process without altering the forgetting objective or the optimization algorithm itself. The effectiveness of CAU is evaluated through sample-level and feature-level unlearning tasks across various datasets, demonstrating that it significantly improves the model's ability to unlearn while maintaining predictive utility. The findings suggest that careful consideration of schema-induced overlaps is crucial for effective unlearning in tabular contexts.
Methodology
The authors introduce Conflict-Aware Unlearning (CAU), which modifies the construction of retain constraints by relaxing the preservation requirements for retain records that are closely aligned with the forget set in schema space. This method is applied to both sample-level and feature-level unlearning tasks, allowing for a more nuanced approach to managing the forget-retain conflict.
Results
The evaluation of CAU reveals that it closely matches the performance of a retraining oracle while maintaining predictive utility and effective retain-region behavior. The results indicate that CAU is particularly beneficial in high-overlap scenarios, where traditional methods struggle to balance forgetting and retaining.
Implications
The findings of this research have significant implications for machine learning applications in high-stakes domains where tabular data is prevalent, such as healthcare and finance. By improving the unlearning process, models can better comply with data privacy regulations and adapt to changing data requirements without extensive retraining.
ACE: Adapter Consolidation across Experts for Parameter-Efficient Fine-Tuning of MoE LLMs
NLP
Large Language Models
Efficient ML
- ACE consolidates redundant expert-specific adapters into shared higher-rank modules, improving adaptation efficiency.
- Grouped adapter execution reduces computational overhead by minimizing small matrix multiplication operations.
- ACE achieves higher mean accuracy and faster training times compared to traditional expert-wise LoRA methods.
- The method maintains the same parameter budget while enhancing gradient supervision and execution efficiency.
Read more
ACE: Adapter Consolidation across Experts for Parameter-Efficient Fine-Tuning of MoE LLMs
Summary
The paper introduces ACE (Adapter Consolidation across Experts), a novel method for parameter-efficient fine-tuning (PEFT) of mixture-of-experts (MoE) models. Traditional PEFT approaches, such as LoRA, attach separate low-rank adapters to each expert, leading to fragmentation in adaptation capacity, sparse gradient supervision, and inefficient execution. The authors observe that many expert-specific adapters become functionally similar during fine-tuning, indicating redundancy. ACE addresses this by grouping similar experts and replacing their individual adapters with shared higher-rank LoRA modules, optimizing the use of a fixed parameter budget. Additionally, ACE implements grouped adapter execution, consolidating computations into fewer, larger operations, thus enhancing efficiency. The method was evaluated across 12 datasets and four MoE architectures, demonstrating significant improvements in adaptation quality and training speed without increasing peak memory usage.
Methodology
ACE employs a similarity-based grouping technique to identify functionally similar expert adapters during fine-tuning. It replaces these adapters with shared higher-rank LoRA modules, optimizing the adaptation capacity under a fixed parameter budget. The method also introduces grouped adapter execution to streamline computations, reducing the number of small GEMM operations required.
Results
ACE consistently outperformed traditional expert-wise LoRA methods across various evaluations, achieving the highest mean accuracy on three MoE backbones while providing a 1.31× to 1.48× speedup in wall-clock training time. For instance, on OLMoE, ACE improved average accuracy from 74.11% to 75.61% across commonsense benchmarks.
Implications
The findings suggest that ACE can significantly enhance the efficiency of fine-tuning large language models, making it a valuable approach for adapting MoE architectures to various downstream tasks. This could lead to more effective utilization of computational resources in NLP applications.
Miles v0.1: Production-Level Post-Training
Reinforcement Learning
Large Language Models
Generative Models
- Miles v0.1 is a full-stack system for post-training in reinforcement learning, focusing on large language models.
- The architecture supports various training backends and weight synchronization methods for flexible deployment.
- The system addresses challenges in throughput and fidelity during rollout generation, crucial for agentic RL.
- An end-to-end case study showcases the system's performance on a large-scale model with significant efficiency.
Read more
Miles v0.1: Production-Level Post-Training
Summary
Miles v0.1 is introduced as a comprehensive, production-ready system designed for frontier post-training, particularly in the context of reinforcement learning (RL) for large language models. The system is built on the principles of cleanliness, verification, and customization, aiming to enhance accuracy, efficiency, reliability, and scalability. The architecture includes rollout engines based on SGLang, a trainer that supports two backends (NVIDIA Megatron-LM and PyTorch FSDP), and various weight-synchronization methods tailored for different deployment scenarios. Miles v0.1 not only facilitates full-parameter RL but also accommodates LoRA RL, on-policy distillation, supervised fine-tuning, and aligns rollout-training with true-on-policy methods. The report details the end-to-end system, including a case study demonstrating fully asynchronous agentic RL on a GLM-5.2 model using 64 NVIDIA GB300 GPUs, achieving a median step time of 263 seconds over the initial 30 steps. The paper emphasizes the challenges of sustaining high hardware utilization and maintaining fidelity in trajectory generation, proposing solutions through optimized request routing and decoupled training processes.
Methodology
The methodology involves a three-stage RL training loop: rollout generation using SGLang engines, policy training with either NVIDIA Megatron-LM or PyTorch FSDP, and weight synchronization to minimize disruptions. The system employs fully asynchronous scheduling to enhance throughput and maintain fidelity in trajectory generation.
Results
The case study demonstrated that Miles v0.1 could effectively run fully asynchronous agentic RL on a GLM-5.2 model, achieving a median step time of 263 seconds across the first 30 measured steps, indicating high efficiency and scalability.
Implications
Miles v0.1 has the potential to democratize access to frontier-scale reinforcement learning, making it more feasible for both researchers and enterprises to implement advanced RL techniques. Its design principles may influence future systems in the field of machine learning, particularly for large-scale applications.
AI and TCAD for Inverse Design and Defect Discovery: From Simple Machine Learning to LLM
Large Language Models
Multimodal
Optimization
- AI can significantly enhance semiconductor device design and defect discovery despite data limitations.
- TCAD-generated data can be effectively used to train ML models that perform at the level of experienced engineers.
- Auto-encoder-based models and noise engineering are effective in learning complex relationships in semiconductor physics.
- The integration of LLMs and MLLMs is crucial for automating and improving TCAD processes.
Read more
AI and TCAD for Inverse Design and Defect Discovery: From Simple Machine Learning to LLM
Summary
This paper explores the integration of artificial intelligence (AI) and Technology Computer-Aided Design (TCAD) in semiconductor device design and defect discovery. The author discusses the limitations of traditional machine learning (ML) approaches in this domain, primarily due to data scarcity and the complexities of semiconductor physics. The paper introduces a simulation-augmented ML framework that utilizes TCAD-generated data to train models capable of performing tasks typically reserved for experienced device engineers. The author demonstrates the effectiveness of auto-encoder-based models and noise engineering techniques in learning latent physics from TCAD data, which can then be applied to experimental datasets. Various examples illustrate the step-by-step construction of models for tasks such as inverse design of PiN diode layer thickness, Ga2O3 Schottky diode doping variations, and transistor contact resistance detection. The paper also emphasizes the potential of large language models (LLMs) and multimodal LLMs (MLLMs) in automating TCAD command file creation and enhancing device design processes. Overall, the work highlights the transformative potential of AI in semiconductor engineering, particularly through the use of simulation-augmented ML.
Methodology
The methodology involves using TCAD simulations to generate data for training machine learning models. The paper discusses various ML techniques, including regression, autoencoders, and noise engineering, to learn relationships between structural parameters and electrical characteristics. The models are validated against both simulated and experimental data.
Results
The results demonstrate that the proposed simulation-augmented ML framework can accurately predict device parameters and identify defects, achieving performance comparable to that of experienced engineers. The models successfully reverse-engineer parameters from experimental IV curves, although challenges with overfitting and generalization to experimental data are noted.
Implications
The findings suggest that AI and simulation-augmented ML can revolutionize semiconductor design processes, making them more efficient and accessible. The potential use of LLMs and MLLMs could further streamline TCAD operations and enhance the design of advanced semiconductor devices.