AI-generated summaries
Today's ML research,
without the noise.
Daily summaries of the latest machine learning papers from arXiv, processed every 8 hours.
24
Papers today
8h
Update frequency
7
Days of history
RDDMPI: Residual Denoising Diffusion Model for Probabilistic Multivariate Time Series Imputation
Time Series
- RDDMPI reformulates MTSI as a baseline-residual decomposition, separating deterministic reconstruction from probabilistic modeling.
- The framework operates in residual space, allowing for focused correction of systematic errors.
- A reliability-aware conditioning mechanism is introduced to manage the influence of baseline predictions.
- RDDMPI shows significant improvements in both accuracy and uncertainty quantification compared to existing methods.
Read more
RDDMPI: Residual Denoising Diffusion Model for Probabilistic Multivariate Time Series Imputation
Summary
This paper addresses the challenge of multivariate time series imputation (MTSI), which is crucial for various applications such as healthcare and traffic monitoring. Traditional methods struggle with the complexity of capturing temporal dynamics and inter-variable dependencies, especially when dealing with missing data. The authors propose RDDMPI, a novel framework that reformulates MTSI as a baseline-residual decomposition. Instead of directly modeling the entire missing signal, RDDMPI first uses a pretrained deterministic model to generate a baseline signal and then applies a diffusion process to model the residual uncertainty. This approach simplifies the generative task by focusing on correcting systematic errors rather than reconstructing the full signal. Additionally, RDDMPI incorporates a reliability-aware conditioning mechanism that adaptively adjusts the influence of baseline predictions based on their reliability. Experimental results demonstrate that RDDMPI consistently outperforms existing methods in terms of reconstruction accuracy and uncertainty quantification across multiple benchmark datasets.
Methodology
RDDMPI employs a two-stage process: first, a deterministic model generates a baseline-completed signal, and second, a conditional diffusion model is trained in residual space to refine the imputation by focusing on missing regions. The diffusion process is guided by both the baseline signal and its latent representation, with an adaptive mechanism to control the influence of the baseline based on its reliability.
Results
Experiments conducted on multiple benchmark datasets indicate that RDDMPI consistently enhances reconstruction accuracy and provides better uncertainty quantification compared to existing imputation methods, demonstrating its effectiveness in handling multivariate time series data.
Implications
The proposed method has significant implications for fields that rely on accurate time series data, such as healthcare monitoring, traffic management, and energy systems, by improving the reliability of imputed values and enabling better decision-making based on incomplete data.
How Wrong Can a Good Predictor Be? Diverging Updates with Vanishing Predictive KL
Theory
- An unbounded internal update gap does not guarantee predictive failure.
- Predictive KL divergence can vanish even when update maps diverge significantly.
- The relationship between internal update gaps and predictive cost is complex and task-dependent.
- Controlled numerical experiments validate the theoretical findings.
Read more
How Wrong Can a Good Predictor Be? Diverging Updates with Vanishing Predictive KL
Summary
This paper investigates the relationship between internal update gaps in predictive models and their performance, particularly in the context of Bayesian updates within stationary symmetric Gaussian Hidden Markov Models (HMMs). The authors demonstrate that a significant divergence in update maps can occur while the predictive Kullback-Leibler (KL) divergence approaches zero. They establish that for any fixed finite K (K ≥ 2), the distance between exact Bayesian mixing and a radial filter can grow linearly with the natural confidence scale, even as the categorical DKL between the two approaches vanishes. The study emphasizes that an unbounded internal update gap does not necessarily imply predictive failure, as both filters can converge to a common confidence cone under certain conditions. The paper also provides controlled numerical experiments to illustrate these findings, highlighting the complex interplay between decoder sensitivity, separating states, and expected loss in predictive performance. The results challenge conventional assumptions about the necessity of accurate Bayesian updates for effective prediction, suggesting that the geometry of updates and the nature of the task play critical roles in determining predictive cost.
Methodology
The authors utilize theoretical proofs and controlled numerical experiments to analyze the behavior of predictive models under stationary symmetric Gaussian HMMs. They explore the dynamics of exact Bayesian mixing versus radial filtering, focusing on the relationship between update maps and predictive KL divergence. The methodology includes examining the convergence of expected terminal KL under specific conditions and utilizing a sweep of Gaussian distributions to illustrate the findings.
Results
The main results indicate that for fixed K, the internal separation between update maps can diverge while the predictive KL divergence approaches zero. The expected terminal KL between filtered posteriors also converges to zero under the stationary law as the horizon increases, particularly in scenarios with rare switching. These findings highlight the nuanced relationship between update geometry and predictive performance.
Implications
The results have significant implications for the design of predictive models, particularly in understanding the trade-offs between model complexity and predictive accuracy. They suggest that designers can focus on the geometry of updates and task-specific characteristics rather than solely relying on accurate Bayesian approximations. This could lead to more efficient and effective model architectures in various applications.
M3-Former: Multimodal Transformer with Mixture-of-Experts for Long-Term Vessel Trajectory Prediction
Multimodal
Large Language Models
Time Series
- M3-Former integrates static vessel attributes and navigational intent for enhanced trajectory prediction.
- The dual-granularity Mixture-of-Experts architecture captures both global and local trajectory dynamics.
- A novel Steering-Weighted Cross-Entropy loss function improves accuracy in critical maneuvering scenarios.
- Experimental results demonstrate significant performance improvements over existing state-of-the-art methods.
Read more
M3-Former: Multimodal Transformer with Mixture-of-Experts for Long-Term Vessel Trajectory Prediction
Summary
This paper introduces M3-Former, a novel framework designed to enhance long-term vessel trajectory prediction by addressing challenges such as behavioral multimodality, limited semantic utilization, and long-term error accumulation. M3-Former integrates vessel static attributes and navigational intent as semantic priors, utilizing a unified multimodal representation space where static information is encoded by a pre-trained large language model (LLM) and aligned with dynamic trajectory features through self-attention mechanisms. The framework employs a dual-granularity Mixture-of-Experts (MoE) architecture, which consists of sequence-level experts for global navigation trends and token-level experts for fine-grained maneuvering behaviors. Additionally, a Steering-Weighted Cross-Entropy loss function is introduced to mitigate the long-tail distribution of sparse turning samples, thereby enhancing prediction accuracy in critical scenarios. Experimental results on a real-world Danish AIS dataset show that M3-Former outperforms state-of-the-art baselines across various prediction horizons, achieving significant reductions in Average Displacement Error (ADE) and Final Displacement Error (FDE). Qualitative and ablation studies confirm that the semantic fusion reduces long-term trajectory drift, while the dual-granularity MoE improves robustness in complex waterways and route-branching situations. The framework establishes a semantic-guided hierarchical prediction paradigm, effectively modeling high-level navigational intent alongside local motion dynamics for robust long-term vessel trajectory forecasting.
Methodology
The M3-Former framework constructs a unified multimodal representation space by encoding static semantic information using a pre-trained LLM and aligning it with dynamic trajectory features through self-attention. It employs a dual-granularity Mixture-of-Experts architecture, with sequence-level experts for global navigation trends and token-level experts for local maneuvering behaviors. The model also incorporates a Steering-Weighted Cross-Entropy loss to address the challenges of sparse turning samples.
Results
M3-Former consistently outperformed state-of-the-art baselines across prediction horizons from 1 to 4 hours. In the 4-hour prediction task, it achieved a 4.4% reduction in Average Displacement Error (ADE) and a 5.1% reduction in Final Displacement Error (FDE) compared to the strongest baseline. Qualitative and ablation analyses confirmed the effectiveness of semantic fusion and the dual-granularity MoE in improving prediction accuracy and robustness.
Implications
The proposed framework has significant implications for maritime transportation, enhancing navigational safety, optimizing route planning, and supporting intelligent maritime surveillance and automated shipping scheduling. Its ability to accurately predict vessel trajectories can lead to improved operational efficiency and safety in complex maritime environments.
Phase-Decoupled, Model-Calibrated Power Control for Disaggregated LLM Serving
Large Language Models
Optimization
Efficient ML
- Proposes a phase-decoupled, model-calibrated power control strategy for LLM serving.
- Demonstrates that optimal power settings depend on the specific model and hardware combination.
- Achieves significant efficiency improvements over traditional static power profiles.
- Confirms that distinct profiles for prefill and decode phases enhance energy recovery.
Read more
Phase-Decoupled, Model-Calibrated Power Control for Disaggregated LLM Serving
Summary
This paper addresses the challenge of power management in datacenter GPU clusters serving large language models (LLMs), particularly under the constraints of disaggregated architectures. The authors propose a novel phase-decoupled, model-calibrated power control strategy that optimizes power settings based on the specific combination of model, quantization, engine, and hardware. They argue that traditional static profiles fail to account for the unique requirements of different phases of LLM serving—prefill and decode—which operate under distinct hardware constraints. The proposed controller dynamically adjusts power settings to maximize efficiency while ensuring compliance with service level objectives (SLOs). Experimental results demonstrate that the new approach significantly outperforms existing vendor profiles, achieving a 20.4% increase in tokens per joule with only a 3.5% increase in mean end-to-end latency, compared to the modest gains of 8.6% tokens/J and 5.2% latency from the Max-Q profile. The findings indicate that a tailored, phase-aware power management strategy can lead to substantial energy savings and improved performance in LLM serving environments.
Methodology
The authors implemented a phase-decoupled controller that calibrates power settings based on real-time measurements of the deployed model and hardware. They conducted experiments on an 8× B200 node serving various LLMs under agentic loads, comparing their approach against NVIDIA's Max-Q profile and other static power management strategies. The methodology involved measuring throughput and latency to derive optimal power settings for both prefill and decode phases.
Results
The proposed controller achieved a 20.4% increase in tokens per joule with a 3.5% increase in mean end-to-end latency, outperforming the Max-Q profile, which yielded only an 8.6% increase in tokens/J with a 5.2% latency increase. Additionally, the calibrated power cap demonstrated superior performance over static clock locking, leading to a 24.1% efficiency gain compared to 15.5% from aggressive static settings. A sustained three-day run showed a 32.3% reduction in electricity consumption for a lane pair.
Implications
This research has significant implications for optimizing power management in datacenters serving LLMs, particularly as the demand for energy-efficient AI solutions grows. The findings suggest that tailored power control strategies can enhance the performance and sustainability of AI workloads, potentially influencing future designs of GPU architectures and power management frameworks.
Particle GFlowNets: Rethinking Generative Marginalization Models
Generative Models
Optimization
Efficient ML
- MaMs can be represented as permutation-conditioned GFlowNets, establishing a theoretical equivalence.
- P-GFlowNets extend the sampling strategy of MaMs to non-autoregressive generative processes.
- The introduction of a rejuvenation criterion based on the Gelman-Rubin statistic significantly speeds up learning convergence.
- P-GFlowNets reduce the average number of forward passes required per gradient step as the MDP horizon increases.
Read more
Particle GFlowNets: Rethinking Generative Marginalization Models
Summary
This paper introduces Particle GFlowNets (P-GFlowNets), a novel approach that reinterprets Generative Marginalization Models (MaMs) as a specific case of permutation-conditioned Generative Flow Networks (GFlowNets). The authors demonstrate that MaMs, which facilitate efficient neural sampling for discrete distributions, can be equivalently represented within the GFlowNet framework. They extend the sampling strategy of MaMs to non-autoregressive generative processes, addressing limitations in previous models that were confined to factorized domains. A key innovation is the introduction of a rejuvenation criterion based on the Gelman-Rubin statistic to enhance learning convergence by refreshing the sampling process. The experiments conducted show that P-GFlowNets significantly accelerate training in large combinatorial spaces, particularly when the cost of Markov Decision Process (MDP) simulation is high compared to evaluating the target distribution. Overall, the work highlights the equivalence of MaMs and GFlowNets while providing a practical solution for improving training efficiency in complex generative tasks.
Methodology
The authors propose Particle GFlowNets, which maintain a persistent Gibbs sampler over terminal states and optimize a learning objective that enforces the detailed balance condition of the underlying Markov chain. They utilize a stochastic reverse transition kernel to facilitate non-deterministic backward moves in the sampling process. The rejuvenation process is implemented based on the Gelman-Rubin statistic to refresh the sampling and enhance convergence.
Results
The experiments reveal that P-GFlowNets significantly accelerate training convergence, especially in scenarios where MDP simulation costs are high. The method demonstrates improved efficiency in generating samples and reduces wall-clock time for learning convergence compared to traditional GFlowNet training methods.
Implications
The findings suggest that P-GFlowNets can be effectively applied in various domains requiring efficient sampling from complex discrete distributions, such as computational biology, combinatorial optimization, and Bayesian inference. The theoretical insights into the equivalence of MaMs and GFlowNets may also inspire further research in generative modeling.
The information geometry of large language models is shared, learned, and controllable
NLP
Large Language Models
Theory
- Output geometries of LLMs show stronger agreement than activation geometries across different architectures.
- Predictive behavior determines a canonical output geometry that is invariant under certain transformations.
- Controlled language assignments can causally influence the learned geometry of models.
- The geometry prescribes minimum-disturbance interventions and predicts their relative costs.
Read more
The information geometry of large language models is shared, learned, and controllable
Summary
This paper explores the information geometry of large language models (LLMs), focusing on the shared structures and controllability of their behaviors. It establishes that the Fisher–Rao geometry of next-token probabilities is crucial in understanding model behaviors, revealing that output geometries across various architectures (transformer, state-space, and recurrent models) exhibit stronger agreement than activation geometries. The study demonstrates that predictive accuracy, model scale, and training enhance alignment with human word choices. Furthermore, it shows that controlled language assignments can dictate learned geometries, and that pretraining corpus statistics can predict fact acquisition without recalibration. The paper introduces a framework where the geometry of model outputs informs predictive behavior, while the anisotropy of the geometry predicts intervention costs across architectures. The findings suggest that the geometry allows for minimum-disturbance interventions and facilitates the transfer of learned updates across different prompts, enhancing tasks such as steering, editing, and fine-tuning.
Methodology
The paper employs a theoretical framework based on Fisher–Rao geometry to analyze the output probabilities of language models. It examines the relationships between model architectures, activation geometries, and predictive behaviors through empirical experiments and statistical analyses. The study also utilizes controlled language assignments and randomised tests to validate its findings.
Results
The research finds that the output Fisher metric provides a unique and identifiable geometry for LLMs, with a mean rank agreement of 0.88 among ten independently trained models. It establishes that deeper statistical evidence delays fact acquisition significantly and that the geometry allows for effective minimum-disturbance interventions. The study also demonstrates that the learned geometry can predict intervention costs and supports the transfer of updates across different prompts.
Implications
The findings have significant implications for the design and control of large language models, suggesting that understanding their geometric properties can lead to better steering, editing, and fine-tuning techniques. This could enhance the usability of LLMs in various applications, including natural language processing tasks and interactive AI systems.
Dynamic language model representations for multi-objective reaction optimisation
Optimization
- Introduces a dynamic representation learning approach for chemical reaction optimization using language models.
- Bypassing traditional descriptor computation, the method adapts representations based on reaction performance.
- Demonstrates improved optimization efficiency in multi-objective settings compared to conventional methods.
- Achieves high yields and enantiomeric excess in practical experimental setups.
Read more
Dynamic language model representations for multi-objective reaction optimisation
Summary
This paper addresses the challenge of optimizing chemical reactions across multiple objectives such as yield, selectivity, and safety, which is crucial in chemical synthesis. Traditional methods of representing reaction components, like one-hot encodings and molecular descriptors, either lack chemical informativeness or do not generalize well across diverse chemical entities. The authors propose a novel approach that dynamically learns reaction representations from textual descriptions of reaction conditions using a fine-tuned language model. This model is trained in conjunction with Gaussian process surrogates, allowing for task-adaptive representations within a multi-objective Bayesian optimization framework. The method was tested on nickel- and palladium-catalyzed cross-couplings, demonstrating faster optimization convergence compared to traditional descriptor libraries and one-hot encoding. The approach was further validated through high-throughput experimentation, achieving significant yields and enantiomeric excess in subsequent reactions, showcasing its potential for practical applications in chemical synthesis.
Methodology
The authors developed a framework that encodes textual descriptions of reaction conditions using a fine-tuned language model. This model generates dense, continuous embeddings that are integrated into a multi-objective Bayesian optimization loop, where independent Gaussian process models are used to predict outcomes based on these embeddings. The architecture is trained jointly, allowing for continuous improvement of representations as more data is collected.
Results
The proposed method achieved optimization convergence in fewer experiments than traditional descriptor-based approaches. In high-throughput experiments involving palladium-catalyzed cyanation and asymmetric hydrogenation, the method delivered conditions that translated to gram-scale production with isolated yields of 94% and 84%, respectively, and a high enantiomeric excess of 99.6%.
Implications
This approach has significant implications for the field of chemical synthesis, particularly in pharmaceutical development, where efficient optimization of reaction conditions is critical. It offers a scalable and adaptable solution for navigating complex reaction spaces, potentially accelerating the discovery and development of new chemical processes.
From Connectivity to Rewards: Dense Reward Learning with Directed State Graphs
Reinforcement Learning
Graph Learning
Robotics
- Introduction of a directed state graph for online construction during exploration.
- Development of a state connectivity model to predict state connectivity strengths.
- Transformation of connectivity strengths into dense auxiliary rewards for improved learning.
- Compatibility of the proposed framework with existing GCHRL architectures.
Read more
From Connectivity to Rewards: Dense Reward Learning with Directed State Graphs
Summary
This paper addresses the challenges of sparse rewards in Goal-Conditioned Hierarchical Reinforcement Learning (GCHRL) by introducing a novel framework called Graph-Guided Quasimetric Dense Reward (G2QDR). The authors highlight that existing GCHRL methods often neglect the intrinsic connectivity information of state graphs, which can hinder efficient learning, especially in quasimetric environments where state transitions are asymmetric. To overcome this limitation, the authors propose an online construction of a directed state graph during exploration, which captures the connectivity and accessibility of states. A state connectivity model is developed to predict the strength of pairwise state connections, which is then transformed into dense auxiliary rewards. These rewards provide continuous guidance across hierarchical levels, enhancing the learning process for both high-level and low-level agents. The framework is compatible with existing GCHRL architectures and is empirically validated across various sparse reward environments, demonstrating significant performance improvements over baseline methods with manageable computational overhead.
Methodology
The authors construct a directed state graph online during exploration, incrementally adding visited states and pruning outdated connections. A neural network is trained on this graph to predict state connectivity, which is then used to generate dense rewards that guide the learning process of both high-level and low-level agents in GCHRL.
Results
The proposed G2QDR framework was evaluated in various MuJoCo environments, showing improved success rates and overall performance compared to baseline GCHRL methods. The results indicate that the integration of connectivity information and dense rewards significantly enhances learning efficiency.
Implications
The findings suggest that incorporating connectivity information into GCHRL can lead to more efficient exploration and learning in environments with sparse rewards. This approach may have broader applications in robotics and other domains where hierarchical task decomposition is beneficial.
Topological Necessities: Mechanism-Invariant Strategic Subgoals for Cross-Embodiment Goal-Conditioned Control
Reinforcement Learning
Robotics
Theory
- Introduction of topological necessities as mechanism-invariant strategic subgoals.
- Development of a transport-weighted carrier for analyzing offline behavior data.
- Demonstration of successful transfer of learned structures across different executors without retraining.
- Achievement of state-of-the-art performance on multiple benchmark tasks.
Read more
Topological Necessities: Mechanism-Invariant Strategic Subgoals for Cross-Embodiment Goal-Conditioned Control
Summary
This paper addresses the challenge of long-horizon goal-conditioned reinforcement learning (GCRL) by proposing a novel framework for defining strategic subgoals that are mechanism-invariant and applicable across different executors and embodiments. The authors introduce the concept of 'topological necessities,' which are defined as an order of unavoidable stages that every successful executor must traverse, independent of the specific executor's dynamics. By leveraging offline trajectories, the authors construct a transport-weighted carrier and utilize homology to identify and certify these topological necessities. The framework allows for the recovery of a route-conditioned order of stages that can be transferred across different embodiments without retraining. The results demonstrate that the proposed method achieves superior performance on various tasks, including PointMaze, AntMaze, and Kitchen, showcasing the effectiveness of the topological approach in facilitating cross-executor learning and strategic subgoal emergence.
Methodology
The authors construct a transport-weighted graph and a rectification embedding to analyze offline trajectories. They employ homology to read the topological properties of the data, identifying certified gates that represent strategic subgoals. The decision-making process incorporates a recursive topological gate hierarchy, allowing for effective navigation through the learned structure.
Results
The proposed framework saturates the PointMaze task with a perfect score and outperforms existing baselines on AntMaze and Kitchen tasks. The method achieves a significant performance margin on multi-route tasks and demonstrates the ability to transfer learned structures across different embodiments, achieving high performance without retraining.
Implications
This research has significant implications for the field of reinforcement learning, particularly in enabling agents to learn from offline data and transfer knowledge across different embodiments. The findings suggest that topological structures can provide a robust framework for defining strategic subgoals, enhancing the efficiency and effectiveness of goal-conditioned control in complex environments.
CoRA-NAS: Coarse Ranking and Anchor-Residual Refinement for Neural Architecture Search
Computer Vision
Optimization
Efficient ML
- CoRA-NAS combines coarse ranking with anchor-residual refinement for efficient NAS.
- The method is label-free and does not require fully trained architecture-accuracy labels.
- CoRA-NAS achieves high ranking quality with a Spearman correlation of up to 0.946 across different datasets.
- The refinement stage allows for capturing learning dynamics with minimal training cost.
Read more
CoRA-NAS: Coarse Ranking and Anchor-Residual Refinement for Neural Architecture Search
Summary
CoRA-NAS introduces a novel approach to Neural Architecture Search (NAS) that combines coarse ranking and anchor-residual refinement to improve the efficiency and effectiveness of architecture evaluation. Traditional NAS methods often rely on fully training candidate architectures, which is computationally expensive. CoRA-NAS addresses this by employing a two-stage process: the first stage ranks architectures based on static signals (capacity and structure) using zero-cost proxies, while the second stage refines these rankings by incorporating a small fraction of training data (approximately 1% of full training) to capture learning dynamics. This method avoids the need for fully trained architecture-accuracy labels, making it label-free and applicable across various datasets. The results demonstrate that CoRA-NAS achieves high ranking quality and selects strong architectures, reaching a CIFAR-100 accuracy of 73.32, which is very close to the ground-truth best of 73.37. The method shows robustness across multiple vision spaces, outperforming existing zero-cost proxies and providing a strong baseline for future NAS research.
Methodology
CoRA-NAS employs a two-stage process: Stage 1 (Rank) uses a consensus of zero-cost proxies to rank architectures based on static signals of capacity and structure. Stage 2 (Refine) involves training a small stratified anchor set for a few epochs to correct the initial rankings using early training curve data, thus incorporating dynamic learning signals.
Results
CoRA-NAS achieved a Spearman correlation of 0.946 on the NB201 dataset and maintained high performance across other datasets (0.715 on NB101, 0.786 on TransNAS-Bench-101, and 0.894 on NATS-SSS). The method reached a CIFAR-100 accuracy of 73.32, closely approaching the best-known accuracy of 73.37.
Implications
The CoRA-NAS framework provides a more efficient and effective approach to NAS, potentially reducing computational costs and time in architecture search processes. Its label-free nature allows for broader applicability in various domains, making it a valuable tool for researchers and practitioners in machine learning.
MUtE: A Dual Framework for Concept Erasure and Counterfactual Interventions
NLP
Large Language Models
Interpretability
- Introduction of MUtE*, a class of optimal erasure functions that define a dual counterfactual mapping.
- Development of a computationally efficient implementation that utilizes translational bias for erasure and counterfactual generation.
- Empirical validation shows significant improvements in algorithmic fairness and the generation of counterfactual texts.
Read more
MUtE: A Dual Framework for Concept Erasure and Counterfactual Interventions
Summary
The paper presents MUtE, a novel framework designed to erase concept-specific information from representations while preserving unrelated information, thereby improving algorithmic fairness and interpretability in NLP tasks. The authors revisit the optimal bounds of concept erasure and introduce a class of erasure functions that naturally induce a dual counterfactual mapping. This duality allows for seamless navigation between concept erasure and counterfactual generation, addressing the challenges of existing methods that often sacrifice utility for privacy or suffer from computational inefficiencies. The proposed implementation leverages a translational bias on counterfactual trajectories, aligning with the geometric manifestations of concepts in modern language models. Empirical evaluations demonstrate the framework's effectiveness in bias mitigation and counterfactual text generation across synthetic datasets and NLP benchmarks.
Methodology
The authors formalize the structure of optimal erasure functions and derive conditions for perfect erasure and maximal utility retention. They propose a practical implementation that incorporates a translational bias to facilitate efficient erasure and counterfactual representation generation. The framework is empirically tested on synthetic data and NLP benchmarks to validate its effectiveness.
Results
The empirical results indicate that the MUtE framework significantly enhances algorithmic fairness and successfully generates coherent counterfactual texts, outperforming existing methods that struggle with the trade-off between privacy and utility.
Implications
The findings suggest that MUtE can be applied to various NLP tasks requiring bias mitigation and interpretability, potentially leading to more equitable AI systems. The dual framework also opens avenues for further research in counterfactual generation and representation learning.
Thompson Sampling for Non-Monotone Convex Ridge Bandits: Monotonicity Is Not Needed for Polynomial Regret
Theory
Optimization
- Thompson Sampling can achieve polynomial regret for non-monotone convex ridge losses.
- The paper provides a new cardinality bound for uninformative configurations, which is crucial for the analysis.
- A self-contained transfer theorem is established for the information ratio to regret under fixed measurable selection rules.
- The results challenge the necessity of monotonicity in achieving low-regret performance in Bayesian bandit settings.
Read more
Thompson Sampling for Non-Monotone Convex Ridge Bandits: Monotonicity Is Not Needed for Polynomial Regret
Summary
This paper addresses the performance of Thompson Sampling (TS) in the context of Bayesian bandit convex optimization, specifically focusing on non-monotone convex ridge losses. Previous work established that TS achieves a Bayesian regret of O(d^(5/2)√n) for monotone convex ridge losses. However, this paper provides a qualitative negative answer to the question of whether monotonicity is necessary for achieving polynomial regret. The author demonstrates that for any prior on 1-Lipschitz convex ridge losses with arbitrary convex, possibly non-monotone links, TS can achieve a Bayesian regret of O(d^(9/2)√n). The methodology involves a novel cardinality bound for uninformative configurations and a self-contained transfer theorem for the information ratio to regret under fixed measurable selection rules. The results indicate that the structure of convex ridge functions is sufficient for low-regret performance of TS, even without monotonicity, although the question of retaining the d^(5/2) dependence from the monotone case remains open.
Methodology
The author employs a combination of theoretical analysis and explicit constructions to derive new bounds on Bayesian regret for TS in the context of non-monotone convex ridge functions. This includes the use of a cardinality bound for uninformative configurations and a transfer theorem for the information ratio to regret.
Results
The main results include a Bayesian regret bound of O(d^(9/2)√n) for TS applied to non-monotone convex ridge losses, demonstrating that monotonicity is not a necessary condition for achieving polynomial regret. Additionally, a cardinality bound for uninformative configurations is established, showing that the configurations can be tightly bounded in size.
Implications
These findings suggest that TS can be effectively applied in more general settings of convex ridge functions without the need for monotonicity, potentially broadening the applicability of Bayesian bandit algorithms in various optimization problems. It also raises further questions about the limits of regret bounds in non-monotone scenarios.
A Dynamic Fusion Large Language Model for Traffic Flow Prediction
Large Language Models
Graph Learning
Time Series
- Introduction of DF-LLM, a model specifically designed for traffic flow prediction.
- Integration of spatiotemporal features through a dedicated embedding and fusion strategy.
- Utilization of a differentiated parameter adaptation strategy for improved model training.
- Incorporation of residual connections to mitigate gradient vanishing in deep networks.
Read more
A Dynamic Fusion Large Language Model for Traffic Flow Prediction
Summary
This paper presents a novel Dynamic Fusion Large Language Model (DF-LLM) aimed at improving traffic flow prediction, a critical component of intelligent transportation systems. Traditional neural networks often face limitations in accuracy due to their reliance on singular feature modeling, while large language models (LLMs) struggle with capturing spatial topological information and spatiotemporal correlations. The DF-LLM integrates three key components: a spatiotemporal embedding module, a spatiotemporal fusion module, and an LLM backbone. The spatiotemporal embedding module generates multi-scale representations of traffic data, while the fusion module employs graph convolution to integrate spatial topology and dynamic dependencies. The LLM backbone utilizes a differentiated parameter adaptation strategy to enhance training efficiency and adaptability to traffic data, alongside a context aggregation attention mechanism to strengthen global dependencies. The model also incorporates residual connections to address gradient vanishing issues in deep networks. Experimental results demonstrate that DF-LLM outperforms existing methods across four datasets, showcasing its effectiveness in traffic flow prediction.
Methodology
The DF-LLM framework processes historical traffic data through a spatiotemporal embedding module that generates multi-scale feature representations, followed by a spatiotemporal fusion module that integrates these features using graph convolution. The LLM backbone employs a context aggregation attention mechanism and a differentiated parameter adaptation strategy to enhance performance and adaptability. Residual connections are used to stabilize training and improve feature retention.
Results
The DF-LLM model achieved competitive or superior forecasting performance compared to existing advanced methods across four datasets, indicating its effectiveness in capturing complex spatiotemporal dependencies in traffic data.
Implications
The proposed DF-LLM has significant implications for intelligent transportation systems, potentially leading to improved traffic management, reduced congestion, and enhanced resource allocation efficiency. Its ability to effectively model spatiotemporal dynamics can be leveraged in various urban planning and traffic optimization applications.
HERALD: High-Fidelity Exemplar Retrieval with Adaptive Landmark Distillation for Heterophily-Aware Graph Condensation
Graph Learning
- HERALD addresses the limitations of existing graph condensation methods in heterophilic settings.
- The framework employs a gradient-free approach, adapting node scoring based on the graph's heterophily.
- HERALD combines multiple scoring criteria to select nodes that are informative and representative.
- Experimental results show that HERALD outperforms state-of-the-art methods on heterophilic graphs.
Read more
HERALD: High-Fidelity Exemplar Retrieval with Adaptive Landmark Distillation for Heterophily-Aware Graph Condensation
Summary
The paper introduces HERALD, a novel framework for graph condensation that addresses the limitations of existing methods in handling heterophilic graphs. Traditional graph condensation techniques often rely on the assumption that adjacent nodes share the same label, which fails in heterophilic contexts where nodes may belong to different classes. HERALD employs a gradient-free approach that adapts node scoring and feature selection based on the graph's heterophily. It utilizes a combination of prototype representativeness, decision-boundary proximity, and Local Intrinsic Dimensionality (LID) for scoring nodes, with weights adjusted according to the heterophily ratio. The framework retains the architecture-agnostic nature of previous methods while enhancing the quality of the condensed graph by focusing on informative nodes that are representative of their classes and relevant to decision boundaries. Experiments conducted on eight benchmark datasets demonstrate that HERALD matches or outperforms state-of-the-art methods on heterophilic graphs and remains competitive on homophilic graphs across various Graph Neural Network (GNN) architectures.
Methodology
HERALD utilizes a gradient-free graph condensation framework that adapts to the heterophily of the input graph. It scores nodes based on a combination of prototype representativeness, decision-boundary proximity, and Local Intrinsic Dimensionality (LID), with weights adjusted according to the graph's heterophily ratio. The condensed subgraph is constructed through score-ordered BFS expansion, Personalized PageRank pruning, and class rebalancing, maintaining a consistent storage budget for comparison with existing methods like BONSAI.
Results
The experiments on eight benchmark datasets reveal that HERALD effectively condenses graphs while preserving node-classification performance, achieving results that match or exceed those of state-of-the-art condensation methods on heterophilic graphs. The framework also demonstrates competitive performance on homophilic graphs across four different GNN architectures.
Implications
HERALD's ability to effectively condense heterophilic graphs has significant implications for applications in various domains where graph structures are complex and diverse, such as social networks, recommendation systems, and biological networks. The framework's architecture-agnostic nature allows for broader applicability across different GNN models, facilitating more efficient training and inference processes.
EMMI: Edge Multi-Modal Intelligence for Communication-Efficient MLLM Inference via Fused Representation Compression
Multimodal
Efficient ML
Large Language Models
- EMMI enables efficient multimodal reasoning on edge devices by compressing and fusing representations before transmission.
- The architecture significantly reduces communication overhead while preserving task-relevant information.
- EMMI maintains comparable accuracy to traditional methods while achieving a 32× reduction in communication payload.
- The approach leads to a 3.4× reduction in end-to-end inference latency in bandwidth-constrained environments.
Read more
EMMI: Edge Multi-Modal Intelligence for Communication-Efficient MLLM Inference via Fused Representation Compression
Summary
The paper introduces Edge Multi-Modal Intelligence (EMMI), a novel architecture designed to facilitate the deployment of multimodal large language models (MLLMs) on resource-constrained edge platforms. EMMI addresses the challenges posed by the high computational, memory, and communication demands of MLLMs by enabling communication-efficient inference. Instead of transmitting raw sensor data or intermediate neural network outputs, EMMI encodes modality-specific information and fuses cross-modal representations at the edge. This results in a compact latent representation that is sent to server resources for high-capacity reasoning. The architecture not only reduces communication overhead but also preserves local data privacy and provides a fixed-size interface between heterogeneous edge devices and server-side MLLMs. Experimental evaluations demonstrate that EMMI can achieve a 32× reduction in communication payload while maintaining comparable accuracy, leading to a 3.4× decrease in estimated end-to-end inference latency under bandwidth-constrained conditions.
Methodology
The EMMI framework employs modality-specific encoding, cross-modal representation fusion, and learned compression techniques at the edge. This process generates a unified multimodal representation that is compactly transmitted to server resources for reasoning, thereby optimizing both computation and communication.
Results
The evaluation on a multimodal benchmark indicates that EMMI can reduce the communication payload by 32× while achieving similar downstream accuracy compared to existing methods. Additionally, it results in a 3.4× reduction in estimated end-to-end inference latency under bandwidth constraints.
Implications
The EMMI architecture has significant implications for deploying advanced multimodal reasoning capabilities in edge environments, such as autonomous systems, smart cities, and IoT applications, where efficient communication and processing are critical.
Counterfactual Marginalisation: Framework for Evaluating Robustness to Nuisance Variables
Generative Models
Interpretability
Computer Vision
- Introduces counterfactual marginalisation as a test-time evaluation procedure for medical image classifiers.
- Develops intervention-aware evaluation metrics to assess model robustness against nuisance variables.
- Demonstrates that the proposed metrics can reveal biases in predictions more effectively than traditional methods.
- Provides a framework that does not require disease labels to measure sensitivity to demographic variables.
Read more
Counterfactual Marginalisation: Framework for Evaluating Robustness to Nuisance Variables
Summary
This paper introduces a novel framework called counterfactual (CF) marginalisation, aimed at evaluating the robustness of classification models, particularly in medical image analysis, against nuisance variables such as demographic attributes. The authors argue that while deep learning models may perform well on test data, they can still rely on shortcuts related to demographics, which can lead to biased predictions. The proposed framework utilizes a CF image generator to create counterfactual versions of test images by intervening on nuisance variables while preserving patient-specific information. This allows for the computation of intervention-aware predictions that help in assessing model performance. The authors define several metrics, including counterfactual marginal risk, interventional expected risk, counterfactual stability, and worst-case counterfactual risk, to quantitatively evaluate the robustness of models. The paper demonstrates that these metrics can better expose biases in predictors compared to traditional evaluation methods, providing a more nuanced understanding of model sensitivity to demographic variations without needing disease labels.
Methodology
The methodology involves using a Structural Causal Model (SCM)-based CF generator to create counterfactual images by intervening on nuisance variables like age and sex. The authors then compute various metrics based on these counterfactual images to evaluate model performance, sensitivity, stability, and calibration. The evaluation is done through Monte Carlo sampling to approximate expected predictions over the interventions.
Results
The results indicate that the counterfactual marginalisation framework effectively identifies biases in medical image classifiers that traditional metrics might overlook. The proposed metrics provide insights into the models' sensitivity to nuisance variables and their calibration, thereby enhancing the understanding of their robustness in clinical applications.
Implications
The implications of this work are significant for the deployment of machine learning models in healthcare, as it provides a systematic approach to evaluate and ensure that models are not unfairly biased by demographic factors. This can lead to more equitable and clinically relevant decision-making processes in medical diagnostics.
DeFiFlowBench: Benchmarking and Improving Safe Executability in Natural-Language DeFi Workflow Synthesis
NLP
Large Language Models
Graph Learning
- Introduction of DEFIFLOWBENCH, a benchmark for evaluating DeFi workflow safety.
- Identification of significant safety gaps in existing natural-language DeFi workflows.
- Development of Koan-Safe, which improves safety outcomes in workflow synthesis.
- Demonstration that high static scores do not guarantee overall safety in trade executions.
Read more
DeFiFlowBench: Benchmarking and Improving Safe Executability in Natural-Language DeFi Workflow Synthesis
Summary
The paper introduces DEFIFLOWBENCH, a benchmark consisting of 207 team-authored prompts aimed at evaluating the safe executability of natural-language DeFi workflows. The authors highlight that even structurally valid workflows can lead to costly trades due to inadequate safety measures. The benchmark assesses graph coverage, configuration completeness, and safety predicates, and tests trade configurations using a local Ethereum Virtual Machine (EVM). The study finds that various prompting methods (direct, constrained, and few-shot) often result in unsafe executions, with 14-19 unsafe outcomes per configuration under a 5% price-impact cap. To address these issues, the authors propose Koan-Safe, a hybrid approach that integrates an intent parser, a generator, and structural repairs with default safety parameters. Koan-Safe achieves a static safety proxy score of 0.67 on held-out prompts, outperforming the best baseline score of 0.33. The results indicate that while structural repairs improve safety, they must be complemented by numeric policy validation to ensure comprehensive trade protections. The paper emphasizes the importance of explicit trade protections and execution-based evaluations in DeFi workflow synthesis.
Methodology
The authors developed DEFIFLOWBENCH, which includes 120 development prompts and 87 held-out prompts covering various DeFi tasks. They employed three static scores to evaluate graph validity, configuration completeness, and safety predicates, alongside an execution check using a local EVM to assess trade configurations. Koan-Safe was created as a hybrid method combining intent parsing, candidate generation, and structural safety enhancements.
Results
Koan-Safe achieved a static safety proxy score of 0.67 on held-out prompts, significantly higher than the best baseline score of 0.33. The study found that existing prompting methods often led to unsafe executions, with 14-19 unsafe outcomes per configuration. The results also highlighted the limitations of default safety parameters and the necessity for numeric policy validation.
Implications
The findings suggest that improving the safety of DeFi workflows requires not only structural repairs but also rigorous numeric policy checks. This has implications for the design of safer DeFi applications and protocols, as well as for future research in natural-language processing for financial systems.
DR-LabStack: Design and Implementation of a Clinician-Facing Web System for Diabetic Retinopathy Prediction
Theory
Interpretability
Computer Vision
- DR-LabStack integrates multiple pretrained diabetic retinopathy prediction models into a single web interface.
- The system addresses the challenges of heterogeneous input requirements and preprocessing for different models.
- Functional evaluations confirmed the system's ability to handle diverse input contracts and produce consistent predictions.
- The design emphasizes a clinician-facing interface, although clinical effectiveness was not evaluated in this study.
Read more
DR-LabStack: Design and Implementation of a Clinician-Facing Web System for Diabetic Retinopathy Prediction
Summary
The paper presents DR-LabStack, a web system designed to facilitate diabetic retinopathy (DR) prediction through a unified interface for multiple pretrained models. The system integrates four distinct models: RuleFit, Pruned RuleFit, Elaborative XGBoost, and Two-level Ensemble, each requiring different input formats and preprocessing steps. DR-LabStack employs a React frontend for user interaction and a Flask backend for model integration, allowing clinicians to input data and receive predictions in a standardized manner. The system's design addresses the challenges of heterogeneous model requirements, ensuring that input ordering and preprocessing are correctly handled. Functional evaluations demonstrated that all models could be effectively loaded and utilized, confirming the system's capability to manage diverse input contracts and produce consistent output. The paper emphasizes the importance of a reusable interaction and serving workflow for clinical applications, although it notes that further evaluation of clinical effectiveness and usability is necessary.
Methodology
The authors designed and implemented a web application using React for the frontend and Flask for the backend. The system integrates four pretrained models, ensuring proper input ordering and preprocessing through model-specific adapters. Functional evaluations were conducted using real model artifacts in an isolated environment, including tests for service behavior and input validation.
Results
The evaluation confirmed that all four models were successfully loaded and exposed their respective input fields. A total of 62 Flask test-client requests characterized service behavior, while 12 limited-vector checks validated invocation-path and threshold consistency. Additionally, 24 browser-component scenarios verified input ordering and result rendering.
Implications
DR-LabStack provides a framework for integrating multiple machine learning models into a single clinical interface, enhancing accessibility for clinicians. This approach could streamline the prediction process for diabetic retinopathy and potentially improve patient outcomes through better-informed clinical decisions.
A Bellman Optimality Equation for Plasticity
Reinforcement Learning
Theory
Optimization
- Introduces a Bellman optimality equation for optimizing plasticity in continual reinforcement learning.
- Reframes the stability-plasticity tradeoff as an empowerment-plasticity tradeoff.
- Establishes theoretical foundations for the Bellman optimality equation and proves its contraction property.
- Empirical evaluations demonstrate the effectiveness of the proposed equation in benchmark environments.
Read more
A Bellman Optimality Equation for Plasticity
Summary
This paper addresses the challenge of managing the stability-plasticity tradeoff in continual reinforcement learning by introducing a Bellman optimality equation specifically for optimizing plasticity. Building on the recent work by Abel et al. (2025), which redefined plasticity as the generalized directed information from an agent's observations to its actions, the authors aim to formalize and optimize plasticity within Markov decision processes (MDPs). The paper establishes a theoretical foundation for a Bellman optimality equation that parallels existing formulations for empowerment, thereby bridging a significant gap in the literature. The authors derive this equation and demonstrate its applicability through empirical evaluations in benchmark environments, highlighting the need for both empowerment and plasticity to be maintained above critical thresholds for robust continual learning agents. The work sets the stage for future research into the optimization of plasticity, which has been largely unexplored compared to empowerment.
Methodology
The authors derive a Bellman optimality equation for plasticity by utilizing directed information theory and adapting existing frameworks for empowerment. They focus on a scenario without external rewards to isolate the information-theoretic value of plasticity. The paper includes a formal derivation of the equation and discusses algorithm optimization techniques, including a multi-weight knapsack dynamic programming approach to manage the computational complexity of the optimization process.
Results
The paper presents a new Bellman optimality equation for plasticity, demonstrating its theoretical validity and practical applicability through empirical evaluations in benchmark and scaled-up environments. The results indicate that the proposed optimization framework effectively manages the tradeoff between empowerment and plasticity, contributing to the development of more robust continual learning agents.
Implications
This work has significant implications for the design of continual learning agents, suggesting that a balanced optimization of both empowerment and plasticity is crucial for effective learning in dynamic environments. The findings could lead to advancements in various applications of reinforcement learning, particularly in scenarios requiring adaptability and resilience.
Semi-Tensor Product-Based Multi-Term Randomized T-SVD and Its Visual Applications
Computer Vision
Efficient ML
Theory
- Introduction of a novel semi-tensor product framework for third-order tensors.
- Development of MSTP-SVD for improved low-rank approximation accuracy.
- Implementation of MRSTP-SVD to reduce computational costs while maintaining accuracy.
- Demonstration of effectiveness through experiments on image and video tasks.
Read more
Semi-Tensor Product-Based Multi-Term Randomized T-SVD and Its Visual Applications
Summary
This paper addresses the limitations of traditional Tensor Singular Value Decomposition (T-SVD) in processing high-dimensional visual data by introducing a novel semi-tensor product framework. The authors highlight that the standard t-product imposes strict dimensional compatibility constraints, which restricts its applicability to real-world tensors. To overcome this, they propose a multi-term semi-tensor product singular value decomposition (MSTP-SVD) that integrates multiple orthogonal decomposition terms, enhancing low-rank approximation accuracy. Additionally, to mitigate the computational costs associated with multi-term modeling, they develop an accelerated multi-term randomized semi-tensor product SVD (MRSTP-SVD) algorithm. This algorithm employs randomized projection and power iteration techniques, achieving a balance between reconstruction accuracy and computational efficiency. Experimental results demonstrate the effectiveness of the proposed methods in image and video compression and completion tasks, showcasing their potential in handling large-scale tensor data efficiently.
Methodology
The authors develop a semi-tensor product for third-order tensors that relaxes dimensional constraints of the standard t-product. They create the MSTP-SVD by combining multiple orthogonal decomposition terms and utilize randomized projection and power iteration techniques to form the MRSTP-SVD algorithm, which accelerates the decomposition process.
Results
The proposed MRSTP-SVD algorithm significantly improves reconstruction accuracy and computational efficiency in tensor decomposition tasks, as evidenced by experiments on image and video compression and completion, outperforming traditional single-term methods.
Implications
The advancements in tensor decomposition techniques presented in this paper have significant implications for various applications in computer vision, machine learning, and data processing, particularly in scenarios involving high-dimensional visual data.
Certifying Lower Bounds for Risk-Sensitive Reinforcement Learning under Adversarial State Perturbations
Reinforcement Learning
Optimization
Theory
- Introduces lower bounds for risk-sensitive reinforcement learning under adversarial state perturbations.
- Extends existing certification methods to support exponential utility for risk-averse performance evaluation.
- Formulates the certification problem as a convex optimization problem with a tractable dual.
- Demonstrates that risk-averse training can yield higher certified lower bounds but may lead to non-monotonic performance.
Read more
Certifying Lower Bounds for Risk-Sensitive Reinforcement Learning under Adversarial State Perturbations
Summary
This paper addresses the vulnerabilities of reinforcement learning (RL) agents to adversarial perturbations in state observations, which can jeopardize their performance in safety-critical applications. The authors extend existing certification methods, which primarily focus on risk-neutral objectives, to risk-sensitive objectives by establishing lower bounds on the exponential utility of cumulative rewards under lp-norm bounded state adversarial perturbations. They introduce a ϕ-divergence relaxation of the perturbation set and formulate the risk-sensitive certification problem as a convex optimization problem, deriving its dual to obtain a tractable approximation of the certified lower bound. Additionally, the authors propose an empirical method that enhances certified lower bounds by allowing the training risk-aversion parameter to be selected independently of the evaluation risk level. Experiments conducted in OpenAI Gym environments and a machine replacement problem demonstrate that risk-averse training generally results in higher certified lower bounds compared to risk-neutral training, especially under larger perturbation budgets. However, the relationship between training-time risk aversion and certified performance is non-monotonic, indicating that overly conservative policies may degrade certification performance. This work represents a significant advancement in providing robust certificates for RL policies, emphasizing the importance of risk sensitivity in certification.
Methodology
The authors develop a convex optimization framework to certify lower bounds on expected utilities of cumulative rewards under bounded adversarial state perturbations. They utilize a ϕ-divergence relaxation to formulate the problem and derive a dual formulation for efficient computation of lower bounds. An empirical method is also proposed to optimize the risk-aversion parameter during training.
Results
The experiments show that risk-averse training generally leads to higher certified lower bounds compared to risk-neutral training, particularly under larger perturbation budgets. However, increasing risk aversion during training initially improves certified performance but can eventually degrade it due to overly conservative policies.
Implications
This research has significant implications for the deployment of RL agents in safety-critical environments, as it provides a framework for certifying the robustness of policies against adversarial perturbations. The findings underscore the necessity of incorporating risk sensitivity into RL training and evaluation processes.
AdamX: Cosine similarity meets gradient descent
Optimization
- Introduction of AdamX, an adaptive optimizer utilizing cosine similarity for update magnitude control.
- Incorporation of a variance rectification scheme for smoother optimization in early training stages.
- Empirical evidence showing competitive convergence rates across diverse datasets and architectures.
- Scalable and model-agnostic design allows easy integration into existing training pipelines.
Read more
AdamX: Cosine similarity meets gradient descent
Summary
The paper introduces AdamX, a novel first-order optimizer that integrates cosine similarity as a mechanism for adapting update magnitudes during optimization. AdamX is designed to be scalable, model-agnostic, and easily integrated into existing training workflows. The authors propose a variance rectification scheme to enhance optimization stability, particularly in the early training phases. Empirical evaluations demonstrate that AdamX achieves competitive convergence rates across various benchmark datasets and architectures, outperforming traditional methods in terms of the number of epochs needed to meet specific performance thresholds. The optimizer builds on the principles of existing adaptive methods like Adam and AMSGrad, while leveraging directional alignment of gradients to modulate update sizes effectively.
Methodology
AdamX employs a moment-normalized update similar to Adam, maintaining exponential moving averages of gradients and their squared magnitudes. It introduces a cosine-similarity controller that modulates update magnitudes based on the alignment of consecutive gradients. The method also includes a monotone second-moment envelope to ensure stable updates. The performance of AdamX is evaluated by measuring the number of epochs required to achieve predefined performance thresholds across various datasets.
Results
AdamX demonstrates improved convergence rates compared to traditional optimizers, requiring fewer epochs to reach performance benchmarks. The empirical results indicate that the integration of cosine similarity into the update mechanism enhances optimization dynamics, particularly in challenging training scenarios.
Implications
AdamX's design could lead to more efficient training processes in machine learning applications, particularly in deep learning tasks where convergence speed is critical. Its model-agnostic nature allows for broad applicability across different architectures and datasets, potentially influencing future optimizer designs.
Musec: MomentUm SpEctral Clipping for Stable Muon-type Training
Optimization
Large Language Models
Theory
- Musec provides a novel spectral clipping approach to stabilize Muon training without architecture-specific modifications.
- Soft Musec offers an efficient implementation that enhances training stability across various datasets.
- The paper establishes the first convergence guarantees for Muon-type optimizers in nonconvex nonsmooth settings.
- Empirical results show that Soft Musec maintains performance while ensuring stability in challenging training conditions.
Read more
Musec: MomentUm SpEctral Clipping for Stable Muon-type Training
Summary
The paper introduces MomentUm SpEctral Clipping (Musec), a novel optimization technique designed to enhance the stability of Muon, an optimizer known for its effectiveness in training large language models. While Muon has demonstrated superior convergence compared to traditional optimizers like Adam and AdamW, it suffers from instability issues characterized by loss spikes and unbounded growth of model weights. Existing solutions primarily focus on architecture-specific modifications, which do not universally address the instability across all model components. Musec addresses this by replacing Muon's spectral flattening with spectral clipping, which limits the singular values of the momentum matrix that exceed a certain threshold while maintaining the spectral structure. This architecture-agnostic approach stabilizes the training process across various model architectures. The authors also present Soft Musec, an efficient implementation of Musec that employs a smooth spectral saturation function and approximates the transformation using coupled Newton-Schulz iterations. The paper establishes convergence guarantees for Musec in nonconvex nonsmooth stochastic optimization, marking the first such guarantees for Muon-type methods in this context. Empirical results demonstrate that Soft Musec significantly improves training stability across diverse datasets and learning rates, outperforming existing Muon variants in scenarios where they typically diverge.
Methodology
The authors propose Musec, which clips singular values of the momentum matrix that exceed a threshold, preserving the spectral structure. They also develop Soft Musec, which uses a smooth spectral saturation function approximated through coupled Newton-Schulz iterations. Theoretical convergence guarantees are established for Musec in nonconvex nonsmooth stochastic optimization settings.
Results
Soft Musec consistently outperforms existing Muon variants in terms of training stability across multiple datasets, including FineWeb, OpenWebText, and C4. It remains stable in scenarios where other Muon variants diverge, while achieving comparable performance under optimal configurations.
Implications
The findings suggest that Musec can be widely applied to improve the stability of training large language models, potentially leading to more robust and efficient training processes in various machine learning applications.
LILA: Calibration-Free Structured Pruning of Large Language Models via Latent Spectral Geometry
Large Language Models
Efficient ML
Theory
- LILA provides a calibration-free approach to structured pruning of LLMs.
- The KS-distance score serves as a novel importance criterion for neuron selection.
- LILA outperforms existing methods like PruneNet and SliceGPT in zero-shot accuracy.
- The method preserves the original architecture while achieving competitive performance.
Read more
LILA: Calibration-Free Structured Pruning of Large Language Models via Latent Spectral Geometry
Summary
The paper introduces LILA (Latent-Informed Layer Analysis), a novel framework for structured pruning of large language models (LLMs) that eliminates the need for calibration data, gradient computations, or auxiliary networks during the pruning process. LILA employs the Kolmogorov–Smirnov (KS) distance to assess neuron importance based on the singular value distributions of weight matrices, allowing for a closed-form spectral rule. The method demonstrates superior performance compared to existing approaches, achieving a 1.57 percentage point improvement over PruneNet in zero-shot accuracy on LLaMA-2-7B at 25% sparsity, and surpassing WikiText-2-calibrated SliceGPT by up to 6.0 percentage points across various sparsity levels. After one epoch of recovery fine-tuning, LILA's performance closely matches that of the heavily calibrated SliceGPT, with only a 0.48 percentage point difference, all while preserving the original model architecture. The paper also provides a theoretical foundation through Neural Tangent Kernel analysis, showing a 22× reduction in functional distortion compared to random pruning. Additionally, LILA can dynamically allocate sparsity budgets, enhancing generative preservation at moderate compression levels and revealing architectural bottlenecks at higher compression rates.
Methodology
LILA utilizes the Kolmogorov–Smirnov distance to evaluate the importance of neurons based on the singular value distributions of weight matrices. It operates without requiring calibration data or auxiliary networks, making it a data-free pruning method. The framework includes various scoring criteria and matrix variants, allowing for rapid pruning with minimal computational overhead.
Results
LILA achieves a 1.57 percentage point improvement over PruneNet in zero-shot accuracy at 25% sparsity on LLaMA-2-7B and outperforms SliceGPT by up to 6.0 percentage points across all sparsity levels. After recovery fine-tuning, LILA's performance is within 0.48 percentage points of SliceGPT, demonstrating its effectiveness without calibration data.
Implications
The findings suggest that LILA can significantly enhance the deployment of large language models in resource-constrained environments by enabling efficient pruning without the need for extensive calibration data. This could facilitate broader accessibility and application of LLMs in various domains.