AI-generated summaries
Today's ML research,
without the noise.
Daily summaries of the latest machine learning papers from arXiv, processed every 8 hours.
24
Papers today
8h
Update frequency
7
Days of history
Align-RAG: Alignment Is All You Need for TSFM In-Context Learning
Time Series
- Align-RAG is a training-free method that improves retrieval-augmented forecasting for frozen TSFMs.
- The method applies amplitude rescaling and phase shifting to align retrieved data with the query.
- Align-RAG outperforms the state-of-the-art trained retrieval adapter, TS-RAG, on multiple datasets.
- The approach demonstrates that frozen TSFMs can dynamically incorporate retrieved context without learned parameters.
Read more
Align-RAG: Alignment Is All You Need for TSFM In-Context Learning
Summary
The paper presents Align-RAG, a novel method for retrieval-augmented forecasting that enhances frozen Time Series Foundation Models (TSFMs) without the need for training or learned fusion modules. Traditional approaches have relied on learned adapters to merge retrieved examples into forecasts, based on the assumption that frozen models cannot dynamically incorporate external context. Align-RAG challenges this assumption by introducing a training-free technique that applies amplitude rescaling and phase shifting to retrieved past-future windows, aligning them with the query before they are processed by the TSFM. The authors demonstrate that this method significantly improves forecasting accuracy across multiple datasets, outperforming the state-of-the-art trained retrieval adapter, TS-RAG, by an average of 3.75% in mean squared error (MSE) across seven datasets. Furthermore, Align-RAG shows improvements in zero-shot MSE on four additional TSFMs, with gains ranging from 2.5% to 13.7% without any tuning for specific backbones. The findings suggest that frozen TSFMs can effectively utilize retrieved context dynamically, and that closed-form alignment should be the default approach for retrieval-augmented forecasting.
Methodology
Align-RAG retrieves past-future windows from a reference corpus and applies closed-form transformations (amplitude rescaling and phase shifting) to align these windows with the query. The aligned pairs are then used as in-context demonstrations for the frozen TSFM, which processes them without any learned parameters.
Results
Align-RAG achieved an average MSE reduction of 3.75% compared to TS-RAG across seven datasets. It also improved zero-shot MSE on four additional frozen TSFMs by 2.5% to 13.7%, demonstrating significant performance gains without any per-backbone tuning.
Implications
The findings indicate that retrieval-augmented forecasting can be significantly enhanced without the need for complex training processes. This has implications for real-time forecasting applications in various domains, such as finance, healthcare, and energy management, where dynamic adaptation to new data is crucial.
Reasoning Errors Have a Region and a Direction in the Residual-Stream Trajectory of LLMs
NLP
Large Language Models
Interpretability
- Introduces a three-stream detector for analyzing reasoning in LLMs.
- Combines motion analysis with coarse and fine location readings to improve reasoning assessment.
- Achieves significant accuracy improvements over existing methods on reasoning benchmarks.
- Demonstrates effectiveness in tasks beyond reasoning, such as factual completion and verification.
Read more
Reasoning Errors Have a Region and a Direction in the Residual-Stream Trajectory of LLMs
Summary
This paper addresses the challenge of distinguishing sound reasoning from flawed reasoning in large language models (LLMs), particularly as they are increasingly used in tasks requiring verifiable reasoning. The authors critique existing trajectory-based methods that analyze layerwise residual-stream displacements, noting that while these methods can suppress stable, token-specific information, they may overlook critical reasoning-related signals. To overcome this limitation, the authors propose a novel three-stream detector that integrates motion with two restricted views of location: a coarse region reader based on vector quantization and a fine direction reader over normalized multi-layer states. This approach allows for the interpretation of motion while maintaining sufficient context to avoid reintroducing shortcut-prone information. The proposed method demonstrates significant improvements in selection accuracy on reasoning benchmarks, outperforming existing methods by up to 12% over displacement-only approaches and 21% over single-layer probing baselines. Additionally, it shows superior performance in factual completion and fact verification tasks, suggesting that reasoning validity is more effectively assessed through state-conditioned motion rather than static states or decontextualized trajectories alone.
Methodology
The authors developed a three-stream validity detector that consists of a motion reader for layerwise displacements, a coarse region reader utilizing vector quantization, and a fine direction reader analyzing normalized multi-layer states. This architecture allows for the selective restoration of reasoning-related state information while suppressing label-correlated signals.
Results
The proposed method improved selection accuracy by up to 12% compared to the state-of-the-art displacement-only methods and 21% over single-layer probing baselines on unseen reasoning benchmarks. It also outperformed other detectors in factual completion and fact verification tasks, indicating its robustness across different reasoning-related applications.
Implications
The findings suggest that enhancing the interpretability of LLMs through a better understanding of their reasoning processes can lead to more reliable applications in critical areas such as healthcare, legal reasoning, and automated decision-making, where sound reasoning is essential.
Hypothesis Testing with Conditional Queries: Learnability and the Value of Interaction
Theory
- Learnability of distribution classes is contingent on positive separation in pairwise conditional probabilities.
- A randomized non-adaptive procedure can simulate adaptive testing with a controlled total variation distance.
- The worst-case fixed-error adaptivity gap is Θ(N^2), indicating a quadratic advantage from interaction.
- The results challenge the assumption that interactive evaluations yield exponential query advantages.
Read more
Hypothesis Testing with Conditional Queries: Learnability and the Value of Interaction
Summary
This paper investigates the dynamics of hypothesis testing within a conditional-query model, focusing on the learnability of distribution classes and the impact of interaction on testing efficiency. The study distinguishes between static benchmarks, where all tests are predetermined, and interactive evaluations, where subsequent tests depend on earlier responses. The author establishes that learnability is achievable if the pairwise conditional probabilities of the two distribution classes exhibit positive separation. Conversely, if this separation is zero, the worst-case error remains at 1/2 regardless of the query budget. The paper introduces a randomized non-adaptive procedure that simulates an adaptive testing strategy, demonstrating that it can achieve a total variation distance within a specified threshold of the adaptive transcript. Moreover, it reveals that while interaction can significantly reduce the number of required tests, the advantage is quadratic rather than exponential. The findings provide a comprehensive understanding of the trade-offs between adaptive and non-adaptive testing strategies in hypothesis evaluation.
Methodology
The paper employs a conditional distribution testing framework on a finite outcome space, analyzing both adaptive and non-adaptive testing strategies. It constructs a randomized non-adaptive procedure to simulate adaptive testing and derives bounds on query complexity based on the separation of conditional probabilities.
Results
The study provides an exact characterization of conditional-query learnability, establishing that a pair of hypothesis classes is learnable if their conditional probabilities are positively separated. It also presents a non-adaptive simulation bound that is independent of the hypothesis classes' cardinality and separation, while demonstrating that the quadratic dependence on the number of outcomes is unavoidable.
Implications
The findings have significant implications for the design of hypothesis testing frameworks in machine learning, particularly in scenarios where adaptive testing is not feasible. They suggest that understanding the structure of distribution classes can lead to more efficient testing strategies, potentially influencing the development of benchmarks and evaluation methodologies in various applications.
Potential Matching Optimal Transport: Continuous Normalizing Flows for Exact $p$-Wasserstein Dynamics
Generative Models
Optimization
Theory
- Introduction of PMOT, a framework for general p-cost optimal transport using CNFs.
- Establishment of zero-loss exactness, ensuring recovery of p-optimal transport dynamics.
- Demonstration of PMOT's effectiveness on synthetic benchmarks and high-dimensional data.
- Flexible terminal distribution matching without precomputed optimal transport couplings.
Read more
Potential Matching Optimal Transport: Continuous Normalizing Flows for Exact $p$-Wasserstein Dynamics
Summary
This paper introduces Potential Matching Optimal Transport (PMOT), a novel framework for general p-cost optimal transport using Continuous Normalizing Flows (CNFs). PMOT parameterizes the velocity field with a scalar potential in the generalized Benamou–Brenier form, allowing for flexible terminal distribution matching. The authors establish a significant theoretical result: under certain regularity and uniqueness conditions, any zero-loss solution of PMOT recovers the corresponding p-optimal transport map and dynamics. The framework is evaluated on synthetic benchmarks, demonstrating its ability to learn p-specific transport maps that align with reference optimal transport solutions. Additionally, PMOT shows competitive performance as a likelihood-based density model on high-dimensional tabular data and exhibits flexible sample-based terminal matching in color transformation tasks. The results highlight PMOT's potential for applications in various optimal transport problems beyond the quadratic cost scenario.
Methodology
PMOT employs a potential-flow framework that parameterizes the CNF velocity field using a scalar potential in the generalized Benamou–Brenier form. It trains the potential gradient with a self-induced matching loss along straight bridges determined by the model's endpoints, allowing for flexible terminal distribution matching without requiring external optimal transport couplings.
Results
The experiments validate PMOT's ability to learn p-specific transport maps that align with reference optimal transport solutions. The framework demonstrates competitive performance in likelihood-based density modeling on high-dimensional tabular data and shows flexibility in terminal matching through MMD-based color transformation experiments.
Implications
PMOT has the potential to enhance various applications in optimal transport problems, particularly those involving non-quadratic costs. Its ability to learn flexible transport dynamics could be beneficial in fields such as generative modeling, data transformation, and probabilistic inference.
GROM: Gradient-Free Rapid One-Shot Machine Unlearning
Large Language Models
Optimization
Efficient ML
- GROM provides a closed-form solution for machine unlearning, eliminating the need for iterative optimization.
- The method is gradient-free, allowing for rapid updates that are significantly faster than traditional approaches.
- GROM achieves superior forgetting-utility trade-offs across various benchmarks, outperforming existing methods.
- The approach effectively removes targeted knowledge from models, resisting quantization attacks that can recover forgotten information.
Read more
GROM: Gradient-Free Rapid One-Shot Machine Unlearning
Summary
The paper introduces GROM, a novel approach to machine unlearning that allows for the rapid and efficient removal of specific knowledge from large language models (LLMs) without the need for iterative fine-tuning. Traditional methods rely on gradient-based optimization, which can be computationally expensive and may not effectively erase sensitive information. GROM addresses these issues by framing the unlearning process as a ridge-regularized least-squares optimization problem, resulting in a closed-form analytical solution that can be computed in seconds. This method enables a direct additive update to targeted weight matrices, ensuring that unwanted content is suppressed while preserving the model's performance on retained data. The authors demonstrate that GROM achieves state-of-the-art forgetting-utility trade-offs across multiple benchmarks, significantly reducing computational overhead and effectively resisting low-bit quantization attacks that can restore forgotten information. The proposed method is shown to be orders of magnitude faster than existing gradient-based techniques, making it a promising solution for safe and efficient machine unlearning.
Methodology
GROM formulates the unlearning process as a ridge-regularized least-squares optimization problem, deriving a closed-form additive update for targeted weight matrices. This update is computed using gradient-free forward passes over the forget and retain datasets, allowing for rapid execution without backpropagation or iterative convergence.
Results
GROM demonstrated state-of-the-art performance in forgetting-utility trade-offs across five benchmarks, including TOFU-5%, TOFU-10%, MUSE-Books, MUSE-News, and WMDP. The method was found to be up to 180 times faster than existing gradient-based approaches, while also effectively removing sensitive information from models.
Implications
The development of GROM has significant implications for the field of machine learning, particularly in applications where data privacy and the right to be forgotten are critical. Its efficiency and effectiveness make it a valuable tool for organizations that need to manage sensitive information in large language models.
Continual Learning in Transition
Theory
Optimization
Large Language Models
- The transition from parameter-centric to system-level adaptation in Continual Learning.
- Identification of three dimensions of learning: When, How, and Where.
- Emergence of on-policy learning and test-time training as key mechanisms.
- Shift in focus from internal model parameters to external harness components.
Read more
Continual Learning in Transition
Summary
This paper explores the evolving landscape of Continual Learning (CL), moving beyond traditional parameter-centric approaches to a more system-level adaptation framework. The authors identify three critical dimensions of this transition: 'When' learning occurs (pre-training, post-training, and inference), 'How' learning is executed (off-policy, on-policy, and beyond-gradient methods), and 'Where' updates take place (internal parameters versus external structures). The paper systematically surveys existing methods within this tri-axial framework, highlighting the shift from classical CL methods that primarily focus on catastrophic forgetting and stability-plasticity trade-offs to more dynamic approaches that incorporate on-policy learning, test-time training, and external memory systems. This transition is essential for developing models capable of adapting to changing environments and accumulating knowledge over time, which is crucial for achieving artificial general intelligence (AGI). The authors discuss the implications of these developments and outline future research directions in the field of CL.
Methodology
The authors conducted a systematic survey of existing continual learning methods, categorizing them based on the identified dimensions of learning. They analyzed the evolution of learning mechanisms, timing, and capability locus, providing a comprehensive overview of the current state of CL research.
Results
The paper reveals that the landscape of continual learning is rapidly evolving, with significant advancements in on-policy learning and external memory systems. It highlights the limitations of classical methods and emphasizes the need for adaptive systems that can learn continuously in dynamic environments.
Implications
The findings suggest that future research in continual learning should focus on developing models that can effectively adapt to new tasks and environments without forgetting previous knowledge. This has significant implications for applications in artificial intelligence, particularly in achieving AGI and enhancing the capabilities of AI systems in real-world scenarios.
Hybrid-Adaptive Thread Tuning to Mitigate Simulation Execution Bottlenecks in High-Performance Reinforcement Learning Inference
Reinforcement Learning
Optimization
Efficient ML
- Identification of the task execution time to scheduling time ratio as crucial for optimal thread count.
- Introduction of AutoThread, a hybrid adaptive method for thread tuning in SiL-RL systems.
- Utilization of a Physics-Informed Neural Operator for dynamic thread count prediction.
- Significant performance improvements in simulation execution speed and throughput.
Read more
Hybrid-Adaptive Thread Tuning to Mitigate Simulation Execution Bottlenecks in High-Performance Reinforcement Learning Inference
Summary
This paper addresses the execution bottlenecks in simulation-in-the-loop reinforcement learning (SiL-RL) systems, where the performance is often limited by the simulator's execution overhead. The authors identify that the optimal thread count for multithreading in these systems is influenced by the ratio of task execution time to scheduling time. To overcome the limitations of existing static multithreading strategies, they propose AutoThread, a hybrid adaptive thread-tuning method that utilizes a Physics-Informed Neural Operator (PINO) to predict the optimal thread count dynamically. AutoThread incorporates a finite-source M/M/1 queueing model to guide the prediction process, allowing for fast and accurate estimations under varying workloads. Additionally, it employs load-aware online fine-tuning to adjust thread allocation in real-time, compensating for any prediction errors. Experimental results demonstrate that AutoThread achieves an average speedup of 18.4% over static methods, with throughput improvements of 1.7x and 1.8x compared to XGBoost and Reinforcer, respectively, and reduces execution time by up to 83.8% compared to state-of-the-art techniques. The authors also release a large-scale multithreaded trajectory dataset for further research in this area.
Methodology
The authors conducted empirical analysis to understand the relationship between thread count and simulation performance. They developed AutoThread, which combines a Physics-Informed Neural Operator for predicting thread counts with a finite-source M/M/1 queueing model to guide the predictions. The method includes load-aware online fine-tuning to adaptively adjust thread allocation during execution.
Results
AutoThread achieved an average speedup of 18.4% compared to static strategies, with throughput improvements of 1.7x and 1.8x over XGBoost and Reinforcer, respectively. It also reduced execution time by up to 83.8% compared to existing state-of-the-art methods.
Implications
The findings and methodologies presented in this paper can significantly enhance the efficiency of reinforcement learning systems that rely on simulation, particularly in real-time decision-making applications such as wireless communication and autonomous systems. The open-source dataset can facilitate further research and development in optimizing multithreaded simulations.
EvoHarness-RL: Learning Self-Evolving Runtime Harness for Long-Horizon LLM Agents
Reinforcement Learning
Large Language Models
Robotics
- EvoHarness-RL introduces a self-evolving runtime harness for long-horizon LLM agents.
- The framework abstracts external execution support into Belief, Progress, and Experience (BPE).
- Two-stage training involves supervised fine-tuning and cost-aware policy optimization.
- Achieved 96.9% success on seen tasks and 86.6% on unseen tasks in the ALFWorld environment.
Read more
EvoHarness-RL: Learning Self-Evolving Runtime Harness for Long-Horizon LLM Agents
Summary
The paper presents EvoHarness-RL, a novel framework designed to enhance the capabilities of long-horizon LLM agents by enabling them to learn and utilize external execution support more effectively. Traditional approaches rely on manually engineered prompts and heuristics for managing external states, which can be inefficient and suboptimal. EvoHarness-RL introduces a self-evolving runtime harness that abstracts the external workspace into three key components: Belief, Progress, and Experience (BPE). This framework allows agents to learn harness policies offline and apply them during runtime to manage external states dynamically. The methodology involves two main training stages: supervised harness fine-tuning to familiarize the agent with the BPE action space, and cost-aware GRPO (Generalized Reinforcement Policy Optimization) to optimize the agent's decision-making regarding when to access and update external states. The framework is instantiated in the ALFWorld environment using a Qwen3-8B LLM, achieving a success rate of 96.9% on seen tasks and 86.6% on unseen tasks. The study reveals two significant dynamics: harness annealing, where the agent learns to use the harness more selectively over time, and harness evolution, where the agent consolidates its knowledge and refines its external state management. These findings suggest that trainable policies for external state management can significantly enhance the performance of long-horizon agents beyond merely improving tools or memory capacity.
Methodology
The methodology consists of two main training stages: (1) Supervised harness fine-tuning to teach the agent the semantics of the BPE action protocol and how to construct useful external states from interaction traces, and (2) Cost-aware GRPO to optimize the agent's policy for selectively reading, updating, and consolidating external states while considering the interaction budget.
Results
EvoHarness-RL achieved a success rate of 96.9% on the seen tasks and 86.6% on the unseen tasks in the ALFWorld environment. The analysis highlighted two dynamics: harness annealing, where the agent learns to use the harness more selectively, and harness evolution, where the agent consolidates its knowledge and refines its external state management.
Implications
The findings suggest that long-horizon LLM agents can significantly benefit from trainable policies for managing external states, which could lead to more efficient and effective task execution in various applications such as robotics, web navigation, and workflow automation.
Is Self-Pretraining really useful to improve diagnosis in medical Time Series?
Time Series
Multimodal
Robotics
- SPT improves transformer model performance on medical time series tasks.
- Improvements in classification accuracy range from 0 to 6 percentage points.
- Deeper models benefit more from SPT due to better utilization of learned representations.
- SPT is effective for both multivariate and univariate time series data.
Read more
Is Self-Pretraining really useful to improve diagnosis in medical Time Series?
Summary
This paper investigates the effectiveness of Self-PreTraining (SPT) in enhancing the performance of transformer-based models applied to medical time series data. The authors explore whether the benefits of SPT, previously observed in long-context benchmarks, can be extended to various medical applications, including rehabilitation robotics, stress detection, and Parkinson's disease detection. The study evaluates transformer architectures trained from scratch versus those utilizing SPT with four different masking strategies aimed at improving temporal and cross-modal representation learning. The results indicate that SPT consistently enhances classification accuracy across different datasets and model configurations, with improvements ranging from 0 to 6 percentage points. Notably, deeper models show greater benefits from SPT, suggesting that the enriched temporal representations learned during pre-training are better utilized. The findings support the notion that SPT is a straightforward and effective strategy for improving transformer performance in medical time series tasks, particularly in data-limited clinical environments.
Methodology
The authors conducted experiments using transformer architectures on three medical time series tasks, training models either from scratch or through SPT. They employed four masking-based objectives to promote representation learning and systematically varied model depth to assess the interaction between model capacity and pre-training benefits.
Results
The study found that SPT consistently improved classification accuracy across different datasets and configurations, with the most significant gains observed in deeper models. The results indicate that SPT is a viable method for enhancing model performance in medical time series applications, particularly under conditions of limited data.
Implications
The findings suggest that SPT can be a valuable tool for improving diagnostic accuracy in medical settings, especially where labeled data is scarce. This approach could enhance the robustness and reliability of machine learning models in clinical applications, potentially leading to better patient outcomes.
Perturbation Sensitivity at Convergence: A Simple Signal for Identifying Spuriously Correlated Samples
Computer Vision
- Introduces a novel method for identifying spurious correlations in machine learning models post-convergence.
- Utilizes perturbation sensitivity to distinguish between spurious and non-spurious samples without group annotations.
- Demonstrates significant improvement in worst-group accuracy on the Waterbirds dataset.
- Eliminates the need for early stopping and hyperparameter tuning associated with previous methods.
Read more
Perturbation Sensitivity at Convergence: A Simple Signal for Identifying Spuriously Correlated Samples
Summary
This paper addresses the challenge of identifying spurious correlations in machine learning models trained via empirical risk minimization. Such models often achieve high average accuracy but fail on subpopulations where the spurious correlation does not hold. Existing methods for identifying affected samples typically rely on early training signals, which require group annotations and can be hyperparameter dependent. The author proposes a novel approach that utilizes perturbation sensitivity after model convergence to identify spurious correlations without needing group labels. The key insight is that samples consistent with spurious correlations are classified by a shared rule, while non-spurious samples are memorized individually, making them more fragile. By applying a fixed perturbation to the model's inputs, the paper demonstrates that predictions for fragile samples are more likely to flip, allowing for effective identification. This method requires only two forward passes per training sample and significantly improves worst-group accuracy on the Waterbirds dataset, showcasing its effectiveness compared to traditional methods that rely on ground-truth labels.
Methodology
The proposed method involves training a model to convergence and then applying a fixed perturbation (Gaussian blur) to each training sample. A binary sensitivity indicator is computed to determine whether the perturbation changes the predicted class, allowing for the identification of fragile, memorized samples. The samples are then partitioned into pseudo-groups based on their sensitivity, and a rebalancing strategy is employed during retraining to improve accuracy.
Results
The method successfully flagged 70.7% of minority samples and only 7.3% of majority samples in the Waterbirds dataset, indicating a strong separation between spurious and non-spurious samples. The worst-group accuracy improved from 57.3% to 80.8% when using the proposed method, compared to 85.8% with ground-truth labels, demonstrating the effectiveness of the approach.
Implications
This research has significant implications for deploying machine learning models in real-world scenarios where subgroup reliability is crucial. The ability to identify and address spurious correlations without requiring extensive labeled data can enhance model robustness and fairness across diverse populations.
The Tamed Subgradient Unadjusted Langevin Algorithm beyond Convexity
Optimization
Theory
Large Language Models
- Introduction of SG-TULA for sampling from non-convex, non-smooth distributions.
- Utilization of taming techniques to stabilize the algorithm in the presence of superlinear gradient growth.
- Derivation of non-asymptotic convergence bounds in Wasserstein-2 distance.
- Validation of the algorithm's effectiveness through empirical comparisons with existing optimization methods.
Read more
The Tamed Subgradient Unadjusted Langevin Algorithm beyond Convexity
Summary
This paper addresses the challenge of sampling from target distributions characterized by non-smooth potentials, superlinear gradient growth, and non-convexity. The authors introduce the Subgradient Tamed Unadjusted Langevin Algorithm (SG-TULA), which is a discretization of Langevin diffusion that utilizes subgradients directly, avoiding the need for computationally intensive smoothing methods. To manage the superlinear growth, taming techniques are applied to ensure stability in the algorithm. The paper provides non-asymptotic convergence bounds in Wasserstein-2 distance, with constants explicitly defined in relation to dimension and inverse temperature, thus improving existing rates for subgradient-based Langevin algorithms. Additionally, the authors present excess risk estimates for the optimization problem associated with the sampling process. They validate their assumptions with explicit constants for the regularized pretraining potential of a large language model (LLM) in the GPT-2 lineage, demonstrating that the boosted coordinate-wise variant of SG-TULA competes effectively against fine-tuned AdamW and Muon, which lack comparable non-asymptotic guarantees.
Methodology
The authors develop SG-TULA as an explicit Euler–Maruyama discretization of Langevin diffusion, which operates on measurable selections of subgradients. Taming techniques are applied to stabilize the drift, allowing the algorithm to handle superlinear growth without the need for predefined thresholds. The convergence bounds are derived using properties of the associated Gibbs measure and the characteristics of the potential function.
Results
The main results include non-asymptotic convergence bounds in Wasserstein-2 distance, with constants that depend polynomially on dimension and inverse temperature. The algorithm achieves a sampling error that translates into optimization guarantees, with explicit constants for the excess risk associated with the optimization problem. The complexity of the algorithm is shown to be O(ϵ−4), indicating efficient performance.
Implications
The findings suggest that SG-TULA can be effectively applied in scenarios involving non-convex optimization problems, particularly in machine learning contexts such as training neural networks where traditional assumptions about smoothness and convexity do not hold. This could lead to improved sampling methods in various applications, including large language models and other complex machine learning systems.
IFlowNets: Extending Generative Samplers to Learn Strategies in Incomplete Information Games
Reinforcement Learning
Generative Models
Theory
- IFlowNets extend generative flow networks to incomplete information games, addressing limitations of existing methods.
- The authors prove that previous approaches to handling uncertainty are inadmissible for valid sampling in incomplete information settings.
- IFlowNets preserve desired properties such as flow matching, which is essential for learning strategies in games.
- Preliminary results indicate that IFlowNets outperform or match the performance of established methods in standard game environments.
Read more
IFlowNets: Extending Generative Samplers to Learn Strategies in Incomplete Information Games
Summary
This paper presents IFlowNets, an extension of generative flow networks (specifically Adversarial Flow Networks or AFlowNets) to address the challenges posed by incomplete information games. The authors identify that existing methods for handling intragraph uncertainty in complete information games are not applicable to incomplete information scenarios, leading to invalid sampling properties. They propose a new framework, Information Flow Networks (IFlowNets), which preserves the essential flow matching properties necessary for valid strategy learning. The paper theoretically demonstrates that IFlowNets generalize AFlowNets and maintains the expected flow-matching property, which is crucial for achieving a generalized Nash equilibrium known as quantal response equilibrium (QRE). Preliminary experiments in three standard game environments reveal that IFlowNets perform comparably or better than traditional methods such as Outcome Sampling Monte Carlo Counterfactual Regret (OS-MCCFR) and standard reinforcement learning approaches in terms of both performance and computational efficiency.
Methodology
The authors develop IFlowNets by generalizing AFlowNets to accommodate incomplete information. They establish necessary theoretical constraints for valid sampling and demonstrate how these constraints can be applied to ensure expected reward-proportional sampling. The methodology includes aggregating flows over information states and adapting the flow matching properties to the context of incomplete information games.
Results
In preliminary tests across three standard game environments, IFlowNets showed performance that is comparable to or better than OS-MCCFR and standard reinforcement learning methods, indicating their effectiveness in learning strategies under incomplete information.
Implications
The development of IFlowNets has significant implications for game theory and reinforcement learning, particularly in scenarios where agents operate under uncertainty. This framework could enhance the design of algorithms for strategic decision-making in various applications, including economics, competitive gaming, and multi-agent systems.
Does Latent Context Help? A Controlled Evaluation of Inverse Reinforcement Learning in Arctic Shipping
Reinforcement Learning
Robotics
Interpretability
- Latent context in IRL does not necessarily capture hidden preferences but may re-encode observable information.
- The nonlinear reward model outperforms the linear model significantly, while adding latent context decreases performance.
- Behavioral variation among vessels can often be explained by observable features rather than hidden factors.
- A context-need diagnostic is proposed to assess the necessity of latent context in decision-making models.
Read more
Does Latent Context Help? A Controlled Evaluation of Inverse Reinforcement Learning in Arctic Shipping
Summary
This paper investigates the effectiveness of latent context in Inverse Reinforcement Learning (IRL) for Arctic shipping, where AI-assisted navigation is crucial due to rapidly changing sea-ice conditions. The authors evaluate three reward models using a dataset of 3,186 Automatic Identification System (AIS) voyages from 202 vessels across nine Arctic shipping seasons. The models compared include a linear shared reward model (MCE-IRL), a nonlinear shared reward model (AIRL), and a latent-context model (PEMIRL) that incorporates vessel-specific latent context. The findings reveal that while the nonlinear reward model significantly improves held-out likelihood by 50.9% compared to the linear model, the addition of vessel-specific latent context actually reduces performance by 16.5%. Through behavioral analysis and context probes, the authors demonstrate that the observed vessel-level variation can largely be explained by route and environmental conditions rather than hidden preferences. The study emphasizes the need for careful evaluation of latent context in AI systems, particularly in safety-critical applications, and introduces a context-need diagnostic to determine when latent context is beneficial.
Methodology
The authors conducted a controlled evaluation using a dataset of AIS-derived voyages, comparing three reward models: MCE-IRL (linear shared reward), AIRL (nonlinear shared reward), and PEMIRL (nonlinear reward with latent context). They analyzed the models' performance based on held-out likelihood, predictive accuracy, route fidelity, and reward transfer, while also employing behavioral analysis and context probes to understand the sources of behavioral variation.
Results
The nonlinear shared reward model (AIRL) achieved a 50.9% improvement in held-out likelihood over the linear model (MCE-IRL). However, the introduction of vessel-specific latent context in PEMIRL resulted in a 16.5% decrease in performance. The analysis indicated that observable route and environmental conditions largely account for the behavioral variation observed among vessels, challenging the necessity of latent context.
Implications
The findings suggest that in safety-critical applications like Arctic shipping, relying on observable features may be sufficient for modeling behavior, potentially reducing the complexity of AI systems. The context-need diagnostic could guide future research in determining when latent context is beneficial, enhancing the reliability of AI deployments in dynamic environments.
Kastor: An efficient fine-tuning strategy for generative emulation of PDE simulations
Generative Models
Efficient ML
Time Series
- Kastor introduces a two-stage inference scheme to reduce error accumulation in PDE simulations.
- Mean Prediction Regularization (MPR) significantly enhances the stability and accuracy of generative models.
- Incorporating spatial gradient matching improves the physical fidelity of simulations.
- Kastor outperforms existing methods in forecasting accuracy and computational efficiency.
Read more
Kastor: An efficient fine-tuning strategy for generative emulation of PDE simulations
Summary
This paper presents Kastor, a novel methodology designed to enhance the efficiency and accuracy of generative emulation for Partial Differential Equation (PDE) simulations. Traditional PDE solvers are computationally intensive, and while machine learning models can serve as faster surrogates, they often struggle with error accumulation and capturing the stochastic nature of complex systems. Kastor addresses these challenges through a two-stage inference scheme that combines a large-stride causal auto-regressive model with a non-causal temporal super-resolution network, effectively reducing error accumulation. Additionally, the authors introduce Mean Prediction Regularization (MPR), which constrains the generative model to predict the deterministic distribution mean under null noise conditions, leading to improved stability and performance. The paper also highlights the importance of spatial gradient matching to enhance the accuracy and physical fidelity of simulations. Extensive evaluations on diverse datasets demonstrate that Kastor outperforms existing methods, achieving a 42.9% average reduction in forecasting error compared to the baseline Walrus fine-tuning methodology across various datasets.
Methodology
The methodology involves a two-stage inference approach that combines a causal auto-regressive model with a non-causal temporal super-resolution network. The Mean Prediction Regularization (MPR) is employed to constrain the generative model's predictions, and spatial gradient matching is utilized to enhance accuracy. The model is trained on numerical simulation data from The Well, covering various domains.
Results
Kastor demonstrates superior performance compared to the baseline Walrus fine-tuning method, achieving a 42.9% average reduction in forecasting error across 10 datasets. The model shows improvements in forecasting accuracy, spectral consistency, and computational efficiency.
Implications
The advancements presented in Kastor could significantly accelerate physical simulations in various scientific and engineering applications, enabling faster design iterations and more efficient exploration of complex physical phenomena.
Beyond Full-Model Rollback: AuroSFT for Adapter-State Multi-Task Fine-Tuning
NLP
Large Language Models
Efficient ML
- Introduces AuroSFT, a parameter-efficient framework for multi-task fine-tuning.
- Utilizes a compact, mergeable adapter state for task-wise rollback and peak detection.
- Freezes the pretrained backbone and optimizes only adapter parameters.
- Achieves higher accuracy than traditional mSFT methods across multiple benchmarks.
Read more
Beyond Full-Model Rollback: AuroSFT for Adapter-State Multi-Task Fine-Tuning
Summary
The paper presents AuroSFT, a novel framework for multi-task supervised fine-tuning (SFT) that addresses the inefficiencies of traditional methods by utilizing a parameter-efficient approach. Traditional multi-task SFT often treats heterogeneous tasks as a single optimization problem, leading to overfitting and suboptimal performance. AuroSFT introduces a compact, mergeable adapter state that allows for task-wise rollback and peak detection without the need for full-model checkpoints. By freezing the pretrained backbone and training only injected adapters, AuroSFT optimizes the training process, allowing for more efficient transitions between tasks. The framework employs a nonlinear low-rank adapter that transforms weight factors rather than input representations, ensuring that updates remain linear and mergeable. Empirical results demonstrate that AuroSFT outperforms the existing mSFT method, achieving an average accuracy of 61.36% across multiple benchmarks, compared to 59.85% for mSFT. This work highlights the importance of task-specific scheduling in multi-task learning and provides a more efficient alternative to full-model rollback methods.
Methodology
AuroSFT employs a scheduling mechanism that detects task-wise validation peaks and rolls back to optimal checkpoints using a compact adapter state. The framework utilizes a nonlinear low-rank adapter that operates on weight factors, preserving linearity and allowing for efficient merging into the frozen backbone. The training process focuses on optimizing adapter parameters while maintaining the pretrained model's integrity.
Results
AuroSFT achieved an average accuracy of 61.36% across ten benchmarks and five lightweight backbones, outperforming the mSFT reference row, which achieved 59.85%. The framework demonstrated improved performance on all tested backbones, confirming its effectiveness in multi-task fine-tuning.
Implications
The AuroSFT framework has significant implications for the development of more efficient multi-task learning systems, particularly in scenarios where computational resources are limited. Its approach to task-specific scheduling and parameter-efficient fine-tuning can enhance the performance of large language models and other machine learning applications.
Multivariate Time Series Forecasting needs Cross Variable Loss
Time Series
Optimization
Graph Learning
- Identifies a critical gap in existing MTSF models that neglect future inter-variable dependencies.
- Introduces Cross-Variable Loss (CvLoss) as a structural regularizer to improve forecasting accuracy.
- Demonstrates that CvLoss enhances the performance of state-of-the-art forecasting models.
- Proposes a graph-structured approach to capture both synchronous and asynchronous interactions.
Read more
Multivariate Time Series Forecasting needs Cross Variable Loss
Summary
This paper addresses the challenges in multivariate time series forecasting (MTSF), particularly the inadequacy of existing forecasting models that primarily focus on historical cross-variable dependencies while neglecting future inter-variable relationships. The authors identify a critical limitation in the Direct Forecasting (DF) paradigm, which generates multi-step forecasts without considering the inherent co-evolution of future variables. This leads to structural errors due to the mismatch between point-wise objectives and the actual dependencies among future predictions. To resolve this issue, the authors propose a novel structural regularizer called Cross-Variable Loss (CvLoss), which constrains forecast residuals based on a cross-variable graph. CvLoss penalizes inconsistencies in residuals across forecast patches, promoting coherence in both synchronous and asynchronous interactions among variables. The experimental results demonstrate that CvLoss significantly enhances the performance of various competitive forecasting models, outperforming traditional learning objectives while maintaining computational efficiency. This work highlights the importance of capturing the structural relationships among future predictions in MTSF and provides a robust framework for improving forecasting accuracy.
Methodology
The authors propose a graph-based regularization scheme called Cross-Variable Loss (CvLoss), which connects different variables at the same time step to capture synchronous interactions and links variables across different time steps to model asynchronous effects. CvLoss applies an â„“1-norm Graph Total Variation regularization on the prediction residuals, enforcing structural consistency among the forecasts. The method is designed to be a plug-and-play regularizer compatible with various forecasting architectures, with an edge-sampling mechanism to reduce computational complexity.
Results
The experimental evaluations show that models incorporating CvLoss consistently outperform traditional forecasting models and learning objectives, achieving superior predictive performance while maintaining low computational overhead. The results indicate that CvLoss effectively captures the complex interdependencies among future predictions, leading to more accurate forecasts.
Implications
The findings suggest that incorporating structural regularization in MTSF can significantly enhance forecasting accuracy across various domains, such as finance and traffic flow prediction. This approach could lead to more reliable decision-making tools in fields where understanding the relationships among multiple time series is crucial.
PPDL: LLM-Based Flows as Probabilistic Programs
Large Language Models
NLP
Theory
- Introduction of PPDL, the first probabilistic prompt programming language for LLM-based flows.
- Decouples inference scaling from core program logic, simplifying the development process.
- Formal semantics clarifying the interaction between prompting and probabilistic constructs.
- Demonstrated versatility through empirical results with various inference engines and benchmarks.
Read more
PPDL: LLM-Based Flows as Probabilistic Programs
Summary
This paper addresses the challenges of building reliable applications that utilize large language models (LLMs), particularly the issue of uncertainty in their outputs. The authors introduce PPDL, a novel probabilistic programming language designed for LLM-based flows, which allows developers to quantify and propagate uncertainty throughout their applications without complicating the core logic. PPDL integrates probabilistic programming concepts into a high-level prompting language, enabling users to specify flows that include LLM calls and constraints. The language features constructs such as 'sample' for LLM calls and 'factor' for updating probabilities based on user-defined constraints. The runtime of PPDL manages the exploration of multiple execution traces, providing a distribution of outputs rather than a single result. The paper formalizes the semantics of PPDL and presents an implementation that supports various probabilistic inference engines. Empirical results demonstrate PPDL's effectiveness across multiple benchmarks and in a case study involving a theorem proving agent for the Rocq theorem prover.
Methodology
The authors developed PPDL by extending a high-level prompting language with probabilistic programming constructs. They formalized its semantics and implemented it to support parallel execution of traces. The methodology included empirical evaluations using different probabilistic inference engines across various benchmarks and a case study.
Results
The results showed that PPDL effectively allows for experimentation with different inference scaling techniques without additional coding. The framework demonstrated versatility in handling multiple LLMs and benchmarks, and the case study illustrated its practical application in theorem proving.
Implications
PPDL has the potential to enhance the reliability of applications that depend on LLMs by providing a structured way to manage uncertainty. It can be particularly useful for developers and researchers working on multi-step workflows involving LLMs, enabling better assessment of output quality and confidence.
Evaluating Machine Learning Models for Post-Wildfire Debris-Flow Prediction
Interpretability
- Systematic evaluation of 15 machine learning models for post-wildfire debris-flow prediction.
- TabPFN model shows the strongest performance with a threat score of 0.637.
- SHAP analysis identifies key predictive features, emphasizing rainfall intensity and storm accumulation.
- Synthetic data augmentation improves model performance, particularly for deep learning approaches.
Read more
Evaluating Machine Learning Models for Post-Wildfire Debris-Flow Prediction
Summary
This paper addresses the critical issue of predicting post-wildfire debris flows, which pose significant hazards to communities and infrastructure during intense rainfall in recently burned areas. The authors systematically evaluate a range of machine learning models to identify reliable predictors for debris-flow events, focusing on model performance, feature importance, and the impact of synthetic data augmentation. Using basin-scale observations from the western United States, they compare 15 models, including a novel foundation model called the Tabular Prior-Data Fitted Network (TabPFN). The study finds that TabPFN achieves the highest performance in unaugmented scenarios, with a threat score of 0.637, closely followed by tree-based models. Feature importance analysis using SHAP reveals that short-duration rainfall intensity and storm accumulation are the most critical features for prediction, while burn severity and terrain features are less influential. The authors also demonstrate that synthetic data augmentation can enhance model performance, particularly for deep learning models, with an average threat score improvement of +0.041. This comprehensive evaluation framework aims to improve the accuracy and reliability of post-wildfire debris-flow predictions, ultimately aiding in hazard mitigation efforts.
Methodology
The authors conducted a systematic evaluation of 15 machine learning models, including the new TabPFN, using basin-scale observations of post-wildfire debris-flow events. They employed repeated stratified cross-validation for performance assessment and utilized SHAP for feature importance evaluation. Additionally, they explored the effects of synthetic data augmentation on model performance.
Results
The TabPFN model achieved the highest unaugmented performance with a threat score of 0.637. Feature importance analysis revealed that short-duration rainfall intensity and storm accumulation were the most significant predictors. Synthetic data augmentation improved the performance of all models except CNN, with the largest mean gain in threat score observed among deep learning models.
Implications
The findings of this study have significant implications for improving operational warning systems for post-wildfire debris flows. By providing a robust framework for model evaluation and feature analysis, the research can enhance prediction accuracy, thereby aiding in the mitigation of hazards to communities and infrastructure in fire-prone regions.
Equipment-centric workpiece localization in near real-time using deep learning-based vision and event-driven finite state machines
Computer Vision
Robotics
- Introduces an equipment-centric framework for workpiece localization in hot forging environments.
- Achieves 100% event detection accuracy and a mean localization error of 317.8 mm.
- Utilizes event-driven finite state machines for continuous inference of workpiece states.
- Implements Keypoint-Guided Attention to enhance activity recognition performance.
Read more
Equipment-centric workpiece localization in near real-time using deep learning-based vision and event-driven finite state machines
Summary
This paper addresses the critical need for continuous workpiece localization in hot forging environments, where heavy workpieces are transferred among various processing stations. Traditional workpiece-centric localization methods struggle under extreme conditions, leading to inaccuracies. The authors propose an innovative equipment-centric framework that utilizes video streams from static 2D cameras to infer workpiece locations indirectly by analyzing equipment operations. This approach couples equipment behavior with workpiece-handling events, allowing for robust localization even in harsh conditions. The framework estimates 3D coordinates of handling equipment and recognizes activities such as grasping and releasing workpieces. Event-driven finite state machines (FSMs) are employed to validate and detect handling events, enabling continuous updates of workpiece states. Experimental validation in a hot forging factory demonstrated the framework's effectiveness, achieving 100% event detection accuracy within a 33-second tolerance window and a mean localization error of 317.8 mm. The integration of a Keypoint-Guided Attention (KPGA) mechanism further improved activity recognition performance compared to baseline models. This structured output not only enhances localization accuracy but also facilitates data-driven visualization of workpiece transfers, contributing to more intelligent and traceable forging operations.
Methodology
The proposed framework analyzes video streams from multiple static 2D cameras to infer workpiece locations by observing equipment operations. It employs event-driven finite state machines to validate handling events based on equipment coordinates and recognized activities, allowing for continuous updates of workpiece states.
Results
The framework demonstrated 100% accuracy in detecting handling events within a 33-second tolerance window, with a mean localization error of 317.8 mm and a system latency of 21 seconds. The integration of the KPGA mechanism significantly improved activity recognition performance over traditional models.
Implications
This research has significant implications for enhancing traceability and quality control in hot forging operations, enabling more intelligent manufacturing processes and reducing errors associated with workpiece localization. The framework can be adapted for various industrial applications requiring robust localization under challenging conditions.
LC-GRPO: Bridging Train-Inference Gap for Flow-Based GRPO with Langevin Correction
Generative Models
Reinforcement Learning
Computer Vision
- Introduces LC-GRPO to bridge the training-inference gap in flow-based generative models.
- Utilizes a two-step approach combining ODE Euler steps with Langevin correction for improved sampling.
- Theoretical justification shows reduced Wasserstein error with Langevin correction.
- Empirical results indicate consistent improvements in reward optimization and generation quality.
Read more
LC-GRPO: Bridging Train-Inference Gap for Flow-Based GRPO with Langevin Correction
Summary
The paper presents LC-GRPO, a novel framework designed to address the discrepancies between training and inference in flow-based generative models, particularly in the context of reinforcement learning (RL). Traditional flow-based models utilize deterministic ordinary differential equations (ODEs) for sampling, while RL requires stochastic rollouts for effective policy exploration. Existing methods attempt to bridge this gap by substituting the ODE with a stochastic differential equation (SDE) during training, but this often leads to blurred samples and a mismatch in the quality of generated outputs. LC-GRPO introduces a two-step approach where each rollout transition first follows an ODE Euler step, aligning it with inference, and then applies a Langevin correction to introduce stochasticity while targeting the correct marginal distribution. This method leverages the flow velocity to obtain the necessary score without needing an additional score model, ensuring that the transition remains isotropic Gaussian with a manageable likelihood for optimization. Theoretical analysis shows that this Langevin correction can effectively reduce the Wasserstein error of the ODE step, leading to more accurate sampling. Experimental results demonstrate that LC-GRPO significantly enhances reward optimization in text-to-image and text-to-video tasks, improves generation quality, and minimizes the gap between stochastic training rollouts and deterministic test-time ODE inference.
Methodology
The methodology involves a two-step rollout process where the first step uses an ODE Euler update to align with inference sampling, followed by a Langevin correction that introduces stochasticity while targeting the marginal distribution. This approach allows for the direct use of flow velocity to obtain the necessary score, avoiding the need for an additional score model.
Results
LC-GRPO outperforms existing methods such as Flow-GRPO and DanceGRPO in terms of reward optimization across multiple datasets (SD3.5-Medium, FLUX.1-Dev, and HunyuanVideo). It maintains a favorable trade-off between reward and quality metrics, producing clearer stochastic rollouts and minimizing discrepancies between training and evaluation rewards.
Implications
The findings suggest that LC-GRPO can enhance the performance of flow-based generative models in reinforcement learning settings, making it a valuable tool for applications in visual generation tasks where high-quality outputs and effective policy exploration are critical.
Consistency Has a Computable Blind Spot: A Commutation Theory of Label-Free Reliability for Vision-Language Figure Reading
Multimodal
Theory
- Introduces a commutation theory that identifies a computable blind spot in label-free reliability for VLMs.
- Develops the Equivariance-Consistency Score (ECS) as a new label-free, training-free error detection method.
- Demonstrates that certain errors remain undetected due to their commutation with answer-transforms, leading to a systematic misreading.
- Empirical validation shows ECS significantly improves error detection rates compared to traditional methods.
Read more
Consistency Has a Computable Blind Spot: A Commutation Theory of Label-Free Reliability for Vision-Language Figure Reading
Summary
This paper addresses the reliability of vision-language models (VLMs) through the lens of invariance and equivariance. It identifies a critical blind spot in label-free reliability assessments, where systematic misreadings can persist despite input perturbations. The authors introduce a commutation theory that mathematically characterizes the errors that remain undetected by existing methods. They demonstrate that errors are invisible to certain edits when they commute with the answer-transform induced by those edits. The paper proposes the Equivariance-Consistency Score (ECS) as a label-free, training-free detection mechanism, and introduces REND-EQUIV, which pairs matched invariance and equivariance sets over identical data. Empirical results confirm the theoretical predictions, showing that the ECS effectively detects a broader range of errors compared to traditional methods. The findings suggest that the blind spot is a structural property of the relation between edits and errors, rather than an implementation issue, and provide a framework for designing more effective error detection suites.
Methodology
The authors utilize algebraic characterization to define the relationship between edits and errors, focusing on commutation properties. They empirically validate their theoretical findings through experiments involving matched edits and real chart instances, comparing the performance of ECS against traditional methods.
Results
The study finds that a two-element edit suite can detect all non-identity affine errors, while traditional swap edits fail to detect many label permutations. The introduction of cyclic relabeling significantly improves detection rates, achieving an eightfold increase in label-error detection in matched samples. The ECS and REND-EQUIV demonstrate consistent performance across different models and datasets, confirming the theoretical predictions.
Implications
The findings have significant implications for the development of more reliable vision-language models, particularly in applications requiring accurate figure reading and interpretation. The proposed methods can enhance the robustness of VLMs against systematic errors, improving their utility in real-world scenarios.
KV-Skill: Forging Expertise in the Model's Native Language
NLP
Large Language Models
Optimization
- KV-Skill introduces an external operator design space for task knowledge in language models.
- The framework allows for two paths to create external capabilities: registration of text skills and reward learning from task outcomes.
- Experiments show significant improvements in task performance compared to traditional text and continuous-skill methods.
- The approach retains task-specific knowledge without measurable forgetting when using a shared interface.
Read more
KV-Skill: Forging Expertise in the Model's Native Language
Summary
The paper introduces KV-Skill, a novel framework for integrating task knowledge into frozen language models through external factorized operators. Traditional methods either rely on text prompts, which require interpretation at each use, or weight adaptations, which complicate knowledge sharing and deployment. KV-Skill addresses these issues by allowing task knowledge to be represented as independent operators that can be loaded on demand. The framework supports two main paths: registration, which converts authored text skills into fixed operators, and reward learning, which develops compact operators based on task outcomes. The authors demonstrate that converting text to KV-Skill significantly enhances procedural knowledge effectiveness across various benchmarks and model architectures. The results indicate that task knowledge can be effectively compressed and deployed separately from model weights, providing a modular approach to integrating expertise into language models.
Methodology
The authors developed KV-Skill as an external factorized associative operator that interacts with a frozen language model through a lightweight interface. They conducted experiments across ten benchmarks and four model backbones, comparing the performance of KV-Skill against traditional text-based skills and other adaptation methods like soft prefixes and LoRA. The two main methods explored were registration, which compiles text skills into operators, and reward learning, which optimizes operators based on task outcomes.
Results
KV-Skill achieved an accuracy of 77.2% on the Qwen3.5-4B LiveMath benchmark, significantly outperforming the original text skill (23.4%), SkillOpt (52.0%), and SoftSkill (64.5%). In matched reward training scenarios, KV-Skill produced the best results in seven out of eight settings compared to other adaptation techniques. The study also found that task knowledge could be compressed into a compact operator with minimal loss of effectiveness, and a shared interface could manage multiple KV-Skills without forgetting.
Implications
The KV-Skill framework has the potential to enhance the modularity and efficiency of language models by allowing for independent loading and sharing of task knowledge. This could lead to more flexible and effective applications in various NLP tasks, enabling models to adapt to new tasks without extensive retraining. Additionally, the separation of knowledge acquisition from deployment could facilitate better knowledge management in AI systems.
DG-FedReuse: Proxy-Gradient-Gated Cached-Update Reuse with Matched Sparse Uplink Accounting
Federated Learning
Efficient ML
Optimization
- DG-FedReuse allows for cached-update reuse in federated learning, enhancing communication efficiency.
- The method employs a stochastic proxy-gradient discrepancy to decide between fresh and cached updates.
- Significant uplink savings were achieved compared to existing methods, although accuracy differences were minimal.
- The study provides a comprehensive audit of the method's performance and limitations.
Read more
DG-FedReuse: Proxy-Gradient-Gated Cached-Update Reuse with Matched Sparse Uplink Accounting
Summary
The paper introduces DG-FedReuse, a novel mechanism designed to enhance communication efficiency in federated learning (FL) by allowing selected clients to utilize age-decayed cached updates under specific conditions. The method employs a stochastic proxy-gradient discrepancy to determine whether to use a fresh local training update or a cached one, constrained by a hard cache-age limit and a minimum quota for fresh updates. The authors conducted experiments across six image classification datasets with 50 virtual clients and varying levels of label heterogeneity. The results indicate that DG-FedReuse achieves significant uplink savings (83.36% to 85.42%) compared to the matched Top-K FedAvg method (76.88%), although the accuracy differences are minimal. The study emphasizes the importance of communication metrics and provides a detailed audit of the proposed method's boundaries, noting that while it offers promising uplink savings, it does not guarantee improvements in generalization, energy savings, or faster convergence compared to existing methods.
Methodology
The authors implemented DG-FedReuse in a simulator, utilizing a stochastic proxy-gradient discrepancy to guide update reuse decisions. They constrained the reuse of cached updates with a hard cache-age limit and a minimum fresh-client quota. The experiments involved six datasets and 50 virtual clients, focusing on communication metrics and accuracy differences under a fixed communication-round budget.
Results
DG-FedReuse achieved uplink savings ranging from 83.36% to 85.42% compared to 76.88% for matched Top-K FedAvg. The accuracy differences between methods ranged from -5.29 to -0.14 percentage points, with best-observed test accuracies showing minimal variation relative to matched FedAvg. The study also noted a reduction in uplink savings when accounting for dense-model downlink sensitivity.
Implications
The findings suggest that DG-FedReuse could be beneficial in scenarios where communication costs are critical, such as in mobile or edge computing environments. However, the limitations in generalization and convergence speed indicate that further research is needed to fully understand its applicability in real-world federated learning systems.
RRC: Unlocking Generative Reward Models in LLM Reinforcement Learning via Ranking-Based Reward Construction
Reinforcement Learning
Large Language Models
Generative Models
- Generative reward models have not been fully utilized in RL due to a mismatch with scalar scoring paradigms.
- The proposed RRC approach enables effective reward construction from relative preference rankings.
- RRC introduces self-competitive and anchor-guided ranking strategies for improved scalability.
- Experiments show significant performance improvements in RL tasks using RRC compared to traditional methods.
Read more
RRC: Unlocking Generative Reward Models in LLM Reinforcement Learning via Ranking-Based Reward Construction
Summary
This paper addresses the limitations of generative reward models in reinforcement learning (RL) for large language models (LLMs). The authors propose a novel approach called Ranking-based Reward Construction (RRC), which leverages the comparative nature of generative reward models to provide more effective learning signals for RL. The RRC approach consists of two strategies: self-competitive ranking, which utilizes comparisons among sampled responses, and anchor-guided ranking, which allows for scalable reward construction using a small set of reference responses. Through extensive experiments on various benchmarks, the authors demonstrate that RRC significantly enhances RL training performance compared to existing reward construction methods. The results indicate that RRC not only improves the effectiveness of generative reward models in RL but also offers scalability by increasing inference-time computation and the number of anchor responses, thereby enabling higher performance ceilings for RL applications.
Methodology
The authors developed the RRC approach, which constructs scalar reward scores from relative preference rankings among sampled responses. It includes two mechanisms: self-competitive ranking, which compares responses generated by the current policy, and anchor-guided ranking, which uses a small number of reference responses to facilitate scalable reward construction.
Results
The RRC approach achieved notable improvements in RL performance across multiple benchmarks, such as increasing scores from 35.8% to 41.3% on AlpacaEval2 and from 8.0% to 11.2% on ArenaHardV2. Additionally, RRC demonstrated scalability in performance with increased inference-time computation and more anchor responses, leading to further enhancements in RL outcomes.
Implications
The findings suggest that RRC can significantly enhance the effectiveness of generative reward models in RL, potentially leading to more advanced applications in natural language processing and other areas where LLMs are utilized. The scalability aspect of RRC also opens new avenues for improving RL performance in complex tasks.