AI-generated summaries
Today's ML research,
without the noise.
Daily summaries of the latest machine learning papers from arXiv, processed every 8 hours.
67
Papers today
8h
Update frequency
7
Days of history
Beyond Simulated Benchmarks: Evaluating Motion Representations for Fall Detection Under Real-World Data Scarcity
Time Series
- Real-world fall data is extremely scarce, making it challenging to train robust fall detection models.
- Simulated datasets often lead to high performance in controlled settings but fail to generalize to real-world applications.
- Interval-based representations achieve the best real-world performance, while symbolic representations with impact descriptors show resilience under data scarcity.
- The study emphasizes the importance of representation choice for effective fall detection in real-world scenarios.
Read more
Beyond Simulated Benchmarks: Evaluating Motion Representations for Fall Detection Under Real-World Data Scarcity
Summary
This paper addresses the challenge of fall detection in older adults using wearable sensors, emphasizing the limitations posed by the scarcity of real-world fall data. The authors highlight that while many existing models achieve high accuracy in controlled laboratory settings, they often fail to generalize to real-world scenarios due to reliance on simulated datasets. The study systematically evaluates various motion representations—interval-based, kernel-based, symbolic, and foundation models—under conditions of limited real-world data. The authors introduce a lightweight symbolic representation, FallLM, which incorporates physically-grounded impact descriptors to enhance interpretability. Through experiments using both simulated and clinically verified real-world datasets, the study reveals that while complex models perform well in simulations, they struggle under data scarcity and domain shifts. In contrast, the interval-based representation shows strong real-world performance, and the symbolic representation with impact descriptors demonstrates resilience against degradation in detection sensitivity. The findings underscore the necessity of evaluating models beyond simulated benchmarks and highlight the critical role of representation choice in developing effective fall detection systems.
Methodology
The authors conducted a systematic evaluation of different motion representations for fall detection using accelerometer data from both simulated (FallAllD) and real-world (FARSEEING) datasets. They employed a streaming event-detection protocol with subject-wise splits, allowing for realistic testing conditions. Various representation types were compared, including interval-based, kernel-based, symbolic, and foundation models, with a focus on their performance under data scarcity and domain shifts.
Results
The study found that highly parameterized kernel and foundation models performed well on simulated data but significantly degraded in performance when faced with real-world data scarcity and domain shifts. The interval-based representation achieved the highest absolute performance in real-world scenarios, while the symbolic representation with impact descriptors showed the least degradation under domain shifts, maintaining detection sensitivity despite lower precision.
Implications
The findings of this study have significant implications for the development of fall detection systems, particularly in geriatric care and remote health monitoring. By highlighting the importance of motion representation choice, the research encourages the design of more robust models that can effectively operate in real-world conditions, ultimately improving safety and response times for older adults at risk of falls.
Active-Trace Complexity Bounds for Moreau--Yosida Unadjusted Langevin Sampling
Theory
Optimization
Efficient ML
- Introduces the concept of reference active trace (Bref) to analyze MYULA discretization error.
- Establishes a complexity bound for the number of iterations needed to achieve desired accuracy.
- Demonstrates that the active trace can be independent of the smoothing parameter λ for structured penalties.
- Provides a Moreau-bias bound that helps in choosing the optimal smoothing parameter.
Read more
Active-Trace Complexity Bounds for Moreau--Yosida Unadjusted Langevin Sampling
Summary
This paper investigates the Moreau–Yosida unadjusted Langevin algorithm (MYULA) for sampling from nonsmooth composite target distributions. The authors focus on the target distribution defined as π(dx) ∝ exp{−f(x) − g(x)}, where f is m-strongly convex with an Lf-Lipschitz gradient, and g is convex and G-Lipschitz. The key contribution is the introduction of the reference active trace, Bref, which is shown to control the leading MYULA discretization error, rather than relying on the global curvature bound. The paper establishes a complexity bound for the number of iterations required to ensure a desired accuracy in terms of the active trace and other parameters, providing a more nuanced understanding of the discretization error in MYULA. The authors also derive a Moreau-bias bound and suggest an optimal choice for the smoothing parameter λ that balances regularization bias and discretization stability. The results indicate that for structured penalties, the active trace can be independent of λ, leading to improved accuracy estimates for the MYULA kernel.
Methodology
The authors develop a distribution-dependent analysis of the Moreau–Yosida unadjusted Langevin algorithm, focusing on the active trace of the Moreau envelope of the nonsmooth term g. They derive bounds on the discretization error and establish relationships between the active trace, curvature, and the number of iterations required for convergence. The analysis incorporates the properties of the Hessian of the Moreau envelope and uses techniques from nonsmooth calculus.
Results
The paper presents a complexity bound that shows the number of iterations N required to ensure that the quadratic Wasserstein distance between the law of the N-th iterate and the smoothed target is within a specified error εalg. The results indicate that the active trace plays a crucial role in determining the discretization error, leading to improved accuracy estimates for various structured penalties. The Moreau-bias bound provides a framework for selecting the smoothing parameter λ to optimize performance.
Implications
The findings have significant implications for Bayesian inference and high-dimensional statistics, particularly in scenarios involving nonsmooth composite targets. The improved understanding of the discretization error and the role of the active trace can enhance the performance of sampling algorithms in practical applications, such as sparse Bayesian inference and imaging.
Training Under Challenge: Executable Certificates and Challenge-Closed Optimality for Neural Networks
Theory
Optimization
- Introduces an executable-certificate framework for neural network training assurance.
- Establishes a method to construct complete alternative models for objective reevaluation.
- Defines a challenge-power modulus to characterize optimality gaps.
- Demonstrates the framework's effectiveness through empirical results on ResNet-18.
Read more
Training Under Challenge: Executable Certificates and Challenge-Closed Optimality for Neural Networks
Summary
This paper introduces 'Training Under Challenge,' an innovative framework aimed at enhancing the training assurance of neural networks through executable certificates. The authors argue that a flat training curve does not necessarily indicate that a neural network has achieved global optimality, as it could be trapped in a local minimum or limited by its representation. The proposed framework allows for the construction of complete alternative models within the same certified class, enabling a reevaluation of the objective function. By establishing a resource-indexed challenge-power modulus, the framework characterizes the largest optimality gap compatible with successful passage of challenges. The authors demonstrate the effectiveness of their approach through a case study involving a ResNet-18 distillation problem, where they achieved bounds on the empirical global-optimality gap. The paper also discusses the importance of coverage mechanisms and presents a method for diagnosing and repairing under-utilized decoders. Overall, this work provides a structured approach to assess and improve the training process of neural networks, ensuring that models are not only trained effectively but also certified for their performance.
Methodology
The authors developed an executable-certificate framework that involves constructing alternative models through architecture-valid procedures. They defined a challenge-power modulus to assess the optimality gap and used block-decrease operators to ensure coverage checkability. The methodology includes a systematic audit of checkpoints and the evaluation of alternative models against a fixed objective.
Results
In the case study involving ResNet-18, the authors successfully covered all 240 audited output directions with eight internal challenges. The realized-residual bounds achieved were within factors of 1.74 to 3.02 of the true optimality gap, demonstrating the framework's effectiveness in identifying and certifying model performance.
Implications
The proposed framework has significant implications for the field of neural network training, providing a structured approach to ensure models are not only trained effectively but also certified for their performance. This could lead to more reliable AI systems in various applications, particularly in critical domains where model assurance is paramount.
RelShap: Relationally Consistent Shapley Explanations
Interpretability
- RelShap integrates relational database constraints into Shapley value computations.
- The framework restricts background data and coalition evaluations to valid relational configurations.
- RelShap outperforms existing methods like Kernel SHAP and Conditional SHAP in identifying dominant features.
- The method is estimator-agnostic and can be combined with existing Shapley value methods.
Read more
RelShap: Relationally Consistent Shapley Explanations
Summary
The paper introduces RelShap, a novel framework designed to provide Shapley value-based feature attributions that respect relational constraints inherent in relational databases. Traditional methods often flatten relational data into single-table formats, leading to misleading feature attributions due to the assumption of feature independence. RelShap addresses this issue by incorporating relational constraints and data provenance into the Shapley value computation, ensuring that both background data and coalition evaluations are limited to valid relational configurations. This estimator-agnostic framework can be integrated with existing methods like Kernel SHAP, Monte Carlo, and Leverage SHAP without altering their sampling properties. The authors demonstrate that RelShap can significantly improve the accuracy of feature attributions, correctly identifying dominant features in controlled settings where existing methods fail. The framework also offers a combinatorial characterization of expected speedup in computation, making it efficient for practical use. Experiments across various datasets and models validate the effectiveness of RelShap in producing more faithful explanations aligned with the data-generating process.
Methodology
RelShap modifies the Shapley value computation by enforcing relational constraints derived from the database schema and query structure. It restricts both the background data and the feature coalitions to only those configurations that are relationally valid. The framework can be applied in a provenance-aware mode, tracing back the data to its original tuples, enhancing the interpretability of the attributions.
Results
Experiments show that RelShap produces more accurate and faithful feature attributions compared to traditional methods, correctly identifying the dominant feature in scenarios where others fail. The framework demonstrates significant runtime efficiency improvements through its combinatorial approach.
Implications
RelShap has the potential to enhance the interpretability of machine learning models trained on relational data, making it particularly useful in domains such as finance, healthcare, and any field where relational databases are prevalent. By providing more accurate explanations, it can improve trust and understanding of model predictions among practitioners.
The Impact of Temporal Context Length and Encoding Strategies on Self-Supervised ECG Representation Learning
Time Series
- Longer temporal contexts (5-10 minutes) improve ECG representation learning and downstream classification accuracy.
- Continuous convolutional patch embeddings outperform discretized vector-quantized tokens, preserving critical waveform details.
- The study emphasizes the need for ECG models that capture slow-varying rhythm dynamics and individual-specific structures.
Read more
The Impact of Temporal Context Length and Encoding Strategies on Self-Supervised ECG Representation Learning
Summary
This paper investigates the effects of temporal context length and encoding strategies on self-supervised learning (SSL) for electrocardiogram (ECG) representation. The authors conduct a controlled study using the Icentia11k dataset, varying the input horizon (16 seconds, 1 minute, 5 minutes, and 10 minutes) and the front-end representation (continuous convolutional patch embeddings versus fixed vector-quantized tokens). The study aims to determine whether shorter training windows and discretized tokenization compromise the information necessary for accurate rhythm inference and patient-level consistency. The results indicate that longer temporal contexts significantly enhance transfer learning and retrieval accuracy, with the best performance observed at 5 and 10 minutes. Additionally, continuous patch embeddings consistently outperform discretized tokens, highlighting the importance of preserving detailed waveform characteristics for clinical applications. The findings advocate for the development of ECG foundation models that prioritize extended context and continuous encoders to improve clinical prediction and patient-specific applications.
Methodology
The authors conducted experiments using the Icentia11k dataset, comparing self-supervised models trained with different temporal contexts and encoding strategies. They evaluated the models based on their performance in a downstream atrial fibrillation classification task and patient-level retrieval metrics to assess consistency across time and activity.
Results
The study found that increasing the temporal context beyond 16 seconds led to improved classification performance and patient-level consistency, with the most significant gains at 5 and 10 minutes. Continuous patch embeddings consistently provided better results than discretized tokens, indicating that quantization may obscure clinically relevant details.
Implications
The findings suggest that ECG representation learning can benefit from longer temporal contexts and continuous encoding strategies, potentially leading to more accurate clinical predictions and better patient-specific applications. This could influence the design of future ECG foundation models and enhance the utility of self-supervised learning in clinical settings.
The Boolean Power of ReLU
Graph Learning
Theory
- ReLU-MPLang is strictly more expressive than TrReLU-MPLang for Boolean queries on graphs with Boolean features.
- The study resolves an open problem regarding the comparative power of ReLU and truncated ReLU in Boolean contexts.
- The findings emphasize the role of activation functions in the expressiveness of graph neural networks.
- A specific Boolean ReLU query is demonstrated to be undefinable in Σ-MPLang for any collection of eventually constant activation functions.
Read more
The Boolean Power of ReLU
Summary
This paper investigates the expressive power of graph neural networks (GNNs) using different activation functions, specifically focusing on the ReLU (Rectified Linear Unit) and truncated ReLU (TrReLU) functions. The authors prove that for finite simple undirected graphs with a single Boolean node feature, the Boolean queries expressible in the Σ-MPLang (a message-passing language for GNNs) form a strict subset of those expressible in ReLU-MPLang. This finding resolves an open question regarding the comparative expressiveness of ReLU-MPLang and TrReLU-MPLang in the context of Boolean queries. The results indicate that ReLU-GNNs are strictly more expressive than GNNs using TrReLU or identity activations when it comes to Boolean queries. The paper highlights the importance of activation functions in determining the capabilities of GNNs, particularly in recognizing node properties after Boolean thresholding.
Methodology
The authors utilize formal proofs to establish the relationships between different message-passing languages (MPLang) and their expressiveness concerning Boolean queries. They define the semantics of various activation functions and their impact on the expressiveness of GNNs, employing structural induction and Booleanization techniques to demonstrate their results.
Results
The main result shows that there exists a ReLU-MPLang expression that cannot be expressed in Σ-MPLang for any collection of eventually constant activation functions, establishing that bool(TrReLU-MPLang) is a strict subset of bool(ReLU-MPLang). This finding confirms that the unbounded nature of ReLU enhances the expressiveness of GNNs beyond that of truncated ReLU.
Implications
The results suggest that when designing GNN architectures, the choice of activation function significantly influences the model's ability to perform Boolean queries, which could be critical in applications involving graph-based data. This insight may guide future research and development in GNNs, particularly in fields that rely on Boolean decision-making.
Chain-of-Thought Shows the Path to a Tree: Realizing Branching Complexity
Theory
Efficient ML
- First CoT realizations of graph traversal algorithms (DFS and Dijkstra) using hard-attention Transformers.
- Explicit small-depth CoT constructions for Strahler number and width of trees, extending previous work on binary trees to arbitrary n-ary trees.
- Demonstrated that Dijkstra's algorithm can outperform comparison-based implementations through constant-time attention operations.
- Established a bijection between trees and Dyck paths, allowing for independent CoT constructions for both representations.
Read more
Chain-of-Thought Shows the Path to a Tree: Realizing Branching Complexity
Summary
This paper addresses the limitations of bounded-depth Transformers in expressing complex computational tasks by leveraging Chain-of-Thought (CoT) prompting. The authors provide concrete implementations of depth-first search (DFS) and Dijkstra's algorithm using hard-attention decoders with a maximum of two layers. These implementations serve as a computational substrate for calculating the Strahler number and width of trees, which are significant measures in branching complexity. The authors demonstrate that the Strahler number can be computed in 2n - 1 steps for an n-vertex tree, while the width can be computed in n - 1 steps. Notably, these constructions do not rely on auxiliary primitives such as layer normalization, marking a significant advancement in the CoT framework. Furthermore, the paper explores the relationship between ordered trees and Dyck paths, providing independent CoT constructions for both measures on these representations. The findings contribute to the understanding of CoT's expressivity and its potential to solve complex problems in a linear-step regime.
Methodology
The authors implemented depth-first search and Dijkstra's algorithm using hard-attention Transformer decoders with a maximum of two layers. They utilized these algorithms as a computational substrate to compute the Strahler number and width of trees. Additionally, they explored the bijection between ordered trees and Dyck paths to provide independent constructions for both measures.
Results
The paper presents that the Strahler number of an n-vertex tree can be computed in 2n - 1 steps, and the width can be computed in n - 1 steps. These results are significant as they demonstrate the capability of CoT to express complex computations in a linear-step framework, particularly for arbitrary n-ary trees.
Implications
The findings suggest that CoT prompting can enhance the expressiveness of Transformers in solving complex computational problems, potentially leading to more efficient algorithms in graph traversal and tree analysis. This work opens up avenues for further research into the capabilities of CoT in various computational contexts.
The data geometry of masking diffusion: Certified-optimal schedules via unmasking growth complexity
Theory
Generative Models
Efficient ML
- Introduction of unmasking growth complexity (UGC) as a measure of data geometry in masking diffusion.
- UGC increments control KL discretization error, enabling a unified analysis of unmasking schemes.
- Development of certified-optimal samplers that achieve specified KL error with high probability.
- Connection of UGC to classical multivariate dependence measures and previous complexity results.
Read more
The data geometry of masking diffusion: Certified-optimal schedules via unmasking growth complexity
Summary
This paper investigates masking diffusion for discrete sampling and introduces a novel measure of data geometry termed unmasking growth complexity (UGC). The UGC provides a path-resolved analysis that directly influences Kullback–Leibler (KL) discretization error, allowing for a unified examination of both Bernoulli-subset and fixed-cardinality unmasking schemes. By employing log-reveal-odds coordinates, the paper develops optimized sampling schedules that adapt computational effort based on data geometry. A significant contribution is the estimation of UGC increments from samples through KL increments along coupled reveal trajectories, leading to the creation of certified-optimal samplers that maintain a high probability of achieving a specified KL error with iteration complexity closely aligned with oracle procedures. The study also connects the aggregate UGC mass to classical multivariate dependence measures and previous complexity analyses of discrete diffusion. In the fine-partition limit, the squared integral of the square-root UGC density is shown to dictate the leading-order optimal Euler discretization error. The findings demonstrate substantial improvements in sampling efficiency, particularly in high-dimensional spaces, with potential gains scaling with the square root of the dimension.
Methodology
The methodology involves analyzing the unmasking growth complexity (UGC) as a path-resolved measure of data dependence during the unmasking process. The authors derive optimized sampling schedules based on UGC increments, estimate these increments from samples, and construct data-dependent samplers that certify optimal performance in terms of KL error. The study employs log-reveal-odds coordinates to facilitate the analysis and optimization of sampling algorithms.
Results
The paper presents certified-optimal sampling algorithms that achieve a prescribed KL error with high probability and iteration complexity comparable to oracle procedures. It establishes that the UGC complexity can be estimated from samples, leading to optimized blockwise schedules and significant improvements in sampling efficiency, particularly in high-dimensional settings.
Implications
The findings have implications for various applications requiring efficient sampling from high-dimensional discrete distributions, including machine translation, image synthesis, and generative AI. The certified-optimal samplers could enhance the performance of algorithms in these domains by providing more reliable and efficient sampling strategies.
Exponential Convex Calibration Dimension for the Multi-Label Jaccard Measure
Theory
- The Jaccard score's loss matrices are nonsingular with affine dimension 2s - 1.
- Exact calibration of the Jaccard score requires exponentially many prediction coordinates.
- Two polynomial-dimensional approximation guarantees are established, allowing for practical surrogate construction.
- A new transfer from F1 to Jaccard provides a polynomial-time rule with controlled regret.
Read more
Exponential Convex Calibration Dimension for the Multi-Label Jaccard Measure
Summary
This paper investigates the calibration dimension of the Jaccard score, a common metric in multi-label classification and binary segmentation. The author establishes that the loss matrices associated with the Jaccard score, shifted-loss, and ordinary loss are nonsingular, with affine dimensions of 2s - 1, where s is the number of labels. The paper proves bounds on the convex calibration dimension, showing that exact calibration requires exponentially many prediction coordinates. Two polynomial-dimensional approximation guarantees are provided, including a method that transforms an existing F1 surrogate into a Jaccard surrogate with controlled regret. The paper combines theoretical insights with practical implications for designing calibrated surrogates in multi-label settings.
Methodology
The author employs a combination of finite MinHash Gram representations and Boolean Möbius inversion to analyze the Jaccard score's properties. The paper derives bounds on the convex calibration dimension and develops approximation methods that leverage existing F1 surrogates and MinHash features.
Results
The paper establishes that the convex calibration dimension for the Jaccard score lies between 2s - 1 and 2s, indicating that exact calibration necessitates exponential dimensions. Additionally, it presents two polynomial-dimensional approximation methods that achieve controlled regret for Jaccard surrogates, demonstrating the feasibility of practical implementations.
Implications
The findings have significant implications for multi-label classification tasks, particularly in areas where the Jaccard score is critical, such as image segmentation. The results suggest that while exact calibration may be computationally intensive, effective approximations can be achieved, enabling more efficient model training and evaluation.
Multi-perspective Imbalance-Conscious 6G Beamforming Optimization and Performance
Optimization
- Network features are more predictive for 6G beamforming optimization than device, environmental, and vision features.
- Unsupervised clustering methods reveal that deployment environment and device type significantly influence clustering outcomes.
- The study emphasizes the need for adaptive beamforming techniques in dynamic 6G-IoT environments.
- Future work will explore deep and reinforcement learning for optimizing performance indicators like throughput and latency.
Read more
Multi-perspective Imbalance-Conscious 6G Beamforming Optimization and Performance
Summary
This paper presents a comprehensive study on optimizing beamforming for 6G-IoT networks through machine learning (ML) techniques. The authors explore both supervised and unsupervised approaches to assess the predictive power of various feature groups, including network, environmental, device, and vision factors. The study highlights the importance of imbalance-aware experiments, revealing that network features outperform other groups in terms of predictive accuracy, as measured by recall, F1-score, and ROC-AUC metrics. Additionally, unsupervised methods such as K-means, DBSCAN, and hierarchical clustering are employed to analyze the influence of deployment environments and device types on clustering outcomes. The findings indicate that bandwidth, IoT sensors, and mobility are critical factors affecting beamforming success. The authors propose future work involving deep and reinforcement learning techniques to further enhance performance metrics like throughput and latency. Overall, this research contributes to the understanding of feature importance in 6G beamforming optimization and sets the stage for more adaptive and intelligent communication systems.
Methodology
The methodology follows the ML development life cycle, including data collection, preprocessing, feature engineering, dataset splitting, modeling with various classifiers (e.g., Logistic Regression, XGBoost, Random Forest), and evaluation of results. The study employs both supervised and unsupervised ML techniques to analyze feature importance and clustering.
Results
The results demonstrate that network features provide superior predictive power for beamforming optimization, with significant performance metrics achieved. Unsupervised clustering analysis indicates that environmental and device characteristics are more influential than mobility attributes. The explainability analysis identifies bandwidth, IoT sensors, and mobility as key factors in beamforming success.
Implications
The findings suggest that prioritizing network features in 6G beamforming optimization can lead to more efficient resource allocation and improved communication performance. The study also lays the groundwork for future research into adaptive ML techniques that can further enhance the capabilities of 6G networks.
The Objective Is the Bottleneck: Latent World Models Encode What Their Planners Cannot Use
Reinforcement Learning
Optimization
Robotics
- The predictor's capacity is not the limiting factor in long-horizon planning.
- The planning objective can saturate and lead to suboptimal performance.
- Long-horizon success is inversely correlated with one-step prediction accuracy.
- A simple change in the planning objective can significantly enhance performance.
Read more
The Objective Is the Bottleneck: Latent World Models Encode What Their Planners Cannot Use
Summary
This paper investigates the limitations of latent world models in planning tasks, specifically focusing on the LeWorldModel applied to the TwoRoom environment. The author argues that the bottleneck in planning is not the predictive capacity of the model but rather the planning objective itself. Through experiments, it is shown that the predictor maintains informative imagination even at long horizons, while the planner's effectiveness diminishes due to its objective function, which can saturate and even invert. The study reveals that the quality of long-horizon planning is inversely related to one-step prediction accuracy, indicating a dissociation between these metrics. A significant finding is that simply replacing the planning objective can drastically improve goal achievement without retraining the model. The paper concludes that reachability, rather than proximity, should be the focus of the planning objective, and emphasizes the need for learned costs to be trained on the distribution relevant to the planner's performance.
Methodology
The study employs a reproduction of the LeWorldModel on the TwoRoom environment, utilizing an 18.03M-parameter predictive architecture. The planning is conducted using cross-entropy method (CEM) search over learned dynamics, with evaluations based on squared Euclidean distance between imagined and goal embeddings. The experiments are run on a standard CPU without retraining the model.
Results
The experiments demonstrate that the planner's performance improves dramatically when the objective is modified, achieving 98.0% success at an offset of 100, compared to 26.0% with the original objective. The predictor's accuracy remains high even at long horizons, with errors only slightly increasing over time. The study also finds that learned costs must be aligned with the planner's scoring distribution to be effective.
Implications
The findings suggest that in designing planning algorithms, focusing on the right objectives is crucial for improving performance. This has implications for various applications in robotics and reinforcement learning, where effective planning is essential for task completion.
Distillation of Foundation Models for Time-dependent PDEs
Efficient ML
Theory
Time Series
- Introduction of TREX, a novel knowledge distillation framework for PDEs.
- Demonstration of effective transfer of knowledge from large foundation models to compact student models.
- Significant reduction in model parameters and inference time while maintaining or exceeding accuracy.
- First systematic study of distilling general pretrained PDE foundation models into smaller models.
Read more
Distillation of Foundation Models for Time-dependent PDEs
Summary
This paper addresses the challenge of using large foundation models for time-dependent partial differential equations (PDEs) in real-time applications due to their computational intensity. The authors introduce a novel knowledge distillation framework called Teacher Rollout Extension (TREX), which enables the transfer of predictive capabilities from a large, pretrained teacher model to a smaller, more efficient student model. TREX generates long synthetic trajectories from the teacher model, enhancing the training dataset without requiring explicit knowledge of initial-condition distributions. This approach allows the student model to learn from the teacher's rollout distribution, improving its performance in low-data regimes. The authors demonstrate that the distilled student models can match or exceed the accuracy of the teacher while significantly reducing the number of parameters and achieving faster inference times. The study is validated through experiments on multiple PDE benchmarks, showcasing the effectiveness of TREX in creating compact surrogate models for fluid dynamics tasks.
Methodology
The methodology involves fine-tuning a pretrained foundation model on a small set of trajectories from the target domain. TREX then generates long synthetic trajectories through teacher rollouts, which are used to augment the training data for a smaller student model. This process includes periodic noise injection to broaden the state-space coverage and enhance the student's learning. The student model is designed to incorporate task-specific inductive biases, such as equivariance, to improve its efficiency and effectiveness.
Results
The experiments conducted on PDE benchmarks, including fluid dynamics tasks, show that the student models trained using TREX can achieve comparable or superior accuracy to the teacher model while drastically reducing the number of parameters and achieving more than an order-of-magnitude speedup in inference time.
Implications
The findings suggest that TREX can facilitate the deployment of machine learning models in real-time applications that require fast inference, such as control tasks in manufacturing and energy systems. The ability to distill large foundation models into efficient surrogates opens new avenues for applying neural networks to complex physical systems described by PDEs.
Weightless Fine-Tuning: Personalizing LLMs via Logit-Space Transport
NLP
Large Language Models
Efficient ML
- Weightless Fine-Tuning (WFT) is a training-free method that approximates the effects of supervised fine-tuning (SFT) without modifying model weights.
- WFT uses a cross-prefix transport operator to compute supervised residuals and transport them to current prompts, effectively capturing the distributional effects of SFT.
- Experimental results show that WFT outperforms other lightweight personalization methods and achieves competitive performance compared to SFT on various tasks.
- WFT requires significantly less computational resources, achieving comparable results to SFT with less than 7% of the effective computation.
Read more
Weightless Fine-Tuning: Personalizing LLMs via Logit-Space Transport
Summary
This paper introduces Weightless Fine-Tuning (WFT), a novel approach for personalizing large language models (LLMs) without the need for supervised fine-tuning (SFT). Traditional SFT is computationally expensive and impractical for scenarios involving numerous authors, as it requires separate weight access, optimization, and storage. WFT circumvents these issues by operating entirely in logit space, utilizing a cross-prefix transport operator to approximate the effects of SFT. This operator is derived from dropout-induced cross-covariance and captures how perturbations in one context influence predictions in another. The authors evaluate WFT on three personalization tasks from the LaMP benchmark, demonstrating that it achieves superior average performance across datasets, competes effectively with SFT on individual tasks, and outperforms other lightweight personalization methods. Notably, WFT achieves results comparable to SFT while utilizing less than 7% of the computational resources, indicating its efficiency and practicality for real-world applications.
Methodology
WFT operates by computing supervised residuals on an author's training sequence and transporting these effects to the current prompt using a cross-prefix transport operator, which is estimated from dropout-induced cross-covariance. This approach allows for logit-space corrections without requiring weight updates.
Results
WFT demonstrated the best average performance across three LaMP personalization benchmarks, matching or exceeding SFT on individual tasks and outperforming other lightweight baselines. It achieved comparable performance to SFT while using less than 7% of the effective computation.
Implications
The findings suggest that WFT can significantly reduce the computational burden of personalizing LLMs, making it feasible to adapt models for numerous authors without the need for extensive retraining. This has potential applications in various domains where personalized content generation is essential, such as writing assistants, customer service bots, and content creation tools.
Vero: Can AI Agents Build Formally Verified Software Repositories?
Theory
- Vero is the first benchmark for evaluating joint implementation and proof synthesis at the repository level.
- The benchmark includes 43 multi-module instances from diverse domains and programming languages.
- An audit mechanism is implemented to identify and correct errors in specifications and reference implementations.
- Current AI agents struggle with complex specifications, solving only 27 out of 43 instances in the best case.
Read more
Vero: Can AI Agents Build Formally Verified Software Repositories?
Summary
The paper introduces Vero, a novel benchmark designed to evaluate AI agents' capabilities in generating formally verified software repositories. Unlike existing benchmarks that focus on individual functions or proof generation with fixed implementations, Vero assesses the joint implementation and proof synthesis at the repository level. It comprises 43 multi-module instances sourced from real-world repositories across various programming languages, including Python, Dafny, Verus, and Coq. Each instance features a Lean 4 repository with curated formal specifications and reference implementations. The benchmark supports both proof-only and code-and-proof evaluation modes, allowing for a comprehensive assessment of AI agents' performance. A unique audit mechanism is included to identify and correct errors in specifications or implementations, enhancing the reliability of the benchmark. The evaluation of frontier coding-agent configurations revealed that the best-performing agent solved only 27 out of 43 instances, indicating significant challenges remain, particularly with specifications involving cross-module invariants and complex mathematical theories. The findings highlight the need for agents to develop reusable lemma libraries and coordinate implementation and proof generation effectively. Vero serves as a concrete testbed for advancing repository-scale verified software synthesis, providing insights into the current limitations and future directions for AI in formal verification.
Methodology
The authors developed Vero through a multi-stage curation process, creating benchmark instances that include formal specifications and reference implementations. They evaluated various AI agent configurations using the Lean toolchain, focusing on both code-and-proof and proof-only modes. An audit mechanism was integrated to address potential errors in the benchmark.
Results
The evaluation showed that the strongest AI agent configuration fully solved only 27 out of 43 instances, with 10 instances remaining unsolved across all configurations. The unsolved instances were primarily those involving complex specifications and cross-module dependencies, indicating significant challenges in current AI capabilities for formal verification.
Implications
Vero provides a foundational framework for assessing and improving AI agents' abilities in generating formally verified software. It highlights the importance of repository-level verification and the need for agents to develop sophisticated reasoning capabilities to handle complex software systems.
Federated Compositional Muon Optimizer for Matrix-Wise Models
Federated Learning
Optimization
Theory
- Introduction of FedCoMuon and FedCoMuon-VR optimizers for distributed matrix-wise compositional optimization.
- Theoretical convergence analysis under non-convex and non-i.i.d. settings, with FedCoMuon-VR showing lower sample complexity.
- Empirical validation demonstrating competitive performance in robust federated learning and task-distributed meta learning.
Read more
Federated Compositional Muon Optimizer for Matrix-Wise Models
Summary
This paper introduces the Federated Compositional Muon (FedCoMuon) optimizer, designed to address the challenges of hierarchical structured problems in federated learning, particularly for matrix-wise models. The authors highlight that existing Muon-based optimizers are not well-suited for compositional optimization tasks that involve nested objectives. The FedCoMuon optimizer integrates compositional gradient tracking with orthogonalized momentum, while a variance-reduced variant (FedCoMuon-VR) is also proposed. Theoretical analysis demonstrates that FedCoMuon-VR achieves a lower sample complexity of O(ϵ−3) for finding an ϵ-stationary solution compared to existing Federated Muon algorithms. The paper includes extensive numerical experiments that validate the effectiveness of the proposed methods in robust federated learning and task-distributed risk-sensitive meta learning, showing competitive performance and improved accuracy over existing baselines.
Methodology
The FedCoMuon optimizer employs compositional gradient tracking combined with orthogonalized momentum for efficient optimization of matrix-wise models. The FedCoMuon-VR variant incorporates a momentum-based variance reduction technique to enhance performance. The algorithms are analyzed theoretically for convergence properties in non-convex and non-i.i.d. scenarios.
Results
The proposed FedCoMuon and FedCoMuon-VR optimizers were shown to outperform existing compositional optimization methods in terms of accuracy and efficiency in various experimental settings, particularly in robust federated learning and task-distributed meta learning tasks.
Implications
The development of FedCoMuon and FedCoMuon-VR optimizers has significant implications for federated learning applications, especially in privacy-sensitive domains like healthcare and mobile intelligence, where hierarchical structured problems are common. The ability to effectively optimize matrix-wise models can lead to improved performance in real-world applications.
When Local Variance Optimality Is Not Enough: RoPE-Aligned Q/K Rotations for Dynamic 4-Bit Quantisation
NLP
Large Language Models
Optimization
- The only commuting orthogonal maps for distinct RoPE frequencies are independent pairwise rotations.
- The derived rotation angle minimizes channel variance but does not improve quantization accuracy in practice.
- The head-shared pairwise configuration results in higher perplexity compared to full-head mixing across multiple checkpoints.
- Different statistics used by the analytic surrogate and the dynamic quantizer contribute to the observed discrepancies in performance.
Read more
When Local Variance Optimality Is Not Enough: RoPE-Aligned Q/K Rotations for Dynamic 4-Bit Quantisation
Summary
This paper investigates the effectiveness of rotation-based post-training quantization methods in the context of transformer models, specifically focusing on the RoPE (Rotary Position Embedding) mechanism. The authors explore whether applying orthogonal transformations that respect the RoPE frequency pair decomposition can enhance quantization accuracy compared to traditional full-head mixing. They establish that the only commuting orthogonal maps for distinct RoPE frequencies are independent pairwise rotations. The authors derive a rotation angle that minimizes channel variance under a pooled-covariance surrogate and verify that their implementation achieves this minimum. However, their experiments reveal that this head-shared pairwise configuration does not improve accuracy in a dynamic W4A4KV4 setting, leading to increased perplexity across various checkpoints. The findings suggest that optimality for a structured surrogate does not necessarily translate to reduced quantization error when the surrogate and mixing support are misaligned with the quantizer's statistics.
Methodology
The authors derive a closed-form rotation angle that minimizes the larger channel variance using a pooled-covariance, position-averaged surrogate. They conduct experiments on various checkpoints, comparing the performance of head-shared pairwise configurations against full-head Hadamard mixing under dynamic quantization settings.
Results
The implementation of the head-shared pairwise configuration did not yield improved accuracy, as indicated by increased perplexity across all evaluated contexts. The findings showed that estimating the shared angle from K alone improved performance but still did not surpass the full-head mixing results. Additionally, broader mixing support was associated with lower quantization error and smaller degradation in performance.
Implications
The results highlight the importance of aligning the surrogate and mixing support with the quantizer's scale-setting statistics in low-bit quantization methods. This study suggests that future work should consider the structural constraints imposed by mechanisms like RoPE when designing quantization strategies for transformer models.
Transferable Above-Ground Biomass (AGB) Estimation Model from Multi-Sensor Data with Sparse Field Calibration
Computer Vision
Multimodal
Optimization
- Introduces a global CNN model for AGB estimation using multi-sensor data.
- Employs a lightweight calibration process to adapt the model to local conditions.
- Achieves competitive accuracy compared to existing biomass estimation methods.
- Demonstrates the effectiveness of combining optical and radar data for biomass mapping.
Read more
Transferable Above-Ground Biomass (AGB) Estimation Model from Multi-Sensor Data with Sparse Field Calibration
Summary
This paper addresses the challenge of accurately estimating above-ground biomass (AGB) in forests using a novel framework that integrates multi-sensor data and a global convolutional neural network (CNN). The authors highlight the limitations of traditional field inventories, which are spatially sparse, and the inadequacies of spaceborne LiDAR data, which often underestimates biomass in high-biomass forests. The proposed model utilizes optical data from Sentinel-2, C-band and L-band synthetic aperture radar (SAR) data, and terrain data to create a globally trained CNN that can be adapted to new landscapes through a lightweight empirical calibration process. By training the model on seasonal composites of biomass reference data from GEDI, the authors ensure that it captures the full range of canopy conditions. The model is calibrated using a small number of local field plots to correct scale and bias, enhancing its accuracy without the need for extensive retraining. The results demonstrate that the global model achieves competitive performance, with an R² of approximately 0.78 and RMSE of about 22 Mg/ha. After local calibration, performance improves to R² ≈ 0.82 and RMSE ≈ 15 Mg/ha, outperforming both the uncalibrated model and existing biomass products. This approach offers a scalable solution for continuous biomass mapping, crucial for carbon accounting and environmental management.
Methodology
The methodology involves training a global CNN on multi-sensor data, including optical and SAR data, using GEDI biomass reference data. The model is adapted to local landscapes through a field calibration process that applies a small number of local plots for scale-and-bias correction, rather than extensive retraining.
Results
The global model achieved an R² of approximately 0.78 and RMSE of about 22 Mg/ha on held-out validation data. After local calibration, the performance improved to R² ≈ 0.82 and RMSE ≈ 15 Mg/ha, surpassing both the uncalibrated global model and the ESA CCI Biomass product.
Implications
This framework has significant implications for forest carbon accounting and management, enabling more accurate and continuous monitoring of biomass across diverse landscapes. It supports environmental policies and initiatives aimed at reducing emissions from deforestation and forest degradation.
Scaling Automatic Research Agents via World Models
Reinforcement Learning
Large Language Models
Efficient ML
- Introduction of World Model RL (WMRL) to replace expensive environment execution in AutoResearch agents.
- Development of Online Debiasing and Inverse-Variance Denoising mechanisms to improve training efficiency and performance.
- Theoretical grounding of the framework with proven improvements in convergence guarantees.
- Empirical validation showing 3-4x acceleration in training and superior performance compared to larger models.
Read more
Scaling Automatic Research Agents via World Models
Summary
This paper addresses the challenges of scaling Automatic Research (AutoResearch) agents, which utilize large language models (LLMs) to conduct empirical research autonomously. The authors identify a critical bottleneck in the reinforcement learning (RL) training process, where the environment execution component of the training trajectory incurs significantly higher costs compared to agent generation. To alleviate this issue, they propose a novel approach called World Model RL (WMRL), which substitutes real environment execution with a simulated world model. This model allows for faster and more scalable training by enabling multiple trajectories to be processed concurrently. Additionally, the authors introduce two mechanisms—Online Debiasing and Inverse-Variance Denoising—to mitigate the biases and noise inherent in the world model's predictions. Theoretical proofs demonstrate that these mechanisms improve convergence guarantees. Empirical results show that WMRL accelerates training by 3-4 times across various tasks and scales, while outperforming larger baseline models. The method also shows promise in transferring to post-training embodied VLA policies, indicating its broader applicability beyond AutoResearch.
Methodology
The authors propose World Model RL (WMRL), which utilizes a simulated world model to replace real environment execution in the training of AutoResearch agents. They implement Online Debiasing and Inverse-Variance Denoising to address biases and noise in the world model's outputs. Theoretical analysis is conducted to establish convergence guarantees, and empirical experiments are performed across various tasks to validate the approach.
Results
WMRL accelerates training by 3-4 times compared to traditional RL methods and achieves performance that exceeds that of larger open-weight models (48B and 120B parameters) using smaller models (4B and 9B parameters). The proposed mechanisms for debiasing and denoising also contribute to improved convergence and performance.
Implications
The findings suggest that WMRL can significantly enhance the efficiency of training AutoResearch agents, making it feasible to scale these agents for more complex empirical research tasks. The generalizability of the method to other domains indicates potential applications in various fields where autonomous decision-making and research are required.
Where You Measure Decides What You Measure: Position Selection in Ablation-Based SAE Evaluation
NLP
Large Language Models
Interpretability
- Measurement position is determined by the dictionary, leading to comparisons at different tokens.
- Fixing the measurement position significantly reduces variance attributed to dictionary differences.
- Additional reporting choices can alter the interpretation of results.
- A new protocol for reporting ablation-based causal numbers is proposed for better comparability.
Read more
Where You Measure Decides What You Measure: Position Selection in Ablation-Based SAE Evaluation
Summary
This paper investigates the impact of measurement position in the evaluation of sparse autoencoders (SAEs) used in language models. The conventional method of measuring the effect of a latent variable by switching it off and observing changes is critiqued, as the position where this measurement occurs is dictated by the dictionary being evaluated. The author demonstrates that different dictionaries, even when encoding similar latents, often select different tokens for measurement, leading to inconsistent evaluations. Through a controlled experiment involving six autoencoders trained from a common initialization, the paper shows that much of the variance attributed to differences in dictionaries can be eliminated by fixing the measurement position. The study also highlights that additional reporting choices, such as how special tokens are handled and the normalization of intervention magnitudes, can significantly affect results. A new evaluation protocol is proposed to ensure comparability across studies, and an audit of five published papers reveals inconsistencies in reporting these conventions. The findings emphasize the need for transparency in the measurement process to improve the reliability of SAE evaluations.
Methodology
The study employs a controlled experimental design with six sparse autoencoders trained from a single initialization to isolate the effects of measurement position. The author compares the measurement positions selected by different dictionaries and assesses the variance in results when measurements are taken at fixed positions versus conventional top-activating positions.
Results
The results indicate that when measurement positions are fixed, the latent×arm variance attributed to differences in dictionaries drops significantly, from 7.6% and 11.9% to near zero. The findings also show that as corpus size increases, the disagreement about measurement positions between dictionaries worsens, rather than improving.
Implications
The findings suggest that researchers should be cautious in interpreting causal numbers from SAE evaluations without clear reporting of measurement positions. The proposed protocol aims to enhance the reliability and comparability of future research in this area, potentially leading to more consistent and interpretable results in the evaluation of language models.
Large-scale Testing Global Optimization Methods with Black-box Adversarial Attacks
Optimization
- Existing global optimization benchmarks are outdated and limited in scope.
- Black-box adversarial attacks present a novel and relevant benchmark for optimization methods.
- The study evaluates the performance of various evolutionary algorithms and metaheuristics on BBAA problems.
- The findings suggest a need for modern benchmarks that reflect real-world challenges in machine learning.
Read more
Large-scale Testing Global Optimization Methods with Black-box Adversarial Attacks
Summary
This paper addresses the limitations of existing global optimization benchmark suites, which are often small and based on outdated analytical functions. The authors propose that black-box adversarial attack (BBAA) problems can serve as a valuable benchmark for global optimization in high-dimensional spaces. They investigate the effectiveness of various evolutionary algorithms and metaheuristics in solving BBAA problems, thereby aligning global optimization methods with contemporary challenges in machine learning. The paper highlights the evolution of benchmarking methods and the significance of adversarial attacks in evaluating the robustness of machine learning models. By demonstrating the application of optimization techniques to BBAA, the authors contribute to the development of more relevant and challenging benchmarks for global optimization research.
Methodology
The authors conducted experiments using several evolutionary algorithms and metaheuristics to solve black-box adversarial attack problems. They compared the performance of these optimization methods on various BBAA scenarios, assessing their effectiveness in generating perturbations that mislead machine learning models.
Results
The results demonstrated that certain evolutionary algorithms and metaheuristics were effective in solving BBAA problems, indicating their potential as robust optimization techniques in high-dimensional spaces. The study also highlighted the inadequacies of traditional benchmarks in capturing the complexities of modern optimization challenges.
Implications
The findings suggest that the integration of BBAA problems into global optimization benchmarks can enhance the relevance and applicability of optimization methods in machine learning. This could lead to improved robustness of machine learning models against adversarial attacks and foster the development of more sophisticated optimization techniques.
Into the ORBIT for Time Series: Training Regimes for Foundation Models
Time Series
- Introduction of ORBIT, a training paradigm for TSFMs that controls effective pre-training distribution.
- Bootstrap Multi-Level Sampling and Omni-Range Incremental Training are key components of ORBIT.
- Falcon-2.0, a simple encoder-only Transformer, is trained under ORBIT to enhance forecasting capabilities.
- Rank-Guided Cross-Depth Alignment improves representation alignment across Transformer depths.
Read more
Into the ORBIT for Time Series: Training Regimes for Foundation Models
Summary
This paper introduces ORBIT (Omni-Range Bootstrap Incremental Training), a novel training paradigm for Time Series Foundation Models (TSFMs) aimed at improving the effective pre-training distribution of heterogeneous time series data. The authors highlight that while architectural innovations have enhanced TSFMs, the training regimes governing data exposure remain under-explored. ORBIT consists of two main components: Bootstrap Multi-Level Sampling, which hierarchically controls data exposure by constructing sample indices through stochastic selection of time series records, target variables, context windows, and prediction horizons; and Omni-Range Incremental Training, which allows for the consumption of variable-length examples throughout a single training run, enabling diverse context lengths and prediction horizons to coexist. The paper details the training of Falcon-2.0, a simple univariate encoder-only Transformer model, and introduces Rank-Guided Cross-Depth Alignment, a training-only objective that aligns representations across different depths without adding inference costs. Evaluations on GIFT-Eval and fev-bench demonstrate that Falcon-2.0 achieves strong zero-shot forecasting performance across various domains and frequencies, confirming the importance of the proposed training regime.
Methodology
The methodology involves two main components: Bootstrap Multi-Level Sampling for hierarchical data exposure control and Omni-Range Incremental Training for consuming variable-length examples. This approach allows for stochastic selection of training samples while maintaining reproducibility and efficient access, enabling the model to learn from diverse contexts and prediction horizons simultaneously.
Results
Falcon-2.0, trained under the ORBIT paradigm, achieved strong zero-shot forecasting performance on benchmark datasets GIFT-Eval and fev-bench. Ablation studies confirmed the significance of stochastic sample construction and the simultaneous exposure to diverse contexts and prediction horizons in enhancing model performance.
Implications
The findings suggest that effective training regimes are crucial for the performance of TSFMs, particularly in handling heterogeneous time series data. The ORBIT paradigm could lead to advancements in forecasting applications across various domains, improving the generalizability and robustness of time series models.
Learning Discrete Decisions for MIPs with Constraint-Aware Diffusion
Optimization
Generative Models
Graph Learning
- Introduction of Constrained Graph Diffusion (CGD) for MIPs, which enforces feasibility in the discrete decision-making process.
- The methodology separates the discrete and continuous components of MIPs, allowing for efficient resolution of the continuous problem post-discrete decision generation.
- Demonstrated effectiveness on two distinct optimization problems, showcasing substantial improvements in solution quality and feasibility.
- Achieved significant computational speedups compared to traditional numerical solvers, enhancing the practicality of solving complex MIPs.
Read more
Learning Discrete Decisions for MIPs with Constraint-Aware Diffusion
Summary
This paper introduces a novel learning-based approach called Constrained Graph Diffusion (CGD) for solving mixed-integer optimization problems (MIPs). MIPs are challenging due to the need to simultaneously determine discrete and continuous decisions while adhering to complex combinatorial constraints. The CGD framework utilizes a graph-based generative diffusion model that learns the discrete components of MIPs and incorporates a training-free feasibility projection operator into the reverse diffusion process. This integration ensures that intermediate samples remain within the feasible set during generation. Once the discrete decisions are established, the remaining optimization task simplifies to a continuous problem, which can be efficiently solved using existing numerical methods. The authors evaluate CGD on two applications: optimal transmission switching for AC optimal power flow and discrete portfolio optimization. The results demonstrate significant improvements in feasibility and solution quality compared to learning-based baselines, achieving speedups of up to 425 times over state-of-the-art numerical solvers for mixed-integer nonlinear programs (MINLPs).
Methodology
The proposed CGD framework employs a graph-based generative diffusion model that learns the distribution of discrete decisions for MIPs. It integrates a feasibility projection operator into the reverse diffusion process to ensure that generated samples are feasible. After generating the discrete decisions, the continuous variables are optimized using standard numerical methods, effectively decoupling the combinatorial and continuous aspects of the problem.
Results
The CGD framework was evaluated on optimal transmission switching for AC optimal power flow and discrete portfolio optimization. The results indicated that CGD not only generated feasible discrete decisions but also improved solution quality relative to existing learning-based approaches. Additionally, it achieved speedups of up to 425 times compared to state-of-the-art numerical solvers for MINLPs.
Implications
The CGD framework has the potential to significantly enhance the efficiency of solving mixed-integer optimization problems across various domains, including power systems, finance, and logistics. Its ability to enforce feasibility during the decision-making process could lead to more reliable and faster solutions in practical applications.
ScreenShot: A Foundation Model for Few-Shot Combination Drug Screening
Efficient ML
- ScreenShot is a foundation model that enables few-shot predictions for combination drug screening without the need for molecular profiling.
- The model is pretrained on a large dataset, allowing it to generalize effectively to new patient samples.
- An innovative active learning strategy is developed to optimize experimental design, reducing costs while maintaining high accuracy.
- ScreenShot outperforms traditional methods and baselines in prediction accuracy and treatment identification.
Read more
ScreenShot: A Foundation Model for Few-Shot Combination Drug Screening
Summary
The paper introduces ScreenShot, a hierarchical transformer model designed for few-shot prediction in combination drug screening. Traditional drug screening methods are limited by the need for extensive molecular profiling and per-cohort training, which can be impractical in clinical settings with limited samples. ScreenShot addresses this by being pretrained on 40 drug screening datasets, encompassing 3,700 drugs and 6,000 biological samples. It utilizes in-context learning to predict drug response based solely on a few observations from a new patient, without requiring fine-tuning or molecular profiling. The model's architecture reflects the nested structure of screening data, allowing it to operate directly on functional measurements. The authors also propose an active learning strategy that leverages ScreenShot's internal representations to optimize experimental design, achieving significant cost savings while maintaining high accuracy in hit detection. The empirical results demonstrate that ScreenShot outperforms existing methods in both prediction accuracy and treatment identification across multiple datasets.
Methodology
ScreenShot employs a hierarchical transformer architecture and utilizes in-context learning to predict drug response based on a small set of drug-dose-viability measurements from new patient samples. It avoids traditional fine-tuning by conditioning directly on observed data through cross-attention mechanisms. Additionally, an active learning strategy is implemented using k-means++ seeding in the model's embedding space to prioritize experiments based on predicted effectiveness.
Results
ScreenShot demonstrated state-of-the-art performance in few-shot prediction and hit detection across four held-out datasets, outperforming baseline models such as XGBoost and TabPFN. The active learning strategy achieved the same hit detection as uniform screening while using only one-third of the experimental budget.
Implications
The development of ScreenShot has significant implications for personalized medicine, particularly in oncology, where rapid and effective identification of drug combinations can improve patient outcomes. Its ability to operate without extensive molecular profiling makes it suitable for clinical settings with limited tissue availability.
Reoptimization Algorithms for Contextual Bandits with Knapsack Constraints
Optimization
Theory
Efficient ML
- Introduces a new algorithm for Contextual Bandits with Knapsack constraints.
- Achieves a significant improvement in average regret bounds compared to existing literature.
- Utilizes a UCB-guided linear programming re-solving heuristic.
- Focuses on practical applications in digital platforms with known resource consumption.
Read more
Reoptimization Algorithms for Contextual Bandits with Knapsack Constraints
Summary
This paper presents new algorithms for the Contextual Bandits with Knapsack (CBwK) framework, addressing the challenge of assigning products to customers under resource constraints. The authors propose a novel extension of the Upper-Confidence-Bound (UCB) algorithms, incorporating reoptimization techniques to improve decision-making in environments where resource consumption is known but rewards are uncertain. The objective is to minimize expected revenue loss compared to an optimal policy that has full knowledge of the reward function. The proposed algorithm achieves an average regret of O(ln³(T)/T), which is a significant improvement over existing bounds in related literature. This work bridges the gap between bandit learning and network revenue management, providing a computationally efficient solution tailored for high-frequency resource allocation scenarios.
Methodology
The authors develop a UCB-guided linear programming re-solving heuristic that periodically updates a fluid LP based on upper confidence bounds of unknown reward parameters. This method captures the opportunity cost of capacity through time-varying dual shadow prices, allowing for efficient online decision-making.
Results
The proposed algorithm achieves an average regret of O(ln³(T)/T), significantly improving upon the O(1/√T) and O(√ln(T)/T) bounds found in existing CBwK and dynamic-pricing literature. This demonstrates the effectiveness of the reoptimization approach in minimizing revenue loss.
Implications
The findings have significant implications for digital platforms that require real-time resource allocation under uncertainty, such as ride-hailing services and online advertising. The algorithm can enhance decision-making efficiency and revenue management in these contexts.
Diagnosing JEPA World Models with Action-Conditioned Predictive Consistency
Reinforcement Learning
Robotics
Theory
- Introduction of Action-Conditioned Predictive Consistency (ACPC) for diagnosing world models.
- Establishment of Invariance Radius (IR) and Separation Rate (SR) as metrics for evaluating model robustness against visual perturbations.
- Proven bounds on the impact of visual perturbations on multi-step prediction errors and planning costs.
- Empirical validation across multiple control tasks and architectures, showing consistent diagnostic trends.
Read more
Diagnosing JEPA World Models with Action-Conditioned Predictive Consistency
Summary
This paper introduces Action-Conditioned Predictive Consistency (ACPC), a novel diagnostic tool for evaluating joint-embedding predictive architectures (JEPAs) in world models. JEPAs aim to predict outcomes in a compact latent space, minimizing the influence of irrelevant visual details. However, visual perturbations can still affect the encoded representations and subsequent predictions. ACPC addresses this by measuring the divergence between predicted trajectories of a clean history and its visually perturbed counterpart when rolled forward under the same action sequence. The authors establish that this divergence bounds the changes in multi-step prediction error and planner cost due to perturbations. They further define two complementary metrics: the Invariance Radius (IR), which quantifies the spread of clean and perturbed rollouts, and the Separation Rate (SR), which assesses the distinguishability of different states post-rollout. Through experiments on four visual control tasks, the authors demonstrate that ACPC effectively predicts changes in prediction errors and costs induced by visual perturbations. The findings indicate that lower IR and higher SR correlate with better performance under perturbations, providing a robust framework for diagnosing and improving world models.
Methodology
The authors developed ACPC to compare the predicted trajectories of a clean history and its perturbed view under the same action sequence. They derived theoretical bounds on the effects of visual perturbations on prediction errors and costs. The methodology involved defining IR and SR metrics to summarize the sensitivity and distinguishability of states after perturbations, followed by experimental validation on various control tasks.
Results
The experiments demonstrated that ACPC effectively predicts the changes in multi-step prediction errors and planning costs due to visual perturbations. The IR and SR metrics provided insights into model robustness, with lower IR and higher SR indicating better performance across different tasks and perturbation types. The results were consistent across two different world-model architectures, reinforcing the diagnostic utility of ACPC.
Implications
The findings suggest that ACPC can serve as a valuable diagnostic tool for improving the robustness of world models in reinforcement learning and control tasks. By identifying models that can maintain performance under visual perturbations, researchers and practitioners can enhance the reliability of AI systems in real-world applications where visual noise is prevalent.
Sustaining Plasticity via Learnable Wavelet Activations in Continual Learning
Theory
Optimization
- Introduces a learnable wavelet activation function to address plasticity loss in continual learning.
- Combines low-frequency and high-frequency components to counteract spectral bias.
- Employs dynamic wavelet injection and targeted regularization to enhance plasticity and stability.
- Provides theoretical guarantees for the framework's effectiveness in L2 approximation.
Read more
Sustaining Plasticity via Learnable Wavelet Activations in Continual Learning
Summary
This paper addresses the challenge of plasticity loss in continual learning, where models struggle to adapt to new tasks while retaining previously learned knowledge. The authors propose a novel learnable wavelet activation function that decomposes activation into low-frequency and high-frequency components, countering the spectral bias of traditional activation functions. The framework includes a dynamic wavelet injection mechanism to enhance plasticity for new tasks and a regularization strategy to stabilize previously learned knowledge. Theoretical guarantees are provided for the hybrid wavelet architecture, demonstrating its efficiency in L2 approximation and the effectiveness of a decoupled learning rate mechanism in restoring network plasticity. Extensive empirical evaluations show that the proposed method outperforms state-of-the-art approaches across various continual learning benchmarks, maintaining superior trainability and generalization throughout the learning process.
Methodology
The authors developed a wavelet-based activation function that integrates a global base approximation with local wavelet details. This design allows for adaptive learning of high-frequency components while maintaining stability through a regularization strategy that constrains activation slopes. The paper includes rigorous mathematical proofs supporting the proposed framework and its mechanisms.
Results
The proposed method demonstrated superior performance in maintaining plasticity and mitigating catastrophic forgetting compared to existing state-of-the-art methods. Empirical evaluations across multiple continual learning benchmarks confirmed its effectiveness in enhancing model trainability and generalization.
Implications
The findings suggest that incorporating learnable wavelet activations can significantly improve the performance of neural networks in continual learning scenarios, potentially leading to more robust models capable of handling dynamic data streams without losing previously acquired knowledge.
Momentum as Residual-Driven Multiplier Correction for Deep Learning Optimization
Optimization
- Introduction of the AIM framework, which provides a unified interpretation of momentum-based optimization.
- Separation of update geometry and acceleration mechanisms in optimizers, enhancing understanding of their interactions.
- Development of RADAR, a new optimizer that combines multiple advanced techniques for improved performance.
- Establishment of stochastic convergence for RADAR through rigorous analysis.
Read more
Momentum as Residual-Driven Multiplier Correction for Deep Learning Optimization
Summary
This paper introduces an innovative framework called ADMM-Inspired Momentum (AIM) that interprets momentum in deep learning optimization as a multiplier-like correction driven by a residual from variable splitting. The AIM framework separates the roles of update geometry and acceleration in momentum-based optimizers, allowing for a clearer understanding of how these components interact. The authors propose a new optimizer, Relativistic Adaptive gradient Descent with Accelerated Residual (RADAR), which integrates relativistic adaptive geometry, decoupled residual correction, and second-order momentum filtering. This design aims to enhance update direction and momentum estimation. The paper establishes stochastic convergence for RADAR through a variance-perturbed Lyapunov drift analysis and demonstrates its effectiveness across various tasks, including supervised vision learning, language modeling, and reinforcement learning, showing consistent improvements over existing adaptive optimizers.
Methodology
The authors developed the AIM framework based on residual-penalty variable splitting, which recovers the exponential moving average of gradients from an ADMM-style multiplier update. They then proposed RADAR, which incorporates relativistic adaptive geometry, decoupled residual correction, and second-order momentum filtering. The convergence of RADAR was established using a variance-perturbed Lyapunov drift analysis.
Results
RADAR demonstrated consistent improvements over strong adaptive optimizer baselines in various tasks, including supervised vision learning, language modeling, and reinforcement learning. The stochastic convergence of RADAR was rigorously established, supporting its theoretical foundation.
Implications
The AIM framework and RADAR optimizer could lead to the development of more efficient and robust optimization algorithms in deep learning, potentially enhancing the performance of a wide range of applications in computer vision, natural language processing, and reinforcement learning.
Learning the Mathematical Property for Designing Low Mutual Coherence Binary Sensing Matrices
Theory
Optimization
Efficient ML
- Introduces a learning-based technique for constructing binary sensing matrices without large datasets.
- Focuses on optimizing the mutual incoherence property for improved signal recovery.
- Utilizes a neural network framework to generate sensing matrices, reducing computational costs.
- Demonstrates superior performance compared to conventional deterministic and random matrix methods.
Read more
Learning the Mathematical Property for Designing Low Mutual Coherence Binary Sensing Matrices
Summary
This paper presents a novel approach to constructing sensing matrices for compressive sensing, focusing on achieving low mutual coherence without relying on large datasets or specific applications. The authors propose a learning-based technique that utilizes mathematical properties to define the loss function for the neural network, which generates binary sensing matrices. The core of the method is the optimization of the mutual incoherence property, which enhances the stability and uniqueness of sparse signal reconstruction. The proposed framework significantly reduces computational costs and storage requirements while maintaining robustness and generality. The results demonstrate that this method outperforms traditional deterministic and random matrix techniques in terms of signal recovery performance, highlighting its effectiveness in practical applications.
Methodology
The authors employ a neural network to construct binary sensing matrices that optimize the mutual incoherence property. The framework does not require extensive training data and instead uses mathematical properties to define the loss function, facilitating the generation of low mutual coherence matrices.
Results
Simulation results indicate that the proposed method consistently outperforms existing conventional methods, including both deterministic and random matrix techniques, in terms of the accuracy and reliability of sparse signal reconstruction.
Implications
The findings suggest that the proposed approach can be effectively applied in various real-world scenarios requiring efficient signal acquisition and reconstruction, particularly in fields where compressive sensing is utilized, such as imaging and data compression.
Comment on 'Modeling rapid language learning by distilling Bayesian priors into artificial neural networks'
Theory
- M&G's methodology does not effectively distill a Bayesian prior into ANNs as claimed.
- The use of cross-entropy loss leads to overfitting and poor generalization in M&G's model.
- Early stopping may not serve as a valid regularization method in the context of M&G's approach.
- The critique highlights the importance of properly defining and implementing Bayesian priors in machine learning.
Read more
Comment on 'Modeling rapid language learning by distilling Bayesian priors into artificial neural networks'
Summary
This paper critiques the methodology proposed by McCoy & Griffiths (M&G) for distilling Bayesian priors into artificial neural networks (ANNs) using Model-Agnostic Meta-Learning (MAML). The authors argue that M&G's approach does not genuinely incorporate a Bayesian prior into the learning process but rather initializes network weights favorably without altering the objective function. They explore the implications of this interpretation and highlight that M&G's model suffers from overfitting and poor generalization to unseen data, contrasting it with genuine Bayesian learners. The critique emphasizes that M&G's reliance on a cross-entropy loss function is inadequate for formal language learning, leading to suboptimal performance as training progresses. The authors also discuss the role of early stopping in the training process and question whether it can be considered a form of prior, ultimately concluding that M&G's claims about their model's Bayesian nature are overstated.
Methodology
The authors analyze M&G's approach by examining the initialization of network weights through MAML and its implications for learning formal languages. They critique the use of cross-entropy loss and assess the model's performance through empirical testing, focusing on overfitting patterns and generalization capabilities.
Results
The critique reveals that M&G's model overfits and generalizes poorly compared to genuine Bayesian learners. The authors provide empirical evidence showing that as training progresses, test loss increases, indicating a deterioration in performance due to overfitting. They also argue that early stopping does not effectively induce a Bayesian prior as claimed by M&G.
Implications
This commentary underscores the necessity for clear definitions and implementations of Bayesian priors in machine learning models. It suggests that researchers should be cautious in claiming Bayesian properties for models that do not explicitly incorporate such priors in their objective functions. The findings may influence future research on language learning and the application of meta-learning techniques.
Federated Learning for Distributed CNC Tool Wear Prediction
Federated Learning
- Federated Learning enables collaborative model training without sharing raw data, addressing privacy concerns.
- The study utilizes the MATWI dataset, which contains diverse sensor data relevant for tool wear prediction.
- Federated models demonstrate performance close to centralized models and significantly better than local client models.
- The findings support the feasibility of federated learning in industrial environments with distributed data.
Read more
Federated Learning for Distributed CNC Tool Wear Prediction
Summary
This paper explores the application of Federated Learning (FL) for predicting tool wear in CNC machining environments, where data is often distributed across multiple machines and organizations. Traditional machine learning approaches face challenges due to data sharing restrictions and the distributed nature of operational data. The authors propose a federated learning framework that allows for collaborative model training without the need to share raw data. Using the MATWI dataset, which includes diverse sensor data and wear labels from various cutting tools, the study simulates a federated learning scenario with multiple clients. The performance of the federated models is compared against centralized models and local client baselines. The results indicate that federated learning achieves performance levels comparable to centralized learning while significantly outperforming local client models. This suggests that federated learning can effectively facilitate collaborative tool wear prediction in distributed CNC manufacturing settings, enhancing product quality and process reliability.
Methodology
The authors implemented two novel federated learning systems that include modules for data preprocessing, local learning algorithms, and a global model aggregation mechanism. The study involved simulating a federated learning environment using the MATWI dataset, which consists of sensor data and wear labels from cutting tools. The federated models were benchmarked against centralized and local client training approaches to evaluate their effectiveness.
Results
The results showed that federated learning models achieved performance metrics that were nearly equivalent to those of centralized models, while significantly outperforming local client models. This indicates that federated learning can effectively leverage distributed data for tool wear prediction without compromising data privacy.
Implications
The findings suggest that federated learning can be a viable solution for collaborative predictive maintenance in CNC machining, potentially leading to improved tool management, reduced downtime, and enhanced overall production efficiency. This approach could be extended to other industrial applications where data privacy and distribution are critical.
DARTree: Speculative Diffusion Decoding with Autoregressive Draft Trees
NLP
Large Language Models
Efficient ML
- DARTree introduces a training-free speculative decoding method that extends AR correction from chains to trees.
- The method constructs a fixed-width candidate tree, allowing for efficient parallel processing of token proposals.
- DARTree achieves significant speedup and higher acceptance rates compared to existing diffusion-based speculative decoding methods.
- The approach effectively decouples causal correction from sequential heap operations, addressing latency bottlenecks.
Read more
DARTree: Speculative Diffusion Decoding with Autoregressive Draft Trees
Summary
The paper presents DARTree, a novel speculative decoding method designed to enhance the efficiency of autoregressive (AR) language models by utilizing a tree structure for token generation. Traditional AR models face latency issues due to their sequential nature, generating one token at a time. Speculative decoding addresses this by proposing multiple future tokens simultaneously. However, existing methods still suffer from latency as they rely on sequential token generation. DARTree innovatively constructs a fixed-width candidate tree that expands and scores all nodes at each depth in a single batch, allowing for parallel processing. This method decouples the autoregressive head inference from sequential operations, significantly improving the speed and acceptance rate of proposed tokens. The authors demonstrate that DARTree achieves superior performance across various benchmarks, including math, code, and chat tasks, outperforming previous methods like DFlash and Domino in terms of both acceptance length and speedup.
Methodology
DARTree employs a fixed-width candidate tree construction method that expands and evaluates nodes in parallel, applying causal correction across multiple branches simultaneously. It utilizes a deferred best-first pruning strategy to select the final verification tree, enhancing efficiency by avoiding interleaved operations that slow down processing.
Results
DARTree achieves the highest average acceptance length and speedup across seven benchmarks, accepting up to 12.97 tokens per verification round. It demonstrates a lossless speedup of up to 9.73× over traditional autoregressive decoding methods, outperforming DFlash by 98.6% and Domino by 27.9% in the same settings.
Implications
The advancements presented in DARTree have significant implications for improving the efficiency of language models in real-time applications, reducing latency for users and enhancing the overall performance of autoregressive models in various tasks such as coding, mathematical problem-solving, and conversational AI.
Defensive Boosting for Online Probabilistic Forecasting
Theory
Optimization
Efficient ML
- The Defensive Booster algorithm provides dual guarantees for online probabilistic forecasting.
- It achieves competitive Brier scores while also reducing classification error under specific conditions.
- The algorithm is efficient, requiring only one weak-class learner compared to previous methods.
- A strongly adaptive variant offers local hard-core certificates, enhancing prediction reliability.
Read more
Defensive Boosting for Online Probabilistic Forecasting
Summary
This paper addresses the challenge of online probabilistic forecasting of binary outcomes in the presence of an adaptive adversary. The authors propose a novel algorithm called the Defensive Booster, which combines the strengths of existing online boosting techniques to provide two key guarantees: competitive Brier score with the best predictor in the span of a weak hypothesis class and a reduction in classification error under a smooth weak-learning condition. The Defensive Booster operates efficiently by utilizing a single weak-class learner and achieves both guarantees simultaneously. The paper also introduces a strongly adaptive variant that provides local hard-core certificates for every time interval, enhancing the robustness of the predictions. Experimental results on synthetic and real data streams demonstrate the algorithm's superior predictive performance and significantly faster runtime compared to prior methods.
Methodology
The authors develop the Defensive Booster as a black-box reduction from an online learning algorithm for weak predictors. The algorithm operationalizes the dual view of boosting, utilizing mistake weights to certify the failure of the weak-learning condition when errors persist. The paper also explores a strongly adaptive variant that maintains guarantees across contiguous time intervals.
Results
The Defensive Booster achieves a Brier score competitive with the best predictor in the span of weak hypotheses, with a regret term that scales favorably. Under the smooth weak-learning condition, it guarantees both Brier score and randomized classification error to be minimized effectively. The strongly adaptive variant maintains these guarantees across time intervals, providing local certificates of performance.
Implications
The findings suggest that the Defensive Booster can be applied in various online learning scenarios where adaptive adversaries are present, improving the reliability of probabilistic forecasts in dynamic environments. Its efficiency also makes it suitable for real-time applications in fields such as finance, healthcare, and any domain requiring rapid decision-making based on probabilistic predictions.
FM-LLM: A frequency-enhanced mixture-of-experts framework for adapting LLMs to time series forecasting
Large Language Models
Time Series
- FM-LLM enables prompt-free adaptation of LLMs for time series forecasting.
- The framework utilizes a Fourier Analysis Network for structured spectral token alignment.
- An asymmetric Mixture-of-Experts architecture allows for specialized modeling of periodic and non-periodic dynamics.
- FM-LLM achieves state-of-the-art performance on multiple forecasting benchmarks.
Read more
FM-LLM: A frequency-enhanced mixture-of-experts framework for adapting LLMs to time series forecasting
Summary
The paper introduces FM-LLM, a novel framework designed to adapt Large Language Models (LLMs) for time series forecasting by leveraging frequency information and a mixture-of-experts architecture. Traditional methods for adapting LLMs to time series rely on textual prompts, which can introduce computational overhead and fail to capture the complex spectral dynamics of time series data. FM-LLM addresses these challenges by employing a Fourier Analysis Network (FAN) to create structured harmonic representations that are directly injected into a frozen LLM, allowing for prompt-free adaptation. The framework features an asymmetric Mixture-of-Experts (MoE) decoder that separates roles between shared experts, which model global periodic patterns, and routed experts, which focus on non-periodic dynamics. A hybrid loss function optimizes both temporal accuracy and spectral consistency, reducing error accumulation during long-horizon predictions. Evaluated on eleven public benchmarks, FM-LLM outperforms existing methods, achieving state-of-the-art results on 59 out of 78 metrics, with significant improvements in Mean Squared Error (MSE) and Mean Absolute Error (MAE). The framework also demonstrates strong transferability in few-shot and zero-shot scenarios, establishing a new paradigm for efficient and interpretable time series forecasting using LLMs.
Methodology
FM-LLM employs a Fourier Analysis Network (FAN) to create structured harmonic representations from time series data, which are then injected into a frozen LLM. The framework features an asymmetric Mixture-of-Experts (MoE) decoder that separates the roles of shared experts for periodic dynamics and routed experts for non-periodic dynamics. A hybrid loss function is used to optimize both temporal accuracy and spectral consistency.
Results
FM-LLM achieves state-of-the-art performance on 59 out of 78 evaluation metrics across eleven public benchmarks, with average improvements of 5.3% in Mean Squared Error (MSE) and 5.6% in Mean Absolute Error (MAE). Maximum gains reach 8.0% for MSE and 8.4% for MAE. The framework also demonstrates strong performance in few-shot and zero-shot forecasting scenarios.
Implications
FM-LLM's approach could significantly enhance the accuracy and efficiency of time series forecasting in various applications, including energy systems, traffic management, and climate modeling. Its ability to operate without prompts may reduce computational overhead and improve real-time decision-making capabilities.
Doubly Robust Estimation of Causal Effect on CVR with Targeted Regularization
Theory
- Introduction of a new doubly robust estimator for CVR causal effects.
- Theoretical guarantees based on semiparametric theory and von Mises expansion.
- Development of a targeted regularization framework to improve numerical stability.
- Extensive validation through experiments on synthetic and real-world data.
Read more
Doubly Robust Estimation of Causal Effect on CVR with Targeted Regularization
Summary
This paper addresses the estimation of causal effects on post-click conversion rates (CVR), a crucial metric in e-commerce and advertising. The authors highlight the challenges posed by sample selection bias when using clicked samples for causal inference. They propose a new doubly robust estimator tailored for chain-structured outcomes like CVR, which is grounded in semiparametric theory. This estimator is designed to be consistent even if one of the nuisance estimators is inconsistent, achieving desirable asymptotic properties. Additionally, the authors introduce a targeted regularization framework to enhance numerical stability and practical applicability. Extensive experiments on synthetic and real-world datasets demonstrate the effectiveness and robustness of their method, revealing that traditional loss debiasing combined with standard causal estimators underperforms compared to their proposed approach. The findings underscore the importance of developing estimators specifically tailored to CVR objectives with solid theoretical foundations.
Methodology
The authors utilize semiparametric theory to derive a new doubly robust estimator for CVR, focusing on chain-structured outcomes. They also develop a targeted regularization framework to enhance the practical applicability of their method, ensuring numerical stability during estimation.
Results
The proposed method shows strong performance and robustness across various datasets, outperforming traditional loss debiasing methods combined with standard causal estimators. The theoretical properties of the estimator ensure consistency and faster convergence rates.
Implications
This research has significant implications for industries relying on CVR for decision-making, such as e-commerce and advertising. The proposed methods can lead to more accurate causal effect estimations, ultimately improving user experience and conversion strategies.
Forward and Inverse Virtual Metrology for Phototransistor Gain: A Hierarchical, Uncertainty-Aware Approach for Small Production Datasets
Optimization
Theory
Efficient ML
- Identified that approximately half of the variance in phototransistor gain is due to differences between process runs.
- Developed a forward gain predictor with uncertainty quantification and an inverse search for recipe optimization.
- Introduced a multi-level data quality assessment tailored to the hierarchical structure of fabrication data.
- Released dataset and analysis code to promote reproducibility in research.
Read more
Forward and Inverse Virtual Metrology for Phototransistor Gain: A Hierarchical, Uncertainty-Aware Approach for Small Production Datasets
Summary
This paper addresses the challenge of predicting the gain of silicon bipolar phototransistors from process parameters in a small production dataset context. The authors highlight the inefficiencies in traditional virtual metrology approaches, which typically rely on large datasets. They analyze a limited fabrication history consisting of 13-14 process runs and discover that a significant portion of the variance in device gain is attributed to differences between runs rather than within them. This finding necessitates a hierarchical modeling approach that explicitly considers the grouping structure of the data. The authors propose a forward gain predictor that incorporates uncertainty quantification and an inverse search method to identify recipes that achieve target gains. Additionally, they develop a multi-level data quality assessment framework to ensure the reliability of the data used in modeling. The paper concludes by releasing the normalized dataset and analysis code for reproducibility, emphasizing the importance of data quality in the modeling process.
Methodology
The authors employed a hierarchical modeling approach to analyze a small dataset from multiple process runs. They decomposed the variance of device gain to understand the contributions from different levels of the fabrication process. A forward model was created to predict gain from process parameters, complemented by an inverse search for recipe design. A Gaussian-process uncertainty layer was integrated to provide uncertainty estimates.
Results
The study revealed that the majority of gain variation is attributed to differences between process runs rather than within-run recipe parameters. The developed models demonstrated honest accuracy in predictions and provided insights into the influence of wafer position within runs on gain outcomes. The multi-level data quality assessment framework was effective in ensuring the reliability of the data used for modeling.
Implications
The findings of this research have significant implications for semiconductor manufacturing, particularly in optimizing the fabrication process of phototransistors. The proposed methodologies can enhance decision-making in process engineering, reduce fabrication costs, and improve the efficiency of small production runs. Additionally, the emphasis on data quality can lead to more reliable predictive modeling in similar contexts.
I-SDPO: Instance-Level Adaptive Self-Distillation Policy Optimization
Reinforcement Learning
Large Language Models
Optimization
- I-SDPO introduces a capability-dependent routing mechanism for self-distillation in reinforcement learning.
- The method effectively addresses the degenerate gradient problem in GRPO by selectively applying self-distillation.
- I-SDPO achieves significant performance improvements on the SciKnowEval benchmark across multiple domains.
- The approach reduces reliance on biased teacher signals as model capability increases, enhancing learning efficiency.
Read more
I-SDPO: Instance-Level Adaptive Self-Distillation Policy Optimization
Summary
The paper introduces I-SDPO (Instance-Level Adaptive Self-Distillation Policy Optimization), a novel approach to enhance Group Relative Policy Optimization (GRPO) in reinforcement learning, particularly for large language models (LLMs). GRPO faces a challenge known as the degenerate gradient problem, where all sampled responses yield similar low rewards, leading to ineffective learning. I-SDPO addresses this by implementing a capability-dependent routing mechanism that determines when to apply privileged self-distillation based on the success of sampled responses. Specifically, if all responses in a rollout group are incorrect, I-SDPO utilizes self-distillation to provide dense supervision. Conversely, if at least one response is correct, the group remains under GRPO, preserving the relative reward structure. This adaptive approach reduces the reliance on potentially biased teacher signals as the model's capability improves. The paper also provides a local analysis of the alignment between teacher and reward directions, highlighting the optimization bias introduced by persistent teacher influence. The proposed method significantly improves performance on the SciKnowEval benchmark, achieving an average mean@16 accuracy of 70.31%, compared to 56.67% with GRPO, demonstrating its effectiveness in enhancing learning dynamics in challenging scenarios.
Methodology
I-SDPO employs a routing decision for each input instance, directing all-incorrect response groups to a privileged self-distillation objective while preserving GRPO for groups with at least one correct response. This design allows for adaptive self-distillation based on the success probability of the model, reducing the expected distillation rate as performance improves.
Results
I-SDPO outperformed GRPO, pure SDPO, and sample-level SRPO on the SciKnowEval benchmark, achieving an average mean@16 accuracy of 70.31%, marking a substantial increase from the 56.67% accuracy achieved by GRPO. The method also demonstrated consistent training dynamics and effective handling of optimization biases.
Implications
The findings suggest that adaptive self-distillation can significantly enhance the training of large language models in reinforcement learning contexts, particularly in scenarios where correct responses are rare. This approach may lead to more efficient learning strategies and improved reasoning capabilities in LLMs.
Rubric Dropout: A Simple Way to Mitigate Reward Hacking in Rubric-as-Reward RL
Reinforcement Learning
Large Language Models
NLP
- Rubric-as-reward RL can lead to reward hacking due to fixed proxy criteria.
- Rubric Dropout mitigates reward hacking by randomly dropping rubric criteria during training.
- The method improves out-of-distribution performance on benchmark tasks.
- Rubric Dropout is simple to implement and requires minimal changes to the reward function.
Read more
Rubric Dropout: A Simple Way to Mitigate Reward Hacking in Rubric-as-Reward RL
Summary
This paper addresses the issue of reward hacking in reinforcement learning (RL) systems that utilize rubrics as rewards, particularly in the context of language models. The authors demonstrate that when training a language model (Qwen3-8B) using Group Relative Policy Optimization (GRPO) on medical and science rubrics, a divergence occurs between the scores from the training judge and a stronger gold judge. This divergence indicates that the model is exploiting the fixed nature of the rubric, leading to reward hacking. To mitigate this, the authors propose a novel technique called Rubric Dropout, which randomly drops a subset of rubric criteria during training, ensuring that the model does not optimize against the same rubric repeatedly. This method is shown to effectively reduce reward hacking while maintaining performance. The paper includes a comprehensive evaluation of the proposed method against baseline approaches, demonstrating that Rubric Dropout consistently improves out-of-distribution (OOD) performance on benchmark tasks without incurring additional costs in domain.
Methodology
The authors implemented Rubric Dropout by randomly dropping a fraction of the rubric's criteria at each training step, ensuring that the model is not consistently optimizing against the same criteria. They measured the divergence between scores from the training judge and a stronger gold judge to assess reward hacking. The performance of models trained with and without dropout was compared across multiple benchmarks.
Results
The results indicated that models using Rubric Dropout achieved higher gold scores on HealthBench-Hard (increased by 1-2 points) and ResearchQA (increased by 6-7 points) compared to the baseline without dropout. Additionally, the dropout method reduced the measures of reward hacking and did not negatively impact in-domain performance. A broad sweet spot for dropout rates was identified between 30-50%.
Implications
The findings suggest that Rubric Dropout can be a valuable technique for improving the robustness of RL systems that rely on rubrics as rewards, particularly in open-ended tasks where quality is difficult to quantify. This approach could enhance the reliability of language models in critical applications such as medical advice and research explanations.
A Multispectral Framework for the Detection of Calcium Carbide-Induced Ripening and Shelf-Life Estimation in Climacteric Fruits
Multimodal
- Developed a non-invasive multispectral framework for detecting CaC2-induced ripening in fruits.
- Utilized visible-near infrared spectroscopy to analyze spectral profiles of mango and banana.
- Achieved high classification accuracy (95% for mango, 81% for banana) using XGBoost algorithms.
- Integrated environmental parameters and spectral features for improved detection and estimation.
Read more
A Multispectral Framework for the Detection of Calcium Carbide-Induced Ripening and Shelf-Life Estimation in Climacteric Fruits
Summary
This study addresses the health risks associated with the use of industrial-grade Calcium Carbide (CaC2) for ripening climacteric fruits, specifically mango and banana. The authors propose a non-invasive multispectral framework that utilizes visible-near infrared (NIR) spectroscopy to differentiate between fruits ripened with CaC2 and those ripened naturally or with ethephon. The framework analyzes spectral profiles at 18 discrete wavelengths (410 nm - 940 nm) using the AS7265x spectral triad sensor, revealing that CaC2-treated samples exhibit distinct spectral intensity drops due to accelerated chlorophyll degradation. A feature engineering strategy incorporates spectral variance, intensity ratios, and environmental parameters, followed by dimensionality reduction using Principal Component Analysis (PCA) to retain over 90% of spectral variance. The resulting feature set is employed to train three eXtreme Gradient Boosting (XGBoost) algorithms for classification and quantitative estimation of ripening progression and shelf life. The model achieves a classification accuracy of 95% for mango samples and 81% for banana samples, demonstrating the framework's effectiveness in detecting CaC2-induced ripening and estimating shelf life.
Methodology
The study employs visible-near infrared spectroscopy to analyze the spectral profiles of mango and banana at 18 wavelengths. A feature engineering strategy is used to extract relevant features, followed by PCA for dimensionality reduction. Three XGBoost algorithms are trained for classification and estimation of ripening progression and shelf life.
Results
The proposed framework achieved a classification accuracy of 95% for mango samples and 81% for banana samples, with a carbide class recall of 0.67 for mango and 0.74 for banana, indicating effective differentiation between CaC2-ripened and naturally ripened fruits.
Implications
The framework has potential applications in food safety and quality control, enabling non-invasive monitoring of fruit ripening methods and ensuring consumer health by identifying unsafe ripening practices.
Mechanism Design for Generative Engines: From Exploitation toward Win-Win Outcomes
Theory
Optimization
Generative Models
- The paper formulates the interaction between content providers and platforms as a repeated Stackelberg game, highlighting the strategic dynamics involved.
- A new mechanism, Verifiable-Content Rewards (VCR), is proposed to align incentives between content creators and platforms, promoting high-quality content.
- Simulation experiments show that VCR outperforms traditional defenses, achieving an average improvement of 12.1 percentage points in defense-utility scores.
- The findings suggest that without proper mechanisms, the generative engine ecosystem risks devolving into citation wars that degrade content quality.
Read more
Mechanism Design for Generative Engines: From Exploitation toward Win-Win Outcomes
Summary
This paper addresses the strategic tension between content providers and platforms in the generative engine ecosystem, where citations play a crucial role in determining visibility and value. The authors identify that the competition for citations can lead to 'citation wars,' where content quality is compromised due to manipulative rewriting strategies employed by content providers. To analyze this interaction, the authors model it as a repeated Stackelberg game with partial monitoring, revealing that repeated adaptations can lead to undesirable outcomes where neither content quality nor platform defenses are effective. To mitigate this issue, they propose a new mechanism called Verifiable-Content Rewards (VCR), which incentivizes content providers to produce high-quality, verifiable content rather than engaging in manipulative practices. The VCR mechanism rewards substantive rewrites that enhance factual accuracy, thus aligning the interests of both content providers and platforms. Experimental results demonstrate that VCR significantly outperforms existing methods, achieving better utility scores and fostering a win-win scenario for both parties involved.
Methodology
The authors model the supplier-platform interaction as a repeated Stackelberg game with partial monitoring. They conduct local best-response analyses to understand the dynamics of citation competition and propose the VCR mechanism to incentivize quality content creation.
Results
The VCR mechanism consistently achieves the highest Net defense-utility score across three benchmarks (E-COMMERCE, GEO-BENCH, RESEARCHY-GEO), outperforming the strongest baseline by an average of 12.1 percentage points. It also demonstrates improved document and answer quality metrics.
Implications
The proposed VCR mechanism has the potential to reshape the generative engine ecosystem by promoting high-quality content creation, reducing manipulative practices, and enhancing trustworthiness in information retrieval systems.
A Local Sinkhorn Framework for Conditional Distribution Reconstruction of Multidimensional Random Fields
Theory
Generative Models
Efficient ML
- Introduction of a local entropically regularized optimal transport framework for stochastic neural networks.
- Derivation of generalization error bounds that characterize the learning efficiency of the proposed framework.
- Demonstration of superior performance in reconstructing stochastic random fields compared to existing methods.
- Establishment of a computationally efficient local distribution matching objective using Sinkhorn divergence.
Read more
A Local Sinkhorn Framework for Conditional Distribution Reconstruction of Multidimensional Random Fields
Summary
This paper introduces a novel local Sinkhorn divergence framework aimed at reconstructing conditional distributions of multidimensional random fields. The authors propose a scalable, entropically regularized optimal transport method that enhances the training of stochastic neural networks (SNNs) for learning random fields. By leveraging the debiased Sinkhorn divergence, the framework establishes a differentiable and computationally efficient local distribution matching objective. The authors derive theoretical generalization error bounds that elucidate the trade-off between approximation bias and statistical efficiency, controlled by the regularization parameter. The proposed method not only accurately reconstructs stochastic random fields and dynamical systems but also demonstrates superior performance compared to existing machine-learning benchmarks in uncertainty quantification. Through various numerical experiments, the framework is shown to effectively balance reconstruction accuracy, computational efficiency, and scalability, making it a practical solution for multidimensional stochastic systems.
Methodology
The authors develop a local Sinkhorn divergence framework that replaces the traditional exact optimal transport computations with a more efficient Sinkhorn divergence approach. This method allows for the training of stochastic neural networks through a differentiable local distribution matching objective, while also providing theoretical insights into generalization error bounds.
Results
The proposed framework successfully reconstructs conditional distributions of multidimensional random fields, outperforming several benchmark machine learning approaches in terms of accuracy and computational efficiency. The numerical examples illustrate the framework's capability to maintain good scalability for complex stochastic systems.
Implications
The local Sinkhorn framework has significant implications for uncertainty quantification and probabilistic scientific machine learning, particularly in fields where accurate modeling of stochastic processes is crucial, such as engineering, environmental science, and finance.
When Offline Evaluation Misleads: A Diagnostic Protocol for Reward and Policy Selection in Delayed-Feedback Contextual Bandits
Reinforcement Learning
Optimization
Theory
- Traditional offline evaluation methods can mislead in delayed-feedback contexts.
- The proposed diagnostic protocol assesses alignment and learnability before trusting reported improvements.
- A denser reward signal enhances online learning efficiency, revealing differences in rewards that appear tied in static estimates.
- Personalization may not always be beneficial; sometimes, it serves as a robustness measure.
Read more
When Offline Evaluation Misleads: A Diagnostic Protocol for Reward and Policy Selection in Delayed-Feedback Contextual Bandits
Summary
This paper addresses the challenges of using contextual multi-armed bandits (CMABs) in scenarios where the reward (downstream conversion) is delayed, making online learning impractical. The authors highlight that traditional offline evaluation methods can mislead decision-making in such contexts. They propose a diagnostic protocol that evaluates reward and policy candidates based on two axes: alignment (whether optimizing the reward aligns with the ultimate goal) and learnability (the ability of the bandit to identify the optimal policy). The protocol is validated using a public benchmark and a synthetic generator, demonstrating its effectiveness in real-world applications, particularly in a large marketplace push notification system. The findings reveal that a single offline estimate can misrepresent the true value of rewards, and that personalization may not always yield the expected benefits, emphasizing the importance of robustness in policy selection. The paper contributes a methodological framework rather than new algorithms, providing insights into the complexities of delayed-feedback CMABs.
Methodology
The authors developed an ordered diagnostic protocol that screens reward and policy candidates based on alignment and learnability. They validated this protocol using synthetic data and a public benchmark, and illustrated its application in a real-world push notification system. The study involved analyzing the effects of delayed feedback on traditional offline evaluation methods and their implications for policy selection.
Results
The study found that traditional offline evaluation methods often misrank rewards due to issues like the interval trap and marginal versus conditional testing. The proposed protocol successfully identified the correct reward and policy candidates in both synthetic and real-world scenarios, demonstrating that a single offline estimate can obscure the true learning dynamics of a CMAB. The findings also indicated that when the best single arm is unidentifiable, the perceived benefits of personalization may be overstated.
Implications
The findings suggest that organizations using CMABs in delayed-feedback contexts should adopt the proposed diagnostic protocol to ensure more reliable decision-making. This approach can lead to better alignment of marketing strategies with actual business objectives and improve the effectiveness of personalized interventions.
A Factor Graph Approach to Scalable Multi-Output Gaussian Process Regression
Theory
Efficient ML
Time Series
- Introduces a factor graph representation for multi-output Gaussian process regression.
- Achieves linear scaling in the number of data points and handles missing observations efficiently.
- Demonstrates competitive performance against traditional methods in terms of accuracy and computational cost.
- Utilizes a nearest-neighbor chain to approximate high-dimensional input geometry.
Read more
A Factor Graph Approach to Scalable Multi-Output Gaussian Process Regression
Summary
This paper presents a novel approach to multi-output Gaussian process (GP) regression using a factor graph framework, addressing the scalability issues associated with traditional methods. Multi-output GP regression typically suffers from cubic scaling with respect to the number of observations and outputs, particularly when different outputs are observed at different inputs. The authors propose a Forney-style factor graph representation that organizes a fixed candidate set of inputs into a one-dimensional nearest-neighbor chain. This formulation allows for the evolution of latent Matérn processes through linear-Gaussian transition factors, enabling efficient posterior computation through exact Gaussian message passing. The proposed method scales linearly with the number of data points and accommodates missing observations without requiring covariance matrix restructuring. The authors compare their factor-graph approach against traditional kernel-matrix methods and sparse-variational inducing-point methods on synthetic datasets and electricity time series forecasting, demonstrating that their method closely tracks the exact kernel-matrix posterior at low input dimensions and maintains competitive performance as dimensions increase. The factor-graph formulation matches the accuracy of all baselines in forecasting while significantly improving computational efficiency.
Methodology
The authors develop a factor graph model that combines state-space Gaussian processes with the Linear Model of Coregionalization (LMC). They construct a nearest-neighbor chain from a fixed candidate set of inputs and perform exact Gaussian message passing for posterior computation. The methodology allows for efficient handling of missing observations and reduces computational complexity to O(C(DL² + L³)) after chain construction.
Results
The factor-graph approach closely approximates the exact kernel-matrix posterior at low input dimensions, with the performance gap increasing as dimensions rise. In electricity time series forecasting, the proposed method matches the accuracy of traditional methods while scaling linearly with the number of data points, contrasting with the infeasibility of exact kernel-matrix methods at larger scales.
Implications
This work has significant implications for applications in sensor networks, environmental monitoring, and clinical time series analysis, where multi-output regression is common and efficient handling of missing data is crucial. The factor graph approach can facilitate more scalable and flexible modeling in these domains.
GENADA: efficient generative time series adversarial attack framework
Time Series
Generative Models
Efficient ML
- GENADA provides a generative approach to adversarial attacks, eliminating the need for iterative optimization.
- The framework allows for efficient perturbation generation through a single forward pass after training.
- Empirical results show that GENADA achieves competitive attack quality with significantly reduced generation time compared to traditional methods.
- The method is validated across multiple datasets and neural architectures, demonstrating its versatility in the time series domain.
Read more
GENADA: efficient generative time series adversarial attack framework
Summary
The paper presents GENADA (GENerative ADversarial Attack), a novel framework for generating adversarial attacks on time series data. Traditional gradient-based adversarial attacks are computationally intensive, requiring multiple iterations and backpropagation through the target model. In contrast, GENADA utilizes a generative model that learns to produce adversarial perturbations in a single forward pass, significantly improving efficiency. The authors propose a training procedure that employs a frozen target model, allowing the generator to create perturbations that maximize the loss of the target classifier without needing gradient computations during inference. The framework is validated on various neural architectures and datasets in the time series domain, demonstrating that GENADA achieves comparable attack quality to existing strong baselines while requiring less time for perturbation generation. The experiments include evaluations on datasets such as PowerCons, GunPoint, and Strawberry, showcasing the method's effectiveness across different model types, including recurrent, convolutional, and transformer-based architectures.
Methodology
GENADA employs a generative model trained to produce adversarial perturbations that maximize the loss of a target classifier. The training process involves using a frozen target model, which allows the generator to create effective perturbations without requiring gradient calculations during inference. The framework includes both single-step and iterative generative attack schemes.
Results
The empirical evaluation indicates that GENADA achieves comparable attack quality to strong baseline methods like FGSM and iFGSM while significantly reducing the time required for generating adversarial perturbations. The experiments conducted on various datasets demonstrate the framework's effectiveness and efficiency across different neural network architectures.
Implications
The development of GENADA has significant implications for enhancing the robustness of time series classification models against adversarial attacks. Its efficiency could lead to broader applications in critical domains such as healthcare and finance, where timely and accurate predictions are essential. Additionally, the framework could inspire further research into generative approaches for adversarial machine learning.
Revisiting Overestimation Bias Problem of Q-learning: Settling Large Discrete Action Space via Action Intersection
Reinforcement Learning
- Identifies the overestimation bias problem in Q-learning exacerbated by large action spaces.
- Proposes an action intersection strategy to enable semi-decoupling of Q-value estimation.
- Demonstrates that the action intersection strategy allows for flexible bias control and fine granularity.
- Shows significant performance improvements over state-of-the-art methods in both tabular and deep RL settings.
Read more
Revisiting Overestimation Bias Problem of Q-learning: Settling Large Discrete Action Space via Action Intersection
Summary
This paper addresses the overestimation bias problem inherent in Q-learning, particularly in environments with large discrete action spaces. The authors identify that the increased randomness in Q-value estimation due to a large action space exacerbates the biases associated with existing methods. They categorize these methods into two paradigms: the coupling paradigm, which consistently overestimates Q-values, and the decoupling paradigm, which tends to underestimate them. To mitigate these biases, the authors propose an innovative action intersection strategy that allows for semi-decoupling of Q-value estimation. This strategy involves sharing a fraction of trajectory data between two Q-functions, enabling them to update using both coupling and decoupling paradigms based on data sharing. The paper demonstrates that this approach provides a flexible bias range and fine granularity in control, leading to improved performance in both tabular and deep reinforcement learning settings. Experimental results show that the proposed method significantly outperforms several state-of-the-art baselines, highlighting its effectiveness in addressing the overestimation bias in Q-learning.
Methodology
The authors introduce an action intersection strategy that combines elements of both coupling and decoupling paradigms in Q-learning. This involves sharing trajectory data between two Q-functions to allow for adaptive updates based on whether data is shared or not. The methodology is tested in both tabular and deep reinforcement learning environments to evaluate its effectiveness.
Results
The experimental results indicate that the proposed action intersection strategy outperforms existing state-of-the-art methods in mitigating overestimation bias. In deep reinforcement learning experiments, the method shows drastic improvements, while tabular experiments provide insights into the reasons for its superior performance.
Implications
This work opens new avenues for addressing overestimation bias in Q-learning, particularly in applications with large action spaces such as recommender systems and complex decision-making tasks. The findings suggest that the action intersection strategy could be a valuable tool for enhancing the reliability of Q-learning in various reinforcement learning applications.
Understanding Backdoor Vulnerabilities in Vertical Federated Learning: The Gap Between Research and Practice
Federated Learning
- Identifies a gap between theoretical research and practical realities in VFL backdoor vulnerabilities.
- Critiques existing methodologies for relying on unrealistic assumptions about threat models.
- Introduces BVBench, a benchmark for fair evaluation of backdoor attacks in VFL.
- Demonstrates that current understanding of VFL backdoor risks is fragile and often misleading.
Read more
Understanding Backdoor Vulnerabilities in Vertical Federated Learning: The Gap Between Research and Practice
Summary
This paper investigates the vulnerabilities associated with backdoor attacks in Vertical Federated Learning (VFL), highlighting a significant gap between theoretical research and practical applications. VFL allows organizations with complementary data to collaborate on machine learning tasks without sharing raw data, creating an asymmetric information structure. However, this asymmetry can be exploited by malicious contributors who can implant backdoors during training and manipulate predictions during inference. The authors argue that existing research on backdoor attacks often relies on unrealistic assumptions and poorly designed evaluation practices, leading to misleading conclusions about the effectiveness of proposed defenses. To address this, the paper introduces a redefined threat model that considers realistic constraints and presents BVBench, a benchmark for evaluating backdoor vulnerabilities in VFL. This benchmark aims to provide a more accurate assessment of the risks and establish a foundation for developing effective defenses against backdoor attacks in practical VFL scenarios.
Methodology
The authors conducted a systematic review of existing literature on backdoor attacks in VFL, identifying methodological flaws and unrealistic assumptions. They proposed a new threat model that reflects practical constraints and developed BVBench, a benchmark for evaluating backdoor vulnerabilities with preloaded state-of-the-art baselines.
Results
The study revealed that many existing attack success rates reported in the literature are misleading due to unrealistic assumptions. By relaxing these assumptions, the authors demonstrated a significant reduction in attack success rates, highlighting the fragility of current defenses and the need for more practical evaluation methods.
Implications
The findings suggest that practitioners may underestimate the risks associated with VFL systems, leading to potential security breaches. The introduction of BVBench aims to improve the evaluation of backdoor vulnerabilities, guiding researchers and practitioners in developing more robust defenses and understanding real-world risks in VFL applications.
Disentangling the Expressivity of RoPE
Theory
NLP
Large Language Models
- RoPE can be categorized into periodic (RoPEP) and conventional variants, each with distinct expressivity properties.
- RoPEP is shown to recognize languages definable in past temporal logic with modular predicates, enhancing its theoretical foundation.
- Conventional RoPE's non-repeating rotations lead to a bounded locality bias, which can hinder performance on tasks needing long-distance context.
- Controlled experiments validate the theoretical claims, showing that periodic RoPE outperforms conventional RoPE in specific language tasks.
Read more
Disentangling the Expressivity of RoPE
Summary
This paper investigates the expressivity of rotary position embeddings (RoPE) in transformers, addressing two prevalent accounts of their success: expressivity linked to periodic position information and mechanistic studies emphasizing positional anchors. The authors formalize these accounts within a framework of fully uniform, finite-precision soft-attention transformers. They demonstrate that periodic RoPE (RoPEP) aligns with past temporal logic using modular predicates, while conventional RoPE operates with non-repeating rotations, leading to a bounded simulation of fixed-offset look-back operators. Through controlled experiments, the paper reveals that periodic RoPE can generalize across modular languages, whereas conventional RoPE exhibits limitations in tasks requiring access to distant context. The findings reconcile theoretical expressivity with practical observations, providing insights into the operational characteristics of RoPE transformers.
Methodology
The authors formalize the expressivity of RoPE within a framework of fully uniform, finite-precision soft-attention transformers. They analyze the periodicity of RoPE components and conduct controlled experiments to evaluate the performance of periodic versus conventional RoPE on formal languages requiring modular predicates.
Results
The experiments reveal that periodic RoPE schedules effectively length-generalize on languages that require modular predicates, while conventional RoPE fails to generalize on certain languages. Specifically, conventional RoPE outperforms in some cases but underperforms in others, indicating a local bias that can interfere with long-distance conditioning.
Implications
The findings suggest that understanding the expressivity of RoPE can inform the design of more effective transformer architectures, particularly in applications where long-context dependencies are crucial. This could lead to improvements in various NLP tasks and enhance the theoretical understanding of transformer models.
Virtual Temperature Sensors in Power Transformers Using Neural Ordinary Differential Equations
Time Series
- Introduction of Neural Ordinary Differential Equations (Neural ODEs) for modeling transformer thermal behavior.
- Integration of simplified heat-transfer equations into the Neural ODE framework for physics-aware predictions.
- Evaluation of the model using data from fifteen transformers, showcasing its robustness and generalization capabilities.
- Demonstration of improved forecasting accuracy compared to traditional numerical and data-driven methods.
Read more
Virtual Temperature Sensors in Power Transformers Using Neural Ordinary Differential Equations
Summary
This paper addresses the critical need for accurate modeling and forecasting of thermal behavior in power transformers, which is essential for ensuring reliability and extending asset lifetime. Traditional numerical methods like finite element methods (FEM) and computational fluid dynamics (CFD) provide high fidelity but are computationally expensive and impractical for real-time applications. On the other hand, lumped-parameter thermal models, while more efficient, often fail to capture dynamic responses under varying conditions. The authors propose a hybrid approach using Neural Ordinary Differential Equations (Neural ODEs) that integrates simplified heat-transfer equations directly into the modeling process. This physics-aware framework allows for continuous-time modeling of system dynamics, enhancing prediction smoothness and robustness against irregular data sampling. The model is evaluated using real-world time-series data from fifteen different transformers across Norway, demonstrating its effectiveness as a standardized and robust forecasting tool for diverse transformer designs and cooling mechanisms. The results indicate that the Neural ODE framework significantly improves the accuracy and generalization capabilities of thermal forecasts compared to traditional methods.
Methodology
The authors developed a Neural ODE framework that incorporates simplified heat-transfer equations relevant to power transformers. This approach allows for continuous-time modeling of thermal dynamics, improving prediction accuracy and robustness to irregular sampling. The model was trained and validated using real-world time-series data from multiple transformers with varying designs and cooling mechanisms.
Results
The Neural ODE framework successfully forecasted transformer thermal behavior, outperforming traditional methods in terms of accuracy and generalization across diverse datasets. The results indicate that the model can effectively serve as a standardized tool for predicting internal temperatures in power transformers.
Implications
The proposed Neural ODE framework has significant implications for condition monitoring and predictive maintenance in power transformers, potentially leading to enhanced operational reliability and extended asset lifetimes. Its ability to provide accurate thermal forecasts can aid in optimizing power system operations and preventing costly failures.
Air Quality Station Simulation via LSTM and Attention-Based Modelling
Time Series
- Introduction of SATADL model for simulating air quality station data during outages.
- Utilization of spatial and temporal attention mechanisms for improved prediction accuracy.
- Demonstrated superior performance over baseline models in forecasting PM10 concentrations.
- Potential application in urban air quality management and smart city initiatives.
Read more
Air Quality Station Simulation via LSTM and Attention-Based Modelling
Summary
This paper addresses the challenge of simulating air quality measurements from malfunctioning stations, which is crucial for maintaining reliable air quality data in urban areas. The authors propose a novel deep learning model named SATADL (SpAtial-Temporal Attention Dual LSTM) that leverages both spatial and temporal attention mechanisms to predict air quality metrics, specifically PM10 concentrations, during periods of station downtime. The model is designed to act as a virtual proxy for offline stations, ensuring continuous data availability. The architecture includes a spatial module that processes inputs from surrounding stations and a temporal module that captures patterns from the offline station's historical data. The performance of SATADL is evaluated against four datasets from global air quality stations, demonstrating its effectiveness in simulating air quality data during hypothetical outages lasting up to 48 hours. The results indicate that SATADL outperforms existing baseline models in terms of prediction accuracy, measured by the coefficient of determination and root mean squared error, thus showcasing its potential as a reliable tool for urban air quality management.
Methodology
The SATADL model is an encoder-decoder architecture that incorporates two main modules: a spatial module that processes data from surrounding stations using spatial attention, and a temporal module that analyzes historical data from the offline station using temporal attention. The outputs from both modules are combined and decoded using LSTM blocks to produce the final air quality predictions.
Results
The SATADL model was tested on four different air quality station datasets, successfully simulating PM10 concentrations during hypothetical failures. It consistently outperformed baseline models in various prediction windows, achieving better scores in both the coefficient of determination and root mean squared error metrics.
Implications
The findings suggest that SATADL can serve as a critical tool for urban planners and environmental agencies, enabling them to maintain accurate air quality assessments even during equipment failures. This capability is essential for public health monitoring and the development of smart city infrastructures.
Intervention-Aware Clinical World Model for Post-Op Outcome Forecasting in Cardiology
Multimodal
- Introduces an intervention-aware latent clinical world model that evolves a 3D anatomical state based on post-procedural events.
- Develops a horizon-token formulation for anytime recurrence forecasting from partial histories during the blanking period.
- Achieves AUROC of 0.756 and AUPRC of 0.777 for recurrence prediction in atrial fibrillation ablation.
- Provides retrospective risk estimates at different horizons, allowing for dynamic updates as new clinical data becomes available.
Read more
Intervention-Aware Clinical World Model for Post-Op Outcome Forecasting in Cardiology
Summary
This paper presents an innovative intervention-aware clinical world model aimed at improving post-operative outcome forecasting in cardiology, specifically for patients undergoing atrial fibrillation ablation. Traditional clinical prediction models often treat post-intervention outcomes as a direct mapping from baseline measurements to future endpoints, neglecting the dynamic nature of recovery, which involves irregular trajectories influenced by clinical observations, medication changes, and repeat interventions. The proposed model encodes baseline imaging into a 3D spatial latent state and updates this state over time using a combination of procedural context, static covariates, elapsed time, and peri-event physiological embeddings. The framework allows for dynamic risk assessment during the 90-day recovery period, leveraging irregular post-procedure records to provide clinically meaningful insights into long-term recurrence risk. The model was evaluated using the DECAAF-II dataset, achieving promising results in predicting recurrence and scar extent, thereby demonstrating its potential to enhance clinical decision-making and patient management.
Methodology
The methodology involves encoding pre-ablation MRI scans into a 3D latent state using a frozen variational autoencoder. The model updates this state with various inputs, including procedural geometry, static covariates, and time-ordered clinical events. A horizon-token mechanism is employed to forecast recurrence and scar extent based on available records by a selected query horizon, allowing for dynamic risk assessment throughout the recovery period.
Results
The model demonstrated an AUROC of 0.756 and an AUPRC of 0.777 for predicting recurrence in atrial fibrillation patients during the 90-day recovery window. Additionally, it achieved a scar-extent mean absolute error of 2.971 percentage points, indicating its effectiveness in estimating post-operative outcomes without requiring follow-up MRI data at inference.
Implications
The intervention-aware clinical world model has significant implications for enhancing clinical decision-making in cardiology. By providing dynamic risk assessments and accommodating irregular clinical events, it can improve patient management strategies and potentially lead to better long-term outcomes for patients undergoing atrial fibrillation ablation.
Dual-Model Sentiment Analysis of Consumer Reviews in the Retail Coffee Sector Using Machine Learning and Deep Learning Approaches
NLP
- Introduction of a dual-model sentiment classification framework combining machine learning and deep learning techniques.
- Comprehensive evaluation of multiple algorithms on a real-world, imbalanced dataset.
- Implementation of a preprocessing pipeline using NLP techniques to enhance input quality.
- Performance analysis based on various metrics, including accuracy and F1-score.
Read more
Dual-Model Sentiment Analysis of Consumer Reviews in the Retail Coffee Sector Using Machine Learning and Deep Learning Approaches
Summary
This study presents a dual-model sentiment analysis framework applied to Starbucks customer reviews, utilizing both classical machine learning and advanced deep learning techniques. The dataset, comprising over 700 reviews from ConsumerAffairs, was preprocessed to address class imbalance and extract sentiment labels based on star ratings. The authors benchmarked traditional classifiers such as Logistic Regression, SVM, Decision Tree, Random Forest, and Naive Bayes against deep learning models including LSTM, RNN, Bidirectional LSTM, GRU, and CNN. Performance metrics like accuracy, precision, recall, and F1-score were employed for evaluation. The results indicated that SVM achieved the highest accuracy at 91.0%, while Bidirectional LSTM excelled in generalization on unseen data. The study emphasizes the impact of class imbalance on model performance, particularly in positive sentiment recall. This research contributes to the understanding of sentiment analysis in the retail coffee sector, highlighting the importance of tailored model selection and preprocessing strategies for optimizing customer experience.
Methodology
The study employed a dual-model approach, comparing traditional machine learning algorithms (Logistic Regression, SVM, Decision Tree, Random Forest, Naive Bayes) with deep learning architectures (LSTM, RNN, Bidirectional LSTM, GRU, CNN). Extensive preprocessing techniques such as stopword removal, lemmatization, and TF-IDF vectorization were applied to the dataset to improve sentiment classification accuracy.
Results
The SVM model achieved the highest accuracy of 91.0%, while the Bidirectional LSTM model demonstrated superior performance in generalizing to unseen data. The analysis revealed significant challenges posed by class imbalance, particularly affecting the recall of positive sentiments across various models.
Implications
The findings underscore the necessity for tailored model selection and preprocessing strategies in sentiment analysis, which can enhance customer experience and inform business strategies in the retail coffee sector. This research also contributes to the broader field of sentiment analysis by providing insights into handling imbalanced datasets.
TailBooster: A Dual-Layer Generative Framework for Extreme Value Augmentation with Operational Validity Enforcement
Generative Models
- TailBooster effectively addresses the rarity of extreme events in historical flight data by augmenting datasets with synthetic extremes.
- The framework combines generative modeling with anomaly detection to ensure operational validity of generated records.
- Significant improvements in prediction accuracy for extreme air time and arrival delays were achieved compared to conventional synthetic data.
- TailBooster is model-agnostic and can be applied across various domains where extreme-event prediction is critical.
Read more
TailBooster: A Dual-Layer Generative Framework for Extreme Value Augmentation with Operational Validity Enforcement
Summary
The paper introduces TailBooster, a novel dual-layer generative framework designed to enhance the prediction of extreme events in air transportation, such as severe arrival delays and abnormal air times. These extreme events are rare in historical flight data, leading to insufficient training signals for machine learning models. TailBooster addresses this challenge by combining generative modeling with two anomaly detection layers. The first layer utilizes statistical methods to extract extreme values, providing a concentrated training signal for a Tabular Variational Autoencoder. The second layer employs deep learning techniques to clean the generated data, ensuring operational validity by discarding records that do not conform to learned operational constraints. The framework was tested on U.S. flight records, focusing on five evaluation dimensions: diversity, statistical similarity, fidelity, operational validity, and utility. The results demonstrated significant improvements in operational validity and utility for extreme-event prediction, with a notable reduction in Mean Absolute Error across various regression algorithms when trained on TailBooster's synthetic records.
Methodology
TailBooster employs a dual-layer approach: a statistical layer that extracts extreme values using the interquartile range and a deep learning layer that applies autoencoder-based cleaning to ensure operational validity. The framework utilizes a Tabular Variational Autoencoder for generating synthetic data while enforcing operational constraints learned from historical flight records.
Results
The application of TailBooster resulted in a 47-49% reduction in Mean Absolute Error for extreme air time predictions and a 29-57% reduction for extreme arrival delay predictions when compared to conventional synthetic data. The framework also demonstrated comparable gains when real records were augmented with synthetic extremes.
Implications
TailBooster has significant implications for air traffic management and operational resilience in aviation. By improving the predictability of extreme events, it can aid airlines and airports in proactive decision-making, enhancing safety and operational efficiency. Additionally, its model-agnostic nature allows for broader applications in other domains facing similar challenges with rare event prediction.
TANGCO: Learning Topology-Aware Capacity Allocation for Overload-driven Cascading Failures
Graph Learning
Reinforcement Learning
Optimization
- TANGCO optimizes capacity allocation to enhance resilience against cascading failures in networked systems.
- The approach utilizes a graph neural network to learn from cascade dynamics, overcoming limitations of traditional heuristics.
- TANGCO demonstrates superior performance in robustness across multiple synthetic and real-world networks.
- The methodology allows for transferability of learned policies to unseen graphs, enhancing practical applicability.
Read more
TANGCO: Learning Topology-Aware Capacity Allocation for Overload-driven Cascading Failures
Summary
The paper presents TANGCO (Topology-Aware Neural Graph-Guided Capacity Optimization), a novel approach for allocating capacity in networked systems to mitigate overload-driven cascading failures. These failures occur when a node exceeds its capacity and transfers its load to neighboring nodes, potentially leading to widespread system collapse. The authors address the challenge of capacity allocation under local load redistribution, where existing methods are limited to hand-designed heuristics. TANGCO employs a graph neural network (GNN) to learn optimal capacity allocations from simulated cascade dynamics, maximizing robustness against failures. The methodology involves training the GNN using policy-gradient learning and heuristic anchors to navigate the non-differentiable nature of the problem. The evaluation of TANGCO on various synthetic and real-world networks demonstrates significant improvements in robustness compared to traditional heuristics, with gains ranging from 1.6% to 246%. The learned policies also show transferability across similar graph structures, indicating the potential for broader applications in network resilience planning.
Methodology
TANGCO employs a graph neural network (GNN) to learn capacity allocations based on the topology of the network and the dynamics of cascading failures. The GNN is trained using policy-gradient learning, leveraging a cascade simulator to provide feedback on allocation effectiveness. A heuristic anchor is used to guide the learning process, addressing the non-differentiable nature of the optimization problem.
Results
TANGCO outperformed four hand-designed heuristics in all 450 synthetic instances and 40 out of 45 real-network conditions tested. The robustness gains achieved ranged from 1.6% to 246%. Additionally, TANGCOpre, a pre-trained variant, matched the performance of network-specific training on unseen real networks, demonstrating the efficiency and effectiveness of the learned policies.
Implications
The findings suggest that TANGCO can significantly improve the resilience of critical infrastructure systems, such as power grids and cloud networks, against cascading failures. Its ability to learn from network topology and dynamics opens avenues for more adaptive and robust capacity planning in various applications, potentially reducing the risk of large-scale outages.
Symmetry-Breaking De Novo Crystal Generation via Markovian Jump Diffusion
Generative Models
- Introduces a novel diffusion-based framework for crystal generation that captures complete crystallographic specifications.
- Utilizes a Markovian jump-diffusion process to model symmetry-breaking dynamics and inter-space-group transitions.
- Demonstrates superior performance of SbCD over existing models in generating crystal structures.
- Addresses the limitations of current generative models that rely on empirical distributions for space groups.
Read more
Symmetry-Breaking De Novo Crystal Generation via Markovian Jump Diffusion
Summary
This paper presents a novel framework for generating crystal structures called Symmetry-breaking Crystal Diffusion (SbCD), which addresses the limitations of existing generative models that fail to produce complete crystallographic specifications. Traditional models often rely on empirical distributions for space groups and site symmetries, which restricts their ability to capture global symmetry and structural dependencies. The authors draw inspiration from spontaneous symmetry breaking in physics, proposing a Markovian jump-diffusion process that allows for the generation of full structure specifications by reversing from lower-symmetry priors. The framework incorporates inter-space-group transitions, enabling the model to traverse different space groups in a physically motivated manner. The paper demonstrates that SbCD significantly outperforms existing symmetry-preserving models in de novo generation tasks, showcasing its potential for advancing the field of materials science by accelerating the discovery of crystals with desired properties.
Methodology
The authors develop a theoretical framework that combines variational bounds to unify structural dependencies among crystal components with a Markovian jump-diffusion process. This approach allows for modeling space-group distributions and symmetry-breaking diffusion processes on both continuous and discrete state spaces. The model is designed to adaptively enforce space-group constraints and capture site-symmetry distributions effectively.
Results
In experiments conducted on datasets MP20 and MPTS-52, the SbCD model significantly outperformed its symmetry-preserving counterparts, demonstrating its capability to generate complete crystallographic structures with improved accuracy and stability.
Implications
The proposed SbCD framework has the potential to revolutionize the field of materials science by enabling faster and more efficient discovery of new crystal structures with tailored properties, which could impact various applications such as energy storage, semiconductor design, and pharmaceuticals.
Towards Socially Compliant Navigation in Deep Reinforcement Learning via Proxemics-Based Reward Modeling
Reinforcement Learning
Robotics
- Introduction of a proxemics-based reward model for DRL in social navigation.
- The model is grounded in Hall's theory of interpersonal distance, enhancing interpretability.
- Validation across multiple DRL navigation methods shows improved social metrics.
- Emphasis on the need for explicit assessment of comfort in navigation tasks.
Read more
Towards Socially Compliant Navigation in Deep Reinforcement Learning via Proxemics-Based Reward Modeling
Summary
This paper addresses the challenge of robot navigation in crowded environments by introducing a proxemics-based reward formulation for deep reinforcement learning (DRL). While existing DRL methods have improved navigation performance, they often neglect social compliance, which is crucial for human-robot interactions. The authors propose a novel reward model that represents each human's personal space using a radial Gaussian-mixture field based on Hall's proxemics theory. This model provides a dense and interpretable social learning signal, encouraging robots to minimize proxemic intrusion while maintaining navigation efficiency. The proposed reward is integrated into established DRL navigation methods and evaluated in various simulated crowd scenarios. The results demonstrate that the proxemics-based reward consistently enhances social metrics while preserving competitive navigation performance compared to traditional reward models. The paper emphasizes the importance of explicitly assessing comfort-aware navigation, rather than relying solely on navigation metrics, and provides insights into reward design factors for social navigation.
Methodology
The authors developed a proxemics-based reward model that computes a robot-centric local cost based on the personal space of pedestrians. This model was integrated into existing DRL navigation frameworks and evaluated through simulations across various crowd scenarios, using both navigation and social metrics to assess performance.
Results
The proposed proxemics-based reward model led to consistent improvements in social metrics during navigation tasks, while maintaining competitive performance in terms of navigation efficiency compared to existing reward formulations. The evaluation demonstrated that the model effectively encouraged smoother and more socially compliant robot behaviors in dense crowds.
Implications
The findings suggest that incorporating social compliance into robot navigation systems can enhance human-robot interactions in crowded environments, making robots more acceptable and effective in real-world applications such as malls, hospitals, and airports. This work could inform future designs of socially aware robotic systems.
Geometric and Behavioral Stratification in Transformer Residual Streams
NLP
Large Language Models
Interpretability
- The prediction direction acts as a content-defined privileged anchor in transformer residual streams.
- Residual-stream variation is geometrically and behaviorally stratified based on proximity to the prediction direction.
- A narrow prediction interface is universal across various transformer models, regardless of size.
- Disrupting high variance directions near the prediction direction leads to immediate task-frame shifts.
Read more
Geometric and Behavioral Stratification in Transformer Residual Streams
Summary
This paper investigates the geometric and behavioral stratification of transformer models' residual streams, focusing on the concept of a 'privileged basis' that emerges during training. The author identifies the 'prediction direction'—the unembedding direction of the token being predicted—as a content-defined anchor that organizes residual stream geometry. The study reveals that residual-stream variation is stratified based on proximity to this prediction direction across various transformer models, including dense and mixture of experts (MoE) architectures ranging from 7B to 120B parameters. The findings indicate that a narrow, scale-invariant prediction interface is crucial for determining the next token, while the remaining residual stream expands with model size but is less relevant for readout. The paper discusses how variance-based analyses only partially capture this organization due to the orthogonality of the prediction direction to principal variance axes. The results highlight that as prompts become more diverse, the model's separation among them becomes more pronounced along the prediction axis rather than through variance. The study concludes that the prediction direction serves as a privileged anchor, distinct from previously described axes, and suggests that this geometric stratification aids in isolating readout structure from computation, with implications for model interpretability and evaluation.
Methodology
The author conducted variance analyses on the residual streams of eighteen transformer models using structured prompt sets to examine the influence of the prediction direction on geometric structure. The analysis focused on identifying the prediction interface and its relationship to the broader residual stream geometry.
Results
The study found that the prediction direction organizes residual-stream geometry, creating a narrow, low-dimensional prediction interface that is consistent across models. The prediction-distal complement was identified as having low variance and weak alignment with readout, yet it remains crucial for model function. The results suggest that behavior in transformer models is driven more by directional structure than by magnitude.
Implications
The findings have significant implications for the interpretability of transformer models and their evaluation, as understanding the geometric organization of residual streams can enhance insights into model behavior and decision-making processes.
Variational Parameter Calibration with Physics-Aware Latent-Space Surrogates
Optimization
Theory
Efficient ML
- Introduces a physics-aware latent-space framework for parameter calibration.
- Demonstrates the importance of observable supervision in training for better latent representation.
- Shows that traditional reconstruction accuracy is inadequate for effective inverse modeling.
- Evaluates the method on CFD benchmarks, highlighting its robustness in realistic measurement settings.
Read more
Variational Parameter Calibration with Physics-Aware Latent-Space Surrogates
Summary
This paper presents a novel approach to variational parameter calibration in parametric dynamical systems using a physics-aware neural network framework. The authors introduce an autoencoder-based latent-space surrogate model that effectively couples deep learning with variational parameter estimation. The model is designed to accurately predict flow fields while retaining essential information about physical parameters through observable supervision during training. The proposed method is evaluated on two computational fluid dynamics (CFD) benchmarks, demonstrating that traditional reconstruction accuracy is insufficient for effective inverse modeling. The results indicate that the physics-aware approach improves the separability and organization of latent representations, leading to reduced calibration errors and variability in challenging measurement scenarios, including noisy and low-resolution data. Overall, the framework shows promise for enhancing the efficiency and accuracy of parameter calibration in complex dynamical systems.
Methodology
The authors developed an autoencoder-based framework that maps physical parameters to predicted flow fields through a latent representation. The model incorporates observable supervision during offline training to enhance the correlation between latent variables and system parameters. The online inverse problem is addressed using a surrogate-induced observation operator, allowing for efficient parameter estimation.
Results
The proposed framework was tested on two CFD benchmarks, revealing that it significantly outperforms traditional surrogate models in terms of calibration accuracy and robustness. The quantitative analysis of latent-space representations showed improved case-level separability and temporal organization, which are critical for effective parameter calibration.
Implications
This work has significant implications for the fields of computational fluid dynamics and other engineering applications where accurate parameter calibration is essential. The physics-aware approach could lead to more efficient simulations and better predictive capabilities in various parametric systems, potentially benefiting industries such as aerospace, automotive, and environmental modeling.
Faithful, Sufficient and Understandable: Rethinking Graph Counterfactual Explanations via Discrete Diffusion Inversion
Graph Learning
Interpretability
Generative Models
- GDCE-I provides a comprehensive framework for generating graph counterfactuals that respect domain-specific constraints.
- The method utilizes a discrete denoising diffusion model combined with a novel inversion technique to ensure data-manifold awareness.
- GDCE-I outperforms existing counterfactual explanation methods across multiple benchmarks.
- The paper introduces a standardized evaluation framework for assessing graph counterfactuals.
Read more
Faithful, Sufficient and Understandable: Rethinking Graph Counterfactual Explanations via Discrete Diffusion Inversion
Summary
This paper addresses the challenge of generating counterfactual explanations for Graph Neural Networks (GNNs), which are often viewed as black-box models lacking interpretability. The authors propose a novel method called Graph Diffusion Counterfactual Explanation via Inversion (GDCE-I) that combines discrete denoising diffusion models with a unique inversion scheme. This approach allows for the generation of counterfactuals that respect the data manifold and cover the full edit space, enabling meaningful modifications to graph structures while adhering to domain-specific rules. The paper also establishes a framework for evaluating graph counterfactuals, ensuring consistency and completeness across methods. Through extensive benchmarking, GDCE-I demonstrates significant improvements over existing methods, particularly in the molecular domain, where it provides interpretable and in-distribution solutions.
Methodology
The authors developed GDCE-I by integrating discrete denoising diffusion models with classifier-free guidance and a Gumbel-Max based inversion technique. This allows for the generation of counterfactuals that can make targeted modifications to graph structures while maintaining adherence to the original data distribution and domain-specific rules.
Results
GDCE-I significantly outperformed existing counterfactual explanation methods across four benchmarks, demonstrating its effectiveness in generating valid and interpretable graph modifications. The qualitative analysis in the molecular domain further confirmed that GDCE-I produces meaningful in-distribution solutions.
Implications
The proposed method has significant implications for fields that rely on graph-structured data, such as chemistry and biology, where interpretability is crucial for decision-making. GDCE-I can enhance the transparency of GNNs, making them more suitable for high-stakes applications like drug discovery and toxicity analysis.
On the global feature importance for interpretable and trustworthy heat demand forecasting
Interpretability
Time Series
- Introduces an ante-hoc XAI methodology for assessing global feature importance in heat demand forecasting.
- Utilizes four different interpretability approaches, including Gradient Boosting and post-hoc methods.
- Addresses challenges in model interpretability related to compliance and customer satisfaction.
- Highlights the importance of understanding model decisions in complex systems like District Heating.
Read more
On the global feature importance for interpretable and trustworthy heat demand forecasting
Summary
This paper presents an ante-hoc Explainable AI (XAI) methodology aimed at assessing the global feature importance of machine learning models used for heat demand forecasting in District Heating Systems (DHS). The motivation behind this research is to enhance the interpretability and trustworthiness of these models, addressing challenges related to compliance, customer satisfaction, and liability risks. The methodology employs four approaches: intrinsic interpretability of Gradient Boosting and three post-hoc methods—Partial Dependence, Accumulated Local Effects, and SHAP. These methods do not rely on feature permutation or perturbations, which can introduce biases. The paper discusses the results of these approaches, highlighting their complementarities and providing specific interpretations relevant to district heating processes. The research emphasizes the importance of understanding model decisions, especially in complex systems like DHS, where accurate forecasting is crucial for operational efficiency and compliance with standards.
Methodology
The methodology involves training a regression model using ensembles of decision trees on historical data from a SCADA system, merged with meteorological data. The model's feature importances are evaluated using intrinsic metrics and post-hoc interpretability methods, including Partial Dependence, Accumulated Local Effects, and SHAP.
Results
The application of the different interpretability methods provided insights into the global feature importance of the heat demand forecasting model. The results indicated that the methods complement each other, offering a comprehensive understanding of the model's decision-making process.
Implications
The findings suggest that enhancing the interpretability of heat demand forecasting models can lead to better compliance with standards, improved customer satisfaction, and reduced liability risks in District Heating Systems. This research can inform the development of more transparent and trustworthy AI systems in energy management.
Long-Horizon Forecasting of Complete Financial Statements with Forma
Time Series
- ProForma-20Q is introduced as a benchmark for forecasting complete financial statements over long horizons.
- Forma, a transformer-based model, outperforms all tested competitors, including classical ML and large language models.
- The model's performance improves with longer forecasting horizons, crucial for accurate valuation.
- Forma's architecture allows for scenario analysis and maintains accounting coherence in forecasts.
Read more
Long-Horizon Forecasting of Complete Financial Statements with Forma
Summary
This paper addresses the challenge of long-horizon forecasting of complete financial statements, a critical aspect of valuation and financial analysis that has not been extensively explored in prior research. The authors introduce ProForma-20Q, a novel benchmark for forecasting 78 financial statement line items across 1 to 20 quarters using historical data and industry codes. The proposed model, Forma, is a transformer-based architecture that processes financial statements as sets of (account, quarter, value) tuples, optimizing for masked-tuple Gaussian likelihood. The results demonstrate that Forma significantly outperforms various classical machine learning models and large language models, particularly as the forecasting horizon extends. The model's forecasts maintain coherence with accounting identities and allow for scenario analysis without retraining, showcasing its practical utility in financial forecasting.
Methodology
The authors developed a transformer-based model, Forma, which encodes financial statements as sets of (account, quarter, value) tuples. The model is trained to maximize the likelihood of masked tuples, producing Gaussian predictive means and variances for future values. The ProForma-20Q benchmark was established to evaluate the model's performance in predicting changes in financial statement line items over multiple quarters.
Results
Forma achieved superior performance compared to classical machine learning approaches and large language models, with a significant increase in R² scores as the forecasting horizon extended. The model's forecasts were coherent with accounting identities, and it demonstrated the ability to provide probabilistic forecasts that could be reconciled without significant accuracy loss.
Implications
The findings suggest that specialized models like Forma can enhance the accuracy of long-term financial forecasting, which is vital for valuation, credit risk analysis, and financial planning. The ability to conduct scenario analysis without retraining also makes it a valuable tool for practitioners in finance.
A Cloud-Edge System for Multimodal Clinical Screening in Resource-Constrained Rural Settings
Multimodal
Efficient ML
Interpretability
- Introduction of a cloud-edge collaborative architecture for multimodal clinical screening.
- Dynamic selection of diagnostic tools based on patient context to optimize modality coverage.
- Evaluation framework includes 100 multimodal cases and simulates rural network conditions.
- Achieved high accuracy and factual grounding while minimizing data transmission to the cloud.
Read more
A Cloud-Edge System for Multimodal Clinical Screening in Resource-Constrained Rural Settings
Summary
This paper presents a novel cloud-edge collaborative architecture designed to enhance multimodal clinical screening in resource-constrained rural settings. The authors identify significant barriers to accessing advanced medical AI technologies in these areas, including limited bandwidth, computational resources, and the need for integrating diverse medical modalities. To address these challenges, the proposed system utilizes lightweight, domain-specific models deployed at the edge to transform raw medical data into structured outputs, which are then synthesized by a cloud-based large language model (LLM) into coherent clinical summaries. An orchestrator on the edge dynamically selects which diagnostic tools to invoke based on patient context, ensuring relevant modalities are acquired during a single patient encounter. The system was evaluated on 100 multimodal clinical cases across various medical domains, demonstrating high oracle accuracy and factual grounding while significantly reducing the amount of data transmitted to the cloud. The findings highlight the importance of architectural design in improving clinical decision-making under real-world constraints, suggesting that hybrid systems can produce more selective and better-grounded evidence compared to traditional cloud-only approaches.
Methodology
The methodology involves a hybrid system architecture where lightweight, domain-specific models operate on the edge to process medical data and generate structured outputs. A cloud-based LLM synthesizes these outputs into clinical summaries. The system was evaluated using a set of 100 multimodal clinical cases under three simulated network profiles, assessing various metrics including accuracy, coverage, and reasoning quality.
Results
The hybrid cloud-edge system achieved oracle accuracy ranging from 0.87 to 0.90 and demonstrated high factual grounding with knowledge graph precision up to 0.96. Coverage precision was reported between 0.95 and 0.99, while the system maintained bandwidth-invariant latency of 25 to 38 seconds and reduced token costs by up to 15 times compared to cloud-only baselines.
Implications
The proposed system has significant implications for improving access to advanced medical diagnostics in rural areas, potentially reducing patient attrition and enhancing clinical decision-making. It highlights the feasibility of deploying AI technologies in resource-limited settings, paving the way for broader applications in telemedicine and rural healthcare.
Terminal Symmetry as a Decision Resource: Statewise Refinement for Anytime Verified Construction
Theory
Optimization
Robotics
- Introduces a decision-resource formulation that leverages terminal symmetry in directed construction tasks.
- Develops SYMBUILD, which implements a transport-refine-certify approach to enhance decision-making.
- Demonstrates significant improvements in verified efficiency across multiple domains.
- Achieves the lowest mean capped verifier cost compared to existing planners in GRN OOD scenarios.
Read more
Terminal Symmetry as a Decision Resource: Statewise Refinement for Anytime Verified Construction
Summary
This paper introduces a novel decision-resource framework based on terminal symmetry for sequential construction tasks, which often exhibit symmetry at completion despite being directed and history-dependent during execution. The author presents a decomposition of terminal symmetry into four components: process evidence, terminal correspondence, realized-state evidence, and a fixed verifier, leading to a transport-refine-certify methodology. The proposed system, SYMBUILD, utilizes an episode-fixed transported process structure and state-restricted process rank to enhance decision-making efficiency. The methodology includes a state-dependent residual rank that is updated after accepted transitions and an ordinal rank meet that guarantees completion under prefix coverage. The paper demonstrates that SYMBUILD significantly improves verified efficiency across various domains, including CAD assembly, Mini-Programs, and exact-fill packing, achieving notable increases in area under the curve (AUC) metrics. Furthermore, SYMBUILD outperforms existing planners in terms of mean capped verifier cost across multiple scales in target-removal episodes, showcasing the effectiveness of the statewise signal transfer across different organizational structures.
Methodology
The methodology involves a decomposition of terminal symmetry into process evidence, terminal correspondence, realized-state evidence, and a fixed verifier. SYMBUILD implements this through an episode-fixed transported process structure, state-restricted process rank, and a state-dependent residual rank that refreshes after transitions. The system uses an ordinal rank meet to ensure completion guarantees and tight verifier-query bounds.
Results
SYMBUILD shows improvements in anytime AUC by up to 6.77, 21.75, and 8.68 points across CAD assembly, Mini-Programs, and exact-fill packing, respectively. It also achieves the lowest mean capped verifier cost in GRN OOD scenes, outperforming other planners.
Implications
The findings suggest that terminal symmetry can be effectively utilized as a reusable decision resource in various directed construction tasks, potentially leading to more efficient planning and execution in robotics, CAD, and other domains requiring sequential decision-making.
Reduced Matrix Multiplication: Input-Adaptive Matrix-Product Reduction for LLM Inference
Large Language Models
NLP
Efficient ML
- RMM is a training-free, input-adaptive method that reduces matrix products in Transformers without modifying model weights.
- The method allows for a predictable accuracy-efficiency trade-off through a simple retention ratio.
- Larger models generally tolerate more aggressive reductions, but this tolerance is task and model-dependent.
- Attention-side computations are more reducible than MLP components, indicating structural asymmetry in Transformers.
Read more
Reduced Matrix Multiplication: Input-Adaptive Matrix-Product Reduction for LLM Inference
Summary
The paper introduces Reduced Matrix Multiplication (RMM), a novel method aimed at optimizing inference costs in Transformer-based language models by adaptively reducing the number of computations involved in high-dimensional matrix multiplications. RMM operates without modifying model weights and allows for a controllable accuracy-efficiency trade-off through a retention ratio. The authors demonstrate that the reduction tolerance varies across different model families, tasks, and components, with larger models generally exhibiting greater tolerance for reduction. The study also highlights a structural asymmetry in Transformers, revealing that attention-side computations are more amenable to reduction compared to MLP components. Additionally, RMM is shown to extend its benefits to multimodal vision-language inference. The implementation of custom kernels on NVIDIA A100 GPUs indicates that RMM can lead to significant runtime improvements, particularly for longer sequences, positioning it as a scalable solution for input-adaptive inference-time optimization.
Methodology
The authors propose RMM, which dynamically selects and computes informative indices along the shared multiplication axis of matrix products during inference. This method does not require training and allows for systematic exploration of redundancy in Transformer computations. The authors also implement custom Triton kernels to evaluate the practical runtime benefits of RMM on NVIDIA A100 GPUs.
Results
The experiments reveal that RMM maintains robustness across various tasks and model sizes, with notable performance improvements in runtime, especially for longer sequences. The findings indicate that RMM can effectively reduce computation while preserving model performance, with attention computations being more amenable to reduction than MLP components.
Implications
RMM presents a scalable approach for optimizing inference in large language models, potentially leading to more efficient deployment in real-world applications. Its adaptability could facilitate faster processing times in various tasks, including multimodal applications, thereby enhancing the usability of Transformer models in practical scenarios.
Consolidator: Learning Persistent Routed Memory Across Context Boundaries
Theory
- Introduces a shared slot-local transform for STM consolidation without replay.
- Demonstrates that consolidated LTM can guide later memory slot selection.
- Shows significant improvements in recall performance with learned consolidation.
- Isolates and evaluates multiple memory functions in a controlled task.
Read more
Consolidator: Learning Persistent Routed Memory Across Context Boundaries
Summary
The paper introduces 'Consolidator', a novel mechanism designed to enhance the retention and utilization of short-term memory (STM) across context boundaries in a Phasor Memory Network (PMNet). The authors argue that simply copying STM to long-term memory (LTM) does not guarantee that the retained information will influence future memory access. Consolidator operates as a shared slot-local operator that transforms routed STM before integrating it into LTM, allowing for the clearing of STM and key-value (KV) caches post-consolidation. The study evaluates this mechanism through a two-segment modulo-10 mapping task, where the second segment updates the mapping at the same memory address. The results demonstrate that direct routing from LTM significantly improves recall of updated mappings, indicating that the consolidated LTM not only serves as retrievable content but also influences subsequent memory access. The findings suggest that learned consolidation can outperform traditional identity accumulation methods, highlighting the importance of a learned transition in memory systems.
Methodology
The authors utilized a Phasor Memory Network (PMNet) architecture, where they implemented the Consolidator mechanism to transform routed STM into LTM. They conducted experiments on a two-segment mapping task, freezing the majority of the model parameters and training only the 12.35K parameters of the Consolidator. The study included various interventions to isolate the effects of learned consolidation and routing on memory recall.
Results
The results indicated that direct routing from LTM improved updated mapping recall from 44.38% to 87.02%, while immediate STM recall remained stable at 89.90%. Learned consolidation outperformed forced identity accumulation by 21.40 percentage points without routing and 68.70 percentage points with routing, demonstrating the effectiveness of the proposed mechanism in enhancing memory access and retention.
Implications
The findings suggest that incorporating learned memory consolidation mechanisms can significantly improve memory systems in machine learning models, particularly in tasks requiring retention of information across context changes. This could have applications in areas such as natural language processing, continual learning, and systems requiring efficient memory management.
CAKE: Compiler-Agent Co-Design for Frontier Kernel Evolution
Optimization
Efficient ML
Theory
- CAKE enables agents to author a typed intermediate representation (IR) for better control over GPU programming.
- The framework provides localized correctness and performance diagnostics, enhancing the feedback loop for kernel evolution.
- Agent-generated kernels show substantial performance improvements, outperforming traditional CUDA/PTX implementations.
- CAKE evolves its IR based on real-world production kernels, ensuring it adapts to emerging workloads and capabilities.
Read more
CAKE: Compiler-Agent Co-Design for Frontier Kernel Evolution
Summary
The paper introduces CAKE, a novel framework that integrates compiler and agent co-design to enhance GPU kernel evolution. Traditional approaches treat the compiler as a black box, leading to inefficiencies in identifying the causes of synchronization failures and performance bottlenecks. CAKE addresses this by enabling agents to work with a typed intermediate representation (IR) that provides explicit control over hardware resources and facilitates detailed performance diagnostics. This allows for a more informed evolution of GPU kernels, as agents can refine their proposals based on structured feedback from the compiler. The framework evolves the Cake IR through a bottom-up approach, leveraging a corpus of production kernels to identify recurring patterns and gaps in capabilities. The results demonstrate significant performance improvements, with agent-generated kernels achieving up to 2.05× speedup over existing implementations. CAKE is designed to support NVIDIA GPUs from Ampere to Blackwell and aims to bridge the gap between expert kernel programming and automated kernel generation.
Methodology
CAKE employs a co-design approach where agents interact with a typed IR that encapsulates hardware-specific scheduling information. The framework evolves the IR based on insights gained from a corpus of production kernels, allowing agents to refine their proposals iteratively. It also incorporates a feedback mechanism that provides detailed diagnostics on correctness and performance, enabling agents to make informed decisions during the evolution process.
Results
The best candidate kernel generated using CAKE achieved a median performance of 1.144× compared to a tuned baseline, while agent-generated Kimi Delta Attention achieved a 2.05× geometric mean speedup over existing implementations. Dispatcher-backed KNN and KMeans families demonstrated performance improvements ranging from 1.42× to 2.12× across various shapes, with several kernel changes proposed for upstream integration.
Implications
The CAKE framework has the potential to revolutionize GPU programming by enabling more efficient kernel evolution and optimization. It could lead to the development of high-performance kernels that adapt to new workloads and hardware capabilities, ultimately enhancing the performance of applications relying on GPU computations.
A Compositional Theory of Curvature in Probabilistic Circuits
Generative Models
Optimization
Theory
- Probabilistic Circuits (PCs) allow for exact inference and tractable curvature measures unlike deep neural networks.
- Global sharpness regularization can lead to underfitting in PCs due to their compositional curvature characteristics.
- The contribution of nodes to the Hessian trace can be decomposed into contextual usage and local sharpness, providing insights into effective regularization.
- An adaptive sharpness-aware regularizer is proposed, which targets nodes based on their local curvature, improving generalization.
Read more
A Compositional Theory of Curvature in Probabilistic Circuits
Summary
This paper presents a novel approach to understanding the curvature of loss surfaces in Probabilistic Circuits (PCs), which are generative models capable of exact inference. The authors argue that traditional global sharpness regularization methods may not be suitable for PCs due to their inherently compositional nature. They demonstrate that the contribution of each sum node to the Hessian trace can be decomposed into two distinct factors: the circuit flow (indicating how heavily a node is utilized) and a local sharpness term (reflecting the node's output distribution). This decomposition reveals that sharpness is contextually dependent and that global regularization can lead to underfitting by misallocating regularization efforts. To address this, the authors propose an adaptive sharpness-aware regularizer that penalizes nodes based on their intrinsic local curvature while maintaining efficient updates. Empirical results show that this targeted approach improves generalization compared to traditional global regularization methods, particularly in low-data scenarios.
Methodology
The authors analyze the curvature of loss surfaces in Probabilistic Circuits by decomposing the contribution of sum nodes to the Hessian trace into two components: contextual usage and local sharpness. They develop an adaptive sharpness-aware regularizer that penalizes nodes based on their intrinsic local curvature, allowing for efficient updates while preserving the benefits of sharpness-aware learning.
Results
The proposed adaptive sharpness-aware regularizer outperforms traditional global regularization methods, particularly in low-data settings, by effectively targeting nodes with high local curvature. This results in improved generalization and reduced underfitting, as evidenced by empirical evaluations.
Implications
The findings suggest that a more nuanced approach to regularization in Probabilistic Circuits can enhance their performance in various applications, including constrained generation, image inpainting, and robust representation learning. This work may influence future research on optimization techniques in generative models.
Balanced Adaptive Prototype Selection for Scalable TabPFN Inference on Large-Scale Tabular Data
Efficient ML
- Identifies bounded inference context as a primary scalability bottleneck for pretrained tabular models.
- Introduces BAPS, a framework for constructing compact inference contexts that preserve critical information.
- Demonstrates that BAPS can achieve approximately 1,953-fold context compression while retaining strong predictive performance.
- Shows that BAPS is compatible with the original TabPFN architecture without requiring model retraining.
Read more
Balanced Adaptive Prototype Selection for Scalable TabPFN Inference on Large-Scale Tabular Data
Summary
This paper addresses the scalability challenges of pretrained tabular foundation models, specifically focusing on the TabPFN model, which has shown strong predictive capabilities but is limited by the size of the inference context. The authors propose a novel framework called Balanced Adaptive Prototype Selection (BAPS) that constructs compact, information-preserving contexts for scalable inference without modifying or retraining the pretrained model. BAPS aims to retain essential features such as representative structure, informative decision boundaries, local density, class balance, and feature-space diversity. The effectiveness of BAPS is demonstrated through experiments on large datasets, including the million-row HIGGS and SUSY datasets, where it achieves significant context compression while maintaining predictive performance. The findings suggest that effective context construction is crucial for extending pretrained tabular models to handle large-scale datasets efficiently.
Methodology
The BAPS framework is designed to optimize the selection of prototypes from large datasets by focusing on preserving key information rather than merely reducing dataset size. It jointly considers various aspects such as class balance, local density, and informative decision boundaries to construct a compact inference context suitable for the TabPFN model.
Results
Experiments conducted on the HIGGS and SUSY datasets revealed that BAPS could maintain strong predictive performance using only 512 prototypes, achieving a context compression ratio of approximately 1,953-fold. This indicates that the framework effectively retains essential predictive information while significantly reducing the size of the inference context.
Implications
The proposed BAPS framework has the potential to enhance the scalability of pretrained tabular foundation models, making them applicable to large-scale datasets in various domains. This could lead to more efficient machine learning applications in industries that rely on large tabular data, such as finance, healthcare, and marketing.
The Time Value of Evolution
Reinforcement Learning
Optimization
Theory
- Formalization of the 'time value of evolution' concept.
- Introduction of Lineage-Value Policy Gradients (LVPG) for evolutionary search.
- Demonstrated improvement in search efficiency and performance through long-horizon credit assignment.
- Reduction in temporary regressions compared to immediate-return optimization.
Read more
The Time Value of Evolution
Summary
This paper addresses the limitations of traditional evolutionary search methods that often undervalue mutations leading to weak offspring, which may later enable stronger descendants. The authors formalize this concept as the 'time value of evolution' within a finite-horizon Markov decision process. They propose a novel framework called Lineage-Value Policy Gradients (LVPG), which separates search control into specialized policy heads over a shared generative backbone. The LVPG architecture includes a bootstrapped critic head that estimates the value of finite-horizon lineage potential and an actor head that modulates mutation intensity based on the remaining search budget. The study empirically evaluates the effectiveness of LVPG against immediate-return optimization across 90 paired runs, demonstrating that long-horizon credit assignment significantly enhances search efficiency and performance, while also reducing temporary regressions. The findings suggest that recognizing the delayed utility of mutations can lead to more selective and effective evolutionary search strategies.
Methodology
The authors developed the LVPG framework, which employs a long-horizon actor-critic approach for automated policy discovery. The architecture includes a generative backbone with separate actor and critic heads, where the critic estimates lineage value from mutation trees and the actor adjusts mutation intensity based on the remaining search budget. The framework was empirically tested against traditional immediate-return optimization methods across multiple controlled experiments.
Results
The LVPG framework significantly improved search efficiency, increasing the validation best-so-far AUC by 0.394 Sharpe units. It also produced fewer temporary regressions and demonstrated a greater ability to recover from them compared to immediate-return optimization methods. The results highlight the advantages of long-horizon credit assignment in evolutionary search.
Implications
The findings suggest that evolutionary algorithms can benefit from incorporating the time value of mutations, leading to more effective search strategies in various applications, including automated trading and program evolution. This approach may enhance the performance of evolutionary algorithms in complex environments where the value of mutations is not immediately apparent.