AI-generated summaries
Today's ML research,
without the noise.
Daily summaries of the latest machine learning papers from arXiv, processed every 8 hours.
57
Papers today
8h
Update frequency
7
Days of history
A Manifold-Aware Topic Modeling Approach via Rank-Based Prototypes
NLP
- MARETopic is the first topic model based entirely on rank-based similarity information.
- The framework anchors topics in real corpus documents, enhancing interpretability.
- An inter-topic MMR step increases vocabulary diversity while maintaining coherence.
- MARETopic achieves faster execution times compared to existing models without requiring gradient updates.
Read more
A Manifold-Aware Topic Modeling Approach via Rank-Based Prototypes
Summary
This paper introduces MARETopic, a novel topic modeling framework that leverages rank-based prototype selection to address limitations in existing topic models that rely on absolute distances in high-dimensional spaces. Traditional models often fail to provide document-level anchors for topics and can struggle with the geometric distortions of embedding spaces. MARETopic operates without training, using a greedy algorithm to select K exemplar documents from a low-dimensional manifold representation of the data. It utilizes Sentence-BERT for document encoding and UMAP for dimensionality reduction, creating ranked lists that capture the ordinal neighborhood structure of the data. The framework includes two variants: MARETopicCorr, which employs a query performance predictor and rank correlation for scoring, and MARETopicDiff, which uses a rank-based diffusion matrix for faster execution. The model achieves superior topic coherence and diversity, with an innovative inter-topic Maximal Marginal Relevance step that enhances vocabulary diversity with minimal coherence loss. Evaluations on three benchmarks demonstrate MARETopic's effectiveness, outperforming both neural and clustering-based models in key metrics.
Methodology
MARETopic employs a training-free approach that encodes documents using Sentence-BERT, projects them onto a low-dimensional manifold via UMAP, and constructs ranked lists of documents. A greedy algorithm selects K exemplar documents based on a scoring function that penalizes redundancy. Two variants of the model, MARETopicCorr and MARETopicDiff, utilize different scoring mechanisms to optimize the selection process.
Results
MARETopic outperformed seven state-of-the-art baselines in terms of topic coherence and diversity on three benchmark datasets (20 Newsgroups, NYT, and Web of Science). MARETopicCorr achieved high scores in Purity and NMI, while MARETopicDiff demonstrated a significant speed advantage, running 1.7 to 1.9 times faster than its counterpart.
Implications
The MARETopic framework offers a robust alternative for topic modeling that enhances interpretability and efficiency. Its reliance on rank-based methods can be beneficial for applications in document classification, novelty detection, and corpus summarization, where understanding the thematic structure is crucial.
Time-Series Foundation Models That Understand Data Revisions
Time Series
- Introduction of VINTAGE-TS, a revision-aware time-series forecasting model.
- Distinction between observation time and information-availability time to improve accuracy.
- Joint predictive distribution to model uncertainty in forecasting revisions.
- Comprehensive evaluation framework including rolling evaluations and audits.
Read more
Time-Series Foundation Models That Understand Data Revisions
Summary
This paper introduces VINTAGE-TS, a novel time-series foundation model designed to account for data revisions that occur after initial publication. Traditional forecasting models often assume that historical data is fixed, which can lead to inaccuracies when revised data becomes available. VINTAGE-TS distinguishes between observation time and information-availability time, allowing it to predict both the first-published value and the value after a specified maturation period. The model employs a joint predictive distribution to capture the uncertainty surrounding these targets. The author outlines a comprehensive evaluation framework using ALFRED for rolling evaluations and compares it with Chronos-2 and other baseline models. The paper also emphasizes the importance of an audit system to ensure that pretraining does not introduce biases from future revisions. Through a synthetic demonstration and sensitivity analysis, the study reveals how hindsight contamination can affect performance metrics. The findings suggest that while the integration of revision histories may enhance forecasting accuracy, the empirical advantages of the model remain to be validated in real-world scenarios.
Methodology
The methodology includes the development of VINTAGE-TS, which integrates explicit timing of information and separate targets for forecasting. The evaluation framework employs ALFRED for rolling evaluations and includes a matched comparison with Chronos-2, alongside conventional and revision-aware baselines. The paper also details an audit process to check for pretraining overlap and potential biases.
Results
The synthetic demonstration and sensitivity analysis indicate that the proposed model can expose variations across different seeds and revision regimes. However, the paper does not claim empirical advantages over existing models until real-world experiments are conducted.
Implications
The findings could lead to improved forecasting models in economics and other fields where data revisions are common. By accurately accounting for the timing of information availability, practitioners may achieve better calibration and predictive performance.
Reachability-Based Formal Verification of Graph Neural Networks with Node and Edge Features
Graph Learning
- Introduction of GraphStar sets for uncertainty representation in GNNs.
- Extension of the NNV framework to support verification of GCN and GINE architectures.
- Demonstration of tighter robustness guarantees than existing verification methods.
- First edge-aware reachability analysis for GINE models under joint perturbations.
Read more
Reachability-Based Formal Verification of Graph Neural Networks with Node and Edge Features
Summary
This paper addresses the challenge of formally verifying Graph Neural Networks (GNNs) that utilize both node and edge features, particularly in the context of electric power systems. The authors extend the Neural Network Verification (NNV) framework by introducing GraphStar sets, which generalize Star sets to account for uncertainties in both node and edge features. This extension allows for the effective propagation of linear message-passing operations and the sound approximation of ReLU nonlinearities in GNN architectures, specifically Graph Convolutional Networks (GCN) and Graph Isomorphism Networks with Edge features (GINE). The proposed GNNV framework is evaluated on three power system tasks—Power Flow (PF), Optimal Power Flow (OPF), and Cascading Failure Analysis (CFA)—across various IEEE test cases, as well as on standard graph classification benchmarks (ENZYMES and PROTEINS). The results demonstrate that GNNV offers tighter robustness guarantees compared to existing methods, particularly CORA, and provides edge-aware robustness guarantees for GINE-based models under joint perturbations of node and edge features.
Methodology
The authors developed GraphStar sets to capture uncertainties in both node and edge features, integrating this into the NNV framework. They implemented verification procedures for GCN and GINE layers, allowing for reachability analysis and robustness guarantees under various perturbation scenarios. The methodology was validated through experiments on power system tasks and graph classification benchmarks.
Results
The GNNV framework provided tighter robustness guarantees than CORA, maintaining verification capabilities at higher levels of perturbation where CORA failed. The framework also successfully delivered edge-aware robustness guarantees for GINE-based models, marking a significant advancement in the formal verification of GNNs.
Implications
The findings suggest that GNNs can be reliably used in safety-critical applications, such as power systems, where formal verification is essential. The ability to account for uncertainties in both node and edge features enhances the robustness of GNNs, potentially leading to broader applications in various domains that rely on graph-structured data.
Spectral-Guided Diffusion: Accelerating Inference via Static Spectral Layer Scheduling
Generative Models
Efficient ML
Large Language Models
- Introduction of Spectral Concentration Ratio (SCR) for efficient scheduling in diffusion models.
- Static scheduling based on pretrained weights eliminates the need for dynamic routing or calibration.
- Achieves 2.8x to 3.0x speedup in inference time without sacrificing output quality.
- Demonstrated effectiveness across various architectures and tasks, including retrieval and reasoning.
Read more
Spectral-Guided Diffusion: Accelerating Inference via Static Spectral Layer Scheduling
Summary
This paper introduces a novel approach to accelerate inference in diffusion models by leveraging a method called Spectral Concentration Ratio (SCR) combined with Frobenius magnitude. The authors propose that pretrained weights can be utilized to identify residual branches in the network that do not need to be recomputed during the inference process. By measuring the energy distribution of singular values, the SCR provides a static sensitivity proxy that assigns lifetimes to residual units, allowing for efficient caching of updates. This method eliminates the need for routers or calibration prompts, resulting in a deterministic scheduling of computations. The authors demonstrate that their approach outperforms existing scheduling methods in terms of quality preservation while achieving significant speedups in inference time, with reported wall-clock speedups of 2.8x to 3.0x compared to eager inference. The results are validated across multiple architectures and tasks, showcasing the effectiveness of the SCR/Frobenius scheduling in maintaining performance while reducing computational overhead.
Methodology
The authors developed the SCR, a metric that quantifies the energy distribution of singular values in the pretrained weights of the model. This metric is combined with Frobenius magnitude to create a static sensitivity proxy that determines the lifetime of residual units. The scheduling is computed offline and reused for every input, allowing for cached updates of residual branches while maintaining the current state of the network. The method was tested on various architectures and tasks to evaluate its performance against existing scheduling techniques.
Results
The proposed method demonstrated a significant improvement in inference speed, achieving wall-clock speedups of 2.8x to 3.0x over traditional eager inference methods. The SCR/Frobenius scheduling preserved output quality better than other scheduling strategies, including random and depth-based approaches. The results were consistent across multiple architectures, including LLaDA-8B and SDXL, and various tasks such as code generation and summarization.
Implications
This research has the potential to enhance the efficiency of inference in large-scale diffusion models, making them more practical for real-time applications. The findings could lead to broader applications in fields requiring rapid generation of text, images, or audio, and may influence future designs of neural architectures that prioritize computational efficiency.
Beyond Compression: Training Latent Representations for Stable Long-Horizon Rollout in Neural Surrogate Solvers
Generative Models
Efficient ML
Theory
- Latent dynamics models (LDMs) can accelerate simulations but struggle with long-horizon stability.
- Instability arises from training focused on reconstruction rather than long-term forecasting.
- Training-level interventions can significantly improve long-horizon rollout performance.
- The proposed methods reduce long-rollout error by ~40% and lower computational costs.
Read more
Beyond Compression: Training Latent Representations for Stable Long-Horizon Rollout in Neural Surrogate Solvers
Summary
This paper addresses the challenges faced by latent neural surrogate solvers (LDMs) in achieving stable long-horizon rollouts for time-dependent physical systems. While LDMs can significantly accelerate simulations by operating in a compressed latent space, they often suffer from instability during long-term predictions due to their training being primarily focused on reconstruction accuracy. The authors identify that this instability arises not from the latent representation itself but from its misalignment with long-horizon forecasting needs. To mitigate this issue, the paper proposes several training-level interventions, including Koopman operator learning, Hamming noise injection, and multi-step rollout fine-tuning. These strategies aim to enhance the latent representations for better predictive performance over extended time horizons. The results demonstrate that these interventions can reduce long-rollout errors by approximately 40% while matching or exceeding the accuracy of full-resolution models on various physics benchmarks. The proposed methods also significantly lower computational costs, requiring two orders of magnitude fewer floating-point operations and half the GPU memory. The findings suggest that neural compression should focus not only on dimensionality reduction but also on restructuring the solution space for stable dynamical evolution, which is crucial for developing reliable neural surrogates in scientific applications.
Methodology
The authors systematically evaluate five training-level interventions aimed at improving the stability of LDMs during long-horizon rollouts. These include spatially structured latent compression, Koopman-inspired constraints, KL divergence regularization, noise injection during training, and recursive training focused on stabilizing performance. The effectiveness of these methods is tested across three distinct dynamic evolution problems: spinodal decomposition, active matter evolution, and crystal-plasticity fatigue simulations.
Results
The proposed training interventions lead to a reduction in long-rollout error by approximately 40%. The models trained with these interventions match or exceed the accuracy of full-resolution models on two physics benchmarks, while also being significantly more efficient, requiring two orders of magnitude fewer floating-point operations and half the GPU memory.
Implications
The findings suggest that improving the training of latent representations can enhance the reliability and efficiency of neural surrogate solvers in scientific applications, particularly in simulations of complex physical systems. This could lead to broader adoption of LDMs in fields requiring high-fidelity simulations, such as materials science and engineering.
UO-FIE: Combining Exact-Label Supervision with Graded Utility for Factivity Inference
NLP
Large Language Models
Efficient ML
- Introduction of UO-FIE, a system that combines exact-label supervision with graded utility for factivity inference.
- Identification of two empirical patterns: majority-label collapse and regression towards nearby intervals.
- Use of a score-matrix-guided categorical learning approach to retain exact-label supervision while addressing class imbalance.
- UO-FIE ranks first in the fine-tuning track of FIE2026, showcasing its effectiveness in utility-aware training.
Read more
UO-FIE: Combining Exact-Label Supervision with Graded Utility for Factivity Inference
Summary
This paper presents UO-FIE, a novel approach for factivity inference in Chinese context-hypothesis pairs, addressing the challenges posed by the Factivity Inference Evaluation 2026 (FIE2026). The task involves classifying pairs into nine ordered factivity intervals, with a unique evaluation metric that rewards both exact predictions and proximity to the correct interval. The authors identify two empirical patterns in preliminary experiments: a tendency for models to predict the dominant class and a regression towards nearby intervals. UO-FIE combines exact-label supervision with graded utility, utilizing a parameter-efficient system that predicts a distribution over the nine classes. The methodology incorporates hard-label supervision, utility-based soft targets, scheduled class weights, and an ordinal loss function. The model is built on Qwen3.5-9B with LoRA, and it achieves first place in the fine-tuning track with a macro utility score of 0.8316, while a separate prompt-based ensemble ranks third in the non-fine-tuning track with a score of 0.8450. The results demonstrate the effectiveness of utility-aware training and the importance of maintaining exact-label supervision while considering the ordinal structure of the task.
Methodology
UO-FIE employs a parameter-efficient architecture based on Qwen3.5-9B, integrating hard-label supervision with utility-based soft targets and an ordinal loss function. The model predicts a distribution over nine classes and utilizes a score matrix to guide learning, accounting for class imbalance and task utility. The training process involves expected-utility decoding and ordinal calibration to optimize decision thresholds.
Results
UO-FIE achieved a macro utility score of 0.8316, ranking first in the fine-tuning track of the FIE2026 evaluation. Additionally, a prompt-based ensemble method achieved a macro utility score of 0.8450, securing third place in the non-fine-tuning track. These results highlight the model's effectiveness in addressing the challenges of factivity inference.
Implications
The findings suggest that combining exact-label supervision with graded utility can significantly enhance performance in tasks involving ordinal classification. This approach may be applicable to other domains requiring nuanced predictions and could inform future research in utility-aligned learning and ordinal classification tasks.
Response-state Learning for Transferable Vibrational Spectroscopic Characterization with Electron Prior
Theory
Graph Learning
Efficient ML
- Introduction of SENK, a framework for accurate vibrational spectral prediction.
- Combines equivariant learning with state-dependent refinement and reliability sensing.
- Outperforms existing models like DetaNet on benchmark datasets.
- Maintains spectral fidelity across small molecules to complex biomolecular systems.
Read more
Response-state Learning for Transferable Vibrational Spectroscopic Characterization with Electron Prior
Summary
This paper introduces the SO(3) Equivariant Neural Kalman Networks (SENK), a novel framework for predicting vibrational spectra that addresses the challenges posed by localized stereoelectronic environments. Traditional vibrational spectral prediction methods struggle with accuracy when external chemical spaces are involved, primarily due to perturbations in intermediate response states. SENK combines an equivariant transformer backbone for learning key vibrational response quantities (Hessian, dipole-derivative, and polarizability-derivative) with an Equivariant Neural Kalman bridge that refines state-dependent predictions and assesses reliability. Additionally, it incorporates an NBO-informed electronic prior pathway that enhances consistency and calibration of spectral predictions. The framework was evaluated on datasets QM9S and QMe14S, demonstrating superior performance compared to DetaNet while maintaining fidelity across a range of molecular systems, including drug-like compounds and biomolecules. SENK effectively identifies and calibrates response states, allowing for reliable vibrational spectroscopy across diverse chemical environments, thus supporting applications in molecular identification and structural analysis.
Methodology
The SENK framework employs an equivariant transformer to map equilibrium geometries to response tensors, followed by an Equivariant Neural Kalman bridge for refining predictions based on state-dependent reliability. An NBO-informed electronic prior pathway is used to guide the calibration of response states, ensuring that local corrections are informed by physical principles.
Results
SENK demonstrated improved accuracy in predicting vibrational spectra across various datasets, outperforming DetaNet on QM9S and QMe14S. The model maintained full-spectrum fidelity and effectively calibrated response states, particularly in complex biomolecular environments, showcasing its robustness and reliability.
Implications
The SENK framework has significant implications for the field of vibrational spectroscopy, enabling more reliable predictions across diverse chemical spaces. This advancement can enhance molecular identification, structural analysis, and applications in drug discovery and materials science.
Diverse Geometries, Frozen Weights: Robust Heterogeneous Treatment-Effect Estimation via Causal Expert Ensembles
Theory
- GeoACE combines multiple causal experts to improve heterogeneous treatment effect estimation.
- The framework utilizes a frozen weight approach to prevent test data leakage.
- OΦ-ACE introduces a new method for creating outcome-free, overlap-aware projections.
- GeoACE outperformed 11 comparators across multiple benchmarks, demonstrating its robustness.
Read more
Diverse Geometries, Frozen Weights: Robust Heterogeneous Treatment-Effect Estimation via Causal Expert Ensembles
Summary
This paper addresses the challenge of estimating heterogeneous treatment effects (HTE) from observational data, where the appropriate inductive bias can vary significantly based on factors such as overlap, treatment imbalance, prognostic structure, and sample size. The authors introduce the Geometry-Diverse Anchor–Correction Expert Ensemble (GeoACE), a framework comprising five causal experts that utilize a common anchor–correction estimator while incorporating complementary geometries that are aware of overlap and guided by outcomes. The ensemble's weights are determined solely from internal validation predictions and are fixed prior to testing, ensuring that test data does not influence the learned weights. A novel expert, OΦ-ACE, is introduced, which constructs an outcome-free, overlap-aware statistical projection to enhance the ensemble's robustness. The performance of GeoACE is evaluated against 11 other methods across eight benchmark protocols, demonstrating its effectiveness in reducing the mean squared error of treatment effect estimates. The results indicate that the five-expert ensemble consistently outperforms simpler models and supports the notion of using diverse expert libraries and leakage-free aggregation as a strategy for robust treatment effect estimation.
Methodology
The GeoACE framework consists of five causal experts that leverage a shared anchor–correction backbone. The ensemble weights are learned from internal validation predictions and frozen before expert refitting and test evaluation. The OΦ-ACE expert provides a unique outcome-free, overlap-aware projection, enhancing the diversity of the ensemble. The performance is assessed using various benchmark protocols, comparing against multiple existing methods.
Results
GeoACE showed significant improvements in mean squared error across seven benchmarks, winning 998 out of 1,225 paired tasks. It ranked first on three benchmarks (IHDP100, IHDPA, IHDPB) and second on another (NEWS), with a minimal difference from the leading method. The ensemble achieved the lowest average rank across benchmarks, although statistical tests indicated no significant differences among methods.
Implications
The findings suggest that employing a diverse set of causal experts and a robust aggregation strategy can enhance the reliability of treatment effect estimates in observational studies. This approach may have applications in various fields, including healthcare, economics, and social sciences, where understanding the impact of interventions is crucial.
Generalized Graph Variational Autoencoders: Bounded Divergences Control Posterior Collapse
Generative Models
Graph Learning
Theory
- Introduction of the Generalized Graph Variational Autoencoder (GGVA) that utilizes bounded divergences.
- Demonstration that the choice of divergence affects the model's ability to retain information.
- Experimental results show GGVA improves node classification performance compared to VGAE.
- Bounded divergences delay but do not prevent posterior collapse in variational models.
Read more
Generalized Graph Variational Autoencoders: Bounded Divergences Control Posterior Collapse
Summary
This paper introduces the Generalized Graph Variational Autoencoder (GGVA), which modifies the variational graph autoencoder (VGAE) by replacing the Kullback-Leibler (KL) divergence with divergences from the Rényi-Tsallis family. The authors argue that the KL divergence, while convenient, is unbounded and can lead to posterior collapse, where the model fails to learn meaningful representations. The GGVA allows for a bounded divergence, specifically using the Tsallis divergence for orders q < 1, which prevents the gradient from vanishing completely. The paper provides a controlled comparison between the GGVA and VGAE, demonstrating that the choice of divergence significantly impacts the retained information in the posterior. Through experiments on various graph datasets, the GGVA shows improvements in retained information and performance on node classification tasks, although it does not achieve better link-prediction accuracy than the VGAE. The findings suggest that while bounded divergences can delay posterior collapse, they do not completely prevent it.
Methodology
The GGVA modifies the standard VGAE by substituting the KL divergence with divergences from the Rényi-Tsallis family, allowing for a parameterized control over the divergence used in the latent penalty. The model retains the same architecture and training protocol, focusing solely on the divergence's impact. Experiments are conducted on ten different graph datasets, analyzing the effects of varying the order q of the divergence on retained posterior information and performance metrics.
Results
The GGVA shows up to a 49× increase in retained posterior information compared to the VGAE across various datasets. In node classification tasks, the GGVA achieves up to +0.14 macro-F1 score improvement over the VGAE on the CiteSeer dataset. However, it does not surpass the link-prediction accuracy of the VGAE on larger real graphs, indicating that while bounded divergences can improve certain aspects of model performance, they do not fully resolve issues related to posterior collapse.
Implications
The findings suggest that using bounded divergences in variational models can enhance the learning of meaningful representations in graph-based data. This approach may be beneficial in applications where maintaining informative latent representations is crucial, such as in social network analysis, citation networks, and other graph-related tasks.
MORE-PLR: multi-output regression employed for partial label ranking
Theory
- Introduces MORE-PLR, a framework leveraging multi-output regression for partial label ranking.
- Transforms incomplete rankings with ties into multivariate regression targets using an encoder.
- Includes a PLR post-hoc layer to convert MOR outputs into bucket orders.
- Demonstrates improved computational efficiency with linear runtime compared to existing methods.
Read more
MORE-PLR: multi-output regression employed for partial label ranking
Summary
This paper presents MORE-PLR, a novel framework that utilizes multi-output regression (MOR) to tackle the partial label ranking (PLR) problem. PLR is a supervised learning scenario that allows for the prediction of bucket orders over a set of labels, accommodating ties in rankings, which traditional label ranking (LR) methods do not support. The authors propose an encoder that transforms incomplete rankings with ties into multivariate regression targets, thus enabling MOR learners to effectively handle PLR tasks. The framework consists of three main components: an encoding module, a base MOR learner, and a PLR post-hoc layer that converts MOR outputs into the desired bucket order. This approach not only generalizes existing PLR methods but also improves computational efficiency, exhibiting linear runtime compared to the quadratic runtime of state-of-the-art pairwise comparison methods. The experimental results demonstrate that MORE-PLR achieves competitive accuracy on benchmark datasets, showcasing the effectiveness of various encoder and post-hoc layer combinations. The paper also extends previous work by introducing additional encoder variants and a new post-hoc layer, enhancing the flexibility and performance of the framework.
Methodology
The methodology involves encoding bucket orders into a structured multivariate representation compatible with MOR learners. The framework is composed of an encoding module, a base MOR learner, and a PLR post-hoc layer that reconstructs the bucket order from the MOR outputs. The paper explores different encoder variants and post-hoc layers to optimize performance.
Results
The MORE-PLR framework achieves satisfactory accuracy on benchmark datasets, with performance varying based on the choice of encoder, MOR learner, and PLR post-hoc layer. The experimental evaluations indicate that the proposed method is competitive with state-of-the-art PLR techniques, particularly in terms of runtime efficiency.
Implications
The implications of this research extend to various applications in preference learning, such as recommender systems, opinion polling, and any domain where qualitative rankings are prevalent. The ability to handle ties in rankings can significantly enhance the performance of models in real-world scenarios where such data is common.
Sample-Weighted End-to-End Trace-Norm Geometry for Multitask Learning
Theory
- Introduces a sample-size-weighted trace norm for measuring multitask complexity.
- Derives the empirical Rademacher complexity for the proposed trace norm.
- Demonstrates that separated measures can lose information about task interactions.
- Shows significant performance improvements in multitask learning through weighted joint nuclear regularization.
Read more
Sample-Weighted End-to-End Trace-Norm Geometry for Multitask Learning
Summary
This paper addresses the complexities of multitask learning by proposing a sample-size-weighted trace norm as a measure of intrinsic multitask complexity. Traditional multitask models often treat shared representations and task-specific outputs separately, which can lead to loss of information regarding their interactions. The author derives the empirical Rademacher complexity for the proposed trace norm and demonstrates its advantages over conventional methods. The study reveals that the sample-size-weighted trace norm captures the end-to-end relationship between task coefficients and input-space predictors more effectively than separated measures. The paper also discusses the geometric implications of the shared representation on the task Gram matrix and introduces a convex regularizer based on the trace norm. Experimental evaluations across multiple unseen datasets show that the proposed weighted joint nuclear regularization significantly improves performance compared to unweighted methods and other regularization techniques. The findings suggest that the proposed approach can enhance generalization in multitask learning settings by better capturing the relationships between tasks.
Methodology
The author employs theoretical analysis to derive the empirical Rademacher complexity of the sample-size-weighted trace norm. The study includes explicit constructions to illustrate the limitations of separated measures and evaluates the proposed regularizer through experiments on multiple datasets, comparing it against traditional methods.
Results
The weighted joint nuclear regularization improved average population excess over unweighted nuclear regularization by 0.00764, with a 95% confidence interval of [0.00465, 0.01110]. Additionally, correct task counts enhanced performance metrics significantly, indicating the effectiveness of the proposed method in various imbalanced settings.
Implications
The findings suggest that using the sample-size-weighted trace norm can lead to better generalization in multitask learning scenarios. This approach may be particularly beneficial in applications where tasks are interrelated and share common representations, such as in multi-label classification or multi-output regression tasks.
Edge AI on Constrained Devices for Binary Sleep-Wake Classification in Dynamic Environments
Multimodal
Efficient ML
Time Series
- Development of an Edge AI system for sleep-wake classification on constrained devices.
- Integration of inertial sensing and visual pose classification to mitigate motion artifacts.
- High accuracy rates achieved in both motion-based detection and pose classification.
- Feasibility of local, privacy-preserving sleep detection in dynamic environments.
Read more
Edge AI on Constrained Devices for Binary Sleep-Wake Classification in Dynamic Environments
Summary
This paper presents an innovative Edge AI-based system designed for detecting sleep and wake states in dynamic, non-stationary mobile environments using resource-constrained embedded hardware. Traditional methods that rely on accelerometer data for activity metrics are often hindered by motion and vibration artifacts, which complicate the accurate classification of sleep states. To overcome these challenges, the authors developed a multimodal pipeline implemented on an ESP32-S3 microcontroller, integrating inertial sensing for head movement analysis with visual pose classification. The system employs a dual-core architecture with FreeRTOS, allowing for parallel execution of real-time data acquisition and on-device inference. The sleep detection process follows a two-stage strategy: first, it detects low movement over a temporal window, followed by visual validation of poses. Experimental results demonstrate high accuracy rates of 96.5% for motion-based detection and 89% for pose classification, indicating robust binary sleep-wake classification capabilities. Field tests validate the system's feasibility in real-world mobile scenarios, showcasing the potential for privacy-preserving, local sleep detection on edge hardware. The study also highlights limitations regarding sensing intrusiveness, dataset scale, and system integration, providing insights into the challenges of deploying reliable sleep-wake detection in practical applications.
Methodology
The authors designed a multimodal pipeline that combines inertial sensing and visual pose classification, implemented on an ESP32-S3 microcontroller. The system utilizes a dual-core architecture with FreeRTOS for real-time data acquisition and inference, employing a two-stage sleep detection strategy that includes low-movement detection followed by visual pose validation.
Results
The system achieved 96.5% accuracy in motion-based sleep detection and 89% accuracy in pose classification. Field tests confirmed the system's effectiveness in real-world mobile scenarios, demonstrating robust binary sleep-wake classification.
Implications
The findings suggest that reliable sleep-wake detection can be achieved on resource-constrained edge devices, paving the way for applications in health monitoring, safety-critical systems, and other areas requiring unobtrusive, long-term monitoring in dynamic environments.
PoEM: Predicting RL Outcomes from Existing Policies
Reinforcement Learning
Generative Models
Multimodal
- PoEM allows for predicting RL outcomes without additional training on new reward functions.
- The framework utilizes existing post-trained models to approximate new policies based on linear combinations of rewards.
- Log-policies from previous RL training span a low-dimensional space, enabling effective policy composition.
- The method shows strong performance across both text and image generation tasks.
Read more
PoEM: Predicting RL Outcomes from Existing Policies
Summary
The paper introduces PoEM, a novel framework designed to predict the outcomes of reinforcement learning (RL) without the need for additional training on new reward functions. The authors address the computational challenges associated with post-training models through RL, which often require extensive resources and can be unstable. PoEM leverages existing post-trained models to approximate the RL policy for a new reward function by utilizing linear combinations of existing rewards. The framework is based on the observation that even when rewards are not linearly related, the log-policies from previous RL training often span a low-rank subspace. By estimating the coefficients for these combinations using regression on reward outcomes, PoEM can effectively simulate RL outcomes. The authors validate their approach across various tasks in both text and image modalities, demonstrating that PoEM can recover significant reward gains compared to directly trained RL policies.
Methodology
PoEM operates by taking a set of existing post-trained models and a new reward function to estimate the target RL policy. The authors derive a closed-form solution for the new policy based on linear combinations of existing log-policies. They use regression techniques to recover the coefficients for these combinations from samples scored under the new reward, allowing for the simulation of RL outcomes without running additional RL training.
Results
Experimental results indicate that PoEM can closely approximate the outcomes of directly trained RL policies on composite rewards, achieving performance that is nearly equivalent to running RL on the target reward. In tests involving diverse public reward models and image generation tasks, PoEM consistently produced results that were closer to the directly trained RL policies than leading single-expert methods.
Implications
The PoEM framework has significant implications for the efficiency of RL training, particularly in scenarios where multiple reward functions need to be considered. It can streamline the process of adapting models to new tasks, reducing computational costs and time while maintaining high performance. This approach could be particularly beneficial in real-world applications where rapid adaptation to changing objectives is necessary.
Spectral Graph Neural Networks with Hermite Polynomials: A Comprehensive Study
Graph Learning
Theory
Optimization
- Introduction of HermNet, a spectral GNN model based on Hermite polynomials.
- HermNet's sparse propagation mechanism does not require eigendecomposition or a learned basis.
- Controlled experiments show HermNet's superiority over matched polynomial-basis alternatives under limited training budgets.
- Curvature regularization improves HermNet's performance, especially with a common penalty.
Read more
Spectral Graph Neural Networks with Hermite Polynomials: A Comprehensive Study
Summary
This paper presents a comprehensive study of spectral graph neural networks (GNNs) utilizing Hermite polynomials, introducing a model named HermNet. HermNet combines a nodewise predictor with normalized Hermite propagation, characterized by its sparse recurrence that avoids the need for eigendecomposition or a learned basis. The study differentiates the basic model from optional enhancements such as coordinate calibration, response normalization, and Gaussian derivative regularization. The authors analyze the optimization behavior of different polynomial bases, including Hermite, under limited training budgets, focusing on factors like spectral signal energy and label sampling. Controlled synthetic experiments reveal that HermNet outperforms other polynomial-basis alternatives in specific regimes, particularly with short training budgets. The paper also discusses how curvature regularization can enhance HermNet's performance. However, longer training periods diminish the advantages of the plain model. Real-data comparisons indicate some accuracy deficits, and further architectural and numerical studies identify limitations. Overall, the analysis clarifies the conditions under which Hermite propagation is beneficial and how various enhancements impact performance.
Methodology
The authors propose HermNet, which employs normalized Hermite propagation and a nodewise predictor. They analyze the model's performance through synthetic experiments, comparing it against other polynomial bases. The study also examines the effects of optional enhancements like calibration and regularization on the model's performance.
Results
The experiments demonstrate that HermNet achieves lower prediction error compared to matched polynomial-basis models, particularly in scenarios with limited training updates. While HermNet shows advantages in short training budgets, these benefits diminish with longer training. Real-data evaluations reveal some accuracy deficits, indicating the need for further architectural and numerical refinements.
Implications
The findings suggest that Hermite polynomials can be effectively utilized in spectral GNNs, providing insights into the conditions under which they outperform traditional polynomial bases. This work may influence future designs of GNN architectures and regularization techniques, particularly in scenarios with constrained training resources.
Beyond Average Safety: Chance-Constrained LLM Fine-tuning
NLP
Large Language Models
Optimization
- Introduces a chance-constrained optimization framework for safety-preserving LLM fine-tuning.
- Develops a differentiable majorization of the empirical chance constraint for tractable optimization.
- Implements a constraint-aware gradient descent method to ensure safety during fine-tuning.
- Demonstrates superior performance in reducing safety regressions compared to traditional methods.
Read more
Beyond Average Safety: Chance-Constrained LLM Fine-tuning
Summary
This paper addresses the challenge of fine-tuning large language models (LLMs) while preserving safety, particularly in safety-critical contexts. The authors critique existing methods that focus on average safety loss, which can obscure severe failures in specific instances. They propose a novel chance-constrained formulation for safety-preserving fine-tuning, which limits the proportion of safety examples that can exceed a specified degradation threshold relative to a reference model. To tackle the discontinuity in the empirical chance constraint, the authors introduce a differentiable majorization of the violation rate, allowing for a tractable conservative constraint. They develop a constraint-aware gradient descent method that modifies the fine-tuning direction minimally to maintain feasibility within the defined safe set. Extensive experiments demonstrate that their approach consistently outperforms existing baselines across various tasks and models, suggesting a shift in perspective towards treating safety preservation as a reliability-constrained optimization problem rather than merely average-risk regularization.
Methodology
The authors formulate safety-preserving fine-tuning as a chance-constrained optimization problem, introducing a differentiable majorization to handle the discontinuous nature of the empirical chance constraint. They develop a constraint-aware gradient descent method that adjusts the fine-tuning gradient to ensure compliance with the safety constraints, thereby maintaining a safe set in parameter space.
Results
The proposed method was tested on three different tasks and models, including Qwen 3.5 and Llama 3.1, showing consistent improvements in fine-tuning performance while significantly reducing the rates of safety regressions compared to standard weighted and constrained baselines.
Implications
This work has significant implications for the development of safer AI systems, particularly in applications where LLMs are deployed in sensitive environments. By framing safety as a reliability constraint, the proposed methodology can enhance the robustness of LLMs against harmful outputs during fine-tuning.
Precise Convergence Speed of Clipped SGD
Optimization
Theory
- Introduces a tightened convergence analysis for Clip-SGD on (L0, L1)-smooth functions.
- Extends the domain of validity for step-size conditions, allowing for larger step sizes.
- Strengthens convergence criteria from minimal to average gradient norms.
- Improves the final achievable loss characterization.
Read more
Precise Convergence Speed of Clipped SGD
Summary
This paper presents a refined convergence analysis of the Clipped Stochastic Gradient Descent (Clip-SGD) algorithm applied to (L0, L1)-smooth functions. The author builds upon previous work by Koloskova et al. (2023) to simplify the proofs and enhance the understanding of convergence behavior. The analysis extends the validity of the step-size condition from η ≤ 1/(9β) to η < 1/β, where β is defined as L0 + cL1, with c being the clipping constant. The paper strengthens the convergence criterion from a minimal gradient norm to an average gradient norm, allowing for a more precise characterization of convergence speed. The final achievable loss is improved from O(min(σ²/c, σ)) to a more accurate 6 min(σ²/c, 3σ). The study emphasizes the importance of (L0, L1)-smooth functions in modeling the behavior of Hessian spectral norms during training, providing a clearer framework for analyzing the performance of Clip-SGD.
Methodology
The paper employs a theoretical approach to analyze the Clip-SGD algorithm, focusing on the properties of (L0, L1)-smooth functions. It utilizes descent conditions and projection operators to derive convergence criteria and loss bounds. The analysis is built on previous work, particularly simplifying the proofs from Koloskova et al. (2023) and leveraging properties of â„“2-projections.
Results
The author establishes a new convergence criterion that allows for a more precise understanding of the convergence speed of Clip-SGD. The results indicate that under the new conditions, the algorithm can achieve a lower final loss and operate effectively with larger step sizes than previously allowed. The paper also provides explicit constants for the convergence rates, enhancing the practical applicability of the findings.
Implications
The findings have significant implications for optimizing the performance of stochastic gradient descent algorithms, particularly in scenarios where gradient clipping is necessary. The improved convergence criteria can guide practitioners in selecting hyperparameters more effectively, potentially leading to faster convergence and better model performance in various machine learning applications.
FlashLoop: Fast and Memory-Efficient Looped Transformers via Lazy Updates
NLP
Large Language Models
Efficient ML
- FlashLoop significantly reduces inference overhead in Looped Transformers.
- The framework achieves up to 1.64× speedup and 6× reduction in KV-cache memory.
- It retains lossless accuracy while improving practical efficiency.
- Key innovations include Token-Sparse Updates, Sparse Attention, and KV-Residual Quantization.
Read more
FlashLoop: Fast and Memory-Efficient Looped Transformers via Lazy Updates
Summary
The paper introduces FlashLoop, a novel inference framework designed to enhance the efficiency of Looped Transformers, which have gained attention for their parameter-efficient approach to increasing computational depth through repeated application of shared Transformer blocks. Despite their potential, traditional Looped Transformers face challenges in practical inference due to increased computational overhead and memory requirements as loop depth increases. The authors identify that much of the additional computation and storage is redundant, as state changes become concentrated on a small subset of tokens, and attention-output differences are dominated by a sparse subset of key columns. FlashLoop addresses these inefficiencies by implementing three key strategies: Token-Sparse Updates, which minimize redundant updates by reusing states of converged tokens; Sparse Attention, which selectively focuses on important keys; and KV-Residual Quantization, which compresses the KV cache into a quantized base state with low-bit updates. The framework is training-free and significantly reduces both computation and memory usage, making it feasible to scale Looped Transformers to greater depths and longer contexts while maintaining accuracy.
Methodology
The authors systematically analyze the dynamics of Looped Transformers to identify redundancy in computation and memory usage. They propose FlashLoop, which utilizes Token-Sparse Updates to minimize unnecessary state updates, Sparse Attention to focus on significant keys, and KV-Residual Quantization to compress the KV cache. These methods are implemented efficiently on GPUs through optimized kernels and hardware-aware pipelining.
Results
FlashLoop demonstrates substantial improvements over traditional Looped Transformers, achieving up to 1.64× end-to-end speedup and reducing KV-cache memory requirements by up to 6×, while maintaining lossless accuracy across various models.
Implications
The advancements presented in FlashLoop could lead to more efficient deployment of Looped Transformers in real-world applications, particularly in scenarios requiring deep computation with limited resources. This could enhance the practicality of using such models in NLP tasks and other domains where memory and speed are critical.
A Contraction Framework for Stochastic Operators with Bootstrapping: Application to TD Learning
Reinforcement Learning
Theory
Optimization
- Introduces a contraction framework for stochastic operators in TD learning.
- Establishes convergence without assuming linearity and for arbitrary target-update periods.
- Derives finite-time bounds for i.i.d. samples in TD learning.
- Demonstrates that convergence rates can be predicted and validated through simulations.
Read more
A Contraction Framework for Stochastic Operators with Bootstrapping: Application to TD Learning
Summary
This paper introduces a contraction framework for analyzing the convergence of temporal-difference (TD) learning algorithms that utilize bootstrapping. Traditional convergence guarantees for TD learning often depend on specific structures of the update process, such as linear approximations or gradient-based methods, and assume bounded sampling errors. The authors propose a more general approach by modeling the sampled updates as stochastic operators, allowing for a broader analysis that does not require gradient structures and accommodates growing sampling errors. They derive finite-time convergence bounds for TD learning with independent and identically distributed (i.i.d.) samples and arbitrary target-update periods. The results indicate that the iterates converge geometrically in root mean square to a neighborhood around the fixed point, contingent on the sensitivity to the frozen target being less than the contraction slack of the inner map. The framework encompasses existing deterministic and stochastic convergence results as special cases and is validated through simulations that align with the predicted contraction rates and error floor scaling with step size.
Methodology
The authors formulate the TD learning algorithm from an operator-theoretic perspective, treating the update process as a stochastic operator. They analyze the conditions for convergence using contraction arguments, allowing for flexibility in the update structure and accommodating sampling errors. The mathematical framework is applied to derive convergence results for TD learning with bootstrapped targets.
Results
The study establishes that the iterates of the TD learning algorithm converge geometrically in root mean square to a region around the fixed point, provided certain conditions regarding the sensitivity to the frozen target and contraction slack are met. The framework successfully reproduces existing results as special cases and aligns with empirical simulations.
Implications
This framework has significant implications for the analysis and implementation of TD learning and other iterative algorithms that utilize bootstrapping. It provides a more robust theoretical foundation for understanding convergence in complex scenarios, potentially improving the performance and reliability of reinforcement learning applications in various domains.
Let Training Guide Selection: Online Synthetic Data Filtering via Real-Anchored Utility
Computer Vision
NLP
Large Language Models
- FROST is a model- and task-agnostic framework for online synthetic data filtering.
- It estimates synthetic data utility through gradient feedback anchored in real training data.
- FROST improves real-task performance while reducing the amount of synthetic training data needed.
- The framework demonstrated significant performance gains in a large-scale industrial application.
Read more
Let Training Guide Selection: Online Synthetic Data Filtering via Real-Anchored Utility
Summary
The paper introduces FROST, an innovative online framework for filtering synthetic data during training, addressing the challenges posed by noise and distribution mismatch in synthetic datasets. Traditional methods often focus on fidelity or diversity, neglecting the evolving needs of the learner. FROST leverages gradient feedback from real training data to estimate the utility of synthetic samples, dynamically calibrating batch utility based on recent training history. This approach allows for selective filtering of synthetic data, retaining only the most beneficial samples. Experiments demonstrate that FROST can filter out 20-30% of synthetic data while enhancing real-task performance across benchmarks in image classification and LLM fine-tuning for text-to-SQL. Additionally, its application in a large-scale industrial ads re-ranking system yielded significant performance improvements over existing production baselines, showcasing its effectiveness and generalizability.
Methodology
FROST employs an online filtering mechanism that utilizes gradient feedback from real training data to assess the utility of synthetic samples. It maintains an exponential moving average of gradients from real training batches and scores synthetic samples based on their alignment with this reference. The framework aggregates sample utilities and calibrates batch-level filtering, applying sample-level filtering selectively to batches that fall outside a defined utility band.
Results
FROST outperformed existing synthetic data selection methods, achieving accuracy improvements of 0.58-1.14% in image classification and 0.9% in text-to-SQL tasks while using 20-30% fewer synthetic samples. In a large-scale industrial ads re-ranking system, FROST improved Normalized Entropy (NE) from 0.211% to 0.096%, significantly surpassing the 0.02% gain threshold considered significant in prior studies.
Implications
The findings suggest that FROST can effectively enhance the training process in various machine learning applications by optimizing the use of synthetic data. Its ability to dynamically filter data based on real-time training needs could lead to more efficient training processes and better model performance in scenarios where real data is scarce or expensive to obtain.
Federated Learning of AnDE Classifiers
Federated Learning
- Introduces FedAnDE, a federated learning framework for AnDE classifiers.
- Supports any dependency order n, enhancing model expressiveness.
- Demonstrates improved privacy by sharing only model weights instead of sensitive data.
- Experimental results show superior performance of discriminative models over Naive Bayes.
Read more
Federated Learning of AnDE Classifiers
Summary
This paper introduces a federated learning framework for training Averaged n-Dependence Estimators (AnDE) in distributed environments, addressing the need for privacy-preserving machine learning methods. The proposed method, named FedAnDE, allows for local learning of model weights while aggregating them globally, supporting any dependency order n. This approach enhances privacy by avoiding the transmission of semantically meaningful parameters. The authors also federate generative AnDE models as a comparative baseline, applying optional differential privacy to the aggregation of probability tables. Experiments conducted on 12 discrete datasets demonstrate that discriminative models with n ≥ 1 consistently outperform the federated Naive Bayes model (n = 0), and that the privacy-preserving aggregation method is effective with minimal accuracy loss. The findings establish federated AnDE as a viable framework for privacy-preserving probabilistic modeling in modern federated learning settings.
Methodology
The FedAnDE framework involves local training of weighted AnDE models on client datasets, where clients share only the discriminative weights with a central server. The server aggregates these weights across multiple clients, allowing for a collaborative model update while keeping generative parameters private. The methodology includes federated rounds of communication and optimization using the L-BFGS-B algorithm to maximize conditional log-likelihood.
Results
The experimental evaluation on 12 categorical datasets shows that increasing the dependency order n leads to improved accuracy in federated discriminative models. The generative baseline models experience greater degradation under privacy constraints as the number of clients increases, highlighting the advantages of the proposed discriminative approach.
Implications
The findings suggest that federated learning can effectively incorporate probabilistic models, providing a privacy-preserving alternative to traditional methods. This has significant implications for applications in sensitive domains such as healthcare and finance, where data privacy is paramount.
SMILESGNN: Interpretable Clinical Toxicity Prediction via SMILES-Graph Cross-Attention Fusion
Graph Learning
Multimodal
Interpretability
- Introduction of SMILESGNN, a multimodal architecture for toxicity prediction that combines SMILES and graph representations.
- Achieves high predictive performance on ClinTox with AUC-ROC of 0.987 and F1 score of 0.906 using only 0.4M parameters.
- Facilitates interpretable predictions through GNNExplainer, allowing identification of substructures linked to toxicity.
- SMILESGNN-PT variant shows competitive performance on Tox21, matching results of larger baseline models.
Read more
SMILESGNN: Interpretable Clinical Toxicity Prediction via SMILES-Graph Cross-Attention Fusion
Summary
The paper presents SMILESGNN, a novel multimodal architecture designed for predicting clinical toxicity of drugs by integrating SMILES (Simplified Molecular Input Line Entry System) representations and graph-based representations through a cross-attention mechanism. This approach addresses critical challenges in drug toxicity prediction, including severe class imbalance, scaffold-based generalization, and the need for interpretable predictions. The architecture consists of a SMILES Transformer encoder and a GATv2 graph encoder, allowing for the extraction of complementary features from both modalities. The authors also introduce SMILESGNN-PT, a variant that utilizes a pretrained ChemBERTa-2 backbone. The model's design facilitates GNNExplainer-based analysis, enabling the identification of toxicophore-like substructures that contribute to toxicity predictions. The performance of SMILESGNN on the ClinTox dataset yielded an AUC-ROC of 0.987 and an F1 score of 0.906, demonstrating its effectiveness with only 0.4 million parameters. Additionally, SMILESGNN-PT achieved competitive results on the Tox21 dataset, indicating its robustness across different tasks. The findings suggest that cross-attention fusion is a viable alternative to concatenation methods, maintaining predictive performance while enhancing interpretability.
Methodology
The methodology involves a dual-pathway architecture that fuses a SMILES Transformer encoder with a GATv2 graph encoder using cross-attention. The model is trained with focal loss to mitigate class imbalance. Data processing includes custom tokenization for SMILES sequences and feature extraction for molecular graphs, ensuring both modalities contribute to the prediction task.
Results
On the ClinTox dataset, SMILESGNN achieved an AUC-ROC of 0.987 ± 0.012 and an F1 score of 0.906 ± 0.039, outperforming single-modality baselines. The SMILESGNN-PT variant achieved a mean AUC-ROC of 0.750 ± 0.002 on the Tox21 dataset, comparable to other strong models.
Implications
The findings suggest that SMILESGNN can significantly improve the early prediction of drug toxicity, potentially reducing late-stage attrition in drug development. The model's interpretability features may aid toxicologists in understanding the underlying causes of toxicity, thereby enhancing drug safety assessments.
Bandit Multiclass PAC Learning: Corrected Lower Bounds, Exact Families, and a Confidence Direct-Sum Phenomenon
Theory
- The previously published lower bound for sample complexity in bandit multiclass PAC learning is shown to be incorrect.
- A new anchored dimension, aBDS(H), is introduced to provide a more accurate characterization of sample complexity.
- Upper bounds are established that do not depend on the number of labels, K, improving previous analyses.
- Exact characterizations for specific families of classes are provided, enhancing understanding of sample complexity.
Read more
Bandit Multiclass PAC Learning: Corrected Lower Bounds, Exact Families, and a Confidence Direct-Sum Phenomenon
Summary
This paper investigates the sample complexity of realizable multiclass PAC learning with bandit feedback, where a learner predicts labels based on observed instances and receives feedback only on the correctness of those predictions. The author critiques previous work that established upper and lower bounds on sample complexity, specifically addressing the incorrectness of the published lower bound and proposing a new anchored dimension, aBDS(H), to refine the analysis. The paper presents a corrected lower bound that incorporates this new dimension and demonstrates that the sample complexity can be significantly smaller than previously stated. Additionally, the author provides upper bounds that eliminate dependence on the number of labels, K, and introduces a fiberization lemma to strengthen the analysis. The findings include exact characterizations for specific families of classes and a negative answer to an open question regarding sample complexity, revealing a confidence direct-sum phenomenon that indicates intrinsic limitations in the sample complexity of certain classes. Overall, the work reshapes the understanding of sample complexity in bandit multiclass PAC learning.
Methodology
The author employs theoretical analysis to critique existing bounds on sample complexity, introduces a new dimension for better characterization, and uses lemmas to derive upper bounds. The study includes constructing specific classes to demonstrate the findings and analyzing their sample complexities.
Results
The paper presents a corrected lower bound for sample complexity that is significantly smaller than previously stated, establishes new upper bounds that eliminate dependence on K, and provides exact characterizations for classes with specific properties. The findings also include a negative answer to an open question regarding uniform-constant sample complexity, highlighting a confidence direct-sum phenomenon.
Implications
The results have implications for the design of learning algorithms in multiclass settings, particularly in scenarios where feedback is limited. The refined understanding of sample complexity can guide future research and algorithm development in bandit learning frameworks.
An Agnostic Sample Compression Scheme for Squared Loss of Near-Linear Size in the Fat-Shattering Dimension
Theory
- Introduces an agnostic sample compression scheme for empirical squared loss with near-linear size.
- Eliminates the dual fat-shattering dimension factor present in previous constructions.
- Achieves compression size independent of sample size, focusing on a fraction of sample points.
- Utilizes synthetic labels and quantized side information for effective reconstruction.
Read more
An Agnostic Sample Compression Scheme for Squared Loss of Near-Linear Size in the Fat-Shattering Dimension
Summary
This paper presents a novel agnostic sample compression scheme for empirical squared loss applicable to any function class F within the range [0, 1]X. The proposed scheme addresses an open problem posed by Attias et al. (2024) regarding the existence of an α-approximate agnostic ℓ2 compression scheme that is independent of sample size. The key contribution is a compression scheme that retains at most O(fat(F, c′α) · log3(2/α)) original labeled examples and auxiliary bits, effectively eliminating the previously required dual fat-shattering dimension factor. The methodology involves a unique approach where the scheme targets only a (1 − ε)-fraction of sample points, leveraging Kegl’s margin bound to achieve a number of rounds that is independent of the sample size. The reconstruction of the function is facilitated by transmitting synthetic labels through quantized side-information bits attached to the stored examples. The results demonstrate that the proposed scheme meets the requirements of the open problem, providing a significant improvement over existing methods by removing the dual dimension dependency and achieving a compression size that is polynomially logarithmic in terms of the accuracy parameter α.
Methodology
The methodology involves constructing a compression scheme that targets a subset of sample points to ensure average-loss guarantees while avoiding the need for sparsification. The scheme employs Kegl’s margin bound to determine the number of rounds required for boosting, which is independent of the sample size. The reconstruction process uses synthetic labels derived from a near-minimizer of the empirical loss, transmitted alongside the original labeled examples as quantized side information.
Results
The main result confirms the existence of a deterministic agnostic sample compression scheme for empirical squared loss that achieves a total size of at most C d log3(2/α), independent of the sample length m. This result resolves a significant open problem in the literature and provides a worst-case exponential improvement over prior methods by removing the dual dimension factor.
Implications
The findings have significant implications for the field of machine learning, particularly in regression tasks where sample compression is critical. The proposed scheme can enhance the efficiency of learning algorithms by reducing the amount of data needed for training while maintaining accuracy, potentially leading to advancements in various applications such as predictive modeling and data-driven decision-making.
Downside-Controlled Online Forecast Combination under Delayed and Revised Outcomes
Time Series
- Introduces a downside-controlled approach to forecast combination that minimizes the risk of worsening predictions.
- Combines frozen forecasters with static and online correctors to adapt to changing error patterns.
- Achieves significant improvements in forecasting accuracy across multiple benchmarks and real-world applications.
- Demonstrates the effectiveness of the method in day-ahead load forecasting for European transmission system operators.
Read more
Downside-Controlled Online Forecast Combination under Delayed and Revised Outcomes
Summary
This paper addresses the challenge of improving forecasts from frozen models, particularly in scenarios where retraining is not feasible. The authors propose a novel approach that combines a frozen forecaster with both static and online correctors, focusing on downside control to ensure that the corrected forecasts do not significantly worsen compared to the original forecasts. The methodology involves using only losses that mature after the forecast horizon, allowing for a more accurate adjustment of predictions. The proposed method is evaluated across seven benchmarks and four base models, including foundation models, demonstrating that the worst deterioration in performance is only 0.15%, while gains can reach up to 11.5%. The approach is particularly effective in improving day-ahead load forecasts for European bidding zones, where it consistently lowers mean mean squared error (MSE) compared to single correctors. The findings highlight the importance of combining different types of correctors to adapt to changing error patterns and improve forecasting accuracy in real-time applications.
Methodology
The authors employ a combination of a frozen forecaster, a static corrector, and an online corrector, using a gate mechanism to adjust weights based on matured losses. The method focuses on downside control, ensuring that the corrected forecasts do not significantly deviate from the original forecasts. The approach is evaluated using various datasets and models, including both traditional and foundation models.
Results
The proposed method shows a worst-case deterioration of only 0.15% across 28 dataset-model pairs, with potential gains reaching 11.5%. In the context of day-ahead load forecasting for seven European bidding zones, the method consistently lowers mean MSE, outperforming single correctors that can increase MSE by up to 102%.
Implications
The findings suggest that the proposed downside-controlled forecast combination approach can be effectively applied in real-time forecasting scenarios, particularly in regulated industries where retraining models is not feasible. This has implications for improving the accuracy of forecasts in various applications, including energy load forecasting and other time-sensitive predictions.
The Mechanics of Delta Learning: Target Design for Generalizable Scientific Machine Learning
Graph Learning
- Residual scale alone is not a sufficient indicator of learnability in delta learning.
- Complex local descriptor baselines can yield small but rough residuals that hinder learning.
- Semi-empirical baselines improve both target scale and normalized roughness, enhancing prediction accuracy.
- The introduction of DIQR provides a pre-training diagnostic for assessing target learnability.
Read more
The Mechanics of Delta Learning: Target Design for Generalizable Scientific Machine Learning
Summary
This paper explores the concept of delta learning in scientific machine learning, particularly focusing on the design of targets to enhance model generalizability. The authors argue that relying solely on residual scale as a heuristic for learnability is inadequate. Through experiments with molecular graph neural networks (GNNs) on total energy targets, they demonstrate that complex local descriptor baselines can produce small residuals that are difficult to learn due to increased roughness in the model's feature space. In contrast, semi-empirical baselines reduce both the scale and roughness of targets, leading to improved predictions both in-domain and out-of-domain. The authors introduce a new diagnostic metric, scale-normalized graph Dirichlet roughness (DIQR), to assess residual learnability prior to training. Their findings suggest that target design should consider both target scale and smoothness, alongside model architecture, to optimize scientific machine learning outcomes.
Methodology
The authors evaluate molecular graph neural networks using a hierarchy of baselines with varying physical priors and accuracies. They analyze the impact of target design on model performance, employing metrics such as normalized mean absolute error (n-MAE) and the newly introduced DIQR to assess target smoothness and learnability. The study involves testing different baseline models and comparing their effects on prediction accuracy and generalizability.
Results
The results indicate that replacing linear descriptor baselines with more accurate multi-layer perceptron (MLP) fits leads to smaller residuals that are harder to learn. Conversely, using the xTB baseline reduces both target scale and normalized roughness, resulting in improved in-domain accuracy and out-of-distribution transfer. The findings highlight that delta learning should be viewed as a two-dimensional target-design challenge, emphasizing the importance of both residual scale and representation-space smoothness.
Implications
The insights from this study could significantly influence the design of machine learning models in scientific domains, particularly in materials discovery and molecular simulations. By emphasizing the importance of target design, researchers can develop more effective models that generalize better across different datasets and applications.
Lightweight Probabilistic Downscaling from a Deterministic Base Model
Efficient ML
- Introduction of lightweight probabilistic downscaling models using a modified U-Net architecture.
- Implementation of a two-stage training curriculum combining deterministic pretraining with probabilistic tuning.
- Achieved state-of-the-art RMSE performance on CORDEX-ML-Bench for temperature and precipitation.
- Utilization of MC dropout and block-sparse attention to enhance model efficiency and accuracy.
Read more
Lightweight Probabilistic Downscaling from a Deterministic Base Model
Summary
This paper addresses the challenge of climate data downscaling, which involves increasing the spatial resolution of climate data derived from coarse global models. The authors introduce a family of lightweight probabilistic machine learning models based on a modified U-Net architecture, adapting recent advancements in training methods from weather forecasting to the downscaling domain. They implement a two-stage training curriculum that combines deterministic pretraining with probabilistic tuning, which has shown to outperform existing state-of-the-art methods in terms of root mean square error (RMSE) on the CORDEX-ML-Bench suite, specifically for daily maximum temperature and precipitation across three regions: the Alps, New Zealand, and South Africa. The study highlights the effectiveness of using Monte Carlo (MC) dropout and block-sparse attention mechanisms, achieving a new benchmark in RMSE while maintaining computational efficiency suitable for single GPU training. The findings suggest that the proposed models can significantly enhance the accessibility and applicability of high-resolution climate data for various societal needs, such as agricultural planning and infrastructure resilience.
Methodology
The authors adapted a U-Net architecture for downscaling and employed a two-stage training approach. The first stage involved deterministic pretraining to minimize mean absolute error (MAE), followed by probabilistic tuning to minimize the Continuous Ranked Probability Score (CRPS) using Monte Carlo (MC) dropout. They also explored different stochastic mechanisms and attention variants to optimize performance.
Results
The proposed models set a new state-of-the-art for RMSE on the CORDEX-ML-Bench suite for both temperature and precipitation, outperforming existing methods while being lightweight enough to train on a single GPU. The combination of MC dropout and block-sparse attention was particularly effective.
Implications
The advancements in lightweight probabilistic downscaling models can significantly improve the accessibility of high-resolution climate data, which is crucial for informed decision-making in sectors like agriculture, disaster preparedness, and climate adaptation strategies.
Beyond Feature Reliability: Repeat-Informed Multifractal Curve Regression for Brain-Age Prediction
Time Series
- Introduction of RMCR, a new framework for brain-age prediction using multifractal analysis.
- RMCR improves prediction accuracy by jointly modeling curve structures and repeat-scan variability.
- Significant reductions in MAE and within-visit repeat differences compared to traditional methods.
- Emphasis on the importance of feature interactions over individual feature reliability.
Read more
Beyond Feature Reliability: Repeat-Informed Multifractal Curve Regression for Brain-Age Prediction
Summary
This paper presents a novel approach for predicting brain age from resting-state fMRI data, addressing the limitations of existing methods that focus on individual feature reliability. The authors introduce Repeat-informed Multifractal Curve Regression (RMCR), which jointly models multifractal curve structures and repeat-scan variability to enhance prediction accuracy and consistency. RMCR effectively learns stable age-predictive patterns by separating second-order scaling from curve shape and applying a repeat-informed penalty that discourages unstable weight changes across fluctuation orders. The method leverages repeated scans during training to estimate joint feature variability, allowing for improved predictions from a single scan during inference. The results demonstrate that RMCR significantly reduces mean absolute error (MAE) compared to a baseline ridge regression approach, achieving a 6.1% reduction on the HCP-A dataset and a 7.9% reduction on the external Cam-CAN cohort, while also lowering within-visit repeat absolute differences by 18.5% on HCP-A. This advancement highlights the importance of considering the joint behavior of features in predictive modeling for brain age assessment.
Methodology
The RMCR framework utilizes multifractal detrended fluctuation analysis (MF-DFA) to extract multifractal curves from resting-state fMRI data. It separates second-order scaling from curve shape and applies a repeat-informed penalty during regression to stabilize predictions across fluctuation orders. The model is trained on repeated scans to capture joint feature variability, allowing for effective age prediction from a single scan.
Results
RMCR achieved a 6.1% reduction in mean absolute error (MAE) on the HCP-A dataset and a 7.9% reduction on the Cam-CAN cohort compared to a matched run-level ridge baseline. Additionally, it lowered the within-visit repeat absolute difference by 18.5% on the HCP-A dataset, demonstrating improved prediction stability and accuracy.
Implications
The findings suggest that RMCR could be a valuable tool for researchers and clinicians in assessing brain age and understanding age-related changes in brain function. The methodology may also be applicable to other domains where prediction stability is crucial, potentially influencing future studies in neuroimaging and related fields.
On the second-order optimization for spiking neural networks
Optimization
Efficient ML
Theory
- Introduction of SpiKFAX, a second-order optimization method for SNNs.
- Development of a Kronecker-factored approximation of the Fisher information matrix tailored for SNNs.
- Empirical results show consistent improvements in accuracy and stability over existing optimizers.
- Addresses the limitations of first-order optimizers in the context of SNNs' sharp loss landscapes.
Read more
On the second-order optimization for spiking neural networks
Summary
This paper addresses the challenges associated with training Spiking Neural Networks (SNNs), which are known for their energy efficiency but suffer from a sharp loss landscape due to non-differentiable spiking activations. The authors introduce SpiKFAX, a second-order optimization method that utilizes a Kronecker-factored approximation of the Fisher information matrix tailored for the unique dynamics of SNNs. This method aims to capture second-order curvature information that is often overlooked by first-order optimizers like Adam. The paper presents empirical evaluations across five different architectures and seven datasets, demonstrating that SpiKFAX significantly enhances both test accuracy and training stability compared to traditional optimizers. The findings suggest that incorporating second-order information can lead to more effective training of SNNs, bridging the gap between second-order optimization techniques and SNN-specific training methods.
Methodology
The authors propose SpiKFAX, which approximates the Fisher information matrix using a Kronecker-factored approach. This method allows for efficient computation while accounting for the time-recurrent and sparse nature of SNNs. The optimization process involves minimizing a non-convex, non-differentiable loss function through preconditioned updates that incorporate curvature information.
Results
The empirical evaluation of SpiKFAX across five model architectures and seven datasets indicates that it consistently outperforms traditional optimizers like SGD, Adam, and AdamW in terms of both accuracy and training stability, particularly in the context of SNNs.
Implications
The findings suggest that second-order optimization methods can significantly improve the training of SNNs, potentially leading to more robust and efficient neural network models for various applications, including neuromorphic computing and energy-efficient AI systems.
RLVR landscapes for iterated multiplications can be benign: Insights from spin-glass theory
Reinforcement Learning
Theory
Large Language Models
- The RLVR landscape for certain algorithmic tasks is benign, with no local minima trapping training.
- Challenges in RLVR arise from diffusive barriers and gradient estimation errors rather than landscape ruggedness.
- Entropy regulation can help mitigate obstacles in RLVR optimization.
- Transformers can effectively learn complex tasks when strongly regulated and provided with a suitable curriculum.
Read more
RLVR landscapes for iterated multiplications can be benign: Insights from spin-glass theory
Summary
This paper investigates the optimization landscape of Reinforcement Learning with Verifiable Rewards (RLVR) applied to algorithmic tasks, specifically iterated group and quasigroup multiplications. The authors establish a mapping between RLVR and spin-glass models, allowing for a rigorous characterization of the optimization landscape. They demonstrate that for a wide range of models with uncorrelated inputs, the RLVR landscape is benign, lacking local minima that could hinder training. Instead, challenges arise from diffusive barriers and gradient estimation errors. The study shows that these obstacles can be mitigated through appropriate entropy regulation. Experimental results indicate that a transformer model trained from scratch using last-token rewards can successfully learn to perform iterated non-Abelian group multiplications, supporting the theoretical findings.
Methodology
The authors mapped entropy-regularized RLVR over myopic tabular policies to an energy-based spin-glass model. They analyzed the landscape using theoretical proofs and numerical experiments, focusing on the behavior of RLVR in the context of iterated multiplications.
Results
The study found that the RLVR landscape is benign for simple untied models, with no suboptimal local maxima. It established that a Monte Carlo sampler could solve these models without a curriculum. However, when weight tying and input correlations were introduced, local maxima appeared. The gap between RLVR optimization and spin-glass model performance was identified, with solutions proposed to close this gap through local policy resets and strong regulators.
Implications
The findings suggest that RLVR can be effectively applied to complex reasoning tasks without being hindered by local minima, provided that appropriate strategies are employed. This has implications for the design of reinforcement learning algorithms and their application in training large language models and other AI systems.
Common Covariance Geometry and Certification for Brownian Kernel Ladders
Theory
Optimization
Efficient ML
- Introduction of minimum-trace common covariance for Brownian kernel ladders.
- Development of statistical, approximation-theoretic, and computational consequences.
- Derivation of a universal Gaussian-complexity bound and exact empirical Kolmogorov-width formula.
- Establishment of deterministic depth laws and conditional Gaussian reverses.
Read more
Common Covariance Geometry and Certification for Brownian Kernel Ladders
Summary
This paper introduces a novel approach to the problem of adaptive kernel methods in machine learning, specifically focusing on Brownian kernel ladders. The author presents the concept of minimum-trace common covariance, which dominates the unrestricted empirical union generated by these kernels. This approach leads to significant advancements in statistical theory, approximation methods, and computational techniques. The paper derives exact formulations for covariance values using absolutely two-summing operators and covariance-dominated multipliers, establishing a universal Gaussian-complexity bound. Furthermore, it introduces a closed last-layer Dirac-trace reduction and a signed Brownian threshold representation, transforming the covariance problem into geometrical frameworks such as threshold, graph-coarea, and effective-resistance geometry. The findings yield deterministic depth laws, conditional Gaussian reverses, and an exact empirical Kolmogorov-width formula that approximates the complete adaptive ball. The paper also explores finite contact, active semidefinite programs, and convex resistance-design relaxations to provide lower and upper certificates for covariance. A practical illustration using a finite covariance-indexed Brownian path highlights the distinction between successful covariance certification and predictive selection, revealing that while all path certificates succeed, the predictive study may overlook certain criteria. Overall, this work identifies a finite-dimensional covariance object that connects unrestricted kernel adaptation, Gaussian geometry, common subspaces, and certifiable computation.
Methodology
The author employs a combination of theoretical frameworks including absolutely two-summing operators, covariance-dominated multipliers, and geometric transformations to analyze the covariance properties of adaptive kernel methods. The paper also utilizes semidefinite programming and convex resistance-design techniques to derive certificates for covariance.
Results
The paper successfully establishes a minimum-trace common covariance that dominates the empirical unions of Brownian kernel ladders. It provides a universal Gaussian-complexity bound and presents an exact empirical Kolmogorov-width formula. The results demonstrate the effectiveness of the proposed methods in achieving covariance certification and highlight the limitations of predictive selection in certain scenarios.
Implications
The findings have significant implications for the development of adaptive kernel methods in machine learning, particularly in enhancing the understanding of covariance structures and improving the reliability of predictive models. The techniques introduced could be applied in various domains requiring robust statistical methods and efficient computation.
Leakage-Safe Machine Learning for Hydrogen Embrittlement Detection in 316L Stainless Steel: A Region-Held-Out Evaluation of Texture and Deep Features in SEM Micrographs
Computer Vision
- Proposes a region-held-out evaluation protocol to prevent data leakage in machine learning models for HE detection.
- Demonstrates that simpler texture-based models can outperform complex deep learning models in specific applications.
- Achieves significant classification metrics, indicating the effectiveness of the proposed methodology.
- Highlights the importance of robust validation protocols in machine learning for materials science applications.
Read more
Leakage-Safe Machine Learning for Hydrogen Embrittlement Detection in 316L Stainless Steel: A Region-Held-Out Evaluation of Texture and Deep Features in SEM Micrographs
Summary
This paper addresses the challenge of detecting hydrogen embrittlement (HE) in 316L stainless steel using machine learning (ML) techniques applied to scanning electron microscopy (SEM) images. Traditional ML evaluations often suffer from data leakage due to image-level splits, where multiple images from the same specimen can lead to overfitting. To mitigate this, the authors propose a region-held-out evaluation protocol, specifically Leave-One-Region-Out (LORO) cross-validation, which isolates distinct microstructural regions during training and testing. The study compares various feature-classifier combinations, including local binary patterns (LBP), grey-level co-occurrence matrices (GLCM), and deep learning embeddings from a convolutional neural network (CNN). The results indicate that the simplest model, LBP combined with a support vector machine (SVM), outperformed more complex models, achieving a balanced accuracy of 0.79, with a hydrogen recall of 0.69 and precision of 0.82. The findings emphasize the importance of avoiding data leakage and demonstrate that texture descriptors can effectively identify hydrogen-charging signatures in SEM micrographs, even with limited samples. This methodology can be extended to larger studies on HE detection in other alloy systems.
Methodology
The authors implemented a region-held-out evaluation protocol using Leave-One-Region-Out (LORO) cross-validation on SEM micrographs of 316L stainless steel. They compared six feature-classifier combinations, including local binary patterns (LBP), grey-level co-occurrence matrices (GLCM), and self-supervised convolutional embeddings from a CNN, to assess their performance in classifying as-received and hydrogen-charged images.
Results
The LBP combined with SVM model achieved the highest balanced accuracy of 0.79, with a hydrogen recall of 0.69 and precision of 0.82. A permutation test confirmed the statistical significance of the results (p = 0.008), indicating that the model's performance was not due to chance. Grad-CAM maps from the CNN highlighted localized features associated with hydrogen-induced changes.
Implications
The findings suggest that a leakage-safe approach to machine learning can enhance the reliability of automated diagnostics for hydrogen embrittlement in structural materials. This methodology can be adapted for broader applications in materials science, particularly in assessing the integrity of alloys used in hydrogen infrastructure.
SwitchPFN: Shared Switching Dynamics for Frozen In-Context Time Series Classification
Time Series
- SwitchPFN addresses the challenges of temporal order preservation and feature consistency in time series classification.
- The method utilizes a shared projection and regime codebook to enhance comparability of features across sequences.
- SwitchPFN achieves state-of-the-art performance on multiple benchmark datasets, significantly outperforming existing models.
- Ablation studies confirm the importance of the representation design choices made in SwitchPFN.
Read more
SwitchPFN: Shared Switching Dynamics for Frozen In-Context Time Series Classification
Summary
The paper introduces SwitchPFN, a novel approach for time series classification using tabular foundation models (TFMs). It addresses two significant challenges in existing time series representations: the loss of temporal order during global aggregation and the inconsistency of feature meanings across sequences due to independently fitted coordinate systems. SwitchPFN learns a shared projection and regime codebook from training sequences, allowing local dynamic operators and transition features to be directly comparable across samples. The proposed method captures local temporal transitions while maintaining a consistent feature definition across different time series. The authors demonstrate that SwitchPFN outperforms existing methods, achieving a mean accuracy of 82.63% on eight UEA datasets, which is 3.54 percentage points higher than the next best model. The paper includes ablation studies and sensitivity analyses to evaluate the representation's components and its performance with limited labeled data, highlighting its robustness and efficiency in time series classification tasks.
Methodology
SwitchPFN maps training sequences and queries to a shared feature space using a common coordinate system. It combines local dynamics, class residuals, ordered paths, and signal summaries into a fixed-width table for the TabPFN framework. The shared projection and regime codebook ensure that features maintain consistent meanings across different sequences, allowing for effective classification.
Results
SwitchPFN achieved a mean test accuracy of 82.63% across eight UEA datasets, outperforming the strongest baseline by 4.47%. The results were supported by comprehensive ablation studies and analyses of parameter sensitivity and label efficiency.
Implications
The findings suggest that SwitchPFN can be effectively applied to various time series classification tasks in fields such as healthcare (e.g., ECG analysis), environmental monitoring, and any domain where temporal data is prevalent. Its ability to maintain feature consistency and efficiency with limited data makes it a valuable tool for practitioners.
To Trust or Not to Trust: Retrieval-Augmented Fact Checking in Speech
NLP
Large Language Models
Audio & Speech
- Introduction of VERISPEAK, a benchmark for speech-based fact verification.
- Significant text-speech modality gap observed in LALMs' performance.
- Retrieval alone is insufficient for effective verification of spoken claims.
- Combining retrieval with reasoning enhances claim-evidence comparison.
Read more
To Trust or Not to Trust: Retrieval-Augmented Fact Checking in Speech
Summary
The paper addresses the growing issue of misinformation in spoken formats, such as news clips and podcasts, and introduces VERISPEAK, a benchmark designed to evaluate fact-checking systems that verify claims made in speech. The benchmark consists of 3,879 spoken claims with balanced true and false labels, allowing for an examination of whether factual verification capabilities can transfer from text to speech. The authors conducted experiments using various Large Audio Language Models (LALMs) to assess the performance of these models in verifying spoken claims compared to written claims. The findings reveal a significant text-speech modality gap, where LALMs that perform well on text often struggle with spoken claims. Additionally, the study shows that retrieval alone does not significantly enhance verification performance, as models tend to conflate retrieved evidence with the spoken claim. However, combining retrieval with explicit reasoning improves the accuracy of claim-evidence comparisons, with a tuned LALM achieving an accuracy of 86.1%. The results emphasize that effective speech misinformation detection requires not just understanding speech but also grounded reasoning over retrieved evidence.
Methodology
The authors developed the VERISPEAK benchmark and evaluated five LALMs across various settings, including text-only verification, speech-input LALMs, retrieval-augmented verification, and reasoning-augmented verification. They systematically analyzed the models' performance in verifying spoken claims against textual evidence.
Results
The experiments demonstrated that LALMs exhibit a consistent performance gap between text and speech modalities. Retrieval methods improved performance only marginally, and reasoning techniques were necessary to effectively differentiate between claims and evidence. The best-performing model achieved an accuracy of 86.1% when both retrieval and reasoning were applied.
Implications
The findings suggest that future speech-based misinformation detection systems must integrate advanced reasoning capabilities alongside speech recognition and retrieval methods. This has implications for developing more robust fact-checking tools in various applications, including news media, social platforms, and public discourse.
MF-SCBO : Multi-fidelity Scalable Constrained Bayesian Optimization
Optimization
- MF-SCBO effectively combines multi-fidelity optimization with constrained Bayesian optimization.
- The method addresses high-dimensional black-box functions with arbitrary fidelity levels and non-nested sampling.
- Experimental results indicate superior convergence performance compared to existing methods.
- The approach utilizes Gaussian processes and introduces innovative trust region center selection strategies.
Read more
MF-SCBO : Multi-fidelity Scalable Constrained Bayesian Optimization
Summary
The paper introduces MF-SCBO, a novel method for Multi-fidelity Scalable Constrained Bayesian Optimization, addressing the challenges of optimizing high-dimensional black-box functions under constraints. Traditional optimization methods struggle with expensive evaluations, particularly in fields like engineering and machine learning. MF-SCBO extends the existing Scalable Constrained Bayesian Optimization (SCBO) framework to incorporate multiple fidelity levels, allowing for the use of cheaper, lower-fidelity evaluations alongside high-fidelity ones. The authors tackle the complexities of high-dimensionality, arbitrary fidelity levels, and non-nested sampling, which have not been adequately addressed in previous works. The method employs Gaussian process surrogates and introduces two variants for trust region centers based on high-fidelity evaluations and predictions. Experimental results demonstrate that MF-SCBO outperforms both the single-fidelity SCBO and other multi-fidelity methods in terms of convergence across a range of benchmark functions and challenging real-world problems, showcasing its effectiveness in reducing computational costs while maintaining optimization quality.
Methodology
The MF-SCBO method builds upon the SCBO framework, utilizing Gaussian process surrogates to model the objective function and constraints. It introduces two variants for determining the trust region center based on either high-fidelity evaluations or predictions. The method is designed to handle multiple fidelity levels and employs a joint maximization of the acquisition function over both input variables and fidelity levels. The performance is evaluated against benchmark functions and real-world applications, focusing on high-dimensional settings.
Results
The experimental evaluation shows that MF-SCBO consistently achieves better convergence rates than both the standard SCBO and the Multi-Fidelity Max-value Entropy Search (MF-MES) across various benchmark problems, which include synthetic and realistic applications with input dimensions ranging from 6 to 100. The results highlight the method's ability to effectively leverage multi-fidelity information to enhance optimization efficiency.
Implications
MF-SCBO has significant implications for fields that require optimization of expensive simulations or experiments, such as aerospace engineering, machine learning hyperparameter tuning, and robotics. By efficiently utilizing multi-fidelity evaluations, the method can reduce computational costs and improve the optimization process in practical applications.
BLADE: Distilled LLM Regularization for Calibrated Knowledge Graph Completion
Graph Learning
Large Language Models
NLP
- BLADE separates latent truth from graph recording for improved calibration in KGC.
- The model distills knowledge from an LLM into a frozen teacher regularizer, enhancing performance without LLM involvement during inference.
- BLADE achieves significant reductions in adaptive ECE compared to traditional methods.
- The model demonstrates superior performance in ECE, Brier score, and NLL over existing calibration techniques.
Read more
BLADE: Distilled LLM Regularization for Calibrated Knowledge Graph Completion
Summary
The paper introduces BLADE, a novel approach to knowledge graph completion (KGC) that addresses the need for calibrated probabilities in ranking models. Traditional KGC models focus on optimizing ranking but often fail to provide reliable probability estimates for the inferred facts. BLADE employs a variational model that separates latent truth from graph recording and distills judgments from an offline language model (LLM) into a frozen teacher regularizer. This design allows for the LLM to be excluded during inference, enabling the model to produce predictive probabilities and quantify epistemic uncertainty through posterior samples. The authors demonstrate that BLADE maintains competitive performance across five benchmarks while significantly improving calibration metrics, including a 60.1% reduction in adaptive expected calibration error (ECE) compared to deep ensembles and 78.1% compared to temperature-scaled RotatE. The model also shows improvements in ECE, Brier score, and negative log-likelihood (NLL) over existing methods, particularly in scenarios with controlled missingness. The findings suggest that while aligned semantics from the LLM contribute to performance, the model's calibration is limited to the specified candidate distributions, indicating a nuanced understanding of the relationship between ranking and probability in KGC.
Methodology
BLADE employs a three-stage process: first, it trains a frozen plausibility teacher using offline language model labels; second, it utilizes a relational encoder to parameterize factorized Gaussian embeddings and a probabilistic decoder to map samples to latent truth probabilities; and finally, during inference, it generates ranking scores and calibrated probabilities from posterior samples without the LLM, using the compact teacher as an optional regularizer.
Results
BLADE outperforms existing KGC models in terms of calibration, achieving a 60.1% reduction in adaptive ECE compared to deep ensembles and 78.1% compared to temperature-scaled RotatE. It also improves ECE, Brier score, and NLL over validation-selected histogram binning and a matched generative ComplEx2 model, with these advantages persisting in controlled missingness scenarios.
Implications
The findings suggest that BLADE can be effectively applied in scenarios requiring reliable probability estimates for knowledge graph completion, potentially benefiting applications in information retrieval, recommendation systems, and automated reasoning where calibrated confidence in predictions is crucial.
CataOPD: Catalytic On-Policy Distillation for Large Language Model Reasoning
NLP
Large Language Models
Reinforcement Learning
- CataOPD acts as a catalyst, expanding the reachability of reasoning tasks for LLMs.
- Self-Rescue Routing and Catalytic-Guided Self-Resolution are key components that enhance trajectory recovery.
- Barrier-Weighted Internalization prioritizes updates on challenging tokens to improve learning efficiency.
- CataOPD shows significant performance improvements on difficult reasoning problems compared to existing methods.
Read more
CataOPD: Catalytic On-Policy Distillation for Large Language Model Reasoning
Summary
The paper introduces CataOPD, a novel approach to enhance reasoning in large language models (LLMs) by addressing limitations in reinforcement learning (RL) and on-policy distillation (OPD). Traditional RL struggles when no correct trajectories are sampled, while OPD is limited by the reasoning capabilities of the student model. CataOPD redefines the teacher's role as a catalyst, allowing for the internalization of verified student-produced trajectories into a catalyst-free policy. The methodology includes Self-Rescue Routing, which utilizes failed groups as routing signals to recover correct trajectories through on-policy self-sampling. For unresolved issues, Catalytic-Guided Self-Resolution employs catalytic guidance to elicit verified trajectories from the guided distribution. Additionally, Barrier-Weighted Internalization focuses updates on critical tokens that are challenging to resolve without guidance. Experimental results demonstrate that CataOPD outperforms existing baselines, particularly on difficult reasoning tasks, and enhances out-of-distribution generalization without reliance on a teacher during inference.
Methodology
CataOPD employs a three-pronged approach: Self-Rescue Routing to recover trajectories from failed samples, Catalytic-Guided Self-Resolution to leverage guidance for trajectory elicitation, and Barrier-Weighted Internalization to focus on critical tokens during updates. This methodology allows the model to internalize its own correct trajectories rather than solely relying on a teacher's guidance.
Results
CataOPD outperformed baseline models across multiple mathematical reasoning datasets, particularly excelling in solving the hardest problems. The model demonstrated improved performance in out-of-distribution scenarios, indicating enhanced generalization capabilities. The results suggest that CataOPD effectively internalizes correct trajectories, allowing for reasoning without teacher intervention.
Implications
The findings suggest that CataOPD could lead to more robust and generalizable LLMs capable of tackling complex reasoning tasks. This approach may have applications in areas requiring advanced reasoning capabilities, such as automated problem-solving, educational tools, and AI-driven decision-making systems.
Limited Structural Reliability in Public Educational Prediction Benchmarks: A Four-Dimension Audit of Seven Datasets
Theory
- Only three out of seven audited educational datasets passed all reliability checks.
- The primary issue affecting benchmark validity is cross-group fragility rather than model sophistication.
- Increasing model complexity does not resolve structural issues in fragile datasets.
- A reusable pre-modeling audit framework is proposed to enhance the reliability of educational prediction benchmarks.
Read more
Limited Structural Reliability in Public Educational Prediction Benchmarks: A Four-Dimension Audit of Seven Datasets
Summary
This paper investigates the structural reliability of seven public educational prediction datasets through a four-dimensional audit framework. The authors conducted pre-modeling reliability checks, which revealed that only three datasets passed all checks, while four datasets either failed group-aware generalization tests or lacked necessary provenance metadata. The audit included assessments for baseline gap, split instability, null separation, and metadata adequacy under group-aware holdout conditions. Notably, the dominant failure mode was identified as cross-group fragility, where performance metrics significantly declined when evaluated across different groups. For instance, the UCI Student dataset's performance dropped from an iid R2 of 0.242 to a group-holdout R2 of -0.097. The findings indicate that increasing model complexity does not mitigate these issues; instead, it can exacerbate instability in fragile datasets. The study emphasizes that benchmark reliability in educational AI is more affected by data structure and evaluation design than by the choice of algorithms. The authors propose a reusable pre-modeling audit as a quality gate to ensure that public educational datasets can support credible benchmark claims.
Methodology
The authors employed a four-dimensional audit protocol to evaluate each dataset, focusing on baseline gap, split instability, null separation, and metadata adequacy. They used linear baselines and standard statistical diagnostics to assess the datasets before model optimization.
Results
The audit revealed that three datasets passed all four reliability checks, while four failed due to issues such as group-aware generalization and lack of metadata. The analysis showed significant performance drops in fragile datasets when evaluated across different groups, indicating a critical vulnerability in educational prediction benchmarks.
Implications
The findings suggest that educational AI researchers and practitioners should prioritize dataset reliability and structure over model complexity. The proposed audit framework can serve as a standard procedure for evaluating future datasets, ensuring that they meet quality standards before being used for benchmarking or deployment.
When Does Unsupervised Learning Succeed or Fail? A PoS Perspective on Reconstruction-Based Anomaly Detection
Theory
Computer Vision
Time Series
- Identifies two failure modes in reconstruction-based anomaly detection: join blindness and meet preference.
- Proposes the concept of a compact nominal union as the optimal representation for anomaly detection.
- Introduces Dynamic Push and Pull and nested manifold carving as novel methods to improve anomaly detection.
- Demonstrates that geometric conditions can predict reconstruction failures in unsupervised learning.
Read more
When Does Unsupervised Learning Succeed or Fail? A PoS Perspective on Reconstruction-Based Anomaly Detection
Summary
This paper investigates the conditions under which reconstruction-based unsupervised learning succeeds or fails in anomaly detection, using the Pursuit of Subspaces (PoS) hypothesis as a theoretical framework. The authors identify two primary failure modes: 'join blindness,' where the model reconstructs anomalies too accurately, and 'meet preference,' where valid nominal variations are discarded. They propose that the ideal learned representation should be a compact nominal union of submanifolds, which typically requires a nonlinear reconstruction map. To address the identified failures, the authors introduce two novel methods: Dynamic Push and Pull, which learns from controlled perturbations without requiring anomaly labels, and nested manifold carving, which recursively applies the same principle in latent space. Experimental results demonstrate that these methods effectively improve reconstruction-based anomaly detection across standard benchmarks and enhance pretrained ECG representations for downstream classification tasks. The findings connect reconstruction failures to identifiable geometric conditions and provide practical mechanisms for learning compact representations.
Methodology
The authors utilize the Pursuit of Subspaces (PoS) hypothesis to characterize the geometric properties of learned representations. They derive predictions about failure modes and remedies, then validate these predictions through experiments that assess the dynamics of individual samples and configurations. The proposed methods, Dynamic Push and Pull and nested manifold carving, leverage controlled perturbations and recursive principles in latent space to enhance learning.
Results
The proposed methods improved reconstruction-based anomaly detection performance across various standard benchmarks and effectively enhanced pretrained ECG representations for downstream classification tasks. The experiments confirmed the predicted changes in latent geometry, validating the theoretical framework established by the PoS hypothesis.
Implications
The findings suggest that understanding the geometric properties of learned representations can lead to more effective anomaly detection methods in unsupervised learning scenarios. The proposed techniques could be applied to various domains where anomaly detection is critical, such as medical diagnostics, fraud detection, and quality control in manufacturing.
An Analytical Theory of Auxiliary Learning
Theory
Optimization
- Introduces a closed-form expression for generalization error in linear networks under auxiliary learning.
- Establishes a fluctuation-dissipation relationship for non-linear activation functions.
- Demonstrates how auxiliary tasks can improve generalization despite competing objectives.
- Validates theoretical findings with numerical experiments.
Read more
An Analytical Theory of Auxiliary Learning
Summary
This paper presents a theoretical framework for understanding auxiliary learning, where neural networks improve performance on a primary task by concurrently training on additional auxiliary tasks. Utilizing a teacher-student model, the authors derive a closed system of differential equations that describe the dynamics of online stochastic gradient descent in the large-input limit. For linear networks, they provide a closed-form expression for the generalization error, highlighting the influence of task correlations and label noise on the benefits of auxiliary learning. The study also extends to non-linear activation functions, where a fluctuation-dissipation analytical theory is developed, linking main and auxiliary errors to single-task errors. The theoretical predictions are validated through numerical experiments, demonstrating that auxiliary tasks enhance generalization by balancing the dynamics towards optimal solutions amidst gradient noise.
Methodology
The authors employ a teacher-student framework to analyze auxiliary learning, deriving differential equations to model the training dynamics of neural networks. They focus on both linear and non-linear networks, using mean-field ordinary differential equations for linear cases and a Lyapunov equation for non-linear activations. Numerical simulations are conducted to support the theoretical findings.
Results
The paper finds that the generalization error in linear networks can be quantitatively expressed in terms of task correlations and label noise. For non-linear networks, a relationship is established between the errors of main and auxiliary tasks, which holds for any activation function. The results indicate that auxiliary tasks can significantly enhance generalization by optimizing the balance between training dynamics and gradient noise.
Implications
The findings suggest that incorporating auxiliary tasks in neural network training can lead to better generalization performance, which has practical implications for multi-task learning scenarios in various applications, including computer vision and natural language processing. Understanding the underlying mechanisms can guide the design of more effective training strategies.
TinyCardioUNet: IMU-to-ECG Translation with Graph-Encoded Inter-Axis Dependencies and Tensor Decomposition-Based Parameter Reduction
Time Series
Graph Learning
Efficient ML
- TinyCardioUNet effectively translates IMU signals into ECG without prior channel selection.
- The model incorporates graph neural networks to capture inter-axis dependencies, enhancing robustness.
- Tensor decomposition is utilized for significant parameter reduction, resulting in a lightweight model.
- Achieved RMSE of 0.098 and Pearson correlation of 0.677 on a public dataset.
Read more
TinyCardioUNet: IMU-to-ECG Translation with Graph-Encoded Inter-Axis Dependencies and Tensor Decomposition-Based Parameter Reduction
Summary
The paper presents TinyCardioUNet, a novel lightweight architecture designed for translating signals from a chest-worn inertial measurement unit (IMU) into electrocardiography (ECG) data. This approach addresses the discomfort associated with traditional ECG monitoring by enabling continuous heart rate (HR) monitoring without electrodes. TinyCardioUNet leverages all six IMU axes without prior channel selection, enhancing its robustness against noise. The architecture incorporates a graph neural network to model inter-axis dependencies among the IMU signals, which allows for better information sharing between axes. Additionally, it employs tensor decomposition for parameter reduction, resulting in a compact model with only 36,000 parameters. The model was evaluated on a public dataset, achieving a root mean square error (RMSE) of 0.098 and a Pearson correlation coefficient of 0.677, demonstrating its effectiveness in ECG reconstruction while maintaining computational efficiency suitable for deployment on resource-constrained devices.
Methodology
The TinyCardioUNet architecture follows a UNet structure, consisting of an encoder, a graph module for refining bottleneck features, and a decoder. It uses 1D convolutions, batch normalization, and a GraphSAGE layer to model inter-axis dependencies. Tensor decomposition is applied for parameter reduction, making the model lightweight and efficient.
Results
On the 'Mechanocardiograms with ECG Reference' dataset, TinyCardioUNet achieved an RMSE of 0.098 and a Pearson correlation coefficient of 0.677, indicating high accuracy in ECG reconstruction. The model's compact size (36.0k parameters) allows for deployment on edge devices.
Implications
The findings suggest that TinyCardioUNet can facilitate continuous heart rate monitoring in various settings, potentially improving cardiovascular health management without the discomfort associated with traditional ECG methods. Its lightweight design makes it suitable for integration into wearable technology.
Automatic Rank Allocation for Low-Rank Adaptation in Large Language Models via â„“p Regularization
NLP
Large Language Models
Efficient ML
- Introduces â„“p-LoRA, a method for automatic rank allocation in LoRA using â„“p regularization.
- Addresses inefficiencies in traditional uniform rank allocation by adapting ranks based on component importance.
- Derives a proximal subproblem that simplifies the optimization of rank allocation.
- Demonstrates competitive performance on NLP tasks compared to existing LoRA baselines.
Read more
Automatic Rank Allocation for Low-Rank Adaptation in Large Language Models via â„“p Regularization
Summary
This paper addresses the challenge of rank allocation in low-rank adaptation (LoRA) for large language models (LLMs), a method that allows parameter-efficient fine-tuning. Traditional approaches allocate ranks uniformly across layers, which can be inefficient as different components may require varying adaptation capacities. The authors propose a novel method called â„“p-LoRA, which utilizes â„“p regularization (with 0 < p < 1) to automatically determine the rank of each adaptation matrix. This method encourages redundancy reduction by promoting sparsity in the adaptation components, allowing for a more efficient allocation of ranks based on the importance of each component. The paper derives a proximal subproblem that simplifies the optimization process, leading to an implicit thresholding criterion for rank selection. Experimental results on natural language understanding and question-answering tasks show that â„“p-LoRA achieves competitive performance compared to existing LoRA methods, demonstrating its effectiveness in optimizing rank allocation.
Methodology
The authors formulate the rank allocation problem as a structured sparsity problem and apply â„“p regularization to the energy of rank-one LoRA components. This approach allows for the automatic identification of redundant components through an implicit thresholding mechanism, reducing the complexity of the optimization process to a two-dimensional problem.
Results
The experiments conducted on natural language understanding and question-answering tasks indicate that â„“p-LoRA achieves performance comparable to existing LoRA methods, validating the effectiveness of the proposed rank allocation strategy.
Implications
The proposed â„“p-LoRA method can significantly enhance the efficiency of fine-tuning large language models by optimizing rank allocation, potentially leading to reduced computational costs and improved performance in various NLP applications.
FlowAtom: Atom-Based Evidence Aggregation for Multi-Label Website Fingerprinting
Theory
- FlowAtom formulates multi-label website fingerprinting from a flow-level perspective, eliminating the need for packet-level segmentation.
- The method constructs shared Atoms from flow representations without requiring website labels, allowing for effective evidence aggregation.
- FlowAtom achieves impressive micro-F1 scores in both closed-world and open-world evaluations across various encrypted traffic types.
- The approach is validated against realistic scenarios, demonstrating its robustness in identifying multiple websites from mixed traffic.
Read more
FlowAtom: Atom-Based Evidence Aggregation for Multi-Label Website Fingerprinting
Summary
The paper presents FlowAtom, a novel approach to multi-label website fingerprinting that addresses the challenge of identifying monitored websites in mixed encrypted traffic. Traditional methods rely on complete traffic traces, which are often impractical in real-world scenarios where users may visit multiple websites simultaneously. FlowAtom introduces a flow-level perspective, aggregating evidence from multiple flows within an observation window without requiring packet-level trace segmentation. The method constructs shared prototypes, termed Atoms, from flow representations using unsupervised learning. By pretraining a flow encoder on unlabeled traffic, FlowAtom aggregates Atom responses across flows to create a fixed-dimensional representation for predicting the set of monitored websites. The evaluation demonstrates FlowAtom's effectiveness, achieving high micro-F1 scores across different encrypted traffic scenarios, significantly outperforming existing baseline methods.
Methodology
FlowAtom consists of three main stages: flow representation pretraining, Atom construction, and window-level multi-label prediction. It employs contrastive learning for pretraining on external unlabeled traffic, constructs Atoms through clustering of flow representations, and aggregates Atom responses using max pooling to generate a permutation-invariant representation for prediction.
Results
In closed-world evaluations, FlowAtom achieved micro-F1 scores of 97.82%, 94.43%, and 93.92% for Direct HTTPS, Trojan, and VMess traffic types, respectively. In open-world evaluations, it maintained micro-F1 scores of 92.37%, 92.64%, and 89.85% under the same conditions, consistently outperforming baseline methods.
Implications
The findings suggest that FlowAtom can significantly enhance the accuracy of website fingerprinting in encrypted traffic scenarios, which has implications for cybersecurity, privacy protection, and traffic analysis. The method's ability to operate without labeled data also opens avenues for further research in unsupervised learning applications in network security.
Evaluating Cross-region Generalization for Wavelet-Diffusion Precipitation Downscaling
Generative Models
Time Series
Theory
- The Wavelet-Diffusion Model (WDM) shows strong potential for cross-region precipitation downscaling.
- Performance varies significantly across different precipitation intensities, with improvements concentrated in higher-reflectivity structures.
- Spatial organization of precipitation fields plays a crucial role in model performance, more so than regional differences.
- The study highlights the need for systematic evaluations of cross-region generalization in precipitation downscaling models.
Read more
Evaluating Cross-region Generalization for Wavelet-Diffusion Precipitation Downscaling
Summary
This study investigates the cross-region and cross-event generalization capabilities of the Wavelet-Diffusion Model (WDM) for precipitation downscaling. The authors evaluate the model's performance across six distinct U.S. regions, representing various precipitation regimes, using low-resolution inputs derived from NOAA Multi-Radar/Multi-Sensor composite reflectivity fields. The research compares a WDM trained solely on Oklahoma data against a model trained on all six regions, alongside traditional nearest-neighbor and bicubic interpolation methods. Performance metrics include image-domain reconstruction, spectral fidelity, and bin-wise precipitation detection. Results indicate that while the Oklahoma-trained WDM performs competitively outside its training region, the all-region WDM achieves superior overall performance, albeit with uneven improvements across different precipitation intensities. Notably, the study finds a strong correlation between the spatial organization of precipitation fields and model performance, suggesting that spatial characteristics are more influential than regional differences in downscaling tasks. The findings support the potential for transferring downscaling models to regions with limited training data and emphasize the importance of generating consistent high-resolution precipitation products globally.
Methodology
The study employs the Wavelet-Diffusion Model (WDM) framework, evaluating its performance using datasets from six U.S. regions with diverse precipitation regimes. The model is trained on both region-specific and multi-region datasets, with performance assessed through various metrics that measure reconstruction quality, spectral fidelity, and precipitation detection accuracy.
Results
The WDM trained on Oklahoma data remains competitive when applied to other regions, while the all-region model demonstrates the best overall performance. However, improvements are not uniform across precipitation intensities, with significant enhancements observed in localized high-reflectivity areas. A strong correlation (0.901) is found between spatial organization (measured by Moran's I) and model performance across all regions.
Implications
The findings suggest that precipitation downscaling models can be effectively transferred to regions with limited local data, enhancing the generation of high-resolution precipitation products globally. This has significant implications for hydrological modeling, flood-risk assessment, and climate adaptation strategies.
Aim Short to Reach Far: Your Frozen World Model Can Plan Better Than You Think
Robotics
Optimization
Reinforcement Learning
- Final-goal scoring can limit effective planning, even with accurate predictions and optimal short-horizon searches.
- Anchored Planning (AP) improves action synthesis by using intermediate targets derived from recorded experiences.
- Both learned and observed intermediate targets enhance performance over traditional final-goal scoring methods.
- The placement of targets and timing of retrieval significantly affect control outcomes.
Read more
Aim Short to Reach Far: Your Frozen World Model Can Plan Better Than You Think
Summary
This paper investigates the limitations of traditional planning methods that rely on visual world models, particularly in how they score predicted outcomes based on their proximity to a goal image. The authors demonstrate that such methods can hinder effective control, as achieving a goal may necessitate initial actions that move away from it. They propose a novel approach called Anchored Planning (AP), which utilizes recorded segments of past experiences to improve action synthesis and ranking. By focusing on intermediate targets rather than final goals, the authors show that their method significantly enhances performance across various tasks, including Cube, PushT, Reacher, and TwoRoom. The findings indicate that the choice of planning targets is crucial, as it influences the effectiveness of action selection and control, even when using a frozen model without additional training. The paper emphasizes that planning targets should be evaluated alongside prediction accuracy to optimize control strategies in visual world models.
Methodology
The authors introduce Anchored Planning (AP), which retrieves recorded segments that resemble current and goal observations. The method scores actions based on their predicted outcomes towards an intermediate target, allowing for improved action synthesis and ranking without retraining the model. The approach leverages both learned targets and those derived from observed experiences to enhance planning performance.
Results
The experiments demonstrate that planning towards observed intermediate targets consistently outperforms the released LeWM planner across all evaluated tasks. The use of intermediate targets leads to higher success rates, particularly at longer goal offsets, and shows that merely improving prediction accuracy does not guarantee better control.
Implications
The findings suggest that incorporating intermediate targets into planning strategies can significantly enhance the performance of visual world models in robotic control tasks. This approach could be applied to various domains where effective planning is critical, such as autonomous navigation and robotic manipulation.
Learning to Discover Interesting Mathematics
Large Language Models
Theory
Generative Models
- Introduces a novel definition of mathematical interestingness based on proof length and statement length.
- Develops a 27B parameter model that predicts proof difficulty more accurately than existing models.
- Demonstrates that optimizing for interestingness leads to a significant increase in the generation of novel theorems.
- Reduces overlap with existing mathematical libraries from 91.9% to 30.6%, indicating the production of out-of-distribution mathematics.
Read more
Learning to Discover Interesting Mathematics
Summary
This paper explores the capabilities of Large Language Models (LLMs) in generating and discovering interesting mathematical theorems. The authors define 'intrinsic interestingness' of a theorem as the ratio of the proof length to the statement length, which correlates with the theorem's utility in mathematical contexts. They introduce a 27B parameter model that predicts proof difficulty conditioned on existing premises, outperforming existing models. By optimizing for interestingness, the model generates theorems with a significant reduction in overlap with established mathematical libraries, indicating the creation of novel mathematical content. The framework enables autonomous mathematical discovery, allowing the system to generate, evaluate, and build upon new theorems iteratively, thus paving the way for self-expanding mathematical libraries that do not rely on human input.
Methodology
The authors formalize proof difficulty as the computational cost of deriving a theorem from a given context using Lean 4. They post-train a large language model to predict proof difficulty and define interestingness and utility metrics. The model is then optimized to generate more interesting theorems, which are evaluated based on their proof lengths and their novelty compared to existing mathematical libraries.
Results
The model successfully quadruples the mean interestingness of generated theorems and significantly reduces the overlap with established mathematical libraries, showcasing its ability to produce novel mathematical statements. The framework also facilitates recursive mathematical discovery, identifying nontrivial statements not present in existing resources.
Implications
This research has the potential to revolutionize mathematical discovery by enabling systems to autonomously generate and evaluate theorems, thus expanding mathematical knowledge without human intervention. It could lead to the development of self-sustaining mathematical libraries that continuously evolve.
Spatio-temporally complementary feature propagation on graphs for longitudinal AADT estimation
Graph Learning
Optimization
Time Series
- Proposes a novel framework for AADT estimation combining sparse and dense data sources.
- Utilizes a feature propagation algorithm on directed graphs with Poisson energy minimization.
- Achieves high computational efficiency and convergence in minutes.
- Demonstrates effective reconciliation of theoretical models with empirical data.
Read more
Spatio-temporally complementary feature propagation on graphs for longitudinal AADT estimation
Summary
This paper addresses the challenge of estimating Annual Average Daily Traffic (AADT) across urban networks, which is crucial for transportation planning but often hindered by the high cost and sparse distribution of physical sensors. The authors propose a novel framework that combines two data sources: temporally dense loop detector data, which is spatially sparse, and a spatially complete but temporally sparse macroscopic transportation model. The methodology employs a feature propagation algorithm on directed graphs, formulated as a Poisson energy minimization problem that considers residuals. By replacing the standard binary adjacency matrix with flow ratio matrices, the model captures real-world vehicle turn ratios at intersections. The proposed algorithm was validated in Zurich, demonstrating high computational efficiency with convergence achieved in minutes. The results indicate that the framework effectively integrates theoretical models with empirical data, achieving a normalized mean absolute error below 10%. This scalable approach offers a viable solution for network-wide AADT estimation, leveraging limited sensor coverage and traffic models.
Methodology
The methodology involves a feature propagation algorithm on directed graphs, formulated as a Poisson energy minimization problem. It replaces the binary adjacency matrix with flow ratio matrices to better capture vehicle turn ratios at intersections.
Results
The algorithm was validated in Zurich, achieving convergence within minutes and yielding a normalized mean absolute error below 10%, indicating a high level of accuracy in AADT estimation.
Implications
The proposed framework provides a scalable solution for estimating AADT across urban networks, which can aid in transportation planning, infrastructure maintenance, and environmental assessments, particularly in areas with limited sensor coverage.
SGA: Uncertainty Quantification for Multi-Step Forecasting in Time Series Foundation Models
Time Series
- SGA effectively quantifies uncertainty in multi-step forecasting for TSFMs.
- The method utilizes a Directed Acyclic Graph to characterize potential forecast branches.
- SGA outperforms existing uncertainty quantification methods in predictive error ranking.
- Larger TSFM models correlate with lower uncertainty estimates, indicating a scaling law.
Read more
SGA: Uncertainty Quantification for Multi-Step Forecasting in Time Series Foundation Models
Summary
This paper addresses the challenge of uncertainty quantification in multi-step forecasting using Time Series Foundation Models (TSFMs). While TSFMs have shown significant improvements in forecasting accuracy, they often exhibit inherent uncertainties that can undermine the reliability of their predictions. The authors propose a novel method called Slicing-Graphing-Alignment (SGA) to quantify this uncertainty. SGA involves three main stages: slicing, graphing, and alignment. In the slicing stage, multiple forecasts are sampled and segmented into slices representing potential forecast branches. The graphing stage constructs a Directed Acyclic Graph (DAG) to describe the topology of these branches, while the alignment stage uses Dynamic Time Warping (DTW) to measure the similarity between slices. The complexity of the resulting graph is then used to quantify the uncertainty of the forecasts. Experimental results demonstrate that SGA outperforms existing methods in ranking predictive errors and provides a more comprehensive sampling coverage, leading to a fundamentally different quantification mechanism. Additionally, the study finds that larger TSFM models tend to exhibit lower uncertainty estimates, suggesting a new empirical scaling law for uncertainty quantification in multi-step forecasting.
Methodology
The SGA method consists of three stages: (1) Slicing - sampling multiple forecasts and segmenting them into slices; (2) Graphing - constructing a Directed Acyclic Graph (DAG) to represent the topology of forecast branches; (3) Alignment - using Dynamic Time Warping (DTW) to align slices and measure their similarity, which helps in quantifying the complexity of the forecasting space and thus the uncertainty.
Results
Experimental validation across 11 TSFMs and 27 datasets shows that SGA significantly improves the ranking of predictive errors with uncertainty estimates compared to existing methods. It also demonstrates a more extensive and precise sampling coverage, and identifies a correlation between larger model scales and lower uncertainty estimates.
Implications
The findings suggest that SGA can enhance the credibility of TSFM forecasts in high-stakes applications such as finance and industrial forecasting by providing reliable uncertainty measures. This could lead to better decision-making based on TSFM predictions.
fable.intermittent: benchmarking probabilistic forecasting methods for intermittent time series
Time Series
- Introduction of fable.intermittent, an R package for benchmarking probabilistic forecasting methods for intermittent time series.
- Development of TWEES, a novel exponential smoothing model with a Tweedie predictive distribution.
- Release of tweedieDistr, a faster implementation of the Tweedie distribution for improved computational efficiency.
- Evaluation of multiple forecasting methods on four datasets, showcasing the accuracy of TWEES.
Read more
fable.intermittent: benchmarking probabilistic forecasting methods for intermittent time series
Summary
The paper introduces fable.intermittent, an R package designed to benchmark various probabilistic forecasting methods specifically for intermittent time series, which are prevalent in fields like spare-parts demand and retail sales. Traditional forecasting methods often provide point estimates, which are inadequate for decision-making due to the asymmetric costs of forecast errors. The authors propose a new model, TWEES, an exponential smoothing model utilizing a Tweedie predictive distribution, which is particularly suited for intermittent data. The package integrates multiple probabilistic forecasting methods into a single framework, simplifying the evaluation process across different time series datasets. Additionally, the authors present tweedieDistr, a faster implementation of the Tweedie distribution that enhances computational efficiency. The paper evaluates the performance of the implemented methods on four datasets, demonstrating the effectiveness of TWEES and the overall utility of the fable.intermittent package.
Methodology
The authors implemented several probabilistic forecasting methods within the fable framework, including TWEES, which combines exponential smoothing with a Tweedie distribution. They also developed tweedieDistr to enhance the speed of Tweedie density evaluations. The methods were evaluated using a straightforward forecasting pipeline across multiple datasets.
Results
The evaluation showed that TWEES performed among the most accurate models for forecasting intermittent time series. The tweedieDistr package provided density evaluations that were up to 20 times faster than existing implementations while maintaining numerical accuracy.
Implications
The fable.intermittent package can significantly aid practitioners in fields requiring accurate forecasting of intermittent demand, such as supply chain management and retail. The integration of multiple models into a single framework allows for systematic comparisons and better-informed decision-making based on probabilistic forecasts.
TAM-Chain: Multi-Scale Thyroid Cytology Classification via Absorbing Markov Chains and Shannon Entropy Uncertainty Quantification for False-Negative Suppression and Domain-Shift Adaptation
Computer Vision
Theory
Efficient ML
- TAM-Chain effectively suppresses false negatives in thyroid cytology classification.
- The framework utilizes multi-scale feature extraction and dynamic stopping criteria.
- Achieved a Macro F1 score of 0.9741 with 0.00% false-negative rate on internal validation.
- Maintains stability and performance under domain shifts, outperforming single-magnification models.
Read more
TAM-Chain: Multi-Scale Thyroid Cytology Classification via Absorbing Markov Chains and Shannon Entropy Uncertainty Quantification for False-Negative Suppression and Domain-Shift Adaptation
Summary
This paper presents TAM-Chain, a novel framework for thyroid cytology classification that addresses critical challenges in Fine-Needle Aspiration Biopsy (FNAB) diagnostics, particularly high false-negative rates and performance degradation under domain shifts. The framework employs Absorbing Markov Chain theory and Shannon Entropy-based Uncertainty Quantification to dynamically model multi-magnification feature extraction at 10×, 20×, and 40× magnifications. TAM-Chain introduces a dynamic early stopping mechanism that optimizes diagnostic efficiency while maintaining safety by suppressing false negatives. Additionally, it incorporates a human-in-the-loop referral system for ambiguous cases, ensuring expert evaluation when necessary. The framework was rigorously evaluated on both internal and external validation sets, demonstrating exceptional performance with a Macro F1 score of 0.9741 and an absolute False-Negative Rate of 0.00% on the internal set, while maintaining robust performance (Macro F1 = 0.7026) under severe domain shifts in the external validation set. Overall, TAM-Chain represents a significant advancement in automated thyroid cytology diagnostics, harmonizing efficiency with stringent clinical safety requirements.
Methodology
The TAM-Chain framework integrates Absorbing Markov Chains for dynamic modeling of multi-scale feature extraction, coupled with Shannon Entropy for uncertainty quantification. It employs a dynamic early stopping mechanism to optimize computational efficiency and a human-in-the-loop referral system for cases with high uncertainty.
Results
On an internal test set of 235 samples, TAM-Chain achieved a Macro F1 score of 0.9741 and an absolute False-Negative Rate of 0.00%. In external validation on a dataset of 1015 samples, it maintained a Macro F1 score of 0.7026, demonstrating resilience to domain shifts.
Implications
The TAM-Chain framework has significant implications for digital pathology, providing a reliable and efficient tool for thyroid cancer diagnostics that can adapt to varying clinical conditions and enhance patient safety.
Growth-Inspired Graph Generation and Inverse Design of Mechanical Lattices via Dot Matrices Database Augmentation and GCNN
Graph Learning
Optimization
Generative Models
- Introduces a morphogenetic framework for mechanical lattice generation based on natural growth processes.
- Utilizes a graph convolutional neural network (GCNN) to predict effective compressive stiffness from lattice topology.
- Demonstrates successful inverse design capabilities for mechanical lattices with a target stiffness of 1000 MPa.
- Extends the framework to include nonlinear materials and curved beam designs for enhanced shape programming.
Read more
Growth-Inspired Graph Generation and Inverse Design of Mechanical Lattices via Dot Matrices Database Augmentation and GCNN
Summary
This paper presents a novel morphogenetic graph-generation framework for mechanical lattices, inspired by natural growth processes. The framework utilizes a discrete dot matrix to define potential nodes, allowing for the sequential growth of lattice structures through cross-layer and intra-layer connections. The authors visualize this growth in two dimensions and implement it in three dimensions using a 3 × 3 × 3 nodal matrix. A dataset of unique three-dimensional lattices is created and evaluated using beam-based finite element analysis (FEA), which is then encoded as graphs. A graph convolutional neural network (GCNN) is employed to learn the mapping between the lattice topology and its effective compressive stiffness. The GCNN, consisting of three graph-convolution layers and dual global pooling, enables rapid structural sampling and inverse design, successfully predicting a target stiffness of 1000 MPa with a validated design stiffness of 1027.49 MPa. The framework is further extended to include parameterized horseshoe-shaped curved beams made from nonlinear materials, allowing for topology-geometry design aimed at achieving specific deformation shapes. This work bridges biological morphogenesis, graph learning, and nonlinear shape programming, offering a new paradigm for augmenting mechanical metamaterials databases.
Methodology
The methodology involves a growth-based structural representation using a discrete dot matrix, where lattice structures are generated through sequential cross-layer and intra-layer growth. The generated lattices are evaluated using finite element analysis (FEA) and represented as graphs. A graph convolutional neural network (GCNN) is trained to learn the topology-property mapping, enabling rapid screening and inverse design.
Results
The GCNN successfully predicted a design with an effective stiffness of 1042.43 MPa for a target of 1000 MPa, validated by finite element analysis at 1027.49 MPa. The framework also demonstrated the ability to design parameterized curved beams, expanding the design space for mechanical metamaterials.
Implications
This research has significant implications for the design of mechanical metamaterials, allowing for more efficient exploration of complex design spaces and the creation of materials with tailored mechanical properties. The integration of biological principles into the design process may lead to innovative solutions in engineering and materials science.
CFD Correction of Open Tip Clearance Flow in a Compressor Cascade Using VAE Latent Space Adaptation
Generative Models
- Development of a non-intrusive correction method for CFD predictions using VAE.
- Utilization of a low-rank latent-space adapter trained on limited experimental data.
- Significant reduction in mean absolute error, root mean square error, and relative L2 error.
- Improvement in the accuracy of CFD predictions for open tip clearance flow.
Read more
CFD Correction of Open Tip Clearance Flow in a Compressor Cascade Using VAE Latent Space Adaptation
Summary
This paper addresses the discrepancies between computational fluid dynamics (CFD) predictions and experimental observations of open tip clearance flow in compressor cascades. The authors propose a non-intrusive correction method that leverages a variational autoencoder (VAE) for latent-space adaptation. Initially, a VAE is trained on a dataset of 166 parametrically sampled CFD total pressure loss fields to create a low-dimensional statistical representation. Subsequently, a low-rank latent-space adapter is trained using only 12 paired CFD-experimental operating conditions. An observation operator is employed to map the corrected high-resolution fields to the experimental observation space, allowing for supervision at specific measurement locations. The results demonstrate a significant reduction in error metrics, indicating that the proposed method enhances the agreement between CFD predictions and sparse experimental data without altering the RANS solver or requiring high-resolution experimental labels.
Methodology
The methodology involves training a variational autoencoder (VAE) on a large dataset of CFD total pressure loss fields to learn a low-dimensional representation. A low-rank latent-space adapter is then trained using a small set of paired CFD-experimental conditions. An observation operator is used to map the corrected fields to the experimental observation space, allowing for targeted supervision.
Results
The proposed method achieved a mean absolute error reduction from 0.1335 to 0.0473, a root mean square error reduction from 0.1717 to 0.0621, and a relative L2 error reduction from 0.5108 to 0.1871 during a 12-fold cross-validation, indicating improved alignment between CFD predictions and experimental observations.
Implications
The findings suggest that the proposed method can effectively bridge the gap between CFD simulations and sparse experimental data, potentially leading to more accurate predictions in aerodynamics and turbomachinery design. This approach could be applied to other complex fluid dynamics problems where experimental data is limited.
Decoupling Knowledge and Privacy: Post-Task Self-Distillation Replay for LLM Continual Learning
NLP
Large Language Models
Theory
- SPARK framework decouples knowledge retention and privacy correction in continual learning.
- Self-Distillation Replay creates a stable checkpoint for effective task learning.
- Post-Task Privacy Correction selectively reduces PII likelihood while preserving non-sensitive behaviors.
- The proposed method addresses the challenge of catastrophic forgetting in neural models.
Read more
Decoupling Knowledge and Privacy: Post-Task Self-Distillation Replay for LLM Continual Learning
Summary
This paper addresses the challenge of privacy-preserving continual learning (PPCL) in large language models (LLMs), focusing on the need to mitigate the reproduction of sensitive content while retaining useful knowledge from previous tasks. The authors introduce a novel framework called SPARK, which decouples the objectives of knowledge retention and privacy correction. SPARK operates in two phases: first, it employs Self-Distillation Replay to learn the current task while preserving behaviors from previous tasks, creating a stable post-task checkpoint. Second, Post-Task Privacy Correction is applied to selectively reduce the likelihood of personally identifiable information (PII) in the outputs while anchoring non-PII behaviors. The authors argue that this approach allows for effective control over sensitive outputs without compromising the model's continual learning capabilities. Extensive evaluations demonstrate that SPARK successfully achieves selective PII suppression while maintaining strong performance in knowledge retention across various settings.
Methodology
The methodology involves a two-phase approach: first, Self-Distillation Replay is used to learn the current task while preserving previous task behaviors, resulting in a stable checkpoint. Second, Post-Task Privacy Correction is applied to this checkpoint to selectively suppress the likelihood of sensitive outputs, ensuring that non-sensitive behaviors are anchored and retained.
Results
The results indicate that SPARK effectively suppresses annotated PII likelihood while preserving strong continual learning utility and knowledge retention across diverse experimental settings. The evaluations demonstrate that the framework outperforms existing methods in terms of selective output control and stability in task acquisition.
Implications
The findings suggest that SPARK can be applied in scenarios where privacy is paramount, such as in healthcare or finance, where models must learn from sensitive data without compromising user privacy. This approach could enhance the deployment of LLMs in real-world applications while ensuring compliance with privacy regulations.
Auditability Is Not One Property: Rule Overlap, Behavioural Agreement, and Composition in Reinforcement Learning
Reinforcement Learning
Theory
Interpretability
- Auditability in RL is decomposed into six independently testable predicates.
- High rule coverage does not imply behavioural agreement among policies.
- Arbitration of rules does not guarantee correctness of the fused policy.
- The protocol's apparent failures can stem from mismatches between induction and deployment methods.
Read more
Auditability Is Not One Property: Rule Overlap, Behavioural Agreement, and Composition in Reinforcement Learning
Summary
This paper investigates the concept of auditability in reinforcement learning (RL) policies, which are often represented as opaque neural networks. The author proposes a decomposition of auditability into six distinct predicates: trace integrity, lossless coding, rule coverage, behavioural agreement, composition quality, and value-model reliability. Each predicate is independently tested, revealing that passing one does not guarantee the success of another. The study introduces a novel protocol for evaluating these predicates, involving a shared symbolizer among agents, a passive observer for rule extraction, and a ledger for trace verification. Key findings indicate that while there may be substantial symbolic overlap among independently trained policies, this does not translate into behavioural agreement. Additionally, the paper highlights that the process of arbitration in rule selection does not equate to policy generation, and the effectiveness of the protocol can be influenced by the task and evaluation methods used. The author emphasizes the limitations of rule-level fusion compared to value-based composition, suggesting that the claims made are exploratory rather than definitive.
Methodology
The author developed a protocol where agents share a calibration-derived symbolizer, and a passive observer converts logged transitions into confidence-scored condition-action rules. An append-only ledger verifies the integrity of the recorded trace, while offline fusion of rules is conducted using a confidence-ranked arbitration policy.
Results
The study found that symbolic overlap among policies did not correlate with behavioural agreement, indicating that high scores in rule coverage do not guarantee effective policy fusion. The protocol's performance varied based on the task and the methods used for rule induction and evaluation.
Implications
The findings suggest that reinforcement learning systems need more nuanced methods for auditing and composing policies. Understanding the limitations of rule-based approaches could lead to better design choices in RL applications, especially in critical decision-making scenarios.
Learnable Time-Frequency Masks for Explaining Time-Series Classifiers
Time Series
Interpretability
- XACT generalizes learned mask paradigms to arbitrary time-frequency representations.
- The framework incorporates a time-frequency objective that enhances explanation quality.
- LRP is adapted for use in wavelet domains, expanding the applicability of existing methods.
- XACT demonstrates improved precision in explanations compared to traditional methods.
Read more
Learnable Time-Frequency Masks for Explaining Time-Series Classifiers
Summary
The paper addresses the challenge of explainability in time-series classifiers, where discriminative information is often encoded in latent frequency or time-frequency features rather than in the raw signal. Existing attribution methods typically operate in either the time domain or a fixed transform domain, limiting their effectiveness. The authors propose a novel framework called eXplainable Attributions from Coefficients of Transforms (XACT), which learns sparse attribution masks over coefficients from arbitrary invertible time-frequency transforms. XACT is evaluated using the Short-Time Discrete Fourier Transform (STDFT), continuous wavelet transform, and discrete wavelet transform. The framework extends the virtual inspection layer approach to wavelet transforms, allowing Layer-wise Relevance Propagation (LRP) to generate explanations in these representations. Results from a synthetic dataset show that XACT produces precise explanations and is less prone to highlighting spurious features compared to existing methods. On two real-world datasets, XACT generates sparse and structured explanations, although no single method outperforms others across all evaluation criteria. The findings suggest that learning explanations directly in time-frequency representations provides a flexible approach for interpreting deep learning models applied to time series data.
Methodology
The authors developed XACT, which learns sparse and smooth masks directly over coefficients of any linear time-frequency operator with a bounded left inverse. The framework was evaluated using three transforms: STDFT, discrete wavelet transform, and continuous wavelet transform. A time-frequency objective was introduced to enhance the learning process, and LRP was adapted for wavelet domains to facilitate explanation generation.
Results
XACT produced precise explanations on a synthetic dataset and was less likely to highlight irrelevant features compared to baseline methods. In real-world applications, it generated sparse and structured explanations, although no method consistently outperformed others across all evaluation metrics.
Implications
The findings suggest that using time-frequency representations for explanation generation can significantly enhance the interpretability of deep learning models in critical domains such as finance, healthcare, and climate science, where time-series data is prevalent.
Safety-oriented pedestrian trajectory prediction at urban intersections using time-to-collision and crossing-zone context
Robotics
Time Series
Optimization
- Introduces a safety-oriented framework for pedestrian trajectory prediction at urban intersections.
- Combines Time-to-Collision (TTC) and crossing-zone context to improve prediction accuracy.
- Demonstrates significant reductions in prediction errors using a weighted loss function.
- Highlights the need for safety-oriented evaluation metrics in trajectory prediction models.
Read more
Safety-oriented pedestrian trajectory prediction at urban intersections using time-to-collision and crossing-zone context
Summary
This paper addresses the critical issue of pedestrian safety at urban intersections by proposing a novel trajectory prediction framework that integrates pedestrian motion history with Time-to-Collision (TTC) data and crossing-zone context. The study utilizes naturalistic trajectory data from the inD dataset, focusing on a specific urban intersection. The authors evaluate various neural network architectures, particularly a pooled Long Short-Term Memory (LSTM) model, which separately encodes TTC histories and crossing-zone context before merging them with pedestrian position data. The prediction performance is measured using Average Displacement Error (ADE) and Final Displacement Error (FDE), along with a novel weighted loss function that emphasizes larger prediction errors. The results demonstrate that incorporating safety-oriented training and contextual information significantly reduces prediction errors, highlighting the importance of considering both interaction dynamics and environmental context in pedestrian trajectory prediction. The findings suggest that while the proposed model shows promise, further validation across diverse datasets and pedestrian behaviors is necessary to enhance its applicability in real-world scenarios.
Methodology
The authors employed a pooled Long Short-Term Memory (LSTM) architecture to encode pedestrian motion history, Time-to-Collision (TTC) information, and crossing-zone context. They evaluated the model's performance using naturalistic trajectory data with a focus on both average and large prediction errors, applying a weighted loss function to prioritize significant errors.
Results
The final model incorporating TTC and crossing-zone context achieved an Average Displacement Error (ADE) of 0.184 m and a Final Displacement Error (FDE) of 0.491 m. Additionally, it reduced the counts of large prediction errors by 33.5% for ADE and 6.3% for FDE compared to a position-only LSTM model.
Implications
The findings suggest that integrating safety-oriented training and contextual information can enhance pedestrian trajectory prediction, potentially improving the effectiveness of Advanced Driver Assistance Systems (ADAS) and automated driving technologies in urban environments.
Stable and Faithful Explanations for Knowledge Tracing
Interpretability
- Introduces a validation protocol for assessing KT model explanations.
- Demonstrates the importance of stability and faithfulness in feature importance rankings.
- Shows that XGBoost can achieve competitive predictive accuracy compared to deep learning models when provided with similar information.
- Highlights the significance of engineered behavioral features in improving model interpretability.
Read more
Stable and Faithful Explanations for Knowledge Tracing
Summary
This paper addresses the challenge of providing interpretable explanations for knowledge tracing (KT) models, which predict student performance but often lack transparency. The authors propose a validation protocol that assesses predictive accuracy, explanation stability, and faithfulness of explanations. They engineered thirteen behavioral features from the ASSISTments datasets (2009 and 2012) and rebuilt the 2009 dataset to mitigate data leakage issues. An Extreme Gradient Boosting (XGBoost) model was utilized and explained using Tree SHapley Additive exPlanations (TreeSHAP), and its performance was compared against four deep learning baselines (DKT, SAKT, AKT, and SimpleKT) under an information-matched protocol. The results indicated that while XGBoost achieved competitive AUC scores, its performance was closely tied to the information provided, rather than the model type. The study also demonstrated that the global feature importance rankings were consistent across various conditions, and removing top-ranked features significantly impacted model performance, highlighting the importance of stable and faithful explanations in KT.
Methodology
The authors engineered behavioral features from the ASSISTments datasets and rebuilt the 2009 dataset to address data leakage. They employed an XGBoost model explained with TreeSHAP and compared its performance against deep learning models under an information-matched protocol, ensuring that all models received the same behavioral signals.
Results
XGBoost achieved AUC scores of 0.777 on the 2012 dataset and 0.786 on the rebuilt 2009 dataset. When restricted to the same information as the deep learning models, XGBoost's performance was comparable, indicating that differences in predictive accuracy were primarily due to the information supplied rather than the model architecture. The stability of feature importance rankings was confirmed across various conditions.
Implications
The findings suggest that feature-based models can provide interpretable and stable explanations for KT, which can enhance instructional strategies in educational settings. This work lays the groundwork for future research on explainable AI in education, emphasizing the need for transparent models that can inform pedagogical decisions.
Canopy: Exploiting Piecewise Smooth Tree Priors for Multi-Fidelity Bandits
Large Language Models
Optimization
Theory
- CANOPY learns the validity of smoothness assumptions from data, improving flexibility in multi-fidelity bandit settings.
- The framework utilizes cheap probes to certify local aggregation bias, directing expensive evaluations more effectively.
- The method demonstrates superior performance across various LLM inference tasks, achieving notable improvements in recall and efficiency.
- Theoretical guarantees are provided for fixed-budget and regret performance, adapting to the presence of smoothness violations.
Read more
Canopy: Exploiting Piecewise Smooth Tree Priors for Multi-Fidelity Bandits
Summary
The paper introduces CANOPY, a novel multi-fidelity tree bandit framework designed to optimize various large language model (LLM) inference tasks by leveraging the hierarchical structure of candidate evaluations. Traditional hierarchical bandit methods require a predefined smoothness schedule, which is often difficult to ascertain in practice, especially when objectives are piecewise smooth with sharp boundaries. CANOPY addresses this limitation by learning the validity of the smoothness prior from data rather than assuming it globally. It employs cheap random-path probes to create an online certificate of local aggregation bias, guiding expensive leaf evaluations towards areas where smoothness violations are detected. The authors provide theoretical guarantees for fixed-budget and regret performance, demonstrating that CANOPY can recover optimal performance in the absence of violations while adapting to dense violations. The experimental results across multiple benchmarks show significant improvements in performance metrics, including recall and efficiency, compared to existing methods.
Methodology
CANOPY employs a multi-fidelity bandit approach that combines cheap internal-node probes for biased estimates with expensive leaf evaluations for accurate results. It constructs an online local-Lipschitz certificate to assess aggregation bias and utilizes a discontinuity-guided adaptive sampling technique to focus evaluations on regions where smoothness assumptions fail. This framework is applied to various LLM inference tasks, including model routing and prompt caching.
Results
Experimental evaluations show that CANOPY achieves 2.9× higher top-10 recall on a 1000-model pool compared to the best structure-blind baseline. It also resolves 1.6× more SWE-bench Verified issues than the best-of-N approach and reduces median time-to-first-token by 3.6× with prefix caching. The method consistently outperforms existing hierarchical bandit strategies across multiple benchmarks.
Implications
The findings suggest that CANOPY can enhance the efficiency and effectiveness of LLM inference tasks, making it a valuable tool for optimizing resource allocation in real-time applications. Its ability to adaptively learn smoothness conditions could lead to broader applications in multi-fidelity optimization problems across various domains.