AI-generated summaries
Today's ML research,
without the noise.
Daily summaries of the latest machine learning papers from arXiv, processed every 8 hours.
57
Papers today
8h
Update frequency
7
Days of history
FlashLoop: Fast and Memory-Efficient Looped Transformers via Lazy Updates
NLP
Large Language Models
Efficient ML
- FlashLoop reduces the computational and memory overhead of Looped Transformers during inference.
- The framework achieves up to 1.64× speedup and up to 6× reduction in KV-cache memory.
- Key innovations include Token-Sparse Updates, Sparse Attention, and KV-Residual Quantization.
- FlashLoop maintains lossless accuracy while improving the practicality of scaling Looped Transformers.
Read more
FlashLoop: Fast and Memory-Efficient Looped Transformers via Lazy Updates
Summary
The paper introduces FlashLoop, a novel inference framework designed to enhance the efficiency of Looped Transformers, which have gained attention for their parameter-efficient approach to increasing computational depth through repeated application of shared Transformer blocks. Despite their potential, the practical advantages of Looped Transformers are often overshadowed by the increased computational and memory overhead associated with additional loops, particularly in terms of inference FLOPs and KV-cache memory. The authors identify that much of this overhead is redundant, as state changes during recurrence become concentrated on a small subset of tokens, and attention-output differences are dominated by a sparse subset of key columns. To address these inefficiencies, FlashLoop employs three key strategies: Token-Sparse Updates, which minimize redundant updates by focusing on active tokens; Sparse Attention, which selectively processes important keys based on previous loop statistics; and KV-Residual Quantization, which compresses the KV cache into a quantized base state with low-bit updates. The framework is implemented efficiently on GPUs, resulting in significant improvements in inference speed and memory usage without sacrificing accuracy.
Methodology
FlashLoop employs a training-free inference framework that focuses on reducing redundancy in Looped Transformers through three main components: Token-Sparse Updates to minimize unnecessary computations, Sparse Attention to selectively focus on important keys, and KV-Residual Quantization to compress the KV cache efficiently. These methods are implemented with hardware-aware optimizations for GPU performance.
Results
The implementation of FlashLoop across various Looped Transformer models demonstrated a lossless retention of accuracy while achieving significant performance enhancements, including up to 1.64× end-to-end speedup and a reduction of KV-cache memory usage by up to 6×, making the inference costs comparable to or lower than those of non-looped models.
Implications
The advancements presented in FlashLoop could enable more efficient deployment of Looped Transformers in real-world applications, particularly in scenarios requiring long context processing and high computational depth without the corresponding increase in resource demands. This could lead to broader adoption of Looped Transformers in NLP tasks and other domains where efficiency is critical.
Time-Series Foundation Models That Understand Data Revisions
Time Series
- Introduction of VINTAGE-TS, a revision-aware time-series forecasting model.
- Distinction between observation time and information-availability time to improve forecasting accuracy.
- Implementation of a joint predictive distribution to model uncertainty in forecasts.
- Comprehensive evaluation framework with rolling evaluations and sensitivity analysis.
Read more
Time-Series Foundation Models That Understand Data Revisions
Summary
This paper introduces VINTAGE-TS, a novel time-series foundation model designed to handle data revisions in forecasting. Traditional forecasting models often assume that historical data is fixed, which can lead to inaccuracies when statistical agencies revise previously published values. VINTAGE-TS addresses this issue by distinguishing between observation time and information-availability time, allowing it to predict both the first-published value and the value available after a specified maturation period. The model employs a joint predictive distribution that captures the uncertainty associated with these targets. The author outlines a comprehensive evaluation framework using ALFRED for rolling evaluations and compares it with the Chronos-2 model, alongside conventional and revision-aware baselines. The paper also discusses the implementation of validity-interval reconstruction and delayed-label filtering, ensuring reproducibility and robustness in diagnostics. Through synthetic experiments and a sensitivity analysis, the paper explores the effects of hindsight contamination on performance metrics. The findings suggest that access to revision histories may enhance forecast accuracy, although no empirical advantage over existing models is claimed at this stage.
Methodology
The methodology involves the development of VINTAGE-TS, which integrates explicit information timing and joint uncertainty modeling. The evaluation framework includes rolling evaluations using ALFRED, comparisons with Chronos-2, and various baseline models. The paper also implements synthetic demonstrations and a sensitivity suite to assess the model's performance under different conditions.
Results
The results indicate that the proposed model can effectively handle data revisions and provide more accurate forecasts by utilizing historical revision data. However, the paper does not claim any empirical advantage over existing models, as the hypothesis regarding the benefits of revision histories is yet to be validated.
Implications
The findings could have significant implications for economic forecasting and other fields where data revisions are common. By improving the accuracy of forecasts in the presence of revised data, VINTAGE-TS may enhance decision-making processes in various domains.
SMILESGNN: Interpretable Clinical Toxicity Prediction via SMILES-Graph Cross-Attention Fusion
Graph Learning
Multimodal
Interpretability
- SMILESGNN integrates SMILES and graph representations using cross-attention for improved toxicity prediction.
- The model addresses severe class imbalance and provides interpretable predictions through GNNExplainer.
- SMILESGNN achieves high performance on ClinTox and Tox21 datasets with a low number of parameters.
- The architecture allows for scaffold-based generalization, crucial for predicting toxicity in novel compounds.
Read more
SMILESGNN: Interpretable Clinical Toxicity Prediction via SMILES-Graph Cross-Attention Fusion
Summary
The paper presents SMILESGNN, a novel multimodal architecture designed for predicting clinical toxicity of drugs by integrating SMILES (Simplified Molecular Input Line Entry System) representations with graph neural networks (GNNs) through a cross-attention mechanism. This approach addresses key challenges in drug toxicity prediction, including severe class imbalance, the need for scaffold-based generalization, and the requirement for interpretable predictions. The architecture combines a SMILES Transformer encoder and a GATv2 graph encoder, allowing for the extraction of complementary information from both modalities. The authors also introduce SMILESGNN-PT, a variant that utilizes a pretrained ChemBERTa-2 backbone. The model is trained using focal loss to mitigate class imbalance and retains an explicit graph branch to facilitate GNNExplainer-based analysis of toxic substructures. The results demonstrate that SMILESGNN achieves competitive performance on the ClinTox dataset with an AUC-ROC of 0.987 and an F1 score of 0.906, while maintaining a low parameter count of 0.4M. On the Tox21 dataset, SMILESGNN-PT shows comparable performance to existing models, confirming the efficacy of the cross-attention fusion approach in maintaining predictive accuracy while enhancing interpretability.
Methodology
The methodology involves a dual-pathway architecture that fuses a SMILES Transformer encoder with a GATv2 graph encoder via cross-attention. The model is trained on the ClinTox and Tox21 datasets, utilizing focal loss to address class imbalance. Data processing includes custom tokenization for SMILES sequences and feature extraction for molecular graphs.
Results
On the ClinTox dataset, SMILESGNN achieves an AUC-ROC of 0.987 ± 0.012 and an F1 score of 0.906 ± 0.039 with only 0.4M parameters. For the Tox21 dataset, SMILESGNN-PT obtains a mean AUC-ROC of 0.750 ± 0.002, demonstrating competitive performance compared to other models.
Implications
The findings suggest that SMILESGNN can significantly enhance the drug discovery process by providing interpretable predictions of toxicity, thereby helping researchers prioritize safer drug candidates and reduce the risk of late-stage attrition in clinical trials.
Improving Calibration of Black-Box Radiology AI Using Test-Time Augmentation
Computer Vision
- Introduces a model-agnostic framework for improving calibration of black-box AI models using test-time augmentation.
- Demonstrates significant reduction in Expected Calibration Error for pulmonary embolism and intracranial hemorrhage detection tasks.
- DualTTA outperforms traditional calibration techniques that require access to model internals.
- Provides a clinically reviewed library of 3D CT augmentations and aggregation strategies.
Read more
Improving Calibration of Black-Box Radiology AI Using Test-Time Augmentation
Summary
This paper addresses the critical issue of calibration in black-box radiology AI systems, which are increasingly used for clinical decision-making. Calibration refers to the alignment between predicted probabilities and actual outcomes, which is essential for safe clinical use. Traditional calibration techniques often require access to model internals or retraining, which is not feasible for proprietary systems. The authors propose a novel model-agnostic framework that utilizes test-time augmentation (TTA) to enhance calibration without needing internal model access. This framework employs geometric and physics-inspired 3D CT perturbations and learns aggregation strategies to improve the reliability of predictions. The proposed method, termed DualTTA, combines different augmentation types and weights them based on their reliability. The framework was evaluated on tasks related to pulmonary embolism and intracranial hemorrhage detection, demonstrating significant improvements in calibration metrics. The results indicate that DualTTA outperforms traditional uncertainty estimation techniques, showcasing its potential as a practical solution for enhancing the calibration of black-box medical AI systems.
Methodology
The authors developed a framework that applies test-time augmentation (TTA) to black-box classifiers, utilizing a library of geometric and physics-inspired 3D CT perturbations. They implemented a learned aggregation strategy, DualTTA, which combines predictions from various augmentations while accounting for their reliability. The framework operates solely on input-output access, making it suitable for proprietary AI systems.
Results
The proposed DualTTA method achieved a 54% reduction in Expected Calibration Error for pulmonary embolism detection and a 43% reduction for intracranial hemorrhage detection. It outperformed several traditional calibration techniques, including Temperature Scaling and MC Dropout, across various calibration metrics.
Implications
The findings suggest that the proposed framework can significantly enhance the reliability of black-box medical AI systems, making them safer for clinical use. This approach could lead to broader adoption of AI in radiology and other medical fields where calibration is crucial for decision-making.
Automatic Rank Allocation for Low-Rank Adaptation in Large Language Models via ℓp Regularization
NLP
Large Language Models
Efficient ML
- Introduces ℓp-LoRA, a novel method for automatic rank allocation in LoRA.
- Utilizes ℓp regularization to encourage sparsity in adaptation matrices.
- Derives a proximal subproblem that simplifies the optimization process.
- Demonstrates competitive performance on natural language tasks compared to existing methods.
Read more
Automatic Rank Allocation for Low-Rank Adaptation in Large Language Models via ℓp Regularization
Summary
This paper addresses the challenge of rank allocation in Low-Rank Adaptation (LoRA) for large language models, a method that allows for parameter-efficient fine-tuning. Traditional approaches to rank allocation often rely on manually designed importance scores, which may not be optimal. The authors propose a new method called ℓp-LoRA, which utilizes ℓp regularization (with 0 < p < 1) to automatically determine the rank of adaptation matrices. This method encourages the elimination of redundant components while retaining those that are essential for performance. By formulating the problem as a structured sparsity issue, the authors derive a proximal subproblem that simplifies the optimization process. Experiments conducted on natural language understanding and question-answering tasks show that ℓp-LoRA achieves competitive performance compared to existing LoRA methods, demonstrating its effectiveness in rank allocation and parameter efficiency.
Methodology
The authors formulate the rank allocation problem as a structured sparsity problem, applying ℓp regularization to the energies of rank-one components in LoRA. This approach leads to an implicit thresholding criterion that identifies and removes redundant components. The optimization is reduced to a two-dimensional problem, allowing for efficient computation.
Results
The proposed ℓp-LoRA method was tested on various natural language understanding and question-answering tasks, yielding competitive results when compared to existing LoRA baselines. The experiments indicate that ℓp-LoRA effectively allocates ranks based on the importance of components, leading to improved performance and efficiency.
Implications
The findings suggest that ℓp-LoRA can enhance the efficiency of fine-tuning large language models by optimizing rank allocation, potentially leading to reduced computational costs and improved model performance in various NLP tasks.
Diverse Geometries, Frozen Weights: Robust Heterogeneous Treatment-Effect Estimation via Causal Expert Ensembles
Theory
- Introduction of GeoACE, a five-expert framework for HTE estimation.
- OΦ-ACE expert enhances robustness by providing an outcome-free, overlap-aware projection.
- Weights for the ensemble are learned from internal validation and frozen to prevent test leakage.
- GeoACE outperformed traditional methods in multiple benchmarks, particularly in reducing mean squared error.
Read more
Diverse Geometries, Frozen Weights: Robust Heterogeneous Treatment-Effect Estimation via Causal Expert Ensembles
Summary
This paper addresses the challenge of estimating heterogeneous treatment effects (HTE) from observational data, where the appropriate inductive bias can vary significantly due to factors like treatment imbalance and sample size. The authors propose a novel framework called Geometry-Diverse Anchor–Correction Expert Ensemble (GeoACE), which consists of five experts that leverage a common anchor–correction estimator while incorporating diverse geometries that are aware of overlap and outcomes. The ensemble weights are determined from internal validation predictions and are frozen prior to testing, ensuring that test data does not influence the model's learning. One of the experts, OΦ-ACE, introduces an outcome-free, overlap-aware statistical projection that enhances the model's robustness. The performance of GeoACE is evaluated against 11 other methods across eight benchmark protocols, demonstrating significant improvements in mean squared error metrics. The findings suggest that the combination of diverse expert geometries and a leakage-free aggregation strategy can enhance the robustness of treatment-effect estimation without asserting universal superiority over existing methods.
Methodology
The GeoACE framework consists of five causal experts built on a shared anchor–correction backbone. The ensemble utilizes a task-level convex weight vector learned from internal validation predictions, which is frozen before the final evaluation. The OΦ-ACE expert provides a lower-dimensional representation of the data, enhancing the model's ability to handle treatment effect heterogeneity. The performance of GeoACE is compared against various existing methods using multiple benchmark datasets.
Results
GeoACE demonstrated a significant reduction in mean squared error across seven benchmarks, winning 998 out of 1,225 paired tasks. It ranked first on several benchmarks, including IHDP100 and IHDPA, and achieved the lowest average rank across all tests. The results indicated that using the frozen expert ensemble with inverse-DR weighting consistently outperformed other methods, although it was statistically indistinguishable from some simpler approaches.
Implications
The findings suggest that employing a diverse set of causal experts and a robust aggregation strategy can improve the reliability of treatment-effect estimates in observational studies. This approach may be particularly beneficial in fields such as healthcare and social sciences, where understanding heterogeneous treatment effects is crucial for decision-making.
RLVR landscapes for iterated multiplications can be benign: Insights from spin-glass theory
Reinforcement Learning
Theory
Large Language Models
- The RLVR landscape for certain algorithmic tasks is benign, lacking local minima.
- Challenges in RLVR arise from diffusive barriers and gradient estimation errors rather than landscape ruggedness.
- The choice of entropy regulator can mitigate obstacles in RLVR optimization.
- Transformers trained with strong regulation can successfully learn complex reasoning tasks.
Read more
RLVR landscapes for iterated multiplications can be benign: Insights from spin-glass theory
Summary
This paper investigates the optimization landscape of Reinforcement Learning with Verifiable Rewards (RLVR) in the context of algorithmic tasks, specifically iterated group and quasigroup multiplications. By mapping RLVR onto an energy-based spin-glass model, the authors provide a theoretical framework to analyze the landscape of RLVR. They demonstrate that for a broad class of models with uncorrelated inputs, the RLVR landscape is benign, lacking local minima that could hinder training. Instead, challenges arise from diffusive barriers and gradient estimation errors. The authors show that these obstacles can be alleviated through careful selection of entropy regulators. Empirical results indicate that a transformer model trained from scratch using last-token rewards can effectively learn the task of iterated non-Abelian group multiplications, supporting the theoretical findings.
Methodology
The authors employ a theoretical framework that maps RLVR onto spin-glass models, allowing for rigorous analysis of the optimization landscape. They utilize myopic tabular policies for numerical experiments and derive closed expressions for the free energy of the spin-glass model associated with specific tasks.
Results
The study reveals that the RLVR landscape is generally benign for untied models, with no suboptimal local maxima. However, local maxima can appear with weight tying and input correlations. The authors demonstrate that Monte Carlo sampling can effectively solve these tasks, while traditional RLVR optimization struggles due to its inability to navigate the landscape effectively. Strongly regulated transformers are shown to perform well in learning the tasks when augmented with a curriculum.
Implications
The findings suggest that RLVR can be effectively applied to complex reasoning tasks without being hindered by local minima, provided that appropriate entropy regulation is employed. This has potential implications for improving the training of large language models and other AI systems that rely on RLVR for learning.
Spectral Graph Neural Networks with Hermite Polynomials: A Comprehensive Study
Graph Learning
Theory
Optimization
- Introduction of HermNet, a spectral GNN model based on Hermite polynomials.
- Analysis of optimization behavior across different polynomial bases under limited training budgets.
- Identification of conditions that enhance HermNet's performance, including curvature regularization.
- Empirical evidence showing HermNet's advantages in synthetic experiments compared to other polynomial-basis models.
Read more
Spectral Graph Neural Networks with Hermite Polynomials: A Comprehensive Study
Summary
This paper presents a comprehensive study of spectral graph neural networks (GNNs) utilizing Hermite polynomials, introducing a novel model called HermNet. HermNet integrates a nodewise predictor with normalized Hermite propagation, characterized by its sparse recurrence that eliminates the need for eigendecomposition or a learned basis. The study distinguishes between the basic HermNet model and optional enhancements such as coordinate calibration, response normalization, and Gaussian derivative regularization. The authors analyze the optimization behavior of different polynomial bases, including Hermite, through various factors such as spectral signal energy, label sampling, and the bias-variance trade-off of regularization. Controlled synthetic experiments reveal that HermNet outperforms other polynomial-basis alternatives under limited training budgets, particularly when curvature regularization is applied. However, the advantages diminish with longer training periods. The paper also discusses the accuracy deficits observed in real-data comparisons and identifies architectural and numerical limitations. Overall, the findings clarify the conditions under which Hermite propagation is beneficial and how calibration and regularization impact performance.
Methodology
The methodology involves the formulation of HermNet for classification and regression tasks, utilizing a sparse propagation algorithm that leverages Hermite polynomials. The study conducts controlled synthetic experiments to evaluate performance against other polynomial-basis models, while also analyzing the effects of various enhancements such as calibration and regularization.
Results
The results indicate that HermNet achieves lower prediction error in synthetic evaluations compared to matched polynomial-basis alternatives, particularly under short training budgets. Curvature regularization further enhances performance. However, the advantages of HermNet diminish with longer training periods, and real-data comparisons reveal accuracy deficits.
Implications
The findings suggest that Hermite polynomials can be effectively utilized in spectral GNNs, particularly in scenarios with limited training resources. The insights into calibration and regularization can inform the design of more efficient GNN architectures, potentially improving their applicability in various graph learning tasks.
Common Covariance Geometry and Certification for Brownian Kernel Ladders
Theory
- Introduction of minimum-trace common covariance for Brownian kernel ladders.
- Development of tools for transforming covariance problems into geometric forms.
- Establishment of a universal Gaussian-complexity bound.
- Illustration of the difference between covariance certification and predictive selection.
Read more
Common Covariance Geometry and Certification for Brownian Kernel Ladders
Summary
This paper introduces a novel approach to the problem of covariance in the context of Brownian kernel ladders, which are used in adaptive kernel methods. The author presents the concept of minimum-trace common covariance that governs the empirical union generated by these kernels. This approach leads to a universal Gaussian-complexity bound and provides a framework for understanding the geometry of covariance in reproducing kernel Hilbert spaces (RKHS). The paper develops several mathematical tools, including a closed last-layer Dirac-trace reduction and a signed Brownian threshold representation, which transform the covariance problem into various geometric forms, such as threshold and effective-resistance geometry. The author also discusses the implications of these tools in terms of deterministic depth laws, Gaussian reverses, and empirical Kolmogorov-width formulas. Furthermore, the paper illustrates the distinction between covariance certification and predictive selection through a finite covariance-indexed Brownian path model, demonstrating that while all path certificates succeed, predictive selection may miss certain criteria. Overall, this work establishes a finite-dimensional covariance object that connects unrestricted kernel adaptation, Gaussian geometry, and certifiable computation.
Methodology
The paper employs mathematical techniques involving absolutely two-summing operators and covariance-dominated multipliers to derive exact formulations for the minimum-trace common covariance. It also utilizes geometric transformations to analyze the covariance problem, including Dirac-trace reductions and threshold representations.
Results
The main results include the derivation of a universal Gaussian-complexity bound, the formulation of an exact empirical Kolmogorov-width formula, and the establishment of deterministic depth laws and conditional Gaussian reverses. The paper also provides lower and upper certificates for covariance through active semidefinite programs.
Implications
The findings have significant implications for adaptive kernel methods in machine learning, particularly in improving the understanding of covariance structures and enhancing the robustness of predictive models. The certification techniques developed could be applied to various machine learning tasks requiring reliable covariance estimation.
An Agnostic Sample Compression Scheme for Squared Loss of Near-Linear Size in the Fat-Shattering Dimension
Theory
Efficient ML
- Introduces an agnostic sample compression scheme for empirical squared loss with near-linear size.
- Eliminates the dual fat-shattering dimension factor present in previous constructions.
- Achieves compression size independent of the sample size, focusing on the fat-shattering dimension.
- Utilizes a boosting method that targets a (1 - ε)-fraction of sample points for efficient learning.
Read more
An Agnostic Sample Compression Scheme for Squared Loss of Near-Linear Size in the Fat-Shattering Dimension
Summary
This paper presents a novel agnostic sample compression scheme for empirical squared loss applicable to any function class F within the range [0, 1]^X. The proposed scheme achieves a compression size of O(fat(F, c'α) · log^3(2/α)), independent of the sample size m, while ensuring that the reconstructed function ˆf maintains an L2 loss within α of the best function in class F. This work addresses an open problem posed by Attias et al. (2024) regarding the existence of an α-approximate agnostic ℓ2 compression scheme of size fat(F, cα) · Polylog(c/α). The key innovation is the elimination of the dual fat-shattering dimension factor, which has been a significant limitation in previous constructions. The methodology leverages a boosting approach that targets a fraction of sample points, allowing for a more efficient aggregation of weak learners without the need for sparsification. The results indicate that the proposed scheme not only meets the requirements of the open problem but also provides a worst-case exponential improvement over existing methods, making it a significant advancement in the field of sample compression for regression tasks.
Methodology
The methodology involves a boosting algorithm that aggregates weak learners without requiring uniform accuracy across all sample points. By focusing on a subset of the sample, the scheme avoids the need for sparsification, which has been a common source of complexity in prior works. The reconstruction process uses quantized side-information bits to convey synthetic labels derived from a near-optimal function, allowing for effective learning while maintaining the desired accuracy.
Results
The main result confirms the existence of a deterministic agnostic sample compression scheme for empirical squared loss that adheres to the size constraints specified in the open problem. The scheme achieves a compression size of O(fat(F, c'α) · log^3(2/α)), demonstrating a significant improvement over previous methods by removing the dual fat-shattering dimension factor.
Implications
This work has potential implications for improving the efficiency of machine learning models, particularly in regression tasks where sample size can be large. The findings could lead to more effective data storage and processing techniques, making it easier to deploy machine learning models in resource-constrained environments.
Temporal Gradient Inversion for Private Trajectory Reconstruction in Embodied Reinforcement Learning
Reinforcement Learning
Robotics
Theory
- Introduction of TRACE, a temporal gradient inversion attack for embodied RL.
- Exploitation of temporal correlations in policy gradients to enhance reconstruction accuracy.
- Theoretical proof of closed-form action recovery and characterization of temporal gain.
- Empirical results show TRACE outperforms existing methods in reconstruction fidelity.
Read more
Temporal Gradient Inversion for Private Trajectory Reconstruction in Embodied Reinforcement Learning
Summary
This paper introduces TRACE (Temporal Reconstruction Attack on Consecutive Encodings), a novel attack method designed to reconstruct private observation-action trajectories from policy gradients in embodied reinforcement learning (RL) settings. The authors highlight the privacy risks associated with gradient inversion attacks, particularly in distributed RL environments where agents transmit gradients while keeping raw sensor data on-device. TRACE leverages temporal correlations between consecutive gradients, which previous single-frame methods overlooked. The method employs an autoregressive model that includes a gradient encoder, a causal transformer for contextualizing trajectory information, and a decoder for reconstructing observations and predicting actions. The authors provide a theoretical foundation for the attack, proving closed-form action recovery and characterizing the temporal reconstruction gain. Empirical evaluations demonstrate TRACE's superior performance in terms of reconstruction fidelity and action recovery compared to existing methods, achieving high PSNR values and rapid reconstruction times across various architectures and environments. The findings suggest that protecting against such attacks may require advanced sequence-aware privacy mechanisms.
Methodology
TRACE employs an autoregressive model that consists of three main components: a gradient encoder that maps gradient updates to compact latent representations, a causal transformer that processes temporal context, and a decoder that reconstructs observations and predicts actions. The method is designed to operate in near real-time, leveraging the temporal structure of the gradient streams.
Results
TRACE achieves an average PSNR of 18.8 dB with near-perfect action recovery, significantly outperforming both optimization-based and learning-based baselines. The method demonstrates robust performance across various unseen environments and architectures, achieving strong zero-shot reconstruction capabilities.
Implications
The findings underscore the need for enhanced privacy mechanisms in distributed RL settings, particularly those involving embodied agents. The results suggest that traditional defenses may be insufficient, prompting the exploration of sequence-aware privacy strategies.
Limited Structural Reliability in Public Educational Prediction Benchmarks: A Four-Dimension Audit of Seven Datasets
Theory
- Only three out of seven audited educational datasets passed all reliability checks.
- The dominant failure mode was cross-group fragility, not weak model performance.
- The audit protocol can effectively identify genuine confounding issues in datasets.
- Increasing model complexity does not necessarily improve stability in fragile datasets.
Read more
Limited Structural Reliability in Public Educational Prediction Benchmarks: A Four-Dimension Audit of Seven Datasets
Summary
This paper conducts a comprehensive audit of seven public educational prediction datasets to assess their structural reliability before modeling. The authors apply a four-dimension framework that includes baseline gap, split instability, null separation, and metadata adequacy to evaluate the datasets. The audit reveals that only three datasets passed all four reliability checks, while the remaining four exhibited issues such as group-aware generalization failures or inadequate provenance metadata. Notably, one dataset was reclassified after correcting for group-identifier features, highlighting the audit's ability to differentiate between true cross-group confounding and artifacts from feature encoding. The findings indicate that the primary threat to benchmark validity in educational AI stems from data structure and evaluation design rather than model sophistication. The paper emphasizes the importance of pre-modeling audits to ensure the reliability of educational datasets, which are often more vulnerable to group-level confounding due to their nested structures. The results underscore the need for rigorous evaluation protocols in educational AI to support credible modeling claims.
Methodology
The authors developed a four-dimension audit protocol to evaluate each dataset using baseline gap, split instability, null separation, and metadata adequacy. This involved applying statistical diagnostics and linear baselines to assess the datasets before model optimization.
Results
The audit found that three datasets met all four reliability criteria, while four failed due to issues like group-aware generalization and lack of necessary metadata. The UCI Student dataset showed a significant drop in performance from iid R2 = 0.242 to group-holdout R2 = −0.097, illustrating the impact of cross-group fragility.
Implications
The findings suggest that educational AI benchmarks may be misleading if dataset reliability is not rigorously assessed. The proposed audit framework can serve as a quality gate for future datasets, ensuring that they support credible modeling claims and contribute to more reliable educational interventions.
Intrinsic-Extrinsic Coupling in Learning Dynamics
Theory
- Intrinsic-extrinsic coupling can significantly influence learning dynamics and outcomes.
- Observation-relative fibers provide a framework for understanding the relationship between current behavior and future learning responses.
- Replay mechanisms can lead to varying contributions to prediction accuracy, sometimes resulting in negative interactions.
- The study distinguishes between local repairs and favorable outputs, emphasizing the complexity of learning dynamics.
Read more
Intrinsic-Extrinsic Coupling in Learning Dynamics
Summary
This paper explores the concept of intrinsic-extrinsic coupling in the learning dynamics of machine learning models. The author formulates this coupling through the continuation-conditioned value of constrained learning-state interventions, emphasizing that a learner's current observations do not solely dictate its future training responses. The study introduces the notion of observation-relative fibers to describe the agreement between present behavior and future learning responses. The methodology involves an executable finite-frame classifier-head that protects current logits while allowing for historical margin repairs under finite-precision checks. The paper distinguishes between local admissibility, continuation-conditioned intervention value, and complete-policy performance. Through a series of experiments, including a class-incremental setting derived from CLINC, the author demonstrates how replay mechanisms can alter the contribution of updates to prediction accuracy. The findings reveal that intrinsic-extrinsic interactions can lead to negative outcomes under certain conditions, challenging the assumption that such coupling always results in positive synergy. The mathematical analysis further clarifies the relationship between feasible local repairs and favorable terminal outputs, while separate coordination tests indicate that content controls can match or exceed development gains. Overall, the paper operationalizes intrinsic-extrinsic coupling by linking state geometry to value assessments and interaction identification, thereby providing insights into the dynamics of learning processes.
Methodology
The paper employs a mathematical framework to analyze intrinsic-extrinsic coupling, utilizing finite-frame classifiers and controlled interventions on learning states. Experiments are conducted in a class-incremental setting, with comparisons made across different training policies and replay mechanisms to assess their impact on prediction accuracy.
Results
The experiments reveal that replay can change the contribution of updates from five correct predictions to zero, indicating that intrinsic-extrinsic interactions can be detrimental. Additionally, the analysis shows that guided allocation strategies outperform standard replay in terms of mean loss across multiple test scenarios, highlighting the root-dependent nature of these effects.
Implications
The findings suggest that understanding intrinsic-extrinsic coupling can lead to improved training strategies in machine learning, particularly in scenarios involving incremental learning and replay mechanisms. This could enhance model performance and adaptability in dynamic environments.
Stable and Faithful Explanations for Knowledge Tracing
Interpretability
- Introduces a validation protocol for assessing KT model explanations.
- Demonstrates that XGBoost can provide stable and interpretable explanations compared to deep learning models.
- Highlights the importance of feature importance stability and faithfulness in educational data mining.
- Rebuilding the ASSISTments dataset mitigated data leakage and improved model performance.
Read more
Stable and Faithful Explanations for Knowledge Tracing
Summary
This paper addresses the challenge of explainability in Knowledge Tracing (KT) models, which are crucial for predicting student performance but often operate as black boxes. The authors propose a validation protocol that assesses predictive accuracy, explanation stability, and faithfulness of explanations in KT models. They engineered thirteen behavioral features from the ASSISTments datasets (2009 and 2012) and rebuilt the 2009 dataset to mitigate data leakage issues. An Extreme Gradient Boosting (XGBoost) model was employed and explained using Tree SHapley Additive exPlanations (TreeSHAP), and its performance was compared against four deep learning baselines (DKT, SAKT, AKT, SimpleKT) under an information-matched protocol. The results showed that XGBoost achieved competitive AUC scores while providing stable and interpretable explanations. The study emphasizes the importance of systematic validation of explanations in KT literature, highlighting that the stability of feature importance rankings and the faithfulness of explanations are critical for effective pedagogical interventions.
Methodology
The authors engineered behavioral features from ASSISTments datasets and rebuilt the 2009 dataset to address data leakage. They employed an XGBoost model explained with TreeSHAP and compared its performance against four deep learning KT models under an information-matched protocol. The study systematically validated explanation stability and faithfulness through various tests.
Results
XGBoost achieved AUC scores of 0.777 on the 2012 dataset and 0.786 on the rebuilt 2009 dataset. The model's performance was comparable to deep learning baselines when restricted to the same information, indicating that differences in performance were due to the information supplied rather than model architecture. The stability of feature rankings was confirmed across multiple folds and seeds.
Implications
The findings suggest that feature-based models like XGBoost can be effectively used in educational contexts to provide interpretable predictions, which can inform instructional strategies. The validation framework established in this study can guide future research in explainable AI, particularly in educational data mining.
A Contraction Framework for Stochastic Operators with Bootstrapping: Application to TD Learning
Reinforcement Learning
Theory
Optimization
- Introduces a contraction framework for stochastic operators applicable to TD learning.
- Establishes convergence guarantees without assuming linearity or fixed update structures.
- Derives finite-time bounds for i.i.d. samples and arbitrary target-update periods.
- Demonstrates geometric convergence of iterates in root mean square to a fixed point.
Read more
A Contraction Framework for Stochastic Operators with Bootstrapping: Application to TD Learning
Summary
This paper presents a novel contraction framework for analyzing the convergence of stochastic operators in the context of temporal-difference (TD) learning, particularly when using bootstrapping techniques. Traditional convergence guarantees for TD learning often rely on specific structures of updates, such as linear approximations or gradient-based methods, and assume uniformly bounded sampling errors. The authors propose a more general approach by modeling the sampled updates as stochastic operators, which allows for a broader analysis without the need for gradient structures. They derive finite-time convergence bounds for independent and identically distributed (i.i.d.) samples and arbitrary target-update periods, demonstrating that the iterates converge geometrically in root mean square to a neighborhood around the fixed point. The framework accommodates non-linear approximators and varying target-update intervals, thus extending the applicability of existing results. The authors validate their theoretical findings through simulations of TD learning, which align with the predicted contraction rates and error floor behaviors based on step sizes.
Methodology
The authors utilize an operator-theoretic perspective to formulate the TD learning algorithm, treating the updates as stochastic operators. They analyze the convergence properties of these operators under the assumption of i.i.d. samples and derive conditions for geometric convergence without relying on linearity or specific update structures.
Results
The paper establishes that the iterates of the TD learning algorithm converge geometrically in root mean square to a ball around the fixed point, provided the sensitivity to the frozen target is less than the contraction slack of the inner map. The results include finite-time bounds applicable to various target-update periods and demonstrate that existing deterministic and stochastic bounds are special cases of the proposed framework.
Implications
The findings have significant implications for reinforcement learning, particularly in environments where bootstrapping and non-linear approximators are prevalent. This framework can enhance the understanding and implementation of TD learning algorithms, potentially leading to more robust and efficient learning strategies in complex decision-making scenarios.
Graph, Loop, and Harness Engineering for Zero-Trust Agentic Data Engineering and Analytical Processing
NLP
Large Language Models
Graph Learning
- Introduction of two frameworks: Zero-Trust Agentic Data Engineering and Zero-Trust Agentic OLAP.
- Utilization of graph, loop, and agent-harness engineering as shared abstractions for workflow management.
- Frameworks ensure that task completion is based on independently verified evidence rather than agent-reported success.
- Empirical evaluation shows improved reliability and security in cloud data engineering and analytical processing.
Read more
Graph, Loop, and Harness Engineering for Zero-Trust Agentic Data Engineering and Analytical Processing
Summary
This paper presents two innovative frameworks aimed at enhancing the reliability and security of cloud data engineering and analytical processing through a zero-trust approach. The first framework, Zero-Trust Agentic Data Engineering, automates the generation, deployment, and verification of comprehensive cloud data-engineering solutions from natural-language tasks. It ensures that the completion of tasks is contingent upon evidence from repositories, deployments, runtime environments, and policies. The second framework, Zero-Trust Agentic OLAP, integrates governed data preparation with verified Online Analytical Processing (OLAP), allowing for production promotion only after thorough validation and evidence-based approval. Both frameworks utilize three core abstractions: graph engineering, which structures workflows with explicit stages and dependencies; loop engineering, which manages bounded recovery from failures; and agent-harness engineering, which enforces a zero-trust execution environment. The authors evaluate these frameworks under various conditions, measuring their effectiveness in verified completion, recovery, authorization enforcement, and OLAP execution. The findings demonstrate that both frameworks significantly enhance the reliability and security of data workflows, addressing common failures in cloud-native data engineering.
Methodology
The authors developed two frameworks that transform natural-language tasks into actionable cloud data-engineering solutions and verified analytical processes. They employed graph engineering to structure workflows, loop engineering for managing failures, and agent-harness engineering to enforce security. The frameworks were empirically evaluated using benchmark suites under various operational conditions, including nominal execution and controlled failures.
Results
The evaluation of both frameworks demonstrated significant improvements in verified completion rates, effective recovery from failures, and robust enforcement of authorization policies. The Zero-Trust Agentic OLAP framework successfully released verified analytical results only after stringent validation processes were met.
Implications
These frameworks can be applied to enhance the automation of data workflows in cloud environments, ensuring that data engineering and analytical processes are both reliable and secure. They are particularly relevant for industries that require stringent compliance and governance in data handling and analysis.
Reachability-Based Formal Verification of Graph Neural Networks with Node and Edge Features
Graph Learning
- Introduction of GraphStar sets for uncertainty representation in GNNs.
- Extension of the NNV framework to support verification of GCN and GINE architectures.
- Demonstration of tighter robustness guarantees compared to existing verification methods.
- First edge-aware reachability analysis for GINE-based models under joint perturbations.
Read more
Reachability-Based Formal Verification of Graph Neural Networks with Node and Edge Features
Summary
This paper addresses the formal verification of Graph Neural Networks (GNNs), particularly in the context of electric power systems where GNNs are used for tasks such as power flow analysis, optimal power flow estimation, and cascading failure analysis. The authors extend the neural network verification (NNV) framework to accommodate graph-structured inputs through the introduction of GraphStar sets, which generalize Star sets to capture uncertainties in both node and edge features. This extension allows for the propagation of linear message-passing operations and the sound approximation of ReLU nonlinearities in GNN architectures, specifically for graph convolutional networks (GCN) and graph isomorphism networks with edge features (GINE). The paper evaluates the proposed GNNV framework on three power system tasks across various IEEE test cases and two standard graph classification benchmarks. The results demonstrate that GNNV offers tighter robustness guarantees compared to existing methods, particularly for GINE-based models under joint perturbations of node and edge features, marking a significant advancement in the formal verification of GNNs.
Methodology
The authors developed GraphStar sets to extend the NNV framework, enabling the verification of GNNs with both node and edge features. They implemented reachability analysis techniques to propagate uncertainties through GNN architectures, specifically focusing on GCN and GINE layers. The methodology involved rigorous testing on power system tasks and standard graph classification datasets to validate the effectiveness of the proposed verification framework.
Results
The evaluation showed that GNNV provides tighter robustness guarantees than CORA, particularly for graph classification models with ReLU activations. The framework successfully maintained verification at perturbation levels where CORA failed and achieved edge-aware robustness guarantees for GINE-based models under joint perturbations of node and edge features.
Implications
The findings suggest that GNNV can enhance the reliability of GNN applications in safety-critical domains, such as electric power systems, by providing formal verification guarantees. This advancement could lead to broader applications of GNNs in other fields requiring robust and verified models, such as molecular modeling and traffic forecasting.
Where Does Exactly-Once Live? Model, Harness, and Tool-Contract Effects on Duplicate Side Effects in LLM Agents
Large Language Models
Theory
NLP
- Exactly-once behavior in LLM agents is influenced by the type of network fault encountered.
- Frontier models can significantly reduce duplication rates when read-back is possible.
- Idempotency keys are crucial in minimizing duplicate actions, reducing rates from 28% to 4%.
- The agent harness has minimal impact on the overall performance regarding exactly-once behavior.
Read more
Where Does Exactly-Once Live? Model, Harness, and Tool-Contract Effects on Duplicate Side Effects in LLM Agents
Summary
This paper investigates the challenges of ensuring exactly-once behavior in tool-using language model (LLM) agents, particularly in the context of network failures that can lead to duplicate actions. The author introduces Limbo, a deterministic sandbox designed to simulate various service contracts and fault conditions, allowing for a comprehensive analysis of how exactly-once behavior can be enforced across different components: the model, the agent harness, and the tool contract. Through extensive experimentation involving 25,930 episodes across multiple models and harnesses, the study reveals that the effectiveness of ensuring exactly-once behavior is highly dependent on the type of fault encountered. When a read-back can confirm the outcome of an action, frontier models exhibit a low duplication rate (0.5%). However, in cases where the request is still in flight or redelivered, duplication rates rise significantly (56% and 74%, respectively), with the tool contract explaining a substantial portion of the variance in these outcomes. The findings highlight the limitations of verification-only policies and demonstrate that providing idempotency keys can drastically reduce duplication rates. The paper concludes that while the harness has minimal impact on performance, agents often fail to recognize when they have duplicated actions, reporting completion in 90% of such cases.
Methodology
The study employs a factorial experimental design using Limbo, a sandbox that simulates six services with realistic contracts and twelve fault modes. The experiments assess the performance of nine recent models across three production agent harnesses under various recovery conditions, grading each episode against a ledger of committed effects.
Results
The results indicate that when immediate read-back is possible, frontier models achieve a duplication rate of only 0.5%. In contrast, when the request is still in flight or redelivered, duplication rates increase to 56% and 74%, respectively. The introduction of idempotency keys leads to a significant reduction in duplicates, while the harness's influence on performance is minimal. Agents reported successful completion in 90% of episodes where duplication occurred.
Implications
The findings suggest that improving the design of tool contracts and incorporating idempotency keys can enhance the reliability of LLM agents in real-world applications. This research also emphasizes the need for better awareness and reporting mechanisms for agents to recognize duplicate actions, which could improve user trust and system reliability.
Thinking Leakage: A Causal Audit of NoThink Post-Training in Hybrid Reasoning Models
Large Language Models
Theory
Interpretability
- Thinking leakage is defined as the causal contribution of existing THINK mode behaviors to NOTHINK post-training gains.
- The study employs bidirectional interventions to demonstrate the causal role of thinking leakage in performance improvements.
- A leakage ratio is derived to quantify the extent to which NOTHINK gains depend on THINK mode behaviors.
- The research shows that post-training gains can obscure whether improvements are due to new capabilities or re-invoked existing behaviors.
Read more
Thinking Leakage: A Causal Audit of NoThink Post-Training in Hybrid Reasoning Models
Summary
This paper investigates the phenomenon of 'thinking leakage' in hybrid reasoning models that utilize both THINK and NOTHINK modes for inference. The authors propose a causal mediation framework to analyze how post-training in NOTHINK mode may inadvertently leverage reasoning capabilities already present in the THINK mode. Through a series of experiments involving three different models and post-training methods, the study reveals that a significant portion of the performance gains observed in NOTHINK mode can be attributed to this leakage, with a leakage ratio ranging from 42% to 79%. The findings indicate that the apparent advantages of post-training may not solely stem from newly acquired capabilities but rather from a re-engagement of existing reasoning behaviors. This work highlights the importance of understanding the underlying mechanisms of performance improvements in hybrid models, suggesting that gains in NOTHINK performance could be misleading if not properly audited for thinking leakage.
Methodology
The authors formalize thinking leakage within a causal mediation framework and operationalize it through a direction in the residual stream of model activations. They conduct experiments on three models using various post-training methods, employing bidirectional interventions to assess the causal impact of thinking leakage on NOTHINK performance.
Results
The analysis reveals that thinking leakage is substantial, with leakage ratios between 42% and 79% across nine checkpoints. Steering the base model's representations towards THINK mode reproduces most of the post-training accuracy gains, while counter-steering reduces these gains significantly, confirming the causal role of thinking leakage.
Implications
The findings suggest that researchers and practitioners should carefully consider the effects of thinking leakage when evaluating hybrid reasoning models. Understanding this phenomenon can lead to more accurate assessments of model capabilities and inform the design of future training methodologies.
The Mechanics of Delta Learning: Target Design for Generalizable Scientific Machine Learning
Graph Learning
- Residual scale alone is insufficient for ensuring learnability in delta learning.
- Complex local descriptor baselines can yield smaller but rougher residuals, making them harder to learn.
- Semi-empirical baselines improve both target scale and normalized roughness, enhancing model performance.
- The introduced DIQR metric serves as a pre-training diagnostic for assessing target smoothness.
Read more
The Mechanics of Delta Learning: Target Design for Generalizable Scientific Machine Learning
Summary
This paper addresses the challenges of generalization in scientific machine learning, particularly in the context of delta learning, which trains models on residual errors relative to physical baselines. The authors argue that merely reducing residual scale is not sufficient for improving model learnability. They demonstrate that complex local descriptor baselines can lead to smaller residual targets that are harder to learn due to increased roughness in architecture-informed proxy spaces. In contrast, using semi-empirical baselines can reduce both the scale and roughness of targets, enhancing prediction accuracy both in-domain and out-of-domain. The paper introduces a new diagnostic measure, scale-normalized graph Dirichlet roughness (DIQR), to evaluate residual learnability prior to training. The authors conduct experiments using molecular graph neural networks, specifically GemNet-T and SchNet, to validate their target design principles and assess the impact of baseline selection on model performance. The findings suggest that target design, alongside model architecture, is crucial for improving generalization in scientific machine learning applications.
Methodology
The authors evaluate molecular graph neural networks on total energy targets, testing various baselines that complement the model's inductive bias. They introduce DIQR as a diagnostic tool for assessing target variation and learnability, and perform experiments using GemNet-T and SchNet architectures to analyze the impact of different baseline strategies on model performance.
Results
The results indicate that replacing linear descriptor baselines with more accurate MLP fits leads to smaller but rougher residuals, negatively impacting learnability. Conversely, using the xTB baseline reduces target scale and normalized roughness, resulting in improved accuracy for both in-domain and out-of-domain predictions. The study establishes that delta learning can be framed as a two-dimensional target-design problem involving both residual scale and representation-space smoothness.
Implications
The findings suggest that careful target design is essential for enhancing the generalizability of scientific machine learning models. This approach can be applied to various domains where complex target properties are involved, potentially leading to more robust and accurate predictive models in materials science and other scientific fields.
TAM-Chain: Multi-Scale Thyroid Cytology Classification via Absorbing Markov Chains and Shannon Entropy Uncertainty Quantification for False-Negative Suppression and Domain-Shift Adaptation
Computer Vision
Theory
Efficient ML
- TAM-Chain effectively suppresses false negatives in thyroid cytology classification.
- The framework utilizes a dynamic early stopping mechanism to optimize magnification levels.
- It incorporates a human-in-the-loop referral system for ambiguous cases.
- Achieved a Macro F1 score of 0.9741 with a 0.00% false-negative rate on internal validation.
Read more
TAM-Chain: Multi-Scale Thyroid Cytology Classification via Absorbing Markov Chains and Shannon Entropy Uncertainty Quantification for False-Negative Suppression and Domain-Shift Adaptation
Summary
This paper presents TAM-Chain, a novel multi-scale classification framework for thyroid cytology that addresses critical challenges in deep learning applications, particularly high false-negative rates and overconfidence in the face of domain shifts. The framework integrates Absorbing Markov Chain theory with Shannon Entropy-based Uncertainty Quantification to enhance diagnostic accuracy and safety. By employing a dynamic early stopping mechanism, TAM-Chain optimizes the magnification levels used for feature extraction, allowing for efficient processing while maintaining high diagnostic fidelity. The system also incorporates a human-in-the-loop referral mechanism to manage ambiguous cases, ensuring that high-risk specimens are evaluated by expert pathologists. Extensive evaluations on both internal and external validation sets demonstrate the framework's effectiveness, achieving a Macro F1 score of 0.9741 with a 0.00% false-negative rate on the internal test set and maintaining robust performance under domain shifts in the external validation set. The results indicate that TAM-Chain harmonizes automated diagnostic efficiency with stringent biological safety requirements, making it a promising solution for digital pathology workflows.
Methodology
The TAM-Chain framework employs a multi-scale approach using Absorbing Markov Chains to model the classification process dynamically. It integrates Shannon Entropy for uncertainty quantification, allowing the system to determine optimal stopping criteria based on uncertainty levels across different magnifications (10×, 20×, 40×). The framework also features a human-in-the-loop mechanism for cases with high uncertainty, ensuring expert evaluation when necessary.
Results
On an internal test set of 235 samples, TAM-Chain achieved a Macro F1 score of 0.9741 with an absolute false-negative rate of 0.00%. In external validation on a dataset of 1015 samples, the framework maintained a Macro F1 score of 0.7026, demonstrating its adaptability and robustness against domain shifts.
Implications
The TAM-Chain framework has significant implications for improving the safety and accuracy of automated diagnostic systems in digital pathology, particularly in thyroid cancer detection. Its ability to suppress false negatives and adapt to domain shifts makes it a valuable tool for clinical workflows, potentially enhancing patient outcomes and reducing the risk of misdiagnosis.
Physics and Data Driven Transformer-Mamba Framework for Flow Field
Multimodal
- Introduction of the TM4FF framework that integrates physics constraints into deep learning for CFD.
- Development of the RWM layer for noise decoupling and improved feature extraction.
- Utilization of a Transformer-based attention mechanism for enhanced feature fusion.
- Incorporation of a physics-informed loss function to ensure adherence to the Navier-Stokes equations.
Read more
Physics and Data Driven Transformer-Mamba Framework for Flow Field
Summary
The paper introduces the Transformer-Mamba for Flow Field (TM4FF) framework, which addresses the limitations of existing deep learning methods in computational fluid dynamics (CFD) such as Physics-Informed Neural Networks (PINNs) and Fourier Neural Operators (FNOs). These traditional methods often struggle with generalization, noise robustness, and adherence to physical laws. The TM4FF framework incorporates three key innovations: a Residual Wavelet Mamba (RWM) layer for effective feature denoising, a Transformer-based attention mechanism for improved feature fusion, and a physics-informed loss function that utilizes Fourier derivatives to enforce the Navier-Stokes equations. By treating fluid flow as a spatiotemporal image sequence and embedding operational parameters into the neural network, TM4FF enhances generalization across varying flow conditions. Experimental results on four CFD datasets demonstrate that TM4FF achieves high accuracy and robust generalization, outperforming existing methods.
Methodology
The TM4FF framework learns an operator that maps operational parameters and previous velocity fields to current flow fields. It employs a dual-branch Residual Wavelet Mamba layer for noise filtering and feature extraction, a Transformer-based attention mechanism for feature fusion, and a physics-informed loss function that incorporates Fourier derivatives to enforce physical laws.
Results
The TM4FF framework achieved state-of-the-art performance in flow field prediction, demonstrating high accuracy and robust generalization across various flow conditions in comparison to existing methods. The results indicate significant improvements in modeling complex temporal dependencies and multi-scale correlations in unsteady flow fields.
Implications
The TM4FF framework has the potential to significantly enhance the efficiency and accuracy of simulations in computational fluid dynamics, making it applicable in various fields such as engineering, meteorology, and environmental science where fluid dynamics play a critical role.
Canopy: Exploiting Piecewise Smooth Tree Priors for Multi-Fidelity Bandits
Large Language Models
Optimization
Theory
- CANOPY learns the validity of smoothness assumptions from data rather than relying on predefined schedules.
- The framework uses cheap probes to identify regions needing expensive evaluations, optimizing resource allocation.
- The method demonstrates superior performance in multiple LLM inference tasks, achieving significant improvements in recall and efficiency.
- Theoretical guarantees are provided for fixed-budget and regret performance, adapting to varying smoothness conditions.
Read more
Canopy: Exploiting Piecewise Smooth Tree Priors for Multi-Fidelity Bandits
Summary
The paper introduces CANOPY, a novel multi-fidelity tree bandit framework designed to optimize large language model (LLM) inference tasks by exploiting the hierarchical structure of candidate evaluations. Traditional hierarchical bandit methods require a predefined smoothness schedule, which can be challenging to determine accurately, especially when objectives are piecewise smooth with sharp boundaries. CANOPY addresses this limitation by learning the validity of the smoothness prior through cheap random-path probes that construct an online certificate of local aggregation bias. This allows the method to direct expensive leaf evaluations towards areas where smoothness violations are detected. The authors provide theoretical guarantees on fixed-budget and regret performance, demonstrating that CANOPY can recover optimal rates when no violations are present and adapt to structure-blind search as violations increase. The experimental results across various benchmarks show significant improvements in performance metrics, such as top-10 recall and time-to-first-token, compared to existing methods.
Methodology
CANOPY employs a multi-fidelity bandit approach that integrates a local-Lipschitz certificate of aggregation bias with a discontinuity-guided adaptive sampling technique. This combination allows the framework to efficiently allocate evaluations based on the detected smoothness of the objective function, optimizing the search process in hierarchical structures.
Results
The experimental evaluation of CANOPY across twelve benchmarks shows that it consistently outperforms existing methods. Notably, it achieves 2.9 times higher top-10 recall on a 1000-model pool and significantly reduces median time-to-first-token in prefix caching scenarios. The method also demonstrates improved performance in routing and caching tasks, with substantial gains in efficiency and effectiveness.
Implications
The findings suggest that CANOPY can be effectively applied to various LLM inference tasks, enhancing the efficiency of model routing, caching, and prompt management. This approach could lead to more responsive and resource-efficient AI systems, particularly in environments with high computational demands.
OPDiv: Optimal Selection of Top-K High-Scoring, Diverse Compounds
Optimization
- OPDiv addresses the trade-off between high-scoring and diverse compound selection in virtual screening.
- The algorithm utilizes integer optimization to find optimal subsets of compounds under diversity constraints.
- Diversity is evaluated using multiple metrics, including fingerprint distance and electrostatic diversity.
- OPDiv serves as a benchmark for assessing the performance of various virtual screening methods.
Read more
OPDiv: Optimal Selection of Top-K High-Scoring, Diverse Compounds
Summary
The paper introduces OPDiv, an algorithm designed to address the challenge of selecting a set of compounds that are both high-scoring and diverse from a large pool of candidates generated during virtual screening. The authors highlight the inherent trade-off between selecting the best-ranking compounds and maintaining diversity, which is crucial for effective drug discovery. OPDiv employs integer optimization to find an optimal subset of molecules that meets specified diversity constraints, thereby facilitating a more balanced exploration of chemical space. The methodology includes evaluating diversity through various metrics such as fingerprint distance, shape, and electrostatic diversity. The authors argue that virtual screening should not only focus on ranking but also consider the diversity of selected compounds, positioning OPDiv as a benchmark for evaluating the performance of different screening pipelines. The results demonstrate that OPDiv can efficiently identify the best compound sets while adhering to diversity thresholds, ultimately enhancing the selection process in drug discovery.
Methodology
The OPDiv algorithm employs integer optimization techniques to select a subset of compounds that maximizes the mean score while satisfying specified diversity constraints. The methodology includes the use of various diversity metrics to evaluate the selected compounds and their distribution in chemical space.
Results
The implementation of OPDiv demonstrated its capability to effectively balance the selection of high-scoring compounds with the need for diversity. The results indicated that OPDiv could achieve optimal selections that meet diversity thresholds, providing a clear advantage over traditional selection methods that may overlook this balance.
Implications
The findings suggest that OPDiv can significantly enhance the compound selection process in drug discovery, allowing researchers to explore a broader chemical space while maintaining a focus on high-quality candidates. This could lead to more effective drug development strategies and improved outcomes in medicinal chemistry.
Spatio-temporally complementary feature propagation on graphs for longitudinal AADT estimation
Graph Learning
Optimization
Time Series
- Introduces a spatio-temporally complementary feature propagation framework for AADT estimation.
- Combines loop detector data and macroscopic transportation models to enhance estimation accuracy.
- Utilizes Poisson energy minimization and flow ratio matrices for effective feature propagation.
- Achieves a normalized mean absolute error below 10% in AADT estimation.
Read more
Spatio-temporally complementary feature propagation on graphs for longitudinal AADT estimation
Summary
This paper addresses the challenge of estimating Annual Average Daily Traffic (AADT) across urban networks, which is crucial for transportation planning but often hindered by the high costs and sparse distribution of physical sensors. The authors propose a novel framework that integrates two data sources: temporally dense but spatially sparse loop detector data and a spatially complete yet temporally sparse macroscopic transportation model. The methodology employs a feature propagation algorithm on directed graphs, formulated as a Poisson energy minimization problem that considers residuals. By replacing the standard binary adjacency matrix with flow ratio matrices, the approach captures real-world vehicle turn ratios at intersections. Validated in Zurich, the algorithm demonstrates high computational efficiency, achieving convergence in minutes. The results show that the proposed framework effectively reconciles theoretical models with empirical data, achieving a normalized mean absolute error below 10%. This scalable method offers a viable solution for network-wide AADT estimation, combining limited sensor coverage with traffic modeling.
Methodology
The methodology involves a feature propagation algorithm on directed graphs, formulated as a Poisson energy minimization problem. It replaces the standard binary adjacency matrix with flow ratio matrices to accurately reflect vehicle turn ratios at intersections, allowing for effective propagation of AADT information from observable to unobservable locations.
Results
The proposed framework was validated in Zurich, achieving a normalized mean absolute error of less than 10%. The algorithm demonstrated high computational efficiency, converging within minutes, and effectively integrated theoretical models with empirical data.
Implications
This research provides a scalable solution for estimating AADT across urban networks, which can significantly aid in transportation planning, infrastructure maintenance, and intelligent transportation systems. The integration of sparse sensor data with robust traffic models can enhance decision-making processes in urban mobility management.
Learnable Time-Frequency Masks for Explaining Time-Series Classifiers
Time Series
Interpretability
- XACT generalizes learned mask paradigms to arbitrary time-frequency representations.
- The framework incorporates a time-frequency objective that enhances explanation quality.
- LRP is successfully adapted for use in wavelet domains, expanding its applicability.
- XACT outperforms existing methods in producing structured explanations while being less prone to highlighting spurious features.
Read more
Learnable Time-Frequency Masks for Explaining Time-Series Classifiers
Summary
This paper addresses the challenge of explainability in time-series classifiers, where important discriminative information is often encoded in latent frequency or time-frequency features rather than in the raw time-domain signal. The authors propose a novel framework called eXplainable Attributions from Coefficients of Transforms (XACT), which learns sparse attribution masks over coefficients from arbitrary invertible time-frequency transforms. XACT is evaluated using three different transforms: the Short-Time Discrete Fourier Transform (STDFT), the Continuous Wavelet Transform (CWT), and the Discrete Wavelet Transform (DWT). The framework extends the virtual inspection layer approach to enable Layer-wise Relevance Propagation (LRP) for generating explanations in these transformed domains. Experimental results on synthetic and real-world datasets demonstrate that XACT produces precise, sparse, and structured explanations, showing improved performance over existing methods while highlighting the importance of learning explanations directly in time-frequency representations.
Methodology
The authors developed XACT, which learns sparse and smooth masks over coefficients of any linear time-frequency operator with a bounded left inverse. The framework includes a two-dimensional stationary smoothness term and an energy-aware penalty to ensure relevance is maintained in meaningful regions of the explanation domain. Additionally, virtual inspection layers were derived for wavelet transforms to facilitate LRP.
Results
XACT produced precise explanations on a synthetic dataset and demonstrated less susceptibility to highlighting irrelevant features compared to baseline methods. On two real-world datasets, it generated sparse and structured explanations, although no single method excelled across all evaluation metrics.
Implications
The findings suggest that learning explanations in time-frequency representations can significantly enhance the interpretability of deep learning models applied to time-series data, which is crucial in domains like finance, healthcare, and climate science where understanding model predictions is essential.
PoEM: Predicting RL Outcomes from Existing Policies
Reinforcement Learning
Generative Models
Multimodal
- PoEM allows for predicting RL outcomes without additional training on new reward functions.
- The framework utilizes existing post-trained models to approximate new policies through linear combinations of log-policies.
- Regression techniques are employed to estimate composition weights from sample outputs.
- Experimental results demonstrate PoEM's effectiveness across both text and image generation tasks.
Read more
PoEM: Predicting RL Outcomes from Existing Policies
Summary
The paper introduces PoEM, a framework designed to predict the outcomes of reinforcement learning (RL) when applied to new reward functions without the need for additional RL training. The authors highlight the computational challenges and instability associated with traditional RL post-training methods, which require retraining whenever the reward model changes. PoEM leverages existing post-trained models and demonstrates that if a new reward function can be expressed as a linear combination of existing ones, the corresponding policy can also be represented as a linear combination of the log-policies of those existing models. Even when the rewards are not linearly related, the authors find that the log-policies often span a low-dimensional subspace, allowing for effective approximation of the new policy. The framework estimates the necessary weighting coefficients using regression on sample outputs, enabling the prediction of RL outcomes without further training. The authors validate PoEM across various tasks in both text and image modalities, showing that it can closely approximate the performance of directly trained RL policies on composite rewards.
Methodology
PoEM operates by taking a set of existing post-trained models and a new reward function. It predicts the new policy by forming a weighted log-mixture of the existing log-policies. The weights for this mixture are estimated using regression on outputs from the existing models, allowing for the simulation of RL outcomes without running additional RL training.
Results
The experimental validation of PoEM shows that it can recover most of the reward gains of directly trained RL policies on composite rewards, with performance closely matching that of traditional RL methods. In tests across various reward models, PoEM's outputs were found to be superior to those of leading single-expert models in most cases.
Implications
PoEM has significant implications for the efficiency of RL training processes, particularly in scenarios where rapid adaptation to new reward functions is necessary. It could streamline the development of RL systems by reducing computational costs and time associated with retraining, making it easier to integrate multiple reward signals.
ICE: Task-Aligned Clifford Latent Fields for Multimodal Graph Foundation Models
Graph Learning
Multimodal
- ICE introduces a novel multimodal graph foundation model that preserves entity semantics and enables complex relational interactions.
- The model utilizes a node-indexed Clifford latent field to explicitly manage topology and multimodal inputs.
- ICE achieves state-of-the-art performance across multiple node classification and link prediction datasets.
- The research emphasizes the significance of higher-order relational information in improving task-specific predictions.
Read more
ICE: Task-Aligned Clifford Latent Fields for Multimodal Graph Foundation Models
Summary
The paper introduces ICE (Interaction-aware Clifford Encoder), a novel multimodal graph foundation model designed to effectively learn from multimodal attributed graphs (MAGs) that integrate entities, visual content, language, and observed relations. The authors argue that traditional methods, which typically compress node information into a single Euclidean vector, fail to preserve essential entity semantics and cross-modal interactions. ICE addresses these limitations by employing a node-indexed Clifford latent field that explicitly incorporates topology, text, and images into a structured representation. The model utilizes edge-aware geometric products to transform these inputs into various relational states while maintaining semantic integrity. The authors conduct extensive experiments across multiple datasets, demonstrating that ICE outperforms existing methods in node classification and link prediction tasks. The findings highlight the importance of preserving higher-order relational information and task-specific access in multimodal graph learning, establishing ICE as a state-of-the-art solution in this domain.
Methodology
ICE employs a node-indexed Clifford latent field to encode multimodal inputs, utilizing edge-aware geometric products to construct higher-order relational states. The model maintains a Grade-1 route for semantic preservation while allowing fresh supervised and few-shot heads to query the latent field for specific tasks. The methodology includes controlled experiments to assess the impact of various components on performance.
Results
ICE ranks first in all evaluated datasets, including six node classification and three link prediction tasks, demonstrating superior performance in both supervised and few-shot settings. The results indicate that the model's design effectively captures complex interactions and preserves essential semantic information, leading to significant improvements over existing methods.
Implications
The findings suggest that ICE can be applied in various domains requiring multimodal graph learning, such as recommendation systems, social network analysis, and knowledge graph construction. The model's ability to preserve relational information could enhance the performance of applications that rely on understanding complex interactions among diverse data types.
LabFactory: Building and Evaluating Executable AI Labs
Theory
Efficient ML
- LABFACTORY framework enables the transformation of scientific briefs into executable AI labs.
- The evaluation focuses on the delivered artifact rather than the builder's progress.
- 28 AI labs were constructed across various scientific tasks, all exceeding performance benchmarks.
- The framework supports diverse task types, including predictive models and analytical tools.
Read more
LabFactory: Building and Evaluating Executable AI Labs
Summary
The paper introduces LABFACTORY, a framework designed to convert scientific briefs into executable AI labs, which are task-specific solvers integrating models, knowledge resources, tools, and a controller behind a fixed interface. The framework allows an AI builder to develop and package a lab in a controlled workspace, which is then executed by a separate host on held-out inputs for evaluation. This approach emphasizes the delivered system as the evaluation object, rather than the builder's progress. The authors document 28 constructions across seven scientific task categories, demonstrating that the delivered labs surpassed their configured reference values in all 33 subtests during host-side execution. The paper outlines a construction and delivery protocol, provides detailed descriptions of the constructed AI labs, and presents execution-grounded case records, showcasing the capabilities of AI agents in building functional scientific tools that remain usable post-construction.
Methodology
The methodology involves a task-conditioned construction process where a builder receives a scientific brief and a workspace to develop an artifact. The builder integrates various resources and tools, validates them, and submits the solver for independent evaluation. The framework records actions and resource usage throughout the construction lifecycle, ensuring a clear separation between the builder and the evaluation process.
Results
The results indicate that all 28 constructed labs exceeded their configured reference values across 33 subtests during host-side execution. The paper provides detailed accounts of the construction processes, including component-level descriptions and execution-grounded case records, demonstrating the effectiveness of the LABFACTORY framework.
Implications
The implications of this work suggest that LABFACTORY can facilitate the development of robust, reusable AI systems for scientific tasks, enhancing the efficiency of scientific workflows and enabling more effective evaluations of AI capabilities in real-world applications.
Growth-Inspired Graph Generation and Inverse Design of Mechanical Lattices via Dot Matrices Database Augmentation and GCNN
Graph Learning
Generative Models
Optimization
- Introduces a growth-inspired framework for generating mechanical lattices based on biological development processes.
- Utilizes a graph convolutional neural network (GCNN) to predict effective stiffness from lattice topology.
- Enables inverse design capabilities for mechanical lattices, achieving precise stiffness targets.
- Extends the framework to include nonlinear materials and parameterized shapes for advanced design applications.
Read more
Growth-Inspired Graph Generation and Inverse Design of Mechanical Lattices via Dot Matrices Database Augmentation and GCNN
Summary
This paper presents a novel morphogenetic graph-generation framework for mechanical lattices, inspired by natural growth processes. The authors propose a method where a discrete dot matrix serves as potential nodes, and the final lattice architecture is developed through sequential cross-layer and intra-layer growth. This approach mimics biological systems, such as leaf venation, where structures emerge over time rather than being designed in a single step. The framework is implemented in both two and three dimensions, utilizing a 3 × 3 × 3 nodal matrix with 27 candidate nodes. The generated lattices are evaluated using beam-based finite element analysis (FEA) and represented as graphs. A graph convolutional neural network (GCNN) is employed to learn the relationship between lattice topology and effective compressive stiffness, enabling rapid predictions and inverse design capabilities. For a target stiffness of 1000 MPa, the predicted design achieved 1042.43 MPa, validated by FEA at 1027.49 MPa. Additionally, the framework extends to parameterized horseshoe-shaped beams made from nonlinear materials, facilitating topology-geometry design for specific deformation shapes. This work bridges biological morphogenesis, graph learning, and nonlinear shape programming, offering a comprehensive generative design framework for architected materials.
Methodology
The authors developed a morphogenetic representation using a discrete dot matrix, where growth occurs through cross-layer and intra-layer connections. The generated lattices are analyzed using finite element analysis, and a GCNN is trained to learn the topology-property mapping, allowing for rapid predictions and inverse design.
Results
The framework successfully predicted a lattice design with an effective stiffness of 1042.43 MPa for a target of 1000 MPa, validated by finite element analysis at 1027.49 MPa. The approach also demonstrated the capability to design complex shapes using nonlinear materials.
Implications
This research has significant implications for the design of mechanical metamaterials, enabling more efficient and innovative approaches to creating structures with tailored mechanical properties. The integration of biological principles into machine learning frameworks could lead to advancements in material science and engineering applications.
A Concentration Bound for Two-Timescale Actor-Critic Algorithm
Reinforcement Learning
Theory
Optimization
- Derives a uniform all-time concentration bound for the two-timescale actor-critic algorithm.
- Establishes that the actor parameter enters a safe region after a finite time with high probability.
- Demonstrates that actor error decreases with the number of updates, enhancing algorithm stability.
- Contributes to the finite-time convergence analysis in reinforcement learning.
Read more
A Concentration Bound for Two-Timescale Actor-Critic Algorithm
Summary
This paper presents a uniform all-time concentration bound for the two-timescale actor-critic algorithm, which is a reinforcement learning method where the actor and critic operate on different learning timescales. The authors derive a high-probability convergence bound for the actor parameter in the long-run average-reward setting, demonstrating that after a certain time, the actor parameter will enter a 'safe region' and remain there with high probability. The derived bound characterizes the actor error, showing it diminishes with the number of updates. This work builds on previous research in actor-critic methods, addressing both asymptotic and non-asymptotic convergence guarantees, and contributes to the understanding of the stability and performance of these algorithms in practical applications.
Methodology
The authors utilize a theoretical framework to derive a concentration bound for the actor parameter in a two-timescale actor-critic algorithm. They analyze the algorithm's behavior using linear function approximation in the context of Markov decision processes, focusing on the long-run average-reward setting. The analysis involves establishing a lower bound on the probability that the actor parameter remains within a specified safe region after a certain time.
Results
The main result is a finite-time high-probability convergence bound for the actor parameter, indicating that it will remain in a safe region after a specified time with high probability. The actor error is characterized as decreasing with the number of updates, providing a quantitative measure of the algorithm's convergence behavior.
Implications
The findings have significant implications for the design and analysis of reinforcement learning algorithms, particularly in ensuring stability and convergence in practical applications. The concentration bounds can be used to inform the development of more robust actor-critic methods and enhance their performance in various reinforcement learning tasks.
Generalized Graph Variational Autoencoders: Bounded Divergences Control Posterior Collapse
Generative Models
Graph Learning
Theory
- Introduces the Generalized Graph Variational Autoencoder (GGVA) that utilizes bounded divergences.
- Demonstrates that the choice of divergence affects the retention of posterior information.
- Shows that the GGVA can retain up to 49 times more posterior information compared to the VGAE.
- Finds that while GGVA improves information retention, it does not enhance link-prediction accuracy.
Read more
Generalized Graph Variational Autoencoders: Bounded Divergences Control Posterior Collapse
Summary
This paper introduces the Generalized Graph Variational Autoencoder (GGVA), which modifies the variational graph autoencoder (VGAE) by replacing the Kullback-Leibler (KL) divergence with a member of the Rényi-Tsallis family of divergences. The authors argue that the KL divergence, while convenient, is unbounded and can lead to posterior collapse, where the model fails to learn useful representations. By using bounded divergences, specifically the Tsallis divergence for orders q < 1, the GGVA maintains a controlled penalty that delays posterior collapse. The paper presents a thorough analysis of the GGVA, demonstrating that it retains significantly more posterior information compared to the VGAE across various graph datasets. The authors also highlight that while the GGVA improves information retention, it does not achieve better link-prediction accuracy on larger real graphs, indicating that boundedness alone does not prevent posterior collapse. This work contributes to the understanding of divergence choices in variational inference for graph-based models and provides a framework for future research in this area.
Methodology
The GGVA replaces the KL divergence in the VGAE with divergences from the Rényi-Tsallis family, allowing for a controlled penalty on the latent space. The model retains the same architecture and training protocol as the VGAE, focusing solely on the divergence used for the nodewise posterior penalty. The authors conduct experiments on various graph datasets to evaluate the performance of the GGVA against the VGAE.
Results
The GGVA significantly improves the retention of posterior information, achieving up to 49 times more information compared to the VGAE on synthetic and real graphs. However, it does not improve link-prediction accuracy on larger real graphs, indicating that while it delays posterior collapse, it does not prevent it.
Implications
The findings suggest that the choice of divergence in variational autoencoders can greatly influence the model's ability to learn meaningful representations, particularly in graph-based settings. This work opens avenues for further exploration of alternative divergences and their effects on model performance and stability.
Language Specificity vs. Domain Diversity: Benchmarking Transformers for Bangla Medical NER
NLP
- Fine-tuned XLM-RoBERTa achieves a new state-of-the-art F1-score of 0.5959 for Bangla medical NER.
- BanglaBERT underperforms compared to multilingual models, highlighting the importance of domain diversity in pretraining.
- A comprehensive evaluation across 3,179 samples provides statistically robust baselines for future research.
- Medicine and Specialist categories are recognized with high reliability, while the Symptom category remains the most challenging.
Read more
Language Specificity vs. Domain Diversity: Benchmarking Transformers for Bangla Medical NER
Summary
This paper addresses the challenges of Medical Named Entity Recognition (NER) in low-resource languages, specifically Bangla, which suffers from high linguistic variability and a lack of annotated corpora. The authors conduct a comprehensive benchmark of three fine-tuned transformer models—BanglaBERT, multilingual BERT (mBERT), and XLM-RoBERTa—against GPT-4o mini under zero-shot and few-shot prompting configurations. Unlike previous studies that evaluated models on a limited sample size of 50, this work utilizes a full test set of 3,179 samples, providing robust statistical baselines. The results indicate that the fine-tuned XLM-RoBERTa model achieves a new state-of-the-art F1-score of 0.5959, surpassing the previous best of 0.5848. The study reveals that BanglaBERT consistently underperforms compared to multilingual models, suggesting that domain diversity in pretraining is more beneficial than language specificity in clinical contexts. A detailed analysis of entity types shows that while Medicine and Specialist categories are recognized reliably, the Symptom category remains challenging. Additionally, the findings indicate that transformer models significantly outperform prompt-only configurations, emphasizing the inadequacy of such methods for structured clinical entity extraction in low-resource settings.
Methodology
The authors conducted a large-scale empirical benchmark comparing three transformer models (BanglaBERT, mBERT, XLM-RoBERTa) and GPT-4o mini under zero-shot and few-shot prompting. They evaluated the models on a complete test set of 3,179 samples and performed a detailed per-entity-type analysis to assess performance across different clinical categories.
Results
The fine-tuned XLM-RoBERTa model achieved an F1-score of 0.5959, surpassing the previous best score of 0.5848. BanglaBERT achieved an F1-score of 0.4937, indicating its inferiority in this context. The analysis revealed high reliability for Medicine and Specialist categories, with F1-scores above 0.83, while the Symptom category had the lowest F1-score of 0.4367.
Implications
The findings suggest that for low-resource languages like Bangla, leveraging multilingual models with diverse domain pretraining can significantly enhance performance in medical NER tasks. This has implications for improving clinical NLP applications, such as electronic health record analytics and clinical decision support systems, in low-resource settings.
Downside-Controlled Online Forecast Combination under Delayed and Revised Outcomes
Time Series
- Introduces a downside-controlled approach to forecast combination for frozen models.
- Combines static and online correctors to improve forecast accuracy without retraining.
- Achieves minimal worst-case deterioration (0.15%) and significant gains (up to 11.5%) across multiple benchmarks.
- Demonstrates effectiveness in real-world applications, particularly in electricity load forecasting.
Read more
Downside-Controlled Online Forecast Combination under Delayed and Revised Outcomes
Summary
This paper addresses the challenge of improving forecasts from frozen models, which cannot be retrained, by proposing a novel approach to forecast combination that emphasizes downside control. The authors introduce a method that combines a static corrector and an online corrector with a frozen forecaster, focusing on minimizing the worst-case deterioration of forecasts while allowing for gains. The study evaluates the proposed method across seven benchmarks and four base models, including foundation models, demonstrating that the worst deterioration is only 0.15% while achieving gains of up to 11.5%. The method is particularly effective in day-ahead load forecasting for European bidding zones, where it consistently lowers mean MSE compared to single correctors. The paper also discusses the implications of learning from provisional outcomes and the importance of adapting to changing error patterns in forecasts.
Methodology
The authors employ a combination of a frozen forecaster, a static corrector, and an online corrector, utilizing a gating mechanism that adjusts weights based on matured losses. The method is evaluated on various datasets, focusing on downside control rather than peak accuracy, and incorporates learning from provisional outcomes.
Results
The proposed method shows a worst-case deterioration of only 0.15% and achieves gains of up to 11.5% across 28 dataset-model pairs. In the context of day-ahead load forecasting for seven European bidding zones, the method consistently lowers mean MSE, outperforming single correctors that can increase MSE by up to 102%.
Implications
The findings suggest that the proposed approach can be effectively applied in scenarios where retraining is not feasible, such as in regulated industries or safety-critical applications. It opens avenues for improving forecast accuracy in real-time data environments, particularly in energy management and economic forecasting.
When Explanations Cannot Be Read: Measuring and Correcting SHAP and LIME Rendering for Right-to-Left Languages
Interpretability
NLP
- Characterizes rendering failures in SHAP and LIME visualizations for RTL languages.
- Introduces a quantitative measure of rendering correctness using OCR.
- Demonstrates that common workarounds for rendering issues are ineffective for Urdu.
- Presents SHAP-RTL, a rendering layer that corrects visualization issues while preserving attribution values.
Read more
When Explanations Cannot Be Read: Measuring and Correcting SHAP and LIME Rendering for Right-to-Left Languages
Summary
This paper addresses the challenges faced by post hoc explanation methods, specifically SHAP and LIME, when applied to right-to-left (RTL) languages such as Urdu, Arabic, Persian, and Hebrew. While the mathematical validity of attribution values remains intact, the visual representation of these values fails due to incorrect token sequencing, broken cursive letterforms, and improper plot layouts. The authors propose SHAP-RTL, a rendering layer that corrects these visualization issues by ensuring proper reading direction and script shaping, while allowing for per-language font selection. The framework is evaluated on hate and offensive language datasets across the four RTL languages using TF-IDF and logistic regression classifiers. The study introduces a novel quantitative measure of rendering correctness assessed through an OCR round trip over 200 feature words per language. The results reveal that common workarounds for rendering issues are ineffective, particularly for Urdu, and highlight the importance of language-aware visualization in enhancing the accessibility of post hoc explainability methods across diverse writing systems.
Methodology
The authors developed SHAP-RTL, a rendering layer that applies complex text layout and per-language font selection to SHAP and LIME visualizations. They quantitatively evaluated rendering correctness using an OCR round trip method on datasets for Urdu, Arabic, Hebrew, and Persian, measuring character error rates to assess the effectiveness of the rendering corrections.
Results
The evaluation showed that default rendering methods resulted in character error rates ranging from 0.820 to 0.979, with the common reshape-and-reorder workaround failing for Urdu at a rate of 0.998. In contrast, SHAP-RTL maintained rendering correctness across different library versions, preserving attribution values and feature ordering.
Implications
The findings underscore the necessity for language-specific adaptations in explainable AI tools, particularly for RTL languages. The SHAP-RTL framework can enhance the interpretability of machine learning models in multilingual contexts, making them more accessible to users who read RTL scripts.
A Particle-Swarm-Assisted Gradient Meta-Learning Algorithm for Joint Transmit Precoding and STAR-RIS Coefficient Optimization
Optimization
- Introduces PSA-GML for optimizing STAR-RIS in multi-user downlink systems.
- Transforms the original non-convex problem into a tractable form to simplify optimization.
- Utilizes PSO for robust initialization and a meta-optimizer for refined updates.
- Achieves significant performance improvements over traditional optimization methods.
Read more
A Particle-Swarm-Assisted Gradient Meta-Learning Algorithm for Joint Transmit Precoding and STAR-RIS Coefficient Optimization
Summary
This paper presents a novel approach for optimizing the transmit precoder and the transmission/reflection coefficients of a simultaneously transmitting and reflecting reconfigurable intelligent surface (STAR-RIS) in a multi-user downlink system, aiming to maximize the weighted sum rate (WSR). The proposed Particle-Swarm-Assisted Gradient Meta-Learning (PSA-GML) algorithm addresses the non-convex nature of the optimization problem by first transforming it into a more tractable form using amplitude-split parameterization and collapsed precoder representation, which inherently satisfies energy conservation constraints and reduces the search space. The algorithm employs Particle Swarm Optimization (PSO) to conduct a global search for STAR-RIS coefficients, providing a robust warm start for the optimization process. This is followed by a coordinate-wise long short-term memory (LSTM) meta-optimizer trained through first-order gradient meta-learning, which refines both the STAR-RIS coefficients and the transmit precoder. The meta-optimizer is trained offline across various channel realizations and can be applied to unseen channels without further adaptation. The results demonstrate that PSA-GML significantly outperforms conventional methods, achieving an 11.06 bits/s/Hz WSR at a transmit SNR of 10 dB, surpassing alternating optimization (AO) by 13.1% and a random-phase scheme by 35.1%. Additionally, the learned optimizer shows robustness against initialization and performs well in interference-limited scenarios, achieving 83.9% of the performance of a hand-designed Adam refinement without manual hyper-parameter tuning.
Methodology
The methodology involves transforming the optimization problem using amplitude-split parameterization and collapsed precoder representation, followed by a global search using PSO for initial coefficient estimation. A coordinate-wise LSTM meta-optimizer is then trained through gradient meta-learning to refine the coefficients and precoder jointly.
Results
The PSA-GML algorithm achieves an 11.06 bits/s/Hz WSR at a transmit SNR of 10 dB with N = 32 elements and K = 4 users, outperforming conventional AO methods by 13.1% and random-phase schemes by 35.1%. In interference-limited conditions, it reaches 83.9% of the performance of a hand-designed Adam optimizer without manual tuning.
Implications
The findings suggest that the PSA-GML algorithm can effectively enhance the performance of wireless networks utilizing STAR-RIS technology, particularly in scenarios requiring high spectral efficiency and adaptability to varying channel conditions. This could lead to more efficient designs in future wireless communication systems, especially as networks transition to 6G.
SwitchPFN: Shared Switching Dynamics for Frozen In-Context Time Series Classification
Time Series
- SwitchPFN addresses the challenges of temporal order preservation and feature consistency in time series classification.
- The method employs a shared projection and regime codebook to enhance comparability of features across sequences.
- SwitchPFN achieves state-of-the-art performance on multiple benchmark datasets, demonstrating its effectiveness.
- Ablation studies and sensitivity analyses provide insights into the model's design choices and data efficiency.
Read more
SwitchPFN: Shared Switching Dynamics for Frozen In-Context Time Series Classification
Summary
The paper introduces SwitchPFN, a novel approach for time series classification using tabular foundation models (TFMs). It addresses the limitations of existing representations that either lose the temporal order of data or create inconsistencies in feature meanings across sequences. SwitchPFN learns a shared projection and regime codebook from training sequences, ensuring that local dynamic operators and transition features are directly comparable across samples. The authors argue that effective representation design for TFMs should capture local temporal transitions while maintaining a consistent feature definition. The proposed method combines switching dynamics, class residuals, ordered paths, and signal summaries into a fixed-width table suitable for TabPFN. The evaluation on eight UEA datasets demonstrates that SwitchPFN achieves a mean test accuracy of 82.63%, outperforming the next best model by 3.54 percentage points. The paper also includes ablation studies and sensitivity analyses to assess the impact of various design choices and the model's efficiency with limited labeled data.
Methodology
SwitchPFN maps training sequences and queries to a common feature space using a shared coordinate system. It integrates local dynamics, class residuals, and ordered paths into a fixed-budget table for TabPFN, ensuring that each feature column retains consistent meanings across different sequences.
Results
SwitchPFN achieved a mean test accuracy of 82.63% on eight UEA datasets, surpassing the second-best model by 3.54 percentage points. The results were supported by comprehensive ablation studies and parameter sensitivity analyses.
Implications
The findings suggest that SwitchPFN can significantly improve time series classification tasks in various fields, including healthcare (e.g., ECG analysis) and environmental monitoring, by providing a more robust representation of temporal data.
Task-Aware Spectral Pruning: A Mixture-of-Masks Framework for Efficient LLM Inference
Large Language Models
Efficient ML
NLP
- Introduces Task-Aware Spectral Pruning (TASP) for efficient LLM inference.
- Utilizes task-specific spectral descriptors to create tailored sparse masks.
- Achieves a 43% reduction in active FLOPs while retaining high model performance.
- Demonstrates significant reduction in decoding latency, improving inference speed.
Read more
Task-Aware Spectral Pruning: A Mixture-of-Masks Framework for Efficient LLM Inference
Summary
The paper presents Task-Aware Spectral Pruning (TASP), a novel post-training framework designed to enhance the efficiency of large language model (LLM) inference by utilizing task-specific spectral descriptors. Unlike traditional static pruning methods that apply a single sparse structure across all prompts, TASP calibrates module-level spectral descriptors based on task-specific ablation effects, allowing for the construction of tailored sparse masks. The framework includes a pilot phase to evaluate the spectral signal's informativeness before full calibration, ensuring that only suitable models are processed. TASP was tested on Llama-3-8B and Llama-3-70B checkpoints, achieving a 43% reduction in active FLOPs while retaining over 97% of the original model's performance. The method also demonstrated significant improvements in decoding latency, reducing it from 45.2 ms/token to 31.3 ms/token, resulting in a 1.44× speedup. The findings suggest that TASP can effectively create a small family of executable subnetworks tailored to specific tasks, thereby addressing the versatility tax associated with traditional pruning methods.
Methodology
TASP employs a multi-stage approach that includes a pilot phase to assess the informativeness of spectral predictors, followed by task-specific calibration of these predictors to construct dependency-closed sparse masks. The framework routes user queries to precompiled masks, maintaining a fixed execution plan throughout the inference process. The methodology emphasizes the separation of scoring, mask multiplicity, and routing to empirically evaluate the contributions of each component.
Results
The application of TASP on Llama-3-70B resulted in a 43% reduction in active FLOPs while achieving a retention of 97.7% of the dense BF16 score. In terms of runtime performance, the method reduced decoding latency from 45.2 ms/token to 31.3 ms/token, achieving a speedup factor of 1.44. The pilot phase effectively filtered out unsuitable models, demonstrating the importance of model-dependent applicability.
Implications
TASP has the potential to significantly enhance the efficiency of LLMs in resource-constrained environments, making it suitable for latency-sensitive applications such as real-time translation, interactive AI systems, and other scenarios where rapid inference is critical. The framework's ability to create task-specific subnetworks could lead to more adaptable and efficient deployment of language models across various tasks.
Decoupled Learning and Selection in Slate Recommendation for Privacy and Stability Under Noisy Scores
Theory
- Introduces a two-step model for slate recommendation that separates score learning from slate selection.
- Establishes differential privacy guarantees that hold under specific conditions for selector inputs.
- Derives a logged margin certificate that certifies stability in the ordered slate against score perturbations.
- Demonstrates empirical results showing reduced ranking churn with increased anchor weight.
Read more
Decoupled Learning and Selection in Slate Recommendation for Privacy and Stability Under Noisy Scores
Summary
This paper formalizes the process of slate recommendation as a two-step operation involving a randomized score learner followed by a deterministic selection mechanism. The authors introduce a framework that allows for differential privacy guarantees to be maintained through the selection process, emphasizing the importance of separating the learning and selection components. They derive a logged margin certificate that ensures stability in the ordered slate, meaning that small perturbations in scores do not lead to changes in the final slate. The paper presents controlled experiments demonstrating the effectiveness of their approach, showing that greater anchor weight can reduce ranking instability caused by score noise. The authors validate their findings through various datasets, including OULAD, MovieLens-25M, and Amazon Musical Instruments, and provide a public repository for reproducibility.
Methodology
The authors develop a model class that consists of a learning operator that generates item scores from data and a deterministic selection operator that maps these scores to a final slate. They utilize differential privacy techniques to ensure privacy during the learning phase and derive a margin-based stability certificate to assess the robustness of the slate against score changes. Controlled experiments and closed-loop simulations are conducted across multiple datasets to validate their theoretical claims.
Results
The experiments reveal that the proposed method exhibits near-linear exponent scaling in stability, with an empirical slope of approximately -0.220. The results indicate that increasing the weight of anchors in the selection process significantly reduces the impact of score noise on the final ranking, leading to improved stability in the slate recommendations. The logged margin certificate is validated through tests on OULAD and EdNet datasets, confirming the theoretical guarantees provided.
Implications
This work has significant implications for the design of recommendation systems, particularly in contexts where privacy and stability are critical. The separation of learning and selection allows for more nuanced privacy guarantees and stability assessments, which can enhance user trust and system reliability in real-world applications.
Policy Complexity, Reaction Time, and Bounded Rationality in Reinforcement Learning
Reinforcement Learning
Theory
Efficient ML
- Introduces MI-SARSA, an RL algorithm that incorporates mutual-information regularization.
- Links policy complexity to reaction time, providing a unified framework for understanding decision-making under cognitive constraints.
- Demonstrates a reward-complexity tradeoff, where simpler policies yield faster reaction times.
- Reveals a robustness-capacity tradeoff in performance under environmental shifts.
Read more
Policy Complexity, Reaction Time, and Bounded Rationality in Reinforcement Learning
Summary
This paper addresses the limitations of standard reinforcement learning (RL) models in capturing the cognitive constraints faced by biological agents during learning and decision-making. The authors introduce MI-SARSA, an on-policy temporal-difference algorithm that incorporates mutual-information regularization to account for the internal costs associated with policy complexity. By penalizing deviations from a learned marginal action prior, MI-SARSA encourages selective use of state information based on its expected return benefit. This approach not only optimizes reward but also predicts reaction times, linking cognitive constraints with behavioral latency. The empirical results demonstrate a reward-complexity tradeoff, where stronger information penalties lead to simpler policies with lower control costs and faster reaction times. Additionally, the model reveals a robustness-capacity tradeoff under environmental shifts, highlighting the balance between performance degradation and asymptotic return. Overall, MI-SARSA serves as a model for bounded sequential learning, providing insights into how cognitive constraints shape biological learning processes.
Methodology
The authors developed MI-SARSA by embedding a mutual-information penalty into a temporal-difference learning framework. This involves maintaining a state-specific policy and an adaptive action prior, penalizing deviations through Kullback-Leibler divergence to minimize mutual information between states and actions.
Results
MI-SARSA showed that stronger information penalties resulted in simpler policies with lower control costs and faster reaction times. The model also indicated that increasing regularization mitigated performance degradation after environmental shifts but reduced asymptotic return, illustrating a tradeoff between robustness and capacity.
Implications
The findings suggest that incorporating cognitive constraints into RL models can enhance our understanding of biological learning and decision-making. This approach may inform the design of more efficient RL algorithms that better mimic human cognitive processes and improve performance in complex environments.
Monitoring Urban Traffic Dynamics at Fine Spatiotemporal Resolution Using Distributed Acoustic Sensing and Deep Learning
Time Series
Audio & Speech
- DAS provides a scalable and privacy-preserving method for urban traffic monitoring.
- Integration of deep learning enhances the processing of DAS data for vehicle detection and traffic state inference.
- Distinct traffic patterns are observed during different urban events, showcasing the system's sensitivity to dynamic conditions.
- The methodology offers a continuous and efficient framework for real-time urban traffic observation.
Read more
Monitoring Urban Traffic Dynamics at Fine Spatiotemporal Resolution Using Distributed Acoustic Sensing and Deep Learning
Summary
This study explores the integration of Distributed Acoustic Sensing (DAS) and deep learning to monitor urban traffic dynamics at high spatiotemporal resolution. DAS utilizes existing underground fiber-optic cables as continuous sensor arrays to capture ground vibrations caused by traffic, enabling real-time monitoring of traffic volume and congestion. The authors developed a deep learning framework that processes raw DAS data to detect vehicle trajectories and infer traffic states. A hybrid training strategy combining synthetic and annotated DAS images was employed to enhance vehicle detection in noisy conditions. The findings reveal that urban events, such as home football games and holidays, produce distinct traffic-state signatures, highlighting the effectiveness of DAS in providing accurate, privacy-preserving traffic monitoring. This approach addresses the limitations of traditional traffic sensing methods, particularly during disruptive events when conventional data sources may be unavailable.
Methodology
The authors deployed DAS along a roadway network and developed a deep learning analytical framework that converts raw ground vibration waveforms into spatiotemporal representations. A hybrid training strategy was used, combining synthetic and manually annotated DAS images to improve vehicle detection under various conditions.
Results
The study found that different urban events generate unique traffic-state signatures. For instance, home football game days resulted in localized congestion with multi-peaked surges, while the Thanksgiving holiday led to sustained traffic suppression and altered diurnal traffic rhythms. The integration of DAS and deep learning proved effective for high-resolution traffic monitoring.
Implications
The findings suggest that DAS combined with deep learning can significantly enhance urban traffic monitoring capabilities, particularly in scenarios where traditional data sources are limited. This approach could inform smart city initiatives and improve urban resilience by providing timely traffic insights during disruptive events.
On the SoS Certifiability of Log-Concave Distributions
Theory
Efficient ML
Optimization
- Establishes SoS certifiability for isotropic log-concave distributions, removing dependence on the Poincaré constant.
- Proves that the polynomial related to moment tensors is a sum of squares for all even m ≥ 2.
- Introduces efficient algorithms with dimension-free error guarantees for high-dimensional statistical estimation.
- Utilizes stochastic localization to derive moment bounds and certificates.
Read more
On the SoS Certifiability of Log-Concave Distributions
Summary
This paper investigates the certifiability of isotropic log-concave distributions through the lens of sum-of-squares (SoS) proofs. The author establishes that for any isotropic log-concave distribution P on R^d, the polynomial (Cm)m∥v∥m^2 - E[X∼P ⟨X, v⟩^m is a sum of squares for every even m ≥ 2, where C is a universal constant. This result eliminates the dependence on the Poincaré constant, thus recovering optimal moment bounds for log-concave distributions. The proof employs stochastic localization to express P as an average of random strongly log-concave measures, allowing for the derivation of subgaussian certificates for centered moments. The findings lead to the development of computationally efficient algorithms that provide dimension-free error guarantees for various high-dimensional statistical estimation tasks. The paper highlights the significance of SoS certificates in enhancing statistical identifiability and algorithmic performance, particularly in robust mean estimation scenarios where the moment bounds play a crucial role in ensuring accuracy in polynomial time.
Methodology
The methodology involves stochastic localization to decompose isotropic log-concave distributions into averages of random strongly log-concave measures. This approach allows the author to derive subgaussian certificates for centered moments and establish the existence of SoS representations for moment bounds. The results are framed within the context of semidefinite programming to certify upper bounds on injective norms.
Results
The main result confirms that every isotropic log-concave distribution is (Cm, m)-certifiably bounded for all even m ≥ 2, with C being a universal constant. This result leads to the conclusion that optimal moment bounds can be achieved with SoS proofs that do not depend on the dimension of the distribution.
Implications
The findings have significant implications for statistical estimation in high-dimensional settings, particularly in robust mean estimation where the moment bounds ensure accurate estimations despite adversarial corruption of samples. The results also enhance the understanding of log-concave distributions in theoretical and practical applications, potentially improving algorithmic performance across various statistical tasks.
CataOPD: Catalytic On-Policy Distillation for Large Language Model Reasoning
NLP
Large Language Models
Reinforcement Learning
- CataOPD acts as a catalyst rather than a target, enhancing reasoning capabilities in LLMs.
- The method addresses challenges in RL and OPD, such as correctness-signal collapse and off-support distillation targets.
- Experimental results show significant performance improvements on challenging reasoning tasks.
- CataOPD enables models to generalize better to out-of-distribution problems.
Read more
CataOPD: Catalytic On-Policy Distillation for Large Language Model Reasoning
Summary
The paper introduces CataOPD, a novel approach to enhance reasoning capabilities in large language models (LLMs) by addressing limitations in existing reinforcement learning (RL) and on-policy distillation (OPD) methods. Traditional RL struggles when no correct trajectories are sampled, while OPD is limited by the reasoning trajectories available under the student's on-policy distribution. CataOPD proposes a framework where the teacher model serves as a catalyst, expanding the reachability of correct trajectories and internalizing verified student-produced trajectories into a catalyst-free policy. The methodology includes Self-Rescue Routing, which utilizes failed groups as routing signals to recover correct trajectories through on-policy self-sampling, and Catalytic-Guided Self-Resolution, which employs catalytic guidance to elicit verified trajectories. Additionally, Barrier-Weighted Internalization focuses updates on critical tokens that are challenging to reason about without guidance. Experimental results demonstrate that CataOPD outperforms existing baselines across multiple reasoning benchmarks, particularly on difficult problems, and enhances out-of-distribution generalization without reliance on a teacher during inference.
Methodology
CataOPD employs three main strategies: Self-Rescue Routing to recover correct trajectories from failed groups, Catalytic-Guided Self-Resolution to elicit verified trajectories using catalytic guidance, and Barrier-Weighted Internalization to focus updates on critical tokens based on guided-to-unaided log-probability gaps.
Results
CataOPD outperformed existing methods across twelve reasoning benchmarks, particularly excelling in solving the hardest problems and demonstrating improved generalization to out-of-distribution tasks. The method maintained the problem-solving capabilities of the base models while enhancing performance on challenging tasks.
Implications
The findings suggest that CataOPD can significantly improve the reasoning abilities of LLMs, making them more capable of tackling previously unsolvable problems and enhancing their applicability in real-world scenarios requiring robust reasoning.
Three Ways Classical Test Theory Misleads for LLM Judges
NLP
Large Language Models
Theory
- Classical Test Theory metrics can mislead when applied to judge evaluations in LLM contexts.
- Internal-consistency coefficients do not capture scorer facets, leading to ambiguous reliability interpretations.
- Dependability indices and accuracy measures can misrepresent judge performance due to conflated variables.
- The study highlights a lack of awareness in existing literature regarding these measurement issues.
Read more
Three Ways Classical Test Theory Misleads for LLM Judges
Summary
This paper critiques the application of Classical Test Theory (CTT) in evaluating the reliability of judges scoring responses generated by Large Language Models (LLMs). The author identifies three significant misinterpretations that arise when borrowing reliability statistics from CTT without considering the unique measurement design of judge evaluations. First, the internal-consistency coefficient, such as KR-20, fails to account for the scorer facet, leading to misleading interpretations of judge reliability. Second, the dependability index Φ(λ) misrepresents classification probabilities, and third, the Livingston–Lewis accuracy conflates judge unreliability with criterion invalidity. The paper emphasizes that these misinterpretations can lead to erroneous deployment decisions and calls for clearer reporting practices to ensure that reliability metrics accurately reflect judge performance. The findings highlight the need for a more nuanced understanding of measurement in the context of LLM evaluations.
Methodology
The author constructed a dataset of 210 short-answer items scored against a rubric with 10 checklist elements. The evaluation involved using an LLM (Claude Haiku 4.5) to score responses, and various reliability statistics were computed to analyze the performance of the judge against the rubric and gold standards.
Results
The analysis revealed that the internal-consistency coefficient (KR-20) varied significantly based on item bank design and judge error, with values ranging from 0.01 to 0.68. The dependability index showed discrepancies in classification probabilities, and the Livingston–Lewis accuracy was found to conflate judge unreliability with criterion validity. The study concluded that existing metrics do not adequately reflect judge-specific reliability.
Implications
The findings suggest that practitioners should exercise caution when interpreting reliability metrics derived from Classical Test Theory in the context of LLM evaluations. Improved reporting practices are necessary to ensure that reliability assessments accurately reflect the performance of judges, which is crucial for making informed deployment decisions.
SpaFactor: Lightweight Spatial Context-Aware Gene Program Modeling for Histology-to-Transcriptomics Inference
Computer Vision
Efficient ML
Multimodal
- Introduces SpaFactor, a lightweight framework for predicting spatial gene expression from H&E images.
- Utilizes a low-rank factorization approach to model the relationship between tissue morphology and gene programs.
- Achieves superior performance in predicting spatially variable genes compared to existing methods.
- Demonstrates the ability to recover biologically organized spatial patterns effectively.
Read more
SpaFactor: Lightweight Spatial Context-Aware Gene Program Modeling for Histology-to-Transcriptomics Inference
Summary
The paper introduces SpaFactor, a novel framework designed to predict spatial gene expression from hematoxylin and eosin (H&E) stained images, addressing the limitations of existing methods that often overlook biological coordination among genes. SpaFactor employs a lightweight low-rank factorization approach that integrates visual representations of tissue morphology with local and regional context. The framework utilizes a residual multi-layer perceptron (MLP) to learn a nonlinear mapping from tissue microenvironments to latent gene programs, which are then decoded into coordinated multi-gene expression predictions. The authors demonstrate that SpaFactor outperforms existing methods across five public cohorts, particularly in predicting spatially variable genes and recovering biologically relevant spatial patterns. This work highlights the potential of lightweight modeling techniques in enhancing predictive accuracy and biological fidelity in spatial transcriptomics.
Methodology
SpaFactor employs a four-block residual MLP that combines a frozen GigaPath embedding of the central spot with parameter-free local and regional summaries of tissue context. This architecture allows for the inference of latent gene program activities, which are decoded through a shared loading matrix to predict gene expressions. The model incorporates tiered high-variance gene weighting and a gene-wise correlation term to prioritize spatially informative patterns.
Results
The model was evaluated across five cohorts, totaling 421 slides and 799,085 spots, demonstrating the best aggregate performance in predicting gene expressions. SpaFactor showed significant improvements in accuracy, particularly for spatially variable genes, and effectively recovered biologically organized spatial patterns, indicating its potential for practical applications in pathology and biomarker discovery.
Implications
SpaFactor's lightweight and efficient approach to modeling spatial gene expression could facilitate broader adoption of spatial transcriptomics in clinical and research settings, enabling large-scale studies without the need for costly and complex experimental setups. This could lead to advancements in biomarker discovery, patient stratification, and understanding of tumor microenvironments.
Sample-Weighted End-to-End Trace-Norm Geometry for Multitask Learning
Theory
- Introduces a sample-size-weighted trace norm for measuring multitask complexity.
- Derives the empirical Rademacher complexity for the fixed-radius class of end-to-end maps.
- Demonstrates significant gaps in orientation and factorization in multitask models.
- Empirical results show improved performance of weighted joint nuclear regularization over traditional methods.
Read more
Sample-Weighted End-to-End Trace-Norm Geometry for Multitask Learning
Summary
This paper investigates the complexities of multitask learning models, which combine shared representations with task-specific outputs. Traditional approaches often treat these components separately, potentially losing critical information about their interactions. The author proposes a novel approach using the sample-size-weighted trace norm of the end-to-end mapping from task coefficients to input-space predictors. By deriving the empirical Rademacher complexity for this fixed-radius class, the paper establishes a more intrinsic measure of multitask complexity. The study reveals that the sample-weighted trace norm can be characterized through optimizing a positive-definite task covariance and through equivalent invertible refactorizations in finite-dimensional spaces. The results demonstrate significant gaps in orientation and factorization, particularly in deep networks where linear layers may cancel each other out. The paper also connects the statistical measure to the geometry of shared maps, leading to an exact Sobolev task Gram matrix determined by multiplicity and local directional distortion. Empirical evaluations across multiple unseen datasets show that the proposed weighted joint nuclear regularization outperforms traditional methods, indicating its effectiveness in improving generalization in multitask learning scenarios.
Methodology
The author employs a theoretical framework to analyze the end-to-end mapping of multitask learning models, focusing on the sample-size-weighted trace norm. The methodology includes deriving the empirical Rademacher complexity, optimizing task covariance matrices, and evaluating the performance of the proposed regularization technique through extensive empirical testing across various datasets.
Results
The study finds that the weighted joint nuclear regularization improves average population excess by 0.00764 compared to unweighted nuclear regularization, with a 95% confidence interval. Additionally, correct task counts enhance performance metrics significantly, indicating the robustness of the proposed approach across various imbalanced rank-suites.
Implications
The findings suggest that incorporating sample-weighted trace norms can lead to better generalization in multitask learning models, potentially influencing future research in neural network architectures and regularization techniques. This approach may also have applications in fields requiring multitask learning, such as computer vision and natural language processing.
Spectral-Guided Diffusion: Accelerating Inference via Static Spectral Layer Scheduling
Generative Models
Efficient ML
Large Language Models
- Introduction of Spectral Concentration Ratio (SCR) for scheduling in diffusion models.
- Static scheduling allows for the reuse of cached updates, improving efficiency.
- Achieves 2.8x to 3.0x speedup in inference time compared to eager inference.
- Outperforms existing scheduling methods in quality preservation.
Read more
Spectral-Guided Diffusion: Accelerating Inference via Static Spectral Layer Scheduling
Summary
This paper introduces a novel approach to accelerate inference in diffusion models through a method called Spectral-Guided Diffusion, which utilizes a static scheduling mechanism based on the Spectral Concentration Ratio (SCR). The authors propose that pretrained weights can be leveraged to identify residual branches in the model that do not need to be recomputed during inference. By measuring the energy distribution of singular values, the SCR provides a proxy for sensitivity, allowing the authors to assign a deterministic lifetime to each scheduled unit. This method enables the reuse of cached updates for frozen units while maintaining the propagation of the current residual stream and external conditioning. The proposed scheduling does not require any router, calibration prompts, or input-dependent search, making it efficient and straightforward. The authors demonstrate that their method outperforms existing scheduling techniques in terms of quality preservation while achieving significant speedups in inference time, with reported wall-clock speedups of 2.8x to 3.0x over traditional eager inference methods. The results indicate that the SCR/Frobenius scheduling approach is effective across various architectures and tasks, including retrieval, reasoning, and open-ended generation.
Methodology
The authors developed a static scheduling method based on the Spectral Concentration Ratio (SCR) and Frobenius magnitude to determine the sensitivity of residual branches in diffusion models. This approach allows for the caching of updates for certain layers, reducing redundant computations during inference. The SCR is calculated offline, and the scheduling is reused for every input without requiring additional training or calibration.
Results
The proposed method achieved a wall-clock speedup of 2.8x to 3.0x over eager inference methods. It maintained quality better than random and other existing scheduling techniques across various architectures, including LLaDA-8B, DiT-XL/2, U-ViT-L, and SDXL. The method was validated through extensive testing on multiple tasks such as retrieval, reasoning, and summarization.
Implications
The findings suggest that leveraging pretrained weights for static scheduling can significantly enhance the efficiency of inference in diffusion models. This approach has potential applications in real-time systems where computational resources are limited, and rapid inference is crucial. It also opens avenues for further research into optimizing other types of neural networks using similar principles.
When Identical Rows Disagree: From Benchmark Identifiability to Replication-Robust Anomaly Detection
Theory
- Identical rows in datasets can lead to misleading anomaly detection results.
- The paper introduces SCOUT, a novel framework for anomaly detection that is robust to row replication.
- An extensive audit of 690 datasets reveals significant issues with train-test overlaps and label conflicts.
- SCOUT maintains performance comparable to existing methods while improving replication-invariant AUROC.
Read more
When Identical Rows Disagree: From Benchmark Identifiability to Replication-Robust Anomaly Detection
Summary
This paper addresses the challenges posed by identical rows in datasets, which can arise from various factors such as business frequency, resampling, or extraction errors. The author identifies that treating such rows as independent samples can lead to misleading evaluations in anomaly detection. The study reveals three main consequences of this ambiguity: an evaluation ceiling imposed by feature-identical rows, sensitivity of row-weighted AUROC to replication, and the learning of a biased law by row-trained detectors. An audit of 690 OddBench datasets uncovers significant overlaps and conflicts in training and testing data, highlighting the need for a more nuanced approach to anomaly detection. The paper introduces SCOUT (Support–Count Orthogonalized Unsupervised Testing), a framework that shifts the focus from row-based to support-based anomaly detection, improving robustness against replication. SCOUT employs a one-class model for distinct support patterns and a conditional count model when necessary, ensuring control over false-positive rates. The results demonstrate that SCOUT is comparable to existing methods like Isolation Forest while enhancing replication-invariant performance. The findings emphasize the importance of understanding the semantics of multiplicity in datasets and propose a diagnostic-to-detector framework for better anomaly detection.
Methodology
The paper conducts an audit of 690 OddBench datasets to identify issues related to identical rows and their implications for anomaly detection. It introduces SCOUT, which utilizes a one-class model for distinct support patterns and a conditional count model for exposure normalization. The methodology emphasizes a shift from row-based to support-based evaluation, incorporating factorwise split-conformal calibration to control false-positive rates.
Results
The analysis of the OddBench datasets revealed that 355 datasets had train-test overlaps, 147 had conflicting labels for identical features, and 137 had test anomalies identical to training normals. SCOUT demonstrated non-inferior performance to row-wise Isolation Forest in raw AUROC, with a slight improvement in replication-invariant AUROC. The results indicate that changing the evaluation unit from rows to observed support significantly affects AUROC metrics across multiple datasets.
Implications
The findings suggest that anomaly detection methods need to account for the complexities introduced by data multiplicity. The SCOUT framework could be applied in various domains where data integrity and anomaly detection are critical, such as finance, healthcare, and cybersecurity. This work encourages a reevaluation of existing benchmarks and methodologies in anomaly detection.
Revalidation Beats Stateful Routing for Scientific Surrogates Under Distribution Shift
Theory
- Revalidation of models on new data outperforms fixed model selection in dynamic environments.
- Stateful adaptive controllers do not provide significant advantages and can worsen performance.
- The benchmark includes a comprehensive evaluation of multiple surrogate models across various tasks and regimes.
- Task-specific model performance varies significantly, indicating the need for tailored approaches.
Read more
Revalidation Beats Stateful Routing for Scientific Surrogates Under Distribution Shift
Summary
This paper investigates the effectiveness of model selection strategies for scientific surrogates in the presence of distribution shifts. The study contrasts the traditional approach of maintaining a fixed surrogate model with the revalidation of candidate models as new data arrives. The authors developed a reproducible streaming benchmark, RegimeShift-Surrogates, which encompasses eight analytic and dynamical tasks across four regimes, utilizing various surrogate models including multilayer perceptrons and Kolmogorov–Arnold networks. The results demonstrate that revalidating models on new data consistently outperforms the selection of a fixed model chosen in hindsight. Specifically, the mean log regret achieved through current revalidation was 0.091, compared to 0.192 for the best fixed model. The study also found that stateful adaptive controllers, which attempt to retain evidence from previous validations, did not improve performance and often led to worse outcomes. The findings highlight the importance of fresh validation evidence in model selection under changing conditions, suggesting that relying on outdated information can be detrimental.
Methodology
The authors constructed a benchmark called RegimeShift-Surrogates, which includes eight task families and four regime schedules. They conducted a confirmatory run involving 30,720 model fits and 3,200 scored deployment windows. Various model selection strategies were compared, including fixed model selection, current-validation, temporally smoothed, and stateful methods. The performance was assessed based on mean log regret against an oracle model selection.
Results
The study found that the mean log regret for the current revalidation strategy was 0.091, significantly better than the 0.192 for the best fixed model. The paired difference of -0.101 was statistically significant, indicating that revalidation is generally superior. None of the stateful alternatives improved upon the current-validation approach, and task-specific results showed varying model effectiveness across different scenarios.
Implications
The findings suggest that in scientific machine learning applications, continuously validating models with fresh data is essential for maintaining performance under changing conditions. This has implications for real-time model deployment in various scientific and engineering fields, where data characteristics may shift over time.
MF-SCBO : Multi-fidelity Scalable Constrained Bayesian Optimization
Optimization
- MF-SCBO effectively integrates multi-fidelity optimization with constrained Bayesian optimization.
- The method addresses high-dimensionality and non-nested sampling challenges.
- Two trust region center selection variants are proposed to improve stability.
- Experimental results indicate better convergence rates compared to existing methods.
Read more
MF-SCBO : Multi-fidelity Scalable Constrained Bayesian Optimization
Summary
The paper presents MF-SCBO, a novel method for multi-fidelity scalable constrained Bayesian optimization, addressing the challenges of high-dimensional black-box optimization problems with constraints. Traditional optimization methods struggle with expensive evaluations, motivating the need for efficient data usage through multi-fidelity optimization. MF-SCBO extends the Scalable Constrained Bayesian Optimization (SCBO) framework to incorporate multiple fidelity levels, allowing for the use of cheaper, lower-fidelity evaluations alongside high-fidelity ones. The method is designed to handle arbitrary fidelity levels, high-dimensional spaces, and non-nested sampling, which are often overlooked in existing approaches. The authors introduce two variants for selecting the trust region center based on high-fidelity evaluations and predictions, and they employ a leave-one-out criterion to enhance predictive accuracy. The effectiveness of MF-SCBO is demonstrated through extensive experiments on benchmark functions and challenging real-world problems, showing superior convergence compared to both single-fidelity SCBO and other multi-fidelity methods.
Methodology
The MF-SCBO method builds upon the SCBO framework by incorporating multi-fidelity evaluations. It utilizes Gaussian process surrogates to model the objective function and constraints, allowing for efficient exploration of the optimization space. The method introduces two strategies for determining the trust region center, based on either high-fidelity evaluations or predictions, and employs a leave-one-out criterion to enhance the reliability of predictions. The acquisition function is adapted to account for both input variables and fidelity levels, optimizing the selection of evaluation points.
Results
The experimental evaluation of MF-SCBO on various benchmark and real-world problems demonstrates that it consistently outperforms both the single-fidelity SCBO and the Multi-Fidelity Max-value Entropy Search (MF-MES) in terms of convergence speed and optimization efficiency, particularly in high-dimensional and constrained settings.
Implications
The MF-SCBO method has significant implications for fields requiring optimization under constraints, such as engineering design, machine learning hyperparameter tuning, and control systems, where evaluation costs are high. Its ability to leverage multiple fidelity levels can lead to more efficient optimization processes and reduced computational budgets.
Error- and Prediction-Driven Motor Learning in the Cortico-Cerebellar Loop
Robotics
- Proposes a cerebellum-inspired control framework for motor learning.
- Demonstrates that multiplexed predictive representations improve online correction accuracy.
- Shows that incorporating feedback in the cerebellar loop accelerates adaptation significantly.
- Finds that single-signal predictions are insufficient under delayed feedback conditions.
Read more
Error- and Prediction-Driven Motor Learning in the Cortico-Cerebellar Loop
Summary
This paper addresses the challenge of robust control in dynamic environments, particularly under delayed sensory feedback, which is critical in both robotics and neuroscience. The authors propose a novel cerebellum-inspired control framework that integrates multiplexed predictive representations with internal feedback mechanisms. This model encodes both kinematic variables and task-relevant error signals, enabling accurate online corrections despite delays in feedback. The study reveals that traditional cerebellar models, which focus solely on forward predictions, are inadequate for explaining the rapid online corrections and adaptations observed in biological systems. By incorporating feedback within the cerebellar loop, the proposed model significantly accelerates adaptation processes, reducing learning time by an order of magnitude. The findings suggest that multiplexing and feedback together provide a unified mechanism for effective online control and rapid learning, thereby enhancing our understanding of motor learning processes in biological systems and their applications in artificial intelligence.
Methodology
The authors developed a recurrent neural network (RNN) model augmented with a cerebellum-inspired module that integrates delayed sensory feedback. They constructed a synthetic instructed-delay center-out reaching task to evaluate the model's performance in terms of online correction and adaptation. The model architecture includes a cortical component represented by an RNN and a cerebellar module that produces corrective signals based on delayed feedback.
Results
The results indicate that the proposed model with multiplexed cerebellar outputs significantly improves the accuracy of online corrections and facilitates faster and more stable adaptation compared to traditional models. The integration of feedback within the cerebellar loop allows the model to reproduce key neural population patterns observed in biological systems, demonstrating its effectiveness in handling delayed sensory feedback.
Implications
The findings have potential implications for the design of intelligent robotic systems that require robust motor control in dynamic environments. The insights gained from this study could inform the development of more adaptive and efficient learning algorithms in artificial intelligence, particularly in areas requiring real-time decision-making and error correction.
Learning from Mixed-Quality Deployment Experience for Robot Manipulation
Robotics
Reinforcement Learning
- PACL combines a Q-conditioned diffusion actor with a chunk-level critic for learning from mixed-quality experiences.
- The method incorporates future latent dynamics prediction to enhance value learning under sparse rewards.
- PACL outperforms strong baselines in both simulated and real-world robot manipulation tasks.
- The approach allows robots to improve their policies using only the experiences gained during autonomous operation.
Read more
Learning from Mixed-Quality Deployment Experience for Robot Manipulation
Summary
This paper introduces Predictive Action Chunk Learning (PACL), a novel approach for improving robot manipulation policies by leveraging mixed-quality deployment experiences. The authors address the challenge of learning from experiences that include successful executions, partial progress, and failures, which are typically not suitable for direct incorporation into imitation learning due to the risk of reinforcing undesirable behaviors. PACL employs a chunk-level critic to evaluate temporally extended action sequences and augments temporal difference learning with future latent predictions, thereby enhancing long-horizon value estimation. The critic converts Q-values into discrete quality conditions that guide a diffusion actor in learning from mixed-quality experiences without treating all behaviors as equivalent. The proposed method is tested in both simulated and real-world robot manipulation tasks, demonstrating consistent improvements over pretrained policies and outperforming established imitation learning and offline reinforcement learning baselines. The findings suggest that PACL effectively utilizes naturally accumulated deployment data for policy refinement, marking a significant advancement in post-deployment learning for autonomous robots.
Methodology
PACL utilizes a chunk-level critic to evaluate sequences of actions and employs future latent dynamics prediction to provide dense supervision for value learning. The critic generates discrete quality conditions from Q-values, guiding a diffusion actor to learn from mixed-quality deployment experiences. During inference, the actor proposes multiple action chunks, and the critic selects the highest-valued candidate.
Results
The experiments demonstrate that PACL consistently enhances the performance of pretrained policies, surpassing the results of strong imitation learning and offline reinforcement learning methods. The ablation studies validate the positive impact of both the Q-conditioned actor and the predictive critic on the overall improvements.
Implications
The findings suggest that PACL can significantly enhance the learning capabilities of autonomous robots by enabling them to refine their policies using naturally accumulated experiences, potentially leading to more robust and adaptable robotic systems in real-world applications.
Decoupling Knowledge and Privacy: Post-Task Self-Distillation Replay for LLM Continual Learning
NLP
Large Language Models
Theory
- Introduces SPARK, a framework for selective sensitive-output control in continual learning.
- Decouples knowledge retention and privacy correction to effectively manage PII outputs.
- Employs Self-Distillation Replay to preserve task behaviors while avoiding reinforcement of sensitive data.
- Demonstrates strong continual learning utility alongside effective PII suppression.
Read more
Decoupling Knowledge and Privacy: Post-Task Self-Distillation Replay for LLM Continual Learning
Summary
This paper addresses the challenge of privacy-preserving continual learning (PPCL) in large language models (LLMs), focusing on the need to mitigate the reproduction of sensitive content while retaining useful knowledge from previous tasks. The authors introduce a novel framework called SPARK, which decouples knowledge retention and privacy correction by first freezing the learned post-task distribution and then applying selective corrections to reduce the likelihood of personally identifiable information (PII) outputs. The methodology involves two main phases: Self-Distillation Replay, which learns the current task while preserving behaviors from previous tasks, and Post-Task Privacy Correction, which targets annotated PII while anchoring non-PII behaviors. The proposed approach is evaluated extensively, demonstrating effective PII suppression without compromising continual learning utility. The results indicate that SPARK successfully balances the dual objectives of knowledge retention and privacy control, providing a robust solution for PPCL in LLMs.
Methodology
The methodology consists of two phases: (1) Self-Distillation Replay, which learns the current task while distilling behaviors from previous tasks to create a stable post-task checkpoint, and (2) Post-Task Privacy Correction, which selectively reduces the likelihood of annotated PII outputs while anchoring non-PII behaviors from both current and previous tasks.
Results
The extensive evaluations show that SPARK achieves significant PII suppression while maintaining high performance in continual learning tasks. The framework effectively balances the need for privacy with the retention of useful knowledge across diverse settings, outperforming existing methods in terms of both privacy control and task acquisition stability.
Implications
The findings have important implications for developing privacy-preserving AI systems, particularly in sensitive domains where data privacy is paramount. The SPARK framework can be applied to various applications involving continual learning in LLMs, ensuring compliance with privacy regulations while enhancing model performance.
Active Client Selection in Federated Trajectory Prediction with Uncertainty-Awareness and Heterogeneous Complexity
Federated Learning
Robotics
Time Series
- Introduces a federated learning framework for trajectory prediction that addresses scene uncertainty and complexity heterogeneity.
- Develops active client selection methods that prioritize clients based on uncertainty and complexity metrics.
- Demonstrates improved performance and faster convergence in federated trajectory prediction compared to traditional methods.
- Provides a reproducible setup for creating heterogeneous federated partitions of trajectory datasets.
Read more
Active Client Selection in Federated Trajectory Prediction with Uncertainty-Awareness and Heterogeneous Complexity
Summary
This paper addresses the challenges of trajectory prediction for autonomous vehicles using Federated Learning (FL). Traditional methods rely on centralized datasets, which are often fragmented in real-world scenarios. The authors propose a novel framework that incorporates uncertainty-awareness and complexity heterogeneity into the client selection process for FL. The framework includes a family of active client selection methods that prioritize clients based on their informativeness, specifically focusing on per-client negative log-likelihood and estimated aleatoric uncertainty. The proposed methods, such as FLTP-AU, FLTP-NLL, and FLTP-DR, aim to enhance model training by selecting clients that provide the most valuable data. Experiments conducted on the Argoverse dataset demonstrate that the proposed FL framework outperforms locally trained models and accelerates convergence, particularly in scenarios with significant heterogeneity in scene complexities. The findings suggest that training on more complex scenes can improve performance on simpler cases, highlighting the importance of tailored client selection in federated trajectory prediction.
Methodology
The authors propose a federated trajectory prediction framework (FLTP) that utilizes an uncertainty-aware objective based on negative log-likelihood. They develop several active client selection methods (FLTP-AU, FLTP-NLL, FLTP-DR) that incorporate uncertainty and complexity metrics to guide the selection of clients for model training. The methods leverage local training data characteristics and scene complexity features to optimize the learning process.
Results
Experiments on the Argoverse dataset show that the proposed federated trajectory prediction methods outperform locally trained models. The uncertainty-aware client selection methods lead to faster convergence and improved performance metrics, including minimum Average Displacement Error (minADE), minimum Final Displacement Error (minFDE), and Miss Rate (MR). The FLTP-DR method, which accounts for both uncertainty and complexity, achieves the best generalization in heterogeneous scenarios.
Implications
The findings suggest that federated learning can effectively leverage decentralized data for trajectory prediction in autonomous vehicles, enhancing model performance and generalization. The active client selection methods can be applied to other domains facing similar challenges of data fragmentation and heterogeneity, potentially improving collaborative learning systems.
Beyond Average Safety: Chance-Constrained LLM Fine-tuning
NLP
Large Language Models
Optimization
- Introduces a chance-constrained formulation for safety-preserving fine-tuning of LLMs.
- Develops a differentiable majorization of the empirical chance constraint for tractable optimization.
- Presents a constraint-aware gradient descent method that maintains feasibility during fine-tuning.
- Demonstrates superior performance in safety preservation compared to traditional methods through empirical experiments.
Read more
Beyond Average Safety: Chance-Constrained LLM Fine-tuning
Summary
This paper addresses the challenge of fine-tuning large language models (LLMs) while preserving safety on critical prompts. Traditional methods often focus on controlling average safety loss, which can obscure severe failures in rare cases. The authors propose a chance-constrained optimization approach that limits the fraction of safety examples whose degradation exceeds a specified threshold. They introduce a differentiable majorization of the empirical chance constraint, allowing for a tractable conservative constraint on safety-regression rates. A novel constraint-aware gradient descent method is developed to ensure that fine-tuning updates remain feasible while minimizing safety regressions. The authors conduct extensive experiments across three tasks and models, demonstrating that their approach consistently outperforms existing baselines in terms of safety preservation and performance improvement. This work advocates for a shift in perspective, treating safety preservation as a reliability-constrained optimization problem rather than merely an average-risk regularization task.
Methodology
The authors formulate safety-preserving fine-tuning as a chance-constrained optimization problem, replacing the discontinuous indicator function of the empirical chance constraint with a differentiable majorizer. They develop a constraint-aware gradient descent method that adjusts the fine-tuning direction to remain within a safe set in parameter space, ensuring minimal modifications to the update while preserving safety.
Results
The proposed method outperformed standard weighted and constrained baselines in extensive experiments across three tasks (SST-2, AG News, GSM8K) and three models (Qwen 3.5 4B, Qwen 3.5 9B, Llama 3.1 8B), demonstrating improved fine-tuning performance and reduced safety-regression rates.
Implications
This research has significant implications for the deployment of LLMs in safety-critical applications, suggesting that fine-tuning strategies should prioritize reliability and safety over average performance metrics. It opens avenues for further research into constrained optimization techniques in machine learning.
Beyond Compression: Training Latent Representations for Stable Long-Horizon Rollout in Neural Surrogate Solvers
Generative Models
Efficient ML
Theory
- Latent dynamics models (LDMs) face instability during long-horizon rollouts due to misalignment in training objectives.
- Training-level interventions can significantly improve long-horizon prediction stability while reducing errors by around 40%.
- The proposed methods include Koopman operator learning, noise injection, and multi-step fine-tuning.
- The resulting models are more computationally efficient, requiring two orders of magnitude fewer floating point operations.
Read more
Beyond Compression: Training Latent Representations for Stable Long-Horizon Rollout in Neural Surrogate Solvers
Summary
This paper addresses the challenges faced by latent neural surrogate solvers (LDMs) in achieving stable long-horizon rollouts for time-dependent physical system simulations. While LDMs offer significant computational advantages by operating in a compressed latent space, they often suffer from rapid error accumulation during long autoregressive predictions. The authors identify that this instability arises not from the latent representation itself but from its conventional training focused solely on reconstruction accuracy. To enhance long-horizon forecasting, the paper proposes several training-level interventions, including Koopman operator learning, Hamming noise injection, and multi-step rollout fine-tuning. These methods aim to align latent representations with the requirements of long-horizon predictions. The findings reveal that while these interventions can degrade traditional metrics like reconstruction accuracy, they significantly reduce long-rollout errors by approximately 40% and achieve comparable or superior accuracy to full-resolution models across various physics benchmarks. The resulting models are also more efficient, requiring significantly fewer computational resources. The study emphasizes the importance of designing neural compression techniques not just for dimensionality reduction but for creating stable dynamical evolution, which is crucial for reliable scientific applications.
Methodology
The authors systematically evaluate five training-level interventions for LDMs, focusing on spatially structured latent compression, Koopman-inspired dynamics constraints, KL divergence regularization, noise injection during training, and recursive training aimed at stabilizing performance. These methods were tested across three distinct dynamic evolution problems: spinodal decomposition, active matter evolution, and crystal-plasticity fatigue simulations.
Results
The interventions led to a reduction in long-rollout error by approximately 40%, with the models achieving accuracy that matches or exceeds that of full-resolution models on two physics benchmarks. Additionally, the models demonstrated significant computational efficiency, requiring 2 orders of magnitude fewer floating point operations and half the GPU memory compared to traditional methods.
Implications
The findings suggest that LDMs can be effectively utilized in scientific and engineering applications where high-fidelity simulations are computationally expensive. The proposed training strategies could enhance the reliability and efficiency of neural surrogate solvers, making them viable tools for complex simulations.
Safety-oriented pedestrian trajectory prediction at urban intersections using time-to-collision and crossing-zone context
Robotics
Time Series
Optimization
- Introduces a safety-oriented framework for pedestrian trajectory prediction at urban intersections.
- Combines pedestrian motion history with Time-to-Collision (TTC) and crossing-zone context.
- Implements a pooled LSTM model to enhance prediction accuracy.
- Demonstrates significant reductions in Average Displacement Error (ADE) and Final Displacement Error (FDE).
Read more
Safety-oriented pedestrian trajectory prediction at urban intersections using time-to-collision and crossing-zone context
Summary
This paper addresses the critical issue of pedestrian safety at urban intersections by proposing a safety-oriented framework for predicting pedestrian trajectories. The authors highlight the importance of accurately predicting pedestrian movements, particularly in environments where interactions with vehicles are frequent. The proposed framework integrates pedestrian motion history with Time-to-Collision (TTC) data and crossing-zone context to enhance prediction accuracy. The study utilizes naturalistic trajectory data from the inD dataset, employing various neural architectures, notably a pooled Long Short-Term Memory (LSTM) model that separately encodes TTC and crossing-zone information before integrating it with pedestrian positions. The performance of the model is evaluated using conventional metrics such as Average Displacement Error (ADE) and Final Displacement Error (FDE), alongside a novel weighted loss function that emphasizes larger prediction errors. The results demonstrate that the incorporation of TTC and crossing-zone context significantly improves prediction accuracy, reducing ADE and FDE while also decreasing the frequency of large prediction errors. The findings suggest that a safety-oriented approach to trajectory prediction can enhance proactive road safety measures, although further validation across diverse pedestrian scenarios and datasets is necessary.
Methodology
The study employs a pooled Long Short-Term Memory (LSTM) neural network architecture that encodes Time-to-Collision (TTC) histories and crossing-zone context separately, integrating them with pedestrian position data. The model is trained using a weighted loss function that prioritizes larger coordinate-wise errors, enhancing the focus on safety-critical prediction scenarios.
Results
The final model configuration, which incorporates TTC and crossing-zone information, achieved an Average Displacement Error (ADE) of 0.184 m and a Final Displacement Error (FDE) of 0.491 m. This configuration also resulted in a 33.5% reduction in ADE exceedance counts and a 6.3% reduction in FDE exceedance counts compared to the position-only LSTM model.
Implications
The findings of this study have significant implications for the development of Advanced Driver Assistance Systems (ADAS) and automated driving technologies, as they highlight the potential for improved pedestrian safety through enhanced trajectory prediction. The integration of safety-oriented metrics in trajectory prediction models can lead to better intervention strategies in high-risk scenarios.