AI-generated summaries
Today's ML research,
without the noise.
Daily summaries of the latest machine learning papers from arXiv, processed every 8 hours.
63
Papers today
8h
Update frequency
7
Days of history
TimeRLM: Recursive Language Models Enable Precise Anomaly Localization in Long-Context Time-Series
Time Series
NLP
Reinforcement Learning
- TimeRLM utilizes Recursive Language Models to improve anomaly localization in long-context time-series data.
- The introduction of the AnomalyXL benchmark allows for precise evaluation of anomaly detection capabilities.
- TimeRLM outperforms existing models, achieving significant improvements in localization and classification tasks.
- Reinforcement learning further enhances the model's performance and efficiency.
Read more
TimeRLM: Recursive Language Models Enable Precise Anomaly Localization in Long-Context Time-Series
Summary
The paper introduces TimeRLM, a Recursive Language Model (RLM) designed to enhance anomaly localization in long-context time-series data. Traditional Time Series Language Models (TSLMs) struggle with long time horizons, leading to decreased performance in anomaly detection. TimeRLM addresses this by maintaining context externally and allowing for recursive interactions with the time-series data through code. The authors propose a new benchmark, AnomalyXL, which includes synthetic long-context anomaly localization tasks. TimeRLM significantly outperforms existing TSLMs and single-pass models on these tasks, achieving high Intersection over Union (IoU) scores for localization and classification. Further improvements are made through reinforcement learning, resulting in a model that is more efficient and effective, even when evaluated on real-world datasets such as ECG and sleep recordings. The findings suggest that recursive interaction with time-series data is a promising approach for long-horizon temporal reasoning and retrieval.
Methodology
The authors developed TimeRLM, which manipulates time-series data using recursive interactions and code execution. They created the AnomalyXL benchmark with programmatically injected anomalies to evaluate performance across various tasks. The model was trained and fine-tuned using reinforcement learning techniques to enhance its anomaly detection capabilities.
Results
TimeRLM achieved a maximum IoU of 0.682 on localization tasks and 0.745 on classify-with-evidence tasks, significantly outperforming TSLMs and other baseline models. The post-trained version of TimeRLM required fewer agent interaction turns to produce final answers and maintained or improved performance on real-world datasets.
Implications
The findings suggest that recursive interaction with time-series data can lead to more effective anomaly detection in various fields, including healthcare, finance, and industrial monitoring. This approach could improve the efficiency of monitoring systems that rely on long-term data streams.
Amortized Interventional Forecasting for Multivariate CIR Processes
Time Series
- Introduces CIR-ACTIVA, an amortized model for causal effect estimation in financial time series.
- Develops a causal multivariate CIR data-generating process to provide necessary ground truth for evaluation.
- Demonstrates superior performance of CIR-ACTIVA over traditional observational forecasting methods.
- Enables the analysis of interventional scenarios, such as stress testing in financial markets.
Read more
Amortized Interventional Forecasting for Multivariate CIR Processes
Summary
This paper addresses the limitations of the Cox-Ingersoll-Ross (CIR) process in financial modeling, specifically its inability to capture causal influences between correlated time series. The authors propose two significant contributions: first, the development of CIR-ACTIVA, an amortized model for estimating distributional causal effects that predicts the response of financial series to external shocks without the need for retraining. Second, they introduce a causal multivariate CIR data-generating process that provides both observational and interventional ground truth data, which is typically unavailable in real market scenarios. The model is evaluated using credit default swap (CDS) spreads as a testbed. The results demonstrate that CIR-ACTIVA outperforms existing baselines in terms of causal selectivity and horizon-resolved calibration, particularly excelling in short-term forecasting relevant for stress testing. This work opens avenues for conducting 'what-if' analyses in financial systems, enabling better decision-making in response to market shocks.
Methodology
The authors leverage structural causal models and amortized causal effect estimation to develop CIR-ACTIVA. The model treats time series data as time-stamped observations and utilizes a horizon-bucketed decoder for predictions. A causal multivariate CIR simulator is also introduced to generate paired observational and interventional samples for training and evaluation.
Results
CIR-ACTIVA outperformed baseline models in causal selectivity and horizon-resolved calibration, particularly excelling in short-term forecasts. The model effectively shields non-affected series from phantom shock effects, demonstrating its robustness in interventional scenarios.
Implications
The findings suggest that CIR-ACTIVA can significantly enhance financial forecasting models, particularly in stress testing and risk management. The ability to conduct causal analyses opens new pathways for understanding market dynamics and improving decision-making processes in finance.
Assessment of Conditional Diffusion Model for Synthetic Histopathology Image Generation
Generative Models
Computer Vision
- Introduction of a two-step training approach for denoising diffusion models.
- Development of domain-specific modified FID and IS metrics using pathology-pretrained foundation models.
- Empirical validation of the correlation between synthetic data quality metrics and downstream task performance.
- Demonstration that increasing data variety is more beneficial for model performance than improving individual image quality.
Read more
Assessment of Conditional Diffusion Model for Synthetic Histopathology Image Generation
Summary
This paper addresses the challenge of data scarcity in computational pathology by investigating the quality of synthetic histopathology images generated using conditional denoising diffusion models. The authors highlight limitations in conventional evaluation metrics like FrΓ©chet Inception Distance (FID) and Inception Score (IS) when applied to histopathology images, as these metrics rely on feature extractors pretrained on ImageNet, which do not align with the characteristics of medical imagery. To overcome these limitations, the authors propose modified evaluation metrics that utilize foundation models pretrained on digital pathology datasets, supplemented by precision-recall based metrics. The study employs a two-step training approach to generate synthetic datasets from four benchmark datasets (MoNuSeg, TNBC, 2018 Data Science Bowl, PanNuke) and assesses the correlation between synthetic data quality metrics and downstream nuclei segmentation performance using metrics such as the aggregated Jaccard index (AJI+) and the Dice coefficient. The findings suggest that pathology-specific metrics provide better discriminative power and that increasing the variety of generated training data correlates more positively with segmentation model performance than enhancing the visual fidelity of individual images.
Methodology
The study utilized conditional denoising diffusion probabilistic models (DDPMs) in a two-step training approach. Separate DDPMs were trained on four distinct histopathology datasets, using instance nuclei segmentation masks to guide the generation of synthetic images. Modified evaluation metrics specific to histopathology were developed, focusing on congruence and coverage, and the correlation between these metrics and downstream segmentation performance was analyzed.
Results
The modified Inception Score showed a higher correlation with downstream task performance (r = 0.6096 with AJI+, p = 0.0122) compared to the original Inception Score (r = 0.0708, p = 0.7944). The study concluded that pathology-specific metrics enhance the evaluation of synthetic data quality and that a greater variety of generated training data positively impacts segmentation model performance.
Implications
The findings suggest that adopting domain-specific evaluation metrics can improve the assessment of synthetic data quality in medical applications, potentially leading to better training datasets for AI models in digital pathology. This could facilitate advancements in computational pathology by providing more robust training data for downstream tasks.
Convex-Hull-Neighborhood Smooth Dual Generalization: Controlling Local Correction Propagation in Offline RL
Reinforcement Learning
Robotics
Theory
- CSDG separates local corrections from in-sample targets to control OOD value contributions.
- The method derives theoretical bounds for correction propagation and performance evaluation.
- Empirical results show improved performance in offline RL tasks without pessimistic penalties.
Read more
Convex-Hull-Neighborhood Smooth Dual Generalization: Controlling Local Correction Propagation in Offline RL
Summary
This paper addresses the challenges of offline reinforcement learning (RL), particularly the amplification of estimation errors at out-of-distribution (OOD) actions due to bootstrapping. The authors introduce a novel approach called Convex Hull Neighborhood Smooth Dual Generalization (CSDG), which reformulates the Bellman backup to include a local correction term that explicitly controls the contribution of generalized values from OOD actions. The CSDG method utilizes asymmetric perturbations to generate in-sample and OOD candidates, smoothing these to define a locally generalized target. A mixture coefficient scales the contribution of this correction in each recursive update. The paper derives several theoretical bounds and criteria for performance evaluation, including exact one-step corrections and fixed-point bounds. Empirical results demonstrate that CSDG achieves strong performance on standard offline RL benchmarks, including Gym-MuJoCo and AntMaze, indicating its effectiveness in stabilizing value estimation and improving policy learning without the need for additional pessimistic penalties.
Methodology
The authors propose CSDG, which organizes local smoothing and correction control through a single adjustment to the Bellman target. Asymmetric perturbations are used to sample candidates for both in-sample and OOD actions, and a mixture coefficient is introduced to regulate the contribution of the local correction in recursive updates. Theoretical analysis provides bounds on the correction's impact and characterizes the implicit policies induced by the method.
Results
CSDG demonstrated strong aggregate performance on benchmark tasks, achieving stable value estimates in offline RL scenarios. The method effectively mitigated the overestimation of OOD actions, leading to improved policy convergence.
Implications
The findings suggest that CSDG can enhance offline RL applications by providing a robust framework for controlling local correction propagation, potentially leading to better policy learning from historical data. This could be particularly beneficial in domains where data collection is expensive or impractical.
WorldDynCache: Risk-Controlled Latent Dynamics Approximation for Diffusion World Model
Generative Models
Efficient ML
Computer Vision
- WorldDynCache addresses the inefficiencies of existing caching methods in diffusion world models.
- The framework includes a risk estimator to track and calibrate approximation defects.
- It employs a condition- and phase-aware surrogate for latent evolution approximation.
- WorldDynCache achieves significant speedups in inference time while preserving high generation quality.
Read more
WorldDynCache: Risk-Controlled Latent Dynamics Approximation for Diffusion World Model
Summary
The paper introduces WorldDynCache, a novel framework designed to enhance the efficiency of diffusion world models by addressing the limitations of existing caching methods. Traditional methods often overlook the cumulative effects of approximation errors and fail to account for changes in latent evolution due to varying conditions. WorldDynCache comprises two main components: a lightweight latent-transition risk estimator that monitors the impact of approximation defects and a condition- and phase-aware lifted latent surrogate that approximates latent evolution without requiring additional transformer evaluations. The framework significantly reduces inference time while maintaining high generation quality. Experimental results demonstrate that WorldDynCache achieves speedups of 4.92Γ and 2.15Γ on HunyuanVoyager-13B and Aether-5B, respectively, outperforming other caching methods in terms of generation quality metrics such as WorldScore, PSNR, SSIM, and LPIPS.
Methodology
WorldDynCache utilizes a risk-controlled approach to approximate latent transitions in diffusion models. It constructs candidate latent states based on previously observed exact transitions under compatible conditions. The framework incorporates a risk estimator to assess the reliability of approximations and decides when to revert to exact computations based on accumulated risks.
Results
The implementation of WorldDynCache resulted in speed improvements of 4.92Γ on HunyuanVoyager-13B and 2.15Γ on Aether-5B. It also achieved superior generation quality compared to existing caching methods, as evidenced by metrics such as WorldScore, PSNR, SSIM, and LPIPS.
Implications
WorldDynCache has the potential to enhance the efficiency of generative models in various applications, particularly in scenarios requiring rapid generation and real-time predictions. Its risk-controlled approach could be applied to other machine learning models that involve sequential decision-making or dynamic environments.
Verifier-Guided Model Discovery for Physical Dynamical Systems with Pretrained Symbolic Transformers
Interpretability
- Introduction of a verifier-guided workflow for symbolic model discovery.
- VG outperforms traditional methods in generalization across initial conditions.
- Successful transfer to high-dimensional physical data, particularly in vortex shedding scenarios.
- Demonstration of the importance of latent dynamics compatibility with pretraining.
Read more
Verifier-Guided Model Discovery for Physical Dynamical Systems with Pretrained Symbolic Transformers
Summary
This paper presents a novel approach to model discovery in nonlinear physical systems using a verifier-guided (VG) workflow that leverages pretrained symbolic transformers. The authors address the challenge of transferring models from synthetic data to high-dimensional physical systems by implementing executable tests for dynamical behavior and physical admissibility. The VG workflow is applied to the canonical Van der Pol oscillators, demonstrating superior performance over traditional methods across various initial conditions. Additionally, the methodology is tested on vortex shedding phenomena, where it successfully discovers reduced-order equations that accurately capture the dynamics without relying on specific candidate libraries or predefined structures. The findings highlight the importance of compatibility between latent dynamics and the pretraining distribution of the transformer, establishing a framework for interpretable and physically auditable forecasting in natural sciences.
Methodology
The authors developed a verifier-guided workflow that utilizes pretrained symbolic transformers to map synthetic ordinary differential equation (ODE) trajectories to explicit models. This workflow incorporates dynamical and physical admissibility tests to select candidate equations from a multi-trajectory pool, ensuring that the discovered models are both interpretable and physically credible.
Results
The VG workflow significantly outperformed the baseline ODEFormer method in recovering governing structures across held-out initial conditions for Van der Pol oscillators. In the context of vortex shedding, VG successfully identified reduced-order equations that accurately reflected shedding dynamics across varying Reynolds numbers, demonstrating stability in predictions even for withheld interpolation and extrapolation values.
Implications
The proposed methodology has the potential to enhance the interpretability and reliability of machine learning models in the natural sciences, facilitating better decision-making in engineering and scientific research. It opens avenues for further exploration of complex physical systems without the need for extensive case-specific knowledge.
CoSynFlow: Conformal Symplectic Neural Flows for Cross-System Prediction of Dissipative Hamiltonian Dynamics
Theory
Optimization
Time Series
- CoSynFlow maintains exact conformal symplecticity for dissipative Hamiltonian dynamics.
- The model enables continuous-time flow learning, allowing predictions for any time t in a single evaluation.
- It facilitates cross-system prediction, allowing a single model to generalize to unseen systems without retraining.
- The architecture preserves the geometric structure of the dynamics by design, addressing limitations of existing neural operator methods.
Read more
CoSynFlow: Conformal Symplectic Neural Flows for Cross-System Prediction of Dissipative Hamiltonian Dynamics
Summary
The paper introduces CoSynFlow, a novel neural flow model designed to learn continuous-time solution maps for dissipative Hamiltonian dynamics while preserving the conformal symplectic structure. Traditional neural operator methods often fail to maintain the geometric integrity of dynamics, particularly in dissipative systems where the symplectic form evolves due to dissipation. CoSynFlow addresses this by combining symplectic shear maps with explicit conformal scaling, ensuring that the learned maps conform to the underlying physics of dissipative Hamiltonian systems. The model is conditioned on a finite-dimensional Hamiltonian descriptor and a dissipation parameter, allowing it to predict solution maps for unseen systems without the need for retraining. The authors demonstrate that CoSynFlow achieves machine precision in structure preservation, exhibits the lowest long-horizon prediction error, and supports physics-informed training, making it a significant advancement in the field of scientific machine learning.
Methodology
CoSynFlow utilizes a combination of symplectic shear maps and explicit conformal scaling to construct time-t maps that adhere to the conformal symplectic structure. The model is conditioned on system-specific parameters, enabling it to generalize across different Hamiltonian systems without retraining. The architecture is designed to produce continuous-time predictions and allows for differentiability in time, which is leveraged for physics-informed training.
Results
The authors trained CoSynFlow on various randomly generated Hamiltonian systems and evaluated its performance on both held-out systems and analytic benchmarks. The results indicate that CoSynFlow maintains structure error at machine precision, achieves the lowest long-horizon prediction errors compared to existing methods, and effectively predicts dynamics beyond the training interval.
Implications
CoSynFlow has the potential to significantly enhance the modeling of complex dynamical systems in physics and engineering, particularly in scenarios involving dissipative processes. Its ability to generalize across systems without retraining could streamline the development of predictive models in scientific research and applications.
Bi-semantic Chemical Embedder for Joint Representation Learning of SMILES and Natural Language
NLP
Multimodal
- CheMatE is a bi-semantic embedding model that captures both SMILES and natural language in a unified representation space.
- The model employs a two-stage training process, combining masked language modeling with contrastive learning.
- CheMatE is trained on a large corpus of SMILES-annotated scientific documents, enhancing its contextual understanding.
- The model achieves strong performance across molecular property prediction and scientific language understanding tasks.
Read more
Bi-semantic Chemical Embedder for Joint Representation Learning of SMILES and Natural Language
Summary
The paper introduces CheMatE, a novel embedding model designed to jointly learn representations of SMILES (Simplified Molecular Input Line Entry System) and natural language. Traditional transformer models have been effective in natural language processing and have been adapted for chemical representations, but they often suffer from overfitting to chemical syntax, losing their foundational semantic capabilities. CheMatE addresses this issue by employing a two-stage training procedure: first, it utilizes continued masked language modeling (MLM) on a large-scale corpus of SMILES-annotated scientific documents, followed by a contrastive learning phase using a synthetic dataset of SMILES-text pairs. This approach allows the model to learn bi-semantic representations that capture both molecular structure and contextual information from scientific literature. The model is evaluated on various downstream tasks related to molecular property prediction and scientific language understanding, demonstrating robust performance and transferability across tasks. CheMatE effectively integrates structural and contextual signals, achieving competitive results compared to specialized chemistry models and general-purpose language models.
Methodology
CheMatE utilizes a two-stage training approach: first, it performs continued masked language modeling on a large corpus of SMILES-annotated scientific documents, followed by a contrastive learning phase using synthetic SMILES-text pairs. The model is built on a ModernBERT backbone and supports long-context sequences of up to 8,192 tokens.
Results
CheMatE shows competitive performance across various downstream tasks, including molecular property prediction and scientific language understanding, outperforming both specialized chemistry models and general-purpose language models. The integration of long-context documents and bi-semantic learning contributes to its robust and transferable representations.
Implications
The development of CheMatE has significant implications for the fields of cheminformatics and natural language processing, enabling more effective integration of chemical data with scientific literature. This model can enhance applications in molecular property prediction, reaction planning, and other chemistry-related tasks, providing a versatile foundation for future research.
On the Implicit Flatness Bias of Sharpness-Aware Minimization: A Linear Stability Analysis with Quantitative Hyperparameter Bounds
Optimization
Theory
- Introduces a linear stability framework for analyzing SAM's implicit flatness bias.
- Establishes a quantitative relationship between perturbation radius, batch size, learning rate, and the sharpness of minima.
- Demonstrates the trade-off between perturbation radius and stable training.
- Validates theoretical predictions through extensive experiments on CIFAR-100.
Read more
On the Implicit Flatness Bias of Sharpness-Aware Minimization: A Linear Stability Analysis with Quantitative Hyperparameter Bounds
Summary
This paper investigates the implicit flatness bias of Sharpness-Aware Minimization (SAM), a technique that enhances generalization in deep learning by optimizing for parameters that maintain low loss under local adversarial perturbations. The authors focus on the perturbation radius (Ο), which is crucial for defining the neighborhood in which SAM measures sharpness. They present a linear stability analysis near an interpolating minimum, proving that every linearly stable minimum adheres to a specific bound on the largest Hessian eigenvalue (Ξ»max). This bound quantitatively links Ο with other hyperparameters like batch size and learning rate, revealing that smaller batch sizes, larger learning rates, or larger perturbation radii lead to flatter minima. The paper also highlights a trade-off: while a larger Ο promotes flatness, it must remain local enough to ensure stable training. The authors validate their theoretical findings through experiments involving 900 models on CIFAR-100, demonstrating that increasing Ο correlates with a decrease in the largest Hessian eigenvalue. They introduce Taylor-Locality Controlled SAM (TLC-SAM), which dynamically adjusts Ο based on Taylor approximation errors, achieving improved flatness bias compared to fixed-radius SAM. Overall, this work provides a quantitative framework for understanding hyperparameter interactions in SAM and offers insights for designing more effective SAM variants.
Methodology
The authors employ a linear stability analysis to derive a bound on the largest Hessian eigenvalue of stable minima in the context of mini-batch SAM. They conduct a controlled empirical study involving 900 models trained on CIFAR-100 to validate their theoretical findings, analyzing the effects of varying the perturbation radius, batch size, and learning rate.
Results
The analysis reveals that increasing the perturbation radius (Ο) generally leads to a smaller largest Hessian eigenvalue, indicating flatter minima. However, excessively large values of Ο can negatively impact test accuracy. The introduction of TLC-SAM, which dynamically adjusts Ο, results in a stronger flatness bias while maintaining stable training.
Implications
This research provides a deeper understanding of how hyperparameters interact in SAM, enabling more informed tuning and design of optimization algorithms in deep learning. The findings could lead to improved generalization in various applications of machine learning, particularly in tasks requiring robust model performance under adversarial conditions.
Who Belongs in the Eval Set? A Capability-Taxonomy-Driven Pipeline for Curating Regression Eval Sets in Agent-Extensibility Platforms
NLP
Large Language Models
Efficient ML
- Introduces a capability-taxonomy-driven pipeline for curating regression evaluation sets.
- Addresses the limitations of existing customer-side evaluation frameworks.
- Utilizes a hybrid classifier and Invocation Quality rater to assess query capabilities.
- Implements a rule-based consolidator for decision-making on query inclusion.
Read more
Who Belongs in the Eval Set? A Capability-Taxonomy-Driven Pipeline for Curating Regression Eval Sets in Agent-Extensibility Platforms
Summary
The paper addresses the challenge of curating regression evaluation sets for agent-extensibility platforms, particularly in the context of Microsoft 365 Copilot. It highlights a regression-economics paradox where each customer onboarding brings a unique evaluation set, but the platform must maintain a limited regression set due to release constraints. The authors propose a novel capability-taxonomy-driven pipeline that processes incoming customer evaluation sets, categorizing queries based on a predefined capability taxonomy of approximately 30 capabilities. The pipeline consists of three main components: a classifier that determines the capabilities exercised by each query, an Invocation Quality (IQ) rater that assesses how well queries cover capabilities, and a consolidator that makes decisions on query inclusion based on coverage and quality. This approach allows for a dynamic and efficient curation of regression sets, ensuring that the evaluation process remains robust and relevant as new queries are introduced. The methodology is designed to be taxonomy-agnostic, making it applicable to various regression evaluation scenarios.
Methodology
The methodology involves a three-component pipeline: a classifier for determining capabilities of queries, an Invocation Quality (IQ) rater for scoring query effectiveness, and a consolidator that evaluates incoming queries against the existing regression set to make decisions on their inclusion. The pipeline operates on a capability taxonomy and is designed to continuously adapt to new customer evaluation sets.
Results
The proposed pipeline effectively curates a regression evaluation set that maximizes capability coverage while adhering to strict query limits. It allows for the identification of superior queries that can replace existing ones, thus maintaining the quality of the regression set over time.
Implications
This work has significant implications for the development of agent-extensibility platforms, enabling more efficient evaluation processes and better resource management. It can be applied to various domains requiring dynamic evaluation set curation, improving the robustness of machine learning models in real-world applications.
CausalOPD: First-Wrong-Step Supervision for Distilling Causal Chain Reasoning
NLP
Large Language Models
Reinforcement Learning
- CausalOPD introduces a curriculum online process distillation framework for causal reasoning.
- The method focuses on identifying the first wrong step in reasoning chains to correct errors effectively.
- Improvements in average path correctness by 23.4 percentage points over standard methods.
- Significant reduction in right-label-wrong-reasoning (RLWR) rate from 15.7% to 4.4%.
Read more
CausalOPD: First-Wrong-Step Supervision for Distilling Causal Chain Reasoning
Summary
The paper introduces CausalOPD, a novel framework for distilling causal chain reasoning from large language models (LLMs) into smaller, locally deployable models. The authors identify that traditional trajectory-imitation methods fail to correct early-stage errors that propagate through a causal reasoning chain. CausalOPD addresses this by employing a curriculum online process distillation approach, which begins with knowledge-augmented teacher trajectories grounded in domain-specific causal rules. The framework focuses on identifying the first wrong step in the reasoning process, allowing for targeted reinforcement learning that corrects specific failures rather than re-optimizing entire responses. The training progresses through a causal-stage curriculum, enhancing the model's ability to identify evidence, infer mechanisms, and attribute conclusions accurately. The results demonstrate significant improvements in path correctness and reductions in incorrect reasoning rates across three distinct domains, showcasing the effectiveness of the proposed method in enhancing causal reasoning capabilities in smaller models.
Methodology
CausalOPD employs a two-phase approach: first, it initializes the student model with reasoning trajectories from a knowledge-augmented teacher. Then, it generates on-policy trajectories where the teacher identifies the first wrong step. This targeted error correction is facilitated through short-horizon reinforcement learning, progressing through a curriculum that aligns with the causal dependencies of reasoning.
Results
CausalOPD achieved an average path correctness improvement of 23.4 percentage points compared to sequence-level OPD and reduced the RLWR rate from 15.7% to 4.4%. The domain-specific 8B student models surpassed proprietary references in all evaluated domains.
Implications
The findings suggest that CausalOPD can enhance the deployment of smaller models in critical reasoning tasks such as clinical diagnosis and legal judgment, where accurate causal reasoning is essential. This approach may lead to more reliable AI systems capable of reasoning in complex, domain-specific scenarios.
Latent-Regime Bias Auditing for Volatility Forecasting
Time Series
- Volatility forecasting should be evaluated as a conditional reliability problem rather than solely through aggregate accuracy metrics.
- The proposed audit framework identifies hidden regime biases and tail underprediction in volatility forecasts.
- Empirical results show that models with competitive aggregate accuracy can still fail in specific market regimes.
- The framework allows for a nuanced understanding of forecast reliability across different economic conditions.
Read more
Latent-Regime Bias Auditing for Volatility Forecasting
Summary
This paper addresses the limitations of traditional aggregate accuracy metrics like RMSE and MAE in evaluating volatility forecasts, particularly in the context of risk management. The authors propose a model-agnostic audit framework that assesses the reliability of volatility forecasts across different latent market regimes. By learning time-series representations of market-state windows and clustering them into regimes using training data, the framework allows for an evaluation of forecast behavior that highlights regime-conditional bias and tail underprediction. The empirical application of this framework to daily volatility forecasting for cryptocurrency and ETF assets reveals that models with high aggregate accuracy can still exhibit significant regime-specific biases and severe tail underprediction. The findings advocate for a shift in forecast evaluation from average accuracy to a focus on the conditions under which forecasts fail, emphasizing the importance of understanding where and how forecasts become unreliable.
Methodology
The authors developed a model-agnostic audit framework that learns time-series representations of market states, clusters them into latent regimes, and evaluates forecast reliability based on regime-conditional metrics. This involves asset-normalization, adversarial representation learning, and alignment of regimes by an economic stress score.
Results
The audit revealed that many models, despite showing strong average performance, had significant biases in specific market regimes. The analysis indicated that aggregate accuracy metrics can obscure critical failures in forecast reliability, particularly during periods of economic stress.
Implications
The findings suggest that volatility forecasting models need to be evaluated not just on average performance but also on their reliability across different market conditions. This has important implications for risk management practices, where understanding the conditions under which forecasts fail is crucial for making informed decisions.
Beyond Gene Reconstruction: Learning Cell Representations through Complementary Transcriptomic Views
Graph Learning
- Introduces CoCoS, a contrastive pretraining framework for single-cell transcriptomics.
- Addresses the gap in existing models that focus on gene-level reconstruction rather than whole-cell representation.
- Implements co-expression-guided gene partitioning and expression-aware contrast-set construction to enhance model training.
- Demonstrates superior performance in downstream tasks such as cell-type annotation and gene regulatory network inference.
Read more
Beyond Gene Reconstruction: Learning Cell Representations through Complementary Transcriptomic Views
Summary
This paper addresses the limitations of existing single-cell transcriptomic models that primarily focus on reconstructing masked gene expression values, which do not optimize for whole-cell representations necessary for downstream tasks. The authors propose a novel contrastive pretraining framework, CoCoS, that learns cell representations through complementary transcriptomic views. The framework introduces three key adaptations: co-expression-guided gene partitioning, expression-aware contrast-set construction, and competence-gated contrastive onset. CoCoS constructs two complementary views of each cell by partitioning genes based on co-expression, ensuring that the identity of the cell is preserved while exposing complementary biological evidence. To avoid shortcuts based on gene identity during negative-set construction, the method employs hard negatives generated by permuting expression values while keeping gene identities unchanged. The competence-gated approach gradually introduces contrastive supervision after the model has learned gene-level dependencies through masked expression modeling. Experiments demonstrate that CoCoS achieves competitive performance in cell-type annotation and gene regulatory network inference, outperforming existing methods in several metrics, particularly in gene regulatory network evaluations.
Methodology
The methodology involves constructing two complementary views of each cell based on co-expression structures, creating hard negatives to prevent model shortcuts, and gradually introducing contrastive learning after initial gene-level dependency training. This is achieved through specific adaptations tailored to the challenges of single-cell transcriptomic data.
Results
The proposed CoCoS framework shows competitive transfer performance across various downstream tasks, achieving the highest mean AUROC and AUPRC in gene regulatory network evaluations compared to other methods. The results indicate that the complementary-view contrastive learning approach is effective for single-cell pretraining.
Implications
The findings suggest that enhancing single-cell transcriptomic models with complementary views can significantly improve their performance in biological tasks, potentially leading to better insights in cell-type classification and gene regulatory networks. This approach may also inspire future research in other domains requiring representation learning from complex data.
Trajectory inference via Acceleration Matching
Time Series
Generative Models
Theory
- Introduction of Acceleration Matching (AM) as a novel algorithm for trajectory inference.
- AM avoids the computational burdens of existing methods by not requiring simulation during training.
- The algorithm generates smooth trajectories by targeting an explicit acceleration field.
- Numerical experiments show AM's competitive performance against existing algorithms.
Read more
Trajectory inference via Acceleration Matching
Summary
This paper addresses the problem of trajectory inference, which involves generating smooth trajectories from unpaired snapshots of observations taken at discrete time points. Existing methods often face computational challenges due to reliance on preprocessing for smoothness or simulation-based training objectives. The authors introduce a novel algorithm called Acceleration Matching (AM), which lifts the interpolation problem to phase space and regresses onto an explicit conditional acceleration field. This approach generates random, smooth trajectories that align with the prescribed marginal distributions while only requiring positional data and avoiding trajectory simulation during training. The authors provide numerical evidence demonstrating that AM is competitive with or superior to existing algorithms across several benchmark problems, showcasing its effectiveness in generating smoother trajectories compared to traditional methods.
Methodology
The methodology involves lifting the trajectory inference problem to phase space and defining a stochastic differential equation (SDE) that incorporates an explicit conditional acceleration field. This allows for the generation of smooth trajectories that meet the marginal constraints without the need for trajectory simulation or extensive preprocessing.
Results
The results indicate that the Acceleration Matching algorithm produces smoother trajectories compared to existing methods, such as piecewise flow matching. The numerical experiments demonstrate that AM is either competitive or superior to current algorithms on standard datasets, confirming its effectiveness in trajectory inference.
Implications
The implications of this work extend to various fields that require trajectory inference from unpaired observational data, such as astronomy, economics, and biological data analysis. The efficiency and effectiveness of AM could lead to advancements in data-driven research methodologies in these domains.
Noise-Aware Shrinkage for Differentially Private Zeroth-Order Fine-Tuning of Large Language Models
NLP
Large Language Models
Optimization
- Introduces SAGE, a noise-aware shrinkage method for adaptive update scaling in DP-ZO.
- Demonstrates that fixed-scale updates do not account for the non-stationary reliability of privatized gradient estimates.
- Theoretical analysis shows SAGE effectively reduces update-risk while preserving useful descent.
- Empirical results indicate SAGE outperforms existing methods under identical privacy constraints.
Read more
Noise-Aware Shrinkage for Differentially Private Zeroth-Order Fine-Tuning of Large Language Models
Summary
This paper addresses the challenges of differentially private zeroth-order optimization (DP-ZO) for fine-tuning large language models (LLMs) by introducing a novel method called SAGE (Shrinkage for Aggregated Gradient Estimates). Existing DP-ZO methods often apply a fixed scale to model updates, which can lead to excessive weighting of noise-dominated updates, thereby degrading model performance. SAGE adapts the scaling of privatized estimates based on their estimated signal quality, effectively reducing the influence of noise during training. The method involves estimating the underlying signal energy by subtracting known Gaussian noise variance from observed moments and stabilizing this estimate through temporal tracking. The theoretical analysis demonstrates that SAGE can suppress the quadratic update-risk term more effectively than the linear descent term, thus preserving useful updates while mitigating noise impact. Experimental results on various models, including RoBERTa-large and OPT, show that SAGE significantly outperforms existing baselines under the same privacy budgets, maintaining the memory efficiency characteristic of DP-ZO.
Methodology
The methodology involves a noise-aware shrinkage approach that estimates the signal energy of privatized gradient scalar estimates by removing the known Gaussian noise floor. This is achieved through temporal tracking and the computation of a bounded shrinkage factor, which is then applied to rescale the estimates adaptively. The approach is purely post-processing, requiring no additional privacy budget or model queries.
Results
The experiments conducted on RoBERTa-large and OPT models across various text classification and natural language understanding tasks show that SAGE consistently improves the utility of private fine-tuning compared to existing aggregation-based DP-ZO methods, all while adhering to the same privacy budgets.
Implications
The findings suggest that SAGE can enhance the performance of differentially private fine-tuning in large language models, making it a valuable tool for applications requiring privacy-preserving machine learning. This could be particularly beneficial in sensitive domains such as healthcare, finance, and personalized services where data privacy is paramount.
Measuring in-context algorithmic reasoning in language models against an exact Bayes-optimal standard
NLP
Large Language Models
Theory
- Introduction of F-ICL benchmark for evaluating algorithmic reasoning in LLMs.
- Exhaustive enumeration of programs allows for the computation of a Bayes-optimal standard.
- Most evaluated models perform worse than a simple keystroke reference despite high accuracy.
- The study reveals a significant gap between model performance and the Bayes-optimal standard.
Read more
Measuring in-context algorithmic reasoning in language models against an exact Bayes-optimal standard
Summary
This paper addresses the challenge of evaluating whether large language models (LLMs) perform genuine algorithmic reasoning or merely pattern completion. The authors introduce F-ICL, an in-context-learning benchmark that provides a ground truth for inductive inference using a Turing-complete machine, referred to as F. They exhaustively enumerate all programs of a specified length and compute the Bayes-optimal posterior under a bounded universal prior. The benchmark allows for a direct comparison of LLMs' predictive distributions against this optimal standard. The study evaluates 37 open models and frontier systems, revealing that while models can achieve up to 92% accuracy on queries, most perform worse than a simple keystroke reference. The findings indicate that many models exhibit behavior that is closer to pattern matching rather than true algorithmic reasoning, with a significant number of transitions from solved to unsolved examples. The paper concludes by releasing F-ICL as an open and reproducible benchmark, providing a new tool for assessing the reasoning capabilities of LLMs.
Methodology
The authors developed the F-ICL benchmark by exhaustively enumerating all programs of a specified length using a Turing-complete machine. They computed the Bayes-optimal posterior in closed form under a bounded universal prior, allowing for a direct comparison of LLMs' outputs against this optimal standard. The benchmark includes three task families and employs a difficulty-aware scoring system to evaluate model performance.
Results
The evaluation of 37 models showed that while they achieved up to 92% accuracy, 45 out of 46 models had distributions that diverged from the Bayes-optimal standard. The study documented 6,545 transitions from solved to unsolved examples, indicating that model performance does not necessarily correlate with accuracy. The gap between model outputs and the optimal standard was not predicted by model scale or training, highlighting the limitations of current LLMs in achieving true algorithmic reasoning.
Implications
The findings suggest that many current LLMs may not be capable of genuine algorithmic reasoning, which has implications for their deployment in tasks requiring robust inference. The F-ICL benchmark provides a new tool for researchers to assess and improve the reasoning capabilities of language models.
Sharp Root Anti-Concentration via Projective Incidence and Ordered Root Laws
Theory
Optimization
Graph Learning
- Establishes a dimension-free characterization of interval-hitting constants for piecewise-Lipschitz functions.
- Removes the previous βN loss in the context of cube-supported coefficients.
- Provides necessary and sufficient conditions for finite interval-hitting constants for monic polynomials.
- Introduces coefficient-based tests for dependent and singular laws.
Read more
Sharp Root Anti-Concentration via Projective Incidence and Ordered Root Laws
Summary
This paper addresses the one-dimensional local root anti-concentration problems in the context of online optimization of piecewise-Lipschitz functions, as posed by Balcan, Pegden, and Sharma. The authors establish that for a homogeneous feature curve and coefficients with a bounded density relative to the uniform law on a symmetric convex body K, the worst-case interval-hitting constant is proportional to a section-averaged projective incidence speed. For cube-supported coefficients, this speed aligns with the projective Lipschitz constant, leading to a sharp, dimension-free characterization that eliminates the previous βN loss. The paper also demonstrates that for monic degree-d polynomials under arbitrary coefficient laws, the interval-hitting constant is finite if and only if the ordered real-root laws have bounded densities, with a sharp factor-d comparison. The authors provide coefficient-based tests for dependent and singular laws and a two-chart bound as a global sufficient condition. The findings are applied to two graph-learning models, resulting in expected regret bounds for a cost-sensitive Gaussian-RBF harmonic classifier and a polynomial-kernel model, showcasing the practical implications of their theoretical results.
Methodology
The authors utilize a combination of projective incidence theory and ordered root laws to derive their results. They analyze the density of coefficients and their impact on the interval-hitting constants, employing geometric area and coarea formulas to establish necessary conditions for anti-concentration. The paper also includes a dispersion framework that integrates local probability estimates with boundary complexity to derive uniform dispersion and regret bounds.
Results
The main results include a sharp characterization of the worst-case interval-hitting constant, which is dimension-free and eliminates the previous βN loss. The authors prove that the interval-hitting constant is finite if and only if the ordered real-root pushforwards have uniformly bounded densities. They also derive expected regret bounds for two graph-learning models, demonstrating the practical utility of their theoretical findings.
Implications
The findings have significant implications for online optimization algorithms, particularly in scenarios where piecewise-Lipschitz functions are involved. The results can enhance the performance of graph-learning models, providing more robust methods for classification tasks in machine learning applications.
Shorter Reasoning, Earlier Answers? An Evaluation of Reasoning Interfaces
NLP
Large Language Models
Efficient ML
- Shortening reasoning traces can reduce costs and latency in LLMs.
- A numeric/concision prompt effectively shortens reasoning without consistent accuracy gains.
- Concise instructions can improve accuracy, particularly under token constraints.
- Lower-effort reasoning often results in higher accuracy compared to high-effort reasoning.
Read more
Shorter Reasoning, Earlier Answers? An Evaluation of Reasoning Interfaces
Summary
This paper investigates the reasoning efficiency of large language models (LLMs) by evaluating different reasoning interfaces that can potentially shorten reasoning time while maintaining accuracy. The authors focus on the trade-off between reasoning length and answer accuracy, particularly under constraints such as deadlines. They conducted experiments using paired runs of the Qwen3-14B and gpt-oss models on two question sets: GPQA Diamond and MMLU-Pro. The study introduces a numeric/concision prompt that limits token usage and assesses the impact of concise instructions on model performance. Results indicate that the Qwen prompt can reduce reasoning traces by 12-17%, with mixed effects on accuracy. The concise instruction improved accuracy on MMLU-Pro by 3.8 percentage points at 512 tokens, while lower-effort reasoning in gpt-oss yielded significantly higher accuracy compared to high-effort reasoning. The findings suggest that shorter reasoning can be beneficial, but the quality of answers may vary, especially when models stop early. The authors advocate for more nuanced evaluation metrics that consider both the timing of completions and the quality of unfinished runs.
Methodology
The authors conducted experiments comparing paired runs of LLMs at matched reasoning horizons. They utilized a numeric/concision prompt for Qwen3-14B and assessed the trained effort settings of gpt-oss models. The study involved analyzing reasoning traces, accuracy at various token limits, and the probability assigned to correct answers.
Results
The Qwen3-14B model showed a 12-17% reduction in reasoning traces with mixed accuracy outcomes. The concise instruction improved MMLU-Pro accuracy by 3.8 percentage points at 512 tokens. For gpt-oss, lower-effort reasoning produced answers that were 14.5-26.3 points more accurate than high-effort answers, primarily due to earlier stopping.
Implications
The findings suggest that optimizing reasoning length can enhance the efficiency of LLMs, particularly in time-sensitive applications. The study highlights the need for improved evaluation protocols that differentiate between the quality of completed and unfinished reasoning, potentially guiding future model training and deployment strategies.
Tight Worst-Case Bounds for the Smallest Eigenvalue of ReLU NTK Gram Matrices
Theory
Optimization
- Establishes a dimension-free lower bound for the smallest eigenvalue of ReLU NTK Gram matrices.
- Proves that Ξ»min(H) is proportional to projective separation βΒ± divided by βlog n.
- Constructs worst-case examples that match the upper bound, demonstrating tightness of the results.
- Highlights the importance of geometric nondegeneracy in neural network optimization.
Read more
Tight Worst-Case Bounds for the Smallest Eigenvalue of ReLU NTK Gram Matrices
Summary
This paper investigates the smallest eigenvalue of the ReLU neural tangent kernel (NTK) Gram matrix, which is critical for understanding the conditioning of linearized training problems in neural networks. The author establishes a dimension-free lower bound for the smallest eigenvalue, Ξ»min(H), of the Gram matrix H, showing that Ξ»min(H) is at least proportional to the projective separation βΒ± divided by the square root of the logarithm of the number of unit vectors n. Specifically, the bound is given as Ξ»min(H) = β¦(βΒ±/βlog n). Furthermore, the author constructs worst-case examples that demonstrate the tightness of this bound, achieving an upper bound of Ξ»min(H) = O(βΒ±/βlog n). This indicates that the relationship between Ξ»min(H) and βΒ± is sharp up to universal constants. The results are significant as they provide a clearer understanding of the geometric properties of the data that influence the behavior of neural networks during training.
Methodology
The author employs a combination of theoretical analysis and constructions based on geometric properties of unit vectors. The proof of the lower bound utilizes the positive Hadamard-power expansion of the arcsine kernel, while the upper bound is derived from analyzing balanced binary linear codes embedded in spherical caps. The results are supported by Fourier diagonalization techniques and probabilistic arguments related to Gaussian distributions.
Results
The paper presents a universal lower bound for the smallest eigenvalue of the ReLU NTK Gram matrix as Ξ»min(H) = β¦(βΒ±/βlog n) for any projectively separated dataset. Additionally, it constructs specific families of unit vectors that achieve the upper bound Ξ»min(H) = O(βΒ±/βlog n), confirming that the established bounds are tight up to universal constants.
Implications
The findings have significant implications for the optimization of neural networks, particularly in understanding how the geometric arrangement of data affects training dynamics and convergence rates. This work can inform the design of neural architectures and training algorithms by emphasizing the importance of data separation in achieving better conditioning of the NTK.
Neural operator learning for collision-aware trajectory planning of spacecraft swarms
Robotics
Optimization
- Introduces a permutation-equivariant neural operator for collision-aware trajectory planning.
- Trained without optimal trajectory labels using self-supervised physics objectives.
- Achieves zero-shot generalization to swarms of 1,000 spacecraft in dense debris fields.
- Matches the accuracy of traditional optimal-control methods while improving swarm proximity.
Read more
Neural operator learning for collision-aware trajectory planning of spacecraft swarms
Summary
The paper addresses the challenge of trajectory planning for autonomous spacecraft swarms in congested orbits, where classical optimization methods struggle due to the exponential growth of collision constraints with swarm size. The authors propose a novel permutation-equivariant neural operator that efficiently maps distributions of spacecraft, targets, and debris to collision-aware trajectories in a single forward pass. This operator is trained using self-supervised physics objectives, avoiding the need for optimal trajectory labels, and incorporates adversarial threats to enhance training. The method demonstrates remarkable generalization capabilities, achieving zero-shot performance for swarms of up to 1,000 spacecraft amidst over 11,000 debris objects. The proposed approach matches the accuracy of traditional optimal-control solvers while significantly reducing the proximity of spacecraft within the swarm. Overall, this physics-grounded operator learning framework presents a scalable and efficient alternative for trajectory planning in crowded orbital environments.
Methodology
The authors developed a two-stage planner that first employs a neural operator to learn distribution-to-trajectory mappings for spacecraft under Keplerian dynamics. This operator processes initial distributions of spacecraft, targets, and debris to generate trajectories in one forward pass. A subsequent batched GaussβNewton method refines these trajectories to ensure they adhere to exact orbital dynamics. The training leverages self-supervised learning techniques and adversarial examples to enhance the model's robustness against potential collisions.
Results
The proposed method was evaluated using real ephemeris data, demonstrating its ability to produce low-cost, collision-aware trajectories that are dynamically consistent. The neural operator successfully generalizes to larger swarms without requiring retraining, maintaining performance across varying debris densities and swarm sizes.
Implications
This research has significant implications for the future of autonomous spacecraft operations, particularly in managing the increasing congestion in Low Earth Orbit (LEO). The ability to efficiently plan trajectories for large swarms could enhance mission safety, reduce fuel consumption, and streamline coordination among multiple spacecraft.
Adaptive Quantum Physics-Informed Neural Networks for Differential Equations with Applications to Fluid Dynamics
Theory
Optimization
Efficient ML
- Introduction of adaptive collocation point sampling to improve accuracy in QPINNs.
- Identification of optimization as a critical bottleneck in QPINNs, alongside expressivity.
- Development of a trainable loss-weighting scheme for balanced training.
- Demonstration of at least 60% improvement in solution accuracy for specific fluid dynamics problems.
Read more
Adaptive Quantum Physics-Informed Neural Networks for Differential Equations with Applications to Fluid Dynamics
Summary
This paper presents a novel hybrid quantum-classical framework that enhances Quantum Physics-Informed Neural Networks (QPINNs) for solving nonlinear partial differential equations (PDEs), particularly in fluid dynamics. The authors introduce adaptive collocation point sampling and loss-aware attention mechanisms to improve accuracy and efficiency in high-dimensional and multiscale systems. By dynamically prioritizing sampling points based on the magnitude of PDE residuals and solution gradients, the proposed method addresses the spectral bias commonly found in traditional PINNs. The study reveals that optimization challenges, rather than just expressivity limitations of quantum circuits, significantly impact the performance of QPINNs. A trainable loss-weighting scheme is also proposed to balance the contributions from physics residuals, boundary conditions, and data fidelity during training. The integration of these strategies with quantum computing techniques results in at least a 60% improvement in solution accuracy for benchmark fluid flows and reaction-diffusion systems. The findings suggest that simply increasing model expressivity is insufficient for effectively resolving complex PDEs, emphasizing the need for robust classical optimization strategies alongside quantum enhancements.
Methodology
The authors employed a hybrid quantum-classical approach, integrating adaptive collocation point sampling and loss-aware attention mechanisms into QPINNs. They utilized a trainable loss-weighting scheme to balance contributions from various training components, and conducted experiments on benchmark fluid flows and reaction-diffusion systems to evaluate performance improvements.
Results
The proposed framework achieved a minimum of 60% improvement in solution accuracy for benchmark fluid dynamics and reaction-diffusion systems. The study demonstrated that addressing optimization challenges is crucial for enhancing the performance of QPINNs, alongside improving quantum expressivity.
Implications
This research provides a scalable pathway for integrating quantum computing with scientific machine learning, particularly in solving complex PDEs in fluid dynamics and other fields. It opens avenues for further exploration of quantum-compatible solvers and optimization strategies in high-dimensional and multiscale systems.
To Describe or Construct Statistical Learning Models Using the Category-theoretical Language
Theory
- Utilizes category theory to describe statistical learning models.
- Introduces a unified descriptive approach for constructing complex models.
- Highlights the construction of the Transformer model as a key example.
- Aims to engage researchers from diverse fields in statistical learning research.
Read more
To Describe or Construct Statistical Learning Models Using the Category-theoretical Language
Summary
This paper explores the intersection of statistical learning and category theory, aiming to provide a novel perspective on understanding and constructing statistical learning models. The author summarizes classical statistical learning models and algorithms, presenting them through a categorical lens to attract researchers from various fields, including mathematics. The paper emphasizes two main contributions: the use of categorical language to describe machine learning models and the introduction of a unified approach for constructing complex models, such as language models. A significant example discussed is the construction of the Transformer model using a localization trick. The paper serves as an introductory resource for those unfamiliar with the intricacies of statistical learning, while also proposing a framework that could facilitate further research and collaboration across disciplines.
Methodology
The paper employs category theory as a descriptive language to analyze and construct statistical learning models. It introduces specific notations and definitions to facilitate understanding and communication of complex concepts in machine learning.
Results
The paper does not present empirical results but rather theoretical insights and frameworks that can be applied to existing models. It successfully demonstrates how categorical language can simplify the description of various statistical learning models and proposes a method for constructing complex models.
Implications
The findings suggest that category theory can provide a powerful framework for understanding and developing statistical learning models, potentially leading to new insights and advancements in machine learning. This approach may encourage interdisciplinary collaboration and innovation in statistical learning research.
Scaling an Autoregressive Transformer for Single-Cell Generation
Generative Models
- Introduction of a self-supervised autoregressive transformer for generating single-cell gene expression vectors.
- Identification of a joint two-exponent scaling law for model size and training data size in single-cell models.
- Demonstration of the correlation between pretraining loss and downstream evaluation metrics.
- Potential applications for fine-tuning the model for perturbation response prediction.
Read more
Scaling an Autoregressive Transformer for Single-Cell Generation
Summary
This paper presents a novel approach to generating single-cell gene expression vectors using a self-supervised autoregressive transformer model. The authors aim to produce additional gene expression vectors for a specific cell type by leveraging existing data. The proposed model combines a causal transformer architecture with a learned quantized Variational Autoencoder (VAE) tokenizer, trained using a cross-entropy loss. The study evaluates the biological fidelity of the generated vectors and investigates the scaling behavior of the pretraining loss in relation to model size and training data volume. Notably, the authors identify a joint two-exponent scaling law for the model, which has not been previously reported in the context of single-cell foundation models. The findings suggest that the pretraining loss correlates with downstream evaluation metrics, indicating the model's effectiveness in generating realistic gene expression profiles. Additionally, the paper discusses the potential for fine-tuning the pretrained model for predicting responses to perturbations, highlighting its applicability in biological research and in-silico experimentation.
Methodology
The authors developed a causal transformer model paired with a learned quantized VAE tokenizer. The model was trained using a cross-entropy loss to generate synthetic gene expression vectors based on existing data from a specific cell type. The scaling properties of the model were analyzed by varying the number of parameters and the amount of training data, leading to the discovery of a joint scaling law.
Results
The model successfully generated gene expression vectors that closely matched the ground truth distribution for the specified cell type. The analysis revealed a joint two-exponent scaling law for the pretraining loss, indicating that both model size and training data size significantly influence performance. The results also showed that the pretraining loss is predictive of the quality of generated vectors.
Implications
This research has significant implications for biological research, particularly in generating synthetic gene expression data, which can augment limited datasets and facilitate in-silico experimentation. The findings on scaling laws may inform future developments in single-cell foundation models and their applications in various biological tasks.
Fused Bayesian Flow Networks for Dual-Target Molecular Design
Generative Models
- Introduction of FusedBFN, a fused Bayesian flow network for dual-target molecular design.
- Utilizes a product-of-experts formulation to integrate information from two target proteins.
- Leverages a pretrained target-aware BFN to address the scarcity of dual-target structural data.
- Implements a chemically aware alignment strategy for better integration of structural features.
Read more
Fused Bayesian Flow Networks for Dual-Target Molecular Design
Summary
This paper presents FusedBFN, a novel fused Bayesian flow network designed for dual-target molecular design, which aims to generate 3D molecules that can interact with two target proteins simultaneously. The authors argue that existing dual-target approaches either focus on sequence generation or inadequately incorporate structural information from both targets, limiting their effectiveness. FusedBFN addresses these limitations by formulating dual-target generation as a distribution fusion problem in a unified continuous parameter space, utilizing a product-of-experts approach to integrate dual-target information throughout the generative process. To overcome the scarcity of dual-target structural data, the model leverages a pretrained target-aware Bayesian flow network as a backbone. The authors also introduce a chemically aware prior-based alignment method and a prior-free pocket alignment strategy to construct aligned dual-target contexts. Extensive experiments demonstrate that FusedBFN effectively generates molecules with strong binding affinities towards both targets while maintaining favorable molecular properties, marking a significant advancement in dual-target drug design.
Methodology
The methodology involves formulating dual-target molecular generation as a distribution fusion problem using a product-of-experts approach. The model updates parameters in a continuous parameter space based on dual-target contexts. A pretrained target-aware Bayesian flow network serves as the backbone, and two alignment strategies (chemically aware and prior-free) are introduced to construct aligned dual-target contexts.
Results
The experiments conducted on a dual-target benchmark indicate that FusedBFN successfully generates molecules with high binding affinity to both targets while preserving desirable molecular properties, outperforming existing methods that either focus on single-target designs or inadequately integrate dual-target information.
Implications
The proposed FusedBFN framework has significant implications for drug discovery, particularly in developing polypharmacological compounds that can effectively target multiple proteins, potentially leading to more effective treatments for complex diseases.
Sphere Retraction Normalizations
Theory
Optimization
Large Language Models
- Unification of residual connections and GeoNorm as retraction-based updates.
- Introduction of two new retraction methods: Proj-SpheretNorm and Cay-SpheretNorm.
- Development of a generalized p-SpheretNorm that encompasses both new methods and GeoNorm.
- Empirical results show Proj-SpheretNorm achieves superior performance on nanoGPT models.
Read more
Sphere Retraction Normalizations
Summary
This paper addresses the challenge of training deep neural networks stably by introducing Sphere Retraction Normalizations (SpheretNorms), which unify residual connections and Geodesic Normalization (GeoNorm) within a single framework on a Riemannian manifold. The authors demonstrate that the exponential map, traditionally used in GeoNorm, is just one of many retraction maps applicable on the hypersphere. They propose three specific SpheretNorms: Proj-SpheretNorm, Cay-SpheretNorm, and a generalized p-SpheretNorm, which allows for a flexible adjustment of the rotation angle based on a tunable parameter p. The paper shows that Proj-SpheretNorm outperforms existing lightweight deep connection schemes, achieving the lowest pretraining and validation losses on the nanoGPT model. The findings suggest that the choice of retraction can significantly influence the performance of deep neural networks, indicating that the exponential map may not be the optimal choice for spherical residual streams.
Methodology
The authors utilize a geometric framework to analyze residual connections on a hypersphere, employing the Gram-Schmidt process for orthogonalization and exploring various retraction maps. They derive Proj-SpheretNorm and Cay-SpheretNorm from the metric projection and Cayley retractions, respectively, and generalize these into a single parameterized family of normalizations, p-SpheretNorm.
Results
The experiments conducted on nanoGPT models demonstrate that Proj-SpheretNorm consistently yields the lowest pretraining and validation losses compared to existing methods. It also shows high accuracy on downstream tasks, indicating its effectiveness as a normalization technique.
Implications
The findings suggest that adopting different retraction methods can enhance the stability and performance of deep learning models, particularly in architectures like Transformers. This could lead to more efficient training processes and better model performance across various applications.
ATFlash: Per-RoPE-Wavelength Attention Windows for Compute/Memory-Efficient LLM Inference
Large Language Models
Efficient ML
- Introduction of a per-RoPE-wavelength distance window for efficient LLM inference.
- Achieves 37-48% pruning of query-key inner-product terms with high accuracy retention.
- Maintains performance across various long-context benchmarks.
- Implemented with minimal modifications to existing FlashAttention kernels.
Read more
ATFlash: Per-RoPE-Wavelength Attention Windows for Compute/Memory-Efficient LLM Inference
Summary
The paper introduces ATFlash, a novel method for improving the efficiency of large language model (LLM) inference by utilizing a per-RoPE-wavelength distance window. This method leverages the structure of rotary position embeddings (RoPE) to prune query-key inner-product terms based on their wavelength, allowing for significant reductions in computation without sacrificing performance. Unlike traditional sliding windows, ATFlash ensures that all tokens remain accessible through low-frequency pairs, maintaining a high match rate and low Kullback-Leibler divergence. The authors demonstrate that ATFlash can prune 37-48% of inner-product terms while preserving 96-98% of the top-1 match rate and achieving substantial speedups in inference on various models, including Qwen2.5 and Llama. The method is implemented with minimal changes to existing FlashAttention kernels, making it portable and efficient for practical applications.
Methodology
The proposed method, ATFlash, employs a per-RoPE-wavelength distance window that prunes query-key inner-product terms based on their wavelength. This approach is implemented as a slice of the reduction axis in online softmax, ensuring that no tokens are discarded. The reduction rate is derived in closed form, showing logarithmic dependence on sequence length, and the method is tested across various models and benchmarks to validate its effectiveness.
Results
ATFlash successfully prunes 37-48% of inner-product terms while maintaining a top-1 match rate of 96-98% and a mean output-distribution KL divergence at the 10^-3 level on LongBench-v2 contexts. The method shows significant speedups, achieving up to 1.31Γ faster inference on Qwen2.5-7B-1M with 57% of inner-product terms pruned. Benchmark scores are broadly preserved, confirming the method's robustness.
Implications
The ATFlash method has the potential to enhance the efficiency of LLM inference in real-world applications, particularly in scenarios requiring long-context processing. Its compatibility with existing FlashAttention implementations allows for easy integration into current systems, making it a valuable tool for developers and researchers working with large language models.
AlphaG-OPD: Reliability-Gated Sibling Counterfactuals for On-Policy Distillation in Symbolic Alpha Factor Discovery
Reinforcement Learning
Generative Models
Optimization
- Introduces AlphaG-OPD, a structural on-policy distillation framework for symbolic alpha factor discovery.
- Separates decision-making into three components: where to teach, what to teach, and how to consolidate teaching.
- Employs reliability gating to ensure only reliable sibling actions are taught, enhancing learning efficiency.
- Maintains the global objective of Trajectory Balance while providing local action guidance.
Read more
AlphaG-OPD: Reliability-Gated Sibling Counterfactuals for On-Policy Distillation in Symbolic Alpha Factor Discovery
Summary
The paper presents AlphaG-OPD, a novel framework for symbolic alpha factor discovery that enhances the process of on-policy distillation (OPD) using reliability-gated sibling counterfactuals. Traditional symbolic alpha factor discovery methods score expressions only after completion, lacking direct feedback on the structural decisions made during the process. AlphaG-OPD addresses this gap by introducing a structured approach that separates the decision-making process into three components: (1) identifying where to teach by exposing grammar-valid sibling actions at partial abstract-syntax-tree (AST) states; (2) determining the reliability of what to teach by evaluating sibling actions under shared completions and applying a KL-bounded target based on agreement and confidence; and (3) managing how long and how strongly to teach through bounded replay and score-indexed expiry. The framework maintains the global objective of Trajectory Balance while providing local action guidance, thus improving the learning efficiency of GFlowNets in symbolic alpha factor discovery. The method was tested across various stock market indices, demonstrating strong performance and robustness across multiple random seeds.
Methodology
AlphaG-OPD employs a structured approach to on-policy distillation by evaluating sibling actions at partial AST states. It uses a reliability-gated mechanism to determine which actions to teach based on agreement among sibling evaluations and confidence bounds. The framework consolidates accepted actions through bounded replay and score-indexed expiry, ensuring efficient use of resources without additional evaluations.
Results
The implementation of AlphaG-OPD showed significant improvements in performance across China's CSI300, CSI500, and CSI1000 indices, as well as the U.S. S&P 500. The method outperformed traditional approaches, demonstrating robustness and effectiveness in various market conditions.
Implications
AlphaG-OPD has potential applications in financial modeling and algorithmic trading, where understanding the impact of structural decisions on performance is crucial. The framework could also be adapted for other domains requiring symbolic reasoning and decision-making under uncertainty.
Stop When Memory Suffices: Evidence-Conditioned Progressive Execution for LLM Agents
Large Language Models
Efficient ML
NLP
- Router-Mem introduces a novel framework for long-horizon agent memory that balances efficiency and answer quality.
- The sufficiency router enables early termination of processing when sufficient evidence is available, reducing unnecessary computation.
- The framework shows strong performance on benchmark datasets, achieving high scores while decreasing average inference time.
- Router-Mem effectively combines retrieval and deeper analysis, optimizing resource allocation based on query needs.
Read more
Stop When Memory Suffices: Evidence-Conditioned Progressive Execution for LLM Agents
Summary
The paper addresses the need for long-term memory mechanisms in large language models (LLMs) to enhance their adaptive intelligence. Existing memory systems either compress historical data for efficient access or conduct extensive searches over broader memory, leading to a trade-off between answer quality and latency. The authors propose Router-Mem, an evidence-conditioned progressive execution framework that optimizes memory usage by first applying a low-cost retrieval prefix to gather evidence. A lightweight sufficiency router then determines if the evidence is sufficient for an early answer, allowing for a single-token decision during inference. If the evidence is inadequate, Router-Mem expands memory blocks and performs deeper analysis. Experiments on AMA-Bench and BEAM demonstrate that Router-Mem achieves significant improvements in answer quality while reducing inference time compared to traditional memory execution methods.
Methodology
Router-Mem employs a two-step process: first, it retrieves evidence using a shared low-cost prefix; second, it uses a sufficiency router to evaluate whether the retrieved evidence is adequate for answering the query. If sufficient, it provides an answer directly; if not, it expands memory blocks for further analysis.
Results
Router-Mem achieved scores of 55.17% and 38.77% on AMA-Bench and BEAM, respectively, while reducing average inference time by 27.3% and 25.5% compared to full memory execution. The framework consistently demonstrated favorable quality-latency trade-offs across various routing thresholds.
Implications
The findings suggest that Router-Mem can significantly enhance the efficiency of LLMs in real-time applications, allowing for quicker responses without compromising the quality of answers. This has potential applications in interactive systems that require adaptive intelligence and long-term memory.
GoT-CD: Graph-of-Thoughts Causal Discovery and the Fragility of Post-hoc Path-Specific Fairness Audits
Graph Learning
Large Language Models
Interpretability
- GoT-CD generates multiple candidate graphs in parallel, enhancing the robustness of causal discovery.
- The method ensures that critical pathways for fairness audits are preserved, addressing a significant gap in existing methodologies.
- GoT-CD outperforms traditional causal discovery methods and LLM-based approaches in terms of structural validity and path-specific fairness.
- The study emphasizes the need for causal discovery methods to be evaluated based on their ability to recover relevant pathways for fairness analysis.
Read more
GoT-CD: Graph-of-Thoughts Causal Discovery and the Fragility of Post-hoc Path-Specific Fairness Audits
Summary
The paper introduces GoT-CD, a novel causal discovery method that leverages the Graph-of-Thoughts framework to enhance the accuracy of causal graphs derived from observational data. Traditional causal discovery methods often fail to ensure that specific pathways relevant for fairness audits are preserved, leading to potential misinterpretations of fairness in predictive models. GoT-CD addresses this by generating multiple candidate graphs in parallel, scoring them based on a deterministic validity function, and merging them under constraints that prevent the introduction of erroneous edges. The authors demonstrate that GoT-CD produces valid directed acyclic graphs (DAGs) across multiple benchmarks, outperforming existing large language model (LLM) methods in terms of structural fidelity and path-specific fairness recovery. The results highlight the importance of evaluating causal discovery methods not just on aggregate structural metrics but also on their ability to retain critical pathways for downstream fairness analyses, particularly in clinical settings where fairness is paramount.
Methodology
GoT-CD employs a Graph-of-Thoughts framework to generate multiple candidate graphs simultaneously. Each graph is scored using a deterministic validity function, and a hard union constraint is applied to prevent the introduction of edges not proposed by any reasoning branch. A post-processing step ensures acyclicity before finalizing the graph. The method is compared against classical and LLM baselines across five benchmarks, with a focus on path-specific fairness audits.
Results
GoT-CD successfully produces valid DAGs on all five benchmarks tested, achieving the highest DAG-valid F1 scores among LLM methods on datasets such as Asia (0.750), Alzheimerβs (0.757), and COVID-Respiratory (0.688). Notably, it accurately recovers a known unfair pathway in the Alzheimerβs benchmark, demonstrating its effectiveness in retaining critical pathways for fairness analysis.
Implications
The findings suggest that causal discovery methods must prioritize the preservation of relevant pathways for fairness audits, particularly in sensitive applications like healthcare. This could lead to more reliable fairness assessments in predictive models, ultimately enhancing trust and accountability in AI systems used for clinical decision support.
Federated generative event models for tokenized electronic health records
Federated Learning
Generative Models
Time Series
- GEM representations are significantly more portable across health systems compared to conventional supervised models.
- Federated learning approaches recover most of the performance of centralized training, especially for data-limited sites.
- The primary challenge is cross-site transfer and adaptation rather than the federated optimization process.
- Centralized multi-site training offers limited improvements over strong local models when large datasets are available.
Read more
Federated generative event models for tokenized electronic health records
Summary
This paper addresses the limitations of electronic health record (EHR) models due to data silos and performance degradation in cross-site applications. The authors propose federated generative event models (GEMs) trained on tokenized EHR data from three independent health systems, totaling 122,251 ICU hospitalizations. The study evaluates the performance of GEMs across various training configurations, including within-site, cross-site, centralized, and federated training. Results indicate that GEMs outperform conventional supervised models in terms of transportability, with lower cross-site penalties in ROC-AUC and PR-AUC metrics. Federated learning approaches (FedAvg and FedAvgM) closely match centralized training performance, particularly benefiting data-limited sites. The findings highlight that the main challenge lies in effectively transferring knowledge across heterogeneous datasets rather than the optimization method used. The authors also introduce 'coreopsis', an open-source framework for federated GEM training, facilitating the integration of tokenized EHR data across institutions.
Methodology
The authors utilized federated learning to train generative event models on tokenized EHR data from three health systems, employing a common data format for harmonization. They conducted evaluations on various clinical prediction tasks using different training configurations to assess model performance.
Results
GEMs achieved the highest mean ROC-AUC scores for both within-site and cross-site evaluations, with cross-site penalties significantly lower than those of conventional models. Federated learning methods approached centralized training performance, with most gains realized within 5-10 communication rounds. The study found that federated models were particularly advantageous for institutions with limited local data.
Implications
The findings suggest that federated GEM training can effectively leverage heterogeneous data from multiple health systems while maintaining patient privacy. This approach has the potential to improve clinical predictions in diverse healthcare settings, ultimately enhancing patient care and outcomes.
Contrast-invariant deep ptychography neural networks
Computer Vision
Optimization
Efficient ML
- Introduces a factorization strategy to decouple object texture from measurement scaling.
- Utilizes a real/imaginary output representation for improved optimization stability.
- Implements a dynamic scaling factorization optimized during test time.
- Demonstrates a 5x reduction in Fourier error across various experimental datasets.
Read more
Contrast-invariant deep ptychography neural networks
Summary
This paper addresses the limitations of ptychography neural networks, particularly their scaling inconsistencies when generalizing to out-of-distribution data. The authors propose a novel factorization strategy that separates the learned object texture from measurement scaling, allowing a single trained network to produce consistent reconstructions across varying illumination conditions. The key innovation is the use of a real and imaginary unit output instead of the traditional amplitude and phase representation, which facilitates a more stable optimization landscape. Additionally, a synthetic object sampling strategy is introduced to minimize phase distribution mismatch between synthetic training data and experimental targets. The proposed improvements lead to significant reductions in Fourier error across multiple experimental datasets, demonstrating the method's effectiveness and versatility.
Methodology
The authors modify the neural network architecture to output normalized rectangular components (real and imaginary) instead of amplitude and phase. They implement dynamic scaling factorization to optimize scaling parameters during testing, use probe-weighted patch stitching for improved reconstruction quality, and apply a synthetic object sampling strategy to address phase distribution mismatches.
Results
The proposed method, PtychoPINN-CI, achieves up to a 5x reduction in Fourier error compared to the previous baseline, PtychoPINN-torch, across five experimental datasets from different beamlines and facilities. The improvements also lead to enhanced qualitative reconstructions of both amplitude and phase.
Implications
This work has significant implications for real-time feedback in ptychographic imaging, enhancing the efficiency of experiments at synchrotrons and X-ray free-electron lasers. The method's adaptability to various neural network architectures suggests broad applicability in other unsupervised learning contexts.
Quantization Effects on Biomedical LLM Reliability
NLP
Large Language Models
- Scoring rule sensitivity can reverse calibration rankings without affecting accuracy.
- Prompt template choice can lead to significant accuracy variations, comparable to model choice.
- INT8 quantization has minimal impact on specialized models, while INT4 effects are heterogeneous.
- Post-hoc temperature scaling can improve calibration but is specific to certain scoring rules.
Read more
Quantization Effects on Biomedical LLM Reliability
Summary
This paper investigates the impact of quantization and prompt design on the reliability of biomedical large language models (LLMs) when used for sentence classification tasks. The authors conduct a controlled evaluation of three variants of the Mistral-7B model on a dataset of 2,000 PubMed RCT sentences, examining the effects of different precision levels (FP16, INT8, INT4) and prompt templates. The study reveals that the choice of scoring rule significantly influences model calibration, with a switch from summed to mean-token log-likelihood scoring reversing the calibration ranking between models while accuracy remains largely unchanged. Additionally, the choice of prompt template can lead to substantial accuracy variations (7-24 percentage points) that can exceed the effects of model choice. The results indicate that INT8 quantization has minimal impact on accuracy for specialized models, while INT4 shows varied effects. The findings emphasize the importance of prompt design and scoring normalization as critical factors in evaluating decoder calibration, highlighting the need for careful experimental design in assessing model reliability.
Methodology
The authors evaluated three Mistral-7B model variants under different quantization levels (FP16, INT8, INT4) and prompt templates. They analyzed the effects of scoring rules on calibration and accuracy using a dataset of 2,000 PubMed RCT sentences.
Results
The study found that switching scoring rules altered calibration rankings significantly, with BioMistral's expected calibration error (ECE) increasing from 0.097 to 0.289 under mean-token scoring. Template choice led to accuracy variations of 7-24 percentage points, and INT8 quantization produced minor accuracy changes (1-2 percentage points) for specialized models, while INT4 effects varied without catastrophic degradation.
Implications
The findings suggest that careful consideration of prompt design and scoring methods is essential for reliable model calibration in biomedical applications. This has implications for deploying LLMs in healthcare, where accurate classifications and well-calibrated confidence estimates are crucial.
Agentic Reinforcement Learning with Self-Distilled Reward Shaping
Reinforcement Learning
Large Language Models
NLP
- ADRS provides a solution to the sparse feedback problem in agentic reinforcement learning by enabling token-level credit assignment.
- The framework incorporates teacher confidence and realized returns to modulate reward shaping effectively.
- Experiments show consistent performance improvements across various interactive benchmarks and RL backbones.
- ADRS maintains skill-free rollouts during inference, enhancing its applicability in real-world scenarios.
Read more
Agentic Reinforcement Learning with Self-Distilled Reward Shaping
Summary
The paper introduces a novel framework called Agentic Reinforcement Learning with Self-Distilled Reward Shaping (ADRS) aimed at enhancing the performance of language model agents in interactive tasks. Traditional reinforcement learning (RL) methods often provide sparse rewards at the trajectory level, making it challenging to assign credit to intermediate decisions. ADRS addresses this limitation by utilizing a training-only privileged setting where a frozen policy rescales tokens from skill-free trajectories based on task-matched procedural skills. The framework tackles three main challenges: score calibration, reliability estimation, and credit integration. It normalizes privileged token scores, modulates them with a Teacher Value Advantage (TVA) gate that relates teacher confidence to realized returns, and incorporates this gated signal into the native RL credit construction. The authors demonstrate that ADRS significantly improves performance on long-horizon tasks across various benchmarks, showing robustness across different RL architectures, reduced-data scenarios, and unseen tasks. The proposed method not only enhances the learning process but also maintains skill-free rollouts during inference, making it a versatile approach for agentic reinforcement learning.
Methodology
ADRS employs a self-distilled reward shaping approach that normalizes teacher scores, estimates their reliability concerning realized returns, and integrates these scores into the reinforcement learning credit path. The framework uses a TVA gate to modulate the influence of teacher scores based on their confidence and the associated returns, ensuring that the feedback is effectively utilized in the learning process.
Results
The experiments conducted across three interactive benchmarks demonstrate that ADRS consistently outperforms existing methods in long-horizon tasks. The improvements are observed across different RL architectures, reduced-data settings, and even in tasks that were not seen during training, indicating the robustness and generalizability of the approach.
Implications
The ADRS framework has significant implications for the development of more effective language model agents capable of performing complex interactive tasks. By improving credit assignment and reward shaping, it can enhance the learning efficiency and effectiveness of RL agents in various applications, including evidence search, navigation, and web interaction.
Subtype Robustness Is Not Just Accuracy: Calibration Under Unseen Subtype Shift
Computer Vision
- Calibration is crucial for evaluating subtype robustness, not just accuracy.
- Models exhibit overconfidence in predictions for unseen subtypes despite accuracy loss.
- Generic image corruption leads to a more significant drop in confidence compared to subtype novelty.
- Recalibration methods do not fully recover lost calibration under unseen subtype conditions.
Read more
Subtype Robustness Is Not Just Accuracy: Calibration Under Unseen Subtype Shift
Summary
This paper investigates the concept of subtype robustness in machine learning, which refers to a model's ability to maintain correct coarse predictions when faced with fine-grained subtypes that were not present during training. The authors highlight that prior research has primarily focused on accuracy as the metric for evaluating subtype robustness, neglecting the importance of calibration. They present a systematic study across multiple datasets (ImageNet, BREEDS, iNaturalist, and CIFAR-100) and five different architectures, revealing that while accuracy drops on unseen subtypes, the model's confidence does not decrease correspondingly, leading to overconfidence in incorrect predictions. The study finds that this calibration failure is distinct from the effects of generic image corruption, where both accuracy and confidence decline together. Additionally, recalibration methods tuned on seen subtypes only partially address the calibration gap, indicating that subtype robustness should be assessed through calibration metrics rather than accuracy alone.
Methodology
The authors formalize subtype robustness as a same-coarse-label, unseen-fine-subtype shift and evaluate models based on matched accuracy, confidence, and calibration gaps. They conduct experiments across various datasets and architectures to analyze the relationship between accuracy and confidence under subtype shifts.
Results
The study finds that across all datasets, coarse accuracy declines on unseen subtypes while confidence remains high, resulting in systematic overconfidence. The drop in accuracy varies by dataset density, and the authors demonstrate that generic image corruption causes a more substantial decrease in confidence than subtype novelty. Recalibration efforts improve but do not eliminate the calibration gap.
Implications
The findings suggest that machine learning models deployed in real-world applications, particularly in hierarchical classification tasks, may be at risk of miscalibration when encountering novel subtypes. This has significant implications for applications such as biodiversity monitoring, where accurate coarse predictions are critical despite the presence of unseen fine-grained categories.
ConformalShift: Targeted Event Reordering Against Adaptive ECG Monitoring
Time Series
- ConformalShift is an event-reordering attack that targets adaptive ECG monitoring systems.
- The attack suppresses the detection of ventricular ectopic beats without modifying ECG data or classifier outputs.
- Experimental results show a suppression rate of 66.7% for Extra Trees and 60.0% for HistGradientBoosting classifiers.
- The study highlights the importance of event order in adaptive conformal prediction systems.
Read more
ConformalShift: Targeted Event Reordering Against Adaptive ECG Monitoring
Summary
This paper introduces ConformalShift, a novel adversarial attack targeting adaptive ECG monitoring systems that utilize conformal prediction. The authors highlight the vulnerability of these systems to event reordering, which can suppress the detection of clinically significant heartbeat classes, particularly ventricular ectopic beats. Unlike traditional adversarial attacks that modify input data or model parameters, ConformalShift manipulates the order of authentic events without altering ECG waveforms, labels, or classifier outputs. The methodology involves searching for feasible permutations of preceding events to lower the ventricular threshold before a target event is evaluated. The effectiveness of ConformalShift is demonstrated through experiments on disjoint MIT-BIH datasets, showing a significant suppression rate of eligible targets compared to random scheduling. The findings underscore the potential for adversarial manipulation in healthcare monitoring systems, emphasizing the need for robust defenses against such timing-based attacks.
Methodology
The authors developed a constrained search method to identify feasible event orders that can suppress the target class in adaptive ECG monitoring. They defined a bounded event reordering attack that manipulates the threshold state of the monitor by changing the order of authentic events while preserving their content. The methodology was evaluated on two ECG datasets using two different classifier families.
Results
ConformalShift successfully suppressed 66.7% of eligible targets for the Extra Trees classifier and 60.0% for the HistGradientBoosting classifier. In contrast, random scheduling resulted in much lower suppression rates of 4.4% and 12.0%, respectively. The attack also demonstrated effectiveness on the INCART dataset, although reducing the displacement budget weakened its performance.
Implications
The results indicate that adaptive ECG monitoring systems are susceptible to timing-based adversarial attacks, which could have serious implications for patient safety and the reliability of automated healthcare systems. This highlights the need for developing robust defenses against such vulnerabilities in real-time monitoring applications.
GENESIS: Towards Explainable Causal Discovery
Graph Learning
Interpretability
Large Language Models
- Genesis ensures decision traceability for each edge in the causal graph.
- The framework integrates motif-level reasoning with statistical validation.
- Achieves 100% decision traceability across various experimental settings.
- Outperforms traditional statistical methods in terms of Structural Hamming Distance.
Read more
GENESIS: Towards Explainable Causal Discovery
Summary
The paper introduces Genesis, an explainable hybrid framework for causal discovery (CD) that addresses the challenges of decision traceability in learning causal structures from observational data. Traditional CD methods often lack transparency in edge decisions, especially in low-sample scenarios where statistical methods struggle. Genesis formalizes the need for decision traceability, ensuring that each edge in the directed acyclic graph (DAG) is justified by auditable evidence, Markov Blanket consistency, or domain reasoning. The framework operates in three stages: first, it identifies and scores three-node structural motifs (chains, forks, colliders) to establish transparent priors; second, it evaluates the plausibility of these motifs using statistical evidence; and finally, it refines the graph by integrating domain knowledge when necessary. Experimental results demonstrate that Genesis achieves 100% decision traceability and outperforms traditional statistical methods in most benchmark datasets while maintaining comparable performance to state-of-the-art LLM-assisted approaches. This work emphasizes the importance of explainability in causal discovery, making it a first-class objective in the field.
Methodology
Genesis decomposes causal graph construction into interpretable decision points, utilizing a three-stage process: (1) LLM Explorer identifies and scores structural motifs, (2) Data Explorer evaluates edge plausibility using statistical evidence, and (3) the graph is refined through Markov Blanket filtering, integrating both statistical and semantic insights.
Results
Genesis achieved 100% decision traceability across all experimental settings and consistently outperformed traditional statistical causal discovery methods on benchmark datasets, while achieving performance levels comparable to advanced LLM-assisted approaches.
Implications
The framework's emphasis on explainability and decision traceability can enhance the reliability of causal analyses in real-world applications such as treatment effect estimation, root cause analysis, and scientific discovery, where understanding the rationale behind causal relationships is crucial.
ReFP-AD: Rectified Flow Preconditioning for Energy-Based Anomaly Detection
Computer Vision
Generative Models
- Introduces ReFP-AD, a method for stable anomaly detection in high-dimensional token spaces.
- Addresses instability in Energy-Based Models due to geometric factors like anisotropy.
- Achieves 98.6% Image AUROC on MVTec-AD and 97.3% on VisA, outperforming prior methods.
- Demonstrates the critical role of geometric reparameterization for effective MCMC sampling.
Read more
ReFP-AD: Rectified Flow Preconditioning for Energy-Based Anomaly Detection
Summary
The paper introduces ReFP-AD, a novel method for anomaly detection that addresses the challenges of modeling heterogeneous normal data without access to anomalous samples. It leverages Energy-Based Models (EBMs) for explicit density estimation in high-dimensional token spaces, which are often unstable due to geometric issues like anisotropy and strong cross-dimensional correlations. The authors propose a geometric reparameterization that transforms high-dimensional embeddings into a well-conditioned latent space using optimal transport (OT)-coupled rectified flow. This transformation enables stable persistent contrastive divergence with preconditioned Stochastic Gradient Langevin Dynamics (SGLD), allowing for effective anomaly scoring based on the learned energy landscape. The method is evaluated on the MVTec-AD and VisA datasets, achieving state-of-the-art performance in anomaly detection and localization, significantly outperforming previous unified EBM baselines. The findings highlight the importance of geometric conditioning in high-dimensional spaces for stable MCMC sampling and accurate anomaly detection.
Methodology
The methodology involves a geometric reparameterization of high-dimensional token embeddings into a well-conditioned latent space using optimal transport-coupled rectified flow. This transformation facilitates stable sampling through persistent contrastive divergence with preconditioned SGLD, allowing for effective training of EBMs in high-dimensional spaces without dimensionality reduction.
Results
ReFP-AD achieved 98.6% Image AUROC and 97.9% Pixel AUROC on the MVTec-AD dataset, and 97.3% Image AUROC and 99.0% Pixel AUROC on the VisA dataset. These results represent significant improvements over previous unified EBM baselines, with gains of up to +10.8% in Image AUROC.
Implications
The findings suggest that geometric conditioning is essential for effective anomaly detection in high-dimensional spaces, which could have applications in industrial inspection, autonomous systems, and scientific imaging where reliable detection of anomalies is critical.
Maglev: Sliding Recurrent Memory
NLP
Large Language Models
Efficient ML
- Maglev combines sliding-window attention with a recurrent memory mechanism for improved language modeling.
- The architecture allows for parallel training through a prefiller that generates memory targets independently of the decoder.
- A memory consistency loss is employed to align the decoder's memory with the prefiller's outputs.
- Empirical results show significant improvements in validation loss and accuracy on downstream tasks.
Read more
Maglev: Sliding Recurrent Memory
Summary
Maglev introduces a novel recurrent Transformer architecture designed to enhance memory management in language models. It generalizes sliding-window attention while maintaining parallelizability during training. The architecture comprises two models: a prefiller (Q) that uses full attention to generate memory targets and a decoder (P) that employs sliding-window attention and recurrent key/value (K/V) injection for next-token prediction. A memory consistency loss aligns the decoder's memory with the prefiller's targets, allowing for efficient inference using only the decoder. The empirical results demonstrate that Maglev outperforms existing sliding-window and latent recurrent Transformer baselines in terms of validation loss and downstream pretraining benchmarks. Additionally, the architecture allows for parameter sharing between the prefiller and decoder, reducing memory usage while retaining performance gains.
Methodology
Maglev employs a two-model architecture consisting of a prefiller (Q) and a decoder (P). The prefiller generates memory targets using full attention in a parallel manner, while the decoder uses sliding-window attention and recurrent K/V injection to predict the next tokens and produce its own memory. The training involves a memory consistency loss to ensure alignment between the decoder's memory and the prefiller's targets, allowing for efficient inference without the prefiller.
Results
The best model of Maglev, with 435M parameters trained on 43.52B tokens, achieved a FineWeb-Edu validation BPB improvement from 0.7413 to 0.7251 and an increase in average downstream accuracy from 54.1 to 56.4 compared to a matched sliding-window Transformer. It also outperformed a matched latent recurrent Transformer.
Implications
Maglev's architecture could lead to more efficient and effective language models capable of handling longer contexts without the computational overhead typically associated with full attention mechanisms. This could have significant applications in various NLP tasks, including text generation, summarization, and conversational agents.
ChaosProbe: A Neurochaotic Lens on Frozen Transformer Input-Embedding Spaces
NLP
Large Language Models
Theory
- ChaosProbe offers a new method for analyzing frozen transformer input-embedding spaces using neurochaotic principles.
- The method generates fixed-length signatures that summarize the response characteristics of embedding matrices.
- The study validates the effectiveness of ChaosProbe through correlation analyses among different transformer models.
- Results indicate that the embedding spaces contain meaningful geometric structures that can be analyzed independently of contextual computations.
Read more
ChaosProbe: A Neurochaotic Lens on Frozen Transformer Input-Embedding Spaces
Summary
The paper introduces ChaosProbe, a novel method for analyzing frozen transformer input-embedding spaces through a neurochaotic lens. While transformer models are typically evaluated based on their performance in tasks, this study emphasizes the importance of understanding the underlying structure of their input-embedding spaces. ChaosProbe utilizes a deterministic, chaos-inspired approach to generate response-based fingerprints for these embeddings. By applying chaotic trajectory transformations to prompt-level embedding matrices, the method summarizes the Firing Rate and Entropy channel responses, producing fixed-length signatures for various models including GPT-2, DistilGPT2, BERT-base-uncased, and RoBERTa-base. The authors conducted a proof-of-concept study with 80 neutral prompts, demonstrating that the method effectively captures the structural relationships among different models. The findings indicate that ChaosProbe can reveal significant similarities and differences in the embedding spaces, providing insights into the geometric organization of these representations prior to contextual processing.
Methodology
ChaosProbe applies a chaotic trajectory-based transformation to the prompt-level embedding matrices of frozen transformers. It summarizes the Firing Rate and Entropy channel responses to create fixed-length signatures for each model. The study involved a proof-of-concept analysis using 80 neutral prompts across four pretrained models, employing statistical measures such as Pearson correlation, Spearman correlation, and cosine similarity to assess the relationships among the embeddings.
Results
The analysis revealed that Pearson and Spearman correlations successfully identified all nearest-neighbor assignments among the models, while Euclidean distance recovered a subset of these assignments. Bootstrap resampling confirmed the stability of the correlations, and validity checks indicated that the generated fingerprints were robust against constant or collapsed responses.
Implications
The findings suggest that ChaosProbe can be a valuable tool for researchers to explore the structural properties of transformer models' embedding spaces, potentially leading to improved model interpretability and insights into their behavior. This could enhance the understanding of how different models process language and contribute to advancements in NLP applications.
FedCritic-MIMO: Communication-Efficient Serverless Federated Critic Learning for Massive-MIMO Resource Control in Open and Disaggregated 6G RANs
Reinforcement Learning
Federated Learning
Efficient ML
- FedCritic-MIMO enables communication-efficient coordination among independently deployed RAN controllers.
- The framework allows local execution of actors while facilitating shared critic parameter exchange.
- Achieves significant performance improvements in network throughput and quality-of-service metrics.
- Reduces critic communication overhead by about 76% compared to traditional methods.
Read more
FedCritic-MIMO: Communication-Efficient Serverless Federated Critic Learning for Massive-MIMO Resource Control in Open and Disaggregated 6G RANs
Summary
This paper introduces FedCritic-MIMO, a novel framework designed for communication-efficient serverless federated multi-agent reinforcement learning (MARL) aimed at resource control in open and disaggregated 6G radio access networks (RANs). The framework addresses the challenge of coordinating independently deployed cell-level controllers that lack a common trainer and operate with local observations. FedCritic-MIMO facilitates the exchange of compatible shared critic parameters among neighboring controllers while allowing each base station (BS) to execute its actor locally. The approach is tailored for reuse-1 multi-cell massive multiple-input multiple-output (massive-MIMO) orthogonal frequency-division multiple access (OFDMA) systems, where distributed RAN controllers manage user scheduling, power allocation, beamforming, and interference under limited signaling. The framework employs wireless-aware event triggering, adaptive layer-wise top-k sparse critic exchange with error feedback, and balanced interference-aware fusion to enhance collaboration. The paper establishes guarantees for finite-time stationarity and consensus for the proposed critic recursion model. In simulations, FedCritic-MIMO outperforms various baselines, achieving superior network throughput, improved user-rate distribution, increased quality-of-service satisfaction, and reduced communication overhead by approximately 76% compared to uncompressed critic exchanges. These findings demonstrate the efficacy of decentralized critic parameter sharing in coordinating RAN controllers without centralized training or parameter aggregation.
Methodology
The methodology involves a serverless federated learning approach where each base station executes its actor locally and exchanges critic parameters with neighboring controllers based on a predefined compatible critic subnetwork. The framework incorporates wireless-aware event triggering and adaptive sparse critic exchange to minimize communication while ensuring effective coordination among controllers.
Results
FedCritic-MIMO demonstrated the best performance-communication tradeoff in simulations, achieving the highest network throughput, improved user-rate distribution, and increased QoS satisfaction. It also significantly reduced interference costs per delivered bit and critic communication overhead compared to various baseline methods.
Implications
The findings suggest that FedCritic-MIMO can be effectively applied in future 6G networks to enhance resource management and coordination among distributed RAN controllers, potentially leading to more efficient and intelligent network operations.
LLMs Can Annotate Attribution Graphs
Large Language Models
Interpretability
- Introduction of a low-cost pipeline for annotating attribution graphs using LLMs.
- Validation of the interpretability of LLM-generated supernodes against human annotations.
- Successful application of the pipeline on a two-hop Capitals task with high recovery rates.
- Proof of concept for open-ended exploration of attribution graphs through automated annotation.
Read more
LLMs Can Annotate Attribution Graphs
Summary
This paper presents a novel pipeline for automating the annotation of attribution graphs generated through circuit tracing in language models. The authors address the labor-intensive manual step of grouping individual features or MLP neurons into supernodes, which is crucial for interpretability. Their approach involves using a language model (GPT-5 mini) to generate descriptions for features based on their top-activating examples and subsequently grouping these descriptions into supernodes. The authors validate the interpretability of the supernodes produced by their pipeline against those created by human annotators using automated metrics. They demonstrate the effectiveness of their method on a two-hop Capitals task, achieving a recovery rate of 97 out of 100 prompts for the intermediate hop supernode. Additionally, the authors explore the potential of their pipeline for open-ended exploration by automatically annotating 1000 attribution graphs from Wikipedia prompts and filtering them for interesting behaviors. This work highlights the feasibility of using simple automation to enhance the interpretability of language models, paving the way for further advancements in automated circuit tracing.
Methodology
The authors developed a pipeline that generates feature descriptions from attribution graphs using an LLM and groups these descriptions into supernodes. The process involves three phases: generating supernodes from the highest-influence features, iteratively assigning remaining features to existing or new supernodes, and refining the supernodes by merging and pruning irrelevant ones.
Results
The pipeline successfully recovered a supernode corresponding to the intermediate hop in 97 out of 100 prompts during the two-hop Capitals task. The supernodes generated were found to be as interpretable as those created by human annotators, according to automated interpretability metrics. The authors also annotated 1000 attribution graphs from Wikipedia prompts, demonstrating the pipeline's capability for open-ended exploration.
Implications
This work suggests that automation can significantly reduce the time and effort required for annotating attribution graphs, making mechanistic interpretability more accessible. It opens avenues for further research in automated circuit tracing and enhances the understanding of language model internals.
HindSearch: Trajectory-Level Hindsight Critique for Search-Augmented Reinforcement Learning
Reinforcement Learning
Large Language Models
NLP
- Introduces trajectory-level hindsight critique (TLHC) to enhance search-augmented RL training.
- Achieves 39.4% average EM on a standard benchmark suite, outperforming prior methods.
- Demonstrates the critical role of hindsight in improving agent performance.
- Utilizes a frozen judge to provide directive critiques of failed trajectories.
Read more
HindSearch: Trajectory-Level Hindsight Critique for Search-Augmented Reinforcement Learning
Summary
The paper introduces HindSearch, a novel approach to improving search-augmented reinforcement learning (RL) agents by utilizing a trajectory-level hindsight critique mechanism. Traditional search-augmented language model (LM) agents are trained using a binary exact-match reward, which only indicates success or failure without providing insights into the reasons for failure. HindSearch addresses this limitation by implementing a hindsight self-distillation procedure where a frozen judge critiques failed trajectories based on the gold answer. This critique serves as an auxiliary on-policy distillation signal for the student agent's search actions. The authors demonstrate that HindSearch significantly enhances performance on a standard seven-benchmark suite, achieving an average exact match (EM) score of 39.4%, surpassing previous search-RL baselines. The study reveals that removing the judge's access to the gold answer diminishes the performance gains, highlighting the importance of hindsight in the learning process. Overall, HindSearch provides a more informative training signal that allows agents to learn from their mistakes effectively.
Methodology
HindSearch employs a trajectory-level hindsight critique mechanism where, after each rollout, a frozen judge evaluates failed trajectories against the gold answer and provides critiques. These critiques are used to condition a frozen teacher model, which generates an auxiliary on-policy distillation loss for the student agent's search actions. The method builds on the GRPO framework and integrates this new critique mechanism to enhance the learning signal from failed attempts.
Results
The implementation of HindSearch resulted in a notable improvement in performance, achieving a 39.4% average EM score on a standard seven-benchmark search-QA suite. This performance exceeds that of previous search-RL baselines, indicating the effectiveness of the trajectory-level hindsight critique approach. Additionally, the study found that removing the judge's access to the gold answer significantly reduced the performance gains, underscoring the importance of the hindsight critique.
Implications
HindSearch has the potential to advance the training of search-augmented RL agents by providing richer feedback mechanisms. This approach can be applied to various domains where agents must learn from complex interactions and failures, leading to more robust and effective learning strategies in search and question-answering tasks.
CoRe-GNN: Multilevel Message passing on Coarsened graphs
Graph Learning
- CoRe-GNN combines inter-cluster and intra-cluster message passing to leverage the strengths of both approaches.
- The architecture is agnostic to the choice of coarsening algorithm, allowing for flexibility in implementation.
- A novel batching scheme enables CoRe-GNN to efficiently handle large graphs with millions of nodes.
- The method shows competitive performance on various node classification tasks, outperforming existing baselines.
Read more
CoRe-GNN: Multilevel Message passing on Coarsened graphs
Summary
The paper introduces CoRe-GNN, a novel architecture designed to enhance the training of Graph Neural Networks (GNNs) on large graphs by addressing memory limitations associated with storing node representations across layers. The authors identify the shortcomings of existing scalable approaches, such as graph coarsening and Cluster-GCN, which either oversimplify node representations or discard long-range connections. CoRe-GNN innovatively combines both inter-cluster and intra-cluster message passing in parallel, allowing it to capture long-range structures while maintaining node-level discriminability. The architecture is flexible with respect to the coarsening algorithm used and incorporates a batching scheme that scales effectively to graphs with millions of nodes. The authors provide theoretical guarantees regarding the preservation of signal propagation and demonstrate through extensive experiments that CoRe-GNN outperforms existing methods on various node classification benchmarks, including homophilic, heterophilic, large-scale, and long-range graphs.
Methodology
CoRe-GNN employs a dual message-passing mechanism that operates on both a coarsened graph (for capturing long-range dependencies) and the original graph (for maintaining node-specific information). This parallel approach is coupled across layers, allowing for effective learning while addressing memory constraints. The authors also introduce a cluster-based batching scheme to optimize the training process on large-scale graphs.
Results
CoRe-GNN consistently outperformed both graph coarsening and Cluster-GCN baselines across various benchmarks, demonstrating superior accuracy in node classification tasks, particularly in long-range scenarios. The architecture maintained memory efficiency while achieving competitive results, validating its effectiveness in handling large graphs.
Implications
The advancements presented in CoRe-GNN have significant implications for applications involving large-scale graph data, such as social network analysis, recommendation systems, and biological network modeling. The ability to efficiently train GNNs on large graphs opens up new avenues for research and practical applications in various domains.
FL-OA: A Byzantine-Robust Federated Learning Framework with Outsourced Auditing for Intelligent Devices
Federated Learning
- FL-OA utilizes outsourced auditing to enhance Byzantine robustness without strong assumptions.
- The framework mitigates divergence among benign updates through innovative local training techniques.
- A parameter importance indicator is introduced to address the curse of dimensionality.
- Theoretical analysis supports the framework's design and effectiveness.
Read more
FL-OA: A Byzantine-Robust Federated Learning Framework with Outsourced Auditing for Intelligent Devices
Summary
The paper introduces FL-OA, a novel federated learning framework designed to enhance robustness against Byzantine attacks, which are a significant threat in distributed learning environments. Traditional defense mechanisms often rely on strong assumptions about the proportion of benign devices or require the server to possess a root dataset for effective auditing. FL-OA circumvents these limitations by outsourcing the auditing process to a third-party organization that holds an additional root dataset, thus enabling robust aggregation without stringent assumptions. The framework also addresses the challenges of divergence among benign updates and the curse of dimensionality by incorporating a gradient ascent step and a correction term during local training, as well as a parameter importance indicator to focus on critical parameters for auditing. The authors provide a theoretical analysis of FL-OA and validate its effectiveness through extensive experiments, demonstrating that it outperforms existing methods in resisting Byzantine attacks.
Methodology
FL-OA employs a collaborative approach where the server partners with a third-party organization for auditing. It integrates a gradient ascent step and a correction term in local training to reduce update divergence and introduces a parameter importance indicator to manage high-dimensional data effectively.
Results
The experimental results indicate that FL-OA significantly outperforms existing Byzantine defense strategies, demonstrating its robustness against malicious updates in federated learning scenarios.
Implications
The proposed framework has potential applications in various fields that utilize federated learning, such as healthcare, finance, and autonomous systems, where data privacy and model integrity are critical.
UNVaMP: Neural Knowledge Tracing with Variational Regularization of Latent Knowledge Dynamics
Theory
Interpretability
Time Series
- UNVaMP integrates observed interactions with internal memory for dynamic knowledge representation.
- The architecture supports both purely neural and hybrid configurations for flexibility and interpretability.
- UNVaMP-MLP shows superior predictive performance compared to other models on multiple datasets.
- The model allows for controlled volatility in knowledge estimates and quantifies uncertainty.
Read more
UNVaMP: Neural Knowledge Tracing with Variational Regularization of Latent Knowledge Dynamics
Summary
The paper introduces the Unified Neural Variational Measurement of Proficiency (UNVaMP) architecture, a novel approach to knowledge tracing that combines observed student-item interactions with internal memory to create evolving latent representations of student knowledge. This architecture allows for accurate predictions of future responses while providing control over the smoothness of learning trajectories. UNVaMP can function as a purely neural model (UNVaMP-MLP) or as a hybrid model (UNVaMP-MIRT) that utilizes an interpretable measurement function over the latent space. The authors demonstrate that UNVaMP-MLP outperforms other models in predictive accuracy across three out of four datasets, while UNVaMP-MIRT maintains competitive performance with the added benefit of interpretability. The architecture also offers mechanisms for controlling volatility in estimating student latent variables, quantifying uncertainty in knowledge state estimates, and accommodating heterogeneous interaction features. Experimental results indicate that auxiliary inputs can influence the predictive behavior of UNVaMP-MIRT, and simulation studies confirm that UNVaMP produces reliable knowledge state estimates under controlled conditions. Overall, UNVaMP is positioned as a practical tool for educational systems, capable of extracting meaningful structures from student-item interactions.
Methodology
The authors developed the UNVaMP architecture, which utilizes neural networks to model latent knowledge dynamics. They compared a purely neural configuration (UNVaMP-MLP) with a hybrid model (UNVaMP-MIRT) that incorporates a one-parameter logistic measurement function. The models were evaluated on multiple datasets to assess predictive performance, interpretability, and the impact of auxiliary inputs.
Results
UNVaMP-MLP achieved the highest predictive accuracy on three out of four datasets, while UNVaMP-MIRT performed closely behind, demonstrating that interpretability comes with a modest predictive cost. The architecture successfully controlled volatility in latent variable estimates and quantified uncertainty, with experimental results showing that auxiliary inputs led to structured changes in predictions.
Implications
The UNVaMP architecture has significant implications for real-world educational systems, providing a robust framework for tracking student knowledge over time. Its ability to recover meaningful structures from interactions can enhance personalized learning experiences and inform instructional strategies.
Approximate Speculative Decoding
NLP
Large Language Models
Efficient ML
- ASD replaces binary first-mismatch truncation with budgeted longest-prefix selection, allowing for more flexible acceptance of mismatches.
- The method does not require a new draft model or fine-tuning, making it easy to implement.
- ASD improves throughput by 3.05% to 15.26% compared to strict verification methods across multiple tasks.
- The approach allows for the reuse of target-greedy suffixes, enhancing efficiency without additional computational costs.
Read more
Approximate Speculative Decoding
Summary
This paper introduces Approximate Speculative Decoding (ASD), a novel approach to enhance the efficiency of autoregressive generation in large language models (LLMs). ASD addresses the limitations of standard greedy verification in speculative decoding, which halts at the first mismatch between draft tokens and target model outputs, leading to the loss of potentially reusable suffixes. Instead, ASD employs a training-free verification strategy that allows for the acceptance of certain mismatches while maintaining a budget for local regret. This method enables the reuse of contiguous target-greedy suffixes without requiring additional forward passes of the target model. The authors demonstrate that ASD can improve throughput significantly over matched strict verification methods and maintain high acceptance rates in various tasks, showcasing its potential to optimize the decoding process in LLMs without necessitating model retraining or fine-tuning.
Methodology
The authors propose a budgeted longest-contiguous-prefix selection problem as a framework for ASD, which measures local target-logit regret at each mismatch. The method accepts mismatches based on a regret threshold, block-level exception cap, and request-level regret budget, allowing for the reuse of subsequent draft tokens that remain target-greedy under the realized prefix.
Results
ASD demonstrated an improvement in fixed-workload throughput by 3.05% to 15.26% over matched strict verification methods and averaged a 7.78% gain across seven Qwen3-14B + DSpark-14B tasks. Additionally, on the DeepSeek-V4-Flash (284B) with DSpark, ASD increased verifier-side acceptance rates by approximately 10% to 16% on tasks like GSM8K and MATH-500.
Implications
The findings suggest that ASD can significantly enhance the efficiency of autoregressive generation in LLMs, making it a valuable technique for real-time applications where rapid response generation is critical. Its training-free nature allows for easy integration into existing systems, potentially leading to broader adoption in various NLP tasks.
Pseudorandom Streams within Diffusion Models Act as Learnable Inputs That Affect Generation Quality
Generative Models
Optimization
Theory
- Pseudorandom streams can be treated as learnable inputs in diffusion models.
- The study distinguishes between next-value predictability and exploitability within the target diffusion system.
- Different pseudorandom orbits lead to significant variations in generation quality and diffusion losses.
- The relationship between pseudorandom inputs and learning systems follows an empirical power law.
Read more
Pseudorandom Streams within Diffusion Models Act as Learnable Inputs That Affect Generation Quality
Summary
This paper explores the role of pseudorandom streams in diffusion models, positing that these streams can act as learnable inputs that significantly influence both training and generation quality. The authors argue that the deterministic nature of pseudorandom number generation (PRNG) on finite-precision hardware introduces a model-dependent structure that can be exploited by neural networks. They introduce a small multilayer perceptron (MLP) to predict the next value in a pseudorandom orbit, assessing the predictability of sequences. Additionally, a diffusion probe is employed to evaluate whether the diffusion model can utilize the structure of these orbits by replacing real images with random tensors. The study finds that different pseudorandom orbits lead to varying diffusion losses and generation quality on datasets like MNIST and CIFAR-10, indicating that the choice of random source is more than a mere implementation detail. The paper highlights the importance of understanding the relationship between the pseudorandom input and the learning system, revealing that the realized training objective is jointly dependent on the model, data, and pseudorandom stream.
Methodology
The authors developed an experimental framework that includes empirical-rank uniformization, Gaussianization, meanβvariance calibration, and screening for strong dynamical structures. They evaluated various deterministic orbits and compared their performance against an IID source using a multilayer perceptron for next-value prediction and a diffusion probe for exploitability assessment.
Results
The experiments demonstrated that different pseudorandom orbits produced markedly different diffusion losses and generation quality on MNIST and CIFAR-10. Both the MLP loss and the diffusion probe loss exhibited strong rank correlations with generation degradation. After normalization by the IID baseline, the losses followed an empirical power law with distinct exponents for the two datasets, indicating a learning-system-dependent relationship.
Implications
The findings suggest that the choice of pseudorandom input can significantly impact the performance of diffusion models, highlighting the need for careful consideration of random sources in machine learning tasks. This could lead to improved model designs that better exploit the structure of pseudorandom streams, enhancing generation quality in various applications.
Pin Once, Swap Light: Subspace-Aligned Centroid-Residual Training for Efficient Ultra-LoRA Serving
Large Language Models
Efficient ML
- SALT introduces a matrix cosine regularizer to align task subspaces into a unified basis.
- The framework decouples representational capacity from physical memory costs, allowing for efficient multi-tenant serving.
- SALT achieves up to 18.5% absolute accuracy gains over existing compression methods while reducing memory usage by up to 16x.
- The method improves serving throughput by up to 51% under PCIe bandwidth pressure and 28% under GPU VRAM constraints.
Read more
Pin Once, Swap Light: Subspace-Aligned Centroid-Residual Training for Efficient Ultra-LoRA Serving
Summary
The paper addresses the challenge of efficiently serving multiple Low-Rank Adapters (LoRAs) in multi-tenant systems while maintaining high task performance. Traditional high-rank adapters provide better performance but are constrained by GPU VRAM and PCIe bandwidth, while ultra-low-rank adapters (r β€ 2) reduce memory usage but degrade performance. To tackle this issue, the authors propose Subspace-Aligned LoRA Training (SALT), a hierarchical fine-tuning framework that consists of three phases: training high-capacity domain centroids on public data, fine-tuning ultra-low-rank task residuals on private data, and dynamically swapping task residuals during inference while keeping the centroid pinned in GPU memory. SALT employs a novel alignment regularizer to unify task subspaces, allowing it to recover high-rank performance with low-rank residuals. The framework significantly reduces memory usage and improves serving throughput under hardware constraints, demonstrating its effectiveness across various large language models (LLMs).
Methodology
The methodology involves a three-phase process: (1) training high-capacity domain centroids using a novel alignment regularizer on public data, (2) fine-tuning ultra-low-rank task residuals on private data while keeping the centroids frozen, and (3) during inference, dynamically swapping the task residuals while the centroid remains pinned in GPU memory.
Results
SALT recovers high-rank accuracy using ultra-low-rank residuals (r β€ 2), achieving up to 18.5% accuracy improvement over state-of-the-art methods. It reduces per-adapter memory usage by up to 16x and enhances serving throughput by up to 51% under PCIe bandwidth constraints and 28% under GPU VRAM limitations, particularly demonstrated with the Llama-3.2-3B model.
Implications
The proposed SALT framework has significant implications for the deployment of large language models in multi-tenant environments, enabling more efficient resource utilization and improved performance for personalized applications. It can facilitate the development of scalable LLM-as-a-Service architectures.
PatTree: a novel approach for automated creation of multimodal, graph-based patient representations for medical classification tasks
Graph Learning
Multimodal
- PatTree automates the integration of multimodal, longitudinal patient data.
- The approach creates a tree-based patient representation without requiring data harmonization.
- PatTree preserves semantic relationships in clinical data, enhancing interoperability.
- Achieved state-of-the-art classification performance in distinguishing cognitive states.
Read more
PatTree: a novel approach for automated creation of multimodal, graph-based patient representations for medical classification tasks
Summary
The paper presents PatTree, a novel approach for the automated creation of multimodal, graph-based patient representations aimed at enhancing medical classification tasks. Traditional methods of integrating clinical data often face challenges due to the heterogeneity and complexity of real-world medical data, which includes missing values, diverse modalities, and inconsistent formats. PatTree addresses these issues by enabling early-stage data integration without the need for pre-standardized inputs. It constructs a unified knowledge graph that preserves semantic relationships among data elements across various modalities and sources, thus facilitating interoperability. The authors demonstrate the effectiveness of PatTree using a subset of the Alzheimerβs Disease Neuroimaging Initiative (ADNI-1) cohort, achieving a balanced accuracy of 98.5% and an F1 score of 0.987 in a three-class classification task distinguishing Alzheimerβs disease, mild cognitive impairment, and cognitively normal individuals. The results indicate that the automated structuring of multimodal medical data can serve as a scalable foundation for clinical AI applications, bypassing the need for extensive data preparation and standardization.
Methodology
PatTree utilizes a graph-based approach to automatically structure heterogeneous clinical data into a unified knowledge graph. It leverages the natural structure of medical data derived from patient journeys and clinical information systems, avoiding the need for data harmonization or named entity recognition (NER). This allows for the direct application of Graph Neural Networks (GNNs) for classification tasks.
Results
In a classification task involving 763 patients from the ADNI-1 cohort, PatTree achieved a balanced accuracy of 98.5% and an F1 score of 0.987, effectively distinguishing between Alzheimerβs disease, mild cognitive impairment, and cognitively normal individuals.
Implications
The PatTree framework has the potential to significantly streamline the process of preparing clinical data for AI applications, making it easier to implement AI-driven decision support systems in healthcare. By reducing the reliance on standardized data inputs, it could enhance the scalability and reproducibility of AI models in clinical settings.
Cross-Model KV Cache Transfer in LLM Families: A Closed-Form Linear Mapping for Prefill Reuse
Large Language Models
Efficient ML
Optimization
- Introduction of cross-model KV cache transfer to reduce prefill costs in LLMs.
- Discovery of substantial linear structure in KV relationships across model families.
- Development of a closed-form ridge mapper that operates efficiently on a per-head basis.
- Validation of the method across multiple model families, achieving high accuracy retention.
Read more
Cross-Model KV Cache Transfer in LLM Families: A Closed-Form Linear Mapping for Prefill Reuse
Summary
This paper addresses the challenge of reusing key-value (KV) caches across different models within a family of large language models (LLMs) during production deployments. The authors propose a method called cross-model KV cache transfer, which allows a target model to reuse the KV cache from a source model, thus avoiding the need for costly prefill operations. The study reveals that there is a significant linear relationship between the KV pairs of source and target models, particularly when they share the same number of KV heads and dimensions. The authors introduce a closed-form ridge mapper that operates on a per-head basis, consisting of three main steps: selecting the most predictive source layers, stripping positional encodings from keys, and fitting a ridge regression model on a small calibration dataset. The results show that this linear mapping retains a high percentage of the target model's standalone accuracy while significantly reducing prefill latency. Additionally, a nonlinear multi-layer perceptron (MLP) is introduced to further enhance retention in cases where the linear model fails. The findings suggest that the proposed method is not only efficient but also practical for real-world applications in LLM serving.
Methodology
The authors propose a closed-form ridge regression approach for cross-model KV cache transfer, which includes selecting the top-k predictive source layers for each target layer, stripping positional encodings from keys, and fitting the model on a small calibration dataset. The method is validated across six matched KV pairs from three different model families.
Results
The linear mapping retains 73-98% of the standalone accuracy of the target model across five benchmarks, with a significant reduction in prefill latency (2.7-25 times faster than traditional methods). The nonlinear MLP extension provides an additional retention improvement of up to 37 percentage points on more difficult pairs.
Implications
The proposed method has potential applications in optimizing the deployment of LLMs in production environments, allowing for more efficient resource utilization and improved user experience during long agentic sessions. It can facilitate smoother transitions between models of different sizes without incurring high computational costs.
A Theory of Conditional Collapse under Low-Rank Weight-Space Ablations: I. The Single-Block Theory and Synthetic Validation
Theory
Interpretability
- Establishes conditions for deterministic output collapse when carriers are deleted from a neural network.
- Demonstrates that activation patching and weight-space ablation produce different effects on network behavior.
- Derives an exact interaction term for a specific architecture, revealing the limitations of treating components as independent.
- Validates theoretical predictions with synthetic experiments on small transformers, showing strong rank correlation with model behavior.
Read more
A Theory of Conditional Collapse under Low-Rank Weight-Space Ablations: I. The Single-Block Theory and Synthetic Validation
Summary
This paper investigates the relationship between activation patching and weight-space ablation in neural networks, focusing on when these two interventions yield consistent results regarding the causal responsibility of network components for specific behaviors. The author presents an idealized model of conditional computation, demonstrating three key results: (1) The conditions under which deleting certain carriers leads to a deterministic collapse of outputs for matched input pairs, (2) The distinct effects of patching versus ablating carriers on network outputs, and (3) An exact formula for the interaction term in a specific architecture involving an attention head and its MLP. The findings are validated through experiments with small transformers on synthetic tasks, revealing strong correlations between the model's predictions and the observed behavior of the edited networks. The paper highlights the nuanced interplay between network components and the implications for mechanistic interpretability in deep learning.
Methodology
The author employs an idealized model of conditional computation, analyzing the effects of deleting and patching carriers in a residual stream. Theoretical results are derived through mathematical proofs, and synthetic experiments are conducted using small transformers to validate these results.
Results
The study finds that deleting carriers leads to a deterministic output collapse under specific conditions, while patching and ablating carriers affect outputs differently. The derived interaction term shows that the independence assumption does not hold in certain architectures, leading to deviations from expected behavior. The synthetic validation demonstrates strong correlations between the model's predictions and the actual behavior of the networks across various ablation configurations.
Implications
The findings have significant implications for mechanistic interpretability in neural networks, providing a clearer understanding of how different components contribute to network behavior. This could inform future research on model design and debugging, as well as enhance the interpretability of deep learning systems.
Sedentary Behavior Classification for Wearable Sensors with a CNN-BiLSTM Model
Time Series
- The CHAP model effectively classifies sedentary behavior using hip-worn accelerometer data.
- Transferability of the hip-trained model to wrist-worn data shows a decrease in accuracy, necessitating adaptation.
- Fine-tuning the model with wrist data improves performance compared to training from scratch.
- The study emphasizes the importance of posture in accurately measuring sedentary behavior.
Read more
Sedentary Behavior Classification for Wearable Sensors with a CNN-BiLSTM Model
Summary
This paper addresses the challenge of accurately classifying sedentary behavior using wearable sensors, particularly focusing on the transferability of a deep learning model from hip-worn to wrist-worn accelerometer data. The authors utilize the CHAP model, a CNN-BiLSTM architecture originally trained on hip accelerometer data, to evaluate its performance on wrist data without retraining, as well as its adaptation through fine-tuning with varying amounts of labeled wrist data. The study employs the iWatch dataset, which includes ground-truth posture labels derived from wearable cameras. Results indicate that while the hip-trained model performs well on hip data, its accuracy decreases on wrist data due to differences in sensor placement. However, fine-tuning the CHAP model consistently outperforms transformer models trained from scratch, suggesting that pretraining on hip data provides a beneficial starting point for wrist deployment. The findings highlight the necessity for wrist-specific adaptations to manage the increased variability in wrist sensor signals.
Methodology
The study employs a CNN-BiLSTM model (CHAP) initially trained on hip accelerometer data and tests its zero-shot performance on wrist data. It further explores the model's adaptation through fine-tuning with varying amounts of labeled wrist data. The performance is compared against transformer models trained from scratch on the same dataset.
Results
The hip-trained CHAP model shows strong performance on hip data but experiences a drop in accuracy on wrist data. Fine-tuning the model with wrist data yields consistent performance improvements, outperforming transformer models. The results indicate that hip-based pretraining is advantageous for wrist deployment, although wrist-specific adaptations are necessary to address signal variability.
Implications
The findings suggest that wearable technology for monitoring sedentary behavior can benefit from transfer learning approaches, allowing for effective use of existing models in different sensor placements. This has potential applications in health monitoring and interventions aimed at reducing sedentary behavior.
Deep Learning-Based Estimation of Ground Reaction Forces in Parkinsonian Gait Using an Optimized Set of IMU Data
Time Series
- Introduces a deep learning framework for estimating ground reaction forces in Parkinsonian gait.
- Achieves high accuracy in estimating vGRFs using a minimal configuration of wearable IMUs.
- Demonstrates significant differences in optimal sensor placement between PD patients and healthy controls.
- Identifies a compact setup that allows for robust estimation with only two IMUs.
Read more
Deep Learning-Based Estimation of Ground Reaction Forces in Parkinsonian Gait Using an Optimized Set of IMU Data
Summary
This study presents a novel deep learning framework for estimating bilateral vertical ground reaction forces (vGRFs) in individuals with Parkinson's disease (PD) using a minimal and optimized configuration of wearable inertial measurement units (IMUs). Traditional methods for gait analysis in PD often rely on laboratory-based systems, which can be impractical for continuous monitoring. The proposed hybrid CNN-BiLSTM model was trained on data from 61 PD patients and 65 healthy controls, utilizing 13 IMUs. The model demonstrated high intra-subject accuracy (RΒ² = 0.98) and strong inter-subject generalization (RΒ² = 0.93 for healthy controls and RΒ² = 0.91 for PD patients). The study found that the configuration of sensors significantly impacted estimation accuracy, with the optimal setup for PD patients involving four IMUs. Notably, reducing the number of IMUs to one led to a marked decrease in accuracy, although a robust estimation was still achievable with just two IMUs. This research supports the development of practical wearable systems for gait analysis in PD, facilitating accessible clinical assessments and personalized rehabilitation strategies.
Methodology
The study employed a hybrid CNN-BiLSTM deep learning model trained on IMU data collected from 61 PD patients and 65 healthy controls. The model was optimized to estimate bilateral vertical ground reaction forces, with a focus on sensor configuration and placement.
Results
The model achieved an intra-subject accuracy of RΒ² = 0.98 and inter-subject generalization of RΒ² = 0.93 for healthy controls and RΒ² = 0.91 for PD patients. The optimal sensor configuration for PD patients involved four IMUs, while a minimal setup with two IMUs still provided robust estimations.
Implications
The findings suggest that wearable IMU-based systems can facilitate continuous gait monitoring in PD, enhancing clinical assessments and enabling personalized rehabilitation strategies. This approach could also be adapted for other pathological conditions, improving accessibility to gait analysis.
NANQ: Noise-Floor-Aware Mixed-Precision Non-Uniform Quantization for Analog Compute-in-Memory
Efficient ML
- NANQ optimizes quantization by considering hardware noise, improving precision allocation.
- The framework assigns finer quantization resolution in low-noise areas and avoids redundancy in high-noise regions.
- Layer-wise bit-widths are determined based on each layer's saturation point under noise, enhancing model performance.
- Experimental results show significant accuracy improvements in vision and language models compared to existing quantization methods.
Read more
NANQ: Noise-Floor-Aware Mixed-Precision Non-Uniform Quantization for Analog Compute-in-Memory
Summary
The paper presents NANQ, a novel noise-aware mixed-precision non-uniform quantization framework tailored for analog compute-in-memory (CIM) architectures. Traditional quantization methods often overlook the impact of hardware noise, leading to inefficient precision allocation in low-bit quantized models. NANQ addresses this by modeling the magnitude-dependent weight noise from actual measurements of an eFlash CIM array, converting this noise profile into an adaptive quantization density. This allows for finer resolution in low-noise regions while avoiding unnecessary precision in areas dominated by noise. Additionally, NANQ employs a layer-wise bit-width assignment strategy based on the saturation point of each layer under hardware noise, utilizing a unified threshold to optimize performance without requiring retraining. Experimental results demonstrate that NANQ significantly enhances model accuracy, achieving an 8.05 percentage point improvement in vision models and a 54.7% reduction in perplexity for language models compared to existing methods like PowerQuant. The framework captures the benefits of additional quantization resources with only 3.2β3.8 equivalent bits, showcasing its efficiency and effectiveness in practical applications.
Methodology
NANQ employs a training-free non-uniform quantization approach that adapts to hardware noise profiles. It uses cumulative-density partitioning to establish quantization boundaries and a noise-aware mixed-precision allocation strategy to determine optimal bit-widths for each layer based on a unified threshold derived from hardware noise characteristics.
Results
NANQ improved vision-model accuracy by 8.05 percentage points and reduced language-model perplexity by 54.7% on average compared to PowerQuant. It achieved the best quantized results in 38 out of 45 model-bit-width configurations, particularly excelling at low precision.
Implications
The findings suggest that NANQ can significantly enhance the performance of neural networks deployed on analog CIM architectures, making it a valuable approach for energy-efficient inference in various applications, particularly in computer vision and natural language processing.
Sparse Weight Decomposition for Efficient Circuit Extraction
Interpretability
Large Language Models
Efficient ML
- SWD allows for efficient circuit extraction from pretrained models without additional training.
- The method matches or exceeds the fidelity of existing approaches while using less than 1% of the data.
- SWD supports full-model replacement of attention and MLP weight matrices after fine-tuning.
- A zero-data variant of SWD enables broader use in mechanistic interpretability analysis.
Read more
Sparse Weight Decomposition for Efficient Circuit Extraction
Summary
The paper introduces Sparse Weight Decomposition (SWD), a novel method for circuit extraction from dense pretrained transformers. Traditional methods for extracting interpretable units often require additional training or optimization, leading to increased computational costs and potential fidelity gaps. SWD addresses this by reparameterizing dense weight matrices into two sparse factors, allowing for the extraction of interpretable circuit units directly from pretrained models without the need for auxiliary representations. The methodology involves factorizing the weight matrix into two components, which define bottleneck units that can be independently scored and selected based on their contribution to task performance. The authors demonstrate that SWD achieves comparable fidelity to existing methods while using significantly less data for training replacements. The experiments show that SWD can effectively replace full model components, maintaining performance after fine-tuning. Additionally, a zero-data variant of SWD is proposed, broadening its applicability for mechanistic interpretability analysis.
Methodology
The authors propose Sparse Weight Decomposition (SWD), which factorizes a dense weight matrix into two sparse matrices. This factorization creates bottleneck units that can be scored and selected for circuit extraction, allowing for a streamlined workflow that does not require training additional models.
Results
SWD matches the fidelity of strong baselines like Transcoder while using significantly less data. It achieves similar circuit sufficiency and necessity targets with fewer active connections across various tasks on models such as GPT-2 and Qwen3.5-27B. The method also demonstrates effectiveness in full-model replacement after fine-tuning.
Implications
The introduction of SWD has significant implications for the field of interpretability in machine learning, particularly in understanding the internal workings of large language models. It allows researchers to extract meaningful circuits from pretrained models efficiently, facilitating better insights into model behavior and decision-making processes.
DiagLoop: A Counterfactual Data Flywheel with Stage-Localized Reinforcement for Diagnostic LLMs
Large Language Models
Reinforcement Learning
NLP
- DiagLoop synthesizes counterfactual scenarios to enhance training data for diagnostic LLMs.
- The model employs a unique stage-localized reinforcement learning approach to improve reasoning accuracy.
- It achieves significant improvements in path correctness over conventional models in various diagnostic contexts.
- The methodology allows for local deployment, addressing privacy and latency concerns in sensitive data environments.
Read more
DiagLoop: A Counterfactual Data Flywheel with Stage-Localized Reinforcement for Diagnostic LLMs
Summary
The paper presents DiagLoop, an innovative approach for training locally deployable diagnostic large language models (LLMs) that can effectively reason about causal relationships in industrial and clinical diagnostics. The authors address the challenges of limited training data, particularly the scarcity of serious cases and the lack of reasoning paths in existing records. DiagLoop utilizes a counterfactual data generation mechanism that synthesizes diverse scenarios based on codified physical relations or clinical guidelines, allowing the model to learn from a broader range of situations without requiring case-level expert annotations. The training process involves a teacher model that proposes counterfactual worlds, a hybrid checker that verifies the admissibility of these worlds, and a student model that learns to abstract symptoms, construct causal chains, and attribute root causes. The methodology incorporates stage-localized reinforcement learning to reward the model's performance at different stages of reasoning, while mechanisms are in place to prevent forgetting and ensure continuous improvement. The results demonstrate that the DiagLoop model significantly outperforms conventional baselines in terms of path correctness across multiple industrial systems and disease categories, showcasing its potential for real-world applications in diagnostics.
Methodology
DiagLoop combines counterfactual data generation, causal weakness mining, and stage-localized reinforcement learning. A training-only teacher generates counterfactual scenarios based on domain specifications, which are then validated by a hybrid checker. The student model learns to reason through symptom abstraction, causal-chain construction, and root-cause attribution, with reinforcement learning applied at each stage to enhance performance.
Results
The DiagLoop model, trained solely on synthesized scenarios, shows an improvement of 11.6 and 5.5 points in strict path correctness over the strongest conventional baseline across eight industrial systems and ten disease categories. It also outperforms proprietary references in path correctness, even under few-shot conditions.
Implications
The findings suggest that DiagLoop could revolutionize diagnostic processes in both industrial and clinical settings by providing a reliable, locally deployable LLM capable of accurate reasoning. This could lead to more effective diagnostics and treatments, ultimately improving patient outcomes and operational efficiency.
Population-Robust Feature Selection via Generalized Welfare Optimization
Optimization
Efficient ML
Interpretability
- Introduction of PopFS for robust feature selection across diverse populations.
- Utilization of a tunable welfare objective to balance predictive performance and protection for lower-benefit populations.
- Scalable optimization strategy using multitask sparse learning and ranked refit search.
- Demonstrated strong performance improvements in both average and worst-case scenarios.
Read more
Population-Robust Feature Selection via Generalized Welfare Optimization
Summary
The paper presents PopFS, a novel method for feature selection that addresses the challenge of deploying machine learning systems across heterogeneous populations. Traditional feature selection methods often optimize for a single population, while robust approaches typically fit one model for all populations. PopFS aims to learn a shared feature set that is robust to population differences, allowing each population to train its own model. The method incorporates a tunable welfare objective, enabling practitioners to balance overall predictive performance with the protection of populations that benefit least. To ensure scalability, PopFS employs multitask sparse learning for initial feature screening and a ranked refit search for final feature selection. The authors demonstrate PopFS's effectiveness across eight population splits from six prediction tasks, achieving improved average and worst-case performance. A case study on COVID-19 nowcasting illustrates how adjusting the welfare objective can enhance performance for underrepresented states while maintaining overall accuracy.
Methodology
PopFS employs a two-step approach: first, it uses multitask sparse learning to reduce the candidate feature pool, and then it conducts a ranked refit search over discrete feature sets to optimize the selection process. This method allows for a direct search over hard feature sets while maintaining scalability.
Results
PopFS consistently outperformed baseline methods by up to 22% in both average and worst-case population performance across various datasets. In a COVID-19 nowcasting study, tuning the welfare objective led to a 40% performance increase for the least-served states without compromising overall performance.
Implications
The findings suggest that PopFS can be effectively applied in various domains where feature selection must accommodate diverse populations, such as healthcare, finance, and social sciences. The ability to balance performance across populations can lead to more equitable machine learning applications.
Adaptive Sampling for Automated Post-Disaster Rapid Damage Assessment via Level-Set Cost-Aware Bayesian Optimization
Optimization
Robotics
Efficient ML
- Introduction of a novel Ordinal Deep Kernel Gaussian Process (ODGP) for modeling spatial correlations in damage assessment.
- Development of a cost-aware level-set acquisition function that optimizes data collection locations based on information gain and travel costs.
- Validation of the framework through synthetic studies and real disaster data, showcasing its effectiveness in rapid damage assessment.
- Demonstration of reduced uncertainty in damage predictions and improved operational efficiency for emergency response.
Read more
Adaptive Sampling for Automated Post-Disaster Rapid Damage Assessment via Level-Set Cost-Aware Bayesian Optimization
Summary
This paper addresses the challenges of post-disaster rapid damage assessment (PDDA) by proposing an adaptive sampling framework that utilizes cost-aware Bayesian optimization combined with level-set estimation. Traditional PDDA methods often rely on static and labor-intensive data collection, which can be inefficient and costly in dynamic post-disaster environments. The authors introduce a novel Ordinal Deep Kernel Gaussian Process (ODGP) that captures complex spatial correlations and handles ordinal damage levels effectively. The framework guides autonomous data collectors, such as UAVs, to the most informative regions while minimizing operational costs. The methodology is validated through controlled synthetic studies and high-fidelity disaster data, demonstrating its ability to efficiently trace damage boundaries and reduce predictive uncertainty. The results indicate that the proposed approach provides accurate and timely damage estimates, facilitating informed emergency responses.
Methodology
The proposed framework employs a cost-aware Bayesian optimization strategy, utilizing an Ordinal Deep Kernel Gaussian Process (ODGP) as the core meta-model. This model captures complex spatial relationships and ordinal damage levels. The adaptive sampling strategy is guided by a level-set acquisition function that prioritizes data collection in high-impact regions while considering operational constraints such as travel distance and resource expenditure.
Results
The framework was validated through controlled synthetic studies and high-fidelity disaster data, demonstrating its capability to efficiently trace damage boundaries and recover underlying damage maps. The results showed that the approach significantly reduces predictive uncertainty and provides timely and accurate damage estimates, which are crucial for effective emergency response.
Implications
The findings suggest that integrating adaptive sampling strategies with advanced data collection technologies can enhance the efficiency and accuracy of post-disaster damage assessments. This has significant implications for emergency management, resource allocation, and strategic planning in disaster response efforts.
SAKI: Score-Aware Low-Rank Key Indexing for Long-Context KV Retrieval
NLP
Large Language Models
Efficient ML
- SAKI optimizes key compression specifically for attention score fidelity, improving retrieval performance.
- The method outperforms key-PCA in all tested models, reducing recall error by 13-30%.
- The theoretical framework predicts empirical results with high accuracy, validating the approach.
- SAKI is training-free and provides a significant improvement in deep layers of transformer models.
Read more
SAKI: Score-Aware Low-Rank Key Indexing for Long-Context KV Retrieval
Summary
The paper introduces SAKI, a novel indexing method for key-value (KV) retrieval in long-context scenarios, addressing limitations in existing low-rank KV-cache indexes that either preserve model weights or key variance. The author derives a new objective for key compression that optimizes attention score fidelity, leading to a two-sided covariance-weighted low-rank approximation. This approach results in an optimal rank-r linear score map that is training-free and outperforms traditional methods like key-PCA across various models. Empirical results demonstrate that SAKI significantly reduces recall error and improves performance in deep layers of models, validating the theoretical predictions of the method. The findings suggest that the proposed objective and its implementation can enhance retrieval efficiency in large language models, making it a promising direction for future research in KV retrieval systems.
Methodology
The methodology involves deriving a two-sided covariance-weighted low-rank approximation for key compression, leading to an optimal rank-r linear score map. The method is training-free, relying on calibration covariances computed from model activations, and does not require traditional projector-based approaches.
Results
SAKI consistently outperformed key-PCA across multiple models (LLaMA-3.1-8B, Qwen2.5-7B, Mistral-7B-v0.1, and Llama-3.2-3B), achieving a 13-30% reduction in recall error at rank 32. The method improved 68-89% of heads per model, with the most significant gains observed in deep layers where query and key geometry diverge. The predicted score-MSE reduction closely matched the measured results, with a Pearson correlation of 0.997.
Implications
The implications of this research extend to enhancing the efficiency of KV retrieval systems in large language models, potentially leading to faster and more accurate responses in applications such as natural language processing and conversational AI. The findings may also inform future developments in data-aware indexing techniques.
Abstention as an Action Can Kill Both the Reward Gradient and the KL Anchor: Collapse Law and Repair for Error-Penalized Reinforcement Learning
Reinforcement Learning
Large Language Models
Theory
- Abstention as a discrete action can lead to the collapse of both reward gradients and KL anchors in reinforcement learning.
- The collapse occurs under specific conditions, including bounded readouts and the penalty for incorrect answers.
- A structural repair method is proposed, moving abstention to a confidence reporting mechanism that avoids shared saturation factors.
- Simulations and experiments confirm the predictions, demonstrating improved model performance through the proposed method.
Read more
Abstention as an Action Can Kill Both the Reward Gradient and the KL Anchor: Collapse Law and Repair for Error-Penalized Reinforcement Learning
Summary
This paper investigates the effects of error-penalized scoring rules in reinforcement learning, particularly focusing on the implications of abstention as an action. The authors demonstrate that when abstention is treated as a discrete action, both the reward gradient and the KL anchor can collapse under certain conditions, leading to a model that increasingly refuses to answer questions. This collapse is characterized by a mean training reward that approaches zero over time, despite the model appearing to improve. The authors identify the mechanism behind this phenomenon, which is linked to the saturation factor of the gate used for decision-making. They propose a structural repair by shifting abstention from the action space to a mandatory confidence report, which is trained to provide calibrated outputs without the risk of collapsing the reward gradient. Simulations and experiments on language models validate their findings, showing that the proposed method enhances coverage, accuracy, and calibration of the models.
Methodology
The authors analyze the behavior of reinforcement learning models under error-penalized scoring rules, focusing on the interaction between reward gradients and KL anchors. They derive mathematical conditions for collapse and propose a new training structure that involves mandatory confidence reporting. Simulations and experiments on language models are conducted to validate their theoretical findings.
Results
The study finds that under the specified conditions, models can drift towards complete abstention, with training rewards decreasing over time. The proposed repair mechanism effectively prevents this collapse, leading to improved coverage, accuracy, and calibration in language models. Experimental results corroborate the theoretical predictions, showing that models trained with the new approach maintain performance on solvable questions.
Implications
The findings suggest that reinforcement learning frameworks need to be carefully designed to avoid the pitfalls of abstention leading to performance collapse. The proposed confidence reporting mechanism could be applied to enhance the reliability of language models and other AI systems that require decision-making under uncertainty.
Unleashing the Potential of Large Language Models: A Blueprint for Real-Time, Enterprise-Ready Deployments
Large Language Models
NLP
Generative Models
- Introduction of a unified LLM operations (LLMOps) architecture for real-time deployments.
- Integration of real-time data ingestion and continual learning to address knowledge staleness and hallucinations.
- Framework designed to meet regulatory compliance and operational demands in high-risk sectors.
- Utilization of software design patterns to optimize latency, cost, and accuracy trade-offs.
Read more
Unleashing the Potential of Large Language Models: A Blueprint for Real-Time, Enterprise-Ready Deployments
Summary
This paper presents a novel architecture for deploying large language models (LLMs) in real-time enterprise settings, addressing critical limitations such as knowledge staleness, catastrophic forgetting, and hallucinations. The authors propose a comprehensive framework that integrates real-time data ingestion, continual learning, retrieval-augmented generation (RAG), and human-in-the-loop feedback into a single operational pipeline. This architecture aims to transform LLMs from static models into adaptive systems that can continuously learn and improve from new data and user feedback. The framework is designed to meet the operational demands of high-risk sectors, ensuring compliance with regulatory requirements while optimizing latency, cost, and accuracy. By utilizing established software design patterns, the proposed system enhances the reliability and consistency of LLM outputs, making them suitable for applications in healthcare, finance, and customer support, where timely and accurate information is critical.
Methodology
The authors developed a framework that incorporates adaptive ingestion patterns, continual learning mechanisms (STAR+FAR), retrieval-augmented generation (SAGE), and automated feedback loops. This architecture was validated through extensive experiments to produce Pareto-optimal configurations, ensuring a balance between latency, cost, and accuracy.
Results
The proposed framework demonstrated significant improvements in the adaptability and reliability of LLMs in real-time applications. It effectively mitigated issues related to knowledge staleness and hallucinations, resulting in enhanced performance in high-stakes environments. The architecture also facilitated compliance with regulatory standards, making it suitable for deployment in sectors requiring strict oversight.
Implications
The findings suggest that the proposed LLMOps architecture can significantly enhance the deployment of LLMs in enterprise contexts, particularly in fields where timely and accurate information is crucial. This could lead to improved decision-making in healthcare, finance, and customer support, ultimately transforming how organizations leverage AI technologies.
SAFE-Merge: Data-Free Continual Model Merging with General Knowledge Preservation
Computer Vision
NLP
Efficient ML
- SAFE-Merge introduces a risk-aware sparse masking approach to select safe parameter updates for merging.
- The framework employs masked low-rank recovery to restore task-specific information without altering masked parameters.
- It achieves the best H-score across multiple vision and language benchmarks, demonstrating effective continual merging.
- The method incurs no additional inference costs, making it efficient for real-world applications.
Read more
SAFE-Merge: Data-Free Continual Model Merging with General Knowledge Preservation
Summary
The paper introduces SAFE-Merge, a novel framework for data-free continual model merging that aims to preserve both pretrained general knowledge and previously acquired tasks without access to task data. Existing methods primarily focus on merging task updates while minimizing interference among downstream tasks, often neglecting the preservation of general knowledge. SAFE-Merge addresses this gap by implementing a risk-aware sparse masking technique that selectively retains parameter updates that are deemed safe for general knowledge. Additionally, it employs masked low-rank recovery to compensate for lost task information without modifying masked parameters. The framework is designed to incur no additional inference cost and has been evaluated across various vision and language benchmarks, consistently achieving superior performance in terms of H-score and accuracy compared to existing methods like NUFILT. The authors emphasize the importance of balancing the preservation of general knowledge with the acquisition of new task-specific information in a data-free setting.
Methodology
SAFE-Merge utilizes a parameter-space method that first employs risk-aware sparse masking to identify safe parameter updates. It contrasts the energy of parameter updates along pretrained general directions with task-specific directions to create a sparse binary mask. Following this, it implements a masked low-rank recovery term that restores lost task information while keeping masked parameters unchanged. The combined updates are then integrated into the model backbone without additional inference costs.
Results
SAFE-Merge consistently outperforms existing methods in terms of H-score across various benchmarks, including vision and language tasks. It shows significant improvements in accuracy, particularly on longer CLIP task sequences, compared to NUFILT, while maintaining a strong balance between continual merging accuracy and generalization to held-out distributions.
Implications
The proposed SAFE-Merge framework has significant implications for deploying specialized models in real-world applications where data privacy and availability are concerns. It allows for efficient model updates while preserving essential knowledge, making it suitable for scenarios such as personalized AI systems and continual learning environments.
Diffusion Policy with Behavioral Advantage Correction for Offline Reinforcement Learning
Reinforcement Learning
Generative Models
Robotics
- Introduces BAC-PE to correct Q-value estimations in offline RL.
- Theoretically analyzes the convergence of BAC-PE and provides an upper bound on Q-function differences.
- Employs diffusion models for effective policy regularization and distribution matching.
- DPBAC algorithm shows superior performance on D4RL tasks compared to existing methods.
Read more
Diffusion Policy with Behavioral Advantage Correction for Offline Reinforcement Learning
Summary
This paper addresses the challenges of offline reinforcement learning (RL), particularly the distribution shift between behavioral data and the learned policy, which can lead to inaccurate Q-value estimations. The authors propose a novel approach called Behavioral Advantage Corrected Policy Evaluation (BAC-PE) that utilizes the Q-function of the behavior policy to correct the learned policy's Q-function, thereby reducing pessimistic conservatism and overestimation bias. The convergence of BAC-PE is theoretically analyzed, providing an upper bound on the difference between the learned and true Q-functions. To further mitigate distribution shift, the authors employ diffusion models to represent both the behavior and learned policies, facilitating distribution matching for accurate policy regularization. The proposed algorithm, Diffusion Policy with Behavioral Advantage Correction (DPBAC), combines BAC-PE with diffusion policy modeling, demonstrating enhanced policy representation capabilities and effectively addressing Q-value estimation bias. Experimental results on various D4RL tasks indicate that DPBAC outperforms existing offline methods, showcasing significant advantages over state-of-the-art algorithms.
Methodology
The authors developed the BAC-PE approach to correct Q-value estimations by leveraging the Q-function of the behavior policy. They theoretically analyzed the convergence of BAC-PE and derived an upper bound on the difference between learned and true Q-functions. Diffusion models were utilized to represent both the behavior and learned policies, enabling effective distribution matching and policy regularization. The DPBAC algorithm integrates these components for improved policy optimization.
Results
Experimental evaluations on multiple D4RL tasks demonstrated that the DPBAC algorithm significantly outperforms existing offline reinforcement learning methods, achieving stronger policy representation and effectively mitigating biases in Q-value estimation.
Implications
The proposed methods can enhance the safety and efficiency of offline reinforcement learning applications, particularly in domains where real-time interaction is costly or risky. The findings may lead to more robust RL systems in robotics, autonomous driving, and other decision-making tasks.