AI-generated summaries
Today's ML research,
without the noise.
Daily summaries of the latest machine learning papers from arXiv, processed every 8 hours.
64
Papers today
8h
Update frequency
7
Days of history
Harness Continual Learning: Continual Adaptation Beyond Model Parameters
NLP
Multimodal
Theory
- Introduction of Harness Continual Learning (HCL) paradigm focusing on harness adaptation rather than model parameter changes.
- Identification of harness-level forgetting as a challenge in continual learning.
- Development of a framework with components for managing harness state and learning.
- Demonstration of capability accumulation and failure recovery in various experimental settings.
Read more
Harness Continual Learning: Continual Adaptation Beyond Model Parameters
Summary
This paper introduces Harness Continual Learning (HCL), a novel paradigm that shifts the focus of continual learning from model parameters to an external harness that adapts around a frozen foundation model. Traditional continual learning approaches primarily modify model parameters to retain learned behaviors, leading to issues like forgetting previous tasks. HCL addresses this by evolving a harness composed of prompts, memories, tools, skills, and routing rules, which can adapt and accumulate experience while the underlying model remains unchanged. The authors define 'harness-level forgetting' as the loss of previously reliable behaviors due to updates in the harness. They propose a framework consisting of four components: Task Interface, Experience Memory, Capability Map, and Adaptive Router, which collectively manage how an agent processes information and executes tasks. The evolution of the harness is governed by a Continual Optimizer that generates candidate updates based on feedback and a Continual Evaluator that assesses these candidates for performance improvement and retention of historical behaviors. Experimental results demonstrate that HCL can effectively accumulate capabilities and recover from failures across various tasks, showing significant performance gains over baseline methods while also revealing measurable harness-level forgetting. The findings suggest that the stability-plasticity trade-off can be explicitly managed, providing a new avenue for research in continual learning.
Methodology
The authors developed a framework for HCL that includes four main components: Task Interface, Experience Memory, Capability Map, and Adaptive Router. They implemented a two-part evolution process involving a Continual Optimizer to propose updates based on feedback and a Continual Evaluator to validate these updates against performance and retention criteria. The methodology was tested through experiments in textual reasoning, multimodal perception, and open-world interaction.
Results
The experimental results showed that HCL outperformed baseline methods by over 10% in various tasks, demonstrating effective capability accumulation and recovery from failures. The study also revealed measurable harness-level forgetting and confirmed that the historical-retention budget could influence the balance between adaptation and retention.
Implications
The findings of this research could lead to more robust continual learning systems that maintain previously learned behaviors while adapting to new tasks. This approach has potential applications in AI systems that require ongoing learning and adaptation without losing prior knowledge, such as in robotics, personal assistants, and interactive AI agents.
Complementary, Not Cumulative: Interaction Effects in Physics-Informed Neural Networks for Navier-Stokes Vortex Shedding
Theory
Optimization
Efficient ML
- Systematic evaluation of various PINN techniques reveals that many do not improve performance when used alone.
- The combination of periodic activations and causal weighting is identified as effective for learning vortex shedding.
- Adding more techniques beyond the effective combination can lead to performance degradation.
- The study highlights the nonlinear interactions between different PINN enhancements.
Read more
Complementary, Not Cumulative: Interaction Effects in Physics-Informed Neural Networks for Navier-Stokes Vortex Shedding
Summary
This paper investigates the interaction effects of various techniques used to enhance Physics-Informed Neural Networks (PINNs) in solving the Navier-Stokes equations for vortex shedding. While PINNs offer a promising alternative to traditional computational fluid dynamics (CFD) solvers by embedding governing equations into the training loss, the study reveals that many proposed enhancements, when tested in isolation, do not outperform a baseline model. The author systematically evaluates a range of architectural changes, activation functions, and loss reweighting schemes on the DFG/Schäfer–Turek unsteady cylinder wake benchmark. The key finding is that combining periodic (SIREN) activations with causal weighting significantly improves performance, achieving an average relative L2 error of 4.1% compared to an OpenFOAM reference solution. However, adding more techniques leads to performance degradation, indicating that effective interventions can interact nonlinearly and that more complex training strategies are not necessarily beneficial. This work emphasizes the importance of understanding the interactions between different PINN techniques rather than validating them in isolation.
Methodology
The author conducted a systematic evaluation of widely-used PINN techniques, including architectural changes, activation functions, and loss reweighting schemes, specifically on the DFG/Schäfer–Turek unsteady cylinder wake benchmark. The performance of these techniques was assessed in isolation and in combination to understand their interaction effects.
Results
The study found that while many techniques did not outperform the baseline when applied individually, the combination of periodic activations (SIREN) and causal weighting significantly improved the model's ability to reconstruct velocity and pressure fields, achieving a 4.1% average relative L2 error compared to a reference solution. However, the introduction of additional techniques led to catastrophic performance degradation, underscoring the complexity of interactions among enhancements.
Implications
The findings suggest that careful consideration of the interactions between different techniques is crucial for improving PINN performance in fluid dynamics applications. This could lead to more efficient and effective modeling of complex fluid behaviors in engineering and scientific research.
Multi-Source Wasserstein Distributionally Robust Graph Learning
Graph Learning
Optimization
Theory
- Introduces MS-WDRO framework for robust graph learning from multi-source data.
- Utilizes Wasserstein barycenter for effective fusion of heterogeneous data sources.
- Establishes non-asymptotic performance guarantees for the proposed method.
- Achieves superior performance in graph recovery and sample efficiency compared to existing methods.
Read more
Multi-Source Wasserstein Distributionally Robust Graph Learning
Summary
This paper addresses the challenge of network topology inference from graph signals, particularly in scenarios where target-domain samples are limited but heterogeneous source-domain data is plentiful. The authors introduce a novel framework called MS-WDRO (Multi-Source Wasserstein Distributionally Robust Optimization) that utilizes the Wasserstein metric to fuse diverse sources into a reliable nominal distribution while preserving their intrinsic geometries. By constructing a weighted Wasserstein barycenter and creating an ambiguity ball around it, the framework minimizes worst-case risk, leading to a regularized Laplacian estimator that is efficiently solved using an ADMM (Alternating Direction Method of Multipliers) scheme. The authors provide non-asymptotic guarantees, including a finite-sample concentration bound for the empirical barycenter and an out-of-sample excess risk bound that decays at a parametric rate. The proposed method is calibrated through an end-to-end differentiable architecture, allowing for data-adaptive hyperparameter tuning. Experimental results demonstrate that MS-WDRO significantly outperforms seven baseline methods in terms of graph recovery accuracy, sample efficiency, and diagnostic utility, particularly in scenarios with limited samples.
Methodology
The methodology involves constructing a weighted Wasserstein barycenter from heterogeneous source data, creating an ambiguity ball around this barycenter to account for uncertainty, and minimizing the worst-case risk using a regularized Laplacian estimator. The optimization is performed using a convergent ADMM scheme, and the hyperparameters are calibrated through a differentiable architecture.
Results
The MS-WDRO framework consistently outperformed seven baseline methods in experiments, showing significant improvements in graph recovery accuracy and sample efficiency, especially in scenarios with limited data. The results also included theoretical guarantees for the performance of the method.
Implications
The proposed framework has potential applications in various fields such as neuroimaging, sensor networks, and social network analysis, where accurate network topology inference is crucial for downstream tasks like anomaly detection and disease biomarker discovery.
Lévy Attention: Single-Pass Predictive Uncertainty for Continuous-Time Attention
Time Series
- Lévy Attention integrates uncertainty estimation directly into the attention mechanism, eliminating the need for separate models or additional passes.
- The mechanism uses a stochastic formulation that allows for closed-form computation of both predictions and their uncertainty.
- Experiments indicate that Lévy Attention maintains high accuracy while significantly improving uncertainty estimation, especially in sparse datasets.
- The method is computationally efficient, providing error bars and uncertainty metrics with minimal additional cost.
Read more
Lévy Attention: Single-Pass Predictive Uncertainty for Continuous-Time Attention
Summary
This paper introduces Lévy Attention, a novel attention mechanism designed to provide predictive uncertainty for irregularly-sampled time series data. Traditional models often fail to convey the trustworthiness of their predictions, especially when dealing with sparse data. Lévy Attention addresses this gap by integrating uncertainty estimation directly into the attention layer, allowing it to report both predictions and their associated uncertainty at no additional computational cost. The mechanism utilizes a stochastic formulation based on an inhomogeneous Poisson random measure, which enables the model to maintain a continuous (time × channel) index space. The output of Lévy Attention is a stochastic integral that captures both the evidence of the predictions and the disagreement among the attended values. The authors demonstrate that this approach not only retains the accuracy of existing models but also improves uncertainty estimation without requiring additional model complexity or training overhead. Experiments show that Lévy Attention outperforms traditional uncertainty estimation methods, particularly in sparse data scenarios, and provides valuable insights into the reliability of predictions.
Methodology
The authors propose a cross-attention operator that functions as a stochastic integral against an inhomogeneous Poisson random measure. This formulation allows for the simultaneous computation of predictions and uncertainty metrics during a single forward pass, leveraging the properties of the Poisson measure to capture evidence and disagreement in the predictions.
Results
Lévy Attention was tested against the t-PatchGNN benchmark, showing a maximum accuracy drop of only 5.6% compared to matched controls, with no loss on the sparsest dataset. The method also outperformed traditional uncertainty estimation techniques, achieving the best sparsification error in dense datasets and demonstrating effective calibration of uncertainty metrics.
Implications
The introduction of Lévy Attention has significant implications for applications involving irregularly-sampled time series data, such as clinical monitoring and sensor networks. By providing a reliable measure of uncertainty alongside predictions, it enhances decision-making processes in critical domains where trust in data is paramount.
Beyond Multimodal Alignment: Certifying Physical Language through Response Substitution and Ordered Execution
Multimodal
Robotics
Theory
- Introduces the Disjoint-Bridge Operator-Substitution Certificate (DBOSC) for certifying multimodal representations.
- Demonstrates that audio and acceleration modalities can interchangeably represent the same physical responses.
- Establishes a hierarchy of capabilities in multimodal learning, including response substitution and ordered execution.
- Shows that the executor's capability is critical for successful execution of programs in multimodal systems.
Read more
Beyond Multimodal Alignment: Certifying Physical Language through Response Substitution and Ordered Execution
Summary
This paper addresses the challenge of certifying the physical language of multimodal representations in world models, which serve as interfaces between perception and physical interaction. The authors introduce the Disjoint-Bridge Operator-Substitution Certificate (DBOSC), which evaluates whether independently trained modality compilers can interchangeably access a frozen response chart based on evidence outside their training panels. The study employs two experiments: the first demonstrates that audio and acceleration representations of unseen surfaces are significantly closer in response space than incorrect pairings, confirming the effectiveness of the shared response coordinate. The second experiment tests ordered execution in a controlled elastoplastic system, revealing that the DBOSC acts as an instrument rather than a verdict, with the executor's capability being crucial for successful execution. The findings establish a hierarchy of distinct achievements in multimodal learning, including attribute access, response substitution, fusion closure, and ordered execution, highlighting the importance of separating these capabilities for better understanding and application.
Methodology
The authors conducted two experiments: the first involved measuring the response space of audio and acceleration modalities on unseen surfaces to validate the DBOSC. The second experiment tested ordered execution in a controlled elastoplastic system, analyzing how the executor's capabilities affected the execution of multimodal programs.
Results
The results indicated that audio and acceleration modalities were 4.5 times closer in response space for the same unseen surface compared to incorrect pairings. In the elastoplastic system, the DBOSC successfully certified ordered execution once the executor was adequately trained, achieving an oracle NMSE of 0.18. Additionally, 14 out of 16 registered checks passed, with the failures attributed to a diagonal restriction in the fused information matrix.
Implications
The findings suggest that separating the capabilities of multimodal systems can lead to more effective designs and implementations in robotics and physical interaction tasks. The DBOSC can serve as a framework for future research in certifying multimodal representations and enhancing their reliability in practical applications.
DeltaMomentum: A Key-Value based Anisotropic Momentum Update via Delta Rule
Optimization
- DELTAMOMENTUM introduces a direction-aware momentum update that adapts forgetting rates based on the frequency of gradient directions.
- The method is built on a key-value structure derived from the gradients of linear layers, enhancing the efficiency of momentum updates.
- It provides theoretical guarantees of validity and performance, outperforming traditional EMA momentum in clearing stale directions.
- DELTAMOMENTUM is a drop-in replacement for existing momentum buffers, maintaining compatibility with other optimization techniques.
Read more
DeltaMomentum: A Key-Value based Anisotropic Momentum Update via Delta Rule
Summary
The paper introduces DELTAMOMENTUM, a novel momentum update method designed to address the anisotropic nature of gradient updates in deep learning. Traditional optimizers utilize an exponential moving average (EMA) for momentum, which applies a uniform forgetting rate across all directions, potentially leading to inefficiencies in training. DELTAMOMENTUM leverages a key-value structure inherent in the gradients of linear layers, treating the input as a key and the output-side error as a value. This approach allows for a direction-aware momentum update where the forgetting rate is dynamically adjusted based on how frequently a direction is queried. The authors prove that DELTAMOMENTUM is a valid momentum update, applies curvature corrections without matrix inversion, and clears stale directions faster than EMA. The method is compatible with existing optimizers and incurs minimal additional computational cost. Experimental results demonstrate that DELTAMOMENTUM significantly reduces the number of training steps required to achieve validation loss comparable to AdamW, with improvements observed across various model sizes and architectures.
Methodology
The authors propose DELTAMOMENTUM as a key-value based momentum update that utilizes the classical delta rule to refresh momentum based on the frequency of direction queries. The method is validated through theoretical proofs and empirical testing on language model pretraining tasks.
Results
In experiments, AdamW with DELTAMOMENTUM (DeltaAdamW) achieved validation loss in up to 46.39% fewer steps at 67M parameters and 22.12% fewer steps at 370M parameters compared to standard AdamW. The performance gains were consistent across different model sizes, including a 1B parameter model, and were also observed with SGD, ResNet-18, and ViT-Tiny on CIFAR-10.
Implications
The introduction of DELTAMOMENTUM could lead to more efficient training of large neural networks, allowing for faster convergence and better utilization of computational resources. Its compatibility with existing optimizers suggests it can be easily integrated into current training pipelines.
Improved Confidence Estimates for Black-Box Large Language Models
NLP
Large Language Models
Interpretability
- Proposes a novel framework for uncertainty quantification in LLMs using labeled datasets.
- Utilizes simple classifiers to predict response correctness based on uncertainty scores and similar query correctness.
- Demonstrates minimal computational overhead and flexibility in aggregating uncertainty scores.
- Empirical results show consistent improvement over existing UQ methods across various datasets.
Read more
Improved Confidence Estimates for Black-Box Large Language Models
Summary
This paper addresses the challenge of uncertainty quantification (UQ) in large language models (LLMs), which is crucial for their deployment in high-stakes environments. Existing UQ methods often rely on zero-shot approaches that do not utilize labeled data, leading to suboptimal performance. The authors propose a novel framework that leverages labeled datasets to enhance confidence estimates for LLM responses. By training simple classifiers that incorporate uncertainty scores and the correctness of similar queries as features, the proposed method significantly improves the reliability of uncertainty estimates. The approach is computationally efficient and flexible, allowing for the aggregation of multiple uncertainty scores into a single confidence estimate. Empirical results demonstrate that this data-driven method consistently outperforms existing UQ techniques across various datasets and models, providing calibrated confidence estimates tailored to specific tasks.
Methodology
The authors partition a labeled dataset into training and reference sets. They augment the training queries with uncertainty scores as features and extract neighborhood-based statistics from the reference set. A supervised binary classification task is then formulated to predict response correctness using these augmented features.
Results
The classifiers developed in this study consistently outperform existing UQ methods, providing calibrated confidence estimates. The results indicate that leveraging labeled data significantly enhances the reliability of uncertainty estimates, with performance varying based on the task-specific learner selected.
Implications
The findings suggest that incorporating labeled data into UQ frameworks can lead to safer and more reliable deployment of LLMs in critical applications such as healthcare and scientific research. This approach could facilitate better decision-making processes by providing more accurate confidence estimates.
Multi-Agent Off-Policy Deep Reinforcement Learning for Smart Campus Coverage
Reinforcement Learning
Optimization
- The optimization of BS placement in non-convex campus environments is NP-hard.
- Multi-Agent DDPG outperforms single-agent approaches in terms of coverage and fairness.
- The proposed framework achieves a Jain’s fairness index of 0.94 and guarantees full coverage.
- The study introduces a boundary projection mechanism to enhance optimization.
Read more
Multi-Agent Off-Policy Deep Reinforcement Learning for Smart Campus Coverage
Summary
This paper explores the deployment of millimeter-wave (mmWave) base stations (BSs) in a complex campus environment, addressing the NP-hard optimization problem of BS placement for maximizing user fairness and coverage. The authors propose a novel approach that formulates the BS placement as a Markov Decision Process (MDP) and evaluates four deep reinforcement learning (DRL) schemes: a discrete single-agent Deep Q-Network (DQN), a spatially partitioned Multi-Agent DQN, a continuous single-agent Deep Deterministic Policy Gradient (DDPG), and a geographically partitioned multi-agent DDPG framework. The study highlights the limitations of traditional optimization methods in non-convex environments and demonstrates that the multi-agent DDPG approach significantly outperforms single-agent methods, achieving full coverage and a Jain’s fairness index of 0.94 in dense user scenarios. The results indicate that the proposed multi-agent framework not only ensures optimal BS placement but also enhances computational efficiency in complex topologies.
Methodology
The authors formulated the BS placement problem as an MDP, utilizing both discrete and continuous DRL methods. They benchmarked four different DRL schemes, including single-agent and multi-agent architectures, to optimize the placement of BSs in a realistic campus topology. The reward function was designed to reflect max-min fairness, focusing on equitable user throughput.
Results
The multi-agent DDPG framework achieved superior performance, ensuring 100% coverage and maintaining minimum SNRs above 19 dB. The Jain’s fairness index reached 0.94, indicating a high level of user fairness in the network. The numerical evaluations demonstrated significant computational efficiency in dense user scenarios.
Implications
The findings suggest that DRL can effectively address complex optimization problems in wireless network deployment, particularly in urban environments. This research could inform future designs of next-generation wireless networks, enhancing connectivity and user experience in smart campuses and similar settings.
Answer-Level Trust Selection for Physical Vision-Language Reasoning
Multimodal
Computer Vision
Interpretability
- Introduces Answer-Level Trust Selection (ATS) for evaluating individual predictions of VLMs.
- ATS is model-agnostic and requires no fine-tuning or access to internal model logits.
- Utilizes eight behavioral diagnostic scores to generate a unified trust score for predictions.
- Demonstrates that ATS can identify hidden failure modes in VLM predictions.
Read more
Answer-Level Trust Selection for Physical Vision-Language Reasoning
Summary
This paper addresses the challenge of evaluating the reliability of individual predictions made by Vision-Language Models (VLMs) in the context of quantitative physical reasoning. While existing benchmarks focus on overall model performance against ground truth, the authors highlight the need for a mechanism to assess the trustworthiness of specific predictions when ground truth is unavailable. They introduce Answer-Level Trust Selection (ATS), a model-agnostic framework that aggregates eight interpretable behavioral diagnostic scores to produce a unified trust score for each prediction. ATS is designed to identify stable-but-wrong estimates and predictions that rely on textual priors rather than visual evidence. The authors conduct extensive evaluations on the Qwen2.5-VL-7B model and benchmark 20 VLMs to characterize their capabilities in quantitative reasoning. The results indicate that ATS effectively identifies hidden failure modes that traditional evaluation methods may overlook, although it may also lead to a reduction in the retention of correct predictions. This work emphasizes the importance of answer-level reliability assessment in the deployment of VLMs for real-world applications.
Methodology
The authors propose ATS as a post-hoc framework that aggregates eight interpretable behavioral diagnostic scores derived from repeated queries and controlled interventions. This approach allows for the assessment of individual predictions without modifying the original model or requiring additional verifiers.
Results
The evaluation of ATS on Qwen2.5-VL-7B and across 20 VLM backbones reveals its effectiveness in identifying stable-but-wrong predictions and those influenced by textual priors. While ATS improves the rejection of failure cases, it may also decrease the retention of correct predictions, highlighting a trade-off in its application.
Implications
The findings suggest that ATS can enhance the reliability of VLM outputs in practical applications, particularly in scenarios where ground truth is not available. This framework could be crucial for deploying VLMs in safety-critical systems that require accurate quantitative reasoning.
MileGPO: Milestone Inference with Local Evidence for Graph-Based Policy Optimization of Long-Horizon LLM Agents
Reinforcement Learning
Large Language Models
Optimization
- MileGPO reveals the limitations of final-goal-distance credit in distinguishing same-state branches.
- The proposed method learns from rollout graphs and on-policy rewards without needing external annotations or auxiliary inference.
- MileGPO achieves state-of-the-art performance on ALFWorld and WebShop benchmarks.
- The method effectively calibrates intermediate credit, leading to improved decision-making in long-horizon tasks.
Read more
MileGPO: Milestone Inference with Local Evidence for Graph-Based Policy Optimization of Long-Horizon LLM Agents
Summary
The paper introduces MileGPO, a novel approach to credit assignment in long-horizon reinforcement learning (RL) for large language model (LLM) agents. Traditional methods struggle with assigning credit to intermediate actions when supervision is limited to final rewards. MileGPO addresses this by deriving process-level credit from grouped on-policy rollouts through three main components: Milestone Discovery (MD), which identifies significant milestones and traps; Reliability-Calibrated Shaping (RCS), which weights these milestones based on their reliability; and Progress-Contrastive Calibration (PCC), which assesses local progress and compares transitions from the same state. The method does not require auxiliary models or additional environment interactions, making it efficient. Experimental results demonstrate that MileGPO achieves state-of-the-art performance on benchmarks ALFWorld and WebShop, with a notably small in-distribution to out-of-distribution gap, indicating strong generalization capabilities. The findings highlight the importance of reliable intermediate credit and the effectiveness of the proposed calibration mechanisms in improving agent performance.
Methodology
MileGPO employs a three-part methodology: Milestone Discovery (MD) identifies significant milestones and traps from successful and failed rollouts; Reliability-Calibrated Shaping (RCS) weights these milestones based on their empirical reliability; and Progress-Contrastive Calibration (PCC) evaluates local progress and compares sibling transitions to refine credit assignment. This approach leverages on-policy rollouts and does not require additional models or interactions with the environment.
Results
MileGPO demonstrated superior performance on the ALFWorld and WebShop benchmarks, achieving state-of-the-art results. The in-distribution to out-of-distribution gap on ALFWorld was only 1.69 points, outperforming previous methods such as GiGPO and GraphGPO. Further analysis indicated that the calibration mechanisms effectively resolved ambiguities in credit assignment, leading to more reliable performance.
Implications
The findings suggest that MileGPO can enhance the training of LLM agents in complex, long-horizon tasks, potentially improving their decision-making capabilities in real-world applications such as web navigation and embodied instruction following. The methodology may also inform future research on credit assignment in reinforcement learning.
Auditing Recorded Predictive Lead Service-Line Classifications Against Physical Verification: A Statewide Study of New York
Theory
- 49% of localities showed zero output variance in predictive classifications, raising concerns about model reliability.
- New York City reported a uniform classification for over 43,000 addresses, indicating potential issues with the predictive model.
- Physical verification revealed lead presence across all construction eras, contradicting the predictive model's findings.
- The study identifies significant gaps in the predictive modeling approach, necessitating more rigorous validation methods.
Read more
Auditing Recorded Predictive Lead Service-Line Classifications Against Physical Verification: A Statewide Study of New York
Summary
This paper investigates the reliability of predictive models used by water utilities in New York State to classify service line materials under the Lead and Copper Rule Revisions (LCRR). The study audits the outputs of these models against physical verification data, focusing on discrepancies and the implications of using statistical methods instead of direct inspections. The author screened 153 localities, finding that 49% of them recorded a single material value for service lines, with varying degrees of consistency with physical verification. Notably, New York City reported a uniform classification of 'Known Other' for over 43,000 addresses, raising concerns about the accuracy of the predictive model. The analysis reveals that while the model-cleared population is newer, it fails to identify lead in service lines, contrasting sharply with physical verification results that show lead presence across various construction eras. The findings highlight the limitations of predictive modeling in accurately assessing lead service lines and emphasize the need for physical verification to ensure public safety.
Methodology
The study employed a screening process across all 153 localities in New York that classified at least 100 addresses using predictive models. It compared model outputs against physical verification data, focusing on zero variance in classifications and consistency with physical checks. The analysis included spatial joins to assess construction eras and utilized various estimators to evaluate the expected number of lead lines.
Results
The analysis found that 49% of localities recorded a single material value, with only 7 showing contradictions against their own physical verification. New York City’s model classified all 43,215 addresses as 'Known Other,' while physical verification indicated lead presence in other areas. The study estimated that the model-cleared population contained approximately 1,150 to 1,450 expected lead lines, highlighting significant discrepancies between predictive modeling and actual physical conditions.
Implications
The findings suggest that reliance on predictive models without adequate physical verification can lead to underreporting of lead service lines, posing risks to public health. The study advocates for stricter adherence to verification protocols and improved model validation to ensure accurate assessments of water infrastructure.
SAGE-XGBoost: Spatially Augmented Graph Embeddings–Machine Learning Framework for Natural Hazards Susceptibility Mapping under Data Scarcity
Graph Learning
- SAGE-XGBoost improves natural hazard susceptibility mapping under data scarcity.
- The framework combines data augmentation with graph embeddings for feature enhancement.
- Significant performance improvements were observed over traditional machine learning models.
- The model achieved high AUC values for both landslide and wildfire susceptibility.
Read more
SAGE-XGBoost: Spatially Augmented Graph Embeddings–Machine Learning Framework for Natural Hazards Susceptibility Mapping under Data Scarcity
Summary
This paper addresses the challenge of natural hazard susceptibility mapping, which is often limited by the availability of labeled data. The authors propose a novel framework called SAGE (Spatially Augmented Graph Embeddings) that enhances prediction capabilities under data-scarce conditions. SAGE integrates controlled noise-based data augmentation with neighborhood-based graph embeddings to derive local spatial statistics. These statistics are processed using principal component analysis and combined with environmental covariates and spatial coordinates to create robust features for the XGBoost model. The SAGE-XGBoost model was evaluated for landslide and wildfire susceptibility mapping, demonstrating significant improvements over conventional machine learning models. The results showed an absolute improvement of over 33 percentage points compared to Spatial XGBoost, achieving AUC values of approximately 0.97 for landslide susceptibility and 0.95 for wildfire susceptibility. Feature importance analysis confirmed the value of graph embeddings in enhancing prediction accuracy and spatial coherence while minimizing local noise amplification. The study concludes that SAGE-XGBoost serves as an efficient and transferable alternative to deep learning methods for environmental hazard assessments, particularly in scenarios with limited supervision.
Methodology
The methodology involves constructing a K-nearest neighbor graph to derive local spatial statistics, which are then reduced using principal component analysis. These features are integrated with environmental covariates and spatial coordinates to train the XGBoost model, resulting in the SAGE-XGBoost framework.
Results
The SAGE-XGBoost model outperformed conventional and spatially explicit machine learning models, achieving an absolute improvement of over 33 percentage points compared to Spatial XGBoost. The model reached AUC values of approximately 0.97 for landslide susceptibility and 0.95 for wildfire susceptibility.
Implications
The SAGE-XGBoost framework offers a robust alternative for natural hazard susceptibility mapping, particularly in data-scarce environments. Its efficiency and transferability make it suitable for various geospatial prediction tasks and environmental hazard assessments.
FAR-DPO: Feasibility-Aware and Robust Direct Preference Optimization for Cyclic Peptide Design
Generative Models
Optimization
- FAR-DPO addresses the challenges of cyclic peptide design by focusing on feasibility and robustness.
- The framework integrates a novel preference construction strategy that accounts for structural and biophysical constraints.
- FAR-DPO shows significant improvements in the success rate of feasible cyclic peptide designs across multiple generative models.
- The methodology includes difficulty-aware optimization to ensure consistent performance across heterogeneous targets.
Read more
FAR-DPO: Feasibility-Aware and Robust Direct Preference Optimization for Cyclic Peptide Design
Summary
Cyclic peptides are gaining traction in drug discovery due to their high binding affinity and structural stability. However, designing cyclic peptides poses significant challenges due to the constraints imposed by cyclization on the feasible design space. Existing generative models often struggle with low yields of feasible designs and limited control over multi-objective trade-offs due to reliance on zero-shot generation or post hoc filtering. To address these issues, the authors propose FAR-DPO (Feasibility-Aware and Robust Direct Preference Optimization), an architecture-agnostic framework that enhances the design of cyclic peptides by integrating feasibility-aware preference construction with difficulty-aware group-robust optimization. FAR-DPO constructs preference pairs through feasibility-gated multi-objective dominance and adaptively reweights difficulty groups based on current preference losses. Evaluations on the CPSea LNR benchmark demonstrate that FAR-DPO significantly improves the success rate of feasible designs, increasing it from 46.89% to 57.79% on PepGLAD and from 47.96% to 49.57% on PepFlow, while also yielding better binding scores. These results indicate FAR-DPO's effectiveness in enhancing both feasibility and robustness across different targets.
Methodology
FAR-DPO employs a two-pronged approach: it constructs feasibility-aware preference pairs from generated cyclic peptides and implements a difficulty-aware optimization strategy that adjusts weights based on the performance of different difficulty groups. This allows the model to focus on improving areas where it currently underperforms, thus enhancing the overall quality of generated designs.
Results
FAR-DPO achieved an increase in the overall success rate of feasible cyclic peptide designs, with improvements from 46.89% to 57.79% on PepGLAD and from 47.96% to 49.57% on PepFlow. The framework also produced better binding scores, particularly for the most challenging targets, demonstrating its effectiveness in generating structurally and biophysically feasible cyclic peptides.
Implications
The development of FAR-DPO has significant implications for drug discovery, particularly in the design of cyclic peptides that can effectively modulate protein-protein interactions. By improving the feasibility and robustness of cyclic peptide designs, this framework could accelerate the identification of novel therapeutics and enhance the efficiency of the drug development process.
Green BOA: Determining the environmental break-even point for ML-based data compression
Efficient ML
- The environmental break-even point for ML-based data compression is sensitive to the carbon intensity of the energy source.
- ML-based compression algorithms can offer better compression ratios but may have higher carbon costs compared to traditional methods.
- The study provides a framework for assessing the carbon footprint of ML training and inference against storage savings.
- Future work will explore a broader range of scenarios and improve throughput for ML-based compression.
Read more
Green BOA: Determining the environmental break-even point for ML-based data compression
Summary
This paper presents the findings of two summer internship projects at the University of Manchester, which investigate the environmental break-even point for machine learning (ML)-based data compression algorithms. The authors focus on a specific ML-based lossless compression algorithm and analyze the carbon-equivalent costs associated with the infrastructure required for ML training and inference against the carbon-equivalent savings from reduced disk storage needs. The study highlights the significant environmental costs of big data experiments, particularly in contexts like the Large Hadron Collider, where data storage demands are immense. The authors employ a proof-of-principle approach to estimate energy use and carbon emissions for ML-based data compression, comparing it with traditional compression methods. They find that the break-even point is highly sensitive to the carbon intensity of the energy sources used in different countries. The results indicate that while ML-based compression can achieve better compression ratios, it may incur higher CO2 costs per unit of data processed compared to standard algorithms, raising questions about the environmental justification for deploying such methods at scale.
Methodology
The authors estimated the energy consumption of a specific ML-based data compression algorithm using an Nvidia T4 GPU, considering various country-specific energy mixes to calculate CO2-equivalent emissions. They compared these emissions to the carbon costs associated with traditional disk storage methods (HDDs and tapes) over a five-year lifespan, defining the break-even point as the dataset size where the carbon costs of ML compression equal the savings from reduced storage needs.
Results
The analysis revealed that the break-even point varies significantly based on the carbon intensity of the location where the ML-based compression is performed. The results indicated that while tape storage has a lower CO2 footprint than HDDs, it offers slower data access. The study also found that ML-based compression, while achieving better compression ratios, incurs higher CO2 costs per unit processed compared to standard algorithms.
Implications
The findings of this study have significant implications for the deployment of ML technologies in data-intensive fields, particularly in terms of environmental sustainability. It highlights the necessity for researchers and practitioners to weigh the benefits of improved data compression against the environmental costs, potentially influencing future research directions and policy decisions regarding ML applications.
Explainable Transformer Models for Clinical Prediction Tasks on Structured Electronic Health Records
NLP
Interpretability
- Introduction of BERT-LER, a transformer model that combines laboratory data representation with interpretability.
- Utilization of percentile-based binning for encoding laboratory results, preserving graded information.
- Demonstrated competitive performance on EHRShot and real-world asthma progression tasks.
- Token-level attributions using Integrated Gradients align with clinically known risk factors.
Read more
Explainable Transformer Models for Clinical Prediction Tasks on Structured Electronic Health Records
Summary
This paper presents BERT-LER, a novel BERT-style model designed for clinical prediction tasks using structured electronic health records (EHRs). The model addresses the challenge of incorporating continuous laboratory measurements into discrete token-based architectures while maintaining interpretability. BERT-LER encodes laboratory test results as discrete tokens through percentile-based binning, allowing it to retain graded information. The model is pretrained and fine-tuned on a large dataset of 75 million patients and evaluated on both the EHRShot benchmark suite and a real-world asthma severity progression study. The authors demonstrate that BERT-LER achieves competitive predictive performance, often exceeding existing benchmark models, particularly on laboratory-related tasks. Additionally, the use of Integrated Gradients for token-level attributions provides clinically relevant insights into model predictions, linking them back to input medical events. This work highlights the potential for integrating strong predictive performance with interpretable outputs in clinical settings, offering a unified framework for EHR modeling that can be applied across various therapeutic areas.
Methodology
The authors developed BERT-LER, a BERT-style model that encodes laboratory test results as discrete tokens using percentile-based binning. The model was pretrained on a large dataset and fine-tuned for specific clinical prediction tasks. Integrated Gradients were employed to generate token-level attributions, linking model predictions to input medical events.
Results
BERT-LER achieved predictive performance that is competitive with existing benchmark models, particularly excelling in tasks related to laboratory data. The model's attributions were clinically relevant, reflecting known risk factors in both benchmark and application-driven settings.
Implications
The findings suggest that BERT-LER can facilitate the integration of machine learning models into clinical workflows by providing interpretable predictions that are clinically meaningful. This approach may enhance decision-making in healthcare by improving the understanding of model outputs.
SCORE: Subject Coordinate Recovery for Label-Free Cross-Subject EEG-to-Image Retrieval
Multimodal
- SCORE identifies that different subjects express similar concept relationships in distinct coordinate directions.
- The framework allows for EEG-to-image retrieval without the need for target labels or encoder updates.
- SCORE achieves superior accuracy in cross-subject retrieval compared to existing methods.
- The method generalizes well across different EEG encoders and remains effective under various conditions.
Read more
SCORE: Subject Coordinate Recovery for Label-Free Cross-Subject EEG-to-Image Retrieval
Summary
The paper introduces SCORE (Subject Coordinate Recovery), a novel framework aimed at improving EEG-to-image retrieval across different subjects without the need for labeled calibration data. Traditional methods excel in within-subject scenarios but struggle with cross-subject retrieval, often yielding low accuracy due to the unique ways different individuals represent similar visual concepts. The authors analyze EEG features and discover that while subjects maintain similar relationships among concepts, they express these relationships along distinct coordinate axes. SCORE addresses this by aligning EEG data from source subjects with a common image space during training and employing a label-free coordinate recovery method at deployment. This approach allows SCORE to recover the target subject's EEG coordinates without requiring additional data or label updates. The framework was evaluated on two public benchmarks, THINGS-EEG2 and Alljoined-1.6M, demonstrating significant improvements in retrieval accuracy over existing methods, thereby facilitating practical applications of EEG-based visual decoding.
Methodology
SCORE combines recovery-aware source training with coordinate alignment during deployment. It aligns source subject EEG data with a common image space and simulates unseen-subject recovery through source-only episodes. At deployment, SCORE selects EEG-image landmarks using hubness-corrected matching and estimates an orthogonal transformation to recover target EEG coordinates without requiring source data or target labels.
Results
In 200-way retrieval tasks on the THINGS-EEG2 and Alljoined-1.6M benchmarks, SCORE achieved Top-1/Top-5 accuracies of 53.23%/83.55% and 12.01%/32.16%, respectively. These results surpass the strongest evaluated baselines by significant margins, demonstrating SCORE's effectiveness across all held-out subjects.
Implications
The SCORE framework has the potential to enhance EEG-based visual decoding applications, making them more accessible and practical for real-world use without the need for extensive calibration or subject-specific data. This could lead to advancements in fields such as brain-computer interfaces and neurocommunication.
Safe Domain Adaptation for Physics: Overcoming Nuisances, Label Shifts, and Simulation Priors
Theory
- Traditional domain adaptation methods assume that differences between training and target domains are nuisances, which is often not true in physics.
- The study reveals that standard adversarial domain adaptation can lead to biased predictions when energy spectra differ between domains.
- An adaptive domain adaptation method is proposed to focus on genuine physical mismatches by reweighting simulated events.
- A label-free model selection rule is introduced to help identify optimal training configurations without target labels.
Read more
Safe Domain Adaptation for Physics: Overcoming Nuisances, Label Shifts, and Simulation Priors
Summary
This paper addresses the challenges of applying domain adaptation (DA) techniques in physics, particularly when neural networks trained on simulations are used on experimental data. The authors highlight that traditional DA methods rely on the assumption that the differences between domains are merely nuisances, which is often not the case in physics. They present a systematic study using a toy air-shower benchmark that reveals three types of mismatches: physical simulation shifts, detector-response nuisances, and energy-spectrum shifts. The standard adversarial domain adaptation methods struggle when the energy spectra differ, leading to biased predictions. To overcome these issues, the authors propose an adaptive domain adaptation approach that reweights simulated events to focus on genuine physical mismatches. Additionally, they introduce a label-free model selection rule to identify optimal training configurations without relying on target labels. The findings indicate that while DA can be effective when spectra are aligned, it can introduce significant biases when they are not, necessitating the proposed adaptive methods.
Methodology
The authors conducted a systematic study using a toy air-shower benchmark to analyze the effects of different types of mismatches on domain adaptation. They compared standard adversarial domain adaptation techniques with their proposed adaptive domain adaptation approach, which reweights simulated events to focus on genuine physical mismatches. They also developed a label-free model selection rule for optimal training configuration identification.
Results
The results demonstrated that standard domain adversarial neural networks perform well when the energy spectra of the source and target domains are aligned. However, when the spectra differ, these methods tend to bias the predictions towards the simulation prior. The proposed adaptive domain adaptation method effectively mitigated these biases by focusing on the actual physical mismatches, leading to more accurate predictions.
Implications
The findings have significant implications for the application of machine learning in physics, particularly in areas where simulations are used to analyze experimental data. The proposed methods can enhance the robustness of neural networks against systematic shifts, improving the reliability of predictions in experimental settings.
NanoSleep: A Parameter-Efficient Hybrid Temporal Convolutional Network for Single-Channel Sleep Stage Classification
Efficient ML
Time Series
- NanoSleep is a compact hybrid architecture for sleep stage classification with only 0.35 million parameters.
- The model effectively combines multi-scale temporal and spectral feature extraction.
- A weighted calibrated focal loss is used to address class imbalance in sleep stage datasets.
- NanoSleep outperforms several state-of-the-art methods while being efficient for deployment on resource-constrained devices.
Read more
NanoSleep: A Parameter-Efficient Hybrid Temporal Convolutional Network for Single-Channel Sleep Stage Classification
Summary
The paper introduces NanoSleep, a compact hybrid temporal convolutional network designed for single-channel sleep stage classification using electroencephalography (EEG). The authors address the challenges of deploying deep learning models on resource-constrained devices, such as high model sizes and class imbalance in sleep stage datasets. NanoSleep integrates a learnable Sinc-convolutional front end, a dual-branch feature extractor for multi-scale temporal and spectral representations, a gated dilated temporal convolutional backbone with channel recalibration, and a conditional random field (CRF) for sequence-level decoding. The model employs a weighted calibrated focal loss to mitigate class imbalance. Evaluated on the Sleep-EDF and Sleep-EDF-Expanded datasets through subject-wise cross-validation, NanoSleep consistently outperforms six baseline methods while maintaining a low parameter count of only 0.35 million trainable parameters. The ablation study conducted confirms the contribution of each component, demonstrating that NanoSleep achieves a balance between accuracy and efficiency, making it suitable for wearable devices and home-based sleep monitoring.
Methodology
NanoSleep employs a hybrid architecture that includes a learnable Sinc-convolutional front end, a dual-branch feature extractor for capturing multi-scale temporal and spectral features, a gated dilated temporal convolutional backbone with channel recalibration, and a CRF for sequence-level predictions. The training framework incorporates imbalance-aware optimization strategies and a weighted calibrated focal loss to enhance performance on minority classes.
Results
The experimental results show that NanoSleep consistently outperforms six baseline models on the Sleep-EDF and Sleep-EDF-Expanded datasets, achieving high classification accuracy with significantly fewer parameters. The ablation study confirms the effectiveness of each component in the architecture.
Implications
NanoSleep's design allows for efficient deployment in wearable devices and home-based sleep monitoring systems, potentially improving accessibility to sleep health assessments. Its ability to maintain high accuracy while being resource-efficient opens avenues for clinical applications in sleep disorder diagnosis and management.
In Two Minds about Lifelong Learning: Exploring Hemispheric Redundancy and Specialisation in Neural Models
Theory
Generative Models
Efficient ML
- Introduction of 4MAS, a dual-hemisphere architecture for continual learning.
- Utilization of a wake-sleep training cycle to enhance memory consolidation.
- Competitive accuracy on Split-MNIST (98.3%), Split-Fashion-MNIST (84.9%), and Split-CIFAR-100 (29.29%).
- Demonstration of low representational drift across tasks.
Read more
In Two Minds about Lifelong Learning: Exploring Hemispheric Redundancy and Specialisation in Neural Models
Summary
This paper addresses the challenge of catastrophic forgetting in machine learning, particularly in the context of continual learning. The authors propose a novel architecture called 4MAS (4 Module Awake/Sleep), inspired by biological learning mechanisms, which incorporates two asymmetric hemispheres that specialize in different memory functions. The architecture mimics the biological processes of experience replay, REM sleep, and bilateral redundancy found in mammalian brains. By implementing a wake-sleep training cycle, the model allows for cross-hemispheric consolidation of knowledge, thereby stabilizing latent representations and minimizing representational drift. The paper presents empirical results demonstrating that 4MAS achieves competitive performance on several benchmark datasets while effectively mitigating forgetting, showcasing the potential of biologically-inspired approaches in enhancing machine learning systems.
Methodology
The authors developed the 4MAS architecture, which consists of two hemispheres with dedicated short- and long-term memory modules. The model alternates between an awake phase for task learning and a sleep phase for cross-hemispheric memory consolidation. This approach leverages generative replay and mimics biological processes such as hippocampal replay and REM sleep to stabilize memory and reduce forgetting.
Results
The 4MAS architecture achieved high accuracy rates on benchmark datasets: 98.3% on Split-MNIST, 84.9% on Split-Fashion-MNIST, and 29.29% on Split-CIFAR-100. The results indicate that the model effectively mitigates catastrophic forgetting while maintaining competitive performance.
Implications
The findings suggest that incorporating biological principles into machine learning architectures can significantly enhance their ability to learn continuously without forgetting. This has potential applications in developing more robust AI systems capable of adapting to new information over time, which is crucial for real-world applications.
A FEM-Based Surrogate Modelling and Optimization Framework for Physics-Constrained Electromagnetic Coil Design
Optimization
- Introduces a FEM-based surrogate modeling framework for electromagnetic coil design.
- Demonstrates the impact of FEM evaluation budget on optimizer performance.
- Compares multiple optimization methods, revealing strengths and weaknesses based on budget constraints.
- Provides an automated Python-MPh-COMSOL workflow for efficient design exploration.
Read more
A FEM-Based Surrogate Modelling and Optimization Framework for Physics-Constrained Electromagnetic Coil Design
Summary
This paper presents a framework for surrogate-assisted optimization of a seven-parameter current-excited coil-core design, constrained by geometric, manufacturing, and mass limitations. The methodology integrates a two-dimensional axisymmetric finite-element method (FEM) model with a Matérn-5/2 Gaussian-process (GP) surrogate model, facilitating efficient exploration of the design space. The study emphasizes the importance of the available FEM evaluation budget on optimizer performance, comparing Bayesian optimization (BO) with expected improvement (EI) against other methods such as COBYLA and BOBYQA. Results indicate that while EI-BO shows rapid improvement with small budgets, COBYLA performs better at early checkpoints, and BOBYQA achieves the highest mean terminal response. The findings suggest that different optimization methods may excel under varying conditions, highlighting the need for careful consideration of budget and performance metrics in simulation-driven design. The contributions include an automated workflow for geometry and constraint handling, diagnostics for prediction accuracy, and a comparative analysis of optimizer performance under budget constraints.
Methodology
The study employs a Python-MPh-COMSOL workflow that integrates a finite-element method (FEM) model with a Gaussian-process (GP) surrogate model. It utilizes sequential Bayesian optimization with expected improvement (EI) to rank design candidates, while also comparing other optimization methods like COBYLA and BOBYQA under controlled conditions.
Results
The results show that EI-BO is effective with small evaluation budgets, while COBYLA is superior at early checkpoints, and BOBYQA yields the best overall mean terminal response. A retrospective analysis indicates that no single optimizer consistently outperforms others across all metrics, emphasizing the context-dependent nature of optimization performance.
Implications
This framework can be applied to other simulation-driven design problems in engineering, particularly where computational resources are limited. The findings may influence the selection of optimization strategies in similar contexts, guiding practitioners in balancing early progress with terminal performance.
CLaST: Context-aware Contrastive VAE for Probabilistic Time Series Forecasting
Generative Models
Time Series
Optimization
- CLaST introduces a VAE framework that preserves contextual similarity for improved time series forecasting.
- The concept of Lag-Invariant Non-stationary Time Series (LINTS) is introduced to model temporal dependencies effectively.
- The contrastive loss function enhances training stability and representation quality compared to traditional mutual information estimators.
- CLaST achieves up to 16.4% improvement in CRPS and 14.4% in NMAE for short-term forecasting, and up to 48.6% and 25.1% improvements for long-term forecasting.
Read more
CLaST: Context-aware Contrastive VAE for Probabilistic Time Series Forecasting
Summary
The paper introduces CLaST, a novel variational autoencoder (VAE) framework designed for probabilistic multivariate time series forecasting. Traditional forecasting models often struggle to capture the intricate temporal dependencies within data, leading to suboptimal latent representations. CLaST addresses this issue by employing a contrastive loss function that preserves contextual similarity between observations. The authors define a new class of time series called Lag-Invariant Non-stationary Time Series (LINTS), which allows for realistic temporal dynamics while maintaining a single constraint on the dependence structure. The proposed method enhances the stability of training and improves the quality of learned representations. Extensive experiments across nine benchmark datasets demonstrate that CLaST outperforms existing strong baseline methods, achieving significant improvements in short-term and long-term forecasting tasks. This work highlights the importance of contextual similarity in enhancing the performance of probabilistic forecasting models.
Methodology
CLaST employs a variational autoencoder architecture that integrates a contrastive loss function to maintain contextual similarity between time series observations. It introduces the LINTS framework to model temporal dependencies effectively, allowing for a more stable training process and better representation learning.
Results
CLaST consistently outperformed strong baseline methods across nine benchmark datasets, achieving up to 16.4% improvement in Continuous Ranked Probability Score (CRPS) and 14.4% in Normalized Mean Absolute Error (NMAE) for short-term forecasting. For long-term predictions, it surpassed the second-best method by up to 48.6% in CRPS and 25.1% in NMAE.
Implications
The findings suggest that incorporating contextual similarity into probabilistic forecasting models can significantly enhance their predictive performance. This approach can be applied across various domains, including finance, energy systems, and healthcare, where accurate time series forecasting is crucial.
Vector Symbolic Policy Gradient
Reinforcement Learning
Theory
Efficient ML
- Introduction of VSPG, a discrete-action policy-gradient actor using unit-norm hypervectors.
- Exact vector-symbolic interpretation of the policy-gradient update as advantage-weighted bundling.
- Creation of fixed-size compressed kernel memories for efficient learning and inference.
- Demonstration of competitive performance and sample efficiency across multiple benchmarks.
Read more
Vector Symbolic Policy Gradient
Summary
The paper introduces Vector-Symbolic Policy Gradient (VSPG), a novel discrete-action policy-gradient actor that utilizes Vector Symbolic Architecture (VSA) to represent actions as unit-norm hypervectors. The authors establish a direct connection between VSA and discrete-action policy gradients, demonstrating that the standard softmax policy-gradient update can be interpreted as a vector-symbolic operation involving advantage-weighted bundling and normalization. This approach allows VSPG to create fixed-size compressed kernel memories that efficiently transfer advantage evidence across similar states without retaining past samples during inference. The authors evaluate VSPG across various benchmarks, including classic control tasks, MiniGrid, and multi-agent SustainGym, showing that it achieves competitive performance and superior sample efficiency compared to traditional neural and linear actor baselines. Notably, VSPG exhibits robust performance under conditions of post-training quantization and random bit flips, making it suitable for deployment in unreliable edge systems. The paper highlights the potential of VSPG to enhance the robustness and efficiency of reinforcement learning agents in real-world applications.
Methodology
The authors employ Vector Symbolic Architecture (VSA) to represent actions as hypervectors and derive the policy-gradient update as a vector-symbolic operation. They utilize advantage-weighted bundling and normalization to update the action representations, ensuring that the method is computationally efficient and robust against noise and quantization.
Results
VSPG achieves competitive returns on classic control tasks, MiniGrid, and multi-agent SustainGym, outperforming traditional neural and linear actors in terms of sample efficiency. The method also demonstrates significantly better performance under conditions of post-training bit-flip corruption, indicating its robustness compared to existing approaches.
Implications
The findings suggest that VSPG could be effectively deployed in resource-constrained environments, such as edge computing systems, where reliability and efficiency are critical. This work opens avenues for further exploration of VSA in reinforcement learning and other domains requiring robust decision-making under uncertainty.
Multi-stage neural operator learning with application for convolutions
Theory
Efficient ML
- Introduction of two multi-stage neural operator learning frameworks: DCNO (supervised) and DGNO (unsupervised).
- Both frameworks improve approximation accuracy by progressively constructing neural operator bases.
- Theoretical analysis supports the convergence and approximation capabilities of the proposed methods.
- Numerical experiments show that the methods achieve high accuracy and efficiency in convolution problems.
Read more
Multi-stage neural operator learning with application for convolutions
Summary
This paper presents two innovative multi-stage neural operator learning frameworks aimed at enhancing the efficiency and accuracy of convolution computations, which are prevalent in various scientific and engineering applications. The first framework, Deep Collocation Neural Operator (DCNO), is a supervised learning method that iteratively refines operator approximations by learning from residuals derived from input-output data pairs. The second framework, Deep Galerkin Neural Operator (DGNO), operates in an unsupervised manner, leveraging the weak form of partial differential equations (PDEs) to train the operator when the target can be represented as a PDE. Both frameworks progressively construct basis operators through multiple training stages, significantly improving approximation accuracy compared to traditional one-shot operator learning approaches. The authors provide theoretical analyses supporting the approximation capabilities of both methods and apply them specifically to convolution problems. Extensive numerical experiments demonstrate that DCNO and DGNO achieve high accuracy, nearing machine precision for convolution tasks, while also offering substantial efficiency gains in scenarios involving numerous queries or parametric variations. The frameworks are further extended to accommodate multi-input operator learning scenarios, showcasing their versatility and robustness in handling variations in both density and kernel of convolutions.
Methodology
The paper proposes two frameworks: DCNO, which uses supervised learning to refine operator approximations through residual learning from data pairs, and DGNO, which employs an unsupervised approach based on the weak form of PDEs to train the operator. Both methods involve multiple training stages to enrich the approximation space, leading to improved accuracy over traditional one-shot learning methods.
Results
The numerical experiments indicate that both DCNO and DGNO achieve high accuracy, approaching machine precision for convolution tasks. The frameworks also provide significant efficiency improvements when handling multiple queries or variations in parameters compared to traditional solvers.
Implications
The proposed frameworks can significantly enhance the computational efficiency and accuracy of convolution evaluations in various applications, such as electrostatics, gravitational simulations, and signal processing. Their ability to handle multi-input scenarios further broadens their applicability in complex scientific computing problems.
A Real-Time Tsetlin Machine-based Non-intrusive Load Monitoring System on MCUs
Efficient ML
Time Series
- Introduction of a Tsetlin Machine-based framework for NILM on MCUs.
- Achieves high precision and recall rates for appliance classification.
- Designed for real-time processing with minimal memory footprint.
- Open-source implementation available for further research and development.
Read more
A Real-Time Tsetlin Machine-based Non-intrusive Load Monitoring System on MCUs
Summary
This paper presents a novel Non-Intrusive Load Monitoring (NILM) system utilizing a Tsetlin Machine (TM) framework, specifically designed for real-time applications on resource-constrained microcontrollers (MCUs). Traditional NILM systems face challenges due to their reliance on computationally intensive algorithms, which limits their deployment in real-time scenarios where data privacy is paramount. The proposed TM-based NILM system reformulates the energy disaggregation problem as a classification task, allowing for efficient processing of household energy consumption data. The system comprises four main components: Edge Detection, Event Pairing, Feature Extraction, and TM-based classification. By employing an event-driven architecture, the system minimizes computational overhead, processing only when significant power transitions are detected. The TM model, known for its lightweight and memory-efficient characteristics, enables the system to achieve an average precision of 90% and recall of 96% for two-appliance classification, and 77% precision and 80% recall for four appliances on the REDD dataset. Furthermore, the model occupies only 18 KB of flash memory and achieves an inference latency of 0.43 ms on an ESP32, demonstrating its suitability for embedded applications.
Methodology
The methodology involves reformulating the NILM problem as a classification task using the Tsetlin Machine. The system processes raw sensor data through Edge Detection to identify power transitions, followed by Event Pairing to group these transitions into complete appliance activity windows. Features are extracted from these windows and classified using the TM model, which operates on binary input features, ensuring computational efficiency suitable for MCUs.
Results
The TM-based NILM system achieved an average precision of 90% and recall of 96% for two-appliance classification, and 77% precision and 80% recall for four appliances on the REDD dataset. The model's memory usage is only 18 KB, with an inference latency of 0.43 ms on an ESP32 microcontroller, indicating its effectiveness for real-time applications.
Implications
The proposed system has significant implications for energy management in smart homes, enabling efficient monitoring of appliance energy consumption while preserving user privacy. Its lightweight design allows for deployment in various embedded systems, potentially leading to widespread adoption in energy-efficient technologies.
Online Test-Time Adaptation for Generalizable Dynamic Graph Anomaly Detection
Graph Learning
- Introduces OTTA-DGAD for online test-time adaptation in dynamic graph anomaly detection.
- Utilizes dynamic prototypes to capture evolving normal and anomalous patterns.
- Implements a memory buffer that retains both general and domain-specific patterns.
- Demonstrates superior performance on diverse datasets compared to existing methods.
Read more
Online Test-Time Adaptation for Generalizable Dynamic Graph Anomaly Detection
Summary
This paper addresses the challenge of dynamic graph anomaly detection (DGAD) in unseen target domains without the need for costly retraining. The authors propose a novel method called OTTA-DGAD, which incorporates online test-time adaptation to effectively handle sequentially arriving unlabeled data chunks. OTTA-DGAD extracts dynamic prototypes from temporal ego-graphs and stores them in a memory buffer that retains general patterns from source domains while integrating new domain-specific patterns. An anomaly scorer compares incoming edge representations against these prototypes to detect both general and domain-specific anomalies. The method also updates the memory buffer using reliable pseudo-labels and enriches each target chunk with relevant representations from previous chunks to mitigate information loss. Extensive experiments demonstrate that OTTA-DGAD achieves state-of-the-art performance across ten real-world datasets, showcasing its effectiveness in adapting to evolving anomaly patterns in dynamic graphs.
Methodology
The methodology involves extracting dynamic prototypes from temporal ego-graphs and storing them in a memory buffer. This buffer selectively retains general patterns from source domains while incorporating new patterns from the target domain. An anomaly scorer compares incoming edge representations against these prototypes. The model updates the memory buffer using reliable pseudo-labels and enriches target chunks with relevant representations from previous chunks to address information loss.
Results
OTTA-DGAD outperforms existing methods in dynamic graph anomaly detection across ten real-world datasets, demonstrating its ability to adapt to continuous domain shifts and effectively capture evolving anomaly patterns.
Implications
The proposed method has significant implications for real-world applications such as fraud detection, transaction monitoring, and social network security, where timely and accurate anomaly detection in dynamic environments is crucial.
Adaptive Probabilistic Shielding by Learning MDPs for Safe Reinforcement Learning
Reinforcement Learning
- Introduces adaptive probabilistic shielding for safe reinforcement learning.
- Combines online model learning with MDPs to estimate transition probabilities.
- Addresses the exploration-exploitation dilemma in the context of safety.
- Empirical evaluations show improved performance in various environments.
Read more
Adaptive Probabilistic Shielding by Learning MDPs for Safe Reinforcement Learning
Summary
This paper addresses the challenge of safe reinforcement learning (RL) by introducing an adaptive probabilistic shielding technique that integrates online model learning with Markov decision processes (MDPs). Traditional probabilistic shielding relies on known transition probabilities to constrain the actions of an RL agent to ensure safety during exploration. However, in many real-world scenarios, these probabilities are not available. The authors propose a method where the transition graph of the MDP is known, but the transition probabilities are estimated as the agent interacts with the environment. This approach allows for the computation of a shield that adapts as the model becomes more accurate. The paper discusses the challenges of when to recompute the shield and how to balance exploration with safety. The authors empirically evaluate their method across various environments, demonstrating that the adaptive shielding can effectively reduce the conservativeness of the shield over time, allowing for safer exploration and improved learning outcomes.
Methodology
The authors developed a model estimator that learns the transition probabilities of an MDP as the RL agent interacts with the environment. They utilized interval MDPs (iMDPs) to capture uncertainty in these estimates, allowing for the computation of safety probabilities using probabilistic model checking. The shield is updated adaptively based on the evolving model estimates, and the authors explored different strategies to balance exploration and safety.
Results
The experimental results indicate that the adaptive shielding approach effectively reduces the conservativeness of the shield over time, leading to improved exploration and learning efficiency. The method was tested across multiple environments, showcasing its robustness and adaptability in ensuring safe RL.
Implications
This research has significant implications for the development of safe RL systems in real-world applications, where uncertainty and incomplete information are prevalent. The adaptive shielding technique can enhance the safety of autonomous agents in various domains, including robotics and automated decision-making systems.
When to Retrain: An Empirical Study of Retraining Policies for Streaming ML Under Concept Drift, Budget, and Latency Constraints
Theory
Efficient ML
Time Series
- Incremental learning significantly impacts the effectiveness of retraining policies.
- Periodic retraining outperforms reactive policies under abrupt and gradual drift.
- Reactive policies exhibit systematic failures, including budget exhaustion and drift blindness.
- A latency-budget interaction can reduce effective retraining budgets by half.
Read more
When to Retrain: An Empirical Study of Retraining Policies for Streaming ML Under Concept Drift, Budget, and Latency Constraints
Summary
This paper investigates the critical decision of when to retrain machine learning models in production systems facing concept drift, particularly under constraints of budget and latency. The study presents a controlled empirical analysis of three retraining policies—periodic retraining, error-threshold triggering, and statistical drift-triggered retraining using ADWIN—against a no-retrain baseline. The experiments were conducted across 3,933 runs involving three drift scenarios (abrupt, gradual, recurring), varying budget levels, latency constraints, and two datasets (LUFlow network-intrusion detection and LendingClub loan default). The findings reveal that the most significant factor influencing model performance is whether the deployed model can learn incrementally. For models with incremental updates, no retraining policy significantly outperformed the no-retrain baseline. However, without incremental learning, periodic retraining proved superior, particularly under abrupt and gradual drift conditions. The study also highlights the limitations of reactive policies, such as budget exhaustion and inefficiencies in drift detection, and introduces a latency-budget interaction that can halve effective retraining budgets. The paper concludes with the release of a comprehensive simulator and datasets for reproducibility.
Methodology
The authors developed an open-source streaming simulator that incorporates various drift scenarios, retraining policies, and explicit budget and latency constraints. They conducted a factorial analysis of the policies across different datasets and drift types, employing statistical significance testing to evaluate the performance of each policy.
Results
The results indicate that with incremental updates, no retraining policy significantly outperformed the no-retrain baseline. In contrast, without incremental updates, periodic retraining led to accuracy improvements of 15–55 percentage points compared to reactive policies, which only excelled under recurring drift conditions. Systematic failures of reactive policies were documented, particularly regarding budget management and drift detection.
Implications
The findings suggest that organizations should prioritize incremental learning capabilities in their ML systems to enhance performance under concept drift. Additionally, the study provides actionable insights for developing retraining strategies that optimize budget and latency constraints, potentially improving the robustness of production ML systems.
Quantifying Event Impacts on Time Series via Multiscale Contrastive Learning
Time Series
- Introduces EventTime, a novel framework for estimating financial impacts of cybersecurity events.
- Incorporates multi-resolution market context and event metadata for enhanced prediction.
- Utilizes a dynamic contrastive learning approach to address sparse supervision challenges.
- Demonstrates superior performance over state-of-the-art models in estimating abnormal losses.
Read more
Quantifying Event Impacts on Time Series via Multiscale Contrastive Learning
Summary
This paper addresses the challenge of estimating the short-term abnormal losses in financial time series resulting from discrete external events, particularly cybersecurity incidents. The authors introduce EventTime, a multi-resolution framework that integrates long-horizon market context, short-horizon pre-event dynamics, and event metadata to enhance prediction accuracy. A key innovation is the event fusion module, which couples temporal representations with event attributes to identify relevant market patterns. To tackle the issue of sparse supervision, the authors propose a dynamic contrastive objective that generates event- and time-series-aware positive and negative pairs during training. They also present SECURE, a dataset that aligns cybersecurity incidents with stock-market time series and various semantic features. Experimental results demonstrate that EventTime consistently outperforms existing time-series and event-aware models in estimating post-event financial losses, showcasing improved event sensitivity, robustness to incomplete metadata, and interpretability in market impact assessments.
Methodology
The methodology involves a multi-resolution framework that combines various temporal scales of market data with event metadata. The event fusion module integrates these elements to identify relevant market patterns, while the dynamic contrastive objective creates training pairs that enhance the model's ability to learn from sparse data.
Results
EventTime outperformed existing models in estimating post-event financial losses, showing more sensitivity to events, greater robustness to incomplete metadata, and improved interpretability of market impacts following cybersecurity disclosures.
Implications
The findings have significant implications for investors, firms, and regulators by providing a more accurate assessment of cyber-risk and enabling better portfolio protection strategies. The model can also enhance understanding of how cybersecurity incidents propagate through financial markets.
When Does Dynamic Ensembling Pay Off? Diagnosing Regionwise Gains in Regression under Distribution Shift
Theory
Optimization
Efficient ML
- Introduction of bDCF5, a diagnostic tool for assessing dynamic ensembling gains in regression under distribution shifts.
- High predictive accuracy of bDCF5 with a Spearman correlation of +0.98 across diverse dataset-shift pairs.
- Development of the Probe-Validated Ensemble Selector to choose between static and dynamic models based on validated gains.
- Demonstrated significant risk reduction in test scenarios using dynamic ensembling compared to static methods.
Read more
When Does Dynamic Ensembling Pay Off? Diagnosing Regionwise Gains in Regression under Distribution Shift
Summary
This paper investigates the effectiveness of dynamic ensembling in regression tasks under distribution shifts, where the input-dependent combination of models may outperform static blends. The authors introduce a diagnostic tool, bDCF5, which estimates the potential gains from reallocating trust across different regions of the input space based on a small labeled sample from the target domain. The study evaluates 12 dataset-shift pairs, demonstrating that bDCF5 accurately predicts regionwise gains with a high Spearman correlation of +0.98. The paper also presents the Probe-Validated Ensemble Selector, which decides between static and dynamic models based on the validated gains from the probe. The results indicate that dynamic ensembling can significantly reduce test risk, achieving improvements of 11% and 16% over static methods in two deployments. Additionally, the authors release OpenRegShift, a reproducible evaluation framework for regression ensembles under distribution shifts, enhancing the accessibility of their findings.
Methodology
The authors utilize a labeled probe from the target domain to learn a partition of the input space, comparing the performance of static and regionwise convex blends. The bDCF5 diagnostic is cross-fitted to estimate the realizable gains from dynamic ensembling. A controlled generator is employed to analyze the conditions under which dynamic gains occur, and the Probe-Validated Ensemble Selector is used to validate and select the best model for deployment.
Results
The bDCF5 diagnostic predicts regionwise gains with a Spearman correlation of +0.98, and a sensitivity analysis maintains a correlation of +0.83. The Probe-Validated Ensemble Selector successfully matched or improved upon static performance in all 12 runs, achieving risk reductions of 11% and 16% in two deployments. The gate mechanism effectively rejected candidates that would have incurred excessive losses.
Implications
The findings suggest that dynamic ensembling can be a powerful strategy in regression tasks, particularly in scenarios with distribution shifts. The developed tools and methodologies can assist practitioners in making informed decisions about model deployment, potentially leading to more robust and accurate predictions in real-world applications.
Credit Without Ground Truth: Auditing Step-Level Credit Assignment in LLM Agents Against Executed Replay
Reinforcement Learning
Large Language Models
Theory
- Existing step-level credit signals do not effectively identify causally important steps in LLM agents.
- Causal contributions are sparse and vary significantly between different model architectures.
- Implicit credit scores correlate more with policy fluency than with actual causal contributions.
- A confidence-only routing rule can reduce costs while maintaining performance at chance levels.
Read more
Credit Without Ground Truth: Auditing Step-Level Credit Assignment in LLM Agents Against Executed Replay
Summary
This paper investigates the effectiveness of step-level credit assignment signals used in training large language model (LLM) agents, specifically in the context of reinforcement learning. The author audits various credit signals, including LLM-judge scores and outcome-conditioned log-probability ratios, against a causal ground truth derived from executed replay in a single-agent tool environment (ALFWorld). The findings reveal that none of the evaluated credit signals can reliably identify which steps causally contribute to outcomes better than chance. The study highlights that causal contributions are sparse, with only 30.5% of decision points showing measurable effects, and that the ability to measure these contributions is model-dependent. The paper also identifies that implicit credit tends to reflect the fluency of the policy rather than its causal effectiveness. A novel confidence-only routing decision rule is proposed, which reduces judge costs while maintaining chance-level pivotal recall. The contributions of this work include a comprehensive measurement of causal ground truth, an audit of existing credit signals, and the introduction of protocols for credit comparisons.
Methodology
The methodology involves executing replay in a controlled environment (ALFWorld) to assess the causal contribution of each decision point by re-sampling alternative actions and measuring shifts in outcome distributions. This approach allows for the establishment of a causal ground truth against which various credit signals can be audited.
Results
The audit revealed that none of the credit signals outperformed their own shuffled controls, indicating that they do not provide meaningful insights into causal contributions. The analysis showed that the implicit credit scores were primarily reflective of the model's fluency, and the proposed confidence-only routing rule effectively reduced judge costs without improving the identification of pivotal steps.
Implications
The findings suggest that current methods for credit assignment in LLM training may need to be reevaluated and improved to ensure that they accurately reflect causal contributions. This could lead to more effective training strategies and better performance of LLM agents in complex decision-making tasks.
Transportable Causal Effect Estimation across Networks under Interference
Graph Learning
Theory
- Introduces TranCE, a novel algorithm for causal effect estimation across different networks under interference.
- Establishes graphical conditions for causal effect transportability using a selection diagram.
- Derives transport formulas for direct, spillover, and total effects in the deployment population.
- Demonstrates the effectiveness of the proposed method through extensive experiments on real-world data.
Read more
Transportable Causal Effect Estimation across Networks under Interference
Summary
This paper addresses the challenge of estimating causal effects across different networks where interference occurs, a scenario often encountered in social sciences and public health. Traditional methods assume that the training and deployment networks are the same, which is rarely the case in practice. The authors propose a novel algorithm called TranCE (Transported Causal Effects) that utilizes a selection diagram to model covariate and structural shifts separately. This approach leads to the derivation of transport formulas for direct, spillover, and total effects in the target deployment population. The TranCE algorithm combines an interventional outcome model with a domain density-ratio correction and cross-fitted inference to ensure robustness. The effectiveness of TranCE is validated through extensive experiments on semi-synthetic benchmarks and a real-world field experiment, demonstrating its capability to accurately transport causal effects from one network to another, thereby enhancing intervention strategies in various applications.
Methodology
The authors extend a selection diagram to the network context to separately model covariate and structural shifts. They derive transport formulas for causal effects and implement these using a doubly robust approach that combines an interventional outcome model with a domain density-ratio correction and cross-fitted inference.
Results
The experiments conducted on semi-synthetic benchmarks and a real-world weather-insurance field experiment show that TranCE effectively estimates causal effects in the deployment population, with results aligning closely with held-out randomized estimates, thereby validating the proposed methodology.
Implications
The findings of this research have significant implications for improving intervention strategies in networked systems, particularly in fields such as social networks and public health, where understanding causal effects across different populations is crucial.
Atrial Fibrillation Detection with Arbitrary Leads via a Codebook-Based Reconstruction-Classification Framework
Time Series
Generative Models
Graph Learning
- DCGCNet introduces a novel codebook-based framework for joint classification and reconstruction of ECG signals.
- The model incorporates a Local-Global Contrastive Module to enhance noise resilience.
- DCGCNet demonstrates exceptional performance with AUC > 0.98 across multiple datasets and conditions.
- The approach effectively addresses the limitations of existing AF detection methods by integrating reconstruction and classification tasks.
Read more
Atrial Fibrillation Detection with Arbitrary Leads via a Codebook-Based Reconstruction-Classification Framework
Summary
This paper addresses the challenge of detecting atrial fibrillation (AF) from electrocardiogram (ECG) signals, particularly in real-world clinical settings where lead configurations can vary significantly. The authors propose the Dual-Codebook Graph Collaborative Network (DCGCNet), an innovative deep learning model that integrates classification and reconstruction tasks into a unified framework. The model employs a codebook-based approach that includes a Local-Global Contrastive Module for enhancing noise resilience and an Adaptive Codebook Vector Quantizer to refine codebook prototypes dynamically. This dual-codebook system allows for effective learning of both local discriminative patterns and global signal fidelity, improving the robustness and generalization of AF detection. The results demonstrate that DCGCNet achieves state-of-the-art performance in both intra-dataset evaluations and cross-dataset generalization, consistently achieving an AUC greater than 0.98 across various testing conditions, including noisy environments. The findings suggest that DCGCNet is a promising solution for reliable AF detection, with significant implications for clinical practice.
Methodology
The authors developed DCGCNet, an end-to-end vector-quantized variational autoencoder that utilizes a shared codebook for joint learning of ECG signal reconstruction and AF classification. Key components include a Local-Global Contrastive Module for noise-invariant representation learning and an Adaptive Codebook Vector Quantizer for dynamic refinement of codebook prototypes.
Results
DCGCNet achieved state-of-the-art performance in standard intra-dataset evaluations and demonstrated strong cross-dataset generalization, consistently attaining an AUC greater than 0.98 across seven diverse settings. The model maintained high diagnostic accuracy even under realistic noisy conditions, such as baseline wander and EMG artifacts.
Implications
The proposed framework has significant implications for the development of automated AF detection systems, potentially facilitating early clinical intervention and improving patient outcomes in real-world settings. Its robustness against noise and adaptability to varying lead configurations make it suitable for large-scale screening and monitoring.
DICS: Data-Informed Centroid Splitting for Decision Tree Classifiers
Efficient ML
Interpretability
Theory
- DICS leverages clustering to create a compact set of candidate splits, enhancing computational efficiency.
- Theoretical analysis shows that DICS maintains predictive performance compared to exhaustive split search.
- DICS can be applied to various tree-based models, including random forests and gradient-boosting machines.
- Extensive experiments validate that DICS achieves similar accuracy while significantly reducing training time.
Read more
DICS: Data-Informed Centroid Splitting for Decision Tree Classifiers
Summary
The paper introduces Data-Informed Centroid Splitting (DICS), a novel approach aimed at improving the computational efficiency of decision tree classifiers. Traditional decision tree training involves exhaustive searches over candidate splits at each node, which can be computationally expensive, especially for large and high-dimensional datasets. DICS addresses this issue by utilizing a clustering-based framework to create a compact set of candidate splits informed by the data's structure. This method significantly reduces the search space for splits while maintaining predictive performance. The authors provide a theoretical analysis confirming that DICS does not degrade the performance of classification trees compared to exhaustive search methods. DICS can be integrated into various tree-based models, including classification trees, random forests, and gradient-boosting models. Extensive experiments demonstrate that DICS achieves comparable accuracy to traditional methods while substantially reducing training time across both synthetic and benchmark datasets, showcasing the advantages of incorporating data-informed priors into the split selection process for scalable classification tree learning.
Methodology
The methodology involves clustering the data to derive a reduced set of candidate splits for decision trees. By focusing on class-aware structures, DICS constructs splits that are both compact and informative, thereby accelerating the training process. The authors also provide a theoretical framework to support the efficacy of DICS in preserving classification performance.
Results
The experiments conducted demonstrate that DICS achieves comparable accuracy to traditional exhaustive search methods while significantly improving training efficiency. This was validated across various synthetic and benchmark datasets, indicating the robustness of the approach.
Implications
The DICS framework has the potential to enhance the scalability and efficiency of decision tree-based models in real-world applications, making them more feasible for large and complex datasets. This could lead to broader adoption of decision trees in various domains such as spam filtering, fraud detection, and targeted advertising.
Reliable Neural Collapse Approximation for Open-World Test-Time Adaptation
Computer Vision
Theory
Efficient ML
- Introduction of neural collapse concept to Open-World Test-Time Adaptation (OWTTA).
- Development of the Reliable Neural Collapse approximation (ReNC) method for effective adaptation.
- Utilization of classifier weights as source domain prototypes to filter OOD samples.
- Demonstration of ReNC's superiority through extensive experiments on open-world benchmarks.
Read more
Reliable Neural Collapse Approximation for Open-World Test-Time Adaptation
Summary
This paper addresses the challenges of Test-Time Adaptation (TTA) in open-world scenarios, where label distribution shifts can occur, rendering traditional TTA methods ineffective. The authors propose a novel method called Reliable Neural Collapse approximation (ReNC) for Open-World Test-Time Adaptation (OWTTA). ReNC leverages the concept of neural collapse as a structural prior to facilitate reliable adaptation to the target domain. The method justifies that pre-trained classifier weights can serve as prototypes for the source domain. By measuring the similarity between target samples and these prototypes, ReNC effectively filters out Out-Of-Distribution (OOD) samples, ensuring that updates are based on reliable in-distribution (ID) samples. Additionally, the authors introduce a neural collapse approximation mechanism that refines these prototypes, allowing them to adapt progressively to the target domain while preserving the neural collapse structure. Extensive experiments conducted on various open-world benchmarks demonstrate the superiority of ReNC over existing methods, highlighting its ability to maintain neural collapse properties in the target domain and providing insights for future model design.
Methodology
The authors propose ReNC, which approximates the neural collapse structure in the target domain. The method involves a parameter-free mechanism to filter OOD samples based on their similarity to prototypes. It also formulates a progressive adaptation process that updates prototypes while ensuring the adaptation of target features towards their corresponding prototypes, thus maintaining the neural collapse geometry.
Results
The experiments show that ReNC outperforms existing OWTTA methods, effectively preserving neural collapse properties in the target domain. This leads to improved model performance under conditions of label and data distribution shifts.
Implications
ReNC provides a reliable framework for adapting models in open-world scenarios, which can be particularly beneficial in fields like medical imaging where label distribution shifts are common. The insights gained from this study can inform future research and development of more robust machine learning models.
FleetSieve: Decision-Critical Profiling for SLO-Aware LLM Fleet Configuration
Large Language Models
Optimization
Efficient ML
- FleetSieve focuses on decision-critical profiling to optimize LLM fleet configurations.
- It models capacity and tail latency jointly to avoid infeasible configurations.
- The method reduces profiling GPU-seconds significantly compared to traditional methods.
- FleetSieve provides a three-state decision certificate indicating feasibility.
Read more
FleetSieve: Decision-Critical Profiling for SLO-Aware LLM Fleet Configuration
Summary
The paper introduces FleetSieve, a novel approach for optimizing the configuration of large language model (LLM) serving fleets by addressing the complexities of tensor-parallel (TP) degrees and replica counts. Traditional exhaustive profiling methods are inefficient as they often measure configurations that do not impact final resource allocation. FleetSieve improves upon this by selecting measurements based on their expected influence on resource-coupled, service-level objective (SLO)-aware decisions. It jointly models capacity and tail latency, allowing it to compare conservative and optimistic resource allocations and to cease profiling when the decision gap is within a specified tolerance. Evaluations on a 31B-parameter model demonstrate that FleetSieve achieves significant reductions in GPU-seconds required for profiling, outperforming uniform random profiling by 6.9% and achieving a mean saving of 5.4% across various configurations. The method also ensures that configurations do not violate SLO constraints, thus enhancing the reliability of LLM serving systems under varying loads.
Methodology
FleetSieve employs a decision-focused profiling approach that prioritizes measurements based on their predicted effect on resource allocation decisions. It maintains uncertainty over capacity and tail latency, evaluates configurations under both conservative and optimistic scenarios, and stops profiling when the decision gap is minimal. The method uses a three-state result system to classify configurations as certified feasible, undecided, or certified infeasible.
Results
FleetSieve achieved a reduction of 22,200 GPU-seconds for profiling on a fixed H100 measurement grid, which is 6.9% less than uniform random profiling. Across 200 random reveal orders, the mean saving was 5.4%, with a notable 21.5% saving for chat workloads. The method successfully avoided configurations that would violate SLO constraints, demonstrating its effectiveness in managing resource allocation under varying loads.
Implications
FleetSieve's approach can significantly enhance the efficiency of LLM serving systems, particularly in environments with diverse request types and stringent latency requirements. Its decision-critical profiling can be applied to other resource-constrained systems, potentially improving performance and resource utilization across various applications.
Towards On-Board Implementation of ML-Based Helicopter Weight Estimator
Time Series
Optimization
Robotics
- Development of a supervised ML model for helicopter weight estimation.
- Alignment with EASA and Eurocae ED-324 standards for safety assurance.
- Implementation of an LSTM architecture for the weight estimator.
- Verification of the model on legacy avionics systems for practical deployment.
Read more
Towards On-Board Implementation of ML-Based Helicopter Weight Estimator
Summary
This paper presents the development and implementation of a supervised Machine Learning (ML) model aimed at estimating helicopter weight during takeoff, utilizing extensive datasets from Airbus's global fleet. The authors detail a learning assurance process that aligns with the European Union Aviation Safety Agency (EASA) guidelines and the ongoing Eurocae ED-324 standards. The proposed methodology includes defining Machine Learning Requirements (MLCR), a Machine Learning Model Description (MLMD), and the implementation of a Long Short-Term Memory (LSTM) recurrent neural network. The implementation is verified on legacy avionics computers, demonstrating its suitability for deployment in airborne systems for critical functions like on-board alerting. The paper emphasizes the importance of accurate weight estimation for enhancing safety and reducing operational costs, particularly in Condition-Based Maintenance (CBM) frameworks. The authors also address the challenges of ensuring compliance with safety standards and the verification of the ML model's performance in real-world environments.
Methodology
The methodology involves a W-shape development process that includes defining ML requirements, processing data, and implementing a recurrent neural network (LSTM). The authors specify MLCR and Data Processing Requirements (DPR), build the MLMD from the trained model, and verify the implementation through high-level tests on deployed targets.
Results
The implementation of the ML-based weight estimator was successfully demonstrated on legacy avionics computers, showing that it meets the defined requirements and is suitable for critical on-board functions. The verification process confirmed the model's performance and compliance with safety standards.
Implications
The successful implementation of this ML-based weight estimator could lead to enhanced safety in helicopter operations, improved Condition-Based Maintenance practices, and reduced operational costs. It also sets a precedent for integrating ML technologies into airborne systems while adhering to stringent safety regulations.
A Locally Tokenized Generative Model for Robust Time-Series Watermarking
Generative Models
Time Series
- Introduces L-VQVAE, a generative model that relies on local tokenization for time-series data.
- Proposes LVQMark, a watermarking method that enhances robustness against post-editing attacks.
- Demonstrates that traditional watermarking methods suffer from false positives due to global re-encoding instability.
- Achieves stable detection power and reduced false-positive rates in attacked non-watermarked samples.
Read more
A Locally Tokenized Generative Model for Robust Time-Series Watermarking
Summary
This paper addresses the challenges of watermarking multivariate time series data, particularly in the context of post-editing attacks that can compromise the reliability of existing watermarking methods. The authors identify that traditional detectors, which utilize globally coupled re-encoding, are susceptible to bidirectional drift of the null distribution, leading to unreliable detection of watermarked samples. To overcome this issue, they propose a new generative model called L-VQVAE, which generates discrete tokens based solely on a bounded temporal neighborhood of the observed signal. This model is paired with a watermarking method, LVQMark, that employs logit-bias injection and robust re-encoding to enhance detection during attacks. The proposed methods were evaluated on four datasets across various domains, demonstrating that they maintain high generation quality while improving detection power and controlling false positives under post-editing attacks.
Methodology
The authors developed L-VQVAE, which generates discrete tokens from a short contiguous window of the time series data, rather than relying on a global representation. LVQMark incorporates a red-green logit bias during the autoregressive sampling process and utilizes a robust encoder to map attacked signals back to their clean token assignments for effective detection.
Results
The experiments showed that LVQMark successfully preserves high generation quality while achieving robust watermark detection. It also demonstrated stable false-positive behavior on non-watermarked samples subjected to post-editing attacks, significantly improving upon existing methods.
Implications
The findings suggest that the proposed methods can enhance the reliability of watermarking in generative models, particularly for applications in finance, energy, and neuroimaging where provenance verification is critical. This could lead to more secure and trustworthy synthetic data generation.
H$^2$EDL: Hyper Evidential Deep Learning for Hierarchical Classification
Computer Vision
Theory
Interpretability
- H2EDL provides a unified framework for uncertainty representation in hierarchical classification.
- The model uses a taxonomy to create a linear-size focal family for hyper-evidential beliefs.
- H2EDL reduces calibration error by approximately half compared to traditional methods.
- The model maintains hierarchical coherence and exposes uncertainty at each decision level.
Read more
H$^2$EDL: Hyper Evidential Deep Learning for Hierarchical Classification
Summary
The paper introduces H2EDL, a novel approach for hierarchical classification that addresses the challenge of uncertainty representation in fine-grained recognition tasks. Traditional methods either provide a single measure of uncertainty at the leaf level or fail to incorporate hierarchical structures effectively. H2EDL leverages the taxonomy of classes to create a hyper-evidential model that captures uncertainty across different levels of a hierarchical label space. By assigning local Dirichlet opinions at each branching node, the model maintains a coherent structure while allowing for uncertainty to be expressed at multiple levels. This dual interpretation enables H2EDL to function both as a hierarchical classifier and as a probabilistic model that reflects belief and uncertainty. The authors demonstrate that H2EDL significantly reduces calibration error compared to cross-entropy baselines, particularly at deeper levels of the hierarchy, while also improving the preservation of coarse category predictions during fine-grained mistakes. The model's design eliminates the need for composite labels and auxiliary structures, making it a more efficient and principled solution for hierarchical classification tasks.
Methodology
H2EDL employs a hyper-evidential framework that assigns local Dirichlet opinions at each branching node of a hierarchical taxonomy. This allows for the computation of leaf probabilities through a sequence of local decisions, capturing uncertainty at multiple levels of the hierarchy. The model is trained using standard leaf supervision, eliminating the need for composite labels or auxiliary networks.
Results
The experimental results show that H2EDL reduces calibration error significantly, especially at deeper hierarchy levels, and achieves similar leaf-level accuracy while preserving the correct coarse category 19% more often when fine-grained mistakes occur. This demonstrates the model's effectiveness in managing uncertainty in hierarchical classification tasks.
Implications
H2EDL has potential applications in various domains requiring hierarchical classification, such as fine-grained image recognition, natural language processing tasks with hierarchical labels, and any scenario where uncertainty representation is crucial for decision-making.
Infrared Universality of Collective Dynamics across Transformer and State-Space Architectures
Theory
Large Language Models
NLP
- Investigates collective dynamics in Transformers and Mamba, a state-space model.
- Finds that both architectures exhibit similar infrared organization despite different microscopic mechanisms.
- Demonstrates that Mamba's long-sequence exponent stabilizes near βM ≃−0.17, indicating long-memory dynamics.
- Extends the understanding of collective dynamics in neural networks beyond Transformers.
Read more
Infrared Universality of Collective Dynamics across Transformer and State-Space Architectures
Summary
This paper investigates whether different neural architectures exhibit common collective dynamics, focusing on Transformers and a state-space model called Mamba. Previous studies on Transformers revealed a weakly infrared-enhanced time-scale density of states (TDOS) associated with long-memory dynamics. The author examines Mamba, which employs a distinct microscopic mechanism for state-space dynamics, allowing for relaxation dynamics to be analyzed at three levels: the intrinsic spectrum of the learned state-space generator, input-conditioned selective rescaling, and the collective TDOS from its Jacobian. The findings indicate that while the microscopic dynamics differ, both architectures develop a similar slow-mode continuum in their collective dynamics. The cumulative spectral analysis shows that the long-sequence Mamba exponent stabilizes near βM ≃−0.17, corresponding to long-memory dynamics close to the marginal 1/t regime. In comparison, Transformers exhibit a related infrared organization with exponents around βTr ∼−0.1. These results suggest that distinct sequence architectures can develop closely related near-marginal slow-mode dynamics, extending the understanding of infrared collective organization beyond Transformers and providing insights into the dynamical structure described by Cognitive Field Theory.
Methodology
The study employs spectral analysis to characterize the relaxation dynamics of the Mamba model at multiple levels, including the intrinsic state-space generator and the collective TDOS measured from its Jacobian. The analysis compares these findings with the dynamics of Transformer models to assess similarities and differences in their collective behavior.
Results
The analysis reveals that Mamba develops a reproducible slow-mode continuum with a well-resolved infrared sector as sequence length increases. The cumulative spectral analysis yields a weakly negative exponent for Mamba, stabilizing near βM ≃−0.17, while Transformers show a related organization with βTr ∼−0.1. This indicates that despite different underlying mechanisms, both architectures exhibit similar collective dynamics.
Implications
The findings suggest that common dynamical principles may emerge across different neural architectures, which could influence the design and understanding of future models in sequence processing and language modeling. This work also supports the theoretical framework of Cognitive Field Theory in explaining collective dynamics in neural networks.
Exact Algebraic Computation of Learning Coefficients for Two-Dimensional Singular Models
Theory
- Introduces a deterministic algorithm for exact computation of local RLCTs in two-dimensional models.
- Demonstrates the limitations of classical information criteria in singular models and proposes WBIC as a solution.
- Reveals that effective model complexity can decrease with depth in polynomial neural networks, challenging conventional wisdom.
- Provides a complexity bound for the proposed algorithm, ensuring its applicability across a broad class of models.
Read more
Exact Algebraic Computation of Learning Coefficients for Two-Dimensional Singular Models
Summary
This paper addresses the limitations of classical information criteria, such as the Bayesian Information Criterion (BIC), in the context of singular models commonly encountered in deep learning. The authors introduce the Widely Applicable Bayesian Information Criterion (WBIC), which utilizes local learning coefficients (λ) to accurately capture the asymptotic behavior of marginal likelihoods. The main contribution is the development of the first deterministic algorithm capable of exactly computing local Real Log Canonical Thresholds (RLCT) for any two-dimensional model whose Kullback-Leibler divergence is contact equivalent to a polynomial. This advancement overcomes the reliance on sampling-based methods, which have been the standard for estimating learning coefficients. The authors provide a complexity bound for their algorithm and demonstrate its effectiveness across various models, including polynomial neural networks (PNNs). The findings reveal that the effective model complexity, as indicated by the RLCT, can decrease with increasing model depth, suggesting a nuanced understanding of identifiability in two-dimensional cases. The exact computation of RLCTs not only serves as a benchmark for sampling methods but also unveils algebraic structures in learning coefficients that are otherwise obscured.
Methodology
The authors developed a deterministic algorithm to compute local RLCTs for two-dimensional models. They derived an upper bound on the algorithm's arithmetic complexity and validated its effectiveness through applications to polynomial neural networks. The methodology involved algebraic geometry principles to resolve singularities in the parameter space, allowing for exact computations rather than relying on sampling methods.
Results
The proposed algorithm successfully computes local RLCTs for various two-dimensional models, including polynomial neural networks. The results indicate that the effective model complexity can decrease with the number of layers in PNNs, suggesting a complex relationship between model depth and identifiability. The exact computations provide a new benchmark for calibrating sampling-based estimators and reveal algebraic structures in learning coefficients.
Implications
The findings have significant implications for model selection in deep learning, particularly in understanding the behavior of singular models. The exact computation of learning coefficients can enhance the reliability of model selection criteria, leading to better performance in practical applications. Additionally, the insights into the relationship between model complexity and depth can inform the design of neural network architectures.
Score the Algebra, Not the Span: Dimension Reduction for Transfer Operator Models of Dynamical Systems
Theory
- Traditional spectral methods can lead to linear masking, omitting entire components from models of dynamical systems.
- The proposed method focuses on scoring the σ-algebra generated by coordinates, allowing for more efficient representation.
- A finite budget of coordinates can capture the entire spectrum of the transfer operator, avoiding the pitfalls of rank-based methods.
- The approach is validated through benchmark systems, demonstrating superior performance in recovering masked components.
Read more
Score the Algebra, Not the Span: Dimension Reduction for Transfer Operator Models of Dynamical Systems
Summary
This paper addresses the challenges of dimension reduction in dynamical systems, particularly those composed of weakly interacting components. Traditional methods, which rely on spectral approaches to model the transfer (Koopman) operator, often lead to linear masking, where entire components are omitted from the model. The authors propose a novel approach that focuses on scoring the σ-algebra generated by the coordinates rather than the span of the modes. This method allows for a more efficient representation of the dynamics, capturing the entire spectrum of the operator with significantly fewer coordinates. The proposed criterion, based on χ2-divergence between the present and future states, ensures that twice the intrinsic dimension of the dynamics suffices for a complete embedding. The authors demonstrate the effectiveness of their approach through various benchmark systems, showing that while rank-based methods may fail to capture masked components, their algebra-based representation can successfully predict these components from limited data.
Methodology
The authors propose a new dimension reduction approach that scores the σ-algebra generated by the coordinates instead of the span of modes. They formalize this through a criterion based on χ2-divergence, allowing for the use of existing estimators to compute the optimal embedding. The method is tested on various benchmark systems to evaluate its effectiveness in capturing the dynamics of multicomponent systems.
Results
The results show that the proposed algebra-based representation can recover all components of the system, even when rank-based methods fail to capture them at lower ranks. Specifically, the authors demonstrate that a representation using ten algebra coordinates can successfully predict masked components, while traditional methods miss these components entirely.
Implications
This work has significant implications for modeling complex dynamical systems in physical and biological contexts, where weakly interacting components are common. The ability to capture the full dynamics with fewer coordinates can lead to more efficient modeling and better predictive capabilities, particularly in scenarios with limited data.
Scale-Aware Pretraining of Time Series Foundation Models via Multi-Patch Token Alignment and Hybrid Masking
Time Series
- SATS introduces a scale-aware token alignment mechanism for time series pretraining.
- The method preserves scale-specific expressiveness while unifying token spaces across patch scales.
- A hybrid masking strategy is designed to capture both fine-grained and long-range temporal dependencies.
- SATS demonstrates superior performance on LSTF benchmarks compared to existing methods.
Read more
Scale-Aware Pretraining of Time Series Foundation Models via Multi-Patch Token Alignment and Hybrid Masking
Summary
This paper presents SATS, a novel approach for pretraining time series foundation models that effectively addresses the challenges posed by heterogeneous datasets with varying sampling frequencies. Traditional methods either use dataset-specific patch sizes leading to fragmented representations or enforce a fixed patch size that overlooks temporal variations. SATS introduces a scale-aware token alignment mechanism that treats patch size as a scale, aligning representation spaces across different scales while maintaining distinct modeling capacities. Additionally, a hybrid masking strategy is employed, combining random and contiguous masking to capture multi-scale temporal structures. Experimental results on LSTF benchmarks show that SATS outperforms competitive baselines, achieving a 9.2% improvement in Mean Squared Error (MSE) and an 8.3% gain in GIFT-Eval MASE, while also enhancing model efficiency by 65.6%. This highlights SATS's effectiveness and scalability for time series pretraining, establishing it as a strong candidate for zero-shot and in-distribution forecasting tasks.
Methodology
The methodology involves a scale-aware token alignment mechanism that aligns representation spaces across different patch sizes using scale-specific feedforward networks (FFNs). It minimizes the distance between mean token embeddings across scales while maximizing the distance between their maximal embeddings. Additionally, a hybrid masking strategy is utilized, combining random and contiguous masking to enhance the model's ability to capture multi-scale temporal structures during masked reconstruction.
Results
SATS achieved a 9.2% improvement in Mean Squared Error (MSE) and an 8.3% gain in GIFT-Eval MASE on LSTF benchmarks compared to competitive baselines. Furthermore, it demonstrated a 65.6% increase in model efficiency over advanced methods, establishing its state-of-the-art performance in time series forecasting.
Implications
The findings suggest that SATS can be effectively utilized for various time series forecasting tasks, enhancing the robustness and generalization of models across diverse datasets. Its scalability and efficiency make it a promising approach for real-world applications in fields such as finance, healthcare, and IoT.
DraftFM: A Foundation Model for Day-Zero Drafting in Magic: The Gathering
Theory
- DraftFM operates effectively in a day-zero drafting context, scoring unseen cards based on public information.
- The model achieved significant predictive accuracy, outperforming random selection and existing models trained on specific sets.
- It provides a novel approach to card representation using frozen embeddings, avoiding reliance on set-specific data.
- The model's predictions for the unreleased set 'The Hobbit' were validated against expert opinions.
Read more
DraftFM: A Foundation Model for Day-Zero Drafting in Magic: The Gathering
Summary
The paper introduces DraftFM, a discrete-choice policy model designed for day-zero drafting in Magic: The Gathering (MTG), where players draft cards from a new expansion before any draft logs are available. DraftFM scores cards based solely on their public attributes, using a 775-dimensional representation that includes structured features and a fixed text embedding, without relying on card identities or usage statistics. This approach allows the model to generalize to unseen cards effectively. The model, with 1.6 million parameters, was trained on 149 million human picks from 29 expansions and demonstrated impressive predictive capabilities, achieving top-1 agreement rates of 50.8%, 60.4%, and 56.7% on held-out expansions, significantly outperforming random selection. Furthermore, it provided a card ranking for the unreleased set 'The Hobbit' shortly before its official release, which aligned well with expert evaluations. The study emphasizes the potential of feature-only representations for drafting in new sets, addressing a gap in existing literature regarding day-zero deployment.
Methodology
DraftFM employs a discrete-choice policy framework, utilizing a 775-dimensional frozen representation for each card based on public data. The model is trained on a large dataset of human picks from multiple expansions, allowing it to generalize to new sets without prior exposure. The architecture is designed to score cards based solely on their attributes and rules text, facilitating day-zero drafting.
Results
The model achieved top-1 agreement rates of 50.8%, 60.4%, and 56.7% on three held-out expansions, significantly higher than the random chance of 7%. When refitted on all observed expansions, it successfully ranked cards for the unreleased set 'The Hobbit', aligning closely with expert evaluations.
Implications
DraftFM's approach could revolutionize drafting strategies in card games, providing players with real-time assistance based on a model that does not require historical data from the specific set. This could enhance competitive play and inform future research on behavioral prediction in games.
Selection, Recombination, or a Fresh Solve? A Candidate-Free Control for Single-Pass Test-Time Aggregation
NLP
Large Language Models
Efficient ML
- Introduces a candidate-free control to measure the net value of candidate conditioning.
- Finds that conditioning improves accuracy when multiple candidates are correct but reduces it when all candidates are wrong.
- Demonstrates that failures in all-wrong scenarios often match displayed candidate answers.
- Shows that explicit answer fields can guide outputs, while masking does not improve accuracy.
Read more
Selection, Recombination, or a Fresh Solve? A Candidate-Free Control for Single-Pass Test-Time Aggregation
Summary
This paper addresses the challenge of test-time aggregation in machine learning when all candidate solutions are incorrect. The author introduces a candidate-free control mechanism to evaluate the effectiveness of candidate conditioning in generating accurate outputs. The study reveals that when multiple candidates are correct, conditioning improves accuracy significantly. However, when all candidates are wrong, conditioning leads to a decrease in accuracy compared to a fresh solve. The findings suggest that the value of candidate context is not straightforward and varies depending on the correctness of the candidates. The paper employs a structured methodology involving five different aggregation arms to assess the impact of candidate conditioning under controlled conditions. The results indicate a regime-dependent reversal in accuracy based on the number of correct candidates, highlighting the complexities of candidate-based aggregation strategies.
Methodology
The study utilizes five aggregation arms: AGGREGATE (with candidates), NO-CANDIDATE (fresh solve), ANSWER-ONLY (answers without reasoning), VOTING (raw answer strings), and ORACLE (any correct candidate). Each arm operates under a shared maximum generation-token allowance, and the evaluation is stratified by the number of correct candidates.
Results
The results indicate that conditioning on multiple correct candidates improves accuracy (∆cand = +0.290), while conditioning on an all-wrong candidate pool decreases accuracy (∆cand = -0.123). The study also finds that off-topic candidates can reproduce much of the accuracy deficit observed in all-wrong scenarios.
Implications
These findings have significant implications for the design of aggregation strategies in machine learning, particularly in scenarios where candidate solutions may be incorrect. Understanding the conditions under which candidate conditioning is beneficial or detrimental can inform future research and applications in efficient reasoning and decision-making.
A comparison between ceiling-mounted FMCW, IR-UWB and Wi-Fi radar for in-bedroom human activity monitoring and sleep interruption detection
Time Series
- Controlled comparison of FMCW, IR-UWB, and Wi-Fi sensing technologies under identical conditions.
- IR-UWB outperforms in cross-subject activity recognition, while FMCW excels in generalization to new environments.
- All technologies demonstrate high performance in sleep monitoring tasks.
- Identifies a trade-off between recognition performance and robustness based on signal characteristics.
Read more
A comparison between ceiling-mounted FMCW, IR-UWB and Wi-Fi radar for in-bedroom human activity monitoring and sleep interruption detection
Summary
This paper presents a comparative study of three radio frequency (RF) sensing technologies—frequency-modulated continuous wave (FMCW) radar, impulse radio ultra-wideband (IR-UWB), and Wi-Fi—specifically for human activity recognition (HAR) and sleep monitoring in healthcare settings. The authors conducted a controlled experiment involving synchronized recordings from 20 participants across six different room layouts. All technologies were evaluated using the same convolutional neural network (CNN) architecture to ensure that performance differences were attributable to the sensing technologies rather than the model design. The study found that IR-UWB achieved the highest cross-subject activity recognition performance at 89.0% macro F1 score, while FMCW demonstrated superior generalization to unseen room layouts with an 83.8% macro F1 score. For sleep monitoring tasks, all technologies exceeded 92% macro F1 in unseen environments. The results highlight a trade-off between recognition performance and environmental robustness, influenced by factors such as range resolution and antenna diversity. The findings provide practical guidelines for designing RF sensing systems in healthcare applications.
Methodology
The study utilized a controlled experimental setup with synchronized recordings from 20 participants across six room layouts. A uniform convolutional neural network (CNN) was employed to evaluate the performance of each sensing technology on both fine-grained human activity recognition and coarse sleep monitoring tasks.
Results
IR-UWB achieved the highest macro F1 score of 89.0% for activity recognition, while FMCW radar showed an 83.8% macro F1 score for generalization to unseen layouts. All technologies exceeded 92% macro F1 in sleep monitoring tasks, indicating strong performance across the board.
Implications
The findings suggest that RF sensing technologies can be effectively utilized for non-intrusive healthcare monitoring, offering privacy-preserving alternatives to camera-based systems. The insights gained can guide the design of future RF sensing systems tailored for healthcare applications.
Towards Reversible Forgetting: Managing Obsolete Knowledge in Continual Enterprise AI Agents
Theory
- Reversible forgetting allows for the suppression of obsolete knowledge without permanent erasure.
- The framework introduces three operational memory states: active, dormant, and retired.
- The Hysteretic Reversible Memory Controller (HRMC) manages knowledge transitions with evidence accumulation and policy gates.
- The approach reframes continual learning as a management of knowledge lifecycle rather than mere retention.
Read more
Towards Reversible Forgetting: Managing Obsolete Knowledge in Continual Enterprise AI Agents
Summary
This paper addresses the challenge of managing knowledge in continual learning systems, particularly for enterprise AI agents operating in dynamic environments. Traditional approaches view forgetting as a failure, focusing on retaining knowledge. However, the authors argue that in non-stationary settings, retaining obsolete knowledge can lead to negative transfer and operational risks. They propose a novel framework called reversible forgetting, which categorizes memory into three states: active, dormant, and retired. This framework allows for the suppression of obsolete knowledge while retaining the ability to reactivate it when relevant again. The authors introduce the Hysteretic Reversible Memory Controller (HRMC), which employs evidence accumulation, asymmetric thresholds, shadow mode testing for reactivation, and policy-gated retirement to manage knowledge effectively. The paper emphasizes that knowledge management should be viewed as a lifecycle rather than a binary retention/deletion process, ensuring that enterprise agents utilize the most relevant information at any given time.
Methodology
The authors propose a conceptual framework for reversible forgetting, operationalized through the Hysteretic Reversible Memory Controller (HRMC). This controller assesses memory relevance based on contextual similarity, utility, reliability, staleness, and potential harm. It uses hysteresis to prevent rapid state changes and requires policy approval for memory retirement, ensuring a structured approach to knowledge management.
Results
The proposed framework effectively reduces the influence of obsolete knowledge while allowing for its potential reactivation. By implementing the HRMC, enterprise agents can adapt to changing environments without losing valuable historical knowledge, thus minimizing operational risks associated with negative transfer.
Implications
The findings suggest that enterprise AI systems can benefit from a more nuanced approach to knowledge management, allowing them to adapt to changing conditions while maintaining a record of past knowledge. This has implications for various sectors, including finance, where knowledge relevance can fluctuate with market conditions.
Tensor Field Models
Generative Models
Theory
Efficient ML
- Introduction of Tensor Field Models (TFMs) as a new mathematical structure for machine learning.
- TFMs utilize a learned operator to map component-section families to time-dependent tangent sections.
- The architecture allows for efficient reuse of condition representations across multiple queries.
- Experiments demonstrate improved performance and faster generation through amortized sampling.
Read more
Tensor Field Models
Summary
This paper presents Tensor Field Models (TFMs), which are mathematical structures that utilize a learned operator to map a product of admissible component-section families to a family of time-dependent tangent sections on a Generative State Manifold. The authors emphasize that the analytic and dynamical constraints are determined by the selection of admissible families rather than by the fundamental definition of the model. The paper introduces constructed, component-separable, and Tensor Bundle TFMs, which enhance the structure of the model. In the conditional realizations explored, a structured condition is mapped componentwise to a reusable collection, allowing for efficient evaluations of multiple queries. The TFMs are trained using Flow Matching, and experimental results indicate that these models can enhance performance and accelerate generation through amortized sampling enabled by reusable condition representations. The paper discusses the mathematical background of Tensor Bundles and Tensor Fields, providing a foundation for understanding the proposed models.
Methodology
The authors develop Tensor Field Models by defining a parameterized Field Operator that maps admissible input and output-section families to a Generative State Manifold. They utilize Flow Matching for training the models and explore various architectures, including constructed and component-separable TFMs, to evaluate their performance in generating vector fields.
Results
The experiments conducted show that Tensor Field Models can significantly improve performance in generating vector fields compared to traditional methods. The use of reusable condition representations allows for faster evaluations, particularly in scenarios involving multiple queries.
Implications
The introduction of Tensor Field Models has the potential to enhance various applications in machine learning, particularly in generative modeling tasks where efficiency and performance are critical. The structured approach to handling conditional information may lead to advancements in fields such as robotics and computer vision.
Beyond Trial Averaging: Anchoring Neural and Visual Representations for Few-Repetition Brain-to-Image Retrieval
Multimodal
- Identification of a non-transitive alignment pattern in few-repetition brain-to-image retrieval.
- Introduction of the NEAR framework that enhances retrieval accuracy by anchoring neural and visual representations.
- Demonstrated improvements in retrieval performance across multiple datasets and modalities.
- Reduction of reliance on repeated stimulus presentations, addressing latency and user burden.
Read more
Beyond Trial Averaging: Anchoring Neural and Visual Representations for Few-Repetition Brain-to-Image Retrieval
Summary
This paper addresses the challenge of brain-to-image retrieval, which typically relies on averaging multiple neural trials per image to enhance retrieval accuracy. The authors identify a significant drop in retrieval performance when only a few repetitions are available, attributing this to both query noise and gallery placement issues. They propose a novel framework called Neural-Anchor-based Retrieval (NEAR), which utilizes a stable neural representation as an anchor to improve retrieval accuracy. NEAR employs two strategies: Query Anchoring, which denoises the few-repetition query, and Gallery Anchoring, which predicts a pseudo anchor for each candidate image. The framework is tested across four datasets, including EEG, MEG, and fMRI, demonstrating consistent improvements in retrieval performance, particularly in few-repetition scenarios. This work highlights the importance of anchoring neural and visual representations to reduce reliance on repeated measurements, making brain-to-image retrieval more feasible for real-world applications.
Methodology
The authors conducted a controlled target-swap experiment to analyze the alignment patterns between few-repetition queries, stable neural representations, and images. They developed the NEAR framework, which consists of two main components: Query Anchoring, which denoises the query towards a stable neural anchor, and Gallery Anchoring, which predicts pseudo anchors for candidate images to enhance retrieval scoring.
Results
The NEAR framework significantly improved retrieval accuracy on the THINGS-EEG2 dataset, increasing 200-way Top-1 accuracy by 5.7 percentage points with one repetition and 9.3 percentage points with four repetitions. The improvements were consistent across all ten participants tested, indicating the robustness of the method across different neural signal types.
Implications
The findings suggest that anchoring neural and visual representations can facilitate more effective brain-to-image retrieval with fewer repetitions, potentially enabling practical applications in neuro-rehabilitation and dream decoding without the need for extensive stimulus presentations.
Graphical Design of Interpretable Architectures
Interpretability
- Introduces a graphical notation for interpretable AI architectures based on Penrose tensor notation.
- Provides a global view of architectures while mapping directly to PyTorch einsum code.
- Describes interpretable architectures by construction, enhancing understanding and reproducibility.
- Demonstrates the notation with a case study on the Steerling-8B language model.
Read more
Graphical Design of Interpretable Architectures
Summary
This paper addresses the challenge of designing interpretable AI architectures by introducing a novel graphical notation adapted from Penrose tensor notation. Traditional representations, such as symbolic equations and probabilistic graphical models, often fail to provide a comprehensive view of architectures or do not accurately depict tensor manipulations, which are crucial for understanding and reproducibility. The proposed notation allows for a clear visualization of tensor operations while directly mapping to PyTorch einsum code, facilitating both design and implementation. The author demonstrates the utility of this notation by describing various interpretable architectures, including concept bottlenecks and neural additive models, and provides a detailed diagram of the Steerling-8B language model, highlighting its residual structure and translating it into concise PyTorch code. This approach not only enhances interpretability but also aids in the reproducibility of AI models.
Methodology
The paper employs a graphical notation derived from Penrose tensor notation to represent AI architectures. This notation is used to analyze and diagram tensor operations, which are then translated into PyTorch einsum code. The author illustrates the methodology through various examples of interpretable architectures and a detailed case study of the Steerling-8B model.
Results
The graphical notation successfully provides a clear and interpretable representation of complex tensor manipulations, yielding insights into the architecture of Steerling-8B. The notation facilitates a direct translation into PyTorch code, demonstrating its practical applicability in designing interpretable AI models.
Implications
The proposed graphical notation has the potential to significantly improve the design and understanding of interpretable AI architectures, making it easier for researchers and practitioners to develop reproducible models. This could lead to advancements in the field of interpretable machine learning, enhancing trust and transparency in AI systems.
Systematic Evaluation of TabPFN-TS for Zero-Shot Probabilistic Heat Load Forecasting in District Heating Networks
Time Series
- TabPFN-TS is evaluated for its effectiveness in zero-shot probabilistic heat load forecasting.
- The study identifies optimal configurations for forecasting, emphasizing the importance of context length and covariate selection.
- TabPFN-TS performs competitively with established models like Chronos-2, particularly in calibration.
- A new Multi-Resolution Residual-Correction Forecaster is proposed to improve forecasting accuracy for longer horizons.
Read more
Systematic Evaluation of TabPFN-TS for Zero-Shot Probabilistic Heat Load Forecasting in District Heating Networks
Summary
This paper evaluates the performance of TabPFN-TS, a zero-shot time-series foundation model, for probabilistic heat load forecasting in district heating networks. Traditional forecasting methods often require system-specific models trained on historical data, which can be cumbersome due to changes in network configurations. TabPFN-TS offers an alternative by adapting at inference time from recent observations without the need for retraining. The study systematically compares TabPFN-TS against state-of-the-art models, including Chronos-2, and assesses various factors such as covariate choice, context length, temporal resolution, and prediction horizon. The findings reveal that a configuration using hourly 24-hour forecasting with a 12-week rolling context and ambient temperature as a covariate performs well. TabPFN-TS achieves a CVRMSE of 13.06%, closely trailing Chronos-2's 12.48%. While Chronos-2 has a lower overall error, TabPFN-TS demonstrates superior empirical calibration. The paper also introduces a Multi-Resolution Residual-Correction Forecaster to enhance longer-horizon planning accuracy based on the empirical results.
Methodology
The methodology involves a systematic evaluation of TabPFN-TS against other time-series foundation models and classical machine learning methods. The study analyzes the impact of different covariates, context lengths, and temporal resolutions on forecasting performance. It also tests the model's transferability across different district heating networks and validates configurations over a full year.
Results
TabPFN-TS achieved a CVRMSE of 13.06% in deterministic accuracy, closely following Chronos-2's 12.48%. The optimal configuration identified was hourly 24-hour forecasting with a 12-week rolling context. Although Chronos-2 had a lower aggregate error, TabPFN-TS showed better empirical calibration. The proposed Multi-Resolution Residual-Correction Forecaster demonstrated potential for improving longer-horizon forecasts.
Implications
The findings suggest that TabPFN-TS can be a viable alternative for heat load forecasting in district heating networks, particularly in dynamic environments where traditional models struggle. The proposed forecasting framework could enhance operational efficiency and support the integration of renewable energy sources in district heating systems.
Finite-Horizon Input-Output Dynamics of Minibatch Perturbations in AdamW
Optimization
Theory
- Formulation of localized minibatch influence as a signed finite-horizon response under paired AdamW trajectories.
- Development of a joint parameter-moment ISO operator to characterize the propagation of perturbations and their effects on loss.
- Establishment of an exact multistep error decomposition that separates various sources of error in the response dynamics.
- Experimental validation of the proposed response mechanism and optimizer-state effects.
Read more
Finite-Horizon Input-Output Dynamics of Minibatch Perturbations in AdamW
Summary
This paper investigates the delayed effects of minibatch perturbations in the AdamW optimizer, which is known for storing past gradient information. The authors formulate AdamW as a finite-horizon input-state-output (ISO) system, allowing them to analyze how a single gradient update influences future training dynamics. By comparing paired training trajectories that differ only by one gradient update, they isolate the evolution of localized perturbations over time. The study derives a signed response operator that maps these perturbations to their future loss effects, revealing how the optimizer's memory affects the magnitude, timing, and sign of these effects. Furthermore, the authors establish a multistep error decomposition that accounts for nonlinearity and activation changes, demonstrating first-order finite-horizon accuracy under specific conditions. Experimental validation supports their theoretical findings, highlighting the significant prospective structure of delayed influences and the recoverability of these effects from ISO approximations.
Methodology
The authors model the AdamW optimizer as a finite-horizon input-state-output (ISO) system. They analyze the dynamics of localized gradient perturbations by comparing paired training trajectories and derive a tangent model to map initial perturbations to future loss responses. They also conduct experiments to validate their theoretical findings and explore the structure of delayed influences.
Results
The study reveals that the effects of a minibatch perturbation can persist over several training steps, with significant implications for loss dynamics. The derived response operator effectively captures the influence of optimizer memory on the magnitude and timing of these effects. The multistep error decomposition provides insights into the sources of error in the response dynamics, confirming the first-order accuracy of their finite-horizon approximation.
Implications
The findings suggest that understanding the delayed effects of minibatch perturbations can lead to improved training stability and performance in neural networks. The insights gained from this study may inform the design of more robust optimization algorithms and strategies for managing transient training instabilities.
Inadvertent Context Leakage in Language Models
NLP
Large Language Models
Theory
- Inadvertent context leakage allows sensitive information to be reconstructed from benign model outputs.
- Controlled experiments show high accuracy in reconstructing numeric secrets from model responses.
- More capable models exhibit greater leakage, suggesting a correlation between model capability and privacy risks.
- The study introduces a suppression mechanism to mitigate leakage and evaluates its effectiveness across various models.
Read more
Inadvertent Context Leakage in Language Models
Summary
This paper investigates the phenomenon of inadvertent context leakage in language models (LLMs), particularly concerning sensitive user data such as calendars, credentials, and health records. The authors explore whether the presence of such secrets in a model's context window can lead to hidden correlations in the model's outputs, allowing for the reconstruction of these secrets even when the model refuses direct extraction attempts. The study introduces a novel adaptive attack method that assumes black-box access to the model, enabling adversaries to engineer prompts that exploit this leakage. Through controlled experiments on eight proprietary models, the authors demonstrate that 2-digit secrets can be reconstructed with near-perfect accuracy, while 4-digit secrets achieve an 82% exact match. The findings suggest that more capable models are more prone to leakage, indicating that this issue is a byproduct of model capability rather than a simple bug. The paper also details two practical attacks: one that infers semantic predicates about user memories and another that extracts full Social Security Numbers from a production-style agent.
Methodology
The authors formalize a threat model based on predicate inference, where an adaptive adversary with black-box access learns to exploit model-specific channels that leak in-context secrets. They conduct controlled experiments on eight proprietary models, using benign prompts to assess the models' ability to protect sensitive information. The study also evaluates a proposed suppression mechanism to reduce leakage.
Results
The experiments reveal that 2-digit secrets can be reconstructed with 100% accuracy on certain models, while 4-digit secrets achieve an 82% exact match. The study finds that conditioning prompts on previously recovered digits significantly improves recovery rates. Additionally, the proposed suppression mechanism shows promise in reducing leakage across different models and contexts.
Implications
The findings highlight significant privacy concerns for LLMs used in sensitive applications, suggesting that developers must implement robust mechanisms to prevent inadvertent leakage. The study also emphasizes the need for ongoing research into the privacy implications of increasingly capable AI systems.
SAE-Xplainers: Rule-Based Feature Interpretation for Extreme Earth Events
Interpretability
- Introduction of a geographic location-aware SAE (GeoTopK) for ExEE analysis.
- Development of SAE-Xplainers for rule-based interpretation of complex features.
- Demonstrated improvements in reconstruction performance and feature utilization.
- Validated on three types of extreme Earth events: fires, tropical cyclones, and atmospheric rivers.
Read more
SAE-Xplainers: Rule-Based Feature Interpretation for Extreme Earth Events
Summary
The paper addresses the challenge of interpretability in deep learning models applied to extreme Earth events (ExEE) such as wildfires, tropical cyclones, and atmospheric rivers. The authors propose a novel approach called SAE-Xplainers, which combines geographic location-based modulation of Sparse Autoencoders (SAEs) with an ensemble of rule-based models to interpret high-dimensional features derived from complex environmental data. The geographic modulation captures local semantic meanings of environmental patterns, enhancing the reconstruction performance and feature utilization of the SAEs. The SAE-Xplainers facilitate the interpretation of these features by converting them into human-understandable rules, consistent with scientific literature. The method is validated across three ExEE types, demonstrating improved interpretability and the ability to identify feature absorption, which refers to the sparsity-driven phenomenon of producing overly specific features from hierarchical concepts. Overall, the proposed framework enhances the transparency and usability of deep learning models in operational settings for climate-related predictions and assessments.
Methodology
The authors developed a geographic location encoder that modulates the inputs to a k-sparse autoencoder (GeoTopK) to ensure location-aware feature extraction. They then employed an ensemble of rule-based models (SAE-Xplainers) to interpret the extracted features, translating them into understandable rules that reflect the underlying environmental patterns.
Results
The proposed GeoTopK SAE outperformed traditional location-agnostic SAEs in terms of reconstruction accuracy and reduced the rate of dead features. The SAE-Xplainers provided faithful interpretations of the features, revealing insights consistent with existing scientific knowledge and facilitating the identification of feature absorption.
Implications
The findings suggest that the SAE-Xplainers framework can significantly enhance the interpretability of deep learning models in climate science, making them more applicable in operational settings for forecasting and assessing extreme weather events. This could lead to better preparedness and risk management strategies in response to climate change.
DecoVAE: a Lightweight Interpretable Trend-Seasonal VAE Framework for Efficient Probabilistic Time Series Forecasting
Time Series
Generative Models
Interpretability
- DecoVAE provides a lightweight and interpretable framework for probabilistic time series forecasting.
- The model explicitly decomposes time series into trend and seasonal components using specialized mechanisms.
- DecoVAE achieves significant accuracy improvements over strong baselines while maintaining high computational efficiency.
- The framework employs a differential regularizer for trend smoothness and complex-domain modeling for seasonality.
Read more
DecoVAE: a Lightweight Interpretable Trend-Seasonal VAE Framework for Efficient Probabilistic Time Series Forecasting
Summary
The paper introduces DecoVAE, a novel lightweight framework designed for probabilistic time series forecasting that effectively separates trend and seasonal components. Traditional forecasting methods often struggle with accurately modeling these distinct dynamics, leading to inefficiencies and a lack of interpretability. DecoVAE addresses these challenges by employing domain-specific inductive biases to decompose time series data into trend and seasonal components. The trend component is modeled using a differential regularizer inspired by the Hodrick-Prescott filter, ensuring structural smoothness, while the seasonal component is captured in the complex frequency domain using a complex Gaussian VAE, which allows for the effective representation of periodic patterns. The authors validate DecoVAE through extensive experiments on seven real-world datasets, demonstrating its superior performance over existing baselines in terms of accuracy and efficiency. The framework achieves significant reductions in Continuous Ranked Probability Score (CRPS) and Normalized Mean Absolute Error (NMAE) for both short-term and long-term forecasting, while also reducing model weight and increasing speed compared to other methods.
Methodology
DecoVAE utilizes a variational autoencoder architecture to explicitly decompose time series data into trend and seasonal components. The trend component is regularized using a Hodrick-Prescott-inspired penalty to ensure smoothness, while the seasonal component is modeled in the complex frequency domain to capture periodic patterns effectively. This dual approach allows for well-calibrated probabilistic forecasts that reflect the distinct dynamics of the time series.
Results
DecoVAE demonstrated up to 14.96% improvement in CRPS and 23.30% in NMAE for short-term forecasting, and up to 52.68% and 26.51% improvements for long-term forecasting compared to strong baseline methods. Additionally, the model's weight was reduced by up to 93%, and its speed was accelerated by up to 74% compared to the second-best method.
Implications
The DecoVAE framework has significant implications for various applications requiring accurate probabilistic time series forecasting, such as finance, energy, weather, and traffic management. Its lightweight and interpretable nature makes it suitable for deployment in resource-constrained environments while providing reliable forecasts.
Mechanistic Tomography: Designed Measurement for Control-Oriented Interpretability
Interpretability
- Introduces mechanistic tomography as a unified framework for measurement in mechanistic interpretability.
- Establishes a common measurement language for various interpretability methods.
- Demonstrates the importance of control-oriented validation in assessing interpretability.
- Provides empirical results showing the effectiveness of the proposed measurement strategies.
Read more
Mechanistic Tomography: Designed Measurement for Control-Oriented Interpretability
Summary
This paper introduces the concept of mechanistic tomography, which aims to unify various measurement methods used in mechanistic interpretability of machine learning models. The author formulates a common measurement problem that encompasses techniques such as coordinate patching, attribution patching, subset interventions, and Hessian-vector products. By expressing the measurement problem in a structured form, the paper provides a framework that allows for a systematic approach to recovering internal mechanisms and intervention effects. The methodology involves starting with the least costly measurements, testing on held-out interventions, calibrating corrections, and expanding the measurement family as needed. The paper emphasizes the importance of control-oriented validation, where an estimate acts as an observer guiding interventions. Through experiments on models like GPT-2-small and Qwen-2.5-7B, the author demonstrates the effectiveness of the proposed methods in recovering interaction effects and improving predictive accuracy. The findings highlight the necessity of a measurement-first approach in mechanistic interpretability, especially as model architectures become less transparent.
Methodology
The methodology involves formulating the measurement problem as a linear equation, where different measurement techniques are analyzed under a common framework. The approach includes starting with simple measurements, testing their effectiveness, calibrating corrections, and expanding the measurement family based on residual errors. The paper also discusses the implications of observer error in control settings.
Results
The experiments conducted on GPT-2-small and Qwen-2.5-7B show that the proposed measurement strategies can effectively recover interaction effects and achieve high predictive accuracy (e.g., R2 = .983 on a finite refusal-response surface). The results indicate that a calibrated additive map can significantly improve predictions, while the necessity of adding interactions is determined by held-out error.
Implications
The findings suggest that mechanistic tomography can enhance the interpretability of machine learning models, particularly in control applications. This approach can lead to better understanding and prediction of model behaviors, which is crucial as models become more complex and less interpretable.
GEAR: Generative Expansion and Real Anchoring for Two-Stage Distillation of Tabular Foundation Models
Efficient ML
- GEAR distills TFMs into efficient MLP or tree-based models for practical deployment.
- The two-stage process includes Generative Expansion and Real Anchoring to enhance model performance.
- Significant improvements in AUC scores over traditional supervised models were observed.
- GEAR reduces inference time by up to 2866 times and memory usage by 3.3 times.
Read more
GEAR: Generative Expansion and Real Anchoring for Two-Stage Distillation of Tabular Foundation Models
Summary
The paper introduces GEAR, a two-stage framework designed to distill Tabular Foundation Models (TFMs) into lightweight predictors suitable for deployment on commodity CPUs. The first stage, Generative Expansion, utilizes synthetic covariates to train a student model on soft targets from a frozen teacher model, thereby expanding the coverage of training data. The second stage, Real Anchoring, re-aligns the student model with the actual target distribution using real labels and out-of-fold predictions from the teacher, effectively mitigating self-labeling leakage. The authors derive a risk certificate to characterize the trade-off between the volume of generated queries and the fidelity of the generator. Experiments conducted on datasets TALENT and TabArena demonstrate that GEAR significantly enhances the performance of MLP and tree-based models compared to traditional supervised learning methods, achieving notable improvements in AUC scores while drastically reducing inference time and memory usage. This modular approach is agnostic to the choice of generator, teacher, and student architectures, making it broadly applicable across various tabular data tasks.
Methodology
The GEAR framework consists of two main stages: Stage 1 involves generating synthetic covariates to train a student model on soft targets from a frozen teacher model, while Stage 2 re-anchors the student model to the real data distribution using actual labels and out-of-fold predictions to avoid self-labeling leakage. The authors also derive a risk certificate to analyze the trade-off between the volume of generated queries and the fidelity of the generator.
Results
GEAR's two-stage MLP models outperformed supervised MLPs by 1.81–2.00 AUC points on binary tasks and 1.19–1.35 points on multiclass tasks. Additionally, GEAR achieved improvements over real-data-only distillation by 1.76–2.19 points for binary tasks and 2.09–2.40 points for multiclass tasks. The framework also demonstrated substantial reductions in median inference time (57–2866× faster) and peak memory usage (1.9–3.3× lower) compared to matched supervised baselines.
Implications
The GEAR framework has significant implications for the deployment of machine learning models in resource-constrained environments, particularly in industries relying on tabular data such as finance and healthcare. By enabling efficient inference without sacrificing accuracy, GEAR can facilitate the broader adoption of advanced machine learning techniques in practical applications.
Orthogonal JEPA: Factorized Predictive States for Latent World Models
Computer Vision
Time Series
Robotics
- Introduction of Orthogonal JEPA for improved latent world modeling.
- Utilization of orthogonal predictive factorization to analyze target states into multiple components.
- Implementation of dedicated prediction branches for each component to enhance predictive capacity.
- Incorporation of regularization techniques to maintain diversity and prevent encoder collapse.
Read more
Orthogonal JEPA: Factorized Predictive States for Latent World Models
Summary
The paper introduces Orthogonal JEPA, a novel framework for latent world modeling that enhances the predictive capabilities of joint-embedding predictive architectures (JEPAs). Traditional JEPAs utilize a single target embedding and prediction pathway, which can lead to inefficiencies in complex systems where multiple signals interact. Orthogonal JEPA addresses this by employing orthogonal predictive factorization, which decomposes target states into multiple components, each predicted by a dedicated branch. This approach allows for better allocation of predictive capacity, reducing redundancy and improving the learning of weaker signals. The framework incorporates several regularization techniques, including orthogonality objectives and variance regularization, to maintain diversity in the learned components and prevent encoder collapse. The effectiveness of Orthogonal JEPA is demonstrated through experiments across various domains, including vision, single-cell transcriptomics, health records, control tasks, and molecular dynamics, showcasing its versatility and robustness in representation quality, forecasting, and planning.
Methodology
The methodology involves the development of Orthogonal JEPA, which uses learned basis matrices to decompose target states into multiple components. Each component is predicted through a separate branch, allowing for a more structured approach to latent state prediction. The framework employs various regularization strategies, including orthogonality objectives and factor-activity regularization, to ensure effective learning and prevent redundancy in the predictive pathways.
Results
The experiments conducted across multiple domains indicate that Orthogonal JEPA significantly improves representation quality and predictive performance compared to standard JEPA architectures. The framework demonstrates enhanced stability in long-horizon predictions and effective planning capabilities in complex systems.
Implications
The implications of this research extend to various fields where latent state modeling is crucial, such as robotics, healthcare analytics, and molecular dynamics. By improving the efficiency and effectiveness of predictive modeling, Orthogonal JEPA can facilitate better decision-making and planning in complex environments.
Unsupervised Anomaly Detection Using Flow Matching on Tabular Data
Generative Models
Theory
Efficient ML
- Investigates flow-matching-based anomaly detection under contaminated training data.
- Highlights the importance of anomaly scoring functions for robustness.
- Demonstrates that trajectory-based scoring improves detection performance.
- Shows that Forest-Flow can outperform TCCM with appropriate scoring functions.
Read more
Unsupervised Anomaly Detection Using Flow Matching on Tabular Data
Summary
This paper addresses the challenge of unsupervised anomaly detection in financial datasets, particularly when training data may contain unlabeled anomalies. The authors investigate the robustness of flow-matching-based anomaly detection methods, specifically Time-Conditioned Contraction Matching (TCCM) and Forest-Flow, under conditions of training set contamination. They emphasize the significance of anomaly scoring functions, revealing that TCCM's original Decision score is sensitive to contamination, while trajectory-based scores, such as Deviation and Reconstruction, provide more stable anomaly signals. The study demonstrates that these trajectory-based scores can enhance the performance of Forest-Flow, making it competitive with TCCM across varying levels of anomaly class imbalance. The findings underscore the critical role of scoring strategies in improving the robustness of anomaly detection methods in real-world applications.
Methodology
The authors compare two flow-matching models, TCCM and Forest-Flow, focusing on their anomaly detection capabilities under contaminated training data. They evaluate three anomaly scoring functions: Decision, Deviation, and Reconstruction, assessing their impact on detection performance and robustness in unsupervised settings.
Results
The study finds that TCCM's Decision score is highly sensitive to contamination, while the Deviation and Reconstruction scores provide more stable anomaly signals. As a result, Forest-Flow, when paired with these trajectory-based scores, becomes competitive with TCCM and, in some cases, outperforms it, particularly in datasets with varying anomaly ratios.
Implications
The findings suggest that improving anomaly scoring methods can significantly enhance the robustness of anomaly detection systems in financial applications, where training data often contains unlabeled anomalies. This has potential applications in fraud detection, manufacturing fault detection, and medical diagnosis.
An Empirical Benchmark of Deep Time-Series Models for Smart Meter Energy Forecasting
Time Series
- Benchmarking of nine deep learning models for energy forecasting on real-world datasets.
- Forecasting accuracy improves with historical context only to a saturation point; accuracy declines with longer prediction horizons.
- Lightweight architectures can achieve competitive accuracy at lower computational costs.
- Transformer models show potential advantages for under-represented population segments, though not consistently statistically robust.
Read more
An Empirical Benchmark of Deep Time-Series Models for Smart Meter Energy Forecasting
Summary
This paper presents an empirical benchmark of nine deep learning models for time-series forecasting applied to smart meter energy consumption data. The authors evaluate these models on two high-resolution datasets, focusing on the effects of historical input length, prediction horizon, and model architecture on forecasting performance. The study finds that while extending the historical context improves accuracy, it only does so up to a certain point, after which additional data yields diminishing returns. Conversely, accuracy declines as the prediction horizon increases. The results indicate that lightweight architectures can achieve competitive accuracy at lower computational costs, and that architectural differences are more pronounced at longer forecasting horizons and with heterogeneous datasets. Additionally, subgroup analyses reveal that Transformer-based models may perform better for under-represented population segments, although this advantage is not consistently statistically significant across datasets. The findings provide practical insights for selecting appropriate forecasting models in smart meter applications.
Methodology
The authors benchmarked nine modern deep learning models, including linear, MLP-based, convolutional, and Transformer architectures, on two publicly available smart meter datasets. They analyzed the impact of historical input length, prediction horizon, and model architecture on forecasting performance, employing statistical significance tests to assess differences across models.
Results
The study found that deep learning models consistently outperformed classical baselines. While extending historical context improved accuracy, it plateaued after a certain point. Accuracy decreased with longer prediction horizons, and lightweight models performed comparably to more complex architectures at lower computational costs. Architectural differences were significant mainly at longer horizons and on heterogeneous datasets.
Implications
The findings suggest that practitioners can achieve effective energy consumption forecasting using lightweight models, which can reduce computational costs without sacrificing accuracy. The insights into model performance across different population segments can help tailor forecasting approaches to diverse user groups, enhancing energy management strategies.
Rationally Enriched Chebyshev Trunk Bases for DeepONet Surrogates of High Péclet Entrance Transport
Theory
Optimization
Efficient ML
- Introduction of Rationally Enriched Chebyshev trunks for DeepONet models.
- Demonstrated significant improvements in predicting solution profiles for high-Péclet transport problems.
- REC-trunk DeepONet reduces profile-error metrics by up to 60.2% compared to traditional models.
- Effective suppression of artificial oscillations in near-wall regions.
Read more
Rationally Enriched Chebyshev Trunk Bases for DeepONet Surrogates of High Péclet Entrance Transport
Summary
This paper presents a novel approach using Rationally Enriched Chebyshev (REC) trunks in DeepONet surrogate models to effectively address high-Péclet entrance transport problems characterized by thin localized boundary or wall layers. The REC trunk integrates Chebyshev polynomial elements with rational dictionary elements derived from the adaptive Antoulas–Anderson (AAA) algorithm. The authors evaluate the performance of the REC-trunk DeepONet against a standard DeepONet and a Chebyshev-trunk DeepONet across three distinct problems: a singularly perturbed scalar boundary-value problem (BVP), the thermal entrance problem with a fixed wall temperature, and the concentration entrance problem with an absorbing wall. The results indicate that the REC-trunk DeepONet outperforms the vanilla DeepONet and shows comparable performance to the Chebyshev-trunk DeepONet, particularly excelling in scenarios where the perturbation parameter is between 1.00 × 10−4 and 1.78 × 10−4, achieving up to a 19.5% reduction in profile-error metrics. Additionally, it significantly reduces profile-error metrics for wall-normal temperature and concentration profiles by up to 60.2% and 32.2%, respectively, while minimizing artificial oscillations near walls as the Péclet number varies from 102 to 104.
Methodology
The authors developed the REC trunk by combining Chebyshev polynomial dictionary elements with rational dictionary elements using the adaptive Antoulas–Anderson (AAA) algorithm. They conducted evaluations across three transport problems, comparing the performance of the REC-trunk DeepONet with standard DeepONet and Chebyshev-trunk DeepONet models over multiple training runs.
Results
The REC-trunk DeepONet showed improved accuracy in predicting scalar profiles, particularly in the critical range of perturbation parameters, and reduced profile-error metrics by up to 19.5% compared to the Chebyshev-trunk DeepONet. For wall-normal temperature and concentration profiles, it achieved reductions of up to 60.2% and 32.2%, respectively, compared to both vanilla and Chebyshev-trunk DeepONets.
Implications
The findings suggest that the REC trunk can significantly enhance the accuracy of machine learning models in solving complex transport problems in various engineering applications, potentially leading to more efficient designs and analyses in fields such as biomedical engineering, chemical engineering, and aerospace engineering.
Forgetting, plasticity, and co-observation: a third facet of continual learning
Theory
- Introduction of the co-observation effect as a critical factor in continual learning.
- Demonstration of performance benefits from simultaneous observation of training data.
- Empirical validation across supervised and self-supervised learning paradigms.
- Critique of existing continual learning methods in light of co-observation.
Read more
Forgetting, plasticity, and co-observation: a third facet of continual learning
Summary
This paper addresses the challenges of continual learning in deep neural networks, particularly focusing on the limitations of catastrophic forgetting and loss of plasticity. The authors argue that these two factors alone do not fully account for the performance gap between naive sequential training and offline joint training. They introduce the concept of 'data co-observation' as a crucial third factor that influences continual learning performance. By systematically investigating the benefits of observing training data together, the authors demonstrate that simultaneous observation enhances generalization capabilities beyond mere knowledge retention. Their empirical studies reveal consistent performance improvements in both supervised and self-supervised learning scenarios when training data is co-observed. The paper also critiques existing continual learning mechanisms, suggesting that while distillation methods primarily aid in knowledge retention, memory replay techniques leverage the advantages of co-observation. The findings emphasize the need for continual learning strategies to address the representational deficits caused by the lack of co-observation in separate training settings.
Methodology
The authors employed a diagnostic methodology based on continual pre-training and model ensembles to isolate the co-observation effect. They evaluated continual learning at the representation level using linear probing, which decouples generalization from the dynamics of a continually updated classifier. An ensemble approach was used to control for forgetting, preserving the model's representational state after each training phase.
Results
The results showed that in data-incremental 'chunking' scenarios, incremental joint training consistently outperformed the ensemble baseline, indicating that the absence of co-observation during optimization significantly impacts performance. This effect was observed without additional confounding factors such as distribution shifts or class imbalances.
Implications
The findings suggest that continual learning methods must integrate strategies to mitigate the representational deficits caused by the lack of co-observation in separate training. This could lead to improved generalization capabilities in real-world applications where data arrives sequentially.
RecPFN: Prior-Fitted Networks for In-Context-Based Recommendations
Efficient ML
Theory
NLP
- RecPFN is the first embedding-based PFN-style ICL approach for sequential recommendation.
- The model is pretrained on synthetic data, enabling strong performance with less computational overhead.
- RecPFN achieves state-of-the-art zero-shot performance and is robust to domain shifts.
- The architecture allows for next-item predictions in a single forward pass without weight updates.
Read more
RecPFN: Prior-Fitted Networks for In-Context-Based Recommendations
Summary
The paper introduces RecPFN, a novel approach to sequential recommendation that leverages prior-fitted networks (PFNs) to enable in-context learning (ICL). RecPFN is pretrained on synthetic clickstream data generated from a broad structural causal prior, allowing it to perform Bayesian-style inference with minimal data. The architecture employs a lightweight decoder-only transformer that predicts the next item based on a small set of domain sequences in a single forward pass, without requiring weight updates. The authors demonstrate that RecPFN achieves state-of-the-art zero-shot performance across eight public benchmarks, while also being competitive with supervised methods in low-compute and low-data scenarios. The approach is noted for its deployment efficiency and robustness to domain shifts, outperforming traditional zero-shot baselines that depend on large real-interaction datasets. The findings suggest that RecPFN offers a practical solution for developing generalizable and data-efficient recommendation systems, paving the way for future research into richer priors, longer-context ICL, and multimodal applications.
Methodology
RecPFN employs a synthetic framework for generating clickstream embedding sequences based on a structural causal model. The model is pretrained on these synthetic sequences to approximate Bayesian inference, allowing it to adapt to new domains using in-context examples during inference. The lightweight transformer architecture processes a small number of domain sequences to predict the next item in a single pass.
Results
RecPFN outperforms existing zero-shot recommendation methods and remains competitive with supervised approaches, particularly in scenarios with limited data and computational resources. The model's performance is validated across eight public benchmarks, demonstrating its effectiveness in real-world applications.
Implications
The development of RecPFN suggests a shift towards more efficient and adaptable recommendation systems that can generalize across different domains without extensive retraining. This has significant implications for industries relying on personalized recommendations, such as e-commerce and content delivery, where rapid adaptation to user preferences is crucial.
Evaluating Neural Cartographic Relief Shading for Urban Environments: A Downtown Calgary Study Using High-Resolution DEM and DSM Data
Computer Vision
- The study compares traditional analytical hillshading methods with a neural-based approach in an urban environment.
- Eduard, a neural system trained on mountainous terrain, shows potential for effective urban shading through parameter tuning.
- The research identifies strengths and weaknesses of both methods, emphasizing the need for tailored neural training for urban contexts.
- High-resolution DEM and DSM data from downtown Calgary provide a relevant test case for evaluating hillshading techniques.
Read more
Evaluating Neural Cartographic Relief Shading for Urban Environments: A Downtown Calgary Study Using High-Resolution DEM and DSM Data
Summary
This paper investigates the effectiveness of analytical and neural-based hillshading techniques in urban settings, specifically focusing on downtown Calgary using high-resolution digital elevation model (DEM) and digital surface model (DSM) data. The study compares traditional single-direction and multi-direction analytical hillshading methods against the neural-based Eduard system, which was originally trained on mountainous landscapes. The central aim is to assess whether parameter tuning in Eduard can yield visually compelling and cartographically useful results in an urban context, despite its training bias. The research emphasizes the importance of terrain type, generalization parameters, and detail representation in producing effective urban relief shading. The findings reveal that while analytical methods maintain reliability in certain aspects, Eduard can produce unexpectedly strong results under specific conditions, highlighting the need for future neural model training tailored to urban environments. The study contributes to the broader discourse on terrain representation by exploring the adaptability of neural approaches to complex urban morphologies.
Methodology
The methodology involves a comparative cartographic experiment that evaluates both analytical and neural-based hillshading methods. The study uses high-resolution DEM and DSM data to assess visual legibility and cartographic effectiveness rather than relying solely on algorithmic accuracy metrics. Parameter settings for both methods are varied to understand their impact on urban morphology representation.
Results
The results indicate that while traditional analytical hillshading methods are reliable, the neural-based Eduard system can produce visually compelling results in urban settings when appropriately tuned. The study highlights specific conditions under which Eduard outperforms analytical methods, as well as scenarios where it fails due to its training bias towards natural landscapes.
Implications
The findings suggest that neural-based approaches can enhance urban cartography and visualization by providing more nuanced representations of complex urban forms. This research advocates for future developments in neural model training specifically aimed at urban environments, potentially improving urban planning and geographic information systems.
Longitudinal Bayesian Learning of Continuous Disease Position across the Alzheimer's Disease Continuum
Time Series
Theory
Generative Models
- DCP provides a continuous representation of disease severity, moving beyond discrete diagnostic categories.
- The Disease Continuum Score (DCS) captures clinically relevant variations and predicts future disease progression.
- DCP integrates longitudinal DTI data with weak clinical supervision for improved disease assessment.
- Extensive validation shows DCS's strong clinical relevance and ability to characterize disease evolution.
Read more
Longitudinal Bayesian Learning of Continuous Disease Position across the Alzheimer's Disease Continuum
Summary
This paper introduces a novel framework called Disease Continuum Positioning (DCP) that leverages longitudinal Bayesian learning to assess continuous disease severity in Alzheimer's disease (AD) using diffusion tensor imaging (DTI). Unlike traditional methods that categorize patients into discrete diagnostic stages, DCP provides a continuous Disease Continuum Score (DCS) that quantifies an individual's position along the AD continuum. The framework integrates longitudinal observations with weak clinical supervision, allowing for a fine-grained characterization of disease severity and capturing clinically meaningful variations. The DCS not only reflects current disease status but also predicts future transitions between cognitive states, such as from cognitively normal (CN) to mild cognitive impairment (MCI) and from MCI to AD. Extensive experiments conducted on the Alzheimer's Disease Neuroimaging Initiative (ADNI) cohort demonstrate that DCP outperforms existing disease progression methods, showcasing its ability to accurately characterize disease severity and preserve longitudinal disease evolution.
Methodology
The authors developed the Disease Continuum Positioning (DCP) framework, which employs a longitudinal Bayesian learning approach to model disease severity as a low-dimensional probabilistic latent variable. This method integrates longitudinal DTI scans and weak clinical supervision to derive the Disease Continuum Score (DCS), which quantifies an individual's position along the Alzheimer's disease continuum.
Results
The DCP framework consistently outperformed existing methods in assessing disease progression in the ADNI cohort. The DCS accurately characterized disease severity, demonstrated strong clinical relevance, and effectively preserved longitudinal disease evolution. Furthermore, it provided reliable predictions for future transitions between cognitive states.
Implications
The DCP framework and the associated DCS have significant implications for early diagnosis, prognosis, and management of Alzheimer's disease. By providing a continuous assessment of disease progression, it can enhance clinical decision-making and improve patient outcomes.