AI-generated summaries
Today's ML research,
without the noise.
Daily summaries of the latest machine learning papers from arXiv, processed every 8 hours.
48
Papers today
8h
Update frequency
7
Days of history
DeltaMomentum: A Key-Value based Anisotropic Momentum Update via Delta Rule
Optimization
- DELTAMOMENTUM introduces a direction-aware momentum update that adapts forgetting rates based on the frequency of gradient directions.
- The method is built on a key-value structure derived from the gradients of linear layers, enhancing the efficiency of momentum updates.
- It provides theoretical guarantees of validity and performance, outperforming traditional EMA momentum in clearing stale directions.
- DELTAMOMENTUM is a drop-in replacement for existing momentum buffers, maintaining compatibility with other optimization techniques.
Read more
DeltaMomentum: A Key-Value based Anisotropic Momentum Update via Delta Rule
Summary
The paper introduces DELTAMOMENTUM, a novel momentum update method designed to address the anisotropic nature of gradient updates in deep learning. Traditional optimizers utilize an exponential moving average (EMA) for momentum, which applies a uniform forgetting rate across all directions, potentially leading to inefficiencies in training. DELTAMOMENTUM leverages a key-value structure inherent in the gradients of linear layers, treating the input as a key and the output-side error as a value. This approach allows for a direction-aware momentum update where the forgetting rate is dynamically adjusted based on how frequently a direction is queried. The authors prove that DELTAMOMENTUM is a valid momentum update, applies curvature corrections without matrix inversion, and clears stale directions faster than EMA. The method is compatible with existing optimizers and incurs minimal additional computational cost. Experimental results demonstrate that DELTAMOMENTUM significantly reduces the number of training steps required to achieve validation loss comparable to AdamW, with improvements observed across various model sizes and architectures.
Methodology
The authors propose DELTAMOMENTUM as a key-value based momentum update that utilizes the classical delta rule to refresh momentum based on the frequency of direction queries. The method is validated through theoretical proofs and empirical testing on language model pretraining tasks.
Results
In experiments, AdamW with DELTAMOMENTUM (DeltaAdamW) achieved validation loss in up to 46.39% fewer steps at 67M parameters and 22.12% fewer steps at 370M parameters compared to standard AdamW. The performance gains were consistent across different model sizes, including a 1B parameter model, and were also observed with SGD, ResNet-18, and ViT-Tiny on CIFAR-10.
Implications
The introduction of DELTAMOMENTUM could lead to more efficient training of large neural networks, allowing for faster convergence and better utilization of computational resources. Its compatibility with existing optimizers suggests it can be easily integrated into current training pipelines.
FleetSieve: Decision-Critical Profiling for SLO-Aware LLM Fleet Configuration
Large Language Models
Optimization
Efficient ML
- FleetSieve focuses on decision-critical profiling to optimize LLM fleet configurations.
- It models capacity and tail latency jointly to avoid infeasible configurations.
- The method reduces profiling GPU-seconds significantly compared to traditional methods.
- FleetSieve provides a three-state decision certificate indicating feasibility.
Read more
FleetSieve: Decision-Critical Profiling for SLO-Aware LLM Fleet Configuration
Summary
The paper introduces FleetSieve, a novel approach for optimizing the configuration of large language model (LLM) serving fleets by addressing the complexities of tensor-parallel (TP) degrees and replica counts. Traditional exhaustive profiling methods are inefficient as they often measure configurations that do not impact final resource allocation. FleetSieve improves upon this by selecting measurements based on their expected influence on resource-coupled, service-level objective (SLO)-aware decisions. It jointly models capacity and tail latency, allowing it to compare conservative and optimistic resource allocations and to cease profiling when the decision gap is within a specified tolerance. Evaluations on a 31B-parameter model demonstrate that FleetSieve achieves significant reductions in GPU-seconds required for profiling, outperforming uniform random profiling by 6.9% and achieving a mean saving of 5.4% across various configurations. The method also ensures that configurations do not violate SLO constraints, thus enhancing the reliability of LLM serving systems under varying loads.
Methodology
FleetSieve employs a decision-focused profiling approach that prioritizes measurements based on their predicted effect on resource allocation decisions. It maintains uncertainty over capacity and tail latency, evaluates configurations under both conservative and optimistic scenarios, and stops profiling when the decision gap is minimal. The method uses a three-state result system to classify configurations as certified feasible, undecided, or certified infeasible.
Results
FleetSieve achieved a reduction of 22,200 GPU-seconds for profiling on a fixed H100 measurement grid, which is 6.9% less than uniform random profiling. Across 200 random reveal orders, the mean saving was 5.4%, with a notable 21.5% saving for chat workloads. The method successfully avoided configurations that would violate SLO constraints, demonstrating its effectiveness in managing resource allocation under varying loads.
Implications
FleetSieve's approach can significantly enhance the efficiency of LLM serving systems, particularly in environments with diverse request types and stringent latency requirements. Its decision-critical profiling can be applied to other resource-constrained systems, potentially improving performance and resource utilization across various applications.
CLaST: Context-aware Contrastive VAE for Probabilistic Time Series Forecasting
Generative Models
Time Series
Optimization
- CLaST introduces a VAE framework that preserves contextual similarity for improved time series forecasting.
- The concept of Lag-Invariant Non-stationary Time Series (LINTS) is introduced to model temporal dependencies effectively.
- The contrastive loss function enhances training stability and representation quality compared to traditional mutual information estimators.
- CLaST achieves up to 16.4% improvement in CRPS and 14.4% in NMAE for short-term forecasting, and up to 48.6% and 25.1% improvements for long-term forecasting.
Read more
CLaST: Context-aware Contrastive VAE for Probabilistic Time Series Forecasting
Summary
The paper introduces CLaST, a novel variational autoencoder (VAE) framework designed for probabilistic multivariate time series forecasting. Traditional forecasting models often struggle to capture the intricate temporal dependencies within data, leading to suboptimal latent representations. CLaST addresses this issue by employing a contrastive loss function that preserves contextual similarity between observations. The authors define a new class of time series called Lag-Invariant Non-stationary Time Series (LINTS), which allows for realistic temporal dynamics while maintaining a single constraint on the dependence structure. The proposed method enhances the stability of training and improves the quality of learned representations. Extensive experiments across nine benchmark datasets demonstrate that CLaST outperforms existing strong baseline methods, achieving significant improvements in short-term and long-term forecasting tasks. This work highlights the importance of contextual similarity in enhancing the performance of probabilistic forecasting models.
Methodology
CLaST employs a variational autoencoder architecture that integrates a contrastive loss function to maintain contextual similarity between time series observations. It introduces the LINTS framework to model temporal dependencies effectively, allowing for a more stable training process and better representation learning.
Results
CLaST consistently outperformed strong baseline methods across nine benchmark datasets, achieving up to 16.4% improvement in Continuous Ranked Probability Score (CRPS) and 14.4% in Normalized Mean Absolute Error (NMAE) for short-term forecasting. For long-term predictions, it surpassed the second-best method by up to 48.6% in CRPS and 25.1% in NMAE.
Implications
The findings suggest that incorporating contextual similarity into probabilistic forecasting models can significantly enhance their predictive performance. This approach can be applied across various domains, including finance, energy systems, and healthcare, where accurate time series forecasting is crucial.
GEAR: Generative Expansion and Real Anchoring for Two-Stage Distillation of Tabular Foundation Models
Efficient ML
- GEAR distills TFMs into efficient MLP or tree-based models for practical deployment.
- The two-stage process includes Generative Expansion and Real Anchoring to enhance model performance.
- Significant improvements in AUC scores over traditional supervised models were observed.
- GEAR reduces inference time by up to 2866 times and memory usage by 3.3 times.
Read more
GEAR: Generative Expansion and Real Anchoring for Two-Stage Distillation of Tabular Foundation Models
Summary
The paper introduces GEAR, a two-stage framework designed to distill Tabular Foundation Models (TFMs) into lightweight predictors suitable for deployment on commodity CPUs. The first stage, Generative Expansion, utilizes synthetic covariates to train a student model on soft targets from a frozen teacher model, thereby expanding the coverage of training data. The second stage, Real Anchoring, re-aligns the student model with the actual target distribution using real labels and out-of-fold predictions from the teacher, effectively mitigating self-labeling leakage. The authors derive a risk certificate to characterize the trade-off between the volume of generated queries and the fidelity of the generator. Experiments conducted on datasets TALENT and TabArena demonstrate that GEAR significantly enhances the performance of MLP and tree-based models compared to traditional supervised learning methods, achieving notable improvements in AUC scores while drastically reducing inference time and memory usage. This modular approach is agnostic to the choice of generator, teacher, and student architectures, making it broadly applicable across various tabular data tasks.
Methodology
The GEAR framework consists of two main stages: Stage 1 involves generating synthetic covariates to train a student model on soft targets from a frozen teacher model, while Stage 2 re-anchors the student model to the real data distribution using actual labels and out-of-fold predictions to avoid self-labeling leakage. The authors also derive a risk certificate to analyze the trade-off between the volume of generated queries and the fidelity of the generator.
Results
GEAR's two-stage MLP models outperformed supervised MLPs by 1.81β2.00 AUC points on binary tasks and 1.19β1.35 points on multiclass tasks. Additionally, GEAR achieved improvements over real-data-only distillation by 1.76β2.19 points for binary tasks and 2.09β2.40 points for multiclass tasks. The framework also demonstrated substantial reductions in median inference time (57β2866Γ faster) and peak memory usage (1.9β3.3Γ lower) compared to matched supervised baselines.
Implications
The GEAR framework has significant implications for the deployment of machine learning models in resource-constrained environments, particularly in industries relying on tabular data such as finance and healthcare. By enabling efficient inference without sacrificing accuracy, GEAR can facilitate the broader adoption of advanced machine learning techniques in practical applications.
Answer-Level Trust Selection for Physical Vision-Language Reasoning
Multimodal
Computer Vision
Interpretability
- Introduces Answer-Level Trust Selection (ATS) for evaluating individual predictions of VLMs.
- ATS is model-agnostic and requires no fine-tuning or access to internal model logits.
- Utilizes eight behavioral diagnostic scores to generate a unified trust score for predictions.
- Demonstrates that ATS can identify hidden failure modes in VLM predictions.
Read more
Answer-Level Trust Selection for Physical Vision-Language Reasoning
Summary
This paper addresses the challenge of evaluating the reliability of individual predictions made by Vision-Language Models (VLMs) in the context of quantitative physical reasoning. While existing benchmarks focus on overall model performance against ground truth, the authors highlight the need for a mechanism to assess the trustworthiness of specific predictions when ground truth is unavailable. They introduce Answer-Level Trust Selection (ATS), a model-agnostic framework that aggregates eight interpretable behavioral diagnostic scores to produce a unified trust score for each prediction. ATS is designed to identify stable-but-wrong estimates and predictions that rely on textual priors rather than visual evidence. The authors conduct extensive evaluations on the Qwen2.5-VL-7B model and benchmark 20 VLMs to characterize their capabilities in quantitative reasoning. The results indicate that ATS effectively identifies hidden failure modes that traditional evaluation methods may overlook, although it may also lead to a reduction in the retention of correct predictions. This work emphasizes the importance of answer-level reliability assessment in the deployment of VLMs for real-world applications.
Methodology
The authors propose ATS as a post-hoc framework that aggregates eight interpretable behavioral diagnostic scores derived from repeated queries and controlled interventions. This approach allows for the assessment of individual predictions without modifying the original model or requiring additional verifiers.
Results
The evaluation of ATS on Qwen2.5-VL-7B and across 20 VLM backbones reveals its effectiveness in identifying stable-but-wrong predictions and those influenced by textual priors. While ATS improves the rejection of failure cases, it may also decrease the retention of correct predictions, highlighting a trade-off in its application.
Implications
The findings suggest that ATS can enhance the reliability of VLM outputs in practical applications, particularly in scenarios where ground truth is not available. This framework could be crucial for deploying VLMs in safety-critical systems that require accurate quantitative reasoning.
In Two Minds about Lifelong Learning: Exploring Hemispheric Redundancy and Specialisation in Neural Models
Theory
Generative Models
Efficient ML
- Introduction of 4MAS, a dual-hemisphere architecture for continual learning.
- Utilization of a wake-sleep training cycle to enhance memory consolidation.
- Competitive accuracy on Split-MNIST (98.3%), Split-Fashion-MNIST (84.9%), and Split-CIFAR-100 (29.29%).
- Demonstration of low representational drift across tasks.
Read more
In Two Minds about Lifelong Learning: Exploring Hemispheric Redundancy and Specialisation in Neural Models
Summary
This paper addresses the challenge of catastrophic forgetting in machine learning, particularly in the context of continual learning. The authors propose a novel architecture called 4MAS (4 Module Awake/Sleep), inspired by biological learning mechanisms, which incorporates two asymmetric hemispheres that specialize in different memory functions. The architecture mimics the biological processes of experience replay, REM sleep, and bilateral redundancy found in mammalian brains. By implementing a wake-sleep training cycle, the model allows for cross-hemispheric consolidation of knowledge, thereby stabilizing latent representations and minimizing representational drift. The paper presents empirical results demonstrating that 4MAS achieves competitive performance on several benchmark datasets while effectively mitigating forgetting, showcasing the potential of biologically-inspired approaches in enhancing machine learning systems.
Methodology
The authors developed the 4MAS architecture, which consists of two hemispheres with dedicated short- and long-term memory modules. The model alternates between an awake phase for task learning and a sleep phase for cross-hemispheric memory consolidation. This approach leverages generative replay and mimics biological processes such as hippocampal replay and REM sleep to stabilize memory and reduce forgetting.
Results
The 4MAS architecture achieved high accuracy rates on benchmark datasets: 98.3% on Split-MNIST, 84.9% on Split-Fashion-MNIST, and 29.29% on Split-CIFAR-100. The results indicate that the model effectively mitigates catastrophic forgetting while maintaining competitive performance.
Implications
The findings suggest that incorporating biological principles into machine learning architectures can significantly enhance their ability to learn continuously without forgetting. This has potential applications in developing more robust AI systems capable of adapting to new information over time, which is crucial for real-world applications.
SCORE: Subject Coordinate Recovery for Label-Free Cross-Subject EEG-to-Image Retrieval
Multimodal
- SCORE identifies that different subjects express similar concept relationships in distinct coordinate directions.
- The framework allows for EEG-to-image retrieval without the need for target labels or encoder updates.
- SCORE achieves superior accuracy in cross-subject retrieval compared to existing methods.
- The method generalizes well across different EEG encoders and remains effective under various conditions.
Read more
SCORE: Subject Coordinate Recovery for Label-Free Cross-Subject EEG-to-Image Retrieval
Summary
The paper introduces SCORE (Subject Coordinate Recovery), a novel framework aimed at improving EEG-to-image retrieval across different subjects without the need for labeled calibration data. Traditional methods excel in within-subject scenarios but struggle with cross-subject retrieval, often yielding low accuracy due to the unique ways different individuals represent similar visual concepts. The authors analyze EEG features and discover that while subjects maintain similar relationships among concepts, they express these relationships along distinct coordinate axes. SCORE addresses this by aligning EEG data from source subjects with a common image space during training and employing a label-free coordinate recovery method at deployment. This approach allows SCORE to recover the target subject's EEG coordinates without requiring additional data or label updates. The framework was evaluated on two public benchmarks, THINGS-EEG2 and Alljoined-1.6M, demonstrating significant improvements in retrieval accuracy over existing methods, thereby facilitating practical applications of EEG-based visual decoding.
Methodology
SCORE combines recovery-aware source training with coordinate alignment during deployment. It aligns source subject EEG data with a common image space and simulates unseen-subject recovery through source-only episodes. At deployment, SCORE selects EEG-image landmarks using hubness-corrected matching and estimates an orthogonal transformation to recover target EEG coordinates without requiring source data or target labels.
Results
In 200-way retrieval tasks on the THINGS-EEG2 and Alljoined-1.6M benchmarks, SCORE achieved Top-1/Top-5 accuracies of 53.23%/83.55% and 12.01%/32.16%, respectively. These results surpass the strongest evaluated baselines by significant margins, demonstrating SCORE's effectiveness across all held-out subjects.
Implications
The SCORE framework has the potential to enhance EEG-based visual decoding applications, making them more accessible and practical for real-world use without the need for extensive calibration or subject-specific data. This could lead to advancements in fields such as brain-computer interfaces and neurocommunication.
Forgetting, plasticity, and co-observation: a third facet of continual learning
Theory
- Introduction of the co-observation effect as a critical factor in continual learning.
- Demonstration of performance benefits from simultaneous observation of training data.
- Empirical validation across supervised and self-supervised learning paradigms.
- Critique of existing continual learning methods in light of co-observation.
Read more
Forgetting, plasticity, and co-observation: a third facet of continual learning
Summary
This paper addresses the challenges of continual learning in deep neural networks, particularly focusing on the limitations of catastrophic forgetting and loss of plasticity. The authors argue that these two factors alone do not fully account for the performance gap between naive sequential training and offline joint training. They introduce the concept of 'data co-observation' as a crucial third factor that influences continual learning performance. By systematically investigating the benefits of observing training data together, the authors demonstrate that simultaneous observation enhances generalization capabilities beyond mere knowledge retention. Their empirical studies reveal consistent performance improvements in both supervised and self-supervised learning scenarios when training data is co-observed. The paper also critiques existing continual learning mechanisms, suggesting that while distillation methods primarily aid in knowledge retention, memory replay techniques leverage the advantages of co-observation. The findings emphasize the need for continual learning strategies to address the representational deficits caused by the lack of co-observation in separate training settings.
Methodology
The authors employed a diagnostic methodology based on continual pre-training and model ensembles to isolate the co-observation effect. They evaluated continual learning at the representation level using linear probing, which decouples generalization from the dynamics of a continually updated classifier. An ensemble approach was used to control for forgetting, preserving the model's representational state after each training phase.
Results
The results showed that in data-incremental 'chunking' scenarios, incremental joint training consistently outperformed the ensemble baseline, indicating that the absence of co-observation during optimization significantly impacts performance. This effect was observed without additional confounding factors such as distribution shifts or class imbalances.
Implications
The findings suggest that continual learning methods must integrate strategies to mitigate the representational deficits caused by the lack of co-observation in separate training. This could lead to improved generalization capabilities in real-world applications where data arrives sequentially.
A Real-Time Tsetlin Machine-based Non-intrusive Load Monitoring System on MCUs
Efficient ML
Time Series
- Introduction of a Tsetlin Machine-based framework for NILM on MCUs.
- Achieves high precision and recall rates for appliance classification.
- Designed for real-time processing with minimal memory footprint.
- Open-source implementation available for further research and development.
Read more
A Real-Time Tsetlin Machine-based Non-intrusive Load Monitoring System on MCUs
Summary
This paper presents a novel Non-Intrusive Load Monitoring (NILM) system utilizing a Tsetlin Machine (TM) framework, specifically designed for real-time applications on resource-constrained microcontrollers (MCUs). Traditional NILM systems face challenges due to their reliance on computationally intensive algorithms, which limits their deployment in real-time scenarios where data privacy is paramount. The proposed TM-based NILM system reformulates the energy disaggregation problem as a classification task, allowing for efficient processing of household energy consumption data. The system comprises four main components: Edge Detection, Event Pairing, Feature Extraction, and TM-based classification. By employing an event-driven architecture, the system minimizes computational overhead, processing only when significant power transitions are detected. The TM model, known for its lightweight and memory-efficient characteristics, enables the system to achieve an average precision of 90% and recall of 96% for two-appliance classification, and 77% precision and 80% recall for four appliances on the REDD dataset. Furthermore, the model occupies only 18 KB of flash memory and achieves an inference latency of 0.43 ms on an ESP32, demonstrating its suitability for embedded applications.
Methodology
The methodology involves reformulating the NILM problem as a classification task using the Tsetlin Machine. The system processes raw sensor data through Edge Detection to identify power transitions, followed by Event Pairing to group these transitions into complete appliance activity windows. Features are extracted from these windows and classified using the TM model, which operates on binary input features, ensuring computational efficiency suitable for MCUs.
Results
The TM-based NILM system achieved an average precision of 90% and recall of 96% for two-appliance classification, and 77% precision and 80% recall for four appliances on the REDD dataset. The model's memory usage is only 18 KB, with an inference latency of 0.43 ms on an ESP32 microcontroller, indicating its effectiveness for real-time applications.
Implications
The proposed system has significant implications for energy management in smart homes, enabling efficient monitoring of appliance energy consumption while preserving user privacy. Its lightweight design allows for deployment in various embedded systems, potentially leading to widespread adoption in energy-efficient technologies.
A FEM-Based Surrogate Modelling and Optimization Framework for Physics-Constrained Electromagnetic Coil Design
Optimization
- Introduces a FEM-based surrogate modeling framework for electromagnetic coil design.
- Demonstrates the impact of FEM evaluation budget on optimizer performance.
- Compares multiple optimization methods, revealing strengths and weaknesses based on budget constraints.
- Provides an automated Python-MPh-COMSOL workflow for efficient design exploration.
Read more
A FEM-Based Surrogate Modelling and Optimization Framework for Physics-Constrained Electromagnetic Coil Design
Summary
This paper presents a framework for surrogate-assisted optimization of a seven-parameter current-excited coil-core design, constrained by geometric, manufacturing, and mass limitations. The methodology integrates a two-dimensional axisymmetric finite-element method (FEM) model with a MatΓ©rn-5/2 Gaussian-process (GP) surrogate model, facilitating efficient exploration of the design space. The study emphasizes the importance of the available FEM evaluation budget on optimizer performance, comparing Bayesian optimization (BO) with expected improvement (EI) against other methods such as COBYLA and BOBYQA. Results indicate that while EI-BO shows rapid improvement with small budgets, COBYLA performs better at early checkpoints, and BOBYQA achieves the highest mean terminal response. The findings suggest that different optimization methods may excel under varying conditions, highlighting the need for careful consideration of budget and performance metrics in simulation-driven design. The contributions include an automated workflow for geometry and constraint handling, diagnostics for prediction accuracy, and a comparative analysis of optimizer performance under budget constraints.
Methodology
The study employs a Python-MPh-COMSOL workflow that integrates a finite-element method (FEM) model with a Gaussian-process (GP) surrogate model. It utilizes sequential Bayesian optimization with expected improvement (EI) to rank design candidates, while also comparing other optimization methods like COBYLA and BOBYQA under controlled conditions.
Results
The results show that EI-BO is effective with small evaluation budgets, while COBYLA is superior at early checkpoints, and BOBYQA yields the best overall mean terminal response. A retrospective analysis indicates that no single optimizer consistently outperforms others across all metrics, emphasizing the context-dependent nature of optimization performance.
Implications
This framework can be applied to other simulation-driven design problems in engineering, particularly where computational resources are limited. The findings may influence the selection of optimization strategies in similar contexts, guiding practitioners in balancing early progress with terminal performance.
Multi-stage neural operator learning with application for convolutions
Theory
Efficient ML
- Introduction of two multi-stage neural operator learning frameworks: DCNO (supervised) and DGNO (unsupervised).
- Both frameworks improve approximation accuracy by progressively constructing neural operator bases.
- Theoretical analysis supports the convergence and approximation capabilities of the proposed methods.
- Numerical experiments show that the methods achieve high accuracy and efficiency in convolution problems.
Read more
Multi-stage neural operator learning with application for convolutions
Summary
This paper presents two innovative multi-stage neural operator learning frameworks aimed at enhancing the efficiency and accuracy of convolution computations, which are prevalent in various scientific and engineering applications. The first framework, Deep Collocation Neural Operator (DCNO), is a supervised learning method that iteratively refines operator approximations by learning from residuals derived from input-output data pairs. The second framework, Deep Galerkin Neural Operator (DGNO), operates in an unsupervised manner, leveraging the weak form of partial differential equations (PDEs) to train the operator when the target can be represented as a PDE. Both frameworks progressively construct basis operators through multiple training stages, significantly improving approximation accuracy compared to traditional one-shot operator learning approaches. The authors provide theoretical analyses supporting the approximation capabilities of both methods and apply them specifically to convolution problems. Extensive numerical experiments demonstrate that DCNO and DGNO achieve high accuracy, nearing machine precision for convolution tasks, while also offering substantial efficiency gains in scenarios involving numerous queries or parametric variations. The frameworks are further extended to accommodate multi-input operator learning scenarios, showcasing their versatility and robustness in handling variations in both density and kernel of convolutions.
Methodology
The paper proposes two frameworks: DCNO, which uses supervised learning to refine operator approximations through residual learning from data pairs, and DGNO, which employs an unsupervised approach based on the weak form of PDEs to train the operator. Both methods involve multiple training stages to enrich the approximation space, leading to improved accuracy over traditional one-shot learning methods.
Results
The numerical experiments indicate that both DCNO and DGNO achieve high accuracy, approaching machine precision for convolution tasks. The frameworks also provide significant efficiency improvements when handling multiple queries or variations in parameters compared to traditional solvers.
Implications
The proposed frameworks can significantly enhance the computational efficiency and accuracy of convolution evaluations in various applications, such as electrostatics, gravitational simulations, and signal processing. Their ability to handle multi-input scenarios further broadens their applicability in complex scientific computing problems.
An Empirical Benchmark of Deep Time-Series Models for Smart Meter Energy Forecasting
Time Series
- Benchmarking of nine deep learning models for energy forecasting on real-world datasets.
- Forecasting accuracy improves with historical context only to a saturation point; accuracy declines with longer prediction horizons.
- Lightweight architectures can achieve competitive accuracy at lower computational costs.
- Transformer models show potential advantages for under-represented population segments, though not consistently statistically robust.
Read more
An Empirical Benchmark of Deep Time-Series Models for Smart Meter Energy Forecasting
Summary
This paper presents an empirical benchmark of nine deep learning models for time-series forecasting applied to smart meter energy consumption data. The authors evaluate these models on two high-resolution datasets, focusing on the effects of historical input length, prediction horizon, and model architecture on forecasting performance. The study finds that while extending the historical context improves accuracy, it only does so up to a certain point, after which additional data yields diminishing returns. Conversely, accuracy declines as the prediction horizon increases. The results indicate that lightweight architectures can achieve competitive accuracy at lower computational costs, and that architectural differences are more pronounced at longer forecasting horizons and with heterogeneous datasets. Additionally, subgroup analyses reveal that Transformer-based models may perform better for under-represented population segments, although this advantage is not consistently statistically significant across datasets. The findings provide practical insights for selecting appropriate forecasting models in smart meter applications.
Methodology
The authors benchmarked nine modern deep learning models, including linear, MLP-based, convolutional, and Transformer architectures, on two publicly available smart meter datasets. They analyzed the impact of historical input length, prediction horizon, and model architecture on forecasting performance, employing statistical significance tests to assess differences across models.
Results
The study found that deep learning models consistently outperformed classical baselines. While extending historical context improved accuracy, it plateaued after a certain point. Accuracy decreased with longer prediction horizons, and lightweight models performed comparably to more complex architectures at lower computational costs. Architectural differences were significant mainly at longer horizons and on heterogeneous datasets.
Implications
The findings suggest that practitioners can achieve effective energy consumption forecasting using lightweight models, which can reduce computational costs without sacrificing accuracy. The insights into model performance across different population segments can help tailor forecasting approaches to diverse user groups, enhancing energy management strategies.
Transportable Causal Effect Estimation across Networks under Interference
Graph Learning
Theory
- Introduces TranCE, a novel algorithm for causal effect estimation across different networks under interference.
- Establishes graphical conditions for causal effect transportability using a selection diagram.
- Derives transport formulas for direct, spillover, and total effects in the deployment population.
- Demonstrates the effectiveness of the proposed method through extensive experiments on real-world data.
Read more
Transportable Causal Effect Estimation across Networks under Interference
Summary
This paper addresses the challenge of estimating causal effects across different networks where interference occurs, a scenario often encountered in social sciences and public health. Traditional methods assume that the training and deployment networks are the same, which is rarely the case in practice. The authors propose a novel algorithm called TranCE (Transported Causal Effects) that utilizes a selection diagram to model covariate and structural shifts separately. This approach leads to the derivation of transport formulas for direct, spillover, and total effects in the target deployment population. The TranCE algorithm combines an interventional outcome model with a domain density-ratio correction and cross-fitted inference to ensure robustness. The effectiveness of TranCE is validated through extensive experiments on semi-synthetic benchmarks and a real-world field experiment, demonstrating its capability to accurately transport causal effects from one network to another, thereby enhancing intervention strategies in various applications.
Methodology
The authors extend a selection diagram to the network context to separately model covariate and structural shifts. They derive transport formulas for causal effects and implement these using a doubly robust approach that combines an interventional outcome model with a domain density-ratio correction and cross-fitted inference.
Results
The experiments conducted on semi-synthetic benchmarks and a real-world weather-insurance field experiment show that TranCE effectively estimates causal effects in the deployment population, with results aligning closely with held-out randomized estimates, thereby validating the proposed methodology.
Implications
The findings of this research have significant implications for improving intervention strategies in networked systems, particularly in fields such as social networks and public health, where understanding causal effects across different populations is crucial.
NanoSleep: A Parameter-Efficient Hybrid Temporal Convolutional Network for Single-Channel Sleep Stage Classification
Efficient ML
Time Series
- NanoSleep is a compact hybrid architecture for sleep stage classification with only 0.35 million parameters.
- The model effectively combines multi-scale temporal and spectral feature extraction.
- A weighted calibrated focal loss is used to address class imbalance in sleep stage datasets.
- NanoSleep outperforms several state-of-the-art methods while being efficient for deployment on resource-constrained devices.
Read more
NanoSleep: A Parameter-Efficient Hybrid Temporal Convolutional Network for Single-Channel Sleep Stage Classification
Summary
The paper introduces NanoSleep, a compact hybrid temporal convolutional network designed for single-channel sleep stage classification using electroencephalography (EEG). The authors address the challenges of deploying deep learning models on resource-constrained devices, such as high model sizes and class imbalance in sleep stage datasets. NanoSleep integrates a learnable Sinc-convolutional front end, a dual-branch feature extractor for multi-scale temporal and spectral representations, a gated dilated temporal convolutional backbone with channel recalibration, and a conditional random field (CRF) for sequence-level decoding. The model employs a weighted calibrated focal loss to mitigate class imbalance. Evaluated on the Sleep-EDF and Sleep-EDF-Expanded datasets through subject-wise cross-validation, NanoSleep consistently outperforms six baseline methods while maintaining a low parameter count of only 0.35 million trainable parameters. The ablation study conducted confirms the contribution of each component, demonstrating that NanoSleep achieves a balance between accuracy and efficiency, making it suitable for wearable devices and home-based sleep monitoring.
Methodology
NanoSleep employs a hybrid architecture that includes a learnable Sinc-convolutional front end, a dual-branch feature extractor for capturing multi-scale temporal and spectral features, a gated dilated temporal convolutional backbone with channel recalibration, and a CRF for sequence-level predictions. The training framework incorporates imbalance-aware optimization strategies and a weighted calibrated focal loss to enhance performance on minority classes.
Results
The experimental results show that NanoSleep consistently outperforms six baseline models on the Sleep-EDF and Sleep-EDF-Expanded datasets, achieving high classification accuracy with significantly fewer parameters. The ablation study confirms the effectiveness of each component in the architecture.
Implications
NanoSleep's design allows for efficient deployment in wearable devices and home-based sleep monitoring systems, potentially improving accessibility to sleep health assessments. Its ability to maintain high accuracy while being resource-efficient opens avenues for clinical applications in sleep disorder diagnosis and management.
Tensor Field Models
Generative Models
Theory
Efficient ML
- Introduction of Tensor Field Models (TFMs) as a new mathematical structure for machine learning.
- TFMs utilize a learned operator to map component-section families to time-dependent tangent sections.
- The architecture allows for efficient reuse of condition representations across multiple queries.
- Experiments demonstrate improved performance and faster generation through amortized sampling.
Read more
Tensor Field Models
Summary
This paper presents Tensor Field Models (TFMs), which are mathematical structures that utilize a learned operator to map a product of admissible component-section families to a family of time-dependent tangent sections on a Generative State Manifold. The authors emphasize that the analytic and dynamical constraints are determined by the selection of admissible families rather than by the fundamental definition of the model. The paper introduces constructed, component-separable, and Tensor Bundle TFMs, which enhance the structure of the model. In the conditional realizations explored, a structured condition is mapped componentwise to a reusable collection, allowing for efficient evaluations of multiple queries. The TFMs are trained using Flow Matching, and experimental results indicate that these models can enhance performance and accelerate generation through amortized sampling enabled by reusable condition representations. The paper discusses the mathematical background of Tensor Bundles and Tensor Fields, providing a foundation for understanding the proposed models.
Methodology
The authors develop Tensor Field Models by defining a parameterized Field Operator that maps admissible input and output-section families to a Generative State Manifold. They utilize Flow Matching for training the models and explore various architectures, including constructed and component-separable TFMs, to evaluate their performance in generating vector fields.
Results
The experiments conducted show that Tensor Field Models can significantly improve performance in generating vector fields compared to traditional methods. The use of reusable condition representations allows for faster evaluations, particularly in scenarios involving multiple queries.
Implications
The introduction of Tensor Field Models has the potential to enhance various applications in machine learning, particularly in generative modeling tasks where efficiency and performance are critical. The structured approach to handling conditional information may lead to advancements in fields such as robotics and computer vision.
Safe Domain Adaptation for Physics: Overcoming Nuisances, Label Shifts, and Simulation Priors
Theory
- Traditional domain adaptation methods assume that differences between training and target domains are nuisances, which is often not true in physics.
- The study reveals that standard adversarial domain adaptation can lead to biased predictions when energy spectra differ between domains.
- An adaptive domain adaptation method is proposed to focus on genuine physical mismatches by reweighting simulated events.
- A label-free model selection rule is introduced to help identify optimal training configurations without target labels.
Read more
Safe Domain Adaptation for Physics: Overcoming Nuisances, Label Shifts, and Simulation Priors
Summary
This paper addresses the challenges of applying domain adaptation (DA) techniques in physics, particularly when neural networks trained on simulations are used on experimental data. The authors highlight that traditional DA methods rely on the assumption that the differences between domains are merely nuisances, which is often not the case in physics. They present a systematic study using a toy air-shower benchmark that reveals three types of mismatches: physical simulation shifts, detector-response nuisances, and energy-spectrum shifts. The standard adversarial domain adaptation methods struggle when the energy spectra differ, leading to biased predictions. To overcome these issues, the authors propose an adaptive domain adaptation approach that reweights simulated events to focus on genuine physical mismatches. Additionally, they introduce a label-free model selection rule to identify optimal training configurations without relying on target labels. The findings indicate that while DA can be effective when spectra are aligned, it can introduce significant biases when they are not, necessitating the proposed adaptive methods.
Methodology
The authors conducted a systematic study using a toy air-shower benchmark to analyze the effects of different types of mismatches on domain adaptation. They compared standard adversarial domain adaptation techniques with their proposed adaptive domain adaptation approach, which reweights simulated events to focus on genuine physical mismatches. They also developed a label-free model selection rule for optimal training configuration identification.
Results
The results demonstrated that standard domain adversarial neural networks perform well when the energy spectra of the source and target domains are aligned. However, when the spectra differ, these methods tend to bias the predictions towards the simulation prior. The proposed adaptive domain adaptation method effectively mitigated these biases by focusing on the actual physical mismatches, leading to more accurate predictions.
Implications
The findings have significant implications for the application of machine learning in physics, particularly in areas where simulations are used to analyze experimental data. The proposed methods can enhance the robustness of neural networks against systematic shifts, improving the reliability of predictions in experimental settings.
MileGPO: Milestone Inference with Local Evidence for Graph-Based Policy Optimization of Long-Horizon LLM Agents
Reinforcement Learning
Large Language Models
Optimization
- MileGPO reveals the limitations of final-goal-distance credit in distinguishing same-state branches.
- The proposed method learns from rollout graphs and on-policy rewards without needing external annotations or auxiliary inference.
- MileGPO achieves state-of-the-art performance on ALFWorld and WebShop benchmarks.
- The method effectively calibrates intermediate credit, leading to improved decision-making in long-horizon tasks.
Read more
MileGPO: Milestone Inference with Local Evidence for Graph-Based Policy Optimization of Long-Horizon LLM Agents
Summary
The paper introduces MileGPO, a novel approach to credit assignment in long-horizon reinforcement learning (RL) for large language model (LLM) agents. Traditional methods struggle with assigning credit to intermediate actions when supervision is limited to final rewards. MileGPO addresses this by deriving process-level credit from grouped on-policy rollouts through three main components: Milestone Discovery (MD), which identifies significant milestones and traps; Reliability-Calibrated Shaping (RCS), which weights these milestones based on their reliability; and Progress-Contrastive Calibration (PCC), which assesses local progress and compares transitions from the same state. The method does not require auxiliary models or additional environment interactions, making it efficient. Experimental results demonstrate that MileGPO achieves state-of-the-art performance on benchmarks ALFWorld and WebShop, with a notably small in-distribution to out-of-distribution gap, indicating strong generalization capabilities. The findings highlight the importance of reliable intermediate credit and the effectiveness of the proposed calibration mechanisms in improving agent performance.
Methodology
MileGPO employs a three-part methodology: Milestone Discovery (MD) identifies significant milestones and traps from successful and failed rollouts; Reliability-Calibrated Shaping (RCS) weights these milestones based on their empirical reliability; and Progress-Contrastive Calibration (PCC) evaluates local progress and compares sibling transitions to refine credit assignment. This approach leverages on-policy rollouts and does not require additional models or interactions with the environment.
Results
MileGPO demonstrated superior performance on the ALFWorld and WebShop benchmarks, achieving state-of-the-art results. The in-distribution to out-of-distribution gap on ALFWorld was only 1.69 points, outperforming previous methods such as GiGPO and GraphGPO. Further analysis indicated that the calibration mechanisms effectively resolved ambiguities in credit assignment, leading to more reliable performance.
Implications
The findings suggest that MileGPO can enhance the training of LLM agents in complex, long-horizon tasks, potentially improving their decision-making capabilities in real-world applications such as web navigation and embodied instruction following. The methodology may also inform future research on credit assignment in reinforcement learning.
SAGE-XGBoost: Spatially Augmented Graph EmbeddingsβMachine Learning Framework for Natural Hazards Susceptibility Mapping under Data Scarcity
Graph Learning
- SAGE-XGBoost improves natural hazard susceptibility mapping under data scarcity.
- The framework combines data augmentation with graph embeddings for feature enhancement.
- Significant performance improvements were observed over traditional machine learning models.
- The model achieved high AUC values for both landslide and wildfire susceptibility.
Read more
SAGE-XGBoost: Spatially Augmented Graph EmbeddingsβMachine Learning Framework for Natural Hazards Susceptibility Mapping under Data Scarcity
Summary
This paper addresses the challenge of natural hazard susceptibility mapping, which is often limited by the availability of labeled data. The authors propose a novel framework called SAGE (Spatially Augmented Graph Embeddings) that enhances prediction capabilities under data-scarce conditions. SAGE integrates controlled noise-based data augmentation with neighborhood-based graph embeddings to derive local spatial statistics. These statistics are processed using principal component analysis and combined with environmental covariates and spatial coordinates to create robust features for the XGBoost model. The SAGE-XGBoost model was evaluated for landslide and wildfire susceptibility mapping, demonstrating significant improvements over conventional machine learning models. The results showed an absolute improvement of over 33 percentage points compared to Spatial XGBoost, achieving AUC values of approximately 0.97 for landslide susceptibility and 0.95 for wildfire susceptibility. Feature importance analysis confirmed the value of graph embeddings in enhancing prediction accuracy and spatial coherence while minimizing local noise amplification. The study concludes that SAGE-XGBoost serves as an efficient and transferable alternative to deep learning methods for environmental hazard assessments, particularly in scenarios with limited supervision.
Methodology
The methodology involves constructing a K-nearest neighbor graph to derive local spatial statistics, which are then reduced using principal component analysis. These features are integrated with environmental covariates and spatial coordinates to train the XGBoost model, resulting in the SAGE-XGBoost framework.
Results
The SAGE-XGBoost model outperformed conventional and spatially explicit machine learning models, achieving an absolute improvement of over 33 percentage points compared to Spatial XGBoost. The model reached AUC values of approximately 0.97 for landslide susceptibility and 0.95 for wildfire susceptibility.
Implications
The SAGE-XGBoost framework offers a robust alternative for natural hazard susceptibility mapping, particularly in data-scarce environments. Its efficiency and transferability make it suitable for various geospatial prediction tasks and environmental hazard assessments.
H$^2$EDL: Hyper Evidential Deep Learning for Hierarchical Classification
Computer Vision
Theory
Interpretability
- H2EDL provides a unified framework for uncertainty representation in hierarchical classification.
- The model uses a taxonomy to create a linear-size focal family for hyper-evidential beliefs.
- H2EDL reduces calibration error by approximately half compared to traditional methods.
- The model maintains hierarchical coherence and exposes uncertainty at each decision level.
Read more
H$^2$EDL: Hyper Evidential Deep Learning for Hierarchical Classification
Summary
The paper introduces H2EDL, a novel approach for hierarchical classification that addresses the challenge of uncertainty representation in fine-grained recognition tasks. Traditional methods either provide a single measure of uncertainty at the leaf level or fail to incorporate hierarchical structures effectively. H2EDL leverages the taxonomy of classes to create a hyper-evidential model that captures uncertainty across different levels of a hierarchical label space. By assigning local Dirichlet opinions at each branching node, the model maintains a coherent structure while allowing for uncertainty to be expressed at multiple levels. This dual interpretation enables H2EDL to function both as a hierarchical classifier and as a probabilistic model that reflects belief and uncertainty. The authors demonstrate that H2EDL significantly reduces calibration error compared to cross-entropy baselines, particularly at deeper levels of the hierarchy, while also improving the preservation of coarse category predictions during fine-grained mistakes. The model's design eliminates the need for composite labels and auxiliary structures, making it a more efficient and principled solution for hierarchical classification tasks.
Methodology
H2EDL employs a hyper-evidential framework that assigns local Dirichlet opinions at each branching node of a hierarchical taxonomy. This allows for the computation of leaf probabilities through a sequence of local decisions, capturing uncertainty at multiple levels of the hierarchy. The model is trained using standard leaf supervision, eliminating the need for composite labels or auxiliary networks.
Results
The experimental results show that H2EDL reduces calibration error significantly, especially at deeper hierarchy levels, and achieves similar leaf-level accuracy while preserving the correct coarse category 19% more often when fine-grained mistakes occur. This demonstrates the model's effectiveness in managing uncertainty in hierarchical classification tasks.
Implications
H2EDL has potential applications in various domains requiring hierarchical classification, such as fine-grained image recognition, natural language processing tasks with hierarchical labels, and any scenario where uncertainty representation is crucial for decision-making.
LΓ©vy Attention: Single-Pass Predictive Uncertainty for Continuous-Time Attention
Time Series
- LΓ©vy Attention integrates uncertainty estimation directly into the attention mechanism, eliminating the need for separate models or additional passes.
- The mechanism uses a stochastic formulation that allows for closed-form computation of both predictions and their uncertainty.
- Experiments indicate that LΓ©vy Attention maintains high accuracy while significantly improving uncertainty estimation, especially in sparse datasets.
- The method is computationally efficient, providing error bars and uncertainty metrics with minimal additional cost.
Read more
LΓ©vy Attention: Single-Pass Predictive Uncertainty for Continuous-Time Attention
Summary
This paper introduces LΓ©vy Attention, a novel attention mechanism designed to provide predictive uncertainty for irregularly-sampled time series data. Traditional models often fail to convey the trustworthiness of their predictions, especially when dealing with sparse data. LΓ©vy Attention addresses this gap by integrating uncertainty estimation directly into the attention layer, allowing it to report both predictions and their associated uncertainty at no additional computational cost. The mechanism utilizes a stochastic formulation based on an inhomogeneous Poisson random measure, which enables the model to maintain a continuous (time Γ channel) index space. The output of LΓ©vy Attention is a stochastic integral that captures both the evidence of the predictions and the disagreement among the attended values. The authors demonstrate that this approach not only retains the accuracy of existing models but also improves uncertainty estimation without requiring additional model complexity or training overhead. Experiments show that LΓ©vy Attention outperforms traditional uncertainty estimation methods, particularly in sparse data scenarios, and provides valuable insights into the reliability of predictions.
Methodology
The authors propose a cross-attention operator that functions as a stochastic integral against an inhomogeneous Poisson random measure. This formulation allows for the simultaneous computation of predictions and uncertainty metrics during a single forward pass, leveraging the properties of the Poisson measure to capture evidence and disagreement in the predictions.
Results
LΓ©vy Attention was tested against the t-PatchGNN benchmark, showing a maximum accuracy drop of only 5.6% compared to matched controls, with no loss on the sparsest dataset. The method also outperformed traditional uncertainty estimation techniques, achieving the best sparsification error in dense datasets and demonstrating effective calibration of uncertainty metrics.
Implications
The introduction of LΓ©vy Attention has significant implications for applications involving irregularly-sampled time series data, such as clinical monitoring and sensor networks. By providing a reliable measure of uncertainty alongside predictions, it enhances decision-making processes in critical domains where trust in data is paramount.
When Does Dynamic Ensembling Pay Off? Diagnosing Regionwise Gains in Regression under Distribution Shift
Theory
Optimization
Efficient ML
- Introduction of bDCF5, a diagnostic tool for assessing dynamic ensembling gains in regression under distribution shifts.
- High predictive accuracy of bDCF5 with a Spearman correlation of +0.98 across diverse dataset-shift pairs.
- Development of the Probe-Validated Ensemble Selector to choose between static and dynamic models based on validated gains.
- Demonstrated significant risk reduction in test scenarios using dynamic ensembling compared to static methods.
Read more
When Does Dynamic Ensembling Pay Off? Diagnosing Regionwise Gains in Regression under Distribution Shift
Summary
This paper investigates the effectiveness of dynamic ensembling in regression tasks under distribution shifts, where the input-dependent combination of models may outperform static blends. The authors introduce a diagnostic tool, bDCF5, which estimates the potential gains from reallocating trust across different regions of the input space based on a small labeled sample from the target domain. The study evaluates 12 dataset-shift pairs, demonstrating that bDCF5 accurately predicts regionwise gains with a high Spearman correlation of +0.98. The paper also presents the Probe-Validated Ensemble Selector, which decides between static and dynamic models based on the validated gains from the probe. The results indicate that dynamic ensembling can significantly reduce test risk, achieving improvements of 11% and 16% over static methods in two deployments. Additionally, the authors release OpenRegShift, a reproducible evaluation framework for regression ensembles under distribution shifts, enhancing the accessibility of their findings.
Methodology
The authors utilize a labeled probe from the target domain to learn a partition of the input space, comparing the performance of static and regionwise convex blends. The bDCF5 diagnostic is cross-fitted to estimate the realizable gains from dynamic ensembling. A controlled generator is employed to analyze the conditions under which dynamic gains occur, and the Probe-Validated Ensemble Selector is used to validate and select the best model for deployment.
Results
The bDCF5 diagnostic predicts regionwise gains with a Spearman correlation of +0.98, and a sensitivity analysis maintains a correlation of +0.83. The Probe-Validated Ensemble Selector successfully matched or improved upon static performance in all 12 runs, achieving risk reductions of 11% and 16% in two deployments. The gate mechanism effectively rejected candidates that would have incurred excessive losses.
Implications
The findings suggest that dynamic ensembling can be a powerful strategy in regression tasks, particularly in scenarios with distribution shifts. The developed tools and methodologies can assist practitioners in making informed decisions about model deployment, potentially leading to more robust and accurate predictions in real-world applications.
Inadvertent Context Leakage in Language Models
NLP
Large Language Models
Theory
- Inadvertent context leakage allows sensitive information to be reconstructed from benign model outputs.
- Controlled experiments show high accuracy in reconstructing numeric secrets from model responses.
- More capable models exhibit greater leakage, suggesting a correlation between model capability and privacy risks.
- The study introduces a suppression mechanism to mitigate leakage and evaluates its effectiveness across various models.
Read more
Inadvertent Context Leakage in Language Models
Summary
This paper investigates the phenomenon of inadvertent context leakage in language models (LLMs), particularly concerning sensitive user data such as calendars, credentials, and health records. The authors explore whether the presence of such secrets in a model's context window can lead to hidden correlations in the model's outputs, allowing for the reconstruction of these secrets even when the model refuses direct extraction attempts. The study introduces a novel adaptive attack method that assumes black-box access to the model, enabling adversaries to engineer prompts that exploit this leakage. Through controlled experiments on eight proprietary models, the authors demonstrate that 2-digit secrets can be reconstructed with near-perfect accuracy, while 4-digit secrets achieve an 82% exact match. The findings suggest that more capable models are more prone to leakage, indicating that this issue is a byproduct of model capability rather than a simple bug. The paper also details two practical attacks: one that infers semantic predicates about user memories and another that extracts full Social Security Numbers from a production-style agent.
Methodology
The authors formalize a threat model based on predicate inference, where an adaptive adversary with black-box access learns to exploit model-specific channels that leak in-context secrets. They conduct controlled experiments on eight proprietary models, using benign prompts to assess the models' ability to protect sensitive information. The study also evaluates a proposed suppression mechanism to reduce leakage.
Results
The experiments reveal that 2-digit secrets can be reconstructed with 100% accuracy on certain models, while 4-digit secrets achieve an 82% exact match. The study finds that conditioning prompts on previously recovered digits significantly improves recovery rates. Additionally, the proposed suppression mechanism shows promise in reducing leakage across different models and contexts.
Implications
The findings highlight significant privacy concerns for LLMs used in sensitive applications, suggesting that developers must implement robust mechanisms to prevent inadvertent leakage. The study also emphasizes the need for ongoing research into the privacy implications of increasingly capable AI systems.
A comparison between ceiling-mounted FMCW, IR-UWB and Wi-Fi radar for in-bedroom human activity monitoring and sleep interruption detection
Time Series
- Controlled comparison of FMCW, IR-UWB, and Wi-Fi sensing technologies under identical conditions.
- IR-UWB outperforms in cross-subject activity recognition, while FMCW excels in generalization to new environments.
- All technologies demonstrate high performance in sleep monitoring tasks.
- Identifies a trade-off between recognition performance and robustness based on signal characteristics.
Read more
A comparison between ceiling-mounted FMCW, IR-UWB and Wi-Fi radar for in-bedroom human activity monitoring and sleep interruption detection
Summary
This paper presents a comparative study of three radio frequency (RF) sensing technologiesβfrequency-modulated continuous wave (FMCW) radar, impulse radio ultra-wideband (IR-UWB), and Wi-Fiβspecifically for human activity recognition (HAR) and sleep monitoring in healthcare settings. The authors conducted a controlled experiment involving synchronized recordings from 20 participants across six different room layouts. All technologies were evaluated using the same convolutional neural network (CNN) architecture to ensure that performance differences were attributable to the sensing technologies rather than the model design. The study found that IR-UWB achieved the highest cross-subject activity recognition performance at 89.0% macro F1 score, while FMCW demonstrated superior generalization to unseen room layouts with an 83.8% macro F1 score. For sleep monitoring tasks, all technologies exceeded 92% macro F1 in unseen environments. The results highlight a trade-off between recognition performance and environmental robustness, influenced by factors such as range resolution and antenna diversity. The findings provide practical guidelines for designing RF sensing systems in healthcare applications.
Methodology
The study utilized a controlled experimental setup with synchronized recordings from 20 participants across six room layouts. A uniform convolutional neural network (CNN) was employed to evaluate the performance of each sensing technology on both fine-grained human activity recognition and coarse sleep monitoring tasks.
Results
IR-UWB achieved the highest macro F1 score of 89.0% for activity recognition, while FMCW radar showed an 83.8% macro F1 score for generalization to unseen layouts. All technologies exceeded 92% macro F1 in sleep monitoring tasks, indicating strong performance across the board.
Implications
The findings suggest that RF sensing technologies can be effectively utilized for non-intrusive healthcare monitoring, offering privacy-preserving alternatives to camera-based systems. The insights gained can guide the design of future RF sensing systems tailored for healthcare applications.
SAE-Xplainers: Rule-Based Feature Interpretation for Extreme Earth Events
Interpretability
- Introduction of a geographic location-aware SAE (GeoTopK) for ExEE analysis.
- Development of SAE-Xplainers for rule-based interpretation of complex features.
- Demonstrated improvements in reconstruction performance and feature utilization.
- Validated on three types of extreme Earth events: fires, tropical cyclones, and atmospheric rivers.
Read more
SAE-Xplainers: Rule-Based Feature Interpretation for Extreme Earth Events
Summary
The paper addresses the challenge of interpretability in deep learning models applied to extreme Earth events (ExEE) such as wildfires, tropical cyclones, and atmospheric rivers. The authors propose a novel approach called SAE-Xplainers, which combines geographic location-based modulation of Sparse Autoencoders (SAEs) with an ensemble of rule-based models to interpret high-dimensional features derived from complex environmental data. The geographic modulation captures local semantic meanings of environmental patterns, enhancing the reconstruction performance and feature utilization of the SAEs. The SAE-Xplainers facilitate the interpretation of these features by converting them into human-understandable rules, consistent with scientific literature. The method is validated across three ExEE types, demonstrating improved interpretability and the ability to identify feature absorption, which refers to the sparsity-driven phenomenon of producing overly specific features from hierarchical concepts. Overall, the proposed framework enhances the transparency and usability of deep learning models in operational settings for climate-related predictions and assessments.
Methodology
The authors developed a geographic location encoder that modulates the inputs to a k-sparse autoencoder (GeoTopK) to ensure location-aware feature extraction. They then employed an ensemble of rule-based models (SAE-Xplainers) to interpret the extracted features, translating them into understandable rules that reflect the underlying environmental patterns.
Results
The proposed GeoTopK SAE outperformed traditional location-agnostic SAEs in terms of reconstruction accuracy and reduced the rate of dead features. The SAE-Xplainers provided faithful interpretations of the features, revealing insights consistent with existing scientific knowledge and facilitating the identification of feature absorption.
Implications
The findings suggest that the SAE-Xplainers framework can significantly enhance the interpretability of deep learning models in climate science, making them more applicable in operational settings for forecasting and assessing extreme weather events. This could lead to better preparedness and risk management strategies in response to climate change.
Complementary, Not Cumulative: Interaction Effects in Physics-Informed Neural Networks for Navier-Stokes Vortex Shedding
Theory
Optimization
Efficient ML
- Systematic evaluation of various PINN techniques reveals that many do not improve performance when used alone.
- The combination of periodic activations and causal weighting is identified as effective for learning vortex shedding.
- Adding more techniques beyond the effective combination can lead to performance degradation.
- The study highlights the nonlinear interactions between different PINN enhancements.
Read more
Complementary, Not Cumulative: Interaction Effects in Physics-Informed Neural Networks for Navier-Stokes Vortex Shedding
Summary
This paper investigates the interaction effects of various techniques used to enhance Physics-Informed Neural Networks (PINNs) in solving the Navier-Stokes equations for vortex shedding. While PINNs offer a promising alternative to traditional computational fluid dynamics (CFD) solvers by embedding governing equations into the training loss, the study reveals that many proposed enhancements, when tested in isolation, do not outperform a baseline model. The author systematically evaluates a range of architectural changes, activation functions, and loss reweighting schemes on the DFG/SchΓ€ferβTurek unsteady cylinder wake benchmark. The key finding is that combining periodic (SIREN) activations with causal weighting significantly improves performance, achieving an average relative L2 error of 4.1% compared to an OpenFOAM reference solution. However, adding more techniques leads to performance degradation, indicating that effective interventions can interact nonlinearly and that more complex training strategies are not necessarily beneficial. This work emphasizes the importance of understanding the interactions between different PINN techniques rather than validating them in isolation.
Methodology
The author conducted a systematic evaluation of widely-used PINN techniques, including architectural changes, activation functions, and loss reweighting schemes, specifically on the DFG/SchΓ€ferβTurek unsteady cylinder wake benchmark. The performance of these techniques was assessed in isolation and in combination to understand their interaction effects.
Results
The study found that while many techniques did not outperform the baseline when applied individually, the combination of periodic activations (SIREN) and causal weighting significantly improved the model's ability to reconstruct velocity and pressure fields, achieving a 4.1% average relative L2 error compared to a reference solution. However, the introduction of additional techniques led to catastrophic performance degradation, underscoring the complexity of interactions among enhancements.
Implications
The findings suggest that careful consideration of the interactions between different techniques is crucial for improving PINN performance in fluid dynamics applications. This could lead to more efficient and effective modeling of complex fluid behaviors in engineering and scientific research.
DecoVAE: a Lightweight Interpretable Trend-Seasonal VAE Framework for Efficient Probabilistic Time Series Forecasting
Time Series
Generative Models
Interpretability
- DecoVAE provides a lightweight and interpretable framework for probabilistic time series forecasting.
- The model explicitly decomposes time series into trend and seasonal components using specialized mechanisms.
- DecoVAE achieves significant accuracy improvements over strong baselines while maintaining high computational efficiency.
- The framework employs a differential regularizer for trend smoothness and complex-domain modeling for seasonality.
Read more
DecoVAE: a Lightweight Interpretable Trend-Seasonal VAE Framework for Efficient Probabilistic Time Series Forecasting
Summary
The paper introduces DecoVAE, a novel lightweight framework designed for probabilistic time series forecasting that effectively separates trend and seasonal components. Traditional forecasting methods often struggle with accurately modeling these distinct dynamics, leading to inefficiencies and a lack of interpretability. DecoVAE addresses these challenges by employing domain-specific inductive biases to decompose time series data into trend and seasonal components. The trend component is modeled using a differential regularizer inspired by the Hodrick-Prescott filter, ensuring structural smoothness, while the seasonal component is captured in the complex frequency domain using a complex Gaussian VAE, which allows for the effective representation of periodic patterns. The authors validate DecoVAE through extensive experiments on seven real-world datasets, demonstrating its superior performance over existing baselines in terms of accuracy and efficiency. The framework achieves significant reductions in Continuous Ranked Probability Score (CRPS) and Normalized Mean Absolute Error (NMAE) for both short-term and long-term forecasting, while also reducing model weight and increasing speed compared to other methods.
Methodology
DecoVAE utilizes a variational autoencoder architecture to explicitly decompose time series data into trend and seasonal components. The trend component is regularized using a Hodrick-Prescott-inspired penalty to ensure smoothness, while the seasonal component is modeled in the complex frequency domain to capture periodic patterns effectively. This dual approach allows for well-calibrated probabilistic forecasts that reflect the distinct dynamics of the time series.
Results
DecoVAE demonstrated up to 14.96% improvement in CRPS and 23.30% in NMAE for short-term forecasting, and up to 52.68% and 26.51% improvements for long-term forecasting compared to strong baseline methods. Additionally, the model's weight was reduced by up to 93%, and its speed was accelerated by up to 74% compared to the second-best method.
Implications
The DecoVAE framework has significant implications for various applications requiring accurate probabilistic time series forecasting, such as finance, energy, weather, and traffic management. Its lightweight and interpretable nature makes it suitable for deployment in resource-constrained environments while providing reliable forecasts.
When to Retrain: An Empirical Study of Retraining Policies for Streaming ML Under Concept Drift, Budget, and Latency Constraints
Theory
Efficient ML
Time Series
- Incremental learning significantly impacts the effectiveness of retraining policies.
- Periodic retraining outperforms reactive policies under abrupt and gradual drift.
- Reactive policies exhibit systematic failures, including budget exhaustion and drift blindness.
- A latency-budget interaction can reduce effective retraining budgets by half.
Read more
When to Retrain: An Empirical Study of Retraining Policies for Streaming ML Under Concept Drift, Budget, and Latency Constraints
Summary
This paper investigates the critical decision of when to retrain machine learning models in production systems facing concept drift, particularly under constraints of budget and latency. The study presents a controlled empirical analysis of three retraining policiesβperiodic retraining, error-threshold triggering, and statistical drift-triggered retraining using ADWINβagainst a no-retrain baseline. The experiments were conducted across 3,933 runs involving three drift scenarios (abrupt, gradual, recurring), varying budget levels, latency constraints, and two datasets (LUFlow network-intrusion detection and LendingClub loan default). The findings reveal that the most significant factor influencing model performance is whether the deployed model can learn incrementally. For models with incremental updates, no retraining policy significantly outperformed the no-retrain baseline. However, without incremental learning, periodic retraining proved superior, particularly under abrupt and gradual drift conditions. The study also highlights the limitations of reactive policies, such as budget exhaustion and inefficiencies in drift detection, and introduces a latency-budget interaction that can halve effective retraining budgets. The paper concludes with the release of a comprehensive simulator and datasets for reproducibility.
Methodology
The authors developed an open-source streaming simulator that incorporates various drift scenarios, retraining policies, and explicit budget and latency constraints. They conducted a factorial analysis of the policies across different datasets and drift types, employing statistical significance testing to evaluate the performance of each policy.
Results
The results indicate that with incremental updates, no retraining policy significantly outperformed the no-retrain baseline. In contrast, without incremental updates, periodic retraining led to accuracy improvements of 15β55 percentage points compared to reactive policies, which only excelled under recurring drift conditions. Systematic failures of reactive policies were documented, particularly regarding budget management and drift detection.
Implications
The findings suggest that organizations should prioritize incremental learning capabilities in their ML systems to enhance performance under concept drift. Additionally, the study provides actionable insights for developing retraining strategies that optimize budget and latency constraints, potentially improving the robustness of production ML systems.
Vector Symbolic Policy Gradient
Reinforcement Learning
Theory
Efficient ML
- Introduction of VSPG, a discrete-action policy-gradient actor using unit-norm hypervectors.
- Exact vector-symbolic interpretation of the policy-gradient update as advantage-weighted bundling.
- Creation of fixed-size compressed kernel memories for efficient learning and inference.
- Demonstration of competitive performance and sample efficiency across multiple benchmarks.
Read more
Vector Symbolic Policy Gradient
Summary
The paper introduces Vector-Symbolic Policy Gradient (VSPG), a novel discrete-action policy-gradient actor that utilizes Vector Symbolic Architecture (VSA) to represent actions as unit-norm hypervectors. The authors establish a direct connection between VSA and discrete-action policy gradients, demonstrating that the standard softmax policy-gradient update can be interpreted as a vector-symbolic operation involving advantage-weighted bundling and normalization. This approach allows VSPG to create fixed-size compressed kernel memories that efficiently transfer advantage evidence across similar states without retaining past samples during inference. The authors evaluate VSPG across various benchmarks, including classic control tasks, MiniGrid, and multi-agent SustainGym, showing that it achieves competitive performance and superior sample efficiency compared to traditional neural and linear actor baselines. Notably, VSPG exhibits robust performance under conditions of post-training quantization and random bit flips, making it suitable for deployment in unreliable edge systems. The paper highlights the potential of VSPG to enhance the robustness and efficiency of reinforcement learning agents in real-world applications.
Methodology
The authors employ Vector Symbolic Architecture (VSA) to represent actions as hypervectors and derive the policy-gradient update as a vector-symbolic operation. They utilize advantage-weighted bundling and normalization to update the action representations, ensuring that the method is computationally efficient and robust against noise and quantization.
Results
VSPG achieves competitive returns on classic control tasks, MiniGrid, and multi-agent SustainGym, outperforming traditional neural and linear actors in terms of sample efficiency. The method also demonstrates significantly better performance under conditions of post-training bit-flip corruption, indicating its robustness compared to existing approaches.
Implications
The findings suggest that VSPG could be effectively deployed in resource-constrained environments, such as edge computing systems, where reliability and efficiency are critical. This work opens avenues for further exploration of VSA in reinforcement learning and other domains requiring robust decision-making under uncertainty.
Evaluating Neural Cartographic Relief Shading for Urban Environments: A Downtown Calgary Study Using High-Resolution DEM and DSM Data
Computer Vision
- The study compares traditional analytical hillshading methods with a neural-based approach in an urban environment.
- Eduard, a neural system trained on mountainous terrain, shows potential for effective urban shading through parameter tuning.
- The research identifies strengths and weaknesses of both methods, emphasizing the need for tailored neural training for urban contexts.
- High-resolution DEM and DSM data from downtown Calgary provide a relevant test case for evaluating hillshading techniques.
Read more
Evaluating Neural Cartographic Relief Shading for Urban Environments: A Downtown Calgary Study Using High-Resolution DEM and DSM Data
Summary
This paper investigates the effectiveness of analytical and neural-based hillshading techniques in urban settings, specifically focusing on downtown Calgary using high-resolution digital elevation model (DEM) and digital surface model (DSM) data. The study compares traditional single-direction and multi-direction analytical hillshading methods against the neural-based Eduard system, which was originally trained on mountainous landscapes. The central aim is to assess whether parameter tuning in Eduard can yield visually compelling and cartographically useful results in an urban context, despite its training bias. The research emphasizes the importance of terrain type, generalization parameters, and detail representation in producing effective urban relief shading. The findings reveal that while analytical methods maintain reliability in certain aspects, Eduard can produce unexpectedly strong results under specific conditions, highlighting the need for future neural model training tailored to urban environments. The study contributes to the broader discourse on terrain representation by exploring the adaptability of neural approaches to complex urban morphologies.
Methodology
The methodology involves a comparative cartographic experiment that evaluates both analytical and neural-based hillshading methods. The study uses high-resolution DEM and DSM data to assess visual legibility and cartographic effectiveness rather than relying solely on algorithmic accuracy metrics. Parameter settings for both methods are varied to understand their impact on urban morphology representation.
Results
The results indicate that while traditional analytical hillshading methods are reliable, the neural-based Eduard system can produce visually compelling results in urban settings when appropriately tuned. The study highlights specific conditions under which Eduard outperforms analytical methods, as well as scenarios where it fails due to its training bias towards natural landscapes.
Implications
The findings suggest that neural-based approaches can enhance urban cartography and visualization by providing more nuanced representations of complex urban forms. This research advocates for future developments in neural model training specifically aimed at urban environments, potentially improving urban planning and geographic information systems.
Mechanistic Tomography: Designed Measurement for Control-Oriented Interpretability
Interpretability
- Introduces mechanistic tomography as a unified framework for measurement in mechanistic interpretability.
- Establishes a common measurement language for various interpretability methods.
- Demonstrates the importance of control-oriented validation in assessing interpretability.
- Provides empirical results showing the effectiveness of the proposed measurement strategies.
Read more
Mechanistic Tomography: Designed Measurement for Control-Oriented Interpretability
Summary
This paper introduces the concept of mechanistic tomography, which aims to unify various measurement methods used in mechanistic interpretability of machine learning models. The author formulates a common measurement problem that encompasses techniques such as coordinate patching, attribution patching, subset interventions, and Hessian-vector products. By expressing the measurement problem in a structured form, the paper provides a framework that allows for a systematic approach to recovering internal mechanisms and intervention effects. The methodology involves starting with the least costly measurements, testing on held-out interventions, calibrating corrections, and expanding the measurement family as needed. The paper emphasizes the importance of control-oriented validation, where an estimate acts as an observer guiding interventions. Through experiments on models like GPT-2-small and Qwen-2.5-7B, the author demonstrates the effectiveness of the proposed methods in recovering interaction effects and improving predictive accuracy. The findings highlight the necessity of a measurement-first approach in mechanistic interpretability, especially as model architectures become less transparent.
Methodology
The methodology involves formulating the measurement problem as a linear equation, where different measurement techniques are analyzed under a common framework. The approach includes starting with simple measurements, testing their effectiveness, calibrating corrections, and expanding the measurement family based on residual errors. The paper also discusses the implications of observer error in control settings.
Results
The experiments conducted on GPT-2-small and Qwen-2.5-7B show that the proposed measurement strategies can effectively recover interaction effects and achieve high predictive accuracy (e.g., R2 = .983 on a finite refusal-response surface). The results indicate that a calibrated additive map can significantly improve predictions, while the necessity of adding interactions is determined by held-out error.
Implications
The findings suggest that mechanistic tomography can enhance the interpretability of machine learning models, particularly in control applications. This approach can lead to better understanding and prediction of model behaviors, which is crucial as models become more complex and less interpretable.
DraftFM: A Foundation Model for Day-Zero Drafting in Magic: The Gathering
Theory
- DraftFM operates effectively in a day-zero drafting context, scoring unseen cards based on public information.
- The model achieved significant predictive accuracy, outperforming random selection and existing models trained on specific sets.
- It provides a novel approach to card representation using frozen embeddings, avoiding reliance on set-specific data.
- The model's predictions for the unreleased set 'The Hobbit' were validated against expert opinions.
Read more
DraftFM: A Foundation Model for Day-Zero Drafting in Magic: The Gathering
Summary
The paper introduces DraftFM, a discrete-choice policy model designed for day-zero drafting in Magic: The Gathering (MTG), where players draft cards from a new expansion before any draft logs are available. DraftFM scores cards based solely on their public attributes, using a 775-dimensional representation that includes structured features and a fixed text embedding, without relying on card identities or usage statistics. This approach allows the model to generalize to unseen cards effectively. The model, with 1.6 million parameters, was trained on 149 million human picks from 29 expansions and demonstrated impressive predictive capabilities, achieving top-1 agreement rates of 50.8%, 60.4%, and 56.7% on held-out expansions, significantly outperforming random selection. Furthermore, it provided a card ranking for the unreleased set 'The Hobbit' shortly before its official release, which aligned well with expert evaluations. The study emphasizes the potential of feature-only representations for drafting in new sets, addressing a gap in existing literature regarding day-zero deployment.
Methodology
DraftFM employs a discrete-choice policy framework, utilizing a 775-dimensional frozen representation for each card based on public data. The model is trained on a large dataset of human picks from multiple expansions, allowing it to generalize to new sets without prior exposure. The architecture is designed to score cards based solely on their attributes and rules text, facilitating day-zero drafting.
Results
The model achieved top-1 agreement rates of 50.8%, 60.4%, and 56.7% on three held-out expansions, significantly higher than the random chance of 7%. When refitted on all observed expansions, it successfully ranked cards for the unreleased set 'The Hobbit', aligning closely with expert evaluations.
Implications
DraftFM's approach could revolutionize drafting strategies in card games, providing players with real-time assistance based on a model that does not require historical data from the specific set. This could enhance competitive play and inform future research on behavioral prediction in games.
Atrial Fibrillation Detection with Arbitrary Leads via a Codebook-Based Reconstruction-Classification Framework
Time Series
Generative Models
Graph Learning
- DCGCNet introduces a novel codebook-based framework for joint classification and reconstruction of ECG signals.
- The model incorporates a Local-Global Contrastive Module to enhance noise resilience.
- DCGCNet demonstrates exceptional performance with AUC > 0.98 across multiple datasets and conditions.
- The approach effectively addresses the limitations of existing AF detection methods by integrating reconstruction and classification tasks.
Read more
Atrial Fibrillation Detection with Arbitrary Leads via a Codebook-Based Reconstruction-Classification Framework
Summary
This paper addresses the challenge of detecting atrial fibrillation (AF) from electrocardiogram (ECG) signals, particularly in real-world clinical settings where lead configurations can vary significantly. The authors propose the Dual-Codebook Graph Collaborative Network (DCGCNet), an innovative deep learning model that integrates classification and reconstruction tasks into a unified framework. The model employs a codebook-based approach that includes a Local-Global Contrastive Module for enhancing noise resilience and an Adaptive Codebook Vector Quantizer to refine codebook prototypes dynamically. This dual-codebook system allows for effective learning of both local discriminative patterns and global signal fidelity, improving the robustness and generalization of AF detection. The results demonstrate that DCGCNet achieves state-of-the-art performance in both intra-dataset evaluations and cross-dataset generalization, consistently achieving an AUC greater than 0.98 across various testing conditions, including noisy environments. The findings suggest that DCGCNet is a promising solution for reliable AF detection, with significant implications for clinical practice.
Methodology
The authors developed DCGCNet, an end-to-end vector-quantized variational autoencoder that utilizes a shared codebook for joint learning of ECG signal reconstruction and AF classification. Key components include a Local-Global Contrastive Module for noise-invariant representation learning and an Adaptive Codebook Vector Quantizer for dynamic refinement of codebook prototypes.
Results
DCGCNet achieved state-of-the-art performance in standard intra-dataset evaluations and demonstrated strong cross-dataset generalization, consistently attaining an AUC greater than 0.98 across seven diverse settings. The model maintained high diagnostic accuracy even under realistic noisy conditions, such as baseline wander and EMG artifacts.
Implications
The proposed framework has significant implications for the development of automated AF detection systems, potentially facilitating early clinical intervention and improving patient outcomes in real-world settings. Its robustness against noise and adaptability to varying lead configurations make it suitable for large-scale screening and monitoring.
Multi-Source Wasserstein Distributionally Robust Graph Learning
Graph Learning
Optimization
Theory
- Introduces MS-WDRO framework for robust graph learning from multi-source data.
- Utilizes Wasserstein barycenter for effective fusion of heterogeneous data sources.
- Establishes non-asymptotic performance guarantees for the proposed method.
- Achieves superior performance in graph recovery and sample efficiency compared to existing methods.
Read more
Multi-Source Wasserstein Distributionally Robust Graph Learning
Summary
This paper addresses the challenge of network topology inference from graph signals, particularly in scenarios where target-domain samples are limited but heterogeneous source-domain data is plentiful. The authors introduce a novel framework called MS-WDRO (Multi-Source Wasserstein Distributionally Robust Optimization) that utilizes the Wasserstein metric to fuse diverse sources into a reliable nominal distribution while preserving their intrinsic geometries. By constructing a weighted Wasserstein barycenter and creating an ambiguity ball around it, the framework minimizes worst-case risk, leading to a regularized Laplacian estimator that is efficiently solved using an ADMM (Alternating Direction Method of Multipliers) scheme. The authors provide non-asymptotic guarantees, including a finite-sample concentration bound for the empirical barycenter and an out-of-sample excess risk bound that decays at a parametric rate. The proposed method is calibrated through an end-to-end differentiable architecture, allowing for data-adaptive hyperparameter tuning. Experimental results demonstrate that MS-WDRO significantly outperforms seven baseline methods in terms of graph recovery accuracy, sample efficiency, and diagnostic utility, particularly in scenarios with limited samples.
Methodology
The methodology involves constructing a weighted Wasserstein barycenter from heterogeneous source data, creating an ambiguity ball around this barycenter to account for uncertainty, and minimizing the worst-case risk using a regularized Laplacian estimator. The optimization is performed using a convergent ADMM scheme, and the hyperparameters are calibrated through a differentiable architecture.
Results
The MS-WDRO framework consistently outperformed seven baseline methods in experiments, showing significant improvements in graph recovery accuracy and sample efficiency, especially in scenarios with limited data. The results also included theoretical guarantees for the performance of the method.
Implications
The proposed framework has potential applications in various fields such as neuroimaging, sensor networks, and social network analysis, where accurate network topology inference is crucial for downstream tasks like anomaly detection and disease biomarker discovery.
Orthogonal JEPA: Factorized Predictive States for Latent World Models
Computer Vision
Time Series
Robotics
- Introduction of Orthogonal JEPA for improved latent world modeling.
- Utilization of orthogonal predictive factorization to analyze target states into multiple components.
- Implementation of dedicated prediction branches for each component to enhance predictive capacity.
- Incorporation of regularization techniques to maintain diversity and prevent encoder collapse.
Read more
Orthogonal JEPA: Factorized Predictive States for Latent World Models
Summary
The paper introduces Orthogonal JEPA, a novel framework for latent world modeling that enhances the predictive capabilities of joint-embedding predictive architectures (JEPAs). Traditional JEPAs utilize a single target embedding and prediction pathway, which can lead to inefficiencies in complex systems where multiple signals interact. Orthogonal JEPA addresses this by employing orthogonal predictive factorization, which decomposes target states into multiple components, each predicted by a dedicated branch. This approach allows for better allocation of predictive capacity, reducing redundancy and improving the learning of weaker signals. The framework incorporates several regularization techniques, including orthogonality objectives and variance regularization, to maintain diversity in the learned components and prevent encoder collapse. The effectiveness of Orthogonal JEPA is demonstrated through experiments across various domains, including vision, single-cell transcriptomics, health records, control tasks, and molecular dynamics, showcasing its versatility and robustness in representation quality, forecasting, and planning.
Methodology
The methodology involves the development of Orthogonal JEPA, which uses learned basis matrices to decompose target states into multiple components. Each component is predicted through a separate branch, allowing for a more structured approach to latent state prediction. The framework employs various regularization strategies, including orthogonality objectives and factor-activity regularization, to ensure effective learning and prevent redundancy in the predictive pathways.
Results
The experiments conducted across multiple domains indicate that Orthogonal JEPA significantly improves representation quality and predictive performance compared to standard JEPA architectures. The framework demonstrates enhanced stability in long-horizon predictions and effective planning capabilities in complex systems.
Implications
The implications of this research extend to various fields where latent state modeling is crucial, such as robotics, healthcare analytics, and molecular dynamics. By improving the efficiency and effectiveness of predictive modeling, Orthogonal JEPA can facilitate better decision-making and planning in complex environments.
Selection, Recombination, or a Fresh Solve? A Candidate-Free Control for Single-Pass Test-Time Aggregation
NLP
Large Language Models
Efficient ML
- Introduces a candidate-free control to measure the net value of candidate conditioning.
- Finds that conditioning improves accuracy when multiple candidates are correct but reduces it when all candidates are wrong.
- Demonstrates that failures in all-wrong scenarios often match displayed candidate answers.
- Shows that explicit answer fields can guide outputs, while masking does not improve accuracy.
Read more
Selection, Recombination, or a Fresh Solve? A Candidate-Free Control for Single-Pass Test-Time Aggregation
Summary
This paper addresses the challenge of test-time aggregation in machine learning when all candidate solutions are incorrect. The author introduces a candidate-free control mechanism to evaluate the effectiveness of candidate conditioning in generating accurate outputs. The study reveals that when multiple candidates are correct, conditioning improves accuracy significantly. However, when all candidates are wrong, conditioning leads to a decrease in accuracy compared to a fresh solve. The findings suggest that the value of candidate context is not straightforward and varies depending on the correctness of the candidates. The paper employs a structured methodology involving five different aggregation arms to assess the impact of candidate conditioning under controlled conditions. The results indicate a regime-dependent reversal in accuracy based on the number of correct candidates, highlighting the complexities of candidate-based aggregation strategies.
Methodology
The study utilizes five aggregation arms: AGGREGATE (with candidates), NO-CANDIDATE (fresh solve), ANSWER-ONLY (answers without reasoning), VOTING (raw answer strings), and ORACLE (any correct candidate). Each arm operates under a shared maximum generation-token allowance, and the evaluation is stratified by the number of correct candidates.
Results
The results indicate that conditioning on multiple correct candidates improves accuracy (βcand = +0.290), while conditioning on an all-wrong candidate pool decreases accuracy (βcand = -0.123). The study also finds that off-topic candidates can reproduce much of the accuracy deficit observed in all-wrong scenarios.
Implications
These findings have significant implications for the design of aggregation strategies in machine learning, particularly in scenarios where candidate solutions may be incorrect. Understanding the conditions under which candidate conditioning is beneficial or detrimental can inform future research and applications in efficient reasoning and decision-making.
Scale-Aware Pretraining of Time Series Foundation Models via Multi-Patch Token Alignment and Hybrid Masking
Time Series
- SATS introduces a scale-aware token alignment mechanism for time series pretraining.
- The method preserves scale-specific expressiveness while unifying token spaces across patch scales.
- A hybrid masking strategy is designed to capture both fine-grained and long-range temporal dependencies.
- SATS demonstrates superior performance on LSTF benchmarks compared to existing methods.
Read more
Scale-Aware Pretraining of Time Series Foundation Models via Multi-Patch Token Alignment and Hybrid Masking
Summary
This paper presents SATS, a novel approach for pretraining time series foundation models that effectively addresses the challenges posed by heterogeneous datasets with varying sampling frequencies. Traditional methods either use dataset-specific patch sizes leading to fragmented representations or enforce a fixed patch size that overlooks temporal variations. SATS introduces a scale-aware token alignment mechanism that treats patch size as a scale, aligning representation spaces across different scales while maintaining distinct modeling capacities. Additionally, a hybrid masking strategy is employed, combining random and contiguous masking to capture multi-scale temporal structures. Experimental results on LSTF benchmarks show that SATS outperforms competitive baselines, achieving a 9.2% improvement in Mean Squared Error (MSE) and an 8.3% gain in GIFT-Eval MASE, while also enhancing model efficiency by 65.6%. This highlights SATS's effectiveness and scalability for time series pretraining, establishing it as a strong candidate for zero-shot and in-distribution forecasting tasks.
Methodology
The methodology involves a scale-aware token alignment mechanism that aligns representation spaces across different patch sizes using scale-specific feedforward networks (FFNs). It minimizes the distance between mean token embeddings across scales while maximizing the distance between their maximal embeddings. Additionally, a hybrid masking strategy is utilized, combining random and contiguous masking to enhance the model's ability to capture multi-scale temporal structures during masked reconstruction.
Results
SATS achieved a 9.2% improvement in Mean Squared Error (MSE) and an 8.3% gain in GIFT-Eval MASE on LSTF benchmarks compared to competitive baselines. Furthermore, it demonstrated a 65.6% increase in model efficiency over advanced methods, establishing its state-of-the-art performance in time series forecasting.
Implications
The findings suggest that SATS can be effectively utilized for various time series forecasting tasks, enhancing the robustness and generalization of models across diverse datasets. Its scalability and efficiency make it a promising approach for real-world applications in fields such as finance, healthcare, and IoT.
RecPFN: Prior-Fitted Networks for In-Context-Based Recommendations
Efficient ML
Theory
NLP
- RecPFN is the first embedding-based PFN-style ICL approach for sequential recommendation.
- The model is pretrained on synthetic data, enabling strong performance with less computational overhead.
- RecPFN achieves state-of-the-art zero-shot performance and is robust to domain shifts.
- The architecture allows for next-item predictions in a single forward pass without weight updates.
Read more
RecPFN: Prior-Fitted Networks for In-Context-Based Recommendations
Summary
The paper introduces RecPFN, a novel approach to sequential recommendation that leverages prior-fitted networks (PFNs) to enable in-context learning (ICL). RecPFN is pretrained on synthetic clickstream data generated from a broad structural causal prior, allowing it to perform Bayesian-style inference with minimal data. The architecture employs a lightweight decoder-only transformer that predicts the next item based on a small set of domain sequences in a single forward pass, without requiring weight updates. The authors demonstrate that RecPFN achieves state-of-the-art zero-shot performance across eight public benchmarks, while also being competitive with supervised methods in low-compute and low-data scenarios. The approach is noted for its deployment efficiency and robustness to domain shifts, outperforming traditional zero-shot baselines that depend on large real-interaction datasets. The findings suggest that RecPFN offers a practical solution for developing generalizable and data-efficient recommendation systems, paving the way for future research into richer priors, longer-context ICL, and multimodal applications.
Methodology
RecPFN employs a synthetic framework for generating clickstream embedding sequences based on a structural causal model. The model is pretrained on these synthetic sequences to approximate Bayesian inference, allowing it to adapt to new domains using in-context examples during inference. The lightweight transformer architecture processes a small number of domain sequences to predict the next item in a single pass.
Results
RecPFN outperforms existing zero-shot recommendation methods and remains competitive with supervised approaches, particularly in scenarios with limited data and computational resources. The model's performance is validated across eight public benchmarks, demonstrating its effectiveness in real-world applications.
Implications
The development of RecPFN suggests a shift towards more efficient and adaptable recommendation systems that can generalize across different domains without extensive retraining. This has significant implications for industries relying on personalized recommendations, such as e-commerce and content delivery, where rapid adaptation to user preferences is crucial.
Graphical Design of Interpretable Architectures
Interpretability
- Introduces a graphical notation for interpretable AI architectures based on Penrose tensor notation.
- Provides a global view of architectures while mapping directly to PyTorch einsum code.
- Describes interpretable architectures by construction, enhancing understanding and reproducibility.
- Demonstrates the notation with a case study on the Steerling-8B language model.
Read more
Graphical Design of Interpretable Architectures
Summary
This paper addresses the challenge of designing interpretable AI architectures by introducing a novel graphical notation adapted from Penrose tensor notation. Traditional representations, such as symbolic equations and probabilistic graphical models, often fail to provide a comprehensive view of architectures or do not accurately depict tensor manipulations, which are crucial for understanding and reproducibility. The proposed notation allows for a clear visualization of tensor operations while directly mapping to PyTorch einsum code, facilitating both design and implementation. The author demonstrates the utility of this notation by describing various interpretable architectures, including concept bottlenecks and neural additive models, and provides a detailed diagram of the Steerling-8B language model, highlighting its residual structure and translating it into concise PyTorch code. This approach not only enhances interpretability but also aids in the reproducibility of AI models.
Methodology
The paper employs a graphical notation derived from Penrose tensor notation to represent AI architectures. This notation is used to analyze and diagram tensor operations, which are then translated into PyTorch einsum code. The author illustrates the methodology through various examples of interpretable architectures and a detailed case study of the Steerling-8B model.
Results
The graphical notation successfully provides a clear and interpretable representation of complex tensor manipulations, yielding insights into the architecture of Steerling-8B. The notation facilitates a direct translation into PyTorch code, demonstrating its practical applicability in designing interpretable AI models.
Implications
The proposed graphical notation has the potential to significantly improve the design and understanding of interpretable AI architectures, making it easier for researchers and practitioners to develop reproducible models. This could lead to advancements in the field of interpretable machine learning, enhancing trust and transparency in AI systems.
Towards Reversible Forgetting: Managing Obsolete Knowledge in Continual Enterprise AI Agents
Theory
- Reversible forgetting allows for the suppression of obsolete knowledge without permanent erasure.
- The framework introduces three operational memory states: active, dormant, and retired.
- The Hysteretic Reversible Memory Controller (HRMC) manages knowledge transitions with evidence accumulation and policy gates.
- The approach reframes continual learning as a management of knowledge lifecycle rather than mere retention.
Read more
Towards Reversible Forgetting: Managing Obsolete Knowledge in Continual Enterprise AI Agents
Summary
This paper addresses the challenge of managing knowledge in continual learning systems, particularly for enterprise AI agents operating in dynamic environments. Traditional approaches view forgetting as a failure, focusing on retaining knowledge. However, the authors argue that in non-stationary settings, retaining obsolete knowledge can lead to negative transfer and operational risks. They propose a novel framework called reversible forgetting, which categorizes memory into three states: active, dormant, and retired. This framework allows for the suppression of obsolete knowledge while retaining the ability to reactivate it when relevant again. The authors introduce the Hysteretic Reversible Memory Controller (HRMC), which employs evidence accumulation, asymmetric thresholds, shadow mode testing for reactivation, and policy-gated retirement to manage knowledge effectively. The paper emphasizes that knowledge management should be viewed as a lifecycle rather than a binary retention/deletion process, ensuring that enterprise agents utilize the most relevant information at any given time.
Methodology
The authors propose a conceptual framework for reversible forgetting, operationalized through the Hysteretic Reversible Memory Controller (HRMC). This controller assesses memory relevance based on contextual similarity, utility, reliability, staleness, and potential harm. It uses hysteresis to prevent rapid state changes and requires policy approval for memory retirement, ensuring a structured approach to knowledge management.
Results
The proposed framework effectively reduces the influence of obsolete knowledge while allowing for its potential reactivation. By implementing the HRMC, enterprise agents can adapt to changing environments without losing valuable historical knowledge, thus minimizing operational risks associated with negative transfer.
Implications
The findings suggest that enterprise AI systems can benefit from a more nuanced approach to knowledge management, allowing them to adapt to changing conditions while maintaining a record of past knowledge. This has implications for various sectors, including finance, where knowledge relevance can fluctuate with market conditions.
Infrared Universality of Collective Dynamics across Transformer and State-Space Architectures
Theory
Large Language Models
NLP
- Investigates collective dynamics in Transformers and Mamba, a state-space model.
- Finds that both architectures exhibit similar infrared organization despite different microscopic mechanisms.
- Demonstrates that Mamba's long-sequence exponent stabilizes near Ξ²M ββ0.17, indicating long-memory dynamics.
- Extends the understanding of collective dynamics in neural networks beyond Transformers.
Read more
Infrared Universality of Collective Dynamics across Transformer and State-Space Architectures
Summary
This paper investigates whether different neural architectures exhibit common collective dynamics, focusing on Transformers and a state-space model called Mamba. Previous studies on Transformers revealed a weakly infrared-enhanced time-scale density of states (TDOS) associated with long-memory dynamics. The author examines Mamba, which employs a distinct microscopic mechanism for state-space dynamics, allowing for relaxation dynamics to be analyzed at three levels: the intrinsic spectrum of the learned state-space generator, input-conditioned selective rescaling, and the collective TDOS from its Jacobian. The findings indicate that while the microscopic dynamics differ, both architectures develop a similar slow-mode continuum in their collective dynamics. The cumulative spectral analysis shows that the long-sequence Mamba exponent stabilizes near Ξ²M ββ0.17, corresponding to long-memory dynamics close to the marginal 1/t regime. In comparison, Transformers exhibit a related infrared organization with exponents around Ξ²Tr βΌβ0.1. These results suggest that distinct sequence architectures can develop closely related near-marginal slow-mode dynamics, extending the understanding of infrared collective organization beyond Transformers and providing insights into the dynamical structure described by Cognitive Field Theory.
Methodology
The study employs spectral analysis to characterize the relaxation dynamics of the Mamba model at multiple levels, including the intrinsic state-space generator and the collective TDOS measured from its Jacobian. The analysis compares these findings with the dynamics of Transformer models to assess similarities and differences in their collective behavior.
Results
The analysis reveals that Mamba develops a reproducible slow-mode continuum with a well-resolved infrared sector as sequence length increases. The cumulative spectral analysis yields a weakly negative exponent for Mamba, stabilizing near Ξ²M ββ0.17, while Transformers show a related organization with Ξ²Tr βΌβ0.1. This indicates that despite different underlying mechanisms, both architectures exhibit similar collective dynamics.
Implications
The findings suggest that common dynamical principles may emerge across different neural architectures, which could influence the design and understanding of future models in sequence processing and language modeling. This work also supports the theoretical framework of Cognitive Field Theory in explaining collective dynamics in neural networks.
Explainable Transformer Models for Clinical Prediction Tasks on Structured Electronic Health Records
NLP
Interpretability
- Introduction of BERT-LER, a transformer model that combines laboratory data representation with interpretability.
- Utilization of percentile-based binning for encoding laboratory results, preserving graded information.
- Demonstrated competitive performance on EHRShot and real-world asthma progression tasks.
- Token-level attributions using Integrated Gradients align with clinically known risk factors.
Read more
Explainable Transformer Models for Clinical Prediction Tasks on Structured Electronic Health Records
Summary
This paper presents BERT-LER, a novel BERT-style model designed for clinical prediction tasks using structured electronic health records (EHRs). The model addresses the challenge of incorporating continuous laboratory measurements into discrete token-based architectures while maintaining interpretability. BERT-LER encodes laboratory test results as discrete tokens through percentile-based binning, allowing it to retain graded information. The model is pretrained and fine-tuned on a large dataset of 75 million patients and evaluated on both the EHRShot benchmark suite and a real-world asthma severity progression study. The authors demonstrate that BERT-LER achieves competitive predictive performance, often exceeding existing benchmark models, particularly on laboratory-related tasks. Additionally, the use of Integrated Gradients for token-level attributions provides clinically relevant insights into model predictions, linking them back to input medical events. This work highlights the potential for integrating strong predictive performance with interpretable outputs in clinical settings, offering a unified framework for EHR modeling that can be applied across various therapeutic areas.
Methodology
The authors developed BERT-LER, a BERT-style model that encodes laboratory test results as discrete tokens using percentile-based binning. The model was pretrained on a large dataset and fine-tuned for specific clinical prediction tasks. Integrated Gradients were employed to generate token-level attributions, linking model predictions to input medical events.
Results
BERT-LER achieved predictive performance that is competitive with existing benchmark models, particularly excelling in tasks related to laboratory data. The model's attributions were clinically relevant, reflecting known risk factors in both benchmark and application-driven settings.
Implications
The findings suggest that BERT-LER can facilitate the integration of machine learning models into clinical workflows by providing interpretable predictions that are clinically meaningful. This approach may enhance decision-making in healthcare by improving the understanding of model outputs.
Multi-Agent Off-Policy Deep Reinforcement Learning for Smart Campus Coverage
Reinforcement Learning
Optimization
- The optimization of BS placement in non-convex campus environments is NP-hard.
- Multi-Agent DDPG outperforms single-agent approaches in terms of coverage and fairness.
- The proposed framework achieves a Jainβs fairness index of 0.94 and guarantees full coverage.
- The study introduces a boundary projection mechanism to enhance optimization.
Read more
Multi-Agent Off-Policy Deep Reinforcement Learning for Smart Campus Coverage
Summary
This paper explores the deployment of millimeter-wave (mmWave) base stations (BSs) in a complex campus environment, addressing the NP-hard optimization problem of BS placement for maximizing user fairness and coverage. The authors propose a novel approach that formulates the BS placement as a Markov Decision Process (MDP) and evaluates four deep reinforcement learning (DRL) schemes: a discrete single-agent Deep Q-Network (DQN), a spatially partitioned Multi-Agent DQN, a continuous single-agent Deep Deterministic Policy Gradient (DDPG), and a geographically partitioned multi-agent DDPG framework. The study highlights the limitations of traditional optimization methods in non-convex environments and demonstrates that the multi-agent DDPG approach significantly outperforms single-agent methods, achieving full coverage and a Jainβs fairness index of 0.94 in dense user scenarios. The results indicate that the proposed multi-agent framework not only ensures optimal BS placement but also enhances computational efficiency in complex topologies.
Methodology
The authors formulated the BS placement problem as an MDP, utilizing both discrete and continuous DRL methods. They benchmarked four different DRL schemes, including single-agent and multi-agent architectures, to optimize the placement of BSs in a realistic campus topology. The reward function was designed to reflect max-min fairness, focusing on equitable user throughput.
Results
The multi-agent DDPG framework achieved superior performance, ensuring 100% coverage and maintaining minimum SNRs above 19 dB. The Jainβs fairness index reached 0.94, indicating a high level of user fairness in the network. The numerical evaluations demonstrated significant computational efficiency in dense user scenarios.
Implications
The findings suggest that DRL can effectively address complex optimization problems in wireless network deployment, particularly in urban environments. This research could inform future designs of next-generation wireless networks, enhancing connectivity and user experience in smart campuses and similar settings.
Online Test-Time Adaptation for Generalizable Dynamic Graph Anomaly Detection
Graph Learning
- Introduces OTTA-DGAD for online test-time adaptation in dynamic graph anomaly detection.
- Utilizes dynamic prototypes to capture evolving normal and anomalous patterns.
- Implements a memory buffer that retains both general and domain-specific patterns.
- Demonstrates superior performance on diverse datasets compared to existing methods.
Read more
Online Test-Time Adaptation for Generalizable Dynamic Graph Anomaly Detection
Summary
This paper addresses the challenge of dynamic graph anomaly detection (DGAD) in unseen target domains without the need for costly retraining. The authors propose a novel method called OTTA-DGAD, which incorporates online test-time adaptation to effectively handle sequentially arriving unlabeled data chunks. OTTA-DGAD extracts dynamic prototypes from temporal ego-graphs and stores them in a memory buffer that retains general patterns from source domains while integrating new domain-specific patterns. An anomaly scorer compares incoming edge representations against these prototypes to detect both general and domain-specific anomalies. The method also updates the memory buffer using reliable pseudo-labels and enriches each target chunk with relevant representations from previous chunks to mitigate information loss. Extensive experiments demonstrate that OTTA-DGAD achieves state-of-the-art performance across ten real-world datasets, showcasing its effectiveness in adapting to evolving anomaly patterns in dynamic graphs.
Methodology
The methodology involves extracting dynamic prototypes from temporal ego-graphs and storing them in a memory buffer. This buffer selectively retains general patterns from source domains while incorporating new patterns from the target domain. An anomaly scorer compares incoming edge representations against these prototypes. The model updates the memory buffer using reliable pseudo-labels and enriches target chunks with relevant representations from previous chunks to address information loss.
Results
OTTA-DGAD outperforms existing methods in dynamic graph anomaly detection across ten real-world datasets, demonstrating its ability to adapt to continuous domain shifts and effectively capture evolving anomaly patterns.
Implications
The proposed method has significant implications for real-world applications such as fraud detection, transaction monitoring, and social network security, where timely and accurate anomaly detection in dynamic environments is crucial.
Longitudinal Bayesian Learning of Continuous Disease Position across the Alzheimer's Disease Continuum
Time Series
Theory
Generative Models
- DCP provides a continuous representation of disease severity, moving beyond discrete diagnostic categories.
- The Disease Continuum Score (DCS) captures clinically relevant variations and predicts future disease progression.
- DCP integrates longitudinal DTI data with weak clinical supervision for improved disease assessment.
- Extensive validation shows DCS's strong clinical relevance and ability to characterize disease evolution.
Read more
Longitudinal Bayesian Learning of Continuous Disease Position across the Alzheimer's Disease Continuum
Summary
This paper introduces a novel framework called Disease Continuum Positioning (DCP) that leverages longitudinal Bayesian learning to assess continuous disease severity in Alzheimer's disease (AD) using diffusion tensor imaging (DTI). Unlike traditional methods that categorize patients into discrete diagnostic stages, DCP provides a continuous Disease Continuum Score (DCS) that quantifies an individual's position along the AD continuum. The framework integrates longitudinal observations with weak clinical supervision, allowing for a fine-grained characterization of disease severity and capturing clinically meaningful variations. The DCS not only reflects current disease status but also predicts future transitions between cognitive states, such as from cognitively normal (CN) to mild cognitive impairment (MCI) and from MCI to AD. Extensive experiments conducted on the Alzheimer's Disease Neuroimaging Initiative (ADNI) cohort demonstrate that DCP outperforms existing disease progression methods, showcasing its ability to accurately characterize disease severity and preserve longitudinal disease evolution.
Methodology
The authors developed the Disease Continuum Positioning (DCP) framework, which employs a longitudinal Bayesian learning approach to model disease severity as a low-dimensional probabilistic latent variable. This method integrates longitudinal DTI scans and weak clinical supervision to derive the Disease Continuum Score (DCS), which quantifies an individual's position along the Alzheimer's disease continuum.
Results
The DCP framework consistently outperformed existing methods in assessing disease progression in the ADNI cohort. The DCS accurately characterized disease severity, demonstrated strong clinical relevance, and effectively preserved longitudinal disease evolution. Furthermore, it provided reliable predictions for future transitions between cognitive states.
Implications
The DCP framework and the associated DCS have significant implications for early diagnosis, prognosis, and management of Alzheimer's disease. By providing a continuous assessment of disease progression, it can enhance clinical decision-making and improve patient outcomes.
Beyond Trial Averaging: Anchoring Neural and Visual Representations for Few-Repetition Brain-to-Image Retrieval
Multimodal
- Identification of a non-transitive alignment pattern in few-repetition brain-to-image retrieval.
- Introduction of the NEAR framework that enhances retrieval accuracy by anchoring neural and visual representations.
- Demonstrated improvements in retrieval performance across multiple datasets and modalities.
- Reduction of reliance on repeated stimulus presentations, addressing latency and user burden.
Read more
Beyond Trial Averaging: Anchoring Neural and Visual Representations for Few-Repetition Brain-to-Image Retrieval
Summary
This paper addresses the challenge of brain-to-image retrieval, which typically relies on averaging multiple neural trials per image to enhance retrieval accuracy. The authors identify a significant drop in retrieval performance when only a few repetitions are available, attributing this to both query noise and gallery placement issues. They propose a novel framework called Neural-Anchor-based Retrieval (NEAR), which utilizes a stable neural representation as an anchor to improve retrieval accuracy. NEAR employs two strategies: Query Anchoring, which denoises the few-repetition query, and Gallery Anchoring, which predicts a pseudo anchor for each candidate image. The framework is tested across four datasets, including EEG, MEG, and fMRI, demonstrating consistent improvements in retrieval performance, particularly in few-repetition scenarios. This work highlights the importance of anchoring neural and visual representations to reduce reliance on repeated measurements, making brain-to-image retrieval more feasible for real-world applications.
Methodology
The authors conducted a controlled target-swap experiment to analyze the alignment patterns between few-repetition queries, stable neural representations, and images. They developed the NEAR framework, which consists of two main components: Query Anchoring, which denoises the query towards a stable neural anchor, and Gallery Anchoring, which predicts pseudo anchors for candidate images to enhance retrieval scoring.
Results
The NEAR framework significantly improved retrieval accuracy on the THINGS-EEG2 dataset, increasing 200-way Top-1 accuracy by 5.7 percentage points with one repetition and 9.3 percentage points with four repetitions. The improvements were consistent across all ten participants tested, indicating the robustness of the method across different neural signal types.
Implications
The findings suggest that anchoring neural and visual representations can facilitate more effective brain-to-image retrieval with fewer repetitions, potentially enabling practical applications in neuro-rehabilitation and dream decoding without the need for extensive stimulus presentations.
Harness Continual Learning: Continual Adaptation Beyond Model Parameters
NLP
Multimodal
Theory
- Introduction of Harness Continual Learning (HCL) paradigm focusing on harness adaptation rather than model parameter changes.
- Identification of harness-level forgetting as a challenge in continual learning.
- Development of a framework with components for managing harness state and learning.
- Demonstration of capability accumulation and failure recovery in various experimental settings.
Read more
Harness Continual Learning: Continual Adaptation Beyond Model Parameters
Summary
This paper introduces Harness Continual Learning (HCL), a novel paradigm that shifts the focus of continual learning from model parameters to an external harness that adapts around a frozen foundation model. Traditional continual learning approaches primarily modify model parameters to retain learned behaviors, leading to issues like forgetting previous tasks. HCL addresses this by evolving a harness composed of prompts, memories, tools, skills, and routing rules, which can adapt and accumulate experience while the underlying model remains unchanged. The authors define 'harness-level forgetting' as the loss of previously reliable behaviors due to updates in the harness. They propose a framework consisting of four components: Task Interface, Experience Memory, Capability Map, and Adaptive Router, which collectively manage how an agent processes information and executes tasks. The evolution of the harness is governed by a Continual Optimizer that generates candidate updates based on feedback and a Continual Evaluator that assesses these candidates for performance improvement and retention of historical behaviors. Experimental results demonstrate that HCL can effectively accumulate capabilities and recover from failures across various tasks, showing significant performance gains over baseline methods while also revealing measurable harness-level forgetting. The findings suggest that the stability-plasticity trade-off can be explicitly managed, providing a new avenue for research in continual learning.
Methodology
The authors developed a framework for HCL that includes four main components: Task Interface, Experience Memory, Capability Map, and Adaptive Router. They implemented a two-part evolution process involving a Continual Optimizer to propose updates based on feedback and a Continual Evaluator to validate these updates against performance and retention criteria. The methodology was tested through experiments in textual reasoning, multimodal perception, and open-world interaction.
Results
The experimental results showed that HCL outperformed baseline methods by over 10% in various tasks, demonstrating effective capability accumulation and recovery from failures. The study also revealed measurable harness-level forgetting and confirmed that the historical-retention budget could influence the balance between adaptation and retention.
Implications
The findings of this research could lead to more robust continual learning systems that maintain previously learned behaviors while adapting to new tasks. This approach has potential applications in AI systems that require ongoing learning and adaptation without losing prior knowledge, such as in robotics, personal assistants, and interactive AI agents.
Score the Algebra, Not the Span: Dimension Reduction for Transfer Operator Models of Dynamical Systems
Theory
- Traditional spectral methods can lead to linear masking, omitting entire components from models of dynamical systems.
- The proposed method focuses on scoring the Ο-algebra generated by coordinates, allowing for more efficient representation.
- A finite budget of coordinates can capture the entire spectrum of the transfer operator, avoiding the pitfalls of rank-based methods.
- The approach is validated through benchmark systems, demonstrating superior performance in recovering masked components.
Read more
Score the Algebra, Not the Span: Dimension Reduction for Transfer Operator Models of Dynamical Systems
Summary
This paper addresses the challenges of dimension reduction in dynamical systems, particularly those composed of weakly interacting components. Traditional methods, which rely on spectral approaches to model the transfer (Koopman) operator, often lead to linear masking, where entire components are omitted from the model. The authors propose a novel approach that focuses on scoring the Ο-algebra generated by the coordinates rather than the span of the modes. This method allows for a more efficient representation of the dynamics, capturing the entire spectrum of the operator with significantly fewer coordinates. The proposed criterion, based on Ο2-divergence between the present and future states, ensures that twice the intrinsic dimension of the dynamics suffices for a complete embedding. The authors demonstrate the effectiveness of their approach through various benchmark systems, showing that while rank-based methods may fail to capture masked components, their algebra-based representation can successfully predict these components from limited data.
Methodology
The authors propose a new dimension reduction approach that scores the Ο-algebra generated by the coordinates instead of the span of modes. They formalize this through a criterion based on Ο2-divergence, allowing for the use of existing estimators to compute the optimal embedding. The method is tested on various benchmark systems to evaluate its effectiveness in capturing the dynamics of multicomponent systems.
Results
The results show that the proposed algebra-based representation can recover all components of the system, even when rank-based methods fail to capture them at lower ranks. Specifically, the authors demonstrate that a representation using ten algebra coordinates can successfully predict masked components, while traditional methods miss these components entirely.
Implications
This work has significant implications for modeling complex dynamical systems in physical and biological contexts, where weakly interacting components are common. The ability to capture the full dynamics with fewer coordinates can lead to more efficient modeling and better predictive capabilities, particularly in scenarios with limited data.
FAR-DPO: Feasibility-Aware and Robust Direct Preference Optimization for Cyclic Peptide Design
Generative Models
Optimization
- FAR-DPO addresses the challenges of cyclic peptide design by focusing on feasibility and robustness.
- The framework integrates a novel preference construction strategy that accounts for structural and biophysical constraints.
- FAR-DPO shows significant improvements in the success rate of feasible cyclic peptide designs across multiple generative models.
- The methodology includes difficulty-aware optimization to ensure consistent performance across heterogeneous targets.
Read more
FAR-DPO: Feasibility-Aware and Robust Direct Preference Optimization for Cyclic Peptide Design
Summary
Cyclic peptides are gaining traction in drug discovery due to their high binding affinity and structural stability. However, designing cyclic peptides poses significant challenges due to the constraints imposed by cyclization on the feasible design space. Existing generative models often struggle with low yields of feasible designs and limited control over multi-objective trade-offs due to reliance on zero-shot generation or post hoc filtering. To address these issues, the authors propose FAR-DPO (Feasibility-Aware and Robust Direct Preference Optimization), an architecture-agnostic framework that enhances the design of cyclic peptides by integrating feasibility-aware preference construction with difficulty-aware group-robust optimization. FAR-DPO constructs preference pairs through feasibility-gated multi-objective dominance and adaptively reweights difficulty groups based on current preference losses. Evaluations on the CPSea LNR benchmark demonstrate that FAR-DPO significantly improves the success rate of feasible designs, increasing it from 46.89% to 57.79% on PepGLAD and from 47.96% to 49.57% on PepFlow, while also yielding better binding scores. These results indicate FAR-DPO's effectiveness in enhancing both feasibility and robustness across different targets.
Methodology
FAR-DPO employs a two-pronged approach: it constructs feasibility-aware preference pairs from generated cyclic peptides and implements a difficulty-aware optimization strategy that adjusts weights based on the performance of different difficulty groups. This allows the model to focus on improving areas where it currently underperforms, thus enhancing the overall quality of generated designs.
Results
FAR-DPO achieved an increase in the overall success rate of feasible cyclic peptide designs, with improvements from 46.89% to 57.79% on PepGLAD and from 47.96% to 49.57% on PepFlow. The framework also produced better binding scores, particularly for the most challenging targets, demonstrating its effectiveness in generating structurally and biophysically feasible cyclic peptides.
Implications
The development of FAR-DPO has significant implications for drug discovery, particularly in the design of cyclic peptides that can effectively modulate protein-protein interactions. By improving the feasibility and robustness of cyclic peptide designs, this framework could accelerate the identification of novel therapeutics and enhance the efficiency of the drug development process.