AI-generated summaries
Today's ML research,
without the noise.
Daily summaries of the latest machine learning papers from arXiv, processed every 8 hours.
57
Papers today
8h
Update frequency
7
Days of history
Sample Count Is Not Enough: Candidate-Generation Strategy Shapes the Energy and Performance of LLM Test-Time Scaling
Large Language Models
Efficient ML
NLP
- Candidate count alone does not adequately describe the system costs of multi-candidate inference.
- Increasing the number of candidates improves accuracy but also increases energy consumption and latency.
- Batched generation calls are significantly more efficient than serial execution of candidates.
- The paper provides recommendations for reporting practices in multi-candidate inference studies.
Read more
Sample Count Is Not Enough: Candidate-Generation Strategy Shapes the Energy and Performance of LLM Test-Time Scaling
Summary
This paper investigates the impact of candidate-generation strategies on the performance and energy efficiency of large language models (LLMs) during test-time scaling. The authors highlight that while the number of generated candidates (N) is often used to measure inference budgets, it does not account for how these candidates are executed. The study evaluates the effect of increasing N on reasoning accuracy using two models, Phi-3-mini and Qwen2.5-1.5B, across 500 GSM8K prompts, finding significant accuracy improvements with higher N. However, the authors emphasize that simply increasing N leads to increased system costs, which are not captured by accuracy metrics alone. They compare four different generation schedules (1×8, 2×4, 4×2, and 8×1) while keeping N fixed at 8, measuring latency, throughput, GPU-hours, and energy consumption. Results indicate that serial execution of candidates incurs significantly higher energy costs and latency compared to batched generation. The findings suggest that fewer generation calls with larger batch sizes are more efficient when candidates are independent, and they call for improved reporting practices in the field to include generation schedules and system metrics.
Methodology
The authors conducted experiments with two LLMs, Phi-3-mini and Qwen2.5-1.5B, using a fixed candidate count of N=8. They compared different generation schedules (1×8, 2×4, 4×2, and 8×1) and measured various performance metrics including latency, throughput, GPU-hours, and energy consumption on A100 GPUs.
Results
The study found that increasing the candidate count from 1 to 8 improved accuracy by 8.4 percentage points for Phi-3-mini and 18.4 points for Qwen2.5-1.5B. However, using eight serial calls resulted in 4.64–4.86 times more energy consumption and 5.77–6.12 times higher latency compared to a single batched call with eight candidates. This pattern was consistent across different GPU nodes and workloads.
Implications
The findings suggest that optimizing candidate generation strategies can lead to significant improvements in both performance and energy efficiency of LLMs. This has implications for the deployment of LLMs in high-performance computing environments, where resource efficiency is critical. The recommendations for reporting practices can enhance reproducibility and comparability of results in future research.
Learning-Induced Dynamical Transition in Recurrent Neural Networks
Theory
- Learning in RNNs can induce a transition from chaotic to stable dynamics.
- A non-equilibrium dynamical mean-field theory (DMFT) is developed to analyze this transition.
- The study identifies critical parameters that separate chaotic and stable regimes during learning.
- The theory predicts the evolution of network outputs and shows quantitative agreement with simulations.
Read more
Learning-Induced Dynamical Transition in Recurrent Neural Networks
Summary
This paper explores how learning in recurrent neural networks (RNNs) can transform chaotic dynamics into stable, task-dependent behavior. The author develops a non-equilibrium dynamical mean-field theory (DMFT) to describe this transition during the learning process. The study reveals that a slow feedback-driven learning mechanism generates an evolving effective feedback strength, which facilitates a transition from chaotic to stable dynamics, characterized by a bifurcation in the DMFT solution. By deriving the two-time correlation function, the author identifies a critical feedback strength and a learning rate-dependent critical time that delineate the chaotic and stable regimes. The theory predicts the time evolution of network outputs during training and aligns well with numerical simulations, demonstrating that learning reorganizes the dynamical regime of RNNs. This work emphasizes the dynamic nature of learning, suggesting that stability emerges through gradual adaptation rather than being a fixed property of the learned structure.
Methodology
The author employs a non-equilibrium dynamical mean-field theory (DMFT) to analyze the learning dynamics of recurrent neural networks. The model incorporates feedback mechanisms and derives the two-time correlation function to track the evolution of network dynamics throughout the learning process.
Results
The study finds that the learning process leads to a bifurcation in the DMFT solution, indicating a transition from chaotic to stable dynamics. The predictions regarding the time evolution of network outputs during training are consistent with numerical simulations, validating the theoretical framework.
Implications
This research provides insights into the mechanisms by which recurrent neural networks can achieve stable computational behavior through learning. It has potential applications in understanding biological neural circuits and improving the design of artificial intelligence systems that rely on RNNs.
QUALS: Corpus Equilibrium for Universal Forecasting via Pattern Quantization and Learnability Synchronization
Time Series
Efficient ML
Optimization
- QUALS addresses data diversity issues in time series forecasting.
- The framework includes pattern quantization and learnability synchronization mechanisms.
- Models trained on QUALS achieve superior zero-shot performance with less data.
- The approach effectively manages skewed pattern distributions and learnability discrepancies.
Read more
QUALS: Corpus Equilibrium for Universal Forecasting via Pattern Quantization and Learnability Synchronization
Summary
The paper introduces QUALS, a novel framework aimed at improving the efficiency of time series forecasting by addressing the challenges posed by data diversity and distribution. Current approaches to universal forecasting often focus on model architecture while neglecting the complexities of the data itself, leading to suboptimal performance. QUALS enhances data efficiency by employing two main mechanisms: a pattern quantization framework that decodes heterogeneous patterns from mixed corpora using vector quantization and uniform binning, and a learnability synchronization framework that calibrates sampling weights for these patterns. This dual approach helps bridge the optimization gap between simple and complex motifs, allowing models to achieve superior zero-shot forecasting performance with significantly reduced training data. Extensive benchmarks demonstrate that pre-training on QUALS consistently yields better outcomes compared to traditional methods, even with a fraction of the original training data.
Methodology
QUALS employs a two-pronged approach: first, it utilizes a pattern quantization framework to systematically decode and categorize diverse temporal patterns from large datasets. Second, it implements a learnability synchronization framework that adjusts sampling weights for different patterns, ensuring that both simple and complex motifs are adequately represented during training.
Results
The results indicate that models pre-trained on the QUALS framework consistently outperform those trained on traditional large-scale datasets, achieving better zero-shot forecasting capabilities even when trained on significantly reduced data volumes. This demonstrates the effectiveness of QUALS in enhancing training efficiency and model performance.
Implications
The findings suggest that QUALS could be applied across various domains that rely on time series data, such as transportation, finance, and public safety, potentially leading to more robust forecasting models that require less data and computational resources.
How Far Can Sub-3B Open Language Models Go in Zero-Shot Essay Scoring on an 8 GB Consumer GPU?
NLP
Large Language Models
- Rubric-decomposed prompting outperforms holistic prompting in most cases.
- The mapping of trait scores to prompt scores is fragile and requires careful calibration.
- Essay length impacts scoring accuracy, with smaller models showing variability in performance.
- The best local model configuration achieved a QWK of 0.388, below human scoring standards.
Read more
How Far Can Sub-3B Open Language Models Go in Zero-Shot Essay Scoring on an 8 GB Consumer GPU?
Summary
This paper investigates the capabilities of sub-3B open language models in performing zero-shot essay scoring on a consumer-grade GPU, specifically focusing on the constraints of local inference without API calls. The authors conduct a controlled study using four instruction-tuned models from two families, evaluating their performance on eight prompts from the ASAP-AES dataset. The study compares two prompting strategies: holistic prompting and rubric-decomposed prompting, assessing their effectiveness in generating scores. Key findings indicate that rubric-decomposed prompting generally outperforms holistic prompting, particularly under batch min-max aggregation. The research also reveals that the mapping of trait scores to prompt scores is sensitive to grader calibration, and essay length significantly affects scoring accuracy. The best-performing local model configuration achieved a macro quadratic weighted kappa (QWK) of 0.388, which, while notable, remains below human inter-rater reliability and a length-only baseline. The authors conclude that sub-3B models are suitable for formative feedback in educational contexts but should not replace human scoring.
Methodology
The study employed a systematic evaluation of four instruction-tuned models (Qwen2.5 and SmolLM2) across eight ASAP-AES prompts. It utilized two prompting strategies (holistic and rubric-decomposed) and analyzed the results using bootstrap confidence intervals, Holm-corrected paired tests, and length-bias assessments, all conducted on a single 8 GB consumer GPU.
Results
The findings showed that rubric-decomposed prompting generally yielded better results than holistic prompting, particularly under batch min-max aggregation. The best local model configuration achieved a macro QWK of 0.388, indicating significant room for improvement compared to human inter-rater reliability (0.769) and a length-only baseline (0.523). Additionally, the study highlighted a systematic decline in signed error with increasing essay length.
Implications
The results suggest that sub-3B open language models can provide useful formative feedback in educational settings, particularly where privacy concerns prevent the use of third-party APIs. However, their limitations indicate that they should complement rather than replace human evaluators in essay scoring.
Radio-Frequency Convolutional Neural Networks
Efficient ML
- RF-CNN repurposes existing wireless communication hardware for efficient CNN inference.
- The approach achieves low energy consumption, down to 0.72 femtojoules per multiply-accumulate operation.
- RF-CNN can handle deep CNNs with millions of parameters while maintaining performance close to full precision.
- The method utilizes the native operations of frequency mixers, avoiding the need for additional dedicated hardware.
Read more
Radio-Frequency Convolutional Neural Networks
Summary
This paper introduces Radio-Frequency Convolutional Neural Networks (RF-CNNs), a novel approach that leverages existing wireless communication hardware, specifically frequency mixers, to perform convolutional neural network (CNN) inference directly on edge devices. Traditional AI models require significant computational resources, which are often impractical for edge devices like smartphones and drones due to constraints in size, weight, power, and cost. RF-CNNs utilize the native operation of frequency mixers, which multiply signals in the time domain, effectively performing convolution in the frequency domain. The authors demonstrate that RF-CNN can run deep CNNs with up to 26.4 million parameters and nine layers, achieving performance close to full-precision digital models. This method significantly reduces energy consumption, achieving as low as 0.72 femtojoules per multiply-accumulate operation, which is two orders of magnitude less than traditional digital processors. The results suggest that RF-CNN can efficiently enable state-of-the-art AI inference on billions of connected devices, paving the way for a unified communication and computing architecture.
Methodology
The RF-CNN architecture involves a central radio broadcasting machine learning weights to edge devices, which generate local inputs. The convolutional operations are performed in a passive frequency mixer that directly computes the convolution using the received weights and local inputs, leveraging the inherent capabilities of the communication hardware.
Results
RF-CNN successfully runs deep CNNs with up to 26.4 million parameters and nine layers, achieving classification accuracy comparable to full-precision digital models. The energy efficiency of the system is highlighted by its consumption of only 0.72 femtojoules per multiply-accumulate operation, significantly lower than traditional digital processors.
Implications
The findings suggest that existing wireless infrastructure can be utilized for efficient AI inference, potentially transforming how edge devices process data. This could enhance applications in various fields, including smart devices, IoT, and real-time signal processing, while addressing challenges related to latency, privacy, and computational resource constraints.
Sharp Reconstruction Bounds for Autoencoders Using the Same Forward Map
Theory
- Establishes sharp reconstruction bounds for autoencoders using the same forward map.
- Derives a reconstruction-derivative error bound based on Jacobian singular values.
- Demonstrates that affine maps can achieve the derived bounds at any depth.
- Validates theoretical predictions with empirical results from a large LiDAR dataset.
Read more
Sharp Reconstruction Bounds for Autoencoders Using the Same Forward Map
Summary
This paper investigates the reconstruction capabilities of autoencoders that utilize the same forward map for both encoding and decoding processes. The authors focus on scenarios where observed coordinates are set to zero, and they derive sharp bounds on reconstruction errors for autoencoders with equal odd input and hidden dimensions (d ≥ 3). The main contribution is the establishment of a reconstruction-derivative error bound, which is expressed as max{1 - M(M - m)/2, 0}, where M and m are the maximum and minimum singular values of the Jacobian of the transformation. The authors demonstrate that affine maps can achieve this bound at any specified depth. They also explore the implications of using a translated radial rotation, which can reconstruct inputs within a specified ball exactly, even with singular values close to one. The theoretical predictions are validated through experiments on a large terrestrial LiDAR forest scan dataset, showing that the mean theoretical bound closely aligns with the mean normalized training error. The results indicate that adding one hidden coordinate significantly reduces reconstruction error, highlighting the importance of dimensionality in autoencoder performance.
Methodology
The authors analyze the geometric properties of autoencoders that apply the same learned forward map for both encoding and decoding. They derive mathematical bounds on reconstruction errors using properties of orientation-preserving diffeomorphisms and their Jacobians. The analysis includes theoretical proofs and empirical validation through experiments on a large dataset.
Results
The study finds that the least uniform reconstruction-derivative error is max{1 - M(M - m)/2, 0}, with affine maps achieving this bound. In experiments with a 798,452-point LiDAR forest scan, the mean theoretical bound at input scale 0.05 was 0.155, which is about 84% of the mean normalized training error of 0.185. Adding one hidden coordinate reduced the mean reconstruction error to below 6 × 10^-6.
Implications
The findings suggest that careful design of autoencoder architectures, particularly regarding the dimensionality of hidden layers, can lead to significant improvements in reconstruction performance. This has potential applications in fields requiring accurate data representation and reconstruction, such as computer vision and remote sensing.
MATCH: Model-Aware Tool Learning with Curriculum Scheduling and Hierarchically Gated Rewards
NLP
Large Language Models
Reinforcement Learning
- Identifies and addresses scheduling and reward gaps in RL-based tool learning.
- Introduces Model-Aware Curriculum Learning (MACL) for adaptive sample difficulty.
- Presents Hierarchical Tool-call Gated Reward (HTGR) for structured reward evaluation.
- Demonstrates superior performance of MATCH over existing baselines in tool learning tasks.
Read more
MATCH: Model-Aware Tool Learning with Curriculum Scheduling and Hierarchically Gated Rewards
Summary
The paper presents MATCH, a novel framework for model-aware tool learning that addresses two significant challenges in reinforcement learning (RL) for large language models (LLMs): the scheduling gap caused by fixed-threshold curricula and the reward gap due to additive credit aggregation. MATCH incorporates two main components: Model-Aware Curriculum Learning (MACL) and Hierarchical Tool-call Gated Reward (HTGR). MACL dynamically adjusts sample difficulty based on the evolving capabilities of the policy, ensuring that the training samples remain aligned with the model's current performance. HTGR provides a structured reward system that evaluates tool-call components hierarchically, granting credit only when all prerequisite conditions are met, thereby avoiding spurious rewards from incorrect tool selections. The framework was empirically validated through experiments on API-Bank and BFCL V3 datasets, demonstrating superior performance compared to existing supervised and RL-based methods. The results indicate that MATCH not only enhances the accuracy of tool learning but also generalizes well across different model architectures.
Methodology
MATCH employs a closed-loop framework combining MACL and HTGR. MACL adapts sample selection based on reward-derived difficulty estimates that evolve with the policy, while HTGR scores tool-call components hierarchically, providing denser feedback and avoiding additive reward leakage. The integration of these components allows for continuous improvement in both policy optimization and sample scheduling.
Results
MATCH achieved overall accuracies of 72.19% on API-Bank and 62.87% on BFCL V3, outperforming key supervised and RL-based baselines. Ablation studies confirmed the complementary benefits of MACL and HTGR, with consistent improvements observed across four different backbone models.
Implications
The proposed framework has significant implications for enhancing the capabilities of LLMs in real-world applications that require external tool invocation, such as automated customer support, data retrieval, and complex problem-solving tasks. By improving the efficiency and accuracy of tool learning, MATCH can facilitate more effective human-computer interactions.
ZeroHAT: Behavior-Conditioned Zero-Shot Human Activity Trace Generation
Generative Models
Time Series
Optimization
- ZeroHAT generates synthetic HATs without requiring target-region data, using only source-region data and contextual information.
- The framework includes innovative components for intent extraction, behavioral cloning, and activity realization.
- ZeroHAT significantly outperforms existing methods in terms of utility and fidelity across multiple target regions.
- The approach demonstrates the effectiveness of transferring behavioral patterns across regions with different POI distributions.
Read more
ZeroHAT: Behavior-Conditioned Zero-Shot Human Activity Trace Generation
Summary
The paper presents ZeroHAT, a novel framework for generating synthetic human activity traces (HATs) in a zero-shot manner, addressing the challenges of high collection costs and privacy concerns associated with real HATs. Unlike existing methods that rely on data from the same region, ZeroHAT leverages behavioral patterns from source regions and contextual information about target regions to create realistic HATs. The framework consists of three main components: a multidimensional consistency-aware intent extractor that captures temporal, semantic, and spatial intents; a cross-region behavioral cloning module that learns region-invariant actions; and a behavior-conditioned activity realization module that constructs a dynamic action-POI graph to ground actions onto target POIs. The evaluation of ZeroHAT on a ten-city benchmark demonstrates its superior performance, achieving significantly higher normalized downstream utility and improved fidelity compared to existing baselines. This work highlights the potential of zero-shot learning in synthetic data generation for urban mobility applications.
Methodology
ZeroHAT employs a behavior-conditioned framework that includes a multidimensional intent extractor for capturing activity characteristics, a cross-region behavioral cloning module for learning invariant actions, and a dynamic action-POI graph for activity realization. It integrates these components using a consistency-guided product-of-experts to generate timestamped activities.
Results
In extensive evaluations, ZeroHAT achieved 4.5–6.4 times the normalized utility of the strongest baseline and improved average fidelity by 15.6%–40.8%. The framework demonstrated high computational efficiency, reaching approximately three times the throughput of the fastest neural baseline while using significantly less GPU memory.
Implications
The findings suggest that ZeroHAT can facilitate the generation of synthetic HATs for urban mobility applications, enhancing mobility prediction, urban simulation, and location-based services without the need for extensive real-world data collection. This approach could also inform future research on zero-shot learning in other domains.
Expected Hypervolume Maximization for Multiobjective Optimization under Uncertainties
Optimization
- Framing multiobjective optimization under uncertainty as a Bayesian decision problem.
- Maximizing the expected hypervolume as a utility function for optimization.
- Utilizing Gaussian Processes as surrogate models to reduce computational costs.
- Introducing active learning strategies to improve surrogate model performance.
Read more
Expected Hypervolume Maximization for Multiobjective Optimization under Uncertainties
Summary
This paper addresses the challenge of multiobjective optimization in the presence of uncertainties by framing it as a Bayesian decision problem. Instead of simply taking the expectation of each objective, the author proposes maximizing the expected hypervolume, which serves as a utility function. The optimization is conducted over a finite set of candidate solutions using stochastic gradient methods, while also considering the issue of dominated points. To mitigate computational costs, Gaussian Processes are employed as surrogate models to approximate the objective functions, allowing for more efficient optimization. Additionally, the paper introduces active learning strategies through acquisition functions that enhance the surrogate model's performance for the specific multiobjective optimization task. The proposed methods are evaluated on simple analytical problems to demonstrate their effectiveness.
Methodology
The methodology involves formulating the multiobjective optimization problem as a Bayesian decision problem, maximizing the expected hypervolume using stochastic gradient methods, and employing Gaussian Processes as surrogate models. Active learning strategies are also integrated to refine the surrogate model.
Results
The results indicate that the proposed approach effectively maximizes the expected hypervolume while managing uncertainties in multiobjective optimization. The use of Gaussian Processes and active learning strategies significantly enhances the optimization process, leading to improved performance on the tested analytical problems.
Implications
The findings suggest that this framework can be applied to various scientific and industrial fields where decision-making involves multiple objectives affected by uncertainties. The approach may lead to more robust optimization solutions in real-world scenarios.
Local Sparsity Enables Unsupervised LLM Safety Detection
Large Language Models
NLP
Theory
- Proposes an unsupervised anomaly detection framework for LLM safety that does not rely on unsafe training data.
- Utilizes the Linear Representation Hypothesis to exploit local sparsity in LLM activations for improved anomaly detection.
- Demonstrates that locally sparse methods can achieve near-optimal performance with minimal computational resources.
- Validates the approach across multiple LLM architectures and safety-specific datasets.
Read more
Local Sparsity Enables Unsupervised LLM Safety Detection
Summary
This paper addresses the limitations of supervised safety methods for large language models (LLMs), which often rely on labeled unsafe examples that may not generalize well due to evolving unsafe distributions. The authors propose an unsupervised approach to safety detection framed as anomaly detection, leveraging the Linear Representation Hypothesis (LRH) that suggests LLM activations exhibit local sparsity. By employing a sparse autoencoder (SAE), the authors develop a framework for locally masked anomaly detection that identifies out-of-distribution inputs without needing unsafe data. The proposed methods, FREQMASK-KM and LEARNEDMASK-LORA, exploit local sparsity in LLM activations to enhance anomaly detection performance. The framework is validated across various LLM architectures and datasets, demonstrating that it can achieve near-optimal performance using only a small fraction of the SAE neurons. The findings indicate that local sparsity is crucial for effective anomaly detection in high-dimensional LLM representations, and the theoretical underpinnings provide a sample-complexity bound that supports the feasibility of this approach.
Methodology
The authors introduce a framework for anomaly detection based on locally masked sparse autoencoders (SAE). They isolate four modeling choices to exploit local sparsity: embedding space, clustering, choice of sparse subspace within clusters, and scoring rules. Two methods are instantiated: FREQMASK-KM, which operates at the cluster level without training, and LEARNEDMASK-LORA, which uses low-rank adapters for each cluster.
Results
The proposed methods were validated on six instruction-tuned LLMs and demonstrated performance on par with or better than standard baselines. The results showed that local sparsity is essential for effective anomaly detection, with the methods achieving high AUROC scores while maintaining low false positive rates.
Implications
This work has significant implications for the deployment of LLMs in real-world applications, particularly in enhancing safety measures without the need for extensive labeled datasets. The unsupervised approach could adapt more readily to new types of unsafe inputs, making LLMs safer and more robust in dynamic environments.
Alliance Beats Isolation: Unifying Heterogeneous Allied Datasets Improves Classifier Performance
Theory
Optimization
Efficient ML
- Introduces the notion of allied datasets with similar class labels but heterogeneous feature spaces.
- Proposes a method for merging feature spaces using matrix completion techniques.
- Demonstrates that classifiers trained on unified datasets outperform those trained on individual datasets.
- Highlights the potential for improved generalization and knowledge transfer across datasets.
Read more
Alliance Beats Isolation: Unifying Heterogeneous Allied Datasets Improves Classifier Performance
Summary
This paper introduces the concept of allied datasets, which are characterized by having similar class labels but disjoint underlying objects and heterogeneous feature spaces. The author proposes a novel method to merge the feature spaces of two allied datasets into a single unified feature space. This is achieved through a matrix completion technique that synthesizes a unified dataset from the merged feature space. The hypothesis is that this unified representation enhances the transfer of classification knowledge between the datasets. Experimental results demonstrate that classifiers trained on this unified dataset consistently outperform those trained separately on the individual allied datasets across various pairs of datasets. The findings suggest that unifying allied datasets can significantly improve classifier performance, providing a practical approach to leverage multiple datasets in machine learning applications.
Methodology
The methodology involves defining allied datasets, merging their feature spaces into a unified representation, and employing matrix completion techniques to create a unified dataset. The performance of classifiers trained on this unified dataset is then compared to those trained on the individual datasets.
Results
The experiments conducted on several pairs of allied datasets showed that classifiers trained on the unified dataset consistently achieved higher accuracy than those trained on the separate allied datasets, validating the hypothesis that merging datasets enhances classification performance.
Implications
The findings have significant implications for various domains such as education, finance, and engineering, where multiple independent datasets exist. By utilizing the proposed unification method, practitioners can improve model performance and generalization, leading to better decision-making and predictive capabilities.
AURA: Adaptive Uncertainty-Routed Analysis for Email Threat Detection
Multimodal
- Introduction of AURA, a multi-modal email threat detection system.
- Utilizes an Adaptive Uncertainty Router to manage prediction uncertainty.
- Achieves high performance on both in-distribution and out-of-distribution datasets.
- Combines URL analysis with semantic content analysis for improved detection.
Read more
AURA: Adaptive Uncertainty-Routed Analysis for Email Threat Detection
Summary
The paper presents AURA, a novel multi-modal email threat detection system designed to combat the increasing sophistication of email spam and phishing attacks, particularly those leveraging large language models. AURA addresses three main limitations of existing detection systems: reliance on handcrafted features, single-signal analysis, and inadequate out-of-distribution evaluation. The system employs an Adaptive Uncertainty Router (AUR) that quantifies prediction uncertainty from a lightweight URL classifier. Depending on the uncertainty level, emails are either classified directly, sent for semantic analysis if they contain no URLs, or escalated to a fine-tuned transformer encoder for deeper analysis. This architecture allows for effective integration of URL structural analysis and semantic content analysis. The system was evaluated on eight diverse training corpora and two real-world datasets, demonstrating robust generalization capabilities. AURA achieved a macro F1-score of 0.9858 on in-distribution data and maintained scores of 0.9502 and 0.9436 on out-of-distribution datasets, indicating its effectiveness in real-world scenarios.
Methodology
AURA employs a two-layer architecture where the first layer uses a lightweight URL classifier to assess prediction uncertainty. Emails with ambiguous classifications are escalated to a fine-tuned DistilBERT encoder for semantic analysis. The outputs from both layers are combined using an evidential fusion layer to produce the final classification.
Results
AURA achieved a macro F1-score of 0.9858 on the in-distribution test set. On the NazPhish-Eval and GuenterTrap-Eval datasets, which are out-of-distribution, it maintained F1-scores of 0.9502 and 0.9436, respectively, demonstrating its robustness against genuine distribution shifts.
Implications
The findings suggest that AURA can significantly enhance email threat detection capabilities, making it a valuable tool for organizations facing evolving phishing and spam threats. Its modular design allows for easy integration into existing security frameworks, potentially improving overall email security.
Improving Offline Goal-Conditioned Reinforcement Learning via Selective Reward Stimulation
Reinforcement Learning
Robotics
Theory
- RSIQL improves offline GCRL by providing additional reward signals at intermediate states.
- The method does not require a hierarchical policy, simplifying the learning process.
- Experiments show significant performance improvements over existing methods.
- The approach addresses the issue of delayed goal-completion supervision effectively.
Read more
Improving Offline Goal-Conditioned Reinforcement Learning via Selective Reward Stimulation
Summary
This paper addresses the challenges of offline goal-conditioned reinforcement learning (GCRL), particularly in scenarios with sparse rewards and long-horizon dependencies. The author identifies that goal-completion information can be temporally distant from the actions that lead to success, complicating the learning process. To tackle this issue, the paper introduces Reward Stimulation Implicit Q-Learning (RSIQL), a non-hierarchical method that enhances training supervision by applying additional reward signals at intermediate states deemed to contribute to goal progress. This approach contrasts with hierarchical methods that typically require learning a separate high-level subgoal policy. RSIQL utilizes an auxiliary goal-conditioned value function to identify these intermediate states and applies reward stimulation accordingly. The experiments conducted on D4RL goal-reaching benchmarks and OGBench demonstrate that RSIQL outperforms traditional goal-conditioned IQL and achieves competitive performance compared to hierarchical methods while maintaining a simpler policy structure.
Methodology
The paper proposes RSIQL, which employs an auxiliary goal-conditioned value function to identify intermediate states that indicate progress toward the final goal. It applies reward stimulation at these states to enhance the training signal, thereby improving the learning of the policy and value functions using an IQL-style offline reinforcement learning objective.
Results
The experimental results indicate that RSIQL consistently outperforms traditional goal-conditioned IQL methods and achieves performance on par with hierarchical offline goal-conditioned methods, demonstrating its effectiveness in improving learning from offline datasets.
Implications
The findings suggest that enhancing training signals at informative intermediate states can significantly improve offline GCRL performance, which has implications for various applications in robotics, navigation, and autonomous systems where online exploration is impractical.
Seismic Site Response Prediction from Sparse Observations Using Finite-Element-Pretrained Latent Dynamics
Time Series
- FLARE-T improves seismic site response predictions by integrating finite-element simulations with sparse field observations.
- The framework learns low-dimensional latent dynamics to connect input and output accelerations at multiple depths.
- Validation results show reduced prediction errors in acceleration histories and response spectra compared to conventional models.
- The approach demonstrates robustness across different earthquake intensities and source models.
Read more
Seismic Site Response Prediction from Sparse Observations Using Finite-Element-Pretrained Latent Dynamics
Summary
This paper addresses the challenge of improving seismic site response predictions, which often deviate from observed data due to limited sensor coverage and event records. The authors propose a novel framework called Transfer-Enabled Forced Latent Autoencoder for Response Equations (FLARE-T), which learns low-dimensional latent dynamics that connect base acceleration inputs to acceleration outputs at multiple depths. By leveraging dense finite-element simulations, FLARE-T constructs a response manifold and calibrates dynamics using sparse field observations. The methodology involves initializing predictions with a short response window while driving the response with complete base motion. The framework was validated using a layered-soil centrifuge test and the Lotung field vertical array, demonstrating significant improvements in multi-depth acceleration histories and pseudoacceleration response spectra compared to traditional finite-element models. The results indicate that FLARE-T effectively reduces prediction errors across various sensor locations and earthquake intensities, showcasing its potential for data-efficient seismic site response analysis.
Methodology
The authors developed the FLARE-T framework, which utilizes a forced latent autoencoder to learn low-dimensional representations of seismic site responses from dense finite-element simulations. The model is trained to map simulated sensor responses into learned coordinates and calibrate these dynamics using limited field records. The predictions are initialized with a short response window and driven by complete base motion.
Results
The evaluation of FLARE-T using a layered-soil centrifuge test and the Lotung field vertical array showed significant improvements in the accuracy of multi-depth acceleration histories and 5%-damped pseudoacceleration response spectra. The framework reduced errors at all evaluated sensors for various earthquake motions, demonstrating its effectiveness in reconciling numerical predictions with observed data.
Implications
FLARE-T provides a promising approach for enhancing seismic site response predictions, particularly in scenarios with limited observational data. This methodology could be applied in engineering practices for seismic design and assessment, improving the reliability of predictions in real-world applications.
PhyRestore: Physics-Structured Latent-Factor Restoration
Time Series
Theory
Optimization
- PhyRestore restores corrupted physical factors from bitemporal spatial observations.
- The framework reconstructs temporal changes through the known RUSLE relationship.
- Factor restoration improves recovery of rare high-magnitude changes under certain conditions.
- The study compares multiple learning pathways for soil-loss prediction.
Read more
PhyRestore: Physics-Structured Latent-Factor Restoration
Summary
The paper addresses the challenge of estimating temporal soil-loss changes in the presence of noisy or corrupted input factors, particularly when significant changes are rare compared to numerous locations with minimal change. The authors introduce PhyRestore, a physics-structured latent-factor restoration framework that aims to restore corrupted physical factors and reconstruct temporal changes using the Revised Universal Soil Loss Equation (RUSLE). The study evaluates PhyRestore in a watershed-scale bitemporal raster setting, focusing on the isolated and simultaneous corruption of rainfall erosivity and cover management factors. The results indicate that factor restoration enhances recovery of high-magnitude changes when the corrupted factors are identifiable, but its effectiveness diminishes under joint corruption and when factor values fall outside the training support. The paper contributes to the understanding of signed temporal soil-loss prediction under controlled factor corruption and compares various learning pathways, including analytical estimates and machine learning models like Direct RF, XGBoost, MLP, and CNN.
Methodology
The authors formulated a framework that estimates clean values of corrupted RUSLE factors from bitemporal spatial context. They evaluated the performance of PhyRestore against degraded analytical estimates and various machine learning models (RF, XGBoost, MLP, CNN) under controlled conditions of factor corruption.
Results
The evaluation showed that PhyRestore significantly improves the recovery of high-magnitude soil-loss changes when the corrupted factors are identifiable. However, its advantages weaken in scenarios of joint corruption and when factor values are outside the training support, highlighting the limitations of the framework in certain conditions.
Implications
PhyRestore has potential applications in environmental monitoring and management, particularly in assessing soil erosion and land degradation. The framework can enhance the reliability of soil-loss estimates in regions where direct measurements are difficult to obtain, thereby aiding in better decision-making for land use and conservation practices.
Online Adaptive Kernel Mixing for Gaussian Process Decision Making
Optimization
Theory
- HACK GPs adaptively select kernels in Gaussian Processes to improve decision-making performance.
- The method treats kernel selection as an online learning problem using AdaHedge for dynamic updates.
- Two variants of HACK are introduced: Mixture of Gaussians and categorical sampling.
- The approach shows improved performance over standard kernels and ensemble methods in empirical evaluations.
Read more
Online Adaptive Kernel Mixing for Gaussian Process Decision Making
Summary
This paper introduces HACK GPs (Hedge Adaptive Cumulative Kernels), a novel approach to improve the performance of Gaussian Processes (GPs) in sequential decision-making tasks such as Bayesian optimization, level set estimation, and Bayesian active learning. The authors argue that the effectiveness of GPs is heavily reliant on the choice of kernels, and standard kernels can lead to suboptimal performance due to kernel misspecification. HACK GPs treat kernel selection as an online learning problem, where each candidate kernel is viewed as an expert. The method employs AdaHedge to dynamically update a distribution over these experts based on their predictive performance, allowing the model to adaptively focus on the most suitable kernels over time. Two variants of HACK are proposed: a mixture of Gaussians (MoG) predictive distribution and categorical sampling of a single kernel. The authors provide theoretical guarantees that the weight distribution will concentrate on the best kernel under certain conditions. Empirical results demonstrate that HACK GPs outperform standard kernels and simple ensemble methods across various tasks, showcasing robust performance and adaptability in decision-making scenarios.
Methodology
The authors propose HACK GPs, which utilize an online learning framework where each candidate kernel is treated as an expert. The AdaHedge algorithm is employed to update the weights of these experts based on their predictive performance, allowing for adaptive kernel selection. Two implementation variants are provided: a mixture of Gaussians predictive distribution and categorical sampling of a single kernel.
Results
Empirical evaluations indicate that HACK GPs consistently outperform standard kernels like Squared Exponential and Matérn-5/2, as well as simple ensemble baselines, across tasks such as Bayesian optimization, level set estimation, and Bayesian active learning. The method demonstrates robust adaptability and improved decision-making capabilities.
Implications
The proposed HACK GPs can significantly enhance the performance of Gaussian Processes in various applications requiring sequential decision-making, such as optimization problems in engineering and machine learning. This approach could lead to more efficient and effective use of GPs in real-world scenarios where kernel misspecification is a concern.
Don't Mask the Environment: Observation Supervision Changes How Agents Explore Under RL
Reinforcement Learning
Large Language Models
- ActObs introduces a new SFT approach that includes observation prediction, enhancing agent exploration in RL.
- The method shows superior performance in pass rates and task diversity compared to traditional action-only SFT.
- Observation supervision helps maintain policy entropy, facilitating better exploration and learning dynamics.
- The findings suggest that SFT objectives significantly influence downstream learning and exploration capabilities.
Read more
Don't Mask the Environment: Observation Supervision Changes How Agents Explore Under RL
Summary
This paper introduces ActObs, a novel approach to supervised fine-tuning (SFT) in reinforcement learning (RL) that incorporates environment observations as predictive targets alongside agent actions. Traditional SFT methods only apply loss to actions, neglecting the potential benefits of modeling the consequences of those actions through observation prediction. The authors argue that this oversight can hinder the initialization of policies for subsequent RL training. By employing ActObs, which unmasks observations in agent trajectories and applies a joint loss to both actions and observations, the policy learns to predict the outcomes of its actions. This method does not require additional data or parameters and enhances the agent's ability to model action consequences. The results demonstrate that ActObs outperforms traditional action-only SFT methods after RL training, achieving higher pass rates across various sampling budgets and solving more distinct tasks, particularly in cross-domain scenarios. The analysis reveals that ActObs maintains higher entropy during RL, allowing for greater exploration and a more robust policy that retains its initial SFT characteristics, ultimately leading to improved performance in downstream tasks.
Methodology
The authors developed ActObs by modifying the SFT loss to include environment observations as predictive targets. They compared ActObs with standard action-only SFT and a control method that first applies observation-only SFT. The performance was evaluated using the GRPO procedure on the Terminal-Bench 2.0 and cross-domain multilingual code-editing tasks.
Results
ActObs consistently outperformed the action-only SFT method in terms of pass rates at various sampling budgets, achieving a 29% relative advantage at pass@1 on the 4B model and a 14% increase at pass@16 on the 8B model. Additionally, it solved more tasks in cross-domain settings, with a notable 43% improvement at pass@1 on the multilingual code-editing benchmark.
Implications
The findings suggest that incorporating observation supervision in RL training can lead to more effective exploration strategies and improved agent performance across diverse tasks. This approach may have applications in various domains where agents need to learn from complex environments and make informed decisions based on feedback.
Stiefel Attention: When the Geometry of Transformer Projection Matrices Dominates Optimizer Choice---and When It Does Not
Optimization
Theory
- Introduction of Stiefel Attention, optimizing transformer projection matrices on the Stiefel manifold.
- Development of a Riemannian Adam optimizer that respects the geometry of the manifold.
- Significant performance improvements on benchmarks, with validation accuracy rising from 61.1% to 97.0%.
- The method's advantages grow with increasing data, indicating a strong relationship between geometry and optimization.
Read more
Stiefel Attention: When the Geometry of Transformer Projection Matrices Dominates Optimizer Choice---and When It Does Not
Summary
This paper introduces Stiefel Attention, a novel approach to optimizing the query and key projection matrices (WQ, WK) in transformer architectures by constraining them to the Stiefel manifold. Traditional methods use Euclidean optimizers without considering the geometric structure of these matrices, which can lead to suboptimal performance. The author proposes a Riemannian Adam optimizer that incorporates a scalar second moment per frame, a trust-region cap, and polar retraction to ensure that the optimization respects the manifold's geometry. The paper presents four propositions that demonstrate the effectiveness of this approach, including that it is steepest descent in the embedded metric and well-conditioned. The results show significant improvements in validation accuracy on benchmarks such as modular arithmetic grokking and CIFAR-10, with the Riemannian Adam achieving 97.0% accuracy compared to a baseline of 61.1%. The findings suggest that constraining WQ and WK to the Stiefel manifold not only enhances generalization but also maintains performance as data increases, challenging the conventional wisdom regarding optimizer choice in transformer models.
Methodology
The paper employs a Riemannian optimization approach using a modified Adam optimizer tailored for the Stiefel manifold. This includes a scalar second moment for each frame, a trust-region cap to ensure stability, and polar retraction to map updates back onto the manifold. The effectiveness of the method is validated through numerical propositions and extensive experiments on benchmark datasets.
Results
The proposed Riemannian Adam optimizer achieved a validation accuracy of 97.0% on modular arithmetic grokking tasks, significantly outperforming the baseline accuracy of 61.1%. Additionally, on CIFAR-10, the method demonstrated an improvement of +8.98 percentage points over 12 paired starts, with performance gains increasing as more data was introduced, indicating that the benefits of the method scale with dataset size.
Implications
The findings suggest that incorporating geometric constraints into the optimization of transformer models can lead to substantial performance improvements. This approach could influence future research on optimizer design and the architecture of neural networks, particularly in applications where attention mechanisms are critical.
Search at the Cost of Sampling: Nearly-Instant Latent Space Bayesian Optimization
Generative Models
Optimization
Efficient ML
- Introduces a new LSBO algorithm that significantly reduces computational overhead.
- Utilizes linear surrogates constrained to a spherical domain for efficient optimization.
- Achieves over 100× speedup compared to existing BO methods while maintaining high sample efficiency.
- Demonstrates effectiveness across molecular design and image generation tasks.
Read more
Search at the Cost of Sampling: Nearly-Instant Latent Space Bayesian Optimization
Summary
This paper addresses the challenge of making Bayesian optimization (BO) practical for de novo discovery pipelines, where generative models are used to generate candidate designs that are then filtered through virtual screens. Traditional BO methods are often too slow due to their reliance on complex surrogate models, especially when evaluations are relatively inexpensive. The authors propose a novel latent space Bayesian optimization (LSBO) algorithm that utilizes linear surrogates constrained to a spherical domain, significantly reducing computational overhead. By deriving nearly closed-form solutions for surrogate modeling and acquisition, the proposed method achieves at least a 100× speedup compared to state-of-the-art approaches while maintaining or improving performance across various benchmarks in molecular and image generation. This advancement allows BO to be effectively integrated into de novo discovery processes where it was previously impractical due to time constraints.
Methodology
The authors replace traditional nonlinear Gaussian processes with linear surrogates constrained to a spherical domain, which allows for closed-form solutions in surrogate modeling and acquisition optimization. This approach reduces the complexity of each iteration from cubic to linear in the number of samples, enabling rapid computations that complete in less than one second per iteration.
Results
The proposed LSBO method matches or exceeds the sample efficiency of existing methods on benchmarks involving latent spaces of up to 16,384 dimensions, while running over 100× faster in wall-clock time. The performance gap widens with larger observation budgets and latent dimensionality, demonstrating the method's scalability and efficiency.
Implications
This work has significant implications for the fields of drug discovery, materials science, and any domain where rapid generation and evaluation of candidate designs are critical. The efficiency of the proposed LSBO method allows for broader applications of Bayesian optimization in high-throughput settings, making it a valuable tool for researchers and practitioners in generative modeling and optimization.
When Does Retrieval Help Time-Series Forecasting?
Time Series
- Retrieval benefits in time-series forecasting are primarily determined by the ratio of lookback window length to dominant seasonal period.
- A simple control method can outperform complex retrieval plug-ins under certain conditions, particularly when the seasonal structure is strong.
- The study introduces a regime map and two statistics to predict the effectiveness of retrieval mechanisms before deployment.
- Retrieval mechanisms are less effective when the training data lacks a concentrated seasonal structure.
Read more
When Does Retrieval Help Time-Series Forecasting?
Summary
This paper investigates the conditions under which retrieval mechanisms enhance time-series forecasting performance. The authors argue that the benefits of retrieval depend on the relationship between the lookback window length (S) and the dominant seasonal period (L). They demonstrate that a simple control method, which repeats the last observed seasonal period, can outperform complex retrieval plug-ins in certain conditions. The study reveals that at a short window length (S=12), this control method reduces mean squared error (MSE) significantly across various benchmarks, while retrieval mechanisms are less effective when the training data lacks a concentrated seasonal structure. The authors introduce a regime map that helps identify when retrieval is beneficial and propose two interpretable statistics to predict the effectiveness of retrieval before deployment. The findings suggest that the effectiveness of retrieval is closely tied to the phase of the seasonal cycle, highlighting the importance of understanding the underlying temporal structure in time-series data.
Methodology
The authors conducted a series of experiments varying the lookback window length (S), dominant seasonal period (L), and forecasting horizon (H) to evaluate the performance of retrieval mechanisms against a simple control method. They analyzed the mean squared error (MSE) across multiple benchmarks and introduced a regime map to categorize the conditions under which retrieval is beneficial. Additionally, they developed two interpretable statistics to assess the potential effectiveness of retrieval before deployment.
Results
The results indicate that at S=12, the simple control method reduces MSE by 8% to 44% on four out of seven benchmarks compared to standard backbones. It outperformed the strongest retrieval plug-in on the ETTm1 dataset and matched its performance on ECL. However, it performed worse on datasets lacking a concentrated seasonal structure, increasing MSE by up to 25%. The correlation between retrieval benefit and the dominant period was found to be +0.71, while the correlation with the forecasting horizon was -0.23.
Implications
The findings suggest that understanding the relationship between lookback windows and seasonal periods can significantly improve time-series forecasting strategies. The proposed regime map and predictive statistics can aid practitioners in selecting appropriate retrieval mechanisms based on the characteristics of their data, potentially leading to more effective forecasting models.
Recursive Quantum Long Short-Term Memory for Stable Short-Horizon Temperature Forecasting
Time Series
- Introduction of Recursive Quantum Long Short-Term Memory (RQLSTM) architecture.
- Empirical evaluation against standard QLSTM for temperature forecasting.
- RQLSTM shows improved convergence and predictive accuracy.
- Lower mean absolute error and root mean squared error compared to QLSTM.
Read more
Recursive Quantum Long Short-Term Memory for Stable Short-Horizon Temperature Forecasting
Summary
This paper presents a novel architecture called Recursive Quantum Long Short-Term Memory (RQLSTM) aimed at improving the stability and accuracy of short-horizon temperature forecasting. The authors compare RQLSTM with a standard Quantum Long Short-Term Memory (QLSTM) model for one-step-ahead predictions of daily minimum and maximum temperatures using historical weather data from Toronto. The study highlights the challenges of QLSTM optimization, which can be sensitive to initialization and temporal context. By employing a recursive metacore transformation, RQLSTM enhances information propagation across sequences while maintaining a compact architecture. The authors conduct experiments using various input window lengths (8, 16, and 32 days) and report metrics such as mean absolute error (MAE), root mean squared error (RMSE), convergence behavior, and generalization gap across 20 random seeds. The results demonstrate that RQLSTM consistently outperforms the standard QLSTM in terms of faster convergence, lower prediction errors, and reduced generalization gaps, indicating its potential for more stable and effective temperature forecasting.
Methodology
The study employs a comparative analysis of RQLSTM and QLSTM architectures using daily weather data. Both models are trained under identical conditions with varying input window lengths. The performance is evaluated based on convergence behavior, predictive accuracy (MAE and RMSE), and generalization gap, with results averaged over 20 random seeds to account for variability.
Results
The Recursive QLSTM consistently achieves a near-optimal test loss earlier than the standard QLSTM. It also exhibits lower mean absolute error (MAE) and root mean squared error (RMSE), along with a smaller generalization gap, indicating better stability and out-of-sample performance.
Implications
The findings suggest that incorporating recursive quantum transformations can enhance the performance of hybrid quantum-classical models in time-series forecasting tasks, particularly in meteorological applications. This could lead to more reliable temperature predictions, which are crucial for various sectors including agriculture, energy management, and climate research.
Bayesian Optimization with Rich Auxiliary Information via LLMs
Optimization
Large Language Models
- LLMs can effectively leverage rich auxiliary information to enhance Bayesian Optimization.
- Two novel methods are proposed for incorporating LLM-derived priors into BO.
- Utilizing auxiliary observations beyond standard (x, y) interactions improves optimization performance.
- The methods outperform both traditional BO and existing LLM-based optimization techniques.
Read more
Bayesian Optimization with Rich Auxiliary Information via LLMs
Summary
This paper explores the integration of rich auxiliary information into Bayesian Optimization (BO) using Large Language Models (LLMs). Traditional BO methods primarily rely on function evaluations, but real-world optimization problems often provide additional valuable data, such as training curves, expert notes, and prior knowledge about optimal regions. The authors propose three methods that leverage LLMs to incorporate this auxiliary information effectively. They investigate two types of auxiliary information: rich prior knowledge about the optimization landscape and auxiliary observations generated during the optimization process. The study reveals that LLMs can significantly enhance BO performance by modeling distributions over promising inputs and optimizing acquisition functions based on LLM-informed insights. The proposed methods were evaluated on hyperparameter optimization benchmarks and a nuclear fusion optimization task, demonstrating consistent improvements over standard BO and existing LLM-based approaches. The findings highlight the potential of LLMs to utilize diverse and unstructured data, thereby enhancing the optimization process in complex real-world scenarios.
Methodology
The authors developed two algorithms that integrate LLM-derived priors into BO: one combines classical Gaussian Process (GP) inference with LLM insights, while the other integrates LLM-informed distributions into BO acquisition functions. They also examined the impact of auxiliary observations on optimization performance across various tasks.
Results
The proposed methods consistently outperformed standard BO and existing LLM-based optimization methods across hyperparameter optimization benchmarks and a real-world nuclear fusion task, demonstrating the effectiveness of leveraging rich auxiliary information.
Implications
The findings suggest that incorporating rich auxiliary information through LLMs can significantly enhance optimization processes in various fields, including hyperparameter tuning, scientific experimentation, and engineering design, leading to more efficient and effective solutions.
Layer-wise Curriculum Learning for Efficient LLM Compression
Large Language Models
Efficient ML
Optimization
- Introduces layer-wise curriculum learning for efficient LLM compression.
- Addresses cumulative error phenomenon through theoretical analysis and practical implementation.
- Utilizes feature caching and multi-threading to enhance computational efficiency.
- Achieves over 50% reduction in GPU memory usage and training hours compared to traditional methods.
Read more
Layer-wise Curriculum Learning for Efficient LLM Compression
Summary
This paper presents a novel approach to compressing large language models (LLMs) through layer-wise curriculum learning. The authors propose a method that facilitates knowledge transfer from a teacher model to a student model by utilizing a curriculum learning strategy that starts with simpler optimization tasks and progresses to more complex ones. The model is divided into multiple segments of layers to enhance computational efficiency during knowledge transfer. The theoretical analysis reveals a cumulative error phenomenon, which the proposed method addresses by accelerating convergence and stabilizing the knowledge transfer process. Additionally, a feature caching method combined with a multi-threading strategy is introduced to handle feature misalignment across layers, optimizing GPU utilization. Experimental results demonstrate that the method achieves state-of-the-art performance while significantly reducing GPU memory usage and training time by over 50% on models like BERT and GPT-2. The approach also outperforms existing pruning methods on LLaMA-family and Qwen models, confirming its effectiveness in memory-constrained environments.
Methodology
The methodology involves partitioning the LLM into segments of layers for layer-wise knowledge transfer. A curriculum learning approach is adopted, where optimization begins with easier tasks and gradually progresses to harder ones. The authors also implement a feature caching method to reduce computational overhead and utilize a direction-based loss for more effective knowledge transfer.
Results
The proposed method shows a convergence speed approximately 1.7 times faster than conventional methods and achieves state-of-the-art performance across various datasets. It effectively reduces GPU memory usage and training hours by more than 50% on BERT and GPT-2, while outperforming other pruning methods on LLaMA-family and Qwen models.
Implications
The findings suggest that layer-wise curriculum learning can significantly enhance the efficiency of LLM compression, making it feasible to deploy large models in resource-constrained environments. This approach could be beneficial for applications requiring rapid model deployment and reduced computational costs.
COMPASS: Ordered Clustered Routing at 100K Scale
Optimization
- COMPASS is a globally-coordinated algorithm that optimizes the OCTSP using raw distance inputs.
- The algorithm can scale to 100K synthetic nodes and 28.5K real-world e-commerce nodes, achieving state-of-the-art results.
- COMPASS avoids a quality ceiling by modeling global dependencies between clusters.
- The paper introduces a new benchmark suite for large-scale OCTSP, facilitating future research.
Read more
COMPASS: Ordered Clustered Routing at 100K Scale
Summary
The paper introduces COMPASS, an innovative algorithm designed to tackle the Ordered Clustered Traveling Salesman Problem (OCTSP), which is critical in large-scale routing scenarios where clusters of nodes must be visited in a specific order. Traditional methods often optimize clusters independently, neglecting the interdependencies between them, which can lead to suboptimal solutions. COMPASS addresses this by employing a combination of search techniques and learning-accelerated routing, utilizing parallel sub-solvers to improve solution quality without a predefined quality ceiling. The algorithm is capable of processing general distance matrices, making it versatile compared to existing solvers that rely on coordinate inputs. The authors demonstrate COMPASS's effectiveness by scaling it to 100,000 synthetic nodes and 28,500 real e-commerce nodes, achieving the largest reported routing solution over asymmetric distances, significantly surpassing previous benchmarks. The paper also presents a formal optimality guarantee for COMPASS and introduces a competitive approximate algorithm called Pairwise Slider, along with a comprehensive benchmark suite for large-scale OCTSP.
Methodology
COMPASS employs a directed graph representation of the global cluster chain, where the shortest path corresponds to the OCTSP solution. It integrates reinforcement learning techniques through an enhanced GNN architecture to compute intra-cluster solution costs. The algorithm orchestrates parallel sub-solvers to explore the solution space effectively, addressing both intra-cluster paths and inter-cluster bridges simultaneously.
Results
COMPASS consistently outperformed alternative methods in empirical tests, achieving optimal solutions for instances with up to 100K nodes and demonstrating the largest routing solution for asymmetric distances with 28.5K real e-commerce nodes. The algorithm's performance illustrates its capability to leverage compute power for improved solution costs without a quality ceiling.
Implications
The development of COMPASS has significant implications for large-scale routing applications, particularly in logistics and transportation, where optimizing routes can lead to substantial cost savings and reduced environmental impact. The benchmarks provided can serve as a foundation for further research into efficient routing algorithms and their applications in various industries.
Dynamic Generalized Gromov-Wasserstein Optimal Transport
Optimization
Theory
Time Series
- TP-DATE provides a dynamic generalization of Gromov-Wasserstein optimal transport.
- The framework allows for structure-aware transport in applications like spatial transcriptomics.
- Travelling-pair flow matching facilitates the modeling of interacting conditional paths.
- TP-DATE outperforms existing static methods in preserving spatial structure and reconstructing dynamics.
Read more
Dynamic Generalized Gromov-Wasserstein Optimal Transport
Summary
This paper introduces a novel framework called Travelling Pair Dynamical Alignment and Trajectory Estimation (TP-DATE) that generalizes Gromov-Wasserstein optimal transport (GW-OT) to dynamic settings, addressing the limitations of existing static formulations. The authors highlight the importance of structure-aware transport in applications such as spatial transcriptomics, where it is crucial to preserve tissue structure while reconstructing continuous trajectories of gene expression over time. The TP-DATE framework is built on a theoretical foundation that encompasses a broad class of static and dynamic Quadratic-form optimal transport (QOT) problems, proving the static-dynamic equivalence. A key innovation is the development of travelling-pair flow matching, which enables the modeling of interacting conditional paths and integrates their interactions into a single vector field. The effectiveness of TP-DATE is demonstrated through experiments on both synthetic and real spatial transcriptomics data, showing improved preservation of spatial structure and enhanced reconstruction of continuous 3D dynamics compared to existing methods.
Methodology
The authors developed a mathematical theory for dynamic Quadratic-form optimal transport (QOT) that includes standard OT, GW-OT, and IGW-OT as special cases. They introduced a simulation-free travelling pair flow matching framework that learns dynamics with interacting conditional paths. The framework was validated through theoretical proofs and computational experiments on spatial transcriptomics data.
Results
The TP-DATE framework demonstrated superior performance in preserving spatial structure and accurately reconstructing continuous 3D dynamics in both synthetic and real datasets, compared to traditional static optimal transport methods.
Implications
The TP-DATE framework has significant implications for biological data analysis, particularly in spatial transcriptomics, where understanding the dynamics of gene expression and tissue structure over time is crucial. It opens avenues for more accurate modeling of complex biological processes and could enhance research in developmental biology and related fields.
COIN-GP: Cooperative Online Learning in Networked Distributed Systems with Partial Measurements via Gaussian Process Regression
Robotics
Theory
Efficient ML
- Introduction of COIN-GP framework for cooperative online learning in distributed systems.
- Utilization of Gaussian Process regression to handle partial measurements and unknown dynamics.
- Development of a novel data collection strategy with theoretical guarantees.
- Derivation of error bounds for state and model estimation.
Read more
COIN-GP: Cooperative Online Learning in Networked Distributed Systems with Partial Measurements via Gaussian Process Regression
Summary
This paper addresses the challenge of jointly estimating system states and unknown dynamics in distributed sensor networks where only partial observations are available. The authors propose a novel framework called COIN-GP, which integrates online distributed Gaussian Process (GP) regression to facilitate accurate state estimation despite incomplete measurements. The framework incorporates a new data collection strategy with theoretical conditions for effective data acquisition. Additionally, the authors derive an error upper bound for both state and model estimation, leveraging the deterministic error bounds characteristic of GPs. Through empirical simulations, COIN-GP demonstrates superior performance compared to existing distributed GP-based methods, particularly in scenarios where subsystems have limited training data. The proposed approach enhances the efficiency of the learning process and improves overall state estimation by enabling subsystems to share predictions and leverage information from neighboring nodes, thereby increasing prediction accuracy and system robustness.
Methodology
The methodology involves the use of Gaussian Process regression within a cooperative learning framework that allows distributed subsystems to share predictions and learn from each other. The approach includes a new data collection strategy and theoretical conditions for data acquisition, along with the derivation of error bounds for estimation accuracy.
Results
Empirical simulations indicate that COIN-GP outperforms existing distributed GP-based methods, particularly in scenarios with limited data availability across subsystems. The framework effectively enhances state estimation accuracy and robustness in dynamic environments.
Implications
The findings suggest that COIN-GP can be applied in various fields such as sensor networks, multi-robot systems, and autonomous vehicles, where accurate state estimation is critical despite incomplete data. The framework's ability to facilitate cooperation among distributed agents can lead to more efficient and reliable system performance.
Evaluating Explanation Methods by the Predictors They Induce
Interpretability
Theory
- Introduces a framework to evaluate explanation methods based on their predictive power.
- Demonstrates that the effectiveness of explanations varies with feature dependence.
- Finds that existing quality metrics can fail to accurately rank explanation methods.
- Provides a controlled study across 13 real datasets and 9 synthetic designs.
Read more
Evaluating Explanation Methods by the Predictors They Induce
Summary
This paper addresses the challenge of evaluating explanation methods for machine learning models, which often rely on indirect and hard-to-compare criteria. The authors propose a novel framework that evaluates explanations based on their ability to reconstruct model predictions from the explanations themselves. By converting various explanation methods—such as partial dependence plots (PDP), accumulated local effects (ALE), SHAP, and LIME—into predictors, the authors measure how well these predictors reproduce the model's predictions on unseen data. This approach provides a direct and quantitative metric for the predictive information contained in the explanations. The study demonstrates that the effectiveness of different explanation methods varies depending on feature dependence, with SHAP performing slightly worse than PDP under independence but leading on dependent real data. The authors also highlight issues with existing quality metrics, showing that some can prefer degraded explanations over intact ones. Through extensive testing across multiple datasets and models, the paper establishes a new standard for evaluating explanation methods in machine learning.
Methodology
The authors convert various explanation methods into predictors by aggregating feature effects and measuring their predictive performance on unseen data. They conduct a controlled study across multiple datasets and models, comparing the performance of different explanation methods and evaluating existing quality metrics.
Results
The study reveals that SHAP performs slightly worse than PDP when features are independent, but outperforms it on dependent real data. Additionally, some established quality metrics fail to recognize the superiority of intact explanations over degraded ones, indicating potential flaws in current evaluation practices.
Implications
This work has significant implications for the field of explainable artificial intelligence, providing a more reliable method for evaluating explanation methods. It encourages the development of explanations that not only describe model behavior but also enhance predictive performance, thereby improving trust and understanding in machine learning applications.
Opinion Dynamics-based Coalition Formation for Federated Learning in Heterogeneous IoT Systems
Federated Learning
Time Series
- Introduces a coalition formation approach for federated learning that adapts to client heterogeneity.
- Utilizes opinion dynamics to form coalitions based on local model weights, enhancing model compatibility.
- Demonstrates significant improvements in forecasting accuracy for water consumption compared to traditional methods.
- Achieves coalition structures with no additional client-side computation or communication overhead.
Read more
Opinion Dynamics-based Coalition Formation for Federated Learning in Heterogeneous IoT Systems
Summary
This paper addresses the challenges of federated learning (FL) in heterogeneous Internet-of-Things (IoT) systems, particularly in the context of short-term water consumption forecasting. Traditional FL methods, such as Federated Averaging (FedAvg), struggle with statistical heterogeneity, leading to suboptimal global models that do not capture client-specific patterns. The authors propose a novel approach that forms client coalitions based on local model weights using a Hegselmann–Krause (HK) bounded-confidence opinion dynamics process. This method allows for adaptive coalition formation, where the number and membership of coalitions emerge from the data rather than being predetermined. The proposed framework is evaluated against several benchmarks, including FedAvg and FedProx, demonstrating significant improvements in forecasting accuracy and stability. The results indicate that the HK-based coalition formation can reduce the mean absolute error (MAE) by up to 54% compared to FedAvg, while maintaining a high global accuracy of 83-85%.
Methodology
The authors model coalition formation as a bounded-confidence opinion dynamics process, specifically using the Hegselmann–Krause model. They develop three variants of the interaction based on Euclidean distance and cosine similarity criteria to determine compatibility among clients. The framework is instantiated for short-term water consumption forecasting using local Long Short-Term Memory (LSTM) models, and the performance is evaluated against various federated learning methods.
Results
The proposed HK-based coalition formation method produced stable coalition structures within ten iterations and significantly reduced the average MAE by up to 54% compared to FedAvg, 39% compared to FedProx, and 24% compared to Per-FedAvg. The global accuracy achieved was between 83% and 85%, indicating a strong performance in the context of heterogeneous IoT data.
Implications
This work has potential implications for improving federated learning in various IoT applications, particularly in scenarios with heterogeneous data distributions. The coalition formation approach can enhance model accuracy and stability, making it suitable for privacy-sensitive environments such as smart cities.
Learning Principal-Agent Contracts for Equitable Smallholder Carbon Farming under Moral Hazard and Adverse Selection
Reinforcement Learning
Optimization
Theory
- The paper highlights the exclusion of smallholder farmers from carbon farming programs due to contract design issues.
- A novel framework is proposed that combines moral hazard and adverse selection in a multi-season contract design.
- The Realised Adoption Share (RAS) metric is introduced to assess the impact of MRV costs and profit maximization on farmer participation.
- The study finds that profit-maximizing contracts disproportionately benefit larger farms, exacerbating smallholder exclusion.
Read more
Learning Principal-Agent Contracts for Equitable Smallholder Carbon Farming under Moral Hazard and Adverse Selection
Summary
This paper addresses the challenges of designing contracts for smallholder carbon farming, focusing on the issues of moral hazard and adverse selection. Smallholder farmers, who are crucial for climate mitigation through carbon farming, often face exclusion from existing carbon programs that favor larger farms. The authors model the contract design as a multi-season principal-agent Stackelberg game within a partially observable Markov decision process (POMDP). They employ reinforcement learning to develop a dynamic, profit-maximizing contract that accounts for the heterogeneity of smallholder farmers, including differences in adoption costs and unobserved effort. The study reveals that profit-maximizing aggregators amplify the exclusion of smallholders rather than merely inheriting it, with large farms achieving a Realised Adoption Share (RAS) of 87.7% compared to only 8.2% for smallholdings. The paper suggests that socializing Measurement, Reporting, and Verification (MRV) costs could mitigate this disparity, enhancing participation among smallholders. Additionally, the framework serves as a decision-support tool for evaluating policy interventions in carbon farming.
Methodology
The authors formulate the contract design problem as a multi-season principal-agent Stackelberg game embedded in a POMDP. They utilize deep reinforcement learning to train an aggregator that learns dynamic contracts while considering the heterogeneity of smallholder farmers in terms of adoption costs and effort levels.
Results
The results indicate that profit-maximizing aggregators significantly amplify the exclusion of smallholders from carbon farming programs. The RAS metric shows that large farms achieve 87.7% of the potential adoption, while smallholdings only reach 8.2%. A counterfactual analysis suggests that equalizing MRV costs could eliminate this disparity and increase overall participation.
Implications
The findings suggest that policymakers should consider socializing MRV costs to enhance smallholder participation in carbon markets. The developed framework and simulator can be used to inform contract and policy design, potentially leading to more equitable access to carbon income for smallholder farmers.
CARE-VI: Conservative Adaptive Reliability Estimation for Value Improvement in Off-Policy Actor-Critic Learning
Reinforcement Learning
Robotics
Theory
- Introduction of CARE-VI, a framework for improving target reliability in off-policy actor-critic learning.
- Development of three components: CARS, SEVA, and DARE, which address candidate selection, value assessment, and residual regulation.
- Theoretical analysis provides error bounds for the proposed methods, ensuring fixed-policy recovery.
- Empirical results show CARE-VI achieves the highest mean return across multiple tasks and configurations.
Read more
CARE-VI: Conservative Adaptive Reliability Estimation for Value Improvement in Off-Policy Actor-Critic Learning
Summary
The paper introduces CARE-VI, a framework designed to enhance the reliability of temporal-difference (TD) targets in off-policy actor-critic learning. The authors identify that direct value improvement can lead to unreliable target values due to issues such as noisy rankings and fixed enhancement weights. To mitigate these risks, they propose three key components: Conservative Adaptive Ranking and Screening (CARS), Selector-Evaluator Value Assessment (SEVA), and Dynamic Adaptive Risk-aware Enhancement (DARE). CARS manages the candidate selection process by retaining a leading prefix of candidates until a significant uncertainty gap is observed. SEVA combines a selector critic for ordering candidates with an evaluator critic for reviewing selected values, capping the reviewed value at the selector reference. DARE adjusts the residual corrections based on candidate reliability and the divergence between selector and evaluator signals. The integration of these components into CARE-VI allows for evidence-regulated target construction while maintaining existing interfaces for critic regression and actor updates. The authors provide theoretical bounds on the errors introduced by each component and demonstrate that CARE-VI achieves superior performance across various MuJoCo tasks, outperforming existing methods.
Methodology
The methodology involves the development of three main components: CARS for adaptive candidate screening, SEVA for value assessment using separate critics, and DARE for regulating residual corrections based on candidate reliability. These components are integrated into the CARE-VI framework, which maintains the existing structures for actor-critic learning while improving the reliability of target values through evidence regulation.
Results
CARE-VI demonstrated the highest mean return in all twelve experimental settings across four MuJoCo tasks when compared to baseline methods such as SAC, TD3, and TD7. The results were supported by grouped ablation studies that confirmed the contributions of each component to the overall performance.
Implications
The findings suggest that CARE-VI can significantly enhance the performance of off-policy actor-critic methods in reinforcement learning applications, particularly in continuous control tasks. The framework's ability to improve target reliability may lead to more stable and effective learning in complex environments.
Distributionally Robust Federated Learning with Multi-Source Data
Federated Learning
Optimization
Theory
- Introduces a global ambiguity set that captures both cross-client mixture uncertainty and within-client distributional ambiguity.
- Establishes out-of-sample performance guarantees for the proposed robust solution.
- Develops a federated algorithm (BiDRO-FL) with proven convergence under relaxed conditions.
- Validates the effectiveness of the proposed framework through simulations.
Read more
Distributionally Robust Federated Learning with Multi-Source Data
Summary
This paper addresses the challenges of distributionally robust federated learning (FL) in scenarios where clients have heterogeneous data-generating distributions. Traditional FL methods often rely on inaccurate estimates of local distributions, which can lead to suboptimal model performance. The authors propose a novel framework that constructs a global ambiguity set, which combines client-specific ambiguity sets to account for both cross-client mixture uncertainty and within-client distributional ambiguity. By leveraging Distributionally Robust Optimization (DRO), the authors derive a high-probability out-of-sample performance guarantee and develop a federated algorithm, BiDRO-FL, that reformulates the problem using a penalty-based approach. The convergence of this algorithm is proven under milder regularity conditions, and simulations demonstrate its effectiveness in improving model robustness and generalization across clients with varying data distributions.
Methodology
The authors utilize Distributionally Robust Optimization (DRO) to create a global ambiguity set that encompasses admissible mixtures of local Wasserstein ambiguity sets. They derive performance bounds and develop a federated algorithm that employs a Lagrangian penalty reformulation to ensure convergence.
Results
The proposed BiDRO-FL algorithm demonstrates improved robustness and generalization capabilities in federated learning settings with heterogeneous client data. The simulations validate the theoretical guarantees and show that the algorithm effectively handles the uncertainties associated with local distribution estimates.
Implications
This work has significant implications for federated learning applications in fields such as healthcare and mobile computing, where data privacy and distributional heterogeneity are critical. The proposed framework can lead to more reliable and fair models that perform well across diverse client data distributions.
Labeled Incidence Structures for Native Transformer Modeling of Text, Knowledge Graphs, and Hypergraphs
NLP
Graph Learning
Theory
- LIS enables a single transformer to process text, KGs, and hypergraphs natively without architecture changes.
- The multiplicative addressing method preserves joint information between roles and relations, unlike additive methods.
- Content-computed operators address storage order issues in knowledge repositories, improving model robustness.
- The framework shows potential for unified training across different structured data types.
Read more
Labeled Incidence Structures for Native Transformer Modeling of Text, Knowledge Graphs, and Hypergraphs
Summary
This paper introduces Labeled Incidence Structures (LIS) as a unified representation for processing text, knowledge graphs (KGs), and hypergraphs within a single transformer architecture. The LIS framework encodes each endpoint in a structured manner, capturing content, role, and relation instance without flattening the data into token sequences. By utilizing a multiplicative form for structural addressing, the authors demonstrate that LIS can maintain the unique, path-independent addresses for tokens, allowing for role- and relation-aware inductive biases in attention mechanisms. The paper argues that traditional additive encodings fail to capture joint information between roles and relations, leading to significant information loss. The authors also address challenges in knowledge repositories, highlighting how content-computed operators can mitigate issues related to storage order and model capacity. Through controlled experiments on synthetic and real data, the authors provide evidence that their unified approach is practical and effective, paving the way for future research in n-ary hypergraph training.
Methodology
The paper develops the LIS framework by defining a structural address for tokens that combines positional, slot, and instance operators. The authors analyze the implications of this representation through theoretical proofs and controlled experiments, comparing the performance of additive versus multiplicative encodings in capturing relational information.
Results
The results indicate that the LIS framework allows for effective processing of diverse structured data types within a single transformer model. The experiments demonstrate that multiplicative encodings significantly outperform additive encodings in preserving joint information, and the content-computed operators effectively mitigate issues related to storage order in knowledge repositories.
Implications
The proposed LIS framework has the potential to enhance the modeling of complex structured data in various applications, including natural language processing, knowledge representation, and graph learning. It could lead to more robust and flexible models that can handle multiple data types without the need for specialized architectures.
A Policy Profile for Croissant: Refusal as a Property of the Dataset
Theory
- Introduces an additive policy profile for Croissant with specified evaluation semantics.
- Defines a closed operator language for dataset operations and conditions.
- Demonstrates that the profile's decisions align with existing descriptor records.
- Presents a minimal evaluation cost for implementing the profile.
Read more
A Policy Profile for Croissant: Refusal as a Property of the Dataset
Summary
This paper introduces an additive policy profile for Croissant, a machine-readable descriptor for ML datasets that utilizes JSON-LD over schema.org. The author identifies a significant gap in the existing Croissant 1.1 specification, which includes data use conditions but lacks a defined evaluation procedure for these conditions. The proposed profile allows datasets to declare the operations they permit and the conditions under which these operations are allowed, utilizing a closed set of five operators with a fully specified decision procedure. The evaluation of this profile is conducted using two distinct corpora, demonstrating that the decisions made by the profile align with the native descriptor records. The results indicate that the added evaluation cost is minimal, and the profile successfully captures the necessary semantics for automated compliance checking. The paper emphasizes that the contribution lies in the evaluation semantics rather than the vocabulary itself, providing a structured approach to dataset governance that separates caller-side and data-side authority.
Methodology
The methodology involves developing an additive policy profile that specifies evaluation semantics for dataset operations. The author employs two distinct corpora for evaluation: one from a real bioinformatics pipeline and another generated from the profile's grammar. The evaluation measures the decision-making process and timing for each operation, ensuring that the results are consistent across different descriptors.
Results
The evaluation shows that the decisions made by the profile match the native descriptor records for the datasets used, with an added evaluation cost of only 11.7 µs compared to a 119 µs decision time. The analysis of the generated corpus confirms that all valid cases agree across different decision records, reinforcing the robustness of the proposed evaluation semantics.
Implications
The implications of this work extend to enhancing dataset governance in machine learning by providing a clear framework for evaluating data use conditions. This could facilitate automated compliance checking and improve the transparency of dataset operations, ultimately contributing to responsible AI practices.
SoftTri: Smooth Triangular Membership Functions for Adaptive Fuzzy Inference Systems
Optimization
Interpretability
Theory
- Introduction of SoftTri, a differentiable triangular membership function that enhances optimization stability.
- Closed-form analytical gradients derived for efficient backpropagation in training.
- SoftTri maintains the interpretability and locality of classical triangular MFs while providing smoothness.
- Experimental results show improved performance over classical triangular MFs and comparable results to Gaussian MFs.
Read more
SoftTri: Smooth Triangular Membership Functions for Adaptive Fuzzy Inference Systems
Summary
This paper introduces SoftTri, a novel differentiable triangular membership function designed to enhance the performance of adaptive fuzzy inference systems (FIS). Traditional triangular membership functions (MFs) are popular due to their interpretability and simplicity; however, their nondifferentiability at knot points poses challenges for gradient-based optimization in adaptive neuro-fuzzy architectures. SoftTri addresses this limitation by employing a smooth soft-hinge mechanism inspired by Swish-type activations, which maintains the geometric structure and localized behavior of classical triangular MFs while ensuring C∞ smoothness with respect to both input variables and membership parameters. The authors derive closed-form analytical gradients for SoftTri, facilitating efficient backpropagation-based learning. The proposed membership function is integrated into a Takagi–Sugeno fuzzy neural network and evaluated across various benchmarks, including one-dimensional and two-dimensional nonlinear approximation tasks, as well as a real-world regression problem using the Airfoil Self-Noise dataset. The experimental results indicate that SoftTri significantly enhances optimization stability and approximation accuracy compared to classical triangular MFs, achieving performance levels comparable to or superior to Gaussian MFs under the same conditions. This work presents a valuable compromise between interpretability and differentiable optimization in modern neuro-fuzzy learning systems.
Methodology
The authors developed SoftTri by replacing hard hinge operations with a differentiable soft-hinge construction. They derived closed-form analytical gradients for the function, enabling efficient backpropagation. The function was integrated into a Takagi–Sugeno fuzzy neural network and tested on various benchmarks to evaluate its performance against classical triangular and Gaussian MFs.
Results
SoftTri demonstrated consistent improvements in optimization stability and approximation accuracy compared to classical triangular membership functions. In experiments, it achieved performance levels comparable to or better than Gaussian MFs, indicating its effectiveness in enhancing gradient-based optimization in neuro-fuzzy systems.
Implications
The introduction of SoftTri could lead to more effective and interpretable fuzzy inference systems, particularly in applications requiring adaptive learning and optimization. This approach may enhance the deployment of fuzzy systems in real-world scenarios where interpretability and performance are critical.
Machine-Learning Assessment of the Predictive Value of Inflammatory Biomarkers for Cognitive Impairment in an Older Hispanic Adult Cohort
Interpretability
- I-309 (CCL1) was identified as a significant predictor of cognitive impairment, with a notable increase in predictive accuracy.
- The study employed a leakage-safe machine learning methodology to ensure robust results.
- The findings underscore the need for external validation of biomarkers in predicting cognitive impairment.
- The research distinguishes between statistical significance and predictive utility in the context of small clinical datasets.
Read more
Machine-Learning Assessment of the Predictive Value of Inflammatory Biomarkers for Cognitive Impairment in an Older Hispanic Adult Cohort
Summary
This study investigates the predictive utility of inflammatory biomarkers for cognitive impairment in older Hispanic adults using a machine learning approach. The authors utilized data from the Panama Aging Research Initiative–Health Disparities (PARI-HD) cohort, comprising 165 participants. They implemented a leakage-safe threshold-likelihood Bernoulli/Categorical Naive Bayes (BNB/CNB) classifier, which was designed to ensure that all data-dependent operations were performed within cross-validation folds to avoid information leakage. The baseline demographic model achieved a ROC-AUC of 0.630±0.017. Among the biomarkers analyzed, I-309 (CCL1) emerged as the most significant predictor, contributing an incremental AUC increase of +0.110. The study emphasizes the importance of distinguishing statistical association from predictive utility, highlighting I-309 as a promising candidate for further validation in predicting cognitive impairment. The findings suggest that while certain inflammatory biomarkers may be associated with cognitive decline, their predictive power varies and requires careful evaluation in clinical settings.
Methodology
The authors used a threshold-likelihood Bernoulli/Categorical Naive Bayes (BNB/CNB) classifier, performing all data-dependent operations within repeated stratified ten-fold cross-validation to prevent leakage. Continuous predictors were transformed into supervised chi-square states, and income was treated as a categorical variable.
Results
The demographic baseline model achieved a ROC-AUC of 0.630±0.017. I-309 (CCL1) was the dominant feature, showing a significant incremental AUC improvement of +0.110. The results were validated across multiple partitions, with a fixed-partition DeLong p-value of 0.0018, indicating the robustness of I-309 as a predictive feature.
Implications
The study suggests that I-309 could serve as an interpretable biomarker for predicting cognitive impairment in clinical settings, pending further validation. The methodology may also be applicable to other small clinical datasets where interpretability is crucial.
Graph-Based Stochastic Power-UCT: Monte-Carlo Graph Search with Power Mean Estimation
Reinforcement Learning
Graph Learning
Theory
- GS-Power-UCT merges states at the same depth to reduce redundancy in simulations.
- The algorithm maintains separate values for states accessed at different depths, preserving estimation accuracy.
- GS-Power-UCT achieves a convergence rate of O(n−1/2) for root estimates, matching traditional methods.
- Two variants of the algorithm are proposed, with GS-Power-UCT-F+ showing improved performance through adaptive horizon control.
Read more
Graph-Based Stochastic Power-UCT: Monte-Carlo Graph Search with Power Mean Estimation
Summary
This paper introduces Graph-Based Stochastic Power-UCT (GS-Power-UCT), a novel algorithm designed to enhance the efficiency of Monte-Carlo Tree Search (MCTS) in stochastic Markov Decision Processes (MDPs). Traditional tree-based MCTS suffers from redundancy when the same state is reached through multiple trajectories, leading to inefficient sample usage. GS-Power-UCT addresses this issue by merging states reached at the same planning depth while maintaining separate values for states accessed at different depths. The authors prove that GS-Power-UCT retains the convergence properties of Stochastic-Power-UCT, achieving a convergence rate of O(n−1/2) for root estimates. The paper also explores two full-state variants: GS-Power-UCT-F, which increases sample sharing but may introduce value mixing, and GS-Power-UCT-F+, which employs an adaptive horizon to mitigate this bias. Experimental results demonstrate that GS-Power-UCT significantly improves sample efficiency compared to both tree-based and graph-based baselines in stochastic planning benchmarks.
Methodology
The authors develop GS-Power-UCT by integrating power mean backups, polynomial exploration bonuses, and a depth-augmented graph search structure. The search graph is constructed as a directed acyclic graph (DAG) where nodes represent states indexed by their depth, allowing for efficient state merging and sample reuse. Theoretical analysis is provided to ensure convergence properties and to quantify the effects of state merging across depths.
Results
The theoretical analysis confirms that GS-Power-UCT preserves the concentration guarantees of Stochastic-Power-UCT, with a root expected error of O(n−1/2). The experimental evaluations on stochastic planning benchmarks reveal that GS-Power-UCT outperforms both tree-based and graph-based MCTS methods in terms of sample efficiency.
Implications
The findings suggest that GS-Power-UCT can be effectively applied to complex decision-making problems in stochastic environments, potentially leading to more efficient algorithms in various applications such as robotics, game playing, and automated planning.
Amortizing Physics-Informed Neural Solvers via Graph Hypernetworks
Graph Learning
Optimization
Theory
- Introduces operator-graph conditioning for amortized PINNs.
- Demonstrates improved accuracy in high-reaction scenarios using term-based descriptors.
- Shows that graph conditioning yields lower mean error on unseen coupling compositions.
- Identifies scenarios where traditional coefficient conditioning performs best.
Read more
Amortizing Physics-Informed Neural Solvers via Graph Hypernetworks
Summary
This paper presents a novel approach to amortizing physics-informed neural networks (PINNs) for solving families of related partial differential equations (PDEs) using graph hypernetworks. The authors propose an operator graph that explicitly represents the relationships between fields, derivatives, terms, and residuals, with coefficients as term attributes. A graph hypernetwork generates initialization codes for a meta-trained factorized PINN, which can be adapted to specific target equations without requiring solution labels. The methodology is evaluated across various PDE families, demonstrating that term-based descriptors improve accuracy in high-reaction scenarios and that graph conditioning outperforms traditional coefficient-vector methods in unseen coupling scenarios. The results indicate that the relational conditioning of equations enhances the adaptation of physics-based solvers, suggesting a shift in how hypernetworks can be utilized for solving complex PDEs.
Methodology
The authors develop an operator graph to represent the relationships among various components of PDEs. A graph hypernetwork is employed to predict configuration codes for a shared factorized PINN, which is then adapted to specific equations by minimizing physics residuals. The approach is evaluated on three families of PDEs, comparing the performance of coefficient-vector, term-set, and graph conditioning methods.
Results
The experiments reveal that term-based descriptors significantly enhance accuracy in high-reaction convection-diffusion-reaction problems. In the two-field Fisher-KPP problem, the graph conditioning method achieved a mean final error that was 35.7% lower than the term set and 67.7% lower than the coefficient vector. However, in a fixed-structure capacitively coupled plasma model, the coefficient vector outperformed other methods, indicating that the effectiveness of relational conditioning varies based on the specific characteristics of the equations.
Implications
The findings suggest that incorporating explicit relationships between equation components can lead to more effective adaptations of physics-informed neural solvers. This approach could be beneficial in various scientific modeling applications where families of PDEs are common, potentially improving the efficiency and accuracy of simulations in fields such as fluid dynamics, materials science, and environmental modeling.
Past, Future, All at Once: Mitigating Stability-Plasticity Dilemma via Post-hoc JANUS Rectification
Theory
Efficient ML
Large Language Models
- Introduces Parameter Space Orthogonality as a feasible condition for preventing catastrophic forgetting.
- Develops a post-hoc, tuning-agnostic framework (JANUS) that rectifies parameter updates after fine-tuning.
- Employs a Multi-step Adaptive Rectification mechanism to dynamically adjust parameter updates.
- Demonstrates significant improvements in knowledge recovery with minimal impact on task performance.
Read more
Past, Future, All at Once: Mitigating Stability-Plasticity Dilemma via Post-hoc JANUS Rectification
Summary
This paper addresses the stability-plasticity dilemma encountered when fine-tuning foundation models on new tasks, which often leads to catastrophic forgetting (CF). The authors critique existing methods that rely on the overly restrictive Subspace Orthogonality condition and propose a novel approach called JANUS (JAcobian NUll Space projection). This post-hoc, tuning-agnostic weight rectification framework introduces the Parameter Space Orthogonality condition, which is both necessary and sufficient for preserving historical performance. By projecting parameter updates into the JANUS, the method effectively recovers lost historical knowledge without disrupting the fine-tuning process. Additionally, the authors introduce a Multi-step Adaptive Rectification mechanism that dynamically adjusts step sizes based on the JANUS shift, ensuring efficient navigation of the parameter space. The JANUS framework integrates seamlessly with various fine-tuning methods and employs techniques like ghost projection and singular value decomposition for enhanced efficiency. Experimental results demonstrate that JANUS significantly mitigates the stability-plasticity dilemma, achieving near-perfect knowledge recovery while maintaining task adaptation.
Methodology
The authors propose the JANUS framework, which rectifies parameter updates post fine-tuning by projecting them into the Jacobian Null Space. This approach is complemented by a Multi-step Adaptive Rectification mechanism that adapts the rectification process based on a proxy metric for forgetting. Techniques such as ghost projection and singular value decomposition are utilized to enhance computational efficiency.
Results
Extensive experiments across various models and tasks show that JANUS effectively recovers historical knowledge while preserving the ability to adapt to new tasks. The method consistently outperforms existing approaches in mitigating catastrophic forgetting, achieving a significant advancement in the stability-plasticity trade-off.
Implications
The findings suggest that the JANUS framework can be widely applied to improve the performance of fine-tuning in large models across diverse applications, potentially enhancing the robustness and adaptability of AI systems in real-world scenarios.
A Noise Optimum in Rehearsal-Free Continual Learning: Isolation, Mechanism, and Scope
Theory
- Injecting stochastic noise into consolidation rules can improve task retention up to an optimal level.
- The optimal noise level is achieved through coherent restoring forces directed towards consolidated weights.
- The coupling of anchor gain to noise variance is a critical factor in achieving retention improvements.
- The noise optimum is dependent on shared task structure and is absent in tasks with permuted structures.
Read more
A Noise Optimum in Rehearsal-Free Continual Learning: Isolation, Mechanism, and Scope
Summary
This paper investigates the effects of injecting stochastic noise into a consolidation rule for rehearsal-free continual learning, revealing an inverted-U relationship between noise and task retention. The study identifies the conditions under which this optimal noise level enhances retention of earlier tasks, demonstrating that coherent restoring forces toward consolidated weights are crucial for achieving this optimum. The research is conducted entirely through simulations on various continual learning benchmarks, including Split-MNIST and FashionMNIST. The findings indicate that the optimal noise level is significantly influenced by the coupling of anchor gain to the injected noise variance, a mechanism not utilized by existing methods like Ornstein-Uhlenbeck Adaptation or MESU. The paper also establishes that the presence of shared task structure is necessary for the observed noise optimum, and it discusses the diminishing advantages of noise coupling as the number of tasks increases. Overall, the study contributes to a deeper understanding of how noise can be strategically employed in continual learning frameworks to mitigate catastrophic forgetting.
Methodology
The research employs simulation experiments across multiple continual learning benchmarks to analyze the effects of injected noise on task retention. A magnitude-matched ladder approach is used to isolate the active ingredients contributing to the retention inverted-U phenomenon, comparing various noise coupling strategies.
Results
The study finds that the retention of tasks improves with injected noise, peaking at an optimal level (σ = 0.02) before declining. The coupling of anchor gain to noise variance recovers most of the retention benefits compared to other methods. The results are consistent across different benchmarks, confirming the necessity of coherent restoring forces and shared task structures for optimal performance.
Implications
The findings suggest that strategically incorporating noise into continual learning systems can enhance their ability to retain knowledge over time, which could lead to more robust AI systems capable of learning continuously without rehearsal. This approach may have applications in various domains where continual learning is essential, such as robotics and adaptive systems.
Personalized Federated Hierarchical Gaussian Processes for Privacy-Preserving Modeling of Heterogeneous Distributed Systems
Federated Learning
- Introduces a hierarchical decomposition for modeling heterogeneous federated clients.
- Employs privacy-preserving federated variational inference to keep raw data local.
- Supports uncertainty-aware predictions, crucial for risk-sensitive applications.
- Demonstrates effectiveness in real-world applications like fault classification and air-quality modeling.
Read more
Personalized Federated Hierarchical Gaussian Processes for Privacy-Preserving Modeling of Heterogeneous Distributed Systems
Summary
The paper introduces Personalized Federated Hierarchical Gaussian Processes (pFedHGP), a novel framework designed for probabilistic regression and classification in heterogeneous distributed systems. The framework addresses the challenges posed by non-i.i.d. data across clients by decomposing each client's latent function into three components: a shared global component, a client-specific deviation that maintains the global kernel structure, and a flexible local residual. This hierarchical approach allows for effective modeling of diverse operational conditions while preserving data privacy. The authors employ sparse inducing-variable approximations and federated variational inference to ensure that raw data remains local, with only low-dimensional statistics being synchronized at the server level. The framework supports uncertainty-aware decision-making through full predictive distributions. In practical applications, pFedHGP achieves perfect fault classification in press tonnage monitoring with minimal labeled data and successfully recovers geographic zones in federated air-quality modeling without centralizing sensitive time series data. The paper also establishes a connection between the hierarchical model and multi-output Gaussian processes, enhancing the understanding of correlated sensor outputs.
Methodology
The methodology involves a three-level hierarchical decomposition of latent functions in Gaussian processes, integrating a global component, client-specific deviations, and local residuals. The training is conducted using a two-stage federated variational inference approach, where clients update local variational factors based on private data while the server synchronizes only compact global statistics.
Results
The pFedHGP framework achieved perfect fault classification in press tonnage monitoring using only 13.77% of labeled cycles and effectively recovered geographic zones in federated air-quality modeling without the need for centralizing station-level data. The results indicate that the model successfully captures both global trends and local variations in heterogeneous environments.
Implications
The implications of this work extend to various domains where data privacy and heterogeneity are critical, such as smart healthcare, predictive maintenance in industrial settings, and urban environmental monitoring. The framework can enhance decision-making processes in these areas by providing reliable uncertainty estimates and maintaining data locality.
SAGG: Sample-Adaptive Gradient Gating for Robust Multimodal Learning under Heterogeneous Corruption
Multimodal
- Identifies a granularity mismatch in existing batch-level gradient modulation under heterogeneous multimodal corruption.
- Proposes SAGG, which utilizes sample-level gating for unbiased gradient estimation.
- Demonstrates that SAGG-based SGD converges at a rate of O(1/√T) without residual bias from corruption.
- Provides a certified robustness framework for evaluating multimodal models.
Read more
SAGG: Sample-Adaptive Gradient Gating for Robust Multimodal Learning under Heterogeneous Corruption
Summary
The paper introduces Sample-Adaptive Gradient Gating (SAGG), a novel framework designed to enhance the robustness of multimodal learning systems under heterogeneous corruption. Traditional multimodal gradient balancing methods apply a uniform modulation to gradients across all samples in a batch, which fails to account for the varying levels of corruption present in individual samples. The authors demonstrate that this approach leads to an irreducible bias in gradient estimation. SAGG addresses this issue by implementing a sample-level gating mechanism that makes binary decisions to retain or discard gradients based on an online quality assessment of each sample. This method ensures unbiased gradient estimation and improves convergence rates to stationary points of the clean loss without incurring a corruption-dependent error floor. The paper also provides a certified robustness framework that connects per-modality Lipschitz constants to classification margins. Experimental results on datasets such as Kinetics-Sounds and UCF-101 show that SAGG outperforms ten existing methods, particularly in high-corruption scenarios where batch-level bias is most pronounced.
Methodology
The authors develop SAGG, which employs a binary retain-or-discard decision for each sample based on an online feature-norm quality test. This is coupled with a truncation mechanism to control variance. The theoretical analysis shows that unbiased estimation requires sample-level gating, and the convergence properties of SAGG-based SGD are established.
Results
SAGG consistently outperforms ten existing multimodal learning methods across various experiments involving Gaussian noise injection, partial modality missing, and natural contribution imbalance. The framework achieves state-of-the-art performance, particularly in high-corruption conditions where traditional methods struggle.
Implications
The findings suggest that adopting sample-level adaptive strategies can significantly enhance the robustness of multimodal learning systems in real-world applications where data corruption is heterogeneous. This could lead to improved performance in fields such as audio-visual processing, sensor fusion, and other multimodal tasks.
Enhanced Agriculture-informed Neural Network by Domain Knowledge
Interpretability
- KAINN integrates domain knowledge into deep learning for improved N2O emission predictions.
- The framework incorporates key environmental processes to enhance model interpretability.
- KAINN outperforms traditional neural network models and the original AINN in accuracy.
- The approach demonstrates reduced uncertainty and improved stability in parameter trajectories.
Read more
Enhanced Agriculture-informed Neural Network by Domain Knowledge
Summary
This paper presents the Knowledge-enhanced Agriculture-informed Neural Network (KAINN), a novel framework designed to improve the prediction of nitrous oxide (N2O) emissions from agricultural practices by integrating domain knowledge into a hybrid neural-mechanistic modeling approach. The authors highlight the challenges in accurately modeling N2O emissions due to the complex interactions between soil, climate, and biochemical processes, compounded by the scarcity of high-quality data. KAINN builds upon the existing Agriculture-informed Neural Network (AINN) by explicitly incorporating critical environmental processes such as fertilizer diffusion, soil respiration rate, and water-filled porosity, which guide the learning process and constrain the solution space. The framework is evaluated using various architectures, including CNN, LSTM, and Transformer models, across multiple growing seasons and input configurations. Results indicate that KAINN consistently outperforms both traditional neural network models and the original AINN in terms of prediction accuracy, achieving lower Root Mean Square Error (RMSE) and Mean Absolute Error (MAE), as well as a higher coefficient of determination (R²). Additionally, the analysis of interface evolution reveals that KAINN produces smoother and more physically consistent parameter trajectories, enhancing interpretability and stability. This work underscores the significance of integrating domain knowledge into deep learning models for environmental applications, providing a scalable approach for reliable and interpretable predictions of N2O emissions in agricultural systems.
Methodology
The study employs a hybrid modeling framework that combines deep learning architectures (CNN, LSTM, Transformer) with mechanistic modeling principles. KAINN integrates domain knowledge related to environmental processes to guide the learning of the neural network, enhancing its predictive capabilities and interpretability.
Results
KAINN consistently achieved lower RMSE and MAE compared to traditional models and the original AINN, along with a higher R² value, indicating superior prediction accuracy and generalization across different environmental conditions. The model also produced smoother parameter trajectories, suggesting enhanced interpretability and reduced uncertainty.
Implications
The KAINN framework has significant implications for environmental modeling, particularly in predicting greenhouse gas emissions from agricultural systems. By integrating domain knowledge, it offers a more reliable and interpretable approach to modeling complex environmental processes, which can inform sustainable farming practices and climate mitigation strategies.
Learning-Based Reconstruction of Optical Properties in Bilayered Media from Single-distance Time-Resolved Reflectance Measurements
Theory
Optimization
Efficient ML
- Proposes a machine learning framework for reconstructing optical properties in bilayered media.
- Generates a robust synthetic dataset using Monte Carlo simulations for training.
- Achieves higher accuracy and faster reconstruction compared to traditional model-based methods.
- Estimates parameter space dimensionality without prior information about layer count.
Read more
Learning-Based Reconstruction of Optical Properties in Bilayered Media from Single-distance Time-Resolved Reflectance Measurements
Summary
This paper addresses the challenge of reconstructing optical properties, specifically absorption and scattering coefficients, in bilayered biological media using time-domain reflectance measurements. Traditional analytical models struggle with structural heterogeneity, leading to inaccuracies in estimating optical properties. The authors propose a machine learning framework as a solution, which is benchmarked against existing model-based algorithms. To train the machine learning model, a synthetic dataset was generated using Monte Carlo simulations at various source-detector distances. The results demonstrate that the machine learning approach not only significantly speeds up the reconstruction process but also achieves higher accuracy compared to traditional methods. Additionally, the machine learning model can estimate the parameter space dimensionality without prior knowledge of the number of layers in the medium. Future work is suggested to enhance accuracy further by employing a joint multi-distance reconstruction approach using multiple time-resolved reflectance curves from the same medium.
Methodology
The authors developed a machine learning pipeline trained on a synthetic dataset generated from Monte Carlo simulations of time-domain reflectance measurements. The framework was validated against state-of-the-art model-based reconstruction methods to assess its efficiency and accuracy.
Results
The machine learning approach demonstrated a significant speed-up in reconstruction times and achieved higher accuracy in estimating the optical properties of bilayered media compared to traditional analytical models. It also provided insights into the dimensionality of the parameter space without requiring prior knowledge of the medium's structure.
Implications
This research has potential applications in biomedical fields, particularly in non-invasive imaging and diagnostics of biological tissues, where accurate reconstruction of optical properties is crucial for understanding tissue composition and structure.
Towards a Unified Modality-Agnostic Multimodal Framework for Cognitive Workload Assessment
Multimodal
- Introduces a modality-agnostic framework for cognitive workload assessment using a hierarchical Transformer architecture.
- Evaluates all combinations of five biosignal modalities in a systematic manner.
- Finds EEG to be the most effective single modality for cognitive workload assessment.
- Demonstrates that combining all five modalities achieves the highest performance scores.
Read more
Towards a Unified Modality-Agnostic Multimodal Framework for Cognitive Workload Assessment
Summary
This paper presents a novel framework for assessing cognitive workload using a unified, modality-agnostic, hierarchical Transformer-based architecture. The authors highlight the importance of cognitive workload assessment in adaptive human-machine systems and emphasize the limitations of existing studies that primarily focus on single-modality or predefined multimodal combinations. The proposed framework processes five heterogeneous biosignal modalities—Electrocardiogram (ECG), Electrodermal Activity (EDA), Respiration (RESP), Peripheral Oxygen Saturation (SpO2), and Electroencephalogram (EEG)—within a single model. A pilot study is conducted involving 11 participants performing three distinct cognitive tasks: abstract reasoning, arithmetic problem solving, and a game task. The study evaluates all 31 possible combinations of the five modalities using a leave-one-subject-out validation approach. The results indicate that EEG is the most effective single modality for cognitive workload assessment, while the combination of all five modalities yields the highest average performance scores. Notably, the proposed method reduces model size by approximately 50% compared to traditional late fusion methods, while also maintaining lower inference times. This work addresses the gap in systematic evaluation of modality combinations and offers a flexible solution for real-time cognitive workload monitoring.
Methodology
The study involved a pilot experiment with 11 participants, where EEG and other physiological signals were collected while participants performed three cognitively distinct tasks. A hierarchical Transformer-based model was developed to process the data from all five modalities without requiring modality-specific components. The evaluation was conducted using a leave-one-subject-out validation approach to assess the performance of all possible modality combinations.
Results
The results showed that EEG was the strongest single modality across all tasks, achieving the highest scores in abstract reasoning and game tasks. The full combination of all five modalities resulted in an average score of 73.02% on the IQ task and 68.08% across all tasks. The proposed framework also significantly reduced model size and inference time compared to traditional methods.
Implications
The findings suggest that the proposed framework can enhance real-time cognitive workload assessment in various high-risk fields, such as aviation and healthcare, where understanding cognitive demand is crucial for safety and performance. The flexibility of the modality-agnostic approach allows for effective monitoring even when sensor resources are limited.
One Intervention per Component is Enough: Towards Identifiability in Linear Stochastic Dynamics from Steady State
Theory
Time Series
Optimization
- One intervention per strongly connected component (SCC) is sufficient for parameter recovery of the OU process.
- A recursive algorithm is developed to topologically order SCCs and isolate marginal dynamics.
- The proposed regularized least-squares estimator effectively minimizes residuals from steady-state equations.
- Theoretical results are validated through empirical experiments, confirming the approach's effectiveness.
Read more
One Intervention per Component is Enough: Towards Identifiability in Linear Stochastic Dynamics from Steady State
Summary
This paper addresses the challenge of recovering parameters of a multivariate Ornstein–Uhlenbeck (OU) process using steady-state observational and interventional data. The author establishes that a single intervention per strongly connected component (SCC) of the drift graph is sufficient to recover all parameters of the OU process, provided that the SCC condensation graph is connected with a single root and certain spectral nondegeneracy assumptions are met. The paper introduces a recursive learning algorithm that topologically orders SCCs, isolates marginal dynamics, and solves a linear system derived from steady-state moment equations. Additionally, a regularized least-squares estimator is proposed to minimize residuals across observational and interventional data. Experimental results demonstrate the effectiveness of the proposed methods in recovering parameters and predicting outcomes of unseen interventions, validating the theoretical findings.
Methodology
The methodology involves establishing identifiability results for the OU process parameters using a combination of observational and interventional data. A recursive algorithm is employed to order SCCs and isolate their dynamics, leading to the formulation of a regularized least-squares optimization problem that minimizes residuals of steady-state mean and covariance equations.
Results
The paper demonstrates that the proposed methods can successfully recover parameters of the OU process from steady-state data with a single intervention per SCC. The experimental validation shows that the approach is robust and effective in predicting the effects of unseen interventions.
Implications
The findings have significant implications for fields such as systems biology and neuroscience, where understanding the dynamics of complex systems from limited data is crucial. The ability to recover parameters from steady-state data can enhance causal inference and predictive modeling in various scientific domains.
Subdomain-aware representation compression for pretrained image embeddings
Computer Vision
Efficient ML
- Dimensionality reduction techniques can be effectively tailored for subdomain representation compression.
- PCA and LDA show significant improvements in both space efficiency and accuracy for pretrained image embeddings.
- The approach allows for effective transfer learning capabilities using compressed representations.
- Experiments demonstrate that compression can outperform traditional full-embedding methods.
Read more
Subdomain-aware representation compression for pretrained image embeddings
Summary
This paper explores the use of dimensionality reduction techniques, specifically Principal Component Analysis (PCA) and Linear Discriminant Analysis (LDA), for compressing pretrained image embeddings in a subdomain-aware manner. The authors argue that traditional dimensionality reduction methods, typically applied uniformly across datasets, can be adapted to focus on subdomains, leading to improved space efficiency and computational performance without sacrificing accuracy. The study involves experiments in both unsupervised and supervised settings, demonstrating that the proposed compression methods can effectively extract important features relevant to specific tasks. Additionally, the paper investigates the transfer learning capabilities of the compressed representations, showing that they can be beneficial even when trained on different but related subdomains. The findings indicate that the proposed methods not only reduce resource requirements but also enhance downstream task performance, making them particularly suitable for edge-device machine learning applications.
Methodology
The authors employed standard dimensionality reduction techniques (PCA and LDA) to compress pretrained image embeddings. They conducted experiments in both unsupervised and supervised settings, using sampled embeddings from subdomains to learn the compression transformation. The effectiveness of the compression was evaluated through clustering tasks and transfer learning scenarios.
Results
The results indicated that the proposed compression methods led to improved space efficiency and enhanced performance in downstream tasks compared to traditional full-embedding approaches. The compression ratios achieved were significantly better than typical usage, and the methods demonstrated strong transfer learning capabilities.
Implications
The findings suggest that the proposed dimensionality reduction techniques can facilitate the deployment of machine learning models on edge devices, allowing for efficient local processing of data without the need for constant access to large remote models. This could enhance privacy and reduce computational costs in various applications.
Radio Frequency Detection and Classification of Microplastics in Water
Theory
Efficient ML
Multimodal
- Introduction of a novel RF dielectric spectroscopic cytometry platform for microplastic detection.
- Achieved high classification performance for eight microplastic classes in deionized water.
- Demonstrated robustness of the method in saline environments.
- Utilized machine learning to analyze RF scattering parameters for material classification.
Read more
Radio Frequency Detection and Classification of Microplastics in Water
Summary
This paper addresses the urgent need for rapid, label-free detection methods for microplastics (MPs) and nanoplastics (NPs) in water, given their increasing prevalence and potential health impacts. The authors present a novel machine learning-assisted radio-frequency (RF) dielectric spectroscopic cytometry (DiSC) platform designed for the detection and classification of MPs in aqueous environments. The study focuses on characterizing eight classes of MPs with diameters ranging from 1 to 10 μm at frequencies between 0.2 and 9 GHz. By analyzing alterations in RF scattering parameters (S-parameters), the authors trained supervised machine learning models to classify materials, achieving macro-average F1-scores, precision, and recall values exceeding 0.71. The method demonstrated robust performance even in saline environments, maintaining classification accuracy for polyethylene terephthalate (PET) at salt concentrations of 3.3% and 6.6%. The findings indicate the feasibility of using RF DiSC combined with machine learning for efficient single-particle classification of microplastics, paving the way for future enhancements in RF calibration, spectral coverage, and dataset size to improve classification performance.
Methodology
The study employed a radio-frequency dielectric spectroscopic cytometry (DiSC) system to measure the RF scattering parameters of microplastics suspended in deionized water. The measurements were conducted at frequencies ranging from 0.2 to 9 GHz. Machine learning models were trained on the collected data to classify different types of microplastics based on their dielectric properties.
Results
The proposed method achieved macro-average F1-scores, precision, and recall values exceeding 0.71 for the classification of eight microplastic classes in deionized water. The classification performance for PET was maintained in saline environments with salt concentrations of 3.3% and 6.6%.
Implications
The findings suggest that the RF DiSC platform could serve as a rapid, efficient method for monitoring microplastic contamination in water, which is critical for environmental health assessments. The approach may also be adapted for detecting and classifying microplastics in various environmental matrices, contributing to better understanding and management of plastic pollution.
Score Centering Stabilizes Off-policy Reinforcement Learning
Reinforcement Learning
Large Language Models
Efficient ML
- Score centering effectively cancels drift caused by training-inference mismatch (TIM).
- It outperforms traditional importance sampling methods under severe quantization conditions.
- The method is additive and can be combined with importance sampling for improved performance.
- The approach allows for stable training of large language models while maximizing hardware utilization.
Read more
Score Centering Stabilizes Off-policy Reinforcement Learning
Summary
This paper addresses the challenges of training large language models (LLMs) using reinforcement learning (RL), particularly focusing on the training-inference mismatch (TIM) that leads to instability during training. The authors identify that the primary cause of instability under TIM is drift, a persistent bias that accumulates over training steps. To mitigate this issue, they introduce a novel correction method called 'score centering,' which is an additive term designed to cancel out the drift. The authors demonstrate that score centering can match or outperform traditional importance sampling methods, especially under conditions of severe quantization. Their experiments with models ranging from 0.6B to 30B parameters show that score centering not only stabilizes training but also enhances performance when combined with importance sampling techniques. This approach allows for more efficient training of LLMs while maintaining high hardware utilization, making it a significant advancement in the field of RL for LLMs.
Methodology
The authors analyze the policy gradient update process to identify the drift term associated with TIM. They propose the score centering correction term and conduct experiments on various model sizes and quantization levels to evaluate its effectiveness compared to existing stabilization methods, including importance sampling.
Results
Experiments show that score centering matches or exceeds the performance of importance sampling methods, particularly as quantization severity increases. The results indicate that score centering stabilizes RL training effectively, leading to better training dynamics and performance in large language models.
Implications
The introduction of score centering has the potential to significantly improve the training stability of large language models in reinforcement learning settings, enabling more efficient use of computational resources and potentially leading to better-performing models in practical applications.
Multi-center Medical Data Mining with FL-Net - A One-stop Shop for Federated Learning
Federated Learning
- FL-Net is a comprehensive federated learning framework tailored for multi-center medical research.
- The framework fulfills five essential requirements for federated clinical research that existing frameworks do not meet.
- FL-Net enables the reuse of harmonized data and workflows, promoting collaboration while ensuring data privacy.
- The framework supports up to 50 concurrent clients in federated workflows, showcasing its scalability.
Read more
Multi-center Medical Data Mining with FL-Net - A One-stop Shop for Federated Learning
Summary
The paper introduces FL-Net, a novel federated learning (FL) framework designed for multi-center medical data mining. The authors identify five critical requirements for effective federated clinical research frameworks, including data harmonization, privacy controls, reusable workflows, data feasibility, and traceability. After analyzing 14 existing FL frameworks, none were found to meet all these requirements. FL-Net addresses these gaps by integrating modular data harmonization, data discovery, and secure workflow execution into a persistent network. The framework allows for the reuse of harmonized data and workflows across studies, enhancing collaboration among institutions while maintaining patient privacy. The authors evaluated FL-Net's capabilities through harmonization and cross-study patient discovery across datasets like MIMIC and US-130, demonstrating its ability to support up to 50 concurrent clients. FL-Net is being developed within two EU projects, aiming to cover over 800,000 patients across 10 hospitals in 9 countries, thus providing a robust foundation for interoperable, reproducible, and privacy-preserving clinical research.
Methodology
The authors conducted a systematic analysis of 14 existing federated learning frameworks against five defined requirements for clinical research. They developed FL-Net, integrating features such as modular data harmonization, secure workflow execution, and version control. The framework was evaluated through practical applications involving data harmonization and patient discovery across different datasets.
Results
FL-Net demonstrated successful harmonization and cross-study patient discovery capabilities, effectively managing workflows with up to 50 concurrent clients. The framework's design allows for the reuse of workflows and harmonized data, fulfilling all identified requirements for federated clinical research.
Implications
FL-Net has the potential to revolutionize multi-center clinical research by enabling secure, privacy-preserving collaboration among institutions. Its modular design can facilitate the integration of diverse datasets, improving the robustness of machine learning models in healthcare applications.
Uni-LaDiR: Latent Diffusion Unifies Multimodal Reasoning
Multimodal
Robotics
NLP
- Uni-LaDiR introduces a unified latent space for multimodal reasoning, improving the integration of information from different modalities.
- The framework employs a unified encoder to generate shared thought tokens, enhancing the reasoning process by reducing modality-specific representation issues.
- Diffusion is used to predict the next reasoning steps, allowing for flexibility in generating multiple valid outputs.
- Joint training of the encoder and diffusion model leads to better task-relevant thought token generation.
Read more
Uni-LaDiR: Latent Diffusion Unifies Multimodal Reasoning
Summary
The paper introduces Uni-LaDiR (Unified Latent Diffusion Reasoner), a novel framework designed to enhance multimodal reasoning by mapping information from various modalities into a shared latent space. Traditional methods often concatenate or interleave modality-specific tokens, which complicates the reasoning process due to the need for the model to bridge representational differences. Uni-LaDiR addresses this by employing a unified encoder that transforms teacher reasoning steps from different modalities into shared thought tokens, preserving essential information for subsequent reasoning and final outputs. The framework utilizes diffusion to predict the next block of thought tokens based on the input and preceding tokens, allowing for multiple valid reasoning paths. The joint training of the encoder and diffusion reasoner with shared weights ensures that the thought tokens are both useful for the task and predictable from the context. The model was evaluated across eleven vision-language model (VLM) benchmarks and two vision-language-action (VLA) suites, demonstrating significant improvements in reasoning accuracy and manipulation success compared to existing methods.
Methodology
The methodology involves a unified encoder that maps heterogeneous modality features into a shared latent reasoning space, generating fixed-width latent blocks as thought tokens. The model uses diffusion to predict the next thought tokens based on the input and previous tokens. Joint training of the encoder and diffusion reasoner is employed to ensure that the thought tokens retain task-relevant information and are predictable from the context.
Results
Uni-LaDiR achieved a relative improvement of 7.3% in average mathematical and logical reasoning accuracy across four VLM benchmarks and a 6.1% increase in manipulation success across ten RLBench tasks. Additionally, flow matching within the unified latent space improved performance by 14.9% over direct L2 prediction and 19.5% over cosine similarity loss.
Implications
The implications of this work suggest that separating modality-specific perception from shared latent reasoning can lead to more effective multimodal reasoning systems. This approach could be applied in various domains requiring integration of diverse data types, such as robotics, natural language processing, and computer vision.
Generative Query Suggestion via Intent Coverage and Query-Level Credit Assignment
NLP
Reinforcement Learning
Generative Models
- Introduces an Intent-Driven Query Suggestion Framework with dual-stage optimization.
- Addresses intent-aware diversity modeling and query-level credit assignment challenges.
- Implements an Intent-Aware Diversity Reward to optimize intent coverage.
- Demonstrates significant improvements in user engagement metrics through extensive testing.
Read more
Generative Query Suggestion via Intent Coverage and Query-Level Credit Assignment
Summary
This paper presents a novel framework for generative query suggestion aimed at enhancing user engagement through better anticipation of user intents. The authors identify two main challenges in this domain: optimizing intent-aware diversity and implementing effective query-level credit assignment. To address these, they propose an Intent-Driven Query Suggestion Framework that employs dual-stage optimization. The first stage involves intent-aware diversity modeling to create intent-aligned supervised fine-tuning data, utilizing an Intent-Aware Diversity Reward (IAD-R) to ensure comprehensive intent coverage. The second stage focuses on query-level credit assignment, which allows for routing individual quality signals to specific query tokens while maintaining a shared slate-level diversity signal. The framework was validated through extensive experiments, including online A/B testing and offline evaluations, demonstrating significant improvements in click-through rates, query quality, and intent coverage. This work highlights the importance of balancing individual query relevance with overall slate diversity in conversational AI systems.
Methodology
The methodology consists of two main stages: a cold-start supervised fine-tuning stage that generates high-quality training data through user-initiative alignment and a reinforcement learning stage that optimizes intent-aware diversity and query-level credit assignment. The model formulates the query suggestion task as a conditional generation problem, aiming to optimize both individual query quality and overall slate diversity.
Results
The experiments conducted on a large-scale production dataset showed marked improvements in click-through rates, query quality, and intent coverage, validating the effectiveness of the proposed framework. The combination of query-specific token-level advantages and a shared slate-level intent-coverage advantage led to enhanced performance in generative query suggestion.
Implications
The findings suggest that optimizing for both individual query relevance and overall slate diversity can significantly enhance user engagement in conversational AI systems. This approach could be applied to various applications involving user interaction, such as search engines, virtual assistants, and recommendation systems.
PosteriorBench: From Point Estimates to Posterior Matching in Evaluating Generative Inverse Solvers
Generative Models
- PosteriorBench provides a comprehensive framework for evaluating generative inverse solvers beyond point estimates.
- The benchmark includes four diverse scientific inverse problems, enabling a broad assessment of solver performance.
- A five-metric evaluation suite allows for detailed analysis of distributional accuracy and uncertainty quantification.
- Neural operators demonstrate improved robustness in resolution, addressing gaps in current solver performance.
Read more
PosteriorBench: From Point Estimates to Posterior Matching in Evaluating Generative Inverse Solvers
Summary
The paper introduces PosteriorBench, a benchmark designed to evaluate the distributional accuracy of generative inverse solvers in scientific inverse problems. Traditional evaluations focus on point estimates, which are inadequate for ill-posed problems where multiple solutions can fit the same observations. PosteriorBench assesses four physics-based inverse problems: Darcy flow inversion, Poisson source recovery, carbon capture and storage, and light transport material inference. High-fidelity reference posteriors are constructed using computationally intensive methods like rejection sampling and Markov chain Monte Carlo. The benchmark employs a five-metric evaluation suite that includes posterior-mean error, posterior-standard-deviation error, maximum mean discrepancy, sliced Wasserstein distance, and radially averaged power-spectrum error to evaluate pointwise accuracy, marginal uncertainty, distributional alignment, and global frequency fidelity. The results indicate significant gaps in distribution matching among current solvers, while highlighting the effectiveness of neural operators in improving resolution robustness. The study emphasizes the importance of posterior variance calibration, suggesting that guidance weights and generation noise play critical roles in achieving accurate posterior representations.
Methodology
The authors constructed high-fidelity reference posteriors using computationally heavy methods such as rejection sampling and Markov chain Monte Carlo. They developed a five-metric evaluation suite to assess the performance of generative inverse solvers across various scientific tasks, focusing on distributional accuracy and uncertainty quantification.
Results
The experiments revealed substantial gaps in distribution matching across existing solvers. Function-space diffusion samplers were identified as strong posterior samplers, while traditional pointwise metrics were found to be misleading in terms of posterior variance and distributional alignment.
Implications
PosteriorBench has the potential to improve the evaluation of generative models in scientific computing, leading to better decision-making and interpretations in fields such as subsurface flow, optical imaging, and fluid dynamics. It encourages the development of more robust generative models that accurately capture posterior distributions.
Is It Still Worth Training a Classical Model in the Era of LLMs? A Crossover Benchmark on Tabular Data
Large Language Models
- Trained classical models outperform frozen LLMs in 86% of evaluated cases with minimal labeled data.
- The crossover point where classical models become preferable is around 6% of the training set.
- Few-shot prompting does not yield consistent performance improvements for LLMs.
- The performance of LLMs is sensitive to the semantics of input features.
Read more
Is It Still Worth Training a Classical Model in the Era of LLMs? A Crossover Benchmark on Tabular Data
Summary
This paper investigates the practical implications of using large language models (LLMs) for labeling tabular data compared to training classical machine learning models. The author introduces the concept of a labeled-data crossover point (N ⋆), which indicates the minimum training set size required for a classical model to outperform a frozen LLM in terms of prediction accuracy. Through a comprehensive evaluation involving 126 students assessing 18 tabular datasets under various prompting configurations, the study finds that trained classical models consistently outperform frozen LLMs, with a median crossover point at approximately 6% of the training set. The findings suggest that even with optimal prompting, classical models require only a small amount of labeled data to achieve better performance than LLMs. The paper also highlights the limitations of in-context few-shot prompting and the variability in LLM performance across different implementations. Overall, the results advocate for collecting a few hundred labels to train gradient-boosted models for typical business applications, rather than relying solely on LLMs.
Methodology
The study involved 126 independent evaluations of small GPT models across 18 tabular datasets, utilizing eight different prompting configurations. The results were paired with power-law learning curves for six classical model families to determine the crossover point where classical models outperform LLMs.
Results
The analysis revealed that trained classical models consistently beat frozen LLMs in 86% of cases, with a median crossover point at approximately 6% of the training set. Few-shot prompting was found to be ineffective, and LLM performance varied significantly across independent implementations.
Implications
The findings suggest that businesses should prioritize collecting a small number of labeled examples to train classical models, rather than relying on LLMs for tabular data predictions. This could lead to more efficient and cost-effective predictive modeling in various business applications.
Smart Insole Human Activity Recognition for Continuous Monitoring in Elderly Care
Robotics
Time Series
Multimodal
- Development of a smart insole platform for continuous monitoring of elderly mobility states.
- Integration of pressure sensors and IMU for comprehensive activity recognition.
- High performance of Histogram-Based Gradient Boosting in classifying mobility states.
- Demonstration of effective participant-independent validation for model robustness.
Read more
Smart Insole Human Activity Recognition for Continuous Monitoring in Elderly Care
Summary
This paper presents a novel wireless smart insole platform designed for human activity recognition (HAR) to facilitate continuous monitoring in elderly care. The system aims to detect critical mobility states such as sitting, standing, walking, and unstable walking, which are often precursors to falls in older adults. The smart insole integrates 16 active pressure-sensing locations and a six-dimensional inertial measurement unit (IMU) to capture plantar-pressure and inertial signals. Data were collected from 15 healthy adults at a frequency of 80 Hz and processed through a machine-learning workflow that included stratified 10-fold cross-validation for model selection and participant-independent 5-fold stratified group cross-validation for performance evaluation. The study found that the Histogram-Based Gradient Boosting (HGB) model achieved macro-F1 scores of 0.954 and 0.959 for the left and right feet, respectively, and 0.980 when using bilateral sensing. A compact 1D-CNN was also evaluated but did not significantly outperform HGB. The findings indicate that low-profile footwear sensing can effectively infer activity states from pressure and IMU measurements, providing a foundation for activity monitoring and fall prevention in elderly care settings.
Methodology
The study utilized a custom smart insole equipped with 16 pressure sensors and a six-dimensional IMU to collect data from 15 healthy adults. The data was segmented into overlapping windows and analyzed using a machine-learning workflow that included model selection through stratified 10-fold cross-validation and performance evaluation using participant-independent 5-fold stratified group cross-validation.
Results
The Histogram-Based Gradient Boosting model achieved macro-F1 scores of 0.954 for the left foot and 0.959 for the right foot, with a score of 0.980 when both feet were considered. The compact 1D-CNN did not significantly outperform the HGB model (p = 0.0625).
Implications
The findings suggest that smart insoles can serve as an effective tool for monitoring mobility states in elderly individuals, potentially leading to improved fall prevention strategies and enhanced safety in elder care environments. The unobtrusive nature of the insoles may increase adherence among users, especially those with cognitive impairments.
Block Parallelism For Efficient Distributed Long-Context Diffusion Language Model Training
NLP
Large Language Models
Efficient ML
- Introduction of Block Parallelism (BP) for BDLM training, reducing cross-rank communication.
- Development of Context-Sharded Block Parallelism (CSBP) to efficiently handle long-context training.
- Significant throughput improvements of 1.18–1.45× for supervised fine-tuning and 1.27–1.33× for autoregressive to BDLM conversion.
- CSBP accelerates speculative decoding training by up to 7.59× at 1M context length.
Read more
Block Parallelism For Efficient Distributed Long-Context Diffusion Language Model Training
Summary
This paper introduces Block Parallelism (BP) and Context-Sharded Block Parallelism (CSBP) as innovative techniques for training Block Diffusion Language Models (BDLMs) efficiently in a distributed manner. BDLMs combine autoregressive dependencies with parallel denoising, but their training is hindered by the limitations of conventional context parallelism (CP), which partitions sequences by token position and incurs high communication costs. The authors propose BP, which assigns each corrupted block computation to a single rank, thereby localizing corrupted K/V and gradients and eliminating unnecessary cross-rank communication. CSBP further enhances BP by sharding the shared clean sequence across ranks, allowing for efficient long-context training without replicating clean prefixes. The methodology results in significant throughput improvements across various models, demonstrating the scalability and efficiency of the proposed techniques. The paper reports substantial speedups in training times and improved performance metrics, showcasing the effectiveness of CSBP in both supervised fine-tuning and autoregressive model conversion.
Methodology
The authors propose a new distributed parallelism dimension called Block Parallelism (BP), which assigns each corrupted block computation to a single rank, thus localizing the necessary K/V and gradients. To scale this approach for long contexts, they introduce Context-Sharded Block Parallelism (CSBP), which shards the shared clean sequence across ranks while maintaining the locality of corrupted K/V. This dual approach minimizes communication overhead and maximizes computational efficiency.
Results
The implementation of CSBP on 16 H200 GPUs at 256K context length resulted in throughput improvements of 1.18–1.45× for supervised fine-tuning and 1.27–1.33× for converting autoregressive models to BDLMs. At 512K context, full-model speedup reached 1.61×. On eight H100 GPUs, CSBP accelerated DFlash2 speculative-decoder training by 2.48× at 512K and 7.59× at 1M context length. Additionally, CSBP achieved higher pass rates across all trained checkpoints in matched 12-hour runs.
Implications
The proposed techniques could significantly enhance the efficiency of training large language models, particularly in applications requiring long-context processing, such as coding assistants and interactive agents. The reduction in training time and resource utilization may facilitate broader adoption of BDLMs in real-world applications.
SETTer: Sparse-Encoder Transformer for Long-term Multivariate Time Series Forecasting
Time Series
- SETTer introduces decoupled self-attention and hybrid masking techniques for improved forecasting.
- The model effectively captures both short- and long-term dependencies in multivariate time series data.
- SETTer provides explainable structures to enhance interpretability of its predictions.
- The model outperforms existing state-of-the-art methods in 88% of scenarios tested.
Read more
SETTer: Sparse-Encoder Transformer for Long-term Multivariate Time Series Forecasting
Summary
The paper presents SETTer, a novel transformer-based model designed for long-term multivariate time series forecasting (LMTF). Traditional forecasting methods struggle with high-dimensional and complex relationships inherent in multivariate time series data. SETTer addresses these challenges by introducing techniques for decoupled self-attention and hybrid masking, which enhance the model's ability to capture both short- and long-term patterns across temporal and channel dimensions. The architecture is enriched with explainable structures that clarify the model's decision-making process. The authors demonstrate that SETTer, even with a single-layer transformer architecture, effectively models long-term dependencies amidst varying data complexities. Extensive experiments on benchmark datasets reveal that SETTer outperforms state-of-the-art models in 88% of tested scenarios, showcasing its robustness and effectiveness in practical applications.
Methodology
SETTer employs a transformer architecture enhanced with decoupled self-attention and hybrid masking techniques. These innovations allow the model to selectively focus on relevant temporal and channel dependencies, mitigating issues like attention oversmoothing and limited capacity commonly found in traditional transformer models. The architecture is designed to be interpretable, providing insights into the decision-making process.
Results
SETTer achieved superior performance compared to state-of-the-art models in 88% of the scenarios evaluated on real-world benchmark datasets for long-term multivariate time series forecasting. The model effectively captured complex patterns and dependencies, demonstrating its robustness in practical applications.
Implications
The advancements presented in SETTer have significant implications for various sectors reliant on accurate long-term forecasting, such as energy management, finance, and healthcare. By improving the accuracy and interpretability of forecasts, SETTer can enhance decision-making processes and operational efficiency in these critical areas.
Compressed Active Subspaces for Scalable Bayesian Inference
Efficient ML
Theory
Optimization
- Introduction of Compressed Active Subspaces (CAS) for scalable Bayesian inference.
- CAS reduces memory requirements for active subspace construction in high-dimensional models.
- Demonstrated scalability on neural networks while maintaining predictive performance.
- Enables robust uncertainty quantification in large models.
Read more
Compressed Active Subspaces for Scalable Bayesian Inference
Summary
The paper introduces Compressed Active Subspaces (CAS), a novel approach designed to enhance the scalability of Bayesian inference in high-dimensional models by addressing the limitations of traditional active subspace methods. Active subspaces are effective for quantifying predictive uncertainty by identifying influential parameter directions. However, they require substantial memory to store full-dimensional model gradients, which becomes impractical for large models. CAS mitigates this issue by employing a structured isometric embedding to map model parameters into a compressed space, allowing for the construction of active subspaces with significantly reduced memory requirements. The authors demonstrate the effectiveness of CAS on neural networks of increasing size, showing that it maintains predictive performance and robust uncertainty estimates while enabling scalable Bayesian inference. This work contributes to the field by providing a method that allows for efficient uncertainty quantification in large-scale models, making Bayesian inference more feasible in practical applications.
Methodology
The authors propose a method that first compresses the model parameter space using a structured isometric embedding. Active subspace construction is then performed within this compressed space, significantly reducing the memory needed to store sampled model gradients. The dominant eigenvectors of the covariance matrix of model gradients are approximated using truncated Singular Value Decomposition (SVD) on the compressed gradients.
Results
The CAS approach was tested on neural networks of varying sizes, demonstrating that it effectively reduces memory usage while preserving predictive accuracy and providing reliable uncertainty estimates. The results indicate that CAS can facilitate Bayesian inference in models that were previously too large for traditional active subspace methods.
Implications
The development of CAS has significant implications for the field of Bayesian deep learning, particularly in applications requiring uncertainty quantification in large models. It opens avenues for more efficient model training and inference in various domains, including scientific computing and engineering, where high-dimensional parameter spaces are common.