AI-generated summaries
Today's ML research,
without the noise.
Daily summaries of the latest machine learning papers from arXiv, processed every 8 hours.
45
Papers today
8h
Update frequency
7
Days of history
Fast and Accurate Anomaly Detection in Time Series
Time Series
- Introduces a novel unsupervised anomaly detection algorithm based on Haar Discrete Wavelet Transform.
- Addresses the challenges of class imbalance and high false positive rates in anomaly detection.
- Demonstrates superior performance over existing unsupervised and self-supervised methods across 343 datasets.
- Utilizes a rigorously derived t-test for assigning anomaly scores to observations.
Read more
Fast and Accurate Anomaly Detection in Time Series
Summary
This paper addresses the challenge of anomaly detection in time series data, particularly focusing on the limitations of existing supervised and unsupervised methods. Anomalies are often rare and can be difficult to label, leading to class imbalance issues in supervised learning. The authors propose a novel unsupervised algorithm called DWTt-test, which utilizes the Haar Discrete Wavelet Transform (DWT) to decompose time series data into different components. This allows the algorithm to effectively monitor both long-term trends and short-term variations. A specially designed t-test is applied to assign anomaly scores to observations. The paper presents extensive experiments across 343 datasets, demonstrating that the proposed method outperforms state-of-the-art unsupervised and self-supervised benchmarks in terms of accuracy and speed. The findings suggest that the DWTt-test can significantly enhance anomaly detection capabilities in various applications, including cybersecurity, finance, healthcare, and IoT systems.
Methodology
The proposed method, DWTt-test, employs the Haar Discrete Wavelet Transform to decompose univariate time series into coarse and detail components. A sliding window mechanism is used to apply a specially designed t-test, which assigns an anomaly score to each observation based on its statistical properties.
Results
The DWTt-test algorithm was tested on 343 datasets and showed significant improvements in both speed and accuracy compared to existing state-of-the-art unsupervised and self-supervised anomaly detection methods.
Implications
The findings of this research have important implications for real-world applications where timely and accurate anomaly detection is critical, such as in cybersecurity for breach detection, in finance for fraud detection, and in healthcare for identifying rare medical conditions.
An Optimisation Framework for the Well-Conditioned Training of Physics-Informed Neural Networks
Optimization
- Introduction of DSGNAR, a scalable second-order optimization framework for PINNs.
- Achieves unprecedented accuracy and speed in solving PDEs compared to state-of-the-art methods.
- Demonstrates significant improvements in relative ℓ2 errors across various complex problems.
- Robust performance regardless of architecture, arithmetic precision, and initial hyperparameters.
Read more
An Optimisation Framework for the Well-Conditioned Training of Physics-Informed Neural Networks
Summary
This paper addresses the challenges faced by Physics-Informed Neural Networks (PINNs) in solving partial differential equations (PDEs) with high precision. The authors introduce a novel optimization framework called DSGNAR (Doubly-Sketched Gauss–Newton with Adaptive Ratio) that effectively tackles the ill-conditioned loss landscape associated with PINNs. By combining a doubly-sketched Gauss–Newton model with an innovative strategy for controlling regularization and step length, DSGNAR achieves significant improvements in both accuracy and computational speed. The framework demonstrates remarkable performance across a variety of complex problems, including nonlinear, chaotic, multi-scale, high-dimensional scenarios, and the Navier–Stokes equations. The results show that DSGNAR can reduce relative ℓ2 errors to as low as 3 × 10−16 in double precision, outperforming existing methods by several orders of magnitude. Additionally, it exhibits robustness to different neural network architectures and initial hyperparameters, making it a versatile tool for scientific computing.
Methodology
The authors developed DSGNAR, which integrates a doubly-sketched Gauss–Newton approach with adaptive control of regularization and step length. This method addresses the ill-conditioning of the loss landscape in PINNs, allowing for efficient optimization and high-precision solutions.
Results
DSGNAR achieved relative ℓ2 errors as low as 3 × 10−16 in double precision, significantly improving results on the Burgers' equation by five orders of magnitude and on high-dimensional Poisson problems by eight orders. The framework also demonstrated rapid convergence, solving the Burgers' equation to an error of 4.75 × 10−7 in under ten seconds.
Implications
The advancements presented in this paper could lead to more accurate and efficient solutions for a wide range of scientific and engineering problems modeled by PDEs, enhancing the applicability of PINNs in fields such as fluid dynamics, biomedical engineering, and geophysics.
Multilayer Q-Matrix-Embedded Neural Network for Cognitive Diagnosis (M-QCDNet): Structure-Aware Deep Learning Architecture for Psychometric Interpretability
Interpretability
- M-QCDNet integrates Q-matrix structures into neural networks to enhance both predictive accuracy and interpretability.
- The model introduces new interpretability metrics to evaluate the alignment of latent skill representations with cognitive theory.
- M-QCDNet offers practical applications in educational settings for early detection of learning difficulties.
- The architecture maintains psychometric meaning while leveraging the representation-learning capabilities of deep neural networks.
Read more
Multilayer Q-Matrix-Embedded Neural Network for Cognitive Diagnosis (M-QCDNet): Structure-Aware Deep Learning Architecture for Psychometric Interpretability
Summary
This paper introduces the Multilayer Q-Matrix-Embedded Neural Network (M-QCDNet), a novel architecture designed for cognitive diagnosis that integrates the structural interpretability of cognitive diagnostic models (CDMs) with deep learning neural networks. M-QCDNet utilizes a Q-matrix to structure item-skill relationships, ensuring that latent mastery profiles are interpretable and consistent with cognitive theory. The model incorporates a unique loss function with an L2 penalty to align skills with the Q-matrix, balancing predictive performance with structural alignment. Additionally, the paper proposes new evaluation metrics, including the Q-Matrix Consistency Ratio (QCR), Cross-Loading Ratio (CLR), and Off-Support Activation (OSA), to assess the alignment of predicted skill activations with item-level skills. M-QCDNet enhances classroom practices by facilitating early detection of learning difficulties and supporting mastery-based interventions. By embedding diagnostic validity into its design, M-QCDNet bridges the gap between psychometric transparency and neural flexibility, contributing to the development of interpretable and actionable AI in cognitive diagnostics.
Methodology
The methodology involves embedding a multilayer Q-matrix structure within a neural network, utilizing a specially designed loss function to penalize misaligned skills, and developing new evaluation metrics to assess the interpretability and alignment of the model's outputs with cognitive structures.
Results
M-QCDNet shows improved performance in capturing latent skill structures compared to previous neural cognitive diagnostic models, with enhanced interpretability and alignment with cognitive theory as measured by the proposed metrics.
Implications
The findings suggest that M-QCDNet can significantly improve cognitive diagnostic assessments in educational contexts, enabling more effective interventions and personalized learning experiences based on accurate skill mastery profiles.
Revisiting Decentralized Online Convex Optimization with Compressed Communication
Optimization
Theory
Efficient ML
- Introduction of two FTRL-type algorithms for D-OCO with compressed communication.
- First algorithm matches existing regret bounds in a full-information setting.
- Second algorithm improves regret bounds and communication costs in a bandit setting.
- Simplified theoretical analysis compared to existing methods.
Read more
Revisiting Decentralized Online Convex Optimization with Compressed Communication
Summary
This paper addresses the decentralized online convex optimization (D-OCO) problem, focusing on the challenge of communication bottlenecks in distributed applications with streaming data. Previous research has primarily explored D-OCO using compressed communication with online gradient descent (OGD) variants, while the best existing algorithms for exact communication are based on follow-the-regularized-leader (FTRL). The authors introduce two novel FTRL-type algorithms for D-OCO that utilize compressed communication for the first time. The first algorithm operates in a full-information setting and achieves regret bounds comparable to existing methods, while the second algorithm is tailored for a bandit setting, significantly improving both regret bounds and communication costs. The key insight is the dual update mechanism of FTRL, which simplifies the application of consensus techniques under communication constraints. The proposed algorithms not only match existing performance metrics but also offer a more elegant theoretical framework and reduced complexity in their design and analysis.
Methodology
The authors developed two FTRL-type algorithms for D-OCO that leverage compressed communication. The first algorithm is designed for full-information settings, while the second is adapted for bandit scenarios. Both algorithms utilize a dual update mechanism to facilitate average consensus with communication compression, simplifying the overall design and analysis.
Results
The first algorithm achieves regret bounds that are on par with existing algorithms in the full-information setting. The second algorithm significantly enhances the regret bounds to O(nT^3/4) and O(nT^2/3(log T)^{1/3}) for convex and strongly convex functions, respectively, while reducing the required communication rounds compared to previous methods.
Implications
The proposed algorithms can be applied in various distributed optimization scenarios, particularly in environments where communication bandwidth is limited. Their simplicity and efficiency make them suitable for real-time applications in decentralized networks, such as federated learning and collaborative filtering.
Black-Box Inference of LLM Architectural Properties with Restrictive API Access
Large Language Models
NLP
Theory
- Introduces NightVision, an attack method for inferring LLM architectural properties under restrictive API access.
- Demonstrates that hidden dimensions, depth, and parameter counts can still be recovered despite limited API information.
- Achieves an average relative error of 23% for hidden dimensions and 53% for depth and parameter counts on large models.
- Highlights the inadequacy of current API restrictions in safeguarding LLM architectural details.
Read more
Black-Box Inference of LLM Architectural Properties with Restrictive API Access
Summary
This paper addresses the challenge of inferring architectural properties of large language models (LLMs) when access to their APIs is restricted. Previous research demonstrated that certain architectural details could be recovered with limited API access, such as top-k logits. In response, LLM providers have tightened API restrictions, limiting access to single logits and removing logit bias functions. The authors introduce 'NightVision', a novel attack method that allows for the estimation of hidden dimensions, depth, and parameter counts of LLMs even under these restrictive conditions. NightVision employs a common set prompting technique to expose log probabilities for the same output tokens, which, combined with spectral analysis, enables the recovery of hidden dimensions. Additionally, it uses timing measurements to estimate depth and parameter count. The method was empirically evaluated on 32 open-source LLMs, achieving an average relative error of 23% for hidden dimensions and 53% for depth and parameter counts in larger models. The findings suggest that current API restrictions do not fully protect sensitive architectural information, indicating a need for enhanced security measures.
Methodology
The authors developed NightVision, which consists of two main components: a common set prompting technique that allows for hidden dimension recovery using single-logit access, and a timing-based recovery procedure that estimates depth and parameter count based on the scaling of inference time with respect to model architecture. The methodology includes empirical evaluations across various open-source LLMs to validate the effectiveness of the approach.
Results
NightVision successfully recovered hidden dimensions with an average relative error of 23% across 32 models, achieving exact recovery in 4 cases and within 10% in 12 cases. For models with over three billion parameters, the method estimated depth and parameter counts with an average relative error of approximately 53%. The accuracy of these estimates was found to depend on the token budget provided to the algorithm.
Implications
The results imply that even with restrictive API access, sensitive architectural properties of LLMs can be inferred, raising concerns about the security of proprietary model information. This has implications for API design and model auditing, suggesting that additional measures, such as defending against timing side channels, may be necessary to protect intellectual property in LLMs.
Spin-Weighted Spherical Harmonics Enable Complete and Scalable E(3)-Equivariant Networks
Theory
Efficient ML
Graph Learning
- Introduction of SpinGTP, which generalizes from scalar functions to Spin-Weighted Spherical Harmonics.
- SpinGTP overcomes the expressivity limitations of the Gaunt Tensor Product by recovering antisymmetric interactions.
- The method maintains the computational efficiency of GTP while providing a more expressive equivariant basis.
- Evaluation shows SpinGTP achieves comparable accuracy to full CGTP and excels in chiral and non-centrosymmetric tasks.
Read more
Spin-Weighted Spherical Harmonics Enable Complete and Scalable E(3)-Equivariant Networks
Summary
This paper addresses the limitations of E(3)-equivariant networks in modeling 3D atomistic systems, particularly the scalability issues arising from the high computational complexity of the Clebsch-Gordan Tensor Product (CGTP). The authors introduce a novel approach called SpinGTP, which utilizes Spin-Weighted Spherical Harmonics (SWSH) to enhance expressivity while maintaining computational efficiency. By generalizing from scalar functions to SWSH, SpinGTP effectively recovers missing antisymmetric interactions that are crucial for tasks involving chiral materials and non-centrosymmetric geometries. The paper presents a comprehensive evaluation of SpinGTP across various benchmarks, demonstrating that it achieves accuracies comparable to full CGTP while explicitly capturing antisymmetric paths. This advancement provides a mathematically rigorous and scalable solution for high-order equivariance in large-scale 3D atomistic simulations, paving the way for improved modeling of complex molecular interactions.
Methodology
The authors developed SpinGTP by leveraging the algebraic properties of Spin-Weighted Spherical Harmonics (SWSH) to recover antisymmetric paths in tensor products. They implemented this operator in a real, parity-labeled SWSH basis, allowing for efficient computation while ensuring expressivity in modeling interactions in 3D atomistic systems.
Results
SpinGTP was evaluated on several benchmarks, including Tetris, 3BPA, SPICE-MACE-OFF, and OC20, demonstrating that it achieves accuracies comparable to the full Clebsch-Gordan Tensor Product. The results indicate superior performance in tasks involving chiral materials and non-centrosymmetric geometries, validating the effectiveness of the proposed method.
Implications
The introduction of SpinGTP has significant implications for the field of computational materials science and molecular modeling, enabling more accurate simulations of complex atomic interactions. This work could lead to advancements in the design of materials and molecules with specific chiral properties, enhancing the capabilities of AI in scientific research.
Predicting Closed-Loop Performance of Latent World Models: Offline Checkpoint Selection for MPC and Model-Based RL Under Non-Markovian Rewards in LunarLander
Reinforcement Learning
Robotics
Optimization
- Introduces a suite of 40 structural validation-time metrics for evaluating world models.
- Proposes the Composite Reward Observability Fraction (CROF) for offline checkpoint selection.
- Demonstrates that CROF effectively predicts closed-loop performance in non-Markovian reward settings.
- Shows significant improvements in data efficiency and performance over traditional model-free approaches.
Read more
Predicting Closed-Loop Performance of Latent World Models: Offline Checkpoint Selection for MPC and Model-Based RL Under Non-Markovian Rewards in LunarLander
Summary
This paper addresses the challenge of predicting the closed-loop performance of learned latent world models in reinforcement learning (RL), particularly in environments with non-Markovian rewards, using the LunarLander-v3 as a case study. The author highlights the difficulty in selecting the appropriate checkpoint from a training run, as traditional metrics like validation loss and multi-step prediction RMSE may not correlate with actual closed-loop performance. To tackle this, the paper introduces a suite of 40 structural validation-time diagnostics based on optimal control theory, which are applied to evaluate the performance of a Recurrent State Space Model (RSSM) trained on LunarLander-v3. The study identifies the Reward Observability Fraction (ROF) as the most effective single predictor of closed-loop quality. By combining ROF with additional structural regularizers, the author proposes the Composite Reward Observability Fraction (CROF) as a comprehensive checkpoint-selection score. The results demonstrate that the CROF-selected model significantly outperforms a model-free A2C baseline, achieving approximately 24.5 return points higher while requiring about 65 times fewer real-environment interactions. This work emphasizes the importance of structural properties in learned dynamics for effective planning and policy optimization in model-based RL.
Methodology
The methodology involves training an RSSM world model on the LunarLander-v3 environment and evaluating its performance using a set of 40 structural validation-time metrics derived from classical control theory. The key metric, CROF, combines the ROF with additional structural scores to select the best checkpoints for closed-loop performance.
Results
The CROF-selected world model was shown to train a model-based A2C policy that outperformed a model-free A2C baseline by approximately 24.5 return points while utilizing about 65 times fewer real-environment interactions. The study also demonstrated that CROF correlates well with zero-shot CEM-MPC returns, indicating its effectiveness in checkpoint selection.
Implications
The findings suggest that using structural validation-time diagnostics can enhance the selection of world models in model-based RL, leading to more efficient training and better performance in environments with complex reward structures. This approach could be applied to various RL tasks, improving the applicability of model-based methods in real-world scenarios.
Privacy-Preserving and Verifiable Approximate Distributed Coded Computing
Federated Learning
Theory
Efficient ML
- Introduction of a unified framework for adversary-resilient distributed learning that addresses privacy and malicious behavior in both federated and decentralized settings.
- Integration of GPBACC with robust aggregation strategies for federated learning to enhance privacy and security.
- Application of approximate decode-and-compare and group testing techniques for decentralized learning to enable verification without a trusted aggregator.
- Empirical evaluation of the framework through practical attack scenarios, demonstrating significant improvements in privacy preservation and adversary resilience.
Read more
Privacy-Preserving and Verifiable Approximate Distributed Coded Computing
Summary
This paper addresses the challenges of privacy leakage and malicious manipulation in distributed machine learning, particularly in federated and decentralized learning paradigms. The authors propose a unified, model-agnostic framework that integrates privacy-preserving techniques with adversary-resistant strategies. The framework employs Generalized Privacy-aware Berrut Approximated Coded Computing (GPBACC) as a core privacy-enhancing technology, which is applicable to various machine learning models. For federated learning, robust aggregation strategies are introduced to counteract the effects of malicious participants, while decentralized learning utilizes approximate decode-and-compare methods and group testing techniques for lightweight verification and adversary isolation. The authors conduct an explicit, attack-driven evaluation of their framework, demonstrating its effectiveness against various privacy attacks and malicious behaviors. The results indicate that the combination of GPBACC with tailored adversary-resistance strategies significantly reduces privacy leakage and enhances resilience against active adversaries, providing a practical foundation for secure distributed machine learning.
Methodology
The authors developed a unified framework that combines GPBACC with specific defense mechanisms tailored for federated and decentralized learning. They implemented robust aggregation strategies for federated settings and approximate decode-and-compare along with group testing for decentralized environments. The framework was evaluated through an attack-driven analysis, where various privacy attacks and malicious behaviors were simulated to assess the effectiveness of the proposed solutions.
Results
The evaluation showed that the proposed framework significantly reduces privacy leakage and improves resilience against active adversaries. The combination of GPBACC with the specific adversary-resistance strategies demonstrated a marked enhancement in both privacy preservation and security across the tested scenarios.
Implications
The findings suggest that the proposed framework can serve as a practical solution for secure distributed machine learning applications in sensitive domains such as healthcare and finance, where privacy and data integrity are paramount. The model-agnostic nature of the techniques also allows for their application across diverse machine learning architectures, facilitating broader adoption in real-world scenarios.
From Approximation to Emergence: A Theory of Deep Learning
Theory
Optimization
Generative Models
- Synthesizes existing theoretical frameworks in deep learning into a coherent narrative.
- Explores the implications of overparameterization and emergent behaviors in modern neural networks.
- Clarifies the assumptions and limitations of various theoretical explanations in deep learning.
- Focuses on approximation, optimization, and generalization while addressing additional complexities.
Read more
From Approximation to Emergence: A Theory of Deep Learning
Summary
This monograph presents a comprehensive theoretical framework for understanding deep learning, focusing on the classical triad of approximation, optimization, and generalization. It expands the discussion to include modern phenomena such as overparameterization, foundation models, and emergent behaviors in deep learning systems. The author synthesizes existing mathematical results into a coherent narrative, clarifying the assumptions that underpin various theoretical explanations. The book emphasizes the interconnectedness of different theoretical routes, revealing genuine mechanisms while highlighting the limitations of current theories. It is designed for researchers and practitioners with a solid mathematical background, aiming to provide a rigorous map of deep learning theory rather than a traditional instructional manual.
Methodology
The author employs a synthetic and analytical approach, reorganizing existing literature around common themes and providing proof-oriented presentations of key results. The manuscript discusses approximation theory, optimization theory, and generalization theory, while also addressing additional topics such as robustness, interpretability, and scaling laws.
Results
The book does not claim original discoveries but rather presents a structured overview of existing theories, highlighting their interconnections and the assumptions that lead to different conclusions. It provides insights into how deep learning models can generalize beyond training data and the mechanisms that underlie their performance.
Implications
This work has significant implications for the theoretical understanding of deep learning, potentially guiding future research directions and informing the development of more robust and interpretable models. It serves as a foundational reference for scholars and practitioners aiming to navigate the complexities of deep learning theory.
SABER: A Semantic-Aligned Brain Network Analysis Framework via Multi-scale Hypergraphs
Graph Learning
Large Language Models
Interpretability
- SABER integrates LLM-derived semantics directly into brain network classification, enhancing decision-making.
- The framework employs multi-scale hypergraphs to capture high-order interactions among brain regions.
- A decision-level semantic alignment mechanism allows for patient-specific semantic information to guide predictions.
- SABER outperforms existing methods in terms of performance and interpretability on benchmark datasets.
Read more
SABER: A Semantic-Aligned Brain Network Analysis Framework via Multi-scale Hypergraphs
Summary
The paper presents SABER, a novel framework for brain network analysis that integrates high-level semantic knowledge from large language models (LLMs) into the decision-making process for diagnosing brain diseases. Traditional methods often treat semantics as auxiliary features, limiting their effectiveness. SABER addresses this by incorporating ROI-level semantics through global self-attention, enriching node representations with whole-brain context. It constructs multi-scale hypergraphs to model functional subnetworks and multi-ROI interactions, overcoming the limitations of conventional graph neural networks (GNNs). A decision-level semantic alignment mechanism is introduced, allowing patient-specific textual embeddings to directly influence predictions without altering the underlying network structure. The framework is evaluated on public datasets (ABIDE and ADHD-200), demonstrating superior performance, stability, and interpretability, particularly in small-sample scenarios.
Methodology
SABER consists of three main stages: (1) Multi-scale node-level encoding, where ROI semantics are injected into node features using global self-attention; (2) Construction of multi-scale hypergraphs to model complex interactions beyond pairwise connectivity; and (3) A graph-level semantic alignment mechanism that integrates patient-specific textual embeddings into the graph representation, influencing predictions directly.
Results
The experiments on the ABIDE and ADHD-200 datasets show that SABER consistently achieves state-of-the-art performance, with improved stability and interpretability, especially in scenarios with limited data.
Implications
The framework has significant implications for clinical diagnostics, particularly in enhancing the interpretability and robustness of brain disease classification models. By leveraging high-level semantics, it can provide deeper insights into brain connectivity and functional roles, potentially aiding in the understanding of complex neurodevelopmental disorders.
Bayesian Sparse Low-Rank Adaptation for Large Language Model Uncertainty Estimation
NLP
Large Language Models
Efficient ML
- DALorRA shifts uncertainty quantification from dense parameter spaces to low-rank adaptation levels, minimizing computational overhead.
- The framework integrates variational Bayesian estimation with ensemble-like inference, enhancing uncertainty quantification.
- Extensive empirical validation shows DALorRA achieves excellent calibration and reasoning accuracy across diverse benchmarks.
- The method dynamically adjusts the LoRA rank, addressing the limitations of fixed rank adaptations in low-data scenarios.
Read more
Bayesian Sparse Low-Rank Adaptation for Large Language Model Uncertainty Estimation
Summary
This paper addresses the issue of overconfidence in large language models (LLMs) during task-specific fine-tuning, which can hinder their reliable deployment. The authors propose a novel framework called Data-Adaptive Lower-Rank Adaptation (DALorRA), which utilizes a variational Bayesian approach to shift uncertainty quantification from the dense parameter space to the low-rank adaptation (LoRA) level. By introducing a stochastic diagonal mask matrix, DALorRA dynamically prunes unnecessary rank components during training, allowing for Bayesian regularization and ensemble-like calibration during inference. The methodology captures discrete structural uncertainty and effectively reduces model complexity, leading to improved calibration without sacrificing reasoning accuracy. Extensive experiments demonstrate that DALorRA outperforms existing methods in uncertainty quantification and maintains high predictive performance across various reasoning benchmarks.
Methodology
The authors introduce DALorRA, which employs a stochastic diagonal mask matrix to model rank-level uncertainty in LoRA. This approach allows for the dynamic adjustment of rank components based on task-specific needs, utilizing variational inference to learn the posterior distribution over the mask, thereby capturing structural uncertainty and reducing unnecessary model complexity.
Results
The experiments conducted demonstrate that DALorRA consistently achieves superior calibration of LLMs compared to existing methods, without compromising reasoning accuracy. The results indicate that the framework effectively mitigates overconfidence in predictions, particularly in low-data regimes.
Implications
The proposed DALorRA framework has significant implications for the deployment of LLMs in real-world applications, where reliable uncertainty estimation is crucial. By improving model calibration and reducing overconfidence, DALorRA enhances the interpretability and trustworthiness of LLM predictions, making them more suitable for sensitive tasks.
SA-HGNN: Sample-Adaptive Hyperbolic Graph Neural Network for EEG-Based Depression Recognition
Graph Learning
- SA-HGNN introduces a dynamic graph construction method tailored to individual brain networks.
- Utilizes hyperbolic geometry to better capture hierarchical relationships in brain connectivity.
- Incorporates an attention mechanism to reduce noise in EEG signals.
- Demonstrates superior performance compared to traditional GNNs in EEG-based depression recognition.
Read more
SA-HGNN: Sample-Adaptive Hyperbolic Graph Neural Network for EEG-Based Depression Recognition
Summary
This paper introduces the Sample-Adaptive Hyperbolic Graph Neural Network (SA-HGNN), a novel approach for recognizing Major Depressive Disorder (MDD) through electroencephalography (EEG) data. Traditional Graph Neural Networks (GNNs) struggle to accurately model the hierarchical structure of brain networks affected by depression due to their reliance on Euclidean space. The SA-HGNN addresses this limitation by employing a hyperbolic space metric, which is better suited for capturing the complex hierarchical relationships inherent in brain connectivity. The model consists of three main components: a Sample-Adaptive Graph Construction module that dynamically builds personalized brain network topologies, a Hyperbolic Graph Convolution module that leverages hyperbolic geometry to enhance representation capabilities, and an Attention Pooling module that filters out noise from EEG signals. Extensive experiments conducted on public EEG datasets demonstrate that SA-HGNN outperforms traditional GNNs, showcasing its robustness and effectiveness in identifying abnormal functional connectivity patterns associated with MDD.
Methodology
The SA-HGNN framework consists of three core modules: (1) Sample-Adaptive Graph Construction (SAGC) for personalized topology creation, (2) Hyperbolic Graph Convolution (HGC) to leverage hyperbolic space for improved representation of hierarchical structures, and (3) Attention Pooling (AP) to filter out redundant noise from EEG signals, enhancing the model's ability to capture genuine topological features.
Results
The experimental results indicate that SA-HGNN significantly outperforms existing GNN methods based on Euclidean metrics across various EEG datasets, demonstrating its effectiveness in both resting-state and task-related paradigms. The model's robustness to noise and its capability to accurately capture abnormal functional connectivity patterns in MDD patients were particularly highlighted.
Implications
The findings suggest that SA-HGNN could serve as a powerful tool for automated diagnosis of Major Depressive Disorder, potentially leading to earlier and more accurate detection of the condition. This approach may also be applicable to other mental health disorders where brain connectivity plays a crucial role.
I2RiMA: Spectral Riemannian Representation with Temporal Attention for Mental Stress Detection based on EEG Signals
Time Series
- I2RiMA constructs frequency-specific covariance matrices and maps them to the SPD tangent space.
- The model employs frequency cluster aggregation for effective feature selection and redundancy reduction.
- An intra-inter slice attention module captures both local and global temporal dependencies in EEG data.
- I2RiMA achieves state-of-the-art performance in cross-subject EEG stress detection.
Read more
I2RiMA: Spectral Riemannian Representation with Temporal Attention for Mental Stress Detection based on EEG Signals
Summary
The paper presents I2RiMA, an Intra-Inter Riemannian Manifold Attention Network designed for cross-subject mental stress detection using EEG signals. Traditional methods struggle with subject-dependent and frequency-specific stress patterns, often relying on time-domain covariance matrices that overlook critical neural oscillations. I2RiMA addresses these limitations by constructing spatial covariance matrices at each frequency point and mapping them to the SPD tangent space, preserving channel-wise geometry and frequency-specific cues. The model incorporates frequency cluster aggregation to select informative spectral components and reduce redundancy, forming compact frequency clusters aligned with EEG rhythms. Additionally, an intra-inter slice attention module is introduced to integrate local slice-level dynamics and global temporal context across EEG sequences. Experimental results demonstrate that I2RiMA outperforms five state-of-the-art baselines, achieving balanced accuracies of 77.59%, 75.88%, and 82.78% on three datasets while maintaining efficiency with only 1.60M parameters and 31.95M FLOPs.
Methodology
I2RiMA utilizes a novel architecture that constructs spatial covariance matrices at each frequency point, preserving the Riemannian geometry of EEG data. It employs frequency cluster aggregation to enhance feature selection and reduce redundancy, while the intra-inter slice attention module allows for the integration of local and global temporal information from EEG sequences.
Results
I2RiMA achieved balanced accuracies of 77.59% on MIST Control, 75.88% on MIST Stress, and 82.78% on SEED datasets, outperforming five state-of-the-art methods. The model's efficiency is notable, with only 1.60M parameters and 31.95M FLOPs.
Implications
The findings suggest that I2RiMA could significantly enhance real-time mental stress detection systems, particularly in non-clinical settings, by providing a robust and efficient method for analyzing EEG signals. This could lead to better mental health monitoring and interventions.
Domain Knowledge Based Temporal-Spatial Graph Convolution Network for ECG Recognition
Graph Learning
Time Series
- Introduces a domain knowledge-based graph structure for ECG recognition.
- Utilizes a double-stream directed graph model to capture both spatial and temporal information.
- Achieves an average F1 score of 88.1% on the ECG classification task.
- Demonstrates improved performance in detecting rare ECG categories.
Read more
Domain Knowledge Based Temporal-Spatial Graph Convolution Network for ECG Recognition
Summary
This paper addresses the challenges of interpretability in AI models, particularly in healthcare applications such as electrocardiogram (ECG) recognition. The authors propose a novel approach that integrates domain knowledge into a graph convolutional network (GCN) for ECG classification. By incorporating key landmark points of the ECG signal (P, Q, R, S, T) as domain knowledge, the model utilizes a double-stream directed graph to capture both intra-cycle and inter-cycle relationships. Spatial directed graphs (SDGs) are used to model the positional relationships among key points, while temporal directed graphs (TDGs) capture the temporal dependencies across ECG cycles. The proposed model was validated using the First Chinese ECG Intelligent Competition dataset, achieving an overall average F1 score of 88.1% and a score of 76.3% for rare categories, outperforming existing state-of-the-art models. The findings suggest that integrating domain knowledge significantly enhances detection performance, particularly for rare ECG abnormalities.
Methodology
The authors developed a domain knowledge-based graph structure that represents key landmarks in ECG signals. They employed a double-stream directed graph model, consisting of spatial directed graphs for intra-cycle relationships and temporal directed graphs for inter-cycle dependencies. This approach allows for effective modeling of the ECG's rhythm and morphology.
Results
The proposed model achieved an overall average F1 score of 88.1% on the First Chinese ECG Intelligent Competition dataset, with a notable average F1 score of 76.3% for rare ECG categories, indicating superior performance compared to existing models.
Implications
The integration of domain knowledge into machine learning models can enhance diagnostic performance in healthcare applications, particularly for rare conditions. This approach could lead to more accurate and interpretable AI systems in medical diagnostics.
Certified World Models as Sensing Clocks: Drift-Aware Deadlines for Active Perception
Robotics
Reinforcement Learning
Theory
- Introduction of a certified sensing clock that indicates when to re-sense based on prediction validity.
- Drift-aware deployment method improves the accuracy of sensing deadlines compared to traditional Lyapunov rates.
- Demonstrated effectiveness on a frozen 3D VN-JEPA model, controlling certificate violations across multiple trials.
- Significant reduction in eventful-tail violations compared to existing reactive scheduling methods.
Read more
Certified World Models as Sensing Clocks: Drift-Aware Deadlines for Active Perception
Summary
This paper introduces a novel framework for active perception in agents using certified world models that estimate the validity horizon of their predictions. The author proposes a proactive sensing clock that dictates when an agent should stop relying on its predictions and re-sense the environment. This clock is derived from an audited equivariant world model and is designed to be drift-aware, addressing the limitations of traditional methods that either overestimate or underestimate the validity of predictions. The paper demonstrates that using calibrated native rollout-drift envelopes provides a more accurate deadline for re-sensing compared to on-manifold Lyapunov rates. The proposed method is instantiated on a frozen 3D VN-JEPA model, showing that the clock effectively controls interval-simultaneous certificate violations across various seeds and data shards. Additionally, in a synthetic benchmark, the certified clock significantly reduces eventful-tail violations compared to existing scheduling methods while maintaining a matched sensing budget. The author also clarifies that while the proposed method shows promise, it does not claim empirical superiority over all non-spectral schedulers, as certain conditions yield comparable performance. Overall, the paper contributes a new primitive for proactive sensing in active perception tasks, enhancing the reliability of predictions in dynamic environments.
Methodology
The methodology involves deriving a proactive sensing clock from an audited equivariant world model, utilizing calibrated native rollout-drift envelopes to establish drift-aware deadlines for re-sensing. The performance is evaluated through both real-world model instantiation and synthetic benchmarks, comparing the proposed method against traditional scheduling techniques.
Results
The results indicate that the certified sensing clock effectively controls interval-simultaneous certificate violations, reducing eventful-tail violations from 0.36 to 0.16 at matched sensing budgets. The clock remains valid across different deployment distributions, demonstrating its robustness and reliability in active perception tasks.
Implications
The findings suggest that integrating certified sensing clocks into active perception systems can enhance the efficiency and reliability of agents operating in dynamic environments. This approach could be particularly beneficial in robotics and autonomous systems where timely and accurate sensing is critical.
One More Time: Revisiting Neural Quantum States from a Reinforcement Learning Perspective
Reinforcement Learning
Optimization
Theory
- Introduces Proximal Wavefunction Optimization (PWO) for training Neural Quantum States.
- Establishes a formal connection between variational energy minimization and policy-gradient reinforcement learning.
- Demonstrates improved stability and convergence speed of NQS training compared to existing methods.
- Fine-tunes a large-scale RWKV-7 model, showcasing the scalability of the proposed method.
Read more
One More Time: Revisiting Neural Quantum States from a Reinforcement Learning Perspective
Summary
This paper explores the optimization of Neural Quantum States (NQS), a framework for approximating quantum many-body wavefunctions, through a reinforcement learning (RL) lens. The authors identify that while autoregressive models in NQS allow for efficient sampling from the Born distribution, their optimization remains a challenge. Existing methods like Adam and stochastic reconfiguration (SR) have limitations in terms of stability and computational efficiency. The authors propose a novel optimization algorithm called Proximal Wavefunction Optimization (PWO), which leverages trust-region methods inspired by Proximal Policy Optimization (PPO) in RL. PWO enhances the stability and convergence speed of NQS training, particularly for large models. The paper demonstrates the effectiveness of PWO on various benchmarks, including Ising and frustrated J1–J2 spin systems, and showcases its scalability by fine-tuning a 1.5 billion parameter RWKV-7 model. This work bridges the gap between variational optimization and reinforcement learning, providing a new perspective on training NQS.
Methodology
The authors present Proximal Wavefunction Optimization (PWO), a trust-region optimization algorithm that clips probability-ratio changes and phase increments to ensure stability. PWO allows for sample reuse across multiple updates and avoids explicit matrix inversion, combining the benefits of first-order optimization with theoretical guarantees. The optimization process is framed as an advantage policy-gradient problem, linking it to reinforcement learning principles.
Results
PWO significantly improves the stability and convergence speed of autoregressive NQS compared to Adam and minSR on standard benchmarks. The method was successfully applied to fine-tune a 1.5 billion parameter RWKV-7 model, achieving optimization at a scale previously unattainable in NQS research.
Implications
The findings suggest that integrating reinforcement learning principles into the optimization of neural quantum states can lead to more efficient and stable training methods. This could enhance the applicability of NQS in quantum physics and related fields, potentially enabling more accurate simulations of quantum systems.
Program-as-Weights: A Programming Paradigm for Fuzzy Functions
NLP
Large Language Models
Efficient ML
- PAW allows for the compilation of fuzzy functions from natural language specifications into efficient neural artifacts.
- The framework significantly reduces memory usage and increases execution speed compared to direct prompting of larger models.
- FuzzyBench, a dataset of 10 million examples, is released to support the training of the neural compiler.
- PAW demonstrates versatility across multiple applications, showcasing its ability to handle various fuzzy tasks.
Read more
Program-as-Weights: A Programming Paradigm for Fuzzy Functions
Summary
The paper introduces a novel programming paradigm called Program-as-Weights (PAW) aimed at addressing the limitations of traditional rule-based programming for fuzzy functions, which are tasks that resist precise specification. The authors propose a three-step process where developers describe a function in natural language, a neural compiler converts this description into a compact neural binary, and a lightweight interpreter executes the binary locally. The PAW framework utilizes a 4B compiler trained on a newly released dataset, FuzzyBench, which contains 10 million examples of fuzzy tasks. The resulting system, using a 0.6B Qwen3 interpreter, demonstrates performance comparable to a larger model (Qwen3-32B) while being significantly more memory-efficient. The authors showcase the versatility of PAW through various applications, including log monitoring, intent-based classification, and semantic search reranking, highlighting its potential to enable local execution of fuzzy functions without reliance on large language model APIs.
Methodology
The methodology involves a two-stage compile pipeline using a 4B Qwen3 model. The first stage is a pseudo-compiler that transforms the natural language specification into a clean pseudo-program. The second stage is a trained LoRA compiler that generates a parameter-efficient module tailored for the specific task. This compiled program is then executed by a lightweight interpreter, allowing for efficient local execution.
Results
The PAW framework achieves a performance of 73.78% exact match on fuzzy tasks using a 0.6B Qwen3 interpreter, outperforming the direct prompting of a larger 32B model (68.70% exact match) while consuming only one-fiftieth of the inference memory. The system runs at 30 tokens per second on a MacBook M3, demonstrating its efficiency and practicality.
Implications
The implications of this work suggest a paradigm shift in how developers approach fuzzy programming tasks, moving away from reliance on large language model APIs towards more efficient, local solutions. This could lead to improved software reproducibility, reduced operational costs, and enhanced performance in real-world applications.
Wind-Aware Reinforcement Learning Control of a Small Quadrotor Using Learned Onboard Wind Estimation in Simulated Atmospheric Turbulence
Reinforcement Learning
Robotics
- Introduces a two-stage learning pipeline for wind estimation and control in quadrotors.
- Achieves high accuracy in wind estimation with a GRU network trained on simulated turbulence.
- Demonstrates significant improvement in trajectory tracking using a PPO controller informed by wind estimates.
- Highlights the regime-dependent value of wind perception in control performance.
Read more
Wind-Aware Reinforcement Learning Control of a Small Quadrotor Using Learned Onboard Wind Estimation in Simulated Atmospheric Turbulence
Summary
This paper addresses the challenges faced by small multirotor aircraft operating in turbulent wind conditions, which can significantly degrade trajectory tracking and control performance. The authors propose a two-stage learning pipeline that first estimates local wind conditions using onboard kinematics and dynamics, and then utilizes this estimate in a reinforcement learning (RL) flight controller. The wind estimation is performed using an attention-augmented gated recurrent unit (GRU) network, trained on thousands of simulated flights through von Kármán turbulence. The wind estimator achieves a root-mean-square error (RMSE) of 0.40 m/s and a direction error of 3.2° on unseen wind regimes. The second stage employs a proximal policy optimization (PPO) controller that leverages the wind estimates, resulting in a 48% reduction in horizontal trajectory tracking error compared to a traditional wind-blind proportional-derivative (PD) controller. The study also includes an ablation analysis that distinguishes between the contributions of kinematic learning and wind perception, demonstrating that the value of wind perception increases with wind speed. The proposed method shows resilience in out-of-distribution wind conditions, where traditional methods fail, indicating its potential for enhancing the autonomy of small uncrewed aircraft systems (sUAS) in challenging environments.
Methodology
The methodology consists of two main components: (1) a wind estimation stage using an attention-augmented GRU network trained on simulated flights to infer the horizontal wind vector from onboard kinematics and dynamics, and (2) a reinforcement learning flight controller based on proximal policy optimization that utilizes the wind estimates to improve trajectory tracking performance.
Results
The wind estimator achieved a per-flight RMSE of 0.40 m/s and a direction error of 3.2°. The PPO controller reduced horizontal trajectory tracking error by 48% compared to a wind-blind PD baseline, outperforming it in all evaluation episodes. The analysis revealed that the contribution of wind perception to performance improvement increases with wind speed, particularly in strong winds.
Implications
The findings suggest that integrating learned wind perception into control systems can significantly enhance the autonomy and performance of small UAVs in turbulent conditions, potentially leading to more reliable operations in various applications such as search and rescue, environmental monitoring, and agricultural surveying.
Beyond the Performance Illusion: Structure-Aware Stratified Partitioning and Curriculum Distributionally Robust Optimization for Spatially Correlated Domains
Computer Vision
Optimization
- Identification of spatiotemporal leakage and hidden stratification as critical issues in performance evaluation.
- Introduction of Structure-Aware Stratified Partitioning (SASP) to create better dataset splits.
- Development of Curriculum Distributionally Robust Optimization (CDRO) to stabilize training under rigorous evaluation conditions.
- Demonstration of improved generalization and confidence calibration across multiple domains.
Read more
Beyond the Performance Illusion: Structure-Aware Stratified Partitioning and Curriculum Distributionally Robust Optimization for Spatially Correlated Domains
Summary
This paper addresses the limitations of traditional performance evaluation methods in machine learning, particularly in spatiotemporally correlated domains such as aerial surveillance, precision agriculture, and medical imaging. The authors identify two critical issues: spatiotemporal leakage, where correlated samples contaminate training and validation splits, and hidden stratification, where performance metrics obscure errors in minority subpopulations. To tackle these challenges, they propose a unified framework that includes Structure-Aware Stratified Partitioning (SASP) and Curriculum Distributionally Robust Optimization (CDRO). SASP ensures that validation splits are semantically disjoint and class-balanced, significantly reducing data leakage and improving the evaluation of generalization. CDRO enhances training stability by focusing on difficult subgroups during the learning process. The proposed methods demonstrate improved generalization and more reliable confidence calibration across various benchmarks, revealing failure modes that traditional random-split evaluations fail to expose.
Methodology
The authors propose a two-part framework: SASP for creating validation splits that minimize spatiotemporal leakage while maintaining class balance, and CDRO for training models with a focus on underperforming subgroups. This approach couples dataset partitioning with model training to enhance evaluation accuracy.
Results
The combination of SASP and CDRO leads to consistently improved generalization performance, more reliable confidence calibration, and the exposure of failure modes that are typically hidden under conventional evaluation methods. The methods were validated across benchmarks in aerial surveillance, precision agriculture, and medical imaging.
Implications
The proposed framework has significant implications for high-stakes AI applications where accurate performance evaluation is crucial. By addressing the shortcomings of traditional evaluation methods, it can enhance the reliability of machine learning models in real-world scenarios, particularly in fields where data is spatially or temporally correlated.
Role-Aware Neural Convex Divergence Heads for Asymmetric Representation Learning
NLP
Theory
Interpretability
- Introduction of a role-aware neural convex divergence head for asymmetric representation learning.
- The method retains classical Bregman properties while allowing for role-specific projections.
- Empirical results show improved directional accuracy over traditional symmetric and unstructured methods.
- The approach is versatile, functioning as a plug-in distance module for various encoders.
Read more
Role-Aware Neural Convex Divergence Heads for Asymmetric Representation Learning
Summary
This paper addresses the challenge of representation learning in contexts where directed relations are significant, such as lexical entailment and citation links. Traditional distance metrics like Euclidean and cosine distances are symmetric and fail to capture the directional nature of these relationships. The authors propose a novel approach called the role-aware neural convex divergence head, which incorporates source and target role projections into the evaluation of a neural Bregman divergence. This method yields a nonnegative structured score in a role-projected space, effectively allowing for asymmetric representation learning. The paper provides a theoretical framework for understanding the properties of this divergence head, including its projected-space identity and source-role convexity. Empirical evaluations across various benchmarks demonstrate that the proposed method consistently outperforms traditional symmetric distances and unstructured asymmetric scorers, particularly in terms of directional accuracy, while maintaining a zero observed negative divergence rate. However, in specific scenarios like large fixed-feature citation prediction, traditional symmetric or hyperbolic methods still exhibit superior ranking accuracy. Overall, the role-aware divergence head serves as a structured and interpretable module for tasks requiring directional relations.
Methodology
The proposed method utilizes two learnable role projections before computing a neural Bregman divergence. This construction allows for the transformation of input embeddings into source and target roles, facilitating the evaluation of asymmetric relationships. The authors provide a theoretical characterization of the method, including analyses of its properties and performance across various benchmarks.
Results
The experiments conducted across ten random seeds on semantic and ontology benchmarks indicate that the role-aware projections significantly enhance directional accuracy compared to plain ICNN-Bregman heads, while maintaining a zero observed negative divergence rate. However, in the context of large fixed-feature citation prediction, traditional symmetric or hyperbolic baselines demonstrated stronger ranking accuracy.
Implications
The role-aware neural convex divergence head can be applied in various domains where directional relationships are crucial, such as natural language processing tasks involving entailment and citation analysis. Its structured and interpretable nature makes it a valuable addition to representation learning frameworks.
Probabilistic Low-Voltage Peak Load Forecasting with Time Series Foundation Models Evaluated on Application-Oriented Metrics
Time Series
- Extensive evaluation of time series foundation models for low-voltage load forecasting.
- Introduction of a novel application-oriented metric for assessing forecasting performance.
- Demonstration of superior performance of Chronos-2 in peak load prediction.
- Investigation of the impact of weather covariates on forecasting accuracy.
Read more
Probabilistic Low-Voltage Peak Load Forecasting with Time Series Foundation Models Evaluated on Application-Oriented Metrics
Summary
This paper addresses the challenges of low-voltage load forecasting in the context of increasing electrification and decentralized energy generation. The authors evaluate the performance of time series foundation models (TSFMs) such as Chronos-Bolt, Chronos-2, and TabPFN-TS against six baseline models using a dataset of 200 real-world low-voltage feeders. The study emphasizes the importance of uncertainty estimation and peak load prediction, which are often overlooked in existing forecasting methods. An ablation study reveals that while weather covariates significantly influence forecasting accuracy, TSFMs can adapt to increased uncertainty when these covariates are omitted. The authors introduce a novel application-oriented metric that links forecasting capabilities to grid asset planning, balancing cost reduction with the risk of failure. The results indicate that Chronos-2 outperforms other models, particularly in peak prediction, and highlight the necessity of integrating probabilistic forecasting methods in low-voltage load forecasting to meet the specific requirements of distribution system operators (DSOs).
Methodology
The authors conducted an extensive evaluation of several TSFMs using a dataset of 200 low-voltage feeders. They compared the performance of TSFMs against baseline models, performed an ablation study to assess the role of weather covariates, and introduced a new metric for application-oriented evaluation of forecasting capabilities.
Results
The study found that Chronos-2 significantly outperformed other models in terms of accuracy, especially for peak load predictions. The ablation study showed that while weather covariates are important, TSFMs can still provide reliable forecasts without them. The proposed application-oriented metric effectively linked forecasting performance to grid management requirements.
Implications
The findings suggest that TSFMs can enhance the forecasting capabilities of DSOs, enabling better planning and operation of low-voltage grids. The introduction of a new evaluation metric can help align forecasting models with practical grid management needs, ultimately supporting the integration of renewable energy sources and improving grid reliability.
SINA: A Fully Automated Circuit Schematic Image to Netlist Generator Using Artificial Intelligence
Computer Vision
NLP
Multimodal
- SINA automates the conversion of circuit schematic images to netlists, enhancing EDA workflows.
- The system achieves high accuracy (96.67%) in netlist generation, surpassing state-of-the-art methods.
- SINA effectively handles both IC and PCB schematics, including complex layouts and hand-drawn designs.
- The integration of deep learning, OCR, and VLMs allows for robust component detection and reference designator assignment.
Read more
SINA: A Fully Automated Circuit Schematic Image to Netlist Generator Using Artificial Intelligence
Summary
The paper presents SINA, an open-source, fully automated pipeline designed to convert circuit schematic images into machine-readable netlists, addressing significant limitations in current Electronic Design Automation (EDA) tools. Existing methods struggle with generalization across Integrated Circuit (IC) and Printed Circuit Board (PCB) schematics, inaccurate component recognition, unreliable connectivity inference, and weak reference designator extraction. SINA integrates deep learning for robust component detection, connected-component labeling for accurate connectivity inference, Optical Character Recognition (OCR) for extracting component reference designators, and a Vision-Language Model (VLM) for reliable assignment of these designators. The system is capable of handling various schematic styles, including hand-drawn and scanned images, without assumptions about color or resolution. Validation of the generated netlists is performed using graph isomorphism techniques, demonstrating a netlist generation accuracy of 96.67%, significantly outperforming existing approaches by a factor of 2.72.
Methodology
SINA employs a multi-faceted approach combining deep learning for component detection, connected-component labeling for connectivity inference, Optical Character Recognition (OCR) for extracting reference designators, and a Vision-Language Model (VLM) for assigning these designators accurately. The system is designed to work with various schematic styles and formats, ensuring broad applicability.
Results
SINA achieved a netlist generation accuracy of 96.67%, which is 2.72 times higher than existing methods. The validation process utilized graph isomorphism techniques to ensure the correctness of the generated netlists.
Implications
The development of SINA has significant implications for the field of Electronic Design Automation, facilitating the automation of circuit design processes, improving the accuracy of netlist generation, and enabling the creation of comprehensive databases for AI-based circuit design models. Its open-source nature encourages further research and development in this area.
BOUNDARY_SYNC: Measuring Communication-Induced Representational Coupling in Multi-Agent LLM Systems
Large Language Models
NLP
Multimodal
- Boundary_Sync protocol effectively measures representational coupling in multi-agent LLMs.
- Text communication leads to significant homogenization of outputs (CAF=0.803).
- Image communication also causes homogenization, comparable to text (CAF=0.834).
- Group size influences the direction of coupling, with smaller groups potentially leading to diversification.
Read more
BOUNDARY_SYNC: Measuring Communication-Induced Representational Coupling in Multi-Agent LLM Systems
Summary
This paper introduces Boundary_Sync, a measurement protocol designed to quantify the representational coupling induced by communication among large language models (LLMs) in multi-agent systems. The study investigates whether inter-agent communication leads to output convergence or divergence, which is crucial for understanding the dynamics of multi-agent architectures. The Coupling Amplification Factor (CAF) is introduced as a metric to measure this coupling, where a CAF value less than 1 indicates homogenization and greater than 1 indicates diversification. Through controlled experiments using GPT-4o, the authors find that text-based communication significantly homogenizes outputs (CAF=0.803), while image-based communication also leads to homogenization (CAF=0.834). Notably, the direction of coupling is influenced by group size; with five agents, both modalities show homogenization, but with three agents, the CAF indicates a shift towards diversification. The results demonstrate that coupling is stateless, driven by immediate peer information, and highlight the implications for designing effective multi-agent LLM systems.
Methodology
The authors conducted controlled experiments using GPT-4o, measuring the Coupling Amplification Factor (CAF) across text-based and image-based communication scenarios. They employed a no-communication ablation and prompt perturbation controls to isolate the effects of communication on output homogenization. The study involved approximately 9,900 API calls across three experimental runs with 30 agents per condition.
Results
The experiments revealed that text communication significantly homogenizes outputs (CAF=0.803), while image communication also results in homogenization (CAF=0.834). The direction of coupling varied with group size; with five agents, both modalities showed homogenization, but with three agents, the CAF values indicated a shift towards diversification (CAF > 1). Additionally, the coupling was found to be stateless, disappearing when peer information was removed.
Implications
The findings have direct implications for the design of multi-agent LLM systems, suggesting that communication can significantly influence the diversity of outputs. Understanding the dynamics of representational coupling can help in creating more effective collaborative reasoning and decision-making systems using LLMs.
QFedAgent: Quantum-Enhanced Personalized Federated Learning for Multi-Agent Activity Recognition
Federated Learning
Multimodal
Robotics
- Introduction of QFedAgent, a quantum-enhanced personalized federated learning framework.
- Utilization of variational quantum circuits for efficient multimodal data fusion.
- Achieved approximately 10× reduction in parameters compared to classical methods.
- Demonstrated high accuracy (97.7%) on the OPPORTUNITY dataset under non-IID conditions.
Read more
QFedAgent: Quantum-Enhanced Personalized Federated Learning for Multi-Agent Activity Recognition
Summary
This paper presents QFedAgent, a novel hybrid quantum-classical framework designed for personalized federated learning (FL) in multi-agent activity recognition tasks. The authors address the challenges posed by heterogeneous and non-independent identically distributed (non-IID) data generated by distributed robotic systems, which can degrade the performance of conventional FL algorithms. To overcome these limitations, QFedAgent employs a variational quantum circuit (VQC) fusion module that efficiently models interactions between accelerometer and gyroscope data through quantum state encoding and entanglement. This approach significantly reduces the number of parameters required for model training, achieving a reduction from 33,000 in classical multi-layer perceptron-based fusion to just 72 quantum rotation parameters. The framework is evaluated on the OPPORTUNITY dataset, demonstrating a mean test accuracy of 97.7% under subject-based non-IID partitions, showcasing its effectiveness in maintaining robust performance while minimizing communication costs and parameter overhead.
Methodology
The QFedAgent framework integrates a shared encoder for processing accelerometer and gyroscope signals using dual CNNs, followed by a VQC fusion layer that captures cross-modal interactions. The model maintains client-specific adapters and classifiers, ensuring that only shared parameters are aggregated during federated training, thus preserving data privacy.
Results
The framework achieved a mean test accuracy of 97.7% on the OPPORTUNITY dataset, demonstrating its effectiveness in handling non-IID data distributions while significantly reducing the complexity of the fusion module.
Implications
QFedAgent has potential applications in privacy-sensitive robotic systems, particularly in fields such as industrial automation and healthcare monitoring, where efficient and secure data processing is crucial. The integration of quantum computing into federated learning could pave the way for more advanced and efficient machine learning models in distributed environments.
The Rollout Infrastructure Tax in Coding-Agent Reinforcement Learning
Reinforcement Learning
Efficient ML
- The rollout infrastructure tax significantly affects the efficiency of coding-agent RL systems.
- Cold-start latency can vary by up to 110× across different execution substrates.
- The choice of substrate can lead to a 1.8× spread in projected worker-hours for large-scale rollouts.
- Optimizing execution substrates should be a core concern in the design of coding-agent RL systems.
Read more
The Rollout Infrastructure Tax in Coding-Agent Reinforcement Learning
Summary
This paper addresses the often-overlooked impact of execution infrastructure on coding-agent reinforcement learning (RL), which relies heavily on numerous interactive software rollouts. The authors introduce the concept of the 'rollout infrastructure tax,' which refers to the latency and costs introduced by the systems executing coding-agent RL trajectories. They conduct a comparative study of four execution substrates: single containers, hosted sandboxes, Kubernetes-orchestrated containers, and cloud virtual machines. The study reveals significant variations in cold-start latency and projected worker-hours for large-scale rollouts, suggesting that optimizing execution substrates should be an integral part of the training system rather than merely an implementation detail. The authors propose a controlled evaluation methodology to measure the rollout infrastructure tax and provide design requirements for more efficient rollout-native substrates.
Methodology
The authors conducted a measurement study comparing four common execution substrates while holding the coding-agent workload constant. They defined the rollout infrastructure tax and developed a controlled evaluation methodology to assess the impact of different substrates on rollout performance.
Results
The study found that substrate choice directly influences rollout performance, with cold-start latencies varying significantly. For one million 150-step trajectories, the choice of substrate resulted in a 1.8× increase in projected worker-hours, equating to an additional 5,316 worker-hours.
Implications
The findings suggest that as coding-agent RL systems scale, addressing the rollout infrastructure tax can lead to substantial efficiency gains. This highlights the need for future RL systems to incorporate substrate optimization as a fundamental aspect of their design.
Set Diffusion: Interpolating Token Orderings Between Autoregression and Diffusion for Fast and Flexible Decoding
NLP
Large Language Models
Generative Models
- Set diffusion allows for flexible-length and flexible-position token generation, enhancing decoding flexibility.
- The set-causal diffusion architecture supports KV cache updates after each inference step, improving efficiency.
- Set diffusion outperforms previous diffusion models in speed-quality tradeoffs across various tasks.
- The model demonstrates superior infilling performance compared to block diffusion.
Read more
Set Diffusion: Interpolating Token Orderings Between Autoregression and Diffusion for Fast and Flexible Decoding
Summary
The paper introduces a novel class of language models called 'set diffusion,' which aims to enhance the decoding process by allowing for flexible-position and flexible-length token sets. Traditional discrete diffusion models, while improving upon autoregressive models in terms of quality, are limited by fixed-length generation and lack support for key-value (KV) caching. Block diffusion attempts to address these issues but is still constrained by rigid left-to-right block structures. Set diffusion overcomes these limitations by defining an autoregressive probability distribution over arbitrary token sets, enabling faster inference and any-order decoding. This approach not only improves the speed-quality tradeoff in tasks such as mathematical reasoning, summarization, and unconditional generation but also outperforms block diffusion in infilling tasks. The authors provide code and model weights for further research and application.
Methodology
The authors propose a set diffusion model that factorizes the likelihood over flexible-position, flexible-length token sets. This model utilizes a set-causal diffusion architecture that allows for KV cache updates after each token generation step, contrasting with previous models that required full block completion before cache updates. The methodology includes a comparative analysis of performance metrics across various tasks, emphasizing the advantages of the new approach over traditional block diffusion and autoregressive models.
Results
Set diffusion achieves state-of-the-art performance in terms of speed and quality across tasks such as mathematical reasoning, summarization, and unconditional generation. It shows significant improvements in infilling tasks compared to block diffusion, demonstrating better speed-quality tradeoffs and greater flexibility in decoding.
Implications
The introduction of set diffusion could lead to advancements in natural language processing applications that require flexible and efficient decoding methods. Its ability to support arbitrary token orderings and KV caching could enhance the performance of language models in real-time applications, such as chatbots and interactive systems, where speed and adaptability are crucial.
Multi-modal Rail Crossing Safety Analysis
Multimodal
- The system combines visual data from railway crossing images with structured accident history data.
- It utilizes Vision-Language Models to analyze and assess crossing safety from a highway user's perspective.
- The proposed pipeline achieves a macro F1 score of 0.757 for risk classification.
- The model estimates FRA safety scores with an RMSE of 0.078, indicating strong predictive capability.
Read more
Multi-modal Rail Crossing Safety Analysis
Summary
This paper addresses the critical safety concerns associated with highway-rail grade crossings in the United States, where over 2,000 collisions and 200 fatalities occur annually. The authors propose a novel AI system that leverages multi-modal data, including visual cues from images of railway crossings and structured data from official accident reports, to assess crossing safety. The study introduces a proof-of-concept pipeline that integrates these data sources to provide safety assessments that align with expert opinions and Federal Railroad Administration (FRA) safety scoring. The methodology involves using Vision-Language Models (VLMs) to analyze visual information and historical incident records, focusing on two main tasks: risk scoring and visual risk inspection. The results demonstrate that the proposed system can effectively classify crossings as high-risk or low-risk with a macro F1 score of 0.757 and estimate FRA-based safety scores with a root mean square error (RMSE) of 0.078 and a correlation of 0.492. The findings suggest that incorporating visual data can enhance the accuracy of risk assessments, providing a scalable solution for improving railway crossing safety.
Methodology
The authors developed a multi-modal pipeline that integrates street-level imagery of railway crossings with historical accident data (FRA Form 57 records). They employed Vision-Language Models to perform two tasks: risk scoring (classifying crossings as high or low risk and predicting continuous risk scores) and visual risk inspection. The methodology involved image-only prompting, combined prompting with structured data, and fine-tuning of the models to improve performance.
Results
The proposed system identified high-risk and low-risk crossings with a macro F1 score of 0.757. Additionally, it estimated FRA-based safety scores with a root mean square error of 0.078 and a correlation of 0.492, demonstrating the effectiveness of the multi-modal approach in aligning with expert assessments.
Implications
This research has significant implications for enhancing railway crossing safety assessments, enabling transportation agencies to prioritize interventions based on more accurate risk evaluations. The integration of visual data can lead to more informed decision-making and resource allocation, ultimately reducing accidents and fatalities at railway crossings.
Do LLMs Truly Generalize in the Molecular Domain? A Perturbation-Based Analysis
Large Language Models
Graph Learning
- LLMs show limited generalization in the molecular domain, with performance sensitive to small structural changes.
- The Molecular Perturbation framework reveals that even single edits can significantly degrade model performance.
- In-Context Tuning (ICT) can improve robustness by anchoring predictions to structurally similar molecules.
- Current LLMs prioritize textual coherence over topological consistency, leading to fragility in predictions.
Read more
Do LLMs Truly Generalize in the Molecular Domain? A Perturbation-Based Analysis
Summary
This paper investigates the generalization capabilities of Large Language Models (LLMs) in the molecular domain, particularly focusing on their performance when faced with structural perturbations of molecules. The authors introduce a Molecular Perturbation framework that generates syntax-valid structural variants of training molecules using controlled Graph Edit Distance (GED). Their analysis reveals that even minor structural edits can lead to significant performance drops on molecular tasks, indicating a narrow local trust region and fragility in the models' sensitivity to structural changes. To address this issue, the paper explores In-Context Tuning (ICT), which conditions predictions on structurally similar molecules, suggesting that ICT can partially expand the local trust region and improve robustness against structural variations. The findings highlight the limitations of current LLMs in capturing the complex relationships in chemical space and propose a direction for enhancing their stability through similarity-based inference.
Methodology
The authors developed a Molecular Perturbation framework that generates structural variants of molecules under controlled Graph Edit Distance (GED). They applied granular Atom and Bond Perturbations to simulate structural changes and analyzed the resulting performance of LLMs on various molecular tasks. Additionally, they examined the effects of In-Context Tuning (ICT) on model robustness by conditioning predictions on nearby training examples.
Results
The empirical analysis demonstrated that LLMs exhibit a narrow local trust region, with performance dropping significantly as GED increases. While ICT improved performance under perturbations, it did not completely eliminate sensitivity to structural changes, indicating that while some robustness can be achieved, challenges remain in fully stabilizing molecular LLMs.
Implications
The findings suggest that enhancing LLMs with similarity-based inference mechanisms could bridge the gap between probabilistic sequence modeling and the rigid constraints of chemical structures. This could lead to more reliable applications of LLMs in molecular discovery and related fields.
X-LogSMask: Expand Transformer for Graph-Structured Data
Graph Learning
- X-LogSMask introduces a logarithmic structural mask for graph-structured data, enhancing interpretability and efficiency.
- The method allows multi-hop information propagation within a single Transformer layer by assigning different powers of the adjacency matrix to attention heads.
- Transformers with X-LogSMask achieve state-of-the-art performance on 13 datasets across various benchmarks.
- The approach maintains the original Transformer architecture while improving its applicability to graph data.
Read more
X-LogSMask: Expand Transformer for Graph-Structured Data
Summary
The paper introduces X-LogSMask, a novel approach to adapt Transformer architectures for graph-structured data. Traditional Transformers utilize an all-to-all self-attention mechanism, which is not well-suited for the sparse and structured nature of graphs. X-LogSMask addresses this by incorporating a logarithmic structural mask that integrates graph topology directly into attention logits. This method effectively suppresses irrelevant node interactions while maintaining feature-dependent attention. By assigning different powers of a normalized adjacency matrix to various attention heads, X-LogSMask allows for multi-hop information propagation within a single layer, enhancing the interpretability and efficiency of the model. The authors demonstrate that Transformers equipped with X-LogSMask achieve state-of-the-art performance on 13 out of 20 benchmark datasets, showcasing its effectiveness in graph learning tasks without altering the core Transformer architecture.
Methodology
X-LogSMask is constructed from a symmetrically normalized adjacency matrix with self-loops, which is logarithmically transformed and added to the attention logits. This design suppresses unsupported node interactions and allows for a structured interpretation of attention heads, facilitating multi-hop information propagation in a single layer.
Results
The implementation of X-LogSMask in Transformers led to state-of-the-art performance on 13 out of 20 node-, edge-, and graph-level benchmarks, demonstrating its effectiveness in graph learning tasks. The lightweight one-layer configuration remained competitive, indicating the method's efficiency.
Implications
The findings suggest that simple, interpretable structural masks can significantly enhance the performance of self-attention mechanisms in graph learning, making it a promising approach for various applications in relational systems such as social networks, molecular structures, and transportation networks.
Finite-Lag Operator Geometry of Recurrent Representations
Theory
Time Series
Optimization
- Introduces finite-lag operator geometry for analyzing recurrent representations.
- Develops a conditional transport law and source-centered transport tensor to capture dynamics.
- Proves structural results including affine covariance and estimator stability.
- Demonstrates the ability to detect deterministic recurrent motion not visible to traditional methods.
Read more
Finite-Lag Operator Geometry of Recurrent Representations
Summary
This paper introduces a novel framework for analyzing recurrent representations in machine learning through finite-lag operator geometry. Traditional methods often treat representations as static point clouds, but this approach recognizes that recurrent hidden states are dynamic and meaningful in the context of their evolution over time. The author develops a conditional transport law, Q∆(dy | x), which captures the relationship between source and successor states over a fixed lag. This law is used to derive a source-centered transport tensor, G∆, which decomposes into conditional spread and coherent displacement, as well as an antisymmetric coordinate circulation, Wρ∆, that summarizes directed flow. The paper proves several structural results, including affine covariance and stability of the estimator, and demonstrates that the framework can detect deterministic recurrent motion that is not captured by traditional infinitesimal geometry. Controlled experiments validate the theoretical predictions and reveal architecture-dependent differences in transport scale and displacement in performance-matched networks. Overall, this work provides a geometric perspective on recurrent dynamics, offering insights into the behavior of recurrent neural networks and their representations.
Methodology
The methodology involves defining a finite-lag conditional transport law based on observed source-successor pairs, estimating it using a dense Gaussian source-smoothing operator. The framework derives a source-centered transport tensor and antisymmetric circulation statistic to analyze the geometry of recurrent representations. Theoretical results are established, and controlled experiments are conducted to validate the findings.
Results
The paper presents a decomposition of the transport tensor into conditional spread and coherent displacement, proving that deterministic recurrent motion can be detected even when traditional methods fail. The framework's observables are shown to be sensitive to architecture and resolution, with controlled experiments confirming the theoretical predictions.
Implications
This work has potential implications for the analysis and design of recurrent neural networks, providing a new geometric perspective that could enhance understanding of their dynamics and improve model performance in various applications, particularly in time series analysis and sequential data processing.
Model Merging as Probabilistic Inference in Fine-Tuning Parameter Space
Theory
Optimization
Efficient ML
- Introduces a probabilistic framework for model merging that improves upon traditional geometric approaches.
- Demonstrates that existing methods can be viewed as special cases of the proposed product-of-experts formulation.
- Addresses the mismatch between Gaussian assumptions and heavy-tailed distributions of directional residuals.
- Implements a heavy-tailed PoE design using Cauchy experts for better model merging outcomes.
Read more
Model Merging as Probabilistic Inference in Fine-Tuning Parameter Space
Summary
This paper presents a novel approach to model merging, which combines multiple task-specific models into a single multi-task model without requiring additional fine-tuning. Traditional methods often rely on geometric properties of solution spaces, which can limit their effectiveness in scoring the utility of task-specific updates. The authors propose a probabilistic framework for model merging, viewing it as a product-of-experts (PoE) scenario where each task-specific model acts as an energy-based expert model (EBM). They identify that existing methods often assume Gaussian distributions for update directions, which do not align well with the observed heavy-tailed distributions of directional residuals. To address this, the authors introduce a heavy-tailed PoE design using Cauchy experts, which better captures the behavior of these residuals and allows for a convergent inference procedure. The proposed framework not only unifies existing merging methods but also enhances their performance, as demonstrated through extensive experiments across various tasks and architectures, leading to significant improvements over state-of-the-art baselines.
Methodology
The authors formulate model merging as MAP inference in the fine-tuning parameter space under a product of task-specific energy-based experts. They analyze existing merging methods as special cases of their framework and develop a heavy-tailed PoE design to better capture the distribution of directional residuals.
Results
The experiments conducted across multiple tasks and architectures reveal that the proposed method significantly outperforms existing state-of-the-art model merging techniques, demonstrating its effectiveness in creating a robust multi-task model without additional fine-tuning.
Implications
This research has potential applications in scenarios where multiple task-specific models need to be combined into a single model, such as in multilingual applications or resource-constrained environments. It also opens avenues for further exploration of probabilistic approaches in model merging and other areas of machine learning.
Extreme Adaptive Transformer for Time Series Forecasting
Time Series
- Introduction of the Extreme-Adaptive Attention mechanism that dynamically adjusts query-key interactions based on event severity.
- Exformer is an encoder-only Transformer framework designed specifically for long-term time series forecasting.
- Demonstrated superior performance in forecasting accuracy on hydrologic datasets compared to existing state-of-the-art models.
- Reduction in computational costs while maintaining effective modeling of imbalanced time series data.
Read more
Extreme Adaptive Transformer for Time Series Forecasting
Summary
This paper addresses the challenges of time series forecasting, particularly in hydrologic contexts where extreme events, such as sudden streamflow peaks, are infrequent but critical. Traditional forecasting models often fail to adequately represent these rare events due to their uniform treatment of time points. The authors propose the Extreme-Adaptive Transformer (Exformer), which introduces an extreme-adaptive attention mechanism that consists of three components: Local, Stride, and Extreme. The Local and Stride components focus on capturing short-term and periodic dependencies among normal observations, while the Extreme component specifically models dependencies related to extreme events. Through experiments on four real-world hydrologic datasets, Exformer demonstrates superior forecasting performance compared to state-of-the-art models, highlighting the importance of explicitly incorporating extreme-aware attention in forecasting frameworks. The results indicate that Exformer not only improves accuracy in predicting extreme events but also reduces computational costs associated with traditional full-attention mechanisms.
Methodology
The Exformer employs an extreme-adaptive attention mechanism that selectively models temporal dependencies based on whether the time step corresponds to a normal or extreme event. This mechanism integrates Local and Stride attention for normal time steps and an Extreme attention component for extreme events, allowing for efficient computation and improved focus on critical patterns.
Results
Exformer outperformed state-of-the-art baselines in terms of RMSE and MAPE metrics across four real-world hydrologic datasets. The model not only achieved better forecasting accuracy but also demonstrated reduced attention computation compared to traditional full-attention approaches. Ablation studies validated the contributions of the Extreme-Adaptive Attention mechanism to the overall performance.
Implications
The findings suggest that incorporating extreme-aware attention mechanisms can significantly enhance the forecasting capabilities of models dealing with imbalanced time series data, particularly in fields such as hydrology, flood monitoring, and resource management. This approach could be applied to other domains where rare but impactful events are present.
A Mathematical Introduction to Diffusion Models
Generative Models
Theory
Optimization
- Introduces diffusion models from a mathematical perspective, emphasizing sampling dynamics.
- Establishes convergence guarantees for Langevin diffusion and its variants.
- Analyzes discretization of continuous diffusion models into practical samplers.
- Explores inference-time control techniques for trained models.
Read more
A Mathematical Introduction to Diffusion Models
Summary
This paper provides a proof-oriented introduction to diffusion models, focusing on their mathematical foundations and applications in sampling. It is structured into five main sections: the first discusses the sampling language and Langevin dynamics, establishing convergence guarantees for Langevin diffusion and its variants. The second section develops continuous-time score-based diffusion models using Gaussian noising and related concepts. The third section addresses the discretization of these models into implementable samplers, analyzing sampling errors. The fourth section explores discrete diffusion on finite state spaces, reformulating the error analysis in this context. The final section delves into inference-time control strategies for trained models, including guidance and reinforcement learning techniques. The notes are designed for beginning graduate students with a background in probability, providing detailed proofs of core definitions, representative estimates under simplifying assumptions, and a roadmap for research-level theorems. Appendices cover essential analytic tools such as Itô calculus and Gaussian identities.
Methodology
The methodology includes a rigorous mathematical framework that combines sampling theory, stochastic differential equations (SDEs), and error analysis. It employs tools like the Fokker–Planck equation, Tweedie's identity, and various discretization techniques to derive convergence guarantees and error bounds for diffusion models.
Results
The paper presents convergence guarantees for Langevin diffusion processes, detailed error analyses for different sampling strategies, and methods for steering inference-time behavior in trained models. It also provides a comprehensive framework for understanding the relationship between continuous and discrete diffusion models.
Implications
The findings have significant implications for the development of generative models, particularly in enhancing the efficiency and accuracy of sampling methods. The insights into inference-time control could lead to improved performance in applications such as image generation and other generative tasks.
Do Newer Lightweight CNNs Perform Better Under Resource Constraints? A Controlled Multigenerational Study of Architecture, Initialization, Training Budget, and Efficiency
Computer Vision
Efficient ML
- EfficientNetV2-S and RepViT-M1.0 lead in accuracy for CIFAR-10/CIFAR-100 and Tiny ImageNet, respectively.
- EfficientNet-B0 offers a strong balance of accuracy and resource efficiency across all datasets.
- MobileNetV3-Small is the fastest model with the lowest GMAC count, performing well under severe resource constraints.
- The study highlights the importance of controlled benchmarking in evaluating model performance.
Read more
Do Newer Lightweight CNNs Perform Better Under Resource Constraints? A Controlled Multigenerational Study of Architecture, Initialization, Training Budget, and Efficiency
Summary
This study investigates the performance of nine lightweight convolutional neural networks (CNNs) under resource constraints, focusing on their predictive accuracy and efficiency across three datasets: CIFAR-10, CIFAR-100, and Tiny ImageNet. The research employs a controlled evaluation protocol to assess models based on top-1 accuracy, macro F1 score, and top-5 accuracy, while also measuring resource demands such as parameter count, FP32 storage, multiply-accumulate operations (GMACs), latency on different hardware, and peak CUDA memory usage. The findings reveal that EfficientNetV2-S achieves the highest top-1 accuracy on CIFAR-10 and CIFAR-100, while RepViT-M1.0 excels on Tiny ImageNet. EfficientNet-B0 emerges as a strong contender, balancing accuracy and resource efficiency, using significantly fewer parameters and GMACs compared to its competitors. MobileNetV3-Small is highlighted as the fastest model with the lowest GMAC count, demonstrating superior performance under severe resource constraints. The study also emphasizes the importance of controlled benchmarking and the nuanced trade-offs between accuracy and resource usage, suggesting that newer architectures may not universally outperform older models. Overall, the research provides valuable insights into model selection for resource-limited environments, emphasizing the need for a multi-objective approach in evaluating lightweight CNNs.
Methodology
The study compares nine lightweight CNN architectures using a controlled training and evaluation protocol across three datasets. Performance metrics include top-1 accuracy, macro F1, and top-5 accuracy, while resource demands are measured through parameters, GMACs, latency, and memory usage. Point-estimate Pareto frontiers are used to analyze accuracy-resource trade-offs, and separate training experiments are conducted to assess the impact of initialization on model performance.
Results
EfficientNetV2-S achieved top-1 accuracies of 97.57% and 86.98% on CIFAR-10 and CIFAR-100, respectively. RepViT-M1.0 led Tiny ImageNet with 79.87% accuracy. EfficientNet-B0 maintained competitive accuracy while using approximately 79% fewer parameters than EfficientNetV2-S. MobileNetV3-Small recorded the lowest GMACs and fastest latency, outperforming MobileNetV4-Conv-S in accuracy despite its higher resource usage.
Implications
The findings suggest that model selection for resource-constrained environments should consider both accuracy and resource efficiency. The study provides a framework for evaluating lightweight CNNs, which can guide future research and practical applications in mobile and embedded systems.
kNNGuard: Turning LLM Hidden Activations into a Training-Free Configurable Guardrail
NLP
Large Language Models
Efficient ML
- kNNGuard operates in the activation space of a frozen LLM, eliminating the need for fine-tuning.
- The framework achieves competitive or superior F1 scores compared to state-of-the-art fine-tuned guardrails while running significantly faster.
- Domain adaptation is simplified to updating a small reference bank, allowing for rapid deployment.
- The methodology includes a layer-ensemble and a fused-ensemble approach to enhance classification accuracy.
Read more
kNNGuard: Turning LLM Hidden Activations into a Training-Free Configurable Guardrail
Summary
The paper introduces kNNGuard, a novel training-free guardrail framework designed for large language models (LLMs) to detect unsafe, off-topic, or adversarial prompts. Unlike traditional guardrails that rely on fine-tuning classifiers, kNNGuard leverages the hidden activations of a frozen LLM, utilizing a small bank of labeled prompts to perform classification through multi-layer k-nearest neighbor (kNN) techniques. The framework demonstrates superior performance across six diverse domains, achieving competitive F1 scores while significantly reducing inference latency. The adaptability of kNNGuard allows for rapid domain-specific updates by simply modifying the labeled bank, making it a practical solution for real-time deployment in production environments. The paper also discusses the impact of system prompts and layer selection on classification performance, providing insights for integrating kNNGuard into existing LLM pipelines.
Methodology
kNNGuard utilizes a training-free approach by extracting hidden activations from a frozen LLM. It employs a small labeled bank of safe and unsafe prompts to perform classification using multi-layer kNN techniques, incorporating Fisher-discriminant-based weighting for layer selection. A fused variant combines activation-space and embedding-space scores for improved robustness.
Results
kNNGuard achieved an average F1 score of 87.4% with a false positive rate of 12.9% across six domains, demonstrating competitive performance against fine-tuned guardrails while operating at 2.7× lower inference latency. The guardrail construction time for a 50-sample bank was under 10 seconds, showcasing significant efficiency compared to traditional fine-tuning methods.
Implications
The kNNGuard framework offers a practical solution for integrating safety measures into LLM applications, enabling rapid adaptation to new domains without the overhead of fine-tuning. This could enhance the deployment of LLMs in sensitive areas such as customer service, healthcare, and security, where prompt safety is critical.
MKGR: Multimodal Knowledge-Graph Representation Learning for Cold-Start Protein-Protein Interaction Prediction
Multimodal
Graph Learning
- MKGR effectively combines protein sequence data with multimodal biomedical knowledge graphs for improved PPI prediction.
- The introduction of a bridge reconstruction objective enhances the robustness of graph learning in sparse data scenarios.
- A pair-level gated fusion mechanism allows for adaptive integration of sequence and graph representations tailored to specific protein pairs.
- Experimental results indicate significant performance improvements over traditional PPI prediction methods in cold-start settings.
Read more
MKGR: Multimodal Knowledge-Graph Representation Learning for Cold-Start Protein-Protein Interaction Prediction
Summary
The paper introduces MKGR, a novel multimodal representation learning framework designed to enhance cold-start protein-protein interaction (PPI) prediction. Cold-start scenarios occur when candidate proteins lack observed PPI edges during training, making traditional models reliant on network topology less effective. MKGR addresses this challenge by integrating protein sequence encoding with four biomedical knowledge graphs: protein-drug, protein-disease, protein-miRNA, and protein-lncRNA associations. The framework consists of a sequence branch that utilizes a pretrained protein language model and a Transformer encoder to extract contextual representations from protein sequences, and a graph branch that employs graph attention networks to learn modality-specific protein embeddings from sparse biomedical associations. A bridge reconstruction objective is introduced to regularize graph learning by recovering shared protein-entity associations, while a pair-level gating module adaptively combines sequence and graph evidence for each candidate protein pair. Experimental results on two benchmark datasets demonstrate that MKGR consistently outperforms existing sequence, network, and knowledge-graph baselines across various evaluation metrics, including accuracy, F1 score, AUC, AUPR, and MCC.
Methodology
MKGR employs a multimodal approach that integrates protein sequence encoding through a pretrained language model and Transformer architecture with graph attention networks for processing biomedical knowledge graphs. The model incorporates a bridge reconstruction objective to enhance graph learning and utilizes a pair-level gating mechanism to fuse sequence and graph information adaptively.
Results
MKGR demonstrated superior performance in cold-start PPI prediction tasks across two benchmark datasets, achieving higher scores in accuracy, F1, AUC, AUPR, and MCC compared to competitive baselines, indicating its effectiveness in leveraging multimodal data.
Implications
The findings suggest that MKGR can significantly enhance the accuracy of PPI predictions in scenarios where experimental data is sparse or unavailable, potentially aiding in functional genomics, disease mechanism discovery, and drug development.
HERMES: A Multi-Granularity Labeling Substrate for Pre-training Data Mixtures
NLP
Large Language Models
Optimization
- HERMES introduces a hierarchical labeling substrate that allows for multiple granularities from a single document annotation.
- The method employs a Learned Semantic Transform and a three-stage RVQ for efficient document labeling.
- HERMES achieves competitive clustering performance while enabling flexible exploration of data mixtures.
- The study highlights the importance of sampling strategies in optimizing model performance across different granularities.
Read more
HERMES: A Multi-Granularity Labeling Substrate for Pre-training Data Mixtures
Summary
The paper introduces HERMES, a novel hierarchical labeling substrate designed to enhance data mixing for pre-training large language models (LLMs). Traditional data-mixing methods rely on fixed label systems that limit the granularity and flexibility of data representation. HERMES addresses this limitation by providing a data-derived hierarchical labeling system that allows for multiple granularities from a single annotation. It employs a Learned Semantic Transform followed by a three-stage residual vector quantization (RVQ) to annotate documents into a coarse-to-fine code structure, enabling control over granularity through prefix lengths. The authors demonstrate that HERMES achieves comparable performance to existing clustering methods while allowing for a more versatile exploration of data mixtures. The study reveals that the choice of sampling strategies within the hierarchical structure significantly impacts model performance, particularly in a controlled experimental setup involving 16 tasks. The findings suggest that HERMES can effectively reframe data mixture design, moving away from fixed label sets to a more dynamic, reusable granularity hierarchy.
Methodology
HERMES utilizes a Learned Semantic Transform followed by a three-stage residual vector quantization (RVQ) to create a hierarchical labeling system. This system annotates documents into a coarse-to-fine code structure, where the prefix length of the code determines the granularity of the labels. The authors conducted experiments to compare the performance of different sampling strategies within this hierarchical framework, focusing on a controlled setup involving 16 tasks.
Results
The results indicate that HERMES performs comparably to traditional clustering methods at coarse granularity levels. Specifically, a controlled study showed that switching from a max-entropy coverage sampling strategy to a quality top-30% strategy improved the macro-average performance across 16 tasks by +0.0253. However, this advantage diminished at finer granularity levels due to a significant contraction in the candidate pool, highlighting the interplay between granularity and sampling strategy.
Implications
HERMES has the potential to significantly improve the efficiency and effectiveness of data mixture design in pre-training large language models. By allowing for dynamic granularity adjustments, it can enhance the model's ability to leverage diverse data sources, ultimately leading to better performance in various NLP tasks.
Liquid Latent State Dynamics for Interpretable Turbofan Degradation Modeling
Time Series
Interpretability
- Introduces liquid neural networks for modeling turbofan degradation dynamics.
- Factorizes latent state into degradation and condition components for better interpretability.
- Achieves improved sensor forecasting accuracy on the C-MAPSS benchmark.
- Demonstrates a clearer temporal degradation axis compared to traditional models.
Read more
Liquid Latent State Dynamics for Interpretable Turbofan Degradation Modeling
Summary
This paper presents a novel approach to aircraft engine health monitoring using liquid neural networks as latent dynamics models. The authors address the challenge of accurately modeling the degradation process of turbofan engines while separating it from variations in operating conditions. The proposed model encodes historical sensor data into a latent state, which is then evolved using a liquid transition model to predict future sensor observations. A key innovation is the factorization of the latent state into degradation and condition components, allowing for more interpretable health state representations. The model is evaluated on the C-MAPSS benchmark, demonstrating improved sensor forecasting accuracy, particularly in complex operating conditions. While the model shows promise in providing a clearer degradation trajectory, it does not yet outperform traditional methods in direct remaining useful life (RUL) regression, indicating that it serves better as an interpretable model rather than a precise lifetime predictor.
Methodology
The methodology involves using liquid neural networks to encode a history of sensor observations into a latent state, which is then evolved using a liquid transition model. The latent state is factorized into degradation and condition components, with specific losses applied to each to ensure effective learning and separation of influences.
Results
The proposed model improved the overall sensor forecasting RMSE from 0.2438 (GRU baseline) to 0.2266, with significant gains observed in subsets FD002 and FD004. The learned degradation state exhibited a Spearman correlation of 0.5960, indicating a clearer representation of degradation dynamics. However, the model did not surpass the GRU baseline in direct RUL regression.
Implications
The findings suggest that liquid latent dynamics can enhance predictive maintenance by providing interpretable models of degradation processes, which are crucial for understanding system health and planning maintenance activities. This approach may bridge the gap between accurate forecasting and inspectable health-state modeling in complex engineering systems.
Multi-Head Recurrent Memory Agents
Large Language Models
NLP
Efficient ML
- Identifies memory retention failure as the main issue in recurrent memory agents for long contexts.
- Proposes a novel Multi-Head Recurrent Memory (MHM) framework to improve memory retention.
- Introduces MHM-LRU, a lightweight implementation that guarantees uniform head utilization.
- Demonstrates significant improvements in memory retention and end-to-end accuracy across various benchmarks.
Read more
Multi-Head Recurrent Memory Agents
Summary
This paper addresses the limitations of recurrent memory agents, which are designed to handle long contexts in large language models (LLMs) but suffer from performance degradation as context length increases. The authors diagnose this issue by breaking down performance into two components: memory capture and memory retention, finding that retention is the primary bottleneck. Existing architectures maintain memory as a single block, leading to overwriting issues that compromise retention. To overcome this, the authors propose the Multi-Head Recurrent Memory (MHM) framework, which partitions memory into independent heads and employs a select-then-update strategy to shield unselected heads from overwriting. The paper introduces a specific implementation, Least-Recently-Updated MHM (MHM-LRU), which ensures uniform utilization of memory heads without additional token overhead. Experimental results demonstrate that MHM-LRU significantly enhances memory retention and accuracy across various context lengths, achieving a retention rate improvement from below 30% to 73.96% at 896K tokens. These findings suggest that architectural optimizations can effectively enhance the reliability of long-context recurrent memory agents.
Methodology
The authors conducted a quantitative analysis to decompose performance into memory capture and retention rates. They developed the Multi-Head Recurrent Memory (MHM) framework, which partitions memory into independent heads and uses a stage-wise select-then-update strategy. The MHM-LRU variant was tested on long-context benchmarks to validate its effectiveness in improving retention and accuracy.
Results
MHM-LRU showed substantial improvements in memory retention rates, increasing from less than 30% to 73.96% at 896K tokens. The framework maintained strong and stable accuracy across the 100K-1M token range, outperforming baseline models that experienced sharp performance degradation.
Implications
The findings suggest that architectural changes can significantly enhance the reliability of LLMs in processing long contexts, making them more effective for real-world applications such as document analysis, multi-step research, and extended dialogue systems.
Beyond Adam: SOAP and Muon for Faster, Label-Efficient Training of Machine Learning Interatomic Potentials
Optimization
Efficient ML
- SOAP and SOAP-Muon optimizers significantly improve convergence speed and accuracy compared to AdamW.
- The performance of optimizers is particularly enhanced under conditions of partial force supervision.
- SOAP-Muon can match the performance of AdamW trained with full force labels while using only 50% of the labels.
- The resulting MLIPs maintain physical fidelity, even with minimal force supervision.
Read more
Beyond Adam: SOAP and Muon for Faster, Label-Efficient Training of Machine Learning Interatomic Potentials
Summary
This paper investigates the impact of optimizer choice on the training of machine learning interatomic potentials (MLIPs), which are crucial for scientific simulations in chemistry and materials science. Traditionally, the MLIP community has relied heavily on the Adam optimizer and its variants, but this study explores the performance of newer matrix-structured optimizers, specifically Muon, SOAP, and a hybrid SOAP-Muon. The authors benchmark these optimizers against AdamW using two significant physical systems: liquid water and the solid acid electrolyte CsH2PO4. The findings reveal that SOAP and SOAP-Muon consistently outperform Adam in terms of convergence speed and final accuracy, particularly under conditions of partial force supervision. The results suggest that the choice of optimizer is a critical yet often overlooked factor in the design of MLIPs, with implications for reducing the reliance on expensive force labels during training. The study demonstrates that the optimized MLIPs maintain physical fidelity, accurately reproducing ab initio calculations and experimental data even with reduced force supervision.
Methodology
The authors integrated Muon, SOAP, and SOAP-Muon optimizers into the NequIP MLIP framework and conducted systematic benchmarking against AdamW. They evaluated the optimizers on two systems of physical significance, assessing their performance under varying levels of force supervision, including energy-only training scenarios.
Results
The study found that SOAP and SOAP-Muon consistently improved both energy and force accuracy while accelerating convergence compared to AdamW. SOAP demonstrated the most robust performance across systems, while SOAP-Muon achieved the best results in specific settings. Notably, SOAP-Muon maintained high accuracy even when trained with only 5% of the force labels, whereas the AdamW model became unstable under similar conditions.
Implications
The findings suggest that adopting advanced optimizers like SOAP and SOAP-Muon can lead to more efficient training of MLIPs, reducing the need for extensive force labels and enabling the development of more accurate models for scientific simulations. This could have significant implications for research in chemistry and materials science, particularly in areas where computational resources are limited.
A More Accurate Algorithm Comparison through A/B Testing using Offline Evaluation Methods
Theory
Optimization
- A/B testing can have a higher algorithm selection error rate than offline evaluation methods.
- The proposed estimator introduces a hypothetical middle algorithm to induce positive correlation.
- The new method improves sample efficiency by achieving the same error rate with half the data.
- Bias-variance analysis supports the advantages of the proposed estimator over traditional methods.
Read more
A More Accurate Algorithm Comparison through A/B Testing using Offline Evaluation Methods
Summary
This paper challenges the conventional wisdom that A/B testing is the gold standard for algorithm selection in online services, revealing that it can produce a higher selection error rate than offline evaluation methods. The authors identify that the sample mean estimator used in A/B testing does not induce positive correlation, which is crucial for minimizing selection errors. In contrast, offline evaluation methods, such as Inverse Propensity Scoring (IPS), generate beneficial correlations through shared data. To address this issue, the authors propose a novel estimator that intentionally induces positive correlation by introducing a hypothetical middle algorithm. This method estimates performance differences in a stepwise manner, leveraging shared data at each step to reduce selection errors. The paper derives the optimal middle algorithm concerning variance and demonstrates the advantages of this approach through bias-variance analysis. Experiments on real-world data show that the proposed estimator achieves the same selection error rate as existing methods while requiring only half of the A/B testing data, indicating a significant improvement in sample efficiency.
Methodology
The authors conducted a pre-experiment comparing A/B testing with offline evaluation methods, specifically using the AVG estimator and the IPS estimator. They proposed a new estimator that incorporates a middle algorithm to induce positive correlation and reduce selection errors, employing shared offline data in a stepwise estimation process.
Results
The experiments demonstrated that the proposed estimator achieved the same selection error rate as existing methods while utilizing only half of the A/B testing data, indicating a twofold improvement in sample efficiency.
Implications
The findings suggest that offline evaluation methods can be more reliable than previously thought, and the proposed estimator could enhance algorithm selection processes in various online services, leading to better decision-making and user experiences.
Efficient Temporal Point Processes via Monotone Alternating Splines
Time Series
Efficient ML
Theory
- Identifies fundamental limitations of Monotone Neural Networks in CCIF modeling.
- Introduces Monotone Alternating Splines as a flexible and efficient alternative.
- Establishes a theoretical foundation for MAS, including generalization error analysis.
- Demonstrates superior performance of MAS on synthetic and real-world datasets.
Read more
Efficient Temporal Point Processes via Monotone Alternating Splines
Summary
This paper addresses the limitations of existing Monotone Neural Networks (MNNs) in modeling Cumulative Conditional Intensity Functions (CCIFs) for Temporal Point Processes (TPPs). The authors identify three structural deadlocks in MNNs: convexity restrictions, saturation limits, and violations of CCIF requirements, which hinder their ability to capture complex temporal dynamics. To overcome these issues, the authors propose a new framework called Monotone Alternating Splines (MAS), which utilizes piecewise monotone splines for interpolation and a globally stable extrapolation component. MAS separates the CCIF into distinct segments, allowing for accurate fitting of complex TPP sequences while ensuring global monotonicity. The paper establishes a theoretical foundation for MAS, analyzing its generalization error and proving its superior fitting capabilities compared to MNNs. Extensive experiments demonstrate that MAS outperforms existing methods on both synthetic and real-world datasets, showcasing its efficiency and flexibility in modeling TPPs.
Methodology
The authors propose Monotone Alternating Splines (MAS) as a new framework for modeling CCIFs. MAS consists of two components: an interpolation component that uses piecewise monotone splines for accurate fitting of TPP sequences, and an extrapolation component that ensures global monotonicity. The theoretical analysis includes decomposing generalization error into interpolation, extrapolation, and complexity errors, and comparing approximation error bounds between MAS and MNNs.
Results
The experiments show that MAS achieves superior performance compared to existing MNN-based methods on both synthetic and real-world datasets, demonstrating its effectiveness in capturing complex temporal dynamics and improving computational efficiency.
Implications
The proposed MAS framework can enhance the modeling of event sequences in various domains such as finance, social networks, and neuroscience, leading to better forecasting and understanding of temporal dynamics.
Self-Gating Attention for Efficient Time Series Forecasting
Time Series
Efficient ML
- Introduction of Self-Gating Attention (SGA) to reduce computational complexity in time series forecasting.
- SGA utilizes a shared learnable matrix for common attention patterns and a residual component for input-specific variations.
- Achieves linear time and memory complexity, making it suitable for resource-constrained environments.
- Demonstrates competitive performance on nine real-world datasets compared to traditional self-attention mechanisms.
Read more
Self-Gating Attention for Efficient Time Series Forecasting
Summary
This paper introduces Self-Gating Attention (SGA), a novel attention mechanism designed to enhance the efficiency of time series forecasting models based on Transformer architectures. Traditional self-attention mechanisms exhibit quadratic time and memory complexity, which can hinder their application in resource-constrained environments. The authors observe that self-attention maps often display redundancy across timestamps due to stable temporal correlations in many real-world time series datasets. To address this, SGA employs a shared learnable matrix to capture common attention patterns and an input-dependent residual component to account for variations specific to each input. This design reduces the complexity of attention score computation to linear time and memory, making it more suitable for high-throughput forecasting systems. The authors integrate SGA into several forecasting backbones and evaluate its performance against standard self-attention and lightweight attention variants across nine diverse datasets, including those from electricity, finance, and climate records. The results demonstrate that SGA not only improves inference efficiency but also maintains competitive forecasting accuracy compared to state-of-the-art methods.
Methodology
The authors propose Self-Gating Attention (SGA), which replaces the standard self-attention mechanism with a shared attention score matrix and an input-dependent residual component. This approach allows for the efficient capture of both common and unique temporal patterns in time series data, reducing the need for redundant computations typically associated with traditional self-attention.
Results
SGA was integrated into various forecasting models and tested on nine publicly available datasets. The results indicate that SGA significantly improves inference efficiency while maintaining competitive forecasting performance compared to standard self-attention and other lightweight attention variants. The MSE results were close to those of traditional methods, confirming the effectiveness of the proposed approach.
Implications
The findings suggest that SGA can be effectively deployed in real-world forecasting applications, particularly in scenarios where computational resources are limited. Its design could facilitate faster and more efficient time series forecasting in various domains, including finance, healthcare, and environmental monitoring.
IonSense-QKG: A Quantum-Readiness Metadata Framework for Lithium-Ion Battery Dataset Discovery
Theory
Optimization
Time Series
- Introduction of IonSense-QKG, a metadata framework for lithium-ion battery datasets.
- Enrichment of dataset metadata with quantum-relevant fields to aid in dataset discovery.
- Development of a Quantum Readiness Score (QRS) for ranking datasets based on their suitability for hybrid quantum-classical machine learning.
- Demonstration of SQL-style queries for efficient dataset discovery.
Read more
IonSense-QKG: A Quantum-Readiness Metadata Framework for Lithium-Ion Battery Dataset Discovery
Summary
The paper introduces IonSense-QKG, a metadata framework designed to enhance the discovery of lithium-ion battery datasets for hybrid quantum-classical machine learning applications. The authors identify the challenges faced by researchers in reusing existing battery datasets due to their heterogeneity in chemistry, modality, scale, and preprocessing complexity. To address these issues, IonSense-QKG enriches the metadata of public battery datasets with quantum-relevant fields such as task type, modality, chemistry, label type, and preprocessing needs. A key feature of this framework is the Quantum Readiness Score (QRS), which ranks datasets based on their suitability for near-term quantum workflows. The authors emphasize that the QRS is not an indication of quantum advantage but serves as a heuristic for dataset prioritization. The paper also demonstrates SQL-style queries for dataset discovery and provides open access to the enriched metadata and associated tools. Overall, IonSense-QKG aims to facilitate reproducible and transparent quantum battery analytics by providing a structured approach to dataset selection.
Methodology
The authors built upon the existing EV-Battery-IonSense index and enriched its metadata with fields relevant to quantum computing. They defined a Quantum Readiness Score (QRS) to rank datasets and demonstrated SQL-style queries for dataset discovery, allowing researchers to efficiently identify suitable datasets for hybrid quantum applications.
Results
The framework successfully categorizes and ranks lithium-ion battery datasets based on their quantum readiness, providing a transparent prioritization heuristic for researchers. The open access to the enriched metadata and associated tools facilitates easier discovery and evaluation of datasets for quantum machine learning tasks.
Implications
IonSense-QKG has the potential to streamline the process of identifying suitable datasets for hybrid quantum-classical machine learning, thereby accelerating research in battery analytics and quantum computing. It supports reproducible research and enhances the accessibility of relevant datasets for researchers in the field.
DeadPool: Resilient LLM Training with Hot-Swapping via Zero-Overhead Checkpoint
Large Language Models
Efficient ML
Optimization
- DEADPOOL enables hot-swapping of failed nodes in LLM training without job termination.
- The system employs an asynchronous in-memory checkpointing strategy to achieve zero overhead during normal execution.
- Recovery from node failures is completed in under 40 seconds, demonstrating high efficiency.
- The approach is evaluated on up to 512 GPUs and 65 billion parameter models, showing scalability.
Read more
DeadPool: Resilient LLM Training with Hot-Swapping via Zero-Overhead Checkpoint
Summary
The paper presents DEADPOOL, a novel fault-tolerance mechanism designed for large language model (LLM) training that addresses the challenges of node failures in distributed GPU environments. Traditional fault-tolerance strategies often incur significant overhead during normal operations or lengthy recovery times when failures occur. DEADPOOL introduces a hot-swapping approach that allows for the replacement of failed compute nodes with spare nodes without terminating the entire training job. This is achieved through two main innovations: an asynchronous in-memory checkpointing mechanism that minimizes overhead during error-free execution and a communicator reconstruction protocol that facilitates real-time node replacement. The authors evaluate DEADPOOL on high-performance computing systems, demonstrating its effectiveness in maintaining zero checkpoint overhead and achieving rapid recovery times of under 40 seconds after node failures. The results indicate that DEADPOOL can significantly enhance the resilience of LLM training processes, making it a promising solution for large-scale distributed training environments.
Methodology
The methodology involves implementing an online hot-swapping recovery mechanism that leverages asynchronous in-memory checkpointing for optimizer state shards. The system also incorporates a distributed communicator reconstruction protocol to replace failed nodes dynamically. The evaluation is conducted on two high-performance computing systems, where node failures are injected into live training jobs to assess the effectiveness of the proposed solutions.
Results
The evaluation results show that DEADPOOL achieves zero overhead during error-free execution and consistently completes recovery from node failures in approximately 40 seconds. This performance is validated across different scales, including up to 512 NVIDIA A100 GPUs and models with up to 65 billion parameters.
Implications
The implications of DEADPOOL are significant for the field of large-scale machine learning, particularly in enhancing the reliability and efficiency of LLM training. By minimizing downtime and computational overhead, DEADPOOL can facilitate more robust training processes in environments where hardware failures are common, potentially leading to faster model development cycles and improved resource utilization.