AI-generated summaries
Today's ML research,
without the noise.
Daily summaries of the latest machine learning papers from arXiv, processed every 8 hours.
66
Papers today
8h
Update frequency
7
Days of history
Similarity search generalisation in contrastive learning with InfoNCE loss
Theory
- Introduces a new continuity bound for InfoNCE loss using Gˆateaux differentiation.
- Shows that the population risk with k negative samples is O(1/k) close to expected cross-entropy.
- Demonstrates that the averaging effect of negative samples stabilizes generalization error as k increases.
- Offers a new perspective on similarity search generalization in contrastive learning.
Read more
Similarity search generalisation in contrastive learning with InfoNCE loss
Summary
This paper investigates the generalization performance of similarity search in contrastive learning, specifically focusing on the InfoNCE loss function. The author demonstrates that the population risk with k negative samples approaches O(1/k) relative to an expected cross-entropy, which measures the deviation between a softmax similarity search using learned embeddings and an idealized search based on the positive sample generator. The study introduces a new continuity bound for the InfoNCE loss derived through Gˆateaux differentiation, which retains the averaging structure over negative samples and incorporates an 'inverse temperature' parameter for tuning. The findings indicate that for Lipschitz embedding functions, the averaging effect of negative samples in InfoNCE loss contributes to stabilizing generalization error as k increases. This work provides a fresh perspective on the theoretical understanding of InfoNCE and its implications for similarity search tasks, moving beyond traditional analyses focused on downstream classification.
Methodology
The author employs theoretical analysis, specifically Gˆateaux differentiation, to derive a continuity bound for the InfoNCE loss. The study also analyzes the relationship between the population risk of InfoNCE and similarity search performance on unseen data, contrasting it with idealized search scenarios.
Results
The paper establishes that the generalization error of similarity search using learned embeddings improves as the number of negative samples increases, with a derived continuity bound that preserves the averaging structure of the InfoNCE loss. This indicates that larger k values lead to better stability in generalization performance.
Implications
The findings have significant implications for the design and training of embedding models in contrastive learning, particularly in applications involving similarity search. The insights can guide practitioners in selecting the number of negative samples to optimize generalization performance.
Pitfalls and Remedies for Multi-Task Bayesian Optimization
Optimization
Theory
- Identified two structural pitfalls in the standard MTGP model that lead to misestimation of task correlations.
- Proposed three remedies to improve correlation estimation in multi-task Bayesian optimization.
- Demonstrated the effectiveness of the remedies through empirical evaluations on synthetic and real-world tasks.
- Highlighted the limitations of existing multi-task optimization models, particularly in cases of negative transfer.
Read more
Pitfalls and Remedies for Multi-Task Bayesian Optimization
Summary
This paper investigates the limitations of Multi-Task Gaussian Processes (MTGP) in Bayesian Optimization (BO), particularly in the context of transfer learning from related tasks. The authors identify two main structural pitfalls that lead to misestimation of cross-task correlations, even in simple cases where transfer learning should succeed. They propose three remedies to address these issues: treating per-task means and scales as model parameters, enforcing non-negative correlations, and co-locating source and target designs. Empirical evaluations demonstrate that these remedies can recover performance levels comparable to target-only baselines in simpler scenarios, although challenges persist in more complex instances. The findings highlight the need for careful consideration of task correlations in multi-task settings and suggest that existing models may not be as effective as previously assumed.
Methodology
The authors conducted controlled experiments comparing the performance of the standard MTGP against their proposed remedies across synthetic multi-task problems and real-world hyperparameter tuning tasks. They analyzed the structural mechanisms causing correlation misestimation and empirically validated their proposed solutions.
Results
The proposed remedies successfully recovered the target-only baseline performance on simpler multi-task problems. However, the broader issues of misestimation persisted in more complex scenarios and across various model variants, indicating that while improvements were made, challenges remain in effectively leveraging multi-task learning.
Implications
The findings suggest that practitioners should be cautious when applying standard MTGP models for transfer learning in Bayesian optimization. The proposed remedies could enhance the effectiveness of multi-task optimization strategies, particularly in scenarios where tasks are closely related. This work encourages further research into the structural aspects of multi-task models to improve their robustness and accuracy.
Fully Trainable Deep Differentiable Logic Gate Networks and Lookup Table Networks
Efficient ML
Theory
- Introduces a novel method for learning connections in LGNs and LUTNs.
- Demonstrates significant performance improvements over fixed-connection networks.
- Achieves high accuracy with drastically fewer gates, enhancing energy efficiency.
- Ensures training stability for deep networks through innovative techniques.
Read more
Fully Trainable Deep Differentiable Logic Gate Networks and Lookup Table Networks
Summary
This paper presents a novel approach for optimizing connections in deep differentiable logic gate networks (LGNs) and lookup table networks (LUTNs). The authors propose a training method that utilizes a probability distribution to select optimal connections for each gate or LUT input pin, allowing for both partial and full optimization of connections while learning the optimal gate types or LUT entries in parallel. The results demonstrate that connection-optimized LGNs outperform traditional fixed-connection LGNs across several benchmarks, including the Yin-Yang, MNIST Handwritten Digits, and Fashion-MNIST datasets, achieving high accuracy with significantly fewer gates. The method achieves 98.92% accuracy on MNIST with two layers of 8000 gates, and 98.45% with one layer, indicating a reduction in the number of gates by nearly a factor of 50 compared to fixed-connection LGNs. The authors also ensure training stability for up to ten layers using high learning rates and straight-through estimators. Additionally, a LUT neuron description is introduced, allowing stable training with backpropagation, and achieving 98.88% accuracy for LUTNs with two layers of 2000 6-input LUTs. Overall, the proposed connection-training algorithm shows promise for improving the efficiency and performance of logic-based neural networks.
Methodology
The authors developed a training method that optimizes connections in LGNs and LUTNs by using a probability distribution to select the best connections for each gate or LUT input pin. This method allows for both partial and full optimization of connections while learning gate types or LUT entries in parallel. The training process incorporates techniques such as high learning rates, straight-through estimators, and trimming constant-output gate types to maintain stability across multiple layers.
Results
The proposed connection-optimized LGNs achieved 98.92% accuracy on the MNIST dataset with two layers of 8000 gates, and 98.45% accuracy with a single layer, requiring nearly 50 times fewer gates than traditional fixed-connection LGNs. The LUTNs achieved an accuracy of 98.88% with two layers of 2000 6-input LUTs, demonstrating the effectiveness of the connection-training algorithm.
Implications
The findings suggest that optimizing connections in logic gate networks can significantly enhance their performance and efficiency, making them more suitable for energy-constrained environments such as edge computing. This approach could lead to advancements in neuromorphic computing and the development of more efficient AI models.
Pattern-Aware Graph Neural Networks for Handling Missing Data
Graph Learning
- Introduces pattern-aware GNNs that incorporate missingness patterns, achieving significant improvements in predictive performance.
- Demonstrates that simple random embeddings can perform comparably to learned embeddings, emphasizing the importance of pattern distinction.
- Shows that attention mechanisms are less critical when pattern information is utilized, with mean aggregation achieving competitive results.
- Finds heterogeneous improvements across datasets, with some showing dramatic gains while others exhibit minimal benefits.
Read more
Pattern-Aware Graph Neural Networks for Handling Missing Data
Summary
This paper addresses the challenge of missing data in real-world datasets, where traditional methods often discard incomplete samples or apply imputation techniques that overlook informative missingness patterns. The authors propose a novel approach using pattern-aware graph neural networks (GNNs) that explicitly encode missingness patterns alongside observed values. They explore four encoding strategies: learned embeddings, frozen random embeddings, statistical features, and hierarchical representations, evaluated across seven UCI datasets with naturally occurring missingness. The results show significant improvements in predictive performance, with an average increase of 17% in balanced accuracy and 22% in F1-macro. Notably, the study finds that even simple random pattern embeddings perform comparably to learned embeddings, indicating that distinguishing between missingness patterns may be more crucial than task-specific optimization. The research also reveals that attention mechanisms are less critical when pattern information is available, as simple mean aggregation yields competitive results. The effectiveness of the proposed methods varies by dataset, highlighting the need for tailored approaches in handling missing data.
Methodology
The authors developed pattern-aware extensions to the GRAPE model, which represents each sample as a bipartite graph. They implemented four different encoding strategies for missingness patterns and evaluated their performance on seven UCI datasets using balanced accuracy and F1-macro as metrics.
Results
The pattern-aware methods achieved an average improvement of 17% in balanced accuracy and 22% in F1-macro across all datasets. The dataset-specific results varied significantly, with the annealing dataset showing an 80% improvement in balanced accuracy, while hepatitis and soybean datasets exhibited minimal gains of 4-5%. Random pattern embeddings achieved a balanced accuracy of 0.650, closely matching the 0.663 of learned embeddings.
Implications
The findings suggest that incorporating missingness patterns can enhance predictive modeling in various domains, particularly in fields like healthcare and social sciences where missing data is prevalent. This approach may lead to more robust models that leverage the informative nature of missingness rather than relying solely on imputation.
AutoMatBench: An Automatic Optimization Toolkit for the Acceleration of Material Properties Prediction Benchmarking
Optimization
- AutoMatBench combines MatBench with OOD performance evaluation to improve benchmarking of MPP models.
- The study reveals significant performance discrepancies across different configurations, necessitating careful evaluation.
- Bayesian optimization in AutoMatBench allows for efficient exploration of configurations, saving computational resources.
- The toolkit provides insights that enhance the understanding of MPP benchmarking beyond existing studies.
Read more
AutoMatBench: An Automatic Optimization Toolkit for the Acceleration of Material Properties Prediction Benchmarking
Summary
This paper introduces AutoMatBench, an automatic optimization toolkit designed to enhance the benchmarking of material property prediction (MPP) models. While MatBench has been a standard tool for evaluating AI models on in-distribution (ID) tasks, it fails to adequately assess out-of-distribution (OOD) performance, which is crucial for discovering new materials. The authors address this gap by integrating MatBench's pipelines with existing OOD evaluation methodologies, allowing for a broader range of benchmarking configurations. They highlight significant discrepancies in model performance across different configurations and emphasize the need for careful consideration of these factors. AutoMatBench employs Bayesian optimization to efficiently explore the configuration space, achieving comparable results to MatBench while reducing computational costs by over 50%. The findings suggest that AutoMatBench not only streamlines the benchmarking process but also uncovers valuable insights into MPP, ultimately aiding in the discovery of new materials.
Methodology
The authors developed AutoMatBench, an automatic optimization toolkit utilizing Bayesian optimization to explore a large configuration space for benchmarking MPP models. They evaluated five general-purpose prediction models across six material property tasks, analyzing the impact of various configuration factors on model performance.
Results
The experiments demonstrated that AutoMatBench could identify optimal configurations within twelve optimization steps, yielding results consistent with MatBench and prior OOD research while saving over half the computational cost. The study also revealed new insights into MPP benchmarking that were not previously reported.
Implications
AutoMatBench has the potential to significantly reduce the time and resources required for benchmarking MPP models, facilitating faster material discovery and optimization. Its methodologies may also be applicable to other fields within AI for Science (AI4S), enhancing the efficiency of model evaluation and deployment.
COAST: Context-Aware Differential Learning for Gene Expression Prediction in Spatial Transcriptomics
Computer Vision
- COAST addresses the limitations of existing methods by focusing on relative expression relationships rather than just absolute values.
- The framework utilizes a Transformer encoder to effectively integrate local and global context features.
- Training involves a joint objective that combines absolute and signed differential regression, enhancing the model's ability to capture spatial relationships.
- Experimental results show significant improvements in prediction accuracy and clinically relevant outcomes across multiple datasets.
Read more
COAST: Context-Aware Differential Learning for Gene Expression Prediction in Spatial Transcriptomics
Summary
The paper presents COAST, a novel context-aware differential learning framework aimed at predicting gene expression from H&E stained histopathology images, addressing the limitations of current spatial transcriptomics (ST) methods. Traditional ST approaches focus on absolute expression levels, which can be sensitive to variability and do not effectively utilize the relative expression relationships between spatially adjacent spots. COAST integrates local and global context features through type-specific modulation and employs a Transformer encoder to capture both fine-grained local patterns and broader tissue structures. The framework is trained with a joint objective that combines absolute expression regression with signed differential regression, allowing it to explicitly supervise spatial relationships. Experimental results across multiple ST datasets demonstrate that COAST consistently outperforms existing methods in correlation and distribution metrics, while also retaining clinically relevant prognostic information for downstream survival predictions, highlighting the practical significance of context-aware differential learning in histology-based gene expression prediction.
Methodology
COAST employs a four-component architecture: a feature extractor for image patches, context-specific feature modulation for adaptive conditioning, a spatio-relational Transformer for aggregation, and separate prediction heads for absolute and differential expression. The model is trained using a joint objective that combines regression tasks to optimize both absolute expression values and relative differences between target and context spots.
Results
The experiments conducted on various spatial transcriptomics datasets indicate that COAST achieves consistent improvements in correlation and distribution-based metrics compared to existing baselines. The reconstructed gene expression representations also maintain clinically meaningful prognostic information, which is crucial for downstream survival prediction tasks.
Implications
The COAST framework has the potential to significantly enhance the prediction of gene expression from histopathology images, making spatial transcriptomics more accessible and practical for clinical applications. By leveraging context-aware differential learning, it could facilitate better understanding of tissue microenvironments and their impact on gene expression, ultimately aiding in precision medicine and cancer research.
Reinforcement Learning with Verifiable Physics: Post-training LLMs with Continuous Rewards
Reinforcement Learning
Large Language Models
Theory
- Introduction of RLVP, a framework combining reinforcement learning with verifiable physics for PDE solver generation.
- Development of a hybrid verifier that integrates binary and continuous rewards to enhance model training.
- Demonstration of effective multi-PDE training, allowing a single model to handle diverse PDE families.
- Smaller models trained with RLVP outperform larger models using traditional prompting techniques.
Read more
Reinforcement Learning with Verifiable Physics: Post-training LLMs with Continuous Rewards
Summary
This paper presents Reinforcement Learning with Verifiable Physics (RLVP), a novel framework for post-training large language models (LLMs) to generate code for solving partial differential equations (PDEs). Traditional methods for PDE solving often rely on expert knowledge and are labor-intensive, while existing LLM approaches primarily focus on inference-time improvements. RLVP addresses the limitations of binary verifiers in reinforcement learning by introducing a hybrid verifier that combines hard program-validity checks with continuous rewards based on solution accuracy and PDE-residual consistency. The authors curate a diverse dataset of PDE families and demonstrate that a single policy can be effectively trained across multiple PDE types, leading to significant improvements in solver generation. The results indicate that smaller models trained with RLVP can outperform larger models using traditional prompting methods, showcasing the potential for zero-shot transfer to unseen PDEs. This work not only enhances the reliability of PDE solvers generated by LLMs but also provides a reproducible post-training pipeline for broader scientific computing applications.
Methodology
The authors employ a reinforcement learning approach with a hybrid verifier that includes hard execution validity checks and continuous rewards based on the accuracy of generated solutions and their consistency with physical laws. The training process involves fine-tuning a single policy across multiple PDE families using a curated dataset of numerical schemes.
Results
The RLVP framework shows an 8-13 percentage point improvement over binary-validity-only baselines and a 38% reduction in normalized root mean square error (nRMSE). The trained policy demonstrates compositionality in numerical motifs, successfully generating solvers for previously unseen PDE problems.
Implications
The RLVP framework has the potential to automate and improve the reliability of PDE solver generation, making it accessible to a broader audience without requiring extensive domain knowledge. This could significantly impact scientific computing, enabling more efficient and accurate modeling in various fields such as fluid dynamics and structural mechanics.
CatRetriever: Contrastive Representation Learning for Slab-to-Bulk Retrieval in Generative Catalyst Discovery
Generative Models
- CatRetriever effectively addresses the slab-to-bulk retrieval problem in catalyst discovery.
- The model achieves high retrieval accuracy, significantly improving the efficiency of identifying bulk structures from slab representations.
- An end-to-end pipeline is developed that combines slab-to-bulk retrieval with generative models and adsorption energy evaluations.
- The approach allows for exploration beyond fixed databases, enhancing the scope of potential catalyst candidates.
Read more
CatRetriever: Contrastive Representation Learning for Slab-to-Bulk Retrieval in Generative Catalyst Discovery
Summary
The paper introduces CatRetriever, a novel contrastive representation learning model designed to address the slab-to-bulk retrieval problem in catalyst discovery. Traditional methods in heterogeneous catalysis often focus on slab-level structures, neglecting the corresponding bulk structures essential for evaluating material properties. CatRetriever aligns slab and bulk crystal representations in a shared latent space, enabling accurate retrieval of bulk candidates from slab queries. The model achieves impressive retrieval accuracy, with R@1 > 91% and R@3 > 98% on evaluation sets. Furthermore, the authors extend CatRetriever into a comprehensive pipeline that integrates bulk retrieval, generative search space expansion, and adsorption energy analysis, facilitating the discovery of catalysts that are structurally compatible and meet specific adsorption energy criteria. This approach not only enhances the efficiency of catalyst discovery but also bridges the gap between surface generative models and bulk material properties, providing a scalable solution for real-world applications in catalyst design.
Methodology
CatRetriever employs contrastive representation learning to encode and project bulk crystals and catalyst surfaces into a shared embedding space. The model learns to bring matched slab-bulk pairs closer together while optimizing for structural and chemical correspondence. A two-stage search strategy is introduced, combining database retrieval with generative modeling to explore bulk candidates beyond existing datasets.
Results
CatRetriever demonstrates a top-1 retrieval accuracy of approximately 92% on the Materials Project dataset. The model's retrieval capabilities are validated on both in-distribution and holdout evaluation sets, achieving R@1 > 91% and R@3 > 98%. The integrated pipeline allows for effective candidate evaluation based on structural compatibility and targeted adsorption energy ranges.
Implications
The development of CatRetriever has significant implications for the field of catalyst discovery, enabling more efficient exploration of chemical spaces and facilitating the design of catalysts with desired properties. By connecting surface generative models with bulk material characteristics, this approach can accelerate the discovery of new materials for energy conversion processes.
Weight-Adjusted Gradients Reveal Parameter Importance and Failure Modes in LLMs
Large Language Models
Interpretability
Efficient ML
- WAG combines zeroth-order and first-order information to estimate parameter importance effectively.
- It identifies a critical subset of parameters that cause significant performance degradation when perturbed.
- WAG reveals the previously unexplored relationship between weights and gradients in LLMs.
- The method is computationally efficient and applicable across various practical scenarios.
Read more
Weight-Adjusted Gradients Reveal Parameter Importance and Failure Modes in LLMs
Summary
This paper introduces Weight-Adjusted Gradients (WAG), a novel approach for estimating parameter importance in Large Language Models (LLMs) by capturing the interaction between model weights and first-order gradient information. The authors argue that understanding which parameters significantly influence model behavior is crucial for enhancing the efficiency, reliability, and interpretability of LLMs. WAG identifies a small subset of parameters whose perturbation leads to substantial performance degradation, including failure modes that existing metrics fail to detect. The findings suggest that the interplay between weights and gradients is critical for understanding model behavior, leading to new questions about the role of different types of information in deep learning. The practical applications of WAG are demonstrated in various contexts, such as expert allocation in mixture-of-expert architectures, parameter-specific unlearning, mixed-precision quantization, and layer selection for knowledge editing. Overall, WAG serves as a unified tool for analyzing and controlling LLMs, paving the way for more principled model-level interpretations.
Methodology
The authors propose WAG as a metric that integrates both weight magnitudes and gradient information to assess parameter importance. This approach allows for efficient computation during standard training or evaluation without the need for second-order information or retraining, making it scalable to large models.
Results
The empirical analysis shows that WAG successfully identifies a small number of parameters that, when modified, lead to significant performance degradation and failure modes, which are often overlooked by traditional importance metrics. The results highlight the importance of considering both weight and gradient information together.
Implications
WAG has practical implications for improving the interpretability and reliability of LLMs. It can be utilized in various applications, including optimizing expert allocation in mixture-of-expert architectures, facilitating targeted unlearning, enabling mixed-precision quantization, and guiding knowledge editing processes.
TSAI-MetaFraud: A Benchmark Dataset for Financial Fraud Transaction and Behavioral Risk Detection in Metaverse Ecosystems
Graph Learning
Multimodal
- Introduction of TSAI-MetaFraud, a comprehensive dataset for fraud detection in metaverse environments.
- Integration of behavioral, transactional, and graph-structured data to reflect real-world complexities.
- Establishment of benchmark tasks for systematic evaluation of fraud detection methods.
- Baseline evaluations provided using machine learning and graph neural network approaches.
Read more
TSAI-MetaFraud: A Benchmark Dataset for Financial Fraud Transaction and Behavioral Risk Detection in Metaverse Ecosystems
Summary
The paper introduces TSAI-MetaFraud, a novel multimodal benchmark dataset designed to address the challenges of fraud detection in metaverse ecosystems. As virtual economies evolve, they face unique risks related to fraud and automated bot activity, necessitating a comprehensive dataset that integrates behavioral, transactional, and graph-structured information. TSAI-MetaFraud captures a wide range of activities within a metaverse environment, including avatar interactions, transaction records, and biometric attributes, while simulating both benign and malicious behaviors. The authors define several benchmark tasks such as transaction fraud detection, cross-modal node classification, and temporal link prediction, providing baseline evaluations using machine learning models and graph neural networks. This dataset aims to facilitate reproducible research and advance the fields of multimodal learning, graph mining, and fraud analytics in the context of emerging metaverse technologies.
Methodology
The authors collected data from a metaverse environment built using OpenSimulator, capturing various aspects of user behavior and financial transactions. They defined benchmark tasks for fraud detection and provided baseline results using machine learning models and graph neural networks to evaluate performance.
Results
The TSAI-MetaFraud dataset was successfully created, incorporating realistic scenarios of both benign and malicious activities. Baseline evaluations demonstrated the dataset's utility for testing various fraud detection methodologies, highlighting the challenges posed by the integrated multimodal data.
Implications
The TSAI-MetaFraud dataset has significant implications for researchers and practitioners in the fields of fraud detection, trustworthy AI, and multimodal learning. It provides a foundation for developing advanced detection techniques that can adapt to the complexities of virtual economies, ultimately enhancing security and trust in metaverse platforms.
Understanding Schedule-Free Methods in Nonconvex Optimization: Rate Guarantees and Escaping Saddles
Optimization
Theory
- Schedule-Free methods eliminate the need for handcrafted learning rate schedules, simplifying optimization in machine learning.
- The paper establishes optimal worst-case convergence rates for SF-GD and SF-SGD in nonconvex optimization.
- A Lyapunov function is constructed to analyze the continuous-time limiting ODE of SF-GD, demonstrating an O(1/T) decay rate for the squared gradient norm.
- SF-GD is shown to avoid strict saddle points with a small one-time perturbation, ensuring convergence to local minimizers.
Read more
Understanding Schedule-Free Methods in Nonconvex Optimization: Rate Guarantees and Escaping Saddles
Summary
This paper investigates Schedule-Free methods in nonconvex optimization, addressing a significant gap in the convergence theory of these methods, which have shown strong empirical performance in machine learning tasks. The authors provide worst-case analyses for Schedule-Free gradient descent (SF-GD) and Schedule-Free stochastic gradient descent (SF-SGD) without requiring auxiliary modifications or restrictive conditions. They utilize a Lyapunov analysis based on the continuous-time limiting ordinary differential equation associated with these methods to demonstrate that both SF-GD and SF-SGD achieve optimal worst-case convergence rates among first-order methods. Additionally, the authors formulate SF-GD as a nonautonomous dynamical system and prove that it can avoid strict saddle points under small perturbations. This work enhances the understanding of the strong performance of Schedule-Free methods and establishes their theoretical foundations in nonconvex settings, which are prevalent in modern machine learning applications.
Methodology
The authors analyze Schedule-Free methods through worst-case convergence analysis, utilizing Lyapunov functions and continuous-time limiting ordinary differential equations. They also apply the performance estimation problem (PEP) framework to study the behavior of averaged evaluation sequences.
Results
The paper proves that SF-GD and SF-SGD achieve optimal convergence rates for first-order methods in nonconvex settings. It also establishes that SF-GD can avoid strict saddle points under small perturbations, ensuring convergence to local minima. The results highlight the strong empirical performance of Schedule-Free methods while providing theoretical guarantees.
Implications
The findings suggest that Schedule-Free methods can be reliably used in nonconvex optimization problems commonly encountered in machine learning, potentially leading to more robust and efficient training algorithms without the complexities of learning rate scheduling.
SciML in the Wild: A Diagnostic Study of When Structural Priors Help and When They Hurt
Time Series
- Heuristic structural priors in PINN and UDE models often degrade predictive performance compared to less-constrained models like ARIMA and NODE.
- The study identifies four failure modes of physics-informed learning under non-stationarity.
- Empirical testing of structural priors is crucial in domains with uncertain or evolving governing equations.
- The effectiveness of SciML methods is highly dependent on the alignment of structural priors with the data-generating process.
Read more
SciML in the Wild: A Diagnostic Study of When Structural Priors Help and When They Hurt
Summary
This paper investigates the effectiveness of Scientific Machine Learning (SciML) methods, particularly in the context of macroeconomic forecasting, where structural priors may not align with the underlying data-generating processes. The authors conduct a comprehensive empirical study across 23 countries, comparing five model families: ARIMA, LSTM, Neural Ordinary Differential Equations (NODE), Physics-Informed Neural Networks (PINNs), and Universal Differential Equations (UDEs). The study reveals that none of the models consistently achieve strong forecasting performance, particularly due to the challenges posed by low-frequency data and regime shifts. Notably, less-constrained models like ARIMA and NODE outperform more-constrained models such as PINN and UDE. The authors argue that structural priors can act as misregularizers when they do not match the data dynamics, leading to four identified failure modes: prior misalignment, regime shifts, structural breaks, and optimization instability. The findings suggest that practitioners should empirically test the utility of structural priors rather than assuming their benefits, emphasizing the need for flexibility in modeling approaches in non-stationary environments.
Methodology
The authors conducted a large-scale empirical evaluation using sparse annual data from 23 countries, applying a unified experimental framework to compare the forecasting performance of ARIMA, LSTM, NODE, PINN, and UDE models across multiple temporal splits and random seeds.
Results
The results indicate that no model consistently excels in forecasting performance, with ARIMA and NODE showing relative superiority over PINN and UDE. The identified failure modes highlight the risks associated with misaligned structural priors in macroeconomic contexts.
Implications
The findings suggest that in macroeconomic forecasting, where data is sparse and non-stationary, flexibility in model structure may be more beneficial than rigid adherence to heuristic priors. This has implications for the design and application of SciML methods in various domains.
Manifold Constrained Tabular Deep Neural Networks
Theory
Efficient ML
Graph Learning
- Introduction of Latent Decision Nodes (LDNs) for unified feature representation.
- Utilization of hyperbolic geometry to model hierarchical decision structures.
- Soft Decision Routing mechanism for differentiable range discretization of numerical features.
- Entropy-aware Capacity Allocation algorithm to balance model complexity and expressiveness.
Read more
Manifold Constrained Tabular Deep Neural Networks
Summary
This paper addresses the challenges faced by tabular deep neural networks (DNNs) in modeling discrete, condition-triggered rules inherent in tabular data. Traditional DNNs operate under Euclidean assumptions, which can lead to inefficiencies in representing the hierarchical decision structures typical of tabular classification tasks. The authors propose HDE-Net, a manifold-constrained DNN that utilizes hyperbolic geometry to better capture these structures. The framework introduces Latent Decision Nodes (LDNs) to unify heterogeneous feature representations and employs Soft Decision Routing for numerical features to approximate range-based local rules in a differentiable manner. An entropy-aware capacity allocation algorithm is also introduced to adaptively assign LDNs based on the information density of numerical features. The proposed model is evaluated on the TALENT-tiny-core classification benchmark, demonstrating superior performance compared to both Gradient-Boosted Decision Trees (GBDTs) and existing tabular DNNs, achieving the best average rank across 30 datasets while maintaining high efficiency.
Methodology
The authors developed HDE-Net, which incorporates Latent Decision Nodes (LDNs) to abstract heterogeneous features into a unified representation. Numerical features are processed using a Soft Decision Routing mechanism that mimics decision tree logic in a differentiable manner. The LDNs are embedded in the Poincaré ball to leverage hyperbolic geometry for hierarchical decision modeling. An entropy-aware capacity allocation algorithm is employed to optimize the number of LDNs based on the information density of features.
Results
HDE-Net achieved the best average rank on the TALENT-tiny-core classification benchmark across 30 datasets, outperforming both industrial GBDTs and other recent tabular DNNs. The results indicate that the proposed model effectively captures the discrete, rule-partitioned structures of tabular data while maintaining high efficiency.
Implications
The findings suggest that hyperbolic geometry can significantly enhance the modeling of tabular data, making HDE-Net a promising approach for applications in fields such as finance and healthcare where tabular data is prevalent. The unified decision-centric framework may also inspire further research into geometric deep learning for other types of structured data.
Neural Collapse Is Forbidden: Information Floors in Language Models
NLP
Large Language Models
Theory
- Within-class variance in language models is an allocation of information storage, not an indication of incomplete neural collapse.
- 79-91% of representational variance is attributed to within-token context variability, while macro-category structure accounts for only 4-12%.
- Token-level weight decay penalizes categories based on type count, affecting next-token prediction as an imbalanced K-class problem.
- A converse floor for binary categories establishes a minimum dispersion proportional to conditional mutual information.
Read more
Neural Collapse Is Forbidden: Information Floors in Language Models
Summary
This paper challenges the conventional understanding of within-class variance in language model representations, arguing that it reflects an allocation of information storage rather than an incomplete neural collapse. The author presents a framework that identifies a law governing this allocation, demonstrating that within-token context variability accounts for 79-91% of representational variance across various models, while macro-category structure contributes only 4-12%. The study employs a dimensionless decomposition of variance and establishes that token-level weight decay influences category representation based on type count rather than occurrence mass. A converse floor is proved for binary categories, indicating that within-category dispersion must be proportional to the conditional mutual information between tokens and context given the category. The findings reveal that identity dispersion tracks model-realized conditional information across multiple models, suggesting that the dynamics of language model training involve reallocating representational variance rather than collapsing it. The paper concludes by emphasizing the importance of understanding the informational aspects of language models, which could reshape future research and applications in the field.
Methodology
The study utilizes a dimensionless three-level decomposition of representational variance across 14 language models, applying statistical analysis to measure within-class variance and its components. It also employs a theoretical framework to prove the relationship between within-category dispersion and conditional mutual information.
Results
The analysis shows that within-token context variability dominates representational variance, and identity dispersion tracks conditional information with high correlation across different models. The findings indicate that the dynamics of language model training involve reallocating representational variance rather than achieving a state of neural collapse.
Implications
This research has significant implications for understanding the behavior of language models during training, suggesting that future work should focus on the informational aspects of model representations. It may influence the design of more effective training strategies and architectures for language models.
All you need is SAMPAT
Theory
Interpretability
Efficient ML
- SAMPAT provides a fully interpretable neural architecture using algebraic expressions.
- It can approximate any smooth function with a three-layer structure.
- The architecture allows for optimization of both model parameters and structure.
- SAMPAT demonstrates competitive performance on various datasets.
Read more
All you need is SAMPAT
Summary
The paper introduces SAMPAT (Smooth Approximation via Multivariate Polynomials and Analytic Transformations), a novel three-layer neural architecture designed to enhance interpretability in machine learning models. Unlike traditional deep neural networks, which often operate as black boxes, SAMPAT provides a closed-form algebraic expression that allows for complete interpretability of the learned functions. The architecture is capable of approximating any smooth function arbitrarily closely, leveraging its ability to represent polynomials, rational expressions, and mixtures of functions. The authors demonstrate through experiments on synthetic and benchmark datasets that SAMPAT achieves competitive performance compared to simpler models, with a two-layer configuration often being sufficient for many tasks. The architecture also allows for the optimization of not just model parameters but also the model structure itself, particularly when skip connections are introduced. This flexibility enables SAMPAT to represent a wide range of methods commonly used in AI/ML, making it a versatile tool for both modeling and analysis.
Methodology
The SAMPAT architecture consists of three layers where the first layer uses logarithmic activation functions, the second layer employs exponential functions, and the third layer combines outputs from the second layer to form multivariate polynomials. The architecture is designed to ensure that it can approximate any continuous function through appropriate weight selection, leveraging the extended Stone-Weierstrass Theorem.
Results
Experiments conducted on synthetic and benchmark datasets show that SAMPAT achieves performance comparable to simpler models while maintaining interpretability. The two-layer version of SAMPAT is often sufficient for many tasks, indicating its efficiency and effectiveness.
Implications
SAMPAT's ability to provide interpretable models could significantly benefit fields requiring transparency in AI/ML applications, such as healthcare, finance, and scientific research. Its flexibility in representing various mathematical forms also opens avenues for advanced modeling of nonlinear systems.
Statistically Undetectable Backdoors in Deep Neural Networks
Theory
- Backdoors can be implanted in DNNs that are statistically undetectable in white-box settings.
- The presence of a backdoor allows for the creation of adversarial examples that are not possible to generate without it.
- The study highlights a significant power asymmetry between model trainers and users, complicating trust in machine learning services.
- The authors provide a theoretical framework for understanding the construction and implications of these backdoors.
Read more
Statistically Undetectable Backdoors in Deep Neural Networks
Summary
This paper explores the potential for adversarial model trainers to implant statistically undetectable backdoors in deep feedforward neural networks (DNNs). The authors demonstrate that these backdoors can remain hidden even in a white-box setting, where the model's weights and architecture are fully known. The backdoor allows for the generation of invariance-based adversarial examples, which can cause significant misclassifications without detection. The authors establish that without the backdoor, creating such adversarial examples is computationally infeasible under standard cryptographic assumptions. The findings reveal a critical power imbalance between model trainers and users, raising concerns about the trustworthiness of outsourced machine learning services. The paper outlines the construction of backdoors using Gaussian matrices and discusses the implications of this asymmetry in the context of model training and deployment.
Methodology
The authors develop a theoretical framework for constructing backdoors in DNNs using Gaussian matrices. They analyze the properties of these backdoors in terms of total variation distance and demonstrate the existence of invariance-based adversarial examples. The methodology includes both theoretical proofs and preliminary empirical experiments to validate the effectiveness of the backdoor approach.
Results
The paper presents theoretical results showing that backdoored models can be statistically indistinguishable from honestly trained models, even when all model parameters are known. Empirical results support the theoretical claims, demonstrating the feasibility of generating adversarial examples through the backdoor while maintaining a low total variation distance from the original model.
Implications
The findings suggest that machine learning practitioners and organizations must be vigilant about the integrity of models obtained from third-party providers. The ability to embed undetectable backdoors poses risks for security-sensitive applications, necessitating the development of robust verification methods to ensure model reliability. Additionally, the research opens avenues for exploring secure model training practices and the potential for using backdoors as authentication mechanisms.
Adaptive Bayes exactly tracks information over intrinsic time
Theory
Optimization
Reinforcement Learning
- Introduces an exact information-accounting identity for Bayesian updates.
- Demonstrates two exact adaptive decompositions of cumulative regret.
- Establishes that favorable learning conditions are intrinsic to the sequence rather than approximations.
- Applies the framework to a wide range of learning scenarios, including bandits and online optimization.
Read more
Adaptive Bayes exactly tracks information over intrinsic time
Summary
This paper presents a novel framework for understanding Bayesian and multiplicative-weights updates in the context of sequential feedback. The author establishes an exact information-accounting identity that relates the learner's excess loss to a chosen comparator to both an immediate payment for uncertainty and a reduction in the information distance to that comparator. This leads to two exact adaptive decompositions of cumulative regret, applicable across various learning paradigms, including Bayesian model averaging, contextual bandits, and online convex optimization. The results indicate that favorable learning conditions manifest as intrinsic properties of the realized sequence rather than as slack in worst-case scenarios. The paper emphasizes the role of information as a common currency in these updates, allowing for a unified understanding of diverse learning algorithms.
Methodology
The methodology involves deriving a one-step identity that captures the relationship between the learner's distribution, the composite losses incurred, and the information distance to a comparator. The author explores two adaptive update strategies based on this identity, allowing for a comprehensive analysis of various learning algorithms through the lens of information theory.
Results
The paper provides exact decompositions of cumulative regret for both prior-retempered and local updates, showing that the cumulative regret can be expressed as a sum of intrinsic-time payments and drifts due to changing temperatures. This leads to a deeper understanding of how different learning scenarios can be unified under a single framework.
Implications
The findings have significant implications for the design and analysis of adaptive learning algorithms, suggesting that understanding the intrinsic properties of learning sequences can lead to more robust and efficient algorithms across various applications, including reinforcement learning and online optimization.
Beyond Euclidean Clipping: Overcoming Exploration Collapse in LLM RL via Riemannian Isometric Policy Optimization
Reinforcement Learning
Large Language Models
Optimization
- Identifies the geometric mismatch in PPO-Clip as the root cause of exploration collapse in LLM RL.
- Proposes Riemannian Isometric Policy Optimization (RIPO) to correct the geometric flaws of PPO-Clip.
- Demonstrates that RIPO allows for balanced exploration and exploitation, enhancing model performance.
- Achieves significant improvements in various benchmarks, showcasing the efficacy of the proposed method.
Read more
Beyond Euclidean Clipping: Overcoming Exploration Collapse in LLM RL via Riemannian Isometric Policy Optimization
Summary
This paper addresses a critical limitation in reinforcement learning (RL) algorithms for large language models (LLMs), specifically the exploration collapse associated with Proximal Policy Optimization with Clipping (PPO-Clip). The authors identify that the fundamental flaw of PPO-Clip arises from its reliance on a Euclidean metric to measure policy discrepancy, which is incompatible with the intrinsic Riemannian geometry of the policy manifold. This mismatch leads to overly conservative updates in low-probability regions and aggressive updates in high-probability regions, ultimately causing exploration collapse. To mitigate this issue, the authors propose Riemannian Isometric Policy Optimization (RIPO), which ensures isometric updates on the Riemannian manifold, thereby balancing exploration and exploitation. RIPO dynamically adapts the clipping boundary to the local Riemannian geometry, allowing for larger updates for low-probability actions while constraining updates for high-probability actions. The paper demonstrates that RIPO achieves a favorable bias-variance trade-off, leading to more stable optimization. Extensive experiments show that RIPO significantly outperforms existing RL algorithms across seven competition-level benchmarks, achieving up to 60% improvement over Group Relative Policy Optimization (GRPO) on the AIME24 benchmark.
Methodology
The authors propose Riemannian Isometric Policy Optimization (RIPO), which reinterprets policy updates on the Riemannian manifold. RIPO dynamically adjusts the clipping boundary based on local Riemannian geometry, allowing for larger updates in low-probability regions and constraining updates in high-probability regions. This approach is theoretically grounded and aims to stabilize optimization while balancing exploration and exploitation.
Results
RIPO was tested across seven competition-level benchmarks, showing significant performance improvements, with up to 60% relative improvement over GRPO on the AIME24 benchmark. The experiments validate the effectiveness of RIPO in overcoming exploration collapse and enhancing reasoning performance in LLMs.
Implications
The findings suggest that addressing geometric mismatches in RL algorithms can lead to more effective training of LLMs, particularly in complex decision-making tasks. RIPO's approach may inspire further research into geometric considerations in reinforcement learning, potentially leading to advancements in various applications requiring long-horizon reasoning.
Bandit PCA with Minimax Optimal Regret
Theory
Optimization
- Establishes a minimax optimal regret bound for Bandit PCA of order r√dT.
- Introduces a novel algorithm combining online mirror descent and multiscale exploration.
- Constructs an adaptive adversary to demonstrate the lower bound of regret.
- Connects Bandit PCA to quantum tomography, highlighting its relevance in quantum state estimation.
Read more
Bandit PCA with Minimax Optimal Regret
Summary
This paper investigates the bandit-feedback version of online principal component analysis (Bandit PCA), where the learner selects unit vectors in response to a sequence of symmetric gain matrices chosen by an adversary. The objective is to minimize regret against the best unit vector in hindsight. The authors improve upon previous algorithms by establishing a minimax regret bound of order r√dT, bridging the gap between existing upper and lower bounds. The proposed algorithm employs online mirror descent on the spectrahedron of density matrices combined with a multiscale exploration strategy. Additionally, the authors construct an adaptive adversary to demonstrate the lower bound, linking the problem to subspace estimation. The findings also draw connections to adaptive-measurement quantum tomography, suggesting broader implications for quantum state estimation.
Methodology
The authors develop a new algorithm that utilizes online mirror descent on the spectrahedron of density matrices, incorporating a multiscale exploration scheme that updates eigenspaces at varying rates based on their spectral magnitudes. They also construct an adaptive adversary to refine the hidden reward subspace, which is crucial for establishing the lower bound of regret.
Results
The paper presents a minimax regret bound of order r√dT, improving upon the previous algorithm's O(d√rT log T) bound. The authors successfully demonstrate that low regret is unattainable without estimating the subspace, thus linking the regret analysis to subspace estimation challenges.
Implications
The findings have significant implications for online learning and decision-making in environments where feedback is limited, particularly in applications related to quantum state estimation and adaptive measurement strategies. The results may influence future research in both classical and quantum machine learning frameworks.
MawForge: Memory-Bounded Expert Materialization for Local Mixture-of-Experts Inference
NLP
Large Language Models
Efficient ML
- MawForge enables practical local inference for large MoE models on memory-constrained devices.
- The split-pack architecture allows for on-demand materialization of expert tensors, reducing memory requirements.
- Validation results show that MawForge can serve large models without exceeding memory limits, with a focus on cache behavior.
- Larger expert caches improve hit rates but may negatively impact throughput due to increased memory pressure.
Read more
MawForge: Memory-Bounded Expert Materialization for Local Mixture-of-Experts Inference
Summary
MawForge presents a novel approach to local inference for Sparse Mixture-of-Experts (MoE) language models, addressing the challenge of memory constraints on consumer-grade hardware. Traditional local inference systems require the entire model, including key-value caches and runtime buffers, to fit into fast memory, which is often impractical. MawForge proposes a split-pack architecture that allows the full model to reside on disk while keeping frequently used tensors in memory and materializing expert tensors on demand. The paper details a comprehensive validation conducted on a MacBook Pro M5 Pro with 24 GB of unified memory, demonstrating that MawForge can effectively serve large GGUF MoE models within a constrained memory budget. The results indicate that while larger expert caches improve hit rates and reduce materialized bytes, they can also lead to increased memory pressure and reduced throughput. The findings emphasize the importance of balancing expert reuse and memory footprint, leading to insights on cache behavior and speculative decoding in MoE serving.
Methodology
The methodology involves a split-pack architecture for local MoE serving, where the full model is stored on disk, common tensors are kept in memory, and routed expert tensors are materialized into a bounded execution cache as needed. A validation matrix was created to test various configurations, including model profiles, cache settings, and context lengths, on a specific hardware setup.
Results
The validation matrix comprised 600 rows across different configurations, with MawForge successfully completing all statically feasible generation tasks under an 18 GiB memory target. The results showed that MawForge could serve large models like Qwen3.6 and Gemma 4 effectively, while also revealing the non-monotonic relationship between cache size and performance metrics such as throughput and memory pressure.
Implications
MawForge's approach could significantly enhance the deployment of large language models on consumer hardware, making advanced NLP applications more accessible. Its findings on cache behavior and memory management may inform future research and development in efficient model serving and inference optimization.
A Novel Graph Fraud Detector via Grouped Attribute Completion and Confidence-Aware Contrastive Learning
Graph Learning
- Introduces GFD-GC framework to tackle incomplete node attributes and class imbalance in graph fraud detection.
- Develops a grouped attribute completion module for effective feature recovery by capturing heterogeneous neighborhood semantics.
- Implements a confidence-aware contrastive learning strategy to enhance fraud representation learning using pseudo-fraud nodes.
- Demonstrates superior performance of GFD-GC over existing state-of-the-art methods in extensive experiments.
Read more
A Novel Graph Fraud Detector via Grouped Attribute Completion and Confidence-Aware Contrastive Learning
Summary
This paper addresses the challenges of graph fraud detection, particularly the issues of incomplete node attributes and extreme class imbalance, which hinder the performance of existing Graph Neural Network (GNN)-based detectors. The authors propose a novel framework named GFD-GC (Graph Fraud Detection with Grouped attribute completion and Confidence-aware Contrastive learning). GFD-GC employs a group-wise aggregation method that mimics heterogeneous neighborhood structures to recover informative node features, thereby enhancing the quality of node representations. Additionally, it introduces a confidence-aware supervised contrastive learning strategy that generates high-confidence pseudo-fraud nodes to supplement the limited labeled fraud nodes. This dual approach aims to improve both feature recovery and representation discrimination. The experimental results demonstrate that GFD-GC significantly outperforms state-of-the-art methods on various real-world datasets, indicating its effectiveness in practical fraud detection scenarios.
Methodology
The GFD-GC framework consists of two main components: (1) a grouped attribute completion module that aggregates information from heterogeneous neighborhoods to recover node features, and (2) a confidence-aware contrastive learning strategy that generates high-confidence pseudo-fraud nodes to augment the training set, improving the representation of fraud nodes in the context of extreme class imbalance.
Results
The experimental evaluations reveal that GFD-GC consistently outperforms existing graph fraud detection baselines across multiple real-world datasets, showcasing its robustness and effectiveness in enhancing fraud detection capabilities.
Implications
The proposed framework has significant implications for real-world applications in various domains such as online payments, e-commerce, and social networks, where fraud detection is critical for maintaining security and integrity. GFD-GC can be applied to improve the reliability of fraud detection systems by effectively addressing the challenges of incomplete data and class imbalance.
Hierarchical Bayesian Quadrature
Theory
Efficient ML
Optimization
- Hierarchical Bayesian Quadrature (HBQ) adapts to nonstationary integrands using a tree-based partitioning approach.
- The method combines local integral estimates through a hierarchical Gaussian process conditioning, enhancing correlation across subdomains.
- HBQ outperforms standard Bayesian Quadrature on nonstationary integrands while matching its performance on stationary ones.
- The algorithm is computationally efficient, avoiding MCMC and reducing the size of local Gram matrices.
Read more
Hierarchical Bayesian Quadrature
Summary
This paper presents Hierarchical Bayesian Quadrature (HBQ), a novel approach to numerical integration that addresses the limitations of traditional Bayesian Quadrature (BQ) methods, particularly in handling nonstationary integrands. BQ typically relies on stationary covariance functions, which can lead to model misspecification when the integrand varies significantly across the domain. HBQ overcomes this challenge by employing a tree-based partitioning of the integration domain, allowing for the use of local stationary Gaussian process (GP) models that adapt to the local behavior of the integrand. The method utilizes a hierarchical structure to recombine local integral estimates while maintaining cross-subdomain correlations, thus optimizing the evaluation budget based on local integrand complexity. The authors demonstrate that HBQ achieves significant performance improvements over standard BQ in scenarios involving nonstationary integrands, while still performing comparably on stationary cases. The algorithm is efficient, requiring no Markov Chain Monte Carlo (MCMC) methods, and introduces minimal computational overhead, making it a practical solution for various scientific computing applications.
Methodology
The authors developed HBQ by creating a tree-based partition of the integration domain, where each leaf node corresponds to a local stationary GP model. The tree grows adaptively based on marginal likelihood optimization and model selection criteria, allowing the method to allocate computational resources effectively according to the complexity of the integrand. This approach enables the integration of local estimates while preserving correlations across different regions of the domain.
Results
In benchmark tests, HBQ demonstrated substantial improvements in accuracy and efficiency for nonstationary integrands compared to standard BQ methods. The results showed that HBQ could adaptively allocate evaluation budgets, leading to reduced relative errors in integral estimates, particularly in complex scenarios such as model evidence computations for epidemiological models.
Implications
The development of HBQ has significant implications for fields that rely on numerical integration, such as scientific computing, engineering simulations, and probabilistic machine learning. By effectively handling nonstationary integrands, HBQ can improve the reliability and efficiency of integral computations in various applications, potentially leading to better model evidence estimates and enhanced decision-making processes.
PRISM Edit: One Vector for All Temporal Answers
NLP
Large Language Models
Optimization
- PRISM Edit allows LLMs to maintain historical accuracy while updating temporal facts.
- The method leverages the model's existing internal mechanisms for temporal modulation.
- A new benchmark, TIMECONFLICT, is introduced to evaluate temporal editing performance.
- PRISM Edit achieves state-of-the-art results with significant improvements over baseline methods.
Read more
PRISM Edit: One Vector for All Temporal Answers
Summary
The paper introduces PRISM Edit, a novel approach to knowledge editing in large language models (LLMs) that addresses the limitations of existing methods when dealing with temporal facts. Traditional editing methods often assume that updates replace old facts entirely, which is problematic for temporal knowledge where historical facts remain valid. The authors utilize causal tracing to demonstrate that LLMs already have a two-stage internal mechanism for processing temporal knowledge: early layers retrieve a time-agnostic representation, while later layers apply temporal context to generate time-specific answers. PRISM Edit optimizes a single polysemous representation that can be modulated by the model's internal mechanisms to yield correct answers across different temporal contexts, without requiring architectural changes. The authors evaluate PRISM Edit on a new benchmark, TIMECONFLICT, and on COUNTERFACT, showing significant improvements in temporal consistency and relative-time scoring compared to existing methods, while also being more efficient.
Methodology
The authors employed causal tracing on (subject, relation, object, time) tuples to analyze how LLMs process temporal knowledge. They developed PRISM Edit, which optimizes a single polysemous vector that the model modulates to produce contextually accurate temporal answers, without modifying the model's architecture or requiring additional modules.
Results
PRISM Edit demonstrated an average improvement of +23.3 in Temporal Consistency (TC) and +33.7 in Current Relative-time Score (CRS) on the TIMECONFLICT benchmark and COUNTERFACT, outperforming the best baseline methods while being over two times faster.
Implications
The findings suggest that LLMs can be more effectively updated to reflect temporal changes without losing historical accuracy, which could enhance applications in dynamic knowledge domains such as news, history, and real-time data processing.
Optimal Top-k Identification from Pairwise Comparisons
Theory
Optimization
Efficient ML
- Introduces an asymptotically optimal algorithm for top-k identification from pairwise comparisons under latent utility models.
- Characterizes the oracle allocation problem as a two-player game, leading to insights on optimal sampling strategies.
- Develops a primal-dual algorithm for online learning of comparison allocations, proving its asymptotic optimality.
- Addresses the importance of adaptivity in selecting pairs for comparison to minimize costs associated with comparisons.
Read more
Optimal Top-k Identification from Pairwise Comparisons
Summary
This paper addresses the active learning problem of fixed-confidence top-k identification using noisy pairwise comparisons. The authors propose an algorithm that sequentially selects pairs of items to compare, aiming to identify the top-k items with a specified error probability (δ) while minimizing the expected number of comparisons. The study highlights the lack of existing asymptotically optimal algorithms for top-k identification in pairwise comparisons under latent utility models. The authors develop a new algorithm that formulates the problem as a saddle-point problem, allowing for a computationally efficient primal-dual procedure to learn optimal comparison allocations online. The paper also discusses the implications of adaptivity in comparison allocation, which is crucial in scenarios where comparisons are costly in terms of time, money, or attention. The proposed algorithm is shown to be asymptotically optimal, matching the information-theoretic lower bound as δ approaches zero.
Methodology
The authors formulate the top-k identification problem as a saddle-point problem, allowing them to derive a primal-dual algorithm that learns the optimal allocation of comparisons online. The algorithm adapts based on observed outcomes, focusing on the most informative comparisons to distinguish the top-k items.
Results
The proposed algorithm is proven to be asymptotically optimal, achieving the expected sample complexity that matches the information-theoretic lower bound as δ approaches zero. The performance of the algorithm is validated through simulations, demonstrating its effectiveness in various scenarios of pairwise comparisons.
Implications
The findings have significant implications for applications that rely on pairwise comparisons, such as recommender systems, crowdsourced evaluations, and decision-making processes in various domains. The adaptive nature of the algorithm can enhance efficiency and reduce costs in identifying top items.
LionVote: Per-Layer Learning Rate Adaptation for Lion
Optimization
- LionVote provides a per-layer adaptive learning rate mechanism for the Lion optimizer.
- The effective learning rate varies significantly across different layer types, necessitating tailored adaptations.
- LionVote outperforms standard Lion and AdamW optimizers in terms of accuracy on ViT-Tiny/CIFAR-100.
- The adaptation mechanism is based on diagnostics of gradient stability and momentum health.
Read more
LionVote: Per-Layer Learning Rate Adaptation for Lion
Summary
The paper introduces LionVote, a novel per-layer learning rate adaptation mechanism designed for the Lion optimizer. It addresses the issue of varying effective learning rates across different layer types in neural networks, particularly in Vision Transformers (ViT) and WideResNet architectures. The study finds that the effective learning rate for attention and MLP parameters is significantly higher than for normalization layers, indicating a 32% disparity that cannot be captured by a single global learning rate. LionVote operates by maintaining a compound level for each parameter tensor, which is updated every 'c' epochs based on two diagnostics: gradient direction stability and momentum health. A validation loss tiebreaker is employed when diagnostics conflict. The results demonstrate that LionVote achieves a top-1 accuracy of 69.7% on ViT-Tiny/CIFAR-100, outperforming the standard Lion optimizer and AdamW. The findings highlight the importance of per-layer adaptation, which varies based on architectural heterogeneity and specific tasks, suggesting that different layer types require tailored learning rates for optimal performance.
Methodology
The methodology involves a per-layer learning rate adaptation mechanism where each parameter tensor maintains a compound level that modulates the base learning rate. This level is updated based on diagnostics related to gradient direction stability and momentum health, with a validation loss tiebreaker used when diagnostics conflict. The mechanism is evaluated on WideResNet and ViT-Tiny architectures across CIFAR-10 and CIFAR-100 datasets.
Results
LionVote achieved a top-1 accuracy of 69.7% on ViT-Tiny/CIFAR-100, surpassing the standard Lion optimizer's accuracy of 69.0% and AdamW's 68.8%. The study quantified that Lion's effective learning rate is 2.6–2.8 times too high for attention and MLP parameters and approximately 2 times too high for normalization layers. The effectiveness of per-layer adaptation was shown to depend on the architectural characteristics and the specific tasks.
Implications
The findings suggest that optimizing learning rates at the layer level can lead to improved performance in neural networks, particularly in architectures like Vision Transformers. This approach may be applicable to other optimizers and could enhance training efficiency and model accuracy across various tasks.
LLM-PDESR: Robust PDE Discovery via Subdomain Weighted Residuals and LLM-Guided Symbolic Hypothesis Generation
Large Language Models
Optimization
Interpretability
- Introduction of LLM-PDESR, an end-to-end framework for PDE discovery.
- Utilization of C4 continuous quintic splines and SWR evaluations to address derivative noise.
- Development of a rigorous benchmark to validate genuine symbolic discovery capabilities.
- Successful extraction of a 1D dynamical surrogate from noisy climate data.
Read more
LLM-PDESR: Robust PDE Discovery via Subdomain Weighted Residuals and LLM-Guided Symbolic Hypothesis Generation
Summary
The paper presents LLM-PDESR, a novel framework for discovering governing partial differential equations (PDEs) from noisy observational data. Traditional symbolic regression methods struggle with the vast search space and high-frequency noise, leading to inaccurate equation identification. LLM-PDESR integrates the structural hypothesis generation capabilities of Large Language Models (LLMs) with a mathematically rigorous evaluation environment. It employs C4 continuous quintic splines for robust differentiation and subdomain weighted residuals (SWR) as low-pass filters to mitigate noise effects. The framework includes a Pareto-driven feedback loop that balances predictive accuracy with structural simplicity. The authors evaluate LLM-PDESR on 23 canonical PDEs and five novel equations, demonstrating its ability to extract a consistent structural skeleton from noisy climate data. The results show that LLM-PDESR significantly outperforms existing methods in structural recovery, noise resilience, and avoiding spurious complexity.
Methodology
LLM-PDESR combines LLMs for hypothesis generation with continuous numerical optimization. It replaces traditional finite difference methods with quintic spline smoothing for robust differentiation and employs SWR evaluations to reduce noise impact. A Pareto-driven feedback loop refines candidate equations based on predictive accuracy and structural parsimony.
Results
The framework was evaluated on a benchmark of 23 canonical PDEs and five structurally novel equations, showing significant improvements in structural recovery and noise resilience. It successfully extracted a 1D dynamical surrogate from noisy ERA5 climate data, validated through rigorous out-of-distribution testing.
Implications
LLM-PDESR has the potential to enhance the discovery of governing equations in various scientific fields, particularly in scenarios with noisy data. Its ability to balance complexity and accuracy could lead to more interpretable models in scientific machine learning.
CoCoT-EEG: Contrastive-Pretrained Multiscale Convolutional Transformer for EEG Decoding
Time Series
- CoCoT-EEG introduces a convolution-first architecture for EEG decoding, enhancing feature extraction from noisy signals.
- The model outperforms existing single-task EEG models and competes with large-scale pretrained models.
- Contrastive pretraining significantly boosts performance on diverse downstream EEG tasks.
- Extensive ablation studies reveal important design considerations for EEG foundation models.
Read more
CoCoT-EEG: Contrastive-Pretrained Multiscale Convolutional Transformer for EEG Decoding
Summary
The paper presents CoCoT-EEG, a novel contrastive-pretrained model designed for decoding electroencephalogram (EEG) data. Traditional approaches to EEG decoding often rely on tokenizing raw EEG signals followed by masked reconstruction pretraining, which has been shown to be suboptimal due to the high noise levels and limited information dimensions inherent in EEG data. CoCoT-EEG addresses these challenges by employing a convolution-first architecture that extracts multi-scale temporal features from the raw EEG signals before tokenization. This model combines 1D convolutional layers with Transformer encoder blocks, allowing for effective representation learning. The authors demonstrate that CoCoT-EEG outperforms existing single-task models and even rivals large-scale pretrained EEG models across various benchmark tasks. Additionally, the paper highlights the advantages of contrastive pretraining in enhancing model performance and generalizability, particularly in noisy environments. Through extensive ablation studies, the authors explore the impact of architectural choices, pretraining objectives, and data augmentations, providing insights into the design of future EEG decoding models.
Methodology
CoCoT-EEG employs a hybrid architecture that first utilizes multi-scale 1D convolutions to extract features from raw EEG signals. The model then patchifies the extracted features before encoding them with a Transformer stack. For pretraining, a MoCo-style contrastive learning framework is used, leveraging data augmentations to generate query and key samples for training with InfoNCE loss.
Results
The results indicate that CoCoT-EEG, trained from scratch, surpasses previous single-task models and rivals state-of-the-art pretrained EEG models on various decoding tasks. The systematic ablation studies confirm the effectiveness of the model architecture and contrastive pretraining in enhancing performance and generalizability.
Implications
The findings suggest that CoCoT-EEG could be applied in brain-computer interfaces (BCIs) and clinical diagnostics, where accurate EEG decoding is crucial. The insights into contrastive learning and model design may also inform future research in EEG representation learning and other domains dealing with noisy time series data.
Diachronic Sample Integration: Robust Tail-Risk Estimation with Generative Models
Generative Models
Theory
Time Series
- Introduction of Diachronic Sample Integration (DSI) for robust tail-risk estimation.
- DSI aggregates samples from multiple training checkpoints to mitigate tail estimation instability.
- Formalization of a bias-variance decomposition for VaR and ES under non-i.i.d. samples.
- Empirical results demonstrate substantial reduction in tail-estimation error compared to existing methods.
Read more
Diachronic Sample Integration: Robust Tail-Risk Estimation with Generative Models
Summary
This paper introduces Diachronic Sample Integration (DSI), a novel test-time inference framework designed to enhance tail-risk estimation in risk-sensitive applications using deep generative models. Traditional generative models often focus on bulk distributional fidelity, which can lead to poor performance in estimating rare adverse scenarios due to localized optimization noise. DSI addresses this issue by aggregating generated samples from multiple checkpoints throughout the training trajectory, rather than relying on a single converged model. This approach allows for a more stable estimation of tail-dependent functionals, such as Value-at-Risk (VaR) and Expected Shortfall (ES), without altering the underlying generative objective. The authors formalize the DSI mechanism through a finite-budget bias-variance theory, demonstrating its effectiveness in reducing tail-estimation error. Empirical results show that DSI significantly outperforms single-checkpoint baselines and state-of-the-art tail-aware methods across various synthetic processes and high-frequency trading data, all while maintaining the original generative model's integrity.
Methodology
The methodology involves a test-time framework that ensembles generated samples from various checkpoints in a stochastic training trajectory. This approach leverages checkpoint-level variability to average out tail fluctuations, thereby enhancing the reliability of tail-risk estimates. The authors also provide a theoretical analysis of the bias-variance trade-off in the context of finite simulation budgets.
Results
The empirical evaluation shows that DSI significantly reduces tail-estimation error when compared to standard single-checkpoint diffusion models and other tail-aware benchmarks. This improvement is achieved under fixed simulation budgets, indicating that DSI is effective in scenarios where computational resources are limited.
Implications
The findings suggest that DSI can be a valuable tool in risk management and financial decision-making, particularly in environments where data scarcity and tail risks are prevalent. By improving the reliability of tail-risk estimates, DSI could enhance the performance of generative models in various risk-sensitive applications.
Advancing Optimal Subset Oracle via Learning Relaxation of Neural Set Functions
Optimization
Theory
Efficient ML
- Introduction of a learned surrogate objective to replace Monte Carlo sampling in optimal subset oracles.
- The proposed ReSet method provides stable and efficient gradients, reducing computational overhead.
- Theoretical guarantees for convergence and approximation ratios are established.
- Demonstrated improvements over existing methods in real-world applications.
Read more
Advancing Optimal Subset Oracle via Learning Relaxation of Neural Set Functions
Summary
This paper addresses the challenge of learning neural set functions, which are crucial for applications like AI-driven drug discovery and product recommendation. The authors propose a novel approach to optimal subset oracles that replaces traditional Monte Carlo sampling methods with a learned surrogate objective, termed Relaxation of Set functions (ReSet). This method aims to improve the efficiency and stability of gradient estimation during variational optimization. By interpreting the evidence lower bound (ELBO) as a continuous relaxation of the set function, the authors demonstrate that their approach reduces computational overhead and accelerates inference. They provide theoretical guarantees for convergence and approximation ratios, linking their method to variational free energy. Experimental results show that ReSet consistently outperforms existing baselines across various real-world tasks, highlighting its effectiveness in optimizing neural set functions without the extensive computational costs associated with traditional methods.
Methodology
The authors reinterpret the evidence lower bound as a continuous relaxation of the set function and propose a differentiable surrogate objective that eliminates the need for Monte Carlo sampling. This approach allows for efficient optimization of the variational distribution, leveraging a greedy strategy for continuous submodular maximization. The paper also derives an approximation guarantee and explores the connection between the ELBO and variational free energy.
Results
The experimental results indicate that the ReSet method consistently outperforms baseline methods across diverse real-world tasks, demonstrating faster convergence and reduced training time compared to traditional Monte Carlo sampling approaches.
Implications
The proposed method has significant implications for applications requiring efficient learning of set functions, such as recommendation systems and drug discovery, by reducing the computational burden and improving the stability of the optimization process.
NeuroMem-FHP: A Likelihood-Free Deep Learning Framework for Parameter Estimation of Fractional Hawkes Process
Time Series
- Introduction of NeuroMem-FHP, a deep learning framework for parameter estimation of FHP.
- Utilization of LSTM and Transformer architectures to bypass traditional likelihood-based estimation.
- Significant performance improvement over classical MLE methods in synthetic and real-world datasets.
- Validation of the framework through predictive validation on high-frequency datasets.
Read more
NeuroMem-FHP: A Likelihood-Free Deep Learning Framework for Parameter Estimation of Fractional Hawkes Process
Summary
This paper introduces the NeuroMem-FHP framework, a novel deep learning approach for estimating parameters of the fractional Hawkes process (FHP), which is a self-exciting point process characterized by long-range dependence. The authors develop two neural network architectures: Long Short-Term Memory (LSTM) networks and Transformers, to directly estimate the model parameters (µ, γ, α, β) from sequences of inter-arrival times without the need for computationally intensive likelihood optimization. Experiments conducted on synthetic datasets reveal that both neural models significantly outperform traditional Maximum Likelihood Estimation (MLE) methods, with the Transformer achieving the best performance (MSE = 0.1634), followed by LSTM (MSE = 0.1752), while MLE yielded a much higher MSE of 2.8032. An ablation study investigates the impact of key hyperparameters on model performance. The framework is further validated on two real-world datasets: AAPL NBBO transaction data and Montgomery County 911 emergency call records, demonstrating that event sequences generated from the estimated parameters closely match the empirical distribution and temporal dynamics of the actual data. The findings suggest that the Transformer-based approach offers an efficient and accurate alternative for parameter estimation in FHP, making it a promising tool for modeling event-driven systems exhibiting long-memory dynamics.
Methodology
The authors developed two neural network architectures, LSTM and Transformer, to estimate parameters of the fractional Hawkes process directly from inter-arrival time sequences. The framework eliminates the need for likelihood optimization, allowing for efficient parameter estimation. Experiments were conducted on synthetic data and real-world datasets to validate the performance of the proposed methods.
Results
The Transformer model achieved the lowest mean squared error (MSE = 0.1634) in parameter estimation, followed by the LSTM model (MSE = 0.1752), while the classical MLE method resulted in a significantly higher MSE of 2.8032. The framework was validated on real-world datasets, showing that the estimated parameters could effectively reproduce the empirical distribution and temporal dependence structure of the observed data.
Implications
The NeuroMem-FHP framework provides a robust and efficient method for parameter estimation in fractional Hawkes processes, which can be applied in various fields such as finance, social networks, and seismology where long-range dependence is critical. This approach may enhance the modeling of event-driven systems and improve predictive analytics in these domains.
Reinforcement Learning for Execution under Dynamic Fees in a Closed-Loop DEX Simulator
Reinforcement Learning
- Introduces a closed-loop simulator to study the impact of dynamic fees on trade execution in DEXs.
- Demonstrates that a small DQN outperforms traditional execution heuristics in dynamic-fee environments.
- Establishes a rigorous evaluation methodology for model-free control in trading environments.
- Finds that the advantage of the DQN diminishes under constant fee conditions, indicating the significance of fee variability.
Read more
Reinforcement Learning for Execution under Dynamic Fees in a Closed-Loop DEX Simulator
Summary
This paper addresses the challenges of executing trades in decentralized exchanges (DEXs) with dynamic fees, which are increasingly relevant for automated market makers (AMMs). Traditional empirical methods fail to capture how order flow responds to changing fees due to the lack of variability in trader-facing fees and the latent nature of trader types. To overcome this limitation, the author constructs a minimal closed-loop simulator that incorporates a dynamic-fee rule, fee-sensitive noise flow, and arbitrage mechanisms. The study evaluates various execution policies, including a small Deep Q-Network (DQN), against tuned benchmarks. The DQN demonstrates a significant reduction in implementation shortfall, outperforming traditional one-step routing methods, particularly in dynamic-fee environments. The findings provide model-conditioned counterfactual evidence regarding execution control in AMMs, emphasizing the importance of dynamic fees in optimizing trade execution strategies.
Methodology
The study employs a closed-loop simulator designed to create an action-responsive environment where the execution trader's actions influence market dynamics. The evaluation method includes a disciplined approach to policy comparison, ensuring that all development choices are frozen before testing on a reserved evaluation block. The simulator incorporates a dynamic-fee rule inspired by equilibrium principles, allowing for the assessment of various execution strategies, including a small DQN and tuned benchmarks.
Results
The DQN achieved a reduction in implementation shortfall of 13.3 basis points (bps) of order notional under the pre-specified agent-last ordering, outperforming tuned one-step routing policies. The advantage was particularly pronounced in dynamic-fee scenarios, while under constant fees, the performance difference was negligible, indicating that the DQN's effectiveness is contingent on fee variability.
Implications
The findings suggest that dynamic fee structures can significantly enhance trade execution strategies in decentralized exchanges. This research could inform the design of future AMMs and trading algorithms, highlighting the need for adaptive strategies that respond to changing market conditions. Additionally, the evaluation methodology proposed could serve as a framework for future studies in market execution and reinforcement learning applications.
Exploratory Analysis of Deep Learning Models for Forecasting Meteorological Parameters in the Agricultural Sector
Time Series
- The study evaluates recurrent and hybrid deep learning models for meteorological forecasting in agriculture.
- Hybrid CNN-GRU models outperformed purely recurrent models in short-term forecasting tasks.
- The analysis utilized a substantial dataset of 134,376 hourly observations from Ioannina, Greece.
- Performance metrics included normalized RMSE, R2, and a composite Weighted Quotient Score (WQS).
Read more
Exploratory Analysis of Deep Learning Models for Forecasting Meteorological Parameters in the Agricultural Sector
Summary
This study presents a comparative evaluation of various deep learning architectures for forecasting key meteorological parameters critical to agriculture, including reference evapotranspiration (ET0), vapour pressure deficit (VPD), wind speed, and wind direction components. Utilizing a dataset of 134,376 hourly observations from Ioannina, Greece, spanning from January 2011 to April 2026, the authors compare single and multi-layer Gated Recurrent Units (GRU) and Long Short-Term Memory (LSTM) networks against hybrid models combining 1D Convolutional Neural Networks (CNN) with GRU and LSTM. The forecasting tasks include both a 24-hour next-day forecast and a 168-hour week-ahead forecast. Performance metrics such as normalized root mean squared error, coefficient of determination, and a composite Weighted Quotient Score (WQS) are employed to evaluate model effectiveness. The results indicate that while purely recurrent models perform well, hybrid CNN-GRU models achieve the highest scores, suggesting that convolutional feature extraction enhances short-term forecasting accuracy. This research underscores the importance of deep learning in improving meteorological predictions for agricultural decision-making.
Methodology
The authors conducted a controlled comparative evaluation of single and multi-layer GRU and LSTM networks against hybrid 1D-CNN-GRU and 1D-CNN-LSTM models. The study involved two forecasting scenarios: a 24-hour next-day forecast and a 168-hour week-ahead forecast, using a dataset sourced from the Open-Meteo Historical Weather API. Performance was quantified using normalized RMSE, R2, and a composite Weighted Quality Score (WQS).
Results
The most effective recurrent models were a 64-unit LSTM for the 24-hour forecast (WQS of 0.816755) and a 1024-unit GRU for the 168-hour forecast (WQS of 0.779465). The hybrid CNN-GRU models achieved the highest scores of 0.827535 and 0.782863 for the 24-hour and 168-hour forecasts, respectively, while CNN-LSTM models provided similar results with fewer parameters. Hybrid models improved WQS by 1.22–1.63% for 24 hours and 0.44–0.45% for 168 hours compared to recurrent baselines.
Implications
The findings suggest that integrating deep learning models, particularly hybrid architectures, can significantly enhance the accuracy of meteorological forecasts, which is crucial for agricultural planning and decision-making. This research could lead to improved irrigation management and crop-water requirements estimation, ultimately supporting better agricultural practices.
EvoClawBench: Can Agents Learn Reusable Skills from Their Own Runs?
Large Language Models
Reinforcement Learning
Robotics
- EvoClawBench is a novel benchmark for evaluating agents' ability to learn reusable skills from their own runs.
- The benchmark includes 100 tasks and supports multiple agent runtimes, allowing for comprehensive evaluation.
- Three evaluation modes (BASELINE, PRESKILL, POSTSKILL) are established to isolate the effects of skill authoring.
- Experimental results indicate that the effectiveness of self-authored skills varies significantly across different models.
Read more
EvoClawBench: Can Agents Learn Reusable Skills from Their Own Runs?
Summary
The paper introduces EvoClawBench, a benchmark designed to evaluate whether agents can learn reusable skills from their own execution runs. Existing benchmarks primarily assess task completion and tool use but do not focus on the agent's ability to convert its own experiences into reusable skills. EvoClawBench consists of 100 tasks with 502 sub-problems across various domains, allowing for a closed-loop evaluation of skill learning. The authors propose three evaluation strategies: BASELINE (direct execution without skills), PRESKILL (skill authoring before execution), and POSTSKILL (skill summarization after the first run). Experimental results reveal that performance varies significantly across different agent runtimes, with self-authored skills yielding mixed outcomes. The findings suggest that learning reusable skills is selective and sensitive to costs, rather than an automatic advantage of skill authoring. The paper contributes a new benchmark, a formal evaluation protocol, empirical findings on runtime dependencies, and an open artifact for further research.
Methodology
The authors developed EvoClawBench to assess agents' skill learning capabilities through a structured evaluation framework. They implemented three distinct strategies: BASELINE, where agents execute tasks directly; PRESKILL, where agents create skills prior to execution; and POSTSKILL, where agents summarize their first-run experiences to create reusable skills for a subsequent execution. The benchmark includes a variety of tasks and sub-problems to facilitate a closed-loop learning environment.
Results
The experiments demonstrated that performance is highly runtime-dependent, with OpenClaw achieving below 20% across models, while nanobot models showed a range from 56.45% to 96.13%. Self-authored skills had mixed effects, with some models improving significantly under POSTSKILL, while others experienced drastic performance drops. These results indicate that the process of learning reusable skills is not straightforward and can lead to varying outcomes based on the agent and context.
Implications
The findings from this research could influence the design of future agent benchmarks and the development of more effective skill learning mechanisms in AI systems. By understanding the nuances of skill reuse and the factors affecting performance, researchers can create more robust agents capable of adapting to new tasks efficiently.
modelDNA: Calibrated Lineage Verification and Merge Decomposition from Sampled Weight Fingerprints
Large Language Models
Theory
Efficient ML
- Introduction of modelDNA, a tool for lineage verification of language models.
- Fingerprinting method that captures lineage signals from minimal data.
- Calibrated verdict engine that prioritizes abstention to avoid false accusations.
- Merge decomposition capability that recovers mixture weights from fingerprints.
Read more
modelDNA: Calibrated Lineage Verification and Merge Decomposition from Sampled Weight Fingerprints
Summary
The paper presents modelDNA, a tool designed to verify the lineage of open-weight language models and decompose merged model weights using sampled weight fingerprints. The authors highlight the inadequacy of current lineage detection methods, which often rely on manual inspection and lack a systematic approach. modelDNA fingerprints models using 100-300 MB of ranged HTTP reads, allowing for efficient comparison against a reference database of foundation models. The tool categorizes lineage evidence into one of eight classes with calibrated probabilities, prioritizing abstention over false accusations. The second major contribution is the ability to recover mixture weights from merged models through constrained least squares, validated against known configurations. The system achieved an AUROC of 1.0 on a benchmark of 15 models, with zero false positives and accurate parent attribution. The paper emphasizes the importance of reliable lineage verification in the context of the growing number of model repositories and the challenges posed by undocumented model parentage.
Methodology
modelDNA fingerprints models by sampling 100-300 MB of data from HTTP reads, comparing these fingerprints against a reference database of foundation models. It employs a calibrated verdict engine that categorizes lineage evidence into eight classes, using statistical methods to impute missing signals. For merge decomposition, it utilizes constrained least squares to recover mixture weights from the fingerprints of merged models.
Results
The modelDNA tool achieved an AUROC of 1.0 on a benchmark of 15 models with documented parentage, successfully attributing the correct parent model in all cases without any false positives. The merge decomposition method accurately recovered mixture weights for various merging configurations, demonstrating high correlation with published values.
Implications
The development of modelDNA has significant implications for the accountability and transparency of model lineage in the rapidly evolving landscape of machine learning. It provides a systematic approach to verifying model origins, which is crucial for maintaining trust in AI systems and preventing disputes over model derivation.
Energy-guided Recursive Model
Theory
Optimization
Efficient ML
- ERM introduces an intrinsic selection principle using Hopfield energies for candidate trajectory evaluation.
- The model outperforms previous recursive reasoning models in structured problem-solving tasks.
- Explicit energy functions improve the efficiency of inference in recursive reasoning frameworks.
- ERM effectively integrates with energy-based techniques like parallel tempering for enhanced sampling.
Read more
Energy-guided Recursive Model
Summary
The Energy-guided Recursive Model (ERM) addresses the limitations of existing recursive reasoning models, which struggle with principled selection mechanisms during test-time computation. Traditional models often rely on heuristic methods such as q-heads or majority voting to select from multiple generated trajectories, leading to inefficiencies. ERM introduces a novel approach by incorporating Hopfield energies as an intrinsic selection principle, allowing for a more structured evaluation of candidate trajectories based on their compatibility with valid task structures. This model leverages Hopfield-type memories tailored to specific tasks, such as Sudoku, Pencil Puzzle Bench (PPBench), and Maze, to rank candidates effectively. The results demonstrate that ERM achieves optimal solutions across these tasks, outperforming previous models like the Probabilistic Tiny Recursive Model and Equilibrium Reasoners. The findings suggest that integrating explicit energy functions into recursive reasoning can significantly enhance inference quality and efficiency.
Methodology
ERM employs a Hopfield-network-based energy mechanism to rank candidate outputs generated by a recursive reasoning model. It utilizes task-specific memory structures to evaluate the compatibility of trajectories, allowing for a principled selection process that distinguishes between valid and invalid candidates.
Results
ERM achieved optimal accuracy rates of 98.97% on Sudoku, 88.04% on Pencil Puzzle Bench, and 99.30% on Maze, demonstrating significant improvements over existing models such as the Probabilistic Tiny Recursive Model and Equilibrium Reasoners.
Implications
The introduction of energy-guided selection in recursive reasoning models could lead to more effective inference mechanisms in various structured problem-solving applications, potentially influencing future developments in AI reasoning systems.
Agentic Skill Optimization over Lie Algebroids
Optimization
Large Language Models
Theory
- LASKO framework models skill edits as interdependent operations within a structured artifact context.
- Utilizes controlled Lie algebroids to capture the relationships and order sensitivity of skill edits.
- Achieves significant speedups in skill optimization through efficient screening tests.
- Highlights the geometric nature of skill optimization, focusing on the interactions between edits.
Read more
Agentic Skill Optimization over Lie Algebroids
Summary
This paper introduces a novel framework for agentic skill optimization called LASKO (Lie Algebroid SKill Optimization), which addresses the complexities involved in editing skills within agentic systems. Unlike traditional approaches that treat skill edits as independent operations, LASKO recognizes that these edits are interdependent and can have varying effects based on their context and order. The framework models skills as structured artifacts and employs a controlled Lie algebroid to capture the relationships between edits and their impacts. By utilizing inexpensive Lie-bracket screening tests, LASKO significantly accelerates the optimization process, achieving up to a 15× speedup in skill validation compared to brute-force methods. The paper emphasizes the importance of understanding the geometric properties of skill edits and their interactions, proposing that this understanding can enhance the efficiency and effectiveness of agentic systems in self-improvement tasks.
Methodology
The methodology involves modeling skills as typed, anchored Markdown objects and employing a controlled Lie algebroid to represent the edit policies. The framework uses Lie-bracket screening tests to prioritize which edit pairs to validate, thereby reducing the computational cost associated with exhaustive validation methods.
Results
LASKO demonstrated an order-of-magnitude speedup in skill optimization, achieving a nearly 15× improvement in a benchmark task involving causal extraction from natural language, compared to a brute-force validation approach using a large language model.
Implications
The findings suggest that LASKO can significantly enhance the efficiency of agentic systems in skill optimization tasks, making it a valuable tool for developing more effective AI agents capable of self-improvement. This could have broad applications in areas such as natural language processing, robotics, and other domains requiring adaptive learning systems.
Two Confounds in Cross-Model Value Comparison: Response Determinism and the Access Harness
NLP
Large Language Models
Theory
- Introduces a separation protocol to distinguish between genuine value differences and response determinism in language models.
- Demonstrates that response determinism varies significantly across models and should be measured for accurate comparisons.
- Identifies the access harness as a confounding factor that alters model value profiles based on deployment context.
- Findings suggest that apparent individuation of models is often inflated by response determinism and deployment conditions.
Read more
Two Confounds in Cross-Model Value Comparison: Response Determinism and the Access Harness
Summary
This paper addresses two significant confounds in cross-model value comparisons of language models: response determinism and the access harness. The authors argue that existing methods conflate genuine value differences with differences in how deterministically models respond to forced choices. They introduce a separation protocol that employs no-rule value dilemmas and a determinism index to differentiate between genuine value disagreements and artifacts of response determinism. The study reveals that determinism varies significantly across models and must be measured rather than assumed. Additionally, the authors identify the access harness as a confounding factor that alters a model's value profile based on the deployment context. Their findings indicate that correcting for determinism reduces the apparent individuation of models, particularly within families, while some cross-family disagreements persist. The paper emphasizes the need for careful measurement and consideration of deployment contexts in model comparisons.
Methodology
The authors developed a separation protocol that includes no-rule value dilemmas, counterbalanced repeated forced-choice measurements, and an explicit determinism index to assess how sharply models commit to choices. They also conducted experiments across nine models from five provider families to analyze the effects of determinism and access harness on value comparisons.
Results
The study found substantial variation in response determinism across models, with extremity values ranging from approximately 0.66 to 0.95. Correcting for determinism led to a reduction in the apparent individuation of models, particularly within families, while some disagreements across families remained robust. The access harness was shown to significantly alter the value profiles of models, indicating that deployment context plays a crucial role in value comparisons.
Implications
The findings highlight the importance of accurately measuring response determinism and considering deployment contexts in cross-model comparisons. This could lead to more reliable assessments of language model values and inform alignment audits and model selection processes.
RDQ: Residual Distribution Quantization for Large Language Models
NLP
Large Language Models
Efficient ML
- Identifies residual stream drift as the main cause of performance degradation in sub-4-bit quantization of LLMs.
- Discovers that a significant majority of layers in LLaMA-3-8B have non-Gaussian distributions and that variance increases substantially with depth.
- Introduces Cascaded Error Compensation (CEC) for effective calibration of quantized layers, correcting for cross-layer drift.
- Achieves state-of-the-art perplexity results on LLaMA-3-8B, Qwen-2.5-7B, and Mistral-7B across all tested bit-widths.
Read more
RDQ: Residual Distribution Quantization for Large Language Models
Summary
This paper introduces RDQ (Residual Distribution Quantization), a novel post-training quantization (PTQ) framework aimed at improving the quantization of large language models (LLMs) below 4-bit precision. The authors identify that the degradation in performance at lower bit-widths is primarily due to residual stream distributional drift, where quantization noise accumulates across transformer layers, leading to significant KL divergence from the FP16 baseline. They empirically demonstrate that 84% of the layers in the LLaMA-3-8B model exhibit non-Gaussian residual distributions and that variance increases dramatically across layers. To address this issue, the authors propose Cascaded Error Compensation (CEC), a sequential calibration procedure that captures the drifted activations at each layer and adjusts the quantization scales accordingly. This method allows for improved performance without additional inference overhead. The results show that RDQ achieves state-of-the-art performance on multiple architectures, significantly outperforming existing methods in terms of perplexity (PPL) across various bit-widths.
Methodology
The methodology involves a detailed analysis of the residual stream dynamics in transformer architectures, identifying the accumulation of quantization errors across layers. The proposed CEC method calibrates each layer sequentially using drifted activations from previously quantized layers, fitting per-channel scales to these distributions. This approach contrasts with traditional methods that assume clean FP16 inputs for calibration.
Results
RDQ achieves significant improvements in perplexity, with results such as 7.55 / 5.62 PPL for LLaMA-3-8B at W3/W4, outperforming the best published baseline (LeanQuant/SpinQuant) by up to 46.4% in certain configurations. The method is validated across multiple architectures, demonstrating robustness and effectiveness.
Implications
The findings suggest that addressing residual stream drift can lead to more efficient deployment of large language models on edge hardware, enabling lower precision quantization without sacrificing performance. This has potential applications in various fields requiring efficient model inference, such as mobile applications and real-time processing.
DiffEEG: A Self-Supervised Denoising Diffusion Model for Learning EEG Generic Representations
Time Series
Reinforcement Learning
Generative Models
- DiffEEG addresses severe annotation scarcity and class imbalance in EEG data for seizure detection.
- The model employs a denoising diffusion pre-training strategy to learn robust EEG representations.
- A reinforcement learning-based fine-tuning mechanism enhances sensitivity to rare seizure events.
- DiffEEG achieves clinically relevant performance metrics, including high accuracy and F1-scores.
Read more
DiffEEG: A Self-Supervised Denoising Diffusion Model for Learning EEG Generic Representations
Summary
The paper presents DiffEEG, a self-supervised foundation model designed to enhance EEG-based seizure detection, addressing the challenges of annotation scarcity and class imbalance in clinical recordings. DiffEEG, comprising 9.6 million parameters, is pretrained on 1.3 million unlabeled EEG segments from the Temple University Hospital Seizure Corpus (TUHSZ) using a denoising diffusion approach. The model employs a 1D U-Net architecture with multi-head self-attention to learn generic neural representations. For downstream tasks, a reinforced decision layer utilizes policy gradient optimization to maximize the F1-score, focusing on sensitivity to rare seizure events rather than overall accuracy. The model achieves 61% accuracy and 59% F1-score for 4-class seizure subtyping, and 81% accuracy with 85% weighted F1 for binary detection, while maintaining a clinically viable seizure recall of 59%. The segment-level evaluation indicates an upper bound accuracy of 97.6%, showcasing the model's strong architectural capacity. DiffEEG demonstrates that combining diffusion-based pre-training with metric-aware reinforcement learning can facilitate clinically deployable seizure monitoring with minimal labeled data requirements.
Methodology
DiffEEG is structured in two stages: first, a 1D U-Net encoder is pretrained using a self-supervised denoising diffusion objective to recover clean EEG segments from corrupted inputs. In the second stage, a reinforced decision layer is applied, utilizing policy gradient optimization to maximize the F1-score, focusing on rare seizure events.
Results
DiffEEG achieves 61% accuracy and 59% F1-score for 4-class seizure subtyping, and 81% accuracy with 85% weighted F1 for binary seizure detection. The model maintains a seizure recall of 59% despite a class imbalance of 6.7% prevalence. Segment-level evaluations indicate an upper bound accuracy of 97.6%.
Implications
The findings suggest that DiffEEG can significantly improve seizure monitoring in clinical settings, reducing the reliance on extensive labeled datasets and enhancing the sensitivity of detection systems for rare seizure events. This model could lead to more effective and efficient tools for managing epilepsy and other neurological disorders.
MLPs are Hebbians: Constructing Efficient Fact-Storing MLPs for Transformers
NLP
Large Language Models
Theory
Efficient ML
- Introduces a closed-form construction of MLPs that achieves optimal fact storage scaling.
- Demonstrates the ability to handle arbitrary input/output geometries.
- Shows that the proposed MLPs can be integrated into Transformer architectures for factual recall.
- Achieves significant parameter efficiency compared to prior constructions.
Read more
MLPs are Hebbians: Constructing Efficient Fact-Storing MLPs for Transformers
Summary
This paper addresses the efficient storage of factual knowledge in large language models (LLMs) through Multi-Layer Perceptrons (MLPs). The authors propose a theoretical framework that explains how MLPs can store facts at an information-theoretically optimal rate, a phenomenon not fully captured by existing models. They introduce a closed-form construction of MLPs that meets three critical properties: optimal fact storage scaling, adaptability to arbitrary input/output geometries, and compatibility with Transformer architectures. The study emphasizes the importance of analyzing the decoding margin of MLPs, which allows for a more robust understanding of fact storage. The proposed construction demonstrates a significant reduction in parameter requirements (10-104× fewer) compared to previous methods while maintaining optimal capacity scaling. Additionally, the authors show that their MLPs can effectively function within Transformer blocks for factual recall tasks, further enhancing the efficiency of LLMs. A proof-of-concept illustrates the potential for modular fact editing by replacing MLPs within Transformers.
Methodology
The authors analyze the decoding margin of MLPs to develop a closed-form construction that aligns with the optimal fact storage properties observed in LLMs. They employ theoretical frameworks to demonstrate the scaling of storage capacity and the impact of embedding geometries on MLP performance. Empirical evaluations are conducted to validate the theoretical findings and to compare the proposed MLPs against existing models.
Results
The proposed MLP construction achieves information-theoretically optimal storage capacity scaling, requiring 10-104× fewer parameters than previous constructions at matched fact counts. The study also confirms that the MLPs can be effectively utilized within Transformer blocks for factual recall tasks, requiring 15-63× fewer parameters than the NTK baseline. The results indicate that the construction maintains optimal performance even under varying embedding geometries.
Implications
This work has significant implications for the design of more efficient LLMs, particularly in enhancing their capacity for factual recall and modularity. The findings could lead to advancements in the development of memory-efficient architectures for various applications in natural language processing and beyond.
Nonlinear Axiomatic Attribution for Cooperative Games
Theory
Optimization
- The Shapley value's linearity can lead to unreliable player rankings in cooperative games.
- Nonlinear axiomatic attribution methods are proposed to address the limitations of the Shapley value.
- The new methods retain necessary axioms while relaxing linearity and efficiency.
- Experimental results show improved performance in player ranking quality using the inclusion AUC metric.
Read more
Nonlinear Axiomatic Attribution for Cooperative Games
Summary
This paper addresses the limitations of the Shapley value in cooperative game theory, particularly its reliance on linearity, which can lead to unreliable player rankings when evaluating contributions. The authors propose a new class of nonlinear axiomatic attribution methods that relax the linearity and efficiency axioms while retaining other necessary axioms such as consistency and equal treatment. By introducing these nonlinear methods, inspired by the least core concept, the authors aim to provide a more reliable framework for evaluating player contributions in cooperative games. The paper demonstrates that the Shapley value's linearity can introduce vulnerabilities in maximizing the inclusion Area Under the Curve (AUC) metric, which is commonly used to assess the quality of player rankings. The proposed methods yield contribution vectors that are optimal solutions to minimization problems aimed at accurately approximating utility functions. Experimental results indicate that these nonlinear methods significantly improve player ranking quality compared to traditional Shapley value variants, particularly in contexts where identifying positively contributing players is crucial.
Methodology
The authors develop a class of nonlinear attribution methods by relaxing the linearity and efficiency axioms of traditional attribution methods. They utilize the concept of the least core and semi-values to create a framework that approximates utility functions more faithfully. The methods are mathematically grounded and involve solving minimization problems to derive contribution vectors. The effectiveness of these methods is evaluated through experiments using the inclusion AUC metric.
Results
The experiments conducted demonstrate that the proposed nonlinear attribution methods outperform traditional Shapley value variants in terms of player ranking quality, particularly when evaluated using the inclusion AUC metric. The theoretical results also confirm that the Shapley value is not reliable for maximizing the inclusion AUC due to its linearity.
Implications
The findings suggest that nonlinear attribution methods can provide more accurate evaluations of player contributions in cooperative games, which can be beneficial in various applications such as data attribution, feature attribution, and context attribution in machine learning models. This work opens avenues for further research into nonlinear methods in cooperative game theory and their applications in machine learning.
Time-Lag-Aware Deep Reinforcement Learning for Flexible Job-Shop Scheduling in PPVC Module Factories
Reinforcement Learning
Optimization
- Long post-operation time-lags in PPVC factories significantly inflate makespan and complicate scheduling.
- The proposed DRL solver incorporates lag-aware dynamics and anticipatory features to improve scheduling decisions.
- The learned policy outperforms traditional scheduling methods and is capable of rapid re-planning in response to disruptions.
- A public benchmark generator is released to facilitate further research in this area.
Read more
Time-Lag-Aware Deep Reinforcement Learning for Flexible Job-Shop Scheduling in PPVC Module Factories
Summary
This paper addresses the challenges of flexible job-shop scheduling in Prefabricated Prefinished Volumetric Construction (PPVC) module factories, where long post-operation time-lags significantly affect production efficiency. The authors highlight that these lags, caused by processes such as concrete curing and paint drying, can inflate the optimal makespan by an average of 67%. To tackle this issue, they adapt a state-of-the-art dual-attention deep reinforcement learning (DRL) solver, enhancing it with three key modifications: lag-aware dynamics with an admissible reward bound, anticipatory lag feature channels, and liveness-masked operation and station-type embeddings. The modifications ensure that the scheduling decisions are made with an awareness of the time-lags, which is crucial for optimizing the scheduling process. The authors provide a public benchmark generator based on an official national prefabrication guidebook and demonstrate that their learned policy outperforms traditional dispatching rules and a genetic-algorithm metaheuristic, achieving results within 4% of a constraint-programming reference. The proposed method is fast, requiring only seconds for re-planning, making it suitable for real-time applications in manufacturing environments.
Methodology
The authors adapt a dual-attention deep reinforcement learning solver by introducing three extensions that account for time-lags in scheduling. These include lag-aware dynamics, anticipatory lag feature channels, and liveness-masked embeddings, ensuring that the scheduling process is optimized for the unique constraints of PPVC factories.
Results
The adapted DRL policy achieved scheduling results within 4% of the optimal constraint-programming reference and outperformed all tested dispatching rules and a genetic-algorithm metaheuristic. The method demonstrated significant advantages under capacity contention and maintained performance across various factory sizes.
Implications
This research has significant implications for improving scheduling efficiency in PPVC module factories, potentially leading to enhanced productivity and reduced operational costs. The findings can also inform the development of more sophisticated scheduling algorithms in other manufacturing contexts where time-lags are a critical factor.
Super-Tuning: From Activation-Aware Pruning to Sparse Fine-Tuning
NLP
Large Language Models
Efficient ML
- Introduction of Super, a sparse PEFT method that selects weights using a training-free saliency score.
- Development of Supra, a hybrid approach combining Super's sparse updates with LoRA.
- Demonstration of the effectiveness of pruning-inspired strategies for selecting sparse supports.
- Evaluation of multiple PEFT methods on arithmetic reasoning tasks, showing superior performance of Super and Supra.
Read more
Super-Tuning: From Activation-Aware Pruning to Sparse Fine-Tuning
Summary
This paper addresses the high computational costs associated with fine-tuning large language models (LLMs) by proposing a novel parameter-efficient fine-tuning (PEFT) method called Super. The authors leverage saliency signals from pruning techniques to determine which model parameters to adapt during fine-tuning. Super utilizes a Wanda-style activation-weighted magnitude score to select a small set of trainable parameters without requiring additional training or gradient computations. Additionally, the authors introduce Supra, a hybrid adapter that combines the sparse updates from Super with low-rank adaptation (LoRA), maintaining a consistent parameter budget through a straightforward budget-splitting rule. The effectiveness of these methods is evaluated through experiments on two Llama-family models using the Math17K dataset, demonstrating that the proposed Super and Supra variants achieve superior accuracy compared to existing PEFT techniques. The findings suggest that pruning-inspired strategies can effectively inform sparse support for fine-tuning, particularly when integrated with low-rank adaptations.
Methodology
The authors propose Super, which selects a small set of weights for fine-tuning based on a Wanda-style activation-weighted magnitude score computed from calibration activations. They also introduce Supra, which integrates these sparse updates with LoRA while adhering to a fixed parameter budget. The methods are evaluated on two Llama-family models using the Math17K dataset, focusing on arithmetic reasoning tasks.
Results
The experiments reveal that the best-performing variants of Super and Supra achieve the highest average accuracy among various tested configurations, outperforming traditional PEFT methods such as LoRA, SIFT, and RoSA. The results indicate that both low-score supports and Wanda-style selections can effectively enhance fine-tuning performance.
Implications
The findings suggest that leveraging pruning-inspired techniques for parameter-efficient fine-tuning can significantly reduce the computational burden associated with adapting large language models. This approach has potential applications in resource-constrained environments where efficient model deployment is critical.
ERP Data Provisioning Financial Control Testing
Optimization
- SEQ-FCT framework combines multiple data provisioning techniques to ensure secure and effective financial control testing.
- The framework preserves financial control behavior while minimizing the risk of sensitive data exposure.
- Evaluation using a synthetic dataset shows high performance in reconciliation and fraud detection metrics.
- The approach emphasizes the importance of governance and validation in data provisioning processes.
Read more
ERP Data Provisioning Financial Control Testing
Summary
The paper presents a framework called Secure ERP Quality Provisioning for Financial Control Testing (SEQ-FCT) aimed at addressing the challenges of securely provisioning ERP data for financial control testing. Traditional methods of data provisioning often expose sensitive information or fail to maintain the necessary relationships within the data. SEQ-FCT combines several techniques, including deterministic masking, synthetic scenario expansion, referential tokenization, policy-based release approval, and automated validation to create a governed data-provisioning pipeline. The framework is evaluated using a synthetic dataset containing 186,000 finance-process records from six subsidiaries, which includes various financial documents and attributes necessary for testing. The results demonstrate that SEQ-FCT achieves high performance metrics, including a reconciliation F1 score of 0.932 and a fraud-trigger recall of 0.887, while maintaining a low leakage-risk score of 0.018. The analysis indicates that integrating masking, synthetic data generation, and governance checks into a single pipeline enhances the reliability of financial process behavior preservation.
Methodology
The SEQ-FCT framework employs a combination of deterministic masking, synthetic data generation, referential tokenization, and policy-based release approval. It uses a synthetic dataset for evaluation, ensuring controlled internal consistency and allowing for the assessment of various financial processes and relationships.
Results
SEQ-FCT achieved a reconciliation F1 score of 0.932, a fraud-trigger recall of 0.887, and a control-failure F1 score of 0.914. The estimated leakage-risk score was 0.018, indicating effective data protection while maintaining process fidelity.
Implications
The SEQ-FCT framework can be applied in various financial organizations to enhance the security and effectiveness of financial control testing. It provides a structured approach to data provisioning that balances privacy and operational needs, potentially influencing best practices in ERP data management.
Domain-Aware Scaling Laws Uncover Data Synergy
NLP
Large Language Models
Theory
- Introduces a formal definition of data synergy in the context of LLM pretraining.
- Develops domain-aware scaling laws that account for first-order and second-order synergies.
- Demonstrates improved predictive accuracy over traditional domain-agnostic scaling laws.
- Provides actionable insights for data curation and acquisition based on synergy estimates.
Read more
Domain-Aware Scaling Laws Uncover Data Synergy
Summary
This paper investigates the impact of data composition on the performance of large language models (LLMs) during pretraining, introducing the concept of data synergy. The authors formalize data synergy as the non-additive interactions between datasets from different domains, which can either enhance or degrade model performance. They distinguish between two types of synergy: domain-to-benchmark synergy, which measures how one domain affects performance on another benchmark, and domain-domain synergy, which captures the interactions between multiple domains in a training mixture. The authors propose domain-aware scaling laws that incorporate these synergies, improving predictive accuracy over traditional domain-agnostic scaling laws. By leveraging observational data from various LLMs with different pretraining mixtures, they derive synergy estimates that provide insights into optimal data combinations. The framework is validated through experiments that demonstrate its ability to predict performance rankings based on the synergy estimates, confirming the importance of data composition in LLM training.
Methodology
The authors leverage observational variation across open-weight LLMs with diverse pretraining mixtures to develop domain-aware scaling laws. They modify standard scaling laws to include terms that capture both first-order domain-to-benchmark synergy and second-order domain-domain synergy. This approach allows for the estimation of synergy coefficients that reflect the interactions between different data domains.
Results
The proposed framework successfully recovers stable and interpretable synergy estimates, revealing positive interactions between code and math domains, as well as negative interactions among certain domain pairs. The synergy estimates accurately predict performance rankings when models are trained on both optimal and anti-optimal data mixtures, demonstrating the framework's effectiveness.
Implications
The findings suggest that careful consideration of data composition can lead to significant improvements in model performance. The insights gained from the synergy estimates can guide researchers and practitioners in curating and acquiring training data more effectively, ultimately enhancing the capabilities of LLMs.
SPARC-Net: A Spectral, Causality-Aware, and Hard-Constrained Physics-Informed Architecture for Stiff and Shock-Dominated Partial Differential Equations
Theory
Optimization
- SPARC-Net addresses multiple concurrent issues in PINNs for stiff PDEs.
- The architecture includes a hard-constraint output ansatz to enforce boundary conditions.
- Significant improvements in accuracy were observed across various benchmark problems.
- The framework is open-source and replicable, promoting further research and application.
Read more
SPARC-Net: A Spectral, Causality-Aware, and Hard-Constrained Physics-Informed Architecture for Stiff and Shock-Dominated Partial Differential Equations
Summary
The paper introduces SPARC-Net, a novel architecture designed to address the challenges faced by Physics-Informed Neural Networks (PINNs) when solving stiff and shock-dominated partial differential equations (PDEs). The authors identify four key issues that contribute to the failure of traditional PINNs: spectral bias against sharp features, imbalanced multi-term optimization, violation of temporal causality, and under-resolved collocation. SPARC-Net integrates an adaptive multi-scale spectral encoder with a learnable spectral gate, a gated residual backbone, adaptive activations, and a hard-constraint output ansatz to enforce initial and boundary conditions. The training framework employs stabilized gradient-norm loss balancing, causality-respecting residual weighting, and residual-based adaptive collocation (RAD). The architecture and training methods are designed to work in concert, addressing the identified pathologies simultaneously. The effectiveness of SPARC-Net is validated against four canonical benchmarks, demonstrating significant reductions in error rates compared to traditional PINNs. The authors provide comprehensive evaluations, including ablation studies and hyperparameter sensitivity analyses, establishing SPARC-Net as a robust solution for challenging PDEs.
Methodology
SPARC-Net employs a unified architecture that combines an adaptive multi-scale spectral encoder, gated residual networks, and adaptive activations. The training process incorporates stabilized gradient-norm loss balancing, floored causality-respecting residual weighting, and residual-based adaptive collocation (RAD) to enhance performance on stiff and shock-dominated PDEs.
Results
SPARC-Net achieved substantial error reductions in various benchmarks: a 22% reduction on the viscous Burgers' equation, a 94% reduction on the Allen-Cahn equation, and a 100% reduction on the reaction equation. Additionally, the convection error was reduced from 5.14×10^-1 to 9.88×10^-5, demonstrating the architecture's effectiveness.
Implications
The development of SPARC-Net has significant implications for solving complex PDEs in various scientific and engineering domains, particularly where traditional methods struggle. Its open-source nature encourages further exploration and adaptation in related fields.
LeRoPE: Learnable RoPE Frequencies Improve Language Modeling
NLP
Large Language Models
- LeRoPE introduces learnable frequencies for Rotary Positional Encodings, enhancing model adaptability.
- The method consistently outperforms traditional RoPE and partial RoPE across various model sizes.
- At the largest model scale, RoPE requires more compute to match LeRoPE's performance.
- A dominant positional band was identified, highlighting its importance in attention mechanisms.
Read more
LeRoPE: Learnable RoPE Frequencies Improve Language Modeling
Summary
This paper introduces Learned RoPE (LeRoPE), a novel approach to Rotary Positional Encodings (RoPE) in language models. Traditional RoPE uses fixed base-frequency hyperparameters to determine the rotation rates of query and key vectors based on their positional offsets. The authors propose a modification where these frequencies are treated as learnable parameters, allowing the model to adaptively optimize them during training. The study involves training a series of language models ranging from 52M to 2.5B parameters, demonstrating that LeRoPE consistently outperforms both standard RoPE and partial RoPE across all model sizes. Notably, at the largest scale, RoPE requires 3.4% more compute (FLOPs) to achieve performance levels comparable to LeRoPE. The analysis reveals the emergence of a dominant positional band that significantly contributes to the attention logits, indicating a refined understanding of how positional information is utilized in language modeling.
Methodology
The authors modified the RoPE framework by incorporating learnable scalars for each frequency band, resulting in a total of 32 parameters for the models. They trained multiple language models from scratch, varying in size from 52M to 2.5B parameters, to evaluate the performance of LeRoPE compared to traditional RoPE and partial RoPE. The models were assessed based on their in-distribution loss and compute requirements.
Results
LeRoPE consistently achieved lower in-distribution loss across all model sizes compared to RoPE and partial RoPE. At the largest model scale (2.5B parameters), RoPE required 3.4% more compute to match the performance of LeRoPE, indicating a significant efficiency improvement with the learnable approach. The analysis also uncovered a consistent frequency profile and a dominant positional band that contributed to the model's performance.
Implications
The findings suggest that allowing frequencies in positional encodings to be learnable can lead to more efficient and effective language models. This approach may enhance the performance of various Transformer-based architectures, potentially influencing future research in positional encoding methods and their applications in NLP tasks.
The Spectral Structure of Latent Treatment Effects
Theory
- Introduces a spectral identification theorem for latent treatment effects using a compressed observable operator.
- Demonstrates that latent treatment effects correspond to the eigenvalues of a difference operator derived from treatment and control moment factorizations.
- Extends the Synthetic Potential Outcomes framework to handle overcomplete proxy systems.
- Establishes high-probability bounds for treatment effects and mixture proportions, improving causal inference reliability.
Read more
The Spectral Structure of Latent Treatment Effects
Summary
This paper addresses the challenge of identifying heterogeneous treatment effects in the presence of unobserved confounding variables, a critical issue in observational causal inference. The authors build upon the Synthetic Potential Outcomes (SPO) framework, which previously allowed for the recovery of treatment effect mixtures through scalar moments derived from observable data. They propose a new approach that utilizes a compressed observable operator, revealing that the latent treatment effects can be represented as the eigenvalues of a difference operator derived from the treatment and control moment factorizations. This spectral representation not only simplifies the identification of treatment effects but also extends the applicability of the SPO framework to overcomplete proxy systems, where the number of observed proxies exceeds the number of latent factors. The authors demonstrate that every scalar moment in the SPO framework can be expressed as a bilinear functional of powers of this operator, thereby providing a more robust and efficient method for estimating treatment effects and their distributions. The findings suggest that the spectral approach can yield high-probability bounds for treatment effects and mixture proportions, enhancing the reliability of causal inference in complex observational studies.
Methodology
The authors develop a spectral framework that identifies latent treatment effects through the eigenvalues of a compressed difference operator. They project observable data onto a shared signal subspace and derive treatment effect estimates from the spectral properties of this operator, allowing for the recovery of treatment effect distributions without needing to reconstruct the joint mixture of latent variables.
Results
The main results indicate that the latent treatment effects can be efficiently identified as the eigenvalues of the compressed difference operator, with the corresponding lifted left eigenvectors providing estimates for the target-proxy feature matrix and latent mixture proportions. The method demonstrates robustness in overcomplete proxy systems and offers high-probability bounds for treatment effects.
Implications
This work has significant implications for fields requiring causal inference from observational data, such as precision medicine and social sciences. The ability to accurately estimate heterogeneous treatment effects can inform better decision-making and policy formulation based on observed outcomes.
EvoLP: Self-Evolving Latency Predictor for Model Compression in Real-Time Edge Systems
Efficient ML
- EvoLP provides an efficient and accurate latency prediction for DNNs on edge devices.
- The framework reduces the search space for latency measurement, improving efficiency.
- EvoLP incorporates a self-evolution mechanism to enhance prediction precision during model compression.
- Experimental results show superior performance compared to state-of-the-art latency predictors.
Read more
EvoLP: Self-Evolving Latency Predictor for Model Compression in Real-Time Edge Systems
Summary
The paper introduces EvoLP, a novel framework designed to predict the inference latency of deep neural network (DNN) models on edge devices, addressing the challenges of real-time applications and resource limitations. Traditional methods for measuring latency are costly and time-consuming, often requiring extensive sampling and interaction with physical devices. EvoLP leverages a Multi-Layer Perceptron (MLP) and a self-evolution scheme to enhance latency prediction accuracy during the model compression process. The framework reduces the search space by focusing on the specific model to be compressed, allowing for efficient sampling and analysis. Experimental evaluations demonstrate that EvoLP significantly outperforms existing latency prediction methods across three different edge devices and four model variants. Furthermore, when integrated into a model compression framework, EvoLP effectively guides the compression process, ensuring that the resulting models meet strict latency constraints while maintaining high accuracy. The authors have made EvoLP publicly available for further research and application.
Methodology
EvoLP utilizes a Multi-Layer Perceptron (MLP) to predict inference latency and employs a self-evolution scheme that allows the predictor to adapt and improve its accuracy during the model compression process. The framework focuses on the specific model being compressed, generating a limited number of samples to analyze, which enhances efficiency and reduces the need for extensive latency measurements.
Results
EvoLP was evaluated on three edge devices and four model variants, demonstrating significantly higher prediction accuracy compared to existing methods. The integration of EvoLP into a model compression framework successfully guided the compression process, ensuring that the compressed models met strict latency constraints while achieving high accuracy.
Implications
The development of EvoLP has significant implications for deploying deep learning applications on edge devices, particularly in scenarios requiring real-time processing and low latency. Its ability to accurately predict latency can facilitate more efficient model compression strategies, ultimately enhancing the performance of embedded systems in various applications such as autonomous driving and augmented reality.
Invariant Learning Dynamics of Transformers in Inductive Reasoning Tasks
NLP
Large Language Models
Theory
- Introduction of the Invariant Manifold of Inductive Reasoning (IMIR) as a theoretical framework for understanding Transformer learning dynamics.
- Establishment of a generalized class of inductive tasks that unifies various existing tasks in the literature.
- Theoretical proof of the existence of the IMIR, enabling a more interpretable analysis of learning dynamics.
- Investigation of the impact of data statistics and random initializations on circuit competition in Transformers.
Read more
Invariant Learning Dynamics of Transformers in Inductive Reasoning Tasks
Summary
This paper introduces a theoretical framework to elucidate the emergence of inductive reasoning capabilities in Transformer language models. Unlike previous studies that focused on specific tasks, the authors propose a generalized class of inductive tasks that encompasses various synthetic tasks, such as in-context n-grams and multi-hop reasoning. They establish the existence of an Invariant Manifold of Inductive Reasoning (IMIR), a low-dimensional subspace of the parameter space where the training dynamics of attention models are confined. This manifold allows for a more interpretable analysis of learning dynamics, as it reduces the complexity from millions of parameters to a few interpretable coordinates. The authors investigate how data statistics influence the competition between in-context and in-weights learning, and how random initializations affect the emergence of 'winning' circuits. The framework also facilitates the automatic detection of learned circuits in trained models, contributing to a predictive theory of Transformer learning dynamics.
Methodology
The authors utilize theoretical proofs to describe the IMIR and its properties, conduct circuit competition studies to analyze the dynamics between different learning strategies, and employ empirical methods to demonstrate the framework's applicability in detecting learned circuits.
Results
The study confirms the existence of the IMIR and illustrates how it constrains the learning dynamics of Transformers. It shows that data distribution significantly influences the competition between in-context and in-weights learning, and that random initializations can lead to distinct 'winning' circuits, revealing sharp phase transitions in initialization space.
Implications
This work provides a foundational framework for understanding the learning dynamics of large language models, which can lead to improved model interpretability, safer AI systems, and more effective training methodologies. It opens avenues for future research on the dynamics of circuit formation in Transformers.
GatedLinear: Adaptive Routing of Complementary Linear Bases for Time Series Forecasting
Time Series
- GatedLinear offers a lightweight framework for time series forecasting by utilizing adaptive routing of complementary linear bases.
- The Tri-Factorized Fusion Gate enables dynamic and interpretable routing decisions tailored to specific temporal dynamics.
- GatedLinear achieves state-of-the-art accuracy on benchmark datasets while maintaining a smaller parameter footprint compared to traditional deep learning models.
- The framework effectively captures diverse temporal patterns, addressing the heterogeneity of real-world time series data.
Read more
GatedLinear: Adaptive Routing of Complementary Linear Bases for Time Series Forecasting
Summary
The paper introduces GatedLinear, a novel framework for time series forecasting that addresses the limitations of traditional deep learning models which often rely on a single computational backbone. GatedLinear is designed to capture the diverse temporal dynamics present in real-world time series data, such as smooth trends, nonstationary drifts, and cyclic patterns. The framework utilizes three specialized linear bases: a global trend-seasonal basis for smooth projections, a difference-based incremental basis for nonstationary drift, and a phase-aligned recurrence basis for cyclic reuse. To effectively manage the routing of these bases, the authors propose a Tri-Factorized Fusion Gate that allows for dynamic, point-wise routing decisions based on channel-specific preferences, forecast horizons, and phase-indexed biases. This approach enables GatedLinear to achieve state-of-the-art forecasting accuracy while maintaining a smaller parameter footprint compared to complex foundational models. The experiments conducted demonstrate that GatedLinear not only outperforms existing methods on standard benchmarks but also provides interpretable routing patterns, making it a promising solution for diverse time series forecasting challenges.
Methodology
GatedLinear employs three linear bases to model different temporal dynamics: a trend-seasonal basis for smooth projections, an incremental basis for nonstationary drift, and a recurrence basis for cyclic patterns. The Tri-Factorized Fusion Gate orchestrates the routing of these bases based on specific characteristics of the forecast point, such as variable, horizon, and future phase index, allowing for fine-grained control over the forecasting process.
Results
The experimental results indicate that GatedLinear achieves competitive or superior accuracy compared to recent complex models on standard time series forecasting benchmarks. The model's ability to provide interpretable routing patterns and operate with fewer parameters highlights its efficiency and effectiveness.
Implications
GatedLinear's approach to adaptive routing in time series forecasting has potential applications across various domains, including finance, energy management, and industrial monitoring, where capturing diverse temporal dynamics is crucial for accurate predictions.
Data-Efficient Deep Learning: Empirical Guidelines for Training Set Size Estimation in Inertial Sensor Classification
Efficient ML
Time Series
- Classification accuracy in inertial sensor tasks follows a logarithmic growth pattern.
- Introduces a stability point metric for optimizing training data collection.
- Models often reach stability with fewer samples than traditional heuristics suggest.
- Provides a unified framework for analyzing performance in binary and multi-class scenarios.
Read more
Data-Efficient Deep Learning: Empirical Guidelines for Training Set Size Estimation in Inertial Sensor Classification
Summary
This paper addresses the challenge of determining the minimum sample size required for effective classification in inertial sensor-based tasks, specifically in human activity recognition (HAR) and smartphone location recognition (SLR). The authors conduct a systematic empirical evaluation of learning curve convergence rates across six diverse datasets, totaling 102.7 hours of inertial data. They find that classification accuracy follows a logarithmic growth pattern, which is consistent regardless of the complexity of the task. A novel stability point metric is introduced to optimize data collection, indicating that models can achieve stability with fewer samples than traditional heuristics suggest. The study provides a unified framework for analyzing classification performance in both binary and multi-class scenarios, ultimately shifting the focus from maximizing data volume to optimizing data efficiency. This research offers practical guidelines for planning data collection campaigns in inertial sensing applications, enabling more efficient use of resources in data gathering.
Methodology
The authors conducted an empirical evaluation of learning curves using six diverse datasets. They analyzed classification performance under both binary and multi-class scenarios and derived an empirical formula for estimating performance relative to dataset size. The stability point metric was defined to quantify the sample size needed for the learning curve to stabilize.
Results
The analysis revealed that classification accuracy consistently follows a logarithmic growth pattern, and models can achieve practical stability with significantly fewer samples than previously recommended by heuristics. The proposed framework allows for extrapolating total data requirements from small-scale pilot studies.
Implications
The findings can significantly impact the planning and execution of data collection campaigns in inertial sensing applications, leading to more efficient use of resources and potentially accelerating the deployment of machine learning models in real-world scenarios.
JEPA for AI-Native 6G: Predictive Representations and Open Challenges
Multimodal
- JEPA offers a novel self-supervised learning approach for AI-native 6G networks.
- The architecture predicts future representations, improving label efficiency and robustness.
- The paper highlights the importance of wireless-aware design choices in JEPA.
- An illustrative case study shows practical applications of JEPA in beam management.
Read more
JEPA for AI-Native 6G: Predictive Representations and Open Challenges
Summary
The paper discusses the transition of sixth-generation (6G) networks towards AI-native operations, emphasizing the integration of learning modules across various network components. It introduces the Joint-Embedding Predictive Architecture (JEPA) as a self-supervised learning paradigm that predicts future representations in latent space, rather than reconstructing raw data. This approach addresses challenges such as limited labeled data, heterogeneous network conditions, and non-stationary environments. The authors provide a tutorial on JEPA's training mechanism, detailing how various data types (e.g., channel state information, beam measurements) can be tokenized and masked for effective learning. An illustrative case study on beam management demonstrates that using a wireless-aware target during self-supervised pretraining enhances label efficiency and robustness under varying deployment conditions. The paper concludes by outlining several open challenges in the field, including multi-timescale prediction, action-conditioned modeling, and the need for efficient deployment and benchmarking in AI-native 6G networks.
Methodology
The authors present a tutorial on the JEPA training mechanism, which involves tokenizing and masking various wireless data types. They illustrate the application of JEPA through a case study on beam management, focusing on predicting future beam-energy targets during self-supervised pretraining.
Results
The case study indicates that utilizing a wireless-aware target during pretraining significantly improves label efficiency and robustness across different deployment conditions compared to traditional supervised learning methods.
Implications
The findings suggest that JEPA can enhance the performance of AI-native 6G networks by enabling more efficient learning from limited labeled data. This has implications for various applications, including network management, resource allocation, and anomaly detection in dynamic environments.
Learning Subgroup Relations Using Siamese Graph Neural Networks
Graph Learning
- Introduction of a Siamese GNN for subgroup prediction in finite groups.
- Integration of graph embeddings from Cayley graphs with algebraic features.
- Achieved a high test accuracy of 95.9% on subgroup relation prediction.
- Demonstrated the effectiveness of combining graph-based and algebraic information.
Read more
Learning Subgroup Relations Using Siamese Graph Neural Networks
Summary
This paper addresses the problem of determining whether one finite group is isomorphic to a subgroup of another, a fundamental issue in computational group theory. The author proposes a novel approach using a Siamese Graph Neural Network (Siamese GNN) that leverages Cayley graph representations of finite groups. Each group is represented as an undirected Cayley graph, which is processed by a shared GNN encoder to produce graph embeddings. These embeddings are then combined with algebraic features derived from the groups to create a joint feature vector, which is classified to predict subgroup relations. The methodology integrates structural and algebraic information, providing a unified framework for subgroup prediction. The experimental results demonstrate the effectiveness of the proposed architecture, achieving a test accuracy of 95.9% on an independent test set, highlighting the potential of geometric deep learning in this domain.
Methodology
The proposed methodology involves representing finite groups as undirected Cayley graphs, which are processed by a shared GNN encoder to obtain graph embeddings. These embeddings are concatenated with algebraic features extracted from the groups to form a joint feature vector, which is then classified to determine subgroup relations.
Results
The proposed Siamese GNN architecture achieved a test accuracy of 95.9% on an independent test set, demonstrating its effectiveness in predicting subgroup relations between finite groups.
Implications
This work has significant implications for computational group theory, providing a machine learning approach that complements traditional algebraic methods. It opens avenues for further research in applying geometric deep learning techniques to algebraic structures and could enhance the understanding of subgroup relationships in various mathematical contexts.
Distance-Preserving Embeddings in Inhomogeneous Random Graphs
Graph Learning
Theory
Efficient ML
- Introduces landmark-based embeddings for inhomogeneous random graphs, improving upon worst-case distortion guarantees.
- Achieves tighter dimension-distortion trade-offs, demonstrating polynomial improvements in embedding dimensions.
- Extends theoretical guarantees to global averages and introduces a GNN-augmented variant for efficient shortest-path approximations.
- Establishes a novel metric sandwiching framework that unifies analysis across various latent spaces.
Read more
Distance-Preserving Embeddings in Inhomogeneous Random Graphs
Summary
This paper addresses the challenge of creating distance-preserving embeddings in graph machine learning, particularly focusing on inhomogeneous random graphs (IHGs) which exhibit structural heterogeneity. Traditional methods for embedding nodes often rely on worst-case distortion guarantees that do not accurately reflect the characteristics of typical large-scale networks. The authors propose a novel approach using landmark-based embeddings that retain shortest paths to a small set of reference nodes, effectively functioning as virtual graph spanners. By analyzing the properties of IHGs, the authors derive tighter dimension-distortion trade-offs compared to classical methods, achieving significant improvements in embedding dimensions for both (1 - ε) and (1 + ε) distortions. They extend their findings to global averages and introduce a GNN-augmented variant that utilizes neural networks to approximate shortest-path queries, demonstrating robust generalization capabilities to real-world networks. This work not only enhances the theoretical understanding of distance-preserving embeddings but also provides practical tools for efficient graph representation and analysis.
Methodology
The authors analyze landmark-based distance embeddings on inhomogeneous random graphs using a probabilistic framework based on multi-type branching processes. They derive sharp point-wise guarantees for embedding dimensions and extend these results to global averages. Additionally, they introduce a GNN-augmented approach that replaces traditional shortest-path queries with neural network surrogates, leveraging the alignment between graph neural networks and dynamic programming principles.
Results
The study shows that landmark-based embeddings can achieve embedding dimensions of Ω(n^(1-ε) log n) for (1 ± ε)-distortions, significantly smaller than classical worst-case bounds. The authors also demonstrate that their GNN-augmented variant maintains fidelity to classical embeddings while enhancing computational efficiency and generalization to larger networks.
Implications
The findings have significant implications for graph machine learning, particularly in applications requiring efficient and accurate representations of complex networks. The proposed methods can enhance tasks such as node classification, link prediction, and routing in large-scale networks, making them more feasible and effective.
Learning from Local Walks on Dynamic Graphs with Bandit Feedback
Graph Learning
Theory
Reinforcement Learning
- Introduces a novel framework for stochastic multi-armed bandits on dynamic graphs with local movement constraints.
- Establishes a structural condition (sliding-window mixing) that ensures stable exploration and navigation.
- Analyzes local explore-then-commit algorithms achieving sublinear expected regret.
- Proposes a reward-aware strategy with formal safety and performance guarantees.
Read more
Learning from Local Walks on Dynamic Graphs with Bandit Feedback
Summary
This paper investigates the stochastic multi-armed bandit problem in the context of dynamic graphs, where the arms correspond to vertices with time-varying edges. The authors highlight the challenge of local movement constraints, which restrict the learner to selecting only its current node or immediate neighbors, complicating the identification and exploitation of the best arm. They introduce a structural condition based on sliding-window mixing that ensures the graph's intrinsic walk remains stable for exploration and navigation. The paper analyzes a family of local explore-then-commit algorithms, proving that they achieve sublinear expected regret under this condition. Additionally, a reward-aware strategy is presented, accompanied by a worst-case safety theorem and a performance gain theorem. The findings emphasize the importance of maintaining navigability in dynamic environments to facilitate effective learning.
Methodology
The authors adopt an explore-then-commit framework, focusing on a canonical walk where the learner can either stay at its current node or move to a neighboring node. They formalize a shift-invariant navigability condition based on the canonical walk's stationary distribution and analyze the performance of various local algorithms under this regime.
Results
The paper demonstrates that under the proposed sliding-window mixing condition, the local explore-then-commit algorithms achieve sublinear expected regret, indicating effective learning in dynamic graph environments. The reward-aware strategy is shown to provide both safety and performance improvements.
Implications
The findings have significant implications for applications in sensor networks, peer-to-peer systems, and any scenario where agents operate under local movement constraints in dynamic environments. The results can inform the design of more efficient learning algorithms in such contexts.
BlockServe: Block-Grained Continuous Batching for High-Throughput Diffusion LLM Serving
Large Language Models
Generative Models
Efficient ML
- Introduction of a block-grained scheduler to reduce straggler-induced compute bubbles in dLLM inference.
- Development of a mixed-state memory manager that supports heterogeneous batch processing without custom kernels.
- Implementation of a compute-aware admission controller that enhances effective batch capacity through token budgeting.
- Achieved 1.9–10.6× throughput improvement over Fast-dLLM with similar generation quality.
Read more
BlockServe: Block-Grained Continuous Batching for High-Throughput Diffusion LLM Serving
Summary
The paper presents BlockServe, a novel continuous batching framework designed to enhance the serving efficiency of diffusion large language models (dLLMs). The authors identify a challenge known as convergence heterogeneity, where requests in a batch complete at different rates, leading to inefficiencies such as compute bubbles and increased tail latency. BlockServe addresses this issue by implementing a block-grained scheduling system that allows for immediate eviction of completed requests at block boundaries, thus preventing slower requests from stalling the entire batch. Additionally, it employs a mixed-state memory manager that facilitates parallel decoding and dual cache usage for heterogeneous batches, and a compute-aware admission controller that dynamically adjusts batch sizes based on a token budget. The framework is evaluated on two models, Dream and LLaDA, across five benchmarks, demonstrating significant throughput improvements over existing methods while maintaining comparable generation quality.
Methodology
BlockServe employs a continuous iteration loop centered on block-grained scheduling and mixed-state execution. It uses a block-grained scheduler to manage requests at the block level, allowing for parallel processing of requests at different stages. The mixed-state memory manager organizes requests into a unified dense tensor layout for efficient execution, while the compute-aware admission controller selects requests based on a token budget to optimize batch size and throughput.
Results
BlockServe demonstrated a throughput increase of 1.9 to 10.6 times compared to Fast-dLLM across five benchmarks, while maintaining comparable quality in generated outputs. This significant improvement highlights the effectiveness of the proposed block-grained scheduling and mixed-state execution strategies.
Implications
The advancements presented in BlockServe could lead to more efficient deployment of diffusion LLMs in real-world applications, particularly in scenarios requiring high-throughput processing of multiple requests. This framework could be instrumental in enhancing the performance of AI systems that rely on large-scale language models, making them more accessible and efficient for various applications.
Learning More from Less: Reinforcement Learning from Hindsight
Reinforcement Learning
Robotics
Multimodal
- Introduces Learning from Hindsight (LfH) to improve sample efficiency in RL for VLA models.
- Utilizes hindsight relabeling to convert failed rollouts into training signals.
- Achieves a 5× improvement in sample efficiency on LIBERO-PRO tasks.
- Demonstrates that relabeling can be more beneficial than dense feedback in sparse reward scenarios.
Read more
Learning More from Less: Reinforcement Learning from Hindsight
Summary
This paper addresses the challenge of sample efficiency in reinforcement learning (RL) for vision-language-action (VLA) models, particularly in robotics where data collection is costly and slow. The authors introduce a novel approach called Learning from Hindsight (LfH), which utilizes hindsight relabeling to convert failed rollouts into useful training signals. Instead of discarding rollouts that do not meet the original task requirements, LfH evaluates these rollouts based on the tasks they actually achieved. A vision-language model (VLM) is employed to relabel both the instruction and reward for these rollouts, allowing the RL policy to learn from both intended successes and unintended but meaningful behaviors. The method is tested on out-of-distribution LIBERO-PRO tasks, demonstrating a significant improvement in sample efficiency—achieving a 5× increase compared to standard RL methods. The results indicate that LfH not only accelerates the learning process but also outperforms traditional dense progress-reward baselines, highlighting the effectiveness of relabeling in sparse reward environments.
Methodology
The methodology involves using a single vision-language model (VLM) to relabel both the instructions and rewards of failed rollouts. The VLM analyzes the achieved behaviors and proposes hindsight instructions, scoring how well each rollout satisfies these new tasks. The RL policy is then trained on both the original and relabeled rollouts, allowing it to learn from a broader set of experiences.
Results
The LfH method significantly enhances sample efficiency, achieving the same success rate as standard RL in one-fifth of the training steps. It outperforms a dense progress-reward baseline, indicating that relabeling failed rollouts can provide more valuable training signals in environments with sparse rewards.
Implications
The findings suggest that LfH can make RL post-training more practical for robotics and other applications where data collection is limited. By effectively utilizing failed attempts, this approach can lead to faster learning and improved performance in complex tasks.
On-Device Adaptive Battery Power Prediction for Electric Vehicles
Time Series
- Introduces on-device adaptive learning for battery power prediction in EVs.
- Demonstrates significant performance improvements through online and offline adaptation strategies.
- Highlights the limitations of a 'one model fits all' approach in dynamic driving conditions.
- Evaluates state-of-the-art models for very short-term forecasting (1-3 seconds).
Read more
On-Device Adaptive Battery Power Prediction for Electric Vehicles
Summary
This paper addresses the challenge of accurate battery power prediction in Electric Vehicles (EVs) by introducing a novel on-device adaptive learning approach. Traditional deep learning models for time-series forecasting often struggle with data distribution shifts, which can degrade their performance. The authors propose a method that allows for continuous adaptation of pretrained battery prediction models directly on resource-constrained EV systems. This involves transforming existing models into adaptable versions that retain essential hyperparameter knowledge. The study explores both online and offline adaptation strategies, demonstrating significant improvements in forecasting performance. The results indicate mean absolute error reductions of up to 7.49% with online adaptation and 14.88% with offline adaptation, showcasing the effectiveness of on-device learning in enhancing battery power predictions in real-world scenarios. The paper emphasizes the importance of adapting to dynamic driving conditions and user behaviors, which significantly influence EV power demand, and presents a comprehensive evaluation of state-of-the-art models for short-term battery power forecasting.
Methodology
The authors trained and evaluated various state-of-the-art deep learning models for battery power prediction, focusing on short-term forecasting horizons of 1 to 3 seconds. They implemented two adaptation strategies: online learning, which allows models to process and adapt to incoming data streams in real-time, and offline learning, which enables models to analyze historical data to learn power consumption patterns. The models were deployed on resource-constrained edge devices to simulate in-vehicle processing capabilities.
Results
The study achieved mean absolute error reductions of up to 7.49% with online adaptation and 14.88% with offline adaptation, indicating substantial improvements in forecasting accuracy compared to unadapted models. These results demonstrate the effectiveness of on-device learning in adapting to changing data distributions and improving battery power predictions.
Implications
The findings suggest that on-device adaptive learning can significantly enhance the reliability of battery power predictions in EVs, leading to better battery management and optimization of power supply. This approach can also reduce reliance on cloud-based solutions, thereby lowering infrastructure costs and improving data privacy.
Group Invariant Spectral Embedding
Graph Learning
Theory
Efficient ML
- Introduces group-invariant affinity kernels for spectral embedding.
- Proves convergence of graph Laplacians to differential operators on quotient spaces.
- Demonstrates improved convergence rates and effective dimension reduction.
- Validates the approach on datasets with SO(2) and SO(3) symmetries.
Read more
Group Invariant Spectral Embedding
Summary
This paper addresses the limitations of standard spectral embedding methods in handling datasets with intrinsic symmetries, such as rotations. Traditional methods treat symmetry-related data points as unrelated, which can lead to suboptimal embeddings. The authors propose a novel approach that incorporates symmetries directly into the affinity kernels used for spectral embedding. They focus on Riemannian data manifolds with symmetries defined by compact Lie groups and demonstrate that graph Laplacians constructed from group-invariant kernels converge to second-order differential operators on the quotient space of the manifold. This results in improved convergence rates and effective dimension reduction, as the effective dimension of the data is reduced according to the dimension of the symmetry group. The authors validate their approach on datasets exhibiting SO(2) and SO(3) symmetries, showing that their method recovers the intrinsic geometry of the data more effectively than standard spectral embedding, even with infinite data.
Methodology
The authors analyze three classes of group-invariant affinity kernels: minimization over the group, integration over the group, and G-invariant feature mappings. They establish theoretical results regarding the convergence of graph Laplacians constructed from these kernels to differential operators on the quotient manifold. The analysis is grounded in the context of Riemannian manifolds and compact Lie groups.
Results
The study demonstrates that using G-invariant kernels leads to a significant improvement in convergence rates compared to standard methods. Specifically, the effective dimension reduction is achieved by exploiting the symmetries in the data, resulting in more efficient embeddings that accurately reflect the intrinsic geometry of the datasets.
Implications
This work has potential applications in various fields where data exhibits symmetry, such as computer vision (e.g., cryo-electron microscopy) and graph learning. The proposed methods can enhance the performance of unsupervised learning tasks by providing more accurate representations of data with inherent symmetries.
Spectral Origins of the Self-Correction Blind Spot in Autoregressive Generation
NLP
Large Language Models
Reinforcement Learning
- Introduces SPARC, a formal model explaining the self-correction blind spot in autoregressive generation.
- Establishes that the blind spot arises when the spectral radius of the error-propagation operator is at least one.
- Derives a precise activation threshold for correction markers, recovering a significant reduction in blind-spot rates.
- Provides a convergence guarantee for RL-based self-correction training, highlighting the necessity of RL over supervised fine-tuning.
Read more
Spectral Origins of the Self-Correction Blind Spot in Autoregressive Generation
Summary
This paper addresses the self-correction blind spot observed in large autoregressive language models (LLMs), where models can correct errors attributed to external sources but fail to do so for errors in their own outputs. The authors introduce SPARC, a spectral-algebraic theory that formalizes this phenomenon by defining an error-propagation operator based on attention Jacobians. They prove that the self-correction blind spot occurs when the spectral radius of this operator is at least one. The paper also derives a sharp activation threshold for correction markers and provides a convergence guarantee for reinforcement learning (RL) based verifier-corrector training. The findings unify the understanding of self-correction across different autoregressive modalities, including text, image, and video generation. Experimental validation demonstrates that the theoretical predictions align closely with empirical results, achieving a 3.2% root mean square error in blind-spot rate predictions.
Methodology
The authors develop a spectral-algebraic theory, SPARC, to analyze the error-propagation dynamics in autoregressive models. They define the error-propagation operator as the product of attention Jacobians and derive theoretical results regarding the spectral radius, activation thresholds, and convergence rates for RL training. Experiments are conducted across multiple model backbones to validate the theoretical predictions.
Results
The paper proves that the self-correction blind spot is linked to the spectral radius of the error-propagation operator. It establishes a clear activation threshold for correction markers and demonstrates that RL-based training converges under specific conditions. Experimental results confirm that the spectral predictions match the observed blind-spot rates with a low error margin.
Implications
The findings have significant implications for improving the reliability of autoregressive models in practical applications, particularly in enhancing their self-correction capabilities. This could lead to more trustworthy deployments of LLMs in various domains, including natural language processing, image generation, and video synthesis.
Auditing Construct Overlap in Explainable Machine Learning: Evidence from Burnout-Depression Prediction Across Student Cohorts
Interpretability
- The study reveals that apparent stability in risk hierarchies from XML models is often due to construct overlap rather than genuine predictive relationships.
- A residualization protocol is introduced to quantify the impact of shared variance between correlated predictors and outcomes.
- The predictive performance of the model significantly decreases when controlling for the correlation between trait anxiety and depression.
- Individual-level clinical predictions are deemed non-actionable due to high uncertainty in prediction intervals.
Read more
Auditing Construct Overlap in Explainable Machine Learning: Evidence from Burnout-Depression Prediction Across Student Cohorts
Summary
This paper investigates the reliability of explainable machine learning (XML) models in predicting composite mental health outcomes, specifically burnout and depression among medical students. The authors apply an ElasticNet pipeline to a primary cohort of 886 medical students and validate it across additional datasets, revealing that the apparent stability of risk hierarchies is largely an artifact of construct overlap between predictors and outcomes. The study highlights that trait anxiety and health satisfaction consistently dominate the predictive hierarchy, but this dominance diminishes significantly when accounting for shared variance through a residualization protocol. The authors demonstrate that the model's predictive power drops drastically when controlling for the correlation between trait anxiety and depression, indicating that the observed stability across cohorts does not reflect a true risk structure. Additionally, the study provides evidence that individual-level predictions are not actionable due to high uncertainty in prediction intervals. The paper's main contribution is the introduction of a residualization protocol to audit construct overlap in XML studies, emphasizing the need for such checks in future research.
Methodology
The authors employed an ElasticNet regression model on a primary cohort of 886 medical students, validated across longitudinal observations and a non-medical student cohort. They introduced a residualization protocol to isolate shared variance between correlated predictors (trait anxiety and depression) and assessed the impact on model performance.
Results
The study found that the model's R2 value dropped from 0.41 to 0.016 when controlling for the correlation between trait anxiety and depression, indicating that the apparent predictive dominance of trait anxiety was largely an artifact of construct overlap. The model maintained a consistent performance across cohorts (R2: 0.41–0.49), but this was attributed to the stable correlation rather than a generalizable risk structure. Prediction intervals averaged 35.4 units on a 0–100 scale, highlighting the limitations of individual-level predictions.
Implications
The findings suggest that machine learning models in mental health research must account for construct overlap to avoid misleading interpretations. The residualization protocol proposed can be applied in future studies to enhance the validity of risk assessments in explainable AI applications.
NL-PAC: Specification Ambiguity and Certified Minimax Risk Floors in LLM-Mediated Supervision
NLP
Large Language Models
Theory
- Introduction of the NL-PAC framework to address specification ambiguity in LLM supervision.
- Establishment of a worst-case risk floor that is independent of sample size, linked to the diameter of admissible labels.
- Certification of risk floors from unlabeled inputs, providing a method for auditing model performance.
- Empirical audits reveal the limitations of certain prompts in achieving admissibility for coherent readings.
Read more
NL-PAC: Specification Ambiguity and Certified Minimax Risk Floors in LLM-Mediated Supervision
Summary
This paper introduces the Natural Language PAC (NL-PAC) framework, which addresses the challenges posed by specification ambiguity in tasks supervised by large language models (LLMs). In scenarios where a natural language specification can be interpreted in multiple ways, the supervision channel may not clarify which interpretation is being used, leading to an identification problem. The NL-PAC framework formalizes this situation by defining admissible labels and candidate targets based on a fixed model's decoding law. The study reveals that the probability of multiple admissible labels corresponds to the diameter of the pointwise-admissible target class, and under target-blind supervision, learners face a worst-case risk that is at least half of this diameter, regardless of sample size. The paper also demonstrates that this risk floor can be certified using held-out unlabeled inputs. Through empirical audits, the authors show that specific prompts can yield positive model-relative certificates, while others do not meet the admissibility conditions necessary for coherent readings. The findings highlight the importance of understanding the interplay between task specification and the supervision channel in LLM-mediated tasks.
Methodology
The methodology involves formalizing the relationship between task specifications and the supervision channel in LLMs. The authors define a pointwise-admissible core and its tolerance class based on thresholded model outputs. They derive theoretical results regarding the minimax risk associated with target-blind supervision and develop finite-sample confidence bounds for certification using held-out unlabeled inputs.
Results
The paper presents four main results: (1) a representation of the pointwise-admissible core and its diameter; (2) the establishment of a minimax risk floor under target blindness; (3) a coherent readings condition that bounds the gap between theoretical values; and (4) a method for certifying the risk floor and diameter using Hoeffding certificates from observed admissible sets.
Implications
The findings have significant implications for the design and evaluation of LLM-mediated systems, particularly in understanding the risks associated with ambiguous task specifications. The ability to certify risk floors from unlabeled inputs can enhance the reliability of LLMs in practical applications, ensuring better alignment between model outputs and intended interpretations.
Gauge dependence and structured-output corruption in sign-branched repetition penalties: measurements across models, inference stacks, and alternative repetition controls
NLP
Large Language Models
Generative Models
- The multiplicative repetition penalty is gauge dependent, leading to inconsistent effects across different models.
- A fixed repetition penalty can result in a significant flip rate of generated tokens, affecting model outputs.
- Applying the repetition penalty to normalized log-probabilities instead of raw logits can eliminate the observed issues.
- The study highlights the importance of understanding the underlying mechanisms of logit manipulation in LLMs.
Read more
Gauge dependence and structured-output corruption in sign-branched repetition penalties: measurements across models, inference stacks, and alternative repetition controls
Summary
This paper investigates the implications of the multiplicative repetition penalty used in various large language model (LLM) inference systems, highlighting its gauge dependence and the resulting structured-output corruption. The author argues that the sign-branching mechanism, which modifies logits based on their sign, is not well-defined due to the arbitrary zero-point of logits across different models. This leads to significant variability in the application of the repetition penalty, resulting in a high flip rate of generated tokens and a drastic drop in valid outputs for structured formats like JSON. The paper presents empirical measurements across multiple models, demonstrating that applying the penalty to normalized log-probabilities instead of raw logits can mitigate these issues. The findings suggest that the current implementation of repetition penalties can lead to inconsistent behavior across models and should be reconsidered to improve output quality.
Methodology
The author conducted empirical measurements on five different LLMs, analyzing the effects of the repetition penalty across various configurations. The study involved decoding models with different constant shifts added to logits to assess the flip rate of generated tokens and the impact on structured outputs. The experiments were performed on datasets such as WikiText-103 and JSON schemas, comparing the performance of standard repetition penalties against normalized log-probabilities.
Results
The results indicated that the sign-branching repetition penalty led to a flip rate of 58-96% of greedy tokens at a typical penalty setting of θ = 1.3. Additionally, the rate of valid, schema-conformant outputs dropped from 97% to 23% when using the penalty on raw logits. In contrast, applying the penalty to normalized log-probabilities resulted in no significant flip rates, demonstrating a more stable and reliable output across models.
Implications
The findings suggest that the current implementation of repetition penalties in LLMs may lead to unpredictable and undesirable outputs, particularly in structured data generation. By adopting normalized log-probabilities for repetition penalties, developers can enhance the consistency and quality of model outputs, which is crucial for applications requiring high fidelity in structured formats.
Action-Factored Multi-Agent Reinforcement Learning for Scalable Quantum Device Tuning
Reinforcement Learning
Optimization
Robotics
- Introduction of QADAPT, a multi-agent reinforcement learning framework for quantum device tuning.
- Adaptive action-space factorization reduces cross-agent interference and improves sample efficiency.
- Zero-shot generalization allows the framework to adapt to unseen quantum device sizes without retraining.
- Empirical evaluations demonstrate superior performance compared to state-of-the-art tuning methods.
Read more
Action-Factored Multi-Agent Reinforcement Learning for Scalable Quantum Device Tuning
Summary
This paper introduces QADAPT, a novel framework for tuning electrostatically-defined quantum-dot arrays using cooperative multi-agent reinforcement learning (MARL). The authors address the challenges of parameter cross-talk and non-stationary environments that complicate the tuning of quantum devices. By employing an action-factored representation of the action space, QADAPT allows agents to operate independently while minimizing interference, thus enhancing learning efficiency. The framework utilizes a centralized training with decentralized execution (CTDE) paradigm, enabling agents to learn shared policies based on local observations. The authors demonstrate that QADAPT achieves zero-shot generalization across various quantum device sizes and configurations, maintaining consistent convergence steps to reach target tuning regimes. Empirical evaluations show that QADAPT outperforms existing state-of-the-art methods, providing a scalable solution for the rapid calibration of large-scale quantum processors.
Methodology
The authors developed a Kalman-guided pipeline to estimate the local gate-to-dot capacitance matrix online, which facilitates a factored representation of the action space. They formulated the quantum dot tuning problem as a cooperative continuous Decentralized Partially Observable Markov Decision Process (Dec-POMDP) and implemented a modular actor-critic architecture with shared parameters specific to gate types. This approach allows for decentralized execution while leveraging centralized training.
Results
QADAPT demonstrated zero-shot generalization across different quantum device sizes and configurations, achieving consistent convergence steps to target regimes. The empirical evaluations indicated that QADAPT outperformed various state-of-the-art baselines in terms of tuning efficiency and accuracy.
Implications
The findings suggest that QADAPT can significantly enhance the scalability and efficiency of quantum device tuning, making it a promising tool for the rapid calibration of large-scale quantum processors. This could facilitate advancements in quantum computing technologies and applications.
Multi-dimensional training-priority weighting based on physical information propagation paths: a unified residual-weighting framework for physics-informed neural networks
Theory
Optimization
- Introduces a unified framework for training priorities in PINNs based on physical information propagation paths.
- Demonstrates that standard PINNs do not respect the natural learning order of premise and dependent regions.
- Utilizes negative-exponential residual weights to enforce training priorities at the loss level.
- Implements a directional compatibility coefficient for managing multiple training priorities.
Read more
Multi-dimensional training-priority weighting based on physical information propagation paths: a unified residual-weighting framework for physics-informed neural networks
Summary
This paper addresses the limitations of Physics-Informed Neural Networks (PINNs) in solving partial differential equations (PDEs) by proposing a unified framework for training priority weighting based on the physical information propagation paths. Traditional PINNs optimize residuals from different regions and constraints equally, which can lead to instability and inaccuracies, especially in complex PDEs. The authors introduce a multi-dimensional priority-constraint framework that prioritizes learning based on the propagation path of physical information, ensuring that premise regions are learned before dependent regions. This approach is grounded in the neural tangent kernel (NTK) training dynamics, which reveals that standard PINNs do not inherently respect this priority. By implementing negative-exponential residual weights, the proposed method effectively translates the physical propagation order into training priorities at the loss level. The framework also includes a directional compatibility coefficient to manage cases with multiple priorities. Experimental results demonstrate significant improvements in convergence behavior and prediction accuracy for problems with clear propagation paths, without altering the network architecture or incurring substantial additional computational costs.
Methodology
The authors analyze the training dynamics of PINNs using neural tangent kernel (NTK) theory to establish why standard PINNs fail to respect physical information propagation paths. They propose a multi-dimensional priority-constraint framework that incorporates negative-exponential residual weights to prioritize learning based on the physical propagation order. The framework is tested on various benchmark cases to evaluate its effectiveness.
Results
The proposed framework consistently improves the convergence behavior and prediction accuracy of PINNs on benchmark problems characterized by clear physical propagation paths or constraint-dominated structures. The results indicate that the method enhances training stability without requiring changes to the underlying neural network architecture.
Implications
This research has significant implications for the application of PINNs in complex scientific computing scenarios, particularly in fields where the physical propagation of information is critical. The framework can enhance the reliability and accuracy of solutions to PDEs, potentially benefiting engineering, physics, and other domains that rely on accurate modeling of physical phenomena.