AI-generated summaries
Today's ML research,
without the noise.
Daily summaries of the latest machine learning papers from arXiv, processed every 8 hours.
24
Papers today
8h
Update frequency
7
Days of history
Doubly Robust Estimation of Causal Effect on CVR with Targeted Regularization
Theory
- Introduces a new doubly robust estimator for causal effect on CVR, addressing sample selection bias.
- Derives theoretical properties of the estimator using semiparametric theory.
- Implements targeted regularization to improve numerical stability and practical applicability.
- Demonstrates superior performance through extensive experiments on various datasets.
Read more
Doubly Robust Estimation of Causal Effect on CVR with Targeted Regularization
Summary
This paper addresses the estimation of causal effects on post-click conversion rate (CVR), a crucial metric in e-commerce and advertising. The authors highlight the challenges posed by sample selection bias when applying existing causal inference methods to clicked samples, which can lead to increased variance and incorrect conclusions. They propose a new doubly robust causal effect estimator specifically designed for chain-structured outcomes like CVR, leveraging semiparametric theory to derive its theoretical properties. This estimator achieves faster convergence rates compared to traditional nuisance parameter estimation and is more robust when using flexible nonparametric methods, such as neural networks. To enhance numerical stability and practical applicability, the authors introduce a framework based on targeted regularization. Extensive experiments on synthetic and real-world datasets demonstrate the effectiveness and robustness of their method, revealing that traditional loss debiasing combined with standard causal estimators underperforms compared to their proposed approach. The paper emphasizes the importance of developing estimators tailored to specific objectives in causal inference.
Methodology
The authors develop a doubly robust estimator for CVR causal effects based on semiparametric theory, focusing on the influence function and von Mises expansion. They also implement targeted regularization to enhance the stability and applicability of the estimation framework.
Results
The proposed method shows strong performance and robustness in extensive experiments conducted on synthetic, semi-synthetic, and real-world data, outperforming traditional loss debiasing methods combined with standard causal estimators.
Implications
The findings suggest that tailored estimators for specific causal inference objectives can significantly improve the accuracy of causal effect estimations in practical applications, particularly in e-commerce and advertising contexts.
Simulation-to-real transfer learning for infrared spectroscopic chemical sensing and analysis from molecules to complex samples
Efficient ML
- UltraIR is a foundation model for IR spectroscopy with over 100 million parameters.
- It employs simulation-to-real transfer learning to enhance chemical inference from IR spectra.
- The model is pretrained on simulated spectra and adapted to specific tasks with limited labeled data.
- UltraIR outperforms traditional methods in various chemical sensing applications.
Read more
Simulation-to-real transfer learning for infrared spectroscopic chemical sensing and analysis from molecules to complex samples
Summary
This paper presents UltraIR, a foundation model designed for infrared (IR) spectroscopy that facilitates simulation-to-real transfer learning for chemical sensing and analysis. Traditional methods of interpreting IR spectra rely heavily on expert knowledge and are often labor-intensive, making them less effective for unfamiliar compounds and complex samples. UltraIR addresses these challenges by utilizing a large-scale pretraining approach on approximately 60 million simulated IR spectra, employing three complementary objectives: spectral reconstruction, molecular fingerprint similarity alignment, and functional-group prediction. The model is then fine-tuned for specific tasks using limited labeled experimental spectra. The authors demonstrate UltraIR's effectiveness across various benchmark evaluations, including functional-group prediction and mixture-component identification, as well as real-world applications such as bacterial classification and soil property prediction. The results indicate that UltraIR significantly outperforms conventional machine-learning methods and task-specific deep-learning models, showcasing its potential for adaptable and data-efficient chemical sensing systems.
Methodology
The methodology involves pretraining UltraIR on a large dataset of simulated IR spectra using three objectives: spectral reconstruction, molecular fingerprint similarity alignment, and functional-group prediction. The pretrained model is then adapted to specific downstream tasks with limited labeled experimental spectra, allowing for efficient learning and transferability across different analytical objectives.
Results
UltraIR demonstrated superior performance in benchmark evaluations for functional-group prediction, molecular structure elucidation, and mixture-component identification. In real-world applications, it effectively classified bacteria, traced the geographic origin of medicinal herbs, quantified constituents, classified microplastics, and predicted soil properties, outperforming conventional machine-learning and task-specific deep-learning baselines.
Implications
The findings suggest that UltraIR can significantly enhance the reliability and scalability of chemical sensing systems, making it applicable in various fields such as environmental monitoring, clinical analysis, and materials characterization. Its ability to adapt with limited labeled data also opens new avenues for research and application in chemical analysis.
Towards Socially Compliant Navigation in Deep Reinforcement Learning via Proxemics-Based Reward Modeling
Reinforcement Learning
Robotics
- Introduction of a proxemics-based reward model for socially compliant robot navigation.
- Validation of the model across multiple DRL navigation methods, showing improved social metrics.
- Systematic analysis of reward design factors for effective social navigation.
- Focus on the importance of comfort-aware navigation assessment in addition to traditional navigation metrics.
Read more
Towards Socially Compliant Navigation in Deep Reinforcement Learning via Proxemics-Based Reward Modeling
Summary
This paper addresses the challenge of robot navigation in crowded environments, emphasizing the need for socially compliant behavior in addition to efficient navigation. The authors propose a novel proxemics-based reward formulation for deep reinforcement learning (DRL) that models human personal space using a radial Gaussian-mixture field derived from Hall's proxemics theory. This reward model provides a dense and interpretable signal that encourages robots to minimize proxemic intrusion while maintaining navigation efficiency. The proposed approach is integrated into established DRL navigation methods and evaluated through simulations across various crowd scenarios, reward baselines, and crowd densities. The results demonstrate that the proxemics-based reward consistently enhances social metrics while maintaining competitive navigation performance compared to existing reward models. The paper also emphasizes the importance of explicitly assessing comfort-aware navigation, rather than relying solely on navigation metrics.
Methodology
The authors developed a proxemics-based reward model that represents each pedestrian's personal space as a radial Gaussian-mixture field. This model computes a local proxemic cost within the robot's field of view, which is integrated into existing DRL navigation frameworks. The evaluation involved simulations across various scenarios to compare the proposed reward model against established baselines.
Results
The experiments showed that the proxemics-based reward formulation significantly improved social compliance metrics in robot navigation while maintaining competitive performance in navigation efficiency. The results highlighted the effectiveness of the proposed reward model across different crowd densities and scenarios.
Implications
The findings suggest that incorporating social compliance into robot navigation can enhance interactions in crowded environments, making robots more acceptable and effective in real-world applications such as malls, hospitals, and airports. This work paves the way for future research in socially aware robotics and the development of more sophisticated navigation systems.
MARCH: Scaling Recurrent Memory with Content-Routed State Anchors
NLP
Large Language Models
Efficient ML
- MARCH introduces a scalable memory architecture for recurrent models that allows for efficient long-context retrieval.
- The architecture utilizes content-conditioned state anchors to maintain historical information without increasing computational complexity.
- MARCH consistently outperforms existing linear attention models in various benchmarks, demonstrating its effectiveness in recall-intensive tasks.
- The method provides a controllable trade-off between historical resolution and memory cost.
Read more
MARCH: Scaling Recurrent Memory with Content-Routed State Anchors
Summary
The paper presents MARCH, a novel architecture designed to enhance the memory capabilities of recurrent neural networks (RNNs) for long-context tasks. Traditional transformers excel in long-context retrieval but suffer from quadratic computational complexity and growing memory requirements. Conversely, recurrent models are efficient but often lose earlier information due to state overwriting. MARCH addresses these limitations by introducing a mechanism that periodically caches recurrent state checkpoints as 'state anchors' and associates them with compact, content-conditioned keys. This allows MARCH to maintain a scalable memory bank that grows with context length, enabling effective retrieval of historical information while preserving computational efficiency. The architecture employs an anchor query mechanism to attend to all available state anchors, facilitating an attention-style aggregation over historical states. The authors demonstrate that MARCH outperforms various linear attention models across several benchmarks, including commonsense reasoning and in-context retrieval, showcasing its ability to enhance long-range memory without compromising efficiency.
Methodology
MARCH employs a memory-augmented recurrent architecture that periodically caches cumulative states as state anchors. Each anchor is linked to a compact learned descriptor, enabling the model to route queries to relevant historical states. The architecture uses an anchor query mechanism to aggregate information from all causally available state anchors, facilitating efficient retrieval of historical data.
Results
The experimental results indicate that MARCH significantly outperforms multiple linear attention variants on tasks such as commonsense reasoning and in-context retrieval. The architecture demonstrates improved long-range memory capabilities while maintaining computational efficiency, with notable performance gains observed across various benchmarks.
Implications
The advancements presented in MARCH could lead to more effective applications in long-document understanding, multi-turn interactions, and in-context learning, where retaining and retrieving information across extended sequences is crucial. This could enhance the performance of large language models in real-world applications.
Dual Spatial-Temporal Attribution: Architecture-Aligned Post-Hoc Explainability for Recurrent Graph Anomaly Detection
Graph Learning
Interpretability
Time Series
- Introduction of X-AddGraph, the first post-hoc explainability framework for AddGraph and GCN+GRU models.
- Development of the Dual Spatial-Temporal Attribution (DSTA) mechanism with three aligned components.
- Preservation of detection performance (βAUC = 0) while providing explanations.
- Long-term attribution identifies more relevant historical snapshots compared to random selection.
Read more
Dual Spatial-Temporal Attribution: Architecture-Aligned Post-Hoc Explainability for Recurrent Graph Anomaly Detection
Summary
The paper addresses the challenge of explainability in deep learning models for anomaly detection in dynamic graphs, specifically focusing on the AddGraph framework, which lacks interpretability. The authors introduce X-AddGraph, a post-hoc explainability framework that employs a Dual Spatial-Temporal Attribution (DSTA) mechanism. This mechanism consists of three components aligned with AddGraph's architecture: a gradient-based relevance attribution for the current adjacency structure (spatial), a direct reading of contextual attention weights (short-term temporal), and a gradient rollback through recurrent hidden states (long-term temporal). The framework preserves detection performance while providing explanations for flagged anomalies. The authors validate their approach on the UCI Message benchmark, demonstrating that X-AddGraph reproduces the detection scores of AddGraph while offering insights into the factors influencing anomaly detection. The long-term attribution component significantly outperforms random selection in identifying relevant historical snapshots, showcasing the framework's unique capabilities. The implementation is publicly available for reproducibility.
Methodology
The authors developed the X-AddGraph framework, which utilizes the DSTA mechanism to provide explainability for the AddGraph anomaly detection model. The DSTA mechanism includes three components: spatial relevance attribution using gradients, short-term attribution through contextual attention weights, and long-term attribution via gradient rollback through recurrent states. The framework is strictly post-hoc, ensuring that the anomaly detection performance remains unchanged.
Results
On the UCI Message benchmark, the AddGraph baseline achieved an average per-snapshot AUC of 0.8705, surpassing previously published results. X-AddGraph maintained this performance while providing explanations. The long-term attribution component was shown to identify historical snapshots with significantly more counterfactual signal than random selections, indicating its effectiveness in explaining anomaly detection decisions.
Implications
The proposed framework enhances the trustworthiness and auditability of anomaly detection systems in regulated environments, making it suitable for applications in financial transaction networks, cybersecurity, and other cooperative information systems where explainability is crucial.
Represent, Then Generate: Multimodal-Conditioned Time-Series Generation under Irregular Missingness
Generative Models
Time Series
Multimodal
- ReCoGen effectively addresses the issue of irregular missingness in multimodal physiological data.
- The framework separates the representation of conditions from the generation of target signals, enhancing performance.
- ReCoGen outperforms existing conditional generators across multiple datasets and tasks.
- The use of masked autoencoders allows for robust and compact representations of time-variant conditions.
Read more
Represent, Then Generate: Multimodal-Conditioned Time-Series Generation under Irregular Missingness
Summary
The paper addresses the challenge of generating continuous physiological time series in clinical settings where many informative signals are either invasive, expensive, or unavailable. Existing conditional generation methods struggle with the heterogeneous and irregularly missing data typical in clinical environments. The authors propose a novel framework called ReCoGen (Represent Conditions, then Generate), which consists of two stages: first, it employs masked autoencoders to create robust representations of each modality, capturing the time-variant conditions despite missing data; second, it utilizes a flow-matching generator to synthesize the target signal by integrating these representations with static clinical variables. The framework was evaluated across three physiological benchmarks, demonstrating superior performance compared to six existing conditional generators, achieving better utility in most cases, even surpassing the utility of real signals. The study highlights the importance of dedicated condition representation in improving the performance of time-series generation models in clinical applications.
Methodology
ReCoGen employs a two-stage approach: Stage I involves training masked autoencoders for each modality to create missingness-tolerant token sequences from time-variant conditions. Stage II uses a flow-matching generator to synthesize the target signal by integrating these tokens with static conditions, leveraging learnable cross-attention mechanisms.
Results
ReCoGen achieved the best downstream utility across all sixteen (dataset, task, metric) settings tested, surpassing six representative conditional generators. In thirteen of these settings, the utility reached or exceeded that of real signals, indicating a significant improvement in the generation of physiological time series.
Implications
The proposed framework has the potential to transform clinical monitoring by enabling the synthesis of hard-to-measure physiological signals from routinely collected data, thus reducing the need for invasive procedures and enhancing patient care.
Adaptive $k$ Nearest Neighbors Classifier via Granular Ball Computing
Efficient ML
Theory
- Introduces an adaptive KNN approach using granular-ball computing.
- Utilizes the Fisher criterion for effective granular ball generation.
- Dynamically determines the effective k value based on local neighborhood structure.
- Demonstrates improved robustness against noise and local perturbations.
Read more
Adaptive $k$ Nearest Neighbors Classifier via Granular Ball Computing
Summary
This paper presents an adaptive and efficient k-Nearest Neighbors (KNN) algorithm utilizing granular-ball computing to address the critical issue of selecting the optimal k value, which significantly affects classification performance. The proposed method operates in two stages: during training, the dataset is coarsely partitioned into granular balls to simplify data distributions, and the Fisher criterion is employed to manage the splitting and stopping of these balls, resulting in a multi-granularity representation. In the prediction phase, the nearest granular ball is identified using a weighted distance mechanism, and an adaptive neighborhood is constructed around the test sample, allowing for a dynamic determination of the effective k value based on the actual number of samples in the neighborhood. This approach enhances robustness against noise and local perturbations, leading to improved classification accuracy. Experimental results indicate that the proposed method outperforms existing KNN variants across various datasets in terms of both accuracy and efficiency, demonstrating its effectiveness in real-world applications.
Methodology
The methodology involves a two-stage process: first, the dataset is partitioned into granular balls to reduce complexity, and the Fisher criterion is used to control the splitting of these balls. In the prediction stage, a weighted distance mechanism identifies the nearest granular ball, and an adaptive neighborhood is constructed to dynamically determine the effective k value based on the number of samples in the neighborhood.
Results
The experimental results show that the proposed granular ball KNN (GBKNN) algorithm significantly outperforms traditional KNN methods and other variants in terms of classification accuracy and computational efficiency across multiple datasets.
Implications
The findings suggest that the adaptive granular ball KNN can be effectively applied in various domains requiring robust classification methods, particularly in high-dimensional and noisy environments. The open-sourced code allows for further exploration and application in real-world scenarios.
Training AI Scientists to Replicate Research
Large Language Models
Reinforcement Learning
Theory
- Introduction of Replica, a task space for replicating research papers.
- Development of Faraday, an AI Scientist that outperforms existing models in replication tasks.
- Implementation of a rubric-based judging system for evaluating replication quality.
- Demonstration of Faraday's scientific rigor and creativity in handling underspecified research problems.
Read more
Training AI Scientists to Replicate Research
Summary
This paper addresses the replication crisis in scientific research, particularly in machine learning, by developing a novel AI agent named Faraday, which is capable of replicating research papers. The authors introduce a scalable task space called Replica, consisting of 310 figure-replication tasks derived from 100 machine learning and AI-for-science papers published between 1990 and 2026. To evaluate the replication quality, they implement an auto-generated rubric-based judge that aligns closely with human assessments. Faraday, a 27B-parameter AI Scientist, utilizes coding agents as tools and demonstrates superior performance compared to existing models like Claude Opus 4.8 and GPT-5.5 on various replication tasks. The study highlights Faraday's ability to adopt a scientifically principled approach, emphasizing creativity and adherence to experimental scope, thus paving the way for AI agents capable of long-term scientific innovation without complex frameworks.
Methodology
The authors constructed the Replica task space by curating 310 tasks from 100 research papers. They trained Faraday using a modified version of the GRPO algorithm, incorporating a per-task rubric-based judge for evaluation. Faraday leverages Codex as a coding tool to perform the replication tasks, which require navigating underspecified research details.
Results
Faraday outperformed Claude Opus 4.8 and GPT-5.5 on 73% of in-distribution ML tasks and 60% of held-out AI-for-science tasks, achieving an average improvement of 6% over Claude and 8% over Codex. Human experts rated Faraday as more effective in replicating research than its competitors, showcasing its ability to implement scientific mechanisms rather than simply generating outputs.
Implications
The findings suggest that AI agents like Faraday could significantly enhance the reliability of scientific research by improving replication efforts. This could lead to more robust scientific knowledge and facilitate further innovations in various fields, particularly in machine learning and AI.
Incremental Evaluation and Training in Relational Deep Learning
Graph Learning
Time Series
Theory
- Introduces an incremental evaluation paradigm for RDL that captures the temporal dynamics of relational databases.
- Demonstrates the prevalence of temporal concept drift in predictive tasks within RDL.
- Presents multiple effective incremental training strategies that outperform traditional training methods.
- Proposes a new evaluation metric focused on near-future predictive accuracy.
Read more
Incremental Evaluation and Training in Relational Deep Learning
Summary
This paper addresses the limitations of current evaluation practices in Relational Deep Learning (RDL), which often rely on static, single-episode dataset snapshots. The authors propose an incremental, multi-episode evaluation and training paradigm that reflects the dynamic, evolving nature of real-world relational databases. By simulating the lifecycle of database growth, they demonstrate that temporal concept drift significantly affects model performance in RDL tasks. The study introduces multiple incremental training regimes for fine-tuning models, showing that transfer learning is both feasible and effective in this context. Additionally, a new temporal evaluation metric prioritizing near-future accuracy is proposed, leading to consistently better performance of incrementally fine-tuned models compared to traditional from-scratch training methods. The findings emphasize the importance of adapting RDL models to the continuous evolution of relational data, thereby enhancing their robustness and applicability in real-world scenarios.
Methodology
The authors developed an incremental, multi-episodic evaluation framework that transitions RDL benchmarking from static snapshots to continuous evaluation. They empirically tested this framework using large-scale datasets, analyzing data evolution and model training dynamics. Various incremental fine-tuning strategies were benchmarked to assess their effectiveness in mitigating model degradation due to temporal concept drift.
Results
The study found that temporal concept drift is a significant factor affecting model performance in RDL tasks. Incrementally fine-tuned models consistently outperformed traditional models trained from scratch, demonstrating the effectiveness of transfer learning in the RDL context. The proposed evaluation metric showed improved alignment with practical operational priorities by focusing on near-future accuracy.
Implications
The findings suggest that RDL models need to be continuously adapted to maintain their predictive performance as relational data evolves. This has implications for industries relying on relational databases, as it emphasizes the need for dynamic model training and evaluation strategies to ensure robust performance over time.
The Objective Is the Bottleneck: Latent World Models Encode What Their Planners Cannot Use
Reinforcement Learning
Optimization
Robotics
- The predictor is not the bottleneck; imagination remains informative beyond the planner's usage.
- The planning objective saturates and can lead to counterproductive outcomes.
- The pathology of planning issues is inherent to the method, not specific to a single implementation.
- Reachability is a more effective planning objective than proximity.
Read more
The Objective Is the Bottleneck: Latent World Models Encode What Their Planners Cannot Use
Summary
This paper investigates the limitations of latent world models, particularly focusing on the planning objectives that constrain their performance in long-horizon tasks. The author reproduces the LeWorldModel on the TwoRoom environment and finds that the bottleneck is not the predictive capability of the model but rather the planning objective itself. The study reveals that while the predictor remains informative for up to seventy-five environment steps, the planner only utilizes information for twenty-five steps. The paper highlights that the squared latent distance, which is minimized during planning, can lead to counterintuitive results where moving away from the goal can actually lower the planner's cost. Furthermore, the author demonstrates that the learned cost must be trained on the distribution relevant to the planner's scoring, rather than on static embeddings. The findings suggest that reachability, rather than proximity, should be the focus of planning objectives. A simple adjustment to the planning objective significantly improves goal achievement rates without the need for retraining, indicating that the method's pathology lies in the objective rather than the predictor itself.
Methodology
The study employs a reproduction of the LeWorldModel on the TwoRoom environment, utilizing an 18.03M-parameter joint-embedding predictive architecture. Planning is conducted using cross-entropy method (CEM) search over the learned dynamics, with specific configurations for candidate sequences and refinement iterations. The experiments are run on a CPU without retraining any models.
Results
The results indicate that the model's predictive accuracy does not correlate with long-horizon planning success. By replacing the planning objective without retraining, the author achieves a significant increase in goal completion rates from 26.0% to 98.0% at an offset of 100. The learned cost based on spatial distance performs worse than a linear cost but leads to better planning outcomes due to its focus on reachability.
Implications
These findings suggest that improving planning objectives in latent world models can lead to substantial enhancements in performance, particularly in environments requiring long-horizon planning. The insights could inform future research on optimizing planning strategies in reinforcement learning and robotics.
Sparse Orthogonal Regression Technique: A Spectral Framework for Equation Discovery, Approximation, and Integration
Theory
Optimization
Time Series
- SORT provides a sparse spectral framework for learning from noisy and irregular data.
- The technique allows for the discovery of ordinary differential equations using orthonormal basis expansions.
- SORT shows improved stability and performance compared to traditional sparse regression methods under challenging conditions.
- The method supports nonlinear approximation and high-dimensional integral estimation.
Read more
Sparse Orthogonal Regression Technique: A Spectral Framework for Equation Discovery, Approximation, and Integration
Summary
The paper introduces the Sparse Orthogonal Regression Technique (SORT), a novel framework designed for learning orthonormal-basis expansions from noisy and irregularly sampled data. SORT employs L1-regularized regression to estimate expansion coefficients directly from observations, eliminating the need for explicit quadrature or analytic inner-product evaluations. The primary application of SORT is the data-driven discovery of ordinary differential equations, where vector fields are represented in chosen orthogonal bases and learned as sparse coefficient expansions. This approach serves as a complementary method to existing techniques such as symbolic regression and SINDy-style sparse identification, focusing on recovering a compact spectral representation that can guide searches for simpler analytic forms. The experiments demonstrate that SORT performs comparably or better than library-based sparse-regression methods, particularly when the orthogonal basis is well-suited to the problem. Additionally, SORT exhibits greater stability under conditions of sparse sampling, noisy derivative estimates, and representation mismatch. The framework also facilitates nonlinear approximation and the estimation of complex, high-dimensional integrals through coefficient readout, making it a versatile tool for system identification, approximation, and integration while emphasizing the importance of basis design in scientific modeling.
Methodology
SORT combines orthonormal basis representations with sparsity-promoting regression techniques. It estimates coefficients from data using L1-regularized regression, allowing for the recovery of sparse expansions without the need for explicit quadrature or analytic evaluations. The framework is evaluated across various tasks, including dynamical-system identification, numerical integration, and nonlinear approximation.
Results
The experiments indicate that SORT matches or surpasses the performance of dense least-squares, kernel-based, and library-based sparse-regression methods, particularly when the orthogonal basis is tailored to the specific problem. The results also highlight the persistence of dominant low-order coefficients as model complexity increases, supporting the concept of order-consistent model growth.
Implications
SORT has significant implications for scientific computing and machine learning, particularly in areas requiring the reconstruction of functional relationships from data. Its ability to provide a reusable representation for system identification, approximation, and integration can enhance the efficiency and accuracy of various applications, including surrogate modeling and data-driven equation discovery.
Intervention-Aware Clinical World Model for Post-Op Outcome Forecasting in Cardiology
Multimodal
- Introduces an intervention-aware latent clinical world model for dynamic post-operative outcome forecasting.
- Utilizes a structured latent state that evolves based on irregular post-procedural events.
- Achieves AUROC of 0.756 and AUPRC of 0.777 for recurrence prediction in atrial fibrillation ablation.
- Provides retrospective risk estimates at various horizons, enhancing clinical decision-making.
Read more
Intervention-Aware Clinical World Model for Post-Op Outcome Forecasting in Cardiology
Summary
This paper presents an innovative intervention-aware clinical world model designed to forecast post-operative outcomes in cardiology, specifically focusing on atrial fibrillation (AF) ablation. Traditional clinical prediction models often simplify the recovery process as a direct mapping from baseline measurements to future outcomes, neglecting the dynamic nature of patient recovery, which includes irregular events such as medication changes and repeat interventions. The proposed model addresses this limitation by representing each patient with a structured latent state that evolves over time based on time-ordered post-intervention events. The model encodes baseline imaging into a 3D spatial latent state and updates it using procedural context, static covariates, elapsed time, and physiological embeddings. The framework is trained using follow-up imaging data to predict recurrence risk and scar extent during the 90-day recovery window. The results demonstrate that the model effectively captures the complexities of post-procedure recovery, achieving notable performance metrics in predicting long-term recurrence risk.
Methodology
The methodology involves encoding pre-ablation MRI into a 3D spatial latent state using a variational autoencoder. The model updates this state based on procedural geometry, static covariates, elapsed time, and clinical events, which are represented as tokens. A horizon-token formulation allows for anytime recurrence forecasting from partial histories during the blanking period. The model is trained using follow-up imaging data to define the target latent state for predicting recurrence and scar extent.
Results
The model was evaluated on the DECAAF-II dataset, achieving an AUROC of 0.756 and an AUPRC of 0.777 for predicting recurrence risk. Additionally, it recorded a mean absolute error (MAE) of 2.971 percentage points for scar extent prediction, demonstrating its effectiveness in capturing the complexities of post-operative recovery without relying on follow-up MRI intensities during inference.
Implications
The intervention-aware clinical world model has significant implications for improving clinical decision-making in cardiology by providing dynamic risk assessments that adapt to patient recovery trajectories. This approach could enhance personalized treatment strategies and improve patient outcomes following AF ablation procedures.
A Contract-Grade Verifier for LLM-Generated GPU Kernels, and a Native Blackwell Backward for the Gated-Linear-Recurrence Family
Large Language Models
Theory
Optimization
- Introduction of a twelve-gate contract-grade verifier for GPU kernels.
- Audit of 2,638 machine-generated kernels reveals significant correctness issues.
- Development of the first native Blackwell tcgen05 training backward for the GDN family.
- The verifier successfully identifies failures in kernels accepted by existing benchmarks.
Read more
A Contract-Grade Verifier for LLM-Generated GPU Kernels, and a Native Blackwell Backward for the Gated-Linear-Recurrence Family
Summary
This paper addresses the reliability of GPU kernels generated by large language models (LLMs), which have been reported to have high correctness rates based on loose testing methods. The authors introduce a contract-grade verifier that operationalizes a Kernel Contracts taxonomy, consisting of twelve adversarial gates that a correct kernel must satisfy. The verifier is tested on 2,638 machine-generated kernels, revealing that 39.5% are fundamentally broken and 62.1% violate at least one contract. The findings are supported by multiple independent validations, including positive controls and agreement with existing correctness benchmarks. Additionally, the authors present a novel native Blackwell tcgen05 training backward for the gated-linear-recurrence family, which is verified against a double-precision oracle. This kernel passes the verifier, demonstrating its robustness. The paper concludes that the correctness signal in kernel generation is weaker than previously suggested, and emphasizes the need for stricter verification methods to ensure reliability.
Methodology
The authors developed a contract-grade verifier that implements twelve adversarial gates to assess kernel correctness. The verifier was applied to a large corpus of machine-generated kernels, and its findings were validated through various independent methods, including positive controls and comparisons with existing correctness benchmarks.
Results
The verifier found that 39.5% of the audited kernels were fundamentally broken, with 62.1% showing at least one contract violation. In contrast, the standard testing method accepted 1,487 broken kernels, while the verifier only accepted 14. The native Blackwell kernel was verified against a double-precision oracle and passed the verifier's tests.
Implications
The findings suggest that existing methods for verifying the correctness of LLM-generated GPU kernels are insufficient, highlighting the need for more rigorous testing frameworks. This could lead to improved reliability in GPU kernel generation and broader applications in high-performance computing.
EGRL: Edge generation-guided relation-aware learning for RNA-protein interaction prediction
Graph Learning
- EGRL introduces implicit meta-path learning to enhance relational semantics in GNNs.
- A graph generator is included to predict interactions for cold-start RNA/protein nodes.
- The framework employs a multi-relation-aware attention mechanism for better interaction modeling.
- EGRL shows significant improvements in AUROC and AUPR metrics over existing methods.
Read more
EGRL: Edge generation-guided relation-aware learning for RNA-protein interaction prediction
Summary
The paper presents a novel framework called Edge Generation-guided Relation-aware Learning (EGRL) aimed at improving RNA-Protein Interaction Prediction (RPIP) using Graph Neural Networks (GNNs). Traditional methods for RPIP often rely on costly wet-lab experiments or simplistic machine learning models that fail to capture the complex relationships between RNA and proteins. Existing GNN approaches are limited by their reliance on homogeneous graphs or predefined meta-paths, which restrict their ability to generalize in scenarios with sparse data or unknown molecules. EGRL addresses these challenges through several innovative components: implicit meta-path learning that dynamically captures relational semantics without manual design, a multi-relation-aware attention mechanism for adaptive interaction pattern fusion, a graph generator that predicts potential edges for cold-start nodes, and a multi-feature fusion predictor for scoring interactions. The framework is jointly trained with a primary task loss and an auxiliary generator loss. Evaluations on four benchmark datasets demonstrate that EGRL not only achieves competitive performance but also excels in cold-start scenarios, significantly improving the Area Under the Receiver Operating Characteristic curve (AUROC) and Area Under the Precision-Recall curve (AUPR) compared to state-of-the-art methods.
Methodology
EGRL employs a heterogeneous GNN framework that integrates implicit meta-path learning, a graph generator for cold-start nodes, and a multi-relational Graph Attention Network (GAT) with multi-feature fusion. The model is trained jointly with a primary task loss for interaction prediction and an auxiliary loss for edge generation.
Results
EGRL achieved an AUROC of 0.867 and an AUPR of 0.861 for cold-start scenarios, representing improvements of 8.6% in AUROC and 5.0% in AUPR over prior state-of-the-art methods. The framework demonstrated competitive performance across four benchmark datasets.
Implications
The advancements in EGRL can significantly enhance the efficiency and accuracy of RNA-protein interaction predictions, which are crucial for understanding cellular functions and can aid in drug design and disease prediction.
Novel Knowledge-Guided Generative Methods for Synthetic Transcriptomic Data
Generative Models
Graph Learning
- Synthetic data generation can mitigate challenges of data scarcity and biases in genomic research.
- MK-TGAN, a novel generative model, effectively integrates biological knowledge through graph neural networks.
- Incorporating prior biological knowledge significantly enhances the realism and utility of synthetic transcriptomic data.
- The study demonstrates state-of-the-art performance of MK-TGAN in generating biologically coherent samples.
Read more
Novel Knowledge-Guided Generative Methods for Synthetic Transcriptomic Data
Summary
This paper addresses the challenges in generating high-quality synthetic transcriptomic data, which is crucial for biomedical research. The authors present a comparative analysis of generative models that incorporate prior biological knowledge through gene graphs to enhance the realism and utility of synthetic data. They introduce three variants of Generative Adversarial Networks (GANs), focusing on a novel model called MK-TGAN (Multi-Kernel Transcriptomic Generative Adversarial Network). MK-TGAN employs multiple graph neural network kernels to explicitly integrate biological knowledge, improving the representation of gene relationships. The study evaluates these models on breast cancer transcriptomic data, particularly focusing on Epithelial-to-Mesenchymal Transition (EMT) related genes. The results demonstrate that MK-TGAN outperforms other models in generating synthetic samples that are both realistic and biologically plausible, highlighting the importance of integrating prior knowledge in generative modeling.
Methodology
The authors developed three GAN variants: Graph-Modulated GAN (GM-GAN), Graph-Regularized GAN (GR-GAN), and MK-TGAN. GM-GAN and GR-GAN incorporate biological knowledge implicitly, while MK-TGAN explicitly utilizes multiple graph neural network kernels to enhance data generation. The models were evaluated on breast cancer transcriptomic data focusing on EMT-related genes.
Results
MK-TGAN achieved superior performance in terms of precision, recall, and downstream biological tasks compared to other generative models. The integration of biological knowledge through graph neural networks resulted in synthetic samples that exhibited higher realism and biological plausibility.
Implications
The findings suggest that integrating biological knowledge into generative models can significantly improve the quality of synthetic transcriptomic data, which can be utilized in various biomedical applications, including data augmentation and hypothesis generation in genomics.
The Impact of Temporal Context Length and Encoding Strategies on Self-Supervised ECG Representation Learning
Time Series
- Longer temporal contexts (5-10 minutes) improve ECG representation learning and downstream classification accuracy.
- Continuous convolutional patch embeddings outperform discretized vector-quantized tokens in capturing clinically relevant details.
- Short temporal windows may not provide sufficient information for accurate rhythm diagnosis, particularly for conditions like atrial fibrillation.
Read more
The Impact of Temporal Context Length and Encoding Strategies on Self-Supervised ECG Representation Learning
Summary
This paper investigates the effects of temporal context length and encoding strategies on self-supervised learning (SSL) for electrocardiogram (ECG) representation. The authors conducted a controlled study using the Icentia11k dataset, varying the input horizon (16 seconds, 1 minute, 5 minutes, and 10 minutes) and the representation method (continuous convolutional patch embeddings vs. fixed vector-quantized tokens). The study aims to determine whether shorter temporal windows and discretized tokenization compromise the information necessary for accurate rhythm inference and patient-level consistency. Results indicate that longer temporal contexts (5 and 10 minutes) significantly enhance transfer learning and retrieval accuracy, capturing slow-varying rhythm dynamics better than shorter windows. Additionally, continuous patch embeddings consistently outperformed discretized tokens, suggesting that quantization may lead to the loss of clinically relevant details. The findings advocate for the development of ECG foundation models that prioritize extended context and continuous encoders for improved clinical predictions and applications.
Methodology
The study employed a controlled experimental framework to compare self-supervised models trained on various temporal windows and encoding strategies. The evaluation metrics included downstream classification tasks for atrial fibrillation and patient-level retrieval metrics to assess consistency across time and activity.
Results
The results demonstrated that increasing the temporal context beyond 16 seconds led to improved performance in both classification and patient-level retrieval tasks, with the best results observed at 5 and 10 minutes. Continuous patch embeddings consistently provided better performance compared to discretized tokens, highlighting the importance of preserving waveform details.
Implications
The findings suggest that ECG representation learning can be significantly enhanced by focusing on longer temporal contexts and using continuous encoders. This has implications for developing more effective ECG foundation models that can improve clinical predictions and support similarity-based applications in healthcare.
HiRoute: Hierarchical Routed Prompt Tuning for Safety Alignment of Large Language Models
NLP
Large Language Models
Efficient ML
- HiRoute establishes a cross-category safety boundary using a shared coarse-grained prompt.
- It dynamically composes fine-grained prompt experts through a hierarchical router based on input risk.
- The framework effectively reduces over-refusal while maintaining high safety and helpfulness in responses.
- HiRoute's performance is validated across various instruction-tuned models and safety benchmarks.
Read more
HiRoute: Hierarchical Routed Prompt Tuning for Safety Alignment of Large Language Models
Summary
The paper introduces HiRoute, a novel framework for safety alignment in large language models (LLMs) that addresses vulnerabilities to harmful requests and jailbreak attacks. Traditional prompt tuning methods often rely on static prompts that fail to differentiate between various risk categories, leading to generic refusals that can degrade model performance. HiRoute overcomes these limitations by implementing an input-adaptive hierarchical prompt-tuning approach. It utilizes a lightweight hierarchical router trained on representations from a frozen LLM to detect harmful intents and predict multi-label risk scores. The framework separates category-agnostic safety control from category-specific response guidance, allowing benign inputs to bypass safety checks while risky inputs are processed using a combination of a shared coarse-grained prompt and fine-grained prompt experts. The authors validate HiRoute's effectiveness across multiple instruction-tuned models, demonstrating high safety rates, reduced over-refusal, and competitive performance on general tasks, thus providing a balanced approach to safety and helpfulness in LLM responses.
Methodology
HiRoute employs a two-stage training process where a hierarchical router is first trained to identify harmful inputs and predict risk scores. After freezing the backbone model and router, preference optimization is applied to learn a shared coarse-grained prompt and a set of fine-grained prompt experts. During inference, benign inputs bypass the safety branch, while risky inputs utilize a combination of the shared prompt and router-weighted fine-grained prompts.
Results
Experiments show that HiRoute achieves high safety rates across multiple benchmarks while significantly reducing over-refusal rates and maintaining competitive performance on general-purpose tasks. The framework demonstrates a robust balance between safety alignment and response helpfulness.
Implications
HiRoute's approach can be applied to enhance the safety and effectiveness of LLMs in various applications, particularly in sensitive domains where harmful requests are a concern. It offers a scalable solution for adapting safety mechanisms to diverse input risks without compromising model performance.
Diagnosing JEPA World Models with Action-Conditioned Predictive Consistency
Reinforcement Learning
Robotics
Theory
- Introduction of Action-Conditioned Predictive Consistency (ACPC) for diagnosing visual perturbations in JEPAs.
- Establishment of Invariance Radius (IR) and Separation Rate (SR) as metrics for assessing model robustness.
- Proven bounds on the impact of visual perturbations on multi-step prediction error and planning costs.
- Empirical validation of ACPC across multiple control tasks and architectures, showing consistent diagnostic trends.
Read more
Diagnosing JEPA World Models with Action-Conditioned Predictive Consistency
Summary
This paper introduces Action-Conditioned Predictive Consistency (ACPC), a diagnostic tool for evaluating joint-embedding predictive architectures (JEPAs) in the context of world models. JEPAs predict outcomes in a compact latent space, which helps avoid modeling irrelevant visual details. However, visual perturbations can still affect the encoded representations and subsequent predictions. ACPC measures the divergence between predicted trajectories of a clean history and its visually perturbed counterpart when rolled forward under the same action sequence. The authors establish that this divergence bounds the changes in multi-step prediction error and planner cost due to perturbations. They define two complementary metrics: the Invariance Radius (IR), which summarizes the spread of clean and perturbed rollouts, and the Separation Rate (SR), which assesses the distinguishability of different states after rollout. Experiments across four visual control tasks demonstrate that ACPC effectively predicts changes in prediction error and planning costs induced by visual perturbations. The findings indicate that lower IR and higher SR correlate with better performance under perturbations, suggesting that ACPC can identify robust models across different tasks and perturbation types.
Methodology
The methodology involves rolling forward a clean history and its perturbed view under the same action sequence to measure the distance between their predicted trajectories. The authors derive theoretical bounds on the changes in prediction error and costs due to visual perturbations. They also conduct experiments on various visual control tasks to evaluate the effectiveness of ACPC, IR, and SR in predicting performance changes.
Results
The experiments reveal that ACPC is a reliable diagnostic tool, with IR and SR effectively predicting the impact of visual perturbations on multi-step prediction error and planning costs. Lower IR values indicate less sensitivity to perturbations, while higher SR values suggest better state distinguishability. The results show that these metrics can identify robust models across different tasks and perturbation types, confirming the utility of ACPC in diagnosing world models.
Implications
The findings suggest that ACPC can be used to enhance the robustness of world models in reinforcement learning and control tasks, potentially leading to more reliable AI systems that can handle visual perturbations. This could have applications in robotics, autonomous systems, and any domain where visual input is subject to noise or distortion.
Perturbation-based Regional Interpretability through Subtraction Mapping (PRISM): naming-error dissociations in language models and post-stroke aphasia
NLP
Large Language Models
Interpretability
- PRISM adapts neuroimaging subtraction analysis for interpreting large language models.
- The framework allows for spatially resolved testing of cognitive operation specialization in LLMs.
- Error profiles from a perturbed LLM align with lesion patterns in aphasia patients, demonstrating the framework's validity.
- Robust phonemic-favoring dissociations were identified in both LLM and patient data.
Read more
Perturbation-based Regional Interpretability through Subtraction Mapping (PRISM): naming-error dissociations in language models and post-stroke aphasia
Summary
The paper introduces PRISM (Perturbation-based Regional Interpretability through Subtraction Mapping), a novel framework for interpreting large language models (LLMs) by applying subtraction analysis from neuroimaging to perturbed transformer models. The authors aim to bridge the interpretability gap in LLMs by providing a spatially resolved, falsifiable method to test whether internal components of LLMs are specialized for distinct cognitive operations. Utilizing the Brain-LLM Unified Model (BLUM), the study demonstrates that error profiles from a perturbed LLM (LLaVA-1.6-Vicuna-13B) correspond to lesion patterns in chronic post-stroke aphasia patients. PRISM maps errors from the Philadelphia Naming Test across seven clinical categories, employing a group analysis that treats perturbation seeds as subjects. The methodology includes correlation-difference lesion-symptom mapping on 213 aphasia patients, revealing robust phonemic-favoring dissociations in both the LLM and patient data. The findings suggest that PRISM effectively tests functional specialization claims in LLMs, paving the way for future confirmatory interventions.
Methodology
The authors developed PRISM by applying subtraction analysis to perturbation-induced error maps from a transformer model and conducting a parallel analysis on chronic post-stroke aphasia patients. The methodology involved mapping errors from the Philadelphia Naming Test, conducting correlation-difference lesion-symptom mapping, and employing threshold-free cluster enhancement for spatial inference.
Results
The study found that both the perturbed LLM and the aphasia patient data exhibited a robust phonemic-favoring dissociation, with significant clusters identified in deep layers of the LLM and in the frontal-perisylvian cortex of patients. The semantic-favoring direction showed a consistent trend but was not statistically significant.
Implications
The PRISM framework offers a new approach to understanding the interpretability of LLMs, potentially influencing future research in cognitive neuroscience and artificial intelligence. It sets the stage for further investigations into the causal mechanisms underlying language processing in both artificial and biological systems.
Sampling Luck Masquerades as Allocation Gain: Auditing Test-Time Budget Allocation for Neural Combinatorial Optimization
Optimization
- First measurement of allocation value for NCO test-time sampling, showing no detectable gain in in-distribution workloads.
- Significant gains (11-12%) observed under distribution shift with proper allocation strategies.
- Quantification of in-sample selection bias, demonstrating that conventional methods can produce misleading results.
- Introduction of a correction procedure that preserves real gains while eliminating phantom gains.
Read more
Sampling Luck Masquerades as Allocation Gain: Auditing Test-Time Budget Allocation for Neural Combinatorial Optimization
Summary
This paper investigates the effectiveness of non-uniform allocation of sample budgets in Neural Combinatorial Optimization (NCO) solvers, which typically use a uniform sampling strategy across instances. The author measures the potential gains from tailored allocation strategies and audits the measurement process itself. The study reveals two main findings: first, while in-distribution workloads show no detectable allocation gain, out-of-distribution scenarios demonstrate significant improvements in performance when allocation is guided by held-out sample statistics. Specifically, the paper reports an 11.5% improvement in best-of-k solutions for certain solvers under distribution shift conditions. Second, the paper quantifies the selection bias inherent in the conventional in-sample measurement procedure, which can falsely indicate gains that do not exist. The author proposes a correction procedure to mitigate this bias and provides a comprehensive pre-registration record of the study's methodology and findings.
Methodology
The study employs a comparative analysis of three pretrained NCO solvers (POMO, AM, SymNCO) to evaluate the effectiveness of uniform versus non-uniform sample allocation. It utilizes an instance-wise null construction to quantify selection bias and conducts confirmatory experiments under distribution shift conditions to assess the impact of allocation strategies.
Results
The results indicate that while uniform allocation yields no detectable gains in in-distribution scenarios, a tailored allocation approach under distribution shift leads to significant performance improvements (11.5% for AM and 12.0% for SymNCO). The study also highlights the presence of selection bias in conventional measurement methods, which can falsely suggest allocation gains that do not exist.
Implications
The findings suggest that optimizing sample allocation based on instance characteristics can lead to substantial performance improvements in NCO tasks, particularly in real-world applications where data distribution may vary. The proposed correction procedure and transparency in reporting can enhance the reliability of future research in this area.
Into the ORBIT for Time Series: Training Regimes for Foundation Models
Time Series
- Introduction of ORBIT, a training paradigm for TSFMs that controls pre-training distribution.
- Bootstrap Multi-Level Sampling allows for hierarchical control of data exposure.
- Omni-Range Incremental Training enables the model to handle variable-length examples efficiently.
- Falcon-2.0 demonstrates strong zero-shot forecasting capabilities across diverse datasets.
Read more
Into the ORBIT for Time Series: Training Regimes for Foundation Models
Summary
This paper introduces ORBIT (Omni-Range Bootstrap Incremental Training), a novel training paradigm aimed at improving the pre-training distribution of Time Series Foundation Models (TSFMs). The authors identify that while architectural innovations have advanced TSFMs, the training regimes that dictate data exposure across heterogeneous time series datasets remain under-explored. ORBIT addresses this by implementing two main components: Bootstrap Multi-Level Sampling and Omni-Range Incremental Training. The former controls data exposure hierarchically, allowing for stochastic selection of time series records, target variables, context windows, and prediction horizons. The latter enables the model to incrementally consume variable-length examples during training, accommodating diverse context lengths and prediction horizons without the need for stage-specific schedules. The authors apply ORBIT to train Falcon-2.0, a simple univariate encoder-only Transformer model, which incorporates missingness-aware tokenization and multi-patch quantile prediction. They also introduce Rank-Guided Cross-Depth Alignment, a training objective that enhances representation rank across Transformer depths without incurring additional inference costs. Evaluations on GIFT-Eval and fev-bench demonstrate that Falcon-2.0 achieves strong zero-shot forecasting performance across various domains and frequencies, underscoring the importance of stochastic sample construction and diverse context exposure in training.
Methodology
The methodology involves two main components: Bootstrap Multi-Level Sampling for hierarchical data exposure control and Omni-Range Incremental Training for efficient consumption of variable-length examples. The authors also introduce a novel training objective, Rank-Guided Cross-Depth Alignment, to enhance representation learning across different depths of the Transformer model.
Results
Falcon-2.0, trained under the ORBIT paradigm, achieved strong zero-shot forecasting performance on benchmark datasets GIFT-Eval and fev-bench. Ablation studies confirmed the significance of stochastic sample construction and the simultaneous exposure to diverse context lengths and prediction horizons.
Implications
The findings suggest that effective training regimes can significantly enhance the performance of TSFMs, making them more robust across various forecasting tasks and domains. This could lead to improved applications in fields such as finance, healthcare, and environmental monitoring where time series forecasting is critical.
Knowledge-guided Pattern Discovery via Coupled Tensor Factorizations
Time Series
Optimization
Interpretability
- Introduces a knowledge-guided approach for pattern discovery that combines real and simulated data.
- Utilizes coupled tensor factorizations to analyze multiway datasets effectively.
- Demonstrates improved pattern discovery performance in metabolomics data.
- Reveals potential discrepancies between real data and computational models.
Read more
Knowledge-guided Pattern Discovery via Coupled Tensor Factorizations
Summary
This paper presents a novel approach for pattern discovery in complex systems, such as the human metabolome and brain, by integrating real observational data with simulated data generated from computational models. The authors propose a knowledge-guided method that utilizes coupled tensor factorizations to analyze multiway datasets, which are often high-dimensional and noisy. By jointly analyzing real and simulated data, the method aims to enhance the interpretability and accuracy of pattern discovery while revealing discrepancies between the data and computational models. The experiments conducted on real metabolomics measurements demonstrate that this approach significantly improves the performance of pattern discovery compared to traditional data-driven methods. The findings suggest that incorporating prior knowledge from computational models can lead to more robust insights in the analysis of complex biological systems.
Methodology
The authors employ coupled tensor factorizations, specifically the CANDECOMP/PARAFAC (CP) model and the PARAFAC2 model, to jointly analyze real observational data and simulated data. The approach allows for the extraction of latent factors while linking features from both datasets through a coupling relation. The methodology is designed to handle the complexities of high-dimensional data and to incorporate prior knowledge from computational models.
Results
The experiments show that the proposed knowledge-guided approach consistently outperforms traditional data-driven tensor factorization methods in terms of pattern discovery accuracy. The coupling of real and simulated data not only enhances the interpretability of the results but also identifies discrepancies between the observed data and the computational models, leading to new insights.
Implications
This research has significant implications for fields that rely on the analysis of complex biological systems, such as metabolomics and neuroscience. By integrating computational models with real data, researchers can achieve more reliable and interpretable results, potentially leading to better understanding and treatment of various health conditions.
Neural Quadratic Forms: A Unified Minimal Model for Sudden Learning and Scaling Laws
Theory
Optimization
- Introduces Neural Quadratic Forms (NQF) as a minimal model for understanding neural network learning dynamics.
- Demonstrates that diverse neural architectures share common behaviors due to underlying symmetry.
- Derives a universal quadratic form for the cost function expansion around initial weights.
- Establishes a connection between training dynamics and LotkaβVolterra equations.
Read more
Neural Quadratic Forms: A Unified Minimal Model for Sudden Learning and Scaling Laws
Summary
This paper introduces a unified minimal model termed Neural Quadratic Forms (NQF) to explain the abrupt learning dynamics and scaling laws observed in neural networks. The authors argue that despite the diverse architectures of neural networks, they exhibit common behaviors characterized by long plateaus in training loss followed by sudden drops. This phenomenon can be attributed to a few relevant collective variables, which are derived from the symmetry inherent in neural architectures. By leveraging this symmetry, the authors derive a universal leading form for the expansion of the cost function around near-zero weights, represented as Tr[WWβΊA(x)], where A(x) is a structure matrix specific to each architecture. The model captures the dynamics of various architectures, including perceptrons, attention layers, and convolutions, under a common framework. The training dynamics are shown to converge on an order parameter M = WWβΊ, leading to a LotkaβVolterra equation that describes the sequential activation of modes during training. The paper confirms its theoretical predictions through numerical experiments across different training methods and architectures, demonstrating the universality of the NQF framework.
Methodology
The authors utilize symmetry principles to derive a universal quadratic form for neural network dynamics. They analyze the training behavior of various architectures by expanding the cost function around near-zero weights and identifying collective variables that govern the learning process. Numerical simulations are conducted to validate the theoretical framework across different neural network architectures.
Results
The study finds that the derived NQF accurately describes the training dynamics of various neural network architectures, confirming that the abrupt learning steps and smooth power laws in training losses can be unified under a single model. The predictions regarding the order parameter and the scaling laws are supported by numerical experiments, demonstrating the model's robustness across different training methods.
Implications
The findings suggest that understanding the collective dynamics of neural networks can lead to better design principles for training and scaling models. The NQF framework may inform future research on optimizing neural architectures and improving training efficiency, potentially impacting fields such as deep learning and artificial intelligence.
Training Under Challenge: Executable Certificates and Challenge-Closed Optimality for Neural Networks
Theory
Optimization
- Introduces an executable-certificate framework for assessing neural network training.
- Defines a challenge-power modulus to characterize optimality gaps.
- Demonstrates the framework's effectiveness on a ResNet-18 distillation problem.
- Highlights the need for coverage mechanisms in validating training outcomes.
Read more
Training Under Challenge: Executable Certificates and Challenge-Closed Optimality for Neural Networks
Summary
This paper introduces 'Training Under Challenge', an executable-certificate framework designed to assess the global optimality of neural networks during training. The authors argue that a flat training curve does not necessarily indicate that a network has reached a global optimum, and propose a method to construct alternative models that can be used to evaluate the same objective. The framework allows for the identification of lower-valued candidates that serve as witnesses to the global-optimality gap. The authors define a challenge-power modulus that characterizes the largest gap compatible with successful passage of challenges. They demonstrate the effectiveness of their approach through a case study involving a ResNet-18 distillation problem, where they cover all output directions and achieve bounds close to the true optimality gap. The paper also discusses the importance of coverage mechanisms and presents a structured methodology for auditing neural network training, emphasizing the need for rigorous evidence to support claims of optimality.
Methodology
The authors propose a framework that constructs complete alternative models through various architectural procedures and reevaluates the same objective. They define a resource-indexed challenge-power modulus and utilize block-decrease operators to check coverage and yield bounds on optimality gaps. The framework includes a systematic audit process for checkpoints, categorizing their status into Red, Yellow, Green, and Inconclusive based on performance relative to a reference model.
Results
In their experiments with a channel-gated ResNet-18 distillation problem, the authors successfully covered all 240 output directions with eight internal challenges. The realized-residual bounds achieved were within factors of 1.74β3.02 of the true optimality gap, demonstrating the framework's capability to provide meaningful assessments of model performance.
Implications
The proposed framework has significant implications for the training assurance of neural networks, providing a structured approach to evaluate and certify model performance. It can be applied in various domains where neural networks are used, ensuring that models are not only trained effectively but also validated against rigorous standards of optimality.