AI-generated summaries
Today's ML research,
without the noise.
Daily summaries of the latest machine learning papers from arXiv, processed every 8 hours.
24
Papers today
8h
Update frequency
7
Days of history
Constant regret in general games via higher-order optimism
Theory
Optimization
- Introduction of the HOOD algorithm, achieving O(N^3 log^2 K) regret in N-player games.
- The algorithm stabilizes play by dampening oscillations, addressing a key issue in previous methods.
- HOOD is horizon-free, allowing for constant regret without prior knowledge of play duration.
- Empirical distribution of play converges to coarse correlated equilibria at a rate of O(1/T).
Read more
Constant regret in general games via higher-order optimism
Summary
This paper presents a novel uncoupled learning algorithm called higher-order optimism with discounting (HOOD) that guarantees O(N^3 log^2 K) individual regret for players in an N-player normal form game with up to K actions per player. The algorithm is a variant of the optimistic follow-the-regularized-leader (OptFTRL) method, incorporating a discounted (N + 1)-th order predictor and entropic regularization to stabilize the sequence of play and mitigate large oscillations. The authors address the challenge of achieving constant regret in general games, where previous methods have struggled. They demonstrate that if all players adopt the HOOD algorithm, their individual regret remains bounded over time, ensuring that regret does not accumulate indefinitely. The approach is horizon-free, meaning it does not require prior knowledge of the play duration, and leads to convergence of the empirical distribution of play to the game's coarse correlated equilibria at a rate of O(1/T). This work builds on existing literature on no-regret learning and offers a significant advancement in the understanding of learning dynamics in multi-player games.
Methodology
The authors developed the HOOD algorithm, which combines a discounted (N + 1)-th order predictor with entropic regularization. This method is designed to dampen oscillations in player strategies, ensuring stable learning dynamics in multi-player settings. The algorithm operates in a deterministic, full-information context where players receive feedback on their mixed payoffs.
Results
The main result is that all players using the HOOD algorithm will have their individual regret bounded by O(N^3 log^2 K) over time, ensuring that regret does not accumulate indefinitely. This result is significant as it confirms the possibility of constant regret in general games, a question that had remained open in the field.
Implications
The findings suggest that the HOOD algorithm can be effectively applied in various multi-player game scenarios, potentially improving learning outcomes in competitive environments. This could have applications in economic models, strategic decision-making, and any domain where multi-agent interactions are present.
From Nowcasting to Forecasting: Adapting a Reanalysis-Trained Cloud Cover Model to Observations
Generative Models
Time Series
Computer Vision
- CloudCast v2 improves cloud-cover forecasting accuracy over a 12-hour range.
- The model is trained on reanalysis data and adapted to satellite observations using conditional flow matching.
- It shows a 10% reduction in mean absolute error compared to CloudCast v1.
- The model retains spatial detail from satellite data while extending forecasting capabilities.
Read more
From Nowcasting to Forecasting: Adapting a Reanalysis-Trained Cloud Cover Model to Observations
Summary
This paper presents CloudCast v2, a machine-learning model designed for 12-hour cloud-cover forecasting that integrates observation-based initial conditions. The model is trained on the Copernicus European Regional Reanalysis data to learn cloud-evolution dynamics and is adapted to satellite-derived cloud fields using a generative method called conditional flow matching. The study highlights the challenges of accurately forecasting cloud cover due to its dependence on atmospheric dynamics and initial-state errors. By combining the strengths of numerical weather prediction (NWP) and observation-based extrapolation, CloudCast v2 aims to provide a coherent forecast that retains spatial detail from satellite observations while extending the forecasting range beyond traditional nowcasting limits. The results indicate a 10% reduction in mean absolute error compared to its predecessor, CloudCast v1, and improved spatial agreement as measured by the fractions skill score after 3-6 hours, depending on cloudiness categories.
Methodology
The authors developed CloudCast v2 by first training a vision transformer-based model on reanalysis data to learn cloud evolution dynamics. The model was then adapted to satellite-derived cloud fields using conditional flow matching, a generative approach that transforms noise into cloud-cover forecasts conditioned on observed initial conditions and NWP inputs.
Results
CloudCast v2 achieved a 10% reduction in mean absolute error compared to CloudCast v1 over the 1-12 hour forecasting range. It also surpassed CloudCast v1 in fractions skill score after approximately 3-6 hours, indicating improved spatial agreement in cloud cover predictions.
Implications
The findings suggest that integrating observation-based initial conditions with machine-learning models can enhance the accuracy and reliability of cloud-cover forecasts, which are critical for applications in meteorology, solar energy management, and climate studies. This approach may pave the way for more effective operational forecasting systems that can adapt to evolving atmospheric conditions.
WeatherNext 3: Increasing resolution and performance of global weather models with raw observations
Time Series
- WN3 generates hourly forecasts using real-time geostationary satellite data.
- Achieves a spatial resolution of 0.1Β° for single-level atmospheric variables.
- Directly predicts satellite-derived precipitation and local temperature/dewpoint conditions.
- Demonstrates lower error rates compared to existing global weather models.
Read more
WeatherNext 3: Increasing resolution and performance of global weather models with raw observations
Summary
The paper presents WeatherNext 3 (WN3), an advanced AI-based weather forecasting model that addresses limitations of previous models by increasing both spatial and temporal resolution while directly utilizing raw observational data. Traditional weather models typically generate forecasts every six hours and rely on analysis data that can inherit biases, particularly in precipitation and temperature. In contrast, WN3 generates forecasts every hour by ingesting low-latency geostationary satellite data, achieving a spatial resolution of 0.1Β° for single-level variables. The model not only predicts standard atmospheric variables but also incorporates satellite-derived precipitation estimates and local geographical features to enhance accuracy. This approach allows WN3 to provide 2m temperature and dewpoint predictions with significantly lower error rates than competing models, even for unseen locations. The paper emphasizes the operational capabilities of WN3, which integrates the traditionally distinct stages of data assimilation, forecasting, and post-processing, thereby pushing the boundaries of global weather prediction performance and granularity.
Methodology
WeatherNext 3 employs a novel architecture that ingests both analysis data and real-time geostationary satellite imagery to predict weather trajectories. It utilizes machine learning techniques to model the distribution of future weather conditions based on these inputs, allowing for high-resolution and timely forecasts.
Results
WN3 establishes a new state-of-the-art in probabilistic medium-range weather forecasting, outperforming previous models in accuracy and resolution. It successfully predicts key variables with significantly reduced error rates, particularly in 2m temperature and dewpoint forecasts, even for locations not included in the training data.
Implications
The advancements in WN3 could lead to more accurate and timely weather forecasts, which are crucial for various sectors including agriculture, disaster management, and climate research. The model's ability to directly utilize raw observations may also inspire further innovations in AI-driven meteorology.
Risk and Anomaly Identification for Distribution Network Optimal Operation Based on Reinforcement Learning and Uncertainty Quantification
Reinforcement Learning
Optimization
Time Series
- Introduces a DRL framework that is explicitly uncertainty-aware for DN operations.
- Decomposes total uncertainty into aleatoric and epistemic components for better risk and anomaly identification.
- Demonstrates the effectiveness of the proposed method through simulation results.
- Addresses the limitations of traditional DRL methods in handling out-of-distribution scenarios.
Read more
Risk and Anomaly Identification for Distribution Network Optimal Operation Based on Reinforcement Learning and Uncertainty Quantification
Summary
This paper addresses the critical challenge of identifying operational risks and anomalies in modern distribution networks (DNs) under uncertainty. The author proposes a deep reinforcement learning (DRL) framework that incorporates uncertainty quantification (UQ) to enhance the reliability of DN operations. The framework distinguishes between two types of uncertainty: aleatoric uncertainty (AU), which is inherent to the environment, and epistemic uncertainty (EU), which arises from a lack of knowledge. By integrating distributional and Bayesian DRL, the proposed method decomposes total uncertainty into these components, allowing for better characterization of risks and anomalies. The epistemic estimates guide exploration during training and facilitate out-of-distribution detection during deployment, while aleatoric estimates help in assessing intrinsic operational risks. Simulation results demonstrate the effectiveness of the proposed DRL agent and the UQ approach in improving the robustness and safety of DN operations, particularly in the face of unpredictable conditions such as extreme weather or cyber-physical attacks.
Methodology
The paper employs a deep reinforcement learning framework that integrates distributional and Bayesian methods to quantify uncertainty. It distinguishes between aleatoric and epistemic uncertainties, using them to inform both training and operational decision-making processes. The framework is tested through simulations that mimic real-world DN conditions.
Results
The simulation results indicate that the proposed DRL agent effectively identifies risks and anomalies, improving operational reliability under various uncertain conditions. The uncertainty quantification approach enhances the agent's ability to make informed decisions, reducing the likelihood of unsafe actions and operational failures.
Implications
The findings suggest that the proposed uncertainty-aware DRL framework can significantly enhance the safety and efficiency of distribution network operations, particularly in environments characterized by high uncertainty and variability, such as those influenced by renewable energy sources. This approach could be applied in real-time energy management systems to improve grid resilience against anomalies and risks.
Kernel Reboot: Breaking the Boundaries of Neural Tangent Kernels for Neural Fields
Computer Vision
- Introduces three novel algorithms to enhance Neural Fields using Neural Tangent Kernels.
- NTK-KIP allows for efficient inpainting from sparse data by learning a distilled support set.
- MetaQuill enables quick adaptation of INRs through meta-learning, enhancing feature learning.
- MetaQuill-KIP combines the advantages of both previous methods for improved reconstruction quality.
Read more
Kernel Reboot: Breaking the Boundaries of Neural Tangent Kernels for Neural Fields
Summary
This paper addresses the challenges of high-quality reconstruction in Neural Fields (NFs) from sparse observations by leveraging Neural Tangent Kernels (NTKs). The authors propose three innovative algorithms: NTK-KIP, MetaQuill, and MetaQuill-KIP. NTK-KIP focuses on learning a distilled support set of coordinates to enable inpainting of large missing regions with minimal observed data, resulting in a compact non-linear representation. MetaQuill introduces a meta-learning approach that allows for quick adaptation of implicit neural representations (INRs) to new scenes by updating only a small task-specific weight offset, thus facilitating feature learning and reusability of prior knowledge. The third algorithm, MetaQuill-KIP, combines the strengths of the first two, starting with a KIP-style non-linear warm start and refining the model with a small offset around a meta-learned initialization. The results demonstrate that MetaQuill-KIP achieves high Peak Signal-to-Noise Ratio (PSNR) reconstructions and semantically plausible inpainting, even under very sparse observations, while requiring less computational effort compared to traditional diffusion-based methods. This work shows that NTK-driven neural fields can be both non-linear and meta-learnable, bridging the gap between analytical kernels and practical few-shot reconstruction.
Methodology
The authors developed three algorithms: NTK-KIP for learning a compact representation from sparse data, MetaQuill for meta-learning a shared initialization for quick adaptation, and MetaQuill-KIP which integrates both approaches to refine the model with minimal adjustments. The algorithms utilize NTK principles to facilitate efficient training and inference.
Results
MetaQuill-KIP achieved high PSNR values in reconstructions and effective inpainting under sparse observations, outperforming traditional diffusion-based methods that require extensive pre-training and fine-tuning.
Implications
The proposed methods could significantly enhance applications in computer vision tasks such as image reconstruction, 3D object modeling, and scene understanding, particularly in scenarios with limited data availability.
Witnesses Explain Anomalies
Interpretability
- WAND is an unsupervised anomaly detector that provides explanations by design.
- The method uses witness directions in feature space to attribute scores to anomalies.
- Scoring is efficient, running in linear time relative to sample size.
- WAND outperforms 16 unsupervised baselines in detection while offering superior explanation quality.
Read more
Witnesses Explain Anomalies
Summary
The paper introduces WAND, an unsupervised anomaly detection method designed to provide explanations for flagged anomalies directly during the scoring process. Traditional anomaly detectors score data points without revealing which features contribute to the scores, often requiring post-hoc methods like SHAP or LIME that are computationally expensive and approximate. WAND, however, computes scores based on the projection of data points onto directions on the unit sphere, measuring how far these projections exceed a sub-Gaussian baseline. The 'witness directions' that indicate anomalies also serve as explanations, providing per-feature attributions without additional computational cost. The method is efficient, with scoring time linear to sample size, and guarantees that every anomaly has a corresponding witness. Empirical evaluations on 47 ADBench datasets demonstrate that WAND achieves competitive detection performance while offering more accurate and faithful explanations compared to existing methods, including SHAP, LIME, and ECOD, at a significantly lower query cost. This positions WAND as a practical solution for interpretable anomaly detection in tabular data.
Methodology
WAND organizes its computation around directions on the unit sphere, scoring each data point based on how far its projection exceeds a calibrated sub-Gaussian baseline. The witness directions that flag anomalies serve as explanations, providing per-feature attributions that are recoverable through gradients. The method ensures linear scoring time and establishes a probe-efficiency guarantee for anomaly detection.
Results
WAND achieved the best mean Friedman rank across 47 ADBench datasets, demonstrating ROC-AUC parity with 16 unsupervised baseline methods. It produced more accurate and faithful explanations compared to post-hoc methods like SHAP and LIME, while maintaining a lower computational cost.
Implications
WAND's design allows for practical applications in fields requiring interpretable anomaly detection, such as finance, healthcare, and cybersecurity, where understanding the reasons behind flagged anomalies is crucial for decision-making.
ObserverBench: Testing Mechanistic Estimates for Intervention and Control
Interpretability
- ObserverBench is a benchmark framework for evaluating internal estimators in mechanistic interpretability.
- Estimation accuracy does not guarantee effective decision-making; observers must be trained to minimize action loss.
- Task-specific evaluations reveal that prediction accuracy and decision quality can diverge significantly.
- Safety tasks require consideration of consequences, as high classification accuracy may lead to poor intervention allocation.
Read more
ObserverBench: Testing Mechanistic Estimates for Intervention and Control
Summary
The paper introduces ObserverBench, a benchmark framework designed to evaluate the adequacy of internal estimators (observers) used in mechanistic interpretability for guiding interventions, control, and safety tasks in machine learning models. The framework addresses the limitations of relying solely on average estimation accuracy, which can lead to poor decision-making in specific scenarios. ObserverBench establishes a set of fixed tasks that define the model, observer information, allowed actions, and evaluation metrics, allowing for a comparative analysis of observers under the same constraints. The study identifies three critical distinctions: task-relevant accuracy, the difference between prediction and decision-making, and the importance of consequence-aware safety. Through experiments on models like GPT-2-small and Qwen2.5-7B, the authors demonstrate that better prediction does not always correlate with improved decision-making, highlighting the need for observers to be trained to minimize action loss directly. The findings emphasize that classification accuracy alone is insufficient for safety tasks, as it may lead to poor allocation of intervention budgets. Overall, ObserverBench provides a structured approach to assess the effectiveness of interpretability methods in practical applications.
Methodology
The authors developed ObserverBench as a set of standardized tasks that fix the model, observer information, allowed actions, and evaluation metrics. Each observer is evaluated based on two outcomes: prediction accuracy and the incurred loss from chosen actions on held-out test cases. The framework allows for comparisons of observers under identical constraints, focusing on operational distinctions in decision-making processes.
Results
Experiments showed that observers trained for better prediction did not consistently lead to improved decision-making outcomes. In specific tasks, such as circuit interventions, observers that accurately predicted mean effects did not always choose actions that minimized loss. Additionally, in safety tasks, observers with perfect classification scores could still allocate intervention budgets poorly, underscoring the necessity for consequence-aware evaluations.
Implications
ObserverBench can be utilized to enhance the design and evaluation of interpretability methods in machine learning, particularly in safety-critical applications. By focusing on the adequacy of internal estimators for specific tasks, it can inform the development of more effective intervention strategies and safety protocols in AI systems.
SimpleDesign: A Joint Model for Protein Sequence and Structure Codesign
Generative Models
Multimodal
- Introduction of SimpleDesign, a single-stage model for protein sequence and structure co-design.
- Elimination of the need for structure tokenization, simplifying the training process.
- Utilization of a Transformer-based architecture for effective modality-specific processing.
- Competitive performance achieved on co-design and generation benchmarks with over 2 million training pairs.
Read more
SimpleDesign: A Joint Model for Protein Sequence and Structure Codesign
Summary
The paper presents SimpleDesign, a novel generative model aimed at the co-design of protein sequences and their three-dimensional structures. Unlike existing models that rely on a multi-stage training process involving autoencoders and latent representations, SimpleDesign proposes a single-stage end-to-end training framework. This model utilizes a Transformer-based architecture that allows for modality-specific processing while maintaining global self-attention across both sequence and structure data. The authors argue that their approach simplifies the training pipeline and enhances performance by directly embedding continuous 3D coordinates without the need for a structure tokenizer. The model was trained on over 2 million sequence-structure pairs and demonstrated competitive performance in co-design and unconditional generation tasks, suggesting that effective protein design can be achieved without complex tokenization schemes. Overall, SimpleDesign bridges the gap between traditional protein language models and flow-based methods, providing a minimalist yet effective solution for protein engineering.
Methodology
SimpleDesign employs a Transformer-based multimodal backbone that integrates discrete cross-entropy for sequence generation and a regression objective for structure generation. The model is trained end-to-end on raw data, allowing for direct interaction between sequence and structure modalities through joint attention mechanisms.
Results
The model achieved competitive performance across various benchmarks for co-design and unconditional generation of protein sequences and structures, indicating its effectiveness compared to existing multi-stage models.
Implications
The findings suggest that simpler, more efficient training frameworks can be developed for protein design, potentially accelerating advancements in drug discovery and protein engineering by reducing computational overhead and complexity.
Selective Hypergraph Refinement for Frozen Graph Clustering
Graph Learning
- Introduces Selective Hypergraph Refinement (SHR) for post-processing frozen graph clustering.
- Utilizes attribute hypergraphs to capture higher-order relationships for refining cluster assignments.
- Implements a selective update mechanism based on evidence from graph structure and node attributes.
- Demonstrates positive clustering performance gains with limited changes to existing assignments.
Read more
Selective Hypergraph Refinement for Frozen Graph Clustering
Summary
This paper addresses the challenge of improving clustering performance in frozen graph models, which are models that have already been trained and whose parameters and structures cannot be modified. The author proposes a novel approach called Selective Hypergraph Refinement (SHR), which utilizes an attribute hypergraph to enhance the clustering results without the need for labels or adjustments to the model. The method generates candidate refinements based on higher-order relations captured by the hypergraph and evaluates their reliability using existing graph structures and node attributes. SHR selectively updates cluster assignments only when there is sufficient evidence to support the change, thereby minimizing the risk of erroneous updates. The study reveals that while there is potential for improvement in clustering outputs, the effectiveness of refinements varies across different datasets and models. The results indicate a positive mean macro gain in clustering performance with a low percentage of changes in hard assignments, suggesting that frozen models retain a limited but significant refinement capacity.
Methodology
The methodology involves using an attribute hypergraph to generate candidate residual directions for cluster refinement. The reliability of these candidates is evaluated without labels, relying on graph structure, node attributes, and matched-null evidence. Only modifications with sufficient support are retained, while others maintain their original assignments. The study includes controlled evaluations across various graph-clustering backbones and datasets to assess the effectiveness of SHR.
Results
The evaluation of SHR across 15 backbone-dataset cells showed a positive mean macro gain of 0.066 percentage points, with only 0.209% of hard assignments changing on average. A broader evaluation across 15 combinations yielded a macro gain of 0.137 percentage points with a mean change ratio of 0.375%. The findings indicate that frozen clustering outputs can be refined, though the degree of improvement varies significantly depending on the specific backbone-dataset pair.
Implications
The findings suggest that there is potential for enhancing the performance of frozen graph clustering models through selective refinement techniques. This could lead to more robust clustering applications in various domains where graph-based data is prevalent, such as social network analysis, bioinformatics, and recommendation systems.
Pushing the (Decision) Boundaries: Dynamically Calibrating Differentially Private Noise to Explainability in Federated Learning
Federated Learning
Interpretability
- Introduction of XCal-FL, a closed-loop training algorithm for dynamic DP noise calibration.
- Demonstrated significant improvements in predictive performance and explanation fidelity over static-noise methods.
- Revealed non-linear dynamics of explanation fidelity in relation to cumulative privacy loss.
- Highlighted the importance of explainability in the privacy-utility trade-off, especially in clinical settings.
Read more
Pushing the (Decision) Boundaries: Dynamically Calibrating Differentially Private Noise to Explainability in Federated Learning
Summary
This paper addresses the challenge of maintaining explanation fidelity in Federated Learning (FL) when incorporating Differential Privacy (DP). The authors introduce XCal-FL, a novel training algorithm that dynamically calibrates DP noise based on three signals: prediction logit variations, counterfactual margins, and saliency concentration. This closed-loop approach allows for real-time adjustments to the noise applied during training, enhancing both predictive performance and explanation fidelity. The study demonstrates that XCal-FL significantly outperforms traditional static-noise methods, achieving over 10% improvement in accuracy and up to 5Γ better explanation fidelity across various medical imaging datasets. The findings also reveal that explanation fidelity exhibits non-linear dynamics in relation to privacy loss, suggesting that explainability is a critical dimension of the privacy-utility trade-off that requires dedicated consideration in decision-critical applications.
Methodology
The authors developed XCal-FL, which utilizes a closed-loop approach to dynamically adjust DP noise during training based on three complementary signals: prediction logit variations, counterfactual margins, and saliency concentration. This method enforces formal DP guarantees through adaptive privacy accounting, allowing for improved model explanations while maintaining privacy.
Results
Experiments conducted on three medical imaging datasets showed that XCal-FL improved predictive performance by over 10% and explanation fidelity by up to 5Γ compared to static-noise FL methods. Additionally, XCal-FL outperformed existing adaptive DP methods in terms of explanation fidelity and achieved better privacy-budget efficiency.
Implications
The findings suggest that explainability should be a key consideration in the design of differentially private FL systems, particularly in sensitive applications like healthcare. The dynamic calibration of noise could lead to more trustworthy AI systems that provide reliable explanations, thereby enhancing clinician confidence and facilitating broader adoption of AI in medical diagnostics.
Guide, Not Bind: Why Defeasible Priors Fail in Augmented Lagrangian Causal Discovery
Graph Learning
Optimization
Theory
- Defeasible priors in causal discovery can lead to significant edge suppression due to early penalty application.
- The DADU relaxation rule fails to satisfy necessary conditions for effective adaptive relaxation.
- Correlation-matching objectives can obscure true causal relationships, making identification difficult.
- A new relaxation operator combined with covariance matching improves edge recovery rates significantly.
Read more
Guide, Not Bind: Why Defeasible Priors Fail in Augmented Lagrangian Causal Discovery
Summary
This paper critiques the use of defeasible priors in differentiable causal discovery methods that employ Augmented Lagrangian (ALM) penalties. The authors argue that the assumption that data-adaptive mechanisms can override incorrect prior constraints is flawed. They identify two main reasons for this failure: first, the sequential penalty-ramping of ALM can suppress true edges before they can be detected by counterfactual checks, leading to an 'early suppression trap.' They introduce DADU, a natural relaxation rule, and demonstrate that it fails to meet necessary conditions for adaptive relaxation, resulting in a high rate of suppressed true edges. Second, they show that the standard correlation-matching objective ties the costs of true edges and their reverses, making identification impossible under certain conditions. The authors propose a new relaxation operator combined with covariance matching, which significantly improves edge recovery rates compared to DADU. However, they find that even with improvements, the majority of the weight of suppressed edges is still misallocated. The paper concludes that the issues are not solely artifacts of the correlation-matching objective, as they worsen under likelihood objectives, indicating a deeper problem with penalty schedules in causal discovery.
Methodology
The authors analyze the performance of differentiable causal discovery methods using Augmented Lagrangian penalties, focusing on the effects of sequential penalty-ramping and correlation-matching objectives. They introduce a new relaxation operator and conduct extensive experiments across various graph sizes to evaluate edge recovery rates.
Results
The proposed relaxation operator, when combined with covariance matching, recovers wrongly-forbidden edges 27 to 52 times more often than DADU. However, a significant portion of the weight of suppressed edges is still misallocated to their reverse directions. The correlation-specific issues were confirmed to be absent under least-squares and likelihood objectives, but the suppression mechanism worsened under likelihood objectives.
Implications
The findings suggest that practitioners should be cautious when encoding expert priors in causal discovery systems, as traditional methods may lead to significant misidentifications. The proposed methods could enhance the robustness of causal discovery frameworks, particularly in complex domains where prior knowledge is uncertain.
Causal Foundation Models
Theory
Efficient ML
- CFMs enable causal inference without the need for retraining on new datasets.
- The paper provides practical resources, including example code and Jupyter notebooks.
- CFMs demonstrate improved performance and speed compared to traditional causal inference methods.
- The authors benchmark CFMs against existing methods to validate their effectiveness.
Read more
Causal Foundation Models
Summary
This paper introduces Causal Foundation Models (CFMs), a novel approach to causal inference that leverages the paradigm of foundation models in machine learning. Traditional causal inference requires a bespoke pipeline for each problem, involving the proposal of causal mechanisms, selection of estimators, and hyperparameter tuning. In contrast, CFMs are pretrained neural networks capable of estimating causal quantities, such as average treatment effects, on new datasets using in-context learning without the need for model updates. The authors provide a practical introduction to CFMs, including necessary background on causal inference and machine learning, along with example code and Jupyter notebooks to facilitate immediate application. The paper benchmarks CFMs against traditional causal inference methods, demonstrating superior performance and inference speed. The authors aim to broaden the audience for CFMs and encourage their adoption in various domains, highlighting their potential for transformative applications in causal inference.
Methodology
The authors present CFMs as pretrained neural networks that utilize in-context learning to estimate causal effects from observational data. They benchmark CFMs against traditional causal inference methods, providing a comparative analysis of performance across various tasks.
Results
CFMs showed top performance on causal inference tasks, outperforming traditional methods in both speed and accuracy. The paper includes benchmarks of the first three publicly available CFMs, demonstrating their effectiveness in diverse causal inference scenarios.
Implications
The introduction of CFMs could revolutionize causal inference practices across multiple domains, such as economics, healthcare, and policy-making, by providing faster and more accurate causal estimates. This could lead to better-informed decision-making and policy formulation.
Modern Transformers Are Implicit Hybrids: From Functional Differentiation to Principled Hybrid Architecture Design
NLP
Large Language Models
- Introduces RFIS and RPD metrics for analyzing attention head functions in Transformers.
- Establishes a two-type taxonomy of retrieval and positional heads based on frequency contributions.
- Proposes the Global Positional Band (GPBand) as a mechanism-level boundary for attention functions.
- Presents the Head-wise Hybrid Architecture (HwH) that effectively combines FA and LA.
Read more
Modern Transformers Are Implicit Hybrids: From Functional Differentiation to Principled Hybrid Architecture Design
Summary
This paper investigates the hybrid architecture of modern RoPE-based Transformers, specifically focusing on the integration of Full Attention (FA) and Linear Attention (LA). The authors highlight that while hybrid architectures are gaining traction, their design often relies on heuristic methods. To address this, they introduce two metrics: RoPE Frequency Importance Score (RFIS) and RoPE Positional Dependence (RPD), which help in understanding the functional organization of attention heads. The study identifies a two-type taxonomy of headsβretrieval and positionalβbased on frequency contributions, separated by a Global Positional Band (GPBand). The authors propose two design principles for hybrid architectures: (i) positional modeling should be local, while global retrieval should be position-independent; and (ii) functions should be assigned at the head level with layer-specific allocation. They instantiate these principles in a new architecture called Head-wise Hybrid Architecture (HwH), which demonstrates improved performance in language modeling and commonsense reasoning, particularly in zero-shot long-context extrapolation tasks. The findings suggest that principled hybrid architecture design could enhance future foundation models.
Methodology
The authors utilized behavioral probes to analyze head-level functional differentiation in RoPE Transformers. They introduced RFIS and RPD as intervention-based metrics to assess the impact of frequency contributions and positional modulation on attention distributions. The study involved controlled experiments with Qwen3-series models and Llama3.1 to validate the proposed taxonomy and design principles.
Results
The application of RFIS and RPD revealed a clear separation of retrieval and positional heads, confirming the existence of the GPBand. The Head-wise Hybrid Architecture (HwH) was shown to maintain strong language modeling capabilities while significantly enhancing retrieval performance and zero-shot long-context extrapolation compared to existing models.
Implications
The findings suggest that a principled approach to hybrid architecture design can lead to more effective Transformer models, particularly in tasks requiring long-context understanding and retrieval capabilities. This could influence future research and development of foundation models in NLP.
LongCounsel-8: A Benchmark Suite for Longitudinal Depression Tracking from Multi-Session Counseling Dialogues
NLP
- Introduction of LONGCOUNSEL-8, a comprehensive benchmark for longitudinal depression tracking.
- Validation of the benchmark through controlled-state fidelity and linguistic analysis.
- Findings indicate that lower score errors do not ensure accurate trend identification.
- Existing methods are less reliable for worsening depression trajectories.
Read more
LongCounsel-8: A Benchmark Suite for Longitudinal Depression Tracking from Multi-Session Counseling Dialogues
Summary
The paper introduces LONGCOUNSEL-8, a benchmark suite designed to facilitate longitudinal depression tracking through multi-session counseling dialogues. The authors identify a significant gap in existing resources, which either lack standardized session-level depression labels or consist of single-session dialogues. To address this, LONGCOUNSEL-8 comprises three independently generated datasets totaling 7,749 five-session counseling trajectories, grounded in real-world client profiles and empirical patterns of depression. The construction of this benchmark overcomes challenges related to maintaining longitudinal consistency, grounding symptom progression in empirical data, and expressing controlled depression states naturally. The authors validate the benchmark's fidelity through controlled-state tests and linguistic analyses, confirming its utility for evaluating depression assessment methods. The findings reveal that lower single-session score errors do not guarantee accurate trend identification, existing methods struggle with worsening trajectories, and additional session history may not always enhance prediction accuracy. Overall, LONGCOUNSEL-8 serves as a foundational resource for advancing the understanding and assessment of mental health changes over time.
Methodology
The authors employed profile-grounded longitudinal simulation, empirically informed state construction, and indirect behavioral realization to create the datasets. They utilized real-world client profiles and depression trajectories to ensure the authenticity of the dialogues, while also employing the PHQ-8 questionnaire for standardized symptom states.
Results
The validation of LONGCOUNSEL-8 demonstrated that simulated self-reports effectively recovered controlled states, supporting label fidelity. Experiments revealed that existing depression tracking methods are less reliable in identifying worsening trends and that more historical context does not always lead to improved accuracy in trend prediction.
Implications
LONGCOUNSEL-8 provides a critical resource for researchers and practitioners aiming to enhance the accuracy of longitudinal depression assessments, moving beyond static evaluations to more dynamic tracking of mental health changes over time.
LeanStream: A Speculate-and-Refine Streaming Framework for Efficient on-Device LLM Inference
Large Language Models
Efficient ML
- LeanStream reduces memory usage by 4.8Γβ7.5Γ compared to previous systems.
- It improves token generation throughput by 1.6Γβ2.1Γ.
- The framework employs a speculate-and-refine strategy for dynamic computation and data loading.
- LeanStream allows for fine-grained overlap between GPU execution and I/O operations.
Read more
LeanStream: A Speculate-and-Refine Streaming Framework for Efficient on-Device LLM Inference
Summary
The paper introduces LeanStream, a novel streaming framework designed to enhance the efficiency of on-device inference for Large Language Models (LLMs). The challenge of on-device LLM inference arises from the limited memory and computational capacity of mobile and embedded devices, which often cannot accommodate the large model weights. Existing systems have attempted to address this by utilizing activation sparsity and offloading weights to external storage, but they face a trade-off between accurate execution decisions and efficient computation-I/O overlap. LeanStream addresses this issue by employing a speculate-and-refine approach that progressively updates computation and data loading priorities based on partial GPU results. This allows for fine-grained overlap between GPU execution and storage I/O, significantly reducing memory usage and improving throughput. The framework has been implemented on both mobile and embedded platforms, demonstrating its effectiveness in optimizing resource utilization while maintaining high performance.
Methodology
LeanStream utilizes a speculate-and-refine framework that begins with an initial prediction based on the most recent context and continuously refines computation and data loading priorities using partial results from the GPU. This approach enables the system to stream required weights in fine-grained blocks, allowing for immediate execution of high-priority sub-models as soon as resources are available.
Results
The implementation of LeanStream on mobile and embedded platforms resulted in a significant reduction in memory usage (4.8Γβ7.5Γ) and an increase in token generation throughput (1.6Γβ2.1Γ) compared to existing on-device LLM inference systems.
Implications
The LeanStream framework has the potential to enhance on-device LLM applications, making them more efficient and responsive while addressing privacy concerns. This could lead to broader adoption of LLMs in mobile and embedded environments, facilitating real-time applications in various domains such as personal assistants, language translation, and more.
Conditioning Degenerate Diffusion Models
Generative Models
Theory
Optimization
- Introduction of loss functions for degenerate diffusion models that yield a control ensuring the same law for conditioned and unconditioned processes.
- Extension of predictable representation results to random initializations, enhancing the applicability of the theory.
- Development of local regression losses that do not depend on ambient densities or scores, broadening the scope of generative modeling.
- Demonstration of the proposed methods through numerical experiments and applications in image generation.
Read more
Conditioning Degenerate Diffusion Models
Summary
This paper addresses the challenges of conditioning degenerate diffusion models, which are generative models that rely on score functions for training guidance. The authors propose a novel approach using causal optimal transport to define approximate loss functions that facilitate minimum-entropy control under minimal assumptions. The work extends the predictable representation results for degenerate diffusions to random initializations and introduces local regression losses that do not require an ambient density or score. The authors demonstrate the effectiveness of their loss functions through numerical experiments and applications in image generation, showcasing their ability to handle cases where the conditional densities are not smooth or do not exist. The findings contribute to the understanding of degenerate diffusion processes and provide a framework for developing generative models in complex scenarios.
Methodology
The authors utilize causal optimal transport to derive loss functions for conditioned diffusion processes. They leverage predictable representation properties of stochastic differential equations (SDEs) and introduce local regression losses that do not require smoothness assumptions on the conditional densities. The methodology includes a closed-loop control framework and a numerical experimentation component to validate the proposed loss functions.
Results
The proposed loss functions successfully identify a minimum-energy Brownian perturbation under local square integrability conditions. The numerical experiments demonstrate the performance of these loss functions across various data increments, highlighting their effectiveness in generating images and handling degenerate diffusion scenarios.
Implications
The findings have significant implications for the development of generative models in fields where conditioning on complex data is necessary, such as physical chemistry, genetics, and economics. The proposed methods can enhance the robustness and applicability of generative models in situations where traditional approaches fail due to degeneracy or non-smoothness of densities.
Prospective Coding Improves Learning in Deep Continuous-Time Recurrent Networks
Audio & Speech
Time Series
Theory
- Introduction of Recursive Quadrature Filters (RQFs) as a new architecture for deep CTRNs.
- Implementation of prospective coding to improve learning efficiency and mitigate gradient attenuation.
- Empirical results show RQFs outperform traditional models in accuracy on tasks like Speech Commands and Path-X.
- RQFs are compatible with existing training algorithms for diagonal state-space models.
Read more
Prospective Coding Improves Learning in Deep Continuous-Time Recurrent Networks
Summary
This paper addresses the challenges faced by deep continuous-time recurrent networks (CTRNs) in learning due to the trade-off between memory and reactivity. The authors introduce Recursive Quadrature Filters (RQFs), a novel architecture inspired by biological neural circuits, which serve as a special case of diagonal state-space models (SSMs). The key innovation is the implementation of prospective coding, where each layer's bottom-up input is made prospective rather than instantaneous. This adjustment mitigates the depth-dependent gradient attenuation that occurs during training, particularly under spatial-only backpropagation. The authors derive the dynamics of RQFs and demonstrate their effectiveness through empirical evaluations, showing that RQFs with prospective inputs outperform their non-prospective counterparts across various configurations. The results indicate that RQFs are not only parameter-efficient but also enhance learning in deep CTRNs, making them suitable for tasks requiring long sequence processing.
Methodology
The authors develop RQFs as a linear dynamical system and derive their dynamics from an energy-based model of cortical function. They apply a prospective coding mechanism to each layer's bottom-up input using a parameter-free two-tap update, which preserves the recurrent transition dynamics. The performance of RQFs is evaluated through full backpropagation through time (BPTT) and spatial-only backpropagation across various tasks.
Results
The study finds that RQFs with prospective inputs achieve significant accuracy improvements, with a six-layer RQF reaching 96.09% accuracy on the Speech Commands dataset and 83.56% on the Path-X task. These results demonstrate the effectiveness of the proposed architecture and coding strategy in enhancing learning in deep CTRNs.
Implications
The findings suggest that RQFs and prospective coding could lead to more efficient training of deep recurrent networks, particularly in applications involving long sequences, such as audio processing and time series analysis. This work may influence future research in neural network architectures and learning algorithms.
OSR: Output Space Redistribution for Adaptive Label Removal in Classification Models
Efficient ML
- OSR provides a training-free solution for label removal in classification models.
- The method operates directly on model outputs, avoiding the need for retraining or access to original training data.
- A two-step filtering process ensures consistency and preserves model utility post-removal.
- OSR demonstrates competitive performance against traditional retraining methods across multiple datasets.
Read more
OSR: Output Space Redistribution for Adaptive Label Removal in Classification Models
Summary
The paper addresses the challenge of label removal in classification systems, particularly in contexts where categories must be dynamically updated or eliminated due to evolving taxonomies. Traditional methods for label removal, which include retraining-based and feature-space-adjustment approaches, often require access to original data, incur high computational costs, and suffer from scalability issues. The authors propose a novel method called Output Space Redistribution (OSR), which operates directly in the output space of classification models. This approach allows for adaptive label removal without the need for retraining or modifications to model parameters. OSR utilizes a two-step filtering process that projects original output confidence vectors to remove the influence of deprecated labels and redistributes the remaining confidences to maintain model utility. The method is designed to be model-agnostic and requires only existing labels and prior output confidences, thus addressing privacy concerns associated with data-dependent solutions. Experimental results demonstrate that OSR achieves competitive performance compared to full retraining while significantly improving computational efficiency and privacy preservation across various classification tasks.
Methodology
The OSR method employs a two-step filtering process: first, it projects the original output confidence vectors to eliminate the impact of deprecated labels, and second, it redistributes the remaining confidence values among the active classes. This approach allows for efficient and scalable label removal without modifying the underlying model architecture or requiring retraining.
Results
Extensive experiments conducted across four datasets show that OSR outperforms traditional retraining methods in terms of computational efficiency while maintaining competitive accuracy. The results indicate that OSR effectively adapts classification models to label removal conditions, demonstrating its scalability and consistency.
Implications
The OSR method has significant implications for real-world applications where classification systems must frequently adapt to changing taxonomies, such as in e-commerce and financial industries. Its ability to preserve model utility while enhancing privacy makes it a valuable tool for developing adaptive machine learning systems.
LeanGRPO: Eliminating Redundant Recomputation in Diffusion RL
Reinforcement Learning
Generative Models
Efficient ML
- Identifies update-stage recomputation as a major bottleneck in trajectory-logprob diffusion RL.
- Introduces LeanGRPO, a framework that eliminates redundant recomputation in diffusion RL.
- Presents two complementary training schedules: LeanGRPO-Retain and LeanGRPO-Reweight.
- Achieves up to 1.83Γ speedup in training while preserving optimization objectives.
Read more
LeanGRPO: Eliminating Redundant Recomputation in Diffusion RL
Summary
LeanGRPO addresses the inefficiencies in diffusion reinforcement learning (RL) methods, specifically the redundant recomputation during the update stage after rollout. Traditional methods like DanceGRPO and FlowGRPO recompute selected timesteps with gradient tracking, which is unnecessary when the policy remains unchanged. LeanGRPO introduces a framework that restructures the data-parallel layout to allow for the reuse of computation graphs and saved activations, thereby eliminating this redundancy. The framework includes two innovative training schedules: LeanGRPO-Retain, which retains computation graphs for backward propagation, and LeanGRPO-Reweight, which uses provisional gradients and delays synchronization until after the trajectory is completed. These methods significantly reduce memory overhead while achieving up to 1.83Γ speedup in training without compromising the optimization objectives. The paper demonstrates the effectiveness of LeanGRPO across various diffusion RL algorithms and model scales, providing insights into GPU memory profiling and practical deployment scenarios.
Methodology
LeanGRPO restructures the data-parallel layout used in conventional GRPO methods, allowing all GPUs to process the same prompt while generating different samples. It introduces two training schedules: LeanGRPO-Retain, which retains computation graphs for reuse during updates, and LeanGRPO-Reweight, which uses provisional gradients to manage memory consumption effectively.
Results
LeanGRPO demonstrated significant improvements in training efficiency, achieving up to 1.83Γ end-to-end speedup compared to traditional methods. The framework was integrated with various diffusion RL algorithms and model backbones, showcasing its versatility and effectiveness across different scenarios.
Implications
The LeanGRPO framework can enhance the efficiency of diffusion RL methods, making them more practical for real-world applications in generative modeling. Its ability to reduce memory overhead while maintaining performance could lead to broader adoption of diffusion RL techniques in various domains.
RobustSeiz: An Open-Source Framework for Benchmarking the Robustness of EEG Seizure Detection Models
Time Series
- RobustSeiz is an open-source framework for benchmarking EEG seizure detection models.
- The framework standardizes EEG datasets and simulates real-world challenges to assess model robustness.
- It reports multiple performance metrics, including sensitivity, precision, and predictive agreement.
- The study demonstrates that held-out accuracy does not guarantee deployment readiness.
Read more
RobustSeiz: An Open-Source Framework for Benchmarking the Robustness of EEG Seizure Detection Models
Summary
The paper introduces RobustSeiz, an open-source framework designed to benchmark the robustness of EEG seizure detection models against real-world challenges such as acquisition variability, artifacts, and adversarial inputs. The authors highlight that while seizure detectors may perform well on clean datasets, they often fail in practical scenarios. RobustSeiz provides a standardized and reproducible protocol for stress-testing these models before deployment. The framework standardizes four public scalp-EEG datasets into Brain Imaging Data Structure (BIDS-EEG) format, allowing for the evaluation of subject-independent detectors. It incorporates various environmental, noise, and adversarial transformations to simulate clinical variability. The framework reports multiple performance metrics, including sensitivity, precision, F1 score, false positives, and predictive agreement. The authors demonstrate RobustSeiz using a contemporary seizure detector on the TUSZ dataset, showcasing how perturbations affect detection quality and onset timing. The study emphasizes that accuracy alone is insufficient for deployment readiness, advocating for a comprehensive robustness assessment that reveals potential failure modes. Overall, RobustSeiz aims to enhance the reliability of seizure detection systems in clinical settings by providing a shared benchmarking standard.
Methodology
The authors standardized four public scalp-EEG datasets into BIDS-EEG format and evaluated subject-independent detectors using controlled perturbations that simulate clinical variability. They employed a Dockerized GPU pipeline for experiments, sweeping over predefined hyperparameter grids and reporting various performance metrics.
Results
RobustSeiz successfully demonstrated the robustness of a contemporary seizure detector on the TUSZ dataset, revealing how different perturbations affect detection quality and onset timing. The framework provided a comprehensive analysis of detection performance across various stressors, highlighting the importance of robustness in clinical applications.
Implications
The RobustSeiz framework has significant implications for the clinical deployment of EEG seizure detection models, ensuring that they are tested against realistic conditions before being used in patient care. It promotes a shift from accuracy-focused evaluations to more comprehensive assessments of model stability and reliability.
Tail-Likelihood Reinforcement Learning
Reinforcement Learning
Optimization
Generative Models
- TailRL optimizes the log-probability of exceeding reward thresholds, enhancing coverage of high-reward outcomes.
- The method is compatible with existing RL pipelines, requiring only a simple modification to the advantage function.
- TailRL shows significant improvements in various tasks, outperforming traditional expected-reward methods.
- The approach provides a robust framework for leveraging rare high-reward samples during training.
Read more
Tail-Likelihood Reinforcement Learning
Summary
The paper introduces Tail-Likelihood Reinforcement Learning (TailRL), a novel approach that optimizes the coverage of high-reward outcomes in reinforcement learning (RL) by focusing on the tail of the reward distribution rather than just the average reward. Traditional RL methods often overlook the importance of rare but high-reward rollouts, which can lead to suboptimal policy performance, especially as sampling increases during training and inference. TailRL addresses this by maximizing the log-probability of exceeding a randomly chosen reward threshold, effectively transforming continuous rewards into a series of binary success events. This method enhances the gradient calculation by placing greater emphasis on rare, high-reward rollouts, thereby improving the policy's performance across various tasks such as object localization, maze navigation, GUI grounding, and code optimization. The authors demonstrate that TailRL can be easily integrated into existing RL frameworks with minimal modifications, making it a practical solution for improving RL outcomes.
Methodology
TailRL reformulates the reinforcement learning objective by maximizing the expected log-likelihood of exceeding a uniformly drawn reward threshold. This approach transforms the continuous reward into a family of binary success events, allowing the gradient to decompose into a harmonic mixture of Best-of-k gradients. The method requires a simple adjustment to the advantage function, making it easy to implement in existing RL frameworks.
Results
The empirical results demonstrate that TailRL achieves performance comparable to supervised learning objectives in object localization, significantly improves success rates in maze navigation, matches the performance of existing methods like RLOO with fewer inference rollouts in GUI grounding, and provides a substantial speedup in code optimization tasks.
Implications
TailRL has the potential to enhance reinforcement learning applications by ensuring that policies maintain coverage of high-reward outcomes, which is crucial for tasks where rare but valuable results are important. This could lead to more efficient training and better performance in real-world applications where high-reward scenarios are infrequent.
TRACE: Spatiotemporal Contact Memory Graph Network Simulator for Granular Dynamics
Graph Learning
- TRACE introduces a novel approach to preserving inter-granular contact history using edge-based memory.
- The model achieves significant reductions in long-rollout position error and final-deposit error compared to existing simulators.
- TRACE demonstrates impressive computational efficiency, achieving speedups of 12.2Γ and 8.9Γ over traditional methods in 2D and 3D, respectively.
- The architecture leverages attention-based message passing and GRUs for effective memory management.
Read more
TRACE: Spatiotemporal Contact Memory Graph Network Simulator for Granular Dynamics
Summary
The paper introduces TRACE, a novel graph-network simulator designed for simulating granular dynamics by efficiently preserving inter-granular contact history. Traditional simulators struggle with the dynamic nature of particle interactions, particularly when contacts form, break, and rearrange. TRACE addresses this by storing interaction history directly on contact edges, utilizing an attention-based message passing mechanism and a gated recurrent unit (GRU) to maintain persistent memory. The model features a physics-structured decoder that predicts contact forces while adhering to physical constraints such as Coulomb friction. TRACE is trained using a two-step approach involving single-step pretraining followed by autoregressive rollout fine-tuning. Evaluations on 2D and 3D granular column-collapse benchmarks demonstrate that TRACE achieves stable and physically consistent long-horizon rollouts, accurately reproducing final deposit geometries and kinetic energy during collapses. Compared to existing models, TRACE significantly reduces position and final-deposit errors while using fewer parameters and maintaining near-zero particle interpenetration. Additionally, TRACE offers substantial speed improvements over traditional methods, making it a promising tool for simulating granular materials in various applications.
Methodology
TRACE employs a graph neural network architecture where particles are represented as nodes and their contacts as edges. It maintains a persistent memory on edges updated through attention-based message passing and GRUs. A physics-structured decoder predicts contact forces while enforcing physical constraints. The training process involves single-step pretraining followed by autoregressive fine-tuning.
Results
TRACE outperforms existing graph network simulators by reducing long-rollout position error by 31β62% and final-deposit error by 58β89% across 2D and 3D benchmarks. It also achieves significant computational speedups, with 12.2Γ and 8.9Γ faster simulations compared to the material point method in 2D and 3D, respectively.
Implications
The development of TRACE has significant implications for the simulation of granular materials in engineering and scientific applications, potentially enabling faster and more accurate modeling of complex phenomena such as landslides, debris flows, and industrial powder handling.
Learnable composition for neural operators
Theory
Efficient ML
Optimization
- Introduction of LATENTDDM, a framework that combines local operator pretraining with a learnable composition module.
- Demonstrated significant error reduction in physical simulations under varying conditions compared to traditional models.
- Evaluation on two complex problems: Darcy flow and pitching airfoil flow, showcasing the framework's adaptability.
- The approach minimizes the need for costly high-fidelity simulations during adaptation.
Read more
Learnable composition for neural operators
Summary
The paper introduces LATENTDDM, a novel framework for neural operators that aims to enhance the adaptability and efficiency of physical simulations under varying conditions. Neural operators serve as fast, differentiable surrogates for physical simulations but often struggle with accuracy when faced with changes in domain geometry, size, or operating conditions. The authors propose a two-step approach: first, pretraining a neural operator to predict fields on small subdomains, and second, freezing this operator while training a lightweight composition module that integrates local predictions for new settings. This method is evaluated on two challenging problems: steady Darcy flow and unsteady incompressible flow around a pitching airfoil. The results demonstrate that LATENTDDM significantly reduces error rates compared to traditional models that process the entire domain at once, achieving 36-56% lower errors in larger Darcy domains and improved performance in fast-pitching airfoil scenarios. The findings suggest that co-designing local pretraining and composition-level transfer can be a promising strategy for developing robust physical foundation models.
Methodology
The methodology involves pretraining a neural operator on small subdomains to predict physical fields, followed by freezing this operator and training a composition module that integrates local predictions for new settings. The framework is tested on two physical problems, comparing its performance against models that process the full domain and a neural domain decomposition method with fixed composition rules.
Results
LATENTDDM achieved 36-56% lower error rates in larger Darcy domains after adaptation with only 16 target simulations. It also improved 20-step field rollouts in fast-pitching airfoil flow, both in zero-shot scenarios and after few-shot calibration, outperforming capacity-matched models and traditional neural DDMs.
Implications
The findings suggest that LATENTDDM can significantly reduce the computational costs associated with adapting neural operators to new physical settings, making it a valuable tool for engineers and researchers in fields requiring accurate physical simulations under varying conditions.
The Gradient Does Not See Rank: Rank-Indifference in Matrix-CODI on ProsQA
NLP
Large Language Models
Interpretability
- Rank-k ablation curves remain flat across multiple training conditions, indicating rank-blindness in matrix-CODI.
- The effective rank of the latent matrix does not correlate with model accuracy, challenging the assumption that rank reflects reasoning paths.
- Various nonlinear readouts also show flat rank-k curves, suggesting that rank is not a functional readout in this context.
- A linear probe on the latent matrix underperforms compared to a pretrained hidden state, further supporting the rank-blindness hypothesis.
Read more
The Gradient Does Not See Rank: Rank-Indifference in Matrix-CODI on ProsQA
Summary
This paper investigates the role of rank in matrix-valued continuous chain-of-thought (CoT) models, specifically focusing on a variant called matrix-CODI. The authors argue that while rank is expected to reflect the number of reasoning paths in the latent matrix representation, their experiments reveal that the accuracy of the model remains largely unchanged when truncating the matrix to lower ranks. This suggests that the model's performance is not sensitive to the rank of the latent matrix, indicating a phenomenon termed 'rank-blindness.' The study includes various training regimes and readout configurations, all yielding flat rank-k ablation curves, which challenge the assumption that rank serves as a functional measure of reasoning capacity in these models. The findings imply that the structural capacity offered by rank may not be effectively utilized in matrix-CODI, raising questions about the interpretability and design of such models.
Methodology
The authors employed a matrix-CODI model with a matrix bottleneck in the latent feedback path. They conducted experiments across different training regimes and readout configurations, measuring the impact of rank truncation on model accuracy through rank-k ablation studies. They also compared performance with a linear probe and a control model (vanilla GPT-2) to assess rank-blindness.
Results
The experiments revealed that the rank-k ablation curves were flat, with accuracy changes within 0.6 percentage points across various configurations. The effective ranks observed were 4, 12, and 13, with an overall accuracy of 81.0 Β± 2.0 percentage points. Additionally, the linear probe on the latent matrix achieved an AUC of 0.673, significantly lower than the 0.846 achieved by the pretrained hidden state.
Implications
The findings suggest that the design of continuous CoT models may need to reconsider the role of rank in latent representations, as it does not appear to serve as a reliable indicator of reasoning capacity. This could influence future research on model interpretability and the development of more effective training strategies.