AI-generated summaries
Today's ML research,
without the noise.
Daily summaries of the latest machine learning papers from arXiv, processed every 8 hours.
53
Papers today
8h
Update frequency
7
Days of history
Grounding Investor Views: Neural Predicates in the Black-Litterman Model
Optimization
Theory
Interpretability
- Introduces neural predicates as a structured mechanism for generating investor views in the Black-Litterman model.
- Establishes a formal mapping from neural predicate outputs to Black-Litterman components (P, q, Ω).
- Replaces subjective uncertainty elicitation with a data-driven, entropy-based measure of uncertainty.
- Enhances interpretability by allowing portfolio weights to be traced back to the underlying data.
Read more
Grounding Investor Views: Neural Predicates in the Black-Litterman Model
Summary
This paper addresses the challenges of portfolio construction under the Black-Litterman model, which requires investors to specify views on asset returns along with uncertainty estimates. The author proposes a novel approach using neural predicates as a structured, probabilistic mechanism for generating these views. By processing structured financial analysis data through a hierarchy of neural predicates, the model outputs probability distributions over market stances, which are then mapped to the Black-Litterman components: the pick matrix (P), view return vector (q), and view uncertainty matrix (Ω). This method provides a data-driven alternative to subjective uncertainty elicitation, enhancing interpretability and enabling end-to-end learning. The paper makes three significant contributions: (i) it establishes a formal mapping from neural predicate outputs to the Black-Litterman components; (ii) it introduces an entropy-based measure of uncertainty that replaces fixed covariance heuristics; and (iii) it presents a compositional method for integrating multiple predicates into a coherent asset view. Overall, the proposed framework aims to improve the reproducibility and scalability of investor views in portfolio optimization.
Methodology
The methodology involves processing structured financial data through a compositional hierarchy of neural predicates, which generate probability distributions over market stances. These outputs are then systematically mapped to the components of the Black-Litterman model, allowing for a structured and interpretable integration of investor views.
Results
The proposed approach demonstrates improved stability and interpretability in portfolio construction compared to traditional methods. It effectively replaces subjective uncertainty with a data-driven framework, leading to more coherent and reproducible investor views.
Implications
The findings suggest that integrating neural predicates into portfolio optimization can enhance decision-making under uncertainty, making it easier for investors to generate and justify their views. This could lead to more robust investment strategies and improved performance in financial markets.
Graph Learning on Ensembles of Cyclic Peptides: An Investigation of Molecular Ensemble Modeling
Graph Learning
- EnsembleEGNN encodes multiple conformers into a single embedding, capturing molecular flexibility.
- Pretraining on a multi-task self-supervised objective significantly enhances model performance.
- The model outperforms traditional sequence-only approaches in predicting molecular properties.
- Co-training with a BERT sequence encoder further improves predictive accuracy.
Read more
Graph Learning on Ensembles of Cyclic Peptides: An Investigation of Molecular Ensemble Modeling
Summary
This paper introduces EnsembleEGNN, a novel molecular ensemble foundation model designed to improve molecular property prediction by encoding conformational ensembles of cyclic peptides. Traditional methods often rely on a single conformation, neglecting the inherent flexibility and thermodynamic context of molecules. EnsembleEGNN addresses this by utilizing Equivariant Graph Neural Network (EGNN) layers to encode each conformer and then pooling these representations with a Set Attention Block (SAB). The model is pretrained on the CREMP dataset using a multi-task self-supervised learning approach that includes masked token recovery, noisy-coordinate reconstruction, and pairwise distance reconstruction. The results demonstrate that while training from scratch yields poor performance (R2 = 0.005), the pretrained EnsembleEGNN achieves significant improvements (R2 = 0.477, Pearson r = 0.699), outperforming a sequence-only BERT baseline. Further co-training with BERT enhances performance to R2 = 0.538 and Pearson r = 0.737, highlighting the effectiveness of conformational ensemble modeling in predicting cyclic-peptide properties.
Methodology
The methodology involves encoding each conformer of cyclic peptides using shared EGNN layers, followed by pooling the representations with a Set Attention Block. The model is pretrained on the CREMP dataset with a multi-task objective that includes various reconstruction tasks, allowing it to learn from the thermodynamic distribution of conformers.
Results
The pretrained EnsembleEGNN model achieved an R2 score of 0.477 and a Pearson correlation of 0.699 on the CREMP-CycPeptMPDB dataset. When co-trained with a BERT sequence encoder, the performance improved to an R2 score of 0.538 and a Pearson correlation of 0.737, demonstrating the effectiveness of the ensemble modeling approach.
Implications
The findings suggest that incorporating conformational ensembles into molecular property prediction models can significantly enhance their accuracy. This has potential applications in drug discovery and molecular design, where understanding the flexibility and dynamics of molecules is crucial.
Multi-Task Learning for Heterogeneous Prediction from Video Game State with Transfer Learning
Multimodal
- MTL can improve performance on video game state predictions compared to single-task learning.
- Effective weighting strategies are crucial for balancing mixed losses and conflicting gradients in MTL.
- Source-task pre-training enhances target-task fine-tuning, especially in limited data scenarios.
- Cross-map transfer demonstrates the utility of shared representations in varying game environments.
Read more
Multi-Task Learning for Heterogeneous Prediction from Video Game State with Transfer Learning
Summary
This paper investigates the application of multi-task learning (MTL) for predicting various outcomes from video game state data, specifically in team-based multiplayer games. The authors propose a shared model that leverages multiple related supervision signals derived from the same structured observations, aiming to enhance generalization and reduce both training and inference costs compared to single-task models. They adapt a multimodal architecture designed for endpoint prediction to accommodate a broader multi-task setting, integrating rasterized vision inputs, global match context, and per-unit state information through an image encoder and attention-based interaction modeling. The study employs a large proprietary dataset from the game World of Tanks to compare the performance of single-task versus multi-task training, evaluate different weighting strategies for mixed losses, and assess the effectiveness of pre-training and fine-tuning under limited target-data conditions. Additionally, the authors explore the potential for cross-map transfer within the same game, highlighting the practical relevance of their findings in scenarios where data from new or unreleased maps is scarce.
Methodology
The authors utilized a hard parameter sharing approach in MTL, employing a multimodal model framework that integrates various input types from video game telemetry. They conducted experiments to analyze the performance of single-task versus multi-task training, evaluated different loss weighting strategies, and tested the impact of pre-training on fine-tuning across different target-data regimes. The study also examined the effects of cross-map transfer within the same game.
Results
The experiments revealed that multi-task learning significantly outperformed single-task learning in terms of prediction accuracy. The findings indicated that appropriate weighting strategies for mixed losses improved model performance and that pre-training on source tasks effectively facilitated fine-tuning on target tasks, particularly when data availability was limited. Furthermore, the shared representations learned through MTL proved beneficial for predictions across different game maps.
Implications
The results suggest that MTL can be a powerful approach for developing AI systems in video games, enabling more efficient training and improved predictive capabilities. This methodology could be applied to other domains where related tasks can benefit from shared representations, potentially leading to advancements in game AI and analytics.
Codec-Gauge: Learning Compression-Friendly Gauges for Transformer KV Caches
NLP
Large Language Models
Efficient ML
- Codec-Gauge learns compression-friendly gauges for KV caches in Transformers.
- The method reduces KL divergence by an average of 44.0% compared to raw coordinates.
- It improves quality preservation for quantization methods like block-uniform and KIVI-style.
- The approach maintains model weights and semantics while optimizing cache representation.
Read more
Codec-Gauge: Learning Compression-Friendly Gauges for Transformer KV Caches
Summary
The paper introduces Codec-Gauge, a novel post-training method designed to enhance the compression of key/value (KV) caches in long-context Transformer models. The authors argue that the coordinate basis of KV vectors significantly influences the fidelity of model behavior during compression. Codec-Gauge learns small orthogonal transforms, or gauges, that optimize the representation of KV tensors for compression. The method employs a frequency-distribution objective that combines a DCT spectral-centroid loss with a smooth rate proxy, effectively concentrating KV energy in low-frequency layouts. The authors evaluate Codec-Gauge across six models and various bit rates, demonstrating an average reduction of 44.0% in KL divergence compared to raw coordinates. The learned gauges also improve quality preservation during quantization processes. The study emphasizes the importance of geometry in compression, showing that the learned gauge can significantly enhance the performance of existing compression backends without altering model weights or semantics.
Methodology
The authors developed Codec-Gauge as a post-training layer that learns orthogonal transforms from frozen-model KV tensors. The learning process focuses on optimizing the coordinate basis presented to compression backends, using a frequency-distribution loss that emphasizes low-frequency energy concentration. The evaluation involved a matched-rate experimental design, comparing the performance of learned gauges against various control methods under fixed backend conditions.
Results
Codec-Gauge achieved a significant reduction in KL divergence (44.0% on average) across multiple models and bit rates. The learned gauges outperformed random, Hadamard, DCT, and PCA/KLT controls, demonstrating improved quality retention during compression and quantization processes. The results were validated through actual compression/decompression measurements and rolling compressed-history scoring.
Implications
The findings suggest that optimizing the coordinate geometry of KV caches can lead to more efficient compression methods for Transformers, potentially reducing memory and bandwidth costs in long-context inference. This could have broad applications in enhancing the performance of large language models and other Transformer-based architectures.
Three-Pronged Spectral Control for Federated Parameter Efficient Fine Tuning
Federated Learning
Optimization
Efficient ML
- TRISHUL addresses spectral misalignment in federated PEFT, enhancing aggregation stability.
- The framework employs nuclear-norm proximal shrinkage to suppress unstable client-specific updates.
- Non-uniform adaptation head allocation improves resource utilization across model layers.
- TRISHUL maintains low communication overhead while improving model performance.
Read more
Three-Pronged Spectral Control for Federated Parameter Efficient Fine Tuning
Summary
The paper addresses the challenges of federated parameter-efficient fine-tuning (PEFT) in the presence of non-IID client heterogeneity, which can lead to high-variance aggregation and poor global model performance. The authors propose TRISHUL, a spectral-control framework designed to enhance the robustness of federated PEFT. TRISHUL utilizes shared frozen multi-head low-rank bases for exact aggregation of updates, applies nuclear-norm proximal shrinkage to suppress client-specific high-rank spectral components, and employs a concave water-filling budget rule for non-uniform allocation of adaptation heads across layers. This approach aims to improve the geometric compatibility of local updates during aggregation while maintaining communication efficiency. The authors validate TRISHUL across various benchmarks, demonstrating improved convergence, stability, and performance compared to federated LoRA baselines, particularly under conditions of strong heterogeneity.
Methodology
TRISHUL combines three main strategies: (1) shared frozen multi-head bases for exact aggregation of low-rank updates, (2) nuclear-norm proximal shrinkage to suppress high-rank spectral components before aggregation, and (3) a concave water-filling rule for non-uniform allocation of adaptation budgets across layers. The framework focuses on controlling the geometry of parameter updates to mitigate the effects of client heterogeneity.
Results
The implementation of TRISHUL across various datasets, including CIFAR-100, SVHN, 20 Newsgroups, MRQA, and GLUE with LLaMA3.2-1B, showed significant improvements in convergence rates, stability, and final performance metrics compared to traditional federated LoRA methods. The results indicate that TRISHUL is particularly effective in scenarios with high client heterogeneity.
Implications
The findings suggest that TRISHUL can be applied in real-world federated learning scenarios where data privacy and communication efficiency are critical. The framework's ability to enhance model performance while controlling for client heterogeneity makes it suitable for applications in mobile devices, IoT, and other decentralized environments.
Mean-to-Score Discrete Diffusion: Posterior-Mean Denoisers for Score Entropy
Generative Models
Theory
- Introduces Mean-to-Score (M2S) to enforce Bayes realizability in discrete diffusion models.
- Identifies and quantifies violations of Bayes realizability in existing SEDD models.
- Demonstrates that M2S significantly improves generative performance metrics.
- Provides a method for projecting scores to ensure nonnegative weights during sampling.
Read more
Mean-to-Score Discrete Diffusion: Posterior-Mean Denoisers for Score Entropy
Summary
This paper introduces the Mean-to-Score Discrete Diffusion (M2S) method, which addresses the limitations of Score Entropy Discrete Diffusion (SEDD) in ensuring Bayes realizability during the generation of discrete data. SEDD's approach, while ensuring nonnegative reverse jump rates, fails to guarantee that all candidate score ratios arise from a single clean-token posterior, leading to violations in Bayes realizability. The authors demonstrate that a significant portion of score vectors in a trained SEDD model do not satisfy this condition, which can result in negative weights during sampling. To rectify this, M2S predicts a clean-token posterior mean and transforms it into the score using a linear map, thus ensuring Bayes realizability by construction. The paper presents empirical results showing that M2S outperforms existing models, achieving lower test BPD and FID scores on CIFAR-10 and significantly improving generative performance on OpenWebText, demonstrating its effectiveness in discrete diffusion modeling.
Methodology
The authors propose the Mean-to-Score (M2S) framework, which involves predicting a clean-token posterior mean and converting it to the score through a linear map. This method is designed to ensure Bayes realizability by construction, as opposed to relying on post-hoc adjustments. The framework is validated through experiments on CIFAR-10 and OpenWebText datasets, comparing performance against existing models.
Results
The M2S model, with 170M parameters, achieved a generative perplexity (PPL) of 143.3 at 128 sampling steps, outperforming the best pure-uniform SEDD baseline which had a PPL of 183.6. Additionally, M2S reduced test BPD from 3.173 to 3.129 and FID-50k from 42.83 to 28.09 on CIFAR-10, indicating substantial improvements in generative quality.
Implications
The findings suggest that enforcing Bayes realizability in generative models can lead to significant improvements in performance, particularly in discrete data generation tasks. This approach may have broader applications in various generative modeling scenarios, enhancing the reliability and quality of generated outputs.
When RLVR Shrinks the Reasoning Boundary: Diagnosing Pass@k Inversion
Reinforcement Learning
Theory
Optimization
- RLVR can improve pass@1 accuracy while decreasing pass@k performance, leading to fewer distinct problems solved.
- The pass@k inversion is attributed to a boundary mode-commitment failure where rare correct trajectories are not reinforced adequately.
- Per-Problem Base Anchoring (PBA) is introduced as a method to preserve rare correct trajectories and improve model performance.
- Experimental results show that PBA enhances both pass@1 and high-budget coverage compared to matched controls.
Read more
When RLVR Shrinks the Reasoning Boundary: Diagnosing Pass@k Inversion
Summary
This paper investigates the phenomenon of pass@k inversion in reinforcement learning with verifiable rewards (RLVR), where models exhibit improved one-sample accuracy but perform worse under repeated sampling. The author identifies that after training, the RLVR policy may solve fewer distinct problems than its base model, particularly for boundary prompts that contain rare correct trajectories. The study introduces a diagnostic and mechanistic framework to understand this failure, attributing it to an absence-of-evidence failure where rare correct trajectories are lost due to insufficient reinforcement during training. The paper proposes a method called Per-Problem Base Anchoring (PBA) to mitigate this issue by sharpening prompts with frozen base correct evidence and anchoring risky prompts to the base distribution. Experimental results demonstrate that PBA improves both pass@1 and high-budget coverage across multiple training seeds on the Omni-MATH-Test benchmark, indicating that it effectively preserves rare verifier-positive trajectories. The findings suggest that careful reasoning post-training is crucial for optimizing models without losing coverage of rare solutions, which is particularly relevant for applications involving visual, spatial, or chart reasoning.
Methodology
The paper employs a diagnostic framing to categorize prompts based on their solvability and introduces a two-mode mechanistic model to explain the pass@k inversion phenomenon. The Per-Problem Base Anchoring (PBA) method is tested across three training seeds using a controlled study on the Omni-MATH-Test benchmark, evaluating its effectiveness through full pass@k curves and matched controls.
Results
The implementation of PBA resulted in significant improvements in both pass@1 and high-budget coverage metrics across multiple training seeds. The diagnostic study confirmed that ordinary RLVR methods tend to lose coverage of base-solvable boundary prompts, while PBA successfully preserved rare verifier-positive trajectories.
Implications
The findings suggest that optimizing reasoning models requires careful consideration of which prompts to reinforce to avoid losing coverage of rare solutions. This has broader implications for the development of reliable verifier-guided reasoning systems in various applications, including visual and spatial reasoning tasks.
Gradient Concentration, Not Weight Saliency, Explains Representation-Level Class Unlearning
Theory
Optimization
Efficient ML
- Saliency-based masking does not significantly enhance representation-level class unlearning compared to random masking.
- Forget gradients are predominantly concentrated in the final layers of the network, influencing all masking strategies similarly.
- Saliency masks show limited class specificity, selecting overlapping parameter subsets across different classes.
- Effective representation-level unlearning may necessitate objectives that act directly on latent representations.
Read more
Gradient Concentration, Not Weight Saliency, Explains Representation-Level Class Unlearning
Summary
This paper investigates the mechanisms behind representation-level class unlearning in machine learning, specifically challenging the effectiveness of saliency-based weight selection methods. The authors conduct a controlled ablation study using CIFAR-10 and CIFAR-100 datasets with a ResNet-18 architecture. They compare the traditional saliency-based masking approach with random masks of equal sparsity and unconstrained updates, while keeping other variables constant. The findings reveal that all three methods yield statistically equivalent representation-level recoverability, suggesting that the concentration of forget gradients in the final layers of the network is the primary driver of effective unlearning, rather than the specific identity of the weights selected by saliency masks. Additionally, the study highlights that saliency masks exhibit weak class specificity, indicating that effective unlearning may require objectives that directly target representations rather than relying solely on sophisticated weight-selection strategies.
Methodology
The authors performed a matched-compute ablation study comparing three conditions: (1) saliency-based masking, (2) random masking with identical sparsity, and (3) unconstrained fine-tuning. They maintained a consistent unlearning objective, optimization schedule, and computational budget across all experiments to isolate the effects of the masking strategies.
Results
The study found that saliency-based masking, random masking, and unconstrained updates produced statistically equivalent representation-level recoverability on both CIFAR-10 and CIFAR-100 datasets. The concentration of gradient energy in the final layers of the network was identified as the key factor influencing the effectiveness of unlearning, rather than the specific weights selected by saliency masks.
Implications
These findings suggest a shift in focus for future research on machine unlearning, advocating for the development of methods that directly manipulate representations rather than relying on complex weight-selection strategies. This could lead to more efficient and effective approaches for data deletion in compliance with privacy regulations.
ReliableTableQA: How Much Supervision Does Reliability Annotation Need?
NLP
Large Language Models
Efficient ML
- Introduction of a ten-category reliability taxonomy for statistical hazards in tabular QA.
- Development of a data generation pipeline producing 50,000 labeled examples.
- Demonstration that a small number of schema-stratified training examples can significantly enhance model performance.
- Findings suggest GRPO is only beneficial when the model is under-trained.
Read more
ReliableTableQA: How Much Supervision Does Reliability Annotation Need?
Summary
The paper presents ReliableTableQA, a framework designed to train large language models (LLMs) to assess the statistical reliability of answers generated from tabular data queries. Unlike traditional systems that focus solely on whether a query is answerable, ReliableTableQA emphasizes the importance of the statistical validity of the answers produced. The authors identify a critical gap in existing systems, which often provide confident but unreliable answers, and introduce the Unreliable Confident Answer Rate (UCAR) as a metric to quantify this issue. The framework includes a ten-category reliability taxonomy that identifies various hazards affecting statistical reliability, such as small sample sizes and multiple comparison issues. The authors developed a program-first data pipeline that generates 50,000 reliability-labeled training examples using a context-free grammar approach. They conducted a controlled study to determine the amount of supervision needed for effective reliability annotation, finding that a small, schema-stratified set of 200 examples significantly improved the model's performance. The results indicate that while reinforcement learning from policy optimization (GRPO) can be beneficial in under-trained scenarios, it does not provide additional advantages once the model is adequately trained. This research reframes reliability annotation as a data-efficiency problem and clarifies the conditions under which reinforcement fine-tuning is beneficial.
Methodology
The methodology involves a program-first data generation approach using a context-free grammar to create diverse SQL queries over real retail schemas. A deterministic profiler assigns reliability labels based on the generated queries. The model is trained using supervised fine-tuning (SFT) on a small, schema-stratified dataset followed by reinforcement learning from policy optimization (GRPO). The performance is evaluated using metrics such as reliability-flag F1 and UCAR.
Results
The ReliableTableQA framework achieved a UCAR of 0.000 and a reliability-flag F1 score of approximately 0.98 across various schemas, with a Rel-F1 score of 0.997 on an unseen retail domain (H&M). The study showed that 200 examples were sufficient to raise the reliability-flag F1 from 0.61 to 0.98, and GRPO only provided measurable benefits when the SFT base was under-trained.
Implications
The findings have significant implications for enterprise data analytics, suggesting that models can be trained to provide more reliable statistical assessments of query results, thereby reducing the risk of acting on misleading data. This could enhance decision-making processes in various business applications that rely on tabular data.
Test-Time Scaling via Error Localization
Large Language Models
NLP
Efficient ML
- TTEL introduces token-level error localization to improve inference-time efficiency.
- The algorithm maximizes the reuse of valid reasoning prefixes by truncating and branching only the faulty suffix of a trajectory.
- TTEL achieves significantly higher token efficiency compared to traditional sampling methods.
- Empirical results show TTEL's superiority on both coding and mathematical reasoning benchmarks.
Read more
Test-Time Scaling via Error Localization
Summary
This paper introduces Test-Time Scaling via Error Localization (TTEL), an innovative inference-time algorithm designed to enhance the performance of large language models (LLMs) on complex reasoning and programming tasks. Traditional methods, such as independent sampling and sequential multi-turn refinement, often lack token-level credit assignment, leading to inefficiencies as valid reasoning prefixes are discarded. TTEL addresses this by utilizing fixed or environmental feedback to perform token-level error localization, allowing the model to identify the exact point of failure in its reasoning. By comparing conditional probabilities under informed feedback against a null-context baseline, TTEL can truncate the reasoning trajectory and branch a new generation from the identified error point, maximizing the reuse of valid prefixes. The authors demonstrate that TTEL significantly improves efficiency, establishing dominating Pareto frontiers across various sequential reasoning domains. For instance, with the Qwen3-8B model on LiveCodeBench, TTEL achieves a pass@64 of 71.0% while generating approximately half as many tokens compared to independent sampling. The algorithm also outperforms competing test-time baselines on math benchmarks AIME-2025 and HMMT-2025, showcasing its effectiveness across different tasks.
Methodology
TTEL employs a feedback-guided tree search approach that utilizes environmental feedback to localize errors during generation. It compares token probabilities before and after receiving feedback to identify positions of disagreement, allowing the model to truncate the reasoning trace at the highest-confidence error and branch a new generation from that point.
Results
TTEL establishes strictly dominating Pareto frontiers in sequential reasoning tasks, achieving a pass@64 of 71.0% on LiveCodeBench with the Qwen3-8B model while generating approximately 360.4k tokens, compared to 735.0k tokens with independent sampling. It also outperforms other test-time baselines on math benchmarks AIME-2025 and HMMT-2025.
Implications
The findings suggest that LLMs can effectively utilize feedback for error correction during inference, leading to more efficient reasoning processes. This could have significant applications in areas requiring complex reasoning, such as programming assistance and advanced mathematical problem-solving.
M$^3$-Gen: Interpretable Multimodal Generation of Gene Expression Profiles Using Clinical and Imaging Data
Generative Models
Multimodal
Interpretability
- M3-Gen integrates clinical metadata and histopathology images to generate gene expression profiles.
- The framework uses contrastive learning to create a unified latent representation of the multimodal data.
- An attention-based mechanism enhances interpretability by linking image regions to generated gene expressions.
- Evaluations show that M3-Gen produces realistic and biologically relevant gene expression profiles.
Read more
M$^3$-Gen: Interpretable Multimodal Generation of Gene Expression Profiles Using Clinical and Imaging Data
Summary
The paper introduces M3-Gen, a novel framework designed to generate gene expression profiles by integrating clinical metadata and histopathology images. The authors highlight the challenges in acquiring gene expression data due to high costs and privacy concerns, which limit its use in multimodal research. M3-Gen employs a Generative Adversarial Network (GAN) conditioned on both clinical and imaging data, learning a unified latent representation through contrastive learning. This approach allows the model to produce biologically coherent gene expression profiles while maintaining interpretability. The framework's attention-based mechanism enables the identification of specific regions in histopathology images that influence gene expression generation, thus providing insights into the underlying biological mechanisms. Evaluations on the TCGA dataset demonstrate that M3-Gen generates realistic and functionally meaningful gene expression data, showcasing its potential in augmenting scarce datasets and enhancing multimodal studies.
Methodology
M3-Gen consists of three main stages: preprocessing of histopathology images and clinical metadata, contrastive pretraining to align visual and textual representations, and a generative model using a Wasserstein GAN with Gradient Penalty (WGAN-GP) that synthesizes gene expression profiles based on multimodal embeddings.
Results
The evaluation on the TCGA dataset indicates that M3-Gen successfully generates gene expression profiles that are realistic and functionally meaningful, demonstrating its effectiveness in multimodal data integration.
Implications
M3-Gen has the potential to enhance biomedical research by providing a method to generate synthetic gene expression data, thereby addressing data scarcity and privacy concerns. Its interpretability features may also facilitate deeper insights into the relationships between clinical variables, imaging data, and gene expression.
ADABORD: a novel AdaBoost approach for ordinal classification
Theory
- ADABORD integrates ordinal information into the AdaBoost framework for improved performance in ordinal classification tasks.
- The use of an Ordinal Gini splitting criterion in decision trees enhances the base estimator's ability to handle ordinal data.
- The absolute ranked probability score (aRPS) is introduced as a novel error function that considers both ordering and distance between classes.
- Extensive experiments show that ADABORD outperforms seven state-of-the-art methods, particularly on complex datasets.
Read more
ADABORD: a novel AdaBoost approach for ordinal classification
Summary
The paper introduces ADABORD, a novel AdaBoost framework tailored for ordinal classification (OC) tasks, where class labels have a natural order. Traditional classification methods often overlook this ordinal structure, leading to suboptimal performance. ADABORD enhances the AdaBoost algorithm by integrating ordinal information into two critical components: the base estimator and the error function. Specifically, it employs decision trees with an Ordinal Gini splitting criterion and utilizes an absolute ranked probability score (aRPS) to update sample weights, which accounts for both the ordering and distance between classes. The authors conducted extensive experiments comparing ADABORD against seven state-of-the-art OC methods using the TOC-UCO repository, the largest benchmark for OC. Results indicate that ADABORD significantly outperforms its competitors, especially in datasets with five or more classes, where the ordinal structure is more pronounced. The authors provide the source code and experimental protocols to promote reproducibility and further research in OC.
Methodology
ADABORD modifies the standard AdaBoost algorithm by incorporating ordinal-specific components: it uses decision trees with an Ordinal Gini splitting criterion for the base estimator and employs the absolute ranked probability score (aRPS) as the error function for updating sample weights. This approach allows for a more effective handling of ordinal relationships in the data.
Results
The experimental results demonstrate that ADABORD significantly outperforms seven existing OC methods, particularly excelling in datasets with five or more classes. Statistical analysis supports the superiority of ADABORD in leveraging ordinal information effectively.
Implications
The development of ADABORD has significant implications for various fields that require ordinal classification, such as healthcare, social sciences, and any domain where outcomes are naturally ordered. The framework can enhance predictive accuracy and decision-making processes in these areas.
SOAP, Muon, and Beyond: Pushing LLM Pretraining Scales
Large Language Models
Optimization
Efficient ML
- SOAP and Muon optimizers demonstrate superior performance over AdamW in large-scale LLM pretraining.
- Algorithmic modifications to SOAP enhance stability and eliminate loss spikes during training.
- A layer-wise distributed optimizer is introduced to improve scalability in large model training.
- The paper provides an open-source library for emerging optimizers to support the research community.
Read more
SOAP, Muon, and Beyond: Pushing LLM Pretraining Scales
Summary
This paper addresses the challenges of optimizing large-scale pretraining of language models using higher-order optimizers, specifically SOAP and Muon, which are shown to outperform the widely used AdamW optimizer. The authors identify instabilities in SOAP when applied at large batch sizes and propose algorithmic enhancements, including per-step QR orthogonalization and improved preconditioning strategies, to stabilize training. They conduct a comprehensive empirical study comparing SOAP, Muon, and AdamW, revealing that the former two maintain training stability and efficiency at batch sizes up to 100 million tokens, while AdamW's performance degrades. Additionally, the paper introduces a layer-wise distributed optimizer compatible with Megatron-LM, designed to optimize memory usage and communication overhead. The authors also release an open-source library for emerging optimization algorithms, fostering further research in this area.
Methodology
The authors adapt and enhance preconditioned gradient methods to address the challenges of large-scale LLM pretraining. They implement per-step QR orthogonalization and improved preconditioning strategies for SOAP, and conduct empirical evaluations using an update-RMS matching framework to compare the performance of SOAP, Muon, and AdamW across various batch sizes and model parameters.
Results
Experiments show that SOAP and Muon consistently outperform AdamW in terms of training stability and efficiency at large batch sizes, particularly up to 100 million tokens. The enhancements made to SOAP eliminate loss spikes, allowing for stable training in larger batch and model sizes. The layer-wise distributed optimizer implementation effectively balances memory usage and communication overhead, facilitating efficient training.
Implications
The findings suggest that higher-order optimizers like SOAP and Muon can significantly improve the efficiency and stability of training large language models, which is crucial for advancing the capabilities of AI systems. The open-source library provided can accelerate research and development in optimization techniques for deep learning.
CLOE: Christoffel Loss Autoencoder for Anomaly Detection
Efficient ML
Theory
Optimization
- CLOE combines autoencoders with Christoffel Function-based anomaly detection for improved performance.
- The method requires minimal hyperparameter tuning, needing only one hyperparameter to be set.
- CLOE is computationally lightweight and designed for high-dimensional data applications.
- The proposed loss function enhances representation learning in the latent space.
Read more
CLOE: Christoffel Loss Autoencoder for Anomaly Detection
Summary
The paper introduces CLOE, a novel approach for semi-supervised anomaly detection that combines an autoencoder for dimensionality reduction with a Christoffel Function-based anomaly detection method applied in the latent space. The authors address the challenges of high-dimensional data and the limitations of existing methods, particularly the need for extensive hyperparameter tuning. CLOE employs a unique loss function based on the Christoffel Function to guide the autoencoder's training, ensuring that the learned representations effectively capture the normal data distribution. The method is designed to be lightweight and computationally efficient, making it suitable for resource-constrained environments. The authors also propose a systematic procedure for setting the detection threshold and tuning the remaining hyperparameter. Experimental results demonstrate that CLOE outperforms several state-of-the-art anomaly detection methods across 15 high-dimensional tabular datasets, showcasing its effectiveness and efficiency.
Methodology
CLOE utilizes an autoencoder to reduce the dimensionality of high-dimensional data, while the latent space is regularized using the empirical Christoffel Function. A differentiable Christoffel Function-based loss is introduced to guide the training of the autoencoder, ensuring that the representations learned are optimal for anomaly detection. The method includes a systematic approach for threshold selection and hyperparameter tuning.
Results
CLOE was tested on 15 high-dimensional tabular datasets from the ADBench benchmark, demonstrating superior performance compared to various state-of-the-art anomaly detection techniques. The results indicate that CLOE effectively captures the support of normal data distributions, leading to robust outlier detection.
Implications
CLOE has significant implications for fields requiring anomaly detection, such as healthcare, finance, and industrial monitoring, particularly in scenarios involving high-dimensional data. Its lightweight nature makes it suitable for deployment in resource-constrained environments.
Multimodal CoLRAG-TF: Triple-Filtered Retrieval for Complex PDFs
Multimodal
NLP
Optimization
- Introduces a four-axis fusion architecture for multimodal retrieval.
- Achieves a high retrieval recall of 0.9909 on a benchmark dataset.
- Implements a hybrid OCR pipeline that improves table coverage significantly.
- Demonstrates a 71.6% improvement in multi-hop answer similarity.
Read more
Multimodal CoLRAG-TF: Triple-Filtered Retrieval for Complex PDFs
Summary
The paper presents Multimodal CoLRAG-TF, a novel architecture designed to enhance retrieval-augmented generation (RAG) over heterogeneous PDF collections, particularly focusing on Japanese disaster lesson documents. The proposed system addresses challenges such as multimodal content, domain-specific terminology, and multi-hop reasoning by integrating a four-axis fusion architecture that combines dense text embeddings, BM25 keyword matching, knowledge-graph triple filtering, and image-based similarity. The architecture constructs a multimodal index from 43 disaster lesson PDFs, utilizing a hybrid OCR pipeline and LLM-based caption generation. A significant aspect of the methodology is the extraction of 11,414 OpenIE triples, indexed with FAISS for efficient retrieval. The system employs a coarse-to-fine retrieval strategy inspired by HippoRAG2, which narrows down the search space effectively. Bayesian optimization is used to calibrate fusion weights, emphasizing the importance of the triple axis to mitigate lexical bias. The evaluation on a benchmark of 457 pairs demonstrates a high retrieval recall of 0.9909 and a 71.6% improvement in multi-hop answer similarity compared to single-hop queries. Additionally, an image-to-lesson retrieval pipeline showcases the versatility of the approach for visual inputs, indicating its broader applicability beyond the disaster management domain.
Methodology
The methodology involves constructing a multimodal index from disaster lesson PDFs, utilizing a hybrid OCR pipeline for text extraction, and employing a four-axis fusion architecture that integrates dense text embeddings, BM25 matching, knowledge-graph triples, and image similarity. The system uses Bayesian optimization for weight calibration and a hierarchical coarse-to-fine retrieval strategy for efficient multi-hop reasoning.
Results
The system achieves a retrieval recall of 0.9909 and a 71.6% improvement in multi-hop answer similarity over single-hop queries. The hybrid OCR pipeline raises table coverage from 68.4% to 86.8%, and the image-to-lesson retrieval pipeline achieves a mean Top-1 score of 0.6556 across 12 disaster images.
Implications
The findings suggest that the proposed triple-filtered multimodal fusion approach is crucial for effective structured reasoning over complex, noisy PDF documents. This framework can be applied to various domains beyond disaster management, enhancing retrieval capabilities in fields that deal with heterogeneous document types.
Emergent Misalignment Recruits a Pre-existing Persona Subspace
NLP
Large Language Models
Theory
- Fine-tuning on narrow bad advice can lead to broad misalignment in language models.
- A pre-existing persona structure in the model is recruited during fine-tuning, affecting unrelated domains.
- Projecting the persona subspace out of the residual stream during fine-tuning prevents misalignment.
- Injecting the persona subspace into an untouched model induces misalignment that increases with dosage.
Read more
Emergent Misalignment Recruits a Pre-existing Persona Subspace
Summary
This paper investigates the phenomenon of emergent misalignment in language models, where fine-tuning on a narrow dataset of poor advice leads to broad misalignment across unrelated topics. The author explores the underlying reasons for this generalization, discovering that fine-tuning recruits a pre-existing persona structure within the model. Using a frozen instruction-tuned model (Qwen2.5-14B-Instruct), the study extracts persona subspaces through contrastive teacher forcing, revealing that four unrelated domains share a low-rank core. The first optimizer step during fine-tuning is shown to couple with broad misalignment, and interventions such as projecting the persona subspace out of the residual stream can prevent this misalignment. The findings indicate that spreading a fixed budget of bad data across multiple domains exacerbates misalignment, and that the structure of the persona subspace is a regular loss minimum with distinct geometric properties. The paper provides a comprehensive analysis of how misalignment can be controlled and the implications of these findings for future model training and alignment strategies.
Methodology
The study employs contrastive teacher forcing to extract persona subspaces from a frozen instruction-tuned model. It conducts experiments to assess the impact of projecting these subspaces during fine-tuning and evaluates the effects of spreading bad data across multiple domains. The analysis includes comparisons with matched random controls and investigates the geometric properties of the persona structure.
Results
The research finds that four unrelated domains share a low-rank persona core, and the first gradient step of fine-tuning is linked to broad misalignment. Projecting the persona subspace out of the activations during fine-tuning reduces misalignment from 27.7% to 0.0%, while injecting it into a never-fine-tuned model increases misalignment to 45.4%. The study also reveals that the structure behaves consistently across independently trained models and that post-hoc weight edits do not alter the misalignment disposition.
Implications
These findings suggest that understanding and controlling persona subspaces in language models is crucial for maintaining alignment during training. The methods developed could inform future strategies for mitigating misalignment in AI systems, particularly in sensitive applications where ethical considerations are paramount.
KroQuant: Kronecker-Structured Block Transforms for Efficient Post-Training Quantization of Diffusion Transformers
Efficient ML
Generative Models
Computer Vision
- KroQuant introduces a learned block-diagonal activation transform that is more expressive than fixed transforms and computationally efficient.
- The method utilizes a Kronecker-LU parametrization, significantly reducing the parameter size compared to dense invertible matrices.
- KroQuant achieves faster inference times while maintaining or improving output quality compared to existing quantization methods.
- The approach is compatible with existing weight-splitting PTQ frameworks, enhancing their performance.
Read more
KroQuant: Kronecker-Structured Block Transforms for Efficient Post-Training Quantization of Diffusion Transformers
Summary
The paper introduces KroQuant, a novel post-training quantization (PTQ) method designed to enhance the quantization of diffusion transformers (DiTs) to 4-bit precision without significantly degrading output quality. Traditional PTQ methods struggle with outlier activations that cannot be effectively represented in low-bit formats, leading to quality loss. KroQuant addresses this by implementing a learned Kronecker-structured invertible transform applied to 32-element blocks of activations, which reduces the parameter footprint compared to existing methods while maintaining or improving quantization accuracy. The method integrates seamlessly into existing weight-splitting PTQ pipelines, such as LoRaQ, and offers a significant speed advantage, being up to 14% faster than the SmoothQuant kernel on an MI350 GPU. The authors demonstrate that KroQuant achieves outputs closer to full precision references than competing methods like SVDQuant and LoRaQ, thus providing a more efficient and effective solution for quantizing large-scale DiTs.
Methodology
KroQuant employs a learned Kronecker-structured invertible transform applied to blocks of activations, allowing for efficient quantization while addressing the issue of activation outliers. The method uses a compact parametrization that reduces the number of parameters needed, and it integrates with existing PTQ frameworks to enhance their performance. The quantization process involves applying a tensor-core GEMM operation for each block, optimizing computational efficiency.
Results
KroQuant demonstrates improved output quality compared to SVDQuant and LoRaQ on datasets such as PixArt-Σ, SANA, and FLUX.1-schnell, achieving results closer to full precision references. Additionally, the KroQuant quantizer kernel is reported to be up to 14% faster than the SmoothQuant kernel on MI350 GPUs, indicating significant improvements in computational efficiency.
Implications
The development of KroQuant has the potential to enhance the deployment of large-scale diffusion transformers in resource-constrained environments by enabling efficient quantization without sacrificing quality. This could facilitate broader adoption of DiTs in applications requiring real-time processing and lower memory usage.
One Round Is All You Need: Analytic Federated Learning for Task-Heterogeneous Multi-Label Medical Image Classification
Federated Learning
Computer Vision
- Introduces a novel analytic federated learning framework for task-heterogeneous multi-label medical image classification.
- Achieves convergence in at most two communication rounds, significantly reducing the communication overhead compared to existing methods.
- Demonstrates substantial performance improvements over the state-of-the-art method FedMLP in various missing-class configurations.
- Proposes closed-form operations to address class-imbalance bias and knowledge propagation challenges in federated settings.
Read more
One Round Is All You Need: Analytic Federated Learning for Task-Heterogeneous Multi-Label Medical Image Classification
Summary
This paper addresses the challenges of federated learning (FL) in the context of multi-label medical image classification, particularly under task heterogeneity where different clinical institutions only annotate a subset of disease categories. Traditional gradient-based FL methods struggle in this scenario, requiring numerous communication rounds to converge and suffering from systematic false-negative biases due to unobserved labels. The authors propose an analytic federated learning (AFL) framework that replaces iterative gradient optimization with closed-form operations, allowing for efficient training in just two communication rounds. The framework includes a balanced label projection to mitigate class-imbalance bias, a per-class absolute aggregation law for optimal classifier assembly, and an optional pseudo-label refinement step to enhance knowledge propagation from annotating to non-annotating clients. Experimental results on the ChestXray14 dataset demonstrate that the proposed method significantly outperforms the state-of-the-art method FedMLP, achieving improvements of up to 18.44 BACC points and 13.24 AUC points while reducing communication requirements. The method is also shown to be backbone-agnostic, generalizing well across various neural network architectures.
Methodology
The proposed framework employs three closed-form operations: a balanced label projection to address class imbalance, a per-class absolute aggregation law for constructing optimal classifiers from client-uploaded statistics, and an optional analytic pseudo-label refinement round to enhance knowledge transfer from clients with annotations to those without.
Results
The proposed analytic federated learning method outperformed the state-of-the-art FedMLP method by up to 18.44 BACC points and 13.24 AUC points in experiments on the ChestXray14 dataset, demonstrating its effectiveness in handling task heterogeneity and reducing communication rounds.
Implications
This work has significant implications for improving collaborative medical image classification across institutions with varying expertise, enhancing the ability to detect co-occurring conditions while adhering to patient privacy regulations. The framework can facilitate more efficient and accurate disease classification in clinical settings.
Generative Bayesian Filtering for State Estimation
Generative Models
Time Series
Theory
- Introduction of Generative Bayesian Filtering (GBF) for state estimation.
- Utilization of pretrained conditional generative models to enhance observation modeling.
- Development of a novel backpropagation scheme for information transfer from high-dimensional observations to low-dimensional states.
- Demonstrated improvements in accuracy and robustness in various applications, particularly under noisy conditions.
Read more
Generative Bayesian Filtering for State Estimation
Summary
This paper introduces Generative Bayesian Filtering (GBF), a novel framework for state estimation in dynamic systems that leverages pretrained conditional generative models, specifically conditional variational autoencoders (CVAE). Traditional filtering methods, such as Kalman filters, often rely on simplistic linear-Gaussian observation models, which fail to capture the complexities of high-dimensional sensor data. GBF addresses this limitation by integrating generative models into a Bayesian prediction-update recursion, transforming the measurement update into a posterior sampling problem that combines dynamical priors with likelihoods induced by CVAEs. This approach enhances the flexibility of observation models and allows for robust state estimation even in the presence of noise and distribution shifts. The authors demonstrate the effectiveness of GBF through experiments on synthetic datasets and real-world applications, including manufacturing system monitoring and arrhythmia diagnosis, showing significant improvements in state estimation accuracy and robustness compared to baseline methods.
Methodology
The GBF framework employs a Bayesian prediction-update recursion, where the measurement update is formulated as a posterior sampling problem. It utilizes pretrained CVAEs to model the likelihood of observations given the latent states, and a score-based sampling technique is developed to sample from the posterior using backpropagation. This allows for a recursive closed-loop framework for state estimation.
Results
The experiments conducted on synthetic datasets and real-world scenarios indicate that GBF significantly outperforms traditional filtering methods in terms of state estimation accuracy and robustness, especially when dealing with noisy observations and distribution shifts.
Implications
The proposed GBF framework has potential applications in various fields, including industrial process monitoring and healthcare, where accurate state estimation from noisy and high-dimensional sensor data is crucial for decision-making and risk assessment.
DataPrep-Bench: Benchmarking LLMs as Training Data Preparators
NLP
Large Language Models
- DataPrep-Bench is the first unified benchmark for evaluating LLMs in data preparation tasks.
- The benchmark assesses both data construction and data quality evaluation capabilities.
- Data-Construction-Skill improves data construction performance significantly.
- The Distributional Alignment Score (DAS) outperforms existing metrics in predicting downstream utility.
Read more
DataPrep-Bench: Benchmarking LLMs as Training Data Preparators
Summary
The paper introduces DataPrep-Bench, a novel benchmarking framework designed to evaluate the effectiveness of large language models (LLMs) in preparing training data. Recognizing the critical role of training data quality in the performance of LLMs, the authors propose a unified benchmark that assesses two key capabilities: data construction and data quality evaluation. Data construction involves transforming raw data sources into supervised training datasets, while data quality evaluation predicts the utility of candidate datasets for downstream tasks. The benchmark operates across six domains and utilizes multiple base models, providing a consistent evaluation protocol. The authors present a skill-guided agent, Data-Construction-Skill, which significantly improves data construction outcomes, and introduce the Distributional Alignment Score (DAS) for data quality evaluation, which demonstrates strong correlation with downstream performance. The results indicate that DataPrep-Bench offers a comprehensive framework for advancing LLM-driven data preparation, enabling researchers to measure progress in both data construction and quality evaluation effectively.
Methodology
The authors developed DataPrep-Bench to evaluate LLMs through two main tracks: data construction and data quality evaluation. For data construction, they introduced a skill-guided agent that enhances the baseline performance on specific tasks. For data quality evaluation, they proposed the Distributional Alignment Score (DAS), which measures the alignment between candidate datasets and domain proxies using Maximum Mean Discrepancy (MMD). The evaluation is grounded in downstream performance metrics across multiple domains.
Results
The Data-Construction-Skill agent achieved an improvement of nearly 20 points in absolute performance on the Llama-3.1-8B Finance model compared to the baseline. The DAS metric demonstrated the highest cross-model correlation in four out of six domains, with a notable performance of r > 0.70 in Math, Science, and Medical domains, outperforming traditional quality and diversity metrics.
Implications
DataPrep-Bench provides a robust framework for researchers and practitioners to evaluate and improve LLM-driven data preparation methods. It facilitates the identification of effective data construction strategies and reliable quality evaluation metrics, ultimately enhancing the training data quality for LLMs and their downstream applications.
Expanding Flow Maps
Generative Models
Graph Learning
NLP
- Introduction of Expanding Generative Flows (EFlows) for flexible generative modeling.
- Development of Expanding Flow Maps (EFMs) that utilize an expand operator and a transport map.
- Extension of the framework to discrete simplex spaces for variable-length discrete generation.
- Demonstration of the framework's effectiveness in various generative tasks, including molecular and language modeling.
Read more
Expanding Flow Maps
Summary
The paper introduces Expanding Flow Maps (EFMs), a novel framework for generative modeling that addresses the limitations of existing flow-based models, which are constrained to fixed dimensions or sequence lengths. The authors propose Expanding Generative Flows (EFlows), which allow for flows between distributions of increasing dimensionality by augmenting the state with conditional noise along an expanding interpolant. EFMs consist of two learnable operations: an expand operator that increases the state space and a transport map that moves the expanded state forward. This framework enables variable-size graph generation and variable-length sequence generation, making it applicable to a variety of real-world tasks where output size is not predetermined. The authors demonstrate the effectiveness of EFlows and EFMs through experiments in molecular conformer generation, discrete molecular graph generation, and language modeling, establishing them as a unified approach for variable-dimensional generation across different domains.
Methodology
The authors propose a piecewise-deterministic Markov process to define EFlows, which incorporates a transport term for smooth denoising and a jump kernel for transitioning to higher dimensions. EFMs are constructed by composing an expand operator that augments the state with noise and a transport map that advances the state along the interpolant. The framework is validated through empirical experiments across multiple domains.
Results
The experiments conducted demonstrate that EFlows and EFMs can effectively generate variable-dimensional outputs in a few steps, outperforming traditional fixed-dimensional flow models. The results indicate successful applications in molecular conformer generation, discrete molecular graph generation, and language modeling, showcasing the framework's versatility and efficiency.
Implications
The proposed framework has significant implications for generative modeling tasks where the output size is variable, such as in image generation, audio synthesis, and natural language processing. It allows for more flexible and efficient generation processes, potentially leading to advancements in various applications that require adaptive dimensionality.
Nipping the Butterfly Effect in the Bud: Self-Output Fine-Tuning for Autoregressive Weather Prediction
Time Series
- Error growth in autoregressive models is caused by a feedback loop between output errors and input distribution shifts.
- Small initial errors can quickly escalate, leading to out-of-distribution states early in the forecasting process.
- Self-Output Fine-Tuning (SOFT) is introduced as a simple, effective strategy to improve long-horizon forecasting.
- SOFT fine-tunes models on their own predictions to enhance distributional robustness.
Read more
Nipping the Butterfly Effect in the Bud: Self-Output Fine-Tuning for Autoregressive Weather Prediction
Summary
This paper addresses the challenge of long-horizon weather forecasting using autoregressive deep learning models, which suffer from rapid error growth over extended forecasting periods. The authors identify that this error growth is driven by a feedback loop between output errors and shifts in input distributions, akin to the butterfly effect in atmospheric science. They propose a novel method called Self-Output Fine-Tuning (SOFT), which fine-tunes the model on its own one-step predictions to mitigate the impact of these errors. SOFT effectively calibrates the biased input distribution encountered at the first autoregressive step, improving the model's robustness against out-of-distribution scenarios. The authors conduct extensive experiments demonstrating that SOFT achieves state-of-the-art performance in long-horizon forecasting tasks, significantly reducing prediction errors and distributional discrepancies. This work emphasizes the need to rethink the autoregressive pipeline in deep learning weather prediction, marking a significant advancement in atmospheric science.
Methodology
The authors analyze the error growth phenomenon theoretically and empirically, deriving an error propagation formula to understand the interaction between current and past errors. They propose SOFT as a plug-and-play strategy that fine-tunes the model using its own one-step predictions to address the distributional shifts that occur during autoregressive forecasting.
Results
The implementation of SOFT leads to a significant reduction in prediction errors and distributional discrepancies in long-horizon forecasting tasks. The method demonstrates state-of-the-art performance compared to existing autoregressive deep learning weather prediction models.
Implications
The findings suggest that addressing the feedback loop in autoregressive models can enhance the accuracy of long-term weather forecasts, which is crucial for climate-related decision-making and atmospheric research. The proposed SOFT method can be integrated into existing forecasting systems, potentially improving their reliability and effectiveness.
Fisher Widths: Local Learning Geometry and Anisotropic Recovery
Theory
Optimization
- Introduction of primal and inverse Fisher widths as complementary statistical measures.
- Establishment of a two-sided estimate for statistical dimension in sparse recovery contexts.
- Demonstration of the relationship between Fisher and inverse-Fisher widths, highlighting their interplay.
- Insights into the geometry of local parameter fluctuations and their implications for learning and recovery.
Read more
Fisher Widths: Local Learning Geometry and Anisotropic Recovery
Summary
This paper investigates the concept of Gaussian-width complexity on statistical manifolds through two functionals: the primal Fisher width and the inverse-Fisher width. The primal Fisher width, denoted as wG(T), reflects local parameter fluctuations in the geometry induced by the Fisher information, while the inverse-Fisher width, wG−1(T), captures the effects of anisotropic Gaussian measurements. The author demonstrates that for Fisher-regular losses, the scale of the Fisher width can be attained on sufficiently small Fisher balls. The paper also explores the geometry of sparse recovery, showing that it depends on both sparsity and the positioning of active coordinates within the Fisher spectrum. A two-sided estimate for the statistical dimension is provided, along with support-sensitive recovery estimates and a natural ordering of supports based on curvature profiles. A significant relationship between the primal and inverse-Fisher widths is established, indicating that while Fisher anisotropy can transfer complexity between geometries, it does not reduce both widths relative to the Euclidean scale.
Methodology
The paper employs mathematical analysis to define and explore the properties of the primal and inverse Fisher widths. It utilizes concepts from differential geometry and statistics, particularly focusing on the Fisher information matrix to assess local geometries and their implications for learning and recovery tasks.
Results
The author proves that the scale of the primal Fisher width is achievable on small Fisher balls and provides a two-sided estimate for statistical dimensions in recovery scenarios. The relationship between the primal and inverse Fisher widths is established, showing that they satisfy a specific inequality that reflects their complementary nature.
Implications
The findings have potential applications in statistical learning theory, particularly in understanding the geometry of parameter spaces in machine learning models. The insights into anisotropic recovery could inform better strategies for sparse recovery in high-dimensional settings, enhancing the efficiency of algorithms in various statistical and machine learning applications.
How Many Bits Can an Adapter Write? Measuring the Capacity and Memorization of Parameter-Efficient Fine-Tuning
NLP
Large Language Models
Efficient ML
- Adapters store less information than full fine-tuning models, with a capacity of a few bits per trainable parameter.
- The capacity of adapters is more dependent on their structural placement within the model than on the number of parameters.
- Supervised fine-tuning can lead to privacy leakage, while reinforcement learning with verifiable rewards does not retain sensitive data.
- The study introduces a measurement protocol for quantifying adapter capacity and its implications for privacy.
Read more
How Many Bits Can an Adapter Write? Measuring the Capacity and Memorization of Parameter-Efficient Fine-Tuning
Summary
This paper investigates the capacity and memorization characteristics of low-rank adapters (LoRA) in the context of parameter-efficient fine-tuning. The authors challenge the prevailing assumption that adapters merely represent skills without retaining sensitive training data. By extending compression-based memorization analysis to frozen-base settings, they quantify the information that adapters can store in bits, revealing that adapters write significantly less than full fine-tuning models. The study finds that the capacity of an adapter is influenced more by its structural placement within the model than by the number of parameters it contains. The authors also demonstrate that while supervised fine-tuning can lead to privacy leakage, reinforcement learning with verifiable rewards does not record sensitive information. They provide a measurement protocol for adapter capacity, linking it to real-world implications of privacy and data retention in machine learning models.
Methodology
The authors employed a compression-based analysis to measure the memorization capacity of low-rank adapters in a frozen-base setting. They used uniformly random token sequences to calibrate their measurements, allowing them to quantify the bits stored by the adapters. The study also compared the performance of adapters in supervised and reinforcement learning contexts to assess privacy implications.
Results
The analysis revealed that low-rank adapters write significantly fewer bits than full models, with a capacity that varies based on the structural placement of parameters. The results indicated that adapters could store nearly twice as much information when parameters are placed in the MLP compared to attention layers. Additionally, the study found that supervised fine-tuning led to privacy leakage, while reinforcement learning did not record sensitive information.
Implications
The findings have significant implications for the design and deployment of machine learning models, particularly in terms of privacy and data retention. Understanding the actual capacity of adapters can inform better practices for sharing model components and mitigating risks associated with sensitive data leakage.
Synthetic minority data is redundant or invalid: a data-dependent validity theory and a de-biased test
Theory
- Traditional validation of synthetic minority data is biased and often misleading.
- A new de-biased estimator provides a more accurate measure of synthetic data validity.
- Validity is a property of the data distribution, not the synthetic generation method.
- Synthetic oversampling can be redundant or invalid depending on class overlap.
Read more
Synthetic minority data is redundant or invalid: a data-dependent validity theory and a de-biased test
Summary
This paper critiques the long-standing practice of using synthetic minority oversampling techniques to address class imbalance in machine learning. The authors argue that the traditional validation methods for synthetic data are fundamentally flawed, as they often rely on the same data used to generate the synthetic points, leading to a systematic bias known as 'parent leakage.' To address this, they propose a de-biased estimator that evaluates the validity of synthetic points against withheld real data, establishing validity as a population quantity. Their findings reveal that the classical validation method significantly underestimates the invalidity of synthetic data, with the de-biased estimator providing a more accurate assessment. The authors demonstrate that the validity of synthetic data is inherently tied to the data's distribution, not the generation method, and that oversampling can be redundant or invalid depending on class overlap. Through extensive empirical analysis across various datasets and classifiers, they show that synthetic minority data often fails to provide meaningful improvements, with most methods resulting in negligible gains. The paper concludes by emphasizing the need for synthetic data to prove its validity and information gain on a case-by-case basis.
Methodology
The authors introduce a de-biased estimator that evaluates synthetic minority points against withheld real data to assess their validity. They conduct empirical analyses across 91 methods and three classifiers on various datasets, including those from medicine and finance, to compare the classical validation method with their proposed approach.
Results
The classical validation method underestimates the invalidity of synthetic data in 96-99% of cases, while the de-biased estimator closely tracks true invalidity. None of the synthetic oversampling methods evaluated were able to demonstrate significant improvements over trivial baselines, with median F1 scores below 0.01 and calibration issues in 77-93% of cases.
Implications
The findings suggest a paradigm shift in how synthetic minority data is validated and utilized in machine learning, emphasizing the need for rigorous testing against real data. This could lead to more reliable models in fields such as medicine and finance, where class imbalance is prevalent.
The Active Ingredient in Muon's Grokking
Optimization
Theory
Interpretability
- Orthogonalization is the key factor behind Muon's faster grokking speed, not spectral scaling.
- Orthogonalizing optimizers reach a lower-norm solution compared to AdamW's sparse-Fourier circuit.
- Reducing Newton–Schulz iterations increases speed but decreases the stability of the grokked solution.
- Grokking speed claims can vary significantly based on the metrics used, highlighting the need for careful measurement.
Read more
The Active Ingredient in Muon's Grokking
Summary
This paper investigates the Muon optimizer's ability to reach the grokking threshold in modular arithmetic faster than the AdamW optimizer. Previous research attributed this speedup to both spectral-norm constraints and orthogonalized momentum. However, this study isolates the contributions of these components through a series of multi-seed and multi-learning-rate experiments. The findings reveal that the speedup is primarily due to orthogonalization, specifically the Newton–Schulz iteration, rather than spectral scaling. The analysis shows that orthogonalizing optimizers achieve generalization at a significantly lower spectral norm and settle into a lower-norm solution. Additionally, reducing the number of Newton–Schulz iterations accelerates reaching the threshold but compromises the stability of the grokked solution, particularly at higher learning rates. The paper emphasizes the importance of measuring grokking speed under stability-aware metrics and provides a reproducible framework for further research.
Methodology
The study employs multi-seed and multi-learning-rate sweeps to assess the effects of orthogonalization and spectral scaling on the Muon optimizer's performance. An ablation study isolates the contributions of each component, while mechanistic analyses explore the characteristics of the solutions reached by different optimization strategies.
Results
The results indicate that orthogonalization alone matches the performance of the full Muon optimizer, while spectral scaling does not provide a speed advantage over AdamW. The orthogonalization approach leads to more stable grokking outcomes, and the optimal number of Newton–Schulz iterations is found to be five for robust performance across varying learning rates.
Implications
These findings have significant implications for the design of optimization algorithms in machine learning, particularly in contexts where generalization is critical. Understanding the dynamics of grokking can inform the development of more efficient and stable optimizers, potentially enhancing performance in various applications.
LLMs Get Lost in Evolving User Intent
NLP
Large Language Models
- LLMs are evaluated primarily in static, single-turn settings, which do not reflect real user interactions.
- The proposed framework allows for the transformation of static tasks into dynamic multi-turn conversations.
- Current LLMs show significant performance degradation when user intent evolves during interactions.
- The study highlights a fundamental gap in LLM capabilities regarding tracking and acting on evolving user intent.
Read more
LLMs Get Lost in Evolving User Intent
Summary
This paper addresses the limitations of large language models (LLMs) in tracking and acting on evolving user intent during multi-turn interactions. While LLMs have shown strong performance in static, single-turn tasks, they struggle significantly when user intent changes dynamically throughout a conversation. The authors introduce a novel framework that transforms static, single-turn tasks into dynamic multi-turn conversations, allowing for the simulation of evolving user intent without the need for new annotations. This framework captures three forms of conversational dynamics: incremental disclosure of task information, revision of previously stated details, and task switching. By anchoring the user’s intent at the final turn and constructing plausible preceding situations, the authors maintain verifiability against the original dataset. The framework is applied to various domains, revealing that even state-of-the-art models experience substantial performance drops in dynamic settings, highlighting a critical gap in current LLM capabilities. The findings suggest that LLMs need to improve their ability to track evolving user intent to be effective collaborative agents in real-world applications.
Methodology
The authors developed a framework that converts static, single-turn datasets into dynamic multi-turn environments by modeling user intent evolution through incremental disclosure, revision, and task switching. This approach allows for controlled testing of LLMs' abilities to track user intent over multiple turns while maintaining verifiability against the original dataset.
Results
The evaluation of various LLMs, including frontier and open-source models, demonstrated that models performing well in static settings experienced substantial drops in performance in dynamic environments. For instance, GPT 5.5's accuracy dropped from 99.0% to 80.5% in the math domain after just six intent transitions, indicating a significant challenge in tracking evolving user intent.
Implications
The findings suggest that future LLM development should focus on enhancing the ability to track and adapt to evolving user intent, which is crucial for effective collaboration in real-world applications. This research could inform the design of more robust LLMs that can better serve as collaborative agents in various domains.
TwistedMerge: Certified Higher-Order Diagnostics and Abstention for Model Merging
Theory
- TwistedMerge formalizes a finite certification problem for model merging, distinguishing between different types of cycle inconsistencies.
- The framework includes a conservative certification pipeline with mechanisms for abstention, ensuring only valid results are reported.
- The authors provide theoretical no-go results and error-control theorems that enhance the understanding of model merging dynamics.
- Experimental validation shows that naive factor averaging is dependent on chosen representatives, while the proposed methods are more stable.
Read more
TwistedMerge: Certified Higher-Order Diagnostics and Abstention for Model Merging
Summary
The paper presents TwistedMerge, a novel framework for model merging that addresses the challenges of combining independently trained models without assuming common parameter charts. The authors formulate the merging process as a finite descent problem, where checkpoints are treated as local objects and alignments as transitions. TwistedMerge introduces a conservative certification pipeline that distinguishes between various types of cycle inconsistencies and obstructions, ensuring that only valid residuals are promoted to cohomology classes after rigorous testing. The framework includes mechanisms for abstention, returning to fallback methods when necessary. The authors validate their approach through theoretical proofs and experimental results, demonstrating that naive factor averaging is unreliable and that cycle residuals do not predict merge degradation. Overall, TwistedMerge positions descent theory as a falsifiable framework for certification and abstention in model merging.
Methodology
The authors develop a certification pipeline that separates various regimes of model merging, including fixed-chart averaging and synchronization-removable gauge inconsistencies. They employ rigorous testing protocols for cohomology classes and validate their framework through controlled experiments and theoretical proofs.
Results
The results indicate that TwistedMerge effectively identifies and certifies valid merging scenarios while abstaining from unreliable predictions. The framework's validation through experiments shows that naive averaging methods are sensitive to parameter choices, and cycle residuals do not correlate with performance degradation.
Implications
TwistedMerge offers a robust framework for model merging that can improve the reliability of combining models in practical applications. Its certification and abstention mechanisms could enhance the performance of ensemble methods in various machine learning tasks.
Agree on the Model, Verify the Inference: GKR Protocols for HND-Based Transformer Inference
Theory
Efficient ML
- Introduction of GKR-HND for verifying outsourced HND Transformer inference.
- Separation of cryptographic verification from public computation to enhance efficiency.
- Validation of the protocol through experiments with pretrained models, confirming its effectiveness.
Read more
Agree on the Model, Verify the Inference: GKR Protocols for HND-Based Transformer Inference
Summary
The paper addresses the challenges of verifying outsourced Transformer inference, which can expose clients to risks such as model substitution and incomplete execution. The authors propose GKR-HND, a verification protocol that leverages the Homomorphic–Nonhomomorphic Decomposition (HND) of Transformers to ensure the integrity of the model during inference without the need for dense-matrix replay. The GKR-HND protocol separates the verification process from public computation, allowing a retained verifier to check the proof transcript and registered weights while delegating the computationally expensive tasks to a worker. This method assumes an honest verifier and non-collusion between the prover and worker, ensuring that any disagreement leads to rejection of the results. The experimental validation of GKR-HND with pretrained HND models demonstrates its effectiveness in verifying the proof path and delegated computations without the overhead of dense-matrix operations.
Methodology
The GKR-HND protocol applies the GKR verification method to the polynomial backbone of HND Transformers. It involves a retained verifier that checks the proof transcript and registered weights, while an assigned computation worker performs the public evaluations. The worker's responses are bound to specific queries, ensuring accountability and integrity in the verification process.
Results
The experimental results show that GKR-HND successfully verifies the delegated proof path for HND models without requiring dense-matrix replay, thus maintaining computational efficiency while ensuring model integrity.
Implications
The proposed GKR-HND protocol has significant implications for secure and efficient outsourced machine learning inference, particularly in scenarios where clients need to verify the integrity of models deployed in untrusted environments. It can enhance trust in cloud-based AI services and facilitate broader adoption of outsourced inference.
Improving Access to Essential Medicines via Decision-Aware Machine Learning
Optimization
Efficient ML
- Introduces a decision-aware machine learning framework for essential medicine allocation.
- Utilizes multi-task learning to improve sample efficiency and address data scarcity.
- Achieved a 19% increase in allocated product consumption in treated districts.
- Demonstrates the scalability of the solution to cover millions of vulnerable populations.
Read more
Improving Access to Essential Medicines via Decision-Aware Machine Learning
Summary
This paper addresses the critical challenge of efficiently allocating essential medicines in Low- and Middle-Income Countries (LMICs), particularly in Sierra Leone, where healthcare systems face significant resource constraints and data limitations. The authors propose a novel decision-aware machine learning framework that utilizes multi-task learning to enhance sample efficiency and employs catalytic priors to ensure equitable distribution of medicines. The framework was deployed as a decision support tool in collaboration with the Sierra Leone national government, leading to a nationwide implementation covering approximately 2 million women and children under five. The evaluation of the system indicated a 19% increase in the consumption of allocated products in treated districts, demonstrating its effectiveness in improving access to essential medicines. This work highlights the potential of machine learning to enhance healthcare delivery in resource-constrained settings, providing a scalable solution to address the challenges of demand forecasting and resource allocation.
Methodology
The authors developed a machine learning framework that predicts demand for essential medicines using historical data and applies stochastic optimization to determine the best allocation. The framework incorporates multi-task learning to share data across healthcare facilities and employs a decision-aware learning algorithm that aligns prediction loss with allocation objectives, utilizing catalytic priors to address data inequity.
Results
The implementation of the decision support tool resulted in a 19% increase in the consumption of allocated essential medicines in the districts where it was deployed. The system was successfully scaled to cover an estimated 2 million women and children under five, demonstrating its efficacy in improving access to healthcare resources.
Implications
This research has significant implications for healthcare systems in LMICs, showcasing how machine learning can be leveraged to optimize resource allocation and improve access to essential medicines. The framework can serve as a model for similar initiatives in other resource-constrained settings, potentially leading to better health outcomes and more efficient use of limited resources.
Do Active SAE Feature Planes Carry More Holonomy? A Preregistered Reversal in Gemma
NLP
Large Language Models
Theory
- The study tests the hypothesis that holonomy concentrates on active SAE feature planes.
- Holonomy was measured using a preregistered experimental design, ensuring unbiased results.
- Active-feature planes showed significantly less holonomy than mixed-feature controls, contradicting the initial prediction.
- The findings suggest that the relationship between semantic meaning and holonomy is more complex than previously thought.
Read more
Do Active SAE Feature Planes Carry More Holonomy? A Preregistered Reversal in Gemma
Summary
This paper investigates the relationship between holonomy and active sparse-autoencoder (SAE) feature planes within the context of the Gemma 2 2B model. The study aims to test the semantic-concentration prediction, which posits that holonomy should be concentrated in active SAE feature planes, indicating a geometric signature of semantic density. Holonomy is measured at the final-token layer of the model by utilizing a local frame and the restricted-Jacobian transport rule. The experimental design, including hypothesis and analysis methods, was preregistered to ensure objectivity. Contrary to the initial hypothesis, the results revealed that active-feature planes exhibited significantly less holonomy compared to matched mixed-feature controls, with a log contrast of -0.29439. The findings suggest that the expected relationship between meaning and holonomy does not hold, and the underlying mechanisms remain unexplored. The paper discusses various potential factors influencing these results, including activation strength and feature engagement, while emphasizing the need for further investigation into the geometric properties of activation space.
Methodology
The methodology involved measuring holonomy at the final-token layer of the Gemma model using a local frame and restricted-Jacobian transport rule. The study utilized three types of planes (active-feature, mixed-feature, and random) to compare holonomy across different selection criteria. The experimental design was preregistered to avoid bias in hypothesis testing and analysis.
Results
The results indicated that active-feature planes carried less holonomy than matched mixed-feature controls, with an adjusted log contrast of -0.29439 and a 95% confidence interval of [-0.43989, -0.14889]. This finding reverses the initial hypothesis that active-feature planes would exhibit greater holonomy.
Implications
The results challenge the assumption that semantic meaning directly correlates with holonomy in activation space, suggesting a more nuanced understanding of how meaning and geometric properties interact in neural representations. This could influence future research in understanding the structure of learned representations in language models.
A Polynomial Architecture-Attribution Co-Design Framework for Exact Aumann-Shapley Attribution in GNNs
Graph Learning
Interpretability
- Introduction of APEX, a co-design framework for exact Aumann-Shapley attribution in GNNs.
- Development of PolyGIN, a polynomial variant of the Graph Isomorphism Network that preserves polynomial mappings.
- Exact computation of path integrals using Gauss-Legendre quadrature, eliminating quadrature error.
- Demonstration of higher attribution fidelity and reduced computational costs compared to traditional methods.
Read more
A Polynomial Architecture-Attribution Co-Design Framework for Exact Aumann-Shapley Attribution in GNNs
Summary
This paper introduces APEX, a novel framework for feature-level and node-level explanations in Graph Neural Networks (GNNs) using Aumann-Shapley attribution. Traditional path-integral attribution methods, such as Integrated Gradients, often rely on numerical approximations that can lead to quadrature errors and increased computational costs. APEX addresses these limitations by employing a polynomial GNN architecture, PolyGIN, which ensures that the model outputs are multivariate polynomials. This structure allows for the exact computation of the attribution integral using Gauss-Legendre quadrature, eliminating the need for heuristic sampling. The authors demonstrate that the derivative along the attribution path has a degree bound, enabling efficient and precise attribution calculations. Experimental results on synthetic and real-world graph benchmarks show that PolyGIN maintains competitive predictive performance while APEX achieves higher attribution fidelity compared to existing methods, significantly reducing the number of evaluations required for path integration.
Methodology
The authors propose a polynomial architecture, PolyGIN, which replaces non-polynomial components in GNNs with polynomial or linear operations. This design ensures that the model outputs remain multivariate polynomials, allowing for the exact computation of the Aumann-Shapley attribution integral using Gauss-Legendre quadrature. The framework transforms the attribution problem from a sampling-dependent numerical approximation to a finite exact computation, significantly improving efficiency.
Results
The experiments conducted show that PolyGIN achieves competitive predictive performance while the APEX framework provides higher attribution fidelity than baseline methods. The number of evaluations required for path integration is substantially reduced, demonstrating the effectiveness of the proposed architecture in practical applications.
Implications
The APEX framework has significant implications for the interpretability of GNNs, making it suitable for applications where reliable explanations are crucial, such as in social networks, recommendation systems, and molecular property prediction. By ensuring exact attribution calculations, it enhances trust and transparency in GNN-based models.
Uncertainty-Aware Trust Estimation for Multi-LLM Systems via Structured Expert Judgement
Large Language Models
NLP
Theory
- Introduces an uncertainty-aware trust estimation framework for aggregating predictions from multiple LLMs.
- Adapts structured expert judgment techniques to evaluate and weight the reliability of LLMs based on their predictions.
- Demonstrates that traditional aggregation methods fail in heterogeneous and contaminated settings, leading to overconfident errors.
- Cooke weighting significantly improves accuracy-reliability balance in challenging aggregation scenarios.
Read more
Uncertainty-Aware Trust Estimation for Multi-LLM Systems via Structured Expert Judgement
Summary
This paper addresses the challenge of aggregating predictions from multiple Large Language Models (LLMs) by introducing an uncertainty-aware trust estimation framework. Traditional aggregation methods often treat all models as equally reliable, which can lead to poor performance, especially in heterogeneous environments where models exhibit varying levels of reliability. The authors propose a novel approach that adapts structured expert judgment techniques, specifically Cooke's method, to evaluate the reliability of LLMs based on their probabilistic predictions. By employing context-aware calibration questions, the method assigns weights to models, penalizing overconfident incorrect predictions and rewarding well-calibrated experts. The evaluation is conducted on benchmark datasets MMLU and MMLU-Pro across different expert panel configurations: homogeneous, heterogeneous, and contaminated. The results indicate that while traditional aggregation methods perform similarly in homogeneous settings, the proposed Cooke weighting significantly enhances accuracy and reliability in more complex scenarios, demonstrating its robustness against unreliable experts. This work emphasizes the necessity of calibrating trust under uncertainty in multi-LLM systems, rather than merely combining predictions.
Methodology
The authors frame multi-LLM aggregation as an uncertainty-aware trust estimation problem, utilizing structured expert judgment methods. They employ Cooke-style log weighting to evaluate expert reliability based on calibration questions, assigning weights that penalize overconfident incorrect predictions and favor well-calibrated experts.
Results
The evaluation shows that Cooke weighting achieves a superior accuracy-reliability balance compared to traditional aggregation methods, particularly in heterogeneous and contaminated expert panels. It effectively reduces overconfident failures and maintains strong performance even when unreliable experts are present.
Implications
This research has significant implications for deploying LLMs in high-stakes domains such as healthcare and legal decision-making, where reliable reasoning under uncertainty is crucial. The proposed framework can enhance the trustworthiness of LLM systems by ensuring that predictions are not only accurate but also accompanied by justified confidence levels.
Exact ReLU realization of affine one-dimensional refinement iterates via residual memory and offset frames
Theory
Efficient ML
- Introduces a residual memory controller for exact backward replay of residual states.
- Proves that finite affine iterates can be realized with fixed-width ReLU networks and depth O(n).
- Utilizes offset frames to ensure seam-safe evaluations of forcing terms.
- Extends results to arbitrary compactly supported continuous piecewise linear forcing terms for M ≥ 3.
Read more
Exact ReLU realization of affine one-dimensional refinement iterates via residual memory and offset frames
Summary
This paper investigates vector-valued affine refinement operators, focusing on the exact realization of finite affine iterates using ReLU networks. The authors build on previous work that established a homogeneous realization theorem, proving that for M ≥ 3, every finite affine iterate admits an exact fixed-width ReLU realization with depth O(n). The key innovation is the introduction of a residual memory controller, which allows for the exact backward replay of residual states necessary for a Horner-type evaluation of the affine forcing sum. Additionally, offset frames are utilized to align forcing atoms away from residual seams, enabling accurate recovery of values. The results extend to arbitrary compactly supported continuous piecewise linear forcing terms for M ≥ 3 and provide a linear-depth upgrade for various recursive constructions, including Hilbert and Morton-type curves. The paper addresses a significant gap in the realization of affine forcing sums, contributing to the understanding of neural network approximation theory.
Methodology
The authors employ a combination of theoretical analysis and construction of neural network architectures. They utilize a residual memory controller to manage the state dynamics and offset frames to handle the forcing terms. The proof involves a backward Horner-type recursion to realize the affine forcing sum accurately.
Results
The main result establishes that for M ≥ 3, every finite affine iterate can be realized with a fixed-width ReLU network and depth O(n). The paper also demonstrates that the same construction applies to ordinary-frame seam-separated forcing for M = 2, and provides a stage-dependent extension for fixed finite-dimensional continuous piecewise linear spans.
Implications
The findings have significant implications for the design of neural networks capable of approximating complex functions with reduced depth and width. This could enhance the efficiency of neural network architectures in applications involving recursive structures and geometric constructions.
End-to-End Learning of Safe Optimal Feedback Control in High Dimensions with Control Barrier Function Layers
Robotics
Optimization
Theory
- Introduces a scalable end-to-end training framework for high-dimensional feedback controllers with embedded CBF safety layers.
- Combines operator splitting and Jacobian-Free Backpropagation to overcome computational bottlenecks in differentiating through QP layers.
- Establishes convergence guarantees for training safety-constrained implicit networks despite nonsmoothness from CBFs.
- Demonstrates effectiveness on various high-dimensional nonlinear control problems, showcasing scalability beyond previous methods.
Read more
End-to-End Learning of Safe Optimal Feedback Control in High Dimensions with Control Barrier Function Layers
Summary
This paper addresses the challenge of learning high-dimensional feedback controllers that adhere to strict safety constraints using Control Barrier Functions (CBFs). Traditional methods have struggled with computational and differentiation bottlenecks, limiting their application to low-dimensional systems. The authors propose a novel framework that combines operator splitting with Jacobian-Free Backpropagation (JFB) to enable scalable end-to-end training of neural feedback controllers while ensuring safety through CBFs. The theoretical foundation is established using nonsmooth analysis techniques, providing convergence guarantees even in the presence of nonsmooth safety layers. The effectiveness of the proposed approach is validated through extensive numerical experiments on high-dimensional multi-agent nonlinear control problems, demonstrating its capability to handle state dimensions up to 1200 and control dimensions up to 400, far exceeding the limitations of previous methods.
Methodology
The authors develop a framework that integrates operator splitting with Jacobian-Free Backpropagation (JFB) to facilitate end-to-end training of neural feedback controllers. This approach decomposes the quadratic programs induced by CBFs into simpler subproblems, allowing for efficient differentiation and avoiding the full KKT system formulation, which is computationally intensive.
Results
The proposed framework was validated through numerical experiments involving high-dimensional nonlinear multi-agent control problems. The results indicate that the method can effectively learn safe feedback control policies in scenarios with state dimensions exceeding 1000 and control dimensions exceeding 100, demonstrating significant scalability compared to existing approaches.
Implications
This work has potential implications for the development of safe control systems in robotics and autonomous vehicles, where high-dimensional state spaces and strict safety requirements are common. The ability to learn safe control policies end-to-end could enhance the reliability and performance of such systems in real-world applications.
Training Large Language Models for Self-Explanation Faithfulness
NLP
Large Language Models
Reinforcement Learning
- Introduces an RL method to directly optimize the faithfulness of self-explanations in LLMs.
- Implements counterfactual interventions to derive rewards for training models.
- Achieves substantial improvements in faithfulness metrics, indicating better alignment between reasoning and decision-making.
- Demonstrates cross-intervention generalization, suggesting potential for broader applicability.
Read more
Training Large Language Models for Self-Explanation Faithfulness
Summary
This paper presents a novel approach to enhance the faithfulness of self-explanations generated by large language models (LLMs) through a reinforcement learning (RL) framework. The authors identify a gap in existing methodologies that evaluate or prompt for better self-explanations but do not optimize model parameters for faithfulness. They propose a method that modifies existing faithfulness metrics into an RL training objective, focusing on explanatory faithfulness. The study investigates whether models can be trained to accurately identify influential factors in their decision-making and whether RL can optimize the disclosure of these factors. Two types of interventions—random-word insertions and user-bias insertions—are employed, with rewards derived from the Phi-CCT correlation metric. The results demonstrate significant improvements in faithfulness metrics for fine-tuned models, with in-distribution scores rising from near-zero to 0.664 and out-of-distribution scores reaching 0.691 on held-out tasks. The findings also reveal interesting cross-intervention generalization effects, indicating model-dependent behaviors. The authors conclude that their RL approach effectively trains models to identify and disclose influential factors, paving the way for more faithful reasoning in LLMs.
Methodology
The authors utilize a reinforcement learning framework that incorporates signals from counterfactual tests to create a reward system for training LLMs. The model is rewarded for producing explanations that accurately mention influential factors while omitting non-influential ones. Two types of interventions—random-word and user-bias insertions—are used to evaluate the model's performance.
Results
The RL fine-tuned models, Llama3.1-8B and Qwen3-8B, show significant improvements in the Phi-CCT faithfulness metric, with in-distribution scores increasing from near-zero to 0.664 and out-of-distribution scores reaching up to 0.691 on held-out tasks like StrategyQA. The study also notes interesting cross-intervention generalization effects, with some models showing non-zero transfer between different types of interventions.
Implications
The findings suggest that RL can be effectively used to enhance the interpretability and transparency of LLMs by ensuring that their self-explanations are more faithful to their internal decision-making processes. This has potential applications in areas requiring high-stakes decision-making and model oversight, such as healthcare, finance, and legal domains.
StabilityBench: Benchmarking Instability in LLMs
NLP
Large Language Models
- StabilityBench transforms single-turn benchmarks into multi-turn interaction histories to better simulate real-world scenarios.
- The framework incorporates user simulations and baiting techniques to evaluate LLMs under more realistic conditions.
- Evaluation of nine LLMs shows significant performance instability, with many models failing on baited versions of benchmarks.
- The study emphasizes the limitations of static evaluation methods and the necessity for dynamic assessment frameworks.
Read more
StabilityBench: Benchmarking Instability in LLMs
Summary
The paper introduces StabilityBench, a novel framework designed to benchmark the instability of large language models (LLMs) in real-world applications, particularly in high-stakes environments like healthcare and government services. Traditional evaluation methods often rely on static, single-turn benchmarks that fail to capture the complexities of multi-turn interactions and user-dependent variability. StabilityBench addresses this gap by transforming single-turn queries into multi-turn interaction histories, incorporating realistic user simulations through demographic proxies and baiting techniques. The authors apply StabilityBench to four established benchmarks across domains such as mathematical reasoning and health question-answering, evaluating nine LLMs. The findings reveal significant performance instability, with many models exhibiting degraded performance under the new evaluation conditions. This highlights the limitations of current static evaluations and underscores the need for more dynamic assessment frameworks. Additionally, the paper proposes StabilityBench-Mini, a cost-effective variant that samples across diversification axes for more realistic evaluations without increasing resource demands.
Methodology
The authors developed a Multi-turn Interaction Simulator that generates multi-turn conversations based on user personas derived from socio-demographic factors. They also created a Baiting Module that injects prompt variations designed to distract or mislead the models. The framework was applied to four existing benchmarks, and performance was assessed using metrics like Bait Degradation Rate and Simulation Degradation Rates.
Results
The evaluation revealed that model performance was consistently unstable, with significant degradation observed in three out of four benchmarks. Many queries that were answered correctly in the original benchmarks were answered incorrectly in the baited versions, indicating a high level of instability in the models. The study quantified these performance shifts using various rates, demonstrating the inadequacy of traditional evaluation methods.
Implications
The findings suggest that current evaluation protocols for LLMs may not adequately reflect real-world performance, particularly in high-stakes applications. The introduction of StabilityBench could lead to improved assessment frameworks that better capture the complexities of user interactions and model behavior over time, ultimately enhancing the reliability of AI systems in critical domains.
Memoir: Should a Model Write to Its Memory While It Thinks?
NLP
Large Language Models
Theory
- Memoir combines multiple memory tiers and a latent recurrence process for enhanced model performance.
- The architecture allows for memory rewriting during pondering iterations, posing a risk of corrupting the learning process.
- Empirical results show that read-only pondering outperforms coupled pondering in terms of recall at early training stages.
- Both architectures achieve similar performance at longer training durations, indicating a trade-off between learning speed and capability.
Read more
Memoir: Should a Model Write to Its Memory While It Thinks?
Summary
The paper introduces Memoir, a novel architecture that integrates per-sample fast memory, shared slow parameters, and a variable-depth latent recurrence process. The central question addressed is whether a model should write to its fast memory during each pondering iteration. The authors conduct experiments comparing two configurations: a coupled arm that allows memory rewriting and a read-only arm. Both configurations are identical in terms of parameters and training conditions. Results indicate that while the read-only arm outperforms the coupled arm in recall after 240 training steps, both arms eventually reach the task ceiling after 960 steps, suggesting that the coupled architecture incurs a learning-speed penalty rather than a capability penalty. The paper also reports that the anticipated negative impact of memory rewriting on the energy signal did not occur, and provides a comprehensive implementation of the proposed architecture.
Methodology
The authors implemented a four-tier memory interface with explicit ownership and lifetime boundaries. They conducted a paired ablation study comparing a coupled memory architecture against a read-only architecture across multiple training seeds. The experiments measured recall performance at two different training durations (240 and 960 steps) to evaluate the impact of memory rewriting.
Results
After 240 training steps, the coupled arm achieved a recall score of 0.5203, while the read-only arm achieved 0.6557, indicating a significant performance difference. After 960 steps, both arms reached a perfect recall score of 1.0000, suggesting that the initial performance gap was due to a learning-speed penalty rather than a fundamental capability difference. Additionally, the expected collapse of the energy margin did not occur, and the energy margin improved.
Implications
The findings suggest that while allowing models to write to memory during processing can introduce risks, it does not necessarily compromise performance in the long term. This could have implications for the design of future memory-augmented models in various applications, particularly in areas requiring adaptive learning and memory integration.
Counterfactual Explainability Framework With CycleGAN And Counterfactual-Classifier Alignment Score For Retinal Disease Classification
Computer Vision
Generative Models
Interpretability
- Introduction of CounterFundus, a CycleGAN-based counterfactual explainability framework for retinal disease detection.
- Development of the Counterfactual-Classifier Alignment Score (CCAS) to assess spatial agreement between counterfactuals and classifier saliency.
- Demonstration of improved classification performance through CCAS-filtered counterfactual augmentation.
- CounterFundus provides visually interpretable explanations that are clinically relevant and meaningful.
Read more
Counterfactual Explainability Framework With CycleGAN And Counterfactual-Classifier Alignment Score For Retinal Disease Classification
Summary
This paper introduces CounterFundus, a novel framework for counterfactual explainability in the context of retinal disease detection using fundus images. The framework leverages CycleGAN for generating counterfactual images that represent healthy counterparts of diseased retinal images, facilitating the localization of disease-associated changes. The authors highlight the limitations of traditional post-hoc saliency methods, which often fail to provide clinically meaningful insights. CounterFundus addresses this gap by providing visually plausible disease-to-normal translations and introduces the Counterfactual-Classifier Alignment Score (CCAS) to quantify the spatial agreement between counterfactual difference maps and classifier saliency. The evaluation demonstrates that the generated counterfactuals align well with classifier-relevant retinal evidence, and ablation studies confirm that CCAS-filtered counterfactual augmentation enhances classification performance. This work establishes CounterFundus as a clinically-grounded explainable AI framework for retinal disease detection, emphasizing the importance of explainability in clinical applications of deep learning.
Methodology
The methodology involves using CycleGAN to generate counterfactual images that represent healthy versions of diseased retinal images. The framework integrates EfficientNet-B5 for retinal disease classification and employs the Counterfactual-Classifier Alignment Score (CCAS) to evaluate the spatial consistency between generated counterfactuals and classifier saliency maps. The study includes ablation experiments to assess the impact of CCAS-filtered counterfactuals on classification performance.
Results
The results indicate that the counterfactual explanations generated by CounterFundus are spatially consistent with classifier-relevant retinal evidence across all dimensions of the CCAS. Additionally, the ablation studies show that incorporating CCAS-filtered counterfactuals significantly improves the classification performance on fundus images, validating the effectiveness of the proposed framework.
Implications
The implications of this research extend to enhancing the interpretability and trustworthiness of deep learning models in medical imaging, particularly for retinal disease detection. By providing clinically meaningful explanations, the CounterFundus framework can facilitate better decision-making in clinical settings and promote the adoption of AI technologies in healthcare.
TOUR: A Trajectory-Level Unlearning Benchmark for Offline Reinforcement Learning
Reinforcement Learning
- TOUR is a novel benchmark for evaluating trajectory-level unlearning in offline RL.
- The benchmark incorporates matched non-member controls and multi-attack privacy audits.
- Common deletion methods may appear effective under single-score evaluations but fail under comprehensive audits.
- The study reveals the importance of retaining task performance while ensuring privacy.
Read more
TOUR: A Trajectory-Level Unlearning Benchmark for Offline Reinforcement Learning
Summary
The paper introduces TOUR, a benchmark designed to evaluate trajectory-level unlearning in offline reinforcement learning (RL). Offline RL agents are trained on fixed behavioral trajectories, making it essential to effectively delete specific trajectories after training to comply with privacy regulations. The authors highlight the challenges in evaluating trajectory deletion, as reductions in membership scores can be misleading, reflecting either successful deletion or policy collapse. TOUR combines trajectory-level partitioning, matched non-member controls, retraining references, retained-performance anchors, and multi-attack privacy auditing to provide a comprehensive evaluation framework. The benchmark is tested across various environments, including D4RL locomotion tasks and an AntMaze extension, revealing that common deletion methods exhibit environment-dependent privacy-utility behaviors. The findings indicate that naive fine-tuning and uniform GA+Refit methods can misrepresent privacy without thorough audits, while TrajDeleter shows variable effectiveness. The study emphasizes the need for robust evaluation metrics that account for multiple factors influencing unlearning in offline RL.
Methodology
TOUR employs a structured evaluation protocol that partitions trajectories into forget, retain, and matched non-member sets. It assesses updates through forget-side membership, retain-side diagnostics, and retained utility, utilizing various deletion baselines and auditing techniques to ensure comprehensive analysis of unlearning effectiveness.
Results
The evaluation of TOUR across different environments showed that common deletion baselines often fail to provide consistent privacy-utility outcomes. Specifically, naive fine-tuning and uniform GA+Refit methods can lead to misleading interpretations of privacy, while TrajDeleter's performance varies significantly depending on the environment. The results underscore the necessity for a multi-faceted approach to auditing unlearning in offline RL.
Implications
The findings of this study have significant implications for the development of privacy-preserving algorithms in offline RL applications, particularly in sensitive domains such as healthcare and autonomous driving. By establishing a robust benchmark for trajectory-level unlearning, the research paves the way for improved compliance with privacy regulations and better understanding of the trade-offs between privacy and utility in RL systems.
Offline RL with Hierarchical Action Chunking
Reinforcement Learning
Robotics
Theory
- Introduction of HiQC, which combines high-level latent planning with low-level action chunking.
- Theoretical analysis shows improved value estimation error bounds compared to traditional methods.
- HiQC achieves superior performance on long-horizon tasks, particularly in navigation scenarios.
- The algorithm enables unbiased k-step value backups, reducing recursion depth for value estimation.
Read more
Offline RL with Hierarchical Action Chunking
Summary
This paper presents Hierarchical Implicit Q-Chunking (HiQC), a novel offline goal-conditioned reinforcement learning (RL) algorithm designed to address the challenges posed by the 'curse of horizon' in long-horizon tasks. Traditional offline RL methods often struggle with value estimation errors that accumulate over extended timeframes, particularly when relying on low-level controllers that execute single actions. HiQC mitigates these issues by integrating high-level latent planning with low-level action chunking, allowing the low-level critic to condition on temporally extended action sequences. This dual decomposition results in unbiased k-step value backups, effectively compressing the planning and execution horizons. Theoretical analysis demonstrates that HiQC achieves a tighter bound on value error compared to standard hierarchical or flat chunking methods. Empirical evaluations on the OGBench suite reveal that HiQC outperforms existing methods, particularly excelling in long-horizon navigation tasks such as humanoid-giant.
Methodology
The methodology involves a two-level hierarchical structure where a high-level planner generates subgoals in a latent space, while a low-level controller executes action chunks. The low-level critic is conditioned on these action chunks, allowing for unbiased k-step returns. Theoretical analysis is conducted under a bounded per-backup error model to establish the benefits of this dual decomposition.
Results
HiQC achieved the highest aggregate performance on the OGBench suite, particularly excelling in long-horizon navigation tasks. The empirical results indicate significant improvements over existing offline RL methods, validating the effectiveness of the hierarchical and chunking approach.
Implications
The findings suggest that HiQC could be applied to various long-horizon tasks in offline RL settings, potentially enhancing the performance of agents in complex environments where traditional methods struggle. This approach may also inform future research in hierarchical reinforcement learning and action abstraction techniques.
Memory-Computation Tradeoffs in Semi Amortized Parametric Optimization
Optimization
Theory
Efficient ML
- Establishes a theoretical framework for memory-computation tradeoffs in semi-amortized optimization.
- Identifies matching upper and lower bounds for memory requirements in strongly convex and β-growth regimes.
- Discovers a phase transition in K, indicating a limit to the benefits of additional memory.
- Introduces a meta proof framework that can be applied to various optimization classes.
Read more
Memory-Computation Tradeoffs in Semi Amortized Parametric Optimization
Summary
This paper investigates the memory-computation tradeoffs in semi-amortized parametric optimization, particularly focusing on how much offline information is necessary to achieve a desired accuracy under a fixed online computation budget. The authors introduce a theoretical framework that quantifies the memory required to ensure ε-accuracy when using K steps of projected gradient descent (PGD) for online optimization. They establish matching upper and lower bounds for memory requirements in the context of µ-strongly convex objectives and near-matching bounds for convex objectives with a β-growth condition. A notable finding is the identification of a phase transition in K, beyond which additional memory does not yield further benefits. The paper also provides a meta proof framework that highlights the convergence rate of the online optimizer and the Lipschitz sensitivity of the solution map as key factors influencing the memory-computation tradeoff. Empirical validation through experiments on parameterized ridge regression supports the theoretical predictions, confirming the established bounds and phase transitions.
Methodology
The authors analyze a semi-amortized optimization setup where an offline phase stores solutions to previously solved problems, and an online phase retrieves these solutions to refine them using K steps of projected gradient descent. They develop a theoretical framework to quantify the memory required for achieving a specified accuracy and acceleration, and conduct experiments on parameterized ridge regression to validate their findings.
Results
The paper presents strong theoretical results, including matching bounds for memory requirements in the context of µ-strongly convex objectives and near-matching bounds for convex objectives with β-growth conditions. The empirical experiments corroborate the theoretical predictions, demonstrating the scaling of memory requirements and the observed phase transition in K.
Implications
The findings have significant implications for the design of learning-enabled decision systems that operate under strict latency constraints, as they provide a clearer understanding of the tradeoffs between memory usage and computation time. This can guide the development of more efficient optimization algorithms in various applications, including real-time decision-making and machine learning.
SenCos-GEM: SENet-Calibrated and Law-of-Cosines-Constrained Geometry-Enhanced Molecular Representation for Property Prediction
Graph Learning
- Introduces SenCos-GEM, a framework for enhanced molecular representation learning.
- Incorporates physics-guided geometric consistency to improve 3D spatial representation.
- Utilizes dynamic feature modulation to prevent catastrophic forgetting during adaptation.
- Achieves state-of-the-art results on MoleculeNet, particularly in conformation-sensitive tasks.
Read more
SenCos-GEM: SENet-Calibrated and Law-of-Cosines-Constrained Geometry-Enhanced Molecular Representation for Property Prediction
Summary
The paper introduces SenCos-GEM, a novel framework for molecular representation learning that enhances property prediction accuracy by addressing limitations in existing methods. Traditional approaches often fail to incorporate explicit physical constraints and are sensitive to geometric noise, leading to issues like catastrophic forgetting during downstream adaptation. SenCos-GEM overcomes these challenges by employing a physics-guided geometric consistency loss based on the law of cosines, which ensures high-fidelity 3D spatial priors. Additionally, the framework integrates lightweight Squeeze-and-Excitation (SE) modules as task-specific adapters and features a dual-modulation prediction head that combines Feature-wise Linear Modulation (FiLM) and SENet mechanisms for dynamic feature recalibration. The model demonstrates state-of-the-art performance on the MoleculeNet benchmark, particularly excelling in 3D conformation-sensitive regression tasks, achieving significant reductions in relative error across various datasets. This work represents a significant advancement in molecular property prediction, highlighting the importance of robust spatial modeling and dynamic feature modulation.
Methodology
SenCos-GEM employs a decoupled approach for molecular representation learning, utilizing a GeoGNN backbone for self-supervised pre-training. It integrates a geometric consistency loss based on the law of cosines and incorporates SE modules for task-specific adaptations. The prediction head utilizes FiLM and SENet mechanisms for dynamic feature recalibration, enhancing the model's adaptability to downstream tasks.
Results
SenCos-GEM achieved state-of-the-art performance on various classification and regression tasks within the MoleculeNet benchmark, with notable relative error reductions of 12.9% (RMSE) on FreeSolv, 5.3% (RMSE) on Lipophilicity, and 8.2% (MAE) on QM9. The model also demonstrated enhanced capabilities in distinguishing stereoisomers and handling conformational perturbations.
Implications
The advancements presented in SenCos-GEM could significantly impact drug discovery and molecular design by providing more accurate predictions of molecular properties, thereby facilitating the identification of promising drug candidates and enhancing the understanding of structure-property relationships.
Context-weighted Discrete Flow Matching
Generative Models
NLP
Large Language Models
- Prediction difficulty in DFM is linked to the density of local context.
- Modification of CTMC dynamics to incorporate local context improves generation quality.
- Context-weighted sampling enhances performance without fine-tuning.
- Scaled cross-entropy loss significantly reduces generative perplexity.
Read more
Context-weighted Discrete Flow Matching
Summary
This paper introduces a novel approach to Discrete Flow Matching (DFM) by incorporating local context information into the generative modeling process. The authors identify that the difficulty of predicting tokens in DFM is closely related to the density of available context in their neighborhoods. To address this, they propose a modification to the Continuous-Time Markov Chain (CTMC) dynamics, allowing the transition probabilities to depend on local context while maintaining the overall marginal probabilities. This leads to a context-weighted sampler that enhances generation quality without significant computational overhead. Additionally, a scaled cross-entropy loss function is introduced to reweight the training signal from different tokens, resulting in a substantial reduction in generative perplexity. The empirical results demonstrate that their approach improves generation quality across various domains, including text and molecular generation, while matching the performance of a strong semi-autoregressive baseline. The findings underscore the importance of local context in discrete generative modeling and suggest that simple context-aware modifications can lead to significant improvements in both sampling and training efficiency.
Methodology
The authors modify the transition dynamics of the Continuous-Time Markov Chain (CTMC) used in Discrete Flow Matching to depend on local context. They derive a context-weighted sampler and a scaled cross-entropy loss function that reweights the training signal based on the context available for each token. The methodology includes empirical evaluations on text and molecular generation tasks to assess the effectiveness of the proposed modifications.
Results
The context-weighted sampler improves MAUVE scores by up to 24% on OpenWebText and increases the number of valid and novel samples in molecular generation by up to 2.8x and 1.9x, respectively. The scaled cross-entropy loss reduces generative perplexity by up to 63% and matches the performance of a strong semi-autoregressive block diffusion baseline while maintaining the flexibility of any-order generation.
Implications
The findings suggest that incorporating local context into generative models can lead to significant improvements in generation quality and efficiency. This has potential applications in various fields, including natural language processing, molecular design, and other areas requiring discrete data generation.
Chronofy: A Temporal-Logical Decay Architecture for Information Validity in Time-Aware Retrieval-Augmented Generation
NLP
Large Language Models
Graph Learning
- Chronofy integrates temporal validity into RAG systems through a three-layer architecture.
- The framework employs learnable exponential decay functions for graph-based retrieval.
- Signal Temporal Logic is applied to evaluate the temporal validity of retrieved knowledge.
- Chronofy significantly reduces temporal hallucination and improves retrieval precision.
Read more
Chronofy: A Temporal-Logical Decay Architecture for Information Validity in Time-Aware Retrieval-Augmented Generation
Summary
Chronofy introduces a novel framework for Retrieval-Augmented Generation (RAG) systems that addresses the issue of temporal hallucination, where outdated information can mislead outputs. The proposed Temporal-Logical Decay Architecture (TLDA) consists of three layers: (1) a temporal subspace embedding that integrates fact age into the representation, (2) a decay-weighted graph retrieval mechanism that utilizes learnable exponential decay functions to prioritize more recent information, and (3) a verification layer employing Signal Temporal Logic (STL) to assess the temporal validity of retrieved knowledge. This framework ensures that the most decayed evidence bounds the output confidence, thereby enhancing the reliability of the generated responses. The authors evaluate Chronofy against temporal knowledge graph forecasting benchmarks and a temporal QA benchmark, demonstrating significant improvements in retrieval precision and reductions in temporal hallucination. The results indicate that explicit modeling of temporal decay not only enhances the accuracy of the RAG systems but also facilitates effective data re-acquisition triggers when temporal context is insufficient.
Methodology
Chronofy utilizes a three-layer neuro-symbolic framework. Layer 1 incorporates temporal subspace embeddings to represent the age of facts. Layer 2 employs decay-weighted graph retrieval using learnable exponential decay functions grounded in Bayesian decision theory. Layer 3 applies STL robustness functions to verify the temporal validity of the knowledge retrieved, ensuring that the output confidence is constrained by the most decayed evidence.
Results
Chronofy was evaluated on temporal knowledge graph forecasting benchmarks and the TimE temporal QA benchmark, showing improved retrieval precision and a reduction in temporal hallucination rates. The framework's explicit temporal decay modeling was found to enhance the reliability of outputs and facilitate effective data re-acquisition triggers.
Implications
The findings suggest that incorporating temporal validity into RAG systems can significantly enhance their performance in domains where the timeliness of information is critical, such as healthcare, finance, and legal compliance. This approach could lead to more accurate decision-making systems that rely on up-to-date information.
AI Assistants Overassist
NLP
Large Language Models
- Introduction of INT-BENCH, a benchmark for evaluating LLM interventions in educational contexts.
- LLMs intervene more frequently and earlier than human tutors, often providing complete solutions.
- Interventions by LLMs can hinder deeper learning and cognitive engagement.
- Empirical analysis shows LLM interventions impact immediate task success but limit generalization to new problems.
Read more
AI Assistants Overassist
Summary
The paper investigates the intervention behavior of large language models (LLMs) when used as educational assistants, introducing a benchmark called INT-BENCH to evaluate their effectiveness in guiding learners. The study reveals that while LLMs can enhance immediate task performance, they often intervene too frequently and too early, providing complete solutions rather than fostering independent reasoning. This behavior contrasts with human tutors, who tend to offer more targeted hints and allow learners to engage more deeply with the problem-solving process. The authors conducted empirical analyses across three domains—code debugging, mathematics, and brain teasers—assessing the impact of LLM interventions on task success and the ability to generalize to new problems. The findings highlight a critical trade-off in AI assistance, suggesting that current LLMs prioritize short-term success over long-term cognitive engagement and learning.
Methodology
The authors developed INT-BENCH, a simulation-based benchmark where a teacher LLM monitors a simulated student's reasoning process. They evaluated LLM interventions based on metrics such as frequency, timing, immediate helpfulness, and generalization to new problems, comparing LLMs with human teachers in similar settings.
Results
The study found that LLMs intervened more frequently and earlier than humans, often providing overly informative feedback that led to immediate task success but limited the student's ability to generalize knowledge to new problems. This behavior suggests a tendency for LLMs to optimize for short-term performance rather than fostering deeper learning.
Implications
The findings indicate that while LLMs can be effective in educational settings, their current intervention strategies may need refinement to better support long-term learning and cognitive engagement. This has implications for the design of AI educational tools, suggesting a need for more nuanced intervention strategies that prioritize learner autonomy and reasoning.
HypNO: A Graph-Based Neural Operator with Physics-Informed Message Passing for Hyperbolic Conservation Laws
Graph Learning
Theory
Efficient ML
- HypNO is a graph-based neural operator specifically designed for hyperbolic conservation laws.
- The model utilizes physics-informed message passing to respect physical properties like upwinding and entropy admissibility.
- HypNO shows strong performance in predicting solutions for complex traffic-flow models, outperforming traditional numerical methods.
- The architecture allows for efficient computation across varying initial conditions, making it suitable for real-time applications.
Read more
HypNO: A Graph-Based Neural Operator with Physics-Informed Message Passing for Hyperbolic Conservation Laws
Summary
The paper introduces HypNO, a novel graph-based neural operator designed specifically for scalar hyperbolic conservation laws. HypNO operates on a space-time graph of finite-volume cells, employing adjacency-factored, physics-informed message passing to ensure compliance with upwinding and entropy admissibility, particularly in the presence of shocks. The authors benchmark HypNO against the Lighthill-Whitham-Richards (LWR) and Aw-Rascle-Zhang (ARZ) traffic-flow models, which are challenging for operator-learning methods due to their complex dynamics involving global transport and shock formation. The results demonstrate that HypNO can accurately predict solution snapshots across various initial conditions while effectively capturing shocks and discontinuities. This work highlights the potential of machine learning to complement traditional numerical methods for solving partial differential equations (PDEs), particularly in scenarios requiring rapid inference across multiple instances of the same governing equations.
Methodology
HypNO employs a graph-based architecture that replaces traditional Fourier operators with causal space-time message passing on a cell graph. The model conditions messages on finite volume interface features, such as flux and characteristic speed, to bias the learning process towards transport along characteristics rather than isotropic smoothing. The architecture is benchmarked against established numerical solvers and state-of-the-art neural operators.
Results
HypNO demonstrates accurate predictions of solution snapshots for the LWR and ARZ models, effectively capturing shocks and discontinuities. The performance of HypNO is compared favorably against traditional numerical methods like WENO, Godunov, and HLL, showcasing its potential as a reliable alternative for solving hyperbolic conservation laws.
Implications
The development of HypNO suggests that graph-based neural operators can significantly enhance the efficiency and accuracy of solving hyperbolic PDEs, particularly in applications requiring rapid inference and real-time predictions. This approach could be beneficial in fields such as traffic flow management, fluid dynamics, and other areas governed by hyperbolic conservation laws.
Semantic-Aware Task Clustering for Constructive and Cooperative Multi-Tasking
Optimization
Federated Learning
Theory
- Introduces a multi-stage optimization problem for CMT-SemCom that integrates semantic-aware clustering.
- Utilizes UMAP and HDBSCAN for task clustering based on semantic features, moving beyond traditional clustering methods.
- Demonstrates significant improvements in task execution performance by mitigating destructive cooperation.
- Addresses the challenges of task grouping in multi-task learning for semantic communication systems.
Read more
Semantic-Aware Task Clustering for Constructive and Cooperative Multi-Tasking
Summary
This paper presents a novel approach to cooperative multi-task semantic communication (CMT-SemCom) by introducing a semantic-aware task clustering method aimed at enhancing task execution performance. The authors argue that cooperative multi-tasking can be either constructive or destructive based on the semantic relationships among tasks. To ensure constructive cooperation, they propose a two-stage optimization framework: first, a semantic clustering phase using hierarchical density-based spatial clustering (HDBSCAN) and uniform manifold approximation and projection (UMAP) to group semantically aligned tasks; second, an end-to-end (E2E) learning phase where training occurs exclusively within these identified groups. The proposed method addresses the limitations of existing multi-task learning approaches that assume all tasks are fully related, which can lead to performance degradation when unrelated tasks are involved. The results from simulations indicate that the proposed framework significantly reduces destructive cooperation and negative transfer, leading to improved accuracy compared to traditional unclustered multi-tasking and individual training methods. This work contributes to the field by providing a structured approach to task grouping that enhances the efficiency and effectiveness of multi-task learning in semantic communication systems.
Methodology
The methodology involves a two-stage optimization process: first, a semantic clustering phase that employs UMAP and HDBSCAN to identify semantically related tasks based on high-dimensional feature outputs from a shared encoder; second, an E2E learning phase where training is conducted within the identified task groups to ensure constructive cooperation.
Results
Simulation results show that the proposed semantic-aware task clustering framework effectively mitigates destructive cooperation and negative transfer, resulting in notable accuracy improvements over unclustered multi-tasking and individual training baselines.
Implications
The findings suggest that the proposed framework can enhance the performance of multi-task learning in intelligent wireless networks, potentially leading to more efficient and effective task-oriented semantic communication systems.
Detecting Neural Network Failures through Spectral Analysis of Internal Activations
Theory
Interpretability
Efficient ML
- Introduction of Spectral Drift as a characteristic failure signature in neural networks.
- Development of the Self-Detecting Neural Networks (SDNN) framework for monitoring internal activations.
- Demonstration of substantial performance improvements over existing confidence-based failure detection methods.
- Empirical validation of the effectiveness of spectral analysis in identifying misclassifications.
Read more
Detecting Neural Network Failures through Spectral Analysis of Internal Activations
Summary
This paper introduces a novel approach for detecting neural network failures by analyzing the internal activations of the network through a method termed Spectral Drift. The author identifies that misclassifications in neural networks exhibit significant spectral instability in their internal activations, which is not visible at the output layer. This instability is quantified as the frequency-domain distance between consecutive layer activations, showing a statistically significant increase in drift during failures compared to correct predictions. To address this issue, the paper proposes the Self-Detecting Neural Networks (SDNN) framework, which utilizes Short-Time Fourier Transform, wavelet decomposition, and statistical moments to monitor spectral dynamics across the network's depth. The SDNN framework is trained using curriculum learning, progressively introducing more challenging examples of misclassifications, distribution shifts, and adversarial perturbations. Experiments conducted on the CIFAR-10 dataset demonstrate that SDNN achieves an average AUROC of 79.0±25.3%, significantly outperforming traditional confidence-based methods such as Max-Softmax and Energy Score, which hover around 50-55%. The findings suggest that spectral analysis of internal activations can provide critical insights into neural network reliability, revealing diagnostic information that is otherwise inaccessible through output-based approaches.
Methodology
The methodology involves monitoring internal activations of neural networks using spectral analysis techniques. The SDNN framework employs Short-Time Fourier Transform and wavelet decomposition to capture multi-scale spectral features, alongside statistical moments to summarize activation distributions. A bidirectional GRU is used to model temporal dependencies across layers. The framework is trained using curriculum learning, which gradually introduces more complex failure scenarios.
Results
The SDNN framework achieved an average AUROC of 79.0±25.3% across three random seeds on the CIFAR-10 dataset, significantly outperforming confidence-based methods like Max-Softmax (50.5%) and Energy Score (52.9%). Ablation studies indicated that wavelet decomposition and statistical features consistently contributed to performance, while the role of STFT was less clear.
Implications
The findings suggest that monitoring internal activations through spectral analysis can enhance the reliability of neural networks, particularly in critical applications such as autonomous driving and medical diagnosis, where misclassifications can have severe consequences. This approach may lead to the development of more robust models capable of detecting failures before they manifest in output predictions.
Conflict Resolution under Degraded Surveillance in Air Corridors Using Multi-Agent Reinforcement Learning
Reinforcement Learning
Robotics
- Developed a DQN-based MARL framework for conflict resolution in AAM under degraded surveillance.
- Trained separate policies for UAVs and eVTOLs using a 14-action maneuvering space.
- Evaluated the framework across 90 traffic density and separation threshold combinations.
- Identified trade-offs between safety and corridor capacity in conflict resolution.
Read more
Conflict Resolution under Degraded Surveillance in Air Corridors Using Multi-Agent Reinforcement Learning
Summary
This study addresses the challenge of maintaining safe separation between aircraft in advanced air mobility (AAM) operations under degraded surveillance conditions, where information may be noisy, delayed, or incomplete. The authors develop a Deep Q-Network (DQN)-based Multi-Agent Reinforcement Learning (MARL) framework for decentralized conflict resolution among heterogeneous small unmanned aerial vehicles (UAVs) and electric vertical takeoff and landing (eVTOL) aircraft within a structured three-dimensional corridor. The framework trains separate policies for the two aircraft categories using local observations and a 14-action space that includes maneuvers such as maintaining course, turning, vertical maneuvering, landing, and speed control. The simulation incorporates various factors affecting aircraft dynamics, including energy use, corridor constraints, observation noise, communication delays, and uncertainties. The trained policies are evaluated across 90 combinations of traffic density and minimum separation thresholds, revealing that loss-of-separation frequency and duration tend to increase with traffic density and separation requirements, although most conflicts are resolved within 1 second. The study finds that under safe conditions, agents maintain their motion approximately 79% of the time, with turning maneuvers accounting for 33% of actions during conflicts. The results highlight trade-offs between safety and corridor capacity, providing a framework for simulating safer AAM conflict-resolution strategies under challenging surveillance conditions.
Methodology
The authors implemented a Deep Q-Network (DQN) based Multi-Agent Reinforcement Learning (MARL) model to simulate decentralized conflict resolution among heterogeneous aircraft. The model incorporates local observations, a defined action space, and various uncertainties related to surveillance and aircraft dynamics. The simulation environment generates diverse traffic scenarios to evaluate the trained policies.
Results
The evaluation revealed that loss-of-separation frequency and duration generally increase with traffic density and separation requirements. Most conflict events were resolved within 1 second, with agents maintaining their motion approximately 79% of the time. The analysis of six Pareto-optimal configurations illustrated the trade-offs between safety and corridor capacity.
Implications
The findings suggest that the proposed MARL framework can enhance the safety and efficiency of AAM operations under degraded surveillance conditions. This approach could be applied to improve air traffic management systems and inform the design of autonomous conflict resolution capabilities for future air mobility solutions.
Adaptive Depth in Looped Transformers: Diagnosing Learned Halting Gates and Trajectory Readouts
Efficient ML
Theory
Large Language Models
- Introduces a trajectory–readout perspective on adaptive depth in looped Transformers.
- Demonstrates that fixed-prior depth supervision can create effective, difficulty-aware trajectories.
- Shows that simple post-hoc confidence readouts can match or outperform learned gates.
- Identifies that issues in adaptive compute performance often stem from trajectory formation rather than gate expressivity.
Read more
Adaptive Depth in Looped Transformers: Diagnosing Learned Halting Gates and Trajectory Readouts
Summary
This paper investigates the adaptive depth in looped Transformers, focusing on the interaction between learned halting gates and trajectory readouts. The authors highlight that traditional learned halting objectives entangle the roles of exit selection and trajectory formation, complicating the diagnosis of adaptive computation performance. Through systematic evaluations on synthetic tasks and large-scale models, they propose a trajectory–readout perspective, demonstrating that fixed-prior depth supervision can yield effective trajectories that enhance stopping signals. Their findings indicate that simple post-hoc confidence readouts can outperform complex learned gates, suggesting that future adaptive-depth methods should treat trajectory formation and exit selection as distinct components. The study emphasizes the need for a nuanced understanding of adaptive depth in looped Transformers, reframing it as a joint problem rather than solely a gate-learning challenge.
Methodology
The authors conducted a systematic diagnostic evaluation of adaptive depth in looped Transformers by varying training objectives, gate parameterizations, post-hoc readouts, and forced exits. They applied these methods to controlled synthetic tasks (modular arithmetic and binary parity) and evaluated pretrained models (Ouro-1.4B and 2.6B) to analyze the effects on trajectory formation and exit selection.
Results
The study found that fixed-prior depth supervision leads to difficulty-aware trajectories that enhance stopping signals. Simple confidence readouts often matched or outperformed learned gates in terms of compute-accuracy tradeoff. Additionally, pretrained ponder gates were competitive but not uniformly optimal, and reductions in average exit depth correlated with practical inference-time speedups.
Implications
The findings suggest that future research in adaptive computation should focus on decoupling trajectory formation from exit selection, potentially leading to more efficient and effective models. This could have significant implications for the design of looped Transformers and adaptive computation strategies in various applications.
Beyond Heavy Log Curation: Perplexity-Based APT Detection via Unsupervised, Context-Augmented Language Models
NLP
Large Language Models
Time Series
- CAPTAIN reduces reliance on heavily curated training data and complex preprocessing pipelines.
- The method incorporates temporal context into the detection process, improving robustness.
- CAPTAIN's perplexity-based scoring mechanism effectively identifies suspicious log entries.
- The approach shows competitive performance against existing APT detection baselines.
Read more
Beyond Heavy Log Curation: Perplexity-Based APT Detection via Unsupervised, Context-Augmented Language Models
Summary
The paper addresses the challenge of detecting Advanced Persistent Threats (APTs) in large-scale logs, which are often dominated by benign events. Traditional machine learning methods require extensive log curation and preprocessing, which can be costly and require domain expertise. The authors propose a novel approach called CAPTAIN (Context-Augmented Perplexity-based Threat Activity log detectIoN), which utilizes general, pre-trained language models with minimal preprocessing. CAPTAIN enhances the detection of APTs by encoding recent log history and incorporating it into a perplexity-based scoring mechanism. This method allows for robust scoring of long log entries while reducing the need for heavy preprocessing. The authors conducted a replication study of a strong baseline method, AIRTAG, and identified limitations such as performance degradation with long log entries and the inability to model context across log events. CAPTAIN addresses these limitations by leveraging temporal context and applying smoothing filters to the perplexity time series. The results demonstrate that CAPTAIN competes effectively with existing methods while requiring less curated input, thus lowering operational costs and improving detection capabilities.
Methodology
CAPTAIN employs a context-augmented approach using pre-trained language models. It encodes recent log history with an encoder model and integrates this context into a decoder input, allowing for perplexity scores that reflect temporal dependencies. Smoothing filters are applied to the perplexity time series to enhance stability in detection.
Results
CAPTAIN demonstrated competitive performance on APT-oriented benchmarks, showing robustness under less curated inputs compared to existing methods. The results indicate that CAPTAIN can effectively detect APTs while minimizing the need for extensive log preprocessing.
Implications
The findings suggest that CAPTAIN can be applied in operational environments where rapid detection of APTs is critical, reducing the burden on analysts and improving the scalability of threat detection systems.
Double-Scoring: Reliable Extraction of Strong Lottery Tickets
Theory
Optimization
- Identified layerwise sparsity selection as a bottleneck in strong ticket extraction.
- Introduced double-scoring, which simplifies the extraction process by removing the need for layerwise sparsity tuning.
- Proved that augmented score-space masking preserves access to original masks.
- Demonstrated substantial improvements in strong-ticket extraction through empirical experiments.
Read more
Double-Scoring: Reliable Extraction of Strong Lottery Tickets
Summary
This paper addresses the challenge of reliably extracting strong lottery tickets from overparameterized neural networks, as proposed by the lottery ticket hypothesis. The authors identify a significant bottleneck in existing methods, particularly in the layerwise sparsity selection process of the edge-popup algorithm. To overcome this limitation, they introduce a novel approach called double-scoring, which utilizes an augmented score-space parameterization. This method eliminates the need for pre-defined sparsity levels for each layer, allowing for more effective extraction of strong lottery tickets. The authors provide theoretical proof that this new parameterization maintains access to all original-coordinate masks and can be interpreted as an edge-popup method applied to a zero-augmented network. Through controlled experiments on FashionMNIST MLP, they demonstrate that double-scoring significantly outperforms traditional edge-popup and other baseline methods, while also showing lower sensitivity to hyperparameters. The findings suggest that the improvements are due to the competitive nature of the augmented score-space rather than merely the addition of trainable parameters.
Methodology
The authors revisited the edge-popup algorithm for extracting strong lottery tickets and proposed double-scoring, which augments the score-space parameterization. This approach allows for optimization over enlarged score tensors instead of requiring fixed layerwise sparsity levels. Theoretical proofs support the efficacy of this method, and controlled experiments validate its performance against existing techniques.
Results
In experiments conducted on FashionMNIST MLP, double-scoring significantly improved the extraction of strong lottery tickets compared to fixed-density edge-popup, random masks, SNIP, and GraSP. It also remained competitive with rewound sparse-training topologies, demonstrating lower sensitivity to sparsity hyperparameters. Ablation studies confirmed that the performance gains were linked to the augmented score-space competition.
Implications
The findings suggest that double-scoring could enhance the reliability and efficiency of extracting strong lottery tickets in neural networks, potentially leading to better model performance and reduced computational costs in training sparse networks.