AI-generated summaries
Today's ML research,
without the noise.
Daily summaries of the latest machine learning papers from arXiv, processed every 8 hours.
50
Papers today
8h
Update frequency
7
Days of history
Cross-Modal Generative Framework for Signal Translation from Fetal-Maternal Electrocardiograms to Fetal Doppler Waveforms
Generative Models
Multimodal
Time Series
- Introduction of a cross-modal generative framework for synthesizing fetal Doppler waveforms from fECG.
- Demonstration of the importance of selective attention to maternal ECG for improved Doppler reconstruction.
- Development of a composite loss function that balances pointwise error, derivative error, and correlation for waveform accuracy.
- Significant reduction in PSD MSE and heart-rate error compared to baseline methods, indicating improved model performance.
Read more
Cross-Modal Generative Framework for Signal Translation from Fetal-Maternal Electrocardiograms to Fetal Doppler Waveforms
Summary
This paper presents a novel cross-modal generative framework designed to translate fetal-maternal electrocardiograms (fECG) into fetal Doppler waveforms, addressing the complementary nature of these signals in assessing fetal cardiovascular function. The authors highlight the importance of understanding both electrical and mechanical factors in fetal circulation, particularly through maternal-fetal cardiac coupling. The proposed framework employs dilated convolutions combined with cross-modal attention to selectively integrate maternal ECG data, while self-attention mechanisms capture long-range temporal dependencies within the fetal signals. The model was trained on 885 synchronized segments from 39 pregnancies, achieving a significant reduction in power spectral density mean squared error (PSD MSE) and heart-rate error compared to baseline methods. The findings suggest that the selective attention to maternal signals enhances Doppler reconstruction, allowing for a more nuanced understanding of the contributions of electrical and mechanical factors in fetal health assessment. This work advances computational modeling in maternal-fetal cardiovascular systems and provides a framework for further exploration of fetal health monitoring.
Methodology
The authors developed a cross-modal generative framework that integrates dilated convolutions with cross-modal attention and self-attention mechanisms. This architecture allows for the selective incorporation of maternal ECG data while capturing long-range dependencies in fetal ECG signals. The model was trained using a dataset of synchronized fetal and maternal ECG and Doppler segments.
Results
The model achieved a PSD MSE of 49.9Β±15.8 dBΒ², which is 51% lower than the two-channel baseline, and a heart-rate error of 4.71Β±0.77 bpm, representing a 1.5% improvement over the baseline. The use of cross-modal attention resulted in a 39% reduction in PSD MSE compared to naive dual-channel concatenation.
Implications
This framework enhances the understanding of maternal-fetal cardiovascular interactions and provides a computational tool for fetal health monitoring. It could lead to improved accessibility of fetal assessments in resource-constrained settings by leveraging inexpensive fECG data to infer mechanical hemodynamic information typically obtained through costly Doppler ultrasound.
Collate: Collaborative Neural Network Learning for Latency-Critical Edge Systems
Federated Learning
Efficient ML
Optimization
- Collate framework allows collaborative learning of heterogeneous models for edge systems.
- Dynamic zeroizing-recovering method adjusts local model architectures to meet latency constraints.
- Proto-corrected aggregation scheme effectively combines models from different edge devices.
- Improvements in accuracy of 1.96% and 3.09% for extended and shrunk models, respectively.
Read more
Collate: Collaborative Neural Network Learning for Latency-Critical Edge Systems
Summary
The paper introduces Collate, a novel framework designed for Federated Learning (FL) that addresses the challenges of deploying machine learning models in latency-critical edge systems. Traditional FL methods focus on improving training efficiency but often overlook inference latency, which is crucial for real-time applications. Collate enables the collaborative learning of heterogeneous models tailored to the specific latency constraints of different edge devices. The authors propose a dynamic zeroizing-recovering method to adjust local model architectures for optimal accuracy while adhering to latency limits. Additionally, a proto-corrected federated aggregation scheme is introduced to effectively combine these heterogeneous models, ensuring high accuracy across diverse systems with a single training process. Experimental results demonstrate that Collate can improve accuracy by an average of 1.96% for extended models and 3.09% for shrunk models under latency constraints, with minimal additional training overhead. This work represents a significant advancement in optimizing FL for edge systems, balancing the trade-off between latency and accuracy.
Methodology
The methodology involves a collaborative training framework that utilizes a dynamic zeroizing-recovering approach to modify local model architectures based on latency requirements. A proto-corrected federated aggregation scheme is employed to merge the outputs of heterogeneous models from various edge systems, ensuring that the final model meets the latency constraints while maintaining high accuracy.
Results
The experiments conducted show that Collate achieves an average accuracy improvement of 1.96% for models extended to meet higher latency constraints and 3.09% for models shrunk for lower latency systems, all while incurring almost no extra training overhead compared to state-of-the-art methods.
Implications
The findings suggest that Collate can be effectively applied in real-time edge applications, such as autonomous vehicles and healthcare devices, where both latency and accuracy are critical. This framework could enhance the deployment of Federated Learning in various industries, ensuring compliance with data privacy regulations while optimizing model performance.
Structure Learning on Clustered Data
Graph Learning
Optimization
Theory
- Introduces a new framework for learning DAGs that accounts for cluster-specific variations.
- Extends classical mixed models to structure learning, ensuring acyclicity in the combined graph.
- Develops a first-order optimization method with provable convergence for scalable learning.
- Establishes statistical identifiability and asymptotic recovery of the true structure.
Read more
Structure Learning on Clustered Data
Summary
This paper addresses the limitations of existing directed acyclic graph (DAG) structure learning methods that assume a homogeneous population, which is often not the case in clustered data. The authors propose a novel approach that combines global structure estimation with local cluster-level effects by extending the fixed- and random-effects framework of classical mixed models to the structure learning context. They introduce a differentiable graph coupling mechanism to ensure that the union of fixed- and random-effects graphs remains acyclic. The computational aspect involves a first-order optimization method with proven convergence and efficient batched updates across clusters. The authors establish the identifiability of their model and demonstrate that it can recover the true structure asymptotically. Through experiments on both synthetic and real data, the proposed method outperforms alternative estimators by detecting dependencies that are typically missed, highlighting its effectiveness for structure learning in clustered settings.
Methodology
The authors develop a mixed DAG framework where the overall structure is represented as a combination of fixed and random effects. They utilize a differentiable graph coupling mechanism to maintain acyclicity while estimating the structure. A first-order optimization method is employed for efficient computation, allowing for batched updates across clusters.
Results
The proposed method successfully recovers both population-level and cluster-specific effects, outperforming alternative approaches in detecting dependencies in synthetic and real datasets, particularly in applications such as proteomics.
Implications
This work has significant implications for causal discovery in heterogeneous populations, particularly in fields like epidemiology and personalized medicine, where understanding cluster-specific effects is crucial.
Contravariance Theory: Strong Alignment for Minimal Solutions to Hard Tasks
Theory
- Weak alignment of DNN representations can guarantee strong alignment of privileged axes.
- Hierarchical alignment in DNNs leads to the emergence of privileged axes through task optimization.
- The choice of metric for inter-network comparison is less sensitive with strong tasks.
- Convergent evolution between artificial networks and biological systems is likely.
Read more
Contravariance Theory: Strong Alignment for Minimal Solutions to Hard Tasks
Summary
This paper explores the relationship between deep neural networks (DNNs) and brain function through the lens of Contravariance Theory. The authors present two main results: first, that weak alignment of network representations via affine mappings leads to strong alignment of privileged axes; and second, that alignment occurs hierarchically within the network, resulting in the emergence of privileged axes through end-to-end task optimization. These findings suggest that the choice of metric for comparing networks is less sensitive when tasks are sufficiently challenging, and that convergent evolution between artificial and biological networks is likely. The research builds on previous NeuroAI studies, which have shown that DNNs can effectively model brain activity across various cognitive domains. The authors argue that the observed similarities between DNNs and brain networks are not merely coincidental but arise from shared optimization constraints, indicating a deeper conceptual alignment between artificial and biological systems.
Methodology
The authors utilize a theoretical framework based on Contravariance Theory, supported by empirical observations from previous NeuroAI research. They analyze the alignment of DNN representations with brain data through linear mapping techniques and examine the hierarchical structure of DNNs in relation to neuroanatomical consistency.
Results
The study demonstrates that for minimal DNN solutions to hard tasks, weak alignment leads to strong alignment of privileged axes, and that hierarchical alignment occurs, resulting in the emergence of these axes through end-to-end optimization. The findings indicate that the choice of metrics for comparing DNNs is robust under strong task constraints, and that convergent evolution is a probable outcome.
Implications
These results have significant implications for the field of NeuroAI, suggesting that DNNs can be used to create more accurate models of brain function. This could lead to advancements in understanding cognitive processes and developing neural control systems. Additionally, the findings may inform the design of future artificial intelligence systems that more closely mimic biological intelligence.
Write-Protected Discrete Bottlenecks for Language-Grounded World Models: A Structural Limitation and Sufficient Fix
Robotics
NLP
Multimodal
- Language gradients entering discrete bottlenecks lead to a structural trade-off affecting learning and diversity.
- A three-layer architectural fix is proposed to address the identified limitations in language-grounded world models.
- The proposed architecture achieves high semantic grounding accuracy while being computationally efficient.
- The findings challenge the end-to-end scaling paradigm in embodied AI, emphasizing the need for architectural separation.
Read more
Write-Protected Discrete Bottlenecks for Language-Grounded World Models: A Structural Limitation and Sufficient Fix
Summary
This paper investigates the integration of language with discrete symbol systems in robot world models, challenging the prevalent end-to-end approach that assumes language gradients can directly influence physical symbol representations. The author identifies a structural limitation in this paradigm, where language gradients entering a discrete bottleneck lead to a trade-off: either the Gumbel-softmax estimator collapses to a minimal number of symbols or anti-collapse strategies maintain diversity but fail to learn semantic labels effectively. The proposed solution consists of a three-layer architectural fix: (1) cutting the gradient chain to prevent language signals from reaching the symbol bottleneck, (2) introducing a gradient-free semantic channel using a non-parametric Memory Table for co-occurrence counting, and (3) employing DP-Means streaming clustering to resolve symbol collisions. This combined approach achieves a grounding accuracy of 97.2%, significantly outperforming the baseline accuracy of 22.2% without the third layer. The findings are validated across multiple encoder architectures and environments, demonstrating that the proposed architecture can generalize effectively while requiring fewer than 2 million parameters and no fine-tuning of large language models. The results challenge the assumption that larger LLMs inherently improve physical grounding, suggesting that the architecture must separate physical perception from language processing.
Methodology
The study employs empirical experiments to test the structural limitations of existing end-to-end approaches in language-grounded world models. It introduces a three-layer fix consisting of gradient cutting, a gradient-free semantic channel, and collision resolution, validating the effectiveness of each layer through causal ablation and testing across various encoder architectures and environments.
Results
The proposed three-layer architecture achieved a grounding accuracy of 97.2% compared to 22.2% without the third layer. The experiments demonstrated zero symbol collapse across 74 independent runs, with the architecture generalizing effectively across different conditions and requiring fewer than 2 million parameters.
Implications
The findings suggest a need for re-evaluating the integration of language and perception in robotic systems, advocating for architectural designs that separate these functions. This could lead to more robust and efficient models in embodied AI, enhancing the understanding and interaction of robots with their environments.
When Thinking Hurts: Epistemic Signals in the Reasoning Chains of Visual Language Models
Multimodal
Large Language Models
Interpretability
- First empirical characterization of answer entropy behavior in thinking-mode VLMs across three model families.
- Demonstrated that thinking chain entropy is a superior predictor compared to answer entropy.
- Identified structured abstention affecting a significant percentage of queries.
- Proposed a practical abstention gate that enhances accuracy without extra inference costs.
Read more
When Thinking Hurts: Epistemic Signals in the Reasoning Chains of Visual Language Models
Summary
This paper investigates the behavior of answer entropy in thinking-mode Visual Language Models (VLMs), which generate reasoning chains before committing to answers. The author characterizes the entropy behavior across three different model families, revealing distinct patterns: complete collapse in Qwen3-VL-8B-Thinking, no collapse in GLM-4.1V-9B-Thinking, and selective thinking in InternVL3-8B. The study finds that thinking chain entropy consistently outperforms answer entropy, suggesting that chain signals are a more reliable predictor of performance. Additionally, the paper documents structured abstention in responses and proposes a practical abstention gate that significantly improves accuracy without additional inference costs. The findings highlight the importance of understanding the implications of reasoning chains in VLMs for uncertainty quantification and model deployment in real-world applications.
Methodology
The study employed controlled ablation experiments comparing thinking-mode VLMs with non-thinking counterparts on identical adversarial samples. It analyzed answer and thinking chain entropy using Shannon entropy metrics derived from a single forward pass through the models. The research also included pilot studies on VQAv2 and HallusionBench benchmarks to validate findings.
Results
The results showed that Qwen3-VL-8B-Thinking exhibited a complete collapse in answer entropy (AUROC = 0.492), while GLM-4.1V-9B-Thinking maintained a higher performance (AUROC = 0.716). InternVL3-8B demonstrated selective thinking with a 50% chain rate. Chain entropy outperformed answer entropy across models, particularly in challenging reasoning tasks, and the proposed abstention gate improved accuracy from 71.0% to 93.8% at 62.7% coverage.
Implications
The findings suggest that understanding the dynamics of reasoning chains in VLMs is crucial for improving uncertainty quantification methods. The proposed abstention gate could be applied in practical scenarios where reliable predictions are necessary, enhancing the deployment of VLMs in real-world applications.
Graph-Regularized Deep Learning for EEG-Based Emotion Recognition with Psychologically-Grounded Label Structure
Graph Learning
Time Series
Audio & Speech
- Introduces a novel graph-based regularization approach for EEG emotion recognition.
- Incorporates psychological proximity into training objectives to improve classification accuracy.
- Demonstrates architecture-agnostic benefits across multiple deep learning frameworks.
- Achieves up to +5.42% accuracy improvement and 39% reduction in implausible misclassifications.
Read more
Graph-Regularized Deep Learning for EEG-Based Emotion Recognition with Psychologically-Grounded Label Structure
Summary
This paper addresses the challenge of EEG-based emotion recognition, which is crucial for mental health monitoring and brain-computer interfaces. Traditional deep learning methods treat emotion classes as isolated labels, neglecting their psychological interdependencies. The authors propose a graph-regularized learning framework that models emotions as nodes in a graph, with edges representing psychological proximity based on dimensional emotion theories. They introduce three complementary regularization strategies: Graph Label Smoothing, Graph Laplacian-based commuting distance, and Sliced Wasserstein Distance, which penalize misclassifications according to the established emotional topology. The framework is evaluated using three backbone architectures: AudioTransformer, Conformer, and DCGNN, demonstrating architecture-agnostic improvements. Experiments on SEED-IV and SEED-V datasets show significant enhancements in classification accuracy and reductions in implausible misclassifications, indicating that the proposed method effectively integrates psychological insights into emotion recognition tasks.
Methodology
The authors construct an emotion graph based on Russell's circumplex model, positioning emotions in a valence-arousal space. They implement three regularization strategies: Graph Label Smoothing for local relationships, Graph Laplacian for global connectivity, and Sliced Wasserstein Distance for distributional transformation. These strategies guide the model to prioritize errors based on psychological proximity, allowing for more nuanced emotion classification.
Results
The proposed framework shows consistent improvements in emotion recognition accuracy across the SEED-IV and SEED-V datasets, with the best case achieving a +5.42% increase in accuracy and a 39% reduction in psychologically implausible misclassifications. The results validate the effectiveness of integrating psychological structures into deep learning models for emotion recognition.
Implications
This research has significant implications for clinical applications in mental health monitoring and affective computing, as it enhances the accuracy and relevance of emotion recognition systems. The framework could be utilized in developing more effective brain-computer interfaces and consumer-grade EEG devices, improving user experience in emotion-sensitive applications.
Evaluating the Generalizability of Foundation Models for Extreme Environmental Events: Case Study of California Wildfire PM2.5
Time Series
- First systematic benchmark of time series foundation models for extreme wildfire PM2.5 forecasting.
- Fully-trained BiLSTM models consistently outperform TSFMs in all evaluation metrics.
- Zero-shot TSFMs improve over naive persistence but struggle with extreme out-of-distribution conditions.
- LoRA fine-tuning enhances performance but does not surpass fully-trained baselines.
Read more
Evaluating the Generalizability of Foundation Models for Extreme Environmental Events: Case Study of California Wildfire PM2.5
Summary
This paper addresses the challenge of accurately forecasting hazardous PM2.5 concentrations from wildfire smoke, which pose significant public health risks. The authors systematically benchmark six time series foundation models (TSFMs) against fully-trained deep learning baselines using a 12-year dataset of PM2.5 levels from 1,375 wildfire incidents across 79 monitoring sites in California. The study employs a leave-one-incident-out (LOIO) cross-validation protocol to evaluate model generalization to unseen fire events. The models are assessed using metrics such as Mean Absolute Error (MAE), Root Mean Square Error (RMSE), and exceedance F1 scores at various U.S. Environmental Protection Agency (EPA) air quality index (AQI) thresholds. Results indicate that while zero-shot TSFMs show some improvement over naive persistence methods, fully-trained BiLSTM models outperform all TSFMs across all metrics, particularly in predicting extreme PM2.5 levels. The findings challenge the assumption that larger pretrained models are superior for environmental forecasting, providing critical insights for deploying models in wildfire air quality prediction.
Methodology
The authors benchmarked six TSFM configurations (including zero-shot and fine-tuned variants) against fully-trained deep learning models (LSTM, BiLSTM, Transformer) using a 12-year PM2.5 dataset. They employed a leave-one-incident-out cross-validation protocol to evaluate model performance on unseen wildfire events, assessing metrics such as MAE, RMSE, and exceedance F1 across different AQI thresholds and forecast horizons.
Results
The fully-trained BiLSTM achieved the lowest MAE (5.16 Β΅g/m3) and the highest exceedance F1 scores across all AQI thresholds, outperforming TSFMs, which showed modest improvements over naive persistence. Zero-shot TSFMs exhibited severe RMSE instability, particularly Chronos-2, while LoRA fine-tuning improved performance but did not exceed the trained baselines.
Implications
The findings suggest that while TSFMs have potential for air quality forecasting, fully-trained models remain more effective for extreme environmental events like wildfires. This research provides actionable insights for public health decision-making regarding wildfire smoke and air quality management.
CAAD: Causality-Aware Multivariate Time Series Anomaly Detection via Multi-Scale Alignment and Structural Causal Consistency
Time Series
- CAAD introduces a causality-aware approach to anomaly detection, focusing on causal consistency rather than just temporal similarities.
- The framework employs multi-scale temporal alignment to capture both fine-grained dynamics and coarse-grained trends.
- Continuous causal verification is achieved through gradient-based Granger signals, allowing real-time detection of causal breakdowns.
- Dual-perspective anomaly scoring combines dynamic and relational causal scores to enhance sensitivity to stealthy anomalies.
Read more
CAAD: Causality-Aware Multivariate Time Series Anomaly Detection via Multi-Scale Alignment and Structural Causal Consistency
Summary
The paper presents CAAD, a novel framework for anomaly detection in multivariate time series data, particularly in complex industrial systems. Traditional methods often focus on temporal similarities and magnitude-level deviations, neglecting the internal causal relationships that can indicate system failures. CAAD reframes anomaly detection as a continuous verification of Granger causality consistency, modeling exogenous variables as residuals to identify anomalies caused by external interventions. The framework incorporates multi-scale alignment to capture system dynamics and employs a gradient-based matrix to monitor causal relationship breakdowns. By quantifying causal deviations in both dynamic evolution and relational topology, CAAD effectively detects subtle causal shifts. Experiments on real-world datasets demonstrate that CAAD achieves high precision in anomaly detection, outperforming existing state-of-the-art methods, particularly in scenarios with strong physical constraints and subtle failure modes.
Methodology
CAAD utilizes a hierarchical temporal modeling architecture for multi-scale alignment, gradient-based Granger causality signals for continuous causal verification, and a dual-perspective scoring system to quantify anomalies from both temporal and structural perspectives.
Results
The CAAD framework was tested on multiple real-world datasets, including SWaT and PSM, achieving an F1 score of 0.95 on the SWaT dataset, which is significantly higher than existing correlation and reconstruction-based methods.
Implications
The findings suggest that integrating multi-scale causal awareness into anomaly detection can enhance the reliability of monitoring complex industrial systems, potentially leading to improved safety and operational integrity.
ArtMine: Discovering and Formalizing Artistic Processes
Generative Models
Multimodal
Theory
- Introduces artistic process discovery as a computational problem for generative AI.
- Proposes ArtMine, integrating evidence construction, abductive reasoning, and self-reflection for artistic process reconstruction.
- Demonstrates the ability to generate coherent production trajectories from heterogeneous historical evidence.
Read more
ArtMine: Discovering and Formalizing Artistic Processes
Summary
The paper presents ArtMine, a novel framework aimed at discovering and formalizing artistic processes from heterogeneous historical evidence. Unlike existing generative AI systems that focus on modeling completed artworks, ArtMine emphasizes the iterative decisions and contextual influences that shape artistic production. The framework synthesizes fragmented documentary evidence, such as archival records and preparatory studies, into a structured repository. A Peircean abductive agent then infers evidence-grounded production steps, which are represented as a compositional graph. The generated steps are optimized through self-reflection, comparing generated artworks with reference pieces to improve accuracy. A preliminary case study demonstrates the effectiveness of ArtMine in reconstructing coherent and interpretable artistic workflows from diverse sources. This work aims to advance human-AI co-creativity by providing insights into the creative process, thereby supporting artistic interpretation, education, and cultural studies.
Methodology
ArtMine employs a deep-research agent to organize unstructured evidence into a structured repository. It utilizes Peircean abductive reasoning to infer plausible artistic processes and employs self-reflection to optimize the generated outputs against reference artworks.
Results
The case study showcases that ArtMine can effectively reconstruct artistic workflows from fragmented documentary evidence, producing coherent and auditable representations of creative processes.
Implications
ArtMine has the potential to enhance human-AI collaboration in creative fields, facilitate artistic education, and contribute to computational studies of cultural production by providing a deeper understanding of artistic processes.
Stop Guessing When to Stop Testing: Efficient Model Evaluation with Just Enough Data
Efficient ML
Theory
- Fixed-size benchmarks are inefficient for diverse model evaluation needs.
- The proposed adaptive evaluation framework uses sequential testing to optimize evaluation efficiency and reliability.
- The framework allows users to define stopping criteria based on their specific evaluation objectives.
- Empirical results show significant cost savings while maintaining statistical significance.
Read more
Stop Guessing When to Stop Testing: Efficient Model Evaluation with Just Enough Data
Summary
This paper addresses the inefficiencies of fixed-size benchmarks in model evaluation, which often lead to excessive computational costs or unreliable results due to their rigidity. The authors propose an adaptive evaluation framework that utilizes sequential testing to allow for dynamic stopping criteria based on specific evaluation needs, such as detecting diminishing returns or achieving a minimum detectable effect size. This approach enables practitioners to balance the trade-off between evaluation efficiency and statistical reliability. The framework was tested on the Open VLM Leaderboard, demonstrating an 80% reduction in computational costs while maintaining statistical significance. The authors argue that adaptive evaluation not only improves efficiency but also enhances transparency in decision-making regarding model evaluations, allowing users to understand the implications of prioritizing efficiency over statistical power.
Methodology
The authors developed an adaptive evaluation framework that integrates sequential testing principles with user-defined stopping criteria. This framework allows for early termination of evaluations based on statistical needs, such as achieving a desired confidence interval width or detecting diminishing returns, rather than adhering to a fixed sample size.
Results
The adaptive evaluation framework demonstrated an 80% reduction in computational costs compared to traditional fixed-size evaluations while maintaining a confidence interval width of Β±2.5 points. The framework was validated through various use cases, showing its effectiveness in managing the efficiency-reliability trade-off across different evaluation scenarios.
Implications
The proposed framework has significant implications for model evaluation practices in machine learning, particularly in resource-constrained environments. It allows for more efficient use of computational resources, reduces unnecessary evaluations, and provides clearer insights into the reliability of model comparisons. This could lead to faster iterations in model development and deployment, ultimately enhancing the overall efficiency of machine learning workflows.
LLT: Local Linear Transformer for PDE Operator Learning
Efficient ML
Theory
Optimization
- LLT combines linear global attention with local spatial mixing to improve PDE operator learning.
- The architecture incorporates coordinate and geometry information to enhance performance.
- LLT demonstrates competitive accuracy and significantly reduced training time compared to existing methods.
- The model is scalable to large unstructured meshes, as evidenced by its application to a car aerodynamics dataset.
Read more
LLT: Local Linear Transformer for PDE Operator Learning
Summary
The paper introduces the Local Linear Transformer (LLT), a novel architecture designed for learning partial differential equation (PDE) solution maps. Traditional transformer models face challenges in PDE applications due to their quadratic scaling with the number of computational nodes and a lack of emphasis on local interactions. LLT addresses these limitations by integrating linear global attention with local spatial mixing, while also incorporating coordinate and geometry information. The model is evaluated on various PDE problems, including elasticity, plasticity, airfoil flow, pipe flow, and Darcy flow, using reference data generated from different discretization methods. The results demonstrate that LLT achieves competitive or lower relative L2 error compared to existing neural operator and transformer baselines, while significantly reducing training time per iteration. Additionally, LLT is successfully applied to a three-dimensional car aerodynamics dataset, showcasing its scalability to large unstructured meshes. Overall, LLT proves to be an accurate and computationally efficient operator for a wide range of PDE problems.
Methodology
LLT employs kernelized linear attention for global communication and a local mixing path for spatial neighborhoods. It includes coordinate encodings, a distance-to-reference-grid encoding, and a skip-connected decoder to enhance local interactions and reduce attention costs.
Results
LLT achieves competitive or lower relative L2 error across multiple PDE problems compared to other neural operator methods. It also reduces wall-clock time per training iteration by factors of 1.8 to 2.5 relative to the Transolver model, demonstrating improved computational efficiency.
Implications
The LLT model has the potential to accelerate numerical simulations in scientific computing, particularly in fields requiring repeated solutions of PDEs with varying parameters. Its efficiency and accuracy make it suitable for real-time applications in engineering and physics.
Efficient Safety Alignment of Language Models via Latent Personality Traits
NLP
Large Language Models
Efficient ML
- Introduction of Latent Personality Alignment (LPA) as a novel method for safety alignment in LLMs.
- LPA achieves near-zero attack success rates on harmful prompts while preserving model utility.
- The training process is lightweight, requiring significantly fewer examples compared to traditional methods.
- Extensive ablation studies clarify the factors contributing to LPA's robustness and efficiency.
Read more
Efficient Safety Alignment of Language Models via Latent Personality Traits
Summary
This paper addresses the vulnerabilities of large language models (LLMs) to adversarial attacks, particularly focusing on the limitations of current safety methods that rely on explicit supervision over harmful content. The authors introduce a novel approach called Latent Personality Alignment (LPA), which utilizes latent adversarial training on a compact set of 66 harm-agnostic statements derived from psychometric personality literature. The hypothesis is that personality-anchored representations can implicitly constrain the subspace exploited by jailbreak attacks. LPA demonstrates remarkable effectiveness, achieving near-zero attack success rates on the HarmBench dataset across various jailbreak methods without ever being exposed to harmful content during training. Importantly, LPA maintains performance on standard benchmarks while being significantly more efficient, completing training in minutes on a single GPU and using 75 times fewer examples than traditional Latent Adversarial Training (LAT). The authors provide extensive ablation studies to clarify the contributions of system prompts, trait selection, and data composition to the robustness and utility of the method. Overall, LPA presents a promising direction for enhancing the safety alignment of LLMs without compromising their utility.
Methodology
The authors propose Latent Personality Alignment (LPA), which employs latent adversarial training on 66 harm-agnostic psychometric statements instead of explicit harmful content. This method combines the generalizability of activation steering with the robustness of adversarial training, leveraging the nonlinearity of latent space to enhance safety without the need for extensive datasets of harmful prompts.
Results
LPA achieved near-zero attack success rates on the HarmBench dataset across various jailbreak methods while maintaining performance on standard benchmarks. The training was completed in minutes on a single GPU, utilizing 75 times fewer examples than traditional LAT methods, demonstrating both efficiency and effectiveness.
Implications
The findings suggest that LPA could serve as a robust and efficient framework for enhancing the safety of large language models, potentially leading to broader applications in developing safer AI systems that are less susceptible to adversarial manipulation.
Super Weights in LLMs and the Failure of Selective Training
Large Language Models
Efficient ML
Theory
- Super Weights do not universally lead to improved training outcomes when targeted in isolation.
- Training Super Weights and their neighborhoods fails, while randomly chosen parameters in the same layers succeed.
- LoRA's low-rank updates across entire layers outperform isolated training of Super Weights.
- The study validates the structural consistency of Super Weights across diverse inputs.
Read more
Super Weights in LLMs and the Failure of Selective Training
Summary
This paper investigates the concept of Super Weights in large language models (LLMs), which are parameters whose removal significantly degrades model performance. The authors challenge the assumption that targeting these Super Weights for training would enhance model accuracy. Through extensive experiments on OLMo-1B and OLMo-7B, they demonstrate that training Super Weights in isolation leads to drastic accuracy drops, often to random-guessing levels, and that expanding the training to include nearby parameters does not yield improvements. In contrast, training randomly selected parameters in the same layers results in performance gains, indicating that the failure is specific to the Super Weight coordinates rather than a general issue with sparsity. The authors also show that parameter-efficient fine-tuning methods like LoRA can achieve significant improvements with minimal parameter updates, emphasizing the importance of structured updates across entire layers rather than focusing on individual weights. Their findings suggest that parameter importance does not equate to trainability in isolation, and effective fine-tuning strategies should consider the relational structure of weights.
Methodology
The authors conducted a series of experiments involving pruning, direct training, neighborhood training, and low-rank updates using LoRA on two models (OLMo-1B and OLMo-7B). They validated the consistency of Super Weights across multiple inputs and performed ablation studies to isolate the effects of training strategies.
Results
The results showed that training Super Weights in isolation resulted in accuracy drops to random-guessing levels, while training randomly selected parameters improved performance. LoRA achieved a significant accuracy increase with only 0.16% of parameters updated, demonstrating the effectiveness of structured updates over isolated parameter training.
Implications
These findings have implications for the design of fine-tuning strategies in LLMs, suggesting that focusing on the relational structure of weights rather than individual parameter importance may lead to better performance. This could influence future research in parameter-efficient training methods and the understanding of model interpretability.
Predicting Male Fertility Using Machine Learning: A Semen Parameters Based Analysis with the VISEM Dataset
- Machine learning can classify male fertility status with high accuracy based on semen parameters.
- The Nearest Centroid classifier achieved the highest accuracy (94.2%) among over 40 tested algorithms.
- The study utilized the VISEM dataset, which includes semen samples from 85 participants.
- Machine learning models can provide objective assessments, reducing observer bias in traditional semen analysis.
Read more
Predicting Male Fertility Using Machine Learning: A Semen Parameters Based Analysis with the VISEM Dataset
Summary
This study addresses the challenge of male infertility, which affects a significant portion of the global population, by employing machine learning algorithms to classify male fertility status based on semen parameters. Utilizing the VISEM dataset, which includes semen samples from 85 participants categorized into Fertile, Sub-Fertile, and Infertile groups according to WHO criteria, the authors conducted a thorough analysis. After preprocessing and feature engineering, over 40 machine learning algorithms were evaluated using the LazyPredict framework. The Nearest Centroid classifier emerged as the most effective model, achieving an accuracy of 94.2%, surpassing other models like Support Vector Machines and Quadratic Discriminant Analysis. The model's performance was validated through 5-fold cross-validation and multiclass ROC-AUC analysis. The findings suggest that machine learning can significantly enhance the accuracy and objectivity of fertility assessments, potentially aiding clinical decision-making in andrology and assisted reproductive technologies. This research highlights the transformative potential of AI in reproductive health diagnostics and the development of patient-specific treatment strategies.
Methodology
The study employed a supervised machine learning approach, utilizing the LazyPredict framework to evaluate over 40 classification algorithms on the VISEM dataset. The dataset was preprocessed and features were engineered before training the models. Performance was assessed using 5-fold cross-validation and multiclass ROC-AUC analysis.
Results
The Nearest Centroid classifier achieved an accuracy of 94.2%, outperforming other models such as Support Vector Machines and Quadratic Discriminant Analysis. The robustness of the model was confirmed through cross-validation and ROC-AUC analysis.
Implications
The study suggests that machine learning can enhance the accuracy and consistency of fertility assessments, providing a valuable decision-support tool for clinicians in andrology and assisted reproductive technologies. This could lead to improved patient-specific treatment strategies and better outcomes in fertility planning.
Towards the Explainability of Temporal Graph Networks via Memory Backtracking and Topological Attribution
Graph Learning
Interpretability
Time Series
- Introduces a framework for explainability in Temporal Graph Networks that considers both memory updates and spatial interactions.
- Utilizes topology attribution and memory backtracking trees to quantify contributions of neighboring and historical events.
- Implements Layer-wise Relevance Propagation (LRP) to ensure the total contributions match model predictions.
- Demonstrates improved explanation fidelity compared to existing methods through extensive experiments.
Read more
Towards the Explainability of Temporal Graph Networks via Memory Backtracking and Topological Attribution
Summary
This paper addresses the challenge of explainability in Temporal Graph Networks (TGNs), which are increasingly used in applications like fraud detection and healthcare forecasting. While TGNs excel in predictive accuracy, they lack transparency regarding how historical events influence their predictions. The authors propose a novel framework that integrates memory backtracking and topological attribution to provide insights into TGN predictions. The framework consists of two main components: the topology attribution tree, which captures the influence of neighboring nodes and their memory vectors, and the memory backtracking tree, which quantifies the impact of historical events on node memory vectors. By applying Layer-wise Relevance Propagation (LRP), the authors ensure that the total contributions of historical events align with the model's output logits. They also introduce optimization objectives to improve the identification of important events, addressing the limitations of existing methods that often yield unfaithful explanations. The proposed method is validated through experiments on nine temporal graph datasets, demonstrating its effectiveness in providing accurate explanations and outperforming state-of-the-art baselines.
Methodology
The authors developed a framework that includes a topology attribution tree to assess the contributions of neighboring events and a memory backtracking tree to evaluate the impact of historical events on node memory vectors. They employed Layer-wise Relevance Propagation (LRP) to ensure that the sum of contributions from historical events equals the model's logits. Additionally, they designed optimization objectives to enhance the identification of significant events, addressing the limitations of traditional top-k selection methods.
Results
The proposed method was tested on nine temporal graph datasets, covering various tasks such as node property prediction, link prediction, and graph classification. The results indicated that the framework provided faithful explanations of TGN predictions and outperformed existing state-of-the-art explanation methods, demonstrating its effectiveness in enhancing the interpretability of TGNs.
Implications
The findings of this research have significant implications for the deployment of TGNs in critical areas such as fraud detection and healthcare, where understanding model predictions is crucial for trust and safety. The proposed explainability framework can help practitioners gain insights into the decision-making processes of TGNs, thereby improving their reliability and acceptance in high-stakes applications.
MatBind: A Shared Embedding Space for Multimodal Materials Characterization
Multimodal
- MatBind aligns four key materials modalities into a unified embedding space.
- The framework enables zero-shot retrieval between modality pairs not explicitly trained together.
- Materials are organized according to meaningful properties without explicit supervision.
- Combining modalities at query time improves retrieval performance significantly.
Read more
MatBind: A Shared Embedding Space for Multimodal Materials Characterization
Summary
The paper introduces MatBind, a novel contrastive learning framework designed to unify the analysis of multimodal materials data, including crystal structures, powder X-ray diffraction (pXRD) patterns, electronic density of states (DOS), and textual descriptions. Traditional approaches analyze these modalities in isolation, which limits the ability to relate and query materials effectively. MatBind addresses this fragmentation by aligning these diverse data sources into a shared embedding space, using crystal structure as the central anchor. The framework enables zero-shot cross-modal retrieval, allowing for emergent connections between modalities that were not explicitly trained together. The authors demonstrate that the learned embedding space organizes materials based on meaningful physical properties without explicit supervision. The results indicate that combining modalities at query time significantly enhances retrieval performance, showcasing the potential of treating heterogeneous materials data as complementary representations of a single physical reality.
Methodology
MatBind employs a contrastive learning approach, utilizing an anchor-based training strategy where crystal structure serves as the central modality. Pairwise contrastive objectives are trained between the crystal structure and each auxiliary modality (pXRD, DOS, and text), resulting in a shared representation that allows for cross-modal retrieval and property classification.
Results
The learned embedding space effectively organizes materials based on physical properties. Cross-modal retrieval performance is high, particularly between crystal structure and text, and crystal structure and DOS. Notably, zero-shot retrieval between previously unaligned modalities, such as DOS and text, can outperform directly trained pairs. Querying with multiple modalities simultaneously resolves ambiguities that single modalities cannot address.
Implications
MatBind's approach could significantly enhance materials discovery and characterization by providing a more integrated view of multimodal data. This framework may facilitate more efficient research in materials science, enabling researchers to leverage diverse data sources for better insights into material properties and behaviors.
Spectral Analysis of Dueling Q-Learning
Reinforcement Learning
Theory
- Introduces a theoretical framework for analyzing unregularized Dueling Q-Learning.
- Establishes convergence guarantees for both deterministic and stochastic versions of the algorithm.
- Utilizes switching linear system theory to derive error bounds and convergence conditions.
- Clarifies the roles of value and advantage updates in the Q-function decomposition.
Read more
Spectral Analysis of Dueling Q-Learning
Summary
This paper presents a theoretical analysis of Dueling Q-Learning, a variant of Q-learning that separates the Q-function into a value function and an advantage function to enhance learning efficiency. While Dueling Q-Learning has shown empirical success, its theoretical foundations remain underexplored, particularly in the context of unregularized tabular updates. The author addresses this gap by providing a direct interpretation of the centered tabular decomposition and establishing convergence guarantees for the unregularized, constant step-size recursion. The analysis employs a switching linear system (SLS) framework to derive an exact representation of deterministic dueling Q-learning and a finite-time error bound for its stochastic counterpart. The findings clarify how value and advantage updates operate on different components of the Q-function, leading to convergence to a neighborhood of the optimal Q-function as the common scalar gain approaches zero. This work contributes to a deeper understanding of Dueling Q-Learning, paving the way for more robust applications in reinforcement learning.
Methodology
The paper employs a spectral analysis approach using switching linear system (SLS) theory to analyze the convergence of Dueling Q-Learning. It interprets the Q-function through an orthogonal decomposition into value and advantage components, allowing for a detailed examination of their respective updates and their effects on learning.
Results
The analysis yields convergence guarantees for the unregularized Dueling Q-Learning algorithm, demonstrating that the error recursion can be expressed as an SLS. A finite-time error bound in expectation is established for the sampled stochastic version, indicating that the algorithm converges to a first-moment neighborhood of the optimal Q-function, with the neighborhood size diminishing as the common scalar gain decreases.
Implications
The findings have significant implications for the theoretical understanding of Dueling Q-Learning, potentially leading to improved algorithm designs and implementations in reinforcement learning tasks. This work may also inspire further research into the convergence properties of other reinforcement learning algorithms.
A law of robustness for two-layer neural networks with arbitrary weights
Theory
- Proves a conjectured law of robustness for two-layer neural networks with arbitrary weights.
- Establishes a Lipschitz constant lower bound for fitting noisy labels in high-dimensional data.
- Introduces a function-space covering method to address the challenges posed by unbounded weights.
- Demonstrates the results hold for various continuous piecewise-linear activations, particularly ReLU.
Read more
A law of robustness for two-layer neural networks with arbitrary weights
Summary
This paper addresses a conjecture by Bubeck, Li, and Nagaraj regarding the robustness of two-layer neural networks with arbitrary weights. The conjecture posits that for generic data, a two-layer neural network with m neurons fitting n noisy labels must have a Lipschitz constant of at least the order of β(n/m). Previous work by Bubeck and Sellke established a universal version of this law under polynomial bounds on parameters, but the unbounded-weight case required a different approach. The author proves the conjectured law for continuous piecewise-linear activations, including ReLU networks, with a logarithmic factor. The results hold for data drawn uniformly from the sphere or Gaussian distributions, and the paper establishes that fitting data below the noise floor necessitates a Lipschitz constant that scales with the number of data points and the width of the network. The proof employs a function-space covering method instead of a parameter-space covering, which is infeasible for unbounded weights. A rigidity lemma is introduced to control the coefficients of kinks in the realized functions, leading to the necessary entropy bounds. The paper also discusses the implications of the results for different activation functions and provides insights into the limitations of the projection method in high-dimensional settings.
Methodology
The author employs a function-space covering approach to prove the robustness law, utilizing a rigidity lemma to control the coefficients of kinks in the realized functions. The analysis is conducted for data drawn from uniform and Gaussian distributions, and the results are derived under high-probability conditions.
Results
The paper establishes that for a fixed-width two-layer neural network with arbitrary weights, fitting n noisy labels below the noise floor requires a Lipschitz constant that scales as c Ξ΅ β(n/Β―m log(C Β―mnd/Ξ΅)), where Β―m is a function of the network width and the number of kinks. The results are shown to hold with high probability for various data distributions.
Implications
The findings suggest that robust interpolation in neural networks necessitates a significant capacity, which could inform the design of neural architectures and training strategies. The results also highlight the limitations of existing methods for analyzing neural network robustness, particularly in high-dimensional settings.
KronQ: LLM Quantization via Kronecker-Factored Hessian
NLP
Large Language Models
Efficient ML
- KronQ integrates gradient covariance into the quantization pipeline, enhancing performance over traditional methods.
- The framework introduces bidirectional incoherence processing to optimize weight distribution across dimensions.
- A new sensitivity metric for mixed-precision allocation is derived from Hessian traces, allowing for better resource allocation.
- Empirical results show significant improvements in perplexity for LLaMA models, particularly at low bit-widths.
Read more
KronQ: LLM Quantization via Kronecker-Factored Hessian
Summary
KronQ introduces a novel post-training quantization (PTQ) framework for large language models (LLMs) that incorporates gradient covariance into the quantization process, challenging the traditional assumption that all output channels contribute equally to the reconstruction objective. By utilizing the Kronecker-factored Hessian approximation, KronQ enhances quantization loss calculations by jointly considering both activation and gradient covariances. The framework employs bidirectional incoherence processing to reduce weight magnitude variance across input and output dimensions and introduces a new sensitivity metric for mixed-precision allocation based on Hessian traces. Empirical evaluations on LLaMA models demonstrate that KronQ significantly outperforms existing methods, achieving a perplexity of 7.93 in 2-bit weight-only quantization, while other methods like GPTQ and GPTAQ fail to maintain performance, indicating the effectiveness of incorporating gradient information in the quantization process.
Methodology
KronQ employs a post-training quantization approach that utilizes the Kronecker-factored Hessian approximation to incorporate gradient covariance into the quantization loss. It introduces bidirectional incoherence processing and derives a sensitivity metric for mixed-precision allocation based on joint Hessian traces, allowing for optimized quantization across different layers of the model.
Results
KronQ was evaluated on LLaMA-2 and LLaMA-3 models, achieving state-of-the-art results with a perplexity of 7.93 for 2-bit weight-only quantization. In contrast, existing methods like GPTQ and GPTAQ produced significantly higher perplexity values, indicating a failure to maintain model performance under similar conditions.
Implications
The incorporation of gradient covariance into quantization processes could lead to more efficient deployment of large language models on resource-constrained hardware, enabling broader accessibility and application of advanced NLP technologies. This approach may also inspire further research into optimizing model compression techniques.
Multi-Modal, Multi-Environment Machine Teaching for Robust Reward Learning
Reinforcement Learning
Robotics
Theory
- Introduces a comprehensive analysis of feedback modalities in machine teaching for reward learning.
- Develops a hierarchical teaching algorithm (HSCOT) that operates across multiple environments.
- Demonstrates that comparisons impose stronger constraints on reward functions than demonstrations.
- Empirical results show lower regret and better generalization with HSCOT compared to uniform teaching methods.
Read more
Multi-Modal, Multi-Environment Machine Teaching for Robust Reward Learning
Summary
This paper addresses the challenge of aligning the behavior of autonomous agents with human intent across diverse operational contexts by developing a robust reward learning framework. The authors highlight the limitations of existing inverse reinforcement learning (IRL) approaches that primarily focus on single-environment, demonstration-only settings, which often lead to overfitting and poor generalization in new environments. They analyze how different feedback modalities, such as comparisons and demonstrations, constrain reward functions, revealing that comparisons provide stronger global constraints in unlimited data scenarios. To tackle the limitations of current methods, the authors propose a hierarchical machine teaching algorithm called Hierarchical Set Cover Optimal Teaching (HSCOT). This algorithm strategically selects informative environments and queries low-cost feedback to efficiently constrain rewards across multiple Markov Decision Processes (MDPs). Empirical evaluations demonstrate that HSCOT significantly reduces regret and enhances generalization to held-out environments compared to uniform teaching methods under the same feedback budgets, underscoring the importance of multi-modal and multi-environment teaching for developing robust reward functions.
Methodology
The authors conducted a theoretical analysis of feedback modalities in reward learning, focusing on their constraints in both unlimited-data and limited-budget scenarios. They developed the HSCOT algorithm, which selects informative environments and strategically queries feedback to optimize reward learning across multiple MDPs. The methodology includes empirical validation to compare the performance of HSCOT against uniform teaching baselines.
Results
The empirical results indicate that HSCOT achieves significantly lower regret and better generalization to held-out environments compared to uniform teaching methods, demonstrating the effectiveness of multi-modal and multi-environment approaches in reward learning.
Implications
The findings suggest that incorporating diverse feedback modalities and considering multiple environments can lead to more robust and generalizable reward functions for autonomous agents, which is crucial for their deployment in real-world applications where operational contexts vary.
NFTR: From Provable Mode-Averaging to Geodesic Subgoal Selection in Offline Goal-Conditioned RL
Reinforcement Learning
Robotics
Theory
- NFTR addresses optimistic bias and mode collapse in HIQL by using Normalizing Flows for subgoal selection.
- The triangle-slack score effectively downweights unreliable subgoals based on geometric consistency.
- NFTR preserves population-level monotonic improvement and provides a clear suboptimality decomposition.
- Empirical evaluations show substantial performance improvements over HIQL in offline GCRL tasks.
Read more
NFTR: From Provable Mode-Averaging to Geodesic Subgoal Selection in Offline Goal-Conditioned RL
Summary
The paper introduces NFTR (Normalizing Flows subgoal policies with Triangle-slack Reweighting), a novel approach to offline goal-conditioned reinforcement learning (GCRL) that addresses the limitations of Hierarchical Implicit Q-Learning (HIQL). HIQL suffers from two main issues: optimistic bias due to stochastic environments and mode collapse from using a unimodal Gaussian policy for subgoal selection. NFTR replaces the Gaussian policy with a conditional Normalizing Flow, which allows for modeling multi-modal subgoal distributions. Additionally, it introduces a triangle-slack score that corrects the advantage-weighted regression (AWR) weights to downweight subgoals that are geometrically inconsistent or unreachable. The triangle-slack score is derived from a triangle inequality and serves as a conservative upper bound on composability violations in stochastic dynamics. The proposed method maintains AWR's population-level monotonic improvement and provides a three-term suboptimality decomposition. Empirical results demonstrate that NFTR significantly outperforms HIQL across various tasks, including stochastic, stitching, and manipulation tasks on the OGBench benchmark.
Methodology
The methodology involves replacing the Gaussian policy in HIQL with a conditional Normalizing Flow to enable multi-modal subgoal modeling. The triangle-slack score is introduced to adjust the AWR weights, ensuring that subgoals with high detour costs are downweighted. The approach is theoretically grounded in the properties of deterministic and stochastic Markov Decision Processes (MDPs) and includes a three-term decomposition of suboptimality.
Results
NFTR demonstrates significant improvements over HIQL in various tasks, achieving better performance in stochastic environments and complex manipulation tasks. The method effectively avoids the Gaussian collapse issue and maintains stability under stochastic dynamics, as evidenced by empirical results on the OGBench benchmark.
Implications
The findings suggest that NFTR can enhance the performance of offline GCRL methods, particularly in robotics and autonomous systems where data collection is limited. The approach may lead to more reliable and efficient goal-reaching policies in complex environments.
Spectral Stability of Pseudoinverse-Based Extreme Learning Machine
Theory
Efficient ML
Optimization
- The smallest singular value of the hidden-layer matrix is crucial for output weight stability in ELM.
- Condition number provides a quantitative measure of the hidden-layer matrix's instability.
- SVD-based methods outperform iterative methods in terms of reliability under ill-conditioning.
- Larger training sample sizes generally improve stability, while larger hidden widths can lead to poorer conditioning.
Read more
Spectral Stability of Pseudoinverse-Based Extreme Learning Machine
Summary
This paper investigates the numerical stability of the Extreme Learning Machine (ELM), which utilizes the MooreβPenrose pseudoinverse for computing output weights. The authors highlight that the stability of ELM is significantly influenced by the conditioning of the hidden-layer matrix, particularly through its spectral properties. They establish that the smallest singular value of the hidden-layer matrix is critical in determining the amplification of perturbations in output weights, while the condition number serves as a quantitative measure of instability. The paper contrasts SVD-based pseudoinverse computation with iterative hyperpower methods, revealing that SVD methods are more reliable under ill-conditioning. Through experiments on synthetic matrices and benchmark datasets, the authors demonstrate that the singular-value structure of the hidden-layer matrix fundamentally governs ELM stability, suggesting that larger sample sizes can enhance stability, whereas excessively large hidden widths may worsen conditioning.
Methodology
The authors conducted a spectral stability analysis of the hidden-layer matrix in ELM using singular value decomposition (SVD). They compared SVD-based pseudoinverse computation with iterative methods like NewtonβSchulz and hyperpower iterations. Experiments were performed on synthetic matrices with controlled singular-value spectra and benchmark datasets (MNIST, Fashion-MNIST, ISOLET) to evaluate the performance of different methods based on convergence behavior, classification accuracy, and stability metrics.
Results
The experiments revealed that iterative pseudoinverse methods were effective for well-conditioned and moderately ill-conditioned matrices but failed completely under severely ill-conditioned conditions. In contrast, SVD-based methods maintained successful runs even in challenging scenarios. The results indicated that the singular-value structure of the hidden-layer matrix is a key factor in determining the reliability of convergence and overall stability of ELM.
Implications
The findings suggest that careful consideration of the hidden-layer matrix's spectral properties can lead to more stable and reliable implementations of ELM. This has potential applications in various domains where ELM is utilized, particularly in scenarios with high-dimensional data or when dealing with ill-conditioned matrices.
Dimensionality Reduction Meets Network Science: Sensemaking on UMAP's kNN Graph
Graph Learning
- UMAP's kNN graph encodes high-dimensional manifold structure lost in 2D projections.
- Standard graph algorithms can be applied to the kNN graph for enhanced data sensemaking.
- PageRank, k-core decomposition, and clustering coefficients reveal insights into data representativeness, density, and local cohesion.
- Graph-based analyses on MNIST and Fashion MNIST datasets show competitive performance compared to traditional clustering methods.
Read more
Dimensionality Reduction Meets Network Science: Sensemaking on UMAP's kNN Graph
Summary
This paper explores the underutilized k-nearest-neighbor (kNN) graph constructed by UMAP, a popular dimensionality reduction technique. While UMAP is typically used to visualize high-dimensional data in a 2D scatter plot, the authors argue that the internal kNN graph retains valuable information about the data's manifold structure that is lost during projection. The paper proposes treating this kNN graph as a first-class analytical resource, allowing for the application of standard graph algorithms to enhance data sensemaking. Specifically, the authors demonstrate how PageRank can identify representative data points, k-core decomposition can reveal dense core regions, and clustering coefficients can detect cohesive neighborhoods. Through quantitative and qualitative evaluations on MNIST and Fashion MNIST datasets, the authors show that these graph-based analyses are competitive with traditional methods like k-medoids and HDBSCAN, providing complementary insights into data structure and relationships.
Methodology
The authors applied standard graph algorithmsβPageRank, k-core decomposition, and clustering coefficientsβto UMAP's kNN graph to analyze data structure. They conducted evaluations on MNIST and Fashion MNIST datasets to compare the effectiveness of these graph-based methods against traditional clustering techniques like k-medoids and HDBSCAN.
Results
The results indicated that PageRank-selected data points achieved superior class balance and representativeness compared to k-medoids, while k-core decomposition revealed a hierarchical structure that traditional clustering methods could not capture. The clustering coefficient analysis identified distinct micro-clusters, demonstrating the effectiveness of graph-based analyses.
Implications
This work suggests that leveraging the kNN graph can enhance the interpretability and utility of UMAP in data analysis workflows. By integrating network science with dimensionality reduction, researchers can gain deeper insights into high-dimensional datasets, potentially improving applications in various fields such as computer vision and data mining.
Steering Neural Network Training through Interpretable Constraints Based on Partial Dependence
Interpretability
Time Series
Theory
- Introduces a novel approach for steering neural network training using partial dependence.
- Focuses on regression problems rather than classification, addressing a gap in existing literature.
- Demonstrates improved model performance and data efficiency through the incorporation of domain knowledge.
- Shows that interpretations from constrained models align better with user-provided knowledge.
Read more
Steering Neural Network Training through Interpretable Constraints Based on Partial Dependence
Summary
This paper addresses the challenge of enhancing the interpretability of machine learning models, particularly neural networks, by introducing a novel approach that incorporates domain knowledge into the training process. The authors propose a method that steers neural network training using partial dependence, ensuring that the model's average response to specific features aligns with known functional relationships in the problem domain. Unlike existing methods that primarily focus on classification tasks, this approach is applied to regression problems, including dynamical systems forecasting. The authors empirically validate their method across various regression scenarios, demonstrating that models trained with these interpretable constraints outperform unconstrained models in terms of performance and data efficiency. Furthermore, the explanations derived from the constrained models are shown to be more faithful to the provided domain knowledge, highlighting the importance of integrating expert insights into the model training process.
Methodology
The authors develop a training algorithm that incorporates partial dependence as a constraint during the training of neural networks. This method ensures that the model's marginal responses to certain input features are consistent with predefined functional domain knowledge. The approach is validated through empirical experiments on various regression tasks.
Results
The results indicate that models trained with the proposed method significantly outperform their unconstrained counterparts, demonstrating better performance metrics, requiring fewer training samples, and exhibiting improved generalization capabilities outside the original input domain. Additionally, the explanations generated by the constrained models are more aligned with the prior knowledge provided by users.
Implications
This work has potential implications for fields where model interpretability is crucial, such as healthcare, finance, and engineering. By integrating domain knowledge into the training process, practitioners can develop more reliable and understandable models, ultimately leading to better decision-making and trust in machine learning systems.
Architecture Generalization with MetaNCA
Efficient ML
Theory
Graph Learning
- Introduction of MetaNCA for self-organizing neural network weights through local rules.
- Utilization of a Weight Transformer architecture for local interactions on computation graphs.
- Demonstrated generation of diverse neural network architectures without backpropagation.
- Generalization to unseen architectures, enhancing adaptability and efficiency.
Read more
Architecture Generalization with MetaNCA
Summary
This paper introduces Meta Neural Cellular Automata (MetaNCA), a novel framework that enables the self-organization of artificial neural network weights through local interactions, inspired by biological neural systems. Unlike traditional methods that rely on backpropagation and fixed architectures, MetaNCA employs a Weight Transformer architecture to learn local update rules that can generate diverse neural network architectures without the need for backpropagation. The framework demonstrates the ability to generate weights for various models, including feedforward MLPs, CNNs, and ResNets, while scaling to networks with up to 2 million parameters. Importantly, MetaNCA exhibits generalization capabilities to unseen architectures, suggesting that training with architectural diversity enhances its adaptability. The work addresses the limitations of current training methods, emphasizing the potential for more efficient and flexible neural network training inspired by biological systems.
Methodology
MetaNCA employs a graph neural cellular automaton approach, where a learned rule network iteratively updates the weights of a task network based solely on local interactions within the computation graph. The Weight Transformer architecture utilizes linear attention to aggregate signals from neighboring weights and hidden states, allowing for the generation of diverse neural network architectures during training.
Results
MetaNCA successfully generates weights for various architectures, including MLPs, CNNs, and ResNets, on datasets like MNIST and CIFAR-100. The framework scales to networks with up to 2 million parameters and demonstrates the ability to generalize to architectures not seen during meta-training, indicating that architectural diversity during training enhances generalization capabilities.
Implications
The findings suggest that MetaNCA could lead to more efficient training methods for neural networks, reducing the reliance on backpropagation and enabling the development of adaptable models that can learn from fewer examples. This approach may have significant implications for resource-constrained environments and applications requiring flexible neural network architectures.
ReCoLoRA: Spectrum-Aware Recursive Consolidation for Continual LLM Fine-Tuning
NLP
Large Language Models
Efficient ML
- ReCoLoRA introduces a spectrum-aware framework for continual fine-tuning of LLMs, mitigating catastrophic forgetting.
- The recursive consolidation mechanism allows for the preservation of knowledge from previous tasks while adapting to new ones.
- ReCoLoRA outperforms existing low-rank adaptation methods in terms of final average scores and parameter efficiency.
- The proposed ReCoLoRA-TaskBank variant serves as an upper bound for task retention by isolating task-specific branches.
Read more
ReCoLoRA: Spectrum-Aware Recursive Consolidation for Continual LLM Fine-Tuning
Summary
The paper introduces ReCoLoRA, a novel framework designed for parameter-efficient continual fine-tuning of large language models (LLMs) that addresses the issue of catastrophic forgetting when adapting to new tasks. Traditional low-rank adaptation methods, such as LoRA, often lead to overwriting of previous task knowledge as new tasks are added. ReCoLoRA employs a spectrum-aware approach, initializing adapters from a randomized singular value decomposition (SVD) of pretrained weights and selecting effective ranks using an elbow criterion. The core innovation is the recursive consolidation mechanism, which re-decomposes the effective weight into a frozen residual, a slowly updated principal component, and a new adapter before each new task. This allows the model to retain knowledge from previous tasks while adapting to new ones. The authors demonstrate the effectiveness of ReCoLoRA through experiments on a six-task continual GLUE sequence, showing superior performance compared to existing methods while training fewer parameters. Additionally, they propose ReCoLoRA-TaskBank, an oracle-routed variant that isolates branches for each task, serving as an upper bound for retention without overwriting.
Methodology
ReCoLoRA employs a two-stage training process where the principal subspace is adapted first, followed by gradual adaptation of the residual capacity. It utilizes randomized SVD for initializing adapters and an elbow criterion for effective rank selection. The recursive consolidation mechanism re-decomposes the effective weight into a frozen residual, a slowly trainable principal component, and a fresh adapter before each new task.
Results
In experiments on a six-task continual GLUE sequence using four different LLM backbones, ReCoLoRA achieved the best final average score on three out of four models, demonstrating superior performance compared to rank-swept LoRA, PiSSA, AdaLoRA, and DoRA baselines. The ReCoLoRA-TaskBank variant achieved a final average score of 0.8957 with no average forgetting across tasks.
Implications
ReCoLoRA has significant implications for the deployment of LLMs in dynamic environments where continual learning is essential. It enables efficient adaptation to new tasks while preserving previously learned knowledge, making it suitable for applications in real-time language processing, interactive AI systems, and personalized user experiences.
Latent Memory Palace: Reasoning for Control as Autoregressive Variational Inference
Reinforcement Learning
Robotics
Generative Models
- Introduces Latent Memory Palace (LMP) for iterative and adaptive reasoning in robotic control.
- Utilizes autoregressive variational inference to organize control-relevant information in latent space.
- Demonstrates strong empirical performance in both simulated and real-world robotic tasks.
- Implements a variable-length action tokenizer (LMP-tok) that improves downstream policy performance.
Read more
Latent Memory Palace: Reasoning for Control as Autoregressive Variational Inference
Summary
This paper introduces the Latent Memory Palace (LMP), a novel framework that enables reasoning for control policies in robotics through autoregressive variational inference. The authors argue that human decision-making is both iterative and adaptive, and they seek to replicate this flexibility in robotic policies. Traditional methods from language models do not translate well to continuous control due to the need for spatial understanding and precise actions. LMP organizes information in a latent space that resembles a memory palace, allowing for iterative retrieval of control-relevant information. The framework formulates reasoning as variational inference with a variable-length autoregressive latent distribution, enabling the generation of latent tokens that represent control actions. The authors derive a reinforcement learning technique to optimize the variational lower bound, resulting in a policy (LMP-Ο) that exhibits strong performance in both simulation and real-world tasks. Additionally, they introduce a variable-length action tokenizer (LMP-tok) that enhances the performance of downstream autoregressive policies. The results demonstrate that LMP-Ο outperforms non-iterative methods and effectively allocates computation during decision-making, showcasing the potential of autoregressive variational inference for adaptive reasoning in robotics.
Methodology
The authors develop the Latent Memory Palace (LMP) framework, which formulates reasoning as variational inference with an autoregressive latent distribution. They optimize the variational lower bound using reinforcement learning techniques, enabling the generation of latent tokens that represent actions. The framework allows for variable-length paths in the latent space, facilitating adaptive computation during decision-making.
Results
LMP-Ο, the resulting policy, achieves significant performance improvements across various robotic learning tasks in both simulation and real-world environments. The analysis indicates that iterative computation outperforms non-iterative approaches, and the ability to adaptively allocate computation plays a crucial role in enhancing performance. The action tokenizer LMP-tok shows substantial improvements over existing tokenization methods.
Implications
The findings suggest that autoregressive variational inference can serve as a powerful framework for enhancing reasoning capabilities in robotic control, potentially leading to more efficient and adaptable robotic systems. This approach may also influence future research in integrating reasoning mechanisms into other domains of machine learning.
Workload-Preserving Differentially Private Synthetic Data for Causal Inference via Maximum-Entropy Calibration
Theory
- Introduction of causal workloads designed for differential privacy that focus on orthogonal moments for causal estimators.
- Development of two methods for utilizing causal workloads: direct moment plug-ins and maximum-entropy synthetic data reconstruction.
- Establishment of theoretical bounds connecting ATE error to workload error, with a detailed decomposition of synthetic data error.
- Introduction of CAUSAL-AIM for adaptive workload selection and NA+MI for improved confidence interval estimation.
Read more
Workload-Preserving Differentially Private Synthetic Data for Causal Inference via Maximum-Entropy Calibration
Summary
This paper addresses the challenges of generating differentially private (DP) synthetic data that preserves causal inference properties, particularly for estimating average treatment effects (ATE). The authors introduce the concept of causal workloads, which are DP query sets designed to capture orthogonal moments relevant for causal estimators. They propose two main approaches for utilizing these causal workloads: a direct moment plug-in method and a maximum-entropy synthetic data reconstruction method. The paper provides theoretical bounds that connect ATE error to the error in the released workload, decomposing synthetic data error into various components including sampling and calibration errors. Additionally, the authors present CAUSAL-AIM, an adaptive workload selector, and a noise-aware multiple-imputation (NA+MI) method for generating confidence intervals from DP synthetic data. Empirical results demonstrate that the proposed methods yield near-nominal coverage for confidence intervals while supporting multiple causal analyses without additional privacy costs. The findings highlight a tradeoff between distributional fidelity and valid causal inference, emphasizing the importance of preserving causal moments in synthetic data generation.
Methodology
The authors propose causal workloads as sets of DP queries that measure orthogonal moments relevant for causal inference. They utilize two main approaches: a direct evaluation of stable estimators based on released moments and a maximum-entropy calibration method to reconstruct synthetic data. Theoretical analysis provides error bounds for ATE estimation, and empirical evaluations assess the performance of the proposed methods against traditional approaches.
Results
The proposed methods, particularly the combination of causal workloads and NA+MI, achieved near-nominal coverage rates (99.8-100%) for confidence intervals at low privacy budgets (Ξ΅ β€ 1). The same synthetic data release supported multiple causal analyses (ATE, ATT, subgroup analyses) without incurring additional privacy costs, demonstrating the efficiency and effectiveness of the approach.
Implications
This work has significant implications for the fields of health, education, and social sciences where sensitive data must be released while preserving privacy. The methods developed can facilitate valid causal inference from synthetic data, enabling researchers to conduct comprehensive analyses without compromising individual privacy.
Deep Learning Method for Stationary Distribution of Reflected Brownian Motion
Theory
Efficient ML
Optimization
- Develops a deep learning method for estimating the Laplace transform of high-dimensional reflected Brownian motion.
- Introduces a tailored loss function and sampling scheme to enhance model performance.
- Achieves near-perfect prediction accuracy for tail probabilities in high-dimensional settings.
- Demonstrates the potential of deep learning in analyzing complex stochastic systems.
Read more
Deep Learning Method for Stationary Distribution of Reflected Brownian Motion
Summary
This paper presents a novel deep learning approach to compute the stationary distribution of reflected Brownian motion (RBM), which is crucial for analyzing high-dimensional stochastic systems. The authors highlight the limitations of existing methods that only provide closed-form solutions for specific cases, making it challenging to compute performance metrics like tail probabilities. The proposed framework leverages the basic adjoint relationship (BAR) to learn the Laplace transform of high-dimensional RBMs. Key innovations include a carefully designed loss function, a targeted training data sampling strategy, and a neural network architecture that maintains efficiency as dimensionality increases. The method is evaluated against RBM instances with known tail probabilities, achieving near-perfect predictions in high-dimensional scenarios. This work represents a significant advancement in the application of deep learning to stochastic systems, offering a scalable tool for performance analysis beyond analytically tractable regimes.
Methodology
The authors utilize a deep learning framework that incorporates a specifically designed loss function based on the basic adjoint relationship (BAR) for reflected Brownian motion. They employ feedforward neural networks to approximate the Laplace transform in the complex domain, facilitating robust numerical inversion methods for tail probability estimation. The architecture is structured to ensure that the number of trainable parameters does not scale with the dimension of the problem, enhancing computational efficiency.
Results
The proposed deep learning method demonstrates high accuracy in predicting tail probabilities for reflected Brownian motion across various high-dimensional instances. The evaluations show that the framework can effectively learn the Laplace transforms, providing a richer characterization of system performance compared to existing methods.
Implications
This research has significant implications for the analysis of multiclass queueing networks and other stochastic systems where reflected Brownian motion is applicable. The ability to compute tail probabilities and other performance metrics efficiently opens new avenues for research and application in operations research, network performance analysis, and beyond.
Deep Learning for Joint Narrowband Interference Cancellation and Soft Demodulation in OFDM Systems
Theory
Efficient ML
Optimization
- Introduces a deep learning framework for joint NBI cancellation and soft demodulation in OFDM systems.
- NBI-CNet reduces computational complexity by up to 60% compared to state-of-the-art methods.
- LLR-CNet enhances soft metric calibration, eliminating error floors from traditional demodulation techniques.
- Demonstrates robust performance under severe and mild interference conditions.
Read more
Deep Learning for Joint Narrowband Interference Cancellation and Soft Demodulation in OFDM Systems
Summary
This paper addresses the challenge of narrowband interference (NBI) in orthogonal frequency-division multiplexing (OFDM) systems, which can severely degrade performance by corrupting subcarriers. Traditional methods such as compressed sensing (CS) for NBI mitigation are limited by high latency and residual errors that affect soft demodulation. The authors propose a unified deep learning framework that integrates NBI cancellation and robust soft demodulation. The framework consists of two main components: NBI-CNet, a convolutional neural network that estimates NBI parameters and removes interference in a single pass, and LLR-CNet, which maps non-Gaussian residuals to calibrated soft metrics. The proposed method significantly reduces computational complexity by up to 60% compared to existing algorithms and effectively eliminates error floors associated with traditional approaches. Simulations show that the framework performs well under severe interference conditions, maintaining a close margin to optimal performance while providing substantial coding gains. The architecture is designed to generalize across different FFT sizes without the need for retraining, making it adaptable to various operational scenarios.
Methodology
The authors developed two neural network architectures: NBI-CNet, which employs a physics-informed convolutional structure to estimate and mitigate NBI in a single forward pass, and LLR-CNet, which acts as a structural whitener to improve the reliability of soft metrics derived from post-mitigation residuals. The framework leverages deep learning to adaptively handle dynamic interference scenarios without requiring prior knowledge of the number of active interferers.
Results
The proposed deep learning framework outperforms traditional methods, achieving a block error rate (BLER) of 10^-4 with a signal-to-interference ratio (SIR) of -10 dB, operating within a 0.2 to 0.5 dB margin of the optimal iterative baseline. Under mild interference conditions, it provides a coding gain exceeding 3 dB, effectively avoiding signal-peak confusion and mitigating the impact of interferer-estimation errors.
Implications
The findings suggest that deep learning can significantly enhance the robustness and efficiency of OFDM systems in interference-prone environments, making it applicable to future wireless networks, including 5G and beyond. This approach can lead to improved communication reliability and performance in dense and heterogeneous network scenarios.
Eigenvalue Calibration for Semantic Embeddings of Large Language Models
NLP
Large Language Models
Theory
- Introduces a novel calibration framework for eigenvalues of semantic embeddings in LLMs.
- Establishes theoretical foundations linking entropy and risk in the context of eigenvalue calibration.
- Demonstrates that current LLMs are systematically overconfident in their predictions.
- Validates the effectiveness of temperature scaling in improving eigenvalue calibration.
Read more
Eigenvalue Calibration for Semantic Embeddings of Large Language Models
Summary
This paper addresses the critical issue of uncertainty quantification in large language models (LLMs) by proposing a novel framework for calibrating the eigenvalues of semantic embeddings. The authors highlight that conventional calibration methods for classification probabilities cannot be directly applied to eigenvalues, which represent latent outcome probabilities in a different mathematical space. They introduce a new approach that interprets LLMs combined with semantic embeddings as density matrix predictors and applies temperature scaling to their eigenvalues to improve calibration. The paper establishes a theoretical foundation for this calibration process, including entropy-risk equivalence and a specific calibration inequality for eigenvalues. Empirical experiments demonstrate that current LLMs exhibit systematic overconfidence, and the proposed calibration method effectively reduces this overconfidence, leading to more reliable uncertainty estimates. The findings advance both the theoretical understanding and practical applications of uncertainty quantification in LLMs.
Methodology
The authors propose a framework that interprets LLMs as density matrix predictors. They apply temperature scaling to the eigenvalues derived from semantic embeddings to optimize calibration. The paper derives a central calibration inequality specific to eigenvalues and establishes entropy-risk equivalence under calibration. Empirical evaluations are conducted across various real-world settings to validate the theoretical findings.
Results
The experiments reveal that LLMs are overconfident in their predictions, with maximum eigenvalues indicating inflated confidence levels. After applying temperature scaling, the predicted eigenvalues are significantly reduced, leading to lower expected calibration errors and more accurate uncertainty estimates. The proposed calibration method shows a marked improvement in the reliability of predictions from LLMs.
Implications
The findings have significant implications for the deployment of LLMs in high-stakes applications where reliable uncertainty estimates are crucial. Improved calibration methods can enhance decision-making processes, model comparisons, and human-AI collaboration by providing more trustworthy confidence measures.
AutoAnchor: Stable Diffusion Unlearning Using Cross-Attention as a Manifold Surrogate
Generative Models
Computer Vision
Theory
- AutoAnchor synthesizes manifold-proximal anchors for stable diffusion unlearning.
- The proposed method addresses the limitations of both anchor-based and anchor-free unlearning techniques.
- Cross-attention consistency loss is introduced as a surrogate for optimizing manifold proximity.
- Experimental results indicate up to 31.04% improvement in targeted concept removal.
Read more
AutoAnchor: Stable Diffusion Unlearning Using Cross-Attention as a Manifold Surrogate
Summary
The paper addresses the challenge of diffusion unlearning in text-to-image models, which is crucial for preventing the generation of harmful or copyrighted content. Existing methods for diffusion unlearning either rely on manually chosen anchors, which can introduce bias, or on anchor-free methods that may lead to unrobust unlearning due to off-manifold drift. The authors formalize these issues under the manifold hypothesis, demonstrating that the absence of a manifold-proximal anchor results in significant normal-space drift, degrading unlearning performance. To overcome these limitations, they propose AutoAnchor, a two-stage framework that automatically synthesizes manifold-proximal anchors. The key innovation is a cross-attention consistency loss that serves as an efficient surrogate for manifold proximity, allowing for robust and unbiased unlearning. Experimental results show that AutoAnchor significantly improves targeted concept removal and non-target utility, and can be integrated into existing diffusion unlearning methods to enhance their performance.
Methodology
The authors propose a two-stage framework, AutoAnchor, which first filters and aggregates valid candidate concepts and then optimizes anchors to be manifold-proximal to the target concept. A novel cross-attention consistency loss is introduced to facilitate this optimization, serving as a computationally efficient proxy for manifold proximity.
Results
AutoAnchor demonstrates robust and unbiased unlearning across various state-of-the-art baselines, achieving up to 31.04% improvement in targeted concept removal and up to 4.18% in non-target utility. Additionally, it enhances existing diffusion unlearning methods by an average of 6.30% for concept removal and 6.65% for utility.
Implications
The findings suggest that AutoAnchor can be a valuable tool for improving the safety and reliability of text-to-image diffusion models, making it easier to mitigate risks associated with harmful or copyrighted content generation. Its integration into existing frameworks could lead to broader applications in responsible AI development.
Beyond Backpropagation: Monte Carlo Method Can Train Deep Neural Networks
Optimization
Theory
Efficient ML
- Introduces a gradient-free Monte Carlo method for training deep neural networks.
- Demonstrates the method's effectiveness without requiring batch normalization or residual connections.
- Validates the approach on deep networks exceeding 20 layers and various architectures.
- Highlights the flexibility of the method in supporting discrete weights and unconventional transfer functions.
Read more
Beyond Backpropagation: Monte Carlo Method Can Train Deep Neural Networks
Summary
This paper presents a novel approach to training deep neural networks using a simple Monte Carlo algorithm, challenging the dominance of backpropagation (BP) in deep learning. The author argues that BP's reliance on gradients leads to issues such as vanishing and exploding gradients, motivating the exploration of gradient-free methods. The proposed Monte Carlo method involves randomly mutating network parameters and accepting changes that reduce loss, allowing for effective training of deep networks without the need for techniques like batch normalization or residual connections. The method demonstrates flexibility in various scenarios, including pure pruning training, support for discrete weights, and the use of unconventional transfer functions like Gaussian. The feasibility of this approach is validated through experiments on deep networks with over 20 layers, wide networks with up to 16,384 hidden neurons, and a simple Transformer architecture, applied to tasks such as image classification (MNIST) and character-level language modeling (Tiny Shakespeare). This work suggests that the Monte Carlo method could provide a complementary perspective on neural network learning mechanisms and offers an alternative route for developing physically inspired deep learning systems.
Methodology
The Monte Carlo algorithm involves randomly selecting and mutating network parameters, accepting changes that lead to a decrease in loss. The method is implemented on a GPU, allowing for efficient training of deep networks. The experiments utilize standard datasets like MNIST and explore various transfer functions, including ReLU and Gaussian.
Results
The Monte Carlo method successfully trained deep networks with more than 20 layers and wide networks with up to 16,384 hidden neurons. It also effectively trained a simple Transformer architecture on both image classification and language modeling tasks, demonstrating its practical applicability and efficiency.
Implications
This research opens up new avenues for training deep neural networks without relying on gradient-based methods, potentially leading to more robust and flexible learning systems. It may also enhance our understanding of neural network dynamics and inspire the development of new architectures and training paradigms.
Self-Adaptive Anomaly Detection with Reinforcement Learning and Human Feedback in Connected Vehicles
Reinforcement Learning
Robotics
Time Series
- Introduces a self-adaptive anomaly detection framework for connected vehicles.
- Integrates reinforcement learning with human feedback for continuous adaptation.
- Utilizes a factorized DQN for selecting appropriate detectors based on service dependencies.
- Demonstrates effective performance in a real-world connected vehicle testbed.
Read more
Self-Adaptive Anomaly Detection with Reinforcement Learning and Human Feedback in Connected Vehicles
Summary
This paper presents a novel online anomaly detection framework designed for connected vehicles, which are increasingly autonomous cyber-physical systems. The authors identify the challenge of monitoring these systems due to their evolving nature, which can lead to concept drift that degrades static diagnostic methods. To address this, the proposed framework integrates three coordinated mechanisms: a factorized deep Q-network (DQN) with self-attention for detector selection, an ensemble of statistical drift detectors that prioritize precision, and a human-in-the-loop retraining mechanism that incorporates expert feedback while preventing catastrophic forgetting. The framework was evaluated on a connected-vehicle testbed running an automated valet parking application, demonstrating its effectiveness in adapting to changes in data distributions while maintaining performance. The results indicate that the attention-augmented agent achieved an F1 score of 0.69, significantly outperforming individual detectors. Following a software update that induced concept drift, the F1 score dropped to 0.52 but recovered to 0.65 after operator-triggered retraining, showcasing the framework's ability to adapt without losing prior knowledge.
Methodology
The framework employs a factorized deep Q-network with self-attention to select the most suitable anomaly detector from a candidate pool for each monitored service, leveraging inter-service dependencies. It also utilizes an ensemble of three statistical drift detectors that raise alarms only when all concur, prioritizing precision. A human-in-the-loop retraining mechanism is implemented, which includes a pending transition buffer and a prioritized replay strategy to incorporate expert feedback while preserving learned behaviors.
Results
The attention-augmented agent achieved an F1 score of 0.69, significantly higher than the maximum score of 0.11 from any single detector. After a software update causing concept drift, the F1 score dropped to 0.52 but improved to 0.65 after retraining, while maintaining the previous score of 0.69 on the earlier distribution.
Implications
This framework has significant implications for the development of robust anomaly detection systems in connected vehicles, enabling continuous monitoring and adaptation to changing operational conditions. It highlights the importance of integrating human expertise into automated systems, which can enhance reliability and safety in autonomous driving applications.
PIT-SUN: A Deployable Empirical Marginal Transform Framework with Expectation-Consistent Recovery for Regression in Recommender Systems
Theory
Optimization
- PIT-SUN addresses the instability of gradients in traditional MSE methods for complex target distributions.
- The framework employs a single empirical marginal table to define key components for expectation-consistent recovery.
- Empirical validation shows robust improvements in performance metrics across diverse datasets.
- The methodology emphasizes the need for a coordinated approach to target transformation and recovery.
Read more
PIT-SUN: A Deployable Empirical Marginal Transform Framework with Expectation-Consistent Recovery for Regression in Recommender Systems
Summary
The paper introduces PIT-SUN, a novel framework designed to improve the estimation of conditional expectations in recommender systems, particularly for predicting continuous business values such as dwell time, GMV, and LTV. Traditional methods face challenges with heavy-tailed, zero-inflated, and multimodal targets, leading to issues like mean collapse and tail shrinkage. While target transformations can help, they often lose expectation consistency upon inversion. The authors propose a solution that combines empirical marginal tables with a bounded normal-score coordinate and a variance-controlled recovery base, allowing for effective estimation of original-space expectations without direct inversion. The framework includes a drift monitoring mechanism and utilizes a multiplicative SUN recovery approach. Extensive experiments demonstrate that PIT-SUN significantly enhances point accuracy, calibration, and ranking quality across various datasets, including synthetic, public benchmarks, and large-scale industrial applications, while maintaining lightweight deployment overhead.
Methodology
PIT-SUN utilizes an empirical marginal table to establish a stable coordinate system, an inverse-quantile lookup for recovery, and a variance-controlled recovery base. It integrates drift monitoring and applies a multiplicative SUN recovery method to estimate expectations in the original space, avoiding direct inversion of transformed predictions.
Results
The experiments conducted on synthetic distributions, public benchmarks, and large-scale industrial datasets indicate that PIT-SUN achieves significant improvements in point accuracy, calibration, and ranking quality, demonstrating its effectiveness and robustness in real-world applications.
Implications
The PIT-SUN framework has the potential to enhance the performance of recommender systems by providing more accurate predictions of user engagement metrics, which can lead to better decision-making in marketing and content delivery strategies.
Selective Left-Shift: Turning Test-Time Compute and Difficulty-based Curation into Training Data for Low-Resource Code Generation
NLP
Large Language Models
Reinforcement Learning
- Introduces a three-phase pipeline for improving code generation in low-resource programming languages.
- Decouples syntax acquisition from algorithmic reasoning to address data scarcity and inference costs.
- Utilizes offline data synthesis and verification strategies to generate high-quality training examples.
- Achieves significant performance improvements on benchmark datasets with reduced data and cost.
Read more
Selective Left-Shift: Turning Test-Time Compute and Difficulty-based Curation into Training Data for Low-Resource Code Generation
Summary
This paper addresses the challenges faced by Large Language Models (LLMs) in code generation for Low-Resource Programming Languages (LRPLs) such as Julia and Ballerina. The authors propose a novel three-phase pipeline that separates syntax acquisition from algorithmic reasoning to overcome the limitations of data scarcity, high inference costs, and ineffective reinforcement learning. The first phase involves 'left-shifting' inference-time compute to an offline data synthesis engine that generates verified training examples using iterative compiler and test feedback. The second phase fine-tunes a Small Language Model (SLM) on this synthetic data to instill strong syntactic priors. The final phase employs Reinforcement Learning with Verifiable Rewards (RLVR), leveraging language-agnostic input/output tests to guide exploration and avoid syntax errors. The proposed method significantly enhances the performance of the Qwen3-8B model, achieving notable improvements in pass rates on benchmark datasets while using substantially less data and cost compared to existing state-of-the-art methods. Additionally, the pipeline demonstrates generalizability to Ballerina, achieving a 49.7% pass rate on MultiPL-E, showcasing its effectiveness across different low-resource languages.
Methodology
The methodology consists of three main phases: (1) an offline data synthesis engine that generates verified training examples using iterative compiler and test feedback, (2) fine-tuning a Small Language Model (SLM) on this synthetic data to embed syntactic priors, and (3) applying Reinforcement Learning with Verifiable Rewards (RLVR) based on difficulty-curated datasets to enhance learning signals.
Results
The proposed pipeline improved pass rates by up to +7.6 points on MultiPL-E and +14.2 points on Agnostics LiveCodeBench for Julia, while using only one-third of the data and one-sixth of the cost compared to the previous state-of-the-art. It also achieved a 49.7% pass rate on MultiPL-E for Ballerina, demonstrating the method's effectiveness across different low-resource languages.
Implications
The findings suggest that the proposed pipeline can significantly enhance code generation capabilities for low-resource programming languages, potentially leading to broader adoption and improved software development practices in diverse programming environments. This approach may also inspire similar methodologies in other domains facing data scarcity challenges.
Image classification via a quantum-inspired strategy involving a mixture of experts
Computer Vision
Theory
Efficient ML
- Introduces a hybrid classical-quantum framework for image classification.
- Utilizes a mixture of experts approach to enhance classification accuracy.
- Demonstrates a significant reduction in failure rates for image classification tasks.
- Shows that the quantum-inspired strategy is computationally feasible on GPU workstations.
Read more
Image classification via a quantum-inspired strategy involving a mixture of experts
Summary
This paper presents a novel hybrid classical-quantum framework for image classification that employs a quantum-inspired strategy utilizing a mixture of experts. The authors propose a method that integrates quantum techniques, such as amplitude encoding and local unitary operations, with classical neural network approaches. The framework involves encoding images into quantum states, applying convolution through quantum operations, and processing the extracted features using multiple experts with different parameters. The final classification is performed by a fully connected neural network. The proposed method is tested on the MNIST and Fashion-MNIST datasets, demonstrating that the joint analysis of multiple experts significantly outperforms individual expert performance and reduces the failure rate of image classification by approximately 50%. The computational overhead of this quantum-inspired approach is manageable on GPU workstations, suggesting its practicality compared to traditional methods. Furthermore, the authors discuss the potential for executing the quantum components on actual quantum processors, indicating a pathway for future research in quantum machine learning.
Methodology
The methodology involves several key components: amplitude encoding of images into quantum states, convolutional smearing using local unitary operations, and the introduction of multiple experts for feature extraction. The features from different experts are then processed by a standard fully connected neural network for classification. The approach is implemented using PyTorch on GPU workstations, with a focus on maintaining efficiency while leveraging quantum principles.
Results
The results indicate that the joint analysis of multiple experts leads to improved classification performance compared to individual expert analyses. Specifically, the proposed method reduces the failure rate of image classification by about 50% on the MNIST and Fashion-MNIST datasets. The computational overhead is found to be moderate, making the approach a viable alternative to existing classical schemes.
Implications
This research has significant implications for the field of image classification, particularly in scenarios where rapid and efficient processing of large datasets is required. The integration of quantum-inspired techniques may pave the way for advancements in machine learning algorithms, potentially leading to more effective solutions in various applications, including surveillance, medical imaging, and autonomous systems.
Ensemble Diversity Optimization for Subjective Supervision
NLP
Optimization
Theory
- Introduces Ensemble Diversity Optimization (EDO) for subjective NLP tasks.
- EDO optimizes ensemble structure and diversity through a unified differentiable objective.
- Employs a signed diversity regularizer to manage annotator disagreement effectively.
- Demonstrates significant improvements in probabilistic calibration and alignment with annotator distributions.
Read more
Ensemble Diversity Optimization for Subjective Supervision
Summary
This paper addresses the challenges of subjective NLP tasks characterized by annotator disagreement, proposing a novel framework called Ensemble Diversity Optimization (EDO). EDO optimizes ensemble weights, effective cardinality, and calibration through a unified differentiable objective, allowing for end-to-end learning of ensemble composition and size. The framework employs a signed diversity regularizer to manage disagreement, either preserving or suppressing it based on validation data. This approach prevents ensemble collapse and facilitates a balance between utility and calibration. EDO integrates a soft F1 surrogate, class-weighted cross-entropy for addressing class imbalance, and reliability-weighted diversity to control intra-ensemble variability. Experiments on four subjective text-classification benchmarks demonstrate that EDO significantly enhances probabilistic calibration, achieving a reduction in cross-entropy by 40-78% compared to various baselines while maintaining competitive F1 scores and better alignment with annotator distributions. The results highlight EDO's effectiveness in modeling human subjectivity in supervised learning settings.
Methodology
The EDO framework utilizes GumbelβSoftmax relaxation for differentiable learning of ensemble structure and cardinality. It incorporates a signed, reliability-weighted diversity regularizer to control the optimization process, allowing for either preservation or suppression of disagreement based on the nature of the annotator variability. The optimization balances predictive utility, calibration, and internal diversity through a multi-objective approach.
Results
EDO was tested on four subjective text-classification datasets, showing a reduction in cross-entropy by 40-78% compared to baseline methods. It also achieved lower Brier scores while maintaining competitive F1 scores and better alignment with the distribution of annotator responses, demonstrating its effectiveness in improving probabilistic calibration.
Implications
The findings suggest that EDO can be a valuable tool for enhancing model performance in subjective NLP tasks, where traditional methods may fail to capture the complexity of human judgment. This approach could be applied in areas such as content moderation, sentiment analysis, and any domain where annotator disagreement is prevalent.
Linear Attention Architectures: Mechanisms, Trade-offs, and Cross-Layer Routing
NLP
Large Language Models
Efficient ML
- Introduces a unified framework for comparing softmax and linear attention architectures.
- Demonstrates that Kimi Delta Attention with Muon optimizer achieves the lowest validation loss.
- Gated DeltaNet shows the highest normalized training throughput among the architectures tested.
- Presents Cross-Layer Value Routing (CLVR) as a lightweight mechanism that improves validation loss.
Read more
Linear Attention Architectures: Mechanisms, Trade-offs, and Cross-Layer Routing
Summary
This paper investigates the limitations of traditional self-attention mechanisms in transformer models, particularly their quadratic computational cost with respect to sequence length. It presents a comparative study of softmax attention and four recent linear attention architectures: DeltaNet, Gated DeltaNet, Kimi Delta Attention, and Gated DeltaNet-2. The authors express these architectures in a unified recurrent-memory framework, highlighting differences in expressivity, memory decay, and implementation complexity. The study includes extensive experiments with 350M-parameter models trained on 15B tokens, focusing on various aspects such as optimizer performance, training throughput, and sequence-length runtime. Notably, Kimi Delta Attention with the Muon optimizer achieved the lowest validation loss, while Gated DeltaNet exhibited the highest training throughput. The paper also introduces lightweight cross-layer routing mechanisms, specifically Cross-Layer Value Routing (CLVR), which showed modest improvements in validation loss for DeltaNet and Gated DeltaNet architectures. The findings aim to clarify the design space of linear attention architectures, emphasizing the trade-offs involved and the potential for future research in cross-layer routing.
Methodology
The authors employ a comparative analysis of various linear attention architectures using a common recurrent-memory notation. They conduct experiments on 350M-parameter models, assessing training throughput, validation loss, and the effects of different optimizers and learning rates. The study also evaluates cross-layer routing mechanisms to enhance memory efficiency.
Results
The experiments reveal that Kimi Delta Attention with the Muon optimizer yields the lowest final validation loss, while Gated DeltaNet achieves the highest training throughput. The introduction of CLVR provides a modest reduction in validation loss for DeltaNet and Gated DeltaNet architectures, indicating potential benefits of cross-layer routing.
Implications
The findings suggest that linear attention architectures can be optimized for both efficiency and performance, paving the way for more scalable models in natural language processing tasks. The insights into cross-layer routing mechanisms may inspire future research to further enhance the capabilities of recurrent memory architectures.
Principled Analysis of Deep Reinforcement Learning Evaluation and Design Paradigms
Reinforcement Learning
Theory
- Critiques the implicit assumption of monotonic performance relationships in DRL research.
- Introduces theoretical foundations on scaling laws affecting algorithm design and evaluation.
- Demonstrates through experiments that many DRL algorithms are biased due to flawed evaluation paradigms.
- Calls for a reevaluation of canonical methodological choices in DRL to improve research accuracy.
Read more
Principled Analysis of Deep Reinforcement Learning Evaluation and Design Paradigms
Summary
This paper presents a comprehensive analysis of the evaluation and design paradigms in deep reinforcement learning (DRL). It highlights the significant progress made in the field over the past decade, particularly through the use of deep neural networks to approximate state-action value functions. The author critiques the common implicit assumption that performance rankings of DRL algorithms have a monotonic relationship with sample complexity, which has led to incorrect conclusions in prior research. The paper introduces theoretical foundations regarding scaling laws in DRL, demonstrating that the performance profile of algorithms does not consistently correlate with data regimes. Through extensive large-scale experiments, the author reveals that many recent algorithms evaluated in low-data regimes are systematically biased due to these implicit assumptions. The findings emphasize the need for a reevaluation of methodological choices in DRL research to avoid misdirecting future research efforts.
Methodology
The paper employs a theoretical analysis of evaluation paradigms and conducts large-scale experiments on a diverse set of DRL algorithms in both low-data and high-data regimes, specifically using the Arcade Learning Environment benchmark to assess performance discrepancies.
Results
The results indicate that the performance profiles of DRL algorithms exhibit a non-monotonic relationship with sample complexity, leading to systematic biases in evaluations. The findings challenge the validity of conclusions drawn from existing low-data regime studies and suggest that many established methodologies may misguide future research directions.
Implications
The implications of this work are significant for the field of reinforcement learning, as it calls for a critical reassessment of evaluation practices and methodological choices. By addressing these biases, researchers can improve the reliability of performance comparisons and foster more accurate advancements in DRL algorithms.
A Practical Investigation of Training-free Relaxed Speculative Decoding
NLP
Large Language Models
Efficient ML
- Relaxed speculative decoding can yield significant speed-ups but requires careful evaluation of model capabilities.
- A unified framework for relaxed speculative decoding helps clarify the relationships and implementations of various methods.
- Benchmarking across different inference settings allows for fair comparisons of relaxed decoding approaches.
- Many relaxed approaches depend on a high-quality drafter model, limiting their applicability for lightweight scenarios.
Read more
A Practical Investigation of Training-free Relaxed Speculative Decoding
Summary
This paper investigates training-free relaxed speculative decoding techniques for accelerating sampling from autoregressive large language models (LLMs). Speculative decoding utilizes a faster auxiliary model to draft tokens, which are then verified by the LLM, traditionally preserving the sampling distribution. The authors explore the potential benefits of relaxing this strict distribution preservation, which can lead to increased speed and capability trade-offs. They unify existing methods into a shared framework and benchmark various approaches across contemporary settings. Key findings indicate that while relaxed methods can enhance speed, they often require careful capability evaluation and may not be suitable for lightweight multi-token prediction drafters. The paper serves as a practical guide for researchers and practitioners, providing insights into the utility of relaxed speculative decoding in real-world applications.
Methodology
The authors present a primer on strict speculative decoding, followed by a taxonomy of relaxed speculative decoding methods. They benchmark these methods on modern drafter-verifier pairs and reasoning benchmarks, analyzing their performance under various inference-time parameters. The study also includes a unified framework to categorize existing approaches and their motivations.
Results
The benchmarking results demonstrate that while relaxed speculative decoding methods can improve speed, they often compromise on the model's capability. The findings highlight the necessity of evaluating the trade-offs between speed and performance when employing relaxed approaches.
Implications
The insights from this paper can guide practitioners in selecting appropriate speculative decoding methods for LLM applications, particularly in scenarios where latency is critical. The findings also contribute to the ongoing development of more efficient decoding strategies in natural language processing.
Optimal Learning Rate Scaling Depends on Data in Deep Scalar Linear Networks
Theory
Optimization
- Optimal learning rate scaling in deep scalar linear networks is data-dependent.
- Data-agnostic scaling rules fail to transfer effectively across network depths.
- The proposed data-dependent scaling leads to constant linear convergence rates.
- Similar effects are observed in deep scalar linear networks with residual connections.
Read more
Optimal Learning Rate Scaling Depends on Data in Deep Scalar Linear Networks
Summary
This paper investigates the gradient descent dynamics in deep scalar linear networks, demonstrating that the optimal learning rate scaling is inherently data-dependent. The authors establish that traditional data-agnostic scaling rules do not effectively transfer across different depths of the network. Instead, they propose a data-dependent scaling rule that leads to a constant linear convergence rate, irrespective of the network's depth. The study also extends to deep scalar linear networks with residual connections, confirming that similar data-dependent effects are observed. The findings challenge existing assumptions about hyperparameter transfer in deep learning, emphasizing the necessity of considering data characteristics when determining optimal learning rates.
Methodology
The authors analyze the gradient descent dynamics of deep scalar linear networks using exact solutions expressed through special functions, including hypergeometric and Lambert W functions. They derive the learning dynamics under a balanced initialization scheme and explore the implications of depth on learning rates, incorporating input correlations to reveal data-independent dynamics.
Results
The research shows that the optimal learning rate scaling is influenced by the data, leading to a constant linear convergence rate across all depths. This contrasts with traditional data-agnostic approaches, which do not maintain stability across varying depths. The findings are validated through theoretical analysis and comparisons with existing literature.
Implications
These results suggest that practitioners in deep learning should consider data characteristics when selecting learning rates, particularly in deep networks. The insights could lead to improved training strategies and hyperparameter tuning, enhancing model performance across various applications.
Who Gets Missed in the Tail? Thresholded Subgroup Underdiagnosis in Long-Tailed Chest X-ray Classification
Computer Vision
- Introduces the concept of thresholded class-subgroup underdiagnosis as a measure of fairness in CXR classification.
- Demonstrates that traditional performance metrics can obscure significant subgroup-specific underdiagnosis.
- Implements a diagnostic ladder to analyze the impact of class-level losses, subgroup weighting, and threshold selection.
- Shows substantial reductions in false negative rates for rare classes through tailored weighting and thresholding strategies.
Read more
Who Gets Missed in the Tail? Thresholded Subgroup Underdiagnosis in Long-Tailed Chest X-ray Classification
Summary
This paper addresses the issue of underdiagnosis in long-tailed chest X-ray (CXR) classification, particularly focusing on how rare-positive patients are missed after applying decision thresholds. The authors investigate this fairness problem through an audit lens, analyzing the performance of a multi-label CXR model on two datasets: VinDr-CXR and MIMIC-CXR/CXR-LT. They introduce a diagnostic ladder that differentiates between class-level losses, subgroup-aware weighting, group robustness, and threshold selection. The study reveals that traditional metrics like macro-mAP do not adequately capture the fairness of the model, as they may mask significant subgroup-specific underdiagnosis. The authors demonstrate that employing group-tail weighting and tail-aware thresholding significantly reduces false negative rates (FNR) for rare classes across different demographic subgroups. The findings emphasize that achieving fairness in CXR classification requires a nuanced understanding of the interplay between the model's findings, the subgroups involved, and the thresholds set for decision-making, rather than relying solely on label frequency or ranking metrics.
Methodology
The authors utilize a diagnostic ladder approach to analyze the performance of a CXR classification model. They apply various methods including binary cross-entropy loss, asymmetric loss, effective-number class weighting, and GroupDRO to assess the impact of these techniques on subgroup-specific performance. The study employs two datasets, VinDr-CXR and MIMIC-CXR/CXR-LT, and evaluates the model's performance using metrics such as false negative rates (FNR) and macro mean average precision (macro-mAP). Bootstrap methods are used for statistical validation of results.
Results
The study reports significant improvements in false negative rates (FNR) for rare classes after applying group-tail weighting and tail-aware thresholding. On the VinDr-CXR dataset, the tail FNR decreased from 0.665 to 0.269, with sex-specific FNR dropping from 0.705 to 0.157 and age-specific FNR from 0.822 to 0.133. For the MIMIC-CXR/CXR-LT dataset, tail FNR reduced from 0.866 to 0.741. Despite these improvements, the authors note that residual missed-positive rates remain high, indicating ongoing challenges in achieving complete fairness.
Implications
The findings underscore the necessity for healthcare AI systems to consider subgroup-specific performance metrics to ensure equitable diagnosis across diverse patient populations. This research could inform the development of more robust CXR classification models that minimize underdiagnosis in rare cases, ultimately improving patient outcomes and addressing fairness in medical AI applications.
SHIFT: Survival Prediction from Incomplete and Heterogeneous Genomic Data
Theory
- SHIFT is a transformer-based model that predicts survival from incomplete genomic data without the need for imputation.
- The model employs a variable-rate masking strategy during training to improve robustness to cross-cohort data variability.
- Incorporating incomplete cohorts in model training can enhance predictive performance on external datasets.
- SHIFT demonstrates strong generalization across different cancer types and heterogeneous genomic panels.
Read more
SHIFT: Survival Prediction from Incomplete and Heterogeneous Genomic Data
Summary
The paper presents SHIFT, a novel transformer-based model designed for survival prediction using incomplete genomic data. Traditional genomic prediction models struggle with the heterogeneity of genomic data across different institutions, often leading to the exclusion of patients with incomplete profiles or reliance on imputation methods that can distort biological signals. SHIFT addresses these challenges by employing a missingness-aware approach that utilizes masked self-attention and a feature-availability mask, allowing it to make predictions based solely on observed genomic features. The model is trained with a variable-rate feature masking strategy to enhance its robustness against varying patterns of missingness. Evaluations on glioblastoma and lung squamous cell carcinoma datasets demonstrate that SHIFT outperforms standard survival baselines and imputation-based methods, showcasing strong generalization across multiple cohorts, including those with significant cross-cohort panel mismatches. The findings suggest that incorporating patients from incomplete cohorts during model development can enhance performance on external datasets, advocating for a more inclusive approach in multi-center survival prediction.
Methodology
SHIFT utilizes a transformer architecture with masked self-attention to handle incomplete genomic inputs. It incorporates a feature-availability mask to ensure predictions are based only on observed features. The training process involves variable-rate feature masking, which simulates different patterns of missingness, allowing the model to learn inter-feature relationships effectively.
Results
SHIFT was evaluated on glioblastoma and lung squamous cell carcinoma datasets, showing superior predictive performance compared to traditional survival models and imputation-based approaches. The model maintained strong generalization capabilities across multiple cohorts, even in scenarios with severe cross-cohort panel mismatches. The inclusion of patients from incomplete cohorts during training was found to improve performance on external validation datasets.
Implications
The findings suggest that missingness-aware modeling can significantly enhance the robustness of survival predictions in precision oncology, allowing for more inclusive use of multi-center genomic data. This approach could lead to improved patient outcomes by leveraging diverse datasets that were previously excluded due to incomplete profiles.
Modular Pretraining Enables Access Control
Large Language Models
Efficient ML
Theory
- Introduces GRAM, a method for modular pretraining that enables selective capability management in AI models.
- Demonstrates that GRAM can effectively disable specific capabilities while preserving others, outperforming traditional methods like post-hoc unlearning.
- Shows significant cost reductions in training compared to training multiple models for different capabilities.
- Evaluates GRAM across various domains, including virology and cybersecurity, confirming its robustness and effectiveness.
Read more
Modular Pretraining Enables Access Control
Summary
The paper addresses the dual-use dilemma in AI, where capabilities can be beneficial or harmful, by proposing a novel pretraining method called gradient-routed auxiliary modules (GRAM). This method allows for the selective updating of modules within a neural network to induce specialization for different capabilities, enabling access control without the need for multiple separately trained models. GRAM operates by augmenting a dense transformer model with smaller MLP modules that are selectively activated during training based on the data being processed. This allows the model to effectively disable certain capabilities at inference time while maintaining performance on retained capabilities. The authors evaluate GRAM on both synthetic and realistic datasets across various domains, demonstrating that it can effectively isolate and manage dual-use capabilities. The results indicate that GRAM not only matches the performance of traditional data filtering methods but also offers significant cost savings in training, achieving a fivefold reduction in costs when managing multiple capability profiles. Furthermore, the capability isolation improves with model scale, suggesting that GRAM's advantages persist as models grow larger.
Methodology
The authors propose GRAM, which augments a dense transformer model with auxiliary MLP modules. During training, the model selectively enables these modules based on the current training batch, allowing for capability-specific updates. At inference, modules can be ablated to disable certain capabilities, effectively simulating a model trained on filtered data.
Results
GRAM closely matches the performance of data-filtered models in both capability retention and removal. It demonstrates robust performance across various datasets, achieving better capability removal than traditional methods, particularly in scenarios with malicious finetuning attempts. The analysis shows that as model size increases, the effectiveness of GRAM in isolating capabilities also improves.
Implications
The findings suggest that GRAM could be a viable solution for AI developers facing the dual-use dilemma, allowing for safer deployment of AI systems with sensitive capabilities. This method could facilitate more responsible AI usage by enabling differentiated access control based on user needs.
Uncertainty-gated selection for block-sparse attention
NLP
Large Language Models
Efficient ML
- Introduces a value-of-information approach to improve block-sparse attention selection.
- Enhances recall by expanding the selection set for uncertain queries without extra parameters.
- Demonstrates compatibility with existing block-scoring methods like Quest.
- Achieves significant performance improvements on standard benchmarks.
Read more
Uncertainty-gated selection for block-sparse attention
Summary
This paper introduces a novel approach to enhance block-sparse attention mechanisms in long-context language models by addressing the limitations of traditional top-k selection methods. The proposed uncertainty-gated selection framework treats the cutoff decision as a value-of-information (VoI) problem, allowing for a more informed selection of key blocks based on the decisiveness of the scores. By expanding the kept set of blocks for queries with uncertain cutoffs, the method improves recall and accuracy without requiring additional parameters or retraining. The approach is compatible with existing block-scoring methods, demonstrating significant performance improvements across various models and architectures. The empirical results show that the uncertainty-gated selection outperforms traditional top-k methods, achieving higher recall rates and maintaining dense accuracy while reducing computational time.
Methodology
The methodology involves a value-of-information (VoI) formulation for the selection cutoff, which computes a normalized cutoff margin for each query. This margin indicates the decisiveness of the top-k selection, allowing the model to expand the selection set for queries with high uncertainty. The approach is backbone-agnostic and can be integrated with various block-scoring methods, enhancing their performance without requiring retraining.
Results
The proposed method, referred to as router-on-Quest, achieved a paired recall of 0.75 on the LongBench-v2 medium dataset, significantly outperforming the traditional top-k method, which recorded a recall of 0.47. The router maintained high accuracy levels, preserving 0.81 and 0.89 of dense accuracy on different models while operating at reduced computational times, demonstrating its effectiveness across multiple architectures.
Implications
The findings suggest that incorporating uncertainty into selection mechanisms can lead to more robust and efficient long-context language models. This approach could be applied to various NLP tasks that require attention mechanisms, potentially improving performance in multi-hop reasoning and retrieval tasks where evidence is critical.
When Does Continual Learning Require Learning
Large Language Models
NLP
Reinforcement Learning
- Continual learning in LLMs should focus on increasing competence as the environment changes.
- The authors introduce a unified framework to evaluate various continual learning methods.
- Different learning strategies exhibit unique strengths and weaknesses depending on the nature of change.
- Prompt-based methods are quick to adapt but degrade on future tasks, while distillation methods are stable but slow to update.
Read more
When Does Continual Learning Require Learning
Summary
This paper addresses the challenge of continual learning in large language models (LLMs), arguing that the current focus on mitigating forgetting is insufficient. The authors propose a new framework that defines continual learning as the process of increasing model competence in response to changes in the environment, which they categorize along two axes: spatial (new domains) and temporal (data drift). They recast traditional LLM benchmarks into sequential problems and introduce a mechanism-agnostic protocol to evaluate various learning methods, including prompt-based approaches, supervised learning, reinforcement learning, and context compression. The study finds that different methods exhibit distinct trade-offs: prompt-based methods adapt quickly but struggle with future tasks, distillation methods accumulate knowledge steadily but are slow to update, and context compression improves efficiency without enhancing learning capabilities. Online reinforcement learning shows the best adaptability to knowledge updates but is sensitive to noise. The findings suggest that continual learning is not a singular capability; rather, it requires different strategies depending on the nature of environmental changes, providing insights for the development of more robust continual learning systems.
Methodology
The authors developed a unified framework for continual learning in LLMs, categorizing changes in the environment along spatial and temporal axes. They recast traditional benchmarks as sequential problems and employed a mechanism-agnostic protocol to compare various learning methods, including prompt optimization, supervised updates, online reinforcement learning, and context compression.
Results
The study revealed consistent trade-offs among the evaluated methods. Prompt-based methods achieved high backward accuracy but suffered in future tasks. Distillation methods maintained knowledge stability but were slow to adapt to new information. Context compression improved efficiency without significantly enhancing learning capabilities. Online reinforcement learning methods adapted well to updates but were sensitive to noise.
Implications
The findings suggest that designing continual learning systems requires an understanding of the specific types of environmental changes and the corresponding update behaviors needed. This could lead to more effective and adaptable LLMs in real-world applications.
Frequency-Domain Multi-Modality Transportation Modeling
Time Series
Multimodal
- FreMo effectively addresses the limitations of existing time-domain methods by leveraging frequency domain characteristics.
- The Modality-Wise Frequency Filter (MFF) enhances the quality of spectral components for each modality.
- The Frequency-Guided Synergy Integrator (FSI) enables selective information sharing across modalities based on frequency reliability.
- FreMo outperforms state-of-the-art methods in multi-modality transportation forecasting across diverse scenarios.
Read more
Frequency-Domain Multi-Modality Transportation Modeling
Summary
This paper addresses the challenges of multi-modality transportation forecasting, which involves integrating various transportation modes such as traffic flow and public transit. Traditional methods often struggle due to the distinct spectral characteristics of different modalities and their uneven interactions across frequencies. The authors propose a novel framework called Frequency-Domain Multi-Modality modeling (FreMo), which operates in the frequency domain to enhance cross-modality synergy. FreMo introduces a Modality-Wise Frequency Filter (MFF) to refine spectral components within each modality, focusing on informative frequencies while reducing noise. Additionally, it features a Frequency-Guided Synergy Integrator (FSI) that selectively aggregates information across modalities based on their frequency-dependent contributions. Extensive experiments on real-world datasets demonstrate that FreMo consistently outperforms state-of-the-art baselines, showcasing superior performance and generalization across various forecasting scenarios. The findings suggest that leveraging frequency domain characteristics can significantly improve multi-modality transportation modeling.
Methodology
The proposed FreMo framework operates in the frequency domain, utilizing a Modality-Wise Frequency Filter (MFF) to refine spectral components and a Frequency-Guided Synergy Integrator (FSI) to selectively aggregate information across modalities based on their frequency-dependent contributions.
Results
FreMo demonstrated consistent superiority over existing state-of-the-art methods in multi-modality transportation forecasting, achieving better performance and generalization across various real-world datasets.
Implications
The findings suggest that adopting frequency domain approaches can enhance the accuracy of multi-modality forecasting, which has significant implications for urban planning, traffic management, and smart city applications.
Dynamics of Gradient Descent with Large Step Size Near a Manifold of Flat Minima
Optimization
Theory
- Generalizes previous convergence results for GD with large step sizes to vector-valued outputs.
- Establishes a normal form for GD dynamics near a manifold of flat minima.
- Introduces a novel method for solving singular partial differential equations relevant to the analysis.
- Demonstrates that flat minima form a fiber bundle structure, enhancing understanding of optimization landscapes.
Read more
Dynamics of Gradient Descent with Large Step Size Near a Manifold of Flat Minima
Summary
This paper extends the theory of gradient descent (GD) dynamics with large step sizes, particularly in the context of overparametrized least-squares problems. The authors build upon previous work that established convergence results for GD near isolated flat minima, generalizing these results to vector-valued outputs and to neighborhoods of manifolds of flat minima. The paper addresses significant technical challenges, including the solution of a singular partial differential equation, and introduces a novel method that may have independent interest. The authors demonstrate that GD with large step sizes behaves like Riemannian gradient descent on the sharpness along the manifold of flat minima, and they provide new structural insights into deep matrix factorization problems. The findings suggest that the set of flat minima forms a fiber bundle over a product of spheres, with sharpness exhibiting Morse-Bott behavior along the manifold.
Methodology
The authors utilize dynamical systems theory to analyze the behavior of gradient descent in the context of overparametrized least-squares problems. They develop a normal form for GD dynamics, addressing the complexities introduced by higher codimension problems and manifolds of flat minima. The analysis involves solving a singular partial differential equation and applying geometric hypotheses to establish convergence results.
Results
The paper successfully extends the convergence results of gradient descent with large step sizes to a broader class of problems, demonstrating that GD behaves as Riemannian gradient descent on the sharpness along the manifold of flat minima. The authors provide a comprehensive framework that includes new structural results for deep matrix factorization, confirming the fiber bundle structure of flat minima and the Morse-Bott nature of sharpness.
Implications
The findings have significant implications for the optimization of deep learning models, particularly in understanding the dynamics of gradient descent in non-convex landscapes. The results may inform strategies for training neural networks more effectively, particularly in scenarios where large step sizes are employed. Additionally, the insights into the structure of flat minima could lead to improved model generalization and performance.