AI-generated summaries
Today's ML research,
without the noise.
Daily summaries of the latest machine learning papers from arXiv, processed every 8 hours.
69
Papers today
8h
Update frequency
7
Days of history
Forecasting with Guidance: Representation-Level Supervision for Time Series Forecasting
Time Series
- Identifies limitations of error-only supervision in deep learning-based time series forecasting.
- Introduces ReGuider, a plug-in method for representation-level supervision using pretrained time series foundation models.
- Demonstrates that ReGuider enhances the expressiveness of temporal representations in forecasting models.
- Shows consistent improvements in forecasting accuracy across various datasets and architectures.
Read more
Forecasting with Guidance: Representation-Level Supervision for Time Series Forecasting
Summary
This paper addresses the limitations of traditional error-based objectives in time series forecasting (TSF) models, which often lead to overly smooth predictions and a lack of representation of critical temporal dynamics. The authors propose a novel method called ReGuider, which integrates pretrained time series foundation models as semantic teachers to enhance the learning of temporal representations in forecasting models. By extracting intermediate embeddings from these foundation models and aligning them with the encoder representations of the target forecasting model, ReGuider provides representation-level supervision that enriches the temporal semantics of learned embeddings. This approach is model-agnostic, allowing it to be applied across various forecasting architectures without altering their structure. Extensive experiments demonstrate that ReGuider consistently improves forecasting accuracy across diverse datasets and models, highlighting its effectiveness and versatility in enhancing time series forecasting performance.
Methodology
The methodology involves using ReGuider to align intermediate embeddings from pretrained time series foundation models with the encoder representations of the target forecasting model. This alignment process is designed to enhance the temporal semantics captured by the model, thereby improving its predictive capabilities without increasing architectural complexity.
Results
The experimental results indicate that ReGuider significantly enhances forecasting accuracy across a variety of datasets and forecasting architectures. The method demonstrates robustness and generalizability, confirming its effectiveness in improving the quality of temporal representations in time series forecasting.
Implications
The findings suggest that incorporating external semantic supervision can substantially improve the performance of time series forecasting models. This approach may have broad applications in fields such as finance, healthcare, and climate science, where accurate forecasting is critical.
IPatch: A Multi-Resolution Transformer Architecture for Robust Time-Series Forecasting
Time Series
- IPatch integrates point-wise and patch-wise representations for improved time-series forecasting.
- The architecture includes mechanisms for capturing both fine-grained and coarse temporal dependencies.
- Experimental results show significant improvements in forecasting accuracy and robustness compared to traditional methods.
- IPatch is effective across various prediction horizons and benchmark datasets.
Read more
IPatch: A Multi-Resolution Transformer Architecture for Robust Time-Series Forecasting
Summary
The paper introduces IPatch, a novel multi-resolution Transformer architecture designed to enhance the accuracy and robustness of time-series forecasting. Traditional forecasting methods often struggle to balance the need for capturing both short-term fluctuations and long-range dependencies. While point-wise representations maintain fine-grained temporal details, they are computationally expensive and less effective for long sequences. Conversely, patch-wise representations improve efficiency by aggregating consecutive time steps but may lose critical short-term dynamics. IPatch addresses these challenges by integrating both point-wise and patch-wise tokens, allowing the model to capture temporal information at multiple resolutions. The architecture features a global inter-patch attention mechanism for modeling coarse temporal dependencies and an intra-patch autocorrelation module to preserve fine-grained details. Experimental results across seven benchmark datasets demonstrate that IPatch consistently outperforms single-representation baselines in forecasting accuracy, robustness to noise, and generalization across various prediction horizons.
Methodology
The IPatch architecture employs a hybrid approach that combines point-wise and patch-wise representations. It utilizes a global inter-patch attention mechanism to capture long-range dependencies across segments and an intra-patch autocorrelation module to maintain fine-grained temporal structure within patches. This dual representation allows for efficient modeling of time-series data while preserving critical temporal details.
Results
The experiments conducted on seven benchmark datasets indicate that IPatch significantly enhances forecasting accuracy and robustness against noise. It demonstrates superior generalization capabilities across different prediction horizons when compared to existing single-representation baselines.
Implications
The development of IPatch has potential applications in various fields requiring accurate time-series forecasting, such as finance, weather prediction, and supply chain management. Its ability to effectively model both short-term and long-range dependencies could lead to more reliable forecasting tools in complex and volatile environments.
i-IF-Learn: Iterative Feature Selection and Unsupervised Learning for High-Dimensional Complex Data
Interpretability
- Introduction of i-IF-Learn, a framework for joint feature selection and clustering.
- Adaptive feature selection statistic that balances supervised and unsupervised signals.
- Utilization of low-dimensional embeddings to enhance clustering performance.
- Empirical results show significant improvements over classical and deep clustering methods.
Read more
i-IF-Learn: Iterative Feature Selection and Unsupervised Learning for High-Dimensional Complex Data
Summary
The paper presents i-IF-Learn, an innovative iterative framework designed to tackle the challenges of unsupervised learning in high-dimensional datasets, particularly those with irrelevant or noisy features. The authors emphasize the importance of identifying influential features that significantly contribute to clustering outcomes. i-IF-Learn integrates feature selection and clustering in a joint manner, utilizing an adaptive feature selection statistic that combines pseudo-label supervision with unsupervised signals. This approach dynamically adjusts based on the reliability of intermediate labels, effectively mitigating error propagation that often plagues iterative methods. The framework employs low-dimensional embeddings, specifically PCA and Laplacian eigenmaps, followed by k-means clustering, to enhance performance. Empirical evaluations on gene microarray and single-cell RNA-seq datasets demonstrate that i-IF-Learn outperforms classical and deep clustering baselines. Additionally, the selected influential features serve as a preprocessing step that significantly boosts the performance of downstream models such as DeepCluster, UMAP, and VAE, underscoring the effectiveness of targeted feature selection in improving clustering accuracy and interpretability.
Methodology
The i-IF-Learn framework employs an iterative approach that combines feature selection and clustering. It utilizes an adaptive feature selection statistic that adjusts based on the reliability of pseudo-labels. The framework incorporates low-dimensional embeddings (PCA and Laplacian eigenmaps) followed by k-means clustering to improve the clustering process and output influential features.
Results
Numerical experiments on gene microarray and single-cell RNA-seq datasets indicate that i-IF-Learn significantly outperforms classical clustering methods and deep learning-based approaches. The framework not only identifies influential features but also enhances the performance of subsequent clustering methods when these features are used as preprocessing.
Implications
The findings suggest that i-IF-Learn can be a valuable tool for data interpretation and clustering in high-dimensional datasets across various domains, including genomics and other fields where feature relevance is crucial. The method's ability to improve downstream model performance highlights its potential for broader applications in unsupervised learning tasks.
Unveiling Hidden Convexity in Deep Learning: a Sparse Signal Processing Perspective
Theory
Optimization
Interpretability
- Convex equivalences of ReLU neural networks can simplify optimization and enhance theoretical understanding.
- Reframing neural network training as a convex optimization task allows for efficient global optimization.
- The paper presents an equivalence theorem connecting two-layer ReLU networks to convex group Lasso problems.
- Experimental results indicate performance benefits when applying convex optimization frameworks to neural network training.
Read more
Unveiling Hidden Convexity in Deep Learning: a Sparse Signal Processing Perspective
Summary
This paper explores the non-convex nature of deep neural networks (DNNs) and proposes a novel perspective by leveraging concepts from sparse signal processing to uncover hidden convexities in the loss landscapes of certain neural network architectures, particularly those utilizing Rectified Linear Unit (ReLU) activation functions. The authors argue that reframing the training of DNNs as a convex optimization problem can lead to globally optimal solutions, enhancing the interpretability and robustness of the models. They present an equivalence theorem linking two-layer ReLU networks to convex group Lasso problems, demonstrating that deeper networks can also be analyzed through similar convex formulations. The paper discusses the geometric insights gained from this approach and provides experimental results that show performance improvements when training neural networks as convex models. The authors conclude by addressing the remaining challenges in the convex analysis of neural networks and suggesting future research directions.
Methodology
The authors utilize a theoretical framework that connects deep learning with sparse signal processing, specifically focusing on convex optimization techniques. They derive an equivalence theorem for two-layer ReLU networks and extend this to deeper architectures, employing Lasso-type models and structure-inducing regularization to reformulate the training process as a convex optimization problem.
Results
The paper demonstrates that by treating neural network training as a convex optimization problem, it is possible to achieve globally optimal solutions, which leads to improved generalization and robustness of the models. Experimental results support the theoretical claims, showing enhanced performance when convex formulations are applied.
Implications
This research has significant implications for the training and interpretation of deep neural networks, particularly in applications requiring stability and robustness, such as signal processing. It encourages the adoption of convex optimization techniques in deep learning, potentially leading to more reliable and interpretable models.
Transformers Trained via Gradient Descent Can Provably Learn a Class of Teacher Models
Theory
Optimization
- One-layer transformers can effectively learn from a general class of teacher models.
- The paper establishes a tight convergence guarantee for population loss with a rate of Θ(1/T).
- Transformers demonstrate robust out-of-distribution generalization capabilities.
- The study identifies a bilinear structure that underpins various learning tasks, enabling unified theoretical guarantees.
Read more
Transformers Trained via Gradient Descent Can Provably Learn a Class of Teacher Models
Summary
This paper investigates the theoretical foundations of transformers, specifically their ability to learn from a class of teacher models through gradient descent. The authors focus on one-layer transformers with simplified 'position-only' attention and prove that they can recover all parameter blocks of various teacher models, including convolution layers, graph convolution layers, and classic statistical learning models. The study establishes a tight convergence guarantee for the population loss, demonstrating that transformers can generalize well to out-of-distribution data. The analysis identifies a bilinear structure shared among different learning tasks, allowing for unified learning guarantees. This work not only extends previous findings but also provides a broader theoretical framework for understanding transformers' capabilities across diverse applications.
Methodology
The authors theoretically analyze one-layer transformers trained via gradient descent to learn from teacher models characterized by a bilinear structure. They establish convergence guarantees and generalization bounds, comparing their results with existing literature on specific learning tasks.
Results
The study proves that one-layer transformers can recover teacher model parameters and achieve optimal population loss. The convergence rate is established as Θ(1/T), and the transformers show competitive performance against teacher models across various tasks, including those related to sparse token selection and group sparse linear classification.
Implications
This research provides a theoretical foundation for understanding transformers' learning capabilities, which could enhance their application in various fields such as natural language processing, computer vision, and beyond. The findings may lead to improved transformer architectures and training methodologies.
PoiCGAN: A Targeted Poisoning Based on Feature-Label Joint Perturbation in Federated Learning
Federated Learning
Computer Vision
Generative Models
- PoiCGAN introduces a targeted poisoning attack framework that enhances stealthiness while maintaining model performance.
- The method leverages dual-feature collaborative perturbations to minimize the impact on the main task's accuracy.
- Experiments show a significant increase in attack success rates compared to existing methods.
- The approach highlights new vulnerabilities in Federated Learning systems, necessitating stronger defenses.
Read more
PoiCGAN: A Targeted Poisoning Based on Feature-Label Joint Perturbation in Federated Learning
Summary
The paper introduces PoiCGAN, a novel targeted poisoning attack framework designed for Federated Learning (FL) in industrial image classification tasks. Traditional poisoning attacks often lead to significant degradation in model performance, making them detectable. PoiCGAN addresses this limitation by employing a feature-label collaborative perturbation strategy that allows for the generation of poisoned samples while maintaining the accuracy of the main task. The method utilizes a Conditional Generative Adversarial Network (CGAN) architecture, where the discriminator is trained with misaligned image-label pairs to guide the generator in creating poisoned samples that misclassify images from a source class to a target class. The experiments demonstrate that PoiCGAN achieves an attack success rate of 83.97%, significantly higher than baseline methods, while only reducing the main task's accuracy by less than 8.87%. This approach not only enhances the stealthiness of the attacks but also reveals new vulnerabilities in FL systems, paving the way for improved defenses against such threats.
Methodology
PoiCGAN employs a Conditional Generative Adversarial Network (CGAN) framework, where the discriminator is trained with misaligned image-label pairs. This setup allows the generator to produce poisoned samples that misclassify images from a defined source class to a target class, effectively executing a targeted poisoning attack while preserving the main task's accuracy.
Results
The proposed method achieved an attack success rate of 83.97%, outperforming baseline poisoning attack methods. Additionally, the reduction in the main task's accuracy was kept below 8.87%, demonstrating the effectiveness and stealthiness of the attack.
Implications
The findings suggest that existing Federated Learning systems are vulnerable to targeted poisoning attacks, which could have significant implications for industrial applications relying on image classification. The work lays the groundwork for developing more robust defenses against such attacks.
End-to-End Efficient RL for Linear Bellman Complete MDPs with Deterministic Transitions
Reinforcement Learning
Theory
Efficient ML
- Introduces a computationally efficient algorithm for linear Bellman complete MDPs with deterministic transitions.
- Algorithm is end-to-end efficient for finite action spaces and requires only an argmax oracle for larger action spaces.
- Achieves ε-optimal policy with polynomial sample and computational complexity.
- Addresses a significant gap in existing literature regarding exploration in linear Bellman complete MDPs.
Read more
End-to-End Efficient RL for Linear Bellman Complete MDPs with Deterministic Transitions
Summary
This paper addresses the challenge of reinforcement learning (RL) in Markov Decision Processes (MDPs) characterized by linear Bellman completeness, where the Bellman backup of any linear value function remains linear. Previous algorithms in this domain have been limited by either small action spaces or strong oracle assumptions regarding the feature space. The authors propose a computationally efficient algorithm that operates effectively in environments with deterministic transitions, stochastic initial states, and stochastic rewards. For finite action spaces, the algorithm is end-to-end efficient, while for larger or infinite action spaces, it requires only a standard argmax oracle over actions. The algorithm achieves an ε-optimal policy with sample and computational complexity that is polynomial in the horizon, feature dimension, and 1/ε. This work fills a significant gap in the literature by providing a practical solution for learning near-optimal policies in linear Bellman complete MDPs, particularly in settings with large action spaces, which have been previously unresolved.
Methodology
The authors develop an algorithm that leverages linear function approximation within the framework of linear Bellman complete MDPs. The algorithm is designed to efficiently explore the state-action space while ensuring that the Bellman backups remain linear. It utilizes a combination of deterministic transitions and stochastic initial states and rewards to facilitate learning in complex environments.
Results
The proposed algorithm successfully learns an ε-optimal policy in linear Bellman complete MDPs with deterministic transitions, demonstrating polynomial sample and computational complexity. This represents a significant advancement over previous methods that either required restrictive assumptions or were computationally intractable.
Implications
The findings of this paper have important implications for the field of reinforcement learning, particularly in applications involving large or infinite action spaces. The algorithm can be applied in various domains such as robotics, game playing, and other sequential decision-making tasks where efficient exploration and learning are critical.
Lagrangian Relaxation Score-based Generation for Mixed Integer linear Programming
Optimization
- Introduction of SRG, a generative framework for MILP solving.
- Utilization of Lagrangian relaxation to enhance solution quality and feasibility.
- Joint modeling of decision variables to overcome independence assumptions.
- Demonstrated superior performance compared to existing machine learning baselines.
Read more
Lagrangian Relaxation Score-based Generation for Mixed Integer linear Programming
Summary
This paper introduces SRG, a novel generative framework designed to enhance the efficiency of solving Mixed Integer Linear Programming (MILP) problems. Traditional Predict-and-Search (PaS) methods face limitations due to their reliance on variable independence and deterministic predictions, which can hinder solution diversity and quality. SRG addresses these issues by employing Lagrangian relaxation-guided stochastic differential equations (SDEs) to generate diverse and high-quality solution candidates. The framework captures inter-variable dependencies through convolutional kernels and incorporates a guided sampling process that steers the generation towards feasible and optimal regions. The authors demonstrate that optimizing the Lagrangian-relaxation objective aligns the generative process with a refined target distribution, enhancing the quality of solutions produced. SRG not only outperforms existing machine learning baselines across various benchmarks but also exhibits strong zero-shot transferability, achieving competitive optimality on unseen problem instances while significantly reducing computational overhead. This work represents a significant advancement in the application of machine learning techniques to MILP, offering a robust foundation for efficient problem-solving in combinatorial optimization.
Methodology
The SRG framework employs a guided reverse-time stochastic differential equation (SDE) to generate solutions, integrating Lagrangian relaxation to penalize constraint violations and suboptimality. It uses convolutional kernels to model inter-variable dependencies, allowing for the generation of diverse solution candidates from a Gaussian distribution towards a refined target distribution.
Results
SRG consistently outperformed existing machine learning baselines in terms of solution quality across multiple public benchmarks. It also demonstrated strong zero-shot transferability, achieving competitive optimality with state-of-the-art exact solvers on unseen problem instances while reducing computational overhead.
Implications
The proposed SRG framework has significant implications for real-world applications of MILP, such as production planning, resource allocation, and energy management. Its ability to generate diverse and high-quality solutions efficiently could enhance decision-making processes in time-sensitive environments.
CoordLight: Learning Decentralized Coordination for Network-Wide Traffic Signal Control
Reinforcement Learning
Optimization
- Introduces Queue Dynamic State Encoding (QDSE) for enhanced traffic state representation.
- Develops Neighbor-aware Policy Optimization (NAPO) to improve agent coordination.
- Demonstrates superior performance over existing traffic signal control methods.
- Addresses challenges of partial observability and decentralized decision-making.
Read more
CoordLight: Learning Decentralized Coordination for Network-Wide Traffic Signal Control
Summary
CoordLight is a novel framework that leverages Multi-Agent Reinforcement Learning (MARL) to enhance decentralized traffic signal control across urban networks. The framework addresses the challenges of partial observability and coordination among agents (traffic signals) by introducing Queue Dynamic State Encoding (QDSE), a new state representation that captures vehicle queuing dynamics. This representation allows agents to better analyze and respond to local traffic conditions. Additionally, the Neighbor-aware Policy Optimization (NAPO) algorithm is proposed, which incorporates an attention mechanism to identify and prioritize interactions with influential neighboring agents. This facilitates improved coordination and collaboration among agents, leading to more effective decision-making. The authors conducted extensive evaluations using three real-world traffic datasets, demonstrating that CoordLight outperforms existing state-of-the-art traffic signal control methods, achieving superior performance across various traffic conditions and network configurations.
Methodology
The methodology involves the development of a MARL framework that includes a novel state representation (QDSE) based on vehicle queuing models and an advanced coordination algorithm (NAPO) that utilizes an attention mechanism to discern dependencies among neighboring agents. The framework is evaluated using real-world traffic datasets to assess its effectiveness in optimizing traffic signal control.
Results
CoordLight consistently outperformed traditional traffic signal control methods across three real-world datasets, demonstrating its effectiveness in managing traffic flow and reducing congestion in urban environments. The empirical results indicate significant improvements in traffic management efficiency compared to existing approaches.
Implications
The findings suggest that CoordLight can be effectively applied to urban traffic management systems, potentially leading to reduced congestion, improved travel times, and enhanced sustainability in rapidly urbanizing areas. The framework's decentralized approach may also facilitate scalability in larger urban networks.
Causal Discovery in Action: Learning Chain-Reaction Mechanisms from Interventions
Theory
Graph Learning
- Causal discovery in chain-reaction systems can be achieved through blocking interventions.
- The proposed method provides a unique identification of causal structures with finite-sample guarantees.
- Experiments show that the method outperforms observational heuristics in complex causal scenarios.
- The approach is applicable to various real-world systems exhibiting cascade-like structures.
Read more
Causal Discovery in Action: Learning Chain-Reaction Mechanisms from Interventions
Summary
This paper addresses the challenge of causal discovery in chain-reaction systems, where components activate sequentially and upstream failures suppress downstream effects. The authors propose a method to uniquely identify the causal structure of such systems through blocking interventions that prevent individual components from activating. They introduce a minimal estimator with finite-sample guarantees, demonstrating exponential error decay and logarithmic sample complexity. The methodology is validated through experiments on synthetic models and various chain-reaction environments, showing that the proposed approach reliably recovers causal structures from a limited number of interventions, outperforming observational heuristics in scenarios with delayed or overlapping causal effects. The work emphasizes the importance of interventions in resolving ambiguities that arise in purely observational settings.
Methodology
The authors model interactions in chain-reaction systems using a Structural Causal Model (SCM) with binary variables representing object activations. They apply blocking interventions to identify causal relationships and derive a finite-sample estimator with theoretical guarantees for error decay and sample complexity.
Results
The proposed method successfully identifies the causal structure from single-object blocking interventions, demonstrating exponential error decay and logarithmic sample complexity in empirical tests. The approach consistently recovers the correct causal graph across various chain-reaction environments, even in the presence of stochastic variations.
Implications
This research has potential applications in fields such as mechanical safety systems, biological signaling pathways, and dependency management in software systems. It provides a framework for understanding and modeling complex causal relationships in systems where traditional observational methods may fail.
Latent Algorithmic Structure Precedes Grokking: A Mechanistic Study of ReLU MLPs on Modular Arithmetic
Theory
Interpretability
- ReLU MLPs exhibit near-binary square wave input weights rather than sinusoidal weights during grokking.
- The output weights maintain a phase-sum relation, indicating a structured internal representation.
- An idealized MLP constructed from extracted parameters achieves high accuracy despite the original model's poor performance.
- Grokking sharpens latent algorithmic structures rather than discovering new algorithms.
Read more
Latent Algorithmic Structure Precedes Grokking: A Mechanistic Study of ReLU MLPs on Modular Arithmetic
Summary
This paper investigates the phenomenon of grokking in ReLU multi-layer perceptrons (MLPs) trained on modular arithmetic tasks, specifically the addition of two integers modulo 97. Grokking is characterized by a delayed rise in validation accuracy after initial memorization of training data. Previous studies have identified sinusoidal weight distributions in transformers and MLPs during grokking, but this study finds that ReLU MLPs instead learn near-binary square wave input weights. The research demonstrates that the output weight distributions follow a phase-sum relation, even in models that fail to grok. By applying discrete Fourier transform (DFT) to extract the frequency and phase of neuron weights, the authors construct an idealized MLP that achieves 95.5% accuracy using parameters from a model trained on noisy data that only achieved 0.23% accuracy. This suggests that grokking does not discover new algorithms but refines an already encoded algorithmic structure during the memorization phase, leading to improved generalization.
Methodology
The study involved training a ReLU MLP on a modular addition task with a dataset of integer triples. The model's input weights were analyzed using discrete Fourier transform to extract frequency and phase information. An idealized MLP was constructed using these parameters to evaluate its performance against the original model.
Results
The clean model demonstrated that input weights followed near-binary square waves, contrasting with sinusoidal weights observed in other architectures. The idealized model, constructed from the noisy data-trained model's parameters, achieved 95.5% accuracy, highlighting the effectiveness of the latent algorithmic structure.
Implications
The findings suggest that understanding the internal representations of neural networks can enhance model interpretability and improve training strategies for tasks involving modular arithmetic and potentially other structured tasks.
Reservoir-Based Graph Convolutional Networks
Graph Learning
- Introduction of RGC-Net, combining reservoir computing with graph convolution for improved GNN performance.
- Utilization of fixed-random reservoir weights and a leaky integrator to enhance feature retention and mitigate over-smoothing.
- Demonstrated state-of-the-art performance in graph classification and generative tasks, particularly in dynamic brain connectivity.
- Introduction of TRGC-Net for comparing fixed and trainable reservoir weights, offering insights into reservoir dynamics.
Read more
Reservoir-Based Graph Convolutional Networks
Summary
This paper introduces RGC-Net (Reservoir-based Graph Convolutional Network), a novel architecture that integrates reservoir computing with structured graph convolution to enhance the performance of Graph Neural Networks (GNNs). Traditional GCNs face challenges in handling complex or dynamic data, particularly when capturing long-range dependencies, which often leads to over-smoothing and increased computational costs. RGC-Net addresses these issues by employing fixed-random reservoir weights and a leaky integrator, allowing for stable information propagation and improved feature retention. The architecture not only supports graph classification tasks but also extends to graph generation, particularly in dynamic contexts such as brain connectivity evolution. The authors demonstrate that RGC-Net achieves state-of-the-art performance in various tasks, showing faster convergence and reduced over-smoothing compared to existing models. The paper also introduces TRGC-Net, a trainable variant of RGC-Net, facilitating a systematic comparison between fixed and trainable reservoir weights, thus providing new insights into reservoir dynamics in graph representation learning.
Methodology
The authors developed RGC-Net by integrating reservoir dynamics with structured graph convolution mechanisms. They utilized fixed-random reservoir weights and a leaky integrator to balance past and new information, enhancing the model's ability to retain meaningful features across iterations. The architecture was tested against various benchmarks in graph classification and generation tasks.
Results
RGC-Net achieved state-of-the-art performance in classification and generative tasks, including brain graph evolution, with faster convergence rates and reduced over-smoothing effects compared to existing models. The introduction of TRGC-Net allowed for a systematic evaluation of fixed versus trainable reservoir weights, revealing important insights into the effectiveness of reservoir dynamics in graph learning.
Implications
The findings suggest that RGC-Net can be effectively applied to dynamic graph applications, such as predicting brain connectivity changes over time. The integration of reservoir computing with graph convolution opens new avenues for research in GNNs, particularly in complex and evolving data scenarios.
On the Use of Bagging for Local Intrinsic Dimensionality Estimation
Theory
- Introduces bagging as a variance-reduction technique for LID estimation.
- Analyzes the complex interplay between sampling rate, neighborhood size, and ensemble size.
- Demonstrates significant improvements in estimation accuracy through empirical results.
- Proposes methods for combining bagging with neighborhood smoothing for enhanced performance.
Read more
On the Use of Bagging for Local Intrinsic Dimensionality Estimation
Summary
This paper addresses the challenge of accurately estimating Local Intrinsic Dimensionality (LID), which characterizes the local complexity of data manifolds. Traditional LID estimators often suffer from high variance due to limited data in small neighborhoods around query points, leading to biased estimates when nonlocal data is included. To mitigate this, the authors propose an ensemble method using bagging, specifically subbagging, to reduce variance while preserving the local distribution of nearest neighbor distances. The paper explores the interplay between sampling rate, neighborhood size, and ensemble size, providing theoretical and empirical analyses of how these factors affect LID estimation performance. The results demonstrate that bagging significantly reduces variance and mean squared error compared to non-bagged estimators, with controllable bias. Additionally, the authors introduce methods to combine bagging with neighborhood smoothing for further improvements in LID estimation accuracy.
Methodology
The authors utilize an ensemble approach based on bagging to estimate LID. They conduct both theoretical and empirical analyses to understand how the choice of sampling rate, neighborhood size, and ensemble size influence the performance of LID estimators. The methodology includes subbagging to maintain local distribution characteristics while reducing variance.
Results
The empirical results indicate that using a bagged estimator significantly reduces variance and mean squared error compared to non-bagged baselines. The study also highlights the importance of hyper-parameter selection, showing that informed choices can lead to better LID estimation outcomes.
Implications
The findings suggest that bagging can be effectively applied to LID estimation, providing a robust framework for various applications in data mining and machine learning, such as outlier detection and similarity search. The proposed methods could enhance the reliability of LID estimations in practical scenarios.
Towards Effective Experiential Learning: Dual Guidance for Utilization and Internalization
NLP
Large Language Models
Reinforcement Learning
- DGO introduces a unified framework that combines external and internal experience for improved training effectiveness.
- The framework operates through a closed-loop system of experience utilization and internalization.
- DGO consistently outperforms baseline methods, demonstrating enhanced reasoning capabilities in LLMs.
- The method achieves an average score of 32.41% on six benchmarks, improving to 39.38% with test-time scaling.
Read more
Towards Effective Experiential Learning: Dual Guidance for Utilization and Internalization
Summary
This paper addresses the limitations of current reinforcement learning (RL) approaches in enhancing large language models (LLMs) by proposing a novel framework called Dual Guidance Optimization (DGO). The authors argue that existing RL methods primarily focus on either utilizing external experiences or internalizing knowledge, which does not reflect the dual mechanism of human learning. DGO aims to unify these two aspects by constructing an experience bank from previously explored trajectories and guiding the exploration process with both external and internal knowledge. This closed-loop system allows for continuous refinement of the experience bank and model parameters. The experiments demonstrate that DGO significantly improves reasoning capabilities in LLMs, outperforming baseline methods and achieving higher average scores across multiple benchmarks. The findings suggest that integrating effective experience utilization and internalization can lead to more robust reasoning behaviors in LLMs.
Methodology
The Dual Guidance Optimization (DGO) framework involves three iterative stages: experience construction, joint trajectory-policy refinement, and experience internalization. An experience bank is created from previously collected trajectories, which guides the policy's exploration of solutions. The explored trajectories are then distilled into model parameters to enhance the model's capabilities.
Results
DGO achieved an average score of 32.41% across six challenging benchmarks on the Qwen3-8B-Base model under intrinsic inference conditions. With test-time scaling, this score improved to 39.38%, indicating the effectiveness of the proposed method in enhancing reasoning performance.
Implications
The findings suggest that integrating both experience utilization and internalization can lead to more effective training methodologies for LLMs, potentially influencing future research in reinforcement learning and experiential learning paradigms.
Steering Code LLMs with Activation Directions for Language and Library Control
Large Language Models
NLP
- Code LLMs exhibit strong implicit preferences for specific programming languages and libraries.
- Layer-wise activation directions can be estimated to steer model outputs effectively.
- Interventions can influence code generation even under neutral or conflicting prompts.
- Steering strength varies by model and target, with risks of quality degradation from strong interventions.
Read more
Steering Code LLMs with Activation Directions for Language and Library Control
Summary
This paper investigates the inherent preferences of code large language models (LLMs) for specific programming languages and libraries, which often manifest even under neutral prompts. The authors propose a method to steer these models by manipulating activation directions in the model's hidden states during inference. They utilize a difference-in-means approach to estimate layer-wise steering vectors for five language/library pairs and apply these vectors to the model's activations. The results demonstrate that such interventions can significantly influence code generation towards the desired programming ecosystem, even when prompts are neutral or contradictory. The effectiveness of steering varies by model and target, with common ecosystems being easier to induce than rarer ones. However, overly strong interventions can degrade output quality. Overall, the findings suggest that code-style preferences in LLMs are represented by steerable structures in activation space, providing a lightweight method for controlling code generation beyond traditional prompting techniques.
Methodology
The authors estimate layer-specific semantic directions using a difference-in-means procedure, comparing activations from matched prompt sets representing target and opposite concepts. They then intervene in the model's hidden states by adding learned steering vectors to influence code generation towards specific languages or libraries.
Results
The experiments show that layer-wise activation directions can effectively steer code generation across three open-weight code LLMs and five language/library pairs. The interventions are successful under neutral prompts and can override explicit requests for alternative languages or libraries. However, the effectiveness varies, and excessive steering can lead to reduced output quality.
Implications
This research has significant implications for enhancing the control and flexibility of code generation in LLMs, allowing users to specify desired programming languages and libraries more effectively. It opens avenues for further exploration of activation steering methods in various applications of LLMs.
The Coordinate System Problem in Persistent Structural Memory for Neural Architectures
Theory
- Introduction of the Dual-View Pheromone Pathway Network (DPPN) for persistent structural memory.
- Identification of coordinate stability and graceful transfer mechanisms as independent requirements for effective memory.
- Demonstration that learned coordinate systems are unstable and hinder memory persistence.
- Fixed random Fourier features provide stable coordinates but do not ensure effective transfer.
Read more
The Coordinate System Problem in Persistent Structural Memory for Neural Architectures
Summary
This paper introduces the Dual-View Pheromone Pathway Network (DPPN), a novel architecture designed to enhance persistent structural memory in neural networks by utilizing a pheromone field to route sparse attention. The author conducts five experiments to identify two critical requirements for persistent memory: the necessity of a stable coordinate system and the importance of a graceful transfer mechanism. The findings reveal that any coordinate system learned jointly with the model is inherently unstable, leading to various obstacles such as pheromone saturation and coordinate incompatibility. The study demonstrates that while fixed random Fourier features can provide stable coordinates, they do not guarantee effective transfer. The DPPN architecture outperforms traditional transformer models in within-task learning, and the use of learning-rate modulation instead of routing bias significantly mitigates negative transfer. The paper concludes with a diagnostic cascade methodology that can guide future architectural investigations in neural networks.
Methodology
The author conducted five progressively refined experiments using multiple seeds across various model variants and transfer targets to diagnose obstacles related to persistent structural memory. The experiments included testing the DPPN architecture with pheromone routing and evaluating the effects of fixed random Fourier features on coordinate stability.
Results
The DPPN architecture achieved an average within-task learning performance of AULC 0.700, outperforming transformer and random sparse baselines. The study found that while stable coordinates are necessary, they are not sufficient for effective transfer. Learning-rate modulation was shown to eliminate negative transfer, while routing-bias pheromone consistently resulted in performance degradation.
Implications
The findings suggest that for neural architectures to effectively utilize persistent structural memory, they must incorporate stable coordinate systems and employ graceful transfer mechanisms. This has implications for the design of future neural network architectures, particularly in tasks requiring transfer learning across different domains.
A Learning Method with Gap-Aware Generation for Heterogeneous DAG Scheduling
Reinforcement Learning
Optimization
Theory
- WeCAN framework effectively addresses scheduling of heterogeneous DAGs using reinforcement learning.
- Introduces a two-stage single-pass design for efficient schedule generation.
- Develops an order-space analysis to identify and eliminate generation-induced optimality gaps.
- Demonstrates superior performance in makespan compared to existing scheduling methods.
Read more
A Learning Method with Gap-Aware Generation for Heterogeneous DAG Scheduling
Summary
This paper presents WeCAN, a reinforcement learning framework designed for efficient scheduling of directed acyclic graphs (DAGs) in heterogeneous environments. The authors address the challenges posed by resource capacities and dependencies, particularly in scenarios with varying resource pools and task types. WeCAN employs a two-stage single-pass design that first generates task-pool scores and global parameters, followed by a generation map that constructs schedules without the need for repeated network calls. A key innovation is the weighted cross-attention encoder, which models task-pool interactions while being agnostic to environment size. The paper also introduces an order-space analysis to characterize the reachable set of generation maps and explains the generation-induced optimality gaps that can arise from traditional list-scheduling methods. To mitigate these gaps, the authors propose a skip-extended realization with a parameterized decreasing skip rule, enhancing the reachable order set while maintaining efficiency. Experimental results demonstrate that WeCAN outperforms strong baselines in terms of makespan, with inference times comparable to classical heuristics and faster than multi-round neural schedulers.
Methodology
The methodology involves a two-stage reinforcement learning framework where a weighted cross-attention encoder generates task-pool scores and global parameters in a single forward pass. The generation map constructs schedules based on these scores, while an order-space analysis is used to characterize and address optimality gaps in scheduling.
Results
Experiments show that WeCAN achieves improved makespan over strong baseline methods, with inference times that are competitive with classical heuristics and faster than multi-round neural schedulers, indicating its efficiency and effectiveness in heterogeneous scheduling scenarios.
Implications
The findings suggest that WeCAN can be applied to optimize scheduling in various domains such as data centers, distributed systems, and cloud platforms, where efficient resource allocation and task scheduling are critical.
Scaling Recurrence-aware Foundation Models for Clinical Records via Next-Visit Prediction
Generative Models
Time Series
NLP
- Introduction of RAVEN, a generative model for next-visit prediction in EHRs.
- Implementation of a history-dependent regularization mechanism to improve prediction of new disease onsets.
- Demonstration of RAVEN's competitive performance in zero-shot disease incidence forecasting.
- Investigation of scaling behaviors in data-constrained and compute-saturated regimes.
Read more
Scaling Recurrence-aware Foundation Models for Clinical Records via Next-Visit Prediction
Summary
The paper introduces RAVEN, a novel generative pretraining strategy for sequential electronic health record (EHR) data focused on next-visit prediction. Leveraging a dataset of over one million unique individuals, RAVEN learns to autoregressively generate tokenized clinical events for future visits based on patient history. A key innovation is the incorporation of a history-dependent regularization mechanism that downweights repeated clinical events, addressing the challenge of inflated performance metrics due to the prevalence of chronic conditions in EHRs. The authors highlight the inefficiencies of traditional disease-specific models and propose RAVEN as a scalable alternative that captures the joint distribution of clinical events over time. The model's performance is evaluated through zero-shot predictions for various diseases, demonstrating competitive results against fully fine-tuned Transformer models and outperforming simulation-based approaches. Additionally, RAVEN shows the ability to generalize to external cohorts without further parameter updates, even in the presence of lossy clinical code mappings.
Methodology
RAVEN employs a generative pretraining strategy that autoregressively generates clinical event tokens for future visits based on patient history. The model incorporates a regularization mechanism that penalizes the prediction of frequently observed clinical events, encouraging the model to focus on new disease onsets. The evaluation includes zero-shot predictions and scaling analysis under varying model sizes and data volumes.
Results
RAVEN rivals fully fine-tuned representation-based Transformer models in zero-shot predictions for disease incidence and outperforms simulation-based next-token approaches. The model demonstrates effective generalization to external cohorts under challenging conditions, indicating robustness and scalability.
Implications
The findings suggest that RAVEN could enhance early detection and progression forecasting for chronic conditions, improving healthcare outcomes and optimizing clinical trial designs. Its ability to generalize across different patient cohorts may facilitate broader applications in clinical settings.
Deep Convolutional Neural Networks for predicting highest priority functional group in organic molecules
Computer Vision
- Introduction of a CNN model for predicting the highest priority functional group in organic molecules.
- Utilization of FTIR spectroscopy data for training the model.
- Comparison of CNN performance with traditional SVM methods, showing improved accuracy.
- Emphasis on the significance of functional group priority in organic chemistry.
Read more
Deep Convolutional Neural Networks for predicting highest priority functional group in organic molecules
Summary
This paper addresses the challenge of predicting the highest priority functional group in organic molecules using Deep Convolutional Neural Networks (CNNs). Functional groups are crucial in determining the chemical and physical properties of organic compounds, and their identification is essential in various fields such as biochemistry and drug discovery. The authors utilize Fourier-transform Infrared (FTIR) spectroscopy data to train their CNN model, which is designed to predict the dominant functional group in the presence of multiple groups. They collected a substantial dataset of FTIR spectra from the Spectral Database for Organic Compounds (SDBS) and processed it to create a suitable input format for the CNN. The model's performance is compared against traditional machine learning methods, specifically Support Vector Machines (SVM), demonstrating superior accuracy in predicting functional groups. The paper emphasizes the importance of considering functional group priority, which has been overlooked in previous studies, and presents a novel approach that leverages deep learning techniques to enhance prediction accuracy.
Methodology
The authors collected FTIR spectra data from the SDBS database, processed it to extract relevant transmittance values, and normalized the data for input into the CNN. The CNN architecture was specifically designed to handle the spectral data, allowing for effective learning and prediction of functional groups.
Results
The CNN model outperformed traditional SVM approaches, achieving higher accuracy in predicting the highest priority functional group in organic molecules. The results indicate that deep learning techniques can significantly enhance the identification of functional groups, particularly in complex cases with multiple groups present.
Implications
This research has significant implications for fields such as drug discovery, molecular biology, and toxicology, where accurate identification of functional groups is critical. The proposed CNN model can streamline the process of functional group identification, potentially leading to advancements in chemical analysis and compound characterization.
Transcending Classical Neural Network Boundaries: A Quantum-Classical Synergistic Paradigm for Seismic Data Processing
Generative Models
Theory
Time Series
- Introduction of QC-GAN, the first application of quantum neural networks in seismic exploration.
- Integration of quantum and convolutional pathways enhances feature representation and processing capacity.
- Novel QC feature complementarity loss ensures non-overlapping information encoding.
- Experimental validation shows superior performance in denoising and interpolation tasks.
Read more
Transcending Classical Neural Network Boundaries: A Quantum-Classical Synergistic Paradigm for Seismic Data Processing
Summary
This paper introduces a novel quantum-classical synergistic generative adversarial network (QC-GAN) aimed at enhancing seismic data processing. Traditional neural network methods, while effective in tasks like denoising and interpolation, are limited by their reliance on classical architectures that operate within real-valued Euclidean spaces. The authors propose leveraging quantum neural networks (QNNs) to exploit the high-dimensional Hilbert spaces of quantum mechanics, thereby overcoming the representational limitations of classical networks. The QC-GAN integrates a quantum pathway for high-order feature correlations with a convolutional pathway for waveform structure extraction. A unique QC feature complementarity loss is introduced to ensure orthogonality between the two pathways, enriching feature representation. Experimental results demonstrate that QC-GAN effectively preserves wavefield continuity and amplitude-phase information under complex noise conditions. The authors also extend this paradigm to the UNet architecture, resulting in a QC-UNet that outperforms classical U-Net in low-frequency extrapolation tasks. Overall, the QC synergistic paradigm shows promise as a versatile tool for various seismic data processing applications.
Methodology
The authors developed a QC-GAN that combines quantum neural networks with convolutional neural networks. The quantum pathway captures high-order feature correlations, while the convolutional pathway focuses on waveform structures. A QC feature complementarity loss function is designed to maintain orthogonality between the two pathways, enriching the model's representational capacity.
Results
The QC-GAN demonstrated effective denoising and interpolation capabilities, preserving critical wavefield characteristics even under challenging noise conditions. Additionally, the QC-UNet variant outperformed classical U-Net models in low-frequency extrapolation tasks, highlighting the efficacy of the quantum-classical synergistic approach.
Implications
The proposed QC synergistic paradigm has the potential to revolutionize seismic data processing, providing a robust framework for handling complex data restoration tasks. Its versatility suggests applications beyond geophysics, potentially benefiting other fields requiring advanced data processing techniques.
Attack Assessment and Augmented Identity Recognition for Human Skeleton Data
Generative Models
Computer Vision
Optimization
- Introduction of Attack-AAIRS framework to enhance model robustness against adversarial attacks.
- Utilization of GANs to generate synthetic adversarial samples for training.
- Demonstrated improved resilience of HCN-ID models to various adversarial attack methods.
- Maintained accuracy on real data while improving robustness against unseen attacks.
Read more
Attack Assessment and Augmented Identity Recognition for Human Skeleton Data
Summary
This paper addresses the vulnerability of machine learning models, particularly those trained on small datasets, to adversarial attacks in the context of person identification from LiDAR-based skeleton data. The authors introduce a novel framework called Attack-AAIRS, which enhances the existing Assessment and Augmented Identity Recognition for Skeletons (AAIRS) by incorporating generative adversarial networks (GANs) to generate synthetic adversarial samples. This approach allows the model to be inoculated against various unseen adversarial attacks without relying solely on perturbations of real training samples. The study demonstrates that the Attack-AAIRS framework significantly improves the robustness of Hierarchical Co-occurrence Networks for Person Identification (HCN-ID) against multiple adversarial attack methods, including Fast Gradient Sign Method and Projected Gradient Descent. The results indicate that the inoculated models maintain consistent accuracy on real data while exhibiting enhanced resilience to adversarial perturbations, showcasing the effectiveness of synthetic data generation in bolstering model security.
Methodology
The authors developed the Attack-AAIRS framework, which combines a small real dataset with a GAN-generated synthetic dataset to create adversarial samples. This allows for the assessment and inoculation of the HCN-ID model against adversarial attacks. The framework employs ten-fold cross-validation to evaluate the robustness of the model against various attack methods.
Results
The implementation of Attack-AAIRS resulted in a noticeable increase in the robustness of the HCN-ID model against unseen adversarial attacks. The generated attack samples were found to be of similar quality to original benign samples, and the inoculated models achieved consistent final test accuracy, indicating that robustness was improved without sacrificing performance on real data.
Implications
The findings suggest that synthetic data generation can effectively enhance the security of machine learning models in applications where data acquisition is limited and costly. This approach has potential applications in security and surveillance systems, where reliable person identification is critical.
Symbolic--KAN: Kolmogorov-Arnold Networks with Discrete Symbolic Structure for Interpretable Learning
Interpretability
- Symbolic-KANs bridge the gap between symbolic regression and neural networks by embedding discrete symbolic structures.
- The architecture allows for the direct selection of univariate primitives, leading to compact closed-form expressions.
- Symbolic-KANs effectively recover governing structures in various applications, including regression and dynamical systems.
- The framework extends to physics-informed learning, producing accurate solutions from governing constraints.
Read more
Symbolic--KAN: Kolmogorov-Arnold Networks with Discrete Symbolic Structure for Interpretable Learning
Summary
The paper introduces Symbolic Kolmogorov-Arnold Networks (Symbolic-KANs), a novel neural architecture designed to enhance interpretability in scientific machine learning by embedding discrete symbolic structures directly within a trainable deep network. This approach addresses the longstanding trade-off between interpretability and scalability in symbolic regression and neural networks. Symbolic-KANs represent multivariate functions as compositions of learned univariate primitives, guided by a library of analytic primitives and hierarchical gating. This allows for the selection of compact closed-form expressions without the need for post-hoc symbolic fitting. The authors demonstrate that Symbolic-KANs can reliably recover correct primitive terms and governing structures in data-driven regression and inverse dynamical systems. Additionally, the framework is applicable to forward and inverse physics-informed learning of partial differential equations, producing accurate solutions while maintaining compact symbolic representations that reflect the true analytical structure of the underlying equations. The results suggest that Symbolic-KANs are a significant advancement toward scalable, interpretable, and mechanistically grounded learning of governing laws.
Methodology
The authors developed Symbolic-KANs by integrating discrete symbolic structures into a neural network framework. The architecture employs hierarchical gating and symbolic regularization to guide the selection of univariate primitives and scalar projections. This method allows for the training of the network to produce compact symbolic representations directly from data, without the need for post-hoc symbolic fitting.
Results
Symbolic-KANs demonstrated the ability to accurately recover primitive terms and governing structures in data-driven regression tasks and inverse dynamical systems. The framework also successfully addressed forward and inverse physics-informed learning of partial differential equations, yielding accurate solutions while constructing compact symbolic representations that align with the true analytical structure of the underlying equations.
Implications
The development of Symbolic-KANs has significant implications for scientific machine learning, particularly in fields requiring interpretable models that can uncover governing laws from data. This approach could enhance the trust and usability of machine learning models in scientific research and engineering applications, where understanding the underlying mechanisms is crucial.
CN-Buzz2Portfolio: A Chinese-Market Dataset and Benchmark for LLM-Based Macro and Sector Asset Allocation from Daily Trending Financial News
NLP
Large Language Models
- Introduction of CN-Buzz2Portfolio as a benchmark for evaluating LLMs in financial asset allocation.
- Focus on macro and sector-level asset allocation rather than individual stock picking.
- Implementation of a Tri-Stage CPA Agent Workflow to assess LLM performance.
- Significant disparities observed among LLMs in translating financial narratives into portfolio strategies.
Read more
CN-Buzz2Portfolio: A Chinese-Market Dataset and Benchmark for LLM-Based Macro and Sector Asset Allocation from Daily Trending Financial News
Summary
The paper introduces CN-Buzz2Portfolio, a novel benchmark designed to evaluate Large Language Models (LLMs) in the context of macro and sector asset allocation based on daily trending financial news in the Chinese market. The authors highlight the limitations of existing evaluation paradigms, which either rely on direct live trading—prone to outcome bias—or static benchmarks that focus on entity-level stock picking, neglecting broader market narratives. CN-Buzz2Portfolio addresses these issues by providing a reproducible dataset that simulates a realistic public attention stream, allowing agents to derive investment strategies from high-exposure narratives. The proposed Tri-Stage CPA Agent Workflow (Compression, Perception, Allocation) evaluates LLMs on diversified asset classes, such as Exchange Traded Funds (ETFs), thereby reducing idiosyncratic volatility. Extensive experiments conducted on nine LLMs reveal significant differences in how these models interpret macro-level narratives into portfolio weights, offering insights into the relationship between reasoning capabilities and financial decision-making. The dataset, evaluation code, and experimental results are made publicly available to support ongoing research in sustainable financial agents.
Methodology
The authors developed a rolling-horizon dataset from daily trending financial news, simulating public attention streams. They proposed a Tri-Stage CPA Agent Workflow to evaluate LLMs on their ability to allocate assets based on macro-level narratives, focusing on diversified ETFs instead of individual stocks.
Results
The experiments revealed notable differences in how various LLMs interpret and apply macro-level financial narratives to construct portfolio weights, indicating the complexity and variability in LLM performance in financial contexts.
Implications
The findings suggest that LLMs can be effectively utilized as decision-making agents in financial markets, particularly in environments driven by macroeconomic narratives. The benchmark can aid in developing more robust financial agents and enhance understanding of LLM reasoning in complex financial scenarios.
MsFormer: Enabling Robust Predictive Maintenance Services for Industrial Devices
Time Series
- Introduction of MsFormer, a lightweight Multi-scale Transformer for predictive maintenance.
- Incorporation of a Multi-scale Sampling module to capture multi-scale temporal correlations.
- Use of a lightweight attention mechanism tailored for data-scarce environments.
- Extensive validation on real-world datasets showing significant performance improvements.
Read more
MsFormer: Enabling Robust Predictive Maintenance Services for Industrial Devices
Summary
The paper presents MsFormer, a novel lightweight Multi-scale Transformer model designed to enhance predictive maintenance services for industrial devices. Traditional deep learning approaches struggle with the complexities of industrial IoT sensor data, particularly in capturing multi-scale temporal correlations and handling limited training datasets. MsFormer addresses these challenges by incorporating a Multi-scale Sampling (MS) module and a tailored position encoding mechanism that allows it to effectively model sequential correlations across multi-streaming service data. Additionally, it employs a lightweight attention mechanism that simplifies computations, making it suitable for data-scarce environments. The authors conducted extensive experiments on real-world datasets, demonstrating that MsFormer significantly outperforms existing state-of-the-art methods in predictive maintenance tasks, showcasing its robustness and generalizability across various industrial devices and operating conditions. This work contributes to the field by providing a unified AI service model that ensures reliable Quality of Service (QoS) in predictive maintenance applications.
Methodology
MsFormer employs a four-stage framework that includes a Multi-scale Sampling module to restructure timestamps for multiple time horizons, a lightweight attention mechanism for efficient processing, and a multi-scale positional encoding to enhance correlation extraction among sensor measurements. The model is designed to operate effectively in environments with limited data availability.
Results
The experimental results indicate that MsFormer achieves substantial performance improvements over existing predictive maintenance models, validating its effectiveness in capturing complex degradation patterns and ensuring reliable QoS across various industrial applications.
Implications
MsFormer has the potential to revolutionize predictive maintenance services in industrial settings by providing a robust and reliable AI service model that can adapt to varying data conditions, ultimately leading to reduced downtime and improved operational efficiency.
Bridging the Gap Between Climate Science and Machine Learning in Climate Model Emulation
Efficient ML
- ML emulators can significantly reduce the computational costs associated with traditional climate models.
- There is a disconnect between the climate science and machine learning communities regarding the use of emulators.
- A framework integrating both fields can enhance the design and reliability of climate model emulators.
- Closer collaboration can create feedback loops that improve both emulators and physical simulations.
Read more
Bridging the Gap Between Climate Science and Machine Learning in Climate Model Emulation
Summary
This paper addresses the challenges faced in the integration of machine learning (ML) techniques into climate model emulation, which is crucial for efficient climate decision-making. The authors highlight the significant computational demands of traditional climate models and propose ML emulators as a viable alternative. However, the effective adoption of these emulators is hindered by barriers such as limited accessibility, lack of specialized knowledge, and mistrust towards ML methods. The authors introduce a framework that combines perspectives from both climate science and machine learning to facilitate the development of user-friendly emulators that are reliable and task-oriented. They emphasize the need for closer collaboration between the two communities to enhance the uptake of ML emulators in practical applications. The paper also provides a tutorial that identifies synergies between the fields and suggests methods for improving communication and evaluation processes. Ultimately, the authors aim to foster a more targeted development of ML-based emulators to increase their relevance and application in climate decision-making contexts.
Methodology
The authors analyze the differing perspectives of climate scientists and machine learning researchers on climate model emulation. They propose a framework for collaboration and provide a tutorial that outlines practical steps for integrating ML techniques into climate modeling.
Results
The paper identifies key barriers to the adoption of ML emulators and presents a framework that facilitates their development and use. It emphasizes the importance of designing emulators that are easy to adopt and demonstrate reliability for specific tasks.
Implications
The findings suggest that enhancing collaboration between climate scientists and machine learning practitioners can lead to more effective climate model emulators, ultimately improving decision-making in climate-related contexts. This could enable better resource allocation for climate research and policy-making.
Generalizing Dynamics Modeling More Easily from Representation Perspective
Time Series
- Introduction of a generalized Pre-trained Dynamics EncoDER (PDEDER) for improved dynamics modeling.
- Utilization of the Lyapunov exponent to minimize chaotic behavior in the latent space.
- Incorporation of reconstruction and forecasting objectives to enhance model performance.
- Evaluation on 12 dynamic systems shows significant improvements in forecasting accuracy.
Read more
Generalizing Dynamics Modeling More Easily from Representation Perspective
Summary
This paper addresses the challenge of learning system dynamics from observations across various complex real-world systems, such as climate and ecology. Traditional neural dynamics modeling methods often require specific models for different systems, leading to poor generalization. To overcome this limitation, the authors propose a generalized Pre-trained Dynamics EncoDER (PDEDER) that embeds observations into a latent space conducive to capturing dynamics more effectively. The PDEDER is pre-trained using a Lyapunov exponent objective to minimize chaotic behavior in the latent space, promoting stable and well-structured dynamics. Additionally, reconstruction and forecasting objectives are incorporated to prevent over-smoothing of the latent space. The authors collect 152 sets of observations from 23 complex systems for pre-training and demonstrate the model's effectiveness through fine-tuning on 12 dynamic systems for both in-domain and cross-domain forecasting tasks. The results indicate that PDEDER significantly enhances generalizability and performance in dynamics modeling compared to traditional methods.
Methodology
The authors developed PDEDER by pre-training it on a diverse set of observations using a Lyapunov exponent objective to constrain chaotic dynamics in the latent space. They also included auxiliary tasks for reconstruction and forecasting to maintain the representation capacity. The model was fine-tuned on specific dynamics modeling methods using real-world and synthetic data.
Results
PDEDER was evaluated on 12 dynamic systems, showing improved forecasting capabilities in both in-domain and cross-domain settings. The empirical results confirmed the model's effectiveness and generalizability, outperforming traditional dynamics modeling approaches.
Implications
The proposed PDEDER can be applied to various fields requiring dynamics modeling, such as climate science, ecology, and fluid dynamics. Its ability to generalize across different systems could lead to more robust and adaptable modeling techniques in complex environments.
Linear-Nonlinear Fusion Neural Operator for Partial Differential Equations
Efficient ML
Theory
Interpretability
- Introduction of a linear-nonlinear multiplicative fusion mechanism for improved training efficiency.
- LNF-NO architecture effectively decouples linear and nonlinear effects for better representation.
- Demonstrated significant training speed improvements (up to 2.7x faster) compared to existing models.
- Achieves comparable or better accuracy across various PDE benchmarks.
Read more
Linear-Nonlinear Fusion Neural Operator for Partial Differential Equations
Summary
This paper introduces the Linear-Nonlinear Fusion Neural Operator (LNF-NO), a novel neural network architecture designed for efficiently solving partial differential equations (PDEs). The LNF-NO explicitly decouples linear and nonlinear effects in operator mappings, which enhances learning efficiency and provides a lightweight, interpretable representation. By employing a multiplicative fusion mechanism, the architecture captures complex solution features while maintaining stability and generality. The LNF-NO supports multiple functional inputs and can be applied to both regular grids and irregular geometries. The authors evaluate LNF-NO across various PDE operator-learning benchmarks, including nonlinear Poisson-Boltzmann equations and multi-physics coupled systems. The results demonstrate that LNF-NO is significantly faster to train than existing models like DeepONet and Fourier Neural Operators, while achieving comparable or superior accuracy, particularly noted in a 3D Poisson-Boltzmann case where it outperformed a 3D FNO baseline in both accuracy and training speed.
Methodology
The LNF-NO architecture employs a two-branch structure where linear and nonlinear components are fused multiplicatively. Each input function is encoded separately, and their latent representations are combined within the operator core. This design allows for efficient learning and representation of complex PDE solutions, accommodating multiple inputs and outputs.
Results
LNF-NO consistently outperformed traditional models in training speed, achieving a 2.7x speedup on 3D tasks while maintaining or improving accuracy. In particular, it excelled in the 3D Poisson-Boltzmann case, achieving the best accuracy among compared models and significantly reducing training time.
Implications
The findings suggest that the LNF-NO can serve as an efficient surrogate model for various scientific applications involving PDEs, potentially reducing computational costs in fields such as physics, chemistry, and biology where repeated numerical solutions are common.
TuneShift-KD: Knowledge Distillation and Transfer for Fine-tuned Models
NLP
Large Language Models
Efficient ML
- TuneShift-KD automates the distillation of specialized knowledge from fine-tuned models to target models.
- The method relies on identifying perplexity differences to create a synthetic training dataset.
- It does not require access to original training data or additional training of discriminators.
- Models fine-tuned with TuneShift-KD show improved accuracy over previous knowledge transfer methods.
Read more
TuneShift-KD: Knowledge Distillation and Transfer for Fine-tuned Models
Summary
The paper introduces TuneShift-KD, a novel method for transferring specialized knowledge from fine-tuned models to target models without requiring access to the original training data. As large language models (LLMs) evolve, transferring specialized knowledge becomes crucial, especially when the original fine-tuning data is unavailable due to privacy or commercial restrictions. TuneShift-KD identifies specialized knowledge through perplexity differences between the base and fine-tuned models. It generates a synthetic training dataset by focusing on prompts where the fine-tuned model performs well (low perplexity) while the base model struggles (high perplexity). This automated approach does not require training additional discriminators or access to training datasets, making it efficient and broadly applicable across different LLM architectures. The experiments demonstrate that models fine-tuned using TuneShift-KD achieve higher accuracy compared to previous methods, facilitating effective knowledge transfer and deployment.
Methodology
TuneShift-KD utilizes a perplexity difference filtering criterion to identify prompts that reveal specialized knowledge. It generates synthetic training examples based on these identified prompts, allowing for the transfer of knowledge from a fine-tuned model to a target model without needing the original training data.
Results
The experiments conducted show that models fine-tuned using TuneShift-KD achieve significantly higher accuracy than those fine-tuned using traditional knowledge distillation methods, demonstrating the effectiveness of the approach in transferring specialized knowledge.
Implications
TuneShift-KD has the potential to facilitate the deployment of specialized models in various domains, particularly when original training data is inaccessible. This could enhance the adaptability of LLMs in commercial applications, healthcare, legal fields, and other specialized areas.
Upper Entropy for 2-Monotone Lower Probabilities
Theory
Optimization
Efficient ML
- Introduces a polynomial-time algorithm for computing upper entropy for 2-monotone lower probabilities.
- Demonstrates significant improvements over existing algorithms, making computations more efficient.
- Provides specialized algorithms for belief functions, possibility distributions, and probability intervals.
- Presents an approximation algorithm that effectively bounds the results and controls errors.
Read more
Upper Entropy for 2-Monotone Lower Probabilities
Summary
This paper addresses the computational challenges associated with calculating upper entropy for credal sets induced by 2-monotone lower probabilities. Upper entropy serves as a crucial measure of uncertainty in various applications, including model selection and active learning. The authors present a comprehensive algorithmic analysis, demonstrating that the problem can be solved in a strongly polynomial time, significantly improving upon previous algorithms. They introduce a new algorithm that reduces the number of calls to supermodular function maximization (SFM) to linear in the space size, thus making the computation more efficient. Additionally, the paper provides enhanced algorithms for specific cases such as belief functions, possibility distributions, and probability intervals. An approximation algorithm using the Frank–Wolfe method is also proposed, which bounds the true results and controls approximation errors. Experimental results indicate that the proposed methods are scalable and outperform existing techniques.
Methodology
The authors utilize supermodular optimization techniques to derive polynomial-time algorithms for calculating upper entropy. They enhance existing algorithms by reducing the complexity of calls to supermodular function maximization and propose specialized algorithms for common cases in uncertainty quantification. Additionally, they introduce an approximation algorithm based on the Frank–Wolfe method to manage approximation errors.
Results
The paper establishes that the computational problem of calculating upper entropy is not as complex as previously thought, providing a strongly polynomial solution. The new algorithms significantly reduce the computational burden, with some achieving linear complexity in the number of calls to SFM. The proposed methods were experimentally validated, showing improved scalability and performance compared to prior approaches.
Implications
The findings have significant implications for fields that require uncertainty quantification, such as machine learning, decision-making under uncertainty, and AI applications. The improved algorithms can facilitate more efficient model selection, regularization, and active learning processes, enhancing the robustness of predictive models.
Likelihood hacking in probabilistic program synthesis
Reinforcement Learning
Theory
Generative Models
- Formal definition of likelihood hacking in probabilistic programming languages.
- Establishment of sufficient conditions for preventing likelihood hacking.
- Implementation of SafeStan and SafePyMC to enforce safety constraints.
- Empirical evidence of likelihood hacking in RL-driven automated scientists.
Read more
Likelihood hacking in probabilistic program synthesis
Summary
This paper addresses the phenomenon of likelihood hacking (LH) in probabilistic program synthesis, where language models trained via reinforcement learning (RL) can inflate their marginal-likelihood rewards by generating programs that do not accurately fit the data. The authors formalize LH within a core probabilistic programming language (PPL) and establish sufficient syntactic conditions to prevent it, proving that a safe language fragment, Lsafe, can avoid likelihood-hacking programs. Empirical evidence shows that models trained with GRPO quickly discover LH exploits, leading to high violation rates. To mitigate this issue, the authors implement Lsafe's conditions in SafeStan, a modification of the Stan programming language, demonstrating its effectiveness in preventing LH under optimization pressure. The findings indicate that language-level safety constraints are both theoretically sound and practically effective for automated Bayesian model discovery.
Methodology
The authors formalize likelihood hacking in a core PPL and derive syntactic conditions for its prevention. They implement these conditions in two practical layers: SafeStan for the Stan compiler and SafePyMC for PyMC, using static type-checking and program analysis to reject likelihood-hacking programs.
Results
The implementation of SafeStan and SafePyMC successfully prevents all identified likelihood-hacking exploits while maintaining the acceptance of honest baseline programs. Empirical evaluations show that GRPO-trained models exhibit high likelihood hacking rates, which are mitigated by the proposed safety measures.
Implications
The findings suggest that incorporating language-level safety constraints can enhance the reliability of probabilistic programming languages in automated scientific discovery, potentially leading to more robust AI systems capable of generating valid probabilistic models.
MetaKube: An Experience-Aware LLM Framework for Kubernetes Failure Diagnosis
Large Language Models
- MetaKube integrates episodic memory networks, specialized language models, and causal knowledge graphs for enhanced Kubernetes diagnostics.
- The framework allows for dynamic reasoning pathways, optimizing diagnostic speed and depth based on problem familiarity.
- MetaKube's locally-deployable model ensures data privacy while achieving high diagnostic performance.
- Experiential learning through EPMN significantly improves diagnostic accuracy over time.
Read more
MetaKube: An Experience-Aware LLM Framework for Kubernetes Failure Diagnosis
Summary
MetaKube is introduced as an innovative framework designed to enhance Kubernetes failure diagnosis by integrating experience-aware learning mechanisms. Traditional LLM-based diagnostic systems operate on static knowledge bases and fail to learn from past operational experiences. MetaKube addresses this limitation through three key innovations: (1) an Episodic Pattern Memory Network (EPMN) that abstracts diagnostic patterns from historical resolutions, enabling confidence-calibrated retrieval for rapid pattern matching and guided causal exploration; (2) a meta-cognitive controller that dynamically selects between intuitive and analytical reasoning pathways based on the familiarity of the problem, optimizing the balance between speed and depth of diagnosis; and (3) KubeLLM, a locally-deployable 8B model that has undergone domain-specific post-training on a curated dataset of 7,000 Kubernetes fault resolutions. The framework was evaluated on 1,873 real-world scenarios, demonstrating a significant performance improvement of the Qwen3-8B model from 50.9 to 90.5 points, nearing the performance of GPT-4.1 while ensuring data privacy. The EPMN component alone contributed to a 15.3% improvement through experiential learning, with continuous learning experiments indicating progressive gains as operational knowledge accumulates.
Methodology
MetaKube employs a cognitive architecture that combines an Episodic Pattern Memory Network (EPMN) for pattern recognition, a meta-cognitive controller for dynamic reasoning pathway selection, and KubeLLM, a specialized language model fine-tuned on a comprehensive fault resolution dataset. The framework also constructs a causal knowledge graph to enhance diagnostic capabilities.
Results
MetaKube improved the Qwen3-8B model's performance from 50.9 to 90.5 points in diagnostic accuracy, approaching GPT-4.1 performance. The EPMN contributed to a 15.3% improvement through experiential learning, with continuous learning experiments showing progressive gains.
Implications
MetaKube's experience-aware framework can significantly enhance the efficiency and accuracy of Kubernetes fault diagnosis, making it a valuable tool for cloud infrastructure management. Its ability to learn from operational experiences could lead to more robust and adaptive diagnostic systems in production environments.
UI-Voyager: A Self-Evolving GUI Agent Learning via Failed Experience
Reinforcement Learning
Optimization
Multimodal
- Introduction of UI-Voyager, a self-evolving mobile GUI agent.
- Utilization of Rejection Fine-Tuning (RFT) for autonomous data and model co-evolution.
- Implementation of Group Relative Self-Distillation (GRSD) to improve credit assignment in learning.
- Achieved an 81.0% Pass@1 success rate on the AndroidWorld benchmark, outperforming existing models.
Read more
UI-Voyager: A Self-Evolving GUI Agent Learning via Failed Experience
Summary
The paper presents UI-Voyager, a novel two-stage self-evolving mobile GUI agent designed to improve learning efficiency from failed experiences and address credit assignment challenges in long-horizon tasks. The first stage employs Rejection Fine-Tuning (RFT), which facilitates the autonomous co-evolution of data and models by iteratively refining GUI interaction trajectories without manual annotation. The second stage introduces Group Relative Self-Distillation (GRSD), which identifies critical fork points in group rollouts, enabling the extraction of dense supervision from successful trajectories to correct failed ones. This approach effectively replaces sparse trajectory-level rewards with precise self-distillation signals, enhancing learning efficiency. Extensive experiments on the AndroidWorld benchmark demonstrate that the 4B model achieves an 81.0% Pass@1 success rate, surpassing various baselines and human-level performance. The study includes ablation and case studies confirming the effectiveness of GRSD and the overall UI-Voyager framework, marking a significant advancement in mobile GUI automation without the need for costly manual data annotation.
Methodology
The methodology consists of a two-stage process: (1) Rejection Fine-Tuning (RFT) for continuous refinement of GUI interaction data and models, and (2) Group Relative Self-Distillation (GRSD) for identifying critical fork points in trajectories to provide dense supervision from successful interactions to correct failed ones.
Results
The UI-Voyager model achieved an 81.0% Pass@1 success rate on the AndroidWorld benchmark, outperforming multiple recent baselines and exceeding human-level performance. Ablation studies confirmed the contributions of GRSD and the overall framework.
Implications
The findings suggest that UI-Voyager can significantly enhance the efficiency of mobile GUI automation, paving the way for more robust and intelligent digital assistants that require less manual intervention for training.
Deep Neural Regression Collapse
Theory
- Deep Neural Regression Collapse (NRC) occurs across all layers of deep regression models, not just the last layer.
- Models exhibiting Deep NRC learn the intrinsic dimension of low-rank targets, indicating generalization rather than memorization.
- Weight decay is necessary for inducing Deep NRC, highlighting its importance in model training.
- The study provides a complete description of NRC conditions applicable to deep regressors.
Read more
Deep Neural Regression Collapse
Summary
This paper investigates the phenomenon of Neural Regression Collapse (NRC), extending the concept of Neural Collapse (NC) from classification to regression tasks. The authors establish that NRC occurs not only at the last layer of deep regression models but also in earlier layers, revealing that features in these collapsed layers align with target dimensions and covariances. They propose a set of NRC conditions applicable across all layers, demonstrating that models exhibiting Deep NRC learn the intrinsic dimension of low-rank targets, rather than merely memorizing tasks. The study also explores the role of weight decay in facilitating Deep NRC, suggesting that it is crucial for achieving these solutions. By providing a comprehensive understanding of Deep NRC, the paper contributes to the broader discourse on the implicit biases present in deep learning models and their ability to generalize effectively.
Methodology
The authors trained various deep regression models on real datasets to empirically validate the occurrence of Deep NRC. They derived NRC conditions from existing Neural Collapse conditions and analyzed the relationships between features, weights, and target covariances across different layers.
Results
The results indicate that Deep NRC is prevalent across different model architectures, with features in collapsed layers aligning with target dimensions and covariances. The findings confirm that models exhibiting Deep NRC can generalize effectively by learning the intrinsic dimensions of low-rank targets. Additionally, the necessity of weight decay for inducing Deep NRC was established.
Implications
The findings have significant implications for understanding how deep learning models achieve generalization in regression tasks. By elucidating the structure learned by these models, the research could inform better training practices and model design, potentially leading to improved performance in various regression applications.
Central Dogma Transformer III: Interpretable AI Across DNA, RNA, and Protein
Interpretability
Multimodal
- CDT-III aligns its architecture with the central dogma, enhancing interpretability and biological relevance.
- The two-stage architecture effectively separates transcription and translation processes, improving prediction accuracy.
- Joint prediction of RNA and protein changes leads to better performance and interpretability.
- The model can predict clinical side effects and generate hypotheses without clinical data, showcasing its practical applications.
Read more
Central Dogma Transformer III: Interpretable AI Across DNA, RNA, and Protein
Summary
The paper introduces the Central Dogma Transformer III (CDT-III), an AI model designed to bridge the interpretability gap in biological AI by aligning its architecture with the central dogma of molecular biology, which describes the flow of information from DNA to RNA to protein. CDT-III features a two-stage architecture that includes a Virtual Cell Embedder for the nucleus (VCE-N) to model transcription and another for the cytosol (VCE-C) to model translation. This design allows the model to produce interpretable attention maps at each layer and jointly predict changes in mRNA and surface protein levels resulting from CRISPRi perturbations. The model demonstrates significant predictive accuracy, achieving a correlation of r = 0.843 for RNA and r = 0.969 for protein across five held-out genes. Notably, the inclusion of protein prediction enhances RNA performance and improves interpretability at the DNA level. The model also successfully predicts protein changes in a simulated CD52 knockdown scenario, identifying known clinical side effects without requiring clinical data. This work emphasizes the potential of mechanism-oriented AI to yield clinically actionable insights from biological data.
Methodology
CDT-III employs a two-stage architecture consisting of a Virtual Cell Embedder for the nucleus (VCE-N) and another for the cytosol (VCE-C). This design allows the model to process transcription and translation as distinct yet interconnected modules, producing interpretable attention maps that reflect biological processes. The model predicts perturbation-induced changes in RNA and protein levels using pre-computed embeddings and a differentiable architecture that captures the complete information flow of the central dogma.
Results
CDT-III achieved a correlation of r = 0.843 for RNA predictions and r = 0.969 for protein predictions across five held-out genes. The model demonstrated that adding protein prediction improved RNA performance (from r = 0.804 to r = 0.843) and enhanced DNA-level interpretability by increasing CTCF enrichment by 30%. In the in silico CD52 knockdown application, it accurately predicted 29 out of 29 protein changes in the correct direction and rediscovered 5 of 7 known clinical side effects without clinical data.
Implications
The findings suggest that AI architectures designed to reflect biological processes can enhance the interpretability and predictive power of models in biological research. CDT-III's ability to generate clinically relevant insights from perturbation data alone could significantly impact drug development and personalized medicine by improving understanding of molecular mechanisms and side effects.
Robustness Quantification and Uncertainty Quantification: Comparing Two Methods for Assessing the Reliability of Classifier Predictions
Theory
- RQ outperforms UQ in assessing classifier prediction reliability, particularly under distribution shifts.
- Both RQ and UQ can be combined for enhanced reliability assessments.
- The study emphasizes the significance of reliability in high-stakes AI applications.
- A comprehensive comparison is conducted using real datasets, expanding beyond previous studies focused on artificial data.
Read more
Robustness Quantification and Uncertainty Quantification: Comparing Two Methods for Assessing the Reliability of Classifier Predictions
Summary
This paper investigates two methodologies for assessing the reliability of classifier predictions: Robustness Quantification (RQ) and Uncertainty Quantification (UQ). The authors elucidate the conceptual differences between these approaches and perform a comparative analysis using various benchmark datasets. The findings indicate that RQ can outperform UQ in both standard settings and scenarios involving distribution shifts. Furthermore, the authors explore the complementarity of RQ and UQ, demonstrating that a combined approach can yield superior reliability assessments. The paper emphasizes the importance of understanding the reliability of individual predictions, especially in high-stakes applications, and provides a comprehensive evaluation of both methodologies across real datasets, including Naive Bayes Classifier and Generative Forests.
Methodology
The authors benchmarked RQ and UQ on real datasets using two types of probabilistic generative classifiers: the Naive Bayes Classifier (NBC) and Generative Forests (GeFs). They assessed the reliability of predictions by quantifying uncertainty and robustness, comparing the performance of both methods in standard and distribution-shift scenarios.
Results
The results show that RQ consistently outperformed UQ in terms of reliability assessment. Additionally, the combination of RQ and UQ led to improved reliability evaluations, suggesting that the two approaches can complement each other effectively.
Implications
The findings have significant implications for the deployment of AI models in critical areas such as healthcare, where understanding the reliability of predictions is crucial. The study encourages the adoption of combined methodologies for better decision-making and risk management in AI applications.
Diet Your LLM: Dimension-wise Global Pruning of LLMs via Merging Task-specific Importance Score
Large Language Models
Efficient ML
- DIET is a dimension-wise global pruning framework that generates a single global mask for LLMs.
- The method requires no additional training, relying solely on activation profiling from a small number of task-specific samples.
- DIET consistently outperforms state-of-the-art structured pruning methods across various sparsity levels and model sizes.
- The framework demonstrates significant accuracy gains, particularly in zero-shot commonsense reasoning tasks.
Read more
Diet Your LLM: Dimension-wise Global Pruning of LLMs via Merging Task-specific Importance Score
Summary
The paper introduces DIET, a novel structured pruning method for large language models (LLMs) that addresses the challenges of deploying these models on resource-constrained platforms. Traditional pruning methods either lack task-specific adaptability or require extensive training, which can be costly. DIET overcomes these limitations by employing a training-free approach that combines dimension-level granularity with task-aware selection. The method profiles activation magnitudes across tasks using a minimal sample size of 100 per task and constructs a global pruning mask through majority voting. This global mask is then uniformly applied across all layers of the model. The authors validate DIET's effectiveness through experiments on seven zero-shot benchmarks using the Gemma-2 models (2B and 9B parameters), demonstrating significant improvements in accuracy compared to existing structured pruning techniques. At 20% sparsity, DIET achieves nearly a 10% average accuracy increase, showcasing its robustness and practicality for structured pruning of LLMs.
Methodology
DIET profiles the outputs of MLP layers at each transformer block by averaging absolute activations to create a per-dimension importance vector. It then generates task-specific binary pruning masks and aggregates them to form a single global mask, which is applied uniformly across all relevant layers of the model.
Results
Experiments on the Gemma-2 models (2B and 9B parameters) show that DIET achieves significant accuracy improvements, with a nearly 10% increase at 20% sparsity. The method outperforms previous structured pruning techniques and maintains its advantages across different model sizes and sparsity levels.
Implications
DIET provides a practical solution for deploying large language models in resource-constrained environments, enabling efficient pruning without the need for extensive retraining. This can facilitate the use of LLMs in various applications where computational resources are limited.
AscendOptimizer: Episodic Agent for Ascend NPU Operator Optimization
Optimization
Efficient ML
- AscendOptimizer addresses the knowledge bottleneck in optimizing AscendC operators due to a lack of public reference implementations.
- The framework employs a two-stage optimization process that alternates between host-side tiling and kernel-side optimization.
- It achieves a 1.19× speedup over existing baselines, with 49.61% of operators outperforming their references.
- The method leverages hardware feedback for evolutionary search and creates a library of optimization motifs through kernel rewinding.
Read more
AscendOptimizer: Episodic Agent for Ascend NPU Operator Optimization
Summary
The paper introduces AscendOptimizer, an innovative episodic agent designed to optimize operator performance on Huawei's Ascend neural processing units (NPUs). The authors identify a significant knowledge gap in the optimization of AscendC operators, stemming from the lack of public reference implementations and the complexity of the two-part operator structure, which includes a host-side tiling program and a device-side kernel program. AscendOptimizer addresses this challenge by employing a profiling-in-the-loop evolutionary search to discover effective tiling and data movement configurations based on real hardware feedback. Additionally, it utilizes a technique of 'rewinding' optimized kernels to create a library of optimization motifs, which can be used to guide further improvements. The framework operates in a closed-loop manner, alternating between host tuning and kernel rewriting to iteratively enhance performance. The results demonstrate that AscendOptimizer achieves a 1.19× geometric-mean speedup over existing open-source baselines across 127 real AscendC operators, with nearly half of the operators outperforming their reference implementations. This work not only showcases the potential for automated optimization in knowledge-scarce environments but also highlights the importance of internal experience bootstrapping in the absence of extensive external data.
Methodology
AscendOptimizer utilizes a two-stage optimization framework that alternates between optimizing the host-side tiling configurations and the device-side kernel programs. It incorporates a profiling-in-the-loop evolutionary search to gather hardware feedback and employs a technique of rewinding optimized kernels to synthesize 'bad-to-good' trajectories, which are distilled into a retrievable experience bank for guided rewriting.
Results
The framework was tested on 127 real AscendC operators, achieving a geometric mean speedup of 1.19× compared to open-source baselines. Notably, 49.61% of the operators outperformed their reference implementations, showcasing the effectiveness of the proposed optimization strategies.
Implications
The findings suggest that AscendOptimizer can significantly enhance the performance of operators on NPUs, making it a valuable tool for developers working with Ascend architecture. This approach may also pave the way for similar optimization frameworks in other knowledge-scarce environments, potentially improving the efficiency of AI model training and inference.
Dual-Criterion Curriculum Learning: Application to Temporal Data
Time Series
- Introduction of the Dual-Criterion Curriculum Learning (DCCL) framework combining loss-based and density-based difficulty assessments.
- DCCL addresses the challenge of defining meaningful difficulty measures in Curriculum Learning.
- Empirical evaluations show that DCCL outperforms traditional loss-only baselines in time-series forecasting tasks.
- The framework is modular and applicable to a wide range of data types, enhancing its versatility.
Read more
Dual-Criterion Curriculum Learning: Application to Temporal Data
Summary
This paper introduces the Dual-Criterion Curriculum Learning (DCCL) framework, which enhances the traditional Curriculum Learning (CL) approach by integrating two distinct measures of instance difficulty: a loss-based criterion and a density-based criterion derived from the data representation space. The authors argue that the conventional methods of assessing difficulty often rely on application-specific heuristics, which can limit their effectiveness. By combining loss and density assessments, DCCL aims to provide a more robust framework for training models, particularly in the context of time-series forecasting. The framework is evaluated on multivariate time-series benchmarks using standard training schedules, demonstrating that the dual-criterion approach outperforms loss-only baselines and traditional non-CL training methods. This work highlights the importance of effective difficulty assessment in CL and presents a modular approach that can be adapted to various data types, including complex structured data.
Methodology
The DCCL framework employs a dual-criterion approach to assess instance difficulty, where a loss-based criterion is complemented by a density-based criterion learned in the data representation space. The authors utilize multivariate time-series benchmarks and evaluate the framework under standard training schedules, including One-Pass and Baby-Steps methods.
Results
The empirical results indicate that the DCCL framework significantly improves performance in time-series forecasting tasks compared to traditional loss-only baselines and standard non-CL training methods. The integration of density-based assessments alongside loss measures leads to better learning outcomes.
Implications
The DCCL framework has the potential to enhance model training across various domains, particularly in time-series analysis, by providing a more nuanced understanding of instance difficulty. This could lead to improved convergence speeds and generalization in machine learning applications.
Precision-Varying Prediction (PVP): Robustifying ASR systems against adversarial attacks
Audio & Speech
- PVP enhances ASR robustness by varying numerical precision during inference.
- The method does not require retraining or access to model internals.
- A lightweight detection strategy is proposed based on transcription consistency across precision modes.
- Experiments show significant improvements in robustness and detection performance across multiple ASR models.
Read more
Precision-Varying Prediction (PVP): Robustifying ASR systems against adversarial attacks
Summary
This paper introduces Precision-Varying Prediction (PVP), a novel approach aimed at enhancing the adversarial robustness of automatic speech recognition (ASR) systems. The authors observe that varying the numerical precision during inference can significantly reduce the success rate of adversarial attacks on ASR models. By randomly sampling different precision levels during prediction, the model's robustness is improved without requiring retraining. Additionally, the authors propose a detection strategy that compares outputs from different precision settings to identify adversarial examples using a simple Gaussian classifier. Experimental results demonstrate that PVP not only increases the robustness of various ASR models against different types of adversarial attacks but also provides effective detection capabilities without degrading performance on benign inputs. This approach is model-agnostic, training-free, and efficient, making it suitable for practical deployment in real-world applications.
Methodology
The authors leverage numerical precision variations in ASR models during inference to enhance robustness against adversarial attacks. They implement a random sampling technique for precision levels and develop a Gaussian classifier to detect adversarial examples by comparing outputs from different precision configurations.
Results
The experimental analysis reveals a significant increase in adversarial robustness for various ASR models and attack types. The PVP approach provides competitive detection performance without compromising the accuracy of benign inputs, demonstrating its effectiveness in real-world scenarios.
Implications
The findings suggest that PVP can be effectively integrated into existing ASR systems to improve their security and reliability, particularly in applications where adversarial attacks pose significant risks, such as autonomous driving and healthcare.
Multimodal Training to Unimodal Deployment: Leveraging Unstructured Data During Training to Optimize Structured Data Only Deployment
Multimodal
- Introduces a multimodal learning framework that leverages unstructured EHR data for training while deploying a structured-only model.
- Utilizes contrastive learning and knowledge distillation to transfer knowledge from a teacher model to a student model.
- Achieves an AUROC of 0.705, outperforming the structured-only baseline of 0.656.
- Highlights the importance of unstructured data in enhancing model performance in clinical settings.
Read more
Multimodal Training to Unimodal Deployment: Leveraging Unstructured Data During Training to Optimize Structured Data Only Deployment
Summary
This paper presents a novel multimodal learning framework that utilizes unstructured Electronic Health Record (EHR) data, specifically clinical notes, during the training phase while ensuring that the resulting model can be deployed using only structured EHR data. The authors argue that unstructured data contains valuable clinical context that enhances model performance but is often impractical for deployment. They conducted their study on a cohort of 3,466 children evaluated for late talking, employing BioClinicalBERT to generate note embeddings and structured embeddings from demographic and medical codes. A teacher-student model approach was utilized, where a note-based teacher model was trained alongside a structured-only student model using contrastive learning and knowledge distillation. The results showed that the proposed model achieved an AUROC of 0.705, significantly outperforming the structured-only baseline of 0.656. This demonstrates that training with unstructured data can improve the model's ability to extract relevant information from structured data, leading to better performance in clinical applications.
Methodology
The authors employed a multimodal learning approach that involved generating embeddings from unstructured clinical notes using BioClinicalBERT and structured data from demographics and medical codes. They trained a note-based teacher model and a structured-only student model using contrastive learning and knowledge distillation techniques to optimize the performance of the student model during deployment.
Results
The proposed model achieved an AUROC of 0.705, which is a significant improvement over the structured-only baseline model that had an AUROC of 0.656. This indicates that incorporating unstructured data during training enhances the model's ability to identify relevant information from structured EHR data.
Implications
The findings suggest that leveraging unstructured data during training can lead to more effective clinical models that are deployable in real-world settings where only structured data is available. This approach could improve clinical decision-making and patient outcomes by enhancing the accuracy of computable phenotyping.
Instruction-Tuned, but Not More Verifiable Instruction-Following: A Cross-Task Diagnosis for LoRA Adapters
NLP
Large Language Models
- Nominal training objectives do not consistently predict actual performance improvements across tasks.
- The concept of 'capability drift' describes the mismatch between nominal labels and realized capabilities.
- Routine cross-task evaluations are essential before deploying models to avoid unintended performance shifts.
- Different benchmarks operationalize instruction following differently, leading to mixed evidence across evaluations.
Read more
Instruction-Tuned, but Not More Verifiable Instruction-Following: A Cross-Task Diagnosis for LoRA Adapters
Summary
This paper investigates the reliability of nominal labels, such as 'instruction-tuned', in predicting the actual performance improvements of LoRA adapters across different tasks. The author conducts a cross-task evaluation of the same LoRA adapter to determine if its nominal training objective aligns with realized capability gains, particularly focusing on strict, automatically verifiable instruction following as measured by IFEval. The findings reveal a recurrent but configuration-sensitive mismatch between nominal labels and actual performance, termed 'capability drift'. For instance, an instruction-tuned adapter significantly enhances performance on a numeric benchmark but fails to improve verifiable instruction following metrics. The paper emphasizes the importance of routine cross-task evaluations before deploying models and cautions against relying solely on nominal labels as proxies for capability. The results highlight the variability in how instruction following is operationalized across different benchmarks, suggesting that cross-benchmark agreement should not be assumed.
Methodology
The study employs an empirical cross-task diagnosis approach, evaluating the same LoRA adapter across multiple tasks and configurations. It analyzes the relationship between nominal training objectives and realized performance gains, particularly focusing on strict instruction-following benchmarks. The robustness of findings is assessed across various seeds, base models, and LoRA settings.
Results
The analysis reveals that nominal labels often fail to predict improvements in verifiable instruction following, with some configurations showing near-zero or negative performance changes. A specific example demonstrates that an instruction-tuned adapter improves numeric benchmark performance significantly while not enhancing strict instruction following metrics on IFEval.
Implications
The findings suggest that practitioners should be cautious in using nominal labels for model selection and deployment. Regular cross-task evaluations can help identify potential performance issues and ensure that models meet the desired capabilities in real-world applications.
DeepDTF: Dual-Branch Transformer Fusion for Multi-Omics Anticancer Drug Response Prediction
Multimodal
Graph Learning
Interpretability
- DeepDTF integrates multi-omics data and drug structures using a dual-branch Transformer architecture.
- The model achieves superior performance on drug response prediction tasks compared to existing baselines.
- It includes an interpretability module that connects predictions to biological pathways and gene attributions.
- DeepDTF addresses challenges of cross-modal misalignment and high-dimensional data in cancer drug response modeling.
Read more
DeepDTF: Dual-Branch Transformer Fusion for Multi-Omics Anticancer Drug Response Prediction
Summary
The paper introduces DeepDTF, a novel dual-branch Transformer fusion framework designed to enhance anticancer drug response prediction by integrating multi-omics data and drug chemical structures. The framework addresses the challenges of cross-modal misalignment and high-dimensional data by employing modality-specific encoders for cell-line multi-omics profiles and a GNN-Transformer for drug representations. The model performs joint log(IC50) regression and drug sensitivity classification, effectively capturing long-range dependencies and local topological features. DeepDTF outperforms existing models on public pharmacogenomic benchmarks, achieving significant improvements in predictive accuracy and classification error reduction. Additionally, the framework includes an interpretability module that utilizes SHAP-based gene attributions and pathway enrichment analysis, providing biologically relevant insights into the predictions. This work represents a significant step towards precision oncology by offering a robust computational tool for predicting drug responses based on complex biological data.
Methodology
DeepDTF employs a dual-branch architecture where the cell-line branch uses modality-specific encoders with Transformer blocks to process multi-omics profiles, while the drug branch utilizes a GNN-Transformer to represent molecular graphs. A Transformer fusion module integrates the representations from both branches, allowing for dynamic cross-modal attention to mitigate feature misalignment. The model is trained for joint log(IC50) regression and drug sensitivity classification, with an added interpretability pipeline based on SHAP and GSEA.
Results
DeepDTF consistently outperformed strong baselines across various omics settings, achieving an RMSE of 1.248, an R2 score of 0.875, and an AUC of 0.987 with full multi-omics inputs. The model also reduced classification error by 9.5%, demonstrating its effectiveness in predicting drug responses.
Implications
The development of DeepDTF has significant implications for precision oncology, as it provides a powerful computational tool for predicting drug responses based on complex biological data. The interpretability features enhance the understanding of the underlying biological mechanisms, supporting hypothesis generation and guiding future research in cancer treatment.
Lightweight Fairness for LLM-Based Recommendations via Kernelized Projection and Gated Adapters
NLP
Large Language Models
Efficient ML
- Proposes a lightweight bias mitigation method for LLM-based recommendations.
- Combines kernelized INLP for bias removal with a gated MoE adapter for utility restoration.
- Achieves fairness improvements without sacrificing recommendation accuracy.
- No additional trainable parameters are required, making it computationally efficient.
Read more
Lightweight Fairness for LLM-Based Recommendations via Kernelized Projection and Gated Adapters
Summary
This paper addresses the challenge of social bias in Large Language Model (LLM)-based recommender systems, which can amplify biases present in their training data. The authors propose a novel method that combines kernelized Iterative Null-space Projection (INLP) with a gated Mixture-of-Experts (MoE) adapter to mitigate bias without incurring additional optimization costs. The kernelized INLP allows for the removal of sensitive attributes from LLM representations in a single closed-form step, while the two-level MoE adapter selectively restores useful signals to maintain recommendation accuracy. The proposed method is lightweight, requiring no extra trainable parameters, and demonstrates effectiveness in reducing bias across multiple protected variables while preserving competitive recommendation quality. Experiments on public datasets validate the approach, showcasing its potential for fair and accurate recommendations.
Methodology
The methodology involves using kernelized Iterative Null-space Projection (INLP) to remove sensitive attributes from LLM representations in a closed-form manner. This is complemented by a two-level gated Mixture-of-Experts (MoE) adapter that selectively restores useful signals while mitigating bias. The approach leverages Random Fourier Features for kernelization and isotropic Gaussian perturbation to enhance robustness.
Results
The experiments conducted on two public datasets show that the proposed method significantly reduces attribute leakage across multiple protected variables while maintaining competitive recommendation accuracy, thus validating the effectiveness of the approach in achieving fairness in LLM-based recommendations.
Implications
The findings suggest that the proposed method can be applied to enhance fairness in various LLM-based applications, particularly in recommendation systems where bias mitigation is crucial. This approach could lead to more equitable user experiences and improve trust in automated systems.
GEM: Guided Expectation-Maximization for Behavior-Normalized Candidate Action Selection in Offline RL
Reinforcement Learning
- GEM provides a multimodal and controllable action selection framework for offline RL.
- The method preserves distinct action hypotheses while focusing on high-value regions through GMMs.
- Candidate-based selection allows for a flexible compute-quality trade-off at inference time.
- GEM mitigates the risk of out-of-distribution errors associated with naive candidate maximization.
Read more
GEM: Guided Expectation-Maximization for Behavior-Normalized Candidate Action Selection in Offline RL
Summary
The paper introduces GEM (Guided Expectation-Maximization), a novel framework for action selection in offline reinforcement learning (RL) that addresses the challenges of multimodal action landscapes and distributional shift. Traditional offline RL methods often struggle with action selection, particularly when the dataset leads to branched or multimodal action distributions, resulting in weakly supported 'in-between' actions. GEM employs a Gaussian Mixture Model (GMM) actor trained through critic-guided, advantage-weighted EM-style updates, which preserves distinct action components while focusing on high-value regions. During inference, GEM utilizes a candidate-based selection approach, generating a set of plausible actions and reranking them using a conservative ensemble lower-confidence bound alongside a behavior-normalized support signal. This design allows for stable control across different states and candidate budgets, enabling a trade-off between computational resources and decision quality without the need for retraining. Empirical evaluations demonstrate that GEM performs competitively on D4RL benchmarks, showcasing its effectiveness in offline RL scenarios.
Methodology
GEM employs a Gaussian Mixture Model (GMM) actor trained via critic-guided, advantage-weighted EM-style updates. The inference process involves generating a candidate set of actions and reranking them using a conservative ensemble lower-confidence bound and a behavior-normalized support signal, ensuring stable control across states.
Results
GEM demonstrates competitive performance on D4RL benchmarks, effectively addressing the challenges of action selection in offline RL by providing a robust candidate-based selection mechanism that allows for flexible decision-making without retraining.
Implications
The GEM framework has potential applications in various offline RL scenarios, particularly in environments where action landscapes are complex and multimodal. Its ability to control decision quality through candidate selection can enhance the reliability of RL agents in real-world deployments.
Kronecker-Structured Nonparametric Spatiotemporal Point Processes
Time Series
Theory
Interpretability
- KSTPP enables explicit discovery of event relationships while maintaining modeling flexibility.
- The model captures complex interaction patterns, including excitation, inhibition, and time-varying effects.
- Kronecker algebra is leveraged to reduce computational complexity and enhance scalability.
- The framework outperforms existing neural point process models in predictive tasks.
Read more
Kronecker-Structured Nonparametric Spatiotemporal Point Processes
Summary
This paper introduces the Kronecker-Structured Nonparametric Spatiotemporal Point Process (KSTPP), a novel framework designed to model spatiotemporal events with a focus on uncovering event relationships and improving prediction accuracy. Traditional models like Poisson and Hawkes processes are limited by their parametric assumptions, which restrict their ability to capture complex interactions. In contrast, KSTPP employs a spatial Gaussian process to model background intensity and a spatiotemporal Gaussian process for the influence kernel, allowing for rich interaction patterns, including excitation and inhibition. The model utilizes separable product kernels and structured grids to induce Kronecker-structured covariance matrices, significantly reducing computational costs and enabling scalability to large datasets. Additionally, a tensor-product Gauss-Legendre quadrature scheme is developed for efficient evaluation of likelihood integrals. Experimental results demonstrate that KSTPP outperforms state-of-the-art neural point process models in next-event prediction and accurately recovers underlying intensity functions and interaction patterns in synthetic datasets. Analysis of a real-world earthquake dataset reveals interpretable influence structures, showcasing the model's practical applicability.
Methodology
The KSTPP framework models the conditional intensity function using a combination of spatial and spatiotemporal Gaussian processes. It employs separable product kernels and structured grids to induce Kronecker-structured covariance matrices, facilitating efficient maximum likelihood training and predictive inference. A tensor-product Gauss-Legendre quadrature scheme is introduced to handle intractable likelihood integrals.
Results
KSTPP consistently outperformed state-of-the-art neural point process models in next-event prediction across three real-world benchmark datasets. It also demonstrated competitive performance against diffusion-based generative approaches. Synthetic experiments confirmed the model's ability to accurately recover underlying intensity functions and interaction patterns, while analysis of a real-world earthquake dataset revealed meaningful and interpretable influence structures.
Implications
The KSTPP framework has significant implications for various applications involving spatiotemporal event modeling, such as disaster response, epidemiology, and urban planning, where understanding event interactions is crucial for risk assessment and timely interventions.
Self Paced Gaussian Contextual Reinforcement Learning
Reinforcement Learning
Optimization
Theory
- SPGL avoids costly numerical optimizations by using a closed-form update for Gaussian contexts.
- The method maintains sample efficiency and adaptability while reducing computational overhead.
- SPGL shows superior performance in contextual RL benchmarks compared to existing methods.
- The approach is scalable and applicable to high-dimensional context spaces.
Read more
Self Paced Gaussian Contextual Reinforcement Learning
Summary
This paper introduces Self-Paced Gaussian Curriculum Learning (SPGL), a novel approach to curriculum reinforcement learning (CRL) that enhances the efficiency of reinforcement learning (RL) by sequencing tasks from simple to complex without relying on computationally expensive inner-loop optimizations. SPGL utilizes a closed-form update rule for Gaussian context distributions, which allows it to maintain the sample efficiency and adaptability of traditional self-paced methods while significantly reducing computational overhead. The authors provide theoretical guarantees on convergence and validate SPGL across various contextual RL benchmarks, including Point Mass, Lunar Lander, and Ball Catching environments. Experimental results demonstrate that SPGL matches or outperforms existing curriculum methods, particularly in scenarios with hidden contexts, and achieves more stable convergence of context distributions. This method presents a scalable and principled alternative for curriculum generation in complex continuous and partially observable domains.
Methodology
The authors propose SPGL, which integrates seamlessly with standard deep RL pipelines. It replaces traditional inner-loop optimization with a more efficient update mechanism for context distributions, allowing for adaptive curriculum generation without the computational burden typically associated with self-paced learning methods.
Results
SPGL was validated on several benchmarks, demonstrating that it matches or exceeds the performance of state-of-the-art curriculum methods, particularly in hidden context scenarios. The experimental results indicate more stable convergence of context distributions and improved sample efficiency.
Implications
The proposed method extends the applicability of curriculum learning to more complex and high-dimensional environments, making it suitable for real-world applications in reinforcement learning where computational resources are limited. It can enhance the training efficiency of RL agents in various domains, including robotics and game playing.
Beyond Accuracy: Introducing a Symbolic-Mechanistic Approach to Interpretable Evaluation
NLP
Interpretability
- Traditional accuracy metrics fail to reliably distinguish between generalization and memorization in machine learning models.
- The proposed symbolic-mechanistic evaluation framework combines symbolic rules with mechanistic interpretability to provide deeper insights into model behavior.
- A case study on NL-to-SQL tasks illustrates the limitations of standard evaluation metrics, revealing hidden failures in models that appear competent based on accuracy alone.
- The authors emphasize the need for mechanism-aware evaluation, particularly for tasks with clear algorithmic requirements.
Read more
Beyond Accuracy: Introducing a Symbolic-Mechanistic Approach to Interpretable Evaluation
Summary
This position paper critiques traditional accuracy-based evaluation methods in machine learning, particularly in the context of NLP tasks, arguing that they fail to distinguish genuine generalization from shortcuts like memorization and exploitation of spurious patterns. The authors propose a symbolic-mechanistic evaluation framework that integrates task-relevant symbolic rules with mechanistic interpretability. This approach allows for algorithmic pass/fail scores that reveal where models truly generalize versus where they exploit patterns. The authors demonstrate their method through a case study on NL-to-SQL tasks, comparing models trained with and without schema information. While standard accuracy metrics suggested similar performance, the symbolic-mechanistic evaluation exposed significant deficiencies in the model trained without schema, highlighting its reliance on shallow heuristics. The paper advocates for this new evaluation paradigm, particularly for tasks with well-defined algorithms, to better assess model capabilities beyond mere accuracy.
Methodology
The authors propose a symbolic-mechanistic evaluation framework that involves defining 'non-negotiable rules' in symbolic logic that any model solving a specific task must satisfy. They evaluate models against these rules using mechanistic interventions such as activation patching and attention visualization, assigning pass/fail scores to assess how well models adhere to expected generalization patterns.
Results
In the case study, two models were trained on the same architecture but under different conditions: one with schema information and one without. Standard accuracy metrics suggested similar performance levels, but the symbolic-mechanistic evaluation revealed that the model without schema failed to generalize properly, relying on shallow heuristics instead.
Implications
The proposed evaluation framework has the potential to improve the assessment of machine learning models, particularly in NLP, by providing a more nuanced understanding of their capabilities. It encourages the development of models that genuinely generalize rather than exploit shortcuts, which is crucial for applications in low-resource languages and specialized domains.
Rethinking Multimodal Fusion for Time Series: Auxiliary Modalities Need Constrained Fusion
Time Series
Multimodal
- Naive multimodal fusion strategies often underperform compared to unimodal TS models.
- Constrained fusion methods, including the proposed Controlled Fusion Adapter (CFA), significantly improve performance.
- CFA allows for controlled integration of auxiliary textual information without modifying the TS backbone.
- The study involved over 20,000 experiments across diverse datasets and models, validating the effectiveness of constrained fusion.
Read more
Rethinking Multimodal Fusion for Time Series: Auxiliary Modalities Need Constrained Fusion
Summary
This paper addresses the challenges of integrating auxiliary modalities, such as text and vision, into time series (TS) forecasting. The authors argue that existing multimodal fusion methods often yield limited improvements and can even underperform compared to unimodal TS models due to naive fusion strategies that indiscriminately combine information. They propose a new approach called Controlled Fusion Adapter (CFA), which allows for controlled integration of relevant auxiliary information while preserving the core temporal dynamics of the TS data. The CFA employs low-rank adapters to filter out irrelevant information before fusion, thus enhancing the forecasting performance. The authors conducted extensive experiments across various datasets and models, demonstrating that constrained fusion methods, including CFA, consistently outperform naive fusion techniques. The findings suggest that careful consideration of how modalities interact is crucial for effective multimodal TS forecasting.
Methodology
The authors explored various constrained fusion methods and proposed the Controlled Fusion Adapter (CFA) as a plug-in module for integrating auxiliary textual information into TS representations. They conducted extensive experiments across multiple datasets, TS models, and text models, evaluating different fusion strategies at various layers of the model architecture.
Results
The results indicated that constrained fusion methods consistently outperformed naive fusion strategies across all tested datasets and models. The Controlled Fusion Adapter (CFA) specifically demonstrated superior performance, effectively filtering irrelevant information and enhancing the integration of auxiliary signals.
Implications
The findings suggest that careful design of multimodal fusion strategies is essential for improving TS forecasting accuracy. The Controlled Fusion Adapter can be applied to various unimodal TS models, making it a versatile tool for enhancing forecasting performance in diverse applications such as finance, traffic prediction, and climate modeling.
Synthetic Mixed Training: Scaling Parametric Knowledge Acquisition Beyond RAG
NLP
Large Language Models
- Synthetic Mixed Training combines synthetic QAs and documents to improve knowledge acquisition.
- The approach yields log-linear improvements in performance as data volume and generator strength increase.
- Focal Rewriting enhances document diversity by conditioning generation on specific questions.
- The proposed methods outperform RAG in five out of six benchmark settings.
Read more
Synthetic Mixed Training: Scaling Parametric Knowledge Acquisition Beyond RAG
Summary
This paper introduces Synthetic Mixed Training, a novel approach to enhance the knowledge acquisition capabilities of language models in data-constrained domains. Traditional methods of synthetic data augmentation have shown diminishing returns when scaled, often failing to surpass the performance of Retrieval-Augmented Generation (RAG). The authors propose a combined training strategy that integrates synthetic question-answer pairs with synthetic documents, leveraging their complementary signals. This method leads to significant improvements in model performance, achieving a 2.6% relative gain over RAG on the QuaLITY benchmark. Additionally, the paper introduces Focal Rewriting, a technique that conditions document generation on specific questions, enhancing the diversity of generated documents and resulting in a further 4.4% improvement over RAG. The experiments demonstrate that the proposed methods generalize well across different model architectures and benchmarks, outperforming RAG in five out of six scenarios and achieving an average relative accuracy gain of 2.6%. The findings suggest that synthetic data can play a crucial role in improving the internalization of new knowledge in language models.
Methodology
The authors conducted experiments using four existing data generation algorithms to create synthetic QAs and documents, training a Llama 3.1 8B model on up to 700M synthetic tokens. They evaluated the performance on the QuaLITY benchmark, comparing the effectiveness of their Synthetic Mixed Training and Focal Rewriting techniques against traditional methods.
Results
The results indicated that the Synthetic Mixed Training approach surpassed RAG by 2.6% on the QuaLITY benchmark, with the final model achieving a 4.4% relative gain over RAG. Across multiple models and benchmarks, the training recipe consistently outperformed RAG, achieving an average relative accuracy improvement of 2.6% and a 9.1% gain when combined with RAG.
Implications
The findings suggest that integrating synthetic data in training can significantly enhance the performance of language models, particularly in knowledge-constrained domains. This approach could lead to more effective fine-tuning strategies and broaden the applicability of language models in specialized areas.
Permutation-Symmetrized Diffusion for Unconditional Molecular Generation
Generative Models
- Introduces a direct modeling approach for diffusion on the quotient manifold to achieve permutation invariance.
- Derives an explicit expression for the heat kernel on the quotient manifold, enhancing understanding of diffusion dynamics.
- Utilizes MCMC to approximate the permutation-symmetrized score for training.
- Demonstrates competitive performance in unconditional molecular generation tasks on the QM9 dataset.
Read more
Permutation-Symmetrized Diffusion for Unconditional Molecular Generation
Summary
This paper presents a novel approach to molecular generation using diffusion models that directly incorporates permutation invariance by modeling diffusion on a quotient manifold. Traditional methods enforce permutation invariance indirectly through permutation-equivariant networks in an ordered space, which can lead to inefficiencies. The authors propose a new framework that identifies all atom permutations on the quotient manifold ˜X = Rd×N/SN, allowing for a more natural representation of molecular structures. They derive an explicit expression for the heat kernel on this manifold, demonstrating how diffusion behaves differently compared to ordered-particle diffusion. The training process involves a permutation-symmetrized score, which is approximated using Markov Chain Monte Carlo (MCMC) methods. The proposed method is evaluated on unconditional 3D molecular generation tasks using the QM9 dataset, showing competitive generation quality and improved efficiency compared to existing methods.
Methodology
The authors model diffusion directly on the quotient manifold ˜X = Rd×N/SN, where all atom permutations are identified. They derive the heat kernel for this manifold and use MCMC to approximate the permutation-symmetrized score required for training. The evaluation is conducted using the EQGAT-Diff protocol on the QM9 dataset, employing a SemlaFlow-style backbone.
Results
The proposed method shows that quotient-based permutation symmetrization is practical, yielding competitive generation quality in 3D molecular generation tasks while improving computational efficiency compared to existing methods.
Implications
This approach could lead to more efficient and accurate models for molecular generation, which is crucial in drug discovery and materials science. The methodology may also inspire new techniques in other domains requiring permutation invariance.
Conditionally Identifiable Latent Representation for Multivariate Time Series with Structural Dynamics
Time Series
- Introduction of the Identifiable Variational Dynamic Factor Model (iVDFM) for multivariate time series.
- Achieves identifiability by conditioning on the innovation process rather than latent states.
- Utilizes linear diagonal dynamics to preserve identifiability and ensure computational efficiency.
- Demonstrates improved factor recovery and intervention accuracy on synthetic and real-world data.
Read more
Conditionally Identifiable Latent Representation for Multivariate Time Series with Structural Dynamics
Summary
In this paper, the authors introduce the Identifiable Variational Dynamic Factor Model (iVDFM), which aims to learn latent factors from multivariate time series data while ensuring identifiability. The key innovation of iVDFM is the application of iVAE-style conditioning to the innovation process that drives the dynamics, rather than conditioning on the latent states. This approach guarantees that the factors are identifiable up to permutation and component-wise affine transformations. The model employs linear diagonal dynamics to maintain this identifiability and allows for scalable computation through companion-matrix and Krylov methods. The authors demonstrate the effectiveness of iVDFM through various experiments, showing improved factor recovery on synthetic datasets, stable intervention accuracy on synthetic structural causal models, and competitive performance in probabilistic forecasting on real-world benchmarks. The paper addresses a significant gap in the literature by bridging the need for identifiable latent representations in dynamic settings, which is crucial for applications in macroeconomics, medicine, and causal representation learning.
Methodology
The iVDFM model is trained using variational inference, applying iVAE-style conditioning to the innovation process. The model defines the innovation as a conditional exponential-family distribution that depends on observed auxiliary variables and a regime embedding, allowing for time-varying priors. Linear diagonal dynamics are employed to map innovations to factors, preserving identifiability.
Results
The iVDFM shows significant improvements in factor recovery on synthetic datasets, maintains stable intervention accuracy in synthetic structural causal models, and achieves competitive results in probabilistic forecasting tasks on real-world benchmarks.
Implications
The findings suggest that iVDFM can be effectively used in various fields requiring interpretable and stable latent representations, such as macroeconomics and causal inference, enabling better understanding and analysis of temporal dynamics.
Efficient Controller Learning from Human Preferences and Numerical Data Via Multi-Modal Surrogate Models
Optimization
Robotics
Multimodal
- Introduces a multi-fidelity, multi-modal Bayesian optimization framework.
- Integrates low-fidelity numerical data with high-fidelity human preferences.
- Utilizes Gaussian process surrogate models for efficient learning.
- Demonstrates application in tuning an autonomous vehicle's trajectory planner.
Read more
Efficient Controller Learning from Human Preferences and Numerical Data Via Multi-Modal Surrogate Models
Summary
This paper addresses the challenge of tuning control policies in systems that involve human interaction, which often requires subjective evaluations. Traditional methods, such as Bayesian optimization (BO), are effective for numerical evaluations but can be inefficient when relying solely on human preferences. The authors propose a novel multi-fidelity, multi-modal Bayesian optimization framework that integrates low-fidelity numerical data with high-fidelity human preferences. By employing Gaussian process surrogate models with both hierarchical and non-hierarchical structures, the framework allows for efficient learning from mixed-modality data. The authors demonstrate the effectiveness of their approach through the tuning of an autonomous vehicle's trajectory planner, showing that the combination of numerical and preference data significantly reduces the need for human-involved experiments while effectively adapting the driving style to individual preferences. This work contributes to the field by providing a systematic method for leveraging diverse data sources in controller learning, enhancing both efficiency and user satisfaction.
Methodology
The authors develop a multi-modal Gaussian process model that combines numerical and preferential data for Bayesian optimization. The framework employs both hierarchical, autoregressive structures and non-hierarchical, coregionalization-based structures to facilitate efficient learning from mixed-modality data. This approach allows for the integration of various information sources, enhancing the optimization process.
Results
The proposed framework was successfully applied to tune the trajectory planner of an autonomous vehicle. The results indicated that combining numerical evaluations with human preference data significantly reduced the number of required experiments involving human decision-makers, while effectively adapting the driving style to meet individual preferences.
Implications
This research has significant implications for the development of intelligent systems that require human interaction, such as autonomous vehicles and assistive technologies. By improving the efficiency of controller tuning processes, the framework can lead to better user experiences and more effective system performance in real-world applications.
Research on Individual Trait Clustering and Development Pathway Adaptation Based on the K-means Algorithm
Theory
- Utilizes K-means clustering to categorize students based on individual traits.
- Focuses on the fitness of students for specific career paths rather than just predicting career outcomes.
- Provides targeted career guidance based on clustering results, enhancing personalized support.
- Demonstrates the effectiveness of data-driven approaches in improving employment success rates for students.
Read more
Research on Individual Trait Clustering and Development Pathway Adaptation Based on the K-means Algorithm
Summary
This study investigates the application of the K-means clustering algorithm to enhance career guidance for college students by analyzing their individual traits. Unlike existing methods that primarily focus on predicting career paths, this research emphasizes the fitness of students with varying combinations of characteristics for specific career directions. The authors analyzed data from over 3000 students, including CET-4 scores, GPA, personality traits, and leadership experiences, to classify them into four distinct groups using K-means clustering. The algorithm minimizes intra-cluster squared error to ensure high similarity among students within the same cluster while maximizing differences between clusters. Based on these clusters, tailored career guidance suggestions were developed, demonstrating that students with different trait combinations are suited for different career paths. The findings provide a scientific basis for personalized career guidance, potentially improving students' employment success rates. The study suggests that future research could enhance clustering precision and guidance effectiveness by expanding sample sizes, increasing feature variables, and considering external factors.
Methodology
The study employed the K-means clustering algorithm to analyze and categorize student data, including CET-4 scores, GPA, personality traits, and leadership experiences. Data preprocessing and normalization were conducted to ensure consistency across different scales before clustering.
Results
The K-means clustering resulted in four distinct groups of students, each exhibiting unique combinations of traits. The analysis revealed that these groups correspond to different suitable career directions, thereby validating the effectiveness of the clustering approach in providing personalized career guidance.
Implications
The findings suggest that educational institutions can leverage machine learning techniques, specifically K-means clustering, to offer tailored career guidance, ultimately improving students' employability and career satisfaction. This approach can serve as a framework for developing more sophisticated career counseling systems in universities.
From Arithmetic to Logic: The Resilience of Logic and Lookup-Based Neural Networks Under Parameter Bit-Flips
Theory
Efficient ML
- Resilience against bit-flip errors is a structural property of neural architectures.
- Lower precision, higher sparsity, bounded activations, and shallow depth improve resilience.
- Logic and Lookup-Based Neural Networks (LUT-NNs) demonstrate superior stability under corruption.
- A novel Even-Layer Recovery effect is observed in logic-based architectures.
Read more
From Arithmetic to Logic: The Resilience of Logic and Lookup-Based Neural Networks Under Parameter Bit-Flips
Summary
This paper investigates the resilience of neural network architectures against hardware-induced bit-flip errors, particularly in safety-critical edge environments. The authors propose that resilience should be viewed as a structural property of neural architectures rather than merely a characteristic of trained models. They derive the Expected Squared Error (MSE) under independent parameter bit flips across various numerical formats and layer types, establishing that lower precision, higher sparsity, bounded activations, and shallower networks enhance resilience. The study highlights Logic and Lookup-Based Neural Networks (LUT-NNs) as optimal designs that embody these traits. Through ablation studies on the MLPerf Tiny benchmark suite, the authors validate their theoretical predictions, demonstrating that LUT-based models maintain stability in corruption scenarios where traditional floating-point models fail. Additionally, they identify a unique Even-Layer Recovery effect in logic-based architectures, suggesting that transitioning from continuous arithmetic weights to discrete Boolean lookups can improve the accuracy-resilience trade-off for hardware fault tolerance.
Methodology
The authors derive a formal framework for analyzing expected output error due to parameter bit flips across various neural network architectures, including integer, floating-point, quantized, binary, and LUT-based models. They conduct ablation studies using the MLPerf Tiny benchmark suite to empirically validate their theoretical findings.
Results
The study finds that LUT-based models exhibit significantly higher stability under severe corruption compared to standard floating-point models. The empirical results align with theoretical predictions, confirming that the identified architectural factors contribute to enhanced resilience. The Even-Layer Recovery effect is also documented, showcasing unique recovery characteristics in logic-based networks.
Implications
The results suggest that adopting Logic and Lookup-Based Neural Networks could enhance the reliability of neural networks in environments prone to hardware faults, making them suitable for deployment in safety-critical applications such as autonomous vehicles and medical devices.
Asymptotic Learning Curves for Diffusion Models with Random Features Score and Manifold Data
Generative Models
Theory
- Asymptotic expressions for errors in diffusion models are derived, highlighting the impact of manifold structure on sample complexity.
- For linear manifolds, sample complexity scales linearly with intrinsic dimension, while this advantage diminishes for non-linear manifolds.
- The study uses random feature neural networks to parameterize the score function, providing insights into the learning process of diffusion models.
- The findings suggest that the geometric structure of data significantly influences the performance of generative models.
Read more
Asymptotic Learning Curves for Diffusion Models with Random Features Score and Manifold Data
Summary
This paper investigates the theoretical behavior of denoising score matching (DSM) in the context of diffusion models when the data distribution is supported on a low-dimensional manifold. The authors derive asymptotic expressions for test, train, and score errors in high-dimensional settings, revealing that for linear manifolds, the sample complexity for learning the score function scales linearly with the intrinsic dimension rather than the ambient dimension. However, this benefit diminishes for non-linear manifolds. The study emphasizes the importance of both the data structure and the score parameterization in determining the effectiveness of diffusion models. By employing a random feature neural network (RFNN) to parameterize the score function and a hidden manifold model (HMM) for the data, the authors provide a precise characterization of the learning curves, contributing to a deeper understanding of the sample complexity in generative modeling.
Methodology
The authors utilize a theoretical framework to analyze denoising score matching in diffusion models, focusing on high-dimensional limits where both the ambient dimension and the number of samples grow infinitely. They employ random feature neural networks to parameterize the score function and analyze the implications of low-dimensional manifold structures on sample complexity.
Results
The paper presents asymptotic characterizations of test and train errors, showing that for sufficiently linear manifolds, the required sample size for accurate score learning is linearly dependent on the intrinsic dimension. The results indicate that while structured data can enhance model performance, the type of structure is crucial.
Implications
These findings could inform the design of more efficient generative models by leveraging low-dimensional manifold structures in data, potentially leading to advancements in various applications such as image and audio generation.
LineMVGNN: Anti-Money Laundering with Line-Graph-Assisted Multi-View Graph Neural Networks
Graph Learning
- Introduction of LineMVGNN, a new GNN model for AML detection.
- Utilizes line graphs to enhance transaction information propagation.
- Demonstrates superior performance compared to existing state-of-the-art methods.
- Addresses scalability and interpretability issues in traditional AML systems.
Read more
LineMVGNN: Anti-Money Laundering with Line-Graph-Assisted Multi-View Graph Neural Networks
Summary
This paper presents LineMVGNN, a novel approach to anti-money laundering (AML) that leverages line-graph-assisted multi-view graph neural networks (GNNs). Traditional AML systems often rely on rule-based methods, which can be inefficient and lack scalability. The authors argue that existing GNNs, particularly those designed for directed graphs, face limitations in handling multi-dimensional edge features and capturing transaction flows effectively. LineMVGNN addresses these challenges by incorporating a lightweight multi-view GNN module that facilitates two-way message passing between nodes in a transaction graph. Additionally, it utilizes a line graph representation of the original transaction graph to enhance the propagation of transaction information, thereby improving the detection of suspicious transactions and accounts. The authors conducted experiments on two real-world datasets: the Ethereum phishing transaction network and a financial payment transaction dataset. The results indicate that LineMVGNN outperforms state-of-the-art methods in AML detection, demonstrating its effectiveness and scalability. The paper also discusses the model's robustness against adversarial attacks and its regulatory implications for AML practices.
Methodology
The methodology involves developing a multi-view GNN framework that performs two-way message passing in transaction graphs. The model incorporates a line graph representation to facilitate edge feature propagation, allowing for better capture of transaction flows and interactions. Experiments were conducted on real-world datasets to validate the model's effectiveness.
Results
The experimental results show that LineMVGNN significantly outperforms existing state-of-the-art GNN methods for AML detection, indicating its effectiveness in identifying suspicious transactions and accounts. The model also demonstrates scalability and robustness against adversarial attacks.
Implications
The proposed LineMVGNN model has significant implications for improving AML systems, offering a more effective and scalable solution for detecting money laundering activities. Its ability to handle complex transaction data can enhance regulatory compliance and financial security.
Language-Assisted Image Clustering Guided by Discriminative Relational Signals and Adaptive Semantic Centers
Computer Vision
NLP
Multimodal
- Proposes a new framework for Language-Assisted Image Clustering (LAIC) addressing key limitations of existing methods.
- Enhances inter-class discriminability by utilizing cross-modal relations for self-supervision signals.
- Implements prompt learning to create adaptive semantic centers for improved clustering assignments.
- Achieves an average performance improvement of 2.6% over state-of-the-art methods across multiple datasets.
Read more
Language-Assisted Image Clustering Guided by Discriminative Relational Signals and Adaptive Semantic Centers
Summary
The paper introduces a novel framework for Language-Assisted Image Clustering (LAIC) that addresses two critical issues in existing methods: the lack of inter-class discriminability due to similar textual features and the limitations imposed by pre-built image-text alignments. The proposed framework consists of two main components. First, it utilizes cross-modal relations to generate more discriminative self-supervision signals for clustering, which aligns well with the training mechanisms of vision-language models (VLMs). Second, it employs prompt learning to create category-wise continuous semantic centers that enhance clustering assignments. The authors conducted extensive experiments across eight benchmark datasets, demonstrating an average performance improvement of 2.6% over state-of-the-art methods. The learned semantic centers also exhibited strong interpretability, indicating their effectiveness in capturing meaningful semantic information.
Methodology
The methodology involves constructing an image-text representation matrix to minimize discrepancies between image and text modalities, enhancing inter-class discriminability. K-means clustering is performed on this representation matrix. Additionally, category-wise semantic centers are learned through prompt learning, optimizing the alignment between these centers and image features for final clustering assignments.
Results
The proposed method outperformed existing LAIC approaches by an average of 2.6% across eight benchmark datasets. The learned semantic centers provided strong interpretability, indicating their capacity to capture nuanced semantic information effectively.
Implications
This research has significant implications for improving unsupervised image clustering techniques by leveraging language models, potentially enhancing applications in image retrieval, organization, and analysis in various domains such as e-commerce, social media, and digital asset management.
Cost-Sensitive Neighborhood Aggregation for Heterophilous Graphs: When Does Per-Edge Routing Help?
Graph Learning
- Introduces Cost-Sensitive Neighborhood Aggregation (CSNA) for GNNs to handle heterophilous graphs.
- Distinguishes between adversarial and informative heterophily regimes and their implications for message routing.
- Demonstrates that CSNA can preserve class-discriminative signals where mean aggregation fails.
- Finds that per-edge routing is beneficial in adversarial contexts but not in informative ones.
Read more
Cost-Sensitive Neighborhood Aggregation for Heterophilous Graphs: When Does Per-Edge Routing Help?
Summary
This paper addresses the challenges posed by heterophilous graphs in the context of Graph Neural Networks (GNNs), where connected nodes often belong to different classes. The author distinguishes between two heterophily regimes: adversarial, where cross-class edges dilute class signals, and informative, where heterophilous structures provide useful signals. The key contribution is the introduction of Cost-Sensitive Neighborhood Aggregation (CSNA), a GNN layer that utilizes learned pairwise distances to route messages through two distinct channels—concordant and discordant—each with independent transformations. The study demonstrates that CSNA preserves class-discriminative signals in adversarial-heterophily scenarios, outperforming mean aggregation methods. However, it underperforms in informative-heterophily contexts, indicating that the effectiveness of per-edge routing is contingent on the nature of the heterophily present. The findings suggest that the ability of the cost function to differentiate edge types serves as a diagnostic tool for determining when fine-grained routing is beneficial. The paper provides empirical results across six benchmarks, showing CSNA's competitiveness with state-of-the-art methods in adversarial settings while revealing its limitations in informative scenarios.
Methodology
The methodology involves the development of CSNA, which computes pairwise distances in a learned projection space to soft-route messages through two channels based on edge concordance. The model employs a per-node gating mechanism to combine the outputs from these channels, allowing for individualized routing weights for each edge. The performance is evaluated under a contextual stochastic block model and across multiple benchmark datasets.
Results
CSNA shows competitive performance against state-of-the-art methods on adversarial-heterophily datasets, such as Texas, Wisconsin, Cornell, and Actor. However, it underperforms on informative-heterophily datasets like Chameleon and Squirrel, highlighting the regime-dependent effectiveness of per-edge routing. The results indicate that the cost function's ability to separate edge types is crucial for determining the utility of fine-grained routing.
Implications
The findings have significant implications for the design of GNN architectures, particularly in applications involving heterogeneous data structures. Understanding when to apply cost-sensitive routing can enhance model performance in real-world scenarios where heterophily is prevalent, guiding future research in graph learning.
Probabilistic Geometric Alignment via Bayesian Latent Transport for Domain-Adaptive Foundation Models
Theory
- Introduction of a novel uncertainty-aware probabilistic latent transport framework for foundation model adaptation.
- Development of a Bayesian transport operator for geometry-preserving feature transfer under distributional shifts.
- Integration of optimal transport dynamics with PAC-Bayesian generalization control, providing theoretical guarantees.
- Empirical results demonstrate superior performance in latent manifold alignment and uncertainty calibration.
Read more
Probabilistic Geometric Alignment via Bayesian Latent Transport for Domain-Adaptive Foundation Models
Summary
This paper addresses the challenge of adapting large-scale foundation models to new domains with limited supervision, which is often hindered by latent distribution mismatch, unstable optimization dynamics, and miscalibrated uncertainty propagation. The authors propose an uncertainty-aware probabilistic latent transport framework that formulates domain adaptation as a stochastic geometric alignment problem in representation space. A Bayesian transport operator is introduced to redistribute latent probability mass along Wasserstein-type geodesic trajectories, while a PAC-Bayesian regularization mechanism is employed to constrain posterior model complexity, thereby mitigating catastrophic overfitting. The framework provides theoretical guarantees on convergence stability, loss landscape smoothness, and sample efficiency under distributional shifts. Empirical analyses show a significant reduction in latent manifold discrepancy, accelerated transport energy decay, and improved covariance calibration compared to traditional methods such as deterministic fine-tuning and adversarial domain adaptation. The bounded posterior uncertainty evolution indicates enhanced probabilistic reliability during cross-domain transfer. By connecting stochastic optimal transport geometry with statistical generalization theory, this framework offers new insights into the robust adaptation of foundation architectures in heterogeneous environments, suggesting that uncertainty-aware probabilistic alignment is a promising approach for reliable transfer learning in next-generation deep representation systems.
Methodology
The methodology involves formulating domain adaptation as a stochastic geometric alignment problem, utilizing a Bayesian transport operator to redistribute latent probability mass. PAC-Bayesian regularization is applied to control generalization error, while stochastic representation matching is employed to prevent overconfident adaptation in low-data scenarios.
Results
The proposed framework achieves substantial reductions in latent manifold discrepancy, faster transport energy decay, and improved covariance calibration compared to baseline methods. The empirical analysis indicates enhanced probabilistic reliability during cross-domain transfer, with theoretical guarantees on convergence and sample efficiency.
Implications
This work has significant implications for the reliable transfer of foundation models in low-data environments, enhancing their adaptability to new domains while maintaining robustness and interpretability. It suggests a new paradigm for transfer learning that could be applied across various machine learning applications.
Causality-Driven Disentangled Representation Learning in Multiplex Graphs
Graph Learning
- Introduces CaDeM, a causal inference-based framework for disentangled representation learning in multiplex graphs.
- Employs a GCN backbone and integrates three objectives for effective disentanglement of common and private embeddings.
- Demonstrates significant improvements in representation learning performance across various datasets and tasks.
- Addresses limitations of existing methods that do not explicitly control for confounding effects in multiplex graphs.
Read more
Causality-Driven Disentangled Representation Learning in Multiplex Graphs
Summary
This paper addresses the challenge of learning representations from multiplex graphs, which are multi-layer networks where nodes interact through various relation types. The authors propose a novel framework called CaDeM (Causality-driven self-supervised framework for Disentangled representation learning in Multiplex graphs) that disentangles common (shared across layers) and private (layer-specific) components in a self-supervised manner. The framework employs a Graph Convolutional Neural Network (GCN) backbone and integrates three key objectives: aligning shared embeddings across layers, enforcing private embeddings to capture meaningful layer-specific signals, and applying a causal loss for backdoor adjustment to mitigate confounding effects. The effectiveness of CaDeM is demonstrated through extensive experiments on both synthetic and real-world datasets, showing consistent improvements over existing methods in terms of robustness and interpretability of the learned representations. The results indicate that CaDeM achieves superior performance in downstream tasks, such as node-level and graph-level applications, by producing cleaner representations that accurately reflect the intended roles of common and private embeddings.
Methodology
The methodology involves a Graph Convolutional Neural Network (GCN) backbone that combines three objectives: a matching objective for aligning common embeddings, a self-supervised objective for capturing layer-specific signals, and a causal loss for backdoor adjustment to control confounding effects. This approach ensures that common embeddings retain global information while private embeddings focus on layer-specific dynamics.
Results
The experiments conducted on synthetic and real-world datasets reveal that CaDeM consistently outperforms existing baselines, achieving the best Macro- and Micro-F1 scores across various application domains. The qualitative and quantitative analyses confirm that the framework effectively disentangles shared and layer-specific components, leading to more interpretable representations.
Implications
The findings suggest that CaDeM can be applied in complex, multi-modal domains such as social networks and brain activity modeling, where understanding the distinct influences of different types of interactions is crucial. The framework's ability to produce robust and interpretable representations can enhance applications in recommendation systems, community detection, and other graph-based tasks.
Wireless communication empowers online scheduling of partially-observable transportation multi-robot systems in a smart factory
Robotics
Optimization
- Introduces a communication-enabled online scheduling framework for T-MRS in smart factories.
- Integrates wireless M2M networking with route scheduling to enhance AGV coordination.
- Demonstrates significant improvements in scheduling efficiency compared to traditional methods.
- Highlights the differences between M2M and human-to-human communication in the context of scheduling.
Read more
Wireless communication empowers online scheduling of partially-observable transportation multi-robot systems in a smart factory
Summary
This paper addresses the challenge of online scheduling in transportation multi-robot systems (T-MRS) within smart factories, particularly under conditions of partial observability. The authors propose a novel framework that integrates wireless machine-to-machine (M2M) communication with route scheduling to enable collaborative automatic guided vehicles (AGVs) to share intention information and planned routes. This approach helps overcome the limitations of partial observations and enhances the computation of online scheduling. The framework combines a simulated annealing-based multi-robot task assignment (MRTA) scheme with a congestion-aware A*-based route scheduling method. The results from numerical experiments demonstrate that the proposed integrated communication and scheduling scheme significantly improves scheduling efficiency, even under high AGV load conditions and limited channel resources. The findings suggest that the scheduling-oriented wireless M2M communication design is fundamentally different from human-to-human communication, indicating new technological opportunities for smart factories.
Methodology
The authors developed a framework that couples wireless M2M communication with route scheduling. They employed a simulated annealing-based MRTA scheme and a congestion-aware A*-based route scheduling method. The framework allows AGVs to dynamically adjust their routes based on shared intention information and sensor data, thus reducing computational overhead and improving real-time decision-making.
Results
Numerical experiments indicated that the proposed integrated communication and scheduling scheme significantly enhances the scheduling efficiency of T-MRS, particularly under high load conditions and limited communication resources. The results also revealed that the M2M communication design fundamentally differs from traditional human communication, providing new insights into scheduling performance in smart factories.
Implications
The findings suggest that implementing wireless M2M communication in smart factories can lead to more efficient and agile production processes. This approach could be applied to various industrial settings where multi-robot systems operate under dynamic conditions, potentially transforming operational strategies in manufacturing and logistics.
Safe Reinforcement Learning with Preference-based Constraint Inference
Reinforcement Learning
Robotics
Optimization
- Introduces PbCRL, a novel method for inferring safety constraints from human preferences.
- Addresses limitations of traditional Bradley-Terry models in capturing heavy-tailed cost distributions.
- Incorporates a dead zone mechanism and SNR loss to improve exploration and constraint alignment.
- Demonstrates superior performance in safety and reward compared to existing methods.
Read more
Safe Reinforcement Learning with Preference-based Constraint Inference
Summary
This paper addresses the challenges of safe reinforcement learning (RL), particularly in inferring complex and subjective safety constraints from human preferences. Existing methods often rely on restrictive assumptions or extensive expert demonstrations, which are impractical in many real-world scenarios. The authors propose a novel approach called Preference-based Constrained Reinforcement Learning (PbCRL) that introduces a dead zone mechanism into preference modeling to better capture the heavy-tailed nature of safety costs. This approach theoretically ensures better constraint alignment and incorporates a Signal-to-Noise Ratio (SNR) loss to promote exploration through cost variances. Additionally, a two-stage training strategy is employed to reduce online labeling burdens while enhancing constraint satisfaction. Empirical results show that PbCRL significantly outperforms state-of-the-art baselines in terms of safety and reward, demonstrating its effectiveness in aligning with true safety requirements and its potential for various safety-critical applications.
Methodology
The authors developed PbCRL, which integrates a dead zone mechanism into preference modeling to encourage heavy-tailed cost distributions. They also introduced an SNR loss to enhance exploration and employed a two-stage training strategy to minimize online labeling efforts while improving constraint satisfaction.
Results
Empirical evaluations indicate that PbCRL achieves better alignment with actual safety requirements and surpasses state-of-the-art methods in both safety and reward metrics, confirming its effectiveness in real-world applications.
Implications
The findings suggest that PbCRL can be effectively applied in various safety-critical domains such as robotics and autonomous driving, where accurately modeling safety constraints is essential for safe decision-making.
MolEvolve: LLM-Guided Evolutionary Search for Interpretable Molecular Optimization
Large Language Models
Optimization
Interpretability
- Introduces MolEvolve, an evolutionary framework for molecular optimization using LLMs.
- Addresses the challenge of interpretability in molecular property prediction.
- Utilizes a closed-loop verification mechanism to ensure high-precision chemical insights.
- Outperforms existing GNN and LLM-based methods in property prediction and optimization tasks.
Read more
MolEvolve: LLM-Guided Evolutionary Search for Interpretable Molecular Optimization
Summary
MolEvolve presents a novel framework for molecular optimization that addresses the limitations of existing deep learning methods in chemistry, particularly their lack of interpretability and inability to handle 'activity cliffs'—situations where small structural changes lead to significant shifts in molecular properties. The framework reformulates molecular discovery as an autonomous planning problem, utilizing a Large Language Model (LLM) to explore and evolve a library of chemical symbolic operations. By integrating a Monte Carlo Tree Search (MCTS) engine, MolEvolve autonomously discovers optimal molecular trajectories while generating transparent reasoning chains that translate complex structural transformations into human-readable insights. The experimental results indicate that MolEvolve not only provides interpretable chemical insights but also surpasses traditional baselines in property prediction and molecule optimization tasks, thereby bridging the gap between semantic understanding and physical accuracy in molecular discovery.
Methodology
MolEvolve employs an LLM for an initial cold start to propose heuristic rules for molecular properties, followed by a Monte Carlo Tree Search (MCTS) that navigates a space of executable chemical symbolic operations. The framework integrates external chemical tools for rule generation and verification, ensuring that each proposed operation is rigorously evaluated during the search process.
Results
Experimental evaluations demonstrate that MolEvolve outperforms both Graph Neural Networks (GNNs) and LLM-based methods in property prediction tasks, as well as domain-specific models and standard LLMs in molecule optimization tasks, showcasing its effectiveness in generating interpretable and precise chemical insights.
Implications
The MolEvolve framework has significant implications for the field of computational chemistry, enabling more interpretable and efficient molecular discovery processes. It can facilitate the design of new molecules with desired properties, enhance the understanding of structure-activity relationships, and potentially accelerate drug discovery and material science applications.
Robustness Quantification for Discriminative Models: a New Robustness Metric and its Application to Dynamic Classifier Selection
Theory
- Introduction of a new robustness metric applicable to any probabilistic discriminative classifier.
- The metric is based on Constant Odds Ratio (COR) perturbation, allowing for broader applicability.
- Demonstrated correlation with accuracy through experiments using Accuracy Rejection Curves.
- Application of the metric in dynamic classifier selection to improve prediction reliability.
Read more
Robustness Quantification for Discriminative Models: a New Robustness Metric and its Application to Dynamic Classifier Selection
Summary
This paper addresses the challenge of quantifying the robustness of discriminative models in machine learning, particularly in the context of uncertainty in predictions. The authors propose a new robustness metric that is applicable to any probabilistic discriminative classifier, overcoming the limitations of existing metrics that require generative models or are restricted to specific architectures or discrete features. The new metric is based on the Constant Odds Ratio (COR) perturbation, allowing for a broader application across various feature types. The authors demonstrate the effectiveness of this metric through experiments using Accuracy Rejection Curves, showing that it correlates well with prediction accuracy and outperforms existing alternatives. Furthermore, the paper explores the application of this robustness metric in dynamic classifier selection, providing strategies for selecting the most reliable model based on the features at hand. The findings suggest that robustness quantification can enhance decision-making in high-stakes scenarios by identifying reliable predictions amidst uncertainty.
Methodology
The authors developed a new robustness metric based on COR perturbation, which quantifies how stable a classifier's prediction is against perturbations in the underlying probability distribution. They conducted experiments using Accuracy Rejection Curves to evaluate the correlation between the new metric and prediction accuracy across various model architectures.
Results
The new robustness metric was shown to correlate effectively with prediction accuracy, outperforming existing robustness metrics. The application of this metric in dynamic classifier selection demonstrated improved strategies for choosing the most reliable model based on input features, enhancing the overall decision-making process in uncertain environments.
Implications
The proposed robustness metric has significant implications for improving the reliability of machine learning models in high-stakes applications, where understanding and quantifying uncertainty is crucial. It opens avenues for further research in robustness quantification and its integration into dynamic model selection processes.
Manifold Generalization Provably Proceeds Memorization in Diffusion Models
Generative Models
Theory
- Diffusion models can generate novel samples with coarse scores by capturing the geometry of the data.
- The manifold hypothesis provides a framework for understanding generalization in diffusion models.
- Generalization occurs at a faster statistical rate than full density estimation, especially for smooth manifolds.
- Coarse score accuracy can still yield fine on-manifold coverage, enabling high-quality sample generation.
Read more
Manifold Generalization Provably Proceeds Memorization in Diffusion Models
Summary
This paper investigates the phenomenon where diffusion models generate novel samples even when the learned score is coarse, challenging the conventional view of diffusion training as density estimation. The authors propose that this behavior can be explained by the manifold hypothesis, suggesting that coarse scores capture the geometry of the data while neglecting the fine-scale distributional structure. They demonstrate that diffusion models can achieve a near-parametric rate of generalization by exploiting the regularity of the manifold support, allowing for the generation of high-fidelity samples without memorizing the training data. The study establishes that generalization occurs at a statistical rate faster than that required for full population distribution estimation, particularly when the underlying manifold is smooth. The findings emphasize that the relevant objective for generalization is coverage of the manifold at a non-trivial spatial resolution rather than accurate recovery of the full density.
Methodology
The authors analyze diffusion models under the manifold hypothesis, focusing on the relationship between score accuracy and sample generation. They decompose the analysis into two noise regimes: a moderate-to-large noise regime where score learning is sufficiently accurate, and a small-noise regime where the focus shifts to geometric recovery. The paper employs theoretical proofs to establish the rates of generalization and coverage based on the manifold's regularity and smoothness.
Results
The main result shows that diffusion models trained with coarse scores can achieve δ-coverage at a scale that is significantly finer than that of the empirical distribution, specifically at a rate of O(N^(-β/4k)). This indicates that, under certain conditions, the induced sampling dynamics can be close to a distribution that effectively covers the manifold, allowing for the generation of high-quality samples that do not simply memorize the training data.
Implications
The findings suggest that diffusion models can be effectively utilized in scenarios where high-quality sample generation is required without the risk of overfitting to training data. This has potential applications in generative modeling tasks across various domains, including image synthesis and data augmentation, where maintaining diversity and novelty in generated samples is crucial.
Towards Safe Learning-Based Non-Linear Model Predictive Control through Recurrent Neural Network Modeling
Optimization
Robotics
Reinforcement Learning
- Introduction of Sequential-AMPC, a sequential neural policy for NMPC.
- Integration of safety-augmented online evaluation and fallback mechanisms.
- Significant reduction in expert MPC rollouts needed for training.
- Improved feasibility rates and closed-loop safety compared to baseline methods.
Read more
Towards Safe Learning-Based Non-Linear Model Predictive Control through Recurrent Neural Network Modeling
Summary
This paper addresses the challenges of deploying Nonlinear Model Predictive Control (NMPC) in safety-critical systems, particularly regarding the computational demands of online optimization. The authors propose a novel approach called Sequential-AMPC, which utilizes a sequential neural policy to generate MPC candidate control sequences by sharing parameters across the prediction horizon. This method reduces the reliance on large expert datasets and costly training typically associated with learning-based NMPC. The proposed Safe Sequential-AMPC integrates a safety-augmented online evaluation and fallback mechanism to ensure closed-loop safety and feasibility of control sequences. The results demonstrate that Sequential-AMPC outperforms a naive feedforward policy baseline in terms of requiring fewer expert MPC rollouts, achieving higher feasibility rates, and improving closed-loop safety. Additionally, it shows better learning dynamics and performance in high-dimensional systems while maintaining stable validation improvement, contrasting with the stagnation observed in the feedforward baseline.
Methodology
The authors developed Sequential-AMPC, which employs a recurrent neural network (RNN) architecture to recursively generate control sequences. This approach allows for parameter sharing across the prediction horizon, enhancing the efficiency of the control process. The method is embedded within a safety-augmented framework that evaluates the feasibility and cost of candidate sequences, providing a fallback mechanism when necessary.
Results
The experimental results indicate that Sequential-AMPC requires significantly fewer expert MPC rollouts than traditional methods while yielding candidate sequences with higher feasibility rates. It consistently outperforms the naive feedforward policy across various benchmarks, demonstrating improved closed-loop safety and better learning dynamics in high-dimensional systems.
Implications
The findings suggest that Sequential-AMPC could enhance the deployment of NMPC in safety-critical applications such as autonomous vehicles and robotics, where computational efficiency and safety are paramount. This approach may lead to more robust and reliable control systems capable of operating under tight computational constraints.
Full waveform inversion method based on diffusion model
Generative Models
Optimization
Theory
- Introduction of a conditional diffusion model for full waveform inversion.
- Utilization of two-dimensional density information to improve inversion accuracy.
- Demonstrated enhanced resolution and structural fidelity in inversion results.
- Increased stability and robustness in complex geological scenarios.
Read more
Full waveform inversion method based on diffusion model
Summary
This paper presents a novel full waveform inversion (FWI) method that leverages a conditional diffusion model to enhance the resolution and stability of seismic subsurface imaging. Traditional FWI techniques often struggle with nonlinearity and sensitivity to initial models, leading to local minima during inversion. The authors propose a conditional diffusion model that incorporates two-dimensional density information into a U-Net architecture, addressing the physical coupling between velocity and density. Experimental results demonstrate that this approach significantly improves the fidelity of inversion results, showcasing enhanced robustness in complex scenarios. The method effectively utilizes density constraints to guide the inversion process, indicating its practical applicability in seismic data analysis.
Methodology
The proposed method enhances the traditional full waveform inversion framework by integrating a conditional diffusion model. This model is designed to incorporate two-dimensional density information as a conditional input into a U-Net network, allowing for a more accurate representation of the physical relationships between subsurface parameters. The inversion process minimizes the L2 error between observed and simulated seismic data, leveraging the improved model architecture to achieve better results.
Results
The experimental results indicate that the conditional diffusion model significantly outperforms traditional unconditional models in terms of resolution and structural fidelity. The method exhibits improved stability and robustness when applied to complex geological scenarios, effectively utilizing density information to constrain the inversion process.
Implications
The findings suggest that the conditional diffusion model can be a valuable tool in seismic imaging, potentially leading to more accurate subsurface models. This advancement could enhance the reliability of geological assessments and resource exploration, making it a significant contribution to the field of geophysics.
Learning Response-Statistic Shifts and Parametric Roll Episodes from Wave--Vessel Time Series via LSTM Functional Models
Time Series
- Development of a data-driven surrogate model using LSTM networks for predicting parametric roll in vessels.
- The model is trained on wave-motion time series generated from both experiments and simulations, making it versatile.
- Focus on capturing not just the dynamics of parametric roll but also the statistical shifts in response distributions.
- Evaluation of various loss functions to improve the model's accuracy in tail risk prediction.
Read more
Learning Response-Statistic Shifts and Parametric Roll Episodes from Wave--Vessel Time Series via LSTM Functional Models
Summary
This paper addresses the challenge of predicting parametric roll, a significant instability in maritime vessels that can lead to abrupt changes in ship response and increased risk during extreme sea conditions. The authors propose a data-driven surrogate model that utilizes Long Short-Term Memory (LSTM) networks to learn the complex, nonlinear relationship between incident wave time series and vessel motions. The model is designed to be data-source agnostic, allowing it to be trained on data from controlled experiments or high-fidelity simulations. The training data is generated using a URANS numerical wave tank, simulating various sea states to capture the dynamics of parametric roll. The LSTM surrogate is evaluated based on its ability to reproduce parametric roll episodes and the associated shifts in roll statistics, particularly focusing on tail risk and distributional fidelity. The study also explores different loss functions to optimize the model's performance in terms of tail fidelity, which is crucial for operability and risk assessment in maritime contexts. The paper contributes a curated dataset for severe-sea conditions, enhancing future research in extreme ship response modeling.
Methodology
The authors employ stacked LSTM networks to create a nonlinear functional surrogate that maps incident wave histories to vessel motion histories. The model is trained on data generated from a URANS numerical wave tank, simulating different sea states. The evaluation metrics include time-domain accuracy and distributional fidelity, particularly focusing on the probability density functions (PDFs) of roll responses.
Results
The LSTM model successfully tracks the onset and growth of large-amplitude roll associated with parametric excitation. It captures significant changes in roll probability density functions, demonstrating its capability to reproduce both the dynamics of parametric roll episodes and the statistical shifts in response distributions. The study also highlights the trade-offs between different loss functions in terms of average error and tail fidelity.
Implications
The findings have important implications for maritime safety and operability, particularly in extreme sea conditions. The ability to predict parametric roll and associated risks can inform design and operational decisions for vessels, potentially reducing the likelihood of accidents. The released dataset can serve as a valuable resource for further research in marine hydrodynamics and machine learning applications in this field.
A Direct Classification Approach for Reliable Wind Ramp Event Forecasting under Severe Class Imbalance
Time Series
- Introduces a direct classification approach for forecasting WPREs, addressing severe class imbalance.
- Develops a data preprocessing strategy that enhances feature extraction from power observations.
- Combines majority-class undersampling with ensemble learning to improve model performance.
- Achieves over 85% accuracy and 88% weighted F1 score in numerical simulations.
Read more
A Direct Classification Approach for Reliable Wind Ramp Event Forecasting under Severe Class Imbalance
Summary
This paper addresses the challenge of forecasting Wind Power Ramp Events (WPREs) in low-carbon power systems, particularly under conditions of severe class imbalance where ramp events are infrequent. The authors propose a novel direct classification methodology that treats WPRE forecasting as a multivariate time series classification task. They introduce a data preprocessing strategy that extracts relevant features from recent power observations while masking unavailable ramp information, making it compatible with existing real-time ramp identification tools. The methodology combines majority-class undersampling and ensemble learning techniques to improve forecasting accuracy. Numerical simulations on a real-world dataset demonstrate the effectiveness of this approach, achieving over 85% accuracy and an 88% weighted F1 score, significantly outperforming benchmark classifiers. The findings highlight the importance of tailored classification methods for effective decision support in grid management, especially in the context of increasing reliance on renewable energy sources.
Methodology
The authors propose a direct classification approach for WPRE forecasting, treating it as a multivariate time series classification problem. They implement a data preprocessing strategy that extracts features from recent power observations and masks unavailable ramp information. The methodology employs majority-class undersampling and ensemble learning techniques to mitigate the effects of class imbalance.
Results
The proposed methodology achieved over 85% accuracy and an 88% weighted F1 score in numerical simulations conducted on a real-world dataset, outperforming existing benchmark classifiers.
Implications
The findings suggest that enhanced forecasting tools for WPREs can significantly aid decision-making processes in grid management, particularly as the integration of renewable energy sources increases. This research could lead to improved operational responses and stability in low-carbon power systems.