AI-generated summaries
Today's ML research,
without the noise.
Daily summaries of the latest machine learning papers from arXiv, processed every 8 hours.
48
Papers today
8h
Update frequency
7
Days of history
How Should Transformers Encode Numeric Values in Electronic Health Records?
NLP
Optimization
Time Series
- Introduces a unified evaluation framework for numeric reasoning in EHR transformers.
- Systematic comparison of discrete, continuous, and hybrid numeric value encodings reveals trade-offs in performance.
- Hybrid token-based approaches with binning provide a robust alternative for numeric encoding.
- All evaluated methods can perform approximate numeric computations reliably.
Read more
How Should Transformers Encode Numeric Values in Electronic Health Records?
Summary
This paper investigates the encoding of numeric values in transformer-based models applied to electronic health records (EHR). The authors systematically compare discrete, continuous, and hybrid encoding strategies through a unified evaluation framework that incorporates synthetic arithmetic tasks and real-world clinical prediction tasks. They identify trade-offs among numeric precision, optimization stability, and architectural flexibility. The study finds that approaches modeling value-concept interactions excel in precision-sensitive tasks, while hybrid token-based methods that use binning prior to projection offer a robust alternative. The optimal number of bins is empirically derived based on dataset size. The results indicate that models achieve reliable approximate numeric reasoning rather than exact arithmetic, suggesting that 'good enough' performance may suffice in clinical applications. Additionally, the clinical benefits of incorporating numeric values are modest and task-dependent, emphasizing the importance of robustness and deployability over maximal precision.
Methodology
The authors developed a reusable test suite that combines synthetic arithmetic tasks with real EHR data to evaluate numeric value encodings. They conducted empirical studies comparing different encoding strategies, assessing their performance on both synthetic and real-world clinical prediction tasks.
Results
The study found that approaches explicitly modeling value-concept interactions performed best on precision-sensitive tasks, while hybrid methods with binning were more robust. All methods demonstrated reliable approximate numeric reasoning, with performance degrading smoothly as task complexity increased. The incorporation of numeric values into clinical predictions yielded modest improvements, indicating that robustness is often more critical than precision.
Implications
The findings suggest that hybrid token-based approaches should be considered a practical default for encoding numeric values in EHR applications. The results also highlight the need for further exploration of encoding strategies in different clinical contexts, particularly for short-term predictions where numeric values may have immediate relevance.
Graph Classification via Network Usable Information: From Representation Evaluation to Structure Selection
Graph Learning
- NETINFOGC extends the NUI framework to graph classification, addressing representation-space mismatches.
- The framework combines propagation-based descriptors with classical centrality measures to capture complementary structural information.
- A training-free, clustering-based NUI estimation procedure provides a model-free proxy for representation quality.
- Empirical findings indicate that degree centrality is a strong representation, often correlating closely with classification accuracy.
Read more
Graph Classification via Network Usable Information: From Representation Evaluation to Structure Selection
Summary
This paper introduces NETINFOGC, a novel framework for graph classification that extends the Network Usable Information (NUI) paradigm to graph-level learning. Unlike traditional graph neural networks (GNNs) that rely on end-to-end training of black-box embeddings, NETINFOGC constructs a family of permutation-invariant graph representations using propagation-based mechanisms and classical structural descriptors, such as graph centrality measures. A key innovation is the introduction of a training-free NUI estimation procedure that evaluates representation quality based on clustering consistency with ground-truth labels, providing a model-free proxy for task-relevant information. The framework also employs sparse-group LASSO regularization to automatically select informative structural descriptors while minimizing redundancy. Experimental results on benchmark datasets reveal that classical centrality measures can be highly competitive, and often outperform learned representations. Furthermore, a strong correlation between estimated NUI and downstream classification accuracy is observed, validating NUI as an effective measure of representation utility. Overall, NETINFOGC offers a unified and interpretable approach to evaluating and utilizing graph representations without the need for extensive neural training.
Methodology
The methodology involves constructing permutation-invariant graph representations from propagation-based mechanisms and classical structural descriptors. A two-stage pipeline is employed: first, a training-free clustering-based procedure estimates NUI by measuring alignment with ground-truth labels; second, sparse-group LASSO regularization is used to select informative descriptors while suppressing redundant ones.
Results
The experiments demonstrate that classical centrality measures are competitive with learned representations, with degree centrality often being the strongest performer. The correlation between estimated NUI and classification accuracy supports the utility of NUI as a pre-training screening criterion.
Implications
The findings suggest that NETINFOGC can be applied in various domains requiring graph classification, such as molecular property prediction, malware detection, and network security, offering a more interpretable and efficient approach to representation evaluation.
SABER: A Semantic-Aligned Brain Network Analysis Framework via Multi-scale Hypergraphs
Graph Learning
NLP
Large Language Models
- SABER integrates LLM-derived semantics directly into brain network classification, enhancing decision-making.
- The framework employs multi-scale hypergraphs to capture complex interactions among brain regions.
- A decision-level semantic alignment mechanism allows for patient-specific semantic information to guide predictions.
- SABER outperforms existing methods on brain network datasets, particularly in small-sample settings.
Read more
SABER: A Semantic-Aligned Brain Network Analysis Framework via Multi-scale Hypergraphs
Summary
The paper presents SABER, a novel framework for brain network analysis that integrates high-level semantic knowledge from Large Language Models (LLMs) into the decision-making process for brain disease diagnosis. Traditional methods often treat semantics as auxiliary features, limiting their effectiveness. SABER addresses this by incorporating ROI-level semantics through global self-attention, enriching node representations and providing a whole-brain context. It constructs multi-scale hypergraphs to model functional subnetworks and multi-ROI interactions, overcoming the limitations of traditional Graph Neural Networks (GNNs) in capturing high-order dependencies. A decision-level semantic alignment mechanism is introduced, allowing patient-specific textual embeddings to directly influence predictions without altering the underlying network structure. The framework demonstrates state-of-the-art performance on public datasets, particularly in small-sample scenarios, enhancing stability and interpretability in brain disease classification.
Methodology
The SABER framework consists of three main stages: (1) Multi-scale node-level brain network encoding, where ROI semantics are injected into node features using global self-attention; (2) Construction of multi-scale hypergraphs to model high-order dependencies and multi-ROI interactions; (3) Implementation of a decision-level semantic alignment mechanism that integrates patient-specific textual embeddings into graph representations for direct influence on predictions.
Results
Experiments conducted on the ABIDE and ADHD-200 datasets show that SABER consistently achieves superior performance compared to existing methods, particularly in scenarios with limited sample sizes. The framework also demonstrates improved stability and interpretability in the classification of brain diseases.
Implications
The SABER framework has significant implications for the diagnosis of brain diseases by leveraging high-level semantic knowledge, potentially leading to more accurate and interpretable diagnostic tools in clinical settings. Its ability to integrate complex brain connectivity patterns with semantic information could enhance the understanding of brain disorders and improve patient outcomes.
A Physics-Regulated Neural Framework for Learning 3D Grain Growth Dynamics
Efficient ML
Theory
- 3D-PRIMME effectively models 3D grain growth dynamics using a local evolution rule.
- The model is trained on only two time steps but can predict grain growth over large spatial domains.
- It maintains consistent kinetics and grain topology across significant increases in system size.
- The framework operates with minimal supervision, making it data-efficient.
Read more
A Physics-Regulated Neural Framework for Learning 3D Grain Growth Dynamics
Summary
This paper presents 3D-PRIMME, a novel deep learning framework designed to model the dynamics of 3D grain growth in polycrystalline materials. The framework addresses the challenges of traditional physics-based models, which often struggle with scalability and computational efficiency when applied to large 3D datasets. 3D-PRIMME is trained using only two consecutive time steps, yet it accurately captures the linear coarsening law and preserves topological statistics over extended time scales. The model operates on local patches of the microstructure rather than the entire domain, allowing it to scale effectively to larger systems without retraining. This approach leverages the locality of grain boundary interactions, enabling the model to learn a scale-independent and temporally stable evolution rule. The authors validate the model across multiple datasets, demonstrating its robustness and efficiency in predicting microstructure evolution with minimal supervision and limited training data.
Methodology
The 3D-PRIMME framework learns local evolution rules governing microstructure dynamics by operating on fixed-size local spatial windows instead of the full microstructure. It encodes the local neighborhood configuration into a meaningful representation, allowing for scalable predictions in large 3D domains. The model incorporates physics regularization and is designed to work with limited training data, enhancing its data efficiency.
Results
The model successfully reproduces the linear coarsening law and preserves topological statistics over extended time scales. It is capable of applying learned evolution rules to domains with up to 10243 grid points and 550,000 grains without retraining, demonstrating its scalability and robustness in predicting grain growth dynamics.
Implications
The development of 3D-PRIMME has significant implications for materials science, particularly in understanding and predicting microstructure evolution in polycrystalline materials. Its ability to efficiently model large-scale grain growth dynamics could accelerate research and development in materials engineering, leading to improved mechanical properties and performance of materials.
TREK: Distill to Explore, Reinforce to Refine
Reinforcement Learning
Large Language Models
NLP
- TREK improves exploration support in GRPO by using distillation for exploration rather than imitation.
- The method can utilize both external and internal proposal sources, making it broadly applicable.
- TREK effectively identifies hard prompts and integrates verified solutions to enhance the student's sampling capabilities.
- Significant performance improvements were observed in mathematical reasoning and agentic tasks.
Read more
TREK: Distill to Explore, Reinforce to Refine
Summary
The paper introduces TREK (Teacher-Routed Exploration via Forward KL), a novel approach to enhance Group Relative Policy Optimization (GRPO) by addressing its limitations in exploring hard prompts. GRPO is effective when the current policy can sample useful reasoning trajectories but struggles with prompts that lie outside its on-policy support. TREK utilizes a staged procedure that employs distillation not for imitation but to expand exploration support. It identifies prompts where the student model has low pass rates, queries a proposal source for verified solutions, and applies a forward-KL phase to integrate these solutions into the student's support. This method is versatile, allowing the use of external or internal teachers, and efficiently identifies valuable samples for consolidation. The results demonstrate significant improvements in mathematical reasoning and agentic tasks, showcasing TREK's ability to enhance exploration and learning efficiency.
Methodology
TREK identifies hard prompts based on the student's low pass rates, queries a proposal source for verified solutions, ranks these proposals by their likelihood under the current student model, and applies a forward-KL learning phase to increase the probability mass on these new modes. After this exploration phase, it returns to standard GRPO for reinforcement learning.
Results
TREK with DeepSeek-V4 proposals improved Qwen3 models on AIME 2024 and AIME 2025, with accuracy increases from 36.9 to 40.3 and from 47.9 to 51.1, respectively. The self-context variant also showed improvements. In agentic tasks, TREK raised the success rate in ALFWorld from 75.8 to 82.8 and in ScienceWorld from 12.5 to 26.7, demonstrating early success in training on challenging tasks.
Implications
The findings suggest that enhancing exploration in reinforcement learning can lead to more effective learning strategies, particularly in complex reasoning tasks. TREK's approach could be applied in various domains requiring robust exploration mechanisms, potentially improving performance in AI systems that rely on reasoning and decision-making.
Integrating Physics-Informed Neural Networks for Safe Reinforcement Learning in a 1-DoF Helicopter System
Reinforcement Learning
Robotics
Optimization
- Integration of a differentiable physics model into the PPO algorithm for safe reinforcement learning.
- Simulated future trajectories are used to penalize anticipated safety violations during training.
- Demonstrated significant reduction in constraint violations while maintaining reliable target tracking.
- Evaluation conducted on a simulated 1-DoF helicopter system with strict pitch constraints.
Read more
Integrating Physics-Informed Neural Networks for Safe Reinforcement Learning in a 1-DoF Helicopter System
Summary
This paper addresses the challenge of ensuring safety in deep reinforcement learning (DRL) applications for industrial cyber-physical systems (ICPSs), specifically focusing on a 1-degree-of-freedom (1-DoF) helicopter system. The authors propose a novel approach that integrates a differentiable physics model into the proximal policy optimization (PPO) algorithm's actor loss function. By simulating short-horizon future trajectories during training, the policy is penalized for anticipated safety violations, independent of the task-reward signal. This method aims to enhance safety without the need for complex reward shaping or additional computational overhead. The approach is evaluated on a simulated Quanser Aero 2 system, demonstrating that the physics-informed soft regularizations significantly reduce constraint violations while maintaining effective target tracking. The results indicate a trade-off between safety and performance, highlighting the potential for this method to provide a scalable solution for safe reinforcement learning deployment in real-world applications.
Methodology
The authors utilize the proximal policy optimization (PPO) algorithm, embedding a physics-informed neural network (PINN) into the loss function. They model the system dynamics using differential equations and employ a 4th order Runge-Kutta (RK4) integration method for trajectory simulation. The safety penalty is derived from predicted future violations of a defined pitch limit, which is integrated into the actor's loss function to guide policy updates.
Results
The experimental results reveal that all configurations converged successfully, with the Naive Baseline achieving high task rewards but violating safety constraints. The PINN Over-penalized model exhibited minimal safety violations but resulted in sluggish tracking performance. The study highlights a clear trade-off between target tracking and constraint satisfaction, emphasizing the importance of tuning the penalty weight for optimal performance.
Implications
This work has significant implications for the deployment of reinforcement learning in safety-critical applications, such as robotics and industrial automation. By providing a method to ensure safety without compromising performance, it opens avenues for more reliable and robust control systems in real-world scenarios.
HiFA4: Training-Free 4-bit FlashAttention on Ascend HIF4 NPUs for LLM Inference
NLP
Large Language Models
Efficient ML
- HiFA4 is the first design for 4-bit FlashAttention evaluated on standard NLP benchmarks.
- Introduces Smooth-QK and P-Reordering to improve quantization accuracy and efficiency.
- Achieves a 37.5% recovery of accuracy gap on Qwen3-8B compared to direct HIF4 quantization.
- Reduces the fraction of inconsistent predictions and accuracy regressions significantly.
Read more
HiFA4: Training-Free 4-bit FlashAttention on Ascend HIF4 NPUs for LLM Inference
Summary
The paper introduces HiFA4, a novel post-training operator-level design that implements 4-bit FlashAttention for large language model (LLM) inference on Ascend HIF4 NPUs. HiFA4 executes the QKT and PV matrix multiplications as 4-bit HIF4 Cube GEMMs while maintaining the online softmax state in FP16. This approach is the first of its kind evaluated on standard NLP benchmarks. The authors propose two key mechanisms: Smooth-QK, which applies a calibration-static rescaling to Q and K to alleviate quantization challenges, and P-Reordering, which ensures that the softmax normalizer is computed from the same quantized weights used in the PV GEMM. The paper provides a theoretical proof of the systematic output-scaling error introduced by conventional methods and validates this through empirical analysis. HiFA4 demonstrates significant improvements in accuracy recovery, reducing quantization-induced decision drift across multiple LLMs, and shows potential for latency reduction by fusing operations. The findings indicate that HiFA4 can effectively bridge the accuracy gap caused by direct quantization while enhancing computational efficiency.
Methodology
The methodology involves two main components: Smooth-QK, which calibrates and rescales Q and K to manage quantization errors, and P-Reordering, which eliminates output errors by ensuring consistent computation of the softmax normalizer. The design leverages 4-bit Cube GEMMs for efficient matrix multiplications while maintaining the softmax state in FP16, facilitating high-throughput LLM inference.
Results
HiFA4 was evaluated on five LLMs, showing a 37.5% recovery of accuracy on Qwen3-8B, a reduction in BF16-inconsistent predictions from 16.3% to 8.2%, and a 57% decrease in accuracy regressions. On Gemma2-9B, it maintained accuracy within 0.7 percentage points of BF16 while reducing regressions by 27%. The remaining components of HiFA4 also demonstrated significant improvements in accuracy regressions across various models.
Implications
The advancements presented in HiFA4 could lead to more efficient and accurate inference for large language models, particularly in resource-constrained environments. The techniques developed may also be applicable to other quantization challenges in machine learning, enhancing the deployment of models on specialized hardware.
Conditional Inference Trees and Forests for Feature Selection
Theory
Efficient ML
Interpretability
- CIT and CIF effectively reduce split-selection bias in feature selection.
- CIF ranks 4th among 17 classification methods and 3rd among 18 regression methods in benchmark tests.
- Adaptive stopping and threshold search significantly affect CIF runtime, with potential increases in fitting time by up to 10.8x.
- The study identifies scenarios where forest feature sampling may exclude informative features.
Read more
Conditional Inference Trees and Forests for Feature Selection
Summary
This paper investigates Conditional Inference Trees (CIT) and Conditional Inference Forests (CIF) as methods for feature selection, aiming to mitigate split-selection bias inherent in traditional tree-based models. The authors propose a benchmark that evaluates the performance of CIT and CIF in ranking features for downstream prediction tasks across various datasets. The methodology separates feature selection from threshold optimization, allowing for more accurate feature ranking while controlling for computational costs. The study employs real-data benchmarks, runtime ablations, and synthetic experiments to assess the effectiveness of these methods. Key findings indicate that CIF ranks competitively among other classification and regression methods, while runtime analyses reveal that certain hyperparameters significantly impact computational efficiency. The paper also highlights potential limitations in high-dimensional settings where informative features may be overlooked due to forest feature sampling. Overall, the results support the use of CIF as a viable top-k feature-ranking method in predictive modeling.
Methodology
The authors utilize a conditional inference framework that separates feature selection from threshold optimization. They conduct empirical evaluations using real-data benchmarks and synthetic simulations, comparing CIT and CIF against other feature selection methods. The study also includes runtime ablation experiments to analyze the impact of hyperparameters on computational efficiency.
Results
CIF demonstrated strong performance in feature ranking, achieving high ranks in both classification and regression tasks across multiple datasets. The runtime ablation studies revealed that disabling adaptive stopping and using exhaustive threshold searches significantly increased fitting times without substantial improvements in ranking quality. Additionally, the analysis of high-dimensional data indicated that some informative features may be neglected due to the sampling strategy employed in forests.
Implications
The findings suggest that CIF can be effectively employed for feature selection in various predictive modeling scenarios, particularly in situations where reducing computational costs is crucial. The insights into runtime efficiency and feature sampling may guide future research and practical applications in machine learning, especially in high-dimensional contexts.
FedACT: Federated Adaptive Coordinate Trust Modulation for Robust Transformer Training under Data Heterogeneity
Federated Learning
Optimization
Large Language Models
- Identification of coordinate trust mismatch as a critical issue in federated AdamW training.
- Introduction of FedACT, which reallocates update magnitudes based on coordinate-wise trust scores.
- Demonstrated improvements in communication-round efficiency and training stability.
- Extensive empirical validation across multiple model types, showing significant performance gains.
Read more
FedACT: Federated Adaptive Coordinate Trust Modulation for Robust Transformer Training under Data Heterogeneity
Summary
The paper introduces FedACT, a novel method for federated learning that addresses the issue of coordinate trust mismatch in adaptive optimization, particularly in the context of training Transformer models. Traditional federated adaptive optimizers often apply uniform updates across coordinates, which can lead to inefficiencies, especially when client data is heterogeneous. FedACT enhances the federated AdamW optimizer by first establishing a globally corrected adaptive direction and then reallocating update magnitudes based on a coordinate-wise trust score. This approach allows for larger updates on coordinates that are well-supported by both local gradients and global corrections, while still applying smaller updates to less reliable coordinates. The authors conducted extensive experiments across various models, including vision Transformers and large language models (LLMs), demonstrating that FedACT consistently outperforms existing federated adaptive baselines, particularly under conditions of strong data heterogeneity. The findings suggest that incorporating coordinate-level trust allocation can significantly improve training stability and performance in federated settings.
Methodology
FedACT employs a two-step process: first, it forms a globally corrected adaptive direction for the AdamW optimizer. Then, it reallocates update magnitudes across coordinates based on a trust score that reflects the reliability of each coordinate, considering both local gradients and global corrections. This method contrasts with traditional approaches that apply uniform updates across all coordinates.
Results
The experiments showed that FedACT consistently improved performance over strong federated adaptive baselines, particularly for Transformer models under heterogeneous data conditions. The results highlighted enhanced communication efficiency and training stability, indicating that the coordinate-level trust allocation effectively complements existing correction methods.
Implications
FedACT's approach could lead to more robust federated learning systems, particularly in applications where data is non-IID and heterogeneous, such as in healthcare, finance, and personalized AI systems. The findings may also influence future research in adaptive optimization techniques in federated learning.
What Does a Discrete Diffusion Model Learn?
Generative Models
Theory
Optimization
- The negative ELBO is exactly equal to data entropy plus the path KL divergence, not just a bound.
- The learned reverse process can be represented in three different ways: denoiser, cavity, and score.
- Different noising processes share the same best achievable negative ELBO, which is the data entropy.
- The paper resolves several puzzles in the literature regarding the optimization objectives of various diffusion models.
Read more
What Does a Discrete Diffusion Model Learn?
Summary
This paper investigates the learning dynamics of discrete diffusion models, focusing on what these models actually optimize during training. The authors derive a rigorous framework based on continuous-time Markov chains (CTMC) and establish the Oracle Distance theorem, which states that the negative evidence lower bound (ELBO) is equal to the data entropy plus the Kullback-Leibler divergence from the oracle reverse process to the learned process. This finding clarifies that the optimization process involves matching the entire reverse trajectory of the model to that of the oracle, rather than merely focusing on endpoint distributions. The paper also identifies three distinct representations of the learned reverse process—denoiser, cavity (bridge plug-in), and score—demonstrating that these can be converted among each other. The authors provide insights into why different parameterizations lead to different learning behaviors, particularly in uniform and masked diffusion scenarios. The theoretical findings are numerically verified on an exactly solvable model, confirming the identities derived in the paper.
Methodology
The authors employ a rigorous mathematical approach to derive the continuous-time Markov chain formulation of discrete diffusion models. They establish the Oracle Distance theorem and analyze the implications of different parameterizations on the learning process. Theoretical results are supported by numerical experiments on a solvable model.
Results
The paper proves that the negative ELBO corresponds to the data entropy and the KL divergence from the oracle process, establishing a clear relationship between different representations of the reverse process. It also shows that the same reverse rate can be expressed in multiple valid forms, providing a comprehensive understanding of the optimization landscape in discrete diffusion models.
Implications
The findings have significant implications for the design and training of generative models, particularly in understanding how different parameterizations affect learning outcomes. This could lead to improved methodologies for training discrete diffusion models in various applications, including language modeling and other categorical data generation tasks.
A Near-Linear-Time Solver for Graph $p$-Laplacian Semi-Supervised Learning via Continuation in $p$
Graph Learning
Efficient ML
Theory
- Introduces a near-linear-time solver for graph p-Laplacian semi-supervised learning.
- Addresses the degeneracy problem in traditional SSL methods when labeled data is limited.
- Utilizes damped chord-Newton continuation in p for efficient solving of nonlinear systems.
- Empirical results show significant speed and accuracy improvements over existing solvers.
Read more
A Near-Linear-Time Solver for Graph $p$-Laplacian Semi-Supervised Learning via Continuation in $p$
Summary
This paper addresses the challenges of graph-based semi-supervised learning (SSL) by proposing a near-linear-time solver for the graph p-Laplacian energy minimization. Traditional SSL methods, which minimize a quadratic Dirichlet energy, struggle when labeled data is scarce, leading to degenerate solutions. The author introduces a nonlinear p-Laplacian energy formulation that overcomes this issue by penalizing large gradients, thus ensuring well-posedness when p exceeds the data dimension d. The proposed method employs damped chord-Newton continuation in p, allowing for efficient linearized system solving via a near-linear Laplacian solver. Empirical results demonstrate that this approach significantly outperforms existing solvers, achieving linear wall-clock time on large graph families and handling extensive datasets, such as a 6.8 × 10^7-edge social network, in a matter of minutes. The solver shows improved accuracy on benchmarks like MNIST, particularly with p = 3, which outperforms the traditional quadratic case. This work effectively bridges the gap between nonlinear SSL methods and scalable computational techniques, making graph p-Laplacian learning feasible at web scale.
Methodology
The paper recasts the p-Laplacian SSL problem as a source-form nonlinear Laplacian flow and solves it using damped chord-Newton continuation in p. This approach ensures that each linearized system remains well-conditioned and can be efficiently handled by a near-linear Laplacian solver, such as approximate Cholesky or LAMG+.
Results
The proposed solver achieves near-linear empirical performance on various graph sizes, with a wall-clock time of m^1.19 compared to m^1.45 for direct factorization. It successfully processes large graphs, outperforming existing methods by 1.5–14 times in speed at matched accuracy. On the MNIST benchmark, the method with p = 3 achieves 64% accuracy with one label per class, significantly higher than the 36% achieved with p = 2.
Implications
This work has significant implications for the field of semi-supervised learning, particularly in scenarios where labeled data is scarce. The proposed solver enables the application of graph-based SSL methods to large-scale datasets, enhancing the ability to propagate labels effectively and improving model performance in various domains.
OmniFocus: Query-Guided Modality-Balanced Token Compression for Omni-Modal Large Language Models
Multimodal
Large Language Models
Efficient ML
- OmniFocus mitigates modality bias in token compression by using query-guided importance estimation for both audio and video.
- The method preserves inter-modal association and intra-modal peak evidence, enhancing the quality of compressed outputs.
- Experiments show that OmniFocus achieves high accuracy and efficiency, outperforming existing compression techniques.
- The approach is training-free, making it accessible for practical applications in resource-constrained environments.
Read more
OmniFocus: Query-Guided Modality-Balanced Token Compression for Omni-Modal Large Language Models
Summary
The paper introduces OmniFocus, a novel query-guided token compression method designed for Omni-modal large language models (OmniLLMs) that process audio and video inputs. Traditional token compression techniques often rely on unimodal guidance, which can lead to modality bias and overlook the temporal locality of relevant evidence. OmniFocus addresses these limitations by performing independent importance estimation for both audio and video modalities, allowing for a balanced compression strategy that preserves salient evidence from each modality while maintaining their alignment. The method computes query-token similarity to assess the importance of temporal chunks, resulting in a symmetric compression design. Experiments conducted on the Qwen2.5-Omni model family across four audio-visual benchmarks demonstrate that OmniFocus achieves superior performance at low token retention ratios, outperforming existing methods and providing a favorable trade-off between accuracy and efficiency.
Methodology
OmniFocus employs a query-guided approach to token compression, assessing the importance of audio and video chunks independently through query-token similarity. It calculates modality-specific importance scores, converts these into local drop ratios, and selects retained tokens based on inter-modal and intra-modal associations. This allows for a balanced compression that addresses the limitations of unimodal-guided methods.
Results
In experiments with the Qwen2.5-Omni model family, OmniFocus achieved a 59.40 accuracy on the DailyOmni benchmark at 25% token retention, along with a 1.38× speedup in prefill time compared to the full-token baseline. The method consistently reached best or tied-best results across multiple benchmarks, demonstrating its effectiveness in maintaining performance while reducing computational costs.
Implications
OmniFocus has significant implications for the deployment of OmniLLMs in real-world applications, particularly in scenarios requiring efficient processing of long-form audio-visual data. Its ability to maintain high accuracy with reduced token usage can enhance the feasibility of using OmniLLMs in resource-constrained environments, such as mobile devices or real-time systems.
Predicting Early Stages Of Alzheimer's Disease And Identifying Key Biomarkers Using Deep Artificial Neural Network And Ensemble Of Machine Learning Methodologies
Theory
- Developed an automated diagnostic system for early-stage Alzheimer's Disease.
- Addressed data challenges such as missing values and class imbalance.
- Utilized advanced feature selection techniques to improve model accuracy.
- Implemented both ensemble and deep learning models for comparative analysis.
Read more
Predicting Early Stages Of Alzheimer's Disease And Identifying Key Biomarkers Using Deep Artificial Neural Network And Ensemble Of Machine Learning Methodologies
Summary
This thesis presents a comprehensive study aimed at developing an automated diagnostic system for the early stages of Alzheimer's Disease (AD) using advanced machine learning techniques. The research highlights the unpredictability of AD symptoms and the challenges in timely diagnosis, which often leads to misdiagnosis and delayed treatment. The study utilizes a dataset from the Alzheimer’s Disease Neuroimaging Initiative, addressing issues such as missing values and class imbalance through iterative imputation and the borderline SVM-SMOTE algorithm. Feature selection is performed using both wrapper-based and embedded techniques to enhance model accuracy. The proposed methodology includes a stacking-based ensemble model comprising Logistic Regression, Extra Tree, Bagging KNN, and LightGBM classifiers, alongside a deep learning model based on Artificial Neural Networks (ANN). A comparative analysis of these models is conducted using performance metrics such as precision, recall, F1-Score, and AUC-ROC to identify the most effective classifier and key biomarkers for early AD diagnosis. The findings aim to assist clinicians in understanding significant biomarkers associated with AD and improving early detection strategies.
Methodology
The study employed a dataset from the Alzheimer’s Disease Neuroimaging Initiative, addressing missing values through iterative imputation and class imbalance using the borderline SVM-SMOTE algorithm. Feature selection was performed using wrapper-based and embedded techniques. A stacking-based ensemble model was created with various classifiers, and a deep learning model (ANN) was also implemented for comparison.
Results
The comparative analysis revealed the best-performing classifier based on precision, recall, F1-Score, and AUC-ROC metrics, along with the identification of significant biomarkers for early diagnosis of Alzheimer's Disease.
Implications
The findings of this research could significantly enhance early diagnosis and intervention strategies for Alzheimer's Disease, potentially improving patient outcomes and reducing the economic burden associated with late-stage dementia care.
SHiPPO: Recurrent Memory with Transported Polynomial Projections
Theory
Time Series
NLP
- SHiPPO enhances memory semantics in recurrent models through transported polynomial projections.
- The framework allows for dynamic channel interactions, moving beyond fixed coordinate systems.
- Empirical results show that SHiPPO can recover order-sensitive changes in memory that traditional methods cannot.
- The methodology includes a restricted realization that maintains efficient updates and decoding.
Read more
SHiPPO: Recurrent Memory with Transported Polynomial Projections
Summary
The paper introduces SHiPPO (Sylvester HiPPO), a novel framework that enhances recurrent memory semantics by utilizing transported polynomial projections. Unlike traditional HiPPO, which operates in fixed channel coordinates, SHiPPO allows for a moving channel frame, enabling more dynamic interactions between memory and input. The authors present a method for transporting both the approximation family and channel metric along a specified right-transport path, resulting in a coefficient matrix that adheres to Sylvester dynamics. This approach preserves the left online-memory operator while incorporating right-action transport, facilitating selective state-space model (SSM) execution. The paper also derives a restricted group-local realization compatible with controller actions, allowing for efficient updates and recurrent decoding. The authors demonstrate that larger write ranks improve prediction accuracy but cannot recover order-sensitive memory changes. In contrast, SHiPPO's transported-memory variants successfully capture these signals. The empirical results support SHiPPO as a grounded memory prior, emphasizing its mechanistic foundations over mere sequence-modeling performance.
Methodology
The authors develop SHiPPO by defining a projection problem that jointly transports the channel metric and approximation family along a right-transport path. They derive Sylvester dynamics for the coefficient matrix and establish a restricted realization that ensures compatibility with controller actions, allowing for efficient updates and recurrent decoding.
Results
The results indicate that SHiPPO outperforms traditional HiPPO in capturing order-sensitive memory changes and improves prediction accuracy with larger write ranks. The framework's ability to transport memory dynamics leads to better performance in selective state-space model execution.
Implications
SHiPPO's approach to recurrent memory could significantly impact the design of state-space models, particularly in applications requiring dynamic memory interactions, such as natural language processing and time series analysis. Its mechanistic grounding may also inspire further research into memory mechanisms in machine learning.
Robustness Meets Uncertainty: Evidential Adversarial Training for Robust Selective Classification
Computer Vision
Theory
- Introduces a standardized benchmark for evaluating robustness and uncertainty in selective classification.
- Proposes Evidential Adversarial Training (EV-AT) to improve both robustness and uncertainty quality.
- Demonstrates that existing adversarial training methods often degrade uncertainty ranking despite improving robustness.
- Shows through experiments that EV-AT enhances adversarial robustness while maintaining reliable uncertainty estimates.
Read more
Robustness Meets Uncertainty: Evidential Adversarial Training for Robust Selective Classification
Summary
This paper addresses the critical need for classifiers in safety-critical applications to be both robust against adversarial attacks and reliable in their uncertainty estimates. The authors identify a gap in existing research regarding the interplay between adversarial robustness and predictive uncertainty, particularly in the context of selective classification. They introduce a unified benchmark that standardizes various factors such as architectures, augmentations, threat models, and evaluation metrics to systematically analyze the robustness-uncertainty trade-off. Through extensive experiments, the authors reveal that many state-of-the-art adversarial training methods improve robustness but degrade uncertainty ranking, leading to poorer selective behavior. To tackle this issue, they propose Evidential Adversarial Training (EV-AT), which utilizes a Dirichlet distribution to model uncertainty and incorporates an evidence-based loss to enhance both clean accuracy and reliable uncertainty. The results demonstrate that EV-AT shifts the Pareto frontier of robustness-uncertainty trade-offs, outperforming existing adversarial training methods and providing a more reliable framework for selective classification under adversarial conditions.
Methodology
The authors developed a unified benchmark to evaluate the robustness-uncertainty trade-off, standardizing various experimental factors. They proposed EV-AT, which models predictions using Dirichlet distributions and employs an evidence-based loss to improve both accuracy and uncertainty. The methodology includes extensive experiments across different datasets and threat models to validate the effectiveness of EV-AT compared to existing adversarial training methods.
Results
The experiments showed that EV-AT significantly improves adversarial robustness while enhancing the quality of uncertainty estimates compared to state-of-the-art adversarial training methods. The results indicate a favorable shift in the robustness-uncertainty trade-off, allowing for better selective classification.
Implications
The findings suggest that integrating uncertainty modeling into adversarial training can lead to more reliable classifiers in safety-critical applications. This approach can enhance decision-making processes where understanding model confidence is crucial, such as in autonomous systems, medical diagnosis, and other risk-sensitive domains.
Probabilistic Low-Voltage Peak Load Forecasting with Time Series Foundation Models Evaluated on Application-Oriented Metrics
Time Series
- Evaluation of TSFMs for low-voltage load forecasting shows superior performance, especially for Chronos-2.
- Ablation study indicates TSFMs can handle increased uncertainty without weather covariates.
- Introduction of a novel application-oriented metric for assessing forecasting capabilities in grid asset planning.
- The study utilizes a real-world dataset, ensuring relevance and applicability of results.
Read more
Probabilistic Low-Voltage Peak Load Forecasting with Time Series Foundation Models Evaluated on Application-Oriented Metrics
Summary
This paper addresses the challenges of low-voltage load forecasting in the context of increasing electrification and decentralized generation. Current forecasting methods often require significant manual effort and lack uncertainty estimation and proper peak prediction. The authors evaluate the performance of three time series foundation models (TSFMs) - Chronos-Bolt, Chronos-2, and TabPFN-TS - against six baseline models on a dataset of 200 real-world low-voltage feeders. The study highlights the superior performance of Chronos-2, particularly in peak load forecasting. An ablation study reveals that while weather covariates are important, TSFMs can adapt to increased uncertainty when they are omitted. The authors introduce a novel application-oriented metric that links forecasting capabilities to grid asset planning, emphasizing the trade-off between cost reduction and minimizing failure risk. This work provides valuable insights for distribution system operators (DSOs) and researchers, demonstrating the effectiveness of TSFMs in low-voltage load forecasting and the importance of probabilistic forecasts.
Methodology
The authors conducted an extensive evaluation of TSFMs on low-voltage load forecasting using a dataset of 200 feeders. They compared the performance of three TSFMs against six baseline models, including tree-based and neural network approaches. An ablation study was performed to assess the impact of weather covariates on forecasting accuracy. A novel application-oriented metric was developed to evaluate the forecasts in relation to grid asset management.
Results
The results demonstrated that Chronos-2 outperformed other models in forecasting accuracy, particularly in peak load predictions. The ablation study confirmed that while weather information is beneficial, TSFMs can still provide reliable forecasts without it. The proposed application-oriented metric effectively linked forecasting performance to practical grid management needs, providing a percentage KPI for DSOs.
Implications
The findings suggest that TSFMs can significantly enhance low-voltage load forecasting, aiding DSOs in managing grid capacity and planning for future demands. The application-oriented metric offers a practical tool for evaluating forecasting models in real-world scenarios, potentially leading to more efficient grid operations and reduced risk of overload.
Wind-Aware Reinforcement Learning Control of a Small Quadrotor Using Learned Onboard Wind Estimation in Simulated Atmospheric Turbulence
Reinforcement Learning
Robotics
- Introduces a two-stage learning pipeline for wind estimation and control in small quadrotors.
- Achieves high accuracy in wind estimation with a GRU network, even in unseen conditions.
- Demonstrates a significant reduction in tracking error using a PPO controller informed by wind estimates.
- Highlights the importance of wind perception, particularly in varying wind speeds.
Read more
Wind-Aware Reinforcement Learning Control of a Small Quadrotor Using Learned Onboard Wind Estimation in Simulated Atmospheric Turbulence
Summary
This paper addresses the challenges faced by small multirotor aircraft operating in turbulent winds, which can significantly affect trajectory tracking and control. The authors propose a two-stage learning pipeline that first estimates local wind conditions using onboard kinematics and dynamics, followed by a reinforcement learning (RL) flight controller that utilizes this wind estimate. The wind estimator employs an attention-augmented gated recurrent network (GRU) trained on thousands of simulated flights through von Kármán turbulence, achieving a root-mean-square error (RMSE) of 0.40 m/s and a direction error of 3.2° in unseen wind conditions. The proximal policy optimization (PPO) controller, which integrates the wind estimator's output, demonstrates a 48% reduction in horizontal trajectory tracking error compared to a traditional wind-blind proportional-derivative (PD) controller. The paper also explores the contribution of wind perception to overall performance, revealing that its impact increases with wind speed. The learned controller shows resilience in out-of-distribution wind conditions, where the baseline fails. These findings suggest that integrating learned wind perception can significantly enhance the autonomy of small uncrewed aircraft systems (sUAS) in challenging wind environments.
Methodology
The methodology involves a two-stage architecture where the first stage uses an attention-augmented gated recurrent network (GRU) to estimate the horizontal wind vector from onboard kinematics and dynamics. The second stage employs a proximal policy optimization (PPO) controller that utilizes the wind estimates to improve trajectory tracking performance. The system is evaluated in a simulated environment with von Kármán turbulence, and various ablation studies are conducted to analyze the contributions of different components.
Results
The wind estimator achieved a per-flight RMSE of 0.40 m/s and a direction error of 3.2° on unseen wind regimes. The PPO controller reduced horizontal trajectory tracking error by 48% compared to a wind-blind PD baseline, outperforming it in all evaluation episodes. The analysis revealed that the contribution of wind perception to performance improvement increases with wind speed, with a skill score of 0.861 over a constant-wind reference. The controller also demonstrated resilience to out-of-distribution winds, where the baseline failed.
Implications
The findings suggest that integrating learned wind perception into control systems can significantly enhance the performance and autonomy of small uncrewed aircraft systems (sUAS) in turbulent environments, potentially expanding their operational capabilities in various applications such as search and rescue, environmental monitoring, and agricultural surveying.
Towards Learning Representations of Policies in Two-Player Zero-Sum Imperfect-Information Games
Reinforcement Learning
Theory
- Introduction of methods for generating datasets of policies in two-player zero-sum games.
- Development of various techniques for learning compact policy representations.
- Creation of downstream tasks for evaluating the effectiveness of learned policy embeddings.
- Demonstration of the presence of useful behavioral representations in learned embeddings.
Read more
Towards Learning Representations of Policies in Two-Player Zero-Sum Imperfect-Information Games
Summary
This paper addresses the challenge of learning effective policy representations in two-player zero-sum imperfect-information games. The authors present three main contributions: the development of methods for creating datasets of policies, the proposal of techniques for learning policy representations, and the introduction of downstream tasks to evaluate the effectiveness of these representations. The evaluation is conducted using Kuhn and Leduc Poker, demonstrating that even basic methods can yield useful behavioral representations in the learned embeddings. This work is notable for being one of the first to systematically compare self-supervised learning techniques for policy representation in games, providing a foundation for future research in this area.
Methodology
The authors propose three methods for creating datasets of policies: random initialization of policy neural networks, the Policy Space Response Oracle (PSRO) algorithm, and a variant of neural population learning (NeuPL). For learning policy representations, they introduce several methods, including a weight autoencoder and a functional encoder that evaluates the behavior of networks rather than their weights. The evaluation is performed on Kuhn and Leduc Poker using the proposed downstream tasks.
Results
The evaluation of the proposed methods on Kuhn and Leduc Poker indicates that the learned embeddings contain useful behavioral representations, despite the simplicity of the methods employed. The systematic comparison of self-supervised learning techniques reveals insights into the effectiveness of different approaches for learning policy representations.
Implications
This research has significant implications for the development of more sophisticated agents in imperfect-information games, enhancing their ability to reason about opponents' policies and improve decision-making. The findings could inform future work in game-theoretic learning and self-supervised learning techniques.
How to Allocate Your Tokens? Scaling Laws with Training Steps and Batch Size
Large Language Models
Optimization
Theory
- Introduction of a three-term scaling law that considers model size, training steps, and batch size.
- The proposed law can be fitted using fewer training runs, reducing the required data to 28%.
- It provides a framework for understanding both optimal and suboptimal batch sizes in training.
- The law aligns with previous empirical findings on critical batch sizes and extends to practical constraints.
Read more
How to Allocate Your Tokens? Scaling Laws with Training Steps and Batch Size
Summary
In this paper, Fabian Schaipp introduces a novel scaling law that integrates model size and training data while explicitly considering the allocation of training steps and batch size, referred to as the three-term law. This law is derived from a comprehensive analysis of training runs and is shown to accurately predict the optimal batch size while requiring fewer training runs for fitting. The proposed law not only aligns with existing empirical findings regarding critical batch sizes but also extends to suboptimal batch sizes, making it applicable in practical scenarios where optimal configurations may not be feasible. The paper emphasizes the importance of hyperparameter scaling laws in deep learning, particularly for large language models (LLMs), and connects empirical observations to theoretical frameworks in optimization. The findings suggest that the proposed scaling law can significantly reduce the number of training runs needed to identify optimal configurations, thereby enhancing the efficiency of training large models.
Methodology
The author proposes a power-law model for loss as a function of model size (N), batch size (M), and training steps (K). The model is fitted using data from training runs of large language models, allowing for the derivation of optimal batch sizes and scaling laws for suboptimal configurations. The methodology emphasizes the use of fewer training runs by leveraging data from suboptimal batch sizes.
Results
The proposed scaling law successfully predicts optimal batch sizes consistent with prior research, demonstrating that only two batch sizes per (N, D) are necessary for robust fitting. This results in a significant reduction in training runs needed. The law also provides non-trivial optimal batch sizes that are independent of model size and accurately describes the scaling of critical batch sizes.
Implications
The findings have significant implications for the training of large language models, particularly in optimizing resource allocation and improving training efficiency. The ability to derive scaling laws for suboptimal batch sizes can aid practitioners in scenarios where hardware constraints limit the use of optimal configurations.
Grokking Is Conditional and Fragile: A Fully-Tractable, Multi-Seed Study at 12K Parameters
Theory
- Grokking is conditional on training-set coverage and output cardinality.
- Weight decay reproduces the inverted-U relationship in grok-rate.
- Grokking is sensitive to floating-point environment perturbations.
- Task decomposition improves data efficiency and generalization.
Read more
Grokking Is Conditional and Fragile: A Fully-Tractable, Multi-Seed Study at 12K Parameters
Summary
This paper investigates the phenomenon of grokking, where a neural network exhibits delayed generalization after fitting its training data. Unlike previous studies that often relied on larger models and single training runs, this research utilizes a smaller, fully tractable transformer model (Glimmer-1-Base) with approximately 11,856 parameters. The study emphasizes the importance of multi-seed analysis to understand grokking as a conditional and fragile phase transition. Six key findings are presented: (1) The grokking transition is influenced by training-set coverage, primarily tracking output cardinality rather than task composition. (2) Weight decay leads to an inverted-U relationship in grok-rate, confirming the validity of the measurement. (3) Grokking is sensitive to perturbations in the floating-point environment, with minor changes affecting a small number of seeds without altering the overall rate. (4) Mechanistic insights reveal that generalizing solutions exhibit a more periodic output map, while the model fails to form a textbook Fourier embedding. (5) Task decomposition enhances data efficiency, allowing a two-specialist pipeline to succeed where a monolithic model fails. (6) The study underscores the necessity of multi-seed control to avoid misleading narratives in grokking research.
Methodology
The study employs a small transformer model (Glimmer-1-Base) and conducts multi-seed experiments to measure grokking rates under controlled numerical environments. The analysis includes a comprehensive examination of the model's weights, attention patterns, and output mappings, allowing for a detailed mechanistic understanding.
Results
The findings indicate that grokking is influenced by the coverage of the training set, with a clear relationship to output cardinality. The inverted-U relationship in grok-rate due to weight decay was confirmed, and the model's sensitivity to environmental perturbations was documented. Task decomposition was shown to significantly enhance performance in scenarios where monolithic models failed.
Implications
The results suggest that understanding grokking requires careful consideration of training conditions and model architecture. The findings may inform future research on model generalization and the design of more efficient learning algorithms, particularly in small-scale settings.
X-LogSMask: Expand Transformer for Graph-Structured Data
Graph Learning
- X-LogSMask introduces a logarithmic structural mask for graph data, enhancing the interpretability of attention mechanisms.
- The method allows for multi-hop information propagation within a single Transformer layer, reducing the need for multiple message-passing layers.
- Transformers with X-LogSMask achieve state-of-the-art performance on 13 datasets across various benchmarks.
- The approach maintains the core Transformer architecture while effectively adapting it for graph-structured data.
Read more
X-LogSMask: Expand Transformer for Graph-Structured Data
Summary
The paper introduces X-LogSMask, a novel approach to adapt Transformers for graph-structured data, addressing the limitations of traditional self-attention mechanisms in handling sparse and structured interactions typical of graphs. The authors propose a multi-head logarithmic structural mask that integrates symmetrically normalized graph topology into attention logits, effectively transforming structural connectivity into a topology-aware gating signal. This method suppresses unsupported node interactions while maintaining feature-dependent attention. By assigning different powers of the normalized adjacency matrix to various attention heads, X-LogSMask allows for multi-hop information propagation within a single layer, enhancing interpretability and efficiency. The authors demonstrate that a standard Transformer encoder can be viewed as one-step message passing on a complete graph, positioning X-LogSMask as a topology-constrained alternative to unrestricted self-attention. The experimental results show that Transformers equipped with X-LogSMask achieve state-of-the-art performance on 13 out of 20 benchmarks, indicating that simple structural masks can effectively enhance graph learning without altering the underlying Transformer architecture.
Methodology
The authors developed X-LogSMask by constructing a symmetrically normalized adjacency matrix with self-loops, which is then transformed logarithmically and added to the attention logits. This process allows for the suppression of irrelevant node interactions while preserving the discriminative properties of attention. The method assigns different powers of the normalized adjacency matrix to different attention heads, enabling each head to focus on specific structural radii and facilitating multi-hop information propagation.
Results
The experiments conducted across 20 node-, edge-, and graph-level benchmarks revealed that Transformers equipped with X-LogSMask achieved state-of-the-art results on 13 datasets, demonstrating significant improvements in performance while remaining competitive in a lightweight one-layer configuration.
Implications
The findings suggest that X-LogSMask can enhance the effectiveness of Transformers in graph learning tasks, making it a valuable tool for applications in various domains such as social networks, molecular biology, and transportation systems, where graph-structured data is prevalent.
Geometry-Aware Bayesian Quantification via Compositional Data Analysis
Theory
- Introduces a geometry-aware KDE model for multiclass quantification that respects the simplex structure of compositional data.
- Utilizes log-ratio transformations and Aitchison geometry to improve density estimation accuracy.
- Implements shrinkage regularization to enhance robustness near simplex boundaries.
- Demonstrates competitive performance against state-of-the-art quantifiers across various datasets.
Read more
Geometry-Aware Bayesian Quantification via Compositional Data Analysis
Summary
This paper addresses the challenge of accurately estimating target label distributions in the presence of label shift, a critical task known as quantification or class prevalence estimation. The authors propose a novel geometry-aware kernel density estimation (KDE) model that leverages log-ratio transformations and Aitchison geometry to handle the compositional nature of posterior probability vectors, which reside on the probability simplex. Traditional KDE methods using Euclidean Gaussian kernels fail to respect this geometry, leading to inaccuracies, especially near the simplex boundaries. The proposed method incorporates a shrinkage regularization technique to enhance robustness in these regions. By framing quantification as a mixture density estimation problem, the authors derive both point estimation and Bayesian inference procedures for class prevalences. Extensive experiments across 42 datasets from diverse domains, including tabular, text, and image data, demonstrate that the proposed method outperforms standard KDE-based quantifiers and competes effectively with state-of-the-art Bayesian quantification methods.
Methodology
The authors develop a geometry-aware KDE model that employs log-ratio transformations to represent posterior probability vectors in Aitchison geometry. This approach is complemented by a shrinkage regularization technique to stabilize estimates near the simplex boundary. The quantification task is framed as a mixture density estimation problem, allowing for both point estimation and Bayesian inference of class prevalences.
Results
The proposed method was tested on 42 datasets, showing competitive performance compared to existing quantification methods. It frequently outperformed standard KDE-based quantifiers and achieved strong results among Bayesian quantification techniques, demonstrating its effectiveness across different data types.
Implications
The findings suggest that incorporating geometric considerations into quantification methods can significantly enhance performance in real-world applications where label shift is prevalent. This approach could be beneficial in fields such as healthcare, finance, and any domain where understanding class distributions is crucial for decision-making.
One Framework for All: Cross-Modal Membership Inference for Generative Models
Generative Models
Multimodal
- Introduces a unified framework for membership inference across multiple generative model modalities.
- Utilizes the property of output distributions approximating training data distributions for effective inference.
- Employs likelihood ratio testing without the need for training additional models.
- Demonstrates superior performance compared to existing methods optimized for single model classes.
Read more
One Framework for All: Cross-Modal Membership Inference for Generative Models
Summary
This paper addresses the significant privacy risks posed by large generative models across various modalities, specifically focusing on membership inference attacks (MIAs). Existing research has primarily treated MIAs in isolation for different generative model classes (text-to-text, text-to-image, and image-to-text), limiting their applicability in real-world scenarios. The authors propose a unified membership inference framework that leverages the property that the output distribution of a generative model can approximate its training data distribution. By modeling the distributions of generated outputs and non-member samples in a shared embedding space, they perform membership inference through likelihood ratio testing. The framework is designed to be modality-agnostic, accommodating the unique characteristics of different data types. Extensive experiments demonstrate that their approach outperforms existing state-of-the-art methods, providing superior membership inference performance across various generative models and datasets, including evaluations against both fine-tuning and pre-training data.
Methodology
The authors develop a unified membership inference framework that models the output distributions of generative models and auxiliary non-member samples in a shared embedding space. They utilize likelihood ratio testing to infer membership status, leveraging modality-appropriate feature extractors to create numerical embeddings that are independent of the underlying data modality.
Results
The experimental results indicate that the proposed approach consistently outperforms existing state-of-the-art methods in membership inference tasks across various generative models and datasets. The framework demonstrates effectiveness in both partial-knowledge and zero-knowledge threat models, highlighting its robustness and applicability.
Implications
The findings suggest that a unified approach to membership inference can enhance the understanding of privacy risks associated with generative models in real-world applications. This framework could be crucial for developing privacy-preserving techniques in AI systems that integrate multiple generative components across different modalities.
Statistically Meaningful Geometry (SMG) Beyond the Euclidean Paradigm, with Application to Generative AI
Generative Models
Theory
Optimization
- Introduction of the Statistically Meaningful Geometry (SMG) framework for over-parameterized models.
- Development of a Two-Fold Inference Paradigm to enhance statistical inference in complex models.
- Utilization of an Ehresmann connection as a dynamic geometric filter to isolate learning trajectories.
- Addressing operational pathologies like generative hallucination and catastrophic forgetting in AI systems.
Read more
Statistically Meaningful Geometry (SMG) Beyond the Euclidean Paradigm, with Application to Generative AI
Summary
This paper presents the Statistically Meaningful Geometry (SMG) framework, which addresses the limitations of conventional statistical methods in the context of over-parameterized models, particularly in generative AI. The authors argue that traditional uniform convergence bounds and empirical risk minimization techniques fail in massive models, such as large language transformers, due to their complex optimization landscapes characterized by flat vertical gauge valleys. These issues lead to operational challenges like generative hallucination and catastrophic forgetting. The SMG framework introduces a new information-geometric paradigm that transitions from deterministic parametric models to infinite-dimensional non-parametric Orlicz statistical manifolds. The authors propose a Two-Fold Inference Paradigm that utilizes an Ehresmann connection 1-form as a dynamic geometric filter to eliminate vertical gauge noise and focus on learning trajectories along non-degenerate horizontal distributions. This approach not only enhances the understanding of the underlying geometry of complex models but also provides a robust framework for addressing challenges in generative AI, such as hallucination containment and continuous adaptation without memory erasure.
Methodology
The authors formalize the SMG framework by modeling the total state space as a differential fiber bundle and introducing a Two-Fold Inference Paradigm. They employ an Ehresmann connection 1-form to filter out noise and focus on significant learning directions, providing a mathematical foundation for their approach.
Results
The paper demonstrates that the SMG framework effectively addresses the challenges posed by over-parameterization in generative AI models. It provides a rigorous mathematical structure that enhances the understanding of model behavior and improves generalization capabilities, particularly in the context of large language models.
Implications
The SMG framework has significant implications for the development of generative AI systems, particularly in improving model robustness against hallucinations and enabling continuous learning without memory loss. It opens new avenues for research in statistical inference and model optimization in high-dimensional spaces.
BOUNDARY_SYNC: Measuring Communication-Induced Representational Coupling in Multi-Agent LLM Systems
Large Language Models
NLP
Multimodal
- Boundary_Sync provides a standardized measurement for communication-induced representational coupling in LLMs.
- Text communication leads to significant homogenization of outputs, while image communication shows comparable effects.
- Group size influences the direction of coupling, with smaller groups potentially leading to diversification.
- Coupling is stateless and dependent on immediate peer information, with no evidence of cumulative convergence.
Read more
BOUNDARY_SYNC: Measuring Communication-Induced Representational Coupling in Multi-Agent LLM Systems
Summary
This paper introduces Boundary_Sync, a measurement protocol designed to quantify the representational coupling induced by communication among large language models (LLMs) in multi-agent systems. The study investigates whether inter-agent communication leads to homogenization or diversification of outputs, using the Coupling Amplification Factor (CAF) as a metric. Controlled experiments with GPT-4o reveal that text communication significantly homogenizes outputs (CAF=0.803), while image communication also shows similar effects (CAF=0.834). The study identifies group size as a key moderator, indicating that with fewer agents, communication can lead to diversification (CAF > 1). The results demonstrate that coupling is stateless, driven by immediate peer information, and highlight the implications for designing effective multi-agent LLM systems. The findings suggest that communication can both unify and diversify agent outputs, depending on the context and structure of the communication.
Methodology
The authors conducted controlled experiments using GPT-4o, measuring the Coupling Amplification Factor (CAF) across text and image communication scenarios. They employed a no-communication ablation and prompt perturbation controls to isolate the effects of communication. The experiments involved 30 agents per condition and approximately 9,900 API calls.
Results
The results indicate that text communication significantly homogenizes outputs (CAF=0.803), while image communication also leads to homogenization (CAF=0.834). Group size was found to be a critical factor, with smaller groups (K=3) showing a shift towards diversification (CAF > 1). The coupling was shown to be stateless, with immediate peer information driving the effects observed.
Implications
The findings suggest that understanding communication dynamics is crucial for designing reliable multi-agent LLM systems. The ability to measure and control representational coupling can help in optimizing agent interactions and ensuring diversity in outputs, which is essential for collaborative reasoning and decision-making tasks.
Multi-modal Rail Crossing Safety Analysis
Multimodal
- Integration of visual and structured data improves safety assessment of railway crossings.
- Vision-Language Models (VLMs) are effective in analyzing multimodal data for risk scoring.
- The proposed system identifies high-risk and low-risk crossings with significant accuracy.
- The methodology addresses critical challenges in data preparation and model training.
Read more
Multi-modal Rail Crossing Safety Analysis
Summary
This paper addresses the critical safety concerns associated with highway-rail grade crossings in the United States, where over 2,000 collisions and 200 fatalities occur annually. The authors propose a novel AI system that integrates visual cues from images of railway crossings with structured data from official accident reports to assess crossing safety. The study explores the effectiveness of Vision-Language Models (VLMs) in analyzing multimodal data to provide safety assessments that align with expert opinions and existing safety scoring systems used by the Federal Railroad Administration (FRA). The proposed pipeline tackles various challenges in data preparation and learning paradigms, aiming to identify high-risk and low-risk crossings. The system achieves a macro F1 score of 0.757 for risk classification and an RMSE of 0.078 with a correlation of 0.492 for estimating FRA-based safety scores, demonstrating its potential to enhance the assessment of railway crossing safety.
Methodology
The authors developed a multimodal pipeline that combines image sequences of railway crossings with historical incident records from FRA Form 57. They evaluated VLMs for two tasks: risk scoring (binary classification and continuous score prediction) and visual risk inspection. The study involved fine-tuning models and employing a routed prediction strategy to manage skewed risk score distributions.
Results
The proposed system achieved a macro F1 score of 0.757 for identifying high-risk and low-risk crossings. For estimating FRA-based safety scores, it produced an RMSE of 0.078 and a correlation of 0.492 with expert assessments, indicating a strong alignment with domain knowledge.
Implications
The findings suggest that AI-assisted tools can significantly enhance the scalability and reliability of railway crossing safety assessments, potentially leading to better resource allocation for safety interventions and improved public safety outcomes.
Single-Channel EEG-Based Cognitive Load Assessment in Online Learning: A Hybrid Deep Learning Approach
Time Series
- The hybrid CNN+LSTM+Attention model achieves up to 78.5% accuracy in cognitive load assessment.
- Regularization techniques are crucial for reducing overfitting and improving model generalization.
- The study advocates for subject-independent evaluation to better assess model performance across different learners.
- An open-source evaluation pipeline and visualization tool are provided to enhance reproducibility and practical application.
Read more
Single-Channel EEG-Based Cognitive Load Assessment in Online Learning: A Hybrid Deep Learning Approach
Summary
This paper investigates the feasibility of using a single-channel EEG device, specifically the NeuroSky MindWave Mobile 2, to assess cognitive load during online learning. The authors aim to differentiate between easy and difficult educational video content by employing a hybrid deep learning model that integrates Convolutional Neural Networks (CNN), Long Short-Term Memory (LSTM) networks, and attention mechanisms. The study utilizes a dataset from Wang et al., involving nine learners, to evaluate the model's performance. The hybrid model achieves an accuracy of up to 78.5% in a within-subject setting, significantly outperforming conventional feature-based classifiers, which only reach 55% accuracy. The authors emphasize the importance of regularization techniques to mitigate overfitting and maintain stable validation accuracy. They advocate for subject-independent evaluation as the standard for assessing generalizability, given the small sample size. To facilitate reproducibility, the authors provide an open evaluation pipeline and a visualization tool that displays cognitive load estimates as a heatmap over the video timeline, aiding educators in identifying challenging content segments.
Methodology
The authors implemented a hybrid model combining CNN, LSTM, and attention mechanisms to process EEG data from a single-channel device. They conducted a within-subject evaluation using a dataset of nine learners, applying regularization techniques to address overfitting and improve validation accuracy. The study also includes an open evaluation pipeline for reproducibility.
Results
The hybrid model achieved a maximum accuracy of 78.5% in distinguishing between easy and difficult educational content, compared to 55% for traditional classifiers. Regularization techniques helped stabilize validation accuracy between 68% and 73%. The authors caution that the within-subject evaluation may be overly optimistic due to the small sample size.
Implications
This research highlights the potential of consumer-grade EEG devices for real-time cognitive load assessment in online learning environments. The findings suggest that EEG can provide valuable insights for educators, enabling timely interventions and improvements in instructional design. The open-source tools developed can facilitate further research and practical applications in educational settings.
Quantize the Target, Quantize the Drafter: Efficient Inference with Qwen3.5-4B
Large Language Models
Efficient ML
Optimization
- Achieved a 6.978× speedup in inference latency over the baseline model.
- Utilized quantization-aware distillation to recover accuracy in the quantized target model.
- Developed a block-diffusion drafter optimized for speculative decoding.
- Implemented sliding-window attention to enhance long-context decoding efficiency.
Read more
Quantize the Target, Quantize the Drafter: Efficient Inference with Qwen3.5-4B
Summary
This paper presents a novel approach to enhance the inference efficiency of the Qwen3.5-4B large language model (LLM) on resource-constrained hardware, specifically an NVIDIA A10G GPU. The authors participated in the Efficient Qwen Competition, aiming to minimize latency while maintaining quality. Their methodology integrates a quantized target model with speculative decoding techniques. To recover accuracy lost during quantization, they employed quantization-aware distillation (QAD) on the target model, ensuring it retains the original quantization grid. Additionally, they developed a lightweight block-diffusion drafter that operates in tandem with the quantized target model, trained through a two-stage process: initially learning from a high-precision model and subsequently adapting to the low-precision target. The drafter's efficiency is further enhanced using quantization and sliding-window attention, which improves long-context decoding latency. The results demonstrate a significant average speedup of 6.978× over the baseline model while meeting the required quality thresholds, achieving a commendable 3rd place in the competition. The authors provide their code and resources for further exploration.
Methodology
The authors combined quantization of the Qwen3.5-4B model with speculative decoding techniques. They applied quantization-aware distillation (QAD) to recover accuracy in the quantized target model and trained a lightweight block-diffusion drafter through a two-stage process. The drafter was optimized with quantization and sliding-window attention to improve inference speed.
Results
The proposed system achieved an average speedup of 6.978× compared to the unoptimized baseline model across various input sizes, while still meeting the required quality thresholds on standard benchmarks. Specifically, the latency for short, medium, and long inputs was significantly reduced, demonstrating the effectiveness of the proposed methods.
Implications
The findings suggest that combining quantization with advanced decoding techniques can significantly enhance the efficiency of large language models, making them more viable for deployment in resource-constrained environments. This work provides insights that could inform future developments in efficient LLM inference.
On the Design Space of Discrete Diffusion Online Adaptation for Molecular Optimization
Generative Models
Optimization
- The study explores the interactions among various design choices in online adaptation for molecular optimization.
- Acquisition, reward shaping, and debiasing techniques provide complementary benefits, especially for small molecules.
- The proposed online fine-tuning recipe outperforms offline methods and inference-time searches under fixed oracle budgets.
- Replay mechanisms stabilize learning and maintain valid molecular exploration.
Read more
On the Design Space of Discrete Diffusion Online Adaptation for Molecular Optimization
Summary
This paper investigates the online adaptation of discrete diffusion models for molecular optimization, focusing on how to effectively utilize a limited oracle budget to generate high-reward molecules. The authors identify several critical design choices in the adaptation loop, including candidate evaluation, reward shaping, feedback reuse, and model debiasing. Through controlled studies on small-molecule binding-affinity and protein-fitness tasks, they demonstrate that these components can provide complementary benefits when integrated into a single online fine-tuning recipe. The findings reveal that acquisition strategies, reward shaping, and debiasing techniques enhance the optimization process, particularly when high-reward candidates are located further from the pretrained generative prior. The proposed approach outperforms traditional offline fine-tuning and inference-time search methods under matched oracle-call budgets, highlighting the importance of a well-structured feedback loop in molecular design tasks.
Methodology
The authors conducted controlled experiments across six small-molecule binding-affinity tasks and three protein-fitness tasks. They analyzed the interactions of various components in the online adaptation loop, including candidate selection, reward shaping, and model updates, using discrete diffusion models. The study employed a fixed online schedule with swappable design choices to assess the effectiveness of different strategies.
Results
The results indicate that the integrated online adaptation approach significantly outperformed baseline methods, particularly in scenarios where high-reward candidates necessitated substantial deviations from the pretrained model. The combination of acquisition strategies, CVaR reward shaping, and Density Entropy Regularization led to notable improvements in optimization outcomes, while replay mechanisms ensured stability in learning.
Implications
The findings suggest that a structured online adaptation framework can enhance molecular optimization tasks, making it applicable in drug discovery and materials science. The insights gained from this study can inform the design of more efficient generative models in various chemical and biological applications.
Expander Sparse Autoencoders: Parameter-Efficient Dictionaries for Mechanistic Interpretability
Interpretability
Efficient ML
Large Language Models
- Introduction of Expander SAEs, a parameter-efficient architecture for sparse coding.
- Demonstrated a significant reduction in learned decoder values while maintaining high reconstruction fidelity.
- Proposed a parallel implementation of OMP that improves decoding efficiency.
- Theoretical proofs supporting the identifiability of k-sparse codes under specific conditions.
Read more
Expander Sparse Autoencoders: Parameter-Efficient Dictionaries for Mechanistic Interpretability
Summary
This paper introduces Expander Sparse Autoencoders (Expander SAEs), a novel architecture designed to enhance the mechanistic interpretability of neural networks while maintaining parameter efficiency. Traditional Sparse Autoencoders (SAEs) face challenges in scalability due to the dense decoder requiring a large number of learned parameters. Expander SAEs address this issue by employing a left-d-regular expander mask, which significantly reduces the number of learned decoder values from mn to dn, where d is much smaller than m. This architecture allows for efficient sparse coding while preserving the fidelity of the reconstruction. The authors demonstrate that varying the sparsity parameter d leads to a trade-off between the number of learned decoder values and reconstruction fidelity across several language models, achieving a notable reduction in parameters while retaining high performance. The paper also presents a parallel implementation of Orthogonal Matching Pursuit (OMP) that leverages the expander structure, enhancing the speed and efficiency of the decoding process. The theoretical contributions include proofs that establish conditions for the identifiability of k-sparse codes, ensuring that the architecture can effectively recover sparse representations.
Methodology
The authors developed Expander SAEs by masking the encoder and decoder dictionaries with a left-d-regular expander graph, which reduces the number of parameters needed for the decoder. They conducted experiments on various language models, analyzing the trade-offs between storage and fidelity. Additionally, they implemented a parallel version of OMP to optimize the decoding process, and provided theoretical analysis to support the architecture's effectiveness.
Results
Experiments showed that the Expander SAE architecture could achieve 84% of the reconstruction fidelity of a full dense decoder while using 293 times fewer learned decoder values in the case of Qwen2.5-3B with d=7. The results indicated a consistent storage-fidelity frontier across different models, validating the proposed architecture's efficiency.
Implications
The findings suggest that Expander SAEs can be effectively used in scenarios where interpretability and parameter efficiency are crucial, such as in large language models and other neural network applications. This work could lead to advancements in understanding neural network representations and improving the efficiency of model training and deployment.
FedAvg for HAR: Exploring the Tradeoff Between Personalized and Generalization Accuracy
Federated Learning
- FedAvg demonstrates improved personalization capabilities while maintaining generalization compared to centralized learning.
- The performance tradeoff between personalization and generalization is influenced by factors like client data heterogeneity and class distribution.
- New evaluation strategies, including activity class exclusion, provide deeper insights into federated learning dynamics.
- Under stressful conditions, such as varying class distributions, the advantages of FedAvg may not be as pronounced.
Read more
FedAvg for HAR: Exploring the Tradeoff Between Personalized and Generalization Accuracy
Summary
This paper investigates the tradeoff between personalized and generalized accuracy in the context of federated learning (FL) applied to Human Activity Recognition (HAR). The authors implement various testing scenarios to evaluate the performance of the FedAvg algorithm compared to centralized and local training paradigms. They explore the impact of client data heterogeneity and class distribution on model performance. The study reveals that while FedAvg enhances personalization and maintains generalization compared to traditional centralized learning, these benefits diminish under challenging conditions, such as varying class distributions among clients. The paper also introduces new evaluation strategies, including an activity class exclusion method, which has not been previously explored in the HAR federated learning literature. Overall, the findings highlight the complexities of achieving an optimal balance between personalization and generalization in federated learning settings.
Methodology
The authors designed and implemented multiple testing scenarios to compare the performance of centralized, local, and federated learning paradigms. They utilized the FedAvg algorithm and explored various metrics to assess model performance across different data distributions and client scenarios. The methodology included a detailed data partitioning strategy to evaluate both global and local model performances.
Results
The experimental results indicated that FedAvg achieved a higher degree of personalization while retaining a significant level of generalization compared to traditional centralized learning. However, this advantage was less clear under conditions of varying class distribution among clients, suggesting that the effectiveness of federated learning can be context-dependent.
Implications
The findings of this study have implications for the design of federated learning systems, particularly in applications where data privacy is crucial, such as mobile health monitoring and smart environments. Understanding the tradeoffs between personalization and generalization can help in developing more robust federated learning algorithms that adapt to diverse client conditions.
Self-explainable Operator Learning for Discovering Spatial Patterns in Functional Data
Interpretability
- Introduces a self-explainable operator learning framework for improved interpretability.
- Reformulates operator learning using integral equations to enhance transparency.
- Enables direct interpretability by linking input regions to output predictions.
- Demonstrates effectiveness in fluid flow problems, providing physically meaningful insights.
Read more
Self-explainable Operator Learning for Discovering Spatial Patterns in Functional Data
Summary
This paper introduces a self-explainable operator learning framework aimed at enhancing interpretability in modeling complex physical systems represented in functional spaces. Traditional operator learning methods, often based on neural networks, lack transparency, making it difficult to understand their predictions. The authors propose a reformulation of operator learning as a linear combination of generalized functional linear models expressed through integral equations. By exploiting the additive decomposability of these equations, the framework divides the input domain into subdomains, allowing for localized integral computations that reveal the contribution of specific regions to the overall prediction. This approach not only enhances interpretability by linking input regions to output patterns but also provides insights into the spatial features driving predictions. The framework is validated through applications in fluid flow problems, such as blood flow and unsteady aerodynamics, demonstrating that the model prioritizes regions with strong feature gradients. The proposed method shows qualitative agreement with existing post-hoc explainability techniques while embedding explainability directly within the operator structure, thus fostering trust in machine learning applications for scientific analysis.
Methodology
The authors reformulate operator learning as a linear combination of generalized functional linear models expressed through integral equations. They utilize additive decomposability to compute localized integrals over subdomains, allowing for the evaluation of contributions from specific input regions to the final predictions.
Results
The framework was applied to function-to-scalar and function-to-function mappings in fluid flow scenarios, revealing that the operator emphasizes regions with strong feature gradients. This approach provided insights into the model's decision-making process and demonstrated qualitative agreement with established post-hoc explainability methods.
Implications
The proposed framework enhances the interpretability of machine learning models in scientific applications, allowing researchers to gain insights into complex physical systems and fostering trust in data-driven analyses. It positions operator learning as a viable alternative to traditional black-box models, particularly in contexts where understanding the rationale behind predictions is crucial.
Finite-Lag Operator Geometry of Recurrent Representations
Theory
Time Series
- Introduction of finite-lag operator geometry for recurrent representations.
- Development of a conditional transport law and source-centered transport tensor.
- Proof of structural results including affine covariance and estimator stability.
- Demonstration of the framework's effectiveness in detecting deterministic recurrent motion.
Read more
Finite-Lag Operator Geometry of Recurrent Representations
Summary
This paper introduces a novel framework for analyzing recurrent representations using finite-lag operator geometry, which is distinct from traditional static snapshot methods. The author develops a conditional transport law, Q∆(dy | x), that captures the dynamics of recurrent hidden states by examining source-successor pairs over a fixed lag. This approach leads to the definition of a source-centered transport tensor, G∆, which decomposes into conditional spread and coherent displacement, alongside an antisymmetric circulation statistic, Wρ∆, that summarizes directed flow. The paper establishes several structural results, including affine covariance and stability of the Gaussian estimator, and demonstrates that deterministic recurrent motion can be detected through G∆ even when traditional methods fail. The methodology is validated through controlled experiments, revealing architecture-dependent differences in transport scale and coherent displacement in performance-matched repeat-copy networks. Overall, the framework provides a comprehensive geometric perspective on recurrent representations, emphasizing the importance of temporal dynamics in understanding neural computations.
Methodology
The methodology involves defining a finite-lag conditional transport law based on observed source-successor pairs, estimating it using a dense Gaussian source-smoothing operator. The resulting transport tensor is analyzed for its geometric properties, and controlled experiments are conducted to validate the theoretical predictions.
Results
The results show that the finite-lag operator geometry can effectively capture the dynamics of recurrent representations, revealing insights into conditional spread, coherent displacement, and directed flow. The framework successfully identifies deterministic recurrent motion that is not observable through traditional infinitesimal methods, and the experiments confirm the theoretical predictions regarding transport scale and stability.
Implications
This framework has potential applications in understanding the dynamics of recurrent neural networks, improving model interpretability, and enhancing the design of architectures for tasks involving temporal dependencies. It also opens avenues for further research into the geometric properties of recurrent representations in various machine learning contexts.
Signal or Noise? Understanding Generative Models for Real-World Sensor Time Series
Generative Models
Time Series
- Introduction of SensorGen, a large-scale framework for evaluating generative models on sensor time series.
- Flow-matching models demonstrate superior performance across diverse generation settings.
- Signal properties, including demographic covariates and time-frequency modeling, enhance generation quality.
- Generated signals provide practical benefits beyond visual realism, improving performance in downstream tasks.
Read more
Signal or Noise? Understanding Generative Models for Real-World Sensor Time Series
Summary
This paper introduces SensorGen, a comprehensive study aimed at understanding generative models for real-world sensor time series data. The authors identify that existing generative modeling approaches are fragmented across various modalities and datasets, which limits the understanding of their effectiveness in real-world applications. SensorGen encompasses a large-scale evaluation of generative models across 14 settings, 4 domains, 7 datasets, and 12 signal modalities. The study evaluates five major families of generative models and reveals three significant findings: (1) flow-matching models consistently outperform others in most settings; (2) the properties of the signals, such as demographic covariates and time-frequency characteristics, significantly influence generation quality; and (3) the generated signals have practical utility beyond mere visual realism, with improvements in scaling and synthetic data enhancing downstream task performance. The findings contribute to a broader understanding of design choices, evaluation protocols, and potential failure modes in sensor data generation.
Methodology
The study employs a unified exploration framework that consolidates various sensor generation tasks, modeling paradigms, and evaluation protocols. It systematically evaluates generative models from five major families, including diffusion models, flow-matching models, autoregressive models, normalizing-flow models, and hierarchical models, across diverse sensor settings.
Results
The results indicate that flow-matching models are a robust baseline for sensor generation tasks. Additionally, incorporating signal properties such as demographic information and time-frequency analysis leads to improved generation outcomes. The study also finds that scaling enhances the quality of generated signals and that synthetic data can significantly boost performance in scenarios with limited real data.
Implications
The findings from this study have significant implications for the development of generative models in sensor data applications, suggesting that careful consideration of model choice and signal properties can lead to better performance. This research could inform future work in areas such as healthcare monitoring, environmental sensing, and other domains reliant on sensor data.
The Rollout Infrastructure Tax in Coding-Agent Reinforcement Learning
Reinforcement Learning
Efficient ML
- Introduction of the 'rollout infrastructure tax' concept, highlighting the impact of execution substrate on RL performance.
- Significant variations in cold-start latency and worker-hour requirements based on substrate choice.
- Development of a controlled evaluation methodology for comparing execution substrates.
- Identification of critical components contributing to the rollout infrastructure tax.
Read more
The Rollout Infrastructure Tax in Coding-Agent Reinforcement Learning
Summary
This paper addresses the often-overlooked impact of execution infrastructure on coding-agent reinforcement learning (RL), which relies heavily on interactive software rollouts. The authors introduce the concept of the 'rollout infrastructure tax,' which refers to the latency and costs introduced by the systems executing coding-agent RL trajectories. They conduct a comparative study of four execution substrates: single containers, hosted sandboxes, Kubernetes-orchestrated containers, and cloud virtual machines. The study reveals significant variations in cold-start latency (up to 110×) and projected worker-hours for large-scale rollouts (up to 1.8× spread for one million 150-step trajectories). The findings suggest that optimizing execution substrates should be an integral part of the training system, rather than merely a deployment consideration. The paper concludes with design requirements for rollout-native substrates to enhance efficiency in coding-agent RL systems.
Methodology
The authors conducted a measurement study comparing four common execution substrates while maintaining a fixed coding-agent workload. They defined and quantified the components of the rollout infrastructure tax, including environment creation time, readiness time, per-action costs, and orchestration overhead. The evaluation was controlled to ensure that the differences observed were solely due to the execution substrates.
Results
The study found that substrate choice significantly affects rollout performance, with cold-start latency varying by up to 110×. For one million 150-step trajectories, the choice of substrate resulted in a 1.8× spread in projected worker-hours, equating to an additional 5,316 worker-hours. The results indicate that small per-rollout savings can compound at scale, emphasizing the importance of optimizing execution substrates.
Implications
The findings suggest that as coding-agent RL systems scale, the execution infrastructure should be treated as a core concern in the design of training systems. This could lead to more efficient training processes and better resource utilization in large-scale RL applications.
A Mathematical Introduction to Diffusion Models
Generative Models
Theory
Optimization
- Introduces a comprehensive mathematical framework for understanding diffusion models.
- Establishes convergence guarantees for Langevin diffusion and its applications.
- Analyzes sampling errors in discretized diffusion models and their implications.
- Explores inference-time control techniques for trained diffusion models.
Read more
A Mathematical Introduction to Diffusion Models
Summary
This paper provides a proof-oriented introduction to diffusion models, focusing on their mathematical foundations and applications in sampling. It is structured into five main sections: the first discusses the sampling language and Langevin dynamics, establishing convergence guarantees for Langevin diffusion and the unadjusted Langevin algorithm (ULA). The second section introduces continuous-time score-based diffusion models, utilizing Gaussian noising and reverse-time stochastic differential equations (SDEs). The third section addresses the discretization of these models into implementable denoising diffusion probabilistic models (DDPMs), analyzing sampling errors associated with various methods. The fourth section explores discrete diffusion on finite state spaces, reformulating the error analysis in this context. Finally, the fifth section focuses on inference-time steering techniques, including guidance and reinforcement learning. The notes are designed for graduate students with a foundational understanding of probability, aiming to bridge the gap to more advanced topics in diffusion models and stochastic processes.
Methodology
The paper employs a proof-oriented approach, developing core definitions and identities related to diffusion models, and providing detailed proofs for key results. It integrates concepts from probability theory, stochastic calculus, and numerical methods to analyze diffusion processes and their applications in generative modeling.
Results
The paper presents convergence guarantees for Langevin diffusion and ULA, detailed error analyses for DDPMs, and establishes the framework for discrete diffusion models. It also outlines effective inference-time steering techniques that enhance the performance of trained models.
Implications
The findings have significant implications for the development of generative models, particularly in enhancing the efficiency and accuracy of sampling methods. The structured approach to understanding diffusion models can aid researchers and practitioners in implementing these techniques in various applications, including image generation and probabilistic modeling.
DeadPool: Resilient LLM Training with Hot-Swapping via Zero-Overhead Checkpoint
Large Language Models
Efficient ML
Optimization
- DEADPOOL enables hot-swapping of failed nodes without job termination.
- Introduces an asynchronous in-memory checkpointing strategy for optimizer states.
- Achieves zero overhead during normal execution and rapid recovery from failures.
- Evaluated on large-scale systems with significant model sizes and GPU counts.
Read more
DeadPool: Resilient LLM Training with Hot-Swapping via Zero-Overhead Checkpoint
Summary
The paper introduces DEADPOOL, a fault-tolerant system designed for large language model (LLM) training that addresses the challenges of node failures in distributed training environments. Traditional fault-tolerance mechanisms often incur significant overhead during normal operations or lengthy recovery times after failures. DEADPOOL innovatively employs a hot-swapping mechanism that allows for the replacement of failed compute nodes with spare nodes without terminating the entire training job. This is achieved through two main strategies: an asynchronous in-memory checkpointing system that minimizes overhead during error-free execution, and a communicator reconstruction protocol that facilitates real-time node replacement. The authors evaluate DEADPOOL on high-performance computing systems using up to 512 NVIDIA A100 GPUs and LLMs with up to 65 billion parameters, demonstrating that it maintains zero checkpoint overhead and can recover from node failures in under 40 seconds. The results indicate that DEADPOOL effectively balances the need for efficient training and robust fault tolerance, making it a significant advancement in LLM training methodologies.
Methodology
The methodology involves implementing an online hot-swapping recovery mechanism that utilizes asynchronous in-memory checkpointing to replicate optimizer state shards across GPUs. The system distinguishes between transient and permanent failures, applying a distributed communicator reconstruction protocol to replace failed nodes dynamically. The evaluation is conducted on two production HPC systems, where node failures are injected into live training jobs to assess the effectiveness of the recovery mechanism.
Results
The evaluation results show that DEADPOOL maintains negligible per-step checkpointing overhead during normal operations and successfully completes node replacements and training restoration in approximately 40 seconds after a failure. This performance is consistent across different scales, including up to 512 GPUs and models with 65 billion parameters.
Implications
The implications of DEADPOOL are significant for large-scale LLM training, particularly in environments where node failures are common. Its ability to provide robust fault tolerance without incurring substantial overhead can enhance the efficiency and reliability of training processes in supercomputing contexts, potentially benefiting a wide range of applications in natural language processing and beyond.
Spin-Weighted Spherical Harmonics Enable Complete and Scalable E(3)-Equivariant Networks
Theory
Efficient ML
Graph Learning
- Introduction of SpinGTP to enhance the expressivity of E(3)-equivariant networks.
- SpinGTP recovers antisymmetric interactions lost in previous tensor product formulations.
- Demonstrated superior performance in tasks involving chiral materials and non-centrosymmetric geometries.
- Achieves comparable accuracy to full CGTP while maintaining scalability.
Read more
Spin-Weighted Spherical Harmonics Enable Complete and Scalable E(3)-Equivariant Networks
Summary
This paper addresses the limitations of E(3)-equivariant networks in modeling 3D atomistic systems, particularly focusing on the computational complexity and expressivity of tensor products. The authors introduce SpinGTP, a novel approach that utilizes Spin-Weighted Spherical Harmonics (SWSH) to enhance the expressivity of Gaunt Tensor Products (GTP) while maintaining computational efficiency. The SpinGTP method recovers missing antisymmetric interactions that are crucial for tasks involving chiral materials and non-centrosymmetric geometries. The authors demonstrate the effectiveness of SpinGTP through extensive benchmarking on various tasks, achieving accuracies comparable to the full Clebsch-Gordan Tensor Product (CGTP) while being more scalable. This work not only provides a mathematically rigorous framework for high-order equivariance but also opens new avenues for large-scale 3D atomistic simulations, making it a significant contribution to the field of AI for science.
Methodology
The authors developed SpinGTP by generalizing the Gaunt Tensor Product to incorporate Spin-Weighted Spherical Harmonics. This approach leverages the algebraic properties of SWSH to recover antisymmetric interactions and allows for a more expressive equivariant basis. The implementation includes specialized SWSH equivariant layers and high-performance computational strategies.
Results
SpinGTP was evaluated across multiple benchmarks, including Tetris, 3BPA, SPICE-MACE-OFF, and OC20. The results indicated that SpinGTP achieved accuracies on par with full CGTP, particularly excelling in tasks that require capturing antisymmetric paths, such as those involving chiral materials.
Implications
The development of SpinGTP has significant implications for the modeling of complex molecular interactions and interatomic potentials in 3D atomistic simulations. Its scalability and expressivity make it a valuable tool for researchers in computational materials science and related fields.
Environmental Drivers of Respiratory Disease: A District Level Analysis
Interpretability
- Developed an 11-year panel dataset integrating environmental and health data across 25 districts in Sri Lanka.
- Achieved high predictive accuracy for respiratory disease rates and PM2.5 concentrations using XGBoost models.
- Identified air quality as the primary driver of respiratory health variance, significantly ahead of forest degradation.
- Introduced the Forest-Air-Health (FAH) Risk Index to rank districts by environmental health risk.
Read more
Environmental Drivers of Respiratory Disease: A District Level Analysis
Summary
This study investigates the environmental factors influencing respiratory disease admissions in Sri Lanka, particularly in light of recent forest degradation and rising pollution levels. The authors constructed an 11-year panel dataset (2014-2024) encompassing 25 administrative districts, integrating satellite-derived vegetation indices, fire activity, pollutant concentrations (PM2.5, NO2, SO2), and population-normalized respiratory admission rates. Two XGBoost models were developed to predict annual respiratory rates and monthly PM2.5 concentrations, achieving high accuracy (R2 = 0.937 and R2 = 0.976, respectively). The Shapley Additive Explanations (SHAP) analysis revealed that air quality burden accounts for 80.1% of the variance in respiratory rates, with forest degradation and fire activity contributing 15.6% and 4.3%, respectively. The study introduced the Forest-Air-Health (FAH) Risk Index to identify districts at high risk for respiratory health issues, with Colombo, Gampaha, and Kalutara identified as the most affected. This research provides a novel, evidence-based framework for correlating environmental degradation with respiratory health, offering a quantitative basis for targeted public health and environmental policies in Sri Lanka.
Methodology
The study constructed a comprehensive panel dataset by integrating various environmental and health metrics over 11 years. XGBoost models were employed to predict respiratory rates and PM2.5 concentrations, with SHAP analysis used to interpret the influence of different environmental factors. The FAH Risk Index was created to rank districts based on their risk levels.
Results
The XGBoost models demonstrated high predictive accuracy, with R2 values of 0.937 for respiratory rates and 0.976 for PM2.5 concentrations. SHAP analysis indicated that air quality burden was the most significant contributor to respiratory health variance (80.1%), while the FAH Risk Index highlighted Colombo, Gampaha, and Kalutara as the districts with the highest health risks.
Implications
The findings underscore the importance of addressing air quality and environmental degradation in public health strategies. The study provides a quantitative framework that can guide policymakers in developing targeted interventions to improve respiratory health outcomes in Sri Lanka.
I2RiMA: Spectral Riemannian Representation with Temporal Attention for Mental Stress Detection based on EEG Signals
Time Series
- I2RiMA constructs frequency-specific spatial covariance matrices to preserve discriminative patterns.
- The model employs frequency cluster aggregation for effective feature selection and redundancy reduction.
- An intra-inter slice attention module captures both local and global temporal dependencies in EEG data.
- I2RiMA achieves state-of-the-art performance on multiple datasets with a compact model architecture.
Read more
I2RiMA: Spectral Riemannian Representation with Temporal Attention for Mental Stress Detection based on EEG Signals
Summary
The paper presents I2RiMA, an innovative Intra-Inter Riemannian Manifold Attention Network designed for detecting mental stress through EEG signals. The authors identify significant challenges in cross-subject EEG stress detection, particularly the subject-dependent and frequency-specific nature of stress-related patterns. Traditional Riemannian methods often overlook the importance of neural oscillations and fail to maintain temporal coherence across EEG slices. To address these issues, I2RiMA constructs spatial covariance matrices at each frequency point, mapping them to the SPD tangent space to preserve channel-wise geometry and frequency-specific cues. Additionally, the model employs frequency cluster aggregation to enhance feature selection and reduce redundancy by forming compact frequency clusters aligned with EEG rhythms. An intra-inter slice attention module is introduced to effectively integrate local slice-level dynamics with global temporal context across EEG sequences. Experimental results demonstrate that I2RiMA outperforms five state-of-the-art baselines, achieving a balanced accuracy of up to 82.78% while maintaining efficiency with only 1.60M parameters and 31.95M FLOPs.
Methodology
I2RiMA utilizes a two-pronged approach: it constructs spatial covariance matrices at each frequency point and maps them to the SPD tangent space, preserving the Riemannian geometry. It incorporates frequency cluster aggregation to select informative spectral components and reduce redundancy. The intra-inter slice attention module integrates local slice-level dynamics with global temporal context, allowing for a comprehensive understanding of EEG sequences.
Results
I2RiMA demonstrates superior performance on three datasets, achieving balanced accuracies of 77.59%, 75.88%, and 82.78% on MIST Control, MIST Stress, and SEED datasets, respectively. The model is efficient, utilizing only 1.60M parameters and 31.95M FLOPs.
Implications
The findings suggest that I2RiMA could significantly enhance the accuracy of mental stress detection in real-world applications, particularly in non-clinical settings. The model's efficiency and effectiveness make it suitable for scalable, everyday EEG-based stress monitoring, potentially aiding in mental health assessments and interventions.
Physiological Noise Augmentation Improves Non-Invasive Brain-to-Speech
Audio & Speech
Time Series
Multimodal
- Introduction of Physiological Noise Augmentation (PNA) for non-invasive brain-to-speech decoding.
- PNA improves decoder robustness by training on augmented data that includes physiological artifacts.
- Achieved a 4.7% increase in decoding accuracy on the MegNIST dataset using EEGNet.
- PNA complements traditional multi-trial averaging techniques to enhance performance.
Read more
Physiological Noise Augmentation Improves Non-Invasive Brain-to-Speech
Summary
This paper presents a novel data augmentation technique called Physiological Noise Augmentation (PNA) aimed at enhancing non-invasive brain-to-speech decoding for patients with neurodegenerative diseases. Traditional methods using MEG and EEG suffer from high word error rates due to low signal-to-noise ratios, particularly in imagined speech contexts. PNA addresses this by training decoders to be invariant to task-agnostic artifacts such as ocular and cardiac activity. The authors employ independent component analysis (ICA) to decompose brain recordings into clean data and noise artifacts, which are then scaled and remixed to create realistic training examples. The results demonstrate that PNA significantly improves decoding accuracy on the MegNIST dataset, achieving a 4.7 percentage point increase over models trained solely on real data. This work highlights the potential of artifact-aware augmentation strategies in enhancing the robustness of non-invasive brain-computer interfaces.
Methodology
The methodology involves the use of independent component analysis (ICA) to isolate physiological artifacts from brain recordings. These artifacts are then scaled and remixed to create augmented training data that preserves labels while introducing realistic noise. The augmentation is designed to expose the decoder to a broader range of artifact realizations, promoting invariance to nuisance structures.
Results
The application of PNA resulted in a 4.7 percentage point improvement in decoding accuracy on the MegNIST dataset, reaching an overall accuracy of 77.6% with the EEGNet architecture. This demonstrates the effectiveness of the proposed augmentation technique in enhancing the performance of non-invasive brain-to-speech systems.
Implications
The findings suggest that PNA could significantly improve the performance of non-invasive brain-computer interfaces, making them more viable for clinical applications in restoring communication for patients with speech impairments. The approach may also be applicable to other areas of neural decoding and signal processing where physiological noise is a concern.
MKGR: Multimodal Knowledge-Graph Representation Learning for Cold-Start Protein-Protein Interaction Prediction
Multimodal
Graph Learning
- MKGR effectively combines protein sequence data with multimodal biomedical knowledge graphs for PPI prediction.
- The framework introduces a bridge reconstruction objective to enhance graph learning under sparse conditions.
- A pair-level gated fusion mechanism allows for adaptive integration of sequence and graph representations.
- Experiments show significant performance improvements over traditional PPI prediction methods.
Read more
MKGR: Multimodal Knowledge-Graph Representation Learning for Cold-Start Protein-Protein Interaction Prediction
Summary
This paper introduces MKGR, a novel multimodal representation learning framework aimed at improving cold-start protein-protein interaction (PPI) prediction. The challenge of cold-start scenarios arises when new proteins lack observed PPI edges during training, making traditional models reliant on network topology less effective. MKGR addresses this by integrating region-aware protein sequence encoding with four biomedical knowledge graphs that capture protein-drug, protein-disease, protein-miRNA, and protein-lncRNA associations. The framework consists of a sequence branch that utilizes a pretrained protein language model and a Transformer encoder to extract contextual representations from protein sequences, while the graph branch employs graph attention networks to learn modality-specific embeddings from sparse biomedical associations. A bridge reconstruction objective is introduced to regularize graph learning by recovering shared protein-entity associations, and a pair-level gating module adaptively combines sequence and graph evidence for each candidate protein pair. Experimental results on two benchmark datasets demonstrate that MKGR consistently outperforms existing sequence, network, and knowledge-graph baselines across various evaluation metrics, including accuracy, F1 score, AUC, AUPR, and MCC.
Methodology
MKGR employs a two-branch architecture: a sequence branch that encodes protein sequences using a pretrained model and a Transformer, and a graph branch that utilizes graph attention networks to model protein-entity relationships from biomedical knowledge graphs. The bridge reconstruction objective aids in learning robust representations despite sparse data, while the pair-level gating mechanism allows for tailored integration of the sequence and graph modalities for each protein pair.
Results
The MKGR framework was evaluated on two benchmark datasets under novel-old and novel-novel cold-start settings, demonstrating consistent performance improvements over competitive baselines in terms of accuracy, F1 score, AUC, AUPR, and MCC.
Implications
The findings suggest that MKGR can significantly enhance the prediction of protein-protein interactions in scenarios where data is sparse or incomplete, which is crucial for advancing research in functional genomics, understanding disease mechanisms, and facilitating drug discovery.
Rank-Then-Act: Reward-Free Control from Frame-Order Progress
Reinforcement Learning
Computer Vision
Multimodal
- RTA enables learning control policies from video without environment rewards.
- The framework uses a VLM trained as a progress scorer on shuffled video clips.
- A correlation-based reward signal is introduced, leveraging Spearman rank correlation.
- RTA outperforms existing methods on various control benchmarks.
Read more
Rank-Then-Act: Reward-Free Control from Frame-Order Progress
Summary
The paper introduces Rank-Then-Act (RTA), a novel framework for learning control policies from expert video demonstrations without relying on environment rewards. RTA employs a Vision-Language Model (VLM) trained offline as a progress-based ordinal scorer using a Group Relative Policy Optimization (GRPO) objective. This approach focuses on recovering temporal ordering from visual semantics, avoiding trivial solutions that rely on chronological inputs. Instead of using the VLM as a scalar reward model, RTA defines a correlation-based reward function, calculating the Spearman rank correlation between predicted progress rankings and true temporal indices. This design provides a bounded, scale-invariant learning signal, facilitating stable transfer across tasks and environments. The framework is evaluated on both discrete and continuous control benchmarks, demonstrating that RTA consistently matches or outperforms existing video-based reward learning methods and rank-based baselines. The results indicate that correlation-structured supervision over ordinal signals is sufficient for effective policy learning, presenting a scalable alternative to explicit reward design.
Methodology
RTA consists of two stages: first, training a Vision-Language Model (VLM) as a progressive ordinal estimator using Group Relative Policy Optimization (GRPO) on shuffled video segments. Second, it defines a reinforcement learning signal based on the Spearman rank correlation between predicted ordinal progress and true temporal indices within a sliding window, enabling reward-free policy learning.
Results
RTA was evaluated on discrete control tasks (PyBoy: Catrap, Kirby) and continuous control tasks (PointMaze, MetaWorld), consistently matching or outperforming prior video-based reward learning methods and rank-based baselines. The results highlight the effectiveness of the correlation-based reward signal in facilitating control policy learning without explicit rewards.
Implications
The findings suggest that RTA can be applied in environments where reward design is challenging or unavailable, such as retro games and real-world robotics. The framework's ability to learn from video alone could lead to advancements in generalist agents and broader applications in reinforcement learning.
Stable Global Weighting of Flow Mixtures using Simplex Exponential Moving Average
Generative Models
Optimization
Theory
- Introduces a two-stage framework for variational inference using normalising flows.
- Employs a Simplex Exponential Moving Average for stable global weighting of flow mixtures.
- Demonstrates improved performance on various posterior benchmarks compared to existing methods.
- Decouples expert training from mixture optimization, enhancing stability and efficiency.
Read more
Stable Global Weighting of Flow Mixtures using Simplex Exponential Moving Average
Summary
This paper introduces AMF-VI-sEMA, a novel two-stage framework for variational inference using normalising flows, which addresses the limitations of existing mixture-based flow formulations. The framework employs a stable global weighting mechanism based on a Simplex Exponential Moving Average (sEMA) update. In the first stage, a diverse set of flow architectures (RealNVP, MAF, RBIG) are trained independently to specialize in different structural regimes. In the second stage, the parameters of these experts are frozen, and global mixture weights are learned through a temperature-controlled softmax of average log-likelihoods, followed by a smooth EMA update. This approach allows for a data-agnostic gating mechanism that reallocates capacity adaptively without the need for per-sample gating or gradient backpropagation through weights. The authors evaluate the framework on ten posterior benchmarks, demonstrating that AMF-VI-sEMA consistently outperforms its predecessor and avoids the catastrophic transport failures seen in single-flow baselines, while maintaining stable weight trajectories with minimal computational overhead.
Methodology
The AMF-VI-sEMA framework consists of two stages: first, independent training of multiple flow architectures to specialize in different distributional structures; second, freezing expert parameters and learning global mixture weights through a temperature-controlled softmax of average log-likelihoods, followed by a smooth EMA update on the probability simplex.
Results
The empirical evaluation shows that AMF-VI-sEMA achieves consistent improvements in negative log-likelihood (NLL) over its predecessor, AMF-VI, and avoids the catastrophic transport failures of single-flow models. The framework maintains stable weight trajectories across all datasets tested, with Neff values greater than 1.4, while incurring minimal computational overhead.
Implications
The proposed framework has significant implications for Bayesian inference and variational approximation, particularly in applications requiring accurate representation of multimodal posteriors. Its architecture-agnostic nature allows for broad applicability across various domains in machine learning.
Black-Box Inference of LLM Architectural Properties with Restrictive API Access
Large Language Models
NLP
Theory
- Introduces NightVision, a method for inferring LLM architectural properties under restrictive API access.
- Demonstrates that hidden dimensions can be estimated without access to logit biases or top-k logits.
- Empirical evaluation shows NightVision can recover hidden dimensions with 23% average relative error.
- Depth and parameter count can be estimated with 53% average relative error for models with over 3 billion parameters.
Read more
Black-Box Inference of LLM Architectural Properties with Restrictive API Access
Summary
This paper addresses the challenge of inferring architectural properties of large language models (LLMs) when access to their APIs is restricted. Previous research demonstrated that certain architectural details could be recovered with limited API access, but recent changes by LLM providers have further restricted this access. The authors introduce 'NightVision', a novel attack method that estimates key architectural parameters such as hidden dimension, depth, and parameter count using only minimal API outputs (single logit probabilities) and timing measurements. NightVision employs a common set prompting technique to recover hidden dimensions and a timing-based approach to estimate depth and parameter count. The authors empirically validate NightVision on 32 open-source LLMs, achieving an average relative error of 23% for hidden dimensions and 53% for depth and parameter count in larger models. The findings suggest that even with restrictive APIs, significant architectural details can still be inferred, highlighting potential vulnerabilities in current API designs.
Methodology
The methodology involves a two-pronged approach: first, using common set prompting to recover hidden dimensions from single-logit outputs; second, employing timing measurements to estimate depth and parameter count based on the scaling of inference costs with model architecture. The authors also provide theoretical bounds on the number of API calls required for effective inference.
Results
NightVision successfully recovers hidden dimensions with an average relative error of 23% across 32 tested LLMs, achieving exact recovery in 4 cases and within 10% in 12 cases. For models with over 3 billion parameters, depth and parameter count are estimated with an average relative error of approximately 53%. The accuracy of these estimates is shown to depend on the token budget and model properties.
Implications
The ability to infer architectural properties from restricted API access raises concerns for LLM providers regarding the security of their intellectual property. It suggests that further measures, including defenses against timing side channels, may be necessary to protect sensitive model details.
Scaling Laws for Grid-Based Approximate Nearest Neighbor Search in High Dimensions
Theory
Efficient ML
- Multiprobe grid algorithm shows favorable scaling properties in high-dimensional spaces.
- Identified a unique d-scaling crossover where grid-based methods outperform others in throughput.
- Near-linear scaling with dataset size (N) and lower indexing costs compared to traditional ANN methods.
- Theoretical models derived for query cost and recall provide insights into algorithm performance.
Read more
Scaling Laws for Grid-Based Approximate Nearest Neighbor Search in High Dimensions
Summary
This paper addresses the scaling behavior of grid-based approximate nearest neighbor (ANN) search algorithms, particularly focusing on a multiprobe grid algorithm. The authors systematically characterize the performance of this algorithm in relation to dataset size (N) and dimensionality (d). They identify a unique d-scaling crossover in the GloVe embedding family, where the multiprobe grid maintains a consistent dimensional scaling exponent, unlike other ANN methods that suffer from decreased throughput as dimensionality increases. The multiprobe grid algorithm demonstrates near-linear scaling with respect to dataset size while exhibiting lower indexing costs compared to competing methods. The findings suggest that grid-based methods can be advantageous in scenarios where high-dimensional data and indexing costs are critical factors. The paper also connects the scaling properties of ANN algorithms to the design of efficient transformer architectures, as self-attention mechanisms in transformers can be viewed as ANN operations. The authors provide a theoretical framework for understanding the cost and recall of the multiprobe grid algorithm, supported by empirical evaluations against various baseline ANN algorithms, revealing a log-linear relationship between query performance and recall.
Methodology
The authors developed a theoretical model for the multiprobe grid algorithm, which involves PCA-reducing the dataset to a lower-dimensional space for cell selection, followed by re-ranking candidates in the original high-dimensional space. They conducted empirical evaluations using the ann-benchmarks framework, comparing their algorithm against established ANN methods like Voyager, PyNNDescent, Annoy, and FAISS-IVF.
Results
The multiprobe grid algorithm exhibited a log-linear relationship between query performance (QPS) and recall, indicating that performance is heavily influenced by grid geometry. Empirical results showed that the algorithm scales near-linearly with dataset size, achieving competitive performance against other ANN methods, particularly in high-dimensional settings.
Implications
The findings suggest that grid-based ANN methods, particularly the multiprobe grid, can be effectively utilized in high-dimensional data scenarios, such as those encountered in transformer architectures. This could lead to more efficient designs in machine learning systems that rely on ANN operations.
FlatManifold: Robust Continual Learning under Severe Label Noise and Domain Shifts via Intrinsic Manifold Flattening
Robotics
Optimization
Theory
- FlatManifold effectively mitigates the effects of severe label noise and domain shifts in continual learning.
- The framework utilizes a Nyström manifold flattening map for robust feature distribution mapping.
- It incorporates a continual topology brake term to prevent catastrophic forgetting.
- Extensive evaluations show superior performance compared to traditional continual learning methods.
Read more
FlatManifold: Robust Continual Learning under Severe Label Noise and Domain Shifts via Intrinsic Manifold Flattening
Summary
The paper presents FlatManifold, a novel framework for robust continual learning in non-stationary environments characterized by severe label noise and domain shifts. Traditional continual learning methods struggle with label noise, leading to catastrophic forgetting and overfitting to corrupted samples. FlatManifold addresses these challenges by employing a Nyström manifold flattening map that projects feature distributions into an orthogonalized Reproducing Kernel Hilbert Space (RKHS). This approach smooths the optimization process and mitigates the impact of label noise by maintaining a fixed target topology. Additionally, it incorporates a continual topology brake term to prevent catastrophic forgetting by leveraging the covariance matrix of past experiences. The framework was evaluated on real-world multi-session robotics datasets, demonstrating its effectiveness even under conditions with 40% symmetric label noise and extreme cross-session domain shifts. The results indicate that FlatManifold significantly outperforms standard sequential optimization baselines, showcasing the robustness of structural linearization against label corruption.
Methodology
FlatManifold employs a Nyström manifold flattening map based on the kernel trick to project feature distributions into an orthogonalized RKHS. This method facilitates the smoothing of gradients affected by label noise and incorporates a continual topology brake term to maintain historical knowledge.
Results
The framework was tested on multi-session robotics datasets, achieving significant performance improvements even with 40% label noise and under extreme domain shifts. FlatManifold outperformed standard sequential optimization methods, proving its robustness against label corruption.
Implications
FlatManifold has potential applications in autonomous robotics, particularly in environments with dynamic changes and noisy data. Its robust continual learning capabilities can enhance the performance of Visual Place Recognition systems in real-world scenarios.
MPSelectTune: Prompt-type Selection for Fine-tuning improves Concept Unlearning in LLMs
NLP
Large Language Models
- Introduces MPSelectTune for effective concept unlearning in LLMs.
- Utilizes a two-stage approach combining Multi-Prompt Tuning and Selection Tuning.
- Demonstrates significant improvements in main task accuracy while reducing concept accuracy.
- Addresses the impact of prompt variation on unlearning performance.
Read more
MPSelectTune: Prompt-type Selection for Fine-tuning improves Concept Unlearning in LLMs
Summary
This paper introduces MPSelectTune, a novel approach for concept unlearning in large language models (LLMs) that addresses the challenges posed by biased or harmful concepts present in pre-trained models. The authors highlight that existing unlearning methods often overlook the impact of prompt variation, which is crucial for LLMs that rely on prompts for task performance. MPSelectTune employs a two-stage methodology: the first stage, Multi-Prompt Tuning, utilizes multiple prompt types and a multi-task loss to fine-tune the model, while the second stage, Selection Tuning, focuses on fine-tuning using the worst-performing prompt type for concept prediction. This strategy aims to minimize the concept accuracy of the highest accuracy-prompt type, thereby enhancing the overall unlearning performance. Experimental results demonstrate that MPSelectTune achieves significant improvements in main task accuracy (2-15%) while effectively reducing the worst-case concept accuracy by up to 17% compared to recent baselines. The method also shows a substantial decrease in spurious correlations between task and concept prediction accuracies, indicating a more robust unlearning process.
Methodology
The proposed MPSelectTune framework consists of two main stages. In the first stage, Multi-Prompt Tuning, the model is fine-tuned using multiple prompt types and a multi-task loss that incorporates both task and concept predictions. The second stage, Selection Tuning, focuses on fine-tuning the model using the worst-performing prompt type for concept prediction, which is hypothesized to enhance the model's ability to unlearn biased concepts effectively.
Results
The experimental evaluation on five benchmark unlearning tasks reveals that MPSelectTune achieves a 2-15% increase in main task prediction accuracy while reducing the worst-case concept accuracy by up to 17% compared to existing methods. Additionally, the method significantly lowers the spurious correlation between task and concept prediction accuracies, demonstrating its effectiveness in concept unlearning.
Implications
The findings suggest that MPSelectTune can be a valuable tool for enhancing the safety and ethical compliance of LLMs by effectively unlearning harmful concepts. This approach can be applied in various domains where LLMs are used, particularly in sensitive applications such as hiring, law enforcement, and content moderation.