AI-generated summaries
Today's ML research,
without the noise.
Daily summaries of the latest machine learning papers from arXiv, processed every 8 hours.
24
Papers today
8h
Update frequency
7
Days of history
Literati: Towards Anytime Optimal Shape Generalized Trees via AO*
Optimization
Interpretability
Theory
- Introduces Literati, the first optimal SGT induction algorithm.
- Utilizes an AND/OR graph formulation to optimize tree structure and shape function complexity.
- Implements an AO*-based algorithm with enhancements for improved anytime performance.
- Demonstrates superior accuracy compared to existing decision tree methods across multiple datasets.
Read more
Literati: Towards Anytime Optimal Shape Generalized Trees via AO*
Summary
The paper presents Literati, a novel algorithm for optimal Shape Generalized Tree (SGT) induction, addressing the limitations of traditional decision tree methods that often produce suboptimal and complex structures. While existing SGT induction algorithms are greedy and lack optimality guarantees, Literati formulates the SGT induction problem as an AND/OR graph search, allowing for the joint optimization of tree structure and shape function complexity. The authors develop an AO*-based algorithm enhanced with a secondary heuristic for OR-node selection and a round-robin policy for AND-node exploration, which improves anytime performance while ensuring optimality. The evaluation of Literati across 24 real-world datasets demonstrates its superiority over state-of-the-art tree induction methods in terms of both training and test accuracy, showcasing its potential for more efficient and interpretable decision-making in machine learning applications.
Methodology
The authors propose a novel AND/OR graph formulation for SGT induction, which allows for the simultaneous optimization of tree structure and shape function complexity. They develop an AO*-based search algorithm that incorporates a secondary heuristic for selecting OR-nodes and a round-robin exploration policy for AND-nodes, enhancing the algorithm's performance while maintaining optimality guarantees.
Results
Literati consistently outperforms state-of-the-art tree induction baselines in both training and test accuracy across 24 real-world datasets, demonstrating its effectiveness in producing optimal decision trees with improved expressivity and compactness.
Implications
The development of Literati has significant implications for machine learning applications that require interpretable models, particularly in fields where decision trees are favored for their transparency. By enabling optimal SGT induction, Literati can facilitate better decision-making processes in various domains, including finance, healthcare, and beyond.
Distribution-Consistent Inference for Dynamic Sparse Mixture-of-Experts
NLP
Large Language Models
Efficient ML
- Performance degradation in SMoE models under reduced expert activation is linked to increased output variance and scale.
- Layer-wise Distribution Alignment (LDA) is introduced as a lightweight, inference-time correction method.
- LDA effectively recovers performance lost due to distributional shifts while maintaining sparse inference efficiency.
- The method is applicable across different SMoE architectures and routing strategies, including dynamic expert pruning.
Read more
Distribution-Consistent Inference for Dynamic Sparse Mixture-of-Experts
Summary
This paper addresses the limitations of existing Sparse Mixture-of-Experts (SMoE) models, which typically use a fixed top-k expert selection policy that does not adapt to the varying complexity of input tokens. The authors identify that reducing the number of activated experts during inference leads to a distributional shift in the model's outputs, which negatively impacts downstream performance. To mitigate this issue, they propose a novel method called Layer-wise Distribution Alignment (LDA), which aligns the outputs of reduced-routing configurations with the reference distribution of the default top-k routing. LDA operates at inference time, leveraging layer-wise calibration statistics to correct the representations without requiring retraining or architectural changes. The authors demonstrate that LDA significantly improves the performance of SMoE models across various tasks and routing strategies, effectively balancing performance and computational efficiency.
Methodology
The authors conducted a systematic analysis of how SMoE representations change with varying numbers of activated experts. They developed LDA, which applies per-dimension moment corrections to align outputs from reduced expert activations with a reference distribution derived from the default routing configuration. This approach is implemented at inference time, utilizing calibration statistics without necessitating retraining.
Results
The implementation of LDA led to significant performance improvements across multiple benchmarks and routing strategies. For instance, in general knowledge and commonsense reasoning tasks, LDA improved average performance by up to 20% compared to models without LDA. The results indicate that LDA successfully mitigates the negative effects of reduced expert activation on model performance.
Implications
The findings suggest that LDA can enhance the efficiency and effectiveness of large language models that utilize sparse expert architectures. This has potential applications in various NLP tasks, particularly in scenarios where computational resources are limited, allowing for more adaptable and performant models in real-world applications.
Exact Degeneracy Under Balanced k-Shot Sampling: Consequences for Small-Sample Discriminant Analysis on LLM Embeddings
NLP
Large Language Models
Theory
- Balanced k-shot sampling causes exact degeneracy in KLPCDA, affecting its discriminative power.
- Two KLPCDA variants have arbitrary eigenvector selection due to identical eigenvalues, while a third variant has a void objective.
- A logistic regression probe outperforms KLPCDA variants in most evaluated scenarios.
- Extending the support set from k ≤ 10 to k = 30–50 significantly improves accuracy.
Read more
Exact Degeneracy Under Balanced k-Shot Sampling: Consequences for Small-Sample Discriminant Analysis on LLM Embeddings
Summary
This paper investigates the effects of balanced k-shot sampling on small-sample discriminant analysis, particularly focusing on Kernelized Linear Principal Component Discriminant Analysis (KLPCDA) applied to embeddings from large language models (LLMs). The author demonstrates that balanced k-shot sampling induces a degeneracy in KLPCDA, where the within-class scatter operator becomes a scaled orthogonal projector. This leads to two variants of KLPCDA having identical signal eigenvalues, making their eigenvector selection arbitrary, while a third variant's objective collapses to zero. The paper provides a closed-form diagnosis of this issue and proposes a tie-breaking method to repair the affected variants. The evaluation of the repaired framework in few-shot text classification across four datasets reveals that a cross-validated logistic regression probe outperforms KLPCDA variants in three out of four datasets. Additionally, the study finds that extending the support set significantly improves performance, closing the accuracy gap with bidirectional encoders. The author also provides four quantified guidelines for practitioners and releases the estimator and evaluation harness for further research.
Methodology
The author analyzes the impact of balanced k-shot sampling on KLPCDA by deriving closed-form expressions for the within-class scatter operator. The study includes empirical evaluations on four datasets using various embedding sizes and compares KLPCDA against logistic regression and other baselines. The paper also introduces a tie-breaking mechanism to address the degeneracy issue.
Results
The findings indicate that KLPCDA does not outperform a cross-validated logistic regression probe in three out of four datasets. The proposed tie-breaking method repairs two KLPCDA variants, and extending the support set significantly reduces the accuracy gap with bidirectional encoders. The study also identifies that three geometric separability metrics fail to explain performance differences between embedding models.
Implications
The results suggest that practitioners should be cautious when using KLPCDA for few-shot classification with LLM embeddings, as traditional guidance from other domains may not apply. The findings also highlight the importance of increasing the number of labeled examples to enhance model performance.
A practical DIRECT-type algorithm for medium-scale black-box global optimization
Optimization
- Introduction of X-DTC-GL, a novel DIRECT-type algorithm for medium-scale optimization.
- Dynamic partitioning and hybridization techniques significantly enhance performance.
- Achieved approximately 12% improvement in solvability and 27% in solution quality over existing methods.
- Fastest convergence on 40% of benchmark instances and best runtime on 17% of problems.
Read more
A practical DIRECT-type algorithm for medium-scale black-box global optimization
Summary
This paper presents X-DTC-GL, a novel algorithm designed to enhance the performance of DIRECT-type global optimization methods, particularly for medium-scale black-box optimization problems. The traditional DIRECT algorithm, while effective for low-dimensional problems, suffers from slow convergence as dimensionality increases, limiting its applicability. To overcome this, the authors introduce dynamic partitioning and hybridization techniques. The dynamic partitioning method refines the search space adaptively using local one-dimensional surrogate models, allowing for efficient subdivision of promising hyper-rectangles. The hybridization strategy incorporates a hill-climbing method to exploit these promising regions. The authors conducted extensive experiments across four benchmark suites, demonstrating that X-DTC-GL outperforms existing DIRECT-type algorithms, achieving approximately 12% improvement in solvability and 27% in solution quality. Performance-profile analyses reveal that X-DTC-GL exhibits the fastest convergence on about 40% of instances and the best runtime performance on around 17% of problems, solidifying its competitiveness in the realm of black-box optimization.
Methodology
The methodology involves the development of the X-DTC-GL algorithm, which utilizes dynamic partitioning to refine the search space based on local surrogate models and a hybridization strategy that employs hill-climbing to exploit promising regions. The algorithm is tested against various benchmark suites to evaluate its performance.
Results
The results indicate that X-DTC-GL significantly outperforms existing DIRECT-type algorithms, with improvements of approximately 12% in solvability and 27% in solution quality. The algorithm also shows superior convergence rates and runtime performance in a substantial number of benchmark instances.
Implications
The advancements presented in this paper could lead to more efficient and robust algorithms for black-box optimization tasks across various fields, including engineering design, machine learning, finance, and drug discovery. The improved performance of X-DTC-GL may facilitate its application in complex optimization scenarios where traditional methods struggle.
A Later Test Set Is Not a New Domain: Pretraining Familiarity Survives a Contamination-Free Hold-Out
Time Series
- Introduces a contamination-free evaluation protocol for time-series models.
- Finds that pretrained models perform better in familiar domains but not universally.
- Identifies that seasonal strength and spectral entropy do not explain performance differences.
- Demonstrates that familiarity with a domain persists beyond temporal hold-outs.
Read more
A Later Test Set Is Not a New Domain: Pretraining Familiarity Survives a Contamination-Free Hold-Out
Summary
This paper addresses the evaluation of time-series foundation models, which are often assessed on public datasets that predate their training, leading to ambiguity between generalization and memorization. To mitigate this issue, the authors propose a contamination-free hold-out protocol, where test sets are constructed from observations that were not available during model training. The study evaluates thirteen forecasting models across five domains, revealing that pretrained models outperform classical methods in some areas but not in others. Notably, the advantage of pretrained models is linked to their familiarity with the domain of the test data rather than intrinsic properties of the time series itself. The findings suggest that benchmarks should include domain-specific hold-outs to accurately assess model performance and that practitioners should consider the training domain of models when selecting forecasting methods.
Methodology
The authors constructed a hold-out test set consisting of observations published after the release of the models being evaluated. They compared the performance of thirteen forecasting models, including classical and pretrained models, across seven forecasting groups from five domains. The evaluation focused on metrics such as Mean Absolute Scaled Error (MASE) and employed statistical tests to assess significance.
Results
Pretrained models achieved superior performance in five out of seven groups, with the largest gain observed in forecasting weekly Wikipedia pageviews, where they outperformed classical methods by 28% lower MASE. However, in some cases, classical baselines like the Theta model performed better. The study found no correlation between seasonal strength or spectral entropy and model performance, indicating that familiarity with the training corpus played a significant role.
Implications
The findings suggest that practitioners should be cautious when interpreting model performance based on standard benchmarks, as familiarity with the training data can skew results. The study advocates for the development of domain-specific hold-outs in evaluations to provide a clearer understanding of model capabilities and limitations.
SCCM : Stream Cruise Control Method for Automated Drift Detection and Adaptation
Time Series
Optimization
Theory
- SCCM provides a unified framework for drift detection and adaptation in online regression.
- The method enables early-response drift detection and quantifies drift magnitude for better adaptation.
- SCCM dynamically tunes hyperparameters, enhancing model adaptability to evolving data distributions.
- The framework is evaluated on diverse datasets, showing improved performance over traditional methods.
Read more
SCCM : Stream Cruise Control Method for Automated Drift Detection and Adaptation
Summary
The paper introduces the Stream Cruise Control Method (SCCM), a novel framework designed for automated drift detection and adaptation in online regression tasks. Concept drift, where the underlying data distribution evolves over time, poses significant challenges for predictive modeling. Traditional methods often rely on fixed hyperparameters and reactive adaptation strategies, which can lead to performance degradation before corrective measures are implemented. SCCM addresses these issues by providing an early-response mechanism for drift detection, quantifying drift magnitude, and dynamically tuning hyperparameters in real-time. The framework operates as a pre-update control layer, allowing it to monitor predictive performance and adapt before the model processes new data. SCCM is model-agnostic and can be integrated with various online regression models. The evaluation of SCCM on 18 synthetic datasets and eight real-world datasets demonstrates its effectiveness in handling different types of drift, including abrupt, incremental, and alternating gradual drift, while significantly improving predictive performance compared to existing detector-adaptation baselines.
Methodology
SCCM operates as a pre-update control layer that continuously monitors predictive performance, detects potential drift, quantifies its magnitude, and adjusts hyperparameters accordingly before the model updates with new data. It integrates drift detection, drift-magnitude quantification, and adaptive hyperparameter control into a single process, allowing for real-time adaptability.
Results
The evaluation of SCCM on 18 synthetic datasets and eight real-world datasets showed that it significantly outperformed eight baseline detector-adaptation methods in terms of predictive performance, effectively managing various types of concept drift.
Implications
SCCM's ability to adaptively manage concept drift in real-time makes it suitable for applications in dynamic environments such as traffic management, financial forecasting, and autonomous systems, where timely and accurate predictions are critical.
A positive resolution of the gap-entropy conjecture
Theory
- The paper provides a positive resolution to the gap-entropy conjecture for best-arm identification.
- It establishes a tight bound on the expected number of samples needed to identify the best arm, incorporating both gap and entropy terms.
- The results extend previous work by removing restrictions on instances and algorithms, making the findings more broadly applicable.
- The authors demonstrate that the gap-entropy relationship is crucial for understanding sampling costs in adaptive sampling scenarios.
Read more
A positive resolution of the gap-entropy conjecture
Summary
This paper addresses the gap-entropy conjecture in the context of fixed-confidence best-arm identification for independent unit-variance Gaussian arms. The authors prove that the optimal expected number of samples required to identify the best arm, averaged over all permutations of arm labels, is within constant factors of a quantity that combines the sum of the inverse squared gaps (H) and an entropy term (Ent(I)). They establish that for any algorithm that identifies the optimal arm with high probability, the expected sample size is bounded by H(log(1/δ) + Ent(I)), with an additional term related to the gap to the closest competitor. The authors also demonstrate that their results hold for a broader class of algorithms and instances than previously considered, effectively resolving the conjecture positively. This work significantly advances the understanding of the relationship between sampling costs and the structure of gaps in best-arm identification problems.
Methodology
The authors utilize theoretical analysis to derive bounds on the expected sample sizes required for best-arm identification. They define key quantities such as the sum of the inverse squared gaps (H) and the gap entropy (Ent(I)), and analyze their relationships to establish the main results. The proofs involve careful consideration of the structure of gaps and the performance of various algorithms under the Gaussian model.
Results
The authors prove that the optimal expected number of samples for identifying the best arm is of the form H(log(1/δ) + Ent(I)), where H captures the baseline cost of distinguishing the best arm and Ent(I) measures the distribution of sampling effort across scales. They also show that an algorithm can achieve this bound uniformly across all instances, with an additional term related to the gap to the closest competitor.
Implications
The findings have significant implications for adaptive sampling strategies in multi-armed bandit problems, particularly in scenarios where the gaps between arms are not uniform. The results can inform the design of more efficient algorithms for best-arm identification, potentially leading to faster convergence and reduced sample complexity in practical applications.
NEXUS-MI: Communication-Aware Federated Personalization for Gateway-Coordinated Motor-Imagery Brain-Computer Interfaces
Federated Learning
- NEXUS-MI addresses the challenges of EEG variability in MI-BCIs through a federated learning framework.
- The framework incorporates communication-aware synchronization to optimize both accuracy and communication costs.
- Evaluation showed a 42% reduction in communication traffic while maintaining cohort-level accuracy.
- Subject-level performance revealed vulnerabilities, highlighting the need for robust personalization strategies.
Read more
NEXUS-MI: Communication-Aware Federated Personalization for Gateway-Coordinated Motor-Imagery Brain-Computer Interfaces
Summary
The paper presents NEXUS-MI, a novel framework for federated personalization in motor-imagery brain-computer interfaces (MI-BCIs) that addresses the challenges of variability in EEG signals across subjects and sessions. Traditional federated learning approaches often overlook the complexities of intermittent gateway availability and synchronization, which are critical in real-world applications. NEXUS-MI formulates synchronization as a coupled learning-and-communication control problem, allowing for local processing of raw EEG data while maintaining a shared model backbone through an edge coordinator. The framework was evaluated using two datasets: BCI Competition IV Dataset 2a and OpenBMI, focusing on the impact of communication-aware synchronization on decoding accuracy and communication costs. Results indicated that the communication-aware approach reduced server-to-client traffic by approximately 42% while maintaining cohort-level accuracy. However, subject-level performance varied, revealing vulnerabilities that could lead to significant accuracy losses. The findings emphasize the importance of considering gateway synchronization as a design variable in federated MI personalization, advocating for a holistic evaluation of personalized accuracy, communication costs, and update freshness.
Methodology
The authors developed the NEXUS-MI framework, which integrates a gateway-coordinated federated learning approach. They conducted offline evaluations using two datasets, focusing on the effects of different synchronization policies on decoding accuracy and communication efficiency. The methodology included paired subject-level comparisons and robustness assessments through hierarchical bootstrap techniques.
Results
NEXUS-MI demonstrated a significant reduction in communication traffic (approximately 42%) while maintaining cohort-level accuracy across both datasets. However, subject-level analysis revealed potential accuracy losses of up to 12 percentage points compared to an ideal-link reference, indicating variability in performance that necessitates further investigation.
Implications
The findings suggest that effective gateway synchronization is crucial for the deployment of reliable MI-BCI systems in real-world settings. This research could inform the design of future neurorehabilitation technologies that leverage federated learning to enhance user experience and performance.
ALIGN-HOLD: Experience Alignment for Real-Time Hold Control in Large-Scale Ride-Hailing Matching at DiDi
Reinforcement Learning
Optimization
- ALIGN-HOLD improves hold control in ride-hailing systems by learning from implicit marketplace preferences.
- The framework constructs preference pairs from multiple data views to enhance reward modeling.
- A 28-day A/B test showed significant improvements in trip completion rates and driver income.
- The system effectively reduces passenger cancellations before and after driver acceptance.
Read more
ALIGN-HOLD: Experience Alignment for Real-Time Hold Control in Large-Scale Ride-Hailing Matching at DiDi
Summary
The paper presents ALIGN-HOLD, a novel framework designed for real-time hold control in large-scale ride-hailing systems, specifically implemented at DiDi. The primary goal of hold control is to selectively defer driver-order pairs to optimize matching opportunities, thereby enhancing the overall passenger-driver experience. Traditional systems like EXHOLD rely on handcrafted reward structures that combine various trip-related metrics, which can be challenging to calibrate due to the heterogeneous nature of marketplace preferences and the noisy feedback from user interactions. ALIGN-HOLD shifts the paradigm from manual reward engineering to experience alignment by learning hold policies directly from implicit marketplace preferences. It constructs complementary preference pairs from order and driver trajectories, as well as local matching graphs, and employs a Reward Model (RM) trained through balanced multi-view sampling and model-adaptive hard preference sampling. The RM provides context-dependent rewards during policy learning, enabling the system to filter out low-identifiability interactions. The framework was deployed in a 28-day A/B test, covering around 100,000 passenger requests daily, and demonstrated significant improvements in trip completion rates and driver income, while also reducing passenger cancellations. The results validate the effectiveness of the proposed components and the framework's applicability in real-world scenarios.
Methodology
ALIGN-HOLD utilizes a multi-view sampling approach to construct preference pairs from order and driver trajectories, along with local matching graphs. It trains a Reward Model (RM) using model-adaptive hard preference sampling, which guides contextual-bandit policy learning by providing context-dependent rewards.
Results
The deployment of ALIGN-HOLD resulted in statistically significant improvements in trip completion rates and driver income, while also reducing passenger cancellations before and after driver acceptance. The framework's components were validated through complementary ablation studies and diagnostics.
Implications
The successful implementation of ALIGN-HOLD suggests that experience alignment can effectively enhance real-time decision-making in dynamic marketplaces, potentially leading to better user experiences and operational efficiencies in ride-hailing and similar platforms.
EFQ-Softmax: Exp-Free Quantization for Softmax
NLP
Large Language Models
Efficient ML
- EFQ-Softmax eliminates the high-precision exp-then-quantize step in softmax computations.
- The method directly generates low-bit probability codes from shifted attention scores.
- Significant improvements in task performance metrics across various models.
- Reduces vector-stage latency by over 40% on the A5 vector unit.
Read more
EFQ-Softmax: Exp-Free Quantization for Softmax
Summary
The paper introduces EFQ-Softmax, a novel low-bit probability-generation method designed to enhance the efficiency of softmax computations in Transformer models. Traditional methods often compute softmax probabilities using high-precision exponentials before quantization, leading to inefficiencies due to a mismatch between high-precision probability producers and low-bit matrix consumers. EFQ-Softmax addresses this by directly mapping shifted attention scores to block-scaled E2M1 operands, eliminating the need for intermediate high-precision representations. The method employs a single affine rule to generate nonnegative E2M1 probability codes from shifted scores, maintaining the integrity of the online attention mechanism while improving computational efficiency. The authors evaluate EFQ-Softmax across various tasks, including language and vision-language workloads, demonstrating its effectiveness in maintaining model quality while significantly reducing latency in probability generation. The results indicate that EFQ-Softmax can replace the conventional exp-then-quantize path without compromising performance, thus offering a promising approach for optimizing Transformer inference.
Methodology
The EFQ-Softmax method involves selecting an exponent-only scale from local maximums, mapping shifted scores to a normalized residual domain, and generating E2M1 probability codes using a single affine rule. This approach integrates seamlessly into the existing online attention framework, preserving essential updates and normalizations.
Results
EFQ-Softmax improved the Qwen3-8B seven-task mean score from 0.6749 to 0.6773 and the Qwen3-VL nine-task mean from 0.7826 to 0.8000. It also maintained visual quality on the WAN2.2 model comparable to FP16 and MXFP4 baselines while reducing vector-stage latency by 40.33% across sequence lengths from 16K to 128K.
Implications
The findings suggest that EFQ-Softmax can significantly enhance the efficiency of Transformer models, particularly in applications requiring low-latency inference, such as real-time language processing, vision-language tasks, and video generation. This method could lead to broader adoption of low-bit quantization techniques in various AI applications.
Deep Neural Networks for Learning Intent from sEMG Signals to Support Hardware Devices for Post-Stroke Neurorehabilitation
Robotics
Time Series
Efficient ML
- Developed a reproducible workflow for decoding finger intent from sEMG signals in stroke patients.
- Compared the performance of LSTM, CNN, and GNN architectures for multi-label intent prediction.
- Identified a compact CNN model suitable for real-time hardware integration.
- Demonstrated improved performance through cross-channel knowledge distillation.
Read more
Deep Neural Networks for Learning Intent from sEMG Signals to Support Hardware Devices for Post-Stroke Neurorehabilitation
Summary
This paper addresses the challenge of decoding finger-specific motor intent from high-density surface electromyography (sEMG) signals in post-stroke neurorehabilitation. The authors utilize a dataset called PhysioMio, which includes recordings from stroke patients, to develop a robust machine learning framework for predicting multi-finger intent. The study employs various deep learning architectures, including Long Short-Term Memory (LSTM), Convolutional Neural Networks (CNN), and Graph Neural Networks (GNN), to evaluate their performance in intent decoding. The results indicate that while LSTM achieves the highest subset accuracy, GNN excels in macro F1 and AUPRC metrics. The authors also conduct an architecture search, identifying a compact CNN model suitable for embedded systems, which is retrained to work with a reduced number of channels compatible with hardware devices. The final model demonstrates improved performance through cross-channel knowledge distillation, establishing a pathway for integrating sEMG-based intent prediction into rehabilitation hardware. This work highlights the importance of a hardware-aware approach in developing machine learning solutions for clinical applications.
Methodology
The study involved a systematic approach to processing sEMG signals, including label alignment, filtering, denoising, and feature extraction. Various deep learning models (LSTM, CNN, GNN) were trained and evaluated on the PhysioMio dataset. An architecture search was conducted to identify the best-performing CNN model, which was then optimized for a four-channel input suitable for embedded hardware. Cross-channel knowledge distillation was applied to enhance model performance.
Results
The LSTM model achieved a subset accuracy of 0.545, while the GNN model reached a macro F1 score of 0.706 and a macro AUPRC of 0.776. The optimized CNN-Large model attained a subset accuracy of 0.593 and a macro F1 of 0.714. The compact CNN-Micro model, after retraining and knowledge distillation, achieved a subset accuracy of 0.5219 ± 0.0114 and a macro F1 of 0.6095 ± 0.0058.
Implications
The findings suggest that machine learning models can effectively decode motor intent from sEMG signals in stroke rehabilitation, paving the way for the development of responsive assistive devices. The approach can enhance the usability of rehabilitation technologies by ensuring they are compact and efficient enough for real-time applications.
Accountable and uncertainty-aware evaluation of sensor-based AI under distribution shift: devices, subjects, and nearly three years underground
Time Series
Theory
Interpretability
- Introduction of a staged evaluation protocol for sensor-based AI that accounts for distribution shifts.
- Demonstration of the protocol on geomagnetic localization in underground mines, highlighting real-world applicability.
- Quantification of performance degradation over time and under changing conditions, emphasizing the limitations of traditional evaluation methods.
- Use of quantile-based decision rules to better assess deployment risks compared to mean-based reporting.
Read more
Accountable and uncertainty-aware evaluation of sensor-based AI under distribution shift: devices, subjects, and nearly three years underground
Summary
This paper addresses the challenges of evaluating sensor-based AI systems, particularly in the context of distribution shifts that occur when models are deployed in real-world scenarios. The authors propose a novel evaluation protocol that incorporates accountability and quantifies uncertainty, treating model evaluation as a measurement with defined reference levels. The protocol consists of four cumulative generalization stages that hold out devices, subjects, and time, allowing for a more robust assessment of model performance. The authors demonstrate this protocol through a case study on geomagnetic localization using smartphones in two underground mines, where they re-evaluate unchanged models on data collected 34 months post-training. The results highlight the inadequacy of traditional evaluation methods, revealing significant performance degradation due to distribution shifts. The paper emphasizes the importance of quantile-based decision rules over mean-based assessments, showcasing how the proposed protocol can provide a clearer understanding of deployment risks and performance reliability.
Methodology
The authors developed a staged evaluation protocol that systematically holds out devices, subjects, and recording epochs to assess model performance under distribution shifts. This protocol includes quantiles of repeated trainings against chance references and employs an out-of-present-scope rate to identify misdirection in class predictions. The methodology was applied to infrastructure-free geomagnetic localization using smartphone-based recurrent classifiers, with extensive training runs conducted across two different underground mine sites.
Results
The evaluation protocol revealed that the 5% quantile of present-conditioned precision was 0.39 over 299 repeated trainings, significantly exceeding chance levels. The study also identified a bimodal configuration that passed mean-based tests but had a 5% quantile well below chance, illustrating the risks of relying on mean performance metrics. The analysis of performance degradation over 34 months indicated that class loss, annotation ambiguity, and genuine signature changes contributed to the observed decline in model accuracy.
Implications
The proposed evaluation protocol has significant implications for the deployment of sensor-based AI systems, particularly in environments where conditions can change dramatically over time. By providing a more accountable and uncertainty-aware framework for evaluation, this work can enhance the reliability of AI systems in critical applications such as indoor localization, search and rescue operations, and other scenarios where traditional evaluation methods may fail.
From Fixed Keys to Readable Schemas: Small Language Models for Vehicle Agent Function Calls
NLP
Large Language Models
Efficient ML
- Introduces a benchmark for vehicle function calling with a focus on seen, unseen, and out-of-scope requests.
- Compares Functional Token and Schema-in-Prompt representations across various small language models.
- Demonstrates that Schema-in-Prompt allows for better generalization to unseen functions compared to Functional Tokens.
- Finds that model scaling offers limited benefits for seen functions, with optimal performance at 0.6B parameters.
Read more
From Fixed Keys to Readable Schemas: Small Language Models for Vehicle Agent Function Calls
Summary
This paper investigates the use of small language models (SLMs) for translating natural-language requests into vehicle function calls, focusing on two representation methods: Functional Tokens (FT) and Schema-in-Prompt (SIP). The authors introduce a benchmark comprising 9,822 examples across 79 vehicle functions derived from Android Automotive, which includes both seen and unseen functions. The study reveals that while FT allows for compact inference, it fails to generalize to unseen functions, achieving zero accuracy on held-out functions. In contrast, SIP can generalize effectively, improving performance with model scaling, albeit at the cost of increased memory usage and latency. The paper emphasizes that the representation of the function surface significantly influences the capabilities and limitations of SLMs in vehicle function calling, rather than model size alone. The authors provide a theoretical analysis explaining the differences in generalization and inference costs associated with the two approaches, highlighting the trade-offs between flexibility and computational efficiency.
Methodology
The authors constructed a vehicle function-calling benchmark from Android Automotive specifications, categorizing functions into seen, unseen, and out-of-scope requests. They conducted controlled experiments comparing the FT and SIP representations across SLMs ranging from 270M to 1.7B parameters, analyzing function selection, refusal rates, latency, and memory usage.
Results
The results indicate that FT performs poorly on unseen functions, achieving zero accuracy, while SIP shows significant improvement with scale. For seen functions, the performance of smaller models can match that of larger ones, with the best results observed at 0.6B parameters. SIP also demonstrates better handling of out-of-scope requests by refusing appropriately, unlike FT, which may invoke unavailable functions.
Implications
The findings suggest that for in-vehicle assistants, the choice of function representation is crucial for effective natural language understanding and action execution. This research could inform the design of more efficient and flexible vehicle control systems, enhancing user experience and safety.
Uncertainty-Aware Sea-Ice Type Mapping with Multiple Ice Charts
Computer Vision
- Multi-annotator label uncertainty arises from subjective interpretations of sea-ice conditions by different ice services.
- The study quantifies both annotation and model uncertainty, revealing their interrelationship.
- Incorporating multiple annotator information improves model predictions, especially near the ice edge.
- Soft supervision leads to higher correlation between model uncertainty and multi-annotator disagreement.
Read more
Uncertainty-Aware Sea-Ice Type Mapping with Multiple Ice Charts
Summary
This paper addresses the challenges of sea-ice stage of development (SoD) mapping using deep learning, focusing on the uncertainties inherent in operational ice charts. These charts, created by trained analysts interpreting satellite data, often reflect subjective judgments and can vary between different ice services, leading to multi-annotator label uncertainty. The authors propose a methodology to quantify both annotation uncertainty and model uncertainty, examining their relationship. They find that incorporating information from multiple annotators improves the correlation between model uncertainty and multi-annotator disagreement, particularly near the ice edge. The study highlights the importance of treating uncertain annotations appropriately during model training to avoid misleading confidence in predictions. The results indicate that soft supervision can enhance model performance, achieving a correlation of 0.704 in areas of high disagreement. Monte Carlo dropout is identified as the most effective method for uncertainty estimation, yielding well-calibrated confidence estimates.
Methodology
The authors developed a framework to quantify multi-annotator label uncertainty and model uncertainty. They analyzed the relationship between these uncertainties by comparing predictions from deep learning models with varying levels of supervision, particularly focusing on soft supervision. The study utilized Monte Carlo dropout for uncertainty estimation.
Results
The study found that soft supervision improved the correlation between model uncertainty and multi-annotator disagreement, achieving a maximum correlation of 0.704 near the ice edge. Monte Carlo dropout was shown to provide the best calibration for confidence estimates, with an expected calibration error of 0.050.
Implications
The findings suggest that better handling of uncertainty in sea-ice mapping can enhance operational monitoring and decision-making in polar navigation. The methodology can be applied to other domains where expert annotations are subject to variability, improving the robustness of machine learning models.
Privacy-Preserving Split Learning for Federated LLM Fine-Tuning
Large Language Models
Federated Learning
- Introduction of PrivPair, a framework for privacy-preserving federated LLM fine-tuning.
- Utilization of lightweight adapters for obfuscating and recovering smashed data.
- Demonstrated strong privacy protection against reconstruction attacks.
- Maintains a standard federated learning workflow with minimal utility loss.
Read more
Privacy-Preserving Split Learning for Federated LLM Fine-Tuning
Summary
This paper addresses the challenges of fine-tuning large language models (LLMs) in a federated learning (FL) setting while ensuring data privacy. The authors introduce a novel framework called PrivPair, which combines split learning (SL) and federated learning to allow participants to collaboratively train LLMs without sharing raw data. The key innovation of PrivPair is a learned obfuscate-and-recover mechanism that employs lightweight adapters on the client side to obfuscate intermediate activations sent to the server, thus preventing the leakage of sensitive input data. The server processes these obfuscated activations and returns them to the client, where they are recovered for training. The proposed method effectively mitigates the privacy risks associated with the autoregressive nature of LLMs, which can leak input data through intermediate activations. Experimental results demonstrate that PrivPair achieves strong privacy protection with only modest utility loss and system overhead, making it a practical solution for federated LLM fine-tuning.
Methodology
The authors propose a two-adapter mechanism within the PrivPair framework. The first adapter obfuscates the smashed data before it is sent to the server, while the second adapter recovers the data after processing by the server. This approach allows the server to train a complete model without accessing raw data, thus preserving privacy.
Results
The experiments conducted show that PrivPair effectively protects against data reconstruction attacks while incurring only modest utility loss and computational overhead. The framework allows for the deployment of a complete model on the server side, maintaining the integrity of the federated learning process.
Implications
The proposed method has significant implications for industries that handle sensitive data, such as healthcare and finance, enabling them to collaboratively fine-tune LLMs without compromising privacy. It opens avenues for more secure federated learning practices in various applications.
Strangers to Themselves: What Language Models Say About Themselves Is Generic
NLP
Large Language Models
Theory
- Language models exhibit weak self-knowledge, with low correlation between self-reports and actual behavior.
- Predictions about a model's behavior are often as accurate when based on generic AI descriptions or other models' behaviors.
- Finetuning on a model's own behavior improves predictions but alters the behavior being predicted.
- First-person framing leads to biased, overly flattering self-reports from models.
Read more
Strangers to Themselves: What Language Models Say About Themselves Is Generic
Summary
This paper investigates the self-knowledge of language models by examining how well they can predict their own behavior in various scenarios. The authors conducted nine behavioral evaluations to compare the predictions made by the models about their own actions against predictions made using generic descriptions of AI agents and other models' behaviors. The findings reveal that self-reports from language models are largely generic and do not provide reliable insights into their specific behaviors. The correlation between a model's self-prediction and its actual behavior was found to be weak (r = +0.04), with slight improvements when models were shown their own history or specific evaluation items, but still not surpassing the predictive power of general AI descriptions or other models' assessments. The study also highlights that while finetuning on a model's own behavior can improve narrow self-predictions, it alters the behavior being predicted and does not generalize across different contexts. Overall, the results suggest that language models possess a theory of AI behavior rather than a true self-model, and their self-descriptions often reflect a bias towards more favorable reporting.
Methodology
The authors employed a series of behavioral evaluations to measure how well language models could predict their own actions under various conditions. They compared self-predictions against predictions derived from generic AI descriptions and other models' behaviors. The methodology included varying the information available to the models, the subject of the questions, and the form of the questions asked.
Results
The study found that self-reports from language models had a weak correlation with actual behavior (r = +0.04), which improved only modestly with additional context. Predictions based on generic AI agent descriptions performed similarly well (r = +0.28), and other models' self-reports were more predictive of the target model's behavior than its own (r = +0.35). Finetuning improved narrow self-predictions but did not generalize across different behaviors.
Implications
The findings suggest that developers should be cautious when relying on language models' self-descriptions for understanding their behavior. The results indicate a need for better evaluation methods that do not depend solely on self-reports, as they may not accurately reflect a model's capabilities or tendencies.
Applying foundation model embeddings towards urban livability evaluation
Multimodal
- Introduces a framework for evaluating urban livability using foundation model embeddings.
- Analyzes the impact of different geospatial data types on livability predictions.
- Demonstrates improved prediction performance through the use of embeddings.
- Addresses the challenge of data scarcity in socioeconomic indicator measurement.
Read more
Applying foundation model embeddings towards urban livability evaluation
Summary
This paper addresses the challenge of accurately measuring socioeconomic indicators in data-scarce regions, which hampers effective policy interventions and resource allocation. The authors propose a systematic framework that utilizes foundation model embeddings, specifically AlphaEarth, AnySat, and TerraMind, to identify predictive geospatial indicators relevant to urban livability. By analyzing the influence of various geospatial data types on livability predictions, the framework aids researchers in prioritizing the most informative features for their applications. The study demonstrates how these embeddings can enhance prediction performance while accounting for complex spatial dependencies. The methodology not only extracts actionable insights from satellite imagery but also provides a principled approach to urban livability evaluation, particularly in areas with limited observational data. The paper emphasizes the integration of diverse data sources through geospatial embeddings, which can significantly reduce data engineering overhead and improve the accuracy of livability assessments.
Methodology
The authors build upon a transformer-based architecture to incorporate various foundation model embeddings. They analyze the interactions of these embeddings with different data modalities and conduct interpretability analyses to understand their influence on the model's decision-making processes.
Results
The study finds that using foundation model embeddings significantly enhances the predictive performance for urban livability outcomes. The systematic framework allows for the identification of the most informative geospatial indicators, facilitating better resource allocation and policy interventions in urban planning.
Implications
This research has significant implications for urban planning and policy-making, particularly in regions with limited data. By leveraging high-resolution geospatial data and foundation model embeddings, stakeholders can make informed decisions to improve urban livability and address the challenges posed by rapid urbanization.
An Exponential Deterministic--Randomized Gap in ERM-Oracle Complexity for Thresholds on an Unknown Order
Theory
- Demonstrates a clear separation in performance between deterministic and randomized learners in online learning with oracles.
- Establishes that deterministic learners face a lower bound of M + Q ≥ T - ε, while randomized learners achieve O(log T) expected calls and mistakes.
- Highlights the role of the oracle's selection rule as a complexity parameter in addition to the learning class and oracle interface.
- Provides insights into the necessity of randomization in overcoming adversarial oracle responses.
Read more
An Exponential Deterministic--Randomized Gap in ERM-Oracle Complexity for Thresholds on an Unknown Order
Summary
This paper investigates the complexity of online learning with oracles, specifically focusing on the transductive online learning of thresholds on an unknown total order of instances. The author addresses a question posed by Attias, Hanneke, and Ramaswami regarding whether randomization can reduce the number of oracle calls needed for effective learning. The main contribution is a demonstrated gap in performance between deterministic and randomized learners when using a fixed natural oracle, specifically the minimal-prefix rule. The results show that deterministic learners incur a combined cost of mistakes and oracle calls that is at least T - ε, while randomized learners can achieve O(log T) expected calls and mistakes. The paper also explores the implications of the oracle's selection rule on learning complexity, revealing that the selection behavior of the oracle is a critical complexity parameter. The findings suggest that randomization provides a significant advantage in scenarios where the oracle's responses are adversarial or extremal, highlighting the importance of oracle design in online learning frameworks.
Methodology
The paper employs theoretical analysis to establish bounds on the performance of deterministic and randomized learners under specific oracle rules. It utilizes the minimal-prefix and maximal-prefix rules as benchmarks to derive lower and upper bounds on mistakes and oracle calls, and it examines the implications of oracle selection behavior on learning complexity.
Results
The main results indicate that for deterministic learners, the total cost of mistakes and oracle calls is at least T - ε, while randomized learners can achieve O(log T) in both metrics. Additionally, the paper shows that the oracle's selection rule significantly influences the complexity of learning, with deterministic learners suffering under adversarial conditions.
Implications
These findings have implications for the design of online learning algorithms and the development of oracle interfaces. Understanding the gap between deterministic and randomized approaches can inform algorithm selection in practical applications, especially in scenarios where oracle responses may be adversarial.
Explaining f-Divergence-Based Regularization via Local Curvature and Sharpness-Aware Minimization
Theory
Optimization
- Establishes a formal connection between divergence-based regularization and Sharpness-Aware Minimization (SAM).
- Demonstrates that both methods induce curvature-sensitive penalties in parameter space.
- Validates the analysis using the asymmetric α-skew Jensen-Shannon divergence (JSD) family.
- Shows that stronger curvature penalization is associated with flatter local minima and better predictive performance.
Read more
Explaining f-Divergence-Based Regularization via Local Curvature and Sharpness-Aware Minimization
Summary
This paper explores the relationship between divergence-based regularization and Sharpness-Aware Minimization (SAM) in the context of improving generalization in deep learning. The authors derive a unified local second-order expansion for f-divergence regularization, demonstrating that both methods induce curvature-sensitive penalties in parameter space. They show that divergence regularization yields a Fisher-weighted quadratic form, while SAM penalizes sharpness through the dominant Hessian eigenvalue. The analysis reveals that the local geometric perspective extends to input-space perturbations, where divergence-based regularization induces a pullback quadratic form, offering a broader perturbation framework than standard SAM. The authors validate their findings using the asymmetric α-skew Jensen-Shannon divergence (JSD) family, showing that stronger curvature penalization correlates with flatter local minima and improved predictive performance across benchmark datasets. This work provides a deeper understanding of the geometric underpinnings of regularization techniques and their implications for lifelong learning agents.
Methodology
The authors utilize a theoretical framework based on local second-order geometry to analyze f-divergence regularization and its relationship with SAM. They derive a unified local expansion for f-divergence regularization and conduct empirical studies using the asymmetric α-skew Jensen-Shannon divergence to validate their theoretical findings.
Results
The empirical results indicate that models exhibit better accuracy and negative log-likelihood near the regime of maximal curvature penalization. Loss-landscape visualizations reveal that stronger curvature penalization corresponds to flatter local minima, supporting the theoretical analysis.
Implications
This research has significant implications for the design of regularization techniques in deep learning, particularly in contexts requiring robust generalization, such as lifelong learning agents. Understanding the geometric properties of regularization methods can lead to improved model performance and adaptability to changing environments.
An Explainable Machine Learning Framework for Predicting Blood-Brain Barrier Permeability Using Molecular Descriptors
Interpretability
- Developed an explainable machine learning framework for predicting BBB permeability.
- Utilized molecular descriptors from the MoleculeNet BBBP dataset with RDKit.
- Optimized XGBoost model achieved high predictive performance metrics.
- Identified key molecular descriptors influencing BBB permeability predictions.
Read more
An Explainable Machine Learning Framework for Predicting Blood-Brain Barrier Permeability Using Molecular Descriptors
Summary
This study presents an explainable machine learning framework aimed at predicting blood-brain barrier (BBB) permeability, a crucial factor in the development of central nervous system (CNS) therapeutics. The framework utilizes molecular descriptors derived from the MoleculeNet BBBP dataset, employing the RDKit cheminformatics toolkit. A total of 15 physicochemical descriptors from 2,039 compounds were analyzed using four supervised machine learning algorithms: Logistic Regression, Support Vector Machine (SVM), Random Forest, and Extreme Gradient Boosting (XGBoost). Hyperparameter optimization was conducted using GridSearchCV, and model interpretability was assessed through SHapley Additive exPlanations (SHAP). The optimized XGBoost model exhibited superior predictive performance, achieving an accuracy of 88.97%, precision of 88.92%, recall of 97.76%, F1-score of 93.13%, and ROC–AUC of 0.9282. The robustness of the model was further validated through stratified five-fold cross-validation, yielding a mean ROC–AUC of 0.8982 ± 0.0130. Key molecular descriptors identified as influential in predicting BBB permeability included TPSA, HBD, and LogP. Overall, the proposed framework offers a reliable, interpretable, and efficient computational tool for early-stage screening of CNS drug candidates.
Methodology
The study employed four supervised machine learning algorithms (Logistic Regression, SVM, Random Forest, and XGBoost) to predict BBB permeability using molecular descriptors. Hyperparameter optimization was performed using GridSearchCV, and model interpretability was assessed using SHAP. The dataset consisted of 2,039 compounds with corresponding binary labels indicating BBB permeability.
Results
The optimized XGBoost model achieved an accuracy of 88.97%, precision of 88.92%, recall of 97.76%, F1-score of 93.13%, and ROC–AUC of 0.9282. Stratified five-fold cross-validation resulted in a mean ROC–AUC of 0.8982 ± 0.0130, demonstrating the model's robustness. Key molecular descriptors identified were TPSA, HBD, and LogP.
Implications
The framework provides a computationally efficient and interpretable approach for predicting BBB permeability, which can significantly aid in the early-stage screening of CNS drug candidates, potentially accelerating drug discovery processes.
EEGBind: Detecting Source-Level Interictal Epileptiform Discharges via EEG-Centric Multimodal Binding
Multimodal
Time Series
- EEGBind is an EEG-centric multimodal binding framework for source-level IED classification.
- The framework effectively integrates synchronized video context to enhance classification accuracy.
- A view-consistent repair strategy is introduced to improve robustness against unstable candidate contexts.
- EEGBind achieved a weighted-F1 score of 0.8395, ranking first in the NeuroMM 2026 Grand Challenge.
Read more
EEGBind: Detecting Source-Level Interictal Epileptiform Discharges via EEG-Centric Multimodal Binding
Summary
The paper presents EEGBind, a novel framework designed for the source-level classification of interictal epileptiform discharges (IEDs) using an EEG-centric multimodal binding approach. IEDs are critical for epilepsy diagnosis and treatment planning, as they help identify the origin of epileptiform activity. Traditional methods often struggle with the subtlety and variability of EEG signals, particularly when distinguishing between different source regions. EEGBind addresses these challenges by treating EEG as the primary modality and integrating synchronized video context to enhance classification accuracy. The framework employs a binding mechanism that utilizes ST-EEGFormer as the EEG backbone, while a view-consistent repair stage improves robustness against unstable video contexts. The authors evaluate EEGBind on the NeuroMM 2026 Grand Challenge Track 3 benchmark, achieving a weighted-F1 score of 0.8395, outperforming other methods. This work highlights the effectiveness of EEG-centric multimodal binding in improving source-level IED classification, suggesting that meaningful integration of auxiliary modalities can enhance clinical decision-making in epilepsy management.
Methodology
EEGBind employs a binding mechanism that uses ST-EEGFormer as the EEG backbone, integrating synchronized video context to provide auxiliary evidence for classification. The framework includes a view-consistent repair stage that refines multimodal representations while preserving the learned source-class boundaries, addressing the modality-role mismatch between EEG and video data.
Results
On the NeuroMM 2026 Grand Challenge Track 3 benchmark, EEGBind achieved a weighted-F1 score of 0.8395, outperforming competing methods and demonstrating the effectiveness of the EEG-centric multimodal binding approach for source-level IED classification.
Implications
The findings suggest that integrating multimodal data, particularly video context, can significantly enhance the accuracy of source-level classifications in clinical settings, potentially improving presurgical evaluations and treatment planning for epilepsy patients.
One Loop, Two Gains: Can Active Learning win the Lottery for Free?
Efficient ML
Computer Vision
Theory
- The Improve & Prune (I&P) method integrates magnitude pruning into active learning cycles, reducing computational overhead.
- I&P can produce sparse models that match the accuracy of dense models while achieving high sparsity levels.
- The method alleviates key bottlenecks in deep active learning, making it more scalable for large datasets.
- Empirical results show that winning tickets can emerge under non-stationary conditions typical of active learning.
Read more
One Loop, Two Gains: Can Active Learning win the Lottery for Free?
Summary
This paper explores the intersection of deep active learning (DAL) and the lottery ticket hypothesis, which suggests that sparse subnetworks (winning tickets) can be found within larger networks. The authors propose a novel method called Improve & Prune (I&P) that integrates iterative magnitude pruning into the active learning retraining cycles, allowing for the discovery of winning tickets without incurring additional computational costs. The study investigates whether I&P can produce winning tickets in the non-stationary data environment typical of active learning. The authors conduct experiments across various acquisition functions, architecture families, and image classification datasets, demonstrating that I&P can yield sparse models that maintain the accuracy of their dense counterparts while achieving up to 95% sparsity. This approach addresses significant computational challenges in DAL, particularly in large-scale settings, by reducing the costs associated with model retraining and acquisition scoring.
Methodology
The authors developed the Improve & Prune (I&P) method, which combines iterative magnitude pruning with deep active learning. The method operates within the existing active learning framework, allowing for pruning at each iteration without additional training costs. The pruning mask is dynamically adjusted to accommodate the non-stationary nature of the data, enabling the reactivation of previously pruned parameters when necessary.
Results
The experiments conducted across six acquisition functions and four image classification datasets revealed that I&P consistently produced sparse models that achieved comparable accuracy to their dense counterparts. The models maintained high performance even at sparsities of up to 95%, effectively demonstrating the emergence of winning tickets throughout the active learning process.
Implications
The findings suggest that integrating pruning into active learning can significantly enhance the efficiency of model training in scenarios with limited labeled data. This approach has potential applications in fields requiring rapid model deployment under resource constraints, such as medical imaging and industrial analysis.
Learning with Covariance Matrices: Principal Component Analysis Meets Learning with Graphs
Graph Learning
Theory
Computer Vision
- PCA is limited by instability and inefficiency in multiscale data processing.
- The paper introduces coVariance neural networks (VNNs) as an extension of PCA using graph signal processing.
- VNNs provide improved stability and reproducibility in learning outcomes compared to traditional PCA.
- The connection between PCA and graph convolutional filters enhances the theoretical understanding of both methods.
Read more
Learning with Covariance Matrices: Principal Component Analysis Meets Learning with Graphs
Summary
This paper explores the integration of Principal Component Analysis (PCA) with graph signal processing (GSP) to enhance learning from covariance matrices. Covariance matrices are crucial for understanding linear dependencies in multivariate datasets, but traditional PCA faces challenges such as instability due to finite sample sizes and inefficiency in handling multiscale data. The authors propose a novel approach that leverages the graph representation of covariance matrices, allowing for a more robust and theoretically grounded learning framework. By connecting PCA to graph convolutional filters, the paper introduces coVariance neural networks (VNNs), which maintain the benefits of PCA while addressing its limitations. The theoretical analysis demonstrates that VNNs can achieve stability and reproducibility in learning outcomes, making them a promising alternative to conventional PCA-based methods. This work lays the groundwork for future advancements in machine learning applications across various domains, including neuroimaging and environmental modeling.
Methodology
The authors develop a theoretical framework that connects PCA with graph signal processing, specifically through the use of covariance matrices as graph representations. They analyze the stability of VNNs and establish conditions for reproducibility in learning outcomes, contrasting these with traditional PCA methods.
Results
The theoretical results indicate that VNNs can achieve guaranteed stability and reproducibility in learning outcomes across independently collected datasets, addressing the limitations of PCA in terms of sample-induced stochastic perturbations.
Implications
The findings suggest that VNNs could serve as a more reliable and efficient alternative to PCA in various machine learning applications, particularly in fields requiring the analysis of multiscale data. This could lead to advancements in areas such as neuroimaging, environmental modeling, and other domains that rely on covariance matrix analysis.
Fine-Tuning a KV Cache Concatenation-Aware Model or Recomputing KV Caches? Why Not Both?
NLP
Large Language Models
Efficient ML
- Combines fine-tuning and selective KV cache recomputation for improved response quality.
- Demonstrates that existing methods inadequately handle long-context inputs.
- Achieves a 9.7-point improvement in RULER score for 124k-token inputs.
- Reduces time to first token (TTFT) by 80% compared to full attention.
Read more
Fine-Tuning a KV Cache Concatenation-Aware Model or Recomputing KV Caches? Why Not Both?
Summary
This paper addresses the challenge of improving response quality in Retrieval-Augmented Generation (RAG) systems when handling long input contexts. As the input context length increases, the computational workload for generating responses also rises, leading to longer time to first token (TTFT). The authors propose a novel approach that combines model fine-tuning with selective recomputation of key-value (KV) caches to enhance accuracy and efficiency. The study reveals that existing methods struggle to maintain response quality as context length grows, highlighting the need for improved techniques. The proposed method effectively reduces deviations in KV cache entries, resulting in a significant improvement in accuracy for long-context inputs. Experiments conducted on the RULER benchmark demonstrate that the combined approach yields a 9.7-point increase in RULER score over baseline methods that only recompute KV caches, while also achieving an 80% reduction in TTFT compared to full attention mechanisms.
Methodology
The authors propose a dual approach that involves fine-tuning the model with consideration for KV cache concatenation and selectively recomputing a subset of KV caches. This method aims to mitigate the accuracy degradation observed in long-context scenarios by effectively managing the deviations in KV cache entries.
Results
The proposed method resulted in a 9.7-point improvement in the RULER score for a 124k-token input compared to a baseline that only recomputed KV caches. Additionally, it achieved an 80% reduction in TTFT when KV caches were loaded from SSD, demonstrating significant efficiency gains.
Implications
This research has potential implications for enhancing the performance of RAG systems, particularly in applications requiring the processing of extensive context inputs. The findings suggest that combining model fine-tuning with selective cache recomputation can lead to more efficient and accurate language generation models.