AI-generated summaries
Today's ML research,
without the noise.
Daily summaries of the latest machine learning papers from arXiv, processed every 8 hours.
24
Papers today
8h
Update frequency
7
Days of history
MAG: MAnifold Guided Semi-Supervised Multi-modal In-Context Learning
Multimodal
Large Language Models
Graph Learning
- MAG effectively addresses label scarcity in few-shot multi-modal ICL by utilizing unlabeled data.
- The framework employs a two-stage strategy for demonstration selection, enhancing efficiency and relevance.
- Textual representations are crucial for initial relevance propagation, while both modalities are needed for final selection.
- MAG demonstrates substantial performance gains across diverse benchmarks, particularly in reasoning-intensive tasks.
Read more
MAG: MAnifold Guided Semi-Supervised Multi-modal In-Context Learning
Summary
The paper introduces MAG (MAnifold Guided semi-supervised in-context demonstration selection), a novel framework aimed at enhancing few-shot in-context learning (ICL) with multi-modal large language models (MLLMs). The authors address the challenge of label scarcity in ICL, which limits the effectiveness of demonstration selection. MAG leverages abundant unlabeled multi-modal data through a semi-supervised approach, formulating demonstration selection as a propagation problem on a multi-modal graph. The framework consists of two stages: the first stage employs relevance score propagation to identify a compact set of high-impact unlabeled samples for pseudo-labeling, thereby reducing inference costs; the second stage selects the final demonstrations using multi-modal relevance. The authors find that textual representations are more effective for relevance propagation, while both visual and textual modalities are essential for high-quality demonstration selection. Experiments across eight multi-modal benchmarks show that MAG outperforms strong baselines in label-scarce scenarios, achieving significant improvements with limited pseudo-labeling budgets.
Methodology
MAG employs a two-stage framework that utilizes graph-based relevance score propagation to filter unlabeled samples for pseudo-labeling. The first stage identifies relevant unlabeled samples, while the second stage selects the most effective demonstrations from both labeled and pseudo-labeled data. This method balances scalability with demonstration quality, allowing efficient use of large unlabeled datasets.
Results
The experiments conducted on eight multi-modal benchmarks reveal that MAG consistently outperforms existing methods that rely solely on labeled data. The framework shows particularly pronounced improvements in tasks requiring complex cross-modal reasoning, demonstrating the effective incorporation of unlabeled data.
Implications
The findings suggest that leveraging unlabeled multi-modal data can significantly enhance the performance of ICL in scenarios with limited labeled data. This has potential applications in various fields, including visual emotion recognition, scene text understanding, and visual question answering, where labeled data is often scarce.
History-informed Lagrangian Neural Networks
Robotics
Time Series
Theory
- HiLNN infers hidden velocities and adapts system parameters from position-only observations.
- The framework utilizes a recurrent encoder to extract a latent context from historical position data.
- HiLNN employs a differentiable RK4 rollout scheme for optimized trajectory predictions.
- Empirical results show superior accuracy and physical consistency compared to traditional LNNs and other baselines.
Read more
History-informed Lagrangian Neural Networks
Summary
The paper presents History-informed Lagrangian Neural Networks (HiLNN), a novel framework aimed at improving the long-horizon forecasting of mechanical systems using only position observations. Traditional Lagrangian Neural Networks (LNNs) require complete state inputs, including velocity, which limits their applicability in scenarios where only position data is available. HiLNN addresses this limitation by leveraging the temporal position sequences to extract a latent context through a recurrent encoder. This context not only estimates the unobserved initial velocity but also adapts the mass matrix, potential energy, and damping coefficients of the Lagrangian system. The authors employ a differentiable RK4 rollout scheme for end-to-end optimization under multi-step trajectory supervision and energy-consistency regularization. Empirical evaluations demonstrate that HiLNN significantly enhances prediction accuracy and maintains energy consistency across various mechanical systems, including conservative, dissipative, and variable-parameter systems, outperforming existing state-of-the-art methods.
Methodology
The HiLNN framework uses a recurrent encoder to derive a latent context from historical position data, which is then used to condition a structured Lagrangian dynamics model. The model is optimized using a differentiable RK4 rollout scheme, allowing for end-to-end training under multi-step trajectory supervision and energy-consistency regularization.
Results
HiLNN demonstrated improved long-term prediction accuracy and maintained precise energy profiles across conservative, dissipative, and variable-parameter systems, outperforming traditional LNNs, Hamiltonian Neural Networks (HNN), Neural ODEs, and MLP predictors in empirical evaluations.
Implications
The development of HiLNN has significant implications for fields requiring accurate mechanical forecasting from limited observations, such as robotics, control systems, and scientific modeling. Its ability to adapt to varying dynamics and maintain physical plausibility can enhance the reliability of predictive models in real-world applications.
Federated Compositional Muon Optimizer for Matrix-Wise Models
Federated Learning
Optimization
- Introduction of FedCoMuon and FedCoMuon-VR optimizers for matrix-wise compositional optimization.
- Theoretical convergence analysis under non-convex and non-i.i.d. settings.
- FedCoMuon-VR achieves lower sample complexity than existing FedMuon algorithms.
- Extensive experiments demonstrate competitive performance and improved accuracy.
Read more
Federated Compositional Muon Optimizer for Matrix-Wise Models
Summary
This paper introduces the Federated Compositional Muon (FedCoMuon) optimizer, designed to address the challenges of distributed matrix-wise compositional optimization in federated learning settings. The authors identify a gap in existing Muon-based optimizers, which are primarily tailored for single-level stochastic objectives, and propose FedCoMuon to effectively handle hierarchical structured problems. The FedCoMuon optimizer integrates compositional gradient tracking with orthogonalized momentum, while a variance-reduced variant, FedCoMuon-VR, is also introduced, leveraging a momentum-based variance reduction technique. The paper provides a theoretical analysis of the convergence properties of both algorithms under non-i.i.d. and non-convex conditions, demonstrating that FedCoMuon-VR achieves a lower sample complexity of O(ϵ−3) for finding an ϵ-stationary solution compared to existing FedMuon algorithms. Empirical evaluations on robust federated learning and task-distributed risk-sensitive meta learning show that the proposed methods outperform existing compositional baselines, achieving superior accuracy in various scenarios.
Methodology
The FedCoMuon optimizer combines compositional gradient tracking with orthogonalized momentum to optimize matrix-wise models in a federated learning context. The FedCoMuon-VR variant incorporates a momentum-based variance reduction technique to enhance performance. The algorithms are analyzed theoretically for their convergence properties, focusing on sample and communication complexities.
Results
The proposed FedCoMuon and FedCoMuon-VR optimizers show significant improvements in sample complexity, with FedCoMuon-VR achieving O(ϵ−3) complexity for finding ϵ-stationary solutions. Numerical experiments indicate that these optimizers outperform existing compositional baselines, achieving the best reported accuracy in several settings, particularly in robust federated learning and task-distributed meta learning.
Implications
The development of FedCoMuon and FedCoMuon-VR has potential applications in privacy-sensitive and resource-constrained environments, such as mobile intelligence, healthcare, and recommendation systems, where federated learning is crucial. The ability to effectively optimize matrix-wise models in these contexts can lead to more robust and efficient AI systems.
When Local Variance Optimality Is Not Enough: RoPE-Aligned Q/K Rotations for Dynamic 4-Bit Quantisation
NLP
Large Language Models
Optimization
- The only commuting orthogonal maps for distinct RoPE frequencies are independent pairwise rotations.
- The derived rotation angle minimizes channel variance but does not improve quantization accuracy in practice.
- The head-shared pairwise configuration results in higher perplexity compared to full-head mixing.
- Estimating the shared angle from K alone improves performance but does not close the gap with full-head mixing.
Read more
When Local Variance Optimality Is Not Enough: RoPE-Aligned Q/K Rotations for Dynamic 4-Bit Quantisation
Summary
This paper investigates the effectiveness of rotation-based post-training quantization methods, specifically focusing on RoPE (Rotary Position Embedding) aligned Q/K rotations in the context of dynamic 4-bit quantization. The authors explore whether a transformation that respects the frequency pair decomposition of RoPE can enhance quantization accuracy compared to full-head mixing. They establish that for distinct frequencies, the only commuting orthogonal maps are independent pairwise rotations. The authors derive a rotation angle that minimizes channel variance under a pooled-covariance surrogate and verify that their implementation achieves this minimum. However, their experiments reveal that this head-shared pairwise configuration does not improve accuracy and, in fact, increases perplexity across various checkpoints. The findings suggest that optimality in a structured surrogate does not necessarily lead to reduced quantization error when misaligned with the quantizer's scale-setting statistic.
Methodology
The authors derive a closed-form rotation angle that minimizes the larger channel variance using a pooled-covariance, position-averaged surrogate. They conduct experiments on various checkpoints, comparing the performance of head-shared pairwise configurations against full-head Hadamard mixing under dynamic W4A4KV4 quantization.
Results
The experiments show that the head-shared pairwise configuration consistently increases perplexity across all evaluated contexts compared to full-head mixing. While estimating the shared angle from K improves performance, it does not eliminate the performance gap relative to full-head mixing. Additionally, broader mixing support is associated with lower quantization error and perplexity degradation.
Implications
These findings highlight the limitations of local variance optimality in quantization methods and suggest that careful consideration of mixing support and alignment with quantization statistics is crucial for improving model performance in low-bit quantization scenarios.
A Probe Direction Is a Property of Its Prompt
NLP
Large Language Models
Theory
- The choice of prompt significantly influences model evaluation scores.
- Reported scores can vary widely based on prompt wording, affecting perceived model performance.
- A single-prompt design is insufficient for reliable comparisons across models.
- The paper advocates for a multi-prompt evaluation framework to enhance reliability.
Read more
A Probe Direction Is a Property of Its Prompt
Summary
This paper investigates the reliability of model evaluations in machine learning, particularly focusing on how the choice of prompt influences the reported scores of models. The author critiques the standard method of evaluating models by contrasting activations from prompts that announce an evaluation against those that do not. The study reveals that the reported scores are significantly affected by the specific wording of the prompts used, rather than solely reflecting the model's capabilities. By conducting factorial experiments that vary prompt framing while keeping task text constant, the author demonstrates that the direction and magnitude of scores can change based on prompt choice. This variability indicates that the model's sense of being evaluated is not the sole determinant of the scores, as the prompt itself introduces a substantial amount of variance. The findings suggest that a single-prompt design is inadequate for reliable model comparisons, and the paper provides recommendations for a more robust evaluation framework that includes multiple prompts. The study also emphasizes the importance of transparency in reporting prompt choices in model evaluations.
Methodology
The author employs a factorial experimental design, varying prompt framing while keeping the task text constant. This approach allows for the isolation of the effects of prompt choice on model evaluation scores. The analysis utilizes generalizability theory to assess the variance components related to model performance and prompt framing.
Results
The results indicate that the reported scores are heavily influenced by the choice of prompt, with different framings leading to different trends in model performance. The correlation between model size and evaluation scores can change sign based on prompt selection, demonstrating that the prompt is a significant factor in the evaluation process. Additionally, the study finds that the variance attributed to the model itself is small compared to the variance introduced by prompt interactions.
Implications
The findings suggest that researchers should be cautious when interpreting model evaluation scores, as they may not accurately reflect model capabilities due to prompt influence. The proposed multi-prompt evaluation framework could lead to more reliable assessments of model performance, fostering better comparisons across different models and contributing to the development of more robust machine learning systems.
Beyond Simulated Benchmarks: Evaluating Motion Representations for Fall Detection Under Real-World Data Scarcity
Time Series
- Real-world fall detection is hindered by the extreme scarcity of actual fall data.
- Simulated datasets often lead to overestimated performance in laboratory settings.
- Interval-based representations achieve the best real-world performance, while symbolic representations with impact descriptors show robustness under data scarcity.
- The study emphasizes the importance of representation choice for generalization from simulated to real-world conditions.
Read more
Beyond Simulated Benchmarks: Evaluating Motion Representations for Fall Detection Under Real-World Data Scarcity
Summary
This paper addresses the critical issue of fall detection for older adults, emphasizing the challenges posed by the scarcity of real-world fall data. While wearable sensors have been explored for detecting falls, the rarity of actual fall events necessitates reliance on simulated datasets, which often lead to overestimated performance in controlled environments. The authors systematically evaluate various motion representations—interval-based, kernel-based, symbolic, and foundation model representations—using accelerometer signals from both simulated and clinically verified real-world datasets. They introduce a lightweight symbolic representation, FallLM, which combines symbolic tokens with physically-grounded impact descriptors to enhance interpretability. The study employs a streaming evaluation pipeline to assess the robustness of these representations under data scarcity and cross-dataset transfer. Results indicate that while complex models perform well on simulated data, they struggle with real-world applicability. The interval-based representation shows the best real-world performance, but the augmented symbolic representation demonstrates resilience against domain shifts and retains detection sensitivity under extreme data scarcity. This work highlights the necessity of evaluating motion representations beyond simulated benchmarks and underscores the importance of representation choice for effective fall detection in real-world scenarios.
Methodology
The authors conducted a systematic evaluation of different motion representations for fall detection using accelerometer data. They compared interval-based, kernel-based, symbolic, and foundation model representations through a unified streaming evaluation pipeline, employing subject-wise splits for realistic testing. The study utilized both simulated and clinically verified real-world datasets, focusing on cross-dataset transfer and robustness under data scarcity.
Results
The findings reveal that highly parameterized kernel and foundation models perform well on simulated data but degrade significantly when faced with real-world data scarcity and domain shifts. The interval-based representation achieved the highest absolute performance in real-world scenarios, while the augmented symbolic representation maintained detection sensitivity under extreme data scarcity, albeit with lower precision.
Implications
This research has significant implications for the development of fall detection systems, particularly in geriatric care and remote health monitoring. By emphasizing the importance of motion representation choice, the findings can guide future research and technology development to improve the reliability and effectiveness of fall detection in real-world applications.
Personalized Scorer Modeling: A Learning-Based Framework for Deriving Robust Sleep Stage Labels from Multiple Experts
Time Series
Multimodal
- Introduces a framework for improving sleep stage classification by addressing inter-scorer variability.
- Utilizes multi-scored datasets to derive more reliable sleep stage labels.
- Employs confusion matrices to model scorer-specific behavior and aggregate probabilities for labeling.
- Demonstrates improved classification metrics compared to traditional hypnograms.
Read more
Personalized Scorer Modeling: A Learning-Based Framework for Deriving Robust Sleep Stage Labels from Multiple Experts
Summary
This paper addresses the challenge of sleep stage classification, which is crucial for diagnosing and treating sleep disorders. Traditional methods often rely on a single reference hypnogram, which can lead to inaccuracies due to inter-scorer variability. The authors propose a novel framework called Personalized Scorer Modeling (PSM) that leverages multi-scored datasets, where multiple experts annotate the same polysomnographic recordings. By analyzing two publicly available datasets (DOD-H and DOD-O), the authors extract 30 features from EEG and EMG signals and develop a learning-based hypnogram (LBH) that models the behavior of each scorer using confusion matrices. These matrices are normalized to estimate the probability of each sleep stage based on the labels assigned by the scorers, which are then aggregated to produce a final label for each epoch. The performance of the LBH is evaluated using random forest, support vector machine, and multilayer perceptron classifiers, demonstrating significant improvements over traditional methods. The study concludes that personalized scorer modeling enhances the reliability of sleep stage labels without disregarding individual expert contributions.
Methodology
The study analyzes two multi-scored sleep datasets (DOD-H and DOD-O) and extracts 30 features from EEG and EMG signals. A learning-based hypnogram is generated by modeling the behavior of each scorer through confusion matrices, which are normalized to estimate stage-specific probabilities. These probabilities are aggregated to determine the final sleep stage label for each epoch. The proposed method is evaluated using various classifiers under different signal configurations.
Results
The proposed LBH achieved an accuracy of 86.07%, precision of 85.46%, and F1-score of 85.29% on the DOD-H dataset, and 86.04% accuracy, 85.21% precision, and 84.70% F1-score on the DOD-O dataset. These results indicate a consistent improvement over the original dataset hypnogram and the best-scorer hypnogram.
Implications
The findings suggest that personalized scorer modeling can enhance the construction of reference hypnograms in multi-scored sleep datasets, leading to more reliable automatic sleep staging systems. This approach could improve clinical practices and research in sleep medicine by providing more accurate sleep stage classifications.
Understanding Backdoor Vulnerabilities in Vertical Federated Learning: The Gap Between Research and Practice
Federated Learning
- Identifies a gap between theoretical research on backdoor attacks in VFL and practical applications.
- Highlights unrealistic assumptions in existing methodologies that lead to overestimated attack success rates.
- Introduces BVBench, a benchmark for fair evaluation of backdoor vulnerabilities in VFL.
- Recommends redefining threat models to align with realistic operational constraints.
Read more
Understanding Backdoor Vulnerabilities in Vertical Federated Learning: The Gap Between Research and Practice
Summary
This paper investigates the vulnerabilities associated with backdoor attacks in Vertical Federated Learning (VFL), highlighting a significant gap between theoretical research and practical application. VFL allows organizations to collaborate on machine learning tasks without sharing sensitive data, but this information asymmetry can be exploited by malicious participants. The authors argue that existing studies on backdoor attacks often rely on unrealistic assumptions and poorly designed evaluation practices, leading to an overestimation of the effectiveness of current defenses. To address these issues, the paper proposes a redefined threat model that accounts for realistic constraints and introduces BVBench, a benchmark for evaluating backdoor vulnerabilities in VFL. The findings reveal that many existing attack strategies may not be as effective in real-world scenarios, emphasizing the need for more robust defenses and a better understanding of practical vulnerabilities in VFL systems.
Methodology
The authors conducted a systematic review of existing literature on backdoor attacks in VFL, identifying methodological and evaluation design flaws. They proposed new threat models and developed BVBench to facilitate comprehensive and practical evaluations of backdoor vulnerabilities.
Results
The study found that existing attack strategies often rely on unrealistic assumptions, leading to inflated success rates in controlled environments. The introduction of BVBench revealed the fragility of current understandings of VFL backdoor risks, demonstrating that many attacks are less effective under realistic conditions.
Implications
The findings suggest that practitioners may be underestimating the risks associated with VFL systems, potentially leading to inadequate defenses. The proposed BVBench can serve as a foundation for future research, guiding the development of more effective security measures in federated learning environments.
Finding the Needle in a Haystack: Test-Time Analog Circuit Representation Adaptation for Bayesian Optimization
Optimization
- TTARO adapts circuit representations in real-time during the optimization process, improving alignment with the optimization objective.
- The framework is compatible with various acquisition functions and Gaussian-process kernels, making it versatile for different optimization scenarios.
- TTARO demonstrates significant performance improvements over traditional fixed-embedding BO methods and DKL in analog circuit topology searches.
Read more
Finding the Needle in a Haystack: Test-Time Analog Circuit Representation Adaptation for Bayesian Optimization
Summary
This paper presents Test-Time Analog Representation Adaptation for Bayesian Optimization (TTARO), a novel online framework that enhances Bayesian optimization (BO) for analog circuit topology search by adapting circuit representations during the optimization process. Traditional representation-based BO methods often use fixed embeddings, which can misalign with the optimization objective, leading to inefficient searches. TTARO addresses this by continuously updating circuit representations based on the figure of merit (FoM) of evaluated circuits. The framework jointly learns a nonlinear feature transformation and a Gaussian-process surrogate model as new FoM observations are gathered. The authors conduct extensive experiments across various circuit encoders, kernels, and acquisition functions, demonstrating that TTARO significantly outperforms conventional BO methods and Deep Kernel Learning (DKL) by aligning the search space with the optimization objective. The results indicate that TTARO reduces regret AUC by an average of 15.2% compared to standard BO and 20.7% compared to DKL, showcasing its effectiveness in optimizing analog circuit designs.
Methodology
TTARO employs an online deep-kernel Bayesian optimization approach that continuously refines circuit representations based on newly observed FoM labels. It integrates a nonlinear feature transformation with a Gaussian-process surrogate model, updating both after each evaluation to enhance the search process.
Results
In extensive experiments involving 160 configurations across two public circuit topology benchmarks, TTARO achieved an average reduction in regret AUC of 15.2% compared to conventional BO and 20.7% compared to DKL, with some configurations showing reductions as high as 46.7%. This indicates a substantial improvement in optimization efficiency.
Implications
The findings suggest that adaptive representation learning can significantly enhance the efficiency of optimization in complex design spaces, particularly in electronic design automation. This approach could be applied to other domains requiring optimization over structured, discrete spaces.
The data geometry of masking diffusion: Certified-optimal schedules via unmasking growth complexity
Theory
Generative Models
Efficient ML
- Introduction of unmasking growth complexity (UGC) as a measure of data geometry in masking diffusion.
- Establishment of a unified analysis framework for Bernoulli-subset and fixed-cardinality unmasking schemes.
- Development of certified-optimal samplers with high-probability guarantees on KL error.
- Demonstration of significant dimension-dependent improvements in sampling efficiency.
Read more
The data geometry of masking diffusion: Certified-optimal schedules via unmasking growth complexity
Summary
This paper investigates the concept of masking diffusion for discrete sampling, introducing a novel measure of data geometry termed unmasking growth complexity (UGC). The UGC provides a path-resolved analysis that directly influences the Kullback-Leibler (KL) discretization error, facilitating a unified examination of both Bernoulli-subset and fixed-cardinality unmasking schemes. By employing log-reveal-odds coordinates, the paper derives optimized sampling schedules that adapt computational resources according to the underlying data geometry. A significant contribution is the ability to estimate UGC increments from samples, leading to the development of certified-optimal samplers that achieve a specified KL error with high probability and maintain iteration complexity close to that of oracle procedures. The paper also connects the UGC mass to classical multivariate dependence measures and previous complexity analyses of discrete diffusion. In the fine-partition limit, the squared integral of the square-root UGC density is shown to dictate the leading-order optimal Euler discretization error. The findings demonstrate substantial improvements in sampling efficiency, particularly in high-dimensional settings, highlighting the potential for dimension-dependent gains over traditional coarse schedules.
Methodology
The methodology involves analyzing the unmasking growth complexity (UGC) through a path-resolved approach, estimating local increments from samples along coupled reveal trajectories. The paper derives optimized sampling schedules based on UGC, employing log-reveal-odds coordinates to adapt computational efforts to the data geometry.
Results
The main results include the derivation of certified-optimal sampling algorithms that achieve a specified KL error with high probability, as well as the identification of substantial improvements in sampling efficiency, particularly in high-dimensional spaces. The UGC framework also provides a refined understanding of the discretization error in relation to classical dependence measures.
Implications
The findings have significant implications for the design of efficient sampling algorithms in various applications, including machine translation, image synthesis, and generative AI. The ability to adaptively optimize sampling schedules based on data geometry can enhance performance in high-dimensional settings.
Defensive Boosting for Online Probabilistic Forecasting
Theory
Efficient ML
Optimization
- Introduces the Defensive Booster algorithm for online probabilistic forecasting.
- Achieves dual guarantees: competitive Brier scores and reduced classification error under weak-learning conditions.
- Utilizes a single weak-class learner for efficiency, unlike previous methods requiring multiple learners.
- Provides local hard-core certificates for weak-learning conditions, enhancing robustness.
Read more
Defensive Boosting for Online Probabilistic Forecasting
Summary
This paper addresses the challenge of online probabilistic forecasting of binary outcomes in the presence of an adaptive adversary. The authors propose a novel algorithm called the Defensive Booster, which combines the strengths of existing online boosting techniques to achieve two key guarantees: competitive Brier scores with the best predictor in the span of weak hypotheses and the ability to drive classification error to zero under a smooth weak-learning condition. The Defensive Booster operates efficiently by utilizing a single weak-class learner, contrasting with previous methods that require maintaining large ensembles of weak learners. The paper demonstrates that the Defensive Booster not only meets the performance benchmarks of online gradient boosting but also excels in scenarios where the weak-learning condition is satisfied, providing a robust framework for probabilistic forecasting in dynamic environments. Experimental results on synthetic and real data streams indicate that the Defensive Booster significantly outperforms prior baselines in predictive performance while achieving faster runtimes.
Methodology
The Defensive Booster algorithm is developed as a black-box reduction from an online learning algorithm for weak hypothesis classes. It operationalizes the dual view of boosting, utilizing mistake weights to certify the failure of the weak-learning condition when necessary. The algorithm is designed to be strongly adaptive, allowing it to maintain performance guarantees over contiguous intervals of time.
Results
The Defensive Booster achieves a Brier score competitive with the best predictor in the span of weak hypotheses on every adaptive sequence. When the smooth weak-learning condition holds, it guarantees both Brier score and randomized classification error to be minimized effectively. The algorithm also provides local hard-core certificates indicating when the weak-learning condition fails, enhancing its reliability. Experimental results show substantial improvements in predictive performance and runtime efficiency compared to existing methods.
Implications
The findings suggest that the Defensive Booster can be effectively applied in various online learning scenarios where adversarial conditions are present, such as financial forecasting, real-time decision-making systems, and adaptive risk management. Its efficiency and robustness make it a valuable tool for practitioners in fields requiring accurate probabilistic predictions.
Robust data-driven discovery of fractional differential equations via weak formulations and Pareto-based subset selection
Theory
- Weak-Pareto combines weak formulations with Pareto-based subset selection for discovering fractional differential equations.
- The method effectively mitigates noise amplification issues associated with fractional differentiation.
- Weak-Pareto demonstrates superior robustness and accuracy in recovering equations from noisy data compared to traditional methods.
- The framework allows for continuous-order optimization, avoiding the pitfalls of fixed-order dictionaries.
Read more
Robust data-driven discovery of fractional differential equations via weak formulations and Pareto-based subset selection
Summary
This paper presents a novel framework called Weak-Pareto for the data-driven discovery of fractional differential equations (FDEs) from noisy measurements. Traditional methods struggle with FDEs due to the amplification of high-frequency noise and the challenge of unknown derivative orders. Weak-Pareto addresses these issues by combining an adjoint-consistent weak formulation of fractional terms with a Pareto-based subset selection approach. The framework effectively transfers fractional operators from measured fields to smooth test functions, minimizing noise sensitivity. Coefficients are fitted using ridge regression within a differential-evolution search that optimizes the orders of the fractional derivatives. The method demonstrates robustness against noise, successfully recovering the correct support across various benchmarks, including fractional advection-diffusion and Burgers equations, even under significant noise levels. The results indicate that Weak-Pareto outperforms traditional strong-form methods, particularly in noisy environments, and achieves more consistent operator recovery with lower computational costs compared to neural network baselines. Additionally, a two-dimensional extension of Weak-Pareto is capable of recovering coordinate-dependent orders, showcasing its versatility in complex systems.
Methodology
The Weak-Pareto framework employs an adjoint-consistent weak formulation to derive fractional terms, replacing pointwise differentiation with smoothing integration. It utilizes ridge regression for coefficient fitting and a differential-evolution search for optimizing fractional orders. The Pareto-based subset selection ensures that only the most relevant terms are included, enhancing model parsimony and robustness.
Results
Weak-Pareto successfully recovers the correct structure of fractional differential equations from both clean and noisy measurements across various benchmarks. It retains accuracy even under high levels of multiplicative and additive noise, outperforming traditional strong-form methods and neural network approaches in terms of operator recovery consistency and computational efficiency.
Implications
The findings suggest that Weak-Pareto can be applied to various fields where fractional dynamics are relevant, such as physics, biology, and finance. Its robustness against noise makes it particularly useful for real-world applications where data is often imperfect. The ability to discover nonlocal dynamics could lead to better modeling of complex systems.
Towards Socially Compliant Navigation in Deep Reinforcement Learning via Proxemics-Based Reward Modeling
Reinforcement Learning
Robotics
- Introduction of a proxemics-based reward model for DRL navigation.
- Validation of the model across multiple DRL methods and crowd scenarios.
- Demonstrated improvements in social metrics without sacrificing navigation efficiency.
- Emphasis on the importance of comfort-aware navigation assessment.
Read more
Towards Socially Compliant Navigation in Deep Reinforcement Learning via Proxemics-Based Reward Modeling
Summary
This paper addresses the challenge of robot navigation in crowded environments, emphasizing the need for social compliance alongside task efficiency. Traditional deep reinforcement learning (DRL) methods often prioritize navigation tasks without adequately considering social interactions, leading to behaviors that may be collision-free but socially inappropriate. The authors propose a novel proxemics-based reward model that incorporates Hall's proxemics theory to define personal space as a radial Gaussian-mixture field. This model generates a robot-centric local cost that encourages robots to minimize proxemic intrusion while navigating efficiently. The proposed reward is integrated into established DRL navigation methods and evaluated through simulations across various crowd scenarios. The results demonstrate that the proxemics-based reward consistently enhances social metrics while maintaining competitive navigation performance compared to existing reward models. The paper also provides insights into reward design factors critical for improving social navigation in robotic systems.
Methodology
The authors developed a proxemics-based reward model grounded in Hall's theory of interpersonal distance, representing personal space as a radial Gaussian-mixture field. This model computes a local proxemic cost within the robot's field of view and is integrated into existing DRL navigation frameworks. The evaluation involved simulations across various crowd densities and scenarios, comparing the proposed reward against established baselines using both navigation and social metrics.
Results
The proposed proxemics-based reward model consistently improved social metrics in simulation experiments while maintaining competitive navigation performance compared to other reward formulations. The results highlighted the effectiveness of the model in promoting smoother and more socially compliant robot behaviors in dense crowds.
Implications
The findings suggest that incorporating social compliance into robot navigation systems can enhance human-robot interactions in crowded environments, making robots more acceptable and effective in real-world applications such as malls, hospitals, and airports. This work lays the groundwork for future research in socially aware robotic navigation.
Exploring Oversmoothing with Householder Matrices
Graph Learning
Theory
- Introduces Householder matrices as a method to combat oversmoothing in GNNs.
- Proves that HouseGNN preserves the Euclidean norm of node representations.
- Demonstrates scale and sign invariance of the Householder reflector.
- Shows that pairwise distances between nodes can vary with orthogonal transformations.
Read more
Exploring Oversmoothing with Householder Matrices
Summary
This paper addresses the issue of oversmoothing in deep graph neural networks (GNNs), where node representations converge towards a low-information subspace as the network depth increases. The author introduces the Householder Graph Neural Network (HouseGNN), which updates node embeddings using Householder reflections instead of the standard graph convolutional network (GCN) approach. By employing aggregated neighborhood messages to estimate a reflection direction, HouseGNN preserves the Euclidean norm at each node and depth, thereby mitigating oversmoothing. The paper proves three key properties of HouseGNN: (i) preservation of node-wise Euclidean norm at every internal layer, (ii) scale and sign invariance of the Householder reflector in the message, and (iii) the ability for pairwise distances between nodes to change due to mismatches in node-wise orthogonal operators. This novel approach offers a new perspective on combating oversmoothing in GNNs and enhances the discriminative power of node representations.
Methodology
The Householder Graph Neural Network (HouseGNN) employs Householder reflections to update node embeddings. Instead of directly modifying hidden states, it uses neighborhood messages to define a reflection direction, followed by a norm-preserving nonlinearity called GroupSort. This approach results in a piecewise orthogonal layer that maintains the Euclidean norm across all nodes and depths.
Results
The paper demonstrates that HouseGNN effectively mitigates the oversmoothing problem commonly faced by deep GNNs. The theoretical properties established show that the method maintains the discriminative variation of node representations, which is crucial for tasks such as node classification and link prediction.
Implications
The findings suggest that Householder transformations can be a powerful tool in enhancing the performance of GNNs, particularly in applications where maintaining the distinctiveness of node representations is critical. This could lead to improved outcomes in various graph-based learning tasks, including social network analysis, molecular property prediction, and more.
The Boolean Power of ReLU
Graph Learning
Theory
- ReLU-MPLang is strictly more expressive than TrReLU-MPLang for Boolean queries.
- The study resolves an open problem regarding the expressiveness of different activation functions in GNNs.
- Boolean queries derived from ReLU activations can express properties that are not expressible by truncated ReLU activations.
- The findings emphasize the importance of activation function selection in GNN architectures.
Read more
The Boolean Power of ReLU
Summary
This paper investigates the expressive power of graph neural networks (GNNs) using different activation functions, specifically focusing on the ReLU (Rectified Linear Unit) and truncated ReLU (TrReLU) functions. The authors prove that the Boolean queries expressible in the Σ-MPLang (a message-passing language for GNNs) are a strict subset of those expressible in ReLU-MPLang, thereby resolving an open question regarding the comparative expressiveness of these two languages in the context of Boolean queries. The study establishes that ReLU-GNNs are strictly more expressive than GNNs using TrReLU or identity activations when it comes to Boolean queries on graphs with Boolean features. The findings highlight the impact of activation function choice on the ability of GNNs to recognize properties of nodes after thresholding their outputs to Boolean values, demonstrating that the unbounded nature of ReLU not only influences numerical embeddings but also the Boolean characteristics that can be derived from them.
Methodology
The authors utilize a formal approach to define and compare the expressive powers of Σ-MPLang and ReLU-MPLang. They construct specific Boolean queries and demonstrate their non-definability in Σ-MPLang through a series of logical proofs and structural inductions, focusing on finite simple undirected graphs with a single Boolean node feature.
Results
The main result shows that there exists a Boolean ReLU query that cannot be expressed in Σ-MPLang for any collection of eventually constant activation functions. This establishes that bool(TrReLU-MPLang) is a strict subset of bool(ReLU-MPLang), confirming that ReLU-GNNs have superior expressive capabilities for Boolean queries.
Implications
These findings have significant implications for the design of GNN architectures, particularly in applications where Boolean properties of graphs are critical. The results suggest that using ReLU activations can enhance the ability of GNNs to capture and express complex node properties, which may be beneficial in various domains such as social network analysis, recommendation systems, and biological network modeling.
Perturbation-based Regional Interpretability through Subtraction Mapping (PRISM): naming-error dissociations in language models and post-stroke aphasia
Interpretability
Large Language Models
NLP
- PRISM adapts subtraction analysis from neuroimaging to interpret LLMs, providing a structured framework for mechanistic interpretability.
- The framework demonstrates that perturbation-induced error profiles in LLMs can be compared to lesion patterns in patients with aphasia.
- Both LLMs and aphasia patients show a robust phonemic-favoring dissociation, indicating shared cognitive processing patterns.
- The methodology allows for spatially resolved testing of functional specialization claims in LLMs.
Read more
Perturbation-based Regional Interpretability through Subtraction Mapping (PRISM): naming-error dissociations in language models and post-stroke aphasia
Summary
This paper introduces PRISM (Perturbation-based Regional Interpretability through Subtraction Mapping), a novel framework for interpreting large language models (LLMs) by applying subtraction analysis, a method commonly used in human neuroimaging, to perturbed transformers. The authors aim to bridge the interpretability gap in LLMs by providing a spatially resolved and falsifiable approach to test whether internal components of LLMs are specialized for distinct cognitive operations. Building on the Brain-LLM Unified Model (BLUM), which demonstrated that error profiles from a perturbed LLM correspond to lesion patterns in aphasic patients, PRISM maps the Philadelphia Naming Test categories and conducts pairwise subtractions of error classes. This analysis is run in parallel with a study of 213 chronic post-stroke aphasia patients, using correlation-difference lesion-symptom mapping. The results reveal a robust phonemic-favoring dissociation in both LLMs and human patients, indicating that deep layer clusters in the LLM and frontal-perisylvian cortical clusters in patients replicate consistently. The semantic-favoring direction shows a non-significant trend. PRISM thus offers a new method for testing functional specialization in transformer models, with further confirmatory interventions suggested for future work.
Methodology
The authors developed PRISM by applying subtraction analysis to LLM perturbation data and conducting parallel analyses on aphasia patients. They utilized the Philadelphia Naming Test to categorize errors and performed pairwise subtractions of these error classes. The analysis involved correlation-difference lesion-symptom mapping on patient data, with both LLM and patient analyses structured to match in subject and spatial dimensions.
Results
The study found a significant phonemic-favoring dissociation in both the LLM and the patient cohort, with consistent replication of deep layer clusters in the LLM and frontal-perisylvian cortical clusters in patients. The semantic-favoring direction was observed as a non-significant trend, suggesting further exploration is needed.
Implications
PRISM provides a new avenue for understanding the cognitive operations underlying LLMs, potentially leading to improved interpretability and trust in AI systems. It may also facilitate better alignment between AI models and human cognitive processes, with applications in both AI development and clinical neuroscience.
When Can You Trust Offline Evaluation of Equal-Cost Top-k Allocation? A Controlled, Reproducible Benchmark and Practitioner's Guide
Theory
- Weak overlap is primarily determined by logger-target action alignment rather than logging sharpness.
- Cross-fitting the outcome nuisance does not eliminate reuse bias; honest policy-level splitting is necessary.
- Propensity-estimation error is the most significant factor affecting performance in offline evaluations.
- The paper provides a reproducible benchmark and practical guidance for practitioners in the field.
Read more
When Can You Trust Offline Evaluation of Equal-Cost Top-k Allocation? A Controlled, Reproducible Benchmark and Practitioner's Guide
Summary
This paper addresses the challenges of offline evaluation in equal-cost top-k allocation, particularly focusing on the implications of weak overlap in logged data. The author benchmarks six estimators across five datasets and two known-effect sweeps, validating the mechanisms against a non-simulated paired reference. Key findings indicate that weak overlap is influenced more by logger-target action alignment than by logging sharpness alone. The study reveals that the optimizer's curse persists even with cross-fitting of outcome nuisance, necessitating honest policy-level splitting to mitigate reuse bias. Additionally, propensity-estimation error emerges as a significant factor degrading performance, with out-of-fold estimates adversely affecting Inverse Propensity Score (IPS) methods. The paper provides a comprehensive guide for practitioners on when offline evaluation can be trusted and how to select appropriate estimators, emphasizing the importance of understanding the logging environment and the implications of estimator choice.
Methodology
The study employs a benchmarking approach, testing six different estimators across multiple datasets while controlling for known effects. It integrates empirical evidence with theoretical analysis to assess the impact of logging strategies and estimator choices on evaluation accuracy. The author uses a non-simulated paired reference for validation and conducts a series of experiments to analyze the effects of various factors on overlap and estimator performance.
Results
The results indicate that the effective sample size (ESS) is significantly affected by the alignment of logger and target actions, with misalignment leading to poor overlap. The study finds that honest policy-level splitting reduces bias significantly compared to traditional cross-fitting methods. Additionally, the propensity estimation error is shown to be a critical factor that can invert overlap diagnostics, emphasizing the need for accurate propensity estimates in offline evaluations.
Implications
The findings have significant implications for practitioners involved in budget-constrained allocation decisions, as they provide insights into the reliability of offline evaluation methods. The guidance on logging design and estimator selection can help organizations make more informed decisions about deploying targeting rules based on logged data. This work also contributes to the broader understanding of off-policy evaluation in machine learning, particularly in contexts where accurate estimations of treatment effects are crucial.
Symmetry-Breaking De Novo Crystal Generation via Markovian Jump Diffusion
Generative Models
- Introduces a novel framework (SbCD) for generating complete crystallographic structures.
- Utilizes Markovian jump-diffusion to model symmetry-breaking dynamics.
- Outperforms existing models in generating crystals with full structural specifications.
- Addresses limitations of traditional methods that rely on empirical sampling of space groups.
Read more
Symmetry-Breaking De Novo Crystal Generation via Markovian Jump Diffusion
Summary
This paper presents a novel framework for generating crystalline structures, addressing limitations in existing generative models that fail to produce complete crystallographic specifications. The authors introduce the Symmetry-breaking Crystal Diffusion (SbCD) model, which utilizes a Markovian jump-diffusion process to facilitate symmetry-breaking dynamics in crystal generation. By reversing from low-symmetry priors, SbCD effectively captures global symmetry and structural dependencies, which are often overlooked in traditional methods that sample space groups from empirical distributions. The framework is inspired by spontaneous symmetry breaking in physics and is designed to model inter-space-group transitions. The authors demonstrate that their approach significantly outperforms existing symmetry-preserving models in de novo generation experiments, showcasing its potential for advancing the field of materials science and accelerating crystal discovery.
Methodology
The authors develop a theoretical variational bound objective that unifies structural dependencies among crystal components. They leverage a Markovian jump-diffusion process to model space-group distributions, allowing for the simulation of symmetry-breaking processes. The framework adapts to both continuous and discrete state spaces, enforcing space-group constraints and enabling the generation of stable crystal structures.
Results
In de novo generation experiments on datasets MP20 and MPTS-52, the SbCD model demonstrated substantial improvements over symmetry-preserving counterparts, effectively generating complete crystallographic specifications, including Wyckoff positions and space groups, which are crucial for accurate crystal representation.
Implications
The SbCD framework has significant implications for materials science, potentially accelerating the discovery of new crystalline materials with desired properties. By providing a more accurate generative modeling approach, it could lead to advancements in various applications, including pharmaceuticals, semiconductor design, and energy storage technologies.
I-SDPO: Instance-Level Adaptive Self-Distillation Policy Optimization
Reinforcement Learning
Large Language Models
Optimization
- I-SDPO addresses the degenerate gradient problem in GRPO by adapting the use of self-distillation based on the success of responses.
- The routing decision for self-distillation is made at the instance level, allowing for more effective learning from both successful and unsuccessful trajectories.
- I-SDPO achieves state-of-the-art performance on the SciKnowEval benchmark across multiple scientific domains.
- The method automatically adjusts the expected distillation rate as the model's performance improves, reducing reliance on the teacher over time.
Read more
I-SDPO: Instance-Level Adaptive Self-Distillation Policy Optimization
Summary
This paper introduces I-SDPO (Instance-Level Adaptive Self-Distillation Policy Optimization), a novel approach to address the limitations of Group Relative Policy Optimization (GRPO) in reinforcement learning for large language models (LLMs). GRPO struggles with the degenerate gradient problem, where all sampled responses are incorrect, leading to negligible policy gradients. To mitigate this, I-SDPO employs a capability-dependent routing mechanism that determines whether to use privileged self-distillation or GRPO based on the success of responses in a rollout group. Specifically, if all responses are incorrect, the model utilizes self-distillation to provide dense supervision; if at least one response is correct, it retains the GRPO framework. This method reduces the expected distillation rate as the model's success probability increases, effectively balancing the influence of the teacher and the reward signal. The authors demonstrate that I-SDPO significantly improves performance on the SciKnowEval benchmark, achieving an average mean@16 accuracy of 70.31%, compared to 56.67% with GRPO, highlighting its effectiveness in enhancing reasoning capabilities in LLMs.
Methodology
I-SDPO employs a routing mechanism that decides whether to apply privileged self-distillation or retain GRPO based on the correctness of responses in a rollout group. This instance-level decision-making allows the model to leverage self-distillation only when necessary, thus optimizing the learning process without introducing excessive bias from the teacher.
Results
I-SDPO outperforms GRPO and other methods, achieving an average mean@16 accuracy of 70.31% on the SciKnowEval benchmark, marking a significant improvement over the 56.67% accuracy achieved with GRPO. The maximum domain gain observed was 18.24 points, demonstrating the effectiveness of the proposed method.
Implications
The findings suggest that adaptive self-distillation can enhance the training of large language models, particularly in scenarios where correct responses are rare. This approach could be applied to various reinforcement learning tasks, improving the efficiency and effectiveness of model training.
Scaling Automatic Research Agents via World Models
Reinforcement Learning
Large Language Models
Efficient ML
- Introduction of World Model RL (WMRL) to replace costly environment execution in AutoResearch agents.
- Implementation of Online Debiasing and Inverse-Variance Denoising to enhance the reliability of the world model.
- Theoretical proof of improved convergence guarantees with the proposed mechanisms.
- Empirical validation showing 3-4x training acceleration and superior performance compared to larger models.
Read more
Scaling Automatic Research Agents via World Models
Summary
This paper addresses the challenge of scaling Automatic Research (AutoResearch) agents, which utilize reinforcement learning (RL) to conduct empirical research autonomously. The authors identify a critical bottleneck in the training process: while agent generation can be efficiently batched, environment execution incurs significant costs as each execution requires isolated resources. To mitigate this issue, they introduce World Model RL (WMRL), which substitutes real environment execution with a world model that simulates outcomes based on agent-generated solutions. This approach allows for scalable training without the limitations of real execution. The authors also propose two mechanisms—Online Debiasing and Inverse-Variance Denoising—to correct biases and noise in the world model's outputs, thereby improving convergence guarantees. Empirical results demonstrate that WMRL accelerates training by 3-4 times across various tasks and agent scales, outperforming larger baseline models. The method also shows versatility by extending to post-training embodied VLA policies, indicating its broader applicability beyond AutoResearch.
Methodology
The authors propose a framework where a world model simulates the execution outcomes of AutoResearch agents, allowing for efficient training. They introduce two correction mechanisms to address biases and noise in the world model's predictions, and they provide theoretical grounding for the convergence improvements achieved through these methods.
Results
WMRL accelerates training by 3-4 times across various tasks and agent scales, while outperforming larger models (48B and 120B parameters) with smaller post-trained agents (4B and 9B parameters). The proposed mechanisms also ensure that the performance matches or exceeds that of traditional RL training with real environments.
Implications
The findings suggest that WMRL can significantly enhance the efficiency of training AutoResearch agents, making it feasible to scale their capabilities. The generalizability of the method to other domains indicates potential applications in various fields requiring autonomous empirical research and decision-making.
Sampling Luck Masquerades as Allocation Gain: Auditing Test-Time Budget Allocation for Neural Combinatorial Optimization
Optimization
- First measurement of allocation value for NCO test-time sampling, revealing no detectable gain in in-distribution workloads.
- Quantification of in-sample selection bias, showing that traditional measurement methods can produce misleading gains.
- Demonstration of a significant allocation gain (11-12%) under distribution shift conditions with a pre-registered confirmatory experiment.
- Introduction of a budget-accounted policy that retains performance gains while managing sample costs.
Read more
Sampling Luck Masquerades as Allocation Gain: Auditing Test-Time Budget Allocation for Neural Combinatorial Optimization
Summary
This paper investigates the effectiveness of non-uniform allocation of sample budgets in Neural Combinatorial Optimization (NCO) solvers, which traditionally use a uniform sampling strategy. The author measures the potential gains from instance-wise allocation compared to the conventional uniform approach. The study reveals two significant findings: first, while in-sample evaluations suggest a small allocation gain of 2.2-2.6% across three pretrained solvers (POMO, AM, SymNCO), this gain is not detectable out-of-sample, indicating a bias in the in-sample measurement. Second, under distribution shift conditions, a pre-registered experiment demonstrates that allocation guided by held-out sample statistics can improve performance by 11.5% to 12.0% without charging for the guiding signal. The paper also introduces a budget-accounted policy for deployment that retains gains while charging for sample acquisition. Furthermore, it provides a correction procedure for selection bias and a comprehensive pre-registration record, ensuring transparency in the research process.
Methodology
The study employs a comparative analysis of allocation strategies in NCO solvers, measuring the performance of uniform versus instance-wise sample allocation. It uses both in-sample and out-of-sample evaluations to quantify allocation gains and biases, alongside a pre-registered experimental design to validate findings under distribution shifts.
Results
The results indicate that while in-sample evaluations suggest a small allocation gain, out-of-sample measurements show no detectable gain. However, under distribution shifts, the allocation guided by held-out sample statistics leads to significant performance improvements of 11.5% to 12.0%. A budget-accounted policy also retains gains while managing sample acquisition costs.
Implications
The findings suggest that NCO solvers may benefit from tailored sample allocation strategies, particularly under varying workload distributions. The quantification of selection bias emphasizes the need for careful evaluation methodologies in machine learning research, potentially influencing future practices in resource allocation and performance measurement.
Into the ORBIT for Time Series: Training Regimes for Foundation Models
Time Series
- Introduction of ORBIT, a training paradigm for TSFMs that controls effective pre-training distribution.
- Bootstrap Multi-Level Sampling and Omni-Range Incremental Training are key components of ORBIT.
- Falcon-2.0, trained under ORBIT, shows strong zero-shot forecasting capabilities.
- Rank-Guided Cross-Depth Alignment improves representation alignment across Transformer depths.
Read more
Into the ORBIT for Time Series: Training Regimes for Foundation Models
Summary
This paper introduces ORBIT (Omni-Range Bootstrap Incremental Training), a novel training paradigm for Time Series Foundation Models (TSFMs) that aims to enhance the effective pre-training distribution of heterogeneous time series data. The authors identify that existing training regimes inadequately address challenges such as cross-domain imbalance, varying context requirements, prediction horizons, and handling of missing data. ORBIT consists of two main components: Bootstrap Multi-Level Sampling, which hierarchically controls data exposure by creating a global training stream and stochastic sample indices for time series records, target variables, context windows, and prediction horizons; and Omni-Range Incremental Training, which allows for the simultaneous consumption of variable-length examples during training without the need for stage-specific schedules. The authors implement ORBIT in training Falcon-2.0, a simple univariate encoder-only Transformer model that incorporates missingness-aware tokenization and quantile prediction. They also introduce Rank-Guided Cross-Depth Alignment, a training objective that aligns representations across different depths of the Transformer. The results demonstrate that Falcon-2.0 achieves strong zero-shot forecasting performance across various domains and frequencies, with ablation studies highlighting the significance of stochastic sample construction and diverse context exposure.
Methodology
The methodology involves two main components: Bootstrap Multi-Level Sampling for hierarchical data exposure control, and Omni-Range Incremental Training for consuming variable-length examples during a single training run. The training process utilizes stochastic sampling techniques to create a diverse set of training examples while maintaining reproducibility.
Results
Falcon-2.0, trained using ORBIT, achieved strong zero-shot forecasting performance across diverse domains and frequencies. The ablation studies confirmed the effectiveness of the stochastic sample construction and the simultaneous exposure to various context lengths and prediction horizons.
Implications
The findings suggest that ORBIT can significantly improve the training of TSFMs, leading to better forecasting capabilities in real-world applications. The approach can be beneficial for industries relying on time series data, such as finance, healthcare, and supply chain management.
Learning Under Treatment-Induced Label Indeterminacy with Expert Annotations of Counterfactual Outcomes: A Case Study in Neurological Prognostication
Theory
- Introduces the concept of treatment-induced label indeterminacy in clinical prediction models.
- Proposes a framework for evaluating prediction models that separates certain and uncertain cases.
- Develops a prediction model that balances accuracy on certain cases with alignment to expert estimates for uncertain cases.
- Demonstrates that traditional evaluation metrics can miss important insights in uncertain cases.
Read more
Learning Under Treatment-Induced Label Indeterminacy with Expert Annotations of Counterfactual Outcomes: A Case Study in Neurological Prognostication
Summary
This paper addresses the challenge of developing clinical prediction models in scenarios where treatment decisions render outcomes unobservable, specifically in post-cardiac-arrest neurological prognostication. The authors analyze a cohort of 2,497 patients, among whom 1,429 had indeterminate outcomes due to treatment limitations. Independent clinical experts provided estimates of counterfactual outcomes for these uncertain cases. The study proposes a novel framework for evaluating prediction models that distinguishes between certain cases (with observable outcomes) and uncertain cases (with expert estimates). A prediction model is developed that incorporates both types of target labels, allowing for a trade-off between accuracy on certain cases and alignment with expert estimates for uncertain cases. The findings reveal that conventional evaluation metrics may overlook critical failure modes in uncertain cases, emphasizing the need for tailored evaluation strategies in clinical settings where treatment decisions impact outcome observability.
Methodology
The authors employed a binary classification approach to predict favorable versus unfavorable outcomes in post-cardiac-arrest patients. They utilized expert assessments as imperfect reference signals for uncertain cases and developed a model that integrates both certain and uncertain case labels. The evaluation framework explicitly separates the performance metrics for both types of cases.
Results
The proposed model achieved similar AUROC scores for certain cases compared to baseline models but showed significant differences in Brier scores and probability estimates for uncertain cases. The results indicated a trade-off between improving alignment with uncertain case labels and accuracy on certain cases, revealing that standard evaluation metrics could obscure critical performance issues.
Implications
This research has significant implications for clinical decision-making, particularly in high-stakes environments like post-cardiac arrest care. By improving the evaluation of prediction models in the presence of treatment-induced label indeterminacy, clinicians can better understand the risks and benefits of treatment options, ultimately leading to improved patient outcomes.
Simulation-to-real transfer learning for infrared spectroscopic chemical sensing and analysis from molecules to complex samples
Efficient ML
- Introduction of UltraIR, a foundation model for IR spectroscopy with over 100 million parameters.
- Utilization of simulation-to-real transfer learning to enhance data efficiency and reliability in chemical inference.
- Demonstrated strong performance across multiple analytical tasks and real-world applications.
- Pretraining on simulated IR spectra allows for effective adaptation to downstream tasks with limited labeled data.
Read more
Simulation-to-real transfer learning for infrared spectroscopic chemical sensing and analysis from molecules to complex samples
Summary
This paper presents UltraIR, a foundation model designed for infrared (IR) spectroscopy aimed at enhancing chemical sensing and analysis from molecular to complex samples. Traditional methods for interpreting IR spectra are labor-intensive and often rely on expert knowledge, making them difficult to scale and less reliable for unfamiliar compounds. The authors propose a simulation-to-real transfer learning approach that leverages a large dataset of simulated IR spectra (approximately 60 million) to pretrain a model with over 100 million parameters. This model learns a shared spectral representation through three pretraining objectives: spectral reconstruction, molecular fingerprint similarity alignment, and functional-group prediction. The pretrained model can then be adapted to specific tasks using limited labeled experimental spectra. The paper demonstrates UltraIR's effectiveness across various benchmark evaluations, including functional-group prediction and molecular structure elucidation, as well as real-world applications such as bacterial classification and soil property prediction. The results indicate that UltraIR outperforms conventional machine-learning methods and shows strong performance in scenarios with limited labeled data, highlighting its potential for scalable and reliable chemical analysis.
Methodology
The authors developed UltraIR by pretraining on a large dataset of simulated IR spectra using three complementary objectives. The model learns a shared spectral representation, which is then fine-tuned for specific tasks using limited labeled experimental data. This approach enables effective transfer learning from simulated to real-world scenarios.
Results
UltraIR demonstrated superior performance in various benchmark tasks, including functional-group prediction and molecular structure elucidation, as well as in real-world applications such as bacterial classification and soil property prediction. The model effectively adapted to tasks with limited labeled data, outperforming conventional machine-learning and task-specific deep-learning baselines.
Implications
The findings suggest that UltraIR can significantly enhance the reliability and scalability of chemical sensing systems, making it applicable in diverse fields such as environmental monitoring, clinical analysis, and materials characterization. Its ability to perform well with limited labeled data may facilitate broader adoption of machine learning in chemical analysis.