Results for “Inference”

arXiv (research papers) Wenbo Zhu

Manufacturing Differential Sparsity: Planner-Hardware Co-Design for Efficient World Model Planning

JEPA-style latent world models let embodied agents plan toward new goals with model predictive control, but they are still too slow for real-life embodied control: sampling-based MPC repeats predictor inference tens of thousands of times per plan, and even on high-end GPUs one planning step can take tens of seconds. We observe substantial latent locality in this repeated-inference workload: the predictor maps close actions to close latents, and inside a quantized predictor…

Read at arXiv (research papers)

arXiv (research papers) Fengyuan Liu SI

Communicate the Change: Efficient Expert Parallelism via Activation Compression

Scaling has driven the success of large language models (LLM), with sparse mixture-of-experts (MoE) architectures enabling greater parameter capacity without a proportional increase in computation. At large scales, MoE models rely on expert parallelism to distribute experts across devices. Each MoE layer requires two synchronous collective communication steps during inference: dispatching hidden states to the activated experts and combining the resulting expert outputs. As…

Read at arXiv (research papers)

Semiconductor Engineering Technical Paper Link SI

Reducing High-Bandwidth Flash Latency In GPU Systems (Georgia Tech, Nvidia, Stanford)

Researchers at Georgia Tech, NVIDIA Research, and Stanford University published “SlipStream: Enabling HBF for GPU Systems by Mitigating the Latency and SRAM Overheads.” Abstract Excerpt “Large language model (LLM) inference demands ever more memory capacity for larger models and batch sizes. High-Bandwidth Flash (HBF) replaces the DRAM dies of HBM with NAND flash and provides... »

Read at Semiconductor Engineering

arXiv (research papers) Alexandre Fournier-Montgieux Tech policy

Portrait Fitness as a Pretext: Probing Implicit Demographic Bias in VLMs

We audit face-related bias in VLMs using BALPORTRAITS, a new balanced dataset of 9,000 fictional portraits across ethnicity, gender, and age segments. Sixteen VLMs estimate portrait fitness scores for occupations, brands, activities, and banking or insurance products within an advertising-casting framework designed to avoid direct inference of protected attributes. Item-level estimations are aggregated with scenario-specific prestige scores to compare rankings across…

Read at arXiv (research papers)

arXiv (research papers) Ştefana-Arina Tăbuşcă Interdisciplinary School of Doctoral Studies Tech policy

What You Post Is What You Symptom: Estimating Psychometric Networks from Social Media Discourse

Symptom networks in psychopathology are estimated almost exclusively from self-report questionnaires, restricting inference to whatever symptom set a given instrument covers. We introduce a framework for symptom-resolved network estimation directly from social media discourse and apply the framework to depression and eating disorders, with a third anorexia corpus as cross-dataset validation. To our knowledge, we are the first to estimate symptom-resolved psychometric…

Read at arXiv (research papers)

NVIDIA Blog Shruti Koparkar SI

NVIDIA Sweeps MLPerf Training and Endpoints as Vera Rubin Debuts With Leading Performance

In its MLPerf benchmarks debut, with results released today, the NVIDIA Vera Rubin platform delivers leading performance across training, post-training and agentic inference. NVIDIA swept every MLPerf Training v6.1 benchmark and, in its first MLPerf Endpoints submission, delivered more than 10x generational gains in throughput per watt for agentic AI — measuring how much agentic

Read at NVIDIA Blog

arXiv (research papers) Hongyu Ma

WinDEX: Metric 3D Hand Motion Reconstruction in the Wild for Dexterous Teleoperation and Policy Learning

Human manipulation data are widely used to train robots for human-like dexterity. However, low-cost, large-scale collection in the wild remains challenging. Existing methods remain limited in geometrically grounded metric-scale reconstruction, real-world generalization and inference speed. We present WinDEX, integrating (1) geometric-temporal optimization for metric 3D hand annotation from in-the-wild egocentric stereo videos, combined with visual-inertial SLAM for…

Read at arXiv (research papers)

arXiv (research papers) Hua Zhang

Linglong Agent Service: A Runtime for Centralized Agent Reasoning and Distributed Execution

LLM agents can use remote tools and cloud sandboxes, but integrating reasoning, session state, and heterogeneous execution sites remains a runtime design problem. We present Linglong Agent Service (agentsvc), a self-hosted HTTP service and embeddable Python runtime that treats execution site as a configurable dimension alongside model, tools, and context. A parent service retains inference orchestration, tool descriptions, and conversation history; each inference request…

Read at arXiv (research papers)

arXiv (research papers) Guangzhi Sun Security

AudioTrace: File-Level Membership Detection for Audio Large Language Models

Significant privacy and copyright concerns are raised by the use of publicly available audio recordings to train audio large language models (LLMs), yet effective auditing methods remain limited. Existing membership inference attacks typically assume knowledge of complete training examples, including their associated textual supervision. In practice, however, audio recordings may be paired with privately generated transcripts, captions, or question-answer pairs that are…

Read at arXiv (research papers)

Tech.eu John Reynolds Startups

AI inference startup Daisytuner raises €6M seed

A German startup whose technology tunes AI models for different chips during inference has raised €6 million in seed funding, it said today.Daisytuner has raised the funds in a round co-led by Elaia a...

Read at Tech.eu

arXiv (research papers) Zhe Dong

Building Fully On-Device GUI Agents with Mobile NPUs

Mobile GUI agents are increasingly used to automate everyday smartphone tasks, but most existing systems still rely on cloud-hosted models or remote inference, introducing repeated communication latency, privacy exposure, and dependence on network availability. Modern smartphones integrate neural processing units (NPUs) that provide an attractive substrate for fully on-device LLM inference. However, directly deploying conventional GUI-agent pipelines on mobile NPUs is…

Read at arXiv (research papers)

arXiv (research papers) Ahmed Tariq Independent researcher SI

Routing Beyond Task Categories: A Cost-Aware LAB-Bench Study

We tested whether features of biology questions or of a model's first answer can improve accuracy within an inference budget. In a LAB-Bench pilot, we compared a category router with one that adds six structural features. Neither reached the primary 80% accuracy target on development. At the 50% and 60% secondary targets, both chose Haiku for all 120 held-out questions and scored 71/120. We then froze a new comparison before collecting responses on 144 additional questions…

Read at arXiv (research papers)

arXiv (research papers) Shuning Zhang

AgenticJevNav: Agentic Object Search and Navigation via Event-Driven Verification

Agentic navigation must maintain goals and adapt actions from incomplete evidence despite execution failures and delayed inference. We present AgenticJevNav, which links goal-directed planning, evidence memory, event-triggered verification, and bounded execution. The Agent plans from RGB, odometry, memory, and feedback, retaining goal ownership while authorizing a bounded action set. Jev reviews structured evidence after blocked/failed execution or two consecutive…

Read at arXiv (research papers)

arXiv (research papers) Ziqing Mei

HBF-EventLab: A Discrete-Event Simulator for Studying Host-Device Coordination in High Bandwidth Flash

High Bandwidth Flash (HBF) combines NAND flash density with high-bandwidth accelerator connectivity, offering additional memory capacity for large language model inference. However, limited hardware availability makes it difficult to evaluate how host behavior and device resource constraints jointly affect HBF request processing. We present HBF-EventLab, an open-source, transaction-level discrete-event simulator based on the Open Compute Project HBF specification…

Read at arXiv (research papers)

arXiv (research papers) Chloe Wong

MIAO: Microscaling Intermediate Accumulation Optimization

Microscaling (MX) quantization enables lower-precision large language model (LLM) inference with limited accuracy degradation, reducing memory requirement and multiplication cost. However, despite reduced operand precision, accumulation is often retained at high floating-point precisions, leaving a substantial hardware cost that operand quantization alone does not address. In particular, systolic arrays, as used in tensor accelerators such as Google TPU and AWS Trainium…

Read at arXiv (research papers)

arXiv (research papers) Akshay Vyas SI

Neural Dual Bounds: Valid-by-Construction JGLP Warm-Starts for MAP and Constrained MAP

We present Neural Dual Bounds, a calibration-free framework that produces deterministic upper bounds for MAP inference in probabilistic graphical models using a single neural-network forward pass. The framework trains a network to output the edge messages of a mini-bucket style upper bound on a given join-graph decomposition in a self-supervised manner by directly minimizing a smooth dual objective. The output of this network can be used at test time to warm-start a standard…

Read at arXiv (research papers)

arXiv (research papers) Mostafa Darvishi SI

From Sensors to Silicon: Causal Cross-Layer Self-Healing Edge AI for Secure and Energy-Adaptive IoT Systems

The migration of artificial intelligence from centralized cloud infrastructure to resource-constrained Internet-of-Things (IoT) devices reduces communication latency and improves data locality, but it also tightly couples sensing quality, inference behavior, microarchitectural activity, hardware reliability, security, and energy management within the same physical node. Existing edge-intelligence solutions predominantly address these dimensions in isolation, making it…

Read at arXiv (research papers)

arXiv (research papers) Charan Teja Nagisettygari SI

How Much Determinism Can Fine-Tuning Buy? Margin Regularisation Against Batch-Induced Token Flips

Greedy decoding at zero temperature is widely assumed to be deterministic, although the exact same prompt may yield different tokens depending on the number of other prompts sharing the batch for inference. The currently available solutions operate during inference, trading determinism for throughput. Can we train a model in such a way that fewer decoding steps would be susceptible? We characterize the perturbation empirically across six model families and three numerical…

Read at arXiv (research papers)

arXiv (research papers) Fatih Temiz

Wireless Edge-Assisted Adaptive Latent Reasoning for Vision-Language-Action Models in Physical AI

Vision-language-action (VLA) models enable embodied artificial intelligence (AI) by translating visual observations and language instructions into robotic actions. Emerging use cases in physical AI are expected to drive the need for efficient inference and communication, as the computational demands of the VLA models pose challenges for real-time control with limited onboard resources. Test-time scaling through latent reasoning allows these models to refine hidden…

Read at arXiv (research papers)

arXiv (research papers) Pratyush Kumar Singh SI

Learning inadequacy of Bayesian neural operators for PDE-constrained inference and design

Partial differential equations (PDEs) underpin predictive modeling across science, engineering and medicine. Their use in digital twins, parameter inference and optimal design requires repeated solutions across parameters and scenarios. For nonlinear, transient and multiphysics systems, these calculations can make numerical simulation prohibitively expensive. Here we introduce the residual-informed Bayesian neural operator (RI-BNO), which learns operator inadequacy using a…

Read at arXiv (research papers)

arXiv (research papers) Alba Maria Tellez Fernandez

Refinement vs. Replanning: Inference Compute Allocation in Robot Policies

A frozen chunked robot policy can spend deployment-time inference compute in two coupled ways: more work within each policy call to refine a predicted action chunk, or more frequent policy calls to replan from fresh observations. We study this allocation without updating model weights. Our evaluation procedure reports success with both nominal compute rate and complete attempt cost, selects comparisons on discovery tasks, and tests prespecified comparisons on held-out tasks…

Read at arXiv (research papers)

AWS Machine Learning Blog WonGu Rho SI

Production MLOps at GS Caltex with Amazon SageMaker Canvas, part 1

GS Caltex connects Amazon SageMaker Canvas to Amazon SageMaker Pipelines so process engineers' no-code models run in production. This post shows how ML engineers reuse validated preprocessing flows and models, cutting deployment from about a week to a few hours and running batch inference every 30 minutes.

Read at AWS Machine Learning Blog

AWS Machine Learning Blog Malav Shastri SI

OpenFold3 on Amazon SageMaker JumpStart: Predict biomolecular structures

Deploy OpenFold3-preview from Amazon SageMaker JumpStart to predict 3D structures of proteins, DNA, RNA, and ligand complexes in your own AWS account. The post covers asynchronous inference, multi-GPU parallelism, bringing your own MSAs for accuracy and privacy, and scaling to zero to reduce cost.

Read at AWS Machine Learning Blog

arXiv (research papers) Maryam Ghasemian Security

Quantifying Membership Inference Risks in Genomic Data Sharing: The Role of Attacker's Auxiliary Information

Genomic data sharing is essential for biomedical research but introduces significant privacy risks, including membership inference attacks, in which an adversary determines whether a specific individual's data was used in a study. Existing privacy protections are typically applied after a release format has already been chosen, leaving researchers without a principled method to assess dataset-specific risk beforehand. We propose a server-assisted, principal component…

Read at arXiv (research papers)

AWS Machine Learning Blog Belair Albahagwi SI

Cost-optimizing AssemblyAI speech-to-text inference on AWS with Amazon EC2 Spot

Learn how AssemblyAI runs cost-optimized speech-to-text inference at scale on AWS, using Amazon ECS, Amazon EC2 Spot GPU instances, and an Amazon SQS-based retry architecture to achieve 70% cost savings while maintaining SLA commitments across European Regions.

Read at AWS Machine Learning Blog

AWS Machine Learning Blog Aamna Najmi SI

Cost attribution and controls for AI developer and productivity tools on Amazon Bedrock

Amazon Bedrock provides built-in, zero-instrumentation cost attribution that traces every inference call back to the IAM principal that made it. This post shows how to attribute per-user Bedrock costs for AI developer and productivity tools using IAM Identity Center, session tags, and CUR 2.0, and how to enforce per-user usage limits.

Read at AWS Machine Learning Blog

arXiv (research papers) Yimin Gao

Mutable Inference: Architecting LLM Accelerators for Inference-Time Learning

Long-context LLMs must retain growing histories, but full attention makes each new token increasingly expensive. Inference-time learning (ITL) offers another approach. The model keeps training on the context during inference, retaining what it learns in its weights. Recent work shows that ITL can match full-context attention quality while attending only to a recent fixed-size window. This bounds attention costs but adds Learning (backward computation and weight updates)…

Read at arXiv (research papers)

AWS Machine Learning Blog Shantanu Tripathi SI

Build a model improvement flywheel with Amazon SageMaker HyperPod Data Capture

Amazon SageMaker HyperPod Data Capture records live inference inputs and outputs to Amazon S3. This post walks through one end-to-end example: capturing a support router's production traffic, surfacing a new-product blind spot with Amazon Athena, fine-tuning on the captured data with the HyperPod training operator, and measuring the before-and-after lift.

Read at AWS Machine Learning Blog

arXiv (research papers) Abhilash Durgam SI

Typing Matters: Muscle, Language and Typing Behaviour in Live Keystroke Decoding from sEMG

Surface electromyography (sEMG) from the wrists can recover typed text without a keyboard. It is usually posed as a signal-inference problem: characters are read out of the muscle signal and character error rate is reported offline. We argue that this framing removes most of the problem. The decoder is constrained simultaneously by the muscles, by the language being typed, and by typing behaviour such as typos, corrections, backspaces and line breaks, and a usable system…

Read at arXiv (research papers)

arXiv (research papers) Yuhang Wang Security

CANTraj: Recovering Vehicle Trajectories from CAN Signals in Naturalistic Driving

Open driving releases and usage-based insurance programs often claim to prevent privacy breaches by omitting GPS records when sharing or collecting vehicle data. However, removing GPS coordinates from driving logs does not erase the location information carried by vehicle signals. While prior route-inference attacks have demonstrated that speed, steering, and driving events can be matched to road networks to reconstruct trajectories, existing studies rely on inconsistent…

Read at arXiv (research papers)

Older stories