Results for “Inference”

arXiv (research papers) Ishaan Datta SI

VelServe: Taming Decode Jitters in LLM Inference with Tail-Calibrated Scheduling

Interactive LLM services must meet latency service-level objectives (SLOs) for the time to first token (TTFT) and the time between tokens (TBT), as streaming and soft real-time applications require predictable token delivery. We identify decode jitters in both aggregated and disaggregated serving. A jitter delays every request in the batch, and long responses span hundreds of steps, so even rare overruns cause many SLO violations. In aggregated serving, SLO-aware schedulers…

Read at arXiv (research papers)

arXiv (research papers) Dingwei Zhao

Efficient Parameter Estimation through Quantum Extreme Learning

Quantum sensing protocols rely on the efficient processing of measurement outcomes to infer unknown physical parameters. Conventional optimal estimators, such as Bayesian inference and maximum likelihood estimation, require to construct a likelihood function whose efficient classical construction requires full knowledge of the dynamics, which may practically be infeasible. Model-free experimental construction of the likelihood is also very resource intensive. Here, we…

Read at arXiv (research papers)

arXiv (research papers) Jonathan Pan Security

From Bytes to Behavior: Request-Level Attribution from Live GPU Hidden States

Existing memory forensics for machine learning systems typically targets static artifacts, such as model weights or prompt logs, rather than the transient computational states of a live inference process. This gap limits an examiner's ability to analyze the specific behavior of a running large language model (LLM) service that cannot be restarted or modified. We present a method for acquiring per-layer residual-stream activations from a live GPU process using a non-modifying…

Read at arXiv (research papers)

arXiv (research papers) Marco Giunti Tech policy

The Transformer Paradigm: Dynamic Processing and a New Principle of Inference

We argue that the Transformer revolution is not adequately understood as a merely quantitative expansion of deep learning, but as the emergence of a new paradigm based on a different principle of inference. We interpret the Transformer as a system in which token representations are transformed by parameterized transformations, or transforming concepts. During inference, some of these transformations are dynamically constructed from the current input: previously acquired…

Read at arXiv (research papers)

arXiv (research papers) Md Omar Faruque

CHARMING: Parallel Edge AI Acceleration Across Tensor Processing and In-Memory Computation

Edge AI increasingly relies on specialized accelerators for low-latency inference under tight resource and energy constraints, yet their efficiency remains limited by data movement. Prior work, CHARM, mitigates this overhead by overlapping data movement with computation. We observe that in-memory computing can further supplement tensor processing as a parallel accelerator while eliminating certain data movements entirely. We present CHARMING, a heterogeneous edge AI…

Read at arXiv (research papers)

SiliconANGLE AI Ryan Stevens SI

Kubernetes gets a second act as AI agents turn one prompt into thousands of inference calls

The rise of agentic AI systems is giving Kubernetes, long valued for its reliability and predictability, an unexpected second act. As AI spending shifts from training models toward serving them and running agents, the open-source community is developing tools to handle the growing volume of inference requests. The orchestration platform was built from the start

Read at SiliconANGLE AI

AWS Machine Learning Blog Sai Pavan Kasani SI

Four Amazon Bedrock inference choices that affect cost

Your Amazon Bedrock inference bill can keep climbing even after you tune prompts and application logic. Four request-level choices (cache duration, service tier, reasoning effort, and maxTokens) change how many tokens the service processes and the rate applied. We show how we tested each and how to verify the effect.

Read at AWS Machine Learning Blog

arXiv (research papers) Haozhe Wang SI

Metal Sandbox: Isolated Execution and Command Scheduling for GPU Coding Agents on Apple Silicon

On-device inference requires kernels that make effective use of local accelerators. Efficient training and evaluation of models that generate Apple Metal kernels benefit from native macOS tools and concurrent agent sessions. These sessions alternate between model requests, development commands, and GPU execution, leaving GPU capacity idle for much of a rollout. We present Metal Sandbox, infrastructure that combines restricted per-instance file access, native command…

Read at arXiv (research papers)

arXiv (research papers) Zhezhen Cao Security

Preserve the Vulnerability, Break the Alert: Testing the Robustness of CodeQL Taint Tracking with Natural Rewrites

Static application security testing (SAST) tools such as CodeQL serve as security gates in continuous integration and code review, and a gate is also read in reverse. When a previously reported alert disappears after a change, developers and the tools around them often take the vulnerability to be gone. That inference holds only if the analyzer is robust to changes that keep the vulnerability. We test this robustness for CodeQL's JavaScript taint tracking with natural…

Read at arXiv (research papers)

arXiv (research papers) Jiachun Jin SI

SALA: Semantically Aligned Latent Actions Enable Dual-Axis Scaling of Embodied Foundation Models

Embodied foundation models hinge on diverse robot interaction data, which are costly to collect. Egocentric human videos offer a complementary source of experience, and latent action models (LAMs) provide a popular means of leveraging these videos for policy supervision. Compared with world-action models (WAMs), LAMs can offer greater training efficiency and potentially lower inference costs by avoiding explicit pixel prediction during policy learning and execution. However…

Read at arXiv (research papers)

arXiv (research papers) Hannah Atmer

Broadcast, Don't Replicate: Eliminating Weight HBM in Datacenter LLM Serving

Large language model (LLM) inference at datacenter scale is fundamentally bottlenecked by memory capacity and bandwidth limits. Because model weights are read-only and identical across instances, conventional architectures that replicate these parameters across the local DRAM of tens of thousands of accelerators incur massive capital expenditures, significant static power dissipation, and redundant memory-fetch energy. A recent proposal to use optical fiber loops as…

Read at arXiv (research papers)

arXiv (research papers) Yihan Xia Security

Infrastructure for Verifiable AI: Where Proof Costs Go and How Hardware Can Help

Users of artificial intelligence services, including those powered by large language models and other transformer-based models, need assurance that inference and training follow the declared computation without exposing private models or data. Zero-knowledge proofs can provide this assurance, but proof generation remains expensive. Hardware acceleration research often reports gains for individual cryptographic operations, leaving their effect on the full cost of proving…

Read at arXiv (research papers)

Inc42 Shrishti Bisht Startups

OpenAI Brings API Inference Residency To India To Deepen Enterprise AI Adoption

ChatGPT maker OpenAI has introduced local inference and data processing for its APIs in India, adding to its efforts to…

Read at Inc42

Help Net Security Mirko Zorz Security

Stolen Claude API keys can turn Claude Code into a remote-controlled implant

Darktrace researchers turned Claude Code into a remote-controlled implant in a lab test, steering the coding agent through the memory stores feature of Anthropic’s Claude platform and pulling a 1 GB file out through the Claude Files API. Once the agent was set up, every job and every result traveled through api.anthropic.com, the endpoint the agent already calls for inference, so the attacker’s traffic looked like a coding agent doing its job. The technique needs … More →

Read at Help Net Security

Semiconductor Engineering Liz Allan SI

Rethinking Memory For AI Inference

Processing data closer to memory could reduce power and boost performance, but software, thermal, and architectural challenges remain.

Read at Semiconductor Engineering

Semiconductor Engineering Kushal Koolwal SI

Beyond Bandwidth: AI Inference Is Reshaping The Memory Wall

Longer contexts, higher concurrency, multimodal models, and AI agents will continue changing where memory pressure appears.

Read at Semiconductor Engineering

Analytics India Magazine Siddharth Jindal SI

OpenAI Launches API Inference Residency in India

The move could help financial services firms and other regulated businesses meet data localisation requirements while deploying OpenAI applications.

Read at Analytics India Magazine

SiliconANGLE AI Jonathan Anthony SI

Google’s GKE Labs turns reinforcement learning into a Kubernetes workload

Model fine-tuning through reinforcement learning is fast becoming one of the most demanding jobs in AI infrastructure. It mixes training, inference and sandboxed agent work in a single loop, often with research code tangled into the systems beneath it. Google LLC has been tackling that problem from inside its Kubernetes engineering group, and recently donated

Read at SiliconANGLE AI

AWS Machine Learning Blog Markus Adhiwiyogo SI

The attendee’s guide to the AWS re:Invent 2026 AI/ML track

AWS re:Invent 2026 includes more than 140 AI/ML track sessions. This attendee's guide helps you find the breakouts, chalk talks, workshops, and builders' sessions that fit your role and goals, spanning AI chips, models and inference, agents, agent trust, and sustainability, so you can build a smarter re:Invent schedule.

Read at AWS Machine Learning Blog

AWS Machine Learning Blog Victor Wang SI

Introducing usage-based pricing for Amazon SageMaker AI bidirectional streaming

AWS Marketplace now supports usage-based metering for Amazon SageMaker bidirectional streaming inference, so model providers can charge for what customers actually consume during a live stream. Deepgram is the launch partner, bringing usage-based speech-to-text and text-to-speech billing to AWS Marketplace.

Read at AWS Machine Learning Blog

arXiv (research papers) Kuang-Ting Huang

TQIR: Trellis Quantization with Integer Reconstruction for Bandwidth-Efficient Edge LLM Acceleration

Autoregressive large language model (LLM) inference is often limited by external-memory bandwidth, while existing ultra-low-bit trellis quantization reconstructs floating-point weights and leaves runtime Hadamard-transform and decoder costs unresolved. This paper presents TQIR, an algorithm--hardware co-design that decouples storage precision ($S$) from decoded-weight ($W$) and activation ($A$) compute precisions, enabling a better accuracy--hardware trade-off. TQIR performs…

Read at arXiv (research papers)

arXiv (research papers) Zhan Chen

AMARA: Compiler-Driven Accelerator Architecture Design for Graphical Probabilistic Sampling

Sampling methods are fundamental primitives for training and inference on probabilistic models. However, practical deployment of sampling algorithms such as Markov Chain Monte Carlo (MCMC) remains difficult on general-purpose processors. Existing hardware accelerators improve sampling efficiency, but they are typically specialized for a narrow class of models, graph topologies, or update schedules, making it difficult to support the diverse workloads used in real…

Read at arXiv (research papers)

Inc42 Ankush Das Startups

Can Indian Startups Win The AI Inference Race?

Nations worldwide have spent the last few years obsessing over bigger models and the compute power needed to train them.…

Read at Inc42

Tech in Asia Aiko Gao Ishida Startups

Lenovo launches AI token factory to tap China inference demand

The system routes complex tasks to larger AI models and simpler interactions to smaller ones.

Read at Tech in Asia

arXiv (research papers) Guofu Liao Security

VRMaaS: Verifiable Reward-Model-as-a-Service for Trustworthy LLM Alignment

Reward models (RMs) are increasingly deployed as service components in LLM alignment pipelines, where payment, audit, moderation, and governance workflows act on the scores they return. An inference proof alone does not show whether a score belongs to the intended request, model checkpoint, numerical configuration, and verifier: in a controlled comparison that isolates the accepted statement from prover scale, a generic inference proof wrapped for on-chain checking remains…

Read at arXiv (research papers)

arXiv (research papers) Jaerock Kwon SI

The Posterior-Prior Epistemic Gap: When World Model Imagination Fails to Carry Task-Relevant Signals

Model-based planners that roll out imagined futures assume the transition model faithfully propagates task-relevant signals---a property rarely measured directly. In a single-seed case study of a Recurrent State Space Model world model with active-inference planning in the CARLA driving simulator, we identify a \textbf{posterior-prior gap}---posterior (observation-filtered) states retain obstacle information that imagined rollout states progressively lose: a supervised…

Read at arXiv (research papers)

AWS Machine Learning Blog Aris Tsakpinis SI

Introducing Kimi K3.1 Pro on Amazon Bedrock

Kimi K3.1 Pro from Moonshot AI is now available on Amazon Bedrock. Learn how to get started through the OpenAI-compatible Chat Completions, Responses, and Converse APIs, optimize inference with explicit prompt caching, and use the model as a coding assistant in OpenCode.

Read at AWS Machine Learning Blog

arXiv (research papers) Anish Saxena

SlipStream: Enabling HBF for GPU Systems by Mitigating the Latency and SRAM Overheads

Large language model (LLM) inference demands ever more memory capacity for larger models and batch sizes. High-Bandwidth Flash (HBF) replaces the DRAM dies of HBM with NAND flash and provides up to 16x the capacity at comparable read bandwidth. However, HBF can incur a read latency of nearly 20$μ$s, 20x that of HBM, so on-demand HBF accesses slow inference by up to 12.8x. Conventional cache-line prefetchers require 20x more in-flight tracking state to cover this latency…

Read at arXiv (research papers)

The Hacker News Security

SGLang RCE Exploit Breaches NEAR AI, Steals Data From 18 Agents and About 100 Users

An attacker exploited a remote code execution flaw in SGLang, an open-source inference engine, and ran code on one of NEAR AI's inference servers last Sunday. The attacker could also have seen requests being processed on that server that day, including requests sent in confidential mode, the company said earlier this week. That mode processes requests inside a trusted execution environment (TEE

Read at The Hacker News

Older stories