Results for “Inference”

arXiv (research papers) Mohammed Nayeem Khan SI

Bits or permutations? Spending a fixed inference budget on weight precision or on option-order averaging in small local language models

Local language models can save memory by storing weights in fewer bits, and they can reduce sensitivity to the order of answer options by averaging a few reorderings. We ask whether the first makes the second worse, and which is the cheaper way to spend a fixed budget. On Qwen3-0.6B and Qwen3-1.7B with simulated round-to-nearest weight quantization (8, 6, 4 and 3 bits, fp16 reference), read from the logits of the letters A to D on ARC-Easy, ARC-Challenge, MMLU and HellaSwag…

Read at arXiv (research papers)

arXiv (research papers) Bo Dai SI

An Energy-Based Perspective on Machine Learning

Energy-based models (EBMs) define a distribution through a scalar energy, $p_θ(x)\propto\exp(-E_θ(x))$, and are usually presented as one model family among many. This tutorial has a broader goal: beyond introducing EBMs, it uses them as a single lens that connects the different aspects of machine learning. Any EBM is specified by three choices, the energy, which determines what the model represents; the inference procedure, which samples from or optimizes the energy; and the…

Read at arXiv (research papers)

arXiv (research papers) Zhengxiang Huang SI

EngineAgent: Agentizing On-Device LLM Inference Engines with Inference Stack Hierarchy Awareness and Hardware Abilities

Rapidly evolving LLM architectures and heterogeneous mobile hardware make supporting new models in on-device inference engines a recurring and costly system challenge. Each model may require coordinated changes across conversion, graph construction, operators, and hardware backends. Existing coding agents often miss these cross-layer dependencies and struggle to diagnose failures across host--device execution paths. We present EngineAgent, an agentic system for automating…

Read at arXiv (research papers)

arXiv (research papers) Haitian Li

Elastic-WAM: Adaptive Spatiotemporal Imagination for Efficient Robot Control

World-Action Models improve robot control by predicting future visual dynamics, but iterative video denoising incurs substantial inference overhead. Recent approaches reduce this cost through partial future denoising, yet typically rely on fixed refinement budgets. Our empirical observations reveal that preferred refinement depths vary across tasks and states, while different future regions exhibit varying relevance to action generation. Motivated by these findings, we…

Read at arXiv (research papers)

The Register Connor Jones SI

Fireworks AI gets burned twice as customer data exposure widens

Inference biz pins latest breach on cloud provider and LLMJacking miscreants

Read at The Register

arXiv (research papers) Tomer Solberg Security

Sable: A Small-Field Sumcheck Argument for Bit-Exact Inference of Large Language Models

We present Sable, an argument system and single-GPU prover for the statement that a committed, integer-quantized transformer produced a given token sequence from a given prompt, bit for bit. Sable fixes an integer function F_O with no floating-point step, in which every rounding, normalization, softmax and sampling decision is determined by its inputs, and proves it with a constraint system whose relations provably never wrap modulo the 31-bit KoalaBear prime. The proof…

Read at arXiv (research papers)

arXiv (research papers) Xingqi Wang SI

Specify Privacy Yourself: Assessing Inference-Time Personalized Privacy Preservation Ability of Large Vision-Language Models

Large Vision-Language Models (LVLMs) have demonstrated remarkable capabilities but raise significant privacy concerns due to their abilities to infer sensitive personal information from images with high precision. While current LVLMs are relatively well aligned to protect universal privacy, e.g., credit card data, we argue that privacy is inherently personalized and context-dependent. This work pivots towards a novel task: can LVLMs achieve Inference-Time Personalized…

Read at arXiv (research papers)

arXiv (research papers) Bikrant Bikram Pratap Maurya SI

Domain-Invariant and Language-Model-Assisted Acoustic Side-Channel Attacks on Keyboards

Acoustic side-channel attacks (ASCA) recover typed text from the sound of keystrokes, using only a commodity microphone and no access to the victim's device. Prior work establishes that such attacks succeed under idealized laboratory conditions - a single keyboard, a fixed microphone, a quiet room, and training and testing on the same device but leaves open the question that matters for security: whether keystroke inference survives the open world of unseen keyboards…

Read at arXiv (research papers)

YourStory Startups

OpenAI unveils a 'Control Room' safety feature that shuts down AI agents from going rogue in 35 seconds

OpenAI said Control Room will connect every research inference workload, including agentic sampling for model training and evaluation, data processing, and more. It will work together with its misalignment monitoring systems.

Read at YourStory

eeNews Europe SI

Analog in memory compute for low power AI inference

Analog In-Memory Compute (IMC) is driving the promise of achieving energy efficiencies on the order of one femtoJoule per operation in a computing system for AI. By performing certain computational tasks in place in memory, the need to shuttle data back and forth between the processing and memory units is avoided, slashing the energy consumption

Read at eeNews Europe

arXiv (research papers) David Hobbs Space

ExoPlanetLab: Bayesian orbit fitting of Gaia-DR4 epoch astrometry, radial velocities and relative astrometry

Gaia Data Release 4 (DR4) provides epoch astrometry from which thousands of giant planets and many substellar and stellar companions can be characterised. We present ExoPlanetLab, an open-source package for Bayesian orbit inference from Gaia epoch astrometry, radial velocities (RVs) and relative astrometry, validated on simulated and real data. The model orbit is compared with the epoch residuals after removing the part absorbed by the single-star fit. A Student-t likelihood…

Read at arXiv (research papers)

arXiv (research papers) Vicenç Torra Security

On Private Knowledge Retrieval and Private Inference for secure multi-agent systems

Multi-agent systems provide a convenient framework for the massive deployment of agents based on Large Language Models (LLMs). It provides a sound framework where agents can interact with existing software to perform actions in an autonomous way. Agentic AI is the new term to denote these AI-based autonomous systems. In real multi-agent systems, agents need to deal with malicious agents some of which can exploit security vulnerabilities to do harm. A memory poisoning attack…

Read at arXiv (research papers)

arXiv (research papers) Shuao Jia

Tri-Stack-PIM: A Heterogeneous Multi-Die 3D-Integrated Processing-in-Memory Architecture for Efficient LLM Inference

Large language model (LLM) inference during autoregressive decoding is fundamentally memory-bandwidth-bound. Existing three-dimensional (3D) stacked processing-in-memory (PIM) architectures alleviate the off-chip bandwidth wall, yet remain constrained by several limitations: physical layer (PHY), controllers, and direct memory access (DMA) engines compete with arithmetic units for advanced-node base-die area; hybrid-bonding keep-out zones fragment the base-die floorplan and…

Read at arXiv (research papers)

arXiv (research papers) Linru Zhang Security

MOAI: Module-Optimizing Architecture for non-interactive secure transformer Inference

Privacy concerns have been raised in Large Language Models (LLM) inference when models are deployed in Cloud Service Providers (CSP). Homomorphic encryption (HE) offers a promising solution by enabling secure inference directly over encrypted inputs. However, the high computational overhead of HE remains a major bottleneck. To address this challenge, we propose MOAI, an efficient HE-based, non-interactive framework for secure transformer inference. MOAI gains significant…

Read at arXiv (research papers)

Frontiers in Sustainable Food Systems Hailong Yan

Can rural e-commerce policies enhance the total factor productivity of agricultural enterprises? Causal inference based on dual machine learning models

IntroductionRural e-commerce policies, as an important component of the digital economy, are profoundly reshaping the agricultural value chain and food supply chain, and are of great significance for enhancing the economic efficiency of the grain industry.MethodsBased on the panel data of 231 A-share agricultural listed enterprises in China from 2010 to 2024, this paper takes the comprehensive demonstration policy of e-commerce entering rural areas in China as the standard…

Read at Frontiers in Sustainable Food Systems

ServeTheHome Vic A

Google TPU 8tx Pairs AMD EPYC Turin Hosts with Confidential Computing while TPU 8i Max Doubles Memory Bandwidth

The new Google TPU 8tx pairs AMD EPYC Turin hosts with confidential computing while TPU 8i Max doubles memory bandwidth for inference

Read at ServeTheHome

AWS Machine Learning Blog Venu Kanamatareddy SI

NVIDIA Nemotron 3.5 Super VL is now available in Amazon SageMaker JumpStart

NVIDIA Nemotron 3.5 Super VL, a 120B-parameter multimodal vision-language model, is now available in Amazon SageMaker JumpStart. Deploy the NVFP4 or BF16 variant in a few steps, then run text, image, and video inference for document intelligence, computer-use agents, and multimodal agentic workloads.

Read at AWS Machine Learning Blog

The New Stack Amanda Caswell

Google claims its new AI software cuts inference latency by over 90% — but there’s a catch

Google’s eighth-generation TPU announcement on Monday included more than new chips, introducing TPU Sync, an open-source system designed to speed

Read at The New Stack

arXiv (research papers) Suryansh Talukdar SI

LogCDM: Interpretable Log Anomaly Detection via Behavioral Graph Drift and Adaptive Signal Fusion

Log-based anomaly detection is increasingly dominated by large pretrained models, from BERT-scale encoders to 8B-parameter language models, which require GPUs at inference time and are difficult to deploy on the monitoring nodes where logs are produced. We present LogCDM, a compact detector that scores each log session using three complementary signals: the next-event surprise of a small causal Transformer, the distance between a graph-attention embedding of the session's…

Read at arXiv (research papers)

MarkTechPost Asif Razzaq SI

Google Unveils TPU 8i Max and TPU 8tx: A Dual-Chip TPU and a Confidential TPU

Google added TPU 8i Max and TPU 8tx to its eighth-generation TPU lineup. TPU 8i Max packs 2 TPU 8i chips per package for long-context agents. TPU 8tx adds hardware confidential inference. Bloomberg reports 2027 availability.

Read at MarkTechPost

MarkTechPost Michal Sutter SI

Baidu’s ERNIE Evolve Automated Research System Beats NanoChat and NanoGPT Records

Baidu’s ERNIE Evolve is an automated research system that edits code, runs experiments and keeps only verified gains. It beat public bests on NanoChat and NanoGPT Speedrun and sped up ERNIE-5.5-Flash inference.

Read at MarkTechPost

arXiv (research papers) Andrew Trask SI

Double Blind Evals: Resolving the Dual Confidentiality Dilemma in AI Safety Auditing

AI safety evaluations often involve sensitive prompts and proprietary models. Evaluators want to keep their evaluation prompts secret from model owners to prevent purposeful or incidental data leakage, ensuring longer usefulness of their benchmark. Model owners in turn wish to keep their proprietary model weights and inference code within their own secure infrastructure to protect their IP. We present a way forward for "double-blind evals", which provide cryptographic…

Read at arXiv (research papers)

arXiv (research papers) Hongyu Luo

Compute Regeneration through Re-Productization: Reorganizing Retired Edge AI Silicon into a Delivered Multilingual Service Robot

Useful edge hardware can lose product support while retaining substantial computation. We treat this as systems design: compute regeneration here means re-productization, retaining the compute module while redesigning the surrounding product organization. Our case is GEdgeBot, a multilingual service robot built around a retired 32 GB Jetson AGX Xavier. A source-grounded inventory shows the box alone supplies inference, while the combined system exposes five on-device AI…

Read at arXiv (research papers)

arXiv (research papers) Zhengyang Cheng

Efficient VLA Inference via Retrieval-Based Speculation and Hardware Acceleration

Vision-language-action (VLA) models enable robots to understand and execute complex instructions, but their autoregressive (AR) action generation introduces significant latency on resource-constrained robotic platforms. Speculative decoding can accelerate AR-based inference by parallelizing action generation and verification; however, existing approaches rely on costly draft models, which are inefficient on edge platforms. In this paper, we propose Zora, a hardware-software…

Read at arXiv (research papers)

arXiv (research papers) Zhiyu An

When Is High-Bandwidth Flash Cost-Effective for LLM Inference?

Can high-bandwidth flash (HBF) make LLM inference cheaper than adding GPUs? By storing weights closer to the GPU than SSD, HBF may reduce the number of GPUs needed for model capacity. However, this integration is only justified if it meets service targets and its cost is recovered. We simulate this integration using public component data across nine dense and mixture-of-experts (MoE) models, five workloads, two GPU platforms, and 1--100 GPUs. At fixed demand, HBF reduces the…

Read at arXiv (research papers)

Journal articles (OpenAlex) Zhuoyu Shi, Sen Yang, Xiangzhou Li · Agriculture

Monocular Metric-Depth Adaptation for Agricultural Perception: A CitrusFarm Case Study with Embedded Inference

Adapting monocular metric depth to agricultural imagery requires balancing target-domain accuracy against the scope of model updates and embedded execution cost. We evaluate SARE-Depth on CitrusFarm using a frozen teacher-guided Depth Anything V2 Small base. Depth-Preserving Residual Head Refinement (DP-RHR) learns image-level log-scale and gated local corrections, followed by fixed depth-conditioned Range-Expert (RE) refinement. Before adaptation, fixed-checkpoint…

Read at Journal articles (OpenAlex)

arXiv (research papers) Satvik Maurya

Augur: An Online Learning-based Hardware Data Prefetcher using Reservoir Computing

Machine-learning-based hardware prefetchers can learn complex access patterns, but practical deployment demands online adaptation, low-latency inference, and modest hardware cost. We present Augur, a hardware delta prefetcher based on next-generation reservoir computing (NGRC) that meets all three requirements. Augur formulates delta prediction as a history-based regression problem. It constructs a fixed-size reservoir from hashed delta features over a short context window…

Read at arXiv (research papers)

The Register Thomas Claburn SI

Fireworks says you’re wasting money on AI unless you’re using all the models

The case for managed inference, or automatically routing queries to the right model for the right job

Read at The Register

arXiv (research papers) Karthikeyan B. Kalyanasundara

PENSA: A Spatial Accelerator for BitNet b1.58 Inference on an HBM-equipped AMD Alveo FPGA

Ternary large language models (tLLMs) replace general multiplication in dominant linear layers with sign-controlled accumulation, shifting hardware bottlenecks toward packed-weight delivery, reduction, and operator occupancy. PENSA is an end-to-end FPGA accelerator for the BitNet b1.58 2B4T model on an AMD Alveo U50. Two row-partitioned ternary matrix engines consume packed TQ1_0 weights from eight independent high-bandwidth memory (HBM) pseudo-channels, each processing…

Read at arXiv (research papers)

Older stories