NVIDIA Releases DeepStream 9.1: Brings Agentic AI to Vision AI With 13 Skills and Multi-View 3D Tracking

NVIDIA recently released DeepStream 9.1. The update addresses an ongoing issue with video analytics. Tracking a single object from multiple cameras often requires manual camera calibration and complex calculations. DeepStream 9.1 addresses this with two additions: Multi-View 3D Tracking (MV3DT) and AutoMagicCalib (AMC). Both go as agent capabilities of coding agents. As a result, developers go from concept to running pipeline faster.

What is DeepStream 9.1

To understand the review, start with the basic platform. DeepStream is NVIDIA’s toolkit for streaming analytics for AI-based video and image understanding. It provides a GStreamer-based framework for multi-streaming, multi-model rendering on NVIDIA GPUs. Pipelines include hardware-accelerated decoding and encoding, TensorRT inference, object tracking, and message broker integration.

Building on that foundation, version 9.1 adds five notable features:

1. 13 agent skills are coding agents.
2. MV3DT’s ability to track different cameras.
3. AMC’s auto-calibration capability.
4. NVIDIA JetPack 7.2 supports Jetson Orin and Thor edge devices.
5. An integrated GitHub open source repository under CC-BY-4.0 AND Apache-2.0.

How MV3DT Tracks Objects Across Cameras

Among those additions, MV3DT is the main skill, so consider how it works. At its core, the MV3DT project is derived from multiple limited cameras to a shared 3D coordinate system. It then associates observations of the same object across camera views. Finally, it provides a globally consistent single object ID.

Specifically, data flow goes through four stages. For detection, each camera stream uses an object detector. MV3DT supports three models out of the box:

• PeopleNetTransformer: transformer based people detector, automatic for pedestrian scenes.
• PeopleNet v2.6.3: a high-performance detector based on the DetectNet_v2 architecture.
• RT-DETR 2D: multi-class detector for pedestrians, transporters, and forklifts.

Next, for a single 3D view, each camera uses a 3×4 projection matrix stored in a YAML measurement file. This includes 2D bounding boxes in 3D projects using ground plane projection. Then, in multi-view aggregation, the tracker shares tracklets using Message Queuing Telemetry Transport (MQTT). MQTT is a lightweight pub/sub messaging protocol. When two cameras look at the same person, they match the tracklets by proximity in 3D world space.

After collation, the results are distributed in three ways. The On Screen Display (OSD) shows a tiled grid with 2D and 3D bounding boxes. Bird’s-Eye View (BEV) provides a top-down route map. Kafka messages deliver protobuf metadata for each frame, including sensor ID, object ID, and 3D bounding box.

How AutoMagicCalib Removes Manual Setup

MV3DT relies on calibrated cameras, which traditionally means test boards and downtime. Instead, AMC measures the network by analyzing tracked objects in existing video files or streams. It estimates the internal parameters of each camera (focal length, focal point, lens distortion). It also estimates external parameters (rotation, translation, global position).

Under the hood, the pipe runs in five stages. These are per-camera trajectory extraction, single-view correction, multi-view tracklet matching, batch correction, and optional VGGT correction. VGGT (Visual Geometry Grounded Transformer) helps when the movement of the object is limited. AMC runs as a microservice with REST APIs and a web interface. Users only provide an image of the structure and a few alignment points.

Agentic skills workflow

According to MV3DT and AMC, the delivery method is the skills themselves. Rather than editing configuration files, you define the intent in natural language. Skills work with Claude Code, Codex, Cursor, and similar agents. The setup is brief:

git clone 
cd DeepStream
# Copy skills into your agent's skill directory (Codex shown)
mkdir -p ~/.codex/skills
cp -r skills/* ~/.codex/skills/

After launching the agent, one prompt uses the reference application:

deploy mv3dt on the 12-camera sample dataset

From there, the MV3DT capability verifies the requirements, pulls the container, and installs the Kafka and Mosquitto vendor services. It also retrieves the model weights, generates the pipeline configuration, and runs tracking. Notably, if the calibration files are missing, it triggers AMC capabilities by default.

DeepStream 9.0 vs 9.1

In context, the table below shows what has changed between releases.

Use Cases with examples

Given these capabilities, map features in concrete applications:

• Warehouse security: track a worker near forklifts across platforms with one ID, using RT-DETR 2D.
• Sales figures: follow the consumer between camera positions to measure dwell time without re-identification errors.
• Smart building monitoring: calculate occupancy on all floors and feed Kafka metadata to dashboards.
• Robots and smart cities: share static global links for navigation and incident updates.

Interactive Descriptor

To see how, the demo embedded below lives a single person moving between the camera’s three fields of view. Switch between individual 2D cameras and MV3DT 3D integration to view the object’s ID remains the same.

‘; camsEl.appendChild(d); }); // building pipeline steps var steps=[
{t:’1 · Detection’,d:’Per-camera detector finds 2D boxes (PeopleNetTransformer / RT-DETR 2D).’},
{t:’2 · Monocular 3D’,d:’3×4 projection matrix back-projects boxes to 3D via ground-plane.’},
{t:’3 · Association’,d:’MQTT shares tracklets; match by proximity in 3D world space.’},
{t:’4 · Output’,d:’Global ID to OSD, BEV, and Kafka protobuf metadata.’}
]; var stepsEl=document.getElementById(‘steps’); steps.forEach(function(s,i){ var d=document.createElement(‘div’); d.className=”step”; d.id=’step’+i; d.innerHTML=’

STEP ‘+(i+1)+’

‘+s.t+’

‘+s.d+’

‘; stepsEl.appendChild(d); }); function lerp(a,b,f){return a+(ba)*f;} function personPos(tt){tt=((tt%1)+1)%1; // wrapping [0,1) so negatives are safe
var n=path.length, p=tt*n, i=Math.floor(p)%n, f=p-Math.floor(p);
if(i<0)i+=n;
var a=path[i]b=way[(i+1)%n]; return {x:lerp(a[0],b[0],f), y:lerp(a[1],b[1],f)}; } function inFov(c,px,py){ var dx=px-cx, dy=py-cy, dist=Math.hypot(dx,dy); if(dist>c.range) return false; var a=Math.atan2(dy,dx); var diff=Math.atan2(Math.sin(ac.ang),Math.cos(ac.ang)); return Math.abs(diff)<=c.fov; } function draw(){ ctx.clearRect(0,0,W,H); // floor grid ctx.strokeStyle="#181818"; ctx.lineWidth=1; for (var gx=0;gx<=W;gx+=30){ctx.beginPath();ctx.moveTo(gx,0);ctx.lineTo(gx,H);ctx.stroke();} for (var gy=0;gy<=H;gy+=30){ctx.beginPath();ctx.moveTo(0,gy);ctx.lineTo(W,gy);ctx.stroke();} ctx.strokeStyle="#2a2a2a"; ctx.strokeRect(8,8,W-16,H-16); var p=personPos