Implementation of custom bytetrack demo for real time stream analysis [GSoC'2026]

ci/lib_search.py

@@ -113,6 +113,7 @@ def check_dir(start_dir):
         'ovms-c/dist',
         'requirements.txt',
         'requirements_win.txt',
+        'bytetrack_ovms.pbtxt',
         'resnet_images.txt',
         "resnet_labels.txt",
         'rest_sdk_v2.10.16.patch',

demos/mediapipe/bytetrack/README.md

@@ -0,0 +1,121 @@
+# ByteTrack Demo Setup
+
+End-to-end demo: video source (webcam / file / RTSP) → OpenVINO Model Server (YOLOX Tiny+ ByteTrack) → output (screen / file / RTSP).
+
+---
+
+## 1. Model Preparation
+
+The detector stage of the pipeline runs on [OpenVINO](https://github.com/openvinotoolkit/openvino)-optimized YOLOX models. The following FP16 variants are currently supported:
+
+| Model | HuggingFace Repo |
+|---|---|
+| YOLOX-Tiny (fp16 precision)| `OpenVINO/yolox_tiny-fp16-ov` |
+| YOLOX-Tiny (int8 precision)| `OpenVINO/yolox_tiny-int8-ov` |
+
+`yolox_tiny-fp16-ov` is the default used in this demo.
+
+### Install requirements
+
+```bash
+pip install -r requirements.txt
+```
+
+### Download a model
+
+```bash
+python download_models.py --model OpenVINO/yolox_tiny-fp16-ov
+```
+
+> Swap `--model` for any of the four repo IDs listed above to use a different YOLOX size. The script also downloads the COCO class list used for labeling detections.
+
+This populates the local model directory that `config.json` (used by the OpenVINO Model Server in step 3) points to, and that ByteTrack consumes downstream for tracking.
+
+---
+
+## 2. Start an RTSP relay server
+
+If you don't already have one running, start MediaMTX (or an equivalent RTSP server) so the streams below have somewhere to publish to:
+
+```bash
+docker run --rm -d -p 8554:8554 -e RTSP_PROTOCOLS=tcp bluenviron/mediamtx:latest
+```
+
+> Only needed if you plan to use RTSP input and/or output (see Demo 3 below). It isn't required for the local webcam→screen or video-file→mp4 demos.
+
+---
+
+## 3. Start the OpenVINO Model Server
+
+```bash
+docker run -d -v $PWD:/demo -p 9000:9000 openvino/model_server:latest --config_path /demo/config.json --port 9000
+```
+
+---
+
+## 4. Run the Demo
+
+The `client.py` script (from `real_time_stream_analysis`) supports several combinations of input and output, so the same server can be exercised in different ways depending on what you have available.
+
+### Demo A — Local webcam → screen
+
+Reads directly from a local camera and renders the tracked output in a window.
+
+```bash
+python client.py --grpc_address localhost:9000 --input_stream 0 --output_stream screen
+```
+
+- `--input_stream 0` — camera device ID `0` (use `1`, `2`, etc. for additional cameras).
+- `--output_stream screen` — opens a live preview window instead of writing to a file or stream.
+
+### Demo B — Video file → video file
+
+Reads from an encoded video file and writes the annotated result to a new video file.
+
+```bash
+curl -L "https://raw.githubusercontent.com/FoundationVision/ByteTrack/main/videos/palace.mp4" -o video.mp4
+python client.py --grpc_address localhost:9000 --input_stream video.mp4 --output_stream output.mp4
+```
+
+- `--input_stream video.mp4` — path to the source video.
+- `--output_stream output.mp4` — path where the tracked/annotated video is saved.
+
+### Demo C — RTSP → RTSP
+
+Full end-to-end streaming demo: publish a webcam feed to an RTSP endpoint, run detection + tracking on it, and publish the annotated result to a second RTSP endpoint.
+
+**1. Publish your webcam as an RTSP input stream**
+
+```bash
+ffmpeg -f dshow -video_size 1280x720 -i video="HP True Vision FHD Camera" -f rtsp -rtsp_transport tcp rtsp://localhost:8554/channel1
+```
+
+**2. Run the client against the RTSP input/output**
+
+```bash
+python client.py --grpc_address localhost:9000 --input_stream rtsp://localhost:8554/channel1 --output_stream rtsp://localhost:8554/channel2 --model_name ByteTrack --input_name input_video
+```
+
+**3. View the output stream**
+
+Option 1 (recommended):
+
+```bash
+ffplay -rtsp_transport tcp -vf "scale=704:704,format=yuv420p" rtsp://localhost:8554/channel2
+```
+
+Option 2 (verbose logging):
+
+```bash
+ffplay -loglevel verbose -rtsp_transport tcp rtsp://localhost:8554/channel2
+```
+
+---
+
+## Summary of I/O Options
+
+| Demo | Input | Output | RTSP server required? |
+|---|---|---|---|
+| A | Local webcam (`0`) | `screen` | No |
+| B | Video file (`video.mp4`) | Video file (`output.mp4`) | No |
+| C | RTSP stream | RTSP stream | Yes |

demos/mediapipe/bytetrack/bytetrack_ovms.pbtxt

@@ -0,0 +1,138 @@
+input_stream: "IMAGE:input_video"
+output_stream: "IMAGE:output"
+
+node: {
+  calculator: "ImageTransformationCalculator"
+  input_stream: "IMAGE:input_video"
+  output_stream: "IMAGE:transformed_input_video"
+  node_options: {
+    [type.googleapis.com/mediapipe.ImageTransformationCalculatorOptions] {
+      output_width: 416
+      output_height: 416
+    }
+  }
+}
+
+node {
+  calculator: "OpenVINOConverterCalculator"
+  input_stream: "IMAGE:transformed_input_video"
+  output_stream: "TENSORS:image_tensor"
+  node_options: {
+    [type.googleapis.com/mediapipe.OpenVINOConverterCalculatorOptions] {
+      enable_normalization: true
+      use_custom_normalization: true
+      custom_div: 1.0
+      custom_sub: 0.0
+    }
+  }
+}
+
+node {
+  calculator: "OpenVINOModelServerSessionCalculator"
+  output_side_packet: "SESSION:session"
+  node_options: {
+    [type.googleapis.com/mediapipe.OpenVINOModelServerSessionCalculatorOptions]: {
+      servable_name: "yolox_tiny-fp16-ov"  # servable name inside OVMS
+      servable_version: "1"
+    }
+  }
+}
+
+node {
+  calculator: "OpenVINOInferenceCalculator"
+  input_side_packet: "SESSION:session"
+  input_stream: "OVTENSORS:image_tensor"
+  output_stream: "OVTENSORS2:detection_tensors"
+  node_options: {
+    [type.googleapis.com/mediapipe.OpenVINOInferenceCalculatorOptions]: {
+          input_order_list :["image"]
+          output_order_list :["boxes","labels"]
+        }
+  }
+}
+
+### OpenVINOYoloXTensorsToDetectionsCalculator was developed for TFLite specific model, but we can use OVYoloXTensorsToDetectionsCalculator for OpenVINO model.
+
+#node{
+#    calculator: "OpenVINOYoloXTensorsToDetectionsCalculator"
+#    input_stream: "TENSORS:detection_tensors"
+#    output_stream: "DETECTIONS:detections"
+#    node_options: {
+#    [type.googleapis.com/mediapipe.OpenVINOYoloXTensorsToDetectionsCalculatorOptions] {
+#      conf_thresh: 0.1
+#    }
+#  }
+# }
+
+node {
+  calculator: "OVYoloXTensorsToDetectionsCalculator"
+  input_stream: "TENSORS:detection_tensors"
+  output_stream: "DETECTIONS:detections"
+
+  node_options: {
+    [type.googleapis.com/mediapipe.OVYoloXTensorsToDetectionsCalculatorOptions] {
+      conf_thresh: 0.1
+      input_size: 416.0
+    }
+  }
+}
+
+# Performs non-max suppression to remove excessive detections.
+node {
+  calculator: "NonMaxSuppressionCalculator"
+  input_stream: "detections"
+  output_stream: "filtered_detections"
+  node_options: {
+    [type.googleapis.com/mediapipe.NonMaxSuppressionCalculatorOptions] {
+      min_suppression_threshold: 0.45
+      max_num_detections: 100
+      overlap_type: INTERSECTION_OVER_UNION
+      return_empty_detections: true
+    }
+  }
+}
+
+
+# Maps detection label IDs to the corresponding label text. The label map is
+# provided in the label_map_path option.
+node {
+  calculator: "DetectionLabelIdToTextCalculator"
+  input_stream: "filtered_detections"
+  output_stream: "output_detections"
+  node_options: {
+    [type.googleapis.com/mediapipe.DetectionLabelIdToTextCalculatorOptions] {
+      label_map_path: "/demo/coco_80cl.txt"
+    }
+  }
+}
+
+node {
+  calculator: "ByteTrackCalculator"
+  input_stream: "DETECTIONS:output_detections"
+  output_stream: "DETECTIONS:tracked_detections"
+  options: {
+    [mediapipe.ByteTrackCalculatorOptions.ext] {
+      track_high_threshold:0.7
+      track_low_threshold:0.55
+      new_track_threshold:0.35
+      matching_threshold: 0.8
+      track_buffer: 60
+      fuse_score: false
+    }
+  }
+}
+
+# Converts the detections to drawing primitives for annotation overlay.
+node {
+  calculator: "DetectionColorByIdCalculator"
+  input_stream: "DETECTIONS:tracked_detections"
+  output_stream: "RENDER_DATA:detections_render_data"
+}
+
+# Draws annotations and overlays them on top of the input images.
+node {
+  calculator: "AnnotationOverlayCalculator"
+  input_stream: "IMAGE:input_video"
+  input_stream: "detections_render_data"
+  output_stream: "IMAGE:output"
+}

demos/mediapipe/bytetrack/config.json

@@ -0,0 +1,16 @@
+{
+    "model_config_list": [
+        {"config": {
+            "name": "yolox_tiny-fp16-ov",
+            "base_path": "yolox_tiny-fp16-ov"
+    }
+    }
+    ],
+    "mediapipe_config_list": [
+    {
+        "name":"ByteTrack",
+        "base_path":"./",
+        "graph_path":"bytetrack_ovms.pbtxt"
+    }
+    ]
+}