Skip to content

dynamic shape arg#4233

Open
apbose wants to merge 5 commits into
mainfrom
abose/dynamic-shapes-passthrough
Open

dynamic shape arg#4233
apbose wants to merge 5 commits into
mainfrom
abose/dynamic-shapes-passthrough

Conversation

@apbose

@apbose apbose commented May 5, 2026

Copy link
Copy Markdown
Collaborator
  • Add name_dims to torch_tensorrt.Input — tag a dynamic axis with a name (Input(..., name_dims={0: "B"})); the same name across inputs is exported as one shared torch.export.Dim, letting users share a dim (e.g. a batch axis on input_ids and attention_mask) without a separate dynamic_shapes argument or any torch.export knowledge at the call site.
  • Unblock kwarg-only nn.Module calls — relax the arg_inputs is None assertion when kwarg_inputs is given, normalize arg_inputs=None to [] (instead of [None]), and stop inputs or arg_inputs from overwriting an empty inputs tuple with None.

@meta-cla meta-cla Bot added the cla signed label May 5, 2026
@github-actions github-actions Bot added component: tests Issues re: Tests component: core Issues re: The core compiler component: api [Python] Issues re: Python API component: dynamo Issues relating to the `torch.compile` or `torch._dynamo.export` paths labels May 5, 2026
@github-actions github-actions Bot requested a review from zewenli98 May 5, 2026 04:54

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are some changes that do not conform to Python style guidelines:

--- /home/runner/work/TensorRT/TensorRT/tests/py/dynamo/models/test_shared_dynamic_dim.py	2026-05-05 04:54:37.448873+00:00
+++ /home/runner/work/TensorRT/TensorRT/tests/py/dynamo/models/test_shared_dynamic_dim.py	2026-05-05 04:55:00.216122+00:00
@@ -14,10 +14,11 @@
caller supply a shared ``Dim`` directly to ``torch_tensorrt.compile`` --
mirroring the ``torch.export.export(dynamic_shapes=...)`` signature -- so the
shared-batch case compiles end to end without the caller having to pre-export
the module themselves.
"""
+
import unittest

import pytest
import torch
import torch.nn as nn

@apbose apbose force-pushed the abose/dynamic-shapes-passthrough branch from 64bcdc4 to adffa87 Compare June 4, 2026 17:31

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are some changes that do not conform to Python style guidelines:

--- /home/runner/work/TensorRT/TensorRT/tests/py/dynamo/models/test_shared_dynamic_dim.py	2026-06-04 17:31:18.741538+00:00
+++ /home/runner/work/TensorRT/TensorRT/tests/py/dynamo/models/test_shared_dynamic_dim.py	2026-06-04 17:31:43.478755+00:00
@@ -13,10 +13,11 @@
name; the same name across inputs is exported as a single shared ``Dim``. All
the dynamic-shape intent lives on the ``Input`` objects -- no separate
``dynamic_shapes`` argument and no ``torch.export`` knowledge required at the
call site.
"""
+
import unittest

import pytest
import torch
import torch.nn as nn

@apbose apbose force-pushed the abose/dynamic-shapes-passthrough branch from adffa87 to ae1ce2f Compare June 4, 2026 17:43
@apbose apbose requested review from cehongwang and narendasan June 4, 2026 18:04

@cehongwang cehongwang left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks ok

@apbose

apbose commented Jun 4, 2026

Copy link
Copy Markdown
Collaborator Author

@narendasan are you suggesting something like this? B and S are the named tuples
B = shared_dims("B", min=1, opt=4, max=8) # batch
S = shared_dims("S", min=16, opt=128, max=512) # sequence

torch_tensorrt.compile(
model, ir="dynamo",
kwarg_inputs={
"input_ids": Input(shape=(B, S), dtype=torch.int64),
"attention_mask": Input(shape=(B, S), dtype=torch.int64),
},
)

@narendasan

Copy link
Copy Markdown
Collaborator

No it was more like two different ideas.

Idea 1: Copy the export dynamic dimension system

torch_tensorrt.compile(model, 
    arg_inputs=[
        torch_tensorrt.Input({
            "min":(1,2,3,4), 
            "opt":(1,4,3,4),
            "max":(1,8,3,4),
        }, shared_dims={1:"channel"}),
        torch_tensorrt.Input({
            "min":(2,2,), 
            "opt":(4,2,),
            "max":(8,2,),
        }, shared_dims={0:"channel"}),
    ])     

Idea 2 (we can do later): Named dimensions

from collections import namedtuple

input_shape1 = namedtuple('input_shape', ['n', 'c', 'h', 'w'])
input_shape2 = namedtuple('input_shape', ['c', 'seq',])

torch_tensorrt.compile(model, 
    arg_inputs=[
        torch_tensorrt.Input({
            "min":input_shape1(1,2,3,4), 
            "opt":input_shape1(1,4,3,4),
            "max":input_shape1(1,8,3,4),
        }),
        torch_tensorrt.Input({
            "min":input_shape2(2,2,), 
            "opt":input_shape2(4,2,),
            "max":input_shape2(8,2,),
        }),
    ])     

@apbose apbose force-pushed the abose/dynamic-shapes-passthrough branch from f134c58 to d1e4c28 Compare June 8, 2026 16:16

@narendasan narendasan left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM please add an example on how to use this feature then merge.

@github-actions github-actions Bot added the documentation Improvements or additions to documentation label Jun 11, 2026
@apbose apbose force-pushed the abose/dynamic-shapes-passthrough branch from f907b64 to 9f9055a Compare June 12, 2026 19:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

cla signed component: api [Python] Issues re: Python API component: core Issues re: The core compiler component: dynamo Issues relating to the `torch.compile` or `torch._dynamo.export` paths component: tests Issues re: Tests documentation Improvements or additions to documentation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants