Allow textures to be randomized without rebuilding render context

[bd-pmorais]

This PR is motivated by work adding material and texture randomization to mjlab. I noticed while working on that that randomizing textures via mjmodel worked in the warp renderer but not the classic mujoco renderer. It turns out that it's trivial to enable this behavior and make the renderers more consistent by moving some cached texture fields to mjvGeom with no performance penalty at the cost of a minor ABI break.

[yuvaltassa]

This feature is already supported natively by the mjrf (Filament) API.
I don't think we should touch the OpenGL API. At this point its main feature is that it is stable.
@haroonq would you like to explain how to do what this PR does with the new API?

[haroonq]

Sure, but I'm curious why this doesn't work for the classic renderer.

Are you saying that mjr_uploadTexture doesn't work? Or that it is insufficient for what you're trying to do? If the latter, can you explain your goals in more detail?

[yuvaltassa]

https://mujoco.readthedocs.io/en/latest/APIreference/APIfunctions.html#mjr-uploadtexture

[bd-pmorais]

Sure, but I'm curious why this doesn't work for the classic renderer.

Are you saying that mjr_uploadTexture doesn't work? Or that it is insufficient for what you're trying to do? If the latter, can you explain your goals in more detail?

For context domain randomization in mjlab currently works by randomizing scalar/integer fields in mjmodel. For everything except texid (and texrepeat, texuniform) this works great, even for visual randomizations, because the viewers pick those changes up.

The reason mjr_uploadTexture wouldn't work for this case is because we're swapping between a pool of textures already compiled into the model rather than mutating the contents of the texture buffers (mutating texid rather than texture). This approach is simpler/cheaper than reuploading textures when you can specify the full set of textures you want to randomize at model definition/compilation time. Similarly, although you can update the texture texid points at with mjr_uploadTexture, texrepeat and texuniform can't be changed without recreating the entire render context currently.

The other reason to make this change in the OpenGL renderer is because it's commonly used (via mujoco.viewer.launch_passive) as a debugging viewer and this would make something (changing texid online) that works in the mujoco_warp batch renderer also visible in the debugging viewer which would eliminate a source of confusion (the warp batch renderer currently has no equivalent to the interactive mujoco.viewer, even mjwarp-viewer calls mujoco.viewer). That's actually the main motivation for the change.

Worth noting this isn't really a new feature per se on the OpenGL API. The scalar material fields (rgba, emission, specular, shininess, reflectance) are already carried on mjvGeom and refreshed live by mjv_updateScene, which is why every other visual randomization already shows up in the passive viewer. The texture parameters are the lone exception: they're cached in mjrContext at mjr_makeContext time, so they can only change by rebuilding the whole context. This PR makes them follow the same live path the appearance fields use.

If it helps, I'm not proposing active development on the OpenGL renderer beyond this one change. If you're firm on no changes to the OpenGL renderer I'm happy to defer but it would be a nice quality of life improvement for debugging texturing randomization in mjlab.

[haroonq]

That makes sense.

In that case, I'd suggest a much simpler implementation. For starters, let's not remove any existing functionality (i.e. do not remove any fields from mjrContext). Then, simply add the new fields to mjvGeom, but give them invalid sentinel values in mjv_initGeom. Finally, in render_gl3, just check to see if those fields have non-default values and, if so, use them instead of the material values.

@yuvaltassa Thoughts?

[yuvaltassa]

I agree with @haroonq , you can do this in a non breaking way

[bd-pmorais]

Thanks for the feedback! I've narrowed the implementation per @haroonq's comment

[haroonq]

Thanks! Can you apply a similar change to UpdateGeomMaterial in src/experimental/filament/compat/scene_geom_util.cc?