do_inv=True (inverse prediction) never converges - inbetween frames stay stuck at reference pose

[NDJ-create]

Hi, thank you for this great work!

I'm trying to run the inverse prediction workflow (Step 2 in the README) on a custom hand pose (closed fist → open hand), and the resulting inbetween GIF never actually transitions — frame 0 and frame 9 (out of 17 total) look nearly identical to the reference pose, as if the interpolation stays stuck near the reference throughout.

Setup: single custom stroke drawing, ~365 total strokes after combining both directions (197 confident forward strokes + 168 completed strokes from the inversion SVG).

Steps I followed:

1. Ran run_forward.sh successfully (main1-4 all completed with return code 0).
2. Ran prepare_inverse_data.py, which generated test_examples/[0inv]/ with the alternated ref/tar raster images and X_tar0.svg.
3. Completed the missing strokes in a different color following the Krita tutorial, saved as X.svg, placed in test_examples/[0inv]/svg/.
4. Set do_inv = True and ran run_inverse.sh (optical flow, preprocess_inverse.py, main1-4) — all completed without errors.
5. Ran combine_two_directions.py and the full run_inbetweening.sh sequence.

Along the way I found (and had to manually fix) a few issues that might be relevant:

• preprocess_inverse.py expects the completed drawing at [0inv]/svg/X.svg (not X_tar0.svg as the auto-generated file is named) — this isn't explicit anywhere except by reading the code.
• The first time I ran this, I had manually reconstructed test_examples/[0inv]/raster_black/ (copying the same ref/tar images used in the forward pass) instead of letting prepare_inverse_data.py generate it — which produces different (correctly alternated) ref/tar images. After fixing this, the result was still stuck.
• I confirmed via utils2_local_transform.py that connect_state is required for do_inv=True, generated correctly by preprocess_inverse.py's find_connected_points().

Even with all of the above addressed, the final GIF frames don't show any real transition between reference and target pose. Is there a known additional step or config detail I might be missing for the inverse workflow to converge properly? Happy to share more diagnostic details (intermediate frames, vector params) if helpful.

Thanks again for open-sourcing this work!

[MarkMoHR]

Hi, you can debug step by step:

1. Check results in the outputs/stroke_correspondence_results folder. Do the correspondence results in the forward prediction make sense? If so, go to step 2.
2. Check results in the outputs/stroke_correspondence_results/[0inv] folder. Do the correspondence results in the inverse prediction make sense? If so, go to step 3.
3. Check results in the outputs/stroke_correspondence_results/[1comb] folder. Do the combined correspondence results make sense?

I guess the correspondence results should be problematic.

[NDJ-create]

Thanks for the quick reply!

I checked outputs/stroke_correspondence_results (forward) and the correspondence looks correct — strokes are matched reasonably between reference and target.

I'll check outputs/stroke_correspondence_results/[0inv] and outputs/stroke_correspondence_results/[1comb] next and report back with details (visualizations if the correspondence looks wrong).

Separate question while I have you: I've also run into a memory issue on more complex drawings (300+ strokes after splitting). During main2_local_transform.py (Local Layer Transformation), I'm seeing a steady linear growth in system RAM (~1GB/sec, confirmed via monitoring, independent of GPU/VRAM usage) that eventually triggers an OOM kill before the process completes — even after reducing test_max_batch_size down to 1. This happens on both Colab and Kaggle instances with ~30GB RAM. Is this a known limitation, or is there a recommended stroke count ceiling / chunking strategy for drawings above a certain complexity?

Thanks again for your time!