Skip to content

01-environment-setup.md

Latest commit

 

History

History
23 lines (18 loc) · 1.61 KB

01-environment-setup.md

File metadata and controls

23 lines (18 loc) · 1.61 KB
Raw

← Back to README

1. Environment setup

conda create -y -n multimodal python=3.10
conda activate multimodal

pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu121
pip install lightning torchmetrics transformers peft albumentations opencv-python pytorchvideo multipledispatch tbparse pandas matplotlib pyyaml
pip install onnx onnxruntime-gpu --extra-index-url https://download.pytorch.org/whl/cu121
pip install colorama gdown

Notes:

  • A GPU with CUDA is strongly recommended (all provided configs default to accelerator: gpu); adjust trainer.accelerator in the sub configuration files to cpu otherwise.
  • onnx/onnxruntime-gpu are not required for train.py/test.py, but demo_model.py imports them unconditionally, so install them if you plan to run the demo (step 8), even with optimize: false. pip install tensorrt-cu12 is only needed if you additionally set optimize: true.
  • If you hit RuntimeError: Couldn't find appropriate backend to handle uri ... .wav, install pysoundfile (pip install pysoundfile).
  • Full, pinned Conda environment exports used on the original (Windows) development machines are available under envs/ if you prefer to reproduce that exact setup instead.
  • No manual model download is needed for the encoders: openai/whisper-large-v3 and openai/clip-vit-large-patch14-336 are pulled automatically from Hugging Face (public checkpoints, no login required) the first time you instantiate a model — expect a multi-GB download on first run.

Next: 2. Download the dataset