Add multimodal embedding support for VertexAI (Phase 1)

[Ndunge-Makau]

What this does

This PR adds support for multimodal embeddings in RubyLLM, enabling embedding images and videos alongside text for use cases like semantic image search, video content analysis, and cross-modal retrieval.

Changes:

• Added Vertex AI's multimodalembedding model to the list of known models
• Implemented multimodal embedding support for Vertex AI provider
• Updated models.json (auto-generated from rake task)

Example usage:

# Create embeddings
RubyLLM.embed "Ruby is elegant and expressive"

# Embedding: #<RubyLLM::Embedding:0x00... >
# Embedding.vectors: {text: [0.214382708, -0.126103446, ... ]}


# Create multimodal embeddings (with supported models)
RubyLLM.embed "Ruby is elegant and expressive" with: ["image.png", "video.mp4"], model: "multimodalembedding"

# Embedding: #<RubyLLM::Embedding:0x00... >
# Embedding.vectors: Vector:  {text: [-0.00527974777, ...], image: [0.0258393418, ... ], video: [{"endOffsetSec" => 0, "startOffsetSec" => 16, "embedding" => [-0.0250446387, 0.0323432237, ...]}, ... }

Type of change

• Bug fix
• New feature
• Breaking change
• Documentation
• Performance improvement

Scope check

• I read the Contributing Guide
• This aligns with RubyLLM's focus on LLM communication
• This isn't application-specific logic that belongs in user code
• This benefits most users, not just my specific use case

Required for new features

• I opened an issue before writing code and received maintainer approval
• Linked issue: #529

PRs for new features or enhancements without a prior approved issue will be closed.

Quality check

• I ran overcommit --install and all hooks pass
• I tested my changes thoroughly
  • For provider changes: Re-recorded VCR cassettes with bundle exec rake vcr:record[provider_name]
  • All tests pass: bundle exec rspec
• I updated documentation if needed
• I didn't modify auto-generated files manually (models.json, aliases.json)

AI-generated code

• I used AI tools to help write this code
• I have reviewed and understand all generated code (required if above is checked)

API changes

• Breaking change
• New public methods/classes
• Changed method signatures
• No API changes

[kaka-ruto]

LGTM. Well done @Ndunge-Makau

[kaka-ruto]

Good job here

[Ndunge-Makau]

Thanks!

[crmne]

Thanks for the contribution.

My preference is multimodal embeddings for all providers that support them, not only Vertex AI. The issue/PR scope sounded provider-wide, which is why I approved it.

If you want to ship Vertex-only, please make that explicit in both issue and PR (title + description) as "phase 1: Vertex AI only," and add follow-up issues for other providers.

Please confirm direction before I do line-by-line review:

1. multi-provider in this PR, or
2. explicit Vertex-only phase 1.

Also please fix:

• potential regression in Vertex text embeddings
• unrelated models.json churn

[Ndunge-Makau]

Hi @crmne,

Thanks for the feedback. I'd like to confirm that this PR is explicitly for Vertex AI only as phase 1, and I'll create follow-up issues for other providers.

I've also fixed the regression issue in Vertex text embeddings. Regarding the models.json changes—these were auto-generated by the rake task after I added Vertex's multimodalembedding model to the list of known models. Let me know if you'd still want the previous version restored.

[crmne]

Thanks for working on this. I think the direction is useful, but this needs rework before we can merge it.

Main blockers:

1. The branch conflicts with current main in lib/ruby_llm/embedding.rb, lib/ruby_llm/provider.rb, and lib/ruby_llm/models.json.

2. lib/ruby_llm/providers/vertexai/embeddings.rb uses present?, but RubyLLM does not load ActiveSupport at runtime. The specs pass because spec_helper requires active_support/core_ext, but plain runtime usage fails with NoMethodError.

3. The PR changes existing Vertex text embedding payloads from content / outputDimensionality to text / dimension. That looks correct for the multimodal embedding API, but it likely regresses existing Vertex text embedding models like text-embedding-004.

4. Image gs://... inputs should be sent as image.gcsUri; currently only video GCS URIs are handled that way.

5. The tests mostly exercise private helpers. We need coverage through the public API, e.g. RubyLLM.embed(..., model: "multimodalembedding", provider: :vertexai, with: ...), plus regression coverage for existing Vertex text embeddings.

6. The README example should quote the model name: model: "multimodalembedding".

I think the safest shape is to keep existing Vertex text embedding behavior unchanged, add a separate multimodal payload path only for multimodalembedding, avoid ActiveSupport runtime helpers, and add public API specs/VCR coverage for both text-only and multimodal cases.