Skip to content

[Feature Request] Add more 1v1 text-only negotiation tasks for LLM-only / non-VLM users #2

Description

@LUTIAN233

Problem

Currently, the single_buyer_product_seller category has 28 tasks, but the breakdown by modality is highly skewed:

Type Tasks Can run with LLM-only?
Text-only, price-only Task1, Task2, Task3 ✅ Yes
Multimodal + contract Task4–Task28 (25 tasks) ❌ Requires VLM (image input)

Users who run local LLMs via vLLM/SGLang without vision support (e.g., --language-model-only) can only evaluate on 3 out of 28 single-buyer-single-seller tasks. This makes the benchmark nearly unusable for text-model evaluation or RL training with local models.

Request

Add more text-only, 1v1 negotiation tasks that:

  1. Do not require image input — no product_info.image_url, runnable with BaseLLM / OpenAILLM / CustomLLM / VLLMLLM
  2. Cover more diverse scenarios — the 4 scenario families (E-commerce, Taxi, Food Delivery, Rent House) already have rich text metadata per task (product descriptions, pricing rules, surcharge breakdowns) that work perfectly fine without images
  3. Optionally include contract_config — multi-dimensional contracts can also be done purely via text, since all field_descriptions, continuous_bounds, discrete_options are already text-encoded in the prompt

Proposed approach

The simplest way would be to provide "text-only variants" of the existing s1–s25 scenario tasks — same contract_config, same product_info text, same scoring, just dropping the product_info.image_url line. These could be registered under environment IDs like Task_s1_text-v0.

Alternatively, a batch of 10–20 new text-only tasks with varied difficulty levels (close-price, wide-spread, market-aligned, multi-round grinding) would be very helpful for systematic evaluation of text-only models.

Impact

This would significantly lower the barrier for:

  • Researchers evaluating open-weight LLMs (Qwen, LLaMA, DeepSeek) without VLM capability
  • RL training experiments (e.g., GRPO/PPO on negotiation) that use text-only environments
  • Reproducibility studies on commodity GPU setups where VLM inference is prohibitively expensive
    (written by deepseek)

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions