Autocomplete silently produces nothing when Ollama Modelfile `num_ctx` equals Continue's default `maxTokens` (4096) — `pruneLength` computes a negative prompt budget and prunes prefix/suffix to empty

[japric]

[japric]

Before submitting your bug report

• I've tried finding an answer on the Continue docs site
• I'm not able to find an open issue that reports the same bug (searched pruneLength, num_ctx, autocomplete silent, contextLength maxTokens; closest hit is Error streaming edit diffs: Token limit reached. File/range likely too large for this edit #7544 which is the edit-mode variant with a visible error — this one is the autocomplete-mode variant with no error and no output at all)
• I've seen the troubleshooting guide on the Continue Docs

Relevant environment info

• OS: Ubuntu 22.04
• Continue version: 1.2.1
• IDE version: VSCode 1.103.2
• Model: qwen2.5-coder:1.5b (Ollama 0.13.5, local)
• GPU: NVIDIA MX250 (2 GB VRAM) — relevant only because it motivated me to pin num_ctx to 4096 to keep the model fully in VRAM
• config (~/.continue/config.yaml):

name: Local Agent
version: 1.0.0
schema: v1
models:
  - name: Qwen 2.5 Coder 1.5b AllGPU
    provider: ollama
    model: qwen2.5_coder-1.5b:allgpu
    roles:
      - autocomplete
      - chat
    # NOTE: no explicit contextLength / maxTokens here — both fall back to defaults
  - name: Autodetect
    provider: ollama
    model: AUTODETECT
    roles:
      - autocomplete
      - chat
      - edit
      - apply
context:
  - provider: code
  - provider: docs
  - provider: diff
  - provider: terminal
  - provider: problems
  - provider: folder
  - provider: codebase

The Ollama Modelfile for qwen2.5_coder-1.5b:allgpu contains:

FROM <…qwen2.5-coder-1.5b-instruct-q4_k_m.gguf…>
PARAMETER num_ctx 4096
PARAMETER num_gpu 32

For comparison, the qwen2.5_coder-1.5b:latest tag has no num_ctx line in its Modelfile (everything else identical, same weight blob).

Description

Tab autocomplete silently produces nothing — no completion item, no error message, no log line — when the Ollama Modelfile for the selected autocomplete model contains PARAMETER num_ctx 4096 and the user has not explicitly overridden maxTokens in config.yaml. The same model tag without the num_ctx line (:latest) autocomplete works fine.

Direct curl to http://localhost:11434/api/generate with identical prompts returns correct completions for both tags, so the Ollama side is healthy. The failure is 100% client-side, inside Continue's autocomplete pipeline.

Root cause (traced in the bundled extension.js)

When the Ollama provider is instantiated, Continue calls POST /api/show and parses the parameters string line by line (extension.js:731409–731451). A num_ctx line overwrites this._contextLength:

// extension.js:731428
case "num_ctx":
  this._contextLength = options.contextLength ?? Number.parseInt(value2);
  break;

If the user did not set contextLength in config.yaml, the Modelfile value wins. With num_ctx 4096 in the Modelfile, Continue sees contextLength = 4096.

If the user did not set maxTokens either, maxTokens falls back to DEFAULT_MAX_TOKENS = 4096 (extension.js:239811).

For models without a num_ctx line in their Modelfile, contextLength falls back to DEFAULT_CONTEXT_LENGTH = 32768 (extension.js:239813 + the getter at 243810).

The autocomplete pipeline then calls pruneLength (extension.js:252546):

function pruneLength(llm, prompt2) {
  const contextLength = llm.contextLength;
  const reservedTokens = llm.completionOptions.maxTokens ?? DEFAULT_MAX_TOKENS;
  const safetyBuffer = getTokenCountingBufferSafety(contextLength);   // Math.min(1000, contextLength * 0.02)
  const maxAllowedPromptTokens = contextLength - reservedTokens - safetyBuffer;
  const promptTokenCount = countTokens(prompt2, llm.model);
  return promptTokenCount - maxAllowedPromptTokens;
}

There is no Math.max(0, maxAllowedPromptTokens) clamp. Substituting the two tags:

Tag	contextLength	maxTokens	safetyBuffer	maxAllowedPromptTokens	prune for any non-empty prompt
:latest (no num_ctx)	32768	4096	655.36	28017	negative → no pruning ✅
:allgpu (num_ctx 4096)	4096	4096	81.92	−81.92	always positive → always over-prunes ❌

renderPromptWithTokenLimit (extension.js:252585–252624) enters the prune > 0 branch and computes:

const allowedPrefixTokens = Math.max(0, prefixTokenCount - dropPrefix);  // = 0 when dropPrefix > prefixTokenCount
const allowedSuffixTokens = Math.max(0, suffixTokenCount - dropSuffix);  // = 0 when dropSuffix > suffixTokenCount
prefix = pruneLinesFromTop(prefix, allowedPrefixTokens, helper.modelName);
suffix = pruneLinesFromBottom(suffix, allowedSuffixTokens, helper.modelName);

pruneLinesFromTop(prompt, 0, …) (extension.js:243283–243297) loops while (totalTokens > 0 && start < length) start++ until start === length, then returns lines.slice(length).join("\n") — the empty string. Same for suffix.

The request is then sent to Ollama with prefix === "" and suffix === "\n" (the fallback). The model has no anchor and emits an empty or garbage completion. postprocessCompletion (extension.js:251045) drops it via if (isBlank(completion)) return void 0;, and provideInlineCompletionItems returns undefined (extension.js:254898). No error, no log, no UI feedback.

Note that _streamFim (extension.js:731688–731722) does not gate on contextLength and always fires the request, and num_ctx: this.contextLength (extension.js:731507) is what Continue POSTs back to Ollama as the KV cache size — so the bug is purely in the client-side prompt-budget arithmetic, not in the Ollama round-trip.

Why this is distinct from #7544

#7544 is the same family (contextLength − maxTokens − safetyBuffer goes negative or too small) but in the edit pipeline, where it surfaces as a visible Error streaming edit diffs: Token limit reached. File/range likely too large for this edit. This issue is the autocomplete variant, which has no such error path — the failure is completely silent. The config validator at extension.js:239188 only checks model.defaultCompletionOptions fields that the user wrote explicitly; it does not validate the _contextLength value derived from /api/show, so no warning is emitted either.

Why #7702 does not fix this

#7702 (merged in Continue 1.2.11) overrides the Ollama contextLength getter to fall back to 8192 instead of 32768 when no num_ctx is present. That helps the no-num_ctx case only when the user has not set contextLength in config. It does not help any Modelfile that explicitly sets num_ctx 4096 (Continue still reads 4096 via /api/show), and it does not touch the pruneLength arithmetic at extension.js:252550. So upgrading Continue does not fix this case.

To reproduce

1. Pull a small coder model in Ollama and create two tags — one with num_ctx 4096 pinned, one without:

   ollama pull qwen2.5-coder:1.5b
   ollama show qwen2.5-coder:1.5b --modelfile > /tmp/M.allgpu
   printf 'PARAMETER num_ctx 4096\n' >> /tmp/M.allgpu
   ollama create qwen2.5-coder:1.5b:allgpu -f /tmp/M.allgpu

2. In ~/.continue/config.yaml add one model entry with roles: [autocomplete] and no explicit contextLength or maxTokens:

   models:
     - name: AllGPU
       provider: ollama
       model: qwen2.5-coder:1.5b:allgpu
       roles: [autocomplete]

3. Reload VSCode (Developer: Reload Window).

4. Open a small Python file, e.g.:

   # helloworld
   def hello(): print("Hello, world!")
   
   if __name__ == "__main__":
       █

5. Trigger autocomplete at the cursor. No completion appears. No error in the Continue output channel.

6. Switch the autocomplete model to qwen2.5-coder:1.5b (the :latest tag without num_ctx).

7. Reload VSCode. Trigger autocomplete at the same cursor. hello() is suggested.

8. Direct API sanity check (both tags return the same correct completion):

   curl -s http://localhost:11434/api/generate -d '{
     "model": "qwen2.5-coder:1.5b:allgpu",
     "prompt": "# helloworld\ndef hello(): print(\"Hello, world!\")\n\nif __name__ == \"__main__\":\n    ",
     "stream": false,
     "options": {"temperature": 0.01, "num_predict": 40, "stop": ["\n\n"]}
   }'

Expected behavior

When contextLength is less than or equal to reservedTokens + safetyBuffer, Continue should not compute a negative prompt budget and silently prune the prompt to empty. At minimum:

1. pruneLength should clamp maxAllowedPromptTokens to Math.max(0, …) (or MIN_RESPONSE_TOKENS), so the worst case is "badly trimmed but still sends something", not "sends nothing and tells no-one".
2. The config validator at extension.js:239188 should also validate the /api/show-derived _contextLength vs maxTokens and emit a non-fatal warning so users at least see that their inputs are being truncated.
3. postprocessCompletion should distinguish "blank because the model produced nothing useful" from "blank because the client sent an empty prompt"; the latter should log a warning.

Workarounds (for anyone hitting this today)

Any one of these fixes it:

1. In config.yaml, give the model an explicit small maxTokens so the budget is positive:

   - name: AllGPU
     provider: ollama
     model: qwen2.5-coder:1.5b:allgpu
     roles: [autocomplete]
     defaultCompletionOptions:
       maxTokens: 128    # 4096 − 128 − 81 = 3887 > 0

2. In config.yaml, give the model an explicit larger contextLength (Continue will POST it back to Ollama as num_ctx, so the server KV cache must actually fit):

   defaultCompletionOptions:
     contextLength: 8192

3. Remove the PARAMETER num_ctx 4096 line from the Ollama Modelfile (Continue will fall back to DEFAULT_CONTEXT_LENGTH).

Related issues / PRs

• Error streaming edit diffs: Token limit reached. File/range likely too large for this edit #7544 — same arithmetic bug in the edit pipeline (surfaces as a visible "Token limit reached" error)
• fix: ollama detect context length automatically #7702 — fixes the no-num_ctx default but does not touch pruneLength
• contextLength not being respected for Ollama #2809 — long-standing contextLength vs num_ctx inconsistency between Continue and Ollama
• Malformed stop-parameter on ollama API Requenst (quotes included \"[INST]\" instead of [INST]) #1116 — another bug in the same parameters-string parser (stop tokens), showing this code path has history
• Ollama autocomplete model unloads chat model despite keep_alive: -1 and num_gpu settings #10378 — Ollama autocomplete + num_ctx / num_gpu options not being respected
• Autocomplete pipeline silently fails when default config.ts exists alongside config.yaml on Windows #12544 — autocomplete silently fails (different root cause: config file conflict), same "no error, no log" symptom

Log output

No log output is produced by this bug — that is part of the problem. The Continue output channel shows normal stream providerID=ollama modelID=… step=… lines and then a normal done for Ollama's response; the truncation happens before the request and the discard happens after, both silently.