Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 16 additions & 4 deletions .github/skills/vscode-dev-workbench/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,12 +21,20 @@ If your paths differ, check `server/` in `vscode-dev` for the source root resolu

## Start the dev server

**Critical:** Run `npm run dev` from the **`vscode-dev`** folder, NOT from `vscode`. The `vscode` repo has no `dev` script and will fail with `npm error Missing script: "dev"`. Terminal tools that simplify/strip leading `cd` into separate commands will silently keep the cwd of a previous terminal — always use an absolute `pushd` or verify with `pwd` before `npm run dev`.

```bash
cd vscode-dev
npm run dev # runs watch + nodemon; serves https://127.0.0.1:3000
cd /path/to/vscode-dev # NOT /path/to/vscode
npm run dev # runs watch + nodemon; serves https://127.0.0.1:3000
```

On first start you may see one crash like `Cannot find module './indexes'` — it's the watcher racing the first build. nodemon restarts automatically once `out/` finishes compiling.
If you're driving this through an agent/terminal tool, prefer:

```bash
pushd /absolute/path/to/vscode-dev >/dev/null && pwd && npm run dev
```

On first start you may see one crash like `Cannot find module './indexes'` — it's the watcher racing the first build. nodemon restarts automatically once `out/` finishes compiling. The server is ready when `curl -sk -o /dev/null -w "%{http_code}" https://127.0.0.1:3000/` returns `200`.

## URLs

Expand Down Expand Up @@ -97,15 +105,19 @@ For a true mobile viewport, drive a standalone Playwright script with `devices['

## Testing the Agents window against a local mock agent host

If the scenario touches the Agents window (`/agents` route), you almost always need the mock agent host running. Without it, the Agents window will sit on the sign-in / tunnel-discovery screen and block any real interaction. Start it **in addition to** the dev server — it's a second terminal, not a replacement.

`vscode-dev` supports a `?mock-agent-host=ws://…` URL parameter that short-circuits tunnel discovery and wires the Agents window to a raw WebSocket. Pair it with the mock agent host binary from `microsoft/vscode`:

```bash
cd vscode
cd /path/to/vscode
node out/vs/platform/agentHost/node/agentHostServerMain.js \
--enable-mock-agent --quiet --without-connection-token --port 8765
# Listens on ws://localhost:8765
```

Prerequisite: `out/` in the `vscode` repo must be populated by the `VS Code - Build` task (or `npm run watch`). If `out/vs/platform/agentHost/node/agentHostServerMain.js` is missing, start that task first.

`--enable-mock-agent` registers the `ScriptedMockAgent` from `src/vs/platform/agentHost/test/node/mockAgent.ts` with one pre-existing session. Seed additional sessions via the `VSCODE_AGENT_HOST_MOCK_SEED_SESSIONS` env var, using a comma-separated list of session URIs (for example, `VSCODE_AGENT_HOST_MOCK_SEED_SESSIONS=mock://pre-1,mock://pre-2`). Scripted prompts include `hello`, `use-tool`, `error`, `permission`, `write-file`, `run-safe-command`, `slow`, `client-tool`, `subagent`, etc. (see `mockAgent.ts` for the full list).

Then open:
Expand Down
33 changes: 32 additions & 1 deletion extensions/copilot/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -235,6 +235,27 @@
]
}
},
{
"name": "skill",
"toolReferenceName": "skill",
"displayName": "%copilot.tools.skill.name%",
"icon": "$(book)",
"userDescription": "%copilot.tools.skill.description%",
"modelDescription": "Invoke a skill to handle a user's request with specialized instructions and workflows.\n\nSkills are domain-specific capabilities discovered from SKILL.md files. When a user's task matches an available skill, call this tool to load and apply it. If the user types a slash command (e.g. \"/deploy\", \"/test\"), treat it as a skill invocation.\n\nUsage:\n- Pass the skill name only (no arguments).\n- Examples: skill: \"docx\", skill: \"deploy\", skill: \"fix-ci-failures\"\n\nRules:\n- Available skills appear in system-reminder messages earlier in the conversation.\n- BLOCKING: When a matching skill exists, you MUST call this tool before producing any other output about the task.\n- Never reference a skill without calling this tool.\n- Do not call this tool for a skill that is already active in the current turn (indicated by a <command-name> tag).\n- Do not use this tool for built-in commands such as /help or /clear.",
"when": "config.github.copilot.chat.skillTool.enabled",
"inputSchema": {
"type": "object",
"properties": {
"skill": {
"type": "string",
"description": "The skill name. E.g., \"commit\", \"review-pr\", or \"pdf\""
}
},
"required": [
"skill"
]
}
},
{
"name": "copilot_searchWorkspaceSymbols",
"toolReferenceName": "symbols",
Expand Down Expand Up @@ -1265,7 +1286,8 @@
"problems",
"readFile",
"viewImage",
"readNotebookCellOutput"
"readNotebookCellOutput",
"skill"
]
},
{
Expand Down Expand Up @@ -4411,6 +4433,15 @@
"experimental"
]
},
"github.copilot.chat.skillTool.enabled": {
"type": "boolean",
"default": false,
"markdownDescription": "%github.copilot.config.skill.enabled%",
"tags": [
"advanced",
"experimental"
]
},
"github.copilot.chat.executionSubagent.enabled": {
"type": "boolean",
"default": false,
Expand Down
3 changes: 3 additions & 0 deletions extensions/copilot/package.nls.json
Original file line number Diff line number Diff line change
Expand Up @@ -490,6 +490,9 @@
"copilot.tools.runSubagent.description": "Runs a task within an isolated subagent context. Enables efficient organization of tasks and context window management.",
"copilot.tools.searchSubagent.name": "Search Subagent",
"copilot.tools.searchSubagent.description": "Launch an iterative search-focused subagent to find relevant code in your workspace.",
"copilot.tools.skill.name": "Skill",
"copilot.tools.skill.description": "Execute a skill by name. Skills provide specialized capabilities, domain knowledge, and refined workflows.",
"github.copilot.config.skill.enabled": "Enable the skill tool in Copilot Chat. When enabled, skills are invoked via a dedicated skill tool instead of readFile.",
"github.copilot.config.searchSubagent.enabled": "Enable the search subagent tool for iterative code exploration in the workspace.",
"github.copilot.config.searchSubagent.useAgenticProxy": "Use the agentic proxy for the search subagent tool.",
"github.copilot.config.searchSubagent.model": "Model to use for the search subagent. When useAgenticProxy is enabled, defaults to 'vscode-agentic-search-router-a'. Otherwise defaults to the main agent model.",
Expand Down
3 changes: 3 additions & 0 deletions extensions/copilot/src/extension/intents/node/agentIntent.ts
Original file line number Diff line number Diff line change
Expand Up @@ -120,6 +120,9 @@ export const getAgentTools = async (accessor: ServicesAccessor, request: vscode.
const executionSubagentEnabled = configurationService.getExperimentBasedConfig(ConfigKey.Advanced.ExecutionSubagentToolEnabled, experimentationService);
allowTools[ToolName.ExecutionSubagent] = isGptOrAnthropic && executionSubagentEnabled;

const skillToolEnabled = configurationService.getExperimentBasedConfig(ConfigKey.Advanced.SkillToolEnabled, experimentationService);
allowTools[ToolName.Skill] = skillToolEnabled;

allowTools[CUSTOM_TOOL_SEARCH_NAME] = !!model.supportsToolSearch;

if (model.family.includes('grok-code')) {
Expand Down
25 changes: 21 additions & 4 deletions extensions/copilot/src/extension/intents/node/toolCallingLoop.ts
Original file line number Diff line number Diff line change
Expand Up @@ -383,12 +383,29 @@ export abstract class ToolCallingLoop<TOptions extends IToolCallingLoopOptions =
return undefined;
}

// If the model produced a substantive text response with no tool calls, treat it
// as a final summary and let the loop stop. Nudging in this case typically just
// wastes a turn — the model considers itself done. The user can always continue
// the conversation if it wasn't.
if (result.round.toolCalls.length === 0 && result.round.response.trim().length > 0) {
this._logService.info('[ToolCallingLoop] Autopilot: model produced a text-only response, treating as done');
return undefined;
}

// safety valve — only give up after exhausting all continuation attempts
if (this.autopilotIterationCount >= ToolCallingLoop.MAX_AUTOPILOT_ITERATIONS) {
this._logService.info(`[ToolCallingLoop] Autopilot: hit max iterations (${ToolCallingLoop.MAX_AUTOPILOT_ITERATIONS}), letting it stop`);
return undefined;
}

// If we already nudged once and the model still produced no tool calls, the model
// is effectively done — further nudges just waste tokens. Bail out and let the
// loop stop.
if (this.autopilotStopHookActive && result.round.toolCalls.length === 0) {
this._logService.info('[ToolCallingLoop] Autopilot: prior nudge produced no tool calls, stopping to avoid wasted requests');
return undefined;
}

this.autopilotIterationCount++;
return 'You have not yet marked the task as complete using the task_complete tool. ' +
'You must call task_complete when done — whether the task involved code changes, answering a question, or any other interaction.\n\n' +
Expand Down Expand Up @@ -900,7 +917,7 @@ export abstract class ToolCallingLoop<TOptions extends IToolCallingLoopOptions =
const permLevel = this.options.request.permissionLevel;
if (permLevel === 'autopilot' && this.options.toolCallLimit < 200) {
this.options.toolCallLimit = Math.min(Math.round(this.options.toolCallLimit * 3 / 2), 200);
this.showAutopilotProgress(outputStream, l10n.t('Extending tool call limit with Autopilot...'), l10n.t('Extended tool call limit with Autopilot'));
this.showAutopilotProgress(outputStream, l10n.t('Autopilot: extending tool call limit\u2026'), l10n.t('Autopilot extended tool call limit'));
} else {
lastResult = this.hitToolCallLimit(outputStream, lastResult);
break;
Expand Down Expand Up @@ -951,9 +968,9 @@ export abstract class ToolCallingLoop<TOptions extends IToolCallingLoopOptions =
this.autopilotRetryCount++;
this._logService.info(`[ToolCallingLoop] Auto-retrying on error (attempt ${this.autopilotRetryCount}/${ToolCallingLoop.MAX_AUTOPILOT_RETRIES}): ${result.response.type}`);
if (this.options.request.permissionLevel === 'autopilot') {
this.showAutopilotProgress(outputStream, l10n.t('Request failed, retrying with Autopilot...'), l10n.t('Request failed, retried with Autopilot'));
this.showAutopilotProgress(outputStream, l10n.t('Autopilot: recovering from a request error\u2026'), l10n.t('Autopilot recovered from a request error'));
} else {
this.showAutopilotProgress(outputStream, l10n.t('Request failed, retrying request...'), l10n.t('Request failed, retried request'));
this.showAutopilotProgress(outputStream, l10n.t('Recovering from a request error\u2026'), l10n.t('Recovered from a request error'));
}
await timeout(1000, token);
continue;
Expand Down Expand Up @@ -1003,7 +1020,7 @@ export abstract class ToolCallingLoop<TOptions extends IToolCallingLoopOptions =
const autopilotContinue = this.shouldAutopilotContinue(result);
if (autopilotContinue) {
this._logService.info(`[ToolCallingLoop] Autopilot internal stop hook: continuing because task may not be complete`);
this.showAutopilotProgress(outputStream, l10n.t('Continuing with Autopilot: Task not yet complete'), l10n.t('Continued with Autopilot: Task not yet complete'));
this.showAutopilotProgress(outputStream, l10n.t('Autopilot: verifying task is done\u2026'), l10n.t('Autopilot continued working'));
this.stopHookReason = autopilotContinue;
result.round.hookContext = formatHookContext([autopilotContinue]);
this.autopilotStopHookActive = true;
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -102,10 +102,10 @@ function createTestConversation(turnCount: number = 1): Conversation {
return new Conversation(generateUuid(), turns);
}

function createMockRound(toolCallNames: string[] = []): IToolCallRound {
function createMockRound(toolCallNames: string[] = [], response: string = ''): IToolCallRound {
return {
id: generateUuid(),
response: 'test response',
response,
toolInputRetry: 0,
toolCalls: toolCallNames.map(name => ({
id: generateUuid(),
Expand Down Expand Up @@ -150,10 +150,11 @@ describe('ToolCallingLoop autopilot', () => {
vi.restoreAllMocks();
});

function createLoop(permissionLevel?: string): AutopilotTestToolCallingLoop {
function createLoop(permissionLevel?: string, requestOverrides: Partial<ChatRequest> = {}): AutopilotTestToolCallingLoop {
const conversation = createTestConversation(1);
const request = createMockChatRequest({
permissionLevel,
...requestOverrides,
} as Partial<ChatRequest>);
const loop = instantiationService.createInstance(
AutopilotTestToolCallingLoop,
Expand Down Expand Up @@ -196,15 +197,24 @@ describe('ToolCallingLoop autopilot', () => {
expect(msg).toBeUndefined();
});

it('should keep nudging even with autopilotStopHookActive set', () => {
it('should bail when prior nudge produced no tool calls', () => {
const loop = createLoop('autopilot');

// Simulate that we already nudged once and set the flag
loop.setAutopilotStopHookActive(true);

// Should still return a nudge — autopilotStopHookActive no longer causes early bail
// Should bail — the previous nudge produced no tool calls, so further nudges
// would just waste tokens (the model is effectively done).
const result = loop.testShouldAutopilotContinue(createMockSingleResult());
expect(result).toContain('task_complete');
expect(result).toBeUndefined();
});

it('should skip the nudge when the model returned a text-only response (no tool calls)', () => {
const loop = createLoop('autopilot');
const result = loop.testShouldAutopilotContinue(createMockSingleResult({
round: createMockRound([], 'Here is a summary of what I did.'),
}));
expect(result).toBeUndefined();
});

it('should allow another nudge after autopilotStopHookActive is reset', () => {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -525,6 +525,8 @@ class SkillAdherenceReminder extends PromptElement<SkillAdherenceReminderProps>
constructor(
props: SkillAdherenceReminderProps,
@ICustomInstructionsService private readonly customInstructionsService: ICustomInstructionsService,
@IConfigurationService private readonly configurationService: IConfigurationService,
@IExperimentationService private readonly experimentationService: IExperimentationService,
) {
super(props);
}
Expand All @@ -541,6 +543,14 @@ class SkillAdherenceReminder extends PromptElement<SkillAdherenceReminderProps>
return undefined;
}

const skillToolEnabled = this.configurationService.getExperimentBasedConfig(ConfigKey.Advanced.SkillToolEnabled, this.experimentationService);

if (skillToolEnabled) {
return <Tag name='additional_skills_reminder'>
Always check if any skills apply to the user's request. If so, use the {ToolName.Skill} tool to invoke the skill by name. Multiple skill files may be needed for a single request. These files contain best practices built from testing that are needed for high-quality outputs.<br />
</Tag>;
}

return <Tag name='additional_skills_reminder'>
Always check if any skills apply to the user's request. If so, use the {ToolName.ReadFile} tool to read the corresponding SKILL.md files. Multiple skill files may be needed for a single request. These files contain best practices built from testing that are needed for high-quality outputs.<br />
</Tag>;
Expand Down
54 changes: 54 additions & 0 deletions extensions/copilot/src/extension/tools/common/skillTelemetry.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,54 @@
/*---------------------------------------------------------------------------------------------
* Copyright (c) Microsoft Corporation. All rights reserved.
* Licensed under the MIT License. See License.txt in the project root for license information.
*--------------------------------------------------------------------------------------------*/

import { ICustomInstructionsService, ISkillInfo } from '../../../platform/customInstructions/common/customInstructionsService';
import { IExtensionsService } from '../../../platform/extensions/common/extensionsService';
import { ITelemetryService } from '../../../platform/telemetry/common/telemetry';
import { hash } from '../../../util/vs/base/common/hash';
import { URI } from '../../../util/vs/base/common/uri';

/**
* Sends `skillContentRead` telemetry for a skill invocation.
* Shared between the skill tool and the readFile tool to ensure consistent
* telemetry when skills are loaded through either path.
*
* TODO: Add pluginNameHash and pluginVersion properties once vscode core's
* extensionPromptFileProvider command exposes IAgentPluginService metadata.
*/
export function sendSkillContentReadTelemetry(
telemetryService: ITelemetryService,
customInstructionsService: ICustomInstructionsService,
extensionsService: IExtensionsService,
uri: URI,
skillInfo: ISkillInfo,
content: string,
): void {
const extensionSkillInfo = customInstructionsService.getExtensionSkillInfo(uri);
const extensionId = extensionSkillInfo?.extensionId ?? '';
const extensionVersion = extensionId ? extensionsService.getExtension(extensionId)?.packageJSON?.version ?? '' : '';
const contentHash = content ? String(hash(content)) : '';

const plaintextProps = {
skillName: skillInfo.skillName,
skillPath: uri.toString(),
skillExtensionId: extensionId,
skillExtensionVersion: extensionVersion,
skillStorage: skillInfo.storage,
skillContentHash: contentHash,
};

telemetryService.sendGHTelemetryEvent('skillContentRead',
{
skillNameHash: String(hash(skillInfo.skillName)),
skillExtensionIdHash: extensionId ? String(hash(extensionId)) : '',
skillExtensionVersion: plaintextProps.skillExtensionVersion,
skillStorage: plaintextProps.skillStorage,
skillContentHash: contentHash,
}
);

telemetryService.sendEnhancedGHTelemetryEvent('skillContentRead', plaintextProps);
telemetryService.sendInternalMSFTTelemetryEvent('skillContentRead', plaintextProps);
}
2 changes: 2 additions & 0 deletions extensions/copilot/src/extension/tools/common/toolNames.ts
Original file line number Diff line number Diff line change
Expand Up @@ -74,6 +74,7 @@ export enum ToolName {
ToolSearch = 'tool_search',
ResolveMemoryFileUri = 'resolve_memory_file_uri',
ExecutionSubagent = 'execution_subagent',
Skill = 'skill',
SessionStoreSql = 'session_store_sql',
CoreOpenBrowserPage = 'open_browser_page',
CoreClickElement = 'click_element',
Expand Down Expand Up @@ -266,6 +267,7 @@ export const toolCategories: Record<ToolName, ToolCategory> = {
[ToolName.Memory]: ToolCategory.VSCodeInteraction,
[ToolName.ToolSearch]: ToolCategory.Core,
[ToolName.ResolveMemoryFileUri]: ToolCategory.Core,
[ToolName.Skill]: ToolCategory.Core,
[ToolName.SessionStoreSql]: ToolCategory.Core,
} as const;

Expand Down
1 change: 1 addition & 0 deletions extensions/copilot/src/extension/tools/node/allTools.ts
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@ import './githubTextSearchTool';
import './insertEditTool';
import './installExtensionTool';
import './listDirTool';
import './skillTool';
import './manageTodoListTool';
import './memoryTool';
import './multiReplaceStringTool';
Expand Down
Loading
Loading