Incorporate `jedi-language-server` semantic highlights

[rivershah]

jedi-language-server now implements semantic highlights:

pappasam/jedi-language-server#336 (comment)

Can this please be incorporated. Previously discussed here:

#23049

[karthiknadig]

@rivershah Can you try the latest pre-release of the python extension? this should be included in it. I just added the latest Jedi yesterday.

[rivershah]

@karthiknadig All works with pylance but not so with jedi. At the risk of too much info, here are logs with language server set to Pylance

2025-04-23 04:52:19.137 [info] (Client) Pylance async client (2025.4.1) started with python extension (2025.5.2025042201)
2025-04-23 04:52:19.746 [info] [Info  - 4:52:19 AM] (1465) Server root directory: file:///root/.vscode-server/extensions/ms-python.vscode-pylance-2025.4.1/dist
2025-04-23 04:52:19.748 [info] [Info  - 4:52:19 AM] (1465) Pylance language server 2025.4.1 (pyright version 1.1.398, commit 4f24eccc) starting
2025-04-23 04:52:19.770 [info] [Info  - 4:52:19 AM] (1465) Starting service instance "<project>" for workspace "/workspaces/<project>"
2025-04-23 04:52:19.877 [info] [Info  - 4:52:19 AM] (1465) Loading pyproject.toml file at /workspaces/<project>/pyproject.toml
2025-04-23 04:52:19.882 [info] [Info  - 4:52:19 AM] (1465) Setting environmentName for service "<project>": "3.12.10 (global)"
2025-04-23 04:52:19.883 [info] [Info  - 4:52:19 AM] (1465) Setting pythonPath for service "<project>": "/usr/local/bin/python"
2025-04-23 04:52:19.883 [info] [Info  - 4:52:19 AM] (1465) No include entries specified; assuming /workspaces/<project>
2025-04-23 04:52:19.884 [info] [Info  - 4:52:19 AM] (1465) Auto-excluding **/node_modules
2025-04-23 04:52:19.884 [info] [Info  - 4:52:19 AM] (1465) Auto-excluding **/__pycache__
2025-04-23 04:52:19.884 [info] [Info  - 4:52:19 AM] (1465) Auto-excluding **/.*
2025-04-23 04:52:19.950 [info] [Info  - 4:52:19 AM] (1465) Assuming Python version 3.12.10.final.0
2025-04-23 04:52:20.029 [info] [Info  - 4:52:19 AM] (1465) Found 17 source files
2025-04-23 04:52:20.079 [info] [Info  - 4:52:19 AM] (1465) Loading pyproject.toml file at /workspaces/<project>/pyproject.toml
2025-04-23 04:52:20.082 [info] [Info  - 4:52:19 AM] (1465) Setting environmentName for service "<project>": "3.12.10 (global)"
2025-04-23 04:52:20.083 [info] [Info  - 4:52:19 AM] (1465) Setting pythonPath for service "<project>": "/usr/local/bin/python"
2025-04-23 04:52:20.083 [info] [Info  - 4:52:19 AM] (1465) No include entries specified; assuming /workspaces/<project>
2025-04-23 04:52:20.084 [info] [Info  - 4:52:19 AM] (1465) Auto-excluding **/node_modules
2025-04-23 04:52:20.084 [info] [Info  - 4:52:19 AM] (1465) Auto-excluding **/__pycache__
2025-04-23 04:52:20.084 [info] [Info  - 4:52:19 AM] (1465) Auto-excluding **/.*
2025-04-23 04:52:20.084 [info] [Info  - 4:52:19 AM] (1465) Assuming Python version 3.12.10.final.0
2025-04-23 04:52:20.154 [info] [Info  - 4:52:19 AM] (1465) Found 17 source files
2025-04-23 04:52:20.567 [info] [Info  - 4:52:19 AM] (1465) BG: Priority queue background worker(2) root directory: file:///root/.vscode-server/extensions/ms-python.vscode-pylance-2025.4.1/dist
2025-04-23 04:52:20.569 [info] [Info  - 4:52:19 AM] (1465) BG: Priority queue background worker(2) started
2025-04-23 04:52:20.747 [info] [Info  - 4:52:19 AM] (1465) BG: Indexer background runner(3) root directory: file:///root/.vscode-server/extensions/ms-python.vscode-pylance-2025.4.1/dist (index)
2025-04-23 04:52:20.747 [info] [Info  - 4:52:19 AM] (1465) BG: Indexing(3) started
2025-04-23 04:52:21.456 [info] [Info  - 4:52:19 AM] (1465) BG: scanned(3) 149 files over 1 exec env
2025-04-23 04:52:23.728 [info] [Info  - 4:52:19 AM] (1465) BG: [IDX(3)] Long operation: index execution environment file:///workspaces/<project> (2031ms)
2025-04-23 04:52:23.756 [info] [Info  - 4:52:19 AM] (1465) BG: [IDX(3)] Long operation: index packages file:///workspaces/<project> (2070ms)
2025-04-23 04:52:23.756 [info] [Info  - 4:52:19 AM] (1465) BG: indexed(3) 149 files over 1 exec env
2025-04-23 04:52:23.816 [info] [Info  - 4:52:19 AM] (1465) BG: Indexing finished(3).
2025-04-23 04:52:42.038 [info] [Info  - 4:52:19 AM] (1465) BG: Indexer background runner(4) root directory: file:///root/.vscode-server/extensions/ms-python.vscode-pylance-2025.4.1/dist (refresh)
2025-04-23 04:52:42.038 [info] [Info  - 4:52:19 AM] (1465) BG: Indexing(4) started
2025-04-23 04:52:42.465 [info] [Info  - 4:52:19 AM] (1465) BG: scanned(4) 149 files over 1 exec env
2025-04-23 04:52:42.713 [info] [Info  - 4:52:19 AM] (1465) BG: indexed(4) 149 files over 1 exec env
2025-04-23 04:52:42.767 [info] [Info  - 4:52:19 AM] (1465) BG: Indexing finished(4).

Here are logs with language server set to Jedi (semantic highlights don't work).

2025-04-23 04:54:48.661 [info] semantic_tokens_full <file>
2025-04-23 04:54:48.666 [info] semantic_tokens_range <file> 0:0-69:0
2025-04-23 04:54:50.519 [info] [Debug - 4:54:50 AM] multiple definitions found for name "path" of type 'statement' (7:19):
 <Name full_name='os.path', description='module ntpath'>
 <Name full_name='os.path', description='module posixpath'>
2025-04-23 04:54:50.521 [info] [Debug - 4:54:50 AM] multiple definitions found for name "path" of type 'statement' (7:19):
 <Name full_name='os.path', description='module posixpath'>
 <Name full_name='os.path', description='module ntpath'>
…  
2025-04-23 04:54:51.431 [info] ERROR:pygls.protocol.json_rpc:Exception occurred for message "4": ValueError: not enough values to unpack (expected 1, got 0)
2025-04-23 04:54:51.438 [info] [Error - 4:54:51 AM] Request textDocument/semanticTokens/full failed.
2025-04-23 04:54:51.438 [info]   Message: ValueError: not enough values to unpack (expected 1, got 0)
2025-04-23 04:54:51.757 [info] ERROR:pygls.protocol.json_rpc:Exception occurred for message "5": AssertionError
2025-04-23 04:54:51.758 [info] [Error - 4:54:51 AM] Request textDocument/semanticTokens/range failed.
2025-04-23 04:54:51.758 [info]   Message: AssertionError
…  
2025-04-23 04:56:00.439 [info] semantic_tokens_range <file> 0:0-69:0
2025-04-23 04:56:00.454 [info] ERROR:pygls.protocol.json_rpc:Exception occurred for message "7": ValueError: not enough values to unpack (expected 1, got 0)
…  
2025-04-23 04:56:04.629 [info] ERROR:pygls.protocol.json_rpc:Failed to handle request 11 textDocument/documentHighlight DocumentHighlightParams(text_document=TextDocumentIdentifier(uri='<file>'), position=10:60)
…  
2025-04-23 04:56:04.633 [info] [Error - 4:56:04 AM] Request textDocument/documentHighlight failed.
2025-04-23 04:56:04.633 [info]   Message: ValueError: not enough values to unpack (expected 1, got 0)
…  
2025-04-23 04:56:06.129 [info] semantic_tokens_full <file>
2025-04-23 04:56:06.297 [info] [Error - 4:56:06 AM] Request textDocument/semanticTokens/full failed.
2025-04-23 04:56:06.331 [info] ERROR:pygls.protocol.json_rpc:Failed to handle request 21 textDocument/documentHighlight DocumentHighlightParams(text_document=TextDocumentIdentifier(uri='<file>'), position=75:4)
…  
2025-04-23 04:56:06.332 [info] [Error - 4:56:06 AM] Request textDocument/documentHighlight failed.
2025-04-23 04:56:06.333 [info]   Message: AssertionError
…  
2025-04-23 04:56:06.366 [info] semantic_tokens_range <file> 24:0-116:0
2025-04-23 04:56:06.381 [info] [Error - 4:56:06 AM] Request textDocument/semanticTokens/range failed.
…  
2025-04-23 04:56:07.411 [info] semantic_tokens_full <file>
2025-04-23 04:56:07.421 [info] semantic_tokens_range <file> 0:0-69:0
2025-04-23 04:56:07.426 [info] [Error - 4:56:07 AM] Request textDocument/semanticTokens/full failed.

[karthiknadig]

The fix was that this was not enabled when we sent the options to JediLS:

[karthiknadig]

Verification steps:

1. set python.languageServer setting to Jedi
2. Looks at Output > Python Langauge Server logs for semantic tokens

[rivershah]

Now I see this please. Server also crashes after this error:

2025-04-24 12:13:53.988 [info] [Error - 12:13:53 PM] Request textDocument/semanticTokens/range failed.
2025-04-24 12:13:53.988 [info]   Message: _pickle.UnpicklingError: invalid load key, ' '.
  Code: -32603 
[object Object]
2025-04-24 12:13:54.844 [info] ERROR:pygls.protocol.json_rpc:Failed to handle request 31 textDocument/codeAction CodeActionParams(text_document=TextDocumentIdentifier(uri='file:///[REDACTED_PATH]'), range=100:35-100:35, context=CodeActionContext(diagnostics=[], only=None, trigger_kind=<CodeActionTriggerKind.Automatic: 2>), work_done_token=None, partial_result_token=None)
Traceback (most recent call last):
  File "[REDACTED]/json_rpc.py", line 266, in _handle_request
    self._execute_request(msg_id, handler, params)
  File "[REDACTED]/json_rpc.py", line 188, in _execute_request
    self._send_response(msg_id, handler(params))
                                ^^^^^^^^^^^^^^^
  File "[REDACTED]/notebook_utils.py", line 373, in wrapped
    result = f(notebook_server, notebook_params)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "[REDACTED]/server.py", line 717, in code_action
    extract_function_refactoring = jedi_script.extract_function(
                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "[REDACTED]/helpers.py", line 487, in wrapper
    return func(self, line, column, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "[REDACTED]/__init__.py", line 675, in extract_function
    return extract_function(
           ^^^^^^^^^^^^^^^^^
  File "[REDACTED]/extract.py", line 216, in extract_function
    params, return_variables = list(_find_inputs_and_outputs(module_context, context, nodes))
                                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "[REDACTED]/extract.py", line 332, in _find_inputs_and_outputs
    name_definitions = context.goto(name, name.start_pos)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "[REDACTED]/context.py", line 34, in goto
    names = finder.filter_name(filters, name_or_str)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "[REDACTED]/finder.py", line 36, in filter_name
    names = filter.get(string_name)
            ^^^^^^^^^^^^^^^^^^^^^^^
  File "[REDACTED]/filters.py", line 109, in get
    return self._convert_names(self._filter(
                               ^^^^^^^^^^^^^
  File "[REDACTED]/filters.py", line 145, in _filter
    return list(self._check_flows(names))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "[REDACTED]/filters.py", line 156, in _check_flows
    check = flow_analysis.reachability_check(
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "[REDACTED]/flow_analysis.py", line 85, in reachability_check
    return _break_check(context, value_scope, first_flow_scope, node)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "[REDACTED]/flow_analysis.py", line 93, in _break_check
    reachable = _check_if(context, check_node)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "[REDACTED]/flow_analysis.py", line 120, in _check_if
    types = context.infer_node(node)
            ^^^^^^^^^^^^^^^^^^^^^^^^
  File "[REDACTED]/context.py", line 224, in infer_node
    return infer_node(self, node)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "[REDACTED]/syntax_tree.py", line 157, in infer_node
    return _infer_node_if_inferred(context, element)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "[REDACTED]/syntax_tree.py", line 170, in _infer_node_if_inferred
    return _infer_node_cached(context, element)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "[REDACTED]/cache.py", line 44, in wrapper
    rv = function(obj, *args, **kwargs)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "[REDACTED]/syntax_tree.py", line 175, in _infer_node_cached
    return _infer_node(context, element)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "[REDACTED]/debug.py", line 81, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "[REDACTED]/syntax_tree.py", line 83, in wrapper
    return func(context, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "[REDACTED]/syntax_tree.py", line 218, in _infer_node
    value_set = context.infer_node(element.children[-1])
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "[REDACTED]/context.py", line 224, in infer_node
    return infer_node(self, node)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "[REDACTED]/syntax_tree.py", line 157, in infer_node
    return _infer_node_if_inferred(context, element)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "[REDACTED]/syntax_tree.py", line 170, in _infer_node_if_inferred
    return _infer_node_cached(context, element)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "[REDACTED]/cache.py", line 44, in wrapper
    rv = function(obj, *args, **kwargs)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "[REDACTED]/syntax_tree.py", line 175, in _infer_node_cached
    return _infer_node(context, element)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "[REDACTED]/debug.py", line 81, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "[REDACTED]/syntax_tree.py", line 185, in _infer_node
    return infer_atom(context, element)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "[REDACTED]/syntax_tree.py", line 305, in infer_atom
    return context.py__getattribute__(atom, position=position)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "[REDACTED]/context.py", line 77, in py__getattribute__
    values = ValueSet.from_sets(name.infer() for name in names)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "[REDACTED]/base_value.py", line 430, in from_sets
    for set_ in sets:
                ^^^^
  File "[REDACTED]/context.py", line 77, in <genexpr>
    values = ValueSet.from_sets(name.infer() for name in names)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "[REDACTED]/names.py", line 281, in infer
    return tree_name_to_values(
           ^^^^^^^^^^^^^^^^^^^^
  File "[REDACTED]/__init__.py", line 21, in wrapper
    return built_functions[public_name](*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "[REDACTED]/stdlib.py", line 878, in wrapper
    return func(inference_state, context, tree_name)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "[REDACTED]/django.py", line 177, in wrapper
    result = func(inference_state, context, tree_name)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "[REDACTED]/syntax_tree.py", line 771, in tree_name_to_values
    types = imports.infer_import(context, tree_name)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "[REDACTED]/cache.py", line 44, in wrapper
    rv = function(obj, *args, **kwargs)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "[REDACTED]/imports.py", line 53, in infer_import
    _prepare_infer_import(module_context, tree_name)
  File "[REDACTED]/imports.py", line 116, in _prepare_infer_import
    return from_import_name, tuple(import_path), import_node.level, importer.follow()
                                                                    ^^^^^^^^^^^^^^^^^
  File "[REDACTED]/imports.py", line 296, in follow
    return import_module_by_names(
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "[REDACTED]/imports.py", line 377, in import_module_by_names
    import_module(
  File "[REDACTED]/__init__.py", line 21, in wrapper
    return built_functions[public_name](*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "[REDACTED]/flask.py", line 20, in wrapper
    return callback(inference_state, import_names, module_context, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "[REDACTED]/typeshed.py", line 115, in wrapper
    python_value_set = ValueSet.from_sets(
                       ^^^^^^^^^^^^^^^^^^^
  File "[REDACTED]/imports.py", line 411, in import_module
    file_io_or_ns, is_pkg = inference_state.compiled_subprocess.get_module_info(
                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "[REDACTED]/subprocess.py", line 188, in wrapper
    result = self._compiled_subprocess.run(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "[REDACTED]/subprocess.py", line 293, in run
    self._send(delete_id, None)
  File "[REDACTED]/subprocess.py", line 318, in _send
    is_exception, traceback, result = pickle_load(self._get_process().stdout)
                                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "[REDACTED]/_compatibility.py", line 25, in pickle_load
    return Unpickler(file).load()
           ^^^^^^^^^^^^^^^^^^^^^^
_pickle.UnpicklingError: invalid load key, ' '.

2025-04-24 12:13:54.847 [info] [Error - 12:13:54 PM] Request textDocument/codeAction failed.
2025-04-24 12:13:54.847 [info]   Message: _pickle.UnpicklingError: invalid load key, ' '.
  Code: -32603 
[object Object]

[karthiknadig]

This should be filed on jedi-language-server. All we do is launch it with settings.

We can look into adding settings for Jedi so this can be controlled this easily.