[Remote-SSH] Extension Host with Copilot Fails on VS Code 1.95.3 but Works on 1.93.0 #234355
Comments
|
Does Copilot Chat work? Please try using VS Code bisect to help us narrow down the version which broke this |
|
Thank you for your response. The result of VS Code bisect confirm that the issue is caused by the When VS Code bisect disable GitHub Copilot alone, GitHub Copilot Chat works normally. Additionally, we have verified that Please let us know if further information is needed. |
|
Are you saying that you are using copilot 1.245.0 on both 1.93 and 1.95.3. It's not possible for Copilot Chat 0.22.4 to be used on 1.93 due to breaking API changes. |
|
Sorry for providing incorrect information in the previous update. |
|
We did have a runtime version bump between 1.93 and 1.95, so would be good to know the nature of crash. The node executable has debug symbols so you should be able to analyze the dump file |
|
We found these in our And only For analyze the dump file, if we use here 1.95.3 as an example. Or how should we find Will a core dump file be generated when the 1.95.3 extension host process crash? |
|
We found this log file at Maybe this suggest that the version of If further testing is needed, please let us know. |
|
@technic960183 the path to the node binary is correct, as for the core file location it will depend on your OS configuration. Can you share the your OS information |
|
@deepak1556 Here is the OS-info Also Our system admin said that we didn't config |
|
Thanks, to confirm have you set |
|
Thanks for your reply. After running However, when we tried to login from VS Code again. There was still no core dump file in this directory. The error message from VS Code was
Will a core dump happen automatically? Or we need to setup something to let it happened? For For We do not have it installed. |
|
Yes the dump file should be generated if VSCode server crashes and the core file size if correctly configured via
If you can configure it for a specific user and then connect vscode for that user it should work. |
|
It is also unclear whether you are seeing a non-native exception that triggers the termination of the extension host, can you start VSCode server with the following env variable when connecting to the login node This will generate a report file under |
|
@deepak1556 Now, we are working with our system admin to resolve the problem. For the Update: We are trying to add It will be good to know how should we config the "shell" use by these VSCode server process: We tried to set it in the Also, when we did this in the bash after So we removed We are not sure whether the |
|
The extension host is spawned with login shell environment, so it should get the
Sorry the option was only added in Node.js v23 and is not supported in the version used by our current remote server, removing |
|
Thanks. We did the following step:
Do we need to email the whole report file to you? Or only send it with the real crash by Copilot? We also set |
|
Thanks for confirming, this rules out any uncaught exception or fatal errors from the runtime.
Not required as it is a test report Can you attach the full remote agent logs from the scenario when the extension host terminates by the copilot extension. |
|
This file remote-server.log is the log file Is there any change from VS Code |
Important UpdateAfter our system admin found out and disabled the address space limit policy on the login node, Copilot with VS Code 1.95.3 run correctly! This also explain that why Copilot can run on computing node, as they are no memory limiting rules. |
|
That is great news, thanks for the update. From the logs, we are hitting OOM from Heap::Setup, the maximum code range size that can be requested is 512MB for 64-bit systems without V8 pointer compression. So definitely the OOM was a side effect of the process already reaching the limit before this call could be made. Based on your scenario, you only saw the extension host crash when the copilot extension got activated so the main thread heap was setup without issues, it hints like a worker thread from the extension that failed to setup its heap (a core dump could validate this). Also the OOM didn't create a report since V8 couldn't extract a error callback from the isolate which is what Node.js relies on to generate a report. I am puzzled by how this could happen but that's a different issue.
We bumped Node.js from 20.15.1 to 20.18.0 but there were no version bumps to V8, I doubt a change in the runtime to trigger this. However 32GB sounds good enough size unless we are creating hundreds of worker threads which shouldn't be the case. Can you help validate the memory usage in 1.95
This should give a better idea about the worker_thread counts and virtual memory usage. |
|
We found that 4 of the Copilot & Copilot-Chat enable
From From Copilot & Copilot-Chat disable
From From We have preformed the test that you suggest, and will email you the test report and the Given the information we having now (especially |
|
Thanks for the process info, yeah it is clear that copilot extension was not the primary cause for crash the extension host already reaches 22g for all the other allocations from Node.js and V8. The copilot extension does spawn worker threads which needs their own heap setups that ended up reaching the limit set for the process. I can drill a bit more on my side to understand if there were virtual memory allocation changes between 1.93 and 1.95.
The dump file is not needed at this time since we have pinpointed the crash. Followup question, Is there a requirement to cap the virtual memory on the login node ? would like to understand if the workaround can be applied permanently or not. |
|
Thanks for all the help and guidance. We are exploring alternative ways to achieve this without impacting performance. Or maybe it is possible to ask our system admin to raise the limit. Update:
This is the reason that virtual memory is limited. We might raise the limit for as a temporary solution, but if node.js claim even more address space in the future, it might still be a problem. |
|
We raised the virtual memory limit from 32GB to 40GB and this temporary solve the problem. We are not quite sure, but it seems that this function from V8 is the source of the problem. It return the largest possible address instead of following the maximum Do you consider this as a bug? Or we misunderstood how this code works? |
The function is only used in deciding address space limit for the v8 sandbox which is disabled in our remote server. I don't think the issue is from that. |
deepak1556
added
upstream
Type: Bug
Description
We are encountering an issue with the GitHub Copilot extension when using Visual Studio Code with Remote-SSH on an HPC login node. The extension works flawlessly with VS Code 1.93.0 (4849ca9) but fails on VS Code 1.95.3 (f1a4fb1). Interestingly, VS Code 1.95.3 works fine on the computing node of the same cluster, accessed via SSH forwarding from the login node.
To eliminate variables, we have tested the setup thoroughly:
1.245.0/0.20.3from the log file~/.vscode-server/data/logs/20241121T175642/remoteagent.log).vscode-serverdirectory on the remote machine was cleaned up to ensure a fresh environment.Expected Behavior
The GitHub Copilot extension should initialize and function correctly on both VS Code versions.
Actual Behavior
Test by VS Code Bisect
Done on our login node with VS Code 1.95.3. Result:
Extension Bisect is done and has identified github.copilot as the extension causing the problem.Before, we don't know that it is possible to disable GitHub Copilot alone but not GitHub Copilot Chat.
Later, we found that when VS Code bisect disable GitHub Copilot 1.245.0 alone, GitHub Copilot Chat 0.22.4 works normally on our login node with VS Code 1.95.3.
Logs
Below are logs captured from the Extension Host (remote) process on both the login and computing nodes running VS Code 1.95.3. These logs highlight differences observed during initialization:
Login Node (Fails)
Computing Node (Works)
Additional Notes
Key questions
VS Code version: Code 1.95.3 (f1a4fb1, 2024-11-13T14:50:04.152Z)
OS version: Windows_NT x64 10.0.19045
Modes:
Remote OS version: Linux x64 5.15.0-78-generic
System Info
canvas_oop_rasterization: enabled_on
direct_rendering_display_compositor: disabled_off_ok
gpu_compositing: enabled
multiple_raster_threads: enabled_on
opengl: enabled_on
rasterization: enabled
raw_draw: disabled_off_ok
skia_graphite: disabled_off
video_decode: enabled
video_encode: enabled
vulkan: disabled_off
webgl: enabled
webgl2: enabled
webgpu: enabled
webnn: disabled_off
Extensions (7)
(1 theme extensions excluded)
A/B Experiments
The text was updated successfully, but these errors were encountered: