Breakpoints set in VS Code not hit (docker remote debugging)

[peteat]

I can't seem to hit breakpoints set in VS Code when remotely debugging my go app (which is running in Docker).

I've set up my launch.json:

{
    "version": "0.2.0",
    "configurations": [
        {
            "name": "Remote debug in Docker",
            "type": "go",
            "request": "launch",
            "mode": "remote",
            "program": "${workspaceRoot}",
            "env": {},
            "args": [],
            "remotePath": "${workspaceRoot}",
            "port": 40000, // Port 
            "host": "127.0.0.1", // Docker IP 
            "showLog": true,
            "trace": "verbose"
        }
    ]
}

This is my go.delveConfig (user settings in VS Code):

{
    "useApiV1": false,
    "dlvLoadConfig": {
      "followPointers": true,
      "maxVariableRecurse": 3,
      "maxStringLen": 400,
      "maxArrayValues": 400,
      "maxStructFields": -1
    }
}

It attaches fine, and even hits breakpoints set via runtime.Breakpoint() but will not hit any breakpoints set via VS Code. Debugging the same project in GoLand works fine, I can set and hit breakpoints with no issue.

I have done a lot of googling, and checking documentation, playing with configurations (launch.json, dockerfile, dlv etc) but can't seem to find a fix. Not sure what's wrong, but I have noticed a difference in the VS Code CreateBreakpoint request compared to the GoLand one by looking at the dlv log output (by adding --log --log-output=rpc).

Here's the dlv log (via docker) when debugging via GoLand:

testapp_1  | time="2018-10-15T04:59:46Z" level=debug msg="<- RPCServer.CreateBreakpoint(rpc2.CreateBreakpointIn{\"Breakpoint\":{\"id\":0,\"name\":\"\",\"addr\":0,\"file\":\"C:/Users/<username>/Dev/go/src/testApp/main.go\",\"line\":7,\"Cond\":\"\",\"continue\":false,\"goroutine\":false,\"stacktrace\":0,\"LoadArgs\":null,\"LoadLocals\":null,\"hitCount\":null,\"totalHitCount\":0}})" layer=rpc
testapp_1  | time="2018-10-15T04:59:46Z" level=debug msg="-> *rpc2.CreateBreakpointOut{\"Breakpoint\":{\"id\":1,\"name\":\"\",\"addr\":4833364,\"file\":\"C:/Users/<username>/Dev/go/src/testApp/main.go\",\"line\":7,\"functionName\":\"main.main\",\"Cond\":\"\",\"continue\":false,\"goroutine\":false,\"stacktrace\":0,\"LoadArgs\":null,\"LoadLocals\":null,\"hitCount\":{},\"totalHitCount\":0}} error: \"\"" layer=rpc

But here's the dlv log (via docker) when debugging via VS Code:

testapp_1  | time="2018-10-15T04:52:59Z" level=debug msg="<- RPCServer.CreateBreakpoint(rpc2.CreateBreakpointIn{\"Breakpoint\":{\"id\":0,\"name\":\"\",\"addr\":0,\"file\":\"C:\\\\Users\\\\<username>\\\\Dev\\\\go\\\\src\\\\testApp\\\\main.go\",\"line\":7,\"Cond\":\"\",\"continue\":false,\"goroutine\":false,\"stacktrace\":0,\"LoadArgs\":{\"FollowPointers\":true,\"MaxVariableRecurse\":1,\"MaxStringLen\":64,\"MaxArrayValues\":64,\"MaxStructFields\":-1},\"LoadLocals\":{\"FollowPointers\":true,\"MaxVariableRecurse\":1,\"MaxStringLen\":64,\"MaxArrayValues\":64,\"MaxStructFields\":-1},\"hitCount\":null,\"totalHitCount\":0}})" layer=rpc
testapp_1  | time="2018-10-15T04:52:59Z" level=debug msg="-> *rpc2.CreateBreakpointOut{\"Breakpoint\":{\"id\":0,\"name\":\"\",\"addr\":0,\"file\":\"\",\"line\":0,\"Cond\":\"\",\"continue\":false,\"goroutine\":false,\"stacktrace\":0,\"LoadArgs\":null,\"LoadLocals\":null,\"hitCount\":null,\"totalHitCount\":0}} error: \"could not find C:\\\\Users\\\\<username>\\\\Dev\\\\go\\\\src\\\\testApp\\\\main.go:7\"" layer=rpc

Notice all the extra escapes in the file path for the VS Code call?
They aren't there for the GoLand equivalent call. Are those escapes the problem? Are they preventing dlv from finding the file?

Steps to Reproduce:

1. Create go app (see below)
2. Set up launch.json and delveConfig (as per above)
3. Run go app in docker
   e.g. ./build.sh && docker-compose -f docker-compose.yml up --build
4. Debug the app in VS Code (F5)

main.go:

package main

import "fmt"

func main() {
	v := "test"
	fmt.Println(v)
	fmt.Println("Hello World!")
}

build.sh:

#!/usr/bin/env bash

set -e

echo removing old build artifact
rm testapp || true

echo "building"
GOOS=linux GOARCH=amd64 go build -gcflags "all=-N -l"

echo "build done"

docker-compose.yml:

version: '3.7'

services:
  testapp:
    image: some/testapp:dev
    privileged: true
    build:
      context: .
      dockerfile: ./Dockerfile_dev
    ports:
      - "40000:40000" # delve
      - "8080:8080"
    volumes:
      - ./:/code
      - ${GOPATH}/bin/dlv:/code/dlv
    cap_add:
      - SYS_PTRACE
    security_opt:
      - seccomp:unconfined
      - apparmor:unconfined
    restart: always
    healthcheck:
      test: curl --silent --fail localhost:8080/static/version.txt?healthcheck || exit 1
      interval: 10s
      retries: 5
      timeout: 2s

Dockerfile_dev:

FROM ubuntu
RUN apt-get update && apt install -y tzdata zip ca-certificates curl

# Bring in timezone info; this may not be necessary if go is built inside the container?
WORKDIR /usr/share/zoneinfo
# -0 means no compression.  Needed because go's
# tz loader doesn't handle compressed data.
RUN zip -r -0 /zoneinfo.zip .
ENV ZONEINFO /zoneinfo.zip

WORKDIR /code
#CMD ["./testapp"]
CMD ["./dlv", "--listen=:40000", "--headless=true", "--api-version=2", "--log", "--log-output=rpc", "exec", "./testapp"]

VS Code debug console output:

3:52:59 PM, 10/15/2018
InitializeRequest
InitializeResponse
Using GOPATH: C:\Users\<username>\Dev\go
InitializeEvent
SetBreakPointsRequest
All cleared
Creating on: C:\Users\<username>\Dev\go\src\testApp\main.go (C:\Users\<username>\Dev\go\src\testApp\main.go) :7
Error on CreateBreakpoint: could not find C:\Users\<username>\Dev\go\src\testApp\main.go:7
All set:[null]
SetBreakPointsResponse
ConfigurationDoneRequest
ContinueRequest
ContinueResponse
continue state {"Running":false,"Threads":null,"NextInProgress":false,"exited":true,"exitStatus":0,"When":""}
TerminatedEvent
DisconnectRequest
HaltRequest
DisconnectRequest to parent
DisconnectResponse

Bash console output:

$ ./build.sh && docker-compose -f docker-compose.yml up --build
removing old build artifact
building
build done
The Docker Engine you're using is running in swarm mode.

Compose does not use swarm mode to deploy services to multiple nodes in a swarm. All containers will be scheduled on the current node.

To deploy your application across the swarm, use `docker stack deploy`.

Building testapp
Step 1/7 : FROM ubuntu
 ---> cd6d8154f1e1
Step 2/7 : RUN apt-get update && apt install -y tzdata zip ca-certificates curl
 ---> Using cache
 ---> ad36719bc090
Step 3/7 : WORKDIR /usr/share/zoneinfo
 ---> Using cache
 ---> b603d79ea208
Step 4/7 : RUN zip -r -0 /zoneinfo.zip .
 ---> Using cache
 ---> a4af1a1ad54a
Step 5/7 : ENV ZONEINFO /zoneinfo.zip
 ---> Using cache
 ---> fe54a7095b71
Step 6/7 : WORKDIR /code
 ---> Using cache
 ---> 4491a3030a0e
Step 7/7 : CMD ["./dlv", "--listen=:40000", "--headless=true", "--api-version=2", "--log", "--log-output=rpc", "exec", "./testapp"]
 ---> Using cache
 ---> 5199945d66fa

Successfully built 5199945d66fa
Successfully tagged some/testapp:dev
Starting testapp_testapp_1 ... done
Attaching to testapp_testapp_1
testapp_1  | API server listening at: [::]:40000
testapp_1  | time="2018-10-15T05:04:58Z" level=debug msg="<- RPCServer.GetVersion(api.GetVersionIn{})" layer=rpc
testapp_1  | time="2018-10-15T05:04:58Z" level=debug msg="-> *api.GetVersionOut{\"DelveVersion\":\"Version: 1.1.0\\nBuild: $Id: 1990ba12450cab9425a2ae62e6ab988725023d5c $\",\"APIVersion\":2} error: \"\"" layer=rpc
testapp_1  | time="2018-10-15T05:04:58Z" level=debug msg="<- RPCServer.CreateBreakpoint(rpc2.CreateBreakpointIn{\"Breakpoint\":{\"id\":0,\"name\":\"\",\"addr\":0,\"file\":\"C:\\\\Users\\\\<username>\\\\Dev\\\\go\\\\src\\\\testApp\\\\main.go\",\"line\":7,\"Cond\":\"\",\"continue\":false,\"goroutine\":false,\"stacktrace\":0,\"LoadArgs\":{\"FollowPointers\":true,\"MaxVariableRecurse\":1,\"MaxStringLen\":64,\"MaxArrayValues\":64,\"MaxStructFields\":-1},\"LoadLocals\":{\"FollowPointers\":true,\"MaxVariableRecurse\":1,\"MaxStringLen\":64,\"MaxArrayValues\":64,\"MaxStructFields\":-1},\"hitCount\":null,\"totalHitCount\":0}})" layer=rpc
testapp_1  | time="2018-10-15T05:04:58Z" level=debug msg="-> *rpc2.CreateBreakpointOut{\"Breakpoint\":{\"id\":0,\"name\":\"\",\"addr\":0,\"file\":\"\",\"line\":0,\"Cond\":\"\",\"continue\":false,\"goroutine\":false,\"stacktrace\":0,\"LoadArgs\":null,\"LoadLocals\":null,\"hitCount\":null,\"totalHitCount\":0}} error: \"could not find C:\\\\Users\\\\<username>\\\\Dev\\\\go\\\\src\\\\testApp\\\\main.go:7\"" layer=rpc
testapp_1  | time="2018-10-15T05:04:58Z" level=debug msg="(async 3) <- RPCServer.Command(api.DebuggerCommand{\"name\":\"continue\",\"ReturnInfoLoadConfig\":null})" layer=rpc
testapp_1  | test
testapp_1  | Hello World!
testapp_1  | time="2018-10-15T05:04:58Z" level=debug msg="(async 3) -> rpc2.CommandOut{\"State\":{\"Running\":false,\"Threads\":null,\"NextInProgress\":false,\"exited\":true,\"exitStatus\":0,\"When\":\"\"}} error: \"\"" layer=rpc
testapp_1  | time="2018-10-15T05:04:58Z" level=debug msg="(async 4) <- RPCServer.Command(api.DebuggerCommand{\"name\":\"halt\",\"ReturnInfoLoadConfig\":null})" layer=rpc
testapp_1  | time="2018-10-15T05:04:58Z" level=debug msg="(async 4) -> <nil>null error: \"Process 12 has exited with status 0\"" layer=rpc
Stopping testapp_testapp_1 ... done
Gracefully stopping... (press Ctrl+C again to force)

Note: all output and config has been slightly santised (with <username>)

System info:
VS Code 1.28.1 (user setup)
Windows 10 Enterprise 1803
Docker 18.06.1-ce-win73

Edit: added system info

[jinmatt]

👍

The same thing is happening when debugging without docker, the debugger does not seem to hit API handlers or endpoints.

Everything in the call stack looks like PAUSED. Sceenshot: https://cl.ly/e31600b38c0f

trace:

continue state {"Running":false,"currentThread":{"id":1199227,"pc":31866287,"file":"/Users/jinmatt/go/src/lib/api/http/health_handler.go","line":25,"function":{"name":"lib/api/http.healthHandler.read","value":31866272,"type":0,"goType":0,"optimized":false},"goroutineID":53,"breakPoint":{"id":1,"name":"","addr":31866287,"file":"/Users/jinmatt/go/src/lib/api/http/health_handler.go","line":25,"functionName":"lib/api/http.healthHandler.read","Cond":"","continue":false,"goroutine":false,"stacktrace":0,"LoadArgs":{"FollowPointers":true,"MaxVariableRecurse":1,"MaxStringLen":64,"MaxArrayValues":64,"MaxStructFields":-1},"LoadLocals":{"FollowPointers":true,"MaxVariableRecurse":1,"MaxStringLen":64,"MaxArrayValues":64,"MaxStructFields":-1},"hitCount":{"53":1},"totalHitCount":1},"breakPointInfo":{"arguments":[{"name":"h","addr":824727868248,"onlyAddr":false,"type":"lib/api/http.healthHandler","realType":"lib/api/http.healthHandler","flags":8,"kind":25,"value":"","len":0,"cap":0,"children":[],"base":0,"unreadable":"","LocationExpr":"[block] DW_OP_call_frame_cfa ","DeclLine":25},{"name":"r","addr":824727868264,"onlyAddr":false,"type":"*net/http.Request","realType":"*net/http.Request","flags":8,"kind":22,"value":"","len":1,"cap":0,"children":[{"name":"","addr":824636548864,"onlyAddr":false,"type":"net/http.Request","realType":"net/http.Request","flags":0,"kind":25,"value":"","len":22,"cap":0,"children":[{"name":"Method","addr":824636548864,"onlyAddr":false,"type":"string","realType":"string"[...]
StoppedEvent("breakpoint")
ThreadsRequest
goroutines [{"id":1,"currentLoc":{"pc":16987492,"file":"/usr/local/go/src/runtime/proc.go","line":303,"function":{"name":"runtime.gopark","value":16987280,"type":0,"goType":0,"optimized":true}},"userCurrentLoc":{"pc":32184162,"file":"/Users/jinmatt/go/src/lib/api/http/server.go","line":178,"function":{"name":"lib/api/http.(*server).ListenAndServe","value":32177184,"type":0,"goType":0,"optimized":false}},"goStatementLoc":{"pc":17165051,"file":"/usr/local/go/src/runtime/asm_amd64.s","line":201,"function":{"name":"runtime.rt0_go","value":17164752,"type":0,"goType":0,"optimized":true}},"startLoc":{"pc":16986048,"file":"/usr/local/go/src/runtime/proc.go","line":110,"function":{"name":"runtime.main","value":16986048,"type":0,"goType":0,"optimized":true}},"threadID":0,"unreadable":""},{"id":2,"currentLoc":{"pc":16987492,"file":"/usr/local/go/src/runtime/proc.go","line":303,"function":{"name":"runtime.gopark","value":16987280,"type":0,"goType":0,"optimized":true}},"userCurrentLoc":{"pc":16987492,"file":"/usr/local/go/src/runtime/proc.go","line":303,"function":{"name":"runtime.gopark","value":16987280,"type":0,"goType":0,"optimized":true}},"goStatementLoc":{"pc":16986917,"file":"/usr/local/go/src/runtime/proc.go","line":240,"function":{"name":"runtime.init.4","value":16986864,"type":0,"goType":0,"optimized":true}},"startLoc":{"pc":16986944,"file":"/usr/local/go/src/runtime/proc.go","line":243,"function":{"name":"runtime.forcegchelper","value":16986944,"type":0,"goType":0,"optimized":t[...]
ThreadsResponse [{"id":1,"name":"lib/api/http.(*server).ListenAndServe"},{"id":2,"name":"runtime.gopark"},{"id":18,"name":"runtime.gopark"},{"id":19,"name":"runtime.gopark"},{"id":20,"name":"os/signal.signal_recv"},{"id":21,"name":"lib/vendor/googlemaps.github.io/maps.NewClient.func1"},{"id":3,"name":"runtime.asmcgocall"},{"id":22,"name":"lib/vendor/gopkg.in/redis.v5/internal/pool.(*ConnPool).reaper"},{"id":23,"name":"runtime.gopark"},{"id":27,"name":"lib/vendor/gopkg.in/redis.v5/internal/pool.(*ConnPool).reaper"},{"id":10,"name":"database/sql.(*DB).connectionOpener"},{"id":34,"name":"runtime.asmcgocall"},{"id":33,"name":"runtime.gopark"},{"id":53,"name":"lib/api/http.healthHandler.read"},{"id":32,"name":"runtime.gopark"},{"id":36,"name":"runtime.gopark"},{"id":8,"name":"runtime.gopark"},{"id":50,"name":"runtime.gopark"},{"id":9,"name":"runtime.gopark"},{"id":51,"name":"runtime.gopark"},{"id":52,"name":"runtime.gopark"},{"id":38,"name":"internal/poll.runtime_pollWait"},{"id":66,"name":"io.(*pipe).Read"},{"id":67,"name":"database/sql.(*DB).connectionOpener"},{"id":68,"name":"database/sql.(*DB).connectionResetter"},{"id":69,"name":"lib/vendor/gopkg.in/redis.v5/internal/pool.(*ConnPool).reaper"},{"id":70,"name":"runtime.gopark"},{"id":82,"name":"runtime.gopark"},{"id":98,"name":"runtime.asmcgocall"},{"id":11,"name":"database/sql.(*DB).connectionResetter"},{"id":114,"name":"internal/poll.runtime_pollWait"},{"id":16,"name":"internal/poll.runtime_pollWait"},{"id":115,"name":"runtim[...]
StackTraceRequest
ThreadsRequest
locations [{"pc":31866287,"file":"/Users/jinmatt/go/src/lib/api/http/health_handler.go","line":25,"function":{"name":"lib/api/http.healthHandler.read","value":31866272,"type":0,"goType":0,"optimized":false},"Locals":[],"Arguments":[{"name":"h","addr":824727868248,"onlyAddr":false,"type":"lib/api/http.healthHandler","realType":"lib/api/http.healthHandler","flags":8,"kind":25,"value":"","len":0,"cap":0,"children":[],"base":0,"unreadable":"","LocationExpr":"[block] DW_OP_call_frame_cfa ","DeclLine":25},{"name":"r","addr":824727868264,"onlyAddr":false,"type":"*net/http.Request","realType":"*net/http.Request","flags":8,"kind":22,"value":"","len":1,"cap":0,"children":[{"name":"","addr":824636548864,"onlyAddr":false,"type":"net/http.Request","realType":"net/http.Request","flags":0,"kind":25,"value":"","len":22,"cap":0,"children":[{"name":"Method","addr":824636548864,"onlyAddr":false,"type":"string","realType":"string","flags":0,"kind":24,"value":"GET","len":3,"cap":0,"children":[],"base":824727764992,"unreadable":"","LocationExpr":"","DeclLine":0},{"name":"URL","addr":824636548880,"onlyAddr":false,"type":"*net/url.URL","realType":"*net/url.URL","flags":0,"kind":22,"value":"","len":1,"cap":0,"children":[{"name":"","addr":824727684224,"onlyAddr":false,"type":"net/url.URL","realType":"net/url.URL","flags":0,"kind":25,"value":"","len":9,"cap":0,"children":[{"name":"Scheme","addr":824727684224,"onlyAddr":false,"type":"string","realType":"string","flags":0,"kind":24,"value":"","len":0,"ca[...]
StackTraceResponse
goroutines [{"id":1,"currentLoc":{"pc":16987492,"file":"/usr/local/go/src/runtime/proc.go","line":303,"function":{"name":"runtime.gopark","value":16987280,"type":0,"goType":0,"optimized":true}},"userCurrentLoc":{"pc":32184162,"file":"/Users/jinmatt/go/src/lib/api/http/server.go","line":178,"function":{"name":"lib/api/http.(*server).ListenAndServe","value":32177184,"type":0,"goType":0,"optimized":false}},"goStatementLoc":{"pc":17165051,"file":"/usr/local/go/src/runtime/asm_amd64.s","line":201,"function":{"name":"runtime.rt0_go","value":17164752,"type":0,"goType":0,"optimized":true}},"startLoc":{"pc":16986048,"file":"/usr/local/go/src/runtime/proc.go","line":110,"function":{"name":"runtime.main","value":16986048,"type":0,"goType":0,"optimized":true}},"threadID":0,"unreadable":""},{"id":2,"currentLoc":{"pc":16987492,"file":"/usr/local/go/src/runtime/proc.go","line":303,"function":{"name":"runtime.gopark","value":16987280,"type":0,"goType":0,"optimized":true}},"userCurrentLoc":{"pc":16987492,"file":"/usr/local/go/src/runtime/proc.go","line":303,"function":{"name":"runtime.gopark","value":16987280,"type":0,"goType":0,"optimized":true}},"goStatementLoc":{"pc":16986917,"file":"/usr/local/go/src/runtime/proc.go","line":240,"function":{"name":"runtime.init.4","value":16986864,"type":0,"goType":0,"optimized":true}},"startLoc":{"pc":16986944,"file":"/usr/local/go/src/runtime/proc.go","line":243,"function":{"name":"runtime.forcegchelper","value":16986944,"type":0,"goType":0,"optimized":t[...]
ThreadsResponse [{"id":1,"name":"lib/api/http.(*server).ListenAndServe"},{"id":2,"name":"runtime.gopark"},{"id":18,"name":"runtime.gopark"},{"id":19,"name":"runtime.gopark"},{"id":20,"name":"os/signal.signal_recv"},{"id":21,"name":"lib/vendor/googlemaps.github.io/maps.NewClient.func1"},{"id":3,"name":"runtime.asmcgocall"},{"id":22,"name":"lib/vendor/gopkg.in/redis.v5/internal/pool.(*ConnPool).reaper"},{"id":23,"name":"runtime.gopark"},{"id":27,"name":"lib/vendor/gopkg.in/redis.v5/internal/pool.(*ConnPool).reaper"},{"id":10,"name":"database/sql.(*DB).connectionOpener"},{"id":34,"name":"runtime.asmcgocall"},{"id":33,"name":"runtime.gopark"},{"id":53,"name":"lib/api/http.healthHandler.read"},{"id":32,"name":"runtime.gopark"},{"id":36,"name":"runtime.gopark"},{"id":8,"name":"runtime.gopark"},{"id":50,"name":"runtime.gopark"},{"id":9,"name":"runtime.gopark"},{"id":51,"name":"runtime.gopark"},{"id":52,"name":"runtime.gopark"},{"id":38,"name":"internal/poll.runtime_pollWait"},{"id":66,"name":"io.(*pipe).Read"},{"id":67,"name":"database/sql.(*DB).connectionOpener"},{"id":68,"name":"database/sql.(*DB).connectionResetter"},{"id":69,"name":"lib/vendor/gopkg.in/redis.v5/internal/pool.(*ConnPool).reaper"},{"id":70,"name":"runtime.gopark"},{"id":82,"name":"runtime.gopark"},{"id":98,"name":"runtime.asmcgocall"},{"id":11,"name":"database/sql.(*DB).connectionResetter"},{"id":114,"name":"internal/poll.runtime_pollWait"},{"id":16,"name":"internal/poll.runtime_pollWait"},{"id":115,"name":"runtim[...]

[jpopadak]

This ticket is related to the already open ticket here: #1987

[peteat]

Perhaps, but in my issue the execution never pauses (if you run the above go code the program runs and finishes execution with no pause whatsoever). You can't interrogate any call stack nor does VS Code visually pause. #1987 looks like it manages to successfully set the breakpoint but then has issues once you hit the breakpoint, I can't even get it to set any breakpoints to begin with. Also, the log outputs seem quite different (I'd guess that's because the breakpoints get set).
I'm only a go/vs code beginner (normally I'm a VS & .NET programmer) but I'd suggest it looks like a separate issue.

[ramya-rao-a]

@peteat, @jinmatt, @jpopadak Can you try the latest Insiders?

[jpopadak]

@ramya-rao-a This half works in the Insiders Edition. When I am debugging a test, I am at least getting an error message in the Call Stack tool window. While debugging, it hits the breakpoint, under the thread it shows "Unable to produce stack trace: "Internal debugger error: runtime error: slice bounds out of range runtime.call32(0x1057ada) ..."

When I debug a running program, it hits the breakpoints correctly and shows the variables correctly. However, it jumps to the proc.go file in the gopark method a ton (almost every single time I hit Step Over or Step Into). The debugger seems to correct itself (about 70% of the time) and jumps back to the source code itself that I was stepping through. Let me know if you need logs from either one.

[peteat]

Hi @ramya-rao-a apologies for the delay. I've just tested it on the latest insiders and, I'm afraid, it's no different for me. I still get the Error on CreateBreakpoint: could not find... error on debug start.

[ramya-rao-a]

@peteat, @jinmatt, @jpopadak Can you try the latest beta-version of this extension?

[peteat]

Hi @ramya-rao-a unfortunately it's the same for me with that latest beta (on VS Code and latest insiders) - Error on CreateBreakpoint: could not find...

[joe15000]

Hi, all and @ramya-rao-a,

I just ran into this issue myself, developing on Windows but running the app/delve in a docker container.

Windows 10 version: 10.0.17134
VSCode version: 1.34.0-insider
Go: 1.12
Delve (on Windows Host): 1.2.0

Debugging with Goland works just fine, but when I try with vscode, the debug log in vscode shows me the below, the breakpoints become "unverified" and are never hit. Manually debugging with delve works. Goland works. I also suspect it may be a file path issue between Windows and Unix path conventions.

I have posted a StackOverflow question on this (with a lot more info) before I found this bug report:
https://stackoverflow.com/questions/56037313/breakpoints-in-vscode-win-10-unverified-and-not-hit-when-remote-debugging-go

I have also created a clean, simple GitHub repo to demonstrate the issue (this causes the exact same behaviour for me):
https://github.com/joe15000/vscode_go_debug

1. docker-compose up -d --build
2. Launch debug in vscode with provided launch.json config
3. See the following in the debug console:

SetBreakPointsRequest
All cleared
Creating on: C:\Users\joe\go\src\github.com\joe15000\vscode_go_debug\package1\package1.go (C:/Users/joe/go/src/github.com/joe15000/vscode_go_debug/package1/package1.go) :17
Error on CreateBreakpoint: could not find C:/Users/joe/go/src/github.com/joe15000/vscode_go_debug/package1/package1.go:17
All set:[null]
To client: {"seq":0,"type":"response","request_seq":3,"command":"setBreakpoints","success":true,"body":{"breakpoints":[{"verified":false,"line":17}]}}
SetBreakPointsResponse
From client: setExceptionBreakpoints({"filters":[]})
To client: {"seq":0,"type":"response","request_seq":4,"command":"setExceptionBreakpoints","success":true}
From client: configurationDone(undefined)
ConfigurationDoneRequest
ContinueRequest
To client: {"seq":0,"type":"response","request_seq":5,"command":"configurationDone","success":true}
ContinueResponse
From client: threads(undefined)
To client: {"seq":0,"type":"response","request_seq":6,"command":"threads","success":true,"body":{"threads":[{"id":1,"name":"Dummy"}]}}

Btw, vscode allows me to CTRL-click on the "C:/Users/joe/go/src/github.com/joe15000/vscode_go_debug/package1/package1.go:17" shown in the "Error ... could not find" line - and it opens the file in vscode.

Also leaving this here - not sure if it helps but they look more closely into how the path is passed from vscode to delve:
bazel-contrib/rules_go#1844

As the last answer was from over half a year ago - is this still being investigated / addressed? I really love vscode but if docker debugging is not possible, I unfortunately cannot use it.

[ghost]

I can confirm this happening for me as well - I can debug AWS lambdas via sam local in Goland but not VSCode
Here is my setup

• Windows 10.0.27234r0 64bit
• VsCode: 1.36.1
• VsCode-Go: 0.11.4
• aws-cli 1.16.202
• aws sam cli 0.18.0
• Docker For Windows: 18.09.2

Setup commands /batch script

if exist sam-app RMDIR /S /Q sam-app
docker stop $(docker ps -aq)
docker rm $(docker ps -aq)

sam init  --runtime go1.x

cd sam-app
go get -u github.com/go-delve/delve/cmd/dlv
go get -u github.com/aws/aws-lambda-go/...
setx GOARCH amd64
setx GOOS linux
set GOARCH=amd64
set GOOS=linux
mkdir debugger
go env
go build  -gcflags "all=-N -l" -o ./debugger/dlv github.com/go-delve/delve/cmd/dlv
go build  -gcflags "all=-N -l" -o hello-world/hello-world ./hello-world
mkdir .vscode
copy ../launch.json .vscode/launch.json
start code .
sam local start-api -d 5987 --debugger-path ./debugger --debug-args="-delveAPI=2"

My launch.json file in the same folder as the batch script is

{
    "version": "0.2.0",
    "configurations": [

    {
        "name": "Connect to Lambda container",
        "type": "go",
        "request": "attach",
        "mode": "remote",
        "remotePath": "/var/task/hello-world",
        "port": 5987,
        "host": "127.0.0.1",
        "trace" : "verbose"
      },
    ]
}

[ghost]

I also tried with API version 1 and it still occured - attached is the debug logs
vscode-go-debug.txt

[KarthickEmis]

Even I too confirm , this happening for me as well - I can debug AWS lambdas via sam local in Goland but not VSCode

I tried with new simple go lang program
sam init --runtime go1.x
cd sam-app
code . # to launch VSCode
go get -u github.com/go-delve/delve/cmd/dlv
set GOARCH=amd64
set GOOS=linux
go build -o ./dlv github.com/go-delve/delve/cmd/dlv
go get -u github.com/aws/aws-lambda-go/...
set GOARCH=amd64
set GOOS=linux
go build -gcflags="-N -l" -o hello-world/main ./hello-world
sam local start-api -d 5989 --debugger-path . --debug-args="-delveAPI=2"
debugger is not attaching in VS code .. but code is working( value is returning in browser)
When i try to run the below command ,

sam local start-api -d 5989 --debugger-path . --debug-args="-delveAPI=2" , i can see upto the below lines in terminal
2019-08-28 10:42:16 Mounting C:\go-work\src\sam-app1\sam-app\hello-world as /var/task:ro,delegated inside runtime container
Could not create config directory: mkdir .config: read-only file system.API server listening at: [::]:5989
2019-08-28T05:12:17Z info layer=debugger launching process with args: [/var/task/main],
It get strucked after this .once I restart the debugger in VS code , I am getting value in browser and getting logs in terminal but not hitting break point.
2019-08-28T05:12:57Z debug layer=debugger continuing
?[32mSTART RequestId: 959e2982-4823-1ac4-38a2-db2fb097b6d0 Version: $LATEST?[0m
?[32mEND RequestId: 959e2982-4823-1ac4-38a2-db2fb097b6d0?[0m
?[32mREPORT RequestId: 959e2982-4823-1ac4-38a2-db2fb097b6d0 Duration: 1178.29 ms Billed Duration: 1200 ms Memory Size: 128 MB Max Memory Used: 30 MB ?[0m
2019-08-28 10:42:59 No Content-Type given. Defaulting to 'application/json'.
2019-08-28 10:42:59 127.0.0.1 - - [28/Aug/2019 10:42:59] "GET /hello HTTP/1.1" 200 -
2019-08-28 10:42:59 127.0.0.1 - - [28/Aug/2019 10:42:59] "GET /favicon.ico HTTP/1.1" 403
Note:
I tried -delveAPI=1 also , but not working
versions in my system :
VsCode: 1.37.1
aws-cli - 1.16.26
aws sam cli 0.19.0
Docker For Windows: 19.03.1

My launch.json
{
// Use IntelliSense to learn about possible attributes.
// Hover to view descriptions of existing attributes.
// For more information, visit: https://go.microsoft.com/fwlink/?linkid=830387
"version": "0.2.0",
"configurations": [
{
"name": "Connect to Lambda container",
"type": "go",
"request": "attach",
"mode": "remote",
"remotePath": "/var/task/main",
"port": 5989,
"host": "127.0.0.1"
// "program": "${workspaceRoot}",
// "env": {},
// "args": []
}
]
}

[ghost]

I've been investigating this further including why VSCode does not debug and Goland down and it seems to be down to the fact that VSCode sends delve JSON-RPC commands specifying the location of the file and the line to breakpoint, and Goland sends the memory address to breakpoint at. For the former,when debugging remotely, Delve requires access to a copy of the source code so it knows where to set the breakpoint.

However in AWS SAM, the Delve debugger, and the binary to be debuggged is run inside a temporary docker container, so Delve cannot locate the source file, fails to set the breakpoint, and the program completes.

I tried to fix this by waiting until AWS SAM / delve halted for a debugger to attach, and then copying the source files into the docker instance via the AWS SAM file share, and using vscodes remotePath: to pick it up, but unfortunately Delve did not pick up the files.

This might be solved by having AWS SAM mount the source code, and pointing Delve towards these files so vscode can work.

I have also commented on issu for AWS SAM
aws/aws-sam-cli#1310

However this might be more easily resolved on the AWS SAM end by mounting source code

[ghost]

@KarthickEmis After further investigation into the Delve debugger , I've discovered the following:

1. The compiler go build encodes filepaths and lines in the following format:
   1.a. Windows Subsystem for linux with GOOS=linux : /c/git/linux/exampleFile/main.go
   1.b. Windows with GOOS=linux : C:/Git/exampleFile/main.go
2. The binary is then zipped up ready for deployment
3. AWS SAM is then started via a command on the lines of sam local start-api -t api-gateway.yml -d 5987 --debugger-path ./api/bin --debug-args="-debug -delveAPI=2"
4. A call is made to SAM local which then mounts both the executable and the delve debugger
5. The delve debugger is started and then immediately halts due it's headless commands waiting for a remote debugger to attached, it listens on port 5987
6. VSCode opened in the project directory, and it's launch.json contains something on the lines of

{
	"type": "go",
	"name": "Connect to Lambda container",
	"request": "attach",
	"mode": "remote",
	"remotePath": "/c/git/linux/exampleFile"
	"port": 5987,
	"host": "127.0.0.1",
	"showLog": true,
	"trace": "log"
 }

7. A breakpoint is added in exampleFile/main.go
8. VSCode is attached to the remote debugger
9. VSCode examines the breakpointed files and joins them to the remotePath in the launch.json e.g. '/c/git/linux/exampleFile' + '/main.go'
10. This is then passed to delve which compares the full file path against the one compiled into the executable - if it exists then the breakpoint is set there
11. Debugging is continued until it hits the breakpoint

Breakpoints fail when the compiled path is different then the composite path of remotePath, and the relative path to the file for the VSCode - it is not needed for source code on the remote machine, as this line info is built into the binary.

Where my code was failing was because I built in WSL linux, but my vscode remote path did not have a linux style path to my code directory.

Unfortunately, if I choose the option of compiling in Windows with GOOS=linux, then the paths compiled into the binary look like:

• C:/Git/exampleFile/main.go
  But the path sent by vscode looks like
• C:\Git\exampleFile\main.go

The nature of the slashes are different, and so the breakpoints are not hit. This would require either a change to Go build to change the compiled files on a window system with GOOS=linux, or more easily, to change VSCode so it send paths inP OSIX format when remote connecting to a linux machine. A third option is for delve to rewrite paths passed in so they are in POSIX format

[jackmcguire1]

Great detective work @jdwalker-emis!

I would like to inspect what file path osx produces using linux build, as I suspect it’ll probably be similar to the ones above,

I would say for now it’s an ELI5ing in AWS docs on how to structure remote path for vscode as this is configurable

However I guess raising the issue with delve or go build, Is the way to go as the build process isn’t explicitly handled by vs code

[ramya-rao-a]

cc @quoctruong and @jhendrixMSFT for any additional insights here

[quoctruong]

I think this seems similar to this issue: #2931. The work around was to build with go build -gcflags=-trimpath=c: -asmflags=-trimpath=c: and to replace the remotePath in the launch setting.

[ghost]

@quoctruong I tried the workaround, and checked the debug information - unfortunately setting the trimpath only removed it for the immediate packages and not any of the referenced packages so I would be unable to set breakpoints for those packages. In addition Vscode go is pre fixing my remote path with a forward slash e.g. "/Git/code" vs compiled "Git/code" so this approach is not working.

[BryanHunt]

I just ran into this same problem with Typescript in a remote container. I can set breakpoints, but they are never hit.

[quoctruong]

@BryanHunt For typescript, I think you have to specify remotePath and localPath in the package.json.

[snavarro89]

Im having the same issue with OS X, have tried all solutions proposed but none have worked to stop the breakpoint, is this still happening for you?

[ramya-rao-a]

The fix from #3108 is available in the latest beta version of this extension. Please do try and share feedback

[anthonybgale]

I can confirm that I am able to hit breakpoints now using AWS SAM to launch a linux container with delve and go binaries compiled from Windows.

For anyone still having this problem (like I was before I edited this comment), take care that the "remotePath" element of your launch.json is the absolute path to the source files as compiled on your local system (not the container). As implied above - it's the absolute local path that is added to the DWARF compilation unit file table when you compile the binary.

[ramya-rao-a]

Thanks @anthonybgale

The latest version of the extension now has the fix to this issue