[Jupyter Integration w/ Docker] Debugger breaks at the wrong place

[DonJayamanne]

Environment data

• debugpy version: 1.3.0 (run import debugpy; print(debugpy.__version__) if uncertain)
• OS and version: Linux 5.4.0
• Python version (& distribution if applicable, e.g. Anaconda): 3.9
• Using VS Code or Visual Studio: VS Code Code Spaces

Actual behavior

Debugger stops at interactiveShell.py file at the lie sys.execthook = old_excepthook
This seems to be reproducible only in containers or codespaces.

Expected behavior

Debugger should stop in the code that the user same. In my example i ran sample.py. Hence it should break either in sample.py or a temporary file with the same code.

Please note: In desktop, the issue with source mappings will be resolved in the latest Jupyter extension, see here microsoft/vscode-jupyter#6534

Steps to reproduce:

1. Ensure you have docker up n running
2. Clone the repo https://github.com/DonJayamanne/testDebugPyInContainer
3. Open in VS Code Containers
   3. Open folder in VS Code
   4. When prompted to open in Container click yes
4. Install the Python & Jupyter Extension from marketplace
5. Ensure you select Python 3.9 as your workspace interpreter in python (bottom left statusbar)
6. Pip install ipykernel into the Python 3.9 environment
   8 . Create a file named sample.py and the the following code

# %% 
print('Hello World')

9. Code lenses will appear Run Cell | Run Below | Debug Cell
10. Click Run cell and confirm it gets executed
11. Close the interactive window that opened on the right side (ensure you have just the Python file opened)
12. Now click Debug Cell
13. Notice how the debugger starts and stops at interactiveShell.py file at the lie sys.execthook = old_excepthook

Note:

• Latest ipykernel 6.0 ships with debugpy. You can optionally chose to downgrade to the older version.
• pip uninstall ipykernel debugpy ipython
• python -m pip install ipython==7.15.0 ipykernel==5.5.5

However even with this (old version), the issue repros.

[fabioz]

@DonJayamanne can you provide the logs related to that?

To obtain the logs it's possible to set the following environment variables (they must be set prior to importing the debugger):

PYDEVD_DEBUG_FILE=</path_to/pydevd.log>
PYDEVD_DEBUG=1

[DonJayamanne]

Oh sorry, didn't do that. Karthik said that providing the instructions was sufficient.
Will get the logs.

[fabioz]

Oh sorry, didn't do that. Karthik said that providing the instructions was sufficient.
Will get the logs.

Given that this repros only under a pretty specific scenario, it'd be nice to have those to know that I'm looking at the same thing (I still haven't tried to reproduce it, I'll do that tomorrow).

[DonJayamanne]

Contents are too large, hence uploading files:

• First log up until the debugger breaks in interactiveshell.py
  Step1.log
• This second log is everything after i hit the continue button (& program runs to completion)
  Step2.log

[fabioz]

Hi Don, it seems that these log files are from debugpy (which only provides logging on the messages exchanged), whereas I'm really interested in the log files from pydevd (which provides some internal info on the actual tracing).

Usually when you set the debugging it also creates .pydevd files (but there are some situations where it doesn't do this -- if that's the case and it didn't generate those files, setting the environment variables as I outlined in: #664 (comment) should provide those files).

[DonJayamanne]

environment variables as I outlined in: #664 (comment) should provide those files).

Is it not possible for you to start this in a docker container at this stage (to replicate this issue).
I tried setting environment variables and it didn't work for me, & karthik helped me modify the underlying source code to enable the logging. I'm going to have to do that again here.

[fabioz]

Ok, I've just been able to reproduce it here (I had to install everything from scratch, including wsl/docker on Windows, and it reproduced properly, so, as it's more work than I thought on your side, let me try to diagnose directly here).

[fabioz]

Log obtained:

pydevd.log

An interesting note from the logs: although justMyCode is true in the launch, the library roots are: ['/home/vscode/.local/lib/python3.9/site-packages', '/usr/local/lib/python3.9', '/usr/local/lib/python3.9/site-packages'], whereas the interactiveshell.py is installed in /usr/local/pip-global/IPython/core/interactiveshell.py (and thus is considered user code).

[fabioz]

I'm finishing for the day here.

From what I can see, the debugger integration sends the contents prefixed by breakpoint(), so, something as:

print('Hello World')

becomes:

breakpoint()\nprint('Hello World')

Apparently the kernel is executing line-by-line and not as a single statement (it seems it has something as an run_ast_nodes) so, it'll evaluate the breakpoint(), stop at the first place considered user code (in this case /usr/local/pip-global/IPython/core/interactiveshell.py) and then after the user resumes it will execute print('Hello World').

I still haven't decided on the best approach to fix this.

One approach could be considering anything in /usr/local/pip-global to be library code (the whole difference in this case with the docker approach is that libraries are being installed at that folder, so, it'll stop at that place -- I've tested this and it works for me with this change).

Still it's possible that there's a better approach to the problem as a whole, so, I'm going to sleep on it to think a bit more before providing a pull request...

[fabioz]

I created a pull request to consider everything in pip-global to be library code as this fixes the current issue. I'm still not sure this is the ideal solution... The main reason for that is that relying on stopping or not based on whether it's in user-code or may not be the ideal approach, but other approaches I thought of would need to have a fix on the client.

For instance, the client could request for interactiveshell.py to be explicitly ignored in this case (that'd probably also enable users to run with justMyCode:false as right now this isn't possible).

Also, I noted that the source mapping is not really matching what ipython has (so, breakpoints in the sample.py are never hit and the contents of the file are actually gotten from the server instead of using the contents of the client) -- this is actually out of the scope of this issue, but I think it's worth fixing that on the client side.

i.e.: the paths that are being generated in the kernel are something as: /tmp/ipykernel_15594/4207307487.py, whereas the source mapping is mapping something as "<ipython-input-1-6dcc7365aa9a>" to "/workspaces/testDebugPyInContainer/sample.py" (I suspect this is due to changes in how the kernel generates its cell names or maybe just a different mode is being used instead of what was previously expected in the client -- note: right now this is handled by the client and isn't fixable in the debugger either).

So, in the end the fix I could think of in the debugger side is available in the pull request, but the client could be improved so that this doesn't rely as much on the library filtering...

[DonJayamanne]

ct this is due to changes in how the kernel generates its cell names or maybe just a different mode is being used instead of what was previously expected in the client -- note: right now this is handled by the client and isn't fixable in the debugger either).

Yes, thats correct. we've fixed this in the latest insider build of the Jupyter extension. If you install VS Code Insiders, you'll get the fix in the latest Jupyter extension (which targets vscode insiders).
Here's the fix https://github.com/microsoft/vscode-jupyter/pull/6691/files#diff-f1216ee90d35222d31d042066ee9a29193385fd273457e4ff166ddaf0002d397R381