Connecting PyCharm’s DAP Debugger to an App Running in Kubernetes

When your app runs inside a Kubernetes pod instead of on your laptop, print() statements and log tailing stop being enough pretty quickly. What you really want is the same experience you get locally — breakpoints, variable inspection, step-through execution — but reaching into the cluster. That’s exactly what the Debug Adapter Protocol (DAP) gives you: PyCharm can attach to a remote Python process via debugpy, regardless of where that process physically runs.

Below is a walkthrough using the sample repository k8s-dap-remote-debugger: a small FastAPI app plus two Celery workers, packaged in a Helm chart for a local cluster, with remote debugging wired into all three.

How it works

The core trick is the same everywhere in this repo: instead of launching the “real” process directly (uvicorn, celery worker), the container wraps it in debugpy, which listens on its own port and waits for a debugger to attach.

For the API container, the Deployment template builds this command conditionally on debug.enabled:

command:
  - python
  - -m
  - debugpy
  - --listen
  - 0.0.0.0:
  - --wait-for-client
  - -m
  - uvicorn
  - main:app
  ...

Each Celery worker gets the exact same treatment, just wrapping celery instead of uvicorn, with its own port and its own waitForClient toggle:

- name: 
  command:
    - python
    - -m
    - debugpy
    - --listen
    - 0.0.0.0:
    - --wait-for-client
    - -m
    - celery
    - celery
    - -A
    - celery_app:celery_app
    - worker
    - --loglevel=info
    - --queues=
    - --hostname=@%h
    - --concurrency=

The whole thing runs as one pod with four containers — api, worker-default, worker-math, and a redis sidecar acting as the Celery broker/result backend — all behind a single Service. That’s why one kubectl port-forward call at the end can reach every debug port at once.

The application side is a minimal FastAPI app with two Celery tasks behind it:

# tasks.py
@celery_app.task(name="tasks.echo")
def echo(message: str) -> dict[str, str]:
    ...

@celery_app.task(name="tasks.add")
def add(left: int, right: int) -> dict[str, int | str]:
    ...

# main.py (relevant part)
@app.post("/tasks/echo")
async def enqueue_echo(payload: EchoTaskRequest):
    task = echo.apply_async(args=[payload.message], queue="default")
    return {"task_id": task.id, "status": task.status, "queue": "default"}

@app.post("/tasks/add")
async def enqueue_add(payload: AddTaskRequest):
    task = add.apply_async(args=[payload.left, payload.right], queue="math")
    return {"task_id": task.id, "status": task.status, "queue": "math"}

@app.get("/tasks/{task_id}")
async def get_task(task_id: str):
    return serialize_task(AsyncResult(task_id, app=celery_app))

tasks.echo is routed to the default queue and picked up by worker-default; tasks.add is routed to math and picked up by worker-math. That routing is what makes it possible to target one specific worker’s breakpoints without the other worker’s traffic getting in the way.

Prerequisites

  • Rancher Desktop, set to the containerd engine (Preferences → Container Engine) — nerdctl talks to containerd, not to dockerd (moby), so this matters.
  • helm and kubectl configured against the rancher-desktop context.
  • PyCharm Professional — the Community Edition doesn’t support Python remote debugging.

Redis itself doesn’t need to be installed separately — it ships as a container in the same pod (redis:7-alpine) purely for local development. It’s not something you’d want to run this way outside a laptop-scale cluster.

Step 1. Build the images

git clone https://github.com/churnikov/k8s-dap-remote-debugger.git
cd k8s-dap-remote-debugger

First confirm the context:

kubectl config current-context

It should print rancher-desktop. There are three images to build: the API and one per Celery worker, each worker image built from the same Dockerfile.worker with a different CELERY_WORKER_QUEUE/CELERY_WORKER_NAME build arg. Building into the k8s.io namespace with nerdctl is what makes the images visible to Kubernetes — no image push/pull needed:

nerdctl --namespace k8s.io build -t k8s-remote-debugger:local .
nerdctl --namespace k8s.io build -f Dockerfile.worker \
  --build-arg CELERY_WORKER_QUEUE=default \
  --build-arg CELERY_WORKER_NAME=worker-default \
  -t k8s-remote-debugger-worker-default:local .
nerdctl --namespace k8s.io build -f Dockerfile.worker \
  --build-arg CELERY_WORKER_QUEUE=math \
  --build-arg CELERY_WORKER_NAME=worker-math \
  -t k8s-remote-debugger-worker-math:local .

Step 2. Deploy with Helm

helm upgrade --install k8s-remote-debugger ./chart

The chart already defaults every image to pullPolicy: Never and turns debugging on everywhere: debug.port: 5678 for the API, and 5679 / 5680 for worker-default and worker-math respectively.

[!note] Why pullPolicy: Never matters These images only exist locally — nerdctl builds them straight into containerd’s store, nothing gets pushed to a registry. Never tells the kubelet to use only what’s already on the node. Without it, the kubelet tries a registry pull, which either fails (ErrImageNeverPull) or, worse, silently grabs an unrelated public image with the same name/tag.

Override image names explicitly if needed:

helm upgrade --install k8s-remote-debugger ./chart \
  --set image.repository=k8s-remote-debugger \
  --set image.tag=local \
  --set image.pullPolicy=Never \
  --set workers[0].image.repository=k8s-remote-debugger-worker-default \
  --set workers[1].image.repository=k8s-remote-debugger-worker-math

Without --wait-for-client, the API starts serving traffic immediately — fine once you’re already attached, but a breakpoint on startup code (imports, app initialization, first request handling) can easily fire and pass before PyCharm connects. To pause the API until a debugger attaches, so it hangs at boot until you’re ready:

helm upgrade --install k8s-remote-debugger ./chart --set debug.waitForClient=true

Workers pause the same way, indexed by their position in values.yaml:

helm upgrade --install k8s-remote-debugger ./chart \
  --set workers[0].debug.waitForClient=true \
  --set workers[1].debug.waitForClient=true

Since it’s one pod, be careful with waitForClient here: if you set it on a worker and don’t have PyCharm ready to attach, that container blocks at startup and the pod won’t become ready, taking the API container down with it.

Step 3. Forward the ports

kubectl port-forward service/k8s-remote-debugger 8000:8000 5678:5678 5679:5679 5680:5680

That single command now covers everything:

  • http://127.0.0.1:8000/ — API root
  • http://127.0.0.1:8000/healthz — health check
  • http://127.0.0.1:8000/tasks/echo, /tasks/add, /tasks/{id} — enqueue and poll Celery tasks
  • 127.0.0.1:5678 — API’s debugpy port
  • 127.0.0.1:5679 — worker-default’s debugpy port
  • 127.0.0.1:5680 — worker-math’s debugpy port

Keep it running for the whole debugging session.

Step 4. Set up PyCharm

Because there are three independent debugpy listeners, set up one Python Debug Server configuration per port you actually plan to use:

  1. Run → Edit Configurations… → + → Python Debug Server.
  2. Create one configuration for the API: IDE host name 127.0.0.1, Port 5678.
  3. Create one for each worker you want to debug: same host, Port 5679 (worker-default) or 5680 (worker-math).
  4. For each, add a Path mapping from your local checkout to /app in the container — that’s where Dockerfile/Dockerfile.worker both copy the source (main.py for the API; celery_app.py and tasks.py for the workers). Without this, breakpoints won’t bind to the right source lines.
  5. Ignore PyCharm’s “install debugpy” hint — it’s already in requirements.txt.

PyCharm can run multiple debug sessions at once, so you can have the API and both workers attached simultaneously if you need to trace a request all the way through the queue.

Step 5. Trigger and catch a task

Set a breakpoint in tasks.py, say inside echo(). Start the matching Python Debug Server configuration (port 5679 for worker-default) so PyCharm is listening, then enqueue a task that routes to that worker:

curl -X POST http://127.0.0.1:8000/tasks/echo \
  -H 'content-type: application/json' \
  -d '{"message":"hello celery"}'

Execution should stop on the breakpoint inside PyCharm, with the usual call stack, locals, and debug console. Once you resume, fetch the result:

curl http://127.0.0.1:8000/tasks/<task-id>

The add task on the math queue works the same way against port 5680:

curl -X POST http://127.0.0.1:8000/tasks/add \
  -H 'content-type: application/json' \
  -d '{"left":2,"right":3}'

One thing worth watching for: the chart runs each worker with Celery’s default prefork pool at --concurrency=1, not --pool=solo. debugpy attaches to the parent process; prefork still forks a separate child process to actually execute the task, even with concurrency pinned to one. If a breakpoint inside tasks.py doesn’t fire the way you’d expect, that fork boundary is the usual suspect — overriding the worker’s command to add --pool=solo (single process, no fork, tasks run synchronously) is the standard fix, at the cost of any real concurrency, which is a reasonable trade for a debug session anyway.

Wrap-up

With debugpy wrapping every process — API and both Celery workers alike — and one port-forward reaching all of them, Kubernetes stops being a black box on either side of the queue: you can set a breakpoint in an HTTP handler, in a task body, or in both at once, and step through a request as it crosses from FastAPI into Redis and out the other side in a worker. The pattern from this repo carries over to real services with minimal changes — just remember debug ports are unauthenticated by design, so debug.enabled=false (and dropping the bundled Redis container) belongs in any configuration that leaves your local machine.