Profiling & Debugging

In any real world scenario, an app might start exhibiting undesirable behavior in terms of resource spikes. CPU/Memory spikes are not uncommon in most cases.

Dapr allows users to start an on-demand profiling session using pprof through its profiling server endpoint and start an instrumentation session to discover problems and issues such as concurrency, performance, cpu and memory usage.

Enable profiling

Dapr allows you to enable profiling in both Kubernetes and stand-alone modes.

Stand-alone

To enable profiling in Standalone mode, pass the --enable-profiling and the --profile-port flags to the Dapr CLI: Note that profile-port is not required, and if not provided Dapr will pick an available port.

dapr run --enable-profiling true --profile-port 7777 python myapp.py

Kubernetes

To enable profiling in Kubernetes, simply add the dapr.io/enable-profiling annotation to your Dapr annotated pod:

   annotations:
    dapr.io/enabled: "true"
    dapr.io/app-id: "rust-app"
    dapr.io/enable-profiling: "true"

Debug a profiling session

After profiling is enabled, we can start a profiling session to investigate what’s going on with the Dapr runtime.

Stand-alone

For Standalone mode, locate the Dapr instance that you want to profile:

dapr list
APP ID           DAPR PORT     APP PORT  COMMAND      AGE  CREATED              PID
node-subscriber  3500          3000      node app.js  12s  2019-09-09 15:11.24  896

Grab the DAPR PORT, and if profiling has been enabled as described above, you can now start using pprof to profile Dapr. Look at the Kubernetes examples above for some useful commands to profile Dapr.

More info on pprof can be found here.

Kubernetes

First, find the pod containing the Dapr runtime. If you don’t already know the the pod name, type kubectl get pods:

NAME                                        READY     STATUS    RESTARTS   AGE
divideapp-6dddf7dc74-6sq4l                  2/2       Running   0          2d23h

If profiling has been enabled successfully, the runtime logs should show the following: time="2019-09-09T20:56:21Z" level=info msg="starting profiling server on port 7777"

In this case, we want to start a session with the Dapr runtime inside of pod divideapp-6dddf7dc74-6sq4l.

We can do so by connecting to the pod via port forwarding:

kubectl port-forward divideapp-6dddf7dc74-6sq4 7777:7777
Forwarding from 127.0.0.1:7777 -> 7777
Forwarding from [::1]:7777 -> 7777
Handling connection for 7777

Now that the connection has been established, we can use pprof to profile the Dapr runtime.

The following example will create a cpu.pprof file containing samples from a profile session that lasts 120 seconds:

curl "http://localhost:7777/debug/pprof/profile?seconds=120" > cpu.pprof

Analyze the file with pprof:

pprof cpu.pprof

You can also save the results in a visualized way inside a PDF:

go tool pprof --pdf your-binary-file http://localhost:7777/debug/pprof/profile?seconds=120 > profile.pdf

For memory related issues, you can profile the heap:

go tool pprof --pdf your-binary-file http://localhost:7777/debug/pprof/heap > heap.pdf

Profiling allocated objects:

go tool pprof http://localhost:7777/debug/pprof/heap
> exit

Saved profile in /Users/myusername/pprof/pprof.daprd.alloc_objects.alloc_space.inuse_objects.inuse_space.003.pb.gz

To analyze, grab the file path above (its a dynamic file path, so pay attention to note paste this one), and execute:

go tool pprof -alloc_objects --pdf /Users/myusername/pprof/pprof.daprd.alloc_objects.alloc_space.inuse_objects.inuse_space.003.pb.gz > alloc-objects.pdf