How to debug in Kuryr-Kubernetes

Hello! My name is Sunday Mgbogu and today I am going to discuss the various steps I took before I could solve my challenge of debugging in Kuryr-Kubernetes. As at time I wrote this blog, I was an Outreachy intern for the May’21 cohort with OpenStack on the Kuryr-Kubernetes project. Thanks to my mentors, Maysa De Macedo and Luis Tomas Bolivar. I owe everything I am doing right now to you wonderful folks. I was lucky to have met with you guys and I would continue to appreciate all the wonderful helps, guides and your time committed to see me succeed. So, I am going to share my challenges while starting out on the Kuryr project and how Maysa and Luis made things seem like a breeze.

When I joined the Kury Project, it was quite overwhelming to me how to navigate the codebase. Again, my mentors came to the rescue. For the past weeks, I have been wrapping my head around how to debug the kuryr codes that i write on my local machine.

Mistakes I made

First, I would need to take you through all the steps I tried to take to avoid repeating similar mistakes and getting burned out along the way. I started by trying to run Kuryr-kubernetes service on my regular IDE, Pycharm. Of course, I believe you too might be attempting similar things, but please you can stop. That might work but it requires someone with high technical ability to assist you configure your IDE to speak to the VM and allow you make changes on the IDE and it can ‘RUN’ the kuryr-kubernetes service. Just ignore this as this would actually burn you out. I believe, you can take the steps highlighted on this blog to setup a VM for your dev work. Please take a look and keep this blog handy. How to setup your work environment to become an OpenStack Developer. You would need to modify your local.conf file before you should run the stack.sh script. As at the time of writing this blog post, here is my local.conf file. If you run into this bug simply follow the fix highlighted right there and ./unstack.sh, ./clean.sh and sudo rm -rf /var/run/ovn and ./stack.sh again. Ensure you have a strong network connectivity and please you can start a tmux session to get you restarted in case loss in network connectivity.

Starting to contribute

Here is the thing, after setting up your local environment and you need to start try your hands on some vim handy tool. It is good you spend at least 24hours learning how to use the vim editor. Keep this tool handy for vim as that would definitely save you loads of hours of debugging.

Starting the debug process

I would need to talk about two important python debugging tools. One, the popular python debugger pdb and the recent interactive python debugger ipdb. Of course, you might have heard people talk about this but I would recommend you suspend everything you are doing right now and use the next 24hours to watch youtube videos and read up the documentation of both. I would next discuss the both debugging options and please feel free to choose any option that works best for you.

Python Debugger, pdb

To use this handy tool, all you need to do is to find where you want to start the debugging process and simply trigger the debugger by including this lines of code at the spot.

import pdb; pdb.set_trace()

After making some code changes, you would need to test your code. Simply include the above lines of code at any spot you want to start debugging and restart the kuryr-kubernetes service. Aha!!! That is actually where it begins to get interesting. Honestly, I used the documentation to figure out how to start and stop the kuryr-kubernetes service and I believe by now you might be asking yourself, where in the documentation does this exist. Please, do not burn out yet. The fact, is that you have overcome the most complex task if you manage to get this stage. If you are an Outreachy applicant struggling with your initial contribution, you can just relax as I would teach you two methods of starting your Kuryr-Kubernetes service. First of all, ask yourself, “Did I deploy Kuryr-kubernetes containerized or as a Systemd service?” Relax do not worry yourself as I will teach you how to confirm your Kuryr deployment mode.

If you simply used the above local.conf file that I shared, you can look at the line KURYR_K8S_CONTAINERIZED_DEPLOYMENT=False

You would notice that we issued a command here, telling devstack to deploy kuryr non-containerized. So what are the two modes of deploying Kuryr?

Containerized Kuryr-Kubernetes Deployment

Let us talk about deploying kuryr in a containerized mode. To deploy Kuryr as a containerized service, simply switch the above KURYR_K8S_CONTAINERIZED_DEPLOYMENT=False to True. Then run the stack up and boom, you are now running Kuryr as a containerized service. Now, how do you restart the kuryr-kubernetes service if you are running Kuryr containerized? First, the steps involves

1. make your code changes
2. rebuild the controller image
3. run the kuryr-controller

Make code changes and insert within the code you wish to check a controller log using LOG.debug('some information I want to check'). The controller would print this line in the controller logs and you can use the handy vim search above to run through the controller logs and check your code changes. Try and insert a keyword within the debug function so you can easily grep keyword and retrieve your keyword.

Now let me talk about rebuilding the controller image. To rebuild the controller image, simply follow this instruction right here . After building a new container image, you would need to trigger it so that the kuryr-controller would use the latest image build. Remember your latest changes is actually in the new container image you just built. Simply put, you have to find the already running pod by kubectl get pods and delete it so that the controller can restart with your current container image. So find the pod in the kube-system namespace by kubectl get pods -n kube-system. So delete the kuryr-controller pod by kubectl delete pods -n kube-system kuryr-controller-xxxxxx. Replace with the right kuryr-controller pod name. After deleting the kuryr-controller pod, the kuryr-controller restarts the container with the latest image. Now to see your changes, you need to retrieve the controller logs.

So if you check, you could see that the steps involved in checking for even minor changes like this using the containerized option is very tedious. Here comes the very best option which is running kuryr-kubernetes as a systemd service.

Systemd Kuryr-kubernetes Deployment

Wow! Finally, here is the savior to all your worries (sort of). To deploy kuryr-kubernetes as a systemd service, simply use the above flag on the local.conf and set the containerized deployment to False. Then stack up and find your kuryr-kubernetes.service running as a systemd service. To check simply systemctl status devstack@kuryr-kubernetes. You would likely receive this output. This says the kuryr-kubernetes.service is loaded at /etc/systemd/system/devstack@kuryr-kubernetes.service. Now let us look into this file and see the content by running cat /etc/systemd/system/devstack@kuryr-kubernetes.service . The output is here. You can see the ExecStart line that it runs the code /usr/local/bin/kuryr-k8s-controller --config-file /etc/kuryr/kuryr.conf. So simply issue that command and wait for the controller to drop you into the debugger.

/usr/local/bin/kuryr-k8s-controller --config-file /etc/kuryr/kuryr.conf

You can now use all the mechanics of debugging using pdb.

Python Debugger ipdb