Reliable Embedded Systems: Using The Arduino Watchdog With Crash Tracking

The previous article in this series introduced the “Watchdog Timer “ as a tool to use to build more reliable systems; it will reset the Arduino processor in the event of the application locking up.

However, as in medicine, this is like treating the symptom and not the cause. Why did the application lock-up in the first place?  There are a plethora of reasons for this, from your own code to hardware not doing what you expect. Whatever the reason “Crash Tracking” can help you home in on the problem.

This article describes a concept and an Arduino library that can be used to try and zero in on what happened. If you know what happened then you can try and prevent it happening again.

Knowledge is power… 😁

Intermittent Problems  

Intermittent crashes or unexpected behaviour is the hardest type of problem to track down. When you study the problem it often goes away only to pop-up as soon as your back is turned. The traditional way of debugging Arduino code is to sprinkle the code with serial print statements to show where the code is at any one time. However, this is not something that you can do with a deployed system. In this case, you need something that will log the place in the code that it was when the problem occurred. When you get the unit back to base you can check the logs and hopefully, there will be a smoking gun in there to help you find the problem.

Crash Tracking Using the Watchdog

Enter the concept of “Crash Tracking”.

In a previous article, I introduced the Good ‘Ol Arduino Watchdog and the little known fact that it has the option to trigger an interrupt before it resets the system. In this interrupt routine, you can log state information –  basically, anything that might be useful it tracking down your fault and also where in the program you were when disaster struck. Phil over at MegunoLink has written a nice crash library for the Arduino that does just this. The article is very comprehensive and shows you how to use the library and how to retrieve the place in the code you were when the watchdog fired along with an example:  

You can download the code from Github and when I was setting it up, I re-jigged the structure on disk a bit when placing it in my Arduino Sketchbook’s library. It ended up looking like this:

Pasted Graphic 1.png
Arduino Crash Libary layout

Basically I created an “examples” and “CrashTracking” subfolder and copied the contents of program.cpp into a fresh Arduino Sketch and saved it as “CrashTracking.ino”. I then moved ApplicationMonitor.cpp and ApplicationMonitor.h into the root of Arduino Crash Monitor.

Using with a Sleepy Pi Application

In the article on the Watchdog and Sleepy Pi, I discuss using the watchdog with the Sleepy Pi sleep modes. The bottom line is that if you use any of the Sleepy Forever modes you have no problems. You only need to worry if you use the watchdog for periodic waking, in which case re-setup the crash tracking after each wakeup.

Summary

This is a very powerful technique for improving the quality of your code and tracking down those pesky intermittent problems. I’d highly recommend giving it a goat part of your strategy toolbox.

Leave a Comment

Your email address will not be published. Required fields are marked *