Reliable Embedded Systems: Using The Arduino Watchdog With Crash Tracking

The previous article in this series introduced the “Watchdog Timer “ as a tool to use to build more reliable systems; it will reset the Arduino processor in the event of the application locking up.

However, as in medicine, this is like treating the symptom and not the cause. Why did the application lock-up in the first place?  There are a plethora of reasons for this, from your own code to hardware not doing what you expect. Whatever the reason “Crash Tracking” can help you home in on the problem.

This article describes a concept and an Arduino library that can be used to try and zero in on what happened. If you know what happened then you can try and prevent it happening again.

Knowledge is power… 😁

Intermittent Problems  

Intermittent crashes or unexpected behaviour is the hardest type of problem to track down. When you study the problem it often goes away only to pop-up as soon as your back is turned. The traditional way of debugging Arduino code is to sprinkle the code with serial print statements to show where the code is at any one time. However, this is not something that you can do with a deployed system. In this case, you need something that will log the place in the code that it was when the problem occurred. When you get the unit back to base you can check the logs and hopefully, there will be a smoking gun in there to help you find the problem.

Crash Tracking Using the Watchdog

Enter the concept of “Crash Tracking”.

In a previous article, I introduced the Good ‘Ol Arduino Watchdog and the little known fact that it has the option to trigger an interrupt before it resets the system. In this interrupt routine, you can log state information –  basically, anything that might be useful it tracking down your fault and also where in the program you were when disaster struck. Phil over at MegunoLink has written a nice crash library for the Arduino that does just this. The article is very comprehensive and shows you how to use the library and how to retrieve the place in the code you were when the watchdog fired along with an example:  

#include "ApplicationMonitor.h"
 
Watchdog::CApplicationMonitor ApplicationMonitor;
 
// An LED to flash while we can. 
const int gc_nLEDPin = 13;
 
// countdown until the program locks up. 
int g_nEndOfTheWorld = 15; 
 
// number of iterations completed. 
int g_nIterations = 0;     
 
void setup()
{
  Serial.begin(9600);
  pinMode(gc_nLEDPin, OUTPUT);
  Serial.println("Ready");
 
  ApplicationMonitor.Dump(Serial);
  ApplicationMonitor.EnableWatchdog(Watchdog::CApplicationMonitor::Timeout_4s);
  //ApplicationMonitor.DisableWatchdog();
 
  Serial.println("Hello World!");
}
 
void loop()
{
  ApplicationMonitor.IAmAlive();
  ApplicationMonitor.SetData(g_nIterations++);
 
  Serial.println("The end is nigh!!!");
 
  digitalWrite(gc_nLEDPin, HIGH);   // turn the LED on (HIGH is the voltage level)
  delay(200);               // wait for a second
  digitalWrite(gc_nLEDPin, LOW);    // turn the LED off by making the voltage LOW
  delay(200);               // wait for a second
 
  if (g_nEndOfTheWorld == 0)
  {
    Serial.println("The end is here. Goodbye cruel world.");
    while(1)
      ; // do nothing until the watchdog timer kicks in and resets the program. 
  }
 
  --g_nEndOfTheWorld;

You can download the code from Github and when I was setting it up, I re-jigged the structure on disk a bit when placing it in my Arduino Sketchbook’s library. It ended up looking like this:

Basically I created an “examples” and “CrashTracking” subfolder and copied the contents of program.cpp into a fresh Arduino Sketch and saved it as “CrashTracking.ino”. I then moved ApplicationMonitor.cpp and ApplicationMonitor.h into the root of Arduino Crash Monitor.

Using with a Sleepy Pi Application

In the article on the Watchdog and Sleepy Pi, I discuss using the watchdog with the Sleepy Pi sleep modes. The bottom line is that if you use any of the Sleepy Forever modes you have no problems. You only need to worry if you use the watchdog for periodic waking, in which case re-setup the crash tracking after each wakeup.

Summary

This is a very powerful technique for improving the quality of your code and tracking down those pesky intermittent problems. I’d highly recommend giving it a goat part of your strategy toolbox.