Improve error feedback on ASP.NET applications using Health Monitoring
When I switched from PHP to ASP.NET, I searched an equivalent to error_reporting() function.
I wanted to monitor all errors that happened on our production server. I know production software should be bug free but in the real world, bugs happen in many unexpected ways : an unusual navigation of the user that you hadn’t thought, an external database result which contains something that you didn’t even imagine in your units tests (database mocking is another topic…).
Context : we have a lot ASP.NET applications in production (more than 100), so modifying every application wasn’t an appropriate option.
Here is the steps of my implementation :
1. Natively, DotNet Framework adds in Windows Event Log an warning event when an exception occurs in an ASP.NET application (Yellow Screen of Death).
This mechanism is configurable through System Health Monitoring.
Windows event log is purged automatically but I currently have 3 or 4 weeks of history so it’s enough (“By default, event logs are set with a maximum file size of 512 KB. Then, when a log reaches this limit, events older than seven days are overwritten to prevent the log from exceeding the maximum file size”).
2. I created an executable (C# console project) which reads the event log and send an email to the appropriate developper(s) :
- I use a CSV file with correspondence between “ASP.NET application URL” and the “email’s developer”. Simple but effective to avoid any dependency of an external resource that could fail (such as database, which will prevents me from being notified of -the so-called failure).
- In fact, I still depend on the mail server… (I could provide a fail over solution that uses the command "NET SEND", SNMP… It would be good but a bit luxury at the stage).
3. A Windows Scheduled Task is configured (every 5 minutes) on the machine running IIS to run the executable.
- A text file contains the date and time of the last exception that was sent by mail.
Exceptions are sent by email and only once regardless of how often the scheduled task is executed. If the scheduled task didn’t run for any reason, just restart it and it will send emails back where it left off.
4. Display a Custom Error Page with the opportunity for the user to enter a comment and then sent it by email to the appropriate developer.
This mechanism is configurable through Web.config file. I created an .aspx page as simple as possible (no external dependencies) with a textbox and a "send" button.
This helps to stimulate the user feedback and hide the error details (for security reasons).
=> Conclusion : the main advantage of this hand made tool is that it requires no change in source code of the other applications.
- 90% of users don’t call (or don’t know how to call) the IT department when they encounter an exception.
- A small number of exceptions that occur aren’t visible to the user (mainly session handling errors).
- The most common exceptions are easy to reproduce and fix (80-20 rule ?).
- Then it becomes increasingly difficult to diagnose the type of exception and the call stack isn’t sufficient to understand what happened (I need to know the data entered by the user).
- I also developed a web application to group, sort and filter the exceptions but this will be the subject of another article.
Possible improvements :
- Collect user input data and send it to the developer.
- Find a way to catch and report client side Javascript exceptions to the server. Any idea ?
References :