How to write a useful bug report

This is one of those less glamorous elements of software development, and the one a lot of developers would rather pretend doesn’t exist. The dreaded bug! Yet I guarantee you that only the very simplest software has ever shipped without a bug. I realised, however, that many people subscribe bug reports of differing qualities. Sometimes we’ll get an email saying, simply “the application doesn’t work” or “website is down!” Worse still, when we check monitoring the application or website are very much working as far as we’re concerned. And that difference between what a user perceives and what we perceive does trouble me.

I also like to say that when we get such a report, we should never be satisfied with “works for us!” That’s a terrible answer, because sometimes the user may be experiencing a real problem because they’re using things differently. If, for example, your car didn’t start some mornings and you said it to your mechanics, and they said “well, I just started your car and it was just fine!” it wouldn’t be a help, would it? There’s a problem, and the real issue is that the problem can’t be reproduced. If you tell your mechanic “my car won’t start on frosty mornings” then they might have something more to go on. Although a good mechanic will always ask for more information, if you gather that information in advance you can reduce the number of communications steps in-between problem and resolution. So here are a few handy tips – and they work outside of software development too!

1. Assess whether the bug is important or not

All bugs should be fixed, but as software is complex there are bugs that need to be fixed right now, or which can be bundled into a batch of work. This is like the difference between a radiator running a bit cool and a radiator that’s leaking. The former is minor, the latter an emergency. Please tell us the priority. Do beware of trying to get everything perfect more quickly by saying everything is critical, because that can simply train your software engineers to de-prioritise all of your problems. And that’s a bad outcome.

2. Is it environmental?

Does the bug happen on every computer? On every phone? If you have a couple of devices handy to check with, and the problem only happens on one, then it’s worth mentioning that. If your car doesn’t start well when you’re in Spain, but is always fine in the UK, then that’s an environmental difference. With an application, if it works fine in Chrome but doesn’t work in Safari, then that matters. Similarly, no good software engineer should respond to a bug report of an issue and test only in one browser.

3. Is it temporal?

Does the problem only occur at certain times of the day? When you report a bug, explain exactly when it happened. This allows the developer to do two things – to check if the servers are doing something weird at that time, and to go through server logs to see if there’s anything interesting going on. For example, we had one site going down every day at 1am. A quick check of logs revealed that this was when the system was backed up, and the database had been accidentally set up so that you couldn’t back it up and access it at the same time. It was an easy fix – literally a few minutes. The client never knew about that one – it was our own monitoring that detected it. But the time data being consistent told us that it was a consequence of a regular, scheduled process and made it easy to check.

4. Is it user related?

If you know somebody else who also uses the system, ask them to try to reproduce your problem. Does it happen exactly the same way? If it does or it doesn’t, this is useful data to include – especially if it behaves differently for a different user. Many applications have complex security models, and a developer will likely sign in at a high level to check problems out. If there are differences in usage then this indicative of a problem with the user or security handling part of the system.

5. Explain the events that lead up to the problem

You’ve spent ten minutes filling in a complex form on your application, you hit save, and you get hit with an error. That’s really frustrating and annoying. Log the exact time.

Try again.

Did it work this time? Yes? Doesn’t matter, a problem still happened, so send in the report. Transient problems are still problems and will happen again – with the exact time we can review the server logs and see if there are any clues.

If it still doesn’t work, then it’s probably not transient, but still, send in the time, and the sequence of steps taken before the problem occurred. Was it because you uploaded a 50MB image? Was it a string that was too long? Something too short that wasn’t properly connected to another data item in the database? Your steps leading up to the problem will help us understand how it happened, and the timings will help too!

6. Include screenshots

If the problem is visual in nature, include screenshots, or even screen recordings, of the problem in action. This is especially important for usability issues. A screenshot or screen capture can capture a lot of detail that is very very hard to explain in words.

7. Is it really a bug, or is it a missing feature?

You should check whether the behaviour is actually as expected. A classic problem here is that of file uploads. On the one hand, high availability websites tend to have thirty second time outs. As a consequence, uploading huge files can be problematic and needs special consideration. On a one megabit/s connection, a compressed one megabyte file takes about ten seconds to upload. So a user with a poor connection may struggle to upload a file. What happens if they do? How gracefully do you want it handled? Should the platform pre-emptively check bandwidth or not? Was this asked for, was it something that could be expected, or was it a mistake by the developers? These are the kids of little things that trip up any development project – a developer will generally ensure everything works well over their own typical domestic connections, but if you never asked for the system to work from a wobbly satellite link up in Mogadishu then it’s not necessarily a bug if that use case isn’t well handled. But what about a weak connection from a phone in the Peak District running at 100kb/s? Is it worth spending a week on coding up a solution to that? Setting up special server infrastructure to handle slow connections? Well, maybe, if you have millions of customers who experience that problem regularly. Probably not if you have three.

8. What are the workarounds? Who do they affect?

A workaround is a way of dealing with a limitation or bug in the system. For example, using the example of the uploading problem above, would an effective and cheap workaround be to ask users something like “if you have a slow connection, please consider reducing the size of your files to under 1MB) in order to improve your chances of a successful upload”? But at the same time, if the bug is insisting people who are 14 are allowed to drive, then there’s likely to be no valid workaround.

9. Try to supply relevant data

It can be easy as well to supply too much information! Sorry! If you can keep things relevant, that’s helpful. Chances are they don’t need to know the make and model of your computer, if it’s web related – but knowing the operating system, browser, and browser version can be very useful if the problem happens only on one browser.

10. Be prepared to screenshare

Some problems just aren’t that easy to describe, and sometimes it’s really helpful for a developer to watch you in action as the bug happens. If you work close to one another, that’s easy – they just come over to your desk. But if it’s remote, you’ll need to do a screenshare. It’s worth noting that not all screenshares are made equal – and you may be asked to share your whole screen so that the developer can see what your operating system is playing at.

Example bug report

Problem: The alignment of blocks isn’t correct.

Important: Moderate – it’s not business critical but it’s not good attention to detail and that’s something our brand is famous for.

Description: If you look closely at the blocks in the image below, you’ll see that they don’t align. I’ve drawn a line to show it more clearly. This has happened in all browsers I’ve tried – Firefox, Chrome, and Edge, running on Windows 10. It’s not visible on my phone.

Steps to recreate: Simply visit https://wordpress.org/gutenberg/ and scroll down to the first block – look closely at the two blocks above and you can see they don’t align. You can visit in any Windows browser – Edge, Chrome or Firefox and the problem persists. I haven’t tried on a tablet, phone or alternative operating system.

Time: Happens always, noticed it when it was first released, but forgot to mention the bug – still there as of today.

Workarounds: I wasn’t able to adjust the blocks in any way that corrected this alignment problem without making things even worse.

Summary

Data is everything! The more you can supply to your developer when you raise a bug report, the more chance they have of being able to reproduce the problem in their local environment, to see exactly what’s happening in the system, and to correct it or to even declare it as expected behaviour.

Any ideas on how to improve bug reports? Any horror stories? Feel free to comment below!

Photo by Erik Karits on Unsplash