Beginner's Navigation Guide to the Galaxy of "Report Testing" in Software...

Every application has Reports... But are they mostly over-looked by testers? Do we really devote enough thought to plan on testing reports?

Well, I could not find enough stuff on the 1st few pages returned by Google for the search "Tips and Tricks for testing software reports", which prompted me to post this... Actually, there's another reason too... Reports is 1 part of the system, which cannot be neglected since their importance by the project teams are undermined, but they form a very crucial part of the system... Why so? Mainly because, it is on the information presented in reports that business make a lot of their decisions. This is something which will not be understood by the "Fake Testers" that are in business today.

Trying to broadly classify the reports that are a part of every software solution, I have come across 2 kinds of Reports - "Operational" and "Historical Reports". (If you can think of a broader classification, which I am sure you would, then please write it to me so that I can be proved to be a member of the Fake Software Testing Community).

Operational Reports - Reports in the application that is used to drive day-to-day applications.

Historical Reports - Reports on the basis of data retrieved from the database that forms the basis for key management decisions.

Am sure that you would have encountered both types... (and you can classify even more to it...)

Top 6 aspects that are mostly overlooked by the fake software tester, when planning on testing software reports are, in my humble opinion are as follows:-

1) Data Preparation for Testing Reports
a. Importance of Test Data - It's impossible to prepare test data for testing software reports in a single day!!! You will need to understand the business context and create all your test data accordingly, as close to real-life data as possible. The fake software his this irritating habit to create a "data
dump" with data that comes to his head, without understanding if this would be close to like-live data. This would Rank @ No. 1 in the "Most Horrible Practices of Software Testing" list. The Data that is being prepared should be close to like-live data and it's 1 of the best practices to have the business users take a quick look at this data that'd be tested.

b. Life Span of Application and Frequency of Report Generation - Mostly, you would need to create/generate your own data and so, please understand the life span of the application being tested, before coming up with like live data creation for testing. Also understand the frequency with which the reports would be
generated so that your data preparation can be like that.

c. Sourcing for Reports - If the data can come from an external data feed, please work with the corresponding systems to get the data in place for your testing. If the data is going to change almost on a daily basis, there would be situations wherein you will have to change dates or other attributes in the system. See if you can have an automated program setup to generate this data for you.

Please plan for data preparation for testing Reports as part of your "Data Preparation" Activity.

It is very difficult for me to imagine a fake tester dedicate so much time and plan for the above mentioned activities as part of testing. To quote an example, I remember seeing data prepared by a fake tester contain "Sundays", when the report is on a stock movement over a 5 week period. The fake software tester does not realize that the stock market is closed on Sundays, mostly!!!

2) NFRs for Reports
a. Response Times - Do reports need NFRs? This is 1 question that is asked by the fake software tester. Another set would state that the business never gave them any NFRs. Sample this, if the business user expects a report to be generated in 4 seconds, then that needs to be something which you need to test. A
true tester would question NFRs for Reports and would ensure that testing for this would be incorporated in the report.

b. Report Data Volumes - Another thing that the fake tester does not ask for is for how many records get processed for a particular report, at least a range. There is no point trying to test with 200 trillion records, if the business expectation of data is only around, say 40381 records :)!!!

Response Time and Record Volumes need to be understood before starting to test reports.

3) SQL Related Testing of Reports
This is 1 area that I believe that the fake tester is completely blind to.

a. SQL Profiler - After executing the report, it is a good habit to make use of a database profiler tool (SQL Profiler for SQL and not sure for other
databases), to run a trace and check for performance improvement. Giving the trace to the developer on the test systems would fully tell the developer on what tweaking he needs to do and would execute the report in a much quicker time.

b. Dynamic SQL Query - A lot of freshers miss this. I was once asked to do a check as to why reports take a lot of time to respond. When I looked into the Sproc, I realized that there was a lot of static SQL, when the need was for it to be dynamic. There was no point in me raising a defect stating that the
query takes a long time. Though you are the developer's enemy, please remember it is such small actions, which would go a long way in bridging the gap between you and the development teams.

c. Overnight reports - Now, in case the data is not based on daily transactions, and the report is required during the day, it makes a lot of sense to have the data archived into a separate table and a separate Sproc to get data from this table for the reports. I am really not sure if all the SQL Gurus of the world
would point their guns at me if this is incorrect, but again, please remember I have a lot of traits of the Fake Software Tester, and am currently in the
transition phase :) !!!

4) Print Functionality of Reports
Every Report would print into a paper. Though all the "Go-Green" people of the world would hate me for this, a true tester has to be "Non-Go-Green" and would, I believe waste a lot of paper in testing this functionality.

A true tester, before designing his tests for reports, would go the extra mile to understand the following
a. What is the printer used when the application is like-live
b. What is the printer configuration data when the application is like-live
c. What kind of paper is used for printing? (A4, A3, A2, etc... Any other specific configuration)

Importance of type of Paper - There’s reason for understanding the kind of paper. I remember 1 client of my peer, who used "Ruled Paper" for his testing, so that the test report had data in the corresponding columns. For some reason (I remember that it was incompatibility with other applications), that they got all reports printed on a paper. Now, this type of "Special Paper" was never used for testing by the test team and this resulted in a lot of surprises when the application went live.

Importance of Printer Configuration - Also, if the customer is using landscape printing on A2 for a type of report, then what value does the test team add in testing those reports on A4 sheet using Portrait configurations?

Importance of trying to print many times before Go-LIVE - And, please do a lot of print-outs to see how the print-outs would look in actual, how much ever times you can!!! Only if you take print-outs can you fix problems in areas where the printed copy goes out of the paper borders, totals are printed in a
separate page, proper page alignment for report header and footer, proper alignment for the summary, font related issues, etc.

This area, sadly, is also mostly neglected by the fake tester.

5) Import and Export Functionalities/Search Criteria/Localization
Every report has import and export possibilities. Export, definitely yes, but import???

Of course... I have seen cases where the user wants to import the query from a text file. So, the possibility of "Import" always exists, though it is very minimally used.

Now, we need to understand the various applications and versions of the applications to which the export should support and test with all of those... I do know of cases where the user was using very old versions of Excel and the application refused to export into these versions of the Office software. Sadly, there were a lot of escalations questioning delivery capability, by the time this escalation happened!!!

The search criteria are tested, but needs to be tested to ensure that the user does not search with what is not possible. Search can throw up a lot of defects. Would I want to save a search?

6) And understanding the business context and business logic behind the reports...
And as I had stated at the beginning, please understand the business logic and the business context behind the report that you are testing. Ask the following questions
"Why do you need the report for business?"
"What data is shown and what decisions can be made by this report"
"Why do you feel some columns are not required while some are required?"
"Why do you think that this Report makes sense to business and if not, why do you think that this report is not required"

And to re-iterate, there are a lot of other aspects of software reports that need to be tested. Not just the 6. Listed above is what I feel are the mostly overlooked aspects, which need to be re-looked at. And since Nobody's Perfect, neither am I.... If you really feel that there is more to add, please keep those comments coming in...

Very unfortunately, the fake software tester does not realize the importance of testing reports, which can, indirectly lead to a very wrong business decision, which can potentially bring down the entire company or cause huge losses to the company... It is my wish that there come a day every tester understands the business context of the reports that he tests and tests it using the business perspective...

And until this day arrives, the ranting of this fake software tester would definitely continue.....!!!