Join for free
Omah's Avatar
Omah
Chatterbox
Omah is offline
Ludlow
Joined: Feb 2014
Posts: 10,147
Omah is male  Omah has posted at least 25 times and has been a member for 3 months or more 
 
06-10-2020, 12:30 AM
1

Excel: Why using Microsoft's tool caused Covid-19 results to be lost

https://www.bbc.co.uk/news/technology-54423988

The badly thought-out use of Microsoft's Excel software was the reason nearly 16,000 coronavirus cases went unreported in England. And it appears that Public Health England (PHE) was to blame, rather than a third-party contractor.

The issue was caused by the way the agency brought together logs produced by commercial firms paid to analyse swab tests of the public, to discover who has the virus. They filed their results in the form of text-based lists - known as CSV files - without issue.

PHE had set up an automatic process to pull this data together into Excel templates so that it could then be uploaded to a central system and made available to the NHS Test and Trace team, as well as other government computer dashboards.

The problem is that PHE's own developers picked an old file format to do this - known as XLS. As a consequence, each template could handle only about 65,000 rows of data rather than the one million-plus rows that Excel is actually capable of. And since each test result created several rows of data, in practice it meant that each template was limited to about 1,400 cases. When that total was reached, further cases were simply left off.

Speaking in the House of Commons, the Health Secretary Matt Hancock suggested that the problem had emerged as a result of PHE using a "legacy system" and a decision had been taken two months ago to replace it. Presumably, however, this specific problem had not been spotted. Otherwise PHE would have realised that the flaw would come into effect before the upgrade was complete.

To handle the problem, PHE is now breaking down the test result data into smaller batches to create a larger number of Excel templates. That should ensure none hit their cap.

But the current clunky system needs to be replaced by something more advanced that excludes Excel, as soon as possible.
The use of the term "legacy system" is a legacy itself - AFAIK, it was coined in the 1980's. I first came across it in the 1990's when moving mainframe application systems to networked PC systems in the private sector and then again in the 2000's when I worked for the Civil Service - was shocked to discover how many such systems still existed and had to be supported .....

As for CSV's ..... they were "invented" in the 1970's for use by FORTRAN programs .....

Excel itself is over 30 years old but has been substantially updated; however, the XLS format was superseded in 2007 .....
Tedc's Avatar
Tedc
Senior Member
Tedc is offline
Berkshire, UK
Joined: Aug 2016
Posts: 4,872
Tedc is male  Tedc has posted at least 25 times and has been a member for 3 months or more 
 
06-10-2020, 10:27 AM
2

Re: Excel: Why using Microsoft's tool caused Covid-19 results to be lost

Once again, the users are casting aspersions, towards the Software Packages, rather than at the idiots whom we pay to manage them.

Remember the massive hits, on the NHS, by Ransomware, a little while back? The fixes were available, but, apparently, everyone was too busy to install them.

With so many lives at stake, you'd hope that the various DP departments would take the time to monitor changes, in whatever packages they have in place, and keep them up to date.

Is it too much to expect that someone in the DP section would be reading all update information to see if it is his/her system still measures up?

It's not so much the products, that are at fault, it's the users!

It was the 25th anniversary of the Windows 95 product, in August, I think, and some are still using the original packages!

Worry not, so much, about Legacy products, worry more about Legacy DP staff!


RTFM

Bruce's Avatar
Bruce
Chatterbox
Bruce is offline
Wollongong, Australia
Joined: Apr 2012
Posts: 15,218
Bruce is male  Bruce has posted at least 25 times and has been a member for 3 months or more 
 
10-10-2020, 05:14 AM
3

Re: Excel: Why using Microsoft's tool caused Covid-19 results to be lost

Originally Posted by Tedc ->
Once again, the users are casting aspersions, towards the Software Packages, rather than at the idiots whom we pay to manage them.


You're absolutely right. You would have to ask why on earth they weren't using a database like Access for such important information. Spreadsheets are fine for small, simple, flat databases or tables but for something so important you would think they would be using something more suitable.

The pre 2007 version of Excel only has something like 65000 rows and 256 columns which is probably the reason they lost data. I am guessing that they continue using it because they are using VBA to import the csv information and the post 2007 version of Excel requires a .xlsm file rather than the default .xlsx file to run VBA.

You are right it is total idiocy and amazing that it took so long for someone to notice.
swimfeeders
Chatterbox
swimfeeders is offline
Shropshire
Joined: Oct 2014
Posts: 24,056
swimfeeders is male  swimfeeders has posted at least 25 times and has been a member for 3 months or more 
 
10-10-2020, 05:30 AM
4

Re: Excel: Why using Microsoft's tool caused Covid-19 results to be lost

Originally Posted by Bruce ->
You're absolutely right. You would have to ask why on earth they weren't using a database like Access for such important information. Spreadsheets are fine for small, simple, flat databases or tables but for something so important you would think they would be using something more suitable.

The pre 2007 version of Excel only has something like 65000 rows and 256 columns which is probably the reason they lost data. I am guessing that they continue using it because they are using VBA to import the csv information and the post 2007 version of Excel requires a .xlsm file rather than the default .xlsx file to run VBA.

You are right it is total idiocy and amazing that it took so long for someone to notice.
Hi

Government stupidity.

We have a very sophisticated database in use by Local Authorities and PHE.

It was not specified to be used by the Private Labs who are doing the testing.

This was done on the grounds of cost.

A stupid decision.
 



© Copyright 2009, Over50sForum   Contact Us | Over 50s Forum! | Archive | Privacy Statement | Terms of Use | Top

Powered by vBulletin Copyright ©2000 - 2024, Jelsoft Enterprises Ltd.