Eusprig 2014 Delft – For Excel Builders and Inspectors

Thursday 3 July 2014, Delft, the Netherlands. And it’s FREE thanks to our sponsors!

This is for the two sides of the house – those who use spreadsheets to solve their problems, and those who are concerned about the risks introduced by the use of spreadsheets.

Unlike the serious keynotes of previous years, this year begins with, of all things, Excel and Music. It should start the day for us in good humour!

Both days feature Ray Panko on spreadsheet error detection and auditing.

But wait, there’s more … on Friday 4th July the vendors will be showing their wares – David Colver will talk about Operis OAK, I’ll talk about XLTest.

And if that’s not enough, check out the Excel Experience Day in Amsterdam on May 14th.

Delft July 2014 – Three Spreadsheet Days

Thursday July 3rd
Time Room A: Plenary and Risk track Room B: Building track
9:00 Opening
9:30 SheetMusic – Thomas Levine
10:00 Financial optimization and long term control of complex real estate projects – Jaap Labrie(sponsored)
10:30 Coffee break
11:00 Spreadsheet development for FP&A : Evolution of a 360 degree approach – Paula Jennings Zero Effort Excel Models – Henk Vlootman
11:30 Exploring the limits of Excel – Eloy Caballero Tony De Jonker – Scenario Tools in Excel
12:00 Mapping of the financial modeling ecosystem in large scale infrastructure projects – Haydn Palliser, Rickard Warnelid Bjoern Stiel – SpreadGit
12:30 Lunch
13:30 The FAST standard – Emily Cowan 13:30 Using XML for data storage and –validation in Excel apps: a plea – Frans Bus
14:00 Spreadsheet Model error experiences – Jack Avon Spreadsheet vs ERP – where does the balance lie? – John Kidd
14:30 Demo track 
15:00 Coffee break
15:30 Spreadsheet Inspection (Auditing): Why, How, and How Much? – Ray Panko
16:00 What’s new in Excel 2013 – Steve Kraynak, Leif Brenne
16:45 Closing and drinks

The academics get a special day of their own, July 2nd:
SEMS 14:

•Birgit Hofer, Dietmar Jannach, Thomas Schmitz, Kostyantyn Shchekotykhin and Franz Wotawa. Tool-supported fault localization in spreadsheets: Limitations of current research practice
•Dietmar Jannach, Thomas Schmitz and Kostyantyn Shchekotykhin. Toward Interactive Spreadsheet Debugging
•Raymond Panko. Improving Methods in Spreadsheet Error Research
•Richard Paige, Dimitris Kolovos and Nicholas Matragkas. Spreadsheets are models too
•Birgit Hofer and Franz Wotawa. On the Usage of Dependency-based Models for Spreadsheet Debugging
•Daniel Kulesz. A Spreadsheet Cell-Type Model for Testing
•Pablo Palma. SBBRENG: Spreadsheet Based Business Rule Engine
•Peter Sestoft. End-user development via sheet-defined functions
•Sohon Roy and Felienne Hermans. Dependence Tracing Techniques for Spreadsheets: An Investigation
•Jácome Cunha, Joao Fernandes, Jorge Mendes, Rui Pereira and João Saraiva. MDSheet – Model-Driven Spreadsheets
•Thomas Levine. Automatically assessing the relevance, quality, and usability of spreadsheets
•Thomas Levine. Sheetmusic: Making music from spreadsheets
•Raymond Panko. Are We Overconfident in Our Understanding of Overconfidence?
•Joeri van Veen and Felienne Hermans. Anonymizing Spreadsheet Data and Metadata with AnonymousXL
•Bas Jansen and Felienne Hermans. Using a Visual Language to Create Better Spreadsheets


Posted in Uncategorized | Tagged , , , , , , , | Leave a comment

Dumb autoresponders

I went on holidays before Christmas and set up an email autoresponder to say I was out of the office.
So I left and next day decided to use the free wifi to check the email. Not something one should normally do on holidays, but I was curious. I found about a thousand repeated emails from which had sent out an email at 02:49 and received an autoresponse to which it replied “this email address is not monitored” and to which it received an autoresponse … etc.
After 1400 of these I got my ISP to turn off the autoresponder for me (thanks, Blacknight support people).
I would have thought this would be a well known issue that any email system would handle.
I would have set up the autoresponder to not reply to any messages with “noreply” in the sender, but that could not be done. said they would consider a future feature to detect email loops.
Has any else had that problem and how did you resolve it?
According to wikipedia:

The mail system should retain headers of incoming email while performing any type of auto-forwarding operation.
Auto Responder: Do not send more than ‘x’ replies to the same sender.

Posted in Uncategorized | Tagged , | Leave a comment

Notes from SoftTest “Kick Starting Automated Testing”

Belfast 19 Sep

Paul Gerrard’s talk was a call to review requirements.
He distinguished definitions of Check vs Test.
In Safety tests, they speak of a check for the part being “in good working order”
Look up Behaviour Driven Devt – a requirements driven structured language eg Cucumber, and DefoSpam.
You don’t want to end up with a long, brittle chain of unmaintainable test scripts.
Alternative: robot framework, write your own interface
See, SP.QA free story platform

John McHale, Storm tech, presented Microsoft ALM, VS Ultimate 2012
MS Test Center for manual testing with recording and playback.
Test Impact: What LOC were tested, to see what tests might be impacted by a change in code.

Dorothy Graham and on
Efficiency of finding bugs:
Regression testing 9%
Manual testing 28%
Exploratory testing 58%

See BCS Grove report “Experience-driven Test automation”
High maintenance cost of testware leads to abandonment. What happens to all your scripts if you change tools?
Solution: Separate tests from scripts. (I am reminded of the saying that there is no problem in computer science that cannot be solved by another level of indirection / interface)
Tools don’t replace testers, they support them.
Tester don’t need to learn coding: lose a good tester and gain a poor programmer.
Test automation is a developer’s job.
Deadline pressure usually pushes you back into manual testing.

David Evans, Neuri:
Attitude of learning from mistakes
Testing is never finished, merely stopped; defend your stopping heuristics.
Choosing which features to test = What do we have to let out that we assume is not working?
A bug report does not necessarily cause a bug fix to be added to a work queue.
Every bug (discovered in exploratory testing) is evidence of a missing (unit) test.
Value of a test is speed – timeliness to take action.
He is not fond of end-to-end testing – but I think a counter-example might be the Wobbly Bridge.
Transport analogy: the speed of the bus is never the point, but the service. So, neither is the speed of testing (goes against value argument?)
“Testing is slowing down development” –> Slowing the production of non-functioning code.
Coverage: lines of code are not paths.

Posted in Software Testing | Tagged | Leave a comment

Kick Starting Automated Testing – 19 Sep programme

SoftTest Ireland Inaugural Conference
19th Sept 2013, The Holiday Inn, Ormeau Avenue, Belfast.

Sessions Time Main room (Titanic 1&2) Syrian Venetian
0830 – 0900 Registration and Coffee
0900 – 0915 Opening remarks Brian Lambert – Chair, Softest
0915 – 1015 Paul Gerrard, How to Eliminate Manual Test Lab Feature Checking
1015 – 1115 Tom Roden, Building a lean, mean, BDD test automation machine
John MacHale,Borland Test Lab Easily created automated tests from manual test runs using Microsoft’s ALM tool suite
1115 – 1145 Break – Coffee Exhibition and Test Lab
1145 – 1245 Dorothy Graham Borland Intelligent Mistakes in Test Automation
1245 – 1400 Lunch Demonstration by SQS
1400 – 1500 David Evans, What testers & developers can learn from each other
John MacHale Borland Test Lab Easily created automated tests from manual test runs using Microsoft’s ALM tool suite
1500 – 1530 Break – Coffee Exhibition and Test Lab
1530 – 1630 Graham Abell ios-driver – Automate your iOS applications
1630 – 1715 Plenary Session / Q&A All speakers
1715 – 1730 Closing Remarks Brian Lambert – Chair, Softest
Format – There will be talks and demonstrations taking part in the Syrian and Venetian Rooms during the running of the main speaker programme and during lunch. Delegates are free to choose which session they would like to attend.
To join our group simply subscribe via our website




Paul Gerrard
Paul Gerrard is a consultant, teacher, author, webmaster, developer, tester, conference speaker, rowing coach and a publisher. He has conducted consulting assignments in all aspects of software testing and quality assurance, specialising in test assurance. He has presented keynote talks and tutorials at testing conferences across Europe, the USA, Australia, South Africa and occasionally won awards for them. Educated at the universities of Oxford and Imperial College London, in 2010, Paul won the Eurostar European Testing excellence Award. In 2012, with Susan Windsor, Paul recently co-authored “The Business Story Pocketbook”. Paul is Principal of Gerrard Consulting Limited and is the host of the UK Test Management Forum.

Tom Roden
Tom Roden is the Head of Agile Services at SQS UK where he works as a coach, tester, scrum master, trainer, consultant and general quality enthusiast. Tom has spent the many years as a consultant working with project teams, supporting on-going improvement and transition through process and practice refinement, influenced by agile and lean principles. With a background in testing and test management he specialises as a test and quality coach, setting strategies for testing and working to empower teams to harness the benefits of both exploratory testing and scripted automated testing using test driven practices such as specification by example. Tom has presented at software quality and testing events across the UK and internationally.

Dorothy Graham
Dorothy Graham has been in software testing for 40 years, and is co-author of 4 books: Software Inspection, Software Test Automation, Foundations of Software Testing and Experiences of Test Automation. She is currently working on a new book and wiki with Seretta Gamba on Test Automation Patterns. Dot was programme chair for two EuroSTAR conferences and is a regular speaker at testing conferences worldwide. She has been on the boards of conferences and publications in software testing. She was a founder member of the ISEB Software Testing Board and was a member of the working party that developed the ISTQB Foundation Syllabus. She was awarded the European Excellence Award in Software Testing in 1999 and the first ISTQB Testing Excellence Award in 2012.

David Evans
David Evans is partner at Neuri Consulting. He is a veteran of agile testing and an active member of the agile community, with over 22 years of IT experience. A thought-leader in the field of agile testing, he has trained and provided consulting for clients in the UK, Ireland, Sweden, Germany, Australia, Israel, South Africa and Singapore. David is a regular speaker at international conferences and has had a number of articles published in professional journals.

Graham Abell
Graham is Technical Test Lead at Paddy Power. He has 8 years’ experience in the Financial sector, having worked in Insurance, Poker, Banking and Sports Betting. Graham has worked with many test frameworks – commercial (HP’s QTP, Microsoft’s Coded UI, Froglogic’s Squish) and open source (Selenium, jBehave) In this time he has used Java, Python, C#, Objective C and VBScript. Graham is a core contributor to the ios-driver tool, which allows automation on iOS devices using the Selenium Wire Protocol. In his spare time he cooks pizzas, mills wood and is building a 3D printer to create robots.

We would like to thank all our speakers and contributors to today’s event for
giving freely of their time at this not for profit event.


Soft Test Ireland would like to thank our sponsors Borland, HP, Inter Trade
Ireland, Sogeti, ISA Software Skill- net, and SQS for their continued support.

Posted in Software Testing | Tagged , , | Leave a comment

Conference : Kick Starting Test Automation, Belfast

SoftTest Ireland, the community for software testers in Ireland, is pleased to announce its inaugural whole day event.
The focus of this conference is Automated Testing and will provide a platform for delegates to learn strategies, techniques and practices from peers and leaders in the field. So come along and join us for the day!

You are invited to attend:
Kick Starting Test Automation, Belfast
Kindly Sponsored by Borland, HP, InterTradeIreland, ISA Software Skillnet, Sogeti, and SQS

Thursday 19th September

There will be a range of interesting presentations and workshops including our keynote speakers below:

Dorothy Graham: Intelligent Mistakes in Automation

David Evans: What Testers and Developers Can Learn From Each Other

Paul Gerrard: How to Eliminate Manual Feature Checking

Participants Include: Sogeti, SQS, Borland, Microsoft and HP.

To Book see:

Delegate Rate £55 – Early Bird Rate – £30 (Before 16th August)
Holiday Inn,
Ormeau Avenue,

Posted in Software Testing | Tagged , , , , | Leave a comment

EuSpRIG 2013 Conference report

4-5 July 2013, University of Greenwich, London.

We had a slightly smaller number of attendees this year than we expected, considering the publicity that attended recent spreadsheet errors, such as the London Whale or the Reinhart & Rogoff debacle. Tell me: what would attract you to the EuSpRIG conference? What would you want from it that would make it worthwhile to spend a day learning from experts on spreadsheet risk, development, and audit?
The following are my notes on the presentations. They reflect my personal interests and so are not in any way speaking on behalf of EuSpRIG.
“Multidisciplinary Engineering Models: Methodology and Case Study in Spreadsheet Analytics”; David Birch, Helen Liang, Paul H J Kelly, Glen Mullineux, Tony Field, Joan Ko and Alvise Simondetti, Imperial College London, UK.
David Birch presented a methodology and toolkit for analysing Integrated Resource Management (IRM) spreadsheet models, using the ARUP model as an example. IRM models are large and getting larger all the time, handling a thousand variables in 250,000 formulas over many worksheets. Each worksheet represents one discipline which has a specific team and there is a key role in integrating updates of these sheets into the main model. They analyse a slice of the model, and produce a visualisation of the links between the various disciplines. This enables a knowledgeable reader to validate whether the spreadsheet matches a conceptual model of communication dataflow. They also performed input sensitivity analysis, eg 2,563 simulation runs each taking less than a second on a fast machine. And applied it to a system of 500 connected spreadsheet files. This is, I think, the first paper we have had from the engineering world, it is a welcome change from the traditional focus on financial models. It certainly gave me an idea for an enhancement to my XLTest addin!

“XLSearch: A Search Engine for Spreadsheets”; Michael Kohlhase, and Corneliu Prodescu, Jacobs University Bremen, Germany.
This was a serious computer science paper. I suspect Michael lost a number of the audience at the word “ontology”, but this is a really clever way, smarter than regular expressions, of searching for any given formula pattern in an index collected from spreadsheets. Remarkably, it searches in about 50ms regardless of how many formulas there are. To test it, he is looking for a corporation with at least 1 billion formulas; to get that, they would need several million spreadsheets. If you work in one of the Fortune 100, that may well be true of your organisation, so if you ever wished you could search them all for any given formula, call him.

“Practical Challenges with Spreadsheet Auditing Tools”; Daniel Kulesz, and Jan-Peter Ostberg, University of Stuttgart, Germany
Daniel reflected on the differences between spreadsheet auditing tools and traditional code static analysis tools (TSATs), and on how useful they are to casual users of spreadsheet. TSATs are non-destructive: they provide the report in a view layer that does not change the underlying code, and can refresh if the code changes. Obviously that is easier to do for one-dimensional text than it is for the two-dimensional grid, especially as the object of interest (eg a formula block) can span large ranges. The students found the use of colour coding problematic as they inferred significance from the colours and it was not reversible. They found the tools less customisable than TSATs; and were disappointed that none were localised with a German user interface. They also wanted the software to prioritise the findings for them and point out definite errors rather than a list of possible problems with a high number of “false positives”. As the industry partner did not perceive faulty spreadsheets to be a major risk, they picked an auditing tool on the basis of price (five licenses for less than 1000 euro) rather than features. The German version of the paper gives more details:

Excel 2013 Spreadsheet Inquire; Patrick O’Beirne, Systems Modelling Ltd, Ireland
For my own paper, I reviewed the Inquire add-in for Excel 2013. This is very much a version 1.0 as there are quite a few shortcomings. In particular, it lists formulas individually rather than by distinct formulas (ie a block of one formula copied). I provide a helper addin to make that list more useful:

It is important in that it shows that MS are now drawing attention to the relevance of spreadsheet audit. Other delegates pointed out that this addin is only available in the Professional Plus version of Excel, which excludes most small business licenses. If I was Microsoft I too would probably want to restrict its availability to professional users and get the feedback on bugs before making it more generally available. You can read the paper at

Spreadsheet Auditing in Practice; David Colver, Operis Ltd
David gave a presentation (there is no formal paper available) on the real life world of Private Finance Initiative (PFI) project auditing. Operis offer an opinion backed by a £5M liability insurance so they need to be very sure of what they say. Their scope is far wider than the spreadsheet and includes the contracts and legal documents.

The misuse of spreadsheets in the nuclear fuel industry; Simon Thorne, Cardiff Met. University, UK.
Simon reported on the data falsification incident at BNFL in 1999, where operators copied and pasted spreadsheet data rather than performing the measurements on MOX pellet diameters. The compensation to customers who returned product was over £40M and the CEO was fired by the government. The full report from the HM Nuclear Installations Inspectorate is:

A Quality Checklist for Spreadsheets; Henk Vlootman, Felienne Hermans
Henk and Felienne presented a spreadsheet with a checklist where the user can score the spreadsheet of interest by attributes such as the presence of documentation, structuring, and the evidence of various skills.

Eliminating errors through tables, a new type of worksheet technology; Ted Hawkins, Morphit, UK, Andrew Lemon and Alec Gibson, The Edge Software Consultancy Ltd, UK
This was an eye opener for many – a product that looks like Excel or Access and is in fact à kind of database with numerical computational features at the level of the field, a group of records, or one cell. It comes from work in recording pre-clinical experimental records.

Spatial Modelling Techniques in Microsoft Excel; Stephen Allen, ACBA, UK.
Stephen discussed his strategy for using Excel to solve Sudoku, which led him to reflect on complexity in formulas.

The plenary discussion and debate was quite wide ranging and included contributions recommending the introduction of training in safe ways of using spreadsheets, at the earliest level: school or undergraduate courses in all numerate disciplines such as accounting or engineering.
To discuss these further, join the EuSpRIG forum: send a blank email to eusprig-subscribe at

Posted in Research, Risk | Tagged , , | Leave a comment

Reinhart & Rogoff: The Austerity spreadsheet error

We’ve discussed this on the yahoogroup of the European Spreadsheet Risk Interest Group, of which I’m the current chair. Here’s my summary of The Spreadsheet Error Heard Round the World, with my commentary followed by the background.

1) The authors did not catch the error, and only when they released the spreadsheet did others find it.
Lesson: review by a different pair of eyes can catch errors before they cause loss or embarrassment.
“Peer review” is the gold standard in academic research, and the best self-protection for business users of spreadsheets.
Here’s a story from Edward Krudy of Reuters quoted in the Toronto Star: the person who checked the R&R study also checked their own work:

Student finds glaring spreadsheet errors in study used to justify budget slashing
“I almost didn’t believe my eyes when I saw just the basic spreadsheet error,” said Herndon, 28. “I was like, am I just looking at this wrong? There has to be some other explanation. So I asked my girlfriend, ‘Am I seeing this wrong?’” His girlfriend, Kyla Walters, replied: “I don’t think so, Thomas.”

2) Like most spreadsheet creators, they possibly did not realise at the start how important it would become; in their case, frequently cited in support of austerity and arguably the cause of more grief to the world at large than other problems that have merely cost millions. (See )
Lesson: The more important something is, the more care you have to take with it.

3) Technically, the error could have been spotted by simple tests, such as pressing Ctrl+[ on a formula to show what cells feed into the total.
There are many software tools, such as XLTEST, to point out structural flaws.
There is much guidance on safer spreadsheet construction, books such as “Spreadsheet Check and Control”, “Spreadsheet Safe”, etc.
In the business world, common checks include cross-total balances and reconciliations.

Eusprig papers are published at

Our Annual Conference will be in Greenwich on 4-5 July 2013.

This is a recent blogfire about the unravelling of the Rogoff & Reinhart paper that was influential in austerity policies.

In 2010, economists Carmen Reinhart and Kenneth Rogoff released a paper,
“Growth in a Time of Debt.”
Countries with debt-to-GDP ratios above 90 percent have a slightly
negative average growth rate, in fact.
In a new paper, “Does High Public Debt Consistently Stifle Economic
Growth? A Critique of Reinhart and Rogoff,” Thomas Herndon, Michael Ash,
and Robert Pollin of the University of Massachusetts, Amherst
successfully replicate the results. After trying to replicate the
Reinhart-Rogoff results and failing, they reached out to Reinhart and
Rogoff and they were willing to share their data spreadsheet. This
allowed Herndon et al. to see how how Reinhart and Rogoff’s data was

They find that three main issues stand out. First, Reinhart and Rogoff
selectively exclude years of high debt and average growth. Second, they
use a debatable method to weight the countries. Third, there also
appears to be a coding error that excludes high-debt and average-growth
countries. All three bias in favor of their result, and without them you don’t get their controversial result.

At one point they set cell L51 equal to AVERAGE(L30:L44) when the
correct procedure was AVERAGE(L30:L49). By typing wrong, they
accidentally left Denmark, Canada, Belgium, Austria, and Australia out
of the average. When you fix the Excel error, a -0.1 percent growth rate turns into 0.2 percent growth.

Read more:,0,4073638.story

The reported spreadsheet formula error explains 0.3% of a 2.3% error,
the rest is an argument about selective data, methodology, causation,
correlation, etc. From the look of that table, I doubt if the problem was one of rows added at the bottom. Even if it was, the technical solution from MS is the option “Extend data formats and formulas” which autocorrects formulas at the bottom of columns. Others have checked other averages and found discrepancies which may give rise to suspicion that this was not an unintentional error.

That Reinhart and Rogoff Committed a Spreadsheet Error Completely Misses the Point

After being unable to reproduce R&R’s results with publicly available data, HAP were able to get the spreadsheets (zip) that R&R had used for their calculations.

(POB: the analysis spreadsheet, crucially, is not in that zip)

Posted in Excel/VBA, Research, Risk, Software Testing, Uncategorized | Tagged , , , | Leave a comment