Last year the International Longevity Centre confirmed what society already knew: the UK is aging. While those aged between 15 and 64 will grow on average by 29,000 every year until 2037, those aged 65 and over will rise by 278,800 a year. This reality is staring healthcare providers in the face. The older we get, the more healthcare services we consume.

Now is the time for healthcare providers to review their IT estates and ensure that their technology matches up to this challenge. This is where robust testing becomes even more important. Any outages, data loss or integration failures could have a material impact on day-to-day operations and would severely impact patient care.

The NHS recently delivered the successful replacement for NHS Spine – the world’s largest public healthcare platform. NHS Spine 2 supports critical NHS business applications, providing interoperability and data sharing across its various healthcare and management systems.

> See also: Research exposes crucial gaps in the NHS's cyber security capabilities

As the technological backbone of the NHS, Spine 2 operates 24x7x365, supporting over 27,000 organisations, including hospitals, GP surgeries and pharmacies. It has over 1.1 million users, handles 1,800 transactions per second and incorporates a national demographic database of more than 80 million patients.

A significant focus on test was key to the successful delivery of this system which today routinely exceeds SLA targets. While the combined test team for NHS Spine 2 had over 75 years of experience working in both public as well as private sector healthcare providers, its unique set of requirements challenged existing test practices and provided valuable learning experiences.

Automation

Automation ensures stability in any large Agile delivery project. To enable small regular deliveries, automation should be applied from deployment to unit, integration and system tests.

While this may add complexity to maintenance, it reduces the perception of bottlenecks, builds early confidence and delivers a stable codebase. The additional effort required to create and maintain an appropriate level of automation must be planned for at the project’s outset.

Spine 2’s delivery, at a high level, included Technical (performance, resiliency, soak, stress and volume), UI, Data Migration, functional, user, bespoke automated messaging (synchronous and asynchronous), and considerable unit testing.

This approach allowed for testing to occur early in the project. It also became more straightforward to adapt to the individual needs of each phase. Largely bespoke automated testing formed part of overnight builds and resulted in around 25,000 integration tests occurring across the project’s lifetime. Page-by-page, Object oriented, defect specific and user journey tests were automated. Mocks simulated Spine 2’s thousands of entry points and users.

As a result, there was very high confidence in the system especially with its ability to accurately replicate the API and behaviour of its predecessor. Automation artefacts were managed with the same rigour applied to the codebase. Artefacts were delivered to the same coding standards, allowing them to benefit from clear annotation and become part of the build.

There are benefits to defining a clear framework for tests, ensuring that items such as error handling and multi-browser capability are considered. The project also reaffirmed some important principles:

• Be clear on what test data will be used, where it will come from and how often it will be updated.
• Agree a language style and enforce consistency.
• Having and managing test technical debt enhances automation.
• Continuously maintain and refactor the codebase – both the framework as well as the tests.

Baseline

The Non Functional team for NHS Spine 2 was in place very early – even before an environment was made available to use – enabling the team to define which types of tests would be required, how they would be monitored, which tools would be used and so on.

From this point onwards, as soon as an environment became available, a baseline for those tests was defined. Time was spent experimenting to discover and understand tests such as the rate or distribution of requests over the different endpoints or URL’s.

A baseline establishes a starting point where there is none. Non-Functional Requirements (NFRs) are often not considered until well into the project. Establish them as early as possible – even if much of the project is a functional ‘like-for-like’ replacement.

Collaborate

In any project, there are often many stakeholders with varying perspectives and requirements. This is especially true in a heterogeneous industry such as healthcare where doctors and nurses, receptionists, support staff and management organisations are all defined as stakeholders. This breadth of interested parties impacts practically all aspects and decisions.

With Spine 2’s exceptional size, the team had to consider its functional capability, performance and resiliency. To achieve this, development and test teams from both the NHS and BJSS worked together in blended teams. The level of collaboration in these teams made it virtually impossible for outsiders to differentiate between the two groups, enabling issues to be identified and dealt with quickly.

Testers were also involved in regular iteration planning sessions. This allowed for detailed capacity planning. Business and Assurance representatives attended stand-ups where progress could be understood, and hidden issues identified.

Collaboration was extended to Subject Matter Experts (SMEs) and users who were identified at the outset of the project and invited to Show and Tell sessions. Small, regular deliveries helped them to understand and contribute to the decisions around design and delivery, and it also helped suppress corporate politics.

Large organisations often use more traditional delivery methods and the prospect of introducing an Agile approach is akin to turning around an oil tanker in a small port. Working closely with blended teams introduces many of the Agile ceremonies, but at a pace that is more comfortable and allows for small course corrections.

Data

Tests need data. It’s common to find lots of defects in edge cases and without rich data they become difficult to locate. Like many large organisations, the NHS generates considerable volumes of data.

NHS Spine 2, for example, handles 70 million patient records and transfers 1,800 messages every second. Data volume is a challenge not only for the richness of scenarios that a large dataset can provide, but for the effects of volume on any platform.

One of the biggest challenges in any testing project is being able to obtain production data. This is particularly sensitive in the Healthcare sector where there are legitimate concerns about the use of Patient Identifiable Data (PID).

There is no easy answer to what is the acceptable balance of production data to test requirements. In many cases, legislation prevents even depersonalised data from being used.

Starting with canned data, a dataset can be built through analysis of data journeys. Done early enough, it becomes possible to build a rich set of transitioned data alongside test cases as the project progresses.

Where automated deployment processes are used, take into account the length of time the data will take to load. Consider what size will the data be at go live, how quickly it will grow and how it will look in several years’ time.

Environments

Access to appropriate environments for testing is vital. While any tester would dream of having a fully scaled, like-for-like system, the reality is almost always different. Instead, an environment that is appropriate for the stage of testing should be sufficient. Defining the purpose of testing in that stage drives the specification of the environment.

Spine 2 combined virtual and production environments. For the automated integration testing the team was able to use virtual machines, automatically deploying the code using Open Source tools such as Puppet.

During the GUI phase, the code was deployed to existing integration environments with endpoints stubbed out. Spine 2 used completely new technologies and therefore allowed the test team to physically use the production environment prior to go live for non-functional testing.

It can become easy to abuse test environments. Old data and test code can pollute the environment to a level that it becomes unusable. Roles such as ‘TestOps’ with responsibility for the testing infrastructure can help ensure that environments are available when required, and are maintained throughout the lifecycle of the project.

> See also: How data can reduce the burden of diabetes treatment on the NHS

An aging UK will add to the demand on the healthcare sector’s IT estates, making it likely that replacement and expansion projects will become increasingly complex over the medium term.

While projects such as NHS Spine 2 are very large, ‘once in a generation’ engagements, they provide valuable lessons and challenge the existing testing status quo.  For example:

Decide from the outset how Non Functional Requirements will be tested. Ensure this occurs across all stages of the project. Collaborate as much as you can. Healthcare is heterogeneous and there are many stakeholders. Invest in building strong but frank and honest relationships.

As challenging as it might be, always invest in good quality production-like data – it’s common to find many defects in edge cases and without the rich data they can be difficult to locate.

Finally, automate wherever possible and wherever it makes sense. If you can’t use a production environment, then set up a virtual environment that is as near to production as possible.

IT is renowned for failing to live up to its hype – however, when solutions are carefully tested by skilled practitioners, implementations can deliver lasting operational benefits for healthcare providers and improve patient care.

Sourced from Cassy Calvert, testing services manager, BJSS