Close collaboration with Biglight has allowed Eurostar to increase the number of A/B tests it runs each month by 33%, whilst enhancing the quality of the ideas it tests - helping to increase the pace of development, speed up the delivery of an enhanced customer experience and increase conversion rates by 7%.
A/B testing is central to Eurostar’s iterative design approach, which represents a departure from previous design efforts that, some years ago, were put in place via a single ‘big bang’ cutover.
“The idea this time was to get A/B testing involved as early as possible,” explained Joe Turner, Digital Optimisation Manager at Eurostar. “In order to support our overall deployment schedule we needed to be in a position to run around four tests per month.”
With that in mind, Joe and his team worked with Monetate, which provides Eurostar’s core A/B testing tool, as well as an outsourced development partner in the US to develop some of its tests.
“There are some tests we have to do internally - big content releases and anything affecting the booking pathway for instance,” Joe said. “But other developments, or tweaks to add urgency in the funnel or to test responsive iterations of pages that sit outside the booking pathway, are important too. We needed external resource to work alongside us, to give us the capacity to get both jobs done at the same time.”
It soon became clear, however, that Eurostar was struggling to make the most of the testing capabilities available via Monetate. The team was busy and it was taking up to three weeks to turn around each test idea.
“The agility we needed was not there; we needed to significantly up the pace,” Joe explained. “We could only test if we planned a month or more in advance - a lead time we didn’t always get from our development team -and that was slowing down our roadmap.”
In response, Joe and his team started looking for external support. They embarked on an RFP process that eventually saw them appoint Biglight as Eurostar’s Customer Experience Optimisation partner, with Monetate the core A/B testing tool.
Reflecting on the selection process, Joe said: “When we went out to RFP, the first questions we wanted to ask was ‘do you work with Monetate?’ and ‘do you have a range of skills available in the agency outside of development?’ That was because, while our original request was ‘we want someone to build our tests for us’, we also wanted the flexibility to have design, UX and research skills as well.”
Initially Biglight stood out as the only agency to offer test design input as part of the RFP process: “For cost comparison, we asked all the agencies to price up tests we had already run. Biglight was the only one to come back with additional ideas as well as the costs we asked for - that external perspective and expertise has proven incredibly valuable for us.”
Crucially, Biglight could also give Joe’s team flexible access to the wider range of skills they were seeking. Joe explained: “Biglight’s mix of developer, research, UX, design and analysis skills is pretty much the complete picture for us, and the contract we’ve got gives us access to any of those skills.
“It’s completely flexible so, if we’re low on resources here because we’re releasing a big update, we can call on Biglight to support us.”
Working closely with Biglight has helped transform Eurostar’s A/B testing capability. It has boosted the number of tests Joe’s team is able to run by around 33%, giving it the agility required to support more rapid development, and giving it access to different thinking around test design.
“Now we are easily able to run the four or five tests a month we need to support the overall pace of development,” Joe confirmed. “The days of planning tests out a month or more in advance are gone.”
That has allowed Joe’s team to focus on the tests that must be done in house, and rely on Biglight to look after the rest.
“The big difference has been the close collaboration,” Joe said. “We’re working concurrently on a range of tests, which means we have more time and resource to invest in releasing the big developments, without neglecting the developments that may be smaller on their own, but can add up to significant change for users.”
On top of that, Joe has been pleased with Biglight’s UX and design input. “When we ask for input we get ideas and suggestions we would not have thought of ourselves.
“For example, we asked for ideas for urgency messaging in our booking funnel and Biglight came back with some really interesting stuff, which we are now going to test. That’s really useful because while they are coming up with those ideas, we can be doing other things - it’s a really efficient way for us to work.”
The net result of Eurostar’s increased testing capacity and greater agility has been an acceleration in designing, developing and deploying responsive experiences. Although iterative design and improvement will continue indefinitely, the move across to a fully responsive site is now around 80% complete.
Eurostar’s focus on CRO and A/B testing has proven a wise strategy too: Overall, conversion rates are up by 7%.
“From a customer perspective, more agile testing means we’re able to deliver more changes, moving away from the old site more quickly,” Joe concluded. “The proof of the overall improvements in the customer experience is in the conversion rate improvements, which are really positive so far.”
“Biglight’s mix of developer, research, UX, design and analysis skills is pretty much the complete picture for us, and the contract we’ve got gives us access to any of those skills." Joe Turner - Digital Customer Experience & Development lead, Eurostar