Challenge: Handling massive traffic spikes

Like most e-commerce retailers, Foreo needs to massively scale applications and infrastructure to handle mission-critical events. Black Friday, Cyber Monday, Singles Day in China, and other events drive record sales and revenue for the company. Millions of dollars are at stake when e-commerce systems slow down, or worse - go down, during major sales events, having a material impact on a company's bottom line.

On Black Friday, November 25, 2016, the Foreo.com website began crashing under heavy load at the peak of Black Friday sales, on the cusp of the most lucrative online shopping period of the year. Having a mere 48 hours to resolve the performance issues they were facing before Cyber Monday, Foreo was at a loss as to prevent a public relations nightmare and the potential evaporation of millions in revenue from holiday shoppers faced with prohibitively lengthy load times.

After the emergency situation was addressed, Foreo needed to position themselves for long term success and ensure their sites remained online and highly performant under extreme loads in the future.

Solution: Triage for the short term and prevention for the long term

Within hours of receiving the request for emergency help, Tag1 deployed a team of infrastructure and application experts to begin reviewing performance issues on Foreo.com. Working together non-stop over the next 48 hours, Tag1 collaborated closely with the Foreo team to quickly address the emergency situation, diagnosing and remediating all problems to ensure a successful Cyber Monday that would bring even more traffic.

Performance tuning is a methodical process of triage and iteration. You find the biggest problem that you can quickly solve - the biggest bang for your buck - and fix it. Changing too much at once can cause more problems than it solves. Then you need to test the results of your work to confirm it had the impact you expected before you identify the next biggest problem (which may have changed from your previous assessment based on the change you just made). Knowing where to look and how to work across the entire stack (infrastructure, caching, databases, code, etc.) takes years and years of experience, and executing quickly under the stress and pressure of money being lost by the minute, on less and less sleep, while keeping a cool head is something few can do well.

After the emergency was resolved the teams worked together to ensure future success by auditing and improving application and infrastructure architecture, enhancing development processes & workflows, and implementing advanced DevOps change control that incorporated automated code reviews as well as load tests around each release. Code bases and websites are always under development, and Foreo needs to ensure their site is always ready for a spike in traffic, and that any update or change doesn’t impact performance.

In addition, risk mitigation strategies were established to minimize the impact of any problems that might surface.

• Outside of the automated per-release tests, even more aggressive and in depth load testing and tuning was scheduled leading up to the biggest sales events.
• Readiness fire drills and chaos testing was performed to ensure the teams responsiveness to a variety of problems under pressure.
• Advanced real-time monitoring and alerting systems were put in place to catch small problems before they become revenue impacting
• 24x7 on-call support teams were established around critical events, with staff ready to go at a moments notice to get ahead of any problem that might arise.

With Tag1’s guidance, and a proactive approach to performance, Foreo hasn't had problems since. Their infrastructure and web applications can handle massive traffic spikes, at any time, without any impact, ensuring maximum conversion rates on sales at all times.

Performance Tuning in Practice

In this video presentation, Tag1 Consulting covers the tools, techniques, and processes used to take a proactive approach, readying applications and infrastructure for massive traffic spikes and surges in payment transactions.

We provide actionable insight on how to load test and performance tune systems through real world examples based on work we’ve done together with Foreo. We will also cover best practices to ensure your success during high-traffic events including advanced monitoring and risk mitigation strategies. And to help ensure your applications remain performant as they evolve, we discuss code reviews and performance testing as part of continuous integration systems, as well as other approaches.