BlogEngineering

It's 5:47 on a Friday. Your release just broke signup.

How a perfectly good release breaks a signup page without a single error, why nobody notices for four days, the four kinds of change that cause it, and what the same weekend looks like when you know in minutes.

This is a story about a weekend. It is not about any particular company, but every part of it happens on real websites, to good teams, all the time. If you have shipped software for long enough, parts of it will feel uncomfortably familiar.

Friday, 5:47 pm

The release goes out. It's a good one: a new announcement bar along the top, a tidier header, a handful of small fixes. Code review was happy. Every test passed. Someone opened it on their laptop and their phone, and it looked great.

Everyone logs off for the weekend.

Friday, 6:12 pm

Someone on an iPhone lands on the signup page. On their slightly smaller screen, the new announcement bar is taller than anyone expected, and it is sitting directly on top of the "Create account" button.

They tap it. Nothing.

They tap again. Scroll up, scroll down, try once more.

Then they close the tab.

Saturday

A few hundred more people do exactly the same thing.

The site is up, so the uptime monitor is happy. Nothing crashed, so error tracking has nothing to report. Pages load in a couple of hundred milliseconds. Every monitor the team owns is green.

Nobody writes in. Why would they? There are a dozen other products they can try instead.

Sunday

The ads are still running, so the company is paying for every one of those visitors to arrive at a button they cannot press.

Monday, 10:30 am

Weekly metrics. Signups are down 30%.

Nobody can explain it. Marketing pulls up the campaigns. Product walks through the funnel. Someone wonders whether it's the pricing change from two weeks ago, and someone else blames a competitor's launch.

Everybody opens the signup page on their own laptop. It works perfectly.

Tuesday, 11:05 am

A designer picks up an older iPhone for something completely unrelated, opens the site, and there it is. The announcement bar is covering the button.

The fix takes eleven minutes. The weekend took four days of signups.

The same weekend, with Spectra

Friday, 6:12 pm. The first iPhone visitor taps "Create account" and nothing happens. Spectra sees it immediately: the button is covered on that screen, and the visitor's taps are landing on the bar instead.

Friday, 6:40 pm. It has now happened on enough visits to be certain it is not a fluke. Spectra raises it: "Create account" on the signup page cannot be used on iPhone-sized screens, starting with the 5:47 release.

Friday, 6:45 pm. The developer on call opens the alert and watches a real visitor tap the button three times and leave. There is nothing to debate. It is right there on the screen.

Friday, 7:05 pm. Fixed and shipped.

How long signup stayed broken on iPhones
Found by the team on Tuesday 89 hours
Flagged by Spectra on Friday evening 78 minutes

From the 5:47 pm release to the fix: Tuesday 11:05 am without Spectra, Friday 7:05 pm with it.

Show the numbers
Hours
Found by the team on Tuesday89 hours
Flagged by Spectra on Friday evening78 minutes

Eighty-nine hours against seventy-eight minutes. The same fault, the same fix, the same eleven minutes of work. The only difference is when somebody found out.

Why your release process can't catch this

It is tempting to read this story and conclude that the team should have tested more. They did test. The problem is that testing, however thorough, can only ever cover the devices you test on.

Your customers visit on thousands of combinations of phone, tablet, laptop, browser, screen size, zoom level and language. A fault that only appears at one width on one family of devices will sail through any reasonable test plan, because no reasonable test plan includes that exact combination.

Automated tests have the same limit, and one more besides: they check what you thought to check. Nobody writes a test that asserts "the announcement bar we are adding today will not cover the signup button on a small iPhone", because nobody imagines it until it has happened.

This is not a failure of discipline. It is a matter of scale. The only complete test environment for your interface is your real traffic.

The four kinds of release that break interfaces

Across everything we see, a small number of change types cause the vast majority of broken interfaces. Every one of them looks harmless in review.

1. Something new on the page

A banner, a promotion, a cookie notice, a chat widget, a feedback tab. New elements arrive on top of a layout that was never designed for them, and on some screen sizes they land on something important.

2. Content that changes length

A longer headline, a new menu item, a translated label, a price with more digits. The layout reflows, and a control that sat comfortably in one place is now pushed under, behind or off the edge of something else.

3. A third party that changes without asking

Review badges, chat tools, analytics consent banners and embedded widgets update themselves on their own schedule. Your release did not change, but your page did, and nobody on your team approved it.

4. A style that stops loading

A renamed file, a cache that serves an old version, a font that fails to arrive. The page still loads, but it no longer looks or behaves the way it was built to, and whatever held the layout together is missing.

What release intelligence means

The most expensive question in any incident is "what changed?". Without an answer, a team guesses: the pricing change, the competitor, the campaign, the season. Each guess costs a meeting and a day.

Release intelligence answers that question before anyone has to ask it. When a fault appears, Spectra ties it to the release that introduced it, on the devices where it appears, and refuses to point at an update until it is 99% sure. Instead of "signups are down and nobody knows why", the team hears "the 5:47 release broke Create account on iPhone-sized screens", with the visit attached.

That one sentence is the difference between a four-day mystery and a one-hour fix.

A runbook for the next broken release

Whether or not you use Spectra, these habits shorten the next incident dramatically:

  1. Name every release. A deploy without a version is a fault you cannot trace.
  2. Watch the first hour after every release closely. Most interface faults show up within minutes on real traffic, long before they show up in weekly metrics.
  3. Look at your key pages on at least one small phone, one tablet and one 1280-pixel laptop window before you log off on a Friday.
  4. Treat a sudden drop in one device's conversion as an incident, not a trend. Faults are often confined to one kind of screen.
  5. Make it trivially easy to roll back. The fastest fix for a broken interface is often to undo the change that broke it, then fix it properly on Monday.

Where Spectra comes in

Spectra is interface intelligence for the web, and it turns that four-day nightmare into a one-hour blip.

It watches over every real visit to your site, on every phone, tablet and laptop your customers own, and knows the instant a button stops responding. When a release breaks something, Spectra tells you exactly which button, on which devices, since which release, and puts the visit where it happened right in front of you. No meetings, no theories, no "works on my machine". Just the answer.

It is the safety net every team that ships often should have had years ago.

Ship on Friday. Sleep on Friday night.

See what Spectra sees on your site

Spectra

See every visit through your customer's eyes

Interface intelligence for the web. Live on your site in minutes, watching every visit from the first one.