There's a great blog on LinkedIn about a Formula One engineer and the biggest mistake he made while in F1. To cut a long story short, he put totally the wrong fuel load in the car - but it wasn't the disaster they thought it would be (click through below for the details).
It turns out that their models worked really well around the standard fuel loads, but not for fuel loads they didn't usually use. Because of this, he failed to predict that this accidental low load could actually be a good solution. The key takeaway from the blog is how they reacted to the situation - what started out as an accident turned into a valuable learning opportunity.
In mathematical terms this sounds like a local minimum - you change your conditions a little, test, optimise and end up at the same local min, but change your conditions a lot and you might find a much lower minimum over the hump.
I see this a lot in start-ups: They've got a solution that's working well, but they need to improve it. Often start-ups will iterate around the existing solution, making small tweaks to get marginal gains, but sometimes you need to experiment more widely. If you don't test a totally different approach, you'll never know if it could be far more effective. So once in a while throw out the existing playbook, think things through from scratch, run an experiment and see what you learn.
Create a culture that can extract the absolute maximum from these accidents, not one that hides them away and makes sure they don’t happen again.