What is the overall evaluation criterion (OEC) in A/B testing and conversion rate optimization (CRO)
Published on Jun 18, 2020
If you can’t measure it, you can’t improve it
—Peter Drucker
The overall evaluation criterion (OEC) is a combination of metrics that reflect a companies goals. The OEC should be able to detect improvements over a short period of time while being a good predictor of long-term goals so we’re able to run experiments using it.
The OEC is not an easy to get right. It can take a long time find right metrics with lots of iterations and adjustments along the way.
For example, Google Search could use ad revenue as one of their metrics. Only using ad revenue is dangerous. It could lead to short-term gains (plastering ads all over the website) at the cost of user experience which over time could stop people visiting the website. Instead, Google Search need to consider the user experience which could be measured by the number of times people use the website, the number of successful searches, and engagement. Even those metrics are tricky though. For example, does increased engagement mean people enjoy using Google Search or are struggling to find the right results?
To design and run a good online controlled experiment, you need metrics that meet certain characteristics. They must be measurable in the short term (experiment duration) and computable, as well as sufficiently sensitive and timely to be useful for experimentation. If you use multiple metrics to measure success for an experiment, ideally you may want to combine them into an Overall Evaluation Criterion (OEC), which is believed to causally impact long-term objectives.
—Ronny Kohavi, Trustworthy Online Control Experiments
