The Lean Startup
Many metrics companies gather are meaningless. They are too broad to produce insights that guide the direction of the product. These vanity metrics only produce a cloudy picture of the effects caused by any number of efforts.
Good metrics aren’t produced from looking at the general data set with ill defined criteria. They are produced by clearly splitting up the users into cohorts and applying precise criteria of measurement.
The users can be split up in many ways: by registration status, the number of visits, or specific type of interaction. The split allows you to measure exactly which segment of users is impacted by the new feature or marketing campaign.
The metrics themselves need to be easy to understand. Click through rate can be interpreted in to many ways to be meaningful. The number of new users, on the other hand, is a clear metric that everyone can understand. Using precise measurements allows you to only keep the features that have the desired effect.
In order to make these metrics useful, you can’t release multiple features at once and expect to know which one caused the effect. People are notoriously bad and gauging effect accurately. You will need to work in smaller batches taking the features all the way through an analysis phase before calling them complete. Only then will you be able to understand the full effect of your efforts.