Recently I attempted to define in as objective a way as possible how much value my online calculator is providing to its users. The idea is to come up with an equation which gives me a ’score’ of how much value has been provided to users. With this, it will then be possible to optimise the site to provide as much value as possible. For instance, when considering two possible improvements I can use the equation to estimate the likely increase in value. That then lets me know which one to go ahead with (the one likely to be of most value to the users).

At the end of this post are the notes I made while trying to figure it out. They’re probably not of any use to anyone, but I though I’d share them. Before that, though, I want to outline the process I went though as hopefully this will be a bit more useful.

Step 1: Identify the main source of value you provide

Step 3: Identify time costs

The  other three also incur the time cost involved in typing in the calculation. This is then followed by the time cost in waiting for a response. Hopefully all these costs should be minimal, but it would be wrong to ignore them.

By analysing web logs, etc, it should be possible to get exact values for this time cost in most cases. The only thing that can’t be exactly pinpointed is the time it takes for the user to find the site as this may include, e.g. searching on a different site such as Google.

Step 4: Identify the user-derived benefit and costs

Assuming the user takes the action required to produce the value, three things can happen. Either the user gets the value they were after, they don’t get the value, or they get a wrong answer which may actually cost them (in that they may act on incorrect information).

In the case that the right answer is provided, we know that the user has received some value. While we don’t know how much there is one assumption we can make – it should be greater than the cost the time cost the user expected to incur. Otherwise the user wouldn’t have take the action (assuming rationality). We can figure out the expected time cost in a way similar to the actual time cost calculated in step 3. This just involves estimated averages across all sites (such as average site response time) rather than using data specific to my site. I also made an assumption that a user would expect a return of at least 10% greater than the time invested to make it worthwhile.

In the case that the calculator returns no answer (where I haven’t yet implemented the calculation asked), then the user gets no benefit, but also no real cost other than the time invested.

The final case is that the user gets given the wrong answer. This could potentially be have a large cost to the user if they act on the incorrect information. While hard to quantify, one assumption I felt confident making is that a rational user would cap their potential losses in relation to their potential gain. For this I assumed that the user would cap losses at, at most, 1o times their potential gain.

Step 5: Write out value expressions for each outcome

Step 6: Derive a lower-bound on each expression solely in terms of the gain

We know alpha is the actual time cost the user incurs. This is related to a lower-bound on the gain, G, by the user’s expected time cost (call it alpha_e). With a bit of work (see the notes below for details), we can rewrite the value expressions so that they are expressed in terms of the lower-bound on G. So essentially, we have value expressions written solely in terms of (a lower-bound on) G.

Without getting into to much detail, we can basically total up the value provided to a user when they perform a set of calculations. This will give us a value score for that set in terms of G. While we don’t know what G actually is, it doesn’t matter – as we will be using the value calculation simply to compare one set of calculations to another (separated in time, by user, by calculator version, etc.) we can just drop G entirely.

Conclusion

Now we have an expression for the lower-bound on the value provided by any set of calculations and resulting outcomes. While not perfect (an exact score would be better than a lower-bound), I think it might be the best that can be done without figuring out what user’s are actually doing with their calculation results (and I think that’s their own business). It will still be very useful to be able to compare the lower-bounds as these are likely to be reasonably well correlated with the actual values. Also as the aim is to increase value over time we can be fairly confident that this is occurring if we can see that the lower-bound on the value is increasing over time.

My Rambling Notes (or Stop Here and Don’t Read On)

For the time being I’m restricting the value calculation to the core calculation service. There are four basic outcomes which can occur when someone interacts with main calculator page:

If it is possible to assign a reasonable measure of user value to each item, then it will be possible to combine them into a reasonable total measure of the value provided by the calculator in a given time period, per user, etc.

In general, there will be a fixed time cost involved in performing any calculation (regardless of the result). For the time being, call this alpha. The cost when no calculation is performed will be somewhat less than this. At a rough estimate it will be alpha / 2.

Should a calculation be performed correctly there will be some gain G for the user.

If a calculation is not answered, then this gain G will not be realised. However, should the calculation be incorrectly answered, not only will the gain not be realised, but the user may be significantly affected (e.g. by entering into a bad financial transaction based on the incorrect result). While we can’t say exactly what magnitude of loss this might be, it seems reasonable to expect that any rational user would cap their potential losses at 10 times the possible gain. At the very least, at that risks beyond that level, they could be expected to verify the calculation results with other sources. So the maximum loss can, I think, be taken to be -10*G.

Combining these we can get value expressions for each of the possible outcomes:

This will be more useful if we can express it in a single quantity. To do so we need to look at the relationship between alpha and G.

There is a quantity related to alpha which will be useful to us. This is the time cost the user expects to pay, assuming a reasonably responsive system. Depending on the performance of our system, alpha may be greater or smaller than this. We’ll call this quantity alpha_e.

A rational user would not carry out actions which did not at least repay the expected initial investment plus a bit of ‘profit’ to make it worthwhile and absorb associated risks. Thus we can expect G to have a lower-bound of perhaps 1.1 * alpha_e.

That’s a bit better, but still not quite there. We can get the rest of the way by looking at what exactly alpha and alpha_e are. These, as noted above, are the time cost of performing a calculation and the expected time cost of performing one on an averagely responsive system respectively. They have the same formula:

time_to_input_calc + time_to_perform_calc + time_to_reach_calculator / num_calcs_per_user

The only difference is that alpha_e represents the user’s expectation of these values while alpha represents the actual values.

We can assign approximate values to these for alpha_e. Something like this, perhaps:

alpha_e = time_to_input_calc_e + time_to_perform_calc_e + time_to_reach_calculator_e / num_calcs_per_user_e

= 10 + 3 + 10 / 5

= 15 seconds

This gives a lower-bound on G of 16.5 seconds.

Now, we can perform a similar calculation for alpha. I don’t know exactly, but the result will probably be something similar. Assuming for the time being that it is, we can then say that G >= 1.1 * alpha. We could easily adjust the multiplicative factor should analysis of the data show that alpha and alpha_e show that they differ.

But is relating alpha to a lower-bound on G any use? It’s not precise, but it gives a possible approach.

Totalling up the value for a set of calculations (based on the expressions above) we will get something in terms of G and alpha, e.g.

value = a * G – b * alpha

If we can get an upper-bound on b * alpha in terms of G, then we’ll get a lower-bound on the value in terms of G:

Setting G = 1.1 * alpha, we get our upper-bound on b * alpha:

b * alpha < (b / 1.1) * G

This gives us a lower-bound on the value provided as:

value >= (a – (b / 1.1)) * G

So that is our lower-bound on the value provided to the user by a set of calculations. The a and b variables can easily be calculated by totalling the values of G and alpha for each calculation in the set. The 1.1 factor can be assumed constant (or calculated from the actual data observed). And G is essentially irrelevant to us. As all value calculations will contain a multiple of G, we can simply drop it – it contains no useful information.

So we have, pulling everything together, finally:

value >= (a – (b / c))

where

a = answered_correctly – 10 * answered_incorrectly

b = 0.5 * no_calculation + answered_correctly + answered_incorrectly + not_answered

c = 1.1 * 15 / (time_to_input_calc + time_to_perform_calc + time_to_reach_calculator / num_calcs_per_user)

While not an exact measure, (which wouldn’t really be feasible), it does give us a lower-bound on the value provided (given the assumptions made above). This is almost as useful as we can now operate with this as our measure in value and strive for a constantly increasing lower-bound on the value provided. While the actual value may be somewhere higher than this, improvements in the lower-bound will almost certainly result in the actual value provided being increased (if not perfectly correlated, this should at least happen on average over time).