Reversing Significance Test formula

[Stroop]

Hello

I have a spreadsheet which calculates a P value for a/b conversion tests which are conducted on the website.

I wish to rearrange the formula to get the minimum increase required to get a significant result based on known volumes.

Essentially that means rearranging

(Y-X)/SQRT( ((Y(1-Y))/B) + ((X(1-X))/A)) ) = 1.67

To make y the subject. Here A and B are the number of visitors to the site in the control and test group and X and Y are the number of bookings for the same pots.

I'm getting stuck doing this myself as it gets quite messy once all the y's on one side.

 

[tkhunny]

Hello

I have a spreadsheet which calculates a P value for a/b conversion tests which are conducted on the website.

I wish to rearrange the formula to get the minimum increase required to get a significant result based on known volumes.

Essentially that means rearranging

(Y-X)/SQRT( ((Y(1-Y))/B) + ((X(1-X))/A)) ) = 1.67

To make y the subject. Here A and B are the number of visitors to the site in the control and test group and X and Y are the number of bookings for the same pots.

I'm getting stuck doing this myself as it gets quite messy once all the y's on one side.

"quite messy" - Well, what will you be doing about it? Wade through it. If you're trying to do some meaningful statistics, the algebra should not be your impediment. Give it a go. Let's see how far you get. I wonder if you will get a unique result? Hmmm...

 

[JeffM]

I for one do not understand even the starting formula. Unless y and x are less than 1, this formula does not give answers in the real number system. And if x and y are independent, how will solving for y specify anything?

 

[Stroop]

I for one do not understand even the starting formula. Unless y and x are less than 1, this formula does not give answers in the real number system. And if x and y are independent, how will solving for y specify anything?

The formula is standard for significance testing. Although I mistyped in my initial post. X and Y are the conversion rates not the number of conversions. Doesn't affect the formula though.

Essentially the significance test calculates a t value from the actual difference in conversion rates and the standard error. If this t value exceeds a predetermined value (1.67 in this case) then we seem the test to be successful.

Difference Between Proportions

This lesson describes sampling distribution for the difference between sample proportions. Shows how to compute standard error. Includes problem with solution.

stattrek.com

I'm wanting to reverse this though and to say what's the smallest increase in conversion rate required to reach significance.

 

[Stroop]

So yes x and Y are both less than 1