-
Notifications
You must be signed in to change notification settings - Fork 913
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Experiment] Global FxA CTA on www for Fx users #6629
Comments
@alexgibson can you review the implementation ideas in the experiment plan and let me know what questions you have? @sghoseWI can you review this experiment plan and let me know how we can make it stronger? |
I met with Alex, Craig and Mark to discuss. This is almost ready to pull into delivery. |
Some implementation notes based on our discussion yesterday:
|
Based on the above, I think I can get started on this now, and we can sweat the design and copy details when this is up on demo. |
I pushed an WIP version of this to demo, using existing strings and regular button styles. https://bedrock-demo-agibson.oregon-b.moz.works/en-US/ The FxA button text reads "Create a Firefox Account" and the small link reads "Learn More", but we can choose any copy we decide works best. For Protocol pages, the FxA button uses the standard secondary product themed button (blue outline): For Pebbles and Sandstone pages, the FxA button uses the older green outline secondary button, to remain consistent with thoe pages: Open questions:
|
@hoosteeno -- the experiment plan generally looks great to me. Only suggestions I have are adding in a sample size estimate needed in each treatment group and the control group (along with the corresponding desired confidence/error rate levels). Also would be good to add what specific countries/states we are going to run this in. http://www.raosoft.com/samplesize.html ^ We can use this to estimate the sample size needed in each group -- and also get an idea of how long we need to keep the experiment open. I can get these numbers for us -- I just need to know what the overall population for our experiment is. Am I correct in thinking that this is the sum number of users in the specific countries/states we are experimenting in? |
I'm actually curious about what the control group should do here, is there a benefit of having a control group for this experiment? The default experience (which is usually the control) is just to show a regular download button, but this is an entirely different action to signing up for an account. I guess it depends what we want to get out of this experiment. If we want to know that a) this FxA cta is effective and b) downloads don't really suffer overall, then can we know both of these things without a control group? |
Hi @alexgibson, my thinking is that we would want the control to be whatever standard behavior we want to at least continue to meet (i.e. number of downloads or account sign ups) -- after adding the FxA cta and controlling for whatever independent variables we are interested in factoring in. I think we would still need a control group for this experiment so we can have a baseline and statistically analyze if there is a significant change observed in our experiment from that baseline after the feature change is implemented. This should hopefully help us understand if the FxA cta is effective -- while testing to make sure there is no meaningful decline in our important core metrics like number of account sign ups. |
@sghoseWI I agree. Can you go ahead and update the experiment plan with your recommendations for segment size and treatment duration, as well as anything else you think makes it more measurable? @alexgibson and @sghoseWI, regarding instrumentation...
|
Let’s be sure we pass this by Alex Davis, folks on the Accounts team.
|
Hey @alexgibson here's 2¢ from a design point of view. First, the I'd recommend a couple things: Make the font size a little smaller, align it to the right edge of the button, nudge it up a few pixels. Here's what I did to achieve that result Second, on the /firefox pages, its creating a large vertical gap between the main nav and the subnavigation below |
@alexgibson @hoosteeno One more thing to note: On this page, the |
I think the most appropriate solution here is to hide the button and link on this page, as the page manages it's own state for prompting people to sign up. We don't want to compete with that and create a confusing UX. |
Marking this as blocked until we get final experiment params. |
Just chatted with @alexgibson -- confirming here that I think it would be wise for us to leave the experiment running for 1 week. We only need ~17,000 sessions in both the treatment and control group -- which we can reach very quickly given daily traffic numbers. But allowing the experiment to run for a week will allow us to control for hourly/daily differences observed over the course of the experiment days by randomly sampling from more robust session data. 50% traffic to both groups will be more than enough -- but if there is a compelling reason to reduce the experimental group traffic that I'm not aware of, it shouldn't be a problem as we should still have sufficient samples to analyze post-experiment. |
Thanks @sghoseWI - I'm going to mark this experiment as unblocked and we'll move foreward with the parameters defined in #6629 (comment) @hoosteeno if you have any feedback on the above please add it, else we'll move forward with merging this once it has been code reviewed. |
This looks great to me. |
Based on some new information I've received-- requesting that a hold be placed on this card for now (meeting with @hoosteeno to discuss today). For more context:
Some outstanding points to consider:
|
@hoosteeno and I talked through each of the above points and we have a plan forward @alexgibson, here's what we decided as it pertains to experiment parameters:
Let me know of any questions -- otherwise, I think we can safely remove the hold once the experiment parameters are adjusted. I'll update the wiki to reflect these points. |
@sghoseWI @hoosteeno this sounds good to me, thanks. One question: If we only want to target current Firefox users, how do we class "current"? Do we just include the current release version, or are we also interested in people on pre-release version as well? (Beta, Dev Edition, Nightly). I think this is my last question before we can move forward. |
My understanding is we just want to target the current release version. Thanks @alexgibson! |
@alexgibson we can target >= 57, but it looks like you've coded it to target latest release, and that gives us plenty of data, so that works. |
This is blocked on code review. |
Description: Implement a test showing a Fx Account CTA in the Global Navigation to users visiting mozilla.org w. a Firefox browser instead of the current 'Download Firefox' CTA.
manapage
The text was updated successfully, but these errors were encountered: