Troubleshooting SRM errors with experiments

Affected: All SDKs

Symptoms

The following health error can occur after starting an experiment:

Sample mismatch ratio detected
These results are not valid.

This means LaunchDarkly has detected a mismatch between the audience allocation in your experiment design and the observed exposures in your experiment.

Cause

There are many ways an SRM can occur, but the most common is that the randomization method is incorrect.

If the experiment isn’t assigning users randomly and evenly to each variation (for example, due to an implementation mistake in how the flag is evaluated, how the context is set up, or how the SDK is used), you’ll end up with a skew in variation distribution. Use the experiment health check panel on the results page for real-time feedback on setup and exposure data.

Solution

Check against the list to see if any of the following issues apply:

• If using a JavaScript-based SDK, read Understanding sample ratios to ensure your SDK version will not run into the issues described.
• Make sure to evaluate the experiment flag as close as possible to when the end user encounters the experiment in your application. In most SDKs, this is accomplished by using the SDK's variation call. For more information, read Evaluating flags.
• Add a flush call after evaluating the experiment flag to ensure the feature event is sent to LaunchDarkly and not lost. This is particularly necessary if there's a page redirect that happens when the variation and/or track event is invoked. For more information, read Flushing events. 
• When calling identify, make sure the identify call has completed before performing any flag evaluations. For more information, read Identifying and changing contexts.
• If using the Subscribing to flag changes feature, be aware that the change listener will only fire a feature event if a change in the flag's value is detected. This means that if a context's flag value doesn't change from when the SDK is initialized to when the context is identified, no feature event will be sent to LaunchDarkly, and the context will not be entered into the experiment. The solution for this is to use thevariation method instead as described earlier.
• If using a shared anonymous context key (for example, a generic key like key:"anonymous" for many users), update your implementation to use a unique key per context. Shared keys cause multiple users to be treated as a single context, skewing variation exposure.

If the above checklist does not resolve your issue, submit a support ticket and provide the following information to help expedite the troubleshooting process:

• SDK and SDK version
• Code snippets of the following (if applicable):
  • SDK configuration and initialization
  • How the application evaluates the experiment flag and the surrounding conditions upon which the evaluation is done
  • identifycall
• Enable Enhanced Support