Controlling when participants are added with vanity a/b test <p>Using <a href="https://github.com/assaf/vanity" rel="nofollow">vanity</a> to conduct an A/B test. The test is reliant on a modal that is conditionally shown to certain users. The modal contains different incentives based on which side of the test you see.</p> <p>The code for the modal is in a partial that gets loaded on several pages in our app. </p> <p>Example: </p> <pre><code>&lt;%= render partial: 'modals/example' %&gt; </code></pre> <p>That partial contains a condition for the content that will be shown:</p> <pre><code>&lt;% if ab_test(:whatever) == "something" %&gt; &lt;p&gt;Stuff to show&lt;/p&gt; &lt;% else %&gt; &lt;p&gt;Different stuff to show&lt;/p&gt; &lt;% end %&gt; </code></pre> <p>I'm tracking the relevant metric in the relevant place.</p> <p>The test itself works as expected in terms of showing the content and converting. The problem I'm running into is that because the partial is loaded even when the modal isn't show...everyone that lands on a page that references this partial is being added as a "participant" and I would like to only count people who have actually seen the modal as participants.</p> <p>We use mixpanel as well and I can pivot into running the test with that. I like vanity, though, and would prefer to use it if there's a reasonable solution that isn't going to significantly increase the time it takes to implement. Was hoping someone SO could point out what I'm missing and how to approach solving.</p>