Automations
      Back to home
      1. Help Center
      2. Blueprint
      3. Automations

      A/B testing Automations

      A/B testing (also known as split testing) is essentially an experiment where two or more variants of a process are shown to users at random, and statistical analysis is used to determine which variation performs better for a given conversion goal. In this case, A/B testing can be used to compare two (or more) Automations against each other to determine which one performs better. 

      How to Setup

      1. Head to Automations and either create a new Automation or select a current automation you'd like to edit.
      2. Open the Advance Settings and select 'Contact Filter'.
      3. Select 'A/B test membership'.



      4. Set the Group filters accordingly, choosing the amount of groups that a test needs to be split up into.
      5. In the example below, one automation is tested against another by setting 'Group 1' and '2 Groups'. 'Group 1' represents the target group and '2 Groups' indicates the amount of groups there are. Therefore this is showing that this Automation is set to send messages to 'Group 1' out of a possible 2 groups. 


      6. Create another Automation (or edit a current one) and set the A/B testing to 'Group 2' and '2 Groups'. This then represents the second target group out of the 2 groups. This will result in 50% of the automations being sent to Group 1 and the remaining 50% of the automations being sent to group 2. 

      How Does it Work?

      • A/B test groups prevent people entering a flow if they otherwise would have (e.g if they are NOT a member of an a/b test group).
      • The groups work by dividing the population into N groups and only allowing them into the flow if they are in a specific group of those. E.g I can divide into 3 groups, and group 1 is (roughly) 33%, group 2 is 33% and group 3 is 33%.
      • The group memberships are stationary across all flows. This is important as it allows you to target different groups of the 'same N' in different flows.
      • An example of where this might be useful is in testing the wording of different automations. You might for example, change the wording of the Post Order Check-in message and run an A/B test to see which copy resonates better with the customer.