A/B Testing
Test up to 10 notification variants with control groups, early stopping, statistical significance. Created in Abrova Console; mobile SDKs fetch variants and track events. Auto click tracking on mobile.
Test different push notification variants to optimize engagement and conversion rates. Compare up to 10 variants with statistical significance analysis.
A/B testing, control groups, early stopping and the statistics are included with Abrova Push. Every variant you send counts toward your monthly messages like any other push: 10,000 messages a month are free, and the price above that is set when Abrova Push launches.
Overview#
AbrovaPush A/B testing lets you send different notification variants to different user segments and measure which performs best. Tests are created and run in Abrova Console.
Key Features
- • Up to 10 variants per test
- • Control groups (no message sent)
- • Early stopping when significance reached
- • Statistical confidence analysis
- • Automatic click tracking on mobile
- • Target by segment or percentage
Tracked Metrics
- • Sent — Messages dispatched
- • Delivered — Successfully received
- • Opened — Notification viewed
- • Clicked — User tapped notification
- • Converted — Custom goal achieved
Creating Tests via Dashboard#
Create A/B tests in the AbrovaPush A/B Testing dashboard.
- Create New Test — Click "Create A/B Test" and enter test name and description
- Configure Variants — Add 2-10 variants with different titles, bodies, images, and action buttons. Set traffic percentage for each
- Set Target Audience — Choose "All users", a specific segment, or a percentage of your audience
- Configure Options — Enable control group, early stopping, select winning metric, set test duration
- Start Test — Review and start. Send notifications to begin collecting data
Control Groups#
A control group receives no notification, letting you measure the true impact of your messaging versus no messaging at all.
Example: If your control group has 5% conversion and Variant A has 12%, the notification drove a 7% lift — not just that 12% of users converted.
Early Stopping#
When enabled, the test automatically completes when statistical significance is reached, saving you from running tests longer than necessary.
| Option | Type | Required | Description |
|---|---|---|---|
| Significance Threshold | - | - | Confidence level required (90%, 95%, or 99%) |
| Minimum Sample Size | - | - | Minimum deliveries per variant before checking significance (default 100) |
SDK Integration — Auto Click Tracking#
Mobile SDKs support A/B testing with automatic click tracking. When users tap A/B test notifications, clicks are tracked without additional code.
Android (Kotlin)
// Get variant for a test (with caching)
AbrovaPush.getABTestVariant(
testId = "test_id",
forceRefresh = false,
onSuccess = { variant ->
// Display content based on variant
showBanner(
title = variant.content?.title ?: "Default",
backgroundColor = variant.content?.data?.get("backgroundColor")
)
// Track that variant was displayed
AbrovaPush.trackABTestDisplay(variant)
},
onError = { error -> /* handle */ }
)
// Track click manually (automatic for push notifications)
AbrovaPush.trackABTestClicked(testId, variantId)
// Track conversion
AbrovaPush.trackABTestConverted(testId, variantId)iOS (Swift)
// Get variant for a test
AbrovaPush.shared.getABTestVariant(testId: "test_id") { result in
switch result {
case .success(let variant):
showBanner(
title: variant.content?.title ?? "Default",
color: variant.content?.data?["backgroundColor"]?.value as? String
)
AbrovaPush.shared.trackABTestDisplay(variant: variant)
case .failure(let error):
// handle
}
}
// Track click manually (automatic for push notifications)
AbrovaPush.shared.trackABTestClicked(testId: testId, variantId: variantId)
// Note: the iOS SDK has no conversion-tracking method yet
// Handle notification tap (automatic click tracking)
func userNotificationCenter(_ center: UNUserNotificationCenter,
didReceive response: UNNotificationResponse,
withCompletionHandler completionHandler: @escaping () -> Void) {
// This automatically tracks A/B test clicks
AbrovaPush.shared.handleNotificationResponse(
userInfo: response.notification.request.content.userInfo,
actionIdentifier: response.actionIdentifier
)
completionHandler()
}Flutter
final variant = await AbrovaPush.getABTestVariant('test_id');
if (variant != null) {
showBanner(
title: variant.content?.title ?? 'Default',
color: variant.content?.data?['backgroundColor'],
);
await AbrovaPush.trackABTestDisplay(variant);
}
await AbrovaPush.trackABTestClicked(testId, variantId);
// Note: the Flutter SDK has no conversion-tracking method yetReact Native
const variant = await AbrovaPush.getABTestVariant('test_id');
if (variant) {
showBanner({
title: variant.content?.title ?? 'Default',
color: variant.content?.data?.backgroundColor,
});
await AbrovaPush.trackABTestDisplay(variant);
}
await AbrovaPush.trackABTestClicked(testId, variantId);
// Note: trackABTestConverted is declared in the React Native package but its
// native method is missing, so calling it fails. Conversions are Android-only today.Automatic Click Tracking#
When you send A/B test notifications from the dashboard, the SDK automatically tracks clicks when users tap on the notification or action buttons. No additional code required.
How it works: A/B test notifications include abTestId and variantId in the payload. When the user taps the notification, the SDK extracts these values and sends a click event to the server automatically.
Winner Metrics#
| Option | Type | Required | Description |
|---|---|---|---|
| delivery_rate | delivered / sent | - | Testing delivery optimization |
| open_rate | opened / delivered | - | Testing notification appeal |
| click_rate | clicked / delivered | - | Testing engagement (recommended) |
| conversion_rate | converted / sent | - | Testing business outcomes |
Best Practices#
Change only one element (title, body, or image) per test to understand what drives the improvement
Wait for at least 100-1,000 sends per variant before drawing conclusions
10-20% control group helps measure true lift vs. no messaging at all
Save time by automatically completing tests when statistical significance is reached
95%+ confidence before selecting a winner. The dashboard shows real-time confidence levels