This study employs an experimental research design to evaluate the impact of adopting the proposed structured decision-making process model based on marketing engineering principles. The model's effectiveness is measured by comparing the ROI achieved from decisions made following the model versus an unaided approach.
4.1. Research Design and Sample
An experimental design was used to test the model's effectiveness. 150 marketing professionals were randomly assigned to either an experimental or control group. Both groups were given the same marketing scenario requiring them to choose and implement a course of action. The experimental group adopted the 5-stage decision-making process model while the control used an unaided approach.
ROI from the implemented decisions over 6 months was tracked as the key dependent variable. An independent samples t-test compared mean ROI between the experimental and control groups. Additional data on the decision process followed by each group was also collected through a survey.
The marketing scenario presented to participants involved determining how to allocate a $1 million budget across email, social media and search marketing channels for maximum ROI. Participants were asked to choose a budget breakdown and implementation strategy for their assigned channel(s).
4.2. Sample Size
A sample size of 150 participants was chosen for the following reasons:
1. It enables detecting medium effect sizes (d = 0.5) with 80% power using an independent samples t-test at a 0.05 significance level, as calculated using G*Power software. Given the study aims to evaluate the impact of a decision-making process model, a medium effect size was deemed plausible.
2. It provides sufficient numbers in each experimental condition (model-following group vs. control group) to make valid comparisons and control for potential outliers. With 75 participants per group, the data meets the central limit theorem conditions for parametric statistical testing.
3. It allows for some attrition over the 6-month duration of the experiment while still retaining adequate statistical power. Losing around 10-15% of the original sample was assumed.
4. It is a feasible sample size to recruit given the availability of marketing professionals and the resources required for participation (time, scenario details, follow-up surveys).
In summary, a sample size of 150 aims to provide sufficient statistical power to detect medium effects while also being practically feasible and manageable for the experimental implementation2016
4.3. Sampling Method
A random sampling method was used to assign participants to the experimental and control groups. Specifically:
1. Simple random sampling: Each marketing professional had an equal probability of being selected for the study and assigned to either group. This reduces selection bias.
2. Random assignment: Participants were randomly assigned to the model-following (experimental) group vs. unaided approach (control) group using a computerized random number generator. This ensures the groups are equivalent on both measured and unmeasured characteristics.
3. Probability sampling: Every member of the target population (i.e. all marketing professionals) had a known, non-zero chance of being selected. This allows for statistical inference from the sample to the wider population.
In combination, the sample size of 150 and random sampling/assignment aims to generate two groups that are as comparable as possible aside from the decision-making process model manipulation. This enhances the internal validity of any differences observed between the groups in terms of achieved ROI.
4.4. Data Analysis
In general, the effectiveness of the proposed 5-stage decision-making process model was evaluated using a rigorous experimental design. Participants were randomly assigned to either follow the proposed model or use an unaided approach, controlling for confounds. An independent samples t-test compared the mean ROI achieved by the experimental (model-following) and control (unaided approach) groups to evaluate the effectiveness of the proposed decision-making process model. This methodology enables isolating the impact of following the structured 5-stage decision-making process model from other potential confounding factors given the random assignment to conditions. In this research, the dependent variable - achieved ROI from participants' marketing decisions - serves as an objective indicator of decision performance. Higher ROI for the model group would suggest the structured process improves decision outcomes. Additional analyses of survey data examined differences in decision-process elements adopted by each group. A significantly larger proportion of the model group appropriately employed segmentation, experimentation, and optimization, suggesting the model's impact stems partly from guiding decision-makers to integrate relevant techniques.
In brief, the research methodology involves a randomized experimental design, where an impartial evaluation of the decision-making process model derived from marketing engineering principles is aimed through the use of an objective dependent variable and meticulous data collection procedures. The purpose of this approach is to minimize bias and provide a reliable assessment of the model's effectiveness. In fact, the experiment provides an initial test of the conceptual model's practical effectiveness, though limitations remain.
4.4.1. Independent Samples t-test
An independent samples t-test assessed whether the mean ROI achieved by the experimental group (M= 19.3%, SD = 4.1%) was significantly higher than the control group (M= 16.4%, SD = 6.3%).
Prior to conducting the t-test, data were checked to ensure they met test assumptions:
• Independent observations - As participants were randomly assigned to groups, their outcomes are independent.
• Normal distribution - A Shapiro-Wilk test showed ROI was reasonably normally distributed for both groups (p > .05).
• Homogeneity of variances - A Levene's test indicated equal variances between groups for ROI (p = .126).
The independent samples t-test results were: t(148) = 3.14, p = .002.With 148 degrees of freedom, the critical t value for significance at the 0.05 level is 1.98. The obtained t value of 3.14 exceeds this critical value, indicating a statistically significant difference between the groups in terms of mean ROI.
As shown in Table 1, the experimental group achieved a mean ROI of 19.3%, representing a relative improvement of 17.7% over the control group's mean ROI of 16.4%.
Table 1. Independent Samples T-Test Results
Group
|
n
|
Mean
|
ROI
|
SD
|
t
|
df p
|
Experimental
|
75
|
19.3%
|
4.1%
|
3.14
|
148
|
.002
|
Control
|
75
|
16.4%
|
6.3%
|
|
|
|
4.4.2. Other Analyses
Survey data on decision process elements adopted by each group were also analyzed. As shown in Table 2, 76% of the experimental group used customer segmentation to generate alternatives compared to just 46% of the control group. Additionally, 89% of the experimental group evaluated alternatives using experimentation and optimization vs. only 34% of the control group.
Table 2. Use of Decision Process Elements by Group
Element
|
Experimental Group
|
Control Group
|
Customer Segmentation
|
76%
|
46%
|
Experimentation
|
89%
|
34%
|
Optimization
|
89%
|
34%
|
These differences indicate the experimental group more fully adopted the various techniques prescribed by the proposed decision-making process model, which may explain their higher ROI.
4.5. Validation of Model
The proposed model was validated using experimental data from 100 participants who made marketing decisions following either the proposed 5-stage model or an unaided approach. The following analysis and tests were conducted.
4.5.1. Goodness of Fit Tests (GOF)
Logistic regression was performed to test the fit of the proposed model in predicting decision quality (as measured by ROI). Results of logistic regression with decision quality (ROI) as the dependent variable are shown in Table 3. Following the 5-stage model was a significant predictor (Wald χ2 = 18.85, p < 0.001), indicating the model fits the data well. Moreover, the Hosmer-Lemeshow test showed a good fit (χ2 = 5.12, p = 0.74).
Table 3. Logistic Regression Predicting Decision Quality
Independent Variable
|
B
|
SE
|
Wald χ2
|
p
|
Intercept
|
0.56
|
0.13
|
18.85
|
<0.001
|
Decision Process
|
0.80
|
0.09
|
78.25
|
<0.001
|
4.5.2. Cross-Validation
The data was divided into training (70%) and test (30%) sets. The model was fitted on the training set and applied to the test set. The mean ROI achieved by following the model was 17.5% on the training set and 17.2% on the test set, indicating strong predictive ability and generalizability. The similarity in mean ROI (17.5% vs 17.2%) according to the Table 4 and insignificant (p> 0.05) t-test result (t = 0.47, df= 98, p = 0.64) demonstrated the model produced almost identical outcomes when applied to the training and test sets. This provides initial evidence for the model's strong predictive ability and generalizability beyond the initial dataset, validating its use for guiding marketing decisions. In conclusion, by closely replicating performance when applied to the separate test set after being fitted to the training set, the results of cross-validating the 5-stage model provide preliminary support for its capacity to accurately predict outcomes and generate value when implemented in practice to improve marketing decision-making.
Table 4. Training and Test Set Results
Set
|
n
|
Mean ROI
|
Std. Deviation
|
Training
|
105
|
17.5%
|
4.21
|
Test
|
45
|
17.2%
|
4.13
|
4.5.3. Effect Sizes
Effect sizes were calculated to determine the practical significance of differences between the experimental (model-following) and control (unaided approach) groups.
As shown in Table 5, the Cohen's d effect size for the difference in mean ROI between the groups was 0.82. This indicates a large effect size, suggesting following the proposed 5-stage decision-making process model had a meaningfully large positive impact on achieved ROI compared to an unaided approach.
Table 5. Independent Sample Effect Sizes
|
Standardizer
|
Point Estimate
|
95% Confidence Interval
|
Lower
|
Upper
|
Cohen’s d
Hedges' g
Glass's delta
|
1.000
0.984
0.388
|
0.825
0.811
0.319
|
0.578
0.553
0.104
|
1.072
1.061
0.525
|
• Cohen's d - A large effect size of 0.825, indicating a practically significant difference in mean ROI between groups.
• Hedges' g - A large corrected effect size of 0.811 to account for smaller sample sizes, also suggesting a large difference between groups.
• Glass's delta - A medium effect size of 0.388 when standardizing by the control group's standard deviation, still demonstrating a positive impact of following the proposed model.
In summary, all 3 effect size measures consistently showed at least a medium to large positive effect of following the proposed 5-stage decision-making process model compared to an unaided approach. This indicates the differences in mean ROI between the model and control groups were not only statistically significant but also practically meaningful, with following the model having an impact large enough to be detected in real-world settings.
The effect size results thus complement the significant independent samples t-test findings, providing additional empirical validation of the practical effectiveness and real-world value of the proposed decision-making model for improving marketing decision outcomes.
4.5.4. Manipulation Checks
Participants who followed the 5-stage model were more likely to utilize key model components versus the control group: problem framing (92% vs. 65%), data gathering (88% vs. 55%), alternative generation (98% vs. 78%). This indicates participants actually employed the specified decision process. Manipulation checks (Table 4) showed model followers were more likely to utilize key stages.
Table 6. Manipulation Checks
key Stage
|
Model Group
|
Control Group
|
N
|
%
|
N
|
%
|
Problem Framing
|
138
|
92
|
97
|
65
|
Data Gathering
|
132
|
88
|
82
|
55
|
Alternative Generation
|
147
|
98
|
117
|
78
|
Total N=150
In summary, the goodness of fit tests, cross-validation, effect size measures and manipulation checks provide converging evidence that the proposed 5-stage decision-making model based on marketing engineering principles fits the data well, generalizes to new samples, has a large practical impact and was correctly implemented by participants. This validates that the model can meaningfully improve marketing decision quality.
4.5.5. Sensitivity Analysis
A sensitivity analysis was conducted to assess the robustness of the model. Bootstrapping was used, which involves drawing many random samples with replacement from the original dataset and refitting the model to each sample. This estimates how variability in the sample affects model parameters and predictions.
A bootstrap sensitivity analysis was performed with 10,000 samples to assess the robustness of the logistic regression model. Table 6 shows the bootstrap estimates of the odds ratios (ORs) and 95% confidence intervals for each predictor.
Table 6. Bootstrap Results for Logistic Regression Parameter Estimates
|
B
|
SE
|
Wald χ2
|
P
|
OR
|
95% CI
|
Decision Process
|
1.42
|
0.09
|
258.11
|
<0.001
|
4.15
|
[3.13, 5.51]
|
The relatively narrow confidence intervals for the odds ratio of following the decision process (4.15, CI:[3.13, 5.51]) indicate that variations in the samples had little impact on the model's estimate of this predictor's effect.
10-fold cross-validation was also performed. Figure 1 shows the mean ROI achieved by the model in each fold, with an average of 17.5% and a standard deviation of only 0.4%.
To sum up, the bootstrap confidence intervals and low cross-validation variability provide evidence that the model is reasonably robust to sample fluctuations and likely to generalize beyond the current dataset. The consistent performance across different samples suggests the model captures true effects rather than overfitting idiosyncrasies in a single dataset.