Study design and setting
This is a cross sectional study embedded in a parent study which implemented and evaluated training and supervision of health talk delivery on maternal depression. The parent study was carried out among primary health care workers in Ibadan, Nigeria. This study was set to validate educational materials for the parent study.
This validation study took place in two of five local government areas (LGAS) in Ibadan metropolis, where the experimental arm of the parent study was conducted. In the experimental arm, primary healthcare (PHCs) implemented the parent study while primary health clinics which were a level lower than the PHCs were used for the validation of materials. PHCs offer more comprehensive services including laboratory testing, 24-hours service, minor surgery and admission in addition to the out-patient services to a larger population of approximately 20,000 people while the primary health clinic serve up to 5,000 people, only offering outpatient services and are a referral services for mobile health posts.
The materials for validation
We developed three education materials (poster, leaflet and a song) on maternal depression which are shown in figures 1-3. The maternal child health clients of fifth to seventh grade pretested and evaluated the materials in terms of understanding, clarity and containing satisfactory information. The content of these materials was built on the concept of health belief model (25) with a clear definition of maternal depression, risk factors which addresses perceived susceptibility and consequences of maternal depression. These materials were developed in Nigerians’ three common languages; Yoruba, Pidgin and English. Our study predominantly utilized the Yoruba version because it is the language of the end users (the clients). The content validation of these materials was guided by “The best practice for developing and validating scales for health, social and behavior research (2).
Experts who could make professional contribution to the content, context and concept of the materials validated the suitability of their contents. To be included as an expert, one had to be from one of the following professional backgrounds: mental health, maternal-child health, public health nursing in mental health, health promotion, antenatal care and health communication. Experts who were not bilingual for Yoruba and English language were excluded. Eighteen experts (six per material) were recruited through snowballing(4) . They were given copies of the materials and the tool (Suitability Assessment of Materials) for scoring at home for one week. The song was loaded on their mobile phones. They validated the suitability of the content of the educational materials shown in figure1-3 before being used by the clients in the maternal-child health clinics.
Characteristicsof study participants and selection procedure
The primary study participants were maternal child health service users; nursing mothers and pregnant women. To be included in the study, they had to be accessing routine care at the selected clinics, be able to read Yoruba language and should be available in the waiting area to attend to education sessions. Those who were in a hurry to leave or had a sick child were excluded.
A convenient sample size of 50 was taken that provided a power of 93% at 95% confidence level, margin of error 5%, assuming a loss to follow up of 10% to estimate that 90% ±20% of women rating the materials as suitable (3).
From two selected LGAs, three clinics with the highest patient load according to the clinic record during the period of the study were purposively selected. The three clinics (A, B, C) were selected from Ibadan North LGA and three clinics (X, Y, Z) from Ibadan North East LGA. The clinics names were written and rolled into paper balls that were placed inside a container. Three empty boxes had ‘poster’, ‘leaflet’ and ‘song’ labels written on them. Someone who was not part of the study was invited to randomly pick the first set (A, B, C) into the each of the empty boxes. The same process was repeated for (X, Y, Z). The 50 clients for each material (25 clients from each clinic) were selected randomly as follows: Paper balls labelled “yes” and “no” (50 each) were placed in a box and each eligible client at the clinics was asked to pick one. The clients who picked ‘yes’ each day and consented to participate were recruited. This process was repeated on every clinic day until 50 clients for each material in each clinic were recruited.
Instrument and Data collection procedure
The Suitability Assessment of Materials (SAM), a structured standardized tool (26) was used for assessment of the educational materials. It is adaptable to any materials including audiovisual, audio, print or book and freely available for use. SAM has 22 items spread across six domains; content, context, graphics, topography, simulation and cultural appropriateness. Not all the domains are relevant to all materials. Since SAM does not have a socio demographic information section, we added it as a separate a section on the tool.
The need for translation of SAM became necessary because the end users (maternal-child health clients) at the primary health care had low education. The SAM has never been used in Nigeria but it has been translated to Chinese (24), Turkish language by Akansel’s study on breast cancer educational materials as cited by (27), (28), Swedish language (29). We adapted the methods used in these studies involving forward translation, backward translation, synthesis, experts’ review and pretest in our study. Five translators recruited through snowballing translated SAM to Yoruba. Two translators who are bilingual in Yoruba and English languages translated the original English SAM into Yoruba independently. A master degree holder in English language, whose mother tongue is Yoruba back translated SAM into English. The back translation was then reviewed by two independent English linguists. They synthesized the original English SAM with the back translated English SAM. Through this they were able to find differences in loss of original meaning in the back translated version (30). This necessitated reviewing the Yoruba SAM again until the back translation gave the same meaning as the original English version of SAM. The domains of the Yoruba SAM were adapted for the leaflet, poster and song materials. After this, the researchers pretested the three Yoruba SAM on a population which was similar to the end users of materials for validation. A convenient sample size of 15 maternal-child health service users who could read Yoruba language (7 pregnant women and 8 nursing mothers) participated in the pretest for each Yoruba translated SAM. Three research assistants were recruited to document comments from the clients. All the domains of SAM which were difficult to understand were flagged by the participants. In the process, they were encouraged to speak out loudly about their concerns while reading the SAM to rate the Yoruba version of the educational materials. Meanwhile, the Research Assistants assigned to each person took note of the concerns and suggestions. The responses from the pretest was checked with response of the experts who used original English SAM to validate English version of the materials. The inconsistencies found in the comparison of the responses to the English SAM and translated SAM were corrected by translators and researcher. The Pretest was repeated until the translated SAM elicited right responses.
Data collection process
Content validation of materials by experts is a requirement in the development of educational materials (2). Experts made use of the relevant domains of the original Suitability Assessment of Materials (SAM) to rate the English version of each material. Since the song has no English translation they used the translated Yoruba SAM to rate the song. The ratings of the experts were used to validate the suitability of the content of the materials. The outcome was used to compute content validity index of each materials and comments were used to improve the materials further before the end users’ validation.
The Research Assistants (RAs) who worked on the pretest of Yoruba translated SAM are Yoruba speaking and they are well experienced in research data collection. Each RA was assigned to a material, trained in consent taking and the administration of SAM tool. The RAs used the SAM to ask participants questions and they recorded the rating/comments on the materials for each domain on SAM. The process lasted for one week on the poster and the song but the leaflet took up to 2 weeks because it has more content.
Measurement and Data processing
The primary outcome variable for this study is the validity index of the materials (poster, leaflet and song) as rated by experts and by maternal child health service users. The secondary outcome is the Cronbach coefficient of reliability of the translated Yoruba SAM for poster, leaflet and song.
Validity index of the experts’ rating was first analyzed which gave the materials the first authentication. An inter-rater analysis was carried out on the ratings of the 6 experts to obtain the interrater coefficient of agreement of all the experts. Cronbach alpha of >0.8 cut-off was used and the interclass correlation coefficient on the agreement of the 6 experts was set at p value set at >0.01 level of significance. Interclass coefficient of greater than 0.7 is regarded as moderate agreement while >0.9 is excellent agreement. Following this, validity index was calculated by formulas: I-CVI = Number of experts who rated an item as adequate/the total number of experts; S-CVI/UA=I-CVI rated 1/total no of items (31).
This validity index was computed on the experts’ rating of the educational material with use of the 6 items on SAM (content, literacy, graphics, layout, simulation and cultural appropriateness). The suitability assessment was done for each item with rating as 2 for superior, 1 for adequate and 0 for not adequate. For the purpose of the computation of validation 1, and 2 are regarded as 1=adequate (4).
There are two kinds of Content Validity Index (CVI)(32): Item level- Content Validity Index (I-CVI) and Scale level – Content Validity Index Average (S-CVI/Ave) or Scale level- Content Validity Index/ Universal Agreement (S-CVI/UA). The I-CVI = the number of experts who rated the item 1 or 2 (adequate) divided by the number of experts. S-CVI/UA or S-CVI/Ave are both scale level CVI with different formulas. S-CVI/UA is calculated by adding all items with I-CVI equal to 1 divided by the total number of items. On the other hand, S-CVI/Ave is calculated by taking the sum of the I-CVIs divided by the total number of items. A scale with excellent content validity should be composed of I-CVIs of ≥0.78, S-CVI/UA ≥0.8 or S-CVI/Ave ≥0.9. After this, descriptive analysis was used to analyze the frequencies of the socio demographic characteristics of the client participants and their suitability of assessment of the materials. The SAM score rates ≥70% (Superior: 2); ≥70-40% (adequate: 1) and ≤ 40% (not adequate:0). Chi square tests was used to assess the association of the sociodemographic characteristics and suitability rating with p-value of significance set at 0.05.
This study is part of parent study “Effect of training and supervision of maternal depression inclusive health education delivery among primary health care workers in Ibadan, Nigeria” which received an ethical review approval form the Ministry of Health, Oyo state Nigeria ref no AD 13/479/2016. Written consents were taken from the experts and the maternal child health service users who participated in the study. The consent contained the information about the study and voluntary nature of participation. The consent also assured participants of confidentiality and data protection. No names of individual were collected but codes were used as identifiers on the measuring instruments.