Background Information
On 31st May this year during the World no Tobacco Day, the Dubai Ministry of Health (MOH) in collaboration with Dubai Healthcare City (DHCC) and other stakeholders launched a smoking cessation program that was branded the “No Tobacco Campaign” (George, 2010). The objective of the smoking cessation program was to reduce the prevalence rate of smokers within Dubai and other parts of the UAE by encouraging smokers to adopt healthy behavior and quit smoking.
Towards this end, the program target was to achieve a 24% reduction among addicted smokers who would be enrolled in the No Tobacco Campaign by setting their benchmark at a more optimal level of 50% (George, 2010). The secondary objective of the smoking cessation program was to prevent the relapse of the rehabilitated subjects from resuming smoking habits as well as minimize the incidence rate of people who are non-smokers from taking up the habit.
As such the target group of the smoking cessation program involved both smokers and non-smokers as its focus in reducing the smoking prevalence rate and in promoting healthy behavior. The No Smoking Campaign initiative of the smoking cessation program was structured along with a form of a competitive contest between smokers that were attempting to quit smoking referred to as the Quit and Win contest. This was a form of a competition that pooled together various participants in the smoking cessation program to compete against each other in quitting the habit of smoking and in the process be eligible to win a prize.
To be eligible a person must have been smoking for at least a year and be willing to completely abstain from smoking for a period of not less than one month. At the time of the launch of the program, the smoking prevalence rate in Dubai stood at 20% for males and 3% among females; the highest proportion of the smokers at 42% was among people aged around 18 years while teenagers aged between 13 and 15 years made up 25% of all the smokers (George, 2010). In this part III of the smoking cessation program evaluation, my focus will be on data collection methods and analysis within the program evaluation phases.
Program Evaluation Process
The importance of data collection during program evaluation is an integral part of the process which cannot be overemphasized. Program evaluation is the process of appraisal of a project or part of a project through data collection and analysis of the information gathered to inform on the progress of the program or lack thereof (Hass, 2000). Generally, there are three phases in the program evaluation process which are formative evaluation, summative evaluation and impact evaluation (Hass, 2000). The formative evaluation phase is done during the conceptualization of the program and is intended to support the implementation of the program.
Summative evaluation is done when the program is ongoing to provide an insight on how well the program objectives are being met while impact evaluation is usually done at the end of the program implementation to determine if the changes in the community can be attributed to the program referred as impact. In the context of this program on smoking cessation, the data collection plan for the program will be designed along with these three phases of program evaluation
Data Collection Plan
There are various types of data collection methods that can be chosen when undertaking a program evaluation process; the idea is to utilize data sources that can be relied on to provide an accurate assessment of the program that is being evaluated. For purposes of this program, the data collection method that was used before the launch of the program involved the use of questionnaires on subjects of interest, sort of a baseline survey that informed decision-makers on the status quo as far as smoking is concerned in Dubai. As the smoking cessation program is implemented more data is collected in order to assess the success and impact of the No Tobacco Campaign among the target group.
This meant that the nature of data being collected is both qualitative and quantitative since program evaluation requires analysis of perceptions such as reasons for failure by cases to abstain from smoking as well as statistical summaries. To achieve this, a reliable data collection method is necessary; a good data collection plan must incorporate three key activities; definition of data that need collection, actual collection of data and data analysis plan (CDC.gov, 2008).
In this case the definition of data collection will be contained in pre-collection activities of data which include identification of respondents whom in this case are smokers; method of data collection will be through questionnaires and description of data utilization. The actual data collection will be undertaken among the cases, in this case, smokers at various points in time during the program implementation as well as after implementation. Finally the data analysis plan will utilize both descriptive and inferential statistic techniques since the data collected is both numerical and subjective in nature.
Below is a data collection plan template that can be used as a framework to develop a comprehensive data collection plan that integrates all the relevant data collection methods.
Table 1.
Typical variables of interest that will be captured during data collection stage for this program will include age, duration of smoking, disease conditions, occupation, income level, reason for smoking, reason for failure to quit smoking, longest duration that a respondent has ever abstained from smoking and so forth.
The data collection process should be integrated in every program activity and should essentially be enough to monitor key indicators that the evaluation of the program is pegged on (Brent, 2006). The implication is that each point of contact with the cases through all the activities that are being implemented by all the stakeholders would ideally be an opportunity to collect data of interest which would later be passed on to the relevant department for analysis.
Strengths and Weaknesses for Data Collection Sources
The data collection sources for this program will be from the Ministry of Health, DHCC, national surveys, ExHealth, Green Crescent Insurance Company, target cases and all other partnering organization and health providers that were involved in the implementation of the program. Previous national surveys and ministry of health records would provide important data that is crucial for baseline survey which is vital in establishing the extent of the health problem as has been described so far.
The MOH data records as well as the other implementing stakeholders such as DHCC and Exhealth will provide current data that pertains to the program which will guide decision making. As such the accuracy of these data sources are crucial in enabling implementation of a quality program. In theory there are four data collection sources for most health programs which are medical records, survey studies, health events and data collected from ongoing program implementation (Brent, 2006).
Medical Records
Medical records are the most crucial source of data for health programs since they summarize vital demographic data of target cases and their medical background. This saves a lot of time from having to collect these variables using other research studies and is one of the greatest strengths of this data source. The fact that the medical history of a subject is also summarized in the chart enables valuable inferences to be made from the records and is in fact one of the primary sources of data for heath surveys and retrospective research studies.
However on the other hand this source of data poses limitations in situations where subjects keep shifting their healthcare providers or circumstances where medical history is irrelevant for purposes of the desired program or research study. But in this case this source of data would be crucial for program evaluation and there is minimal weakness for this type of data given that medical history and the demographic details are very important for this evaluation.
Survey Studies
This refers to research survey studies that are done nationally to determine the prevalence or incidence rate of a particular disease condition. From this survey studies we are aware of the prevalence rate of smokers in the country which is found to be at 20% for males and 3% for females. This therefore becomes our baseline from which we shall set the benchmark of the program achievement and determine if change in behavior can be attributed to the implementation of the program. The strength of this data source is that it is more reliable since it is done on many cases which mean the data is more accurate and therefore reliable.
Possible limitations for this data collection source is the fact that figures might not be current given that most research studies are done after several years. Because survey studies rely on samples like all other form of research studies, it becomes hard at times to generalize findings to specific categories of populations.
Health Events and Program Implementation Sources
The routine health events and other activities that are associated with the launch of the program provide excellent opportunities to obtain vital data required for program evaluation. The importance of this source of data is that it can be customized to capture the relevant variables of interest that the program requires. In addition the data obtained is usually current and can be valuable in determination of particular trends that the program requires to focus on. For instance ongoing healthy events can be used to obtain information on current cigarette brands mostly used by the subjects, number of cigarettes smoked and cost of smoking in a day.
The downside for this data source is that it is often prone to error from respondents or due to bias because of the way researchers structure and direct the questions. The fact that data in this case is always collected through face to face interviews introduces another element of inaccuracy caused by respondent perception or ego. For instance a respondent might exaggerate the number of cigarette smoked or brand of cigarette smoked in order not to appear to be “struggling financially” thereby leading to capture of inaccurate data.
Threats in Data Collection Plan
Data collection approach is usually determined by the nature of data that is being collected which is normally divided into two categories; numerical data and subjective data (Gallagher, Stanley, Shearer and Klerman, 2005). Because the data sources varies for each type of data that is collected the challenges associated with data collection also varies within the same program. In general any type of data collection method is threatened by several factors which involve cost constraints, time factors, technological limitations and level of skill workers required to collect the data (Gallagher et al, 2005). One of the biggest challenges that impact on the quality of data collected have to do with timeliness of the data that is being captured as well as its quality, for purposes of this program a third factor of capacity limitation will influence the data collection process of the smoking cessation program.
Access to timely data is crucial for monitoring and evaluation (M&E) purposes of a program, this requires that the procedures for data collection and dissemination be clearly outlined in order to facilitate this process. In this case dissemination of captured data from all the stakeholders to a central M&E office that is done on time is crucial for program evaluation, but submission of data on time is a challenge for many programs that usually involve implementation by various stakeholders. Collecting relevant data from all this stakeholders routinely is a challenge that impacts on the quality of the program evaluation.
Quality of data during collection stage is another factor that negatively impacts on the data collection process; this is because different stakeholders will have different approaches and methods for capturing data that is done by people with varying skills. The risk in this case is the possibility of introduction of bias during the process of data collection by the persons delegated with the responsibility of collecting data.
This means that the data is not consistent across the board which is a characteristic that impacts on the overall quality of the data. Finally the stakeholders involved in the implementation of this program might lack the capacity to collect all the required data from the cases thereby limiting the accuracy of the analysis that can be obtained from such data. This could be because of human resource limitation which means that only a fraction of cases will have their data collected.
Data Analysis Procedures
Organization of collected data is an important prerequisite for data analysis process that is usually done after collection of data. Data organization involves data entry and sorting to ensure that captured variables are easily accessible and well arranged (Schadewitz and Blevins, 1997). The actual data analysis will involve two important steps that include; statistical analysis and qualitative analysis. Statistical analysis which is also referred as quantitative analysis involves application of mathematical concepts to analyze and compare various characteristics of the data variables such as T-tests, anova, and standard deviation among others (Schadewitz and Blevins, 1997).
Quantitative analysis on the other hand involve making inferences that are based on the results of the statistical analysis, this form of analysis goes beyond the facts of the data collected and is mostly exploratory in that it advances hypothesis that are supported by the statistical analysis.
These are the two approaches that I will use to analyze the data collected from the smoking cessation program. For instance the statistical analysis will enable me to determine the change in prevalence before the implementation of the program and after implementation in terms of percentage. I would also be able to determine which category of persons disaggregated by years that the program had the most impact, for example I can determine from the data that incidence rate of smoking for people aged between 13 and 16 years dropped by 50%. When I apply the qualitative analysis on the same dataset I can infer that the success of the program with this group of persons was probably because of the technology used during the campaign that was largely accessible to the young people as opposed to adults and elderly.
Data of Statistical Significance and Practical Significance
Statistical significance in data analysis is a term that is used to describe the probability of mean differences from a dataset that are is not a result of sampling error (Gall, 2004). High statistical significance would therefore mean that the observed statistical descriptions are accurate and not due to any form of sampling errors that could have occurred during the study. Practical significance on the other hand is a measure of the impact that the observed mean difference from statistical analysis would make when applied to real population actual (Gall, 2004).
It is the usefulness of the statistic analysis in practical terms which must be large enough to make observable change when applied practically. The null hypothesis denoted as H0 and the alternative hypothesis (Ha) are some of the two most frequently used statistical significance measures that are used together with P-value to determine if data is statistically significance (Utts, 2005).
To determine the practical significance of data, calculation of the “effect size” of the change is essential, but because the practical significance of any data is usually a factor of the sample size then it means data collected utilizing large sample size have no practical use and vice versa (Gall, 2004). The implication is that data collected from ideal sample size is both statistically and practically significance while this might not be the case for data that involve large sample sizes.
Elements of Program Evaluation Report
The major sections of a program evaluation report include description of the program that is being evaluated which is the introductory part, objectives of the evaluation, description of the program evaluation approach, summary of evaluation findings, discussion of findings and finally the recommendation section (LewinGroup.com, 2009). The description section introduces the limits of the program that is being evaluated and defines the major objectives of the program as well as it overall goal; this is the section that I would introduce the purpose of the Dubai smoking cessation program. The specific objectives of the study are elaborated on the next sections of objective evaluation, after which the evaluation process of the program is discussed in detail. The rest of the section will be on summary of the findings, discussion on findings and their recommendation.
Utilization Processes in Program Evaluation
An ideal utilization process for program findings is one that provides all the implementing stakeholders with feedbacks that is necessary to enable them to positively influence program outcome (Bamberger, Rugh, Church and Fort, 2004). One of the most commonly utilization approach of program evaluation is referred as UpFront which is a form of a Utilization-Focused Evaluation designed by Patton (2004).
As depicted in the diagram the utilization process of the program evaluation touches on the four integral stakeholders of a program which are donors, community benefiting from the program, workers involved in the actual implementation and program designers and managers (Patton, 2004). This is the utilization plan that I would apply to generate stakeholder-specific recommendations that I would emphasize their importance for the success of the whole program.
References
Bamberger, M., Rugh, J., Church, M., & Fort, L. (2004). Shoestring Evaluation: Designing Impact Evaluations under Budget, Time and Data Constraints. American Journal of Evaluation, 25(5):14-37.
Brent, V. (2006). Primary Data Collection Methods. Web.
CDC.gov. (2008). Data Collection Methods for Program Evaluation: Observation. Web.
Gall, M. (2004). Statistical versus Practical Significance. Web.
Gallagher, K., Stanley, A., Shearer, D. & Klerman L. (2005). Challenges in Data Collection, Analysis, and Distribution of Information in Community Coalition Demonstration Projects. J Adolesc Health, 37(3):53-60.
George, T. (2010). A Tobacco Reconceptualization in Psychiatry: Toward the Development of Tobacco-Free Psychiatric Facilities. The American Journal on Addictions, 19(4), 293-311.
Hass, G. (2000). The Three Phases of Program Evaluation. Web.
LewinGroup. (2009). Elements of a Strong Evaluation Report. Web.
Patton, M. (2004). Program Evaluation: Utilization-Focused Evaluation. Web.
Schadewitz, H. & Blevins, D. (1997). A Review of the Data Collection Process. Journal of Technical Writing and Communication, 27(4): 417-424.
Utts, J. (2005). Seeing through Statistics. Ontario, Canada Third: Thomson Learning Inc.