Has the Sample Been Appropriately Selected?

The selection of the subjects for an investigation is a vital research process, since the degree to which research findings can be generalized is largely dependent on the appropriateness of the sample. The target population is the entire group to which the results of the experiment can be generalized, such as all dental hygienists or all individuals with cerebral palsy. Since it is usually not feasible or even necessary to collect data on the entire population, a representative portion or sample is chosen. The best method for choosing a sample is random sampling, where every member of the population has an equal chance of being selected for the sample. Often access to the entire population is not available to the researcher, and a convenience sample of those members who are available is used. In this case, the researcher will try to identify subject characteristics that may have influence on the study and either verify statistically the groups are equal in regard to these characteristics or adjust for these differences during the analysis of the data with statistical tests such as analysis of covariance (ANCOVA)4 and logistic regression.

The size of the sample must be large enough to (1) accommodate the expected loss of subjects during the investigation and (2) demonstrate differences between groups by statistical logic. If the groups are too small, a real difference may not be detected with statistics. Possible misinterpretations are that the experimental treatment or product is not effective (even though it is) or the treatments or products used for both groups are equally effective (when one is actually superior). A rule of thumb is the minimal size of each group should be 26-30 subjects. Conversely, if the groups are too large, a trivial difference that does not have any therapeutic importance may reach statistical significance.4

A formal method for determining sample size is called power analysis, which provides more assurance that real differences or effects, if present, will be identified. Computer programs such as G*Power5 that perform power analysis are now readily available. They allow both an investigator to choose an appropriate sample size and a reader to decide whether a conclusion of "no difference" or "no effect" is warranted, given the actual results of the study. Investigators who perform proper power analyses when designing their studies are much more likely to produce (and publish) clear and unambiguous results, from which conclusions of “effect present” or “effect absent” can be drawn. Studies in which “power” was not addressed directly are more likely to yield inconclusive results.6-7

A research report should clearly describe the characteristics of the sample so the reader can evaluate whether or not the sample is representative of the population and appropriate for the experiment. This is particularly true if the sample was not randomly selected. Included should be such characteristics as age, sex, race or ethnic group, relevant medical conditions, oral hygiene habits, etc. If the reader is not provided with this information, the extent to which the conclusions apply to any other group of individuals or situations remains ambiguous. For instance, a researcher might generalize about the incidence of congenital heart disease in the United States population based on an experimental sample. However, if a description of the sample indicated it was composed of institutionalized patients (who have a much higher incidence of the disease), the reader would know such a generalization was inappropriate. The report should also describe the size of the sample, both at the beginning and the end of the investigation.