HOME HELP FEEDBACK SUBSCRIPTIONS ARCHIVE SEARCH TABLE OF CONTENTS		QUICK SEARCH:   [advanced] Author: Keyword(s): Year:  Vol:  Page:

This Article Full Text (PDF) Alert me when this article is cited Alert me if a correction is posted Services Email this article to a friend Similar articles in this journal Similar articles in PubMed Alert me to new issues of the journal Download to citation manager Citing Articles Citing Articles via HighWire Citing Articles via Google Scholar Google Scholar Articles by Crewson, P. E. Articles by Applegate, K. E. Search for Related Content PubMed PubMed Citation Articles by Crewson, P. E. Articles by Applegate, K. E. Social Bookmarking                      What's this?

Data Collection in Radiology Research

¹ American College of Radiology, 1891 Preston White Dr., Reston, VA 20191.
² Department of Radiology, Rainbow Babies & Children's Hospital, 11100 Euclid Ave., Cleveland, OH 44106-5056.
³ Present address: Department of Radiology, Riley Hospital for Children, 702 Barnhill Dr., Indianapolis, IN 46202-5200.

Received April 3, 2001; accepted after revision April 24, 2001.

 
Series editors: Craig A. Beam, C. Craig Blackmore, Stephen J. Karlik, and Caroline Reinhold.

This is the fourth in the series designed by the American College of Radiology (ACR), the Canadian Association of Radiologists, and the American Journal of Roentgenology. The series, which will ultimately comprise 22 articles, is designed to progressively educate radiologists in the methodologies of rigorous clinical research, from the most basic principles to a level of considerable sophistication. The articles are intended to complement interactive software that permits the user to work with what he or she has learned, which is available on the ACR Web site (www.acr.org).

Project coordinator: Bruce J. Hillman, Chair, ACR Commission on Research and Technology Assessment.

Address correspondence to P.E. Crewson.

Introduction

Top Introduction Define the Primary Research... Identify Data Requirements IRB Approval The Statistical Analysis Plan Designing the Data Collection... Plan for Data Management Quality Control Electronic Data File Conclusion References

 
This paper introduces the basic principles essential for a successful data collection effort. Data collection must begin with a clear research question. The researcher should then carefully identify data needs, anticipate problems with data measurement and missing data, design and pilot test a data collection system, establish quality control, and plan both data entry and statistical analyses. To be successful, all aspects of data collection must focus on the goal of obtaining substantively important data that are consistent, accurate, and unbiased.

"On being asked to talk on the principles of research, my first thought was to arise after the chairman's introduction, to say, `Be careful', and to sit down..." by J. Cornfield [1].

Universally lamented by experienced clinical researchers as an important but often ignored aspect of medical research, good study design and data collection are critical to the success of any clinical study [2, 3]. Although most researchers are, by their very nature, excited by experimentation and analysis, few find enjoyment in the design and implementation of data collection, although these factors are critical to successful research. Too often, researchers pay little attention to how data will be collected, if the data are available or can be measured, or how much data will be incorrect or missing. Even fewer researchers carefully train the data collectors and periodically check their work.

This paper outlines seven basic elements of data collection. We discuss defining the research question, deciding on what data to collect, obtaining institutional review board (IRB) approval, planning statistical analyses, designing the data collection system establishing quality control, and organizing data entry. This article is by no means comprehensive but provides guidelines that we believe will improve clinical research in radiology.

Three general rules of data collection underlie this discussion. First, researchers should assume they will underestimate the amount of time and effort involved in data collection. Second, the more complex the data collection process is, the longer it will take to acquire and enter the data. Finally, systematic and individual data collection errors must be addressed early in the process, because it is unwise to trust human memory or a statistician's creativity to resolve errors in the data.

Define the Primary Research Question

Top Introduction Define the Primary Research... Identify Data Requirements IRB Approval The Statistical Analysis Plan Designing the Data Collection... Plan for Data Management Quality Control Electronic Data File Conclusion References

 
The first step in designing data collection is formulating the research question or questions [4]. The research question should identify the study's end points, also known as the response or outcome variables (see Appendix 1 for a glossary of terms). Examples of common end points in diagnostic imaging studies are diagnostic accuracy, patient quality of life, patient satisfaction, patient comfort, safety, morbidity, impact on patient care, and costs.

The end point is the dependent variable, the variable you wish to better understand. Identifying other variables becomes an exercise in determining what factors might explain variations in the study's end point [4, 5, 6]. These factors, known as independent variables, usually include basic demographics such as age, sex, and race. Other independent variables could include comorbidity, stage of disease, signs or symptoms, laboratory test results, imaging test results, clinician experience and training, type of imaging equipment, and patient movement, to name only a few. To be worthy of inclusion in the study, independent variables should either relate directly to the research question or provide useful controls for defining the study population and sample.

Identify Data Requirements

Top Introduction Define the Primary Research... Identify Data Requirements IRB Approval The Statistical Analysis Plan Designing the Data Collection... Plan for Data Management Quality Control Electronic Data File Conclusion References

 
Deciding what, how, and when to collect data may be the most difficult part of the data collection design [2]. Every principal investigator will be faced with the dilemma of either collecting too little data, thereby weakening the study's results, or attempting to collect too much data, and becoming so overwhelmed that the study is never completed or participation of institutions and personnel wanes from exhaustion. Collecting insufficient data may also significantly impact the statistical analyses. Few studies have the luxury of retrospectively obtaining data after the study has been closed. In turn, endeavoring to collect too much data can result in lack of participation by patients and institutions, excessive amounts of missing data, fatigue of support personnel, and cascading delays in patient accrual, data cleaning, and analysis [2]. This trade-off is particularly important to address in multidisciplinary research where there will be greater demands to collect extraneous data. In characterizing these trade-offs, one author suggests that the right amount of data are "as many as necessary and as few as possible" [4].

Three additional elements must also be considered when determining what data will be collected. Essential for designing data collection forms and creating data files, these elements are the unit of analysis, data precision, and the collection sequence. The involvement of a statistician at this stage of data collection design cannot be overemphasized. They can provide guidance on determining the unit of analysis, data precision, and many other research design issues essential to a defensible statistical analysis.

Unit of Analysis
Determining the unit of analysis is a basic task in designing a study, not only for methodologic reasons, but also because it affects the design of data collection forms, the storage and linking of documentation, and the design of electronic data files. The most common unit of analysis is the individual patient, but there are many other possibilities, such as the institution, the type of procedure, the images, or in the case of reader studies, even individual radiologists.

Data Precision
The degree of accuracy needed in the collected data also deserves early attention [2, 4]. There are likely to be several different ways to measure the data you collect. For example, when recording carotid stenosis is it sufficient to record stenosis to one decimal place (0.4), two decimal places (0.44), or three (0.435)? Obviously, the more precise the measure the better, but the goal of precision may need to be tempered by consideration of the cost in both time and money and the substantive importance of the measure.

Whenever possible, use well-established measurements and common terminology to reduce design time and improve comparability with other studies [6]. In addition, good research design must address reliability (consistency and reproducibility, such as the extent to which a measure obtains similar results on identical patients) and validity (how often the positive test result is correct) [2, 4, 5, 6]. Both are important for establishing the accuracy of the study outcome.

Collection Sequence
Finally, study collaborators should consider the sequence of data collection early in the study design. This will allow for thoughtful preparation of data forms, the design of an adequate data file format, and development of a suitable analysis plan. Many studies incorporate patient follow-up, often at multiple intervals. Follow-up measurements should be recorded on well-focused forms that are coded with a common linking identifier (generally the case identification number) to ensure they can be aggregated with previously collected data.

IRB Approval

Top Introduction Define the Primary Research... Identify Data Requirements IRB Approval The Statistical Analysis Plan Designing the Data Collection... Plan for Data Management Quality Control Electronic Data File Conclusion References

 
Local institutional review board approval is required in most, if not all, clinical studies and deserves to be a major consideration when designing data collection (Appendix 2). Regardless of whether it is a retrospective analysis of collected data or a prospective clinical trial, no data collection should be initiated until all ethical, procedural, and legal requirements are satisfied [7, 8]. IRB requirements will vary, but you should be prepared to address patient confidentiality, potential risks to the patient, and procedures for obtaining informed consent and monitoring for adverse events. Some IRBs will give quick administrative approval for a retrospective study of medical records, whereas others require both full IRB committee review and patient informed consent for this type of study. There is increasing public concern over confidentiality of medical records and increased scrutiny of medical research by the federal government. This concern cannot be overemphasized and is exemplified by a recent New York Times report that a computer hacker accessed thousands of medical records in a cardiology research database at the University of Washington [9].

The Statistical Analysis Plan

Top Introduction Define the Primary Research... Identify Data Requirements IRB Approval The Statistical Analysis Plan Designing the Data Collection... Plan for Data Management Quality Control Electronic Data File Conclusion References

 
The precision and scale of the data (i.e., nominal, ordinal, and interval) will determine or limit the statistical techniques used in the data analysis portion of the study. Collecting patient age in years is fairly precise, but collecting date of birth allows computation of age to years, weeks, and days. Similarly, collecting age by grouping (<35, 36-55, 56) converts the scale of data from interval (computing mean age in years) to ordinal (not useful for computing a mean). As a result, developing a clear statistical analysis plan at this early stage can be very useful [10] not only in providing focus for the data collection effort (such as specifying sample size estimation) but also pointing out weaknesses in the scale and precision of the data before data collection begins.

The statistical analysis plan is a detailed out-line of what data will be analyzed and how. This plan should include clear definitions of variables and statistical end points (descriptive or inferential), a description of the required subgroup analyses, and identification of the most appropriate statistical techniques and their relationship to the research hypotheses. Although a poor research methodology, data are frequently collected without a clear understanding of how it will be analyzed or the scale of data necessary for a particular statistical technique. A useful but time-consuming tool in designing a statistical plan is to draft the tables you will use to present the results of your analysis [5]. This approach is helpful in identifying important comparisons while clarifying statistical method requirements and data needs.

Designing the Data Collection System

Top Introduction Define the Primary Research... Identify Data Requirements IRB Approval The Statistical Analysis Plan Designing the Data Collection... Plan for Data Management Quality Control Electronic Data File Conclusion References

 
Most data are either collected from secondary data sources such as patient records and other administrative databases or from primary sources such as patient interviews, patient surveys, and interpretation of imaging studies by clinical personnel. Collection instruments can be as simple as handwriting data on a paper ledger or as complex as creating a complete computerized internet-accessible direct entry system. Do not assume, however, that sophisticated electronic collection systems will produce error-free data [11]. They will be susceptible to the same problems as paper forms, such as entry errors and missing information. In addition, errant programming could lead to a multitude of problems such as improperly formatted data fields, unsaved entries, and a confusing data file design that requires extensive and time-consuming manipulation.

Regardless of the complexity, factors to consider in designing the data collection system include creating data forms, avoiding systematic bias, and preparing a plan for data administration.

Data Collection Forms
The case report form is a common tool used to collect multiple sources of data (patients, physicians, records) into one document. In developing and designing this type of data form, it is wise to allow for detailed notes, regardless of the number of investigators involved in the study [12]. These notes may or may not be entered into an electronic data file, but they can become invaluable in explaining otherwise unexplainable variations in the data later in the study. Examples could be patient movement or other uncooperative activity, equipment malfunctions, previously undisclosed comorbidities, or exceptions to protocol guidelines that can occur for many reasons including human error and clinical necessity.

Form development is both an art and a science, but there are a few basic rules to follow. First, forms should be self-explanatory to the person entering the data. Second, data should not require extensive interpretation before recording. Third, the unit of measurement should be defined. Using time as an example, specify which unit of measurement is required (hours, days, weeks, months, or years). Level of precision should be evident (fractions of hours, round to nearest full day).

Also, consistent and complete responses should be required for each section of the form. Never leave a section blank. Leaving a section blank may mean the issues are not applicable (which is important to code) or the originator of the data forgot to respond. In the case of missing data, an assumption of irrelevance may be entirely wrong.

Finally, the form should be visually appealing, easy to navigate, and conducive to data entry [2, 4]. It is often helpful to have the coding conventions for data entry included directly on the form (female [0], male [1]).

Pilot Testing
Pretest the forms on individuals who are characteristically the same as those who will fill out the forms in the study; have physicians fill out physician forms, technologists fill out technologist forms, and someone who is not a physician or healthcare worker fill out the patient questionnaires. Involve everyone who will be handling the data collection, data entry, or analysis in the form design and testing process. The initial data collection form may be piloted on a small sample of potential patients to determine whether the desired data are available and whether the data form is easy to complete and enter into the database. The data form can then be revised before the full study has begun. Finally, the principal investigator should not rely on memory to recall study design issues such as units of analyses, data measurement techniques, definitions of each measurement and variable, and time sequence. All members of the research team need a copy of the methodology and a "code book" to serve as a reminder of the methods established for data collection.

Avoid Systemic Bias
Although some biases can be corrected in the analysis, some are fatal and may render the study invalid. Therefore, it is always best to design the study to avoid or minimize these biases. There are many sources of bias to consider, however; some are more closely related to the data collection effort than others. In particular, steps should be taken to maintain objectivity in the data collection system while avoiding bias in patient recruitment and minimizing the effects of "interpretation bias" and "response bias."

Objective measurement of data reduces the likelihood of collection bias, but the degree of objectivity can vary, depending on the means of measurement. As an example, measures of body weight on a calibrated digital scale are unlikely to vary depeding on who weighs the patient. In contrast, if surveyors are asked to indicate whether a patient is underweight, normal in weight, or overweight, much will depend on individual perceptions. It is a subjective measure.

Subjective measures are particularly susceptible to prior knowledge of the treatment arm of a clinical trial [10]. In blinded studies neither the patient nor the data collector know who is in the control group or in the treatment group, therefore minimizing bias. In open studies both the patient and the data collector know who is given treatment. In the event complete blinding is not possible, a blinded clinician could be used to review the data from both groups for consistency. Although a complete discussion is beyond the scope of this paper, be aware that a clinical study may require procedures that fail to completely mirror clinical practice, such as having all available patient information before making an assessment.

If your study calls for patient randomization into multiple study arms, it is essential that the randomization is, if at all possible, either done by a third party or automation. Before obtaining patient consent, the study monitor should have no knowledge of that patient's study arm placement.

In diagnostic imaging, comparisons often involve the same patient receiving two diagnostic tests [13]. Ensuring that the technologists and radiologists are unaware of the competing test results is essential to prevent interpretation bias. This sort of blinding will require two separate, and probably different, data report forms.

Response bias can occur during the follow-up stage of a study because of incomplete responses or patients lost to follow-up [14]. Ill patients may be more likely to complete a follow-up quality of life survey than patients who are not ill. As a result, aggregate quality of life estimates may be lower than what they would have been if all participants responded.

Plan for Data Management

Top Introduction Define the Primary Research... Identify Data Requirements IRB Approval The Statistical Analysis Plan Designing the Data Collection... Plan for Data Management Quality Control Electronic Data File Conclusion References

 
Investigators must develop a plan for ensuring the confidentiality and storage of paper forms and documentation. The preparation of a coding system that provides individual identifiers (case IDs) for each patient is one way to keep data confidential and to support blinding efforts. The same case ID should be used consistently for all data forms related to a particular patient. Maintaining consistent case IDs is especially important for follow-up efforts, because these data may have to be entered separately and then merged with the primary data file at a later date.

In general, all personal identifiable information should be collected and stored separately from the case report form. Even though patient identifying information should be separate from the case report form, each set of data should be stored in a secure location with limited access. Assign responsibilities for data storage and maintenance of the master list that contains both patient information (names and addresses) and assigned case IDs. Similarly, a computer specialist must secure the confidentiality of the electronic files.

Quality Control

Top Introduction Define the Primary Research... Identify Data Requirements IRB Approval The Statistical Analysis Plan Designing the Data Collection... Plan for Data Management Quality Control Electronic Data File Conclusion References

 
Data integrity is the bedrock of any clinical study [2, 10, 14]. Early and ongoing review and cleaning of data during the collection process, while being alert for systematic biases in data collection and processing, is a critical element of ensuring quality control [15]. Preventable errors should be identified and, ideally, corrected early in the study, instead of consuming expensive resources and time cleaning the database after the study has been closed.

Data Cleaning
Cleaning data requires developing a scheme for ensuring that the data are consistent and accurate. Much depends on the study design, but consider monitoring for the following: out-of-range data values; missing data; lack of variability (survey questionnaires can include reversed questions to see whether the respondent is using the scale appropriately); logic traps (check combinations of responses for inconsistency, such as a female record that lists chronic prostatitis as a comorbidity); and date checking (verify forms are completed in sequential order) [12]. An entry error in the year field is much easier to catch early on than it is after the data are entered and combined with other participants.

All members of the research team should understand the goals and design of a study so that they may flag questionable data [2]. The study design should clearly identify the target and study populations and patient selection criteria so that variability among centers and the individual investigators who enroll patients is minimized. Develop and enforce consistent rules for data review and cleaning, including specification of how to handle missing dating. These rules should be delineated before data aggregation in which the temptation to justify certain decisions in favor of a particular outcome is strongest [10]. The principal investigators should also determine whether "interim analysis" is necessary and determine prospectively when and what is analyzed and identify the decision rules for discontinuing the study [10, 16]. An interim analysis is generally done when it is important to monitor the efficacy or safety of two treatments.

Once the data are clean, the "database lock" occurs, the point at which no additional cases or data will be added to the data file. Always assume, however, that there will be data errors even after a complete quality control plan is used [4, 10]. During the statistical analysis, do not be surprised if it becomes necessary to pull original case report forms to answer questions from the analysis. Outliers can be very revealing in a statistical analysis and it is not unusual to want to verify data integrity when the results run counter to theory or prior experience.

Amoral Consequence of Dishonesty
Our discussion of bias thus far has assumed that errors in data collection are the result of unintentional practices, such as misunderstanding instructions or rationalizing postprotocol changes in study design that result in collecting and reporting inaccurate information. In contrast, dishonesty biases data collection through deliberate falsification of either the raw data or the conditions essential for maintaining the integrity of the clinical study (such as proper patient recruitment). Regardless of whether data collection errors are accidental, well-intentioned, or the result of a deliberate fabrication, the amoral consequence is bias [3]. Some will conclude that quality control is a necessary evil to prevent the errors caused by others involved in the study. For most studies, however, the danger of instituting error in data collection rests less with the dishonest than with those well-intentioned researchers who fail to recognize and take steps to mitigate their own potential for bias.

Electronic Data File

Top Introduction Define the Primary Research... Identify Data Requirements IRB Approval The Statistical Analysis Plan Designing the Data Collection... Plan for Data Management Quality Control Electronic Data File Conclusion References

 
It would be hard to conceive of a clinical study that does not require statistical analysis. Regardless of whether the statistical needs are modest (counts, percents, means) or more demanding (multivariate techniques, survival analysis, complicated error estimation), most forms of statistical analysis require the creation of an electronic data file. Therefore, planning the format of the data file at the beginning of the study is essential [1]. There can be a disconnect, however, between what the clinician visualizes as data and what the statistician needs. Although most clinicians are likely to view data in its raw form as patient records, lab results, responses on case report forms, and interview sheets, statisticians view data as numbers in an array of rows and columns.

Although there are many available data file formats and complex organizational structures, such as relational databases, most statisticians prefer the traditional rectangular data file (Table 1). Analogous to the common spreadsheet, each row typically represents one case (a patient) and each column represents a variable (a data element). Ideally, most data entries are numerical codes [1, 4]. As an example, although it is possible to enter "male" or "female" for the sex variable, data entry and subsequent statistical programming are much simpler if numbers (numerical fields) are used in place of words (string fields). In Table 1, female patients are coded "0" and male patients are coded "1." If possible, avoid open-ended entries (e.g., free text comment or description fields) because they will inevitably lead to interpretation error. Popular electronic data files include delimited text files, Excel (Microsoft, Redmond, WA), SAS (SAS Institute, Cary, NC), SPSS (SPSS, Chicago, IL), Access (Microsoft), and Epi Info (Centers for Disease Control and Prevention, Atlanta, GA).

Conclusion

Top Introduction Define the Primary Research... Identify Data Requirements IRB Approval The Statistical Analysis Plan Designing the Data Collection... Plan for Data Management Quality Control Electronic Data File Conclusion References

 
Our discussion has been based primarily on experiences related to collecting data from traditional sources, such as paper case report forms and questionnaires. Much of what has been presented, however, will also be useful as technology shifts to more internet-based collection efforts [17] and alternative software collection systems [18]. We are confident that, regardless of advances in technology and the potential for increased automation in data collection, the basics will remain important.

Data collection requires thoughtful preparation and consistent implementation. To be successful, all aspects of data collection must be focused on the goal of obtaining substantively important data that are consistent, accurate, and unbiased. Data collection begins with a clear research question and is followed by careful attention to identifying data needs, anticipating missing or incorrect data, planning statistical analyses, designing a data collection system, establishing quality control, and planning for data entry. Considerable misspent effort can be avoided if the principal investigators, data managers, and statisticians work together early in the design of a data collection effort.

We have presented elements of a data collection checklist that should be addressed in most, if not all, clinical research. This list is not comprehensive; much will depend on the specifics of a particular study, but recognition of the seven primary issues can dramatically improve the quality of research in radiology.

,

References

Top Introduction Define the Primary Research... Identify Data Requirements IRB Approval The Statistical Analysis Plan Designing the Data Collection... Plan for Data Management Quality Control Electronic Data File Conclusion References

 

1. Feigal D, Black D, Grady D, et al. Planning for data management and analysis. In: SB Hulley, SR Cummings, eds. Designing clinical research: an epidemiologic approach. Baltimore: Williams & Wilkins 1988: 159-171
2. Friedman LM, Furberg CD, DeMets DL. Data collection and quality control. In: Fundamentals of clinical trials. New York: Springer 1998:156 -169
3. Altman DG. Statistics and ethics in medical research: collecting and screening data. BMJ 1980;281 : 1399-1401[Medline]
4. Crombie IK, Davies HTO. Issues in data collection. In: Research in health care: design, conduct and interpretation of health services research. West Sussex, England: Wiley, 1996: 199-222
5. Goldin J, Sayre JW. A guide to clinical epidemiology for radiologists. I. Study design and research methods. Clini Radiol 1996;51:313 -316[Medline]
6. Grady KE, Wallston BS. Research in health care settings. Sage 1998;14:84 -100
7. Office for Human Research Protections. Regulations. Department of Health and Human Services Web site. Available at: http://ohrp.osophs.dhhs.gov. Accessed April 18, 2001
8. Department of Health and Human Services Commission on Research Integrity. Integrity and misconduct in research. Washington: United States Printing Office, 1995. Publication no. 1996-746-425
9. Hackers from abroad obtain data on Washington patients. New York Times, Dec 8, 2000. Avaiable at: www.nytimes.com. Accessed on December 8, 2000.
10. Meyerson LJ, Wiens BL, LaVange LM, et al. Quality control of oncology clinical trials. Hematol Oncol Clin North Am 2000;4:953 -971
11. McManus B. A move to electronic patient records in the community: a qualitative case study of a clinical data collection system problems caused by inattention to users and human error. Top Health Inf Manage 2000;20:23 -37[Medline]
12. Fisher LD, Van Belle G. Data collection: design of forms. In: Biostatistics: a methodology for the health sciences. New York: Wiley, 1993:24 -34
13. Valk PE. Clinical trials of cost effectiveness in technology evaluation. Q J Nucl Med 2000;44:197 -203[Medline]
14. Kane RL. Miscellaneous observations about outcomes research: practical advice. In: Understanding health care outcomes research. Gaithersburg, MD: Aspen, 1997:243 -255
15. Begg CB. Biases in the assessment of diagnostic tests. Stat in Med 1987;6:411 -423
16. Knatterud GL. Comment. Control Clin Trials 1996;17:285 -293[Medline]
17. Wright S, Neill K. Using the World Wide Web for research data collection. Clin Excell Nurse Pract 1999;3:362 -365[Medline]
18. eDict Systems, Inc. eDict Systems, Inc. Web site. Available at: http://www.edictation.com. Accessed April 18, 2001

CiteULike

Complore

Connotea

Del.icio.us

Digg

Reddit

Technorati

This article has been cited by other articles:

				G. T. Sica Bias in Research Studies Radiology, March 1, 2006; 238(3): 780 - 789. [Abstract] [Full Text] [PDF]

This Article Full Text (PDF) Alert me when this article is cited Alert me if a correction is posted Services Email this article to a friend Similar articles in this journal Similar articles in PubMed Alert me to new issues of the journal Download to citation manager Citing Articles Citing Articles via HighWire Citing Articles via Google Scholar Google Scholar Articles by Crewson, P. E. Articles by Applegate, K. E. Search for Related Content PubMed PubMed Citation Articles by Crewson, P. E. Articles by Applegate, K. E. Social Bookmarking                      What's this?