+

DATA
CLEANING

April 2016
 Dealing with messy data

data scrubbing refers to the process of detecting,

Table of Contents
correcting, replacing, modifying or removing messy
data from a record set, table, or database.
Introduction .......................................................................... 0
A. The Data Cleaning Process .......................................... 0 This document provides guidance for data analysts to
find the right data cleaning strategy when dealing with
B. Sources of Error .............................................................. 1 needs assessment data. The guidance is applicable to
C. First Things First ............................................................ 2 both primary and secondary data. It covers situations
where:
D. Screening Data ............................................................... 2
 Raw data is generated by assessment teams using
E. Diagnosing Data ............................................................. 4 a questionnaire.
F. Treatment of Data .......................................................... 4  Data is obtained from secondary sources
(displacement monitoring systems, food security
G. Missing Values ............................................................... 5
data, census data, etc.)
H. Documenting Changes ................................................. 6  Secondary data is compared or merged with the
I. Adapt Process .................................................................. 7 data obtained from field assessments

J. Recoding Variables ........................................................ 7

This document complements the ACAPS technical
K. Quality Control Procedures .......................................... 9 note on How to approach a dataset which specifically
L. Data Integration ........................................................... 10 details data cleaning operations for primary data
entered into an Excel spreadsheet during rapid
M. Key Principles for Data Cleaning ............................. 10
assessments.
N. Tools and Tutorials for Data Cleaning .................... 11
O. Sources and Background Readings ........................ 11 A. The Data Cleaning Process
Annex 1 – Checklist for Data Cleaning ........................ 13
Data cleaning consists primarily in implementing error
Annex 2 – Sample Job Description .............................. 15
prevention strategies before they occur (see data
quality control procedures later in the document).
Introduction However, error-prevention strategies can reduce but
not eliminate common errors and many data errors
will be detected incidentally during activities such as:
No matter how data are collected (in face-to-face
interviews, telephone interviews, self-administered  When collecting or entering data
questionnaires, etc.), there will be some level of error.  When transforming/extracting/transferring data
Messy data” refers to data that is riddled with  When exploring or analysing data
inconsistencies. While some of the discrepancies are  When submitting the draft report for peer review
legitimate as they reflect variation in the context,
others will likely reflect a measurement or entry error. Even with the best error prevention strategies in place,
These can range from mistakes due to human error, there will still be a need for actively and systematically
poorly designed recording systems, or simply because searching for, detecting and remedying
there is incomplete control over the format and type errors/problems in a planned way.
of data imported from external data sources. Such
discrepancies wreak havoc when trying to perform Data cleaning involves repeated cycles of screening,
analysis with the data. Before processing the data for diagnosing, treatment and documentation of this
analysis, care should be taken to ensure data is as process. As patterns of errors are identified, data
accurate and consistent as possible. collection and entry procedures should be adapted to
correct those patterns and reduce future errors.
Used mainly when dealing with data stored in a
database, the terms data validation, data cleaning or
 Dealing with messy data

The four steps of data cleaning: execution (i.e., misuse of instruments, bias, etc.).
Identifying and solving such inconsistencies goes
beyond the scope of this document. It is
recommended to refer to the ACAPS Technical Brief
How sure are you? to get an understanding of how to
deal with measurement errors.

Data entry error: "Data entry" is the process of

transferring information from the medium that
records the response (traditionally responses written
on printed questionnaires) to a computer application.
Under time pressure, or for lack of proper supervision
or control, data is often corrupted at entry time.

Main errors include

Adapted from Van den Broeck J, Argeseanu Cunningham S, Eeckels

R, Herbst K (2005) and Arthur D. Chapman

Screening involves systematically looking for suspect

features in assessment questionnaires, databases, or
analysis datasets.

The diagnosis (identifying the nature of the defective

data) and treatment (deleting, editing or leaving the
data as it is) phases of data cleaning requires an in
depth understanding of all types and sources of errors
possible during data collection and entry processes.

Documenting changes entails leaving an audit trail of

errors detected, alterations, additions and error
Adapted from Kim et Al, 2003; Aldo Benini 2013
checking and will allow a return to the original value if
required.  An erroneous entry happens if, e.g., age is
mistyped as 26 instead of 25.
B. Sources of Error  Extraneous entries add correct, but unwanted
information, e.g. name and title in a name-only
After measurement, data are the object of a sequence field.
of typical activities: they are entered into databases,  Incorrectly derived value occurs when a function
extracted, transferred to other tables, edited, selected, was incorrectly calculated for a derived field (i.e.
transformed, summarized, and presented. It is error in the age derived from the date of birth).
important to realise that errors can occur at any stage  Inconsistencies across tables or files occur e.g.
of the data flow, including during data cleaning itself. when the number of visited sites in the province
Many of the sources of error in databases fall into one table and the number of visited sites in the total
or more of the following categories: sample table do not match.

Measurement errors: Data is generally intended to A large part of the data entry errors can be prevented
measure some physical process, subjects or objects, by using an electronic form (e.g. ODK) and conditional
i.e. the waiting time at the water point, the size of a entry.
population, the incidence of diseases, etc. In some
cases, these measurements are undertaken by human Processing errors: In many settings, raw data are pre-
processes that can have systematic or random errors processed before they are entered into a database.
in their design (i.e., improper sampling strategies) and This data processing is done for a variety of reasons:
1
 Dealing with messy data

to reduce the complexity or noise in the raw data, to duplications by  Data-cleaning errors
analyst
aggregate the data at a higher level, and in some
cases simply to reduce the volume of data being Adapted from Van den Broeck J, Argeseanu Cunningham S, Eeckels
stored. All these processes have the potential to R, Herbst K (2005)
produce errors.
Inaccuracy of a single measurement and data point
Data integration errors: It is rare for a database of may be acceptable, and related to the inherent
significant size and age to contain data from a single technical error of the measurement instrument.
source, collected and entered in the same way over Hence, data cleaning should focus on those errors
time. Very often, a database contains information that are beyond small technical variations and that
collected from multiple sources via multiple methods produce a major shift within or beyond the analysis.
over time. An example is the tracking of the number of Similarly, and under time pressure, consider the
people affected throughout the crisis, where the diminishing marginal utility of cleaning more and more
definition of “affected” is being refined or changed compared to other demanding tasks such as analysis,
over time. Moreover, in practice, many databases visual display and interpretation.
evolve by merging other pre-existing databases. This
merging task almost always requires some attempt to  Understand when and how errors are produced
resolve inconsistencies across the databases during the data collection and workflow.
involving different data units, measurement periods,  Resources for data cleaning are limited.
formats etc. Any procedure that integrates data from Prioritisation of errors related to population
multiple sources can lead to errors. The merging of numbers, geographic location, affected groups and
two or more databases will both identify errors (where date are particularly important because they
there are differences between the two databases) and contaminate derived variables and the final
create new errors (i.e. duplicate records). Table 1 analysis.
below illustrates some of the possible sources and
types of errors in a large assessment, at three basic The following sections of this document offer a step
levels: When filling the questionnaire, when entering by step approach to data cleaning.
data into the database and when performing the
analysis. C. First Things First
Table 1: Sources of data error
The first thing to do is to make a copy of the original
Sources of error
data in a separate workbook and name the sheets
Stage Lack or excess of Outliers and
data inconsistencies
appropriately, or save in a new file.

Measurement  Form missing  Correct value filled out ALWAYS keep the source files in a separate folder and
 Form double, in the wrong box
change its attribute to READ-ONLY, to avoid
collected repeatedly  Not readable
 Answering box or  Writing error modification of any of the files.
options left blank  Answer given is out of
 More than one option expected (conditional)
selected when not
allowed
range D. Screening Data
Entry  Lack or excess of  Outliers and To prepare data for screening, tidy the dataset by
data transferred inconsistencies carried
from the over from transforming the data in an easy to use format.
questionnaire questionnaire Within a tidied dataset:
 Form of field not  Value incorrectly
entered entered, misspelling - Fonts have been harmonised
 Value entered in  Value incorrectly - Text is aligned to the left, numbers to the right
wrong field changed during
- Each variable has been turned into a column and
 Inadvertent deletion previous data cleaning
and duplication  Transformation each observation into a row.
during database (programming) error - There are no blank rows
handling
Processing  Lack or excess of  Outliers and - Column headers are clear and visually distinct.
and Analysis data extracted from inconsistencies carried - Leading spaces have been deleted
the database over from the database
 Data extraction,  Data extraction, coding
Afterwards, examine data for the following possible
coding or transfer or transfer error errors:
error  Sorting errors
 Deletions or (spreadsheets)
2
 Dealing with messy data

 Spelling and formatting irregularities: are

categorical variables written incorrectly? Is the However, screening methods are not only statistical.
date format consistent? For numeric fields, are all Many outliers are detected by perceived non-
of the values numbers? Etc. conformity with prior expectations or the norm. This is
 Lack of data: Do some questions have far fewer for instance based on the analyst's experience, results
answers compared to others? from secondary data review, numerical constraints or
 Excess of data: Are there duplicate entries or plane common sense (weight cannot be negative,
more answers than originally allowed? people cannot have more than 2 parents, women
 Outliers/inconsistencies: Are there values that are cannot bear 35 children, etc.).
so far beyond the typical distribution that they
seem potentially erroneous? A particular problem is that of erroneous inliers, i.e.,
 Remarkable patterns: Are there patterns that data points generated by error but falling within the
suggest that the respondent or enumerator has expected range. Erroneous inliers will often escape
not answered or recorded questions honestly? (i.e. detection. Detection approaches include:
several questionnaires with the exact same  Viewing data in relation to other variables, using
answers)? multivariate views, such as scatter plots or
 Suspect analysis results: Do the answers to some heatmap.
questions seem counterintuitive or extremely  More advanced and resource intensive techniques
unlikely? involves regression analysis, consistency/
plausibility checks (examining the history of each
Common mistakes in needs assessments data point or comparing to a similar location) or by
 Misspelling of place names, particularly when re-measurement. However, under time pressure,
translating between different alphabets (e.g. such examination is rarely feasible. Instead, one
Arabic to English) can examine and/or re-measure or do further
 Use of inconsistent date formats inquiry about a sample of inliers to estimate an
 Totals differ from the results of disaggregate error rate.
questions (e.g. total number of household
members is not matching the aggregation of a Useful screening methods, from simpler to more
different question where respondents are asked complex, are:
to list household members by age and gender)  Screening of columns after sorting
 Values are outside of the acceptable range for  Use summary statistics
values for that question, including negative values  Validated and/or double data entry
in fields that can only have positive values (e.g.  Printouts of variables not passing range checks
price of bread) and of records not passing consistency checks
 Unclear cause of missing data  Frequency distributions and cross-tabulations.
 Merging of datasets with different units of  Graphical exploration of distributions: box plots,
measurement (e.g. different interpretations of the histograms, and scatter plots using visual analysis
term household) or administrative boundaries. software such as Tableau desktop
 In case of multiple choice questions: selecting  Plots of repeated measurements on the same
‘other, please specify’ for a variable that is one of individual, i.e., growth curves
the multiple choice options.  Checking of questionnaires using fixed algorithms
 Malfunctioning skip patterns  Statistical outlier detection.
 Overall lack of consistency within the answers
provided by a respondent (e.g. the response to  In many, if not most instances, data can only be
questions on main needs is not in line with sector cleaned effectively with some human involvement.
specific questions). Know (and/or train data cleaners) common
mistakes and what errors to look for.
Data cleaning can be partly automated through  Exploratory data analysis and data visualization
statistical software packages Descriptive statistic are two main ways to detect data errors.
tools can for instance be used during the screening  Different types of errors call for different detection
phase to predefine expectations, assumptions or approaches – a spell check will recognise
criteria about normal ranges, distribution shapes, and misspelled categorical variables while statistical
strength of relationships. This can facilitate the outlier detection helps identification of extreme
flagging of dubious data, patterns, or results. values.
3
 Dealing with messy data

 Collect additional information, i.e., question the

E. Diagnosing Data
enumerator about what may have happened and, if
possible or necessary, repeat the measurement.
The identification or highlighting of an error is
Such procedures can only happen if data cleaning
followed by diagnosis – finding the cause for this
starts soon after data collection.
error. To clarify suspect data, review all of a
respondent’s answers to determine if the data makes
The diagnostic phase is labour intensive and the
sense within the context. Sometimes it is necessary to
budgetary, logistical, time and personnel requirements
review a cross-section of different respondents’
are typically underestimated or even neglected at the
answers, to identify issues such as a skip pattern that
design stage. Fewer resources are required if
was specified incorrectly.
conditional data entry is used (e.g. through electronic
forms) and if it starts early in data collection process.
There are a multitude of possible diagnoses for each
suspected data point:
 Use common sense, experience, triangulation and
 Missing data: Answers omitted by the respondent
lessons learnt to diagnose the type of error.
(nonresponse), questions skipped by the
 Design your questionnaire form carefully to allow
enumerator or dropout
cross checks between questions.
 Errors: Typos or answers that indicate the question
 Consider the collection of additional information
was misunderstood.
from enumerator to understand the cause of errors
 True extreme: An answer that seems high but can
(i.e. debriefings)
be justified by other answers (i.e., the respondent
working 60 hours a week because they work a full-
time job and a part-time job) F. Treatment of Data
 True normal: A valid record.
 No diagnosis, still suspect: Make a judgment call After identification of missing values, errors, and true
on how to treat this data during the treatment (extreme or normal) values, analysts must decide
phase. what to do with problematic observations:
 Leave it unchanged: The most conservative course
Some data values are clearly logically or biologically of action is to accept the data as a valid response
impossible (men cannot be pregnant; the price of and make no change to it. The larger the sample
bread cannot be negative). Pre-defined cut-off points size, the less one suspect response will affect the
immediately help to detect this type of error. analysis; the smaller the sample size, the more
Sometimes the suspected value falls within the difficult the decision.
acceptable range and the diagnosis is less  Correct the data: If the respondent’s original intent
straightforward. In these cases, it is necessary to can be determined, correct the answer (i.e. after
apply a combination of diagnostic procedures: discussing with the enumerator, it is clear that the
 Go back to previous stages of the data flow to see respondent meant the lack of income instead of
whether a value is consistently the same. This too much income).
requires access to well-archived and documented  Delete the data? The data seems illogical and the
data with justifications for any changes made at value is so far from the norm that it will affect
any stage. descriptive or inferential statistics. What to do?
 Look for information that could confirm the true Delete just this response or delete the entire
extreme status of an outlying data point. For record? Remember that whenever data is deleted,
example, a very low score for weight-for-age (i.e., there is a risk of consciously or subconsciously
−6 Z-scores) might be due to errors in the “cherry picking” the data to obtain the preferred
measurement of age or weight, or the subject may results. To understand the impact of deleting a
be extremely malnourished, in which case other data point, a binary variable can be created (
nutritional variables should also have extremely 1=suspicious record, 0=not suspicious). This new
low values. This type of procedure requires insight variable can be used as a record filter in Pivot
into the coherence of the variables. This insight is tables or in-table filtering to understand the impact
usually available from experience or lessons learnt of potentially erroneous data in the final results.
and can be used to plan and program data  If time and resources allow, re-measure the suspect
cleaning. or erroneous values.

4
 Dealing with messy data

There are some general rules to support a decision on

G. Missing Values
how to treat the data:
 If the person undertaking data entry has entered
Missing values require particular attention. The first
values different from the ones in the questionnaire,
thing is to decide which blank cells need to be filled
the value should be changed to what was recorded
with zeros (because they represent genuine negative
in the questionnaire form. (I.e. the value in the
observations, such as "no", "not present", "option not
questionnaire was 40,000 and the data entry
taken", etc.) and which to leave blank (if the
operator keyed in 4,000 – a zero was left out).
convention is to use blanks for missing or N/A for “not
 When variable values do not make sense, if there is
applicable”). Some analysts replace blank cells with
no data entry error, and there are no notes to help
some explicit missing value code (e.g. using 999 to
determine where the error comes from, leave the
indicate a “do not know”).
data as it is. By changing the value into a more
What to do with those cells remaining blank? Missing
reasonable result, significant bias is introduced
values can be classified as either random or non-
and there is no justification for changing it. The
random:
case should be listed as an outlier (i.e. by using
 Random missing values may occur because the
conditional formatting for instance).
subject inadvertently did not answer some
 When blank cells are found or the record was
questions. The assessment may be overly complex
required even though key informants may not have
or too long, or the enumerator may be tired or not
that type of data or duplicate records were entered,
paying sufficient attention, and miss the question.
then cases must be deleted from the data file.
Random missing values may also occur through
 Impossible values are never left unchanged, but
data entry mistakes. If there are only a small
should be corrected if a correct value can be found,
number of missing values in the dataset (typically,
otherwise they should be deleted. For biological
less than 5%), then it is extremely likely to be a
continuous variables, some within-subject variation
random missing value.
or small measurement variation could be present.
 Non-random missing values may occur because
If a re-measurement is done very rapidly after the
the key informant purposefully did not answer
initial one and the two values are close enough to
some question. This for instance occurs if the
be explained by variation alone, take the average of
question is confusing, not appropriate or perceived
both as the final value.
as sensitive. The missing data is related to one or
 With true extreme values and values that are still
more characteristics of the respondent – e.g. if
suspect after the diagnostic phase, the analyst
female are more likely to refuse a question on the
should examine the influence of such data points,
level of income compared to male respondents.
individually and as a group, on analysis results
before deciding whether or not to leave the data
The default option for dealing with missing values is
unchanged.
to filter and exclude these values from analysis:
 To limit the impact of outliers and extreme values
 Listwise / casewise deletion: All cases (e.g. a
analysts can decide to present the median. This is
respondent) that have missing values are
acceptable as long as clearly explained within the
excluded. If only one variable is analysed, listwise
findings.
deletion is simply analysing the existing data.
 Some authors have recommended that true
When analysing multiple variables, then listwise
extreme values should always stay in the analysis.
deletion removes cases if there is a missing value
In practice, many exceptions are made to that rule.
on any of the variables. The disadvantage is the
The investigator may not want to consider the
loss of data that occurs as all data is removed for
effect of true extreme values if they result from an
a single case, even if some questions were
unanticipated extraneous process. This becomes
answered.
an “a posteriori” exclusion criterion. The data
 Pairwise deletion: Unlike listwise deletion which
points should be reported as “excluded from
removes cases that have missing values on any of
analysis” in the methodology chapter of the final
the variables under analysis, pairwise deletion only
report.
removes the specific missing values from the
analysis (not the entire case). In other words, all
available data is included. When conducting a
correlation on multiple variables, this technique
enables a bivariate correlation between all

5
 Dealing with messy data

available data points, and ignores only those recommended to refer to a statistician in case of a
missing values if they exist on some variables. In small dataset with a large number of missing values.
this case, pairwise deletion will result in different
sample sizes for each variable. Pairwise deletion is Pragmatically, for needs assessment with few
useful when the sample size is small or if missing statistical resources, creating a copy of the variable
values are large because there are not many and replacing missing values with the mean or
values to start with. median may often be enough and preferable to losing
cases in multivariate analysis from small samples.
Try conducting the same test using both deletion
methods to see how the outcome changes. Note that  There are several methods to deal with missing
in these techniques, "deletion" means exclusion within data, including deleting cases with missing values,
a statistical procedure, not deletion (of variables or imputing and the maximum likelihood approach.
cases) from the dataset. However, providing an explanation on why data
are missing ("women could not be interviewed",
A second option is to delete all cases with missing “the last questionnaire section could not be filled
values. Thus, you are left with complete data for all due to lack of time”) may be much more
cases. The disadvantage to this approach is that the informative to end user’s than a plethora of
sample size of the data is reduced, resulting in a loss statistical fixes.
of statistical power and increased error in estimation  Set up a dummy variable with value 0 for those
(wider confidence intervals). It can also affect the who answered the question and value 1 for those
representativeness of a sample: after removing the who did not. Use this variable to show the impact
cases with non-random missing values from a small of different methods.
dataset, the sample size could be insufficient. In  Look for meaning in non-random missing values.
addition, results may be biased in case of non-random Maybe the respondents are indicating something
missing values. The characteristics of cases with important by not answering one of the questions.
missing values may be different than the cases
without missing values.

Another option is imputation: to replace the missing

H. Documenting Changes
values. This technique preserves all cases by
replacing missing data with a probable value based on Documentation of error, alterations, additions and
other available information. A simple procedure for error checking is essential to:
imputation is to replace the missing value with the  Maintain data quality
mean or median. Hot-deck imputation replaces  Avoid duplication of error checking by different
missing values with the value for that same variable data cleaners.
taken from a complete record for a similar person in  Recover data cleaning errors
the same dataset. Once all missing values have been  Determine the fitness of the data for use.
imputed, the data set can then be analysed using  Inform users who may have used the data knowing
standard techniques for complete data. However, this what changes have been made since they last
method can also bias results and p-values. accessed the data

Under certain conditions, maximum likelihood Create a change log within the workbook, where all
approaches have also proven efficient to dealing with information related to modified fields is sourced. This
missing data. This method does not impute any data, will serve as an audit trail showing any modifications,
but rather uses all the data available for the specific and will allow a return to the original value if required.
cases to compute maximum likelihood estimates. Within the change log, store the following information:
 Table (if multiple tables are implemented)
Detailing technicalities, appropriateness and validity of  Column, Row
each techniques goes beyond the scope of this  Date changed
document. Ultimately, choosing the right technique  Changed by
depends on how much data are missing, why this data  Old value
is missing, patterns, randomness and distribution of  New value
missing values, the effects of the missing data and  Comments
how the data will be used for analysis. It is strongly

6
 Dealing with messy data

 Make sure to document what data cleaning steps  Recoding a categorical variable (e.g. ethnicity,
and procedures were implemented or followed, by occupation, an “other” category, spelling
whom, how many responses were affected and for corrections, etc.).
which questions.  Recoding a continuous variable (e.g. age) into a
 ALWAYS make this information available when categorical variable (e.g. age group).
sharing the dataset internally or externally (i.e. by  Combining the values of a variable into fewer
enclosing the change log in a separate worksheet) categories (e.g. grouping all problems caused by
access constraints).
 Combining several variables to create a new
I. Adapt Process variable (e.g., the food consumption score, building
an index based on a set of variables).
 Defining a condition based on certain cut-off points
Once errors have been identified, diagnosed, treated
(e.g., population “at risk” vs. “at acute risk”).
and documented and if data collection/entry is still
 Changing a level of measurement (e.g. from
ongoing, the person in charge of data cleaning should
interval to ordinal scale).
give instructions to enumerators or data entry
operators to prevent further mistakes, especially if
Conceptually, a distinction is needed between:
they are identified as non-random. Feedback will
 Activities related to recoding qualitative data: i.e.
ensure common errors are not repeated and will
responses to open questions
improve the assessment validity and the precision of
 Activities that include transforming and deriving
outcomes. Main recommendations or corrections can
include: new values out of others, such as creating
calculations (i.e. percentage), parsing, merging, etc.
 Programming of data capture, data
Here, the analyst is re-expressing what the data
transformations, and data extractions may need
says (i.e. re-expressing deviation as a % change,
revision.
weighted or moving average, etc.). The data has
 Corrections of questions in the questionnaire form.
(normally) already gone through a cleaning stage
 Amendment of the assessment protocol, design,
before to be transformed.
timing, enumerators training, data collection, and
quality control procedures.
For both types, recoding variables or values can serve
 In extreme cases, it may be necessary to re-
both the purpose of cleaning dirty data and/or
conduct some field assessment (few sites) or
transforming clean data. This section focuses
contact again key informants or enumerators to
primarily on the former rather than the re-expression
ask additional information or more details or
of values which will be tackled more extensively in
confirm some records.
another chapter of the data handbook on data
transformation.
 Data cleaning often leads to insight into the nature
and severity of error-generating processes.
Recoding categorical variables starts with a full listing
 Identify basic causes of errors detected and use
of all variants generated by a variable, together with
that information to improve data collection and the
their frequencies. The variant list can be copied into a
data entry process to prevent those errors to re-
new sheet, to create a table of variants and their
occurring.
desired replacements. ALWAYS keep a copy of the
 Reconsider prior expectations and/or review or
original values, and try out different recoding schemes
update quality control procedures. before settling on a final one.

J. Recoding Variables There are three ways to recode categorical data: 1.

Collapse a categorical variable into fewer categories
Variables might need to be recoded to create new
ones that fit within the analytic needs. The following What is the current occupation of the head of
recoding for instance is common: household?
 Formatting: date (day, month, and year), pre-fixes
to create better sorting in tables Original categories Recoded categories
 Rounding continuous variables Government employee Agriculture
 Syntax: Translation, language style and Agriculture and livestock Non-Agriculture
simplification. Commerce
7
 Dealing with messy data

Student Recoding variables can be tedious. The conceptual

…etc. effort needed in order to produce a meaningfully
recoded category set is often underestimated. Care
Guidelines for collapsing data are as follows: must be taken to evaluate the combined category
 Ordinal variables need to be collapsed in a method sets, to absorb excessive, incoherent or rarely used
that preserves the ordering of categories. categories into broader ones, and to be clear about
 Combine only categories that go together. Do not the rationale for the final number and content of
combine two logically distinct categories just to distinct categories. Also, be aware that any recoding
eliminate categories with small numbers (e.g. lack that reduces the number of categories entails some
of access due to lack of income and lack of access information loss. As in all stages of data analysis,
due to insecurity) as interpretation of data analysts must be alert for errors.
becomes difficult or meaningless.
 The way in which categories are collapsed can Basic tips for effective recoding include:
easily affect the significance level of statistical  Use distinct and easy to remember variable names.
tests. Categories should be collapsed to avoid the Never use the same variable name to denote both
criticism that the data were manipulated just to get the transformed and untransformed variable. For
a certain result. This does not mean there needs to large data sets, a systematic way to name
be a decision before collecting the data (if this was variables is desirable.
the case, there would not be a need to collect  Pay attention to missing values. When recoding is
separate categories). done, the number of cases with missing data
 Do not oversimplify the data. Unnecessary should be the same as before recoding. A check
reduction in the number of categories may reduce that this is so will often be the first clue that
statistical power and obscure relationships in the recoding was in error. A safe procedure is to start
data. As a general guideline, keep intact any the recoding process by setting the new variable to
categories that include 10% or more of your data missing for all cases, and then changing missing
(or 5 cases, for very small samples). values only for those with data on the initial
variables to be recoded. For complicated recoding,
Breaking: There are several reasons for breaking a check a few individual values by hand to make sure
categorical variable into several smaller variables: they were recoded properly, and check the
 Data was collected in a manner easy to collect to distribution of values.
ease the burden of data collection on the subject.  Use graphs to check the accuracy of recoding.
For example, it is easier for the key informant to Recoding is a systematic translation of data
provide a list of issues than to review a long list of values, so scatterplots of raw data vs. recoded
problems. data should show highly organized patterns
 A variable may contain more than one "concept." reflecting the recoding system. Histograms can
For example, consider the ordinal variable show whether your data is now more normally
“severity” below: distributed.
 Use variable codes consistently. For example, with
1. There are no shortages dichotomous "yes/no" variables, always use 0 = no
2. A few people are facing shortages and 1 = yes. For polychotomous variables, always
3. Many people are facing shortages make 0 the reference category/default option.
4. Shortages are affecting everyone  Keep a permanent record of your recoding. For data
entry errors, make the changes directly in the raw
This variable contains two concepts, “shortages” and data file, because there is no need to maintain the
“number of people affected”. It is straightforward to erroneous data points. Recoding should be
code two new variables, shortages (0 = no shortages, undertaken in a separate file, as there might be a
1 = shortage) and number of people (0 = no people, I= need to review the initial data. Most statistical
Few people, 2=Many people, 4= All of them). programs save data in a specially formatted file,
and this file is the one to change. The recoding
Combining is the reverse process of breaking up, such commands should all be put in one program (a do
as combining “shortages” and “number of people” file) that can be executed again. The do file serves
back into the variable “severity”. as a permanent record as well.

8
 Dealing with messy data

prescribed minimum and maximum values (such

K. Quality Control Procedures
as 0 to 120 years for age and should always be
expressed as integer of years, with rules for
When deciding upon an approach to data cleaning, it
rounding up or down for infants).
is useful to consider the different types of errors
 Reference data check are used when the data from
which can be made, and to plan at what point in the
two or more closely related fields can be checked
data workflow prevention measures should be
against external reference tables, i.e. when the
implemented. Best practices include:
recorded values for height, weight and age are
checked against the World Health Organization’s
Roles and responsibilities: Make sure the staff with
standard reference tables.
responsibilities regarding data quality are aware of the
 Skip checks verify whether the skip patterns have
cleaning protocols (see annex 1 for a complete
been followed appropriately. For example, a simple
checklist for needs assessments). Roles and
check verifies that questions to be asked only of
responsibilities related to error detection and
schoolchildren are not recorded for a child who
correction should be clearly defined and
answered "no" to an initial question on school
communicated as part of the job descriptions (see
enrolment.
Annex 2), at each stage of the data collection, entry
and processing.
 Consistency checks verify that values from one
question are consistent with values from another
Ensure that a second pair of eyes review and compare
question, for example, the date of birth and age of
the original data to data entered. Data cleaning should
a given individual.
start in the field (field editing) alongside data
 Typographical checks limit, for instance, the
collection, as questionnaires are reviewed by
transposition of digits like entering “14” rather than
supervisors or field editors on a daily basis. Similarly,
“41” in a numerical input. Such a mistake for age
during data entry, double checks should be
might be caught by consistency checks with
mandatory, especially when:
marital status or family relation. Control totals, for
 There is a process of translation at data entry, to
instance, can significantly reduce typographical
ensure consistency/accuracy of translation.
errors.
 Data entry is distributed across various field
locations and consolidation occurs in a different
 Document the data quality rules to follow, where
location.
focus should be given, and how to solve
errors/issues. Plan double checks.
At the data entry stage, computer-assisted quality
 Communicate clear instructions to enumerators,
control procedures should be used. Additional
team leaders, data entry clerk, at all relevant stages
functionality can be added in the data entry software
of the data flow.
(i.e. Excel, Survey Gizmo, Access, SPSS, STATA, etc.)
 Ensure that data entry staff are familiar with the
to highlight rule violations (null codes, conditional
questionnaire filling procedures, so that mistakes
formatting, etc.) and prevent mistakes (i.e. drop down
can be identified early on and verified/rectified (i.e.
menus). The decision to include those rules in the
rules such as ‘pick only three’ or ‘must add to 100
database must be pragmatic, weighing up the merits
%’).
of having errors detected and rectified by data entry
 Design a data-cleaning plan, including:
staff, versus the time required to set this up and to
- Budget, timeframe and staff requirements.
quickly make necessary adjustments if the initial
- Screening tools.
setup does not work as expected.
- Diagnostic procedures used to discern errors (on
going periodic basis and towards the end of the
Five kind of checks can be automated:
assessment).
 Range checks ensure that every variable in the
- Instructions or training to enumerators and data
survey contains only data within a limited domain
entry staff in case of protocol violation and
of valid values. Categorical variables can have only
consistency check.
one of the values predefined for them on the
- Decision rules that will be applied in the editing
questionnaire (for example, gender can be coded
phase.
only as “1” for males or “2” for females);
chronological variables should contain valid dates,
and numerical variables should lie within

9
 Dealing with messy data

proceed rigorously and methodically to screening,

L. Data Integration
diagnosing and treatment.

A different set of problems can occur when datasets

are integrated or merged with other data. Analysts do M. Key Principles for Data
not always have control over the format and type of
Cleaning
data that is imported from an external data source,
such as a database, text file, or a Web page. Most
Key principles for cleaning data in spreadsheets are
common problems are
as follows:
 Formats: Dates are especially problematic
1. Create a backup copy of the original data in a
(26/02/1977, 26 February 1977, 26-02-1977, etc.).
separate workbook.
Analysts also need to be aware that different
2. Regularly back-up the working file, at successive
applications store dates internally in different
points during collating, cleaning and analysing.
ways. A simple copy-paste from one application to
Save documents with file names that combine
another will thus cause errors across the board.
date and time (yymmdd-time prefixed allow for
 Units: Different metrics are used: litre, gallons,
files to be sorted by order of creation).
gourdes, etc.
3. When integrating or merging different datasets,
 Ranges: Age intervals might differ from one survey
ensure that the data is in a tabular format of rows
to another. Only if the birth date is available, the
and columns with: similar data in each column, all
age interval can be recreated.
columns and rows visible, and no blank rows
 Inconsistency: When merging different data source,
within the range. Check that there are no subtotals,
conflicting information can emerge. Analysts must
totals or other calculated records down the
choose between using both, using the most
columns. Calculated variables can remain on the
recently updated information, the most trusted
right of the data.
source, investigate further or use neither. Duplicate
4. Format the database for readability and easy
records should generally not be deleted, but
navigation: Left align text, number right aligned,
flagged so that they can be identified and excluded
variable title is positioned horizontally, text
from analysis in cases where duplicate records
variables fully visible, column separated by bold
may bias an analysis. While appearing to be
lines, header with background colours, numbers
duplicates, in many cases the records in the two
comma separated every 3 digits, etc.
databases may include some information that is
5. Start with tasks that do not require column such as
unique to each. Just deleting one of the duplicates
spell-checking or using the Find and Replace
(“merge and purge”) is consequently not always a
function.
good option as it can lead to valuable data loss.
6. Afterwards, undertake tasks that do require
 Spelling: Categorical variables and specifically
column manipulation. The general steps for
place names, may have different spellings.
manipulating a column are:
 Loss of bits of data: Some pieces of data, columns
 Insert a new column (B) next to the original
or rows are lost when extracted, i.e. when web
column (A) that needs cleaning.
scrapping or extracting from a pdf (good luck!).
 Transform the data in the column (B).
 Remove the original column (A), which converts
 Data is dirty. Live with it. Analysts assuming that
the new column from B to A.
raw data comes clean and bypassing basic checks
7. Keep the questionnaire close. As each check is
live dangerously.
done, a list of issues will be produced. The
 Check the dataset documentation available. If not
questionnaires should be consulted to double
available (even after request), DO NOT TRUST THE
check or identify the problems.
DATA, even if the source is generally reliable. Start
8. When checking for one type of problem for one site
checking for quality.
or key informant, verify that the data for the other
 Even under time pressure, take the time to screen
variables for that case have been entered correctly.
the data for 15-30mn, focusing first on spelling and 9. Look at the values in all the variables and all the
formatting of the datasets to be merged, then cases for that site, key informant or enumerator.
review outliers (use filters for quick visual Occasionally the data entry person will skip a
detection). If no mistakes are spotted during this variable or a key in the values from the previous
time interval, it is probably of good quality and variable or the subsequent variable, and all the
usable as it is. If mistakes are detected, then

10
 Dealing with messy data

data that have been entered after will not be  Removing duplicate rows
correct.  Finding and replacing text
 Changing the case of text
 Planning and budgeting for data cleaning is  Removing spaces and nonprinting characters from
essential. text
 Organizing data improves efficiency, i.e. by sorting  Fixing numbers and number signs
data on location or records by enumerator.  Fixing dates and times
 Prevention is better than cure. It is far more  Merging and splitting columns
efficient to prevent an error than to have to find it  Transforming and rearranging columns and rows
and correct it later.  Reconciling table data by joining or matching
 The responsibility for generating clean data  Third-party providers
belongs to everyone, enumerators, custodian and
users. Openrefine (ex-Google Refine) and LODRefine are
 Prioritisation reduces duplication. Concentrate on powerful tools for working with messy data, cleaning
those records where extensive data can be cleaned it, or transforming it from one format into another.
at the lowest cost or that are of most value to end Videos and tutorials are available to learn about the
users. different functionalities offered by this software. The
 Feedback is a two-way street: data users or analyst facets function is particularly useful as it can very
will inevitably carry out error detection and must efficiently and quickly gives a feel for the range of
provide feedback to data custodians. Develop variation contained within the dataset.
feedback mechanisms and encourage users to
report errors. Detailed data cleansing tutorials and courses are also
 Education and training improve techniques: Poor available at the school of data:
training of enumerators and data entry operators is  http://schoolofdata.org/handbook/recipes/cleanin
the cause of a large proportion of the errors. Train g-data-with-spreadsheets/
on quality requirements (readability, etc.) and  http://schoolofdata.org/handbook/courses/data-
documentation, cleaning/
 Data cleaning processes need to be transparent
and well documented with a good audit trail to Two specialized tools to accomplish many of these
reduce duplication and to ensure that once tasks are used in ACAPS. The first one is Trifacta
corrected, errors never re-occur. Wrangler, the new version of Data Wrangler by the
 Documentation is the key to good data quality. Stanford Visualization Group. Trifacta Wrangler is a
Without good documentation, it is difficult for user-friendly tool that can automatically find patterns
users to determine the appropriateness for use of in the data based on things selected, and
the data and difficult for custodians to know what automatically makes suggestions of what to do with
and by whom data quality checks have been those patterns. Beautiful and useful. The other
carried out. cleaning star software is Data monarch from
Datawatch, integrating a lot of wrangling, cleaning and
N. Tools and Tutorials for Data enrichment functionalities that can take hours in
Microsoft excel.
Cleaning
O. Sources and Background
Spreadsheets such as Excel offer the capability to
easily sort data, calculate new columns, move and Readings
delete columns, and aggregate data. For data cleaning
of humanitarian assessment data, ACAPS developed  ACAPS. 2013. How to Approach a Dataset –
a specific Technical Note providing a step by step Preparation. Available at:
approach in Excel and detailing cleansing operations, http://www.acaps.org/resourcescats/downloader
supported by a demo workbook. /how_to_approach_a_dataset_part_1_data_prepar
ation/163/1375434553
For generic instructions about how to use excel And its auxiliary workbook, available at:
formulas, functionalities or options to clean data,
several Microsoft office guidance notes are available:
 Spell checking
11
 Dealing with messy data

http://www.acaps.org/resourcescats/downloader http://fsg.afre.msu.edu/survey/Data_Cleaning_Gu
/how_to_approach_a_dataset_data_management/ idelines_SPSS_Stata_1stVer.pdf
164
 Munoz, J. 2005. A Guide for Data Management of
 ACAPS. 2012. Severity Rating, A Data Management Household Surveys. Santiago, Chile, Household
Note. Sample Surveys in Developing and Transition
http://www.acaps.org/resourcescats/downloader Countries.
/severity_rating_data_management_note/87/1376 http://unstats.un.org/unsd/hhsurveys/
302232
 Osborne, J. W. 2013. Best Practices in Data
 Benini, A.. 2011. Efficient Survey Data Entry – A Cleaning: A Complete Guide to Everything You
Template for Development NGOs. Friends in Need to Do Before and After Collecting Your Data.
Village Development Bangladesh (FIVDB). California, SAGE.
http://aldo-
benini.org/Level2/HumanitData/FIVDB_Benini_Effi  Psychwiki. Retrieved 7 September 2009.
cientDataEntry_110314.pdf Identifying Missing Data.
http://www.psychwiki.com/wiki/Identifying_Missi
 Buchner, D. M. Research in Physical Medicine and ng_Data
Rehabilitation.
http://c.ymcdn.com/sites/www.physiatry.org/res  Psychwiki. Retrieved 11 September 2009. Dealing
ource/resmgr/pdfs/pmr-viii.pdf with Missing Data.
http://www.psychwiki.com/wiki/Dealing_with_Mis
 Chapman, A. D. 2005. Principles and Methods of sing_Data
Data Cleaning – Primary Species and Species-
Occurrence Data.  Psychwiki. Retrieved 7 September 2009. Missing
http://www.gbif.org/orc/?doc_id=1262 Values.
http://www.psychwiki.com/wiki/Missing_Values
 Den Broeck, J. V., Cunningham, S. A., Eeckels, R.,
Herbst, K. 2005. Data Cleaning: Detecting,  Sana, M., Weinreb, A. A. 2008. Insiders, Outsiders,
Diagnosing, and Editing Data Abnormalities. South and the Editing of Inconsistent Survey Data.
Africa, Africa Centre for Health and Population Sociological Methods & Research, Volume 36,
Studies. Number 4, SAGE Publications.
http://www.academia.edu/1256179/Insiders_Out
 Dr. Limaye, N. 2005. Clinical Data Management – siders_and_the_Editing_of_Inconsistent_Survey_D
Data Cleaning. ata

 Henning, J. 2009. Data Cleaning.  The Analysis Institute. 2013. Effectively Dealing
http://blog.vovici.com/blog/bid/19211/Data- with Missing Data without Biasing Your Results.
Cleaning http://theanalysisinstitute.com/missing-data-
workshop/
 Joint IDP profiling Service (JIPS). Retrieved July
2013. Manual Data Entry Staff.  Wikipedia. Retrieved 31 July 2013. Data Cleansing.
http://jet.jips.org/pages/view/toolmap http://en.wikipedia.org/wiki/Data_cleansing

 Kassoff, M. 2003. Data Cleaning.

http://logic.stanford.edu/classes/cs246/lectures/l
ecture13.pdf

 Kim et Al. 2003. A Taxonomy of Dirty Data.

http://sci2s.ugr.es/docencia/m1/KimTaxonomy0
3.pdf

 Michigan State University. 2012. Data Cleaning

Guidelines (SPSS and STATA). 1st Edition.
12
 Dealing with messy data

Review records
Annex 1 – Checklist for Data
 If a sampling strategy was used, the records must
Cleaning be verified first. Verify if all the sites have been
entered, including those where the assessment
Prepare for data cleaning was not completed (this is not relevant in case of
Planning is essential. Make sure tools, material and purposive sampling). Compare records with
contacts for cleaning data are available: assessment teams field trip reports or the
 The questionnaire forms spreadsheet where you tracked the visited
 The contacts of team leaders or enumerators, in locations.
case they need to be contacted for questions  Assign and check a unique ID for each site or
 The original database household).
 A translator, if necessary  Check for duplicate cases as a regular routine for
 Visual analysis software (i.e. tableau public) each of the data rows. Remove any blank cases
 Spreadsheet (excel) or database (Access, Stata, where the key variables have been entered but
etc.) software. there are no data in any of the variables. Verify first
 Some would add coffee and music, and a place that the blank cases should be removed and how
without noise and disturbance. this could affect other data in the row.

Identify the data custodian. He/she will generally be Screen, diagnose and treat data
responsible for managing and storing the data, as well  First clean filter questions, i.e. when the population
as for the supervision of the data cleaning, the is asked if they did or had a particular activity
consolidation of the changes and the update and based on a response (yes/no). In that case there
maintenance of the change log. should be data in the following table in the
questionnaire (or column in the database) if the
Establish, document and communicate response is “yes” or there should be no data if the
 Train the data entry operators on the how the response is “no”.
questionnaire is populated. Explain the instructions  Review the skip rules within the questionnaire and
given to enumerators. If possible, include data run the checks in the database to look for invalid or
entry staff in the data collectors training to missing values in variables based on the skip rules.
facilitate internal communication.  Clean questions with min or max response values
 Establish decision rules for when to change a value (“tick three options only”, what are the top three
and when NOT. priorities among the 5 following choice”, etc.).
 Establish procedures to document data that was  Inspect the remaining variables sequentially and as
modified or not collected, i.e. “missing”, or “not they are recorded in the data file. Create a general
collected”. summary table of descriptive statistics, where for
 Explain how to use the change log file. each variable the min, max, mean, median, sum
and count are available.
 Communicate to data entry operators or analysts
the procedures to be followed and who to inform
of detected errors.
 Establish communication channels for
communicating detected errors. Written
communication is recommended.
 For rapid assessments where data analysis,
mapping and visualization generally coincides with
data entry and cleaning, communicate regularly to Screenshot of summary statistics table from Aldo
analysts, GIS officers and graphic designers which Benini, ACAPS Technical note on how to approach a
parts of the datasets are clean and usable. dataset, preparation
Establish clear reporting procedures in case
additional errors are identified. Plan with the team
 If the variable is a categorical/qualitative variable,
which variables are a priority for cleaning.
check if spelling is consistent and run a frequency
count:

13
 Dealing with messy data

o Look at the counts to see if those are consistent manner. Reasons for recoding are:
reasonable for the sample – is the set of data spelling corrections, date (day, month, year)
complete? formatting, translation, language style and
o All values should have labels if the variable is simplification, clustering, pre-fixes to create better
categorical. Check for the range of values. sorting in tables, combination (in categorical
 If the variable is a continuous/interval variable, run variables), rounding (in continuous variables), and
descriptive statistics such as min, max, mode, possibly others.
mean and median.  Sort the file in various ways (by individual variables
o Look at minimum and maximum values. Are or groups of variables) to see if data errors that
they reasonable? Look especially if “0” are were not found previously can be identified.
really “0” and not missing values.
o Is the mean and median as expected? Final considerations
 Inspect data for missing values (blanks, explicit  If the data are cleaned by more than one person,
missing-value codes). Decide: then the final step is to merge all the spreadsheets
o Which blank cells need to be filled with zeros together so that there is only one database. The
(because they represent genuine negative comments or change logs that are made as the
observations, such as ("no", "not present", cleaning progresses should be compiled into one
"option not taken", etc.) document. Problem data should be discussed in
o Which to leave blank (if the convention is to the documentation file.
use blanks for missing or not applicable)  Update cleaning procedures, change log and data
o Which to replace with some explicit missing documentation file as the cleaning progress.
value code (if we want all missing to be Provide feedbacks to enumerators, team leaders or
explicitly coded). data entry operators if the data collection and entry
 Verify that in binary variables (yes/no), the positive process is still ongoing. If the same mistakes are
value is coded as “1”, the negative as “0”. made by one team or enumerators, make sure to
 Check for the distribution of the values (use box inform the culprit.
plots if available). Look at the extremes and check  Be prepared. Data cleaning is a continued process.
them against the questionnaire even if the value is Some problems cannot be identified until analysis
possible and may seem reasonable. If it is an has begun. Errors are discovered as the data is
extreme, other variable may be incorrect as well. being manipulated by analysts, and several stages
Look out for the 5 smallest/largest values. of cleaning are generally required as
 Compare the data between two or more variables inconsistencies are discovered. In rapid
within the same case to check for logical issues. assessments, it is very common that errors are
I.e., can the head of the household be less than 17 detected even during the peer review process.
years old? Compare age with marital status. Is the
person too young to have been married? Do the
proportions sum up to 100%?
 Where there are questions asking about a “unit”,
the data must standardized to a specific unit, i.e.
when a response is collected using the unit
specified by the respondent. For instance, units for
area can be acre, hectare and square meters. To
standardize the area unit, a lookup table can be
used to merge in the conversion value to convert
all areas to hectares.
 Check for consistencies within a set of cases: If
there is a spouse, it is expected the spouse will be
a different gender. The child of the head of
household is not expected to be older than the
head. The parent of the head cannot be younger
than the head.
 Recode variables. Replace unhelpful entries (e.g.
misspellings, verbose descriptions, category
“others”, etc.) with more suitable variants, in
14
 Dealing with messy data

Annex 2 – Sample Job Description

This annex outlines three job description related to data entry and data cleaning: data cleaner, data entry controller and data entry operator. The
word formats are available at http://www.acaps.org/resourcescats/downloader/assessment_team_job_descriptions/97

Title of post: Data Cleaner

Reporting Line
Information Analyst

Requirements
 Assessment and survey experience
 Large scale data entry experience required

Education
Degree in statistics or demographics and/or a degree in IT

Experience
 2-3 years of experience with statistics institute and/or relevant work experience
 Proven experience with data cleansing and management of large volumes of quantitative and qualitative data.
 Proven experience with management and operation of databases.

Language
Fluent in written and spoken English (or survey language use).

Skills
 Professionalism;
 Excellent written and oral communication skills;
 Good knowledge of word processing software (Word, Excel, PowerPoint, email);
 Understanding of the principles of statistical and demographic analysis;
 Understanding of survey techniques;
 Excellent report drafting skills.
 Strong typing skills
 Strong proofreading skills
 Excellent command of IT tools; High level of computer literacy.
 Rigour and accuracy.
 Proven ability to meet deadlines. Ability to work well under pressure;
 Good interpersonal skills and ability work in a multi-cultural environment. Strong ability to work in teams;
 Experience working with the international humanitarian community is an advantage.

Role description and responsibilities:

Under the supervision of the Information analyst, the Data Cleaner is responsible for:
 Ensuring that procedures for checking, coding and entering data are followed;
 Checking the quality of the work conducted by data entry staff during the data checking, coding and entry and providing
all assistance and feedback necessary to improve data entry and reduce or prevent mistakes;
 Keeping a documented overview of the daily work; producing a daily report on data cleansing;
 Making sure that raw data were accurately entered into a computer readable file;
 Checking the character variables contain only valid values;
 Checking that numeric values are within the predetermined ranges;
 Checking for and eliminating duplicate data entries;
 Checking if there are missing values for variables where complete data are necessary;
 Checking for uniqueness of certain values, such as subject IDs;
 Checking for invalid data values and invalid date sequences;
 Follows procedures for data cleaning and editing. Document data problems. Update regularly the master database with
last changes;

Assessment Coordinator: Data cleaner:

Name: _______________________ Name: _________________________
Position: _______________________ Position: __________________________
Signature: _______________________ Signature: __________________________
Date: _______________________ Date: __________________________
15
 Dealing with messy data

Title of post: Data Entry Operator

Reporting Line
 Data Entry Controller
 Information Analyst
Requirements
 Assessment and survey experience
 Large scale data entry experience required

Education
Secondary education, diploma in information/data management an asset

Experience
 1-2 years of experience with statistics institute and/or relevant work experience
 Proven experience with data entry and management of large volumes of quantitative and qualitative data.
 Proven experience with management and operation of databases.

Language
Fluent in written and spoken English (or survey language use).

Skills
 Strong typing skills
 Data entry skills
 Strong proofreading skills
 Analytical skills.
 Excellent command of IT tools;
 High level of computer literacy.
 Rigour and accuracy.
 Proven ability to meet deadlines.
 Good interpersonal skills and ability work in a multi-cultural environment.
 Experience working with the international humanitarian community is an advantage.

Role description and responsibilities:

Under the supervision of the Data Entry Controller or the information analyst, the Data Entry operator is responsible for:

 Checking completed questionnaires before data entry;

 Codifying open and semi-closed questions;
 Identifying questionnaires with mistakes or error, where the identifier is incorrectly completed and ensuring they are
corrected; detect and organize forms that needs re assessment
 Conducting data entry of questionnaires according to the procedures set out during the training;
 Support site staff to record and manage accurately the collected data
 Data quality checks and data editing according to specified procedures
 Feedback to Information analyst and assessment teams leaders about recurrent mistakes to avoid
 Maintenance of a change log in case of data cleaning or editing
 Archive and back up data, using the specified drive path
 Maintain and operate database
 Maintain office machines

Assessment Coordinator: Data entry operator:

Name: _______________________ Name: _________________________
Position: _______________________ Position: __________________________
Signature: _______________________ Signature: __________________________
Date: _______________________ Date: __________________________

16
 Dealing with messy data

Title of post: Data Entry Controller

Reporting Line
Information Analyst

Requirements
 Assessment and survey experience
 Large scale data entry experience required

Education
Degree in statistics or demographics and/or a degree in IT

Experience
 3-5 years of experience with statistics institute and/or relevant work experience
 Proven experience with data entry and management of large volumes of quantitative and qualitative data.
 Proven experience with management and operation of databases.

Language
Fluent in written and spoken English (or international language use).

Skills for data entry clerk

 Professionalism;
 Strong ability to work in teams;
 Excellent written and oral communication skills;
 Ability to work well under pressure;
 Good knowledge of word processing software (Word, Excel, PowerPoint, email);
 Good command of data processing and analysis software, i.e. CsPro and SPSS;
 Understanding of the principles of statistical and demographic analysis;
 Understanding of survey techniques;
 Excellent report drafting skills. Strong typing skills. Strong proofreading skills
 Excellent command of IT tools; High level of computer literacy.
 Rigour and accuracy.
 Proven ability to meet deadlines.
 Good interpersonal skills and ability work in a multi-cultural environment.
 Experience working with the international humanitarian community is an advantage.

Role description and responsibilities:

Under the supervision of the Information analyst, the Data Entry Controller is responsible for:

 Ensuring that procedures for checking, coding and entering data are followed;
 Monitoring data entry staff;
 Checking the quality of the work conducted by data entry staff during the data checking, coding and entry and
providing all assistance necessary;
 Keeping a documented overview of the daily work; producing a daily report on data checking, coding, and entry.
 Write procedures for data cleaning and editing. Supervise data cleaning. Consolidate the data change logs from
data entry operators. Document data problems. Update regularly the master database with last changes.
 Ordering questionnaires and returning them to the archives after the data has been entered;
 Ensuring technical documents are kept in a good condition;
 Ensuring working hours are respected, as well as order and discipline in the workplace;

Assessment Coordinator: Data entry controller:

Name: _______________________ Name: _________________________
Position: _______________________ Position: __________________________
Signature: _______________________ Signature: __________________________
Date: _______________________ Date: __________________________

17