Data Collection and Analysis,

Interpretation and Discussion

1
 Have you ever collected data so far?
What is your experience?

Annotated bibliography
Assignment-
Include 5 thesis
Use the template
Submit – hardcopy on one week after final exam

2
 Data Collection – in Quantitative
and Qualitative, Design research

3
Data Collection
 Recognize potential challenges
 Data might not be complete
 Respondents may not provide data honestly
 Access to the data sources might be difficult
 Solution
 Personal qualities – communication and interpersonal
 Following formal approaches
 May be in some case stop working on the current agenda

4
Data Collection – Quantitative and
Qualitative, design research
 Data collection is after
 developing instruments
 In case of survey, qualitative studies
 Pilot and Main (actual)
 determine the data sources
 In case of experimental , analytical or predictive
 Setting the objectives of the design artifacts and later after the
construction of the artifact.
 In case of design science
 Requires sufficient preparation - has to be reported
 Construct /variable identification and definition
 Determining type of information
5
Cont…
 It is recommended that you create a matrix that aligns your
research questions with the information you assess is needed and
the methods that you will use to collect that information.
 Four areas of information are typically needed for most qualitative studies
as an example : contextual, perceptual, demographic, and theoretical

 Even further constructing a matrix that lists the research

questions along the horizontal axis and the interview questions
(as one method of data collection) down the vertical axis can
further indicate the extent to which your interview questions have
achieved the necessary coverage of your research questions.

6
Cont…

7
Cont…

8
Cont…
 Developing Data collection Instruments
 Two general approaches (in case of people respondent)
 Adopting from similar previous researches
 Interview questions , questionnaire items, observation
checklists
 Mention how and what part is adopted
 Crafting yourself based / inline with specific objectives
 No orphan question or objectives should be observed
 And pilot study/member checking to validate

 Describe procedures of data sources accessing and acquisition (in

case of non-human data sources)
 Read more on – administering questionnaire, interview,

9 observation
Cont…
 Data collection is followed by analysis
 In strict sense in case of quantitative but not in qualitative
 In qualitative you collect- analyze- collect…
 Until “saturation “
 In design research (before the suggestion of the design and
after construction for evaluation)
 Then analysis could start
 by arranging; presentation
 Categories, Tables, graphs etc…
 by description of the nature of the data (like in NLP and DM)
 The case of data understanding in machine learning
 By structuring requirements/set of criteria
10
 Data Analysis in three research
approaches

11
Analysis in Qualitative Studies
 In general, the intent is to make sense out of text and image
data.
 It involves segmenting and taking apart the data (like
Peeling back the layers of an onion) as well as putting it
back together.
 Generally referred as Thematic
 Understanding meanings through identification and
interrelating themes/codes
 Providing “Thick” description is important including rival
explanations.

12
Cont…
 Some Specific Analytic Techniques
 Pattern Matching
 Evaluating the identified patterns/themes against predefined patterns
 Explanation Building/narration
 Using the themes/codes for description building
 Time-Series Analysis
 Watching how themes are changing or unchanged through time
 Cross-Case Synthesis
 Summary of themes identified across cases
 They are intended to deal with problems of developing internal
and external validity in doing case studies
 They all are based on codes
Cont…
 Some general procedures
 Step 1. Organize and prepare the data for analysis.
 This involves transcribing interviews, optically scanning
material, typing up field notes, cataloguing all of the visual
material, and sorting and arranging the data into different
types depending on the sources of information.
 Step 2. Read or look at all the data.
 This step provides a general sense of the information and
an opportunity to reflect on its overall meaning. What
general ideas are participants saying? What is the tone of
the ideas?

14
Cont…
 Step 3. Start coding all of the data.
 Coding is the process of organizing the data by bracketing
chunks (or text or image segments) and writing a word
representing a category in the margins (Rossman & Rallis,
2012).
 Coding is the process of examining the raw qualitative data
in the transcripts and extracting sections of text units
(words, phrases, sentences or paragraphs) and assigning
different codes or labels so that they can easily be retrieved
at a later stage for further comparison and analysis, and the
identification of any patterns.

15
Tesch’s (1990) Eight Steps in the Coding
Process

16
Analysis in qualitative case study
Using Coding in narrative/thematic analysis

17
19
Cont…
 Step 4. Use the coding process to generate a description of the
setting or people as well as categories or themes for analysis.
 Description involves a detailed rendering of information about
people, places, or events in a setting. Researchers can generate codes
for this description.
 Use the coding as well for generating a small number of themes or
categories— perhaps five to seven themes for a research study.
 These themes are the ones that appear as major findings in
qualitative studies and are often used as headings in the findings
sections (or in the findings section of a dissertation or thesis) of
studies.
 They should display multiple perspectives from individuals and be
supported by diverse quotations and specific evidence

20
Cont…
 Beyond identifying the themes during the coding process,
qualitative researchers can do much with themes to build
additional layers of complex analysis.

 For example, researchers interconnect themes into a story line (as in

narratives) or develop them into a theoretical model (as in grounded
theory).
 Themes are analyzed for each individual case and across
different cases (as in case studies) or shaped into a general
description (as in phenomenology).

 Sophisticated qualitative studies go beyond description and

theme identification and form complex theme connections.
21
Cont…
 Step 5. Advance how the description and themes will be
represented in the qualitative narrative.
 The most popular approach is to use a narrative passage to
convey the findings of the analysis.
 This might be a discussion that mentions a chronology of
events, the detailed discussion of several themes (complete with
subthemes, specific illustrations, multiple perspectives from
individuals, and quotations) or a discussion with
interconnecting themes.

22
Cont…
 Step 6. A final step in data analysis involves making an
interpretation in qualitative research of the findings or
results.
 Asking, “What were the lessons learned?” captures the essence
of this idea (Lincoln & Guba, 1985).
 These lessons could be the researcher’s personal interpretation,
couched in the understanding that the inquirer brings to the
study from a personal culture, history, and experiences.
 It could also be a meaning derived from a comparison of the
findings with information gleaned from the literature or
theories.

23
Additional Key issues in Validity and
reliability of qualitative results

24
 Analysis in Quantitative Studies
 Quantitative - Statistical descriptive and inferential or experiments
 Using variables and Understanding numbers is very much
important
 a VARIABLE is a measurable characteristic that varies
 Mainly considering IV and DV (but at times we may need to
include different types of independent variables like
moderating, control variables,…..)
 Lickert scale based data collection will facilitate
 Understand Parameter /Variable settings (assumptions..)
 Do planed experiments
 Conduct as many experiments as required
 Key also –identifying IV and DV
 Example using : ML tool like Weka

25
Cont…
 Intervening or mediating variables stand between the
independent and dependent variables, and they mediate
the effects of the independent variable on the dependent
variable.
 Objective of the mediator variable is to explain the relationship
between IV & DV e.g. IV is not directly influencing DV but rather IV
is indirectly influencing DV through mediator variable. 
 Pictorially, Independent variable --> Mediator variable -->
Dependent variable.
 For example, salary (IV) is positively influencing education
(mediator variable) and then education is positively influencing
health-screening expenses (DV). 
 When the effect of education is removed, the relationship between
salary and health-screening disappears.
26
Cont…
 Moderating variables are independent variables that affect the
direction and/or the strength of the relationship between
independent and dependent variables
 Moderator variable is a third party variable that modify the relationship
between an independent variable (IV) and a dependent variable (DV). 
 Objective of the moderator variable is to measure the strength of the
relationship between the IV & DV. 
 Pictorially, moderator variable's arrow line is pointing to the mid point
of the arrow-lined relationship between independent variable -->
dependent variable. 
 For example, if age is a moderator variable between salary (IV) and
health-screening expenses (DV), then relationship between salary &
health-screening can be stronger for older men and less strong for
younger men

28
Cont…
 Confounding (or spurious) variable, is not actually
measured or observed in a study.
 It exists, but its influence cannot be directly detected.
 Researchers comment on the influence of confounding variables
after the study has been completed, because these variables
may have operated to explain the relationship between the
independent variable and dependent variable, but they were not
or could not be easily assessed (e.g., a confounding variable such
as discriminatory attitudes).

29
Analysis in quantitative survey studies
 Step 1.
 Report information about the number of members of the
sample who did and did not return the survey.
 A table with numbers and percentages describing respondents and no
nrespondents is a useful tool to present this information

 Step 2.
 Discuss the method by which response bias will be determined
and managed .
 Response bias is the effect of nonresponses on survey
estimates.

30
Cont…
 Step 3.
 Discuss a plan/present a descriptive analysis of data for all
independent and dependent variables in the study.
 This analysis should indicate the means, standard deviations,
and range of scores for these variables.
 In some quantitative researches , the analysis stops here with
descriptive analysis, especially if the number of participants
is too small for more advanced, inferential analysis.
 Descriptive statistics
 Used to describe the phenomena with frequency, mean, median, mode

31
Cont…
 Step 4.
 Assuming that you proceed beyond descriptive approaches,
if the proposal contains an instrument with scales or a plan to
develop scales (combining items into scales),
 identify the statistical procedure (i.e., factor analysis) for
accomplishing this.
 Because many correlations between the variables are high,
it was felt that the judge might be confusing some of the
variables, or that some variables might be redundant.
 Thus Conducting factor analysis to determine the fewer
underlying factors might be necessary

32
Cont…
 Thus factor analysis is a statistical approach that can be used
to analyze interrelationships among a large number of variables
and to explain these variables in terms of a smaller number of
common underlying dimensions.
 It is a process in which the values of observed data are
expressed as functions of a number of possible causes in order
to find which are the most important
 This involves finding a way of condensing the information
contained in some of the original variables into a smaller set of
implicit variables (called factors) with a minimum loss of
information

33
Cont…
 For example, suppose you would like to test the observation that customer
satisfaction is based on product knowledge, communications skills and
people skills.
 You develop a new questionnaire about customer satisfaction with 30
questions: 10 concerning product knowledge, 10 concerning communication
skills and 10 concerning people skills.
 Before using the questionnaire on your sample, you pretest it on a group of people similar
to those who will be completing your survey.
 You perform a factor analysis to see if there are really these three factors. If
they do, you will be able to create three separate scales, by summing the items
on each dimension.
 Factor analysis is based on a correlation table.
 If there are k items in the study (e.g. k questions in the above example) then the
correlation table has k × k entries of form rij where each rij is the correlation coefficient
between item i and item j.

34
Cont….
 Step 5.
 Identify the statistics and the statistical computer program for testing
the major inferential research questions or hypotheses in the proposed
study.
 The inferential questions or hypotheses relate variables or compare
groups in terms of variables so that inferences can be drawn from the
sample to a population.
 Provide a rationale for the choice of statistical test and mention the
assumptions associated with the statistic.
 Inferential statistics -Is used to infer about the population from the
sample data and show variable relationship
• T-test to compare means of two groups
• Analysis of Variance (ANOVA)- (When groups more than two),
Regression , ……

35
36
Cont…
 Step 6.
 A final step in the data analysis is to present the results in tables
or figures and interpret the results from the statistical test.
 An interpretation in quantitative research means that the
researcher draws conclusions from the results for the research
questions, hypotheses, and the larger meaning of the results.
 This interpretation involves several issues.

37
Analysis in experimental studies
 In computing fields, in addition to the observation of subjects
before and after treatment, experimental studies refer also
specifically a research in a laboratory using tools and materials like
algorithms
 Conducting Experiments to analyze data
 Assumption
 You have selected (or developed) tools, techniques/algorithms
based on literature
 Inputs
 The problem, and data
 algorithms/techniques,
 various setting options

38
Cont…
 Procedure
 Plan number of experiments by specifying the data input,
algorithms to be used, experimental settings like test
options …
 Write /report the results of each experiments
 Set criteria and compare the experimental results
 Output
 Accuracy/performance etc.. of an experiment
 Lessons learned, new insights, new knowledge,
 Description of how it is going to be used

39
Cont…

Experiments Algorithm Test option Attributes to Instances

be used

Exp-One Randomforest 10-fold 17 10000

validation
Exp-Two J48 10 –fold 17 10000
validation
Exp-Three PART 10 –fold 17 10000
validation
Exp-Four Randomforest 80/20

Exp-Five J48 80/20

Exp-Six PART 80/20

40
Cont…
 Simulation Analysis
 Is a different flavor of laboratory experiment
 Simulation analysis is a descriptive modeling technique.
 As such, simulation analysis does not provide the explicit problem
formulation and solution steps

 Consequently, one must specify in detail a procedure for the development

and use of simulation models to assure successful outcome from a
simulation study.

 While laboratory experiment focus on measuring the impact of

variables on a target issue, simulation is measuring the effectiveness of
a given model/system in an artificial environment
 It also involves experiments in it

41
 Cont…
 Elements of Simulation Analysis
 Problem Formulation: questions for which answer are sought, the variables
involved and measures of system performance to be used
 Data Collection and Analysis: assembling the information necessary to further
refine our understanding of the problem.
 Model Development: building and testing the model of the real system,
selecting simulation tool (programming language), coding the model and
debugging it
 Model Verification and Validation: establish that the model is an appropriate
accurate representation of the real system.
 Model Experimentation and Optimization: precision issues, how large sample
(simulation time) is necessary to estimate the performance of the system. The
design of effective experiments with which to answer the question asked in
the problem formulation.
 Implementation and Simulation result: acceptance of the result by the users
42
and improved decision making stemming from the analysis.
 Cont….
Major Iterative Loops in a Simulation Study
Problem Formulation

Initial Data Collection and Analysis

Model development

Model Verification and Validation

The
Model Experimentation and Optimization
center of
analysis
Implementation of Simulation Results

43
 Validity and Reliability
 Through Standard instruments /procedures and testing
 Construct Validity
 Concepts being studied are operationalized and measured
correctly
 Internal Validity
 Does the data support the conclusion?
 External Validity
 Establish the domain to which a study’s findings can be
generalized
 Can the result be generalized?
 Experimental Reliability
 Demonstrate that the study can be repeated with the same
results
 Is the method/conduct of the research systematic and logical?
Analysis in Design Researches
 Analysis through iterative design and evaluation process
 Suggest, Design, Construct and Evaluate an artifact as a
search process
 Following standards and principles is important (eg OOSE)
 Creativity has also significant place
 Iterations and improvement has to be reported
 Recall the iterative design cycles
 May need input from mainly qualitative type of analysis but
from quantitative also

 Mainly analysis as a deductive process of checking an

abductively identified/ suggested solutions through an
iterative design, development and evaluation process

45
Data collection
Analysis
46
Example: analysis results and their discussion
may take two forms in design science research

47
Main perspectives in design deductive
analysis
 Structure of the artifact: the information space the artifact spans
 basis for deducing all required information about the artifact
 determines the configurational characteristics necessary to enable the
evaluation of the artifact
 Evaluation criteria: the dimensions of the information space which are
relevant for determining the utility of the artifact
 can differ on the purpose of the evaluation
 Evaluation approach: the procedure how to practically test an artifact
 defines all roles concerned with the assessment and the way of
handling the evaluation
 result is a decision whether or not the artifact meets the evaluation
criteria based on the available information.
 Can be qualitative or quantitative

48
Methods
Structure Evaluation Evaluation approach
criteria
process-based meta appropriateness laboratory research
model completeness field inquiries
intended consistency surveys
applications case studies
conditions of
action research
applicability
practice descriptions
products and results
interpretative
of the method
application research
reference to
constructs
Models
Structure Evaluation criteria Evaluation approach
 domain correcteness syntactical validation

 scope, purpose completeness integrity checking

 syntax and clarity sampling using

semantics flexibility selective matching of
 terminology simplicity data to actual external
 intended phenomena or trusted
applicability
surrogate
application implementability
integration tests

risk and cost analysis

user surveys
Instantiations
Structure Evaluation Evaluation approach
criteria
executable implementation in functionality Code inspection
a programming language usability Testing
reference to a design model reliability Code analysis
reference to a requirement performance Verification
specification supportability Accptance study
reference to the
Usability analysis
documentation
reference to quality
management documents
reference to configuration
management documents
reference to project

management documents
Validity and reliability in deign research
 Pragmatic Utility through scientific methods
 Does it solve the initial problem?
 Was it correct and repeatable
 Is it acceptable by the respective beneficiaries
 Rigorous evaluation

52
Exercise
 Relating research problems, objectives and results
 Problem statements should lead to defining a clear objectives
 Objectives should lead to clear analysis and results

 Read the following problem descriptions and

1. Craft a candidate general objectives for the following three
cases.
2. Suggest data sources and data collection methods
3. Suggest potential type of analysis
4. Suggest possible/logical result of the studies and relevant
interpretations expected

53
54
55
 Example 3 -Technology adoption is one of the area of
studies in information technology. E-learning is a
technology that enables effective teaching learning
process. Educational institutes are seen trying to adopt
such platform in their process. However appropriate
adoption requires prior investigation of perception and
feeling of end users. In relation to this there was no any
comprehensive study on the adoption level of e-learning
technology among academic institutions. And factors that
affect such adoption were inconsistent from one study to
another.

56
Sample solutions

57
 Objective
 To identify adoption level and factors affecting e-learning adoption
 Data sources
 Human data sources (students, instructors, educational admins)
 Data collection methods
 It depends but; interview, questionnaire, observation …

 Analysis techniques
 Will depend on the data collection methods
 But can be either descriptive statistics, inferential statistics methods
or thematic analysis
 Expected outputs
 Level of adoption determined
 Factors for adoption listed
 Interpretation and Discussion

59
Interpretation and Discussion
 In strict sense
 Interpretation is telling what the result means
 Discussion is telling how good/better or related the result is
compared to other works.
 Explain the results in light of
 Previous literatures and theories.
 Own RQ and/or objectives
 No clear distinction in case of qualitative
 Involves reporting results of demonstration and evaluation in
case of design research
 May be required to collect data and analysis with the same
procedures as we have seen before
60
 Interpreting the Data
 Interpreting the data means several things, in particular,
it means:
1) Relating the findings to the original research problem
and to the specific research questions and
hypotheses.
Researchers must eventually come full circle to their
starting point – why they conducted a research study
in the first place and what they hoped to discover –
and relate their results to their initial concerns and
questions
Cont…
2) Relating the findings to preexisting literature, concepts,
theories, and research studies.
To be useful, research findings must in some way be
connected to the larger picture – to what people already
know or believe about the topic in question.

Perhaps the new findings confirm a current theoretical

perspective, perhaps they cast doubt on common
“knowledge”, or perhaps they simply raise new questions that
must be addressed before we can truly understand the
phenomenon in question

62
Cont…
3) Determining whether the findings have practical significance
as well as statistical significance.
Statistical significance is one thing; practical significance –
whether findings are actually useful – is something else
altogether.

4) Identifying limitations of the study. Finally, interpreting the

data involves outlining the weaknesses of the study that
yielded them.
No research study can be perfect, and its imperfections
inevitably cast at least a hint of doubt on its findings. Good
researchers know – and they also report – the weaknesses
along with the strengths of their research
TOOLS OF RESEARCH- in relation to data
collection and analysis
 Tools are chosen to facilitate research tasks
 Some researchers need special tools indigenous to particular discipline
 The concern here is with the general tools of research that the majority
of researchers, regardless of discipline and situation, typically need to
derive meaningful and insightful conclusions from the data they collect
 Be careful not to equate tools of research with the methodology of
research
 There are six general tools of research:
1. The library and its resources
2. The computer and its software
3. Techniques of measurement
4. Statistics
5. The human mind
6. Language
1. The Library and Its Resources
 Scholar should know its principal resources and understand its
classification system, and find the shortest route to the
information it contains.
 Both physical and online databases
 The broad definition of the term library includes all the context and
resources in any form that a person get concentrated to think deeply
while reading on a topic of interest
 You learn the library by using the library
 Feel the mood of research in a library
2. The Computer and its Software – Tool of
Research
 Be familiar with at least few selected tools in the area of
research you are engaged
 Software tools like Weka, NLTK, Ns-3 and other networking
monitoring tools, etc…
 Use appropriate tools with logical settings.
 Taking advantage of the Internet
 Web site: journals, publishers, organizations, individuals, etc.
 Search engine: googlescholar , yahoo, Alta Vista, etc.
 E-Mail (Asking questions to authors, experts, etc., Facilitate
collaboration among people, Attached file (reports, etc.)
 News (List servers: E-discussion group., Many groups with
particular interests)
3. Measurement – Tool of Research
 Researchers strive for objectivity: not influenced by own
perceptions, impressions, and biases.
 Therefore, must identify systematic way of measuring a
phenomenon
 In most case numerically
 But also qualitatively
 Old proverb – if it exists, then it can be measured
 If it is researchable, then data must be measurable
4. Statistics as a Tool of Research
 All tools are MORE suitable for some purposes than for others.
 Example : screw driver was designed to INSERT and remove screws –
BUT people often used it for punching holes, scratch away paints,
etc…..misuse……
 So, too, with statistics- Statistics can be a powerful tool when used
correctly (for specific kind of data & research questions)
 BUT can be misleading when applied in other contexts.
 More useful in some academic disciplines than in others.
 REMEMBER, the statistical values obtained are never the end of research
nor the final answer to research problem.
 The final question is “What do the data indicate” not what is their
numerical configuration.
 Statistics give information about the data BUT a conscientious researcher
is not satisfied until the MEANING of this information is revealed.
5. The Human Mind – Tool of Research
 Statistics can tell us the center, the spread, relationship of data BUT
cannot interpret and arrive at a logical conclusion or meaning.
 Only mind can do.
 Mind is the most important tool.
 Nothing equals its powers of comprehension, integrative reasoning
and insight
 Strategies to make use of the human mind to better understand
include :
1. Deductive logic
2. Inductive reasoning
3. Scientific method
4. Critical thinking
5. Collaboration with others
Cont…
 Critical Thinking
 Good researchers engage in critical thinking.
 Involves evaluating information or arguments in terms
of their accuracy and worth
 As an example
 During LR don’t just accept research findings and theories at
face value.
 Scrutinize for faulty assumptions, questionable logic,
weaknesses in methodology, inappropriate statistical analyses,
and unwarranted conclusions.
Cont…
 Critical Thinking: Take a variety of forms, depending on the
context.
 Verbal reasoning – Understanding and evaluating the
persuasive techniques found in oral and written language.
 Argument analysis – Discriminating between reasons that do
and do not support a particular conclusion.
 Decision making – Identifying and judging several
alternatives and selecting the best alternative.
Cont…
 Collaboration with Others
 More heads are better than one.
 A researcher has certain perspectives, assumptions, and
theoretical biases – not to mention holes in knowledge
about subject matter – that limit research approaches of a
project.
 Need to bring colleagues who have perspectives,
backgrounds, and areas of expertise somewhat different –
more cognitive resources to tackle research problem and
how to find meaning.
6. Language as a Tool of Research

 Human kind’s greatest achievements – facilitate

communication and think effectively.
 Can think more clearly and efficiently when can represent
thoughts with specific WORDS and PHRASES
 Good words, even a simple one, can
1. Reduce world’s complexity
2. Facilitate generalization and inference drawing in new
situation.
3. Allow abstraction of the environment
4. Enhance the power of thought
Review questions
 Explain the central concept in qualitative analysis
 Describe the difference between survey experiment and
laboratory experiment methods from IT research perspective
 Explain the essence of analysis in design science research
 What is the purpose of factor analysis?
 Describe knowledge contribution of a typical Design science
research
 How do you interpret your research findings?
 Enumerate at least three major challenges in data collection
phase of a research.

74
A way forward – for Tuesday students
 Today (Dec. 16)
 Chapter four lecture
 Last Paper presentations
 Tuesday (Dec 18 )
 Remaining lecture from chapter four
 Proposal progress report
 Tuesday (Dec. 25)
 Chapter five lecture

75
A way forward – for Wednesday
students
 Today (Dec. 14)
 Chapter four lecture
 Wednesday (Dec 19 )
 Remaining lecture from chapter four
 Last Paper presentation-
 Wednesday (Dec. 26)
 Proposal progress- all
 Sunday (Dec. 30)
 Chapter five lecture

76
 Make sure that your data
collection and analysis is
logical!!!

77