Surveys, Experiments, Case Studies

From CSISWiki
Jump to: navigation, search

Slides available here



  • Method vs Methodology
  • Method: Techniques for gathering evidence; The various ways of proceeding in gathering information
  • Methodology: The underlying theory and analysis of how research does or should proceed, often influenced by discipline
  • Epistemology, Methodology, and Method: a research method is a technique for (or way of proceeding in) gathering evidence while "methodology is a theory and analysis of how research does or should proceed" and "an epistemology is a theory of knowledge"
  • Epistemology (theory of knowledge) is the branch of philosophy concerned with the nature and scope (limitations) of knowledge; addresses questions like:
    • "What is knowledge?"
    • "How is knowledge acquired?"
    • "What do people know?"
    • "How do we know what we know?"
  • Internalism vs externalism: Internalism about justification is the idea that everything necessary to provide justification for a belief must be immediately available in an agent's conscience. Externalism in this context is the view that there are factors other than those which are internal to the believer which can affect the justificatory status of a belief.
  • A priori/a posteriori: The terms a priori ("from the former") and a posteriori ("from the latter") are used in philosophy (epistemology) to distinguish two types of knowledge, justifications or arguments. A priori knowledge or justification is independent of experience ; a posteriori knowledge or justification is dependent on experience or empirical evidence. A priori justification makes reference to experience; but the issue concerns how one knows the proposition or claim in question—what justifies or grounds one's belief in it.
  • Empiricism; Rationalism; Constructivism

( see Wikipedia article on Epistemology; also see this summary)

An Overview of Empirical Research Methods

  • Experimental (Quantitative)
    • True Experiment
    • Quasi-Experiment

  • Descriptive (Qualitative)
    • Case Study
    • Survey/Sampling
    • Focus Groups
    • Ethnography
    • Discourse/Text Analysis

Research methods in SE

  • Controlled experiment (Wohlin et al. 2000)
  • Survey (Pfleger and Kitchenham 2001)
  • Case study (Yin 1994)
  • Ethnography
  • Data collection: Interviews, observation, documentation, ...
  • Design science / constructive research (Hevner et al. 2004, Kasanen et al. 1993)
  • Action / participatory research (Avison et al. 1999, Stringer 1999)

A different view - Research methods in IxD

  • Three ways of studying design (Jacob Buur, Mads Clausen Institute)
    • Ethnographic research - Observing actual practice
    • Action research - Experiments in practice
    • Research through design - Experiments with artefacts

Assessing Methods

  • Research Question(s) is/are key
  • Methods must answer the research question(s)
  • Methodology guides application
  • Epistemology guides analysis
  • All must include 'rigor'


Experimental Research: True Experiment

  • Random sampling, or selection, of subjects (which are also stratified)
  • Introduction of a treatment
  • Use of a control group for comparing subjects who don't receive treatment with those who do
  • Adherence to scientific method (seen as positive, too)
  • Must have both internal and external validity
  • Treatment and control might seem artificial

Experimental Research: Quasi-Experiment

  • Similar to Experiment, except that the subjects are not randomized. Intact groups are often used (for example, students in a classroom).
  • To draw more fully on the power of the experimental method, a pretest may be employed.
  • Employ treatment, control, and scientific method
  • Act of control and treatment makes situation artificial
  • Small subject pools

Positive Aspects of Experimental Research

    • Tests the validity of generalizations
    • Seen as rigorous
    • Identifies a cause-and-effect relationship
    • Seen as more objective, less subjective
    • Can be predictive

Problems with Experimental Research

    • Generalizations need to be qualified according to limitation of research methods employed
    • Controlled settings don't mirror actual conditions; unnatural
    • Difficult to isolate a single variable
    • Doesn't allow for self-reflection (built-in)

Experimental Design in HCI

Survey Research

  • An efficient means of gathering large amounts of data
  • Can be anonymous and inexpensive
  • Feedback often incomplete
  • Wording of instrument can bias feedback
  • Details often left out

Important issues:

  • Data requirements
  • Data generation method
  • Sampling frame
  • Sampling technique
    • probability: random, systematic, stratified, cluster
    • non-probability: purposive, snowball, self-selection, convenience sampling
  • Response rate and non-responses
  • Sample size
  • Accuracy range and confidence level

Case Studies

  • Focus is on individual or small group
  • Able to conduct a comprehensive analysis from a comparison of cases
  • Allows for identification of variables or phenomenon to be studied
  • Multiple sources and methods
  • Time consuming
  • Depth rather than breadth
  • Not necessarily representative or generalizable
  • Types of case studies
    • Exploratory
    • Descriptive
    • Explanatory

Approach to time:

  • Historical
  • Short-term
  • Longitudinal

To Read:

Focus Groups

  • Aid in understanding audience, group, users
  • Small group interaction more than individual response
  • Helps identify and fill gaps in current knowledge re: perceptions, attitudes, feelings, etc.
  • Does not give statistics
  • Marketing tools seen as "suspect"
  • Analysis subjective
Personal tools