home

Key review concepts (By lecture)

Measurement-construct validity Sampling-external validity Design-internal validity Statistics-statistical conclusion validity
 * 1. Introduction **
 * Four Primary Areas of Quantitative Research- **


 * Scales of Measurement: **


 * Nominal **- categorical measurement, we use this when we group observations of the same attribute together, for example, putting people into religious categories (p318)


 * Ordinal ** - goes one step beyond nominal level in that we not only group like observations together but also rank them. For example, the question “ Would you rate your social support as very good, satisfactory, or very poor?” asks for an ordinal judgment. Ordinal measures may use numbers for convenience in coding answers, for example, very good (3), satisfactory (2), and very poor (1). However, such numbers for ordinal categories may serve only as handy, abbreviations or labels, and you could not conclude that someone answering with a “3” had three times as much social support as someone answering with a “3” (p318)


 * Interval ** - not only group equivalent observations together in ordered categories, but also consider the interval between adjacent categories as equal, but no natural zero. Such as temperature, differences make sense, but ratios do not. You can think of the meaning of interval level measurement if you think of it as “equal-internal” measurement (p.329)


 * Ratio **- includes all of the characteristics of the interval level plus the additional one of having a true zero point that allows dividing one measure by another (p.329)


 * <span style="color: black; font-family: Arial,sans-serif; font-size: 10pt;">Measurement – Primary Concepts **

<span style="color: black; font-family: Arial,sans-serif; font-size: 10pt;">**Reliability** - refers to the degree to which observed scores are “free from errors of measurement” (American Psychological Association, 1998, p19). We can gauge reliability by the consistency of scores (p. 76)


 * <span style="color: black; font-family: Arial,sans-serif; font-size: 10pt;">Construct Validity **<span style="color: black; font-family: Arial,sans-serif; font-size: 10pt;">-“Refers to the appropriateness, meaningfulness, and usefulness of the specific inferences” made from the measures (American Psychological Association, 199, p9) (p. 76); how we use the score and how will it reflects the construct


 * __<span style="color: black; font-family: Arial,sans-serif; font-size: 10pt;">Sampling – Primary Concepts __**<span style="color: black; font-family: Arial,sans-serif; font-size: 10pt;">-

<span style="color: black; font-family: Arial,sans-serif; font-size: 10pt;">**Population**-Collection of all elements to whom survey results are to be generalized. (P. 348)
 * <span style="color: black; font-family: Arial,sans-serif; font-size: 10pt;">Population v Sample **<span style="color: black; font-family: Arial,sans-serif; font-size: 10pt;">-
 * <span style="color: black; font-family: Arial,sans-serif; font-size: 10pt;">Sample **<span style="color: black; font-family: Arial,sans-serif; font-size: 10pt;">-Subset of individuals selected from a larger population (p. 350)

<span style="color: black; font-family: Arial,sans-serif; font-size: 10pt;">**Probability Sampling**-The actual selection of elements from the frame must give the elements in the frame an equal probability of selection. Random sampling provides the best way of achieving equal probability sampling (p. 128) <span style="color: black; font-family: Arial,sans-serif; font-size: 10pt;">**Non probability sampling**-includes any method in which the elements have unequal chances of being selected. Tone such method in which they ***have an equal chances of being selected. One such method, called convenience sampling, depends on the availability of respondents. In this procedure, subjects select themselves.**
 * <span style="color: black; font-family: Arial,sans-serif; font-size: 10pt;">Sampling Methods **<span style="color: black; font-family: Arial,sans-serif; font-size: 10pt;"> – e.g., Random Sampling v convenience

<span style="color: black; font-family: Arial,sans-serif; font-size: 10pt;">External Validity **<span style="color: black; font-family: Arial,sans-serif; font-size: 10pt;">-Generalizability of the study’s findings to other populations, places or times (p.345) **

<span style="color: black; font-family: Arial,sans-serif; font-size: 10pt;">Design **<span style="color: black; font-family: Arial,sans-serif; font-size: 10pt;">- ** <span style="color: black; font-family: Arial,sans-serif; font-size: 10pt;">True Experimental Designs **<span style="color: black; font-family: Arial,sans-serif; font-size: 10pt;"> (Random Assignment to Control/Comparison Group) ** <span style="color: black; font-family: Arial,sans-serif; font-size: 10pt;">Quasi-Experimental Designs **<span style="color: black; font-family: Arial,sans-serif; font-size: 10pt;"> (Control/Comparison Group without Random Assignment)- ** <span style="color: black; font-family: Arial,sans-serif; font-size: 10pt;">Pre-Experimental Designs **<span style="color: black; font-family: Arial,sans-serif; font-size: 10pt;"> (No Control/Comparison Group)- ** <span style="color: black; font-family: Arial,sans-serif; font-size: 10pt;">Internal Validity **<span style="color: black; font-family: Arial,sans-serif; font-size: 10pt;">-Truthfulness of the assertion that the observed effect is due to the independent variables) in the study (p. 346) **

<span style="color: black; font-family: Arial,sans-serif; font-size: 10pt;">Statistics **<span style="color: black; font-family: Arial,sans-serif; font-size: 10pt;">- ** <span style="color: black; font-family: Arial,sans-serif; font-size: 10pt;">Descriptive Statistics **<span style="color: black; font-family: Arial,sans-serif; font-size: 10pt;">-describes sample. This is a set of methods to describe data that we have collected. ** <span style="color: black; font-family: Arial,sans-serif; font-size: 10pt;">Inferential Statistics **<span style="color: black; font-family: Arial,sans-serif; font-size: 10pt;">-tests probability of sample data being drawn from population defined by hypothesis, this is a set of methods used to make a generalization, estimate, prediction or decision. **

<span style="color: black; font-family: Arial,sans-serif; font-size: 10pt;">Statistical Conclusion Validity **<span style="color: black; font-family: Arial,sans-serif; font-size: 10pt;">-refers to references about whether it is reasonable to presume that a relationship exists between variables. (Introduction presentation) **

<span style="color: black; font-family: Arial,sans-serif; font-size: 10pt;">2. Ethics <span style="color: black; font-family: Arial,sans-serif; font-size: 10pt;">FERPA **<span style="color: black; font-family: Arial,sans-serif; font-size: 10pt;">-Family Educational Rights and Privacy Act (1974) is a Federal Law that protects the privacy of student education records (Ethics, presentation) ** <span style="color: black; font-family: Arial,sans-serif; font-size: 10pt;">National Research Act and Institutional Review Board (IRB) **<span style="color: black; font-family: Arial,sans-serif; font-size: 10pt;">-Committees established by U.S. federal regulations at each research institution to protect human subjects for abuses through prior review of research proposals ** <span style="color: black; font-family: Arial,sans-serif; font-size: 10pt;">Risk **<span style="color: black; font-family: Arial,sans-serif; font-size: 10pt;">-is exposure to the possibility of physical, psychological or social injury as a result of the study. ** <span style="color: black; font-family: Arial,sans-serif; font-size: 10pt;">Coercion **<span style="color: black; font-family: Arial,sans-serif; font-size: 10pt;">-Coercing people to participate ** <span style="color: black; font-family: Arial,sans-serif; font-size: 10pt;">Anonymous/Confidential- **<span style="color: black; font-family: Arial,sans-serif; font-size: 10pt;">require the participants cannot be identified. (p. 28) ** <span style="color: black; font-family: Arial,sans-serif; font-size: 10pt;">Confidential **<span style="color: black; font-family: Arial,sans-serif; font-size: 10pt;">-require the confidentiality of the participants (p. 28) **

<span style="color: black; font-family: Arial,sans-serif; font-size: 10pt;">3. Reliability
 * <span style="color: black; font-family: Arial,sans-serif; font-size: 10pt;">Definition and relationship to validity-refers to the degree to which observed scores are “free from errors of measurement” We can gauge reliability by the consistency of scores. (p. 76) Reliability is necessary but NOT sufficient condition for validity, reliability can also create a tension with validity. **

<span style="color: black; font-family: Arial,sans-serif; font-size: 10pt;">Reliability Coefficient <span style="color: black; font-family: Arial,sans-serif; font-size: 10pt;">Types and How to Increase <span style="color: black; font-family: Arial,sans-serif; font-size: 10pt;">Test-retest **<span style="color: black; font-family: Arial,sans-serif; font-size: 10pt;">-Short time between tests, stability ** <span style="color: black; font-family: Arial,sans-serif; font-size: 10pt;">Alternate forms **<span style="color: black; font-family: Arial,sans-serif; font-size: 10pt;">-forms match better a content and statistical properties equivalence (error due to form or internal consistency ** <span style="color: black; font-family: Arial,sans-serif; font-size: 10pt;">Inter-rater **<span style="color: black; font-family: Arial,sans-serif; font-size: 10pt;">-Standardizing procedure, including training, rubrics and monitoring scores or rater consistency ** <span style="color: black; font-family: Arial,sans-serif; font-size: 10pt;">Split-half **<span style="color: black; font-family: Arial,sans-serif; font-size: 10pt;">-reliability is higher when forms are matched in content and statistical properties, internal consistency (consistency, across terms-content sampling error, flawed items) ** <span style="color: black; font-family: Arial,sans-serif; font-size: 10pt;">Standard Error of Measurement-(SEM)
 * <span style="color: black; font-family: Arial,sans-serif; font-size: 10pt;">Internal consistency, longer tests-wide range of individual variability or construct. Freedom from distraction, misunderstandings. Use of terms of medium difficulty on cognitive measures (Two primary methods for increasing test, reliability), consistency across items-content sampling error, flawed items. **
 * <span style="color: black; font-family: Arial,sans-serif; font-size: 10pt;">Alternative index for reporting random error based on confidence intervals (Reliability, presentation) **

<span style="color: black; font-family: Arial,sans-serif; font-size: 10pt;">Uses and Advantages
 * <span style="color: black; font-family: Arial,sans-serif; font-size: 10pt;">Interpretation-68% of the time the true score is in the interval of x +/- SEM) 95% of the time the true score is in the interval X +/- 1.96*SEM), it is sample from estimation also important (Reliability, presentation) **

<span style="color: black; font-family: Arial,sans-serif; font-size: 10pt;">4. Construct Validity <span style="color: black; font-family: Arial,sans-serif; font-size: 10pt;">Definition **<span style="color: black; font-family: Arial,sans-serif; font-size: 10pt;">-Construct Validity-“refers to the appropriateness, meaningfulness, and usefulness of the specific inferences” made from the measures (p.76) ** <span style="color: black; font-family: Arial,sans-serif; font-size: 10pt;">Sources of Evidence <span style="color: black; font-family: Arial,sans-serif; font-size: 10pt;">Substantive /Response Process **<span style="color: black; font-family: Arial,sans-serif; font-size: 10pt;">-theoretical and empirical analyses of the response processes of examinees are used to determine the fit between the construct and detailed nature of the examinees actual performance or responses. ** <span style="color: black; font-family: Arial,sans-serif; font-size: 10pt;">Internal **<span style="color: black; font-family: Arial,sans-serif; font-size: 10pt;">-Analysis of the internal structure of a test can indicate the degree to which the relationship among test items and test components conform to the construct or which the proposed test score interpretations are based. ** <span style="color: black; font-family: Arial,sans-serif; font-size: 10pt;">Relationships with External Variables **<span style="color: black; font-family: Arial,sans-serif; font-size: 10pt;">-analysis of the relationship of the test scores to variable external to the test. ** <span style="color: black; font-family: Arial,sans-serif; font-size: 10pt;">Consequences **<span style="color: black; font-family: Arial,sans-serif; font-size: 10pt;">-appraises the value implications of score interpretation as a basis for action as well as the intended and unintended consequences of test use, especially in regard to sources of invalidity related to issues of bias, fairness and distributive justice. ** <span style="color: black; font-family: Arial,sans-serif; font-size: 10pt;">Construct Irrelevant Variance **<span style="color: black; font-family: Arial,sans-serif; font-size: 10pt;">-“surplus construct irrelevancy” ** Construct irrelevant difficulty**-aspects of the task are extraneous to the focal construct-can make the test more difficult for some individuals or groups.** Construct irrelevant easiness**-when extraneous clues in the item or format results in correct responses.** <span style="color: black; font-family: Arial,sans-serif; font-size: 10pt;">Construct Underrepresentation **<span style="color: black; font-family: Arial,sans-serif; font-size: 10pt;">-the test is too narrow and fails to include important dimensions of the construct (Construct Validity, presentation) for example, one test is trying to measure the mathematics curriculum including probability, algebra, measurement, data analysis, etc. however, if the test only includes the terms relating algebra. **
 * <span style="color: black; font-family: Arial,sans-serif; font-size: 10pt;">Content-analysis of the relationship between the test’s content and the construct of interest. It refers to the themes, wording, and format of the items tasks, or question on a test, as well as the procedural **

<span style="color: black; font-family: Arial,sans-serif; font-size: 10pt;">5. Internal Validity (HE SKIPPED THIS SECTION *NEEDS WORK)
 * <span style="color: black; font-family: Arial,sans-serif; font-size: 10pt;">Definition – Isolation-Refers to whether or not the relationship between two variables is influenced by any number of additional variables; **<span style="color: black; font-family: Arial,sans-serif; font-size: 10pt;">refers to the truthfulness of the claim that one variable causes another
 * <span style="color: black; font-family: Arial,sans-serif; font-size: 10pt;">Threats to Validity **
 * <span style="color: black; font-family: Arial,sans-serif; font-size: 10pt;">History **<span style="color: black; font-family: Arial,sans-serif; font-size: 10pt;"> - refers to the threat that some coincidental event outside the study caused the observed change **;** can control by reducing time between measures
 * <span style="color: black; font-family: Arial,sans-serif; font-size: 10pt;">The situation in which some specific event occurs during the study, which in turn affects the results. The specific event is an effect other than the experimental treatments. To prevent this threat **use a control group, shorten time between testing, select a dependent variable less prone to history effects, or insulate participants.

<span style="color: black; font-family: Arial,sans-serif; font-size: 10pt;">**Mortality** - subject attrition from pretest to posttest, which casts doubt on validity of the study; protection of this threat cannot be provided by a control or random assignment <span style="color: black; font-family: Arial,sans-serif; font-size: 10pt;">**Selection** - A threat that occurs when an effect may be due to the difference between the kinds of personsin one kind of experimental group as opposed to another. All quasi-experimental studies suffer from this problem. Refers to the threat of differing groupcharacteristics present at the start of the study. <span style="color: black; font-family: Arial,sans-serif; font-size: 10pt;">**Selection Interactions** - Most of the internal validity threats can interact with selection to produce forces that might spuriously appear as treatment effects.
 * <span style="color: black; font-family: Arial,sans-serif; font-size: 10pt;">Maturation - **<span style="color: black; font-family: Arial,sans-serif; font-size: 10pt;">Time threat to internal validity in which internal or normal developmental processes cause the observed change. To prevent this threat, conduct the study over shorter period of time or use a control group with a comparable maturation rate.
 * <span style="color: black; font-family: Arial,sans-serif; font-size: 10pt;">Test Reactivity **<span style="color: black; font-family: Arial,sans-serif; font-size: 10pt;"> - refers to a response to the pretest that causes the observed change in the outcome variable **; can reduce by disguising the pretest or dropping it completely**
 * <span style="color: black; font-family: Arial,sans-serif; font-size: 10pt;">Instrumentation **<span style="color: black; font-family: Arial,sans-serif; font-size: 10pt;"> - when observed changes results from shifts in the way measures are collected ; can be controlled by carefully standardizing and monitoring the measurement procedures
 * <span style="color: black; font-family: Arial,sans-serif; font-size: 10pt;">Regression to the mean -﻿ **<span style="color: black; font-family: Arial,sans-serif; font-size: 10pt;"> Comes from the tendency of scores from unreliable measures to move toward the mean on retest; affects one group studies that select subjects for their extreme scores; To prevent avoid the selection on the basis of extreme scores or create a control group of extreme scores. **<span style="color: black; font-family: Arial,sans-serif; font-size: 10pt; line-height: 0px; overflow: hidden;">﻿ Can also use highly reliable measures. **
 * Selection maturation (differential maturation)--when experimental groups mature at different rates
 * Selection history (local history) -- treatment groups come from different settings, so they experience different events


 * <span style="color: black; font-family: Arial,sans-serif; font-size: 10pt;">6. Design **
 * <span style="color: black; font-family: Arial,sans-serif; font-size: 10pt;">True Experimental Designs **<span style="color: black; font-family: Arial,sans-serif; font-size: 10pt;"> – Role of Control groups and Random Assignment
 * <span style="color: black; font-family: Arial,sans-serif; font-size: 10pt;">Role of control/comparison group **<span style="color: black; font-family: Arial,sans-serif; font-size: 10pt;">-detect and explain possible confounding variables.
 * <span style="color: black; font-family: Arial,sans-serif; font-size: 10pt;">Role of random assignment **<span style="color: black; font-family: Arial,sans-serif; font-size: 10pt;">-increases isolation by removing the relationship between the treatment and other confounding variables. Results in equivalent group.
 * <span style="color: black; font-family: Arial,sans-serif; font-size: 10pt;">Quasi Experimental Designs- **<span style="color: black; font-family: Arial,sans-serif; font-size: 10pt;">designs that involve some form of control/comparison group, but do NOT use random assignment. Generally, “selection” and “selection-interaction” are the greatest type of threats.
 * <span style="color: black; font-family: Arial,sans-serif; font-size: 10pt;">Pre-Experimental Designs- **<span style="color: black; font-family: Arial,sans-serif; font-size: 10pt;">Experimental design without control/comparison group. Weak design but useful when pretests are not possible (ie., pilot testing)


 * <span style="color: black; font-family: Arial,sans-serif; font-size: 10pt;">Random Assignment (Internal Validity) v Random Selection (External Validity) **
 * <span style="color: black; font-family: Arial,sans-serif; font-size: 10pt;">Random Assignment **<span style="color: black; font-family: Arial,sans-serif; font-size: 10pt;">-method of placing subjects in different conditions so that teach subject has an equal chance of being in any group to avoid systematic subject differences between the groups (p.349)
 * <span style="color: black; font-family: Arial,sans-serif; font-size: 10pt;">Random Selection **<span style="color: black; font-family: Arial,sans-serif; font-size: 10pt;">-drawing a representative group from a population to a method that gives every member of the population an equal chance of being drawn (p. 349).


 * <span style="color: black; font-family: Arial,sans-serif; font-size: 10pt;">Random Assignment, Matching and Statistical Adjustment **
 * Matching**-exact equivalence on select variables-uncertainty on all others.
 * Statistical Adjustment-**argue theoretical model-each model leads to a different solution (Experimental design, presentation)