Research Tips

From PsychWiki - A Collaborative Psychology Wiki

Jump to: navigation, search

Lessons learned from conducting experimental research...

  • Each of us has learned the hard way the many "do's" and "dont's" of conducting experimental research that you can only find out about by hands-on experience. Here is a chance to pass on what you have learned and the tips/tricks that will make it easier for others who are starting to conduct their own research projects...


Contents


[edit]

Design Features

  • Measure the dependent variable as soon as possible after the manipulation; many manipulations wear off quickly. But also be aware that some manipulations only have effects after a short delay or distraction period (e.g., mortality salience manipulations).
  • Use dependent measures, manipulations and manipulation checks that have been used in published research if possible.
  • Ask participants at the end of the study to guess what the study's hypotheses are. This will help you determine whether or not demand characteristics are a problem.
[edit]

Pilot Testing

  • In experimental research, most of the work is pilot testing.
  • Pilot testing is a crucial antidote to assuming you know more about your participants than you actually do. What you think makes people angry may not be what makes college sophomores raised primarily in rural Wisconsin angry. And manipulations that work in Ontario may not work in Oklahoma.
  • Manipulations that seem not be working might in fact be working fine. Perhaps your manipulation check and/or your dependent measures are not working (see more about manipulation checks below).
  • The position of multiple measures prior to a dependent measure may eliminate the effect of the manipulation. Often it makes sense to test a manipulation (with a manipulation check) in a separate pilot study.
  • In the initial stages of a research program, visually inspect the data after running just a few research participants. If your first inspection is ambiguous, run a few more participants and inspect again. You don't need large numbers of participants to find out that your manipulations or measures aren't working as planned.
  • Interview your first few research participants after the debriefing to gain insights into why an experimental manipulation might not be working. You may be making participants angry instead of sad, for example. But also keep in mind the limits of participants' self-reports of their own psychological experiences (Nisbett & Wilson, 1977).
  • During pilot testing, you might save time by running several versions of a manipulation at once (e.g., three different ways of making people feel guilty). Then each method can be compared to one or more control conditions to figure out which manipulation is most powerful.
  • Although this technique violates random assignment and should never be done for research you intend to publish, it is often revealing to compare results from a treatment condition in a current study to results from one or more control conditions (or perhaps treatment conditions) from past studies. This is especially helpful when you are running out of research participants but would still like to know if a new manipulation works.
[edit]

Random Assignment

  • Random assignment is always required in between-group experiments.
  • Replace suspicious or otherwise problematic participants (e.g., those who didn't follow directions) with the next person in random assignment order.
  • Decide whether or not to eliminate a participant's problematic data prior to looking at his/her data.
  • Have research assistants maintain a log in which all strange events can be noted. In most cases a research assistant won't be sure whether or not a participant needs to be "replaced" and the research assistant will be more comfortable describing the problem to you and having you make the call.
  • Have a “good/bad” variable in the data file so that data discarded prior to analysis is not permanently lost. Then create a new data file with only the "good" data in it that you can use for analysis.
  • When practical, check that your random assignment actually worked. It does fail on rare occasions, leading some groups to be very different in crucial ways before the experimental manipulation is even introduced.
[edit]

Control Conditions

  • When manipulating an independent variable, it is optimal to manipulate in a “present versus absent” fashion. But sometimes manipulating “high versus low” (e.g., success vs. failure) is also acceptable, although such a manipulation makes the direction of the effect ambiguous. (For example, is success raising people's mood or is failure lowering it?)
  • Make your control conditions as similar to the treatment conditions as possible. If a treatment condition involves reading an essay, for example, it would be better for participants to read a very similar essay in the control condition (rather than reading nothing at all).
  • In pilot testing, try running multiple control conditions to make sure your control condition is (not) doing anything you don't want. If you are planning an experiment in which anger is the dependent variable, for example, make sure people are no angrier after reading what you assume is a neutral essay than they are after reading no essay at all.
[edit]

Manipulation Checks

  • Check participants' memory for what was manipulated. Participants' motivation is often low and they frequently don't pay attention to the very things that are supposed to influence them.
  • Also check participants' experience of what was manipulated. (This is what is most commonly meant by a manipulation check.) Participants may remember being insulted, but they may not have felt angered by the insult that was administered.
  • Take your manipulation checks seriously, measuring the manipulated construct as thoroughly as possible. For example, if are trying to induce a state or feeling and want to assess this state or feeling with a self-report questionnaire, use multiple items with high internal consistency and be prepared to discard items with low item-total correlations. Most experiments fail, and if yours does, you will rely heavily on your manipulation check to tell you what might have gone wrong and what you should do differently next time.
  • You can check the construct validity of a manipulation check questionnaire by correlating it with other measures to which it should be related. For example, if your manipulation check is supposed to measure people's feelings of anger after the manipulation, it ought to produce a sizable correlation with a measure of trait anger.
  • Scores from manipulation checks can be treated as a mediating variable if the manipulation check and dependent measure are administered in the same study. If your manipulation check is a good measure, then scores on this measure should account for much (ideally all) of the shared variance between the manipulated independent variable and the dependent variable.
  • Manipulation checks are not always necessary, but they are usually helpful. Some people believe that a manipulation check is necessary when your argument is that the manipulation is doing something specific. When you argue that X, and not Y, is producing a result, a mediational analysis using manipulation check scores as a mediating variable is very helpful.
  • When might a manipulation check be unnecessary? If your argument is simply that the manipulation is producing a result and you are not especially concerned about the process behind this result, then you do not need a manipulation check. (Note that this is not especially sophisticated science, but sometimes we want to clearly establish the presence of an effect before we go on to understand the process underlying the effect.)
[edit]

Cell Sizes

  • Appropriate cell size depends on the effect size you are anticipating and the amount of power you want.
  • Having a rough sense of the effect size you can expect (based on past research) and using a power calculator is the best way to decide on cell sizes.
  • If you are curious about how many more participants you might need to run in order for a particular trend in the data to be statistically significant, try copying and pasting your data one or more times and re-conducting your analyses. This allows you to answer questions such as, "If I had the exact same descriptive statistics but twice as many research participants, would the effect be significant?" Please make absolutely sure you delete the fake data after doing this simulation!
  • If you adhere to the information above about random assignment and discarding subjects, you should have reasonably equal cell sizes.
  • If you have unequal cell sizes, you may lose some power. If cell sizes are wildly unequal, some statistical results may be less trustworthy.
  • You can have more confidence in generalizing to the population when cell sizes are equal.
[edit]

Mining your data for ideas

  • Mining your data (e.g., going beyond just the standard statistical tests and looking at all possible relationships in your study even if they are not directly related to your purpose for running the study) is one way to ascertain other possible relationships amongst the data and research ideas.
  • This information may not go into the manuscript, but it is a good source of ideas for you for future research.
  • One idea is to look at “pre-measure” measures for your studies, and/or putting in personality measures into the pre-measure to determine whether there are personality variables that interact/influence variables in your study.
  • If you collect demographic data in the study, this is another source of information for possible future research.
  • One reason to collect demographics at beginning of the study is if one of the demographic variable is an a priori variable in the study so you want to measure it before the manipulation or if you need to know that demographic information in order to assign Ss to conditions.
  • If your manipulation doesn't seem to be working, run correlations between your manipulation check score and your dependent variable so you have at least some idea of whether or not the independent and dependent variables are related.
  • If your dependent measure is a questionnaire, it may be helpful to ask participants at the end of the questionnaire to explain why they responded the way that they did. If your study "fails", open-ended responses may reveal that participants did not interpret something the way that you intended. For example, many participants might express annoyance with a speaker in a video clip--a speaker you thought was quite normal. And if your study "works", you will want some assurance that participants weren't just supplying the answers they knew you wanted to hear (e.g., "As you were probably expecting, the speaker's gender made her message seem weaker"). As you read the open-ended responses, keep in mind that participants frequently don't have conscious access to the true reasons behind their behaviors.
  • If your research involves between-group comparisons, run scatterplots with your groups down on the x-axis and your dependent variable on the y-axis. This will help you see how the scores are clustered across the groups. It will also give you some assurance that outliers are not having an undue influence on differences between groups.
  • If your research involves mean comparisons, try analyzing your data using trimmed means [1], which will allow you to focus on what is happening to the average participants in each condition while ignoring the participants in the tails. This is especially important if you are looking at pilot data collected from relatively small samples.
[edit]

Running subjects

  • It is optimal to run the first few participants yourself. There are almost always "bugs" to be worked out in the experimental protocol and you want to detect these as quickly as possible.
  • If you are at all uncertain about your research assistant's professionalism or conscientiousness in following the experimental protocol, you can ask research participants questions about what the research assistant did during the session on a questionnaire administered at the end of the session.
  • Occasionally interview your participants and research assistants over the course of the experiment to ensure that the protocol is being followed.
  • Try running research assistants through the experimental protocol as participants so that they can see what the experiment is like from the participant's point of view.
  • Research assistants often provide excellent feedback on the clarity of instructions and procedures. If they feel confused while pretending to be research participants, there is a good chance that many of the real research participants will also be confused.
  • Double blind procedures are always best, but sometimes difficult to achieve. At a minimum, consider ways in which participants' condition number can be hidden from the experimenter for part, most or all of the experimental session.
  • Code which research assistant ran which research participant. This will allow you to determine whether a particular research assistant is not following directions, is not convincing to participants or is biasing participants' responses.
  • Run data analysis relatively early in the implementation of the study to ensure that its not a waste of your time to continue and to check whether research assistants are running participants properly.
  • What is the best way to assess suspicion in participants? Funnel debriefing -- if the study is an in-person experiment. Funnel debriefing is when you start with the most abstract and open-ended questions and then funnel down to the most specific and closed-ended questions. If not in-person, such as field studies and online studies, can try using written questions such as asking them what they think the purpose of the study was, or what was your impression of [the person or events] depicted in the study.
  • If conducting the study in the lab, avoid confounds by keeping all extraneous variables constant (such as room, lighting, sounds, research assistants' professionalism, etc.).
  • If conducting the study in the field, it is often difficult or impossible to control extraneous variables so keep careful notes of possible problems and code them when possible in the data file.









◄ Back to Research Tools mainpage

Retrieved from "http://www.psychwiki.com/wiki/Research_Tips"
Content areas...