Tips On Conducting Experiments

From PsychWiki - A Collaborative Psychology Wiki

Lessons learned from conducting experimental research...

• Use this link for Internet Research.

• Each of us has learned the hard way the many "do's" and "dont's" of conducting experimental research that you can only find out about by hands-on experience. Here is a chance to pass on what you have learned and the tips/tricks that will make it easier for others who are starting to conduct their own research projects...

Contents 1 Design Features 2 Pilot Testing 3 Random Assignment 4 Control Conditions 5 Manipulation Checks 6 Cell Sizes 7 Mining your data for ideas 8 Running subjects

Design Features

• Measure the dependent variable as soon as possible after the manipulation; many manipulations wear off quickly. But also be aware that some manipulations only have effects after a short delay or distraction period (e.g., mortality salience manipulations).

• Use dependent measures, manipulations and manipulation checks that have been used in published research if possible.

• Use the strongest manipulation you ethically can. But watch out for demand characteristics.

• Ask participants at the end of the study to guess what the study's hypotheses are. This will help you determine whether or not demand characteristics are a problem.

Pilot Testing

• In experimental research, most of the work is pilot testing.

• Pilot testing is a crucial antidote to assuming you know more about your participants than you actually do. What you think makes people angry may not be what makes college sophomores raised primarily in rural Wisconsin angry. And manipulations that work in Ontario may not work in Oklahoma.

• Manipulations that seem not be working might in fact be working fine. Perhaps your manipulation check and/or your dependent measures are not working (see more about manipulation checks below).

• The position of multiple measures prior to a dependent measure may eliminate the effect of the manipulation. Often it makes sense to test a manipulation (with a manipulation check) in a separate pilot study.

• In the initial stages of a research program, visually inspect the data after running just a few research participants. If your first inspection is ambiguous, run a few more participants and inspect again. You don't need large numbers of participants to find out that your manipulations or measures aren't working as planned.

• Interview your first few research participants after the debriefing to gain insights into why an experimental manipulation might not be working. You may be making participants angry instead of sad, for example. But also keep in mind the limits of participants' self-reports of their own psychological experiences (Nisbett & Wilson, 1977).

• During pilot testing, you might save time by running several versions of a manipulation at once (e.g., three different ways of making people feel guilty). Then each method can be compared to one or more control conditions to figure out which manipulation is most powerful.

• Although this technique violates random assignment and should never be done for research you intend to publish, it is often revealing to compare results from a treatment condition in a current study to results from one or more control conditions (or perhaps treatment conditions) from past studies. This is especially helpful when you are running out of research participants but would still like to know if a new manipulation works.

Random Assignment

• Random assignment is always required in between-group experiments.

• The Research Randomizer is a great tool for random assignment.

• Replace suspicious or otherwise problematic participants (e.g., those who didn't follow directions) with the next person in random assignment order.

• Decide whether or not to eliminate a participant's problematic data prior to looking at his/her data.

• Have research assistants maintain a log in which all strange events can be noted. In most cases a research assistant won't be sure whether or not a participant needs to be "replaced" and the research assistant will be more comfortable describing the problem to you and having you make the call.

• Have a “good/bad” variable in the data file so that data discarded prior to analysis is not permanently lost. Then create a new data file with only the "good" data in it that you can use for analysis.

• When practical, check that your random assignment actually worked. It does fail on rare occasions, leading some groups to be very different in crucial ways before the experimental manipulation is even introduced.

Control Conditions

• When manipulating an independent variable, it is optimal to manipulate in a “present versus absent” fashion. But sometimes manipulating “high versus low” (e.g., success vs. failure) is also acceptable, although such a manipulation makes the direction of the effect ambiguous. (For example, is success raising people's mood or is failure lowering it?)

• Make your control conditions as similar to the treatment conditions as possible. If a treatment condition involves reading an essay, for example, it would be better for participants to read a very similar essay in the control condition (rather than reading nothing at all).

• In pilot testing, try running multiple control conditions to make sure your control condition is (not) doing anything you don't want. If you are planning an experiment in which anger is the dependent variable, for example, make sure people are no angrier after reading what you assume is a neutral essay than they are after reading no essay at all.

Manipulation Checks

• Manipulation checks are separate measured variables that show what an active manipulation concurrently affects besides the dependent variable of interest. Manipulations are NOT intended to verify that the manipulated factor caused variation in the dependent variable. Random assignment, manipulation before measurement of the dependent variable, and statistical tests of effect of the manipulated variable on the dependent variable verify this.

• Manipulation checks are most informative under two conditions. First, when a manipulation fails to influence a dependent variable, they hep the experimenter decipher whether this is due to (a) failure of the experiment to properly execute a manipulation or (b) successful manipulation of an effect that simply has no effect on a dependent variable. Second, they are also informative when a specific mediational process is theorized and a complete mediation model is to be tested, to see whether a manipulated variable (e.g., being insulted) causes variation in a measured manipulation check (e.g., anger), which in turn, causes variation in the dependent variable of interest. Manipulation checks do NOT test whether a manipulation actually drives variation in the dependent variable. Statistical tests of the effect of the manipulated variable on the dependent variable (not the measured manipulation check) affirm this.

• Manipulation checks are not always necessary, but they are usually helpful. Some people believe that a manipulation check is necessary when your argument is that the manipulation drives variation in a dependent variable by creating some specific mediational process (e.g., insults cause time efficiency on a task because manipulations cause anger). When you argue that X, and not Y, is producing a result, a mediational analysis using manipulation check scores as a mediating variable is very helpful. Manipulation checks are also helpful when experimenters do not have direct control over the experimental treatments (e.g., in medical studies, determining whether subjects actually took a mediication or other treatment on their own that they were assigned to).

• When might a manipulation check be unnecessary? If your argument is simply that the manipulation is producing a result and you are not especially concerned about the process behind this result, then you do not need a manipulation check. This is often the case when the experimenter has direct control over the manipulation (e.g., subjects either are either reading green or red text). This is also often the case when a specific mediational process is not of interest to the research question (e.g., subjects unambiguously either were or were not insulted, and it is unimportant to the research question whether subjects were concurrently angered by the insult for it to influence a particular dependent variable).

• Take your manipulation checks seriously, measuring the manipulated construct as thoroughly as possible. For example, if you are trying to induce a state or feeling and want to assess this state or feeling with a self-report questionnaire, use multiple items with high internal consistency and be prepared to discard items with low item-total correlations. Most experiments fail, and if yours does, you will rely heavily on your manipulation check to tell you what might have gone wrong and what you should do differently next time.

• Manipulation checks test participants' memory for what was manipulated. Participants' motivation is often low and they frequently don't pay attention to the very things that are supposed to influence them. This can provide an experimenter with reasons why a manipulation may have failed to influence a dependent variable. However, for a manipulation to successfully influence a dependent variable suggests that subjectys paid enough attention to a manipulation for it to influence them, even without a manipulation check.

• Manipulation checks also test participants' experience of what was manipulated. For example, participants may remember being insulted, but they may not have felt angered by the insult that was administered. This can help clarify whether being insulted, or being angered from being insulted (i.e., a mediational process) causes effects in a dependent variable. However, caution should be noted that individuals (e.g., experimental subjects) do not always have self-insight into what is driving their behavior. For example, a subject may not fully agree that a manipulation influenced him or herself when it did.

Cell Sizes

• Appropriate cell size depends on the effect size you are anticipating and the amount of power you want.

• Having a rough sense of the effect size you can expect (based on past research) and using a power calculator is the best way to decide on cell sizes.

• If you are curious about how many more participants you might need to run in order for a particular trend in the data to be statistically significant, try copying and pasting your data one or more times and re-conducting your analyses. This allows you to answer questions such as, "If I had the exact same descriptive statistics but twice as many research participants, would the effect be significant?" Please make absolutely sure you delete the fake data after doing this simulation!

• If you adhere to the information above about random assignment and discarding subjects, you should have reasonably equal cell sizes.

• If you have unequal cell sizes, you may lose some power. If cell sizes are wildly unequal, some statistical results may be less trustworthy.

• You can have more confidence in generalizing to the population when cell sizes are equal.

Mining your data for ideas

• Mining your data (e.g., going beyond just the standard statistical tests and looking at all possible relationships in your study even if they are not directly related to your purpose for running the study) is one way to ascertain other possible relationships amongst the data and research ideas.

• This information may not go into the manuscript, but it is a good source of ideas for you for future research.

• One idea is to look at “pre-measure” measures for your studies, and/or putting in personality measures into the pre-measure to determine whether there are personality variables that interact/influence variables in your study.

• If you collect demographic data in the study, this is another source of information for possible future research.

• One reason to collect demographics at beginning of the study is if one of the demographic variable is an a priori variable in the study so you want to measure it before the manipulation or if you need to know that demographic information in order to assign Ss to conditions.

• If your manipulation doesn't seem to be working, run correlations between your manipulation check score and your dependent variable so you have at least some idea of whether or not the independent and dependent variables are related.

• If your dependent measure is a questionnaire, it may be helpful to ask participants at the end of the questionnaire to explain why they responded the way that they did. If your study "fails", open-ended responses may reveal that participants did not interpret something the way that you intended. For example, many participants might express annoyance with a speaker in a video clip--a speaker you thought was quite normal. And if your study "works", you will want some assurance that participants weren't just supplying the answers they knew you wanted to hear (e.g., "As you were probably expecting, the speaker's gender made her message seem weaker"). As you read the open-ended responses, keep in mind that participants frequently don't have conscious access to the true reasons behind their behaviors.

• If your research involves between-group comparisons, run scatterplots with your groups down on the x-axis and your dependent variable on the y-axis. This will help you see how the scores are clustered across the groups. It will also give you some assurance that outliers are not having an undue influence on differences between groups.

• If your research involves mean comparisons, try analyzing your data using trimmed means [1], which will allow you to focus on what is happening to the average participants in each condition while ignoring the participants in the tails. This is especially important if you are looking at pilot data collected from relatively small samples.

Running subjects

• It is optimal to run the first few participants yourself. There are almost always "bugs" to be worked out in the experimental protocol and you want to detect these as quickly as possible.

• If you are at all uncertain about your research assistant's professionalism or conscientiousness in following the experimental protocol, you can ask research participants questions about what the research assistant did during the session on a questionnaire administered at the end of the session.

• Occasionally interview your participants and research assistants over the course of the experiment to ensure that the protocol is being followed.

• Try running research assistants through the experimental protocol as participants so that they can see what the experiment is like from the participant's point of view.

• Research assistants often provide excellent feedback on the clarity of instructions and procedures. If they feel confused while pretending to be research participants, there is a good chance that many of the real research participants will also be confused.

• Double blind procedures are always best, but sometimes difficult to achieve. At a minimum, consider ways in which participants' condition number can be hidden from the experimenter for part, most or all of the experimental session.

• Code which research assistant ran which research participant. This will allow you to determine whether a particular research assistant is not following directions, is not convincing to participants or is biasing participants' responses.

• Run data analysis relatively early in the implementation of the study to ensure that its not a waste of your time to continue and to check whether research assistants are running participants properly.

• What is the best way to assess suspicion in participants? Funnel debriefing -- if the study is an in-person experiment. Funnel debriefing is when you start with the most abstract and open-ended questions and then funnel down to the most specific and closed-ended questions. If not in-person, such as field studies and online studies, can try using written questions such as asking them what they think the purpose of the study was, or what was your impression of [the person or events] depicted in the study.

• If conducting the study in the lab, avoid confounds by keeping all extraneous variables constant (such as room, lighting, sounds, research assistants' professionalism, etc.).

• If conducting the study in the field, it is often difficult or impossible to control extraneous variables so keep careful notes of possible problems and code them when possible in the data file.

---

◄ Back to Collecting Data page