Encefalus

Mind Hacks: Through a lab darkly.

I stumbled upon this article the other day. It comments on some articles that criticize the current method of cognitive research. The current trend in cognitive research is based on lab research. We take the subjects, put them into the lab, strip out all the factors that could alter the result in order to isolate the variable that interests us, and, through the experimental procedure, we try to find data that could lead us to a conclusion about this variable.

Lab darkly, not scanner darkly

However, there is a serious objection. The mind has evolved in order to face the challenges of the natural world. By putting the subjects into an artifical environment we strip them out of this environment. This could be somewhat like trying to take a bear into a zoo and study it there, while ignoring the fact that bears live in forests.

Alan Kingstone and his colleagues propose that we need a paradigm shift in cognitive research. In this article (Cognitive Ethology: A new approach for studying human cognition) he defends his opinion that we need a new field called Cognitive Ethology. (see ethology in wikipedia)

The intention was that by minimizing the complexity of the environment and maximizing the experimental control, investigators could create theories that would be universally valid. However, by the mid-1970s it had become very clear that most statements were true if, and only if, particular laboratory conditions were met. In other words, the relationship between factor A and factor B was predictable if, and only if, specific conditions were established within the lab; the relationship
between factors became unpredictable when these laboratory situations were not met. Thus, for example, memory experiments found that what people remembered depended on factors such as (a) what processing they performed on the stimulus materials; (b) what stimulus materials they expected to receive; (c) what materials were actually presented; (d) what people were doing before their memory was measured; (e)how their memory was measured, and so on and so forth. The take home message was that cognitive processes vary and are affected by what is happening elsewhere within the cognitive system, and therefore cognitive processes depend critically on the specific situational context in which a subject is embedded.

[...]the next generation of researchers would take their words to heart and try to find a solution to the issue. In hindsight, this faith has proven to be grossly misplaced, as the next generation of researchers have adopted one of the pathological responses of the past and grounded their neuroimaging investigations on the false assumption that cognitive processes are invariant across situations. It is precisely this false assumption that allows researchers to make the remarkable claim that the cognitive processes that they engage and measure in a simple, artificial brain neuroimaging situation captures the same fundamental cognitive processes and associated neural systems that are engaged in a complex natural situation.  

The current paradigm in Cognitive Science treats the brain as a modular computer. Inspired by the computer revolution in the 70s and 80s, it was a revolutionary, way of thinking at the time. The neuroimaging techniques that were discovered in the 80s and 90s made it even easier to use this approach in science, since we could trace the parts of the brain that are active at any given procedure, find the relationships among them and try to unravel the "modules" of the brain and what they do. 

So, the mind and the brain are really much like a machine with different parts that we can seperate and study. However, thism theory might seem valid to many scientists, but the current paper says that these assumptions are really like mathematical axioms. We just took them for granted, but we don’t know if they’re true and this has caused a lot of problems concerning the philosophy that lies behind the current research and the methodology we follow (also take a look at this article at Scientific American - The Brain Is Not Modular: What fMRI Really Tells Us: Scientific American)

Laboratory research in the field of human cognition is founded on the critical assumption that human cognition is subserved by processes that are invariant and regular across situations. This invariance assumption enables one to conduct a study in the laboratory and then to propose that the process being measured is expressed in everyday life. Importantly, there is a second assumption that falls out of the first. Given that processes are assumed to be invariant across situations, it follows that one can reduce situational variability without compromising the nature of the process one is measuring. Indeed, a basic objective of the experimental nvironment in the laboratory is to gain as much control over a situation as is possible so that any change can be attributed to the variable that is being manipulated.

[...]Unfortunately, a result that is invariant within the strict confines of the laboratory does not mean that it is reproducible outside the lab. Indeed, even a cursory examination of the literature reveals that there are many instances where even the most minor change within a laboratory situation will compromise the replicability of an effect (e.g. Atchley & Kramer, 2001; Berry & Klein, 1993; Bindemann, Burton, & Langton, 2008; Soto-Faraco, Morein-Zamir, & Kingstone, 2005; Wolfe & Pokorny, 1990). In addition, as any researcher knows all too well, failed replications that are published represent just the smallest tip of a very large iceberg of failed replications that are obtained in the laboratory and never published.

The iceberg of failed replications…

Furthermore, the theory behind modules causes us two further problems. Firstly, the researcher has to convince that the procedure that he studies really exists, and is not a product of the lab.

Ironically, any attempt to test the assumption of invariance against real-life situations is met immediately with obstacles that arise from the second assumption of experimental control. The first obstacle is that cognitive concepts often become defined by the experimental controls that are used to examine them. For instance, reflexive attention is often defined as a process that benefits the detection of, and response to, a visual target stimulus that occurs shortly after the abrupt onset of a peripheral, spatially non-predictive, stimulus event. It is not clear whether such a sequence of events ever occurs naturally in real life, and if it did, how this event could be measured.

Secondly, once a researcher he studies the current "module" of the cognitive system, he has to convince the scientific community that he found in the lab has some relation to the real world. After all, psychology is the study of behavior and consciousness, not of behavior and consciousness in the lab :-P. This raises other sort of problems.

Let us accept for the moment that this first obstacle is somehow overcome, and reflexive attention as defined in the laboratory is measured in the real world. A researcher is then immediately posed with the second obstacle of having to make the case that the data collected in the real-life situation are, in fact, a manifestation of the same process being measured in the lab. This is a daunting, and perhaps an ultimately impossible, obstacle to surmount. Our reservation is derived from the very fact that variables that are controlled in the laboratory are not controlled in real life. Therefore, a
real-world effect that appears to be the product of a controlled laboratory effect can always be re-attributed to factors that were uncontrolled in real life. Conversely, the failure to find evidence of a laboratory effect in the real world can be dismissed, as it is a fallacy to conclude that something does not occur simply because one does not find evidence for its existence. Thus, there is no direct way to demonstrate or refute that causal factors found in a simple lab-based setting are also being expressed in a complex real-world situation. Note that the purported real-world relevance of lab-based findings cannot be falsified; such claims, therefore, are, in this most important regard, unscientific.

An unscientific magazine

Here is one more (more at http://www.somethingawful.com/d/photoshop-phriday/reverse-magazines.php)

Here, we could provide an analogy to make this understood better. Let’s say that someone wants to study basketball. He comes to the conclusion that basketball is a modular game constituted of the abilities of shooting, jumping, running etc. Then, he tests some subjects in their shooting ability and tries to predict how good will they do in the game. The researcher here has made two mistakes.

1)He hasn’t studied the ability of shooting to other abilities. Someone might be a bad shooter, but may have a high average score because he can get under the basket and dunk.

2)The ability required in basketball is not just shooting, but shooting under stress. This means that you have to shoot with a guy’s hand into your face, or while running, or without even looking at the basket.

It is well known in sport coaches that the training for a sport, must be sport specific. That means, that should a basketball player try to run faster, he must train with and without a ball, because if he just trains without a ball, when he is called to sprint to the other side of the court with the ball, he’ll probably forget every motor skill he has acquired.

Of course, all sports are modular to some extent. This explains why squatting for example can be a good training strategy for jumping high. But, squatting alone for basketball, which is our example, would be a fundamendal mistake, that professional coaches obviously don’t do.

Modular science

A happy occurence in this paper is the mentioning of non-linear dynamic systems. According to the authors, the current paradigm can explain linear relationships, but cannot explain non-linear ones.

Driving the nail further into this coffin is the fact that general systems theory (see Ward, 2002; Weinberg, 1975) has demonstrated that tight experimental control can be effective at revealing the basic characteristics of simple linear systems but it is ineffective at revealing the characteristics of complex, non-linear systems, which must surely include the human cognitive system. General systems theory holds that certain stable characteristics of complex systems are only revealed, or emerge, when several variables are able to vary together. Of course, this is precisely what is prohibited in controlled laboratory situations, and it is precisely what occurs in uncontrolled natural situations. 

Non-linear dynamic systems could be the future of psychology. Although there are not many handbooks and articles in field, it shows some promise. The main object of this approach is to emply tools from the fields of nonlinear systems and dynamical systems to psychology. I promise to dedicate another post to this topic, since we can’t elaborate on this topic in just a few words. 

So, what the authors propose is, as we said before, is Cognitive Ethology. Ethology, is the study of a behavior in its natural setting.

Something like that, but in a cognitive setting

A second option, and one that we explore in the remainder of this paper, is to first directly study how people behave in their natural real-world environments before moving into the lab. That is, rather than being locked into a laboratory paradigm with the a priori assumption that the paradigm or task that is being applied is tapping into processes that are expressed in
everyday life-situations, one would instead opt to explore first how people behave as they function within a naturally occurring situation. Once this complex problem space is identified and described then one could begin to move into the laboratory to test hypotheses that are generated by real-world observations. We have called this approach ‘Cognitive Ethology’. 

As you see, all these are fairly interesting, and I recommend you to read the paper. The subject of how science should be performed constitutes a great debate. In this article we discussed about cognitive science and simple cognitive experiments. However, there are other approaches and other implications we should discuss about. 

For example, take a look at that: Top 10 Unethical Psychological Experiments - The List Universe. This article describes some experiments that by today’s standards are deemed inhumane. So, it is self-evident that we should take morality into account as well, when we talk about experimentation, since science is not only a matter of validity, but also, a matter of utility, be it in the society or in the individual. In cognitive science the usual subjects are humans, but in other fields of psychology, they are animals. If the lab experimentation can’t guarantee the promised results, maybe ethology is both the moral thing to do and the right choice from an epistimological perspective.

Before we close this article I want to make a mention to Stephen Wolfram and his theory of the empirical study of very simple computational systems that he explains in his book A New Kind of Science. It is an alternative to the traditiona trend in science to see everything through the field of either mathematics (like in physics) or social analysis (like in political philosophy or sometimes in social psychology). I promise I’ll write more about it in future articles.

Finally, I’ll make a mention to something that many of you my not know: On-line experiments! This article here (Top Ten Online Psychology Experiments) presents what it considers the best online experiments. I am learning to program myself right now at c#. Once I finish with that, I’ll take on php with the purpose to start my own experiments on this site!

Stephen Wolfram

Further Reading:

As new kind of science, as data deluge makes the scientific method obsolete…

This entry was posted on Monday, October 20th, 2008 at 3:20 pm and is filed under Cognitive. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.