 Eugen G Tarnow | Eugen G Tarnow January 30 2012 09:47:21 AMSometimes it is hard to believe the type of review one's manuscript gets. And it is hard to figure out how the peer review system could ever change to demand at least a modicum of civility and some standards as to how many reviews are sent out (I thought the standard was three, but in this case it seems to be only one). Enjoy:
Manuscript Number: ACTPSY-D-11-00358
Article Title: Initial Free Recall Distribution Characterized and Explained By Activation Theory of Short Term Memory
Acta Psychologica
Dear Dr. Tarnow,
This is to inform you that the reviewers' comments on your work have been received. You will note that they are advising against publication of your work. Therefore I must reject it.
It is a detailed review of your paper. The author is an expert in the field of memory research and recall and I highly appreciate the suggestion and also the time he or she has invested in evaluating your manuscript. The reviewer suggests a rejection and presents many good arguments for this evaluation. They are very clear and there is no need to repeat them. Based on my own reading I came to the same conclusion why I did not asked for a second review but decided on the existing information to reject the paper.
I am sorry that I cannot provide you better news but I hope that the review will help you to improve your research. I closing I want to add a further comment which was not put forward by the reviewer but which I would like to ask you to consider when you plan to submit your paper elsewhere. Your style of writing does not match the standard of scientific journals. Therefore, after a first view on your manuscript, I first wanted to send it back to you without further considering it for review. I finally sent it out because I wanted to give the paper a chance. However, this might not always happen.
For your guidance, I append the reviewers' comments below.
Thank you for giving us the opportunity to consider your work.
Yours sincerely,
Hubert Zimmer, PhD
Action Editor
Acta Psychologica
Reviewers' comments:
Reviewer #1:
The theoretical and empirical statements the author makes have only the most
tenuous connection to reality. These statements:
``One aspect of the data which so far has been ignored is the initial recall
distribution - the freest of the recalls.''
``There is no previous attempt to explain the initial free recall
distribution.''
from the introduction setting out the motivation for this ms are particularly
troubling, as it can be easily demonstrated that they are simply false.
There are several dozen examples of empirical and model papers that report and
attempt to account for the probability of first recall. In fact, several of
these papers are present in the citation list of the current paper! As a
particularly egregious example, Kahana et al., (2002), from which some of the
fitted data is taken, reports the probability of first recall across
conditions and age groups (see also Kahana, 1996; Howard & Kahana, 2002).
Has the author not read the papers he cites---or even the ones he takes data
from? Or is the author so careless about the factual status of the statements
he makes that it wouldn't make a difference if he had read them? In either
case, the omission is sufficiently serious that these statements by themselves
should be sufficient to reject this paper outright.
The empirical statements the author makes about his analyses appear to have
received only slightly more scrutiny than the theoretical statements made.
The only factual empirical observation I can extract from this paper is that
the shape of the FRP curves looks more or less like the serial position curve.
This has been known for a very long time and does not warrant publication. I
will attempt a more detailed review figure by figure.
Figure 1 reports the serial position curve and FRP curve from one of the
conditions of Murdock (1962). As mentioned earlier, the FRP curves from these
previously published papers have already been well-characterized. The point
seems to be that these look similar to one another. It is worth noting that
the resemblance would not have been nearly so striking had the author chosen
one of the other conditions from that same paper. In most cases, the FRP
shows a non-monotonicity at the end of the list but the serial position curve
does not.
Figure 2 attempts to quantify the similarity of FRP curves to serial position
curves by regressing the one onto the other. There is a recency effect and a
primacy effect in each curve, so it is not surprising that there is a
relatively close correspondence. However, there is also a large systematic
deviations from linearity that is not discussed. This is a consequence of the
fact that at the end of the list the FRP (first recall probability) is not
monotonically increasing (showing a plateau or hook), whereas the P(R) is
monotonically increasing. I can't tell from the figure whether there's also a
systematic deviation due to the primacy effect but I wouldn't be surprised if
there was. There is no discussion of the deviations.
Figure 3 regresses the offset (which as near as I can figure is the intercept
of regressing P(R) onto FRP) onto the lowest probability of recall for each
experiment. There is a high degree of correspondence. I cannot imagine what
I am supposed to infer from this. I can't see how it could be otherwise. The
serial position curve reaches a minimum in the middle. The FRP is zero in
the middle of the FRP curves for these data. Does this reflect anything more
deep than this (very straightforward and already well-understood) observation?
If it does, I cannot guess what it might be.
Figure 4 shows the same type of relationship for the delayed condition of
Kahana et al. (2002). The relationship observed in Figure 3 does not hold.
This is because the FRP does not go to zero in delayed recall (see e.g., Kahana et
al., 2002). Again, the point of this is extremely opaque. It was not
initially clear to me why the immediate data from that same experiment was
excluded. Then on page 9 the reader is told that those data were
excluded because the model does not fit them:
Here we note that the non-delayed initial free recall curves from
Kahana et al (2002) cannot be fitted by the model, presumably because the
initial recall is not particularly free - the participants were asked to
read the word items aloud and the first recall (done verbally) simply
repeats the last read item in 50% of the cases
Figures 5 and 6 present results from the author's theory of memory. This
theory provides a very weak account of the data. Equation 1 says that the
activation of an item is equal to its probability of recall or its probability
of recognition. First, we might note that the probability of recall and
probability of recognition are very different things affected by different
variables. So Eq. 1 taken by itself is already problematic. But then examine
Equation 3. This says that the activation of an item is also a logarithmic
function of the time since it was presented. If these equations are true,
then the probability of recall (or recognition) of an item should be a
logarithmic function of the time since it was presented. It would seem that
the way to test the model would be to take Equation 3 (or perhaps some
appropriately normalized version thereof) and compare it to the FRP curves.
Or the serial position curves. This would enable us to assess whether Eqs 1
and 3 are correct. If that is how the paper proceeded, we would have found
out that Eqs 1 and 3 are not correct. For instance, there is a primacy effect
that describes an increase in the probability of recall with an increase in
the time since its presentation. And the fall-off of the recency effect does
not seem to particularly resemble a logarithmic function in the serial
position curve nor the FRP curve.
Rather than using the data to evaluate the consistency of Eq 1 with Eq 3, Eq 1
is simply assumed to be true. Then, the probability of recall is used as an
empirical method to estimate the activation of each item. A complex
normalization takes place to map the ``activation'' estimated from the SPC
onto the FRP curve. If there is such a mapping, the model must be a good
description of memory. Note that activation drops out of the model entirely
(replaced by probability of recall). But the ability of this model to
describe the FRP curves is taken as evidence for this model of
activation---the circularity of this argument is breathtaking!
Even granting the dubious logic of this excercise, the fit of this model to
the PFR data is nonetheless poor. There are dramatic non-monotonicities in
the Murdock FRP data which are not captured by this model---because they are
not reflected in the serial position curve. The immediate FRPs from the
Kahana et al., (2002) data do not show this non-monotonicity, but we are told
the model doesn't fit those data. Moreover, the model consistently
overestimates the FRP for the early serial positions in the IFR data
(mysteriously, some of these points appear to be missing from this figure).
Neither of these systematic errors is discussed. Had the author paid
attention to these discrepancies, he might have provided some insight into the
ways the first item subjects free recall differs from the set of items they
recall over tens of seconds.
To summarize, the only reliable information communicated by this ms is the
finding that the FRP curve looks more or less like the serial position curve
in free recall. This fact could have been learned from about a dozen papers,
including several cited by this ms. This paper is substandard in just about
every possible dimension a scientific paper could be evaluated. This includes
the quality of writing, scholarship, methodological rigor and intellectual
consistency. It should not be published in Acta Psychologia nor in any
peer-reviewed publication. Eugen G Tarnow October 27 2011 06:14:58 AMScience editors like to set up ethical guidelines for authors in order to better cover their rears. I have criticized these guidelines using surveys elsewhere and I just note that they are set up to pin responsibility for a paper on authors. When something goes wrong, the editors can then point to the authors and say it was their fault. They are not set up to guard scientific originality: authors do not have to reference previous papers but they do have to list anybody who contributed work to the paper whether the particular contribution was scientifically original or not. The lab director says - why don't you study X and if you study X and find something original the lab director has to be on the byline even if they just read the latest newspaper headline. Editors do not enforce any of these rules so the byline credit can go to ghost authors and lab directors and if there is a problem at some point, it can all be pinned at the junior authors by the institutions representing the senior authors. That's how unenforced guidelines run amok. There are ways to follow the guidelines that allow authors to blackmail other authors! But if we authors really want editors to be useful, they need guidelines. Since they are not likely to self-police, here are my suggestions for such. And I will be happy to get an NIH or NSF grant to help enforce those guidelines. 1. Any referees that approve a paper have to be listed with the paper. That way it is much less likely that referees who approve erroneous results can continue to approve erroneous results throughout their careers. 2. Editors should not invite other editors as referees. This limits the diversity of the published science. And it is kind of obvious - judges in our criminal system are not asked to be policemen at the same time. 3. When a published article is found fraudulent, the editorial correspondence, a secret guarded by actual law, has to be immediately opened up to the public. 4. All manuscript processing has to follow the same procedure and the particular procedure has to be published. All published articles should have a summary of the procedure used including what the referee vote was and how many referees were elicited but did not respond in a timely fashion. 5. Statistics for the manuscript process should be publically available including the mean and median times for a decision. 6. A stated procedure has to exist that encourages reviewers to review on-time. As it stands a competitor can hold up an author's paper by simply promising to review it and not doing it. 7. Experiments that reproduce or fail to reproduce an original published result should be automatically published online in the same journal as the original finding with links from the original report. The original authors should have a right to comment on the new finding but not to be able to disapprove the new finding. The journal editors should monitor any disputes and if the original finding is found incorrect, the novel findings should be published as regular articles and the original finding should be marked as incorrect. Eugen G Tarnow August 3 2011 10:35:11 AM1. It takes 0.5 to 3 seconds to put a word in your short term memory by looking at it or listening to it. This "activation" time scale roughly corresponds to the time scale of synaptic exocytosis, i.e. the time it takes to have all the messenger molecules released from the "presynaptic" side of the synapse. The activation time is longer for items that are complex than for items that are simple. 2. If you are not taking in very much information, it takes 3 seconds to 15 minutes to lose a short term memory. It "decays" relatively slowly with time (logarithmically). The time scale and functional behavior is roughly the same as for synaptic endocytosis, i.e. the time it takes for the presynaptic side of a synapse to heal back up and recreate the containers with the messenger molecules. 3. The more information that goes into your short term memory, the faster this information decays. So, for example, if you are asked to read one word every 5 seconds, this word will remain longer in your short term memory than if you are asked to read one word every second. In both cases the short term memory decays. Contrary to popular opinion one memory item does not replace another memory item. A possible explanation for this "presentation rate" dependency is that some catalyst is generated the more memory items are activated and this catalyst speeds up the synaptic endocytosis which makes the memory disappear faster. Eugen G Tarnow July 20 2011 11:24:07 AMIt was just announced that the guidelines for an Alzheimer's diagnosis is no longer an autopsy by itself, since brains with and without AD can often look the same. We are therefore back to the drawing board. We do not know what causes the disease and we also do not know enough to classify it (and much of the research that used the old classification is presumably worthless). Why is it that trillions of dollars (you check it - I did not) spent on medical research fails to impress? To discover just how corrupt medical research is you would have to go underground as a medical researcher, keep your integrity intact, and then be able to remove yourself from a field that offers so much money and so many patients and relatives thinking you are God. I am a neuroscience researcher coming out of physics. I used to think physics was corrupt (people steal authorship credits all the time) but medicine is in another class (though not when it comes to stealing authorship - at least pathology is no worse than is physics - see http://coauthorship.com). If you have caught onto how one day food X is bad for you and another day it is good for you, it is probably the same reason as we now have no idea how to diagnose Alzheimer's: medical researchers refuse to share data. They publish articles and then don't let anybody else look at the data. There are at least three reasons for refusing to share data. 1. The researchers themselves believe they can make a lot of money on the data (which they get from patients for free prodding them and cutting them up and are often paid by taxpayers to collect). 2. They are afraid of competition. Since medical research can be quite expensive to conduct, by withholding the data they have significantly upped the barriers to competition. 3. But most importantly, they are afraid that somebody will discover that their published articles are wrong. And in medicine, it may be that 99% of the research is wrong (you check it - I did not). Those of us who followed the Schoen debacle in physics - in which a physics researcher was suggested for the Nobel Prize but somehow always failed to provide samples backing up his data - know that when data is not shared, it is likely the Emperor is nude. One of the most prestigious journals, PNAS (I am not sure how to pronounce it but it is an abbreviation of Proceedings of the National Academy of Sciences) decided to pretend to do something about this state of affairs. They added a "journal policy" that states: Materials and Data Availability. To allow others to replicate and build on work published in PNAS, authors must make materials, data, and associated protocols available to readers. Authors must disclose upon submission of the manuscript any restrictions on the availability of materials or information. Data not shown and personal communications cannot be used to support claims in the work. Authors are encouraged to use SI to show all necessary data. Authors are encouraged to deposit as much of their data as possible in publicly accessible databases. Such deposition may facilitate access to data during the review process and post-publication.
Authors must make Unique Materials (e.g., cloned DNAs; antibodies; bacterial, animal, or plant cells; viruses; and algorithms and computer codes) promptly available on request by qualified researchers for their own use. Failure to comply will preclude future publication in the journal. It is reasonable for authors to charge a modest amount to cover the cost of preparing and shipping the requested material. Contact pnas@nas.edu if you have difficulty obtaining materials.
But just as other journals use "ethical guidelines" to make themselves look pretty but fail to enforce them, PNAS policy is that the data can be subject to a "transfer agreement". In fact, the journal told me that "It is standard practice for authors to issue materials transfer agreements and PNAS does not adjudicate the terms of those agreements."
In other words, you have to share the data but you can make the sharing contingent of whatever contract you like. Below I attached a contract I was asked to sign when I wanted to see the data behind "Methods to improve the detection of mild cognitive impairment" by William R. Shankle, A. Kimball Romney, Junko Hara, Dennis Fortier, Malcolm B. Dick, James M. Chen,
Timothy Chan, and Xijiang Sun. I put in my initial request for the data on May 2, 2011 upon which Mr. Shankle asked "Can you tell us what the purpose would be?" I then had to explain about the PNAS journal policy on data sharing and after 7 weeks Dennis Fortier sent me the contract these authors needed in place for me to see the data below:
If you read this contract you will find that these researchers want to be my business partners for ever in case I come up with anything commercial from having looked at their data!
I also asked for the data behind "Hippocampal disconnection contributes to memory dysfunction in individuals at risk for Alzheimer’s disease" by Travis R. Stoub, Leyla deToledo-Morrell, Glenn T. Stebbins, Sue Leurgans, David A. Bennett , and Raj C. Shah. I received an answer from Ms. Toledo-Morrell: "It is not clear from this e-mail what exactly you need and what it will be used for. Could you please clarify?" There we go again. After six weeks she told me that should would only share derived scores, z-scores, not the underlying data. PNAS opinion is not yet known but it really is immaterial - if PNAS allows for the use of contracts that they will not check, their journal policy has no teeth. Eugen G Tarnow May 31 2011 06:12:33 AMEditors insist on peer review of scientific articles before they are accepted for publication. Science magazine just asked authors to withdraw a paper ( see http://online.wsj.com/article/SB10001424052702303745304576355852212887170.html ). What I would like to know is who were the editors and peer reviewers that approved of this paper? If peer review is so essential, it surely must be equally essential to keep the peer reviewers responsible? Eugen G Tarnow May 6 2011 05:48:50 AMAfter prompting I finally received a letter from the editor of Journal of Cognitive Psychology. My paper had been sitting in their inbox for more than two months apparently because she was moving. You think there might be an apology in it? a very minimal one! But she also raises one of my pet peeves: unless one performs one more experiment the paper is not considered novel. The editor has no real understanding of what is science, and she is no different from most of the workers in the field... Dear Dr Tarnow, I am writing you concerning your manuscript #ECP-BA 11-34 entitled "The free recall search process introduces errors in short term memory but apparently not in long term memory" which you recently submitted to the Journal of Cognitive Psychology (JCP). I apologize for the delay in getting back to you. In the past few months, the journal has welcomed a huge number of newly submitted manuscripts. On the basis of a careful reading of your manuscript, I have decided to return the manuscript to you without review. JCP receives many more manuscripts than it can publish, and I have to make heavy demands on the time and energy of my Associate Editors and the expert reviewers associated with the journal. This implies that I can only send out manuscripts for review that have a reasonable chance of appealing to JCP’s broad readership. I’m sorry for having to convey this disappointing decision to you. I summarize below some of the key issues with the manuscript that have led me to this view. The half-a-page introduction is very brief, and lacks a clear theoretical and empirically justified motivation for the research question you sought to address. The Introduction is basically an enumeration of different studies. A related problem pertains to the novelty of your study. As you state at the beginning of your method section, ‘ This article does not itself involve any new experiments ..’. Rather, the paper reports a reanalysis of four published studies that were published 15, 16, 41 and 49 years ago. Without a clear and convincing theoretical reason that motivates the re-analysis of data published decades ago, the added value of such a re-analysis remains unclear. As it currently stands, your paper seems more data-driven than theory-driven in nature. This is a particular concern given that one of the major goals of JCP is to advance theoretical understanding of cognitive processes. Second, JCP’s readership includes scholars from all fields within Cognitive Psychology and allied disciplines. Not every reader is an expert in the specialized field your manuscript refers to. Your manuscript is densely written and demands specialized knowledge to understand the line of reasoning and the data that are depicted in the figures. As it currently stands, your densely written manuscript will be difficult to understand for the large majority of JCP’s readers. Finally, as explained in the Notes to Authors on JCP’s website, the style and format of manuscripts should conform to the specifications given in the Publication Manual of the American Psychological Association. Your manuscript does not follow some of the basic rules, including the report of statistical analyses and the presentation of the findings. I am sorry to say that I cannot accept your manuscript for JCP given the extent of the concerns that I outlined above. Because I anticipate that the ultimate conclusion of the review of this manuscript would be for me to not be able to accept it for publication, I felt it best to not subject it to a drawn out review with a foregone conclusion. I am sorry to have brought you this disappointing news, but I wish you all the best in your future endeavors. You are of course now free to submit your manuscript elsewhere should you choose to do so. Yours sincerely, Dr Janet van Hell Editor-in-Chief Journal of Cognitive Psychology jgv3@psu.edu Eugen G Tarnow March 30 2011 05:26:38 PMI got the following request from a referee rejecting my paper (presumably the referee was Marc Howard): "The model really needs to be compared against other models of free recall rather than SOB which is a model of serial recall. So, the obvious candidate is the Temporal Context Model (Howard and Kahana). In particular, I can't see how this model is going to be able to handle conditional response curves which are key data for free recall." I took a little time to address this issue and wrote: Kahana (1996) found that the chance that a recalled item is followed by the next item in the list is higher than that it is followed by a previous item in the list. He constructed “conditional recall probability” (CRP) graphs that he claimed could not be reproduced with the then current computer models. This was followed by the Temporal Context Model (Howard & Kahana, 2001). It argues that the u-shaped free recall curve is really not u-shaped. To minimize the primacy effect and make the u-shape a j-shape they add a distracting task at each item presentation: deciding whether a word is concrete or abstract. The primacy effect is still there for the first list item and Howard & Kahana (2001) concede that “the model has no mechanism to generate primacy, it fails to capture the small one-position primacy effect in the data.” Tarnow (2010b) showed that the rehearsal concept is an ill-defined concept that he argued should be ignored more than it is. One could also argue that too much distraction during a memory test makes the test not a pure memory test anymore. Since a higher presentation rate leads to an increased decay rate even in the non-primacy items (Tarnow, 2010a), the primacy effect makes sense – it is a function of the initial ill-defined presentation rate that mimics a slower presentation rate thus the initial items decay slower. The CRP graphs can be accommodated in the Dynamic Tagging theory as a result of chunking, i.e. the subject associates two items together and makes one memory out of them. Howard & Kahana (201) though write that “it is unlikely that the lag-recency effect is a consequence of direct interitem associations” because there seems to be what they term time scale invariance, i.e. the CRP graphs look the same even if the interitem interval is changed from 2 to 16 seconds. The evidence for this invariance is limited to Fig. 2 in Howard & Kahana (2002) in which the interitem interval is varied from 2 to 16 seconds. Unfortunately, the data in the figure has error bars much too large to prove their point. The data behind the Dynamic Tagging model (Tarnow, 2008) shows the slow decay of short term memory on a logarithmic time scale which makes this point even harder to prove. But the main weakness of the Temporal Context Model is that it was created purely to fit the CRP curves using abstract concepts without any regard to the underlying biology. It has choices, for example, “ff one prefers the response–suppression approach, then TCM can be used to generate a set of activations, which can in turn be modulated by response suppression” (Howard & Kahana, 2001). It can be anything to anyone which also means it has no predictive power and carries no information. Do you agree or disagree? Feel free to post a comment! Eugen G Tarnow March 28 2011 09:25:34 AMA few months ago I requested anonymized data on the short term memory of Rush Alzheimer's Disease Center patients who died. I had first been told that if I wanted to use the data to develop a patent, I would not be given access to it. Recently my request was rejected (yes the "process" took four months!) even though I said I was not going to use it to develop a patent: "Final comment : This project was not considered sufficiently meritorious by the review committee. The analytic plan was not considered acceptable. The DeKosky reference was not from a Rush study and the Scheff data that is from a Rush study is not available to the RADC nor does the RADC know the IDs that were used in that study as nobody in the RADC worked on that study." There was no signature and I was told that Dr. David A. Bennett is so busy he should not be contacted, I believe he is the head of the committee. Today I sent in a FOIA request to the general counsel and quickly got the following answer: "The Illinois Freedom of Information Act (“IFOIA”) only applies to the disclosure of public records of Illinois public bodies and does not extend to the disclosure of records by private entities such as Rush University Medical Center. Furthermore, The Federal Freedom of Information Act (“FOIA”) only applies to federal agencies. Rush is not required under either the IFOIA or the FOIA or any other regulations to provide you the information you have requested and Rush is unable to comply with your request." The problem is: when competitors want to get a hold of data that patients have given a federally supported research center, does it server the patients' interest to deny access? I think not. Eugen G Tarnow January 5 2011 12:34:43 PMI recently had a referee reject a paper of mine stating "Most of the relations that the author reports in the manuscript are not new (relations between response latency and probability of error, number of items recalled and the error frequency, recall order and error rate)." The referee did not state any supporting references. What should the responsibility of the editor be in this situation? Should he/she not ask for supporting references? Should referees be allowed to issue judgements arbitrarily when they are protected by anonymity? Is that not inappropriate? Eugen G Tarnow January 2 2011 09:22:51 PMResponse probability and response time: a straight line, the Tagging/Retagging interpretation of short term memory, an operational definition of meaningfulness and short term memory time decay and search time Cognitive neurodynamics, 2008, 2(4), 347-353. The functional relationship between correct response probability and response time is investigated in data sets from Rubin, Hinton and Wenzel, J Exp Psychol Learn Mem Cogn 25:1161–1176, 1999 and Anderson, J Exp Psychol [Hum Learn] 7:326–343, 1981. The two measures are linearly related through stimulus presentation lags from 0 to 594 s in the former experiment and for repeated learning of words in the latter. The Tagging/Retagging interpretation of short term memory is introduced to explain this linear relationship. At stimulus presentation the words are tagged. This tagging level drops slowly with time. When a probe word is reintroduced the tagging level has to increase for the word to be properly identified leading to a delay in response time. The tagging time is related to the meaningfulness of the words used—the more meaningful the word the longer the tagging time. After stimulus presentation the tagging level drops in a logarithmic fashion to 50% after 10 s and to 20% after 240 s. The incorrect recall and recognition times saturate in the Rubin et al. data set (they are not linear for large time lags), suggesting a limited time to search the short term memory structure: the search time for recall of unusual words is 1.7 s. For recognition of nonsense words the corresponding time is about 0.4 s, similar to the 0.243 s found in Cavanagh (1972). http://www.springerlink.com/content/971t8p4443385231/ |
|