Last updated on January 6th, 2024 at 08:33 am

A tool to address a major health problem—Cognitive Impairment (Ci)

The following article on the RIA appeared in Applied Neuropsychology March of 2017.  The first author, Dr. Gerald Goldstein, was a friend, supervisor and research colleague of Dr. Ruthven.  Dr. Goldstein, now diseased, believed that the RIA had a lower floor of cognitive impairment than standard comprehensive neuropsychological test batteries.  These neuropsych batteries are very powerful in diagnosing structural static and progressive brain damage/disease.  However, the RIA may with further research diagnose reversible brain impairment and perhaps differentiate from the RIA test pattern reversible from irreversible brain impairment.  The whole area of reversible impairment is little understood and deserves greater attention from the research community because of its major impact on every day healthcare. In developing the RIA Dr. Ruthven believed the RIA might be a bridge to a timelier and more cost-efficient path to the final impairing diagnosis.

An Introduction to the Ruthven Impairment Assessment (RIA): A Stability Study

Gerald Goldstein, Jibo He^b, Leslie Ruthven and Jon Walker

^a Mental Illness Research, Educational, and Clinical Center, VA Pittsburgh Healthcare System, Pittsburgh, Pennsylvania; ^department of Psychology, Wichita State University, Wichita, Kansas;  Preferred Mental Management, Inc., Wichita, Kansas; ^research Service, VA Pittsburgh Healthcare System, Pittsburgh, Pennsylvania

While there are a number of recently published neurocognitive tests (Kaufman-Delis Executive Function System, Delis, Kaplan, & Kramer, 2001; Neuropsychological Assessment Battery, White & Stern, 2003; the Cambridge Neuropsychological Test Automated Battery, Robbins et al., 1998; the Measurement and Treatment Research to Improve Cognition in Schizophrenia (MATRICS) Consensus Cognitive Battery [MATRICS], Nuechterlein et al., 2008; and the Repeatable Battery for the Assessment of Neuropsychological Status [RBANS], Randolph, 1998), there may be a need to develop new procedures that have a group of characteristics in common: (a) These test procedures should not have the floor characteristics of many tests developed with patients with structural brain damage. They should assess a wide range of impairment to allow for meaningful scores for people ranging from normal to significantly impaired. (b) The tests should measure complex abilities that are necessary for the more challenging aspects of daily living such as being productive in competitive employment. (c) The tests should be suitable for both clinical and nonclinical settings and capable of monitoring large numbers of participants with interval retesting over time. (d) The tests should be brief in duration, should not involve elaborate materials, and should ideally have the capability of self-administration, not requiring administration by a professional or highly trained examiner. (e) The tests should be repeatable with minimal or no practice-effects improvement over time to allow serial testing of ongoing or changing cognitive status. (f) The system should allow storage of test results over repeated testing and should allow for intervention when clinically indicated. Recently developed screening and diagnostic batteries, notably the RBANS (Randolph, 1998) and MATRICS (Nuechterlein et al., 2008), have many of these characteristics but were designed for clinical populations and are not appropriate for cognitively normal people followed for a span of years.

The Ruthven Impairment Assessment (RIA) was developed while keeping in mind many of the aforementioned test characteristics including the capacity to monitor large numbers of people with regard to assessing their current cognitive status. It was designed to be appropriate for healthy individuals, with the goal of using it clinically following hopefully successful discriminative validity studies with impaired populations and to be repeatable without significant practice effects for monitoring changes in cognitive ability. For that purpose, in addition to the RIA itself, a personal medical history and a brief behavior survey are now under development and are stored in the database for review. Thus, the major goal is the development of a procedure that includes a repeatable test that is challenging to normal individuals and has the capability of identifying impairment when it occurs on longitudinal testing. Its intent is therefore somewhat different from that of the tests listed previously and that are typically used on referral or for research purposes when there is already suspected or known impairment.

The present study is concerned with the stability of the RIA. It does not address the matter of validity with regard to identification of cognitive impairment because the view was taken that clinically significant change in performance can only be identified after the stability of the measure and particularly the influence of practice effects could be assessed in normal individuals. Thus, this study should be understood as preliminary to subsequent validity studies. The definition of stability adopted for the RIA was the absence of a statistically significant (p< .05) difference within participants’ RIA scores across testing sessions. Practice effects would be observed if there was significant improvement in the group’s RIA performance across sessions. This procedure was accomplished with one-way repeated measures analysis of variance (ANOVA; Statistical Package for the Social Sciences [SPSS]; IBM, 2012) with test session as the independent variable. Thus, the present study is based on data from the RIA obtained from normal individuals administered the test over several occasions to seek the presence or absence of statistically significant score changes across testing sessions.

Methods

Participants

The sample consisted of college-age female and male students who were actively engaged in their studies at the time of testing. One subgroup was recruited by Dr. Ruthven from among members of the college table tennis team (n¼ 15), and the other subgroup was recruited by Dr. Jibo He from among psychology students at Wichita State University (n¼ 46). Preliminary analysis showed no demographic or RIA score differences between these two groups, and the two subgroups were combined for statistical purposes. Comprehensive health screenings were not conducted on either group, but inquiry did not reveal any health or psychiatric history that would preclude participation in the study. All participants signed consent forms and agreed to several testing sessions. The intertest time span for the three testings never exceeded 3 weeks and was never less than 24 hr. Participants were paid a nominal fee for participation in the study.

Procedure

Description of the Ruthven Impairment Assessment. The RIA is a brief (15 min or less), computer-delivered performance measure consisting of five subtests, each of which appears on a standard computer screen. For each subtest, instructions appear on the left half of the screen and the right half contains a large standard numbered clock face. As an example, a picture of the screen of Subtest 1 (called Level I on the screen) is presented in Figure 1. The first three subtests are typical reaction time (RT) tests with the participant instructed to press the space bar on the computer key board as quickly as possible according to the adjacent instructions. On all subtests, the participant’s response initiates the feedback of “correct” or “incorrect.” On the three RT subtests, the participant is instructed to press the space bar once when ready to start the task. A description of each of the subtests is in the following paragraphs.

Subtest 1 (Simple RT). The participant presses the space bar when any of the 12 clock numbers light up. It is therefore a simple RT task.

Subtest 2 (Complex RT). The participant presses the space bar when only even clock numbers appear, making it a more complex RT task.

Subtest 3 (Conditional RT). The participant is instructed to press the space bar when the number following two even numbers appears except when the following number is a 5 or an 11, which are the two number exceptions rotating from test to test. With the 5 and 11 number exceptions, if the participant does not press the space bar when a 5 or 11 follows two even numbers, the nonresponse initiates a “correct” on the screen. If the participant presses the space bar after a 5 or an 11 appears, “incorrect” appears on the screen. On these three RT tests, bar pressing must occur within 1.5 s of the stimulus; individual RTs are recorded for only the correct responses.

Subtest 4 (Sequencing). This subtest was meant to provide a measure of sequential attention/memory. It is performed by moving the computer mouse and cursor and left-clicking to repeat the sequence of lighted numbers just presented. For each item of 3, 4, 5, and 6, series of numbers are lighted sequentially, and each of four series is presented twice. The instructions direct the participant to left-click and repeat on the clock the sequence of lighted numbers just presented. When ready to start the task, the participant is instructed to left-click on any part of the clock face to begin. This subtest appears to have a working-memory component.

Subtest 5 (Anticipation). There are 13 items on this subtest. It is meant to be a measure of inference making and pattern identification. The first 3 items are very simple and were included more for explanatory purposes about the nature of the task. All participants are exposed to the first 3 easy items, and the 10 remaining items are randomly generated from various types of items for each testing. The participant is instructed to observe each sequence of five lighted numbers, try to identify a theme or pattern in each five-number series, and anticipate and left-click on the clock numbers that would seem to best continue or complete the five-number series just presented. Item 1 is a lighted number series of 1, 2, 3, 4, and 5, and the participant would respond by left-clicking on the numbers 6 and 7, followed by a “correct” on the screen. Items 4 through 13 are substantially more demanding than the first 3 items, and the task might be considered to measure anticipatory thinking or making inferences about future events based on presently available information. Subtest 5 was created to assess higher-level complex problem solving that was likely to be correlated with general intelligence as well as sensitive to the cognitive consequences of brain dysfunction.

The RIA is potentially self-administering, but in the present study, the participant was instructed by an administrator on how to perform the five subtests and was cautioned to read and understand the subtest instructions prior to starting each of the five tasks. At this point, the administrator left the room and returned after the test had been completed for a debriefing.

Subtest scoring. The RIA is scored by computer, which generates a printed report containing the results. An example report is presented in Figure 2. The following scoring system is used: For Subtest 1 (Simple RT), if the participant presses the space bar within 1.5 s of stimulus onset on any of the 12 lighted clock numbers, the RT is recorded and the participant is given a “correct.” The number correct out of 10 trials administered is recorded, and the individual RT is recorded for each correct response. A mean RT is reported based on the number of correct responses. For Subtests 2 (Complex RT) and 3 (Conditional RT), the scoring is the same as in Subtest 1, but for Subtest 2, the direction is to press the space bar only after the even clock numbers of the 10-item task appear; for Subtest 3, the exception is introduced for numbers 5 or 11 preceding an even

Figure 2.  Ruthven Impairment Assessment score report.

number. The scores are RTs and the number of correct responses out of a possible 10. For Subtest 4, the score is the number of light series sequences correct out of 8. The participant’s score for Subtest 5 is the number of correct patterns identified out of 13.

Research design and data analysis

The aims of this study were to describe the RIA, a new computer-delivered neuropsychological test, and to provide evidence of its stability in a sample of healthy young individuals. The investigators used a repeated measures ANOVA design to evaluate the hypothesis that test results of the group would not differ across the three test sessions. One-way repeated-measures ANOVAs (SPSS; IBM, 2012) were executed for the test session scores. Significant differences were followed up with comparisons using the Scheffe and Tukey tests. Non-significant F ratios would indicate that score differences across test sessions were random fluctuations explainable by chance and, conversely, that there was no causal event such as practice effects operating across the three testing sessions. Thus, the goal of the study was to determine if the RIA was or was not a stable instrument to determine one’s cognitive status longitudinally.

Results

Demographic data

The sample was 42.6% male and 57.4% female. Chi-square analysis for each of the scores revealed no significant (p< .05) differences between women and men for all of the items, and so it was not necessary to perform separate analyses by gender. The mean age for the sample was 24.77 years (SD¼ 5.86 years) and ranged from 19 to 53 years. The median age was 24 years. Pearson r correlations were computed for age with the test items. The results are presented in Table 1. There was a significant (r¼ .28, p< .05) correlation between age and the Subtest 1 (Simple RT) score. RT became slower as age increased. There was a highly significant (r ¼ .37, p¼ .003) negative correlation between age and the Subtest 4 score. Subtest 4 is a measure of sequential reasoning in which participants repeat sequences, and the task possibly involves working memory.

Table 1.  Pearson r correlations between RIA scores and age.

Note. RIA ¼ Ruthven Impairment Assessment; RT ¼ reaction time. *p< .05.

**p< .01.

RIA results

The ANOVA results are presented in Table 2. Inspection of this table indicates the only significant result was for the number of correct items on Subtest 2 (Complex RT), but the Scheffe and Tukey tests indicated that there was not a significant test session performance difference across the three test sessions. These results would indicate that the RIA is stable with healthy, cognitively normal young adults. Group performance scores showed no change over the three testing sessions with only one exception, indicating that RIA performance is free of any measurable practice effect. In most cases, RTs were non significantly slower on repeated testing.

Discussion

The RIA appears to have several of the structural and psychometric characteristics important for the development of new psychological instruments, particularly tests that focus on the often subtle cognitive dysfunction sometimes associated with systemic illness or life events that compromise brain function. However, for purposes of future longitudinal evaluation in clinical studies, it is

Table 2.   ANOVA results for the RIA variables.

It was necessary to do a preliminary analysis of stability of the measure, particularly with regard to evaluating the influence of possible practice effects. The present study had that purpose.

The RIA is potentially fully self-administering. In this study, the participant was instructed by an administrator on how to perform each of the five subtests and was cautioned to read and understand the subtest instructions prior to starting each of the five tasks. Then the administrator left the room and returned after the test had been completed for a debriefing. These instructions can be put on a computer screen and the test can be put online, so that it can be taken at home or in other settings where the Internet is available. The RIA, if validity is proven, may become part of a system for longitudinal assessment that may provide timely and cost-effective identification of cognitively impairing conditions. The results of this study would indicate that longitudinal validity studies with clinical populations can proceed without undue concern about possible practice effects.

The subtests were constructed with the aim of evaluating numerous complex cognitive abilities including speed of information processing measured with RT procedures and higher-level complex problem solving evaluated with the Anticipation subtest that is likely to be correlated with general intelligence as well as sensitive to the cognitive consequences of brain dysfunction. The data in the current study indicate that RT increases with task complexity. There is also some suggestion that simple RT slows with age, as has been noted in the literature for some time (e.g., Botwinick, Brinley, & Robbin, 1959). Perhaps this slowing may occur in younger individuals with emerging brain dysfunction. Longitudinal testing with the RIA might identify that process.

The evidence-based construct validity of the procedure associated with the functions assessed with the

different subtests remains to be determined. These associations were formulated during the development of the RIA on the basis of the existing neuropsychological literature as presented in Lezak, Howieson, Bigler, and Tranel (2012) and Morgan and Ricker (2008). If the RIA or something comparable can achieve the goals of early identification of impairing conditions and enabling an expedited diagnosis and treatment, such an assessment system should improve the quality of health care. We have suggested that to meet the challenges of clinical assessment and treatment planning for disorders that are relatively new to neuropsychology, it will be necessary to develop new tests that capture the sometimes subtle cognitive dysfunction associated with these conditions. In our view, the RIA is therefore worthy of further development as a clinical and research tool, particularly involving studies of the assessment of known diagnostic groups with disorders that may influence neurocognitive function. For validation purposes, there is a need for the application of RIA testing to a variety of clinical groups both in research and clinical settings.

The benefits of more challenging and repeatable tests in neuropsychology should be emphasized. The often simple tests so effective with patients with brain damage may often not be appropriate for patient who have only recently become of interest to neuropsychologists. Often there is no documented structural brain damage in these illnesses. Rather, there is frequently an environmental hazard, systemic disease, or system failure. These disorders include such phenomena as multiple chemical sensitivities (Labarge & McCaffrey, 2000), sleep apnea (Lal et al., 2012), hypoxic conditions sometimes associated with high altitude and other causes sometimes associated with rapid exposure to high altitudes (Cavaletti & Tredici, 1993; Hornbein, Townes, Schoene, Sutton, & Houston, 1989), chronic fatigue syndrome (Busichio, Tiersky, DeLuca, & Natelson, 2004), and systemic disorders such as hypertension (Kupferman, Lande, Adams, & Pavlakis, 2013) and diabetes (Kodl & Seaquist, 2008). Many of these and other disorders are often difficult to diagnose. For impaired patients, there are sometimes major delays before an accurate assessment is made and appropriate treatment is initiated. If the RIA proves to be a reliable and valid test in determining cognitive status and if the current finding holds up that there is no or minimum practice effect with repeated testing over long periods of time, the RIA is likely to signal the onset of a variety of cognitively impairing conditions that require intervention. The RIA may be applicable to many health care settings and could be used productively under the guidance and monitoring of a clinician.

In summary, the RIA, a newly developed, brief, computer-delivered neurocognitive test was described and given to healthy, presumably cognitively normal college students over three testing sessions. ANOVAs were performed comparing testing sessions. The most important finding was that the RIA is relatively free of practice effects and may be used longitudinally for early identification of individuals who may be developing cognitive impairment. With repeated testing, it appears to have the potential for aiding neuropsychologists in monitoring and managing longitudinal assessment, which could lead to making more timely final diagnoses.

Funding

This article is based on work supported in part by the Department of Veterans Affairs, Veterans Health Administration, Office of Research and Development. The project was funded by infrastructure support funding from the VISN 4 Mental Illness Research, Education, and Clinical Center (Director: D. Oslin; Pittsburgh Site Director: G. Haas), VA Pittsburgh Healthcare System.

References

Botwinick, J., Brinley, J. F., & Robbin, J. S. (1959). Modulation of speed of response with age. Journal of Genetic Psychology, 95, 137–144. doi:10.1080/00221325.1959.10534250

Busichio, K., Tiersky, L. A., DeLuca, J., & Natelson, B. H. (2004). Neuropsychological deficits in patients with chronic fatigue syndrome. Journal of the International Neuropsychological Society, 10, 278–285. doi:10.1017/s13556177

04102178

Cavaletti, G., & Tredici, G. (1993). Long-lasting neuropsychological changes after a single high altitude climb. Acta Neurologica Scandinavica, 87, 103–105. doi:10.1111/j.16000404.1993.tb04085.x

Delis, D. C., Kaplan, E., & Kramer, J. H. (2001). The DelisKaplan Executive Function System (D-KEFS). San Antonio, TX: Psychological Corporation.

Hornbein, T. F., Townes, B. D., Schoene, R. B., Sutton, J. R., & Houston, C. S. (1989). The cost to the central nervous system of climbing to extremely high altitudes. New England Journal of Medicine, 321, 1714–1719. doi:10.1056/ nejm198912213212505

IBM. (2012). Microsoft. NET user guide for IBM SPSS Statistics. Armonk, NY: Author.

Kodl, C. T., & Seaquist, E. R. (2008). Cognitive dysfunction and diabetes mellitus. Endocrine Reviews, 29, 494–511. doi:10.1210/er.2007–0034

Kupferman, J. C., Lande, M. B., Adams, H. R., & Pavlakis, S. G. (2013). Primary hypertension and neurocognitive and executive functioning in school-aged children. Pediatric Nephrology, 28, 1601–1610. doi:10.1007/s00467-0122215-8

Labarge, A. S., & McCaffrey R. J. (2000). Multiple chemical sensitivity: A review of the theoretical and research literature. Neuropsychology Review, 10, 183–211.

Lal, C., Strange, C., & Bachman, D. (2012). Neurocognitive impairment in sleep apnea. Chest, 141, 1601–1610. doi:10.1378/chest.11-2214

Lezak, M., Howieson, D., Bigler, E., & Tranel, D. (2012). Neuropsychological assessment (5th ed.) New York, NY: Oxford University Press.

Morgan, J. E., & Ricker, J. H. (2008). Textbook of clinical neuropsychology. New York, NY: Taylor & Francis.

Nuechterlein, K. H., Green, M. F., Kern, R. S., Baade, L. E., Barch, D. M., Cohen, J. D., & Marder, S. R. (2008). The MATRICS Consensus Cognitive Battery, Part 2: Co-norming and standardization. American Journal of Psychiatry, 165, 214–220. doi:10.1176/appi.ajp.2007. 07010042

Randolph, C. (1998). The Repeatable Battery for the Assessment of Neuropsychological Status. Manual. San Antonio, TX: Psychological Corporation.

Robbins, T. W., James, M., Owen, A. M., Sahakian, B. J., Lawrence, A. D., McInnes, L., & Rabbitt, P. M. (1998). A study of performance on tests from the CANTAB battery sensitive to frontal lobe dysfunction in a large sample of normal volunteers: Implications for theories of executive functioning and cognitive aging. Journal of the International Neuropsychological Society, 4, 474–490. doi:10.1017/ s1355617798455073

White, T., & Stern, R. A. (2003). NAB: Neuropsychological Assessment Battery. Lutz, FL: Psychological Assessment Resources.