Better information provides the groundwork for better decisions. This belief is central to The Robert Wood Johnson Foundation's commitment to funding research aimed at improving health and health care.

In pursuit of better information, the Foundation makes grants to research analysts that support a full range of approaches.¹ In most cases, decisions to support research are made after Foundation staff members identify a specific health problem and conclude that better research may help society resolve or ameliorate it. In this sense, funding of research at the Foundation competes with funding for other types of interventions. Research, therefore, is generally of the applied sort, and is designed to make specific contributions to solving a health problem that has been identified as a Foundation priority.

Over the past five years, from 1994 through 1998, the Foundation has awarded 1,154 grants involving research, policy analysis, and evaluation totaling $366 million.² This is small compared with the $11 billion in extramural research grants in 1998 by the National Institutes of Health (NIH) primarily to support biomedical research. And the Foundation's investment is small compared with the $1.9 billion in biomedical research support funded by the Howard Hughes Foundation between 1993 and 1997. However, the $366 million in Foundation grants represents a large share of the total support for behavioral, social, and organizational research in the health sector. As Figure 7.1 shows, in 1998, research, evaluation, and policy analysis accounted for more than 30 percent of the Foundation's awards. As a strategy for addressing the Foundation's goals, funding for these grants has increased steadily since the 1970s.

RATIONALE FOR FUNDING RESEARCH, POLICY ANALYSIS, AND PROGRAM EVALUATION

When The Robert Wood Johnson Foundation was launched as a national philanthropy in 1972, two principles guided the development of funding for research. First, the Foundation decided not to compete with the NIH as a source of funding for medical research. As the Foundation's investments would be small relative to those of the NIH, they could not be expected to have a noticeable impact. This decision translated into a reluctance to fund biomedical research. Second, the Foundation chose not to become a think tank for federal policy. Such a role, it was feared, would embroil the Foundation in partisan politics.

Rather, the Foundation developed its own justification for funding research, policy analysis, and evaluation. Foremost was improving the nation's ability to understand key health and health care issues so that decisions can be made concerning the way Americans maintain health and obtain health care. Research, policy analysis, and evaluation were also expected to inform and influence the decisions of the Foundation itself in designing grant programs.

Testing the Effectiveness of Foundation-Funded Programs

Given the background of some of the Foundation's board members in the pharmaceutical industry, it was natural that early investments in research and evaluation were motivated by analogies to drug trials--a sense of the importance of testing whether programs funded by the Foundation were effective and were not having unintended negative effects^.3 In this sense, program evaluations conducted by independent outside reviewers were a natural companion to the multisite demonstration programs that characterized grant making in the early years of the Foundation.

In at least four notable cases, early interventions supported by the Foundation closely resembled clinical trials in which research and evaluation considerations drove the program design and implementation:

• In 1973 and 1974, researchers at the Educational Testing Service performed a randomized trial of a program aimed at increasing the number of community dentists prepared to care for the physically and mentally handicapped.⁴
• Beginning in 1979, David Olds directed a series of demonstration projects that rigorously studied the impact of providing home visiting by nurses on high-risk mothers and their children.⁵
• The Infant Health and Development Program, conducted in the mid-to-late 1980s, involved an eight-site randomized trial to test the effects of comprehensive early interventions in reducing health and development problems for low-birthweight premature infants.⁶
• From 1977 to 1982, the national preventive dental care program for school-age children assessed the costs and effectiveness of various types and combinations of school-based preventive dental care procedures. ⁷

Each of these early studies included careful evaluation. They marked the beginning of the Foundation's commitment to evaluation, and led to the presumption that most national multisite programs would include an evaluation by outside experts. Over the past twenty-five years, ninety such evaluations of Foundation-funded programs have been done.

Learning About Health and Health Care

In recent years, many Foundation programs focused less on testing new models of care and more on actively promoting social change. This shift has led to a more varied approach to evaluation--where the first central question is how each major Foundation initiative could be considered a learning opportunity. For example, when the Foundation funded a national program to build antismoking coalitions across the states beginning in 1993, the evaluators did not assess whether the program "worked," but used the opportunity to study how coalitions form and operate and to learn principles that could guide future efforts to develop coalitions.⁸

The Foundation also uses research, policy analysis, and evaluation to assess the changes in health care practices and systems that are happening naturally or with funding from sources outside the Foundation. For example, the Health Tracking program involves a series of grants that look at how managed care and other market-oriented changes are affecting health and health care.⁹ Another example is the Foundation's support of the Urban Institute to assess the impact of the federal Children's Health Insurance Program.¹⁰

Obtaining Baseline Information

The Foundation often funds surveys to obtain basic information about important health or health care problems. One series of surveys measured the number of Americans who experienced problems in gaining access to health care. The first such survey was done in 1976 by researchers at the University of Chicago, and since that time the Foundation has supported three periodic access surveys. The access-to-care surveys are examined by Marc Berk and Claudia Schur in the 1997 Anthology.¹¹ Another series of surveys looked at the way physicians practiced medicine; between 1973 and 1976, a set of surveys of 10,000 physicians in twenty-four specialties found that one of every five Americans received general medical care from a specialist physician.¹² A later series of surveys sought to understand how young physicians decided which specialty to enter.¹³

Survey research has been credited with making an important contribution in shaping other investments of the Foundation. The first physician survey convinced the Foundation to support training programs for generalist physicians, and the access-to-care surveys led the Foundation to try to expand primary care by funding primary care group practices within hospitals.¹⁴

An important feature of Foundation-funded survey research has been a commitment to making the data sets public as quickly as possible. In fact, all Foundation-funded surveys that are judged to be of potential use by other researchers are put in the public use data files at the University of Michigan's Institute for Survey Research.¹⁵ This policy has led to the surveys being used by a range of researchers to conduct studies related to health and health care.

Understanding Public Policy Choices

Over time, the Foundation's definition of research began to include policy analysis, which involves background analysis to support the development of new public policies at the federal and state levels. Generally, policy analysis identifies a range of options that address a specified social problem and then makes as clear as possible the likely advantages and disadvantages of each option. Such analysis is sometimes done by academic researchers, but is more often conducted by the staff of government agencies or of nonprofit organizations that have the goal of encouraging information-based policy development.

An example is the Foundation-funded program called State Initiatives in Health Care Reform, examined by Beth Stevens and Lawrence Brown in the 1997 Anthology.¹⁶ This program, which aims at improving access to health care for vulnerable populations in sixteen states, largely supports policy analysts working for state governments. These analysts collect data, examine a range of policy options, use technical assistance provided by national experts, and communicate ideas for reform to executive and legislative leaders in their states. The intention of the program is not to support a specific policy agenda but, rather, to provide resources so that locally identified options to improve access to health care can be researched and then debated.

Understanding Health-Related Behavior

Since 1990, the Foundation's commitment to reducing the harm caused by smoking, alcohol, and illegal drugs has created a demand for information about which public policies and private interventions could reduce the use of health-threatening substances and which behavioral and environmental interventions could help people reduce or stop their use of tobacco, alcohol, and illegal drugs.

The experiences of the Foundation in supporting behavioral and policy research related to substance abuse have been featured in two Anthology chapters: one by Nancy Kaufman and Karyn Feiden (Chapter Eight in this volume) and one by Marjorie Gutman and her colleagues in the 1998-1999 Anthology.¹⁷

Building the Health Care Field

Although most Foundation-funded research, policy analysis, and evaluation focuses on a specific health problem of interest to the Foundation, training programs, which represent 17 percent of Foundation grant making, strive to build the health care field generally. Of nineteen education and training programs currently under way, four focus on training and career development of future researchers. The recipients range from minority scholars advancing as faculty members in medical schools to postdoctoral fellows in economics, political science, and sociology.

THE KINDS OF ANALYSIS SUPPORTED BY THE FOUNDATION AND HOW THEY GET FUNDED

The staff of The Robert Wood Johnson Foundation must make decisions about when research will help advance a Foundation goal and exactly what type of research to support. In this regard, the Foundation hedges its bets by using a range of approaches that vary in the degree to which they are guided by staff members or by investigators and in the degree of flexibility given to funded researchers. The approaches to setting priorities and reviewing proposals can best be understood by considering these categories: research, policy analysis and planning, program evaluation, research-driven demonstrations, and training.

Research Projects

STAFF-INITIATED RESEARCH PROJECTS. Each fall, the staff at the Foundation outlines a two-year grant-making plan that includes a series of strategic objectives--for example, expanding the number of children covered by health insurance. In determining how to further an objective, the staff considers, among other approaches, demonstration programs, communications activities, and research projects.

This autumn planning process often identifies a range of research and analysis ideas that staff members then attempt to turn into funded projects. To do so, they might solicit a proposal from an expert, or they might seek competing proposals on the topic and then, after internal and external review, select one for funding. These projects are usually modest in scale, but they can be ambitious, large, and long-term, such as the Health Tracking initiative discussed in Box 7.1. Although these projects are initiated by the staff, the ideas and the approach are often influenced by experts with whom the staff consults.

AD HOC RESEARCH PROJECTS. Each day's mail at the Foundation brings a steady flow of proposals. Researchers often send a short letter of interest outlining their research idea and why the topic is of practical importance. Each of these ad hoc, or over-the-transom, proposals is reviewed by the staff and, often, by external reviewers. If the topic and proposed research look promising, the staff invites the applicant to prepare a more detailed proposal.

Ad hoc research proposals that address topics related to the Foundation's strategic objectives have the best chance of being funded. Those not related to current priorities can be funded when the staff judges them to represent unusual opportunities to gain information about an important health sector problem. However, the success rate for such proposals is not high--probably around 10 percent. In addition to reviewing the relevance of the topic, the staff examines proposals for the quality of the proposed research, its feasibility, the qualifications of the investigators, and the reasonableness of the budget.

INVITATIONAL INITIATIVES. In some cases, the Foundation attempts to stimulate research in a given area by inviting proposals. These invitational programs give researchers a great deal of flexibility to define projects that they think are important within the area targeted by the Foundation. Each proposal is peer-reviewed; in most cases, selections are made by a national advisory committee. The Foundation often funds an external organization and a program director, who is an expert in the targeted area, to manage the selection, monitoring, and dissemination of the research.

Currently, there are six invitational projects in progress:

• Investigator Awards in Health Policy Research Program: funds a broad range of research on topics with the potential to illuminate or improve health policy ¹⁸

  Box 7.2. Improving Access to Care by Restructuring Provider Payments: An Ad Hoc Research Project. One of the three goals of the Foundation is to improve the care of people with chronic health conditions. It seems essential that health plans be adequately and fairly compensated for the unusually high costs of caring for people with serious health conditions. A reasonable approach to payments seems like a crucial first step to ensuring that high-quality care is received by the chronically ill. Professor Harold Luft and his colleagues at the Institute for Health Policy Studies at the University of California, San Francisco, proposed a project to develop new methods for paying managed-care plans for the special care they provide to the small percentage of beneficiaries that have very expensive health care needs. The project was approved by the Foundation in 1997 for three years at a level of $2.5 million. Under the Improving Access to Care by Restructuring Provider Payments project, researchers at the Institute for Health Policy Studies are collecting detailed information on costs and health status from selected managed-care plans. The researchers will then analyze the cost of treating patients with chronic health conditions and compare it with the costs of treating patients not requiring the same levels of care. Based on the results, the researchers plan to develop a payment system that can be adopted on a trial basis..

• Substance Abuse Policy Research Program: supports investigation of a variety of topics directed at helping the nation reduce harm from substance abuse¹⁹
• Home Care Research Initiative: encourages research on how to target resources for care to disabled people in their own homes
• Changes in Health Care Financing and Organization: funds research related to improving the financing and organization of health care. This program is described in Box 7.3. ²⁰
• Improving Clinical Care for Chronic Illness: supports research to develop better clinical and organizational approaches for serving the chronically ill
• Strengthening the Patient-Provider Relationship in a Changing Health Care Environment: supports analysis of the impact that the changing health care environment has on the patient-provider relationship

RESEARCH NETWORKS. The Foundation also brings together researchers to form networks that investigate a topic. For example, the Research Network on the Etiology of Tobacco Dependence consists of researchers interested in understanding patterns of tobacco initiation, use, and dependence. Under the Substance Abuse Surveillance Initiative, researchers at five institutions monitor changes in state-based substance abuse policy and study how these changes influence the harmful use of substances by young people.

Policy Analysis and Planning

Most of the Foundation's funds for policy analysis and planning go to large national programs that provide resources to states and state-based nonprofit organizations to examine options for improving health, health care, and health policy. The State Initiatives in Health Care Reform program funds policy analysis at the state level. The Turning Point program makes grants to state health departments to rethink their public health system. Over the years, a series of national programs has provided resources for policy analysis and planning directed at the complicated issues surrounding payments for nursing homes and community-based long-term care. Within these national programs, grants are made competitively after eligible organizations submit proposals and panels of experts review them.

Smaller grants fund conferences that bring experts together to consider policy options at the federal and state levels. Other grants support a wide array of thinkers to consider innovative approaches for reforming the way the nation organizes health care, promoting healthy behavior, and dealing with the problems related to tobacco, alcohol, and illegal drugs. Some of these projects are solicited by staff; others are funded through the six investigator-initiated research programs described earlier.

Program Evaluation

Evaluations try to draw lessons from programs that will advance society's and the Foundation's understanding of the barriers to and opportunities for improving health and health care. They also help the staff determine whether, and how, a program can be replicated or modified. Evaluations focus on what can be learned and communicated from a program; they do not monitor individual grant management and administrative performance. The administrative staff at the Foundation and the national program offices that are set up to facilitate the operations of each multisite initiative perform these functions.

The staff members, in collaboration with an outside evaluator, usually determine the approach to a specific evaluation of a national program. Some evaluations are formal and focused on outcome assessment. For instance, the evaluation of Fighting Back, a multisite community demonstration program to reduce the use of and harm associated with alcohol and illegal drugs, collects extensive data on outcomes and participants' experiences, as well as qualitative information on the processes used to carry out the initiative in each of the participating communities.²¹

Other evaluations are less formal and rely on assessment by experts. For example, in 1993, two experts on medical education and research training visited each site of the Foundation's Clinical Scholars Program. After many interviews and a review of project documents, they provided an assessment of the program's contributions to developing physician leaders of health services research. The assessment also made practical suggestions for strengthening the program.²²

Recently, the Foundation has invested in formative evaluations and self-evaluations when it is important for a grantee to obtain regular information about its efforts to attain specified goals. Under formative evaluation, an outside expert provides continuing feedback to a grantee on intermediate outcomes and on the effectiveness of various implementation strategies. Self-evaluation allows a grantee to set up internal mechanisms for assessing intermediate outcomes and progress toward goals. These formative evaluations and self-evaluations often prompt grantees to continue reviewing systematically whether strategies are working.

Grantees that conduct evaluations are sometimes selected as a result of a competitive process using requests for proposals. Often, however, staff members choose a specific research team to conduct an evaluation. The teams selected generally have some expertise--either methodological or substantive--that makes them particularly appropriate for the evaluation.

Research-Driven Demonstrations

Information to improve decisions in the health field sometimes requires carefully designed field trials of new approaches--an innovative new service delivery system, for example, or a new way of financing health care, or an intervention to change health-related behavior. In these situations, the Foundation sometimes funds both a demonstration program and a careful evaluation of it simultaneously. We call these "research-driven demonstrations" because, in the design phase, the research requirements take precedence over other factors. Often, the grantees both conduct the evaluation and operate the demonstrations in a manner that is analogous to clinical trials. Some recent research-driven demonstration programs are summarized in Table 7.1.

These research-driven demonstrations stand in contrast to other national demonstrations where grantees are given wide latitude about how to implement a program, and where evaluation concerns do not drive the design of the programs. In national demonstrations, evaluations use the best possible design given the nature of the program, or they become case studies of single-site initiatives, each directed at a common social problem.

Grantees selected to manage research-driven demonstrations need to be both substantive and research experts in the area of concern. Generally, their participation is solicited by Foundation staff, and the grantees play significant roles in designing the programs.

Research Support Through Training Programs

Perhaps the most flexible research support takes the form of training programs for health researchers. Generally, the Foundation funds training programs to correct perceived deficiencies in the United States health care workforce. In the case of research training programs, however, the training is designed to enable researchers to define and carry out research projects that will advance their careers and create new generations of senior faculty who will address research questions important to the Foundation.

The four active research training programs are the following:²³

• The Clinical Scholars Program, which is the Foundation's oldest and largest training program. It involves seven academic medical centers and had trained 802 young physicians as of July 1999. The physicians enter the program at the end of their clinical training and then learn important nonclinical analytic disciplines such as epidemiology, biostatistics, health economics, bioethics, and research methodology. Most clinical scholars select academic careers, but others assume leadership positions in public health organizations or other governmental and private agencies. ²⁴
• The Minority Medical Faculty Development Program, which provides four-year career development support for young medical school faculty members from underrepresented minorities--African Americans, Hispanic Americans, and Native Americans. The intent of the program is to increase the number of minority faculty members who reach senior ranks in academic medicine and can serve as role models and mentors for prospective and currently enrolled minority medical students. Each year, twelve new faculty scholars are selected for these awards. Past and current scholars are distributed across fifty-eight academic medical centers.²⁵
• The Generalist Physician Faculty Scholars Program, which provides four-year career development support for fifteen young generalist physician faculty each year. Its purpose is to provide academic leadership in the three principal generalist disciplines--family medicine, general internal medicine, and general pediatrics. Past and current scholars can be found in fifty-four academic medical centers.²⁶
• The Scholars in Health Policy Research Program, which supports young faculty candidates who have completed doctoral training in economics, political science, or sociology. Twelve new scholars are selected each year to participate in a two-year training program, applying their disciplines to health-related policy issues.²⁷

ASSESSING THE EFFECTIVENESS OF RESEARCH INVESTMENTS

An enduring question for the Foundation is whether its sizable investment in research, data collection, policy analysis, and evaluation actually leads to improvement in health and health care. And, if so, are the payoffs larger than would have occurred from other types of grant making or from funding other types of research?

It is difficult to argue with the fact that information does matter, especially in the closing years of the twentieth century. However, it is much more difficult to know whether marginal investments in information really affect the decisions society makes about health care. Consider the current body of research on how social and economic factors--especially income variability--influence the health of a population. The first of the studies to show that large income disparities within a country are related to the population's health independent of overall income level was greeted with curiosity, but surely was not seen as convincing evidence. It took a series of replications of the findings using different types of data and research methods to make the findings convincing.²⁸ And it is still not clear how these powerful findings will affect social organization or income distribution policies in the short or long run. Yet a better understanding of how income affects a population's health is surely a good social investment that will shape thinking in ways difficult to quantify.

Another example of the difficulty of knowing the effects of research emerges from debates in the aftermath of the failed Clinton health plan. These debates demonstrate the range of opinion about the role of research. Some argue that information turned out to be unimportant compared to ideological considerations and to the power of paid advertisements and interest group lobbying. Others argue that the country lacked enough objective analysis to understand the problem and to measure the impact of the options that were being considered. The lack of hard data, this argument goes, allowed for the heavy influence of other factors.

My sense is that more information would have improved the quality of the debate concerning health reform. A priority of a foundation like Robert Wood Johnson should be the development of objective analyses of a range of possible insurance reform options that might be considered in the political process.

Assessments of the value of research investments also need to consider that research often pays off in ways that could not have been anticipated at the time of funding. A good example of this is the data from the experiment the Foundation funded on services at the end of life (the Study to Understand Prognoses and Preferences for Outcomes and Risks of Treatment, or SUPPORT). As part of this experiment, detailed information about specific problems people face at the end of life and the range of medical interventions they receive were documented in data files. Although the information was used to answer the specific questions of the research, more recently the data have been used to assess the efficacy of a range of interventions. For example, the research raised questions about the efficacy of the Swan-Ganz catheter, a common intervention to monitor cardiovascular function in critically ill patients.²⁹ The use of the data for a purpose such as this had not been anticipated before the study began.

Research also has an impact in indirect ways by educating researchers and enabling them to play key roles in efforts to change public policy.

In the attempt to reform health policy in the 1990s, for example, many of those who helped shape the way issues were framed were researchers whose careers had been supported by Robert Wood Johnson and other foundations. Although it is difficult to track the relationship between a specific research project and the ability of scholars to play influential roles in the process of social change, the relationship exists. And it is important.

Potential Criticisms of the Foundation's Approach

The specific priorities and approaches to funding research at the Foundation are open to criticism on a number of fronts. For one, the research funded may be too diffuse to have a big impact. In many cases, the Foundation funds research without having a clear vision of what it hopes to accomplish with the information that has been gathered. A specific research topic often is of interest because it addresses a problem related to the Foundation's goals. However, if research is to have sustained impact, there needs to be a vision of how findings can lead to action to solve the problem. Usually it takes a series of research efforts addressing a problem from different perspectives to pave the way for efforts to bring about change or innovation.

The criticism of excessive diffusion in the Foundation's research agenda derives from another aspect of Foundation research funding that is valued: allowing investigators to define topics they think are important. More controlled research agendas would lead to more focus, but might stifle individual creativity.

In recent years, the Foundation has consciously balanced its research portfolio to include some focused research agendas. Fortunately, because of recent growth in Foundation assets, research with focused agendas has increased without reductions in investigator-initiated funding. Examples of focused research agendas include the Health Tracking Initiative, discussed earlier, and Bridging the Gap, an ambitious substance abuse research project that is collecting and analyzing information about substance abuse and state-level initiatives to influence this problem.

A second criticism of the Foundation's research is that it does not fund a wide enough range of viewpoints. Given our emphasis on practicality and findings that can be applied in the short run, our approaches to funding research do not often lead to support for researchers with extreme or unconventional points of view. For example, the Foundation sometimes is accused of funding research that is more liberal than conservative. We also hear criticism that we fund little research that is radical in viewpoint--either on the left or the right--as opposed to incrementalist. This criticism is accurate and reflects the overall goals of the Foundation, which are neither radically to the left nor to the right. Our interest in making noticeable improvements to defined social problems often does favor thinkers who are tinkerers in orientation. Although a few funding vehicles, such as Investigators in Health Policy and Scholars in Health Policy Research, allow for research on almost any topic, the priorities and goals of the Foundation are not likely to attract researchers with radical agendas.

One other criticism is that the Foundation's grant-making cycles are not organized to use research findings to help foster social change. It is true that staff members do not necessarily view research as a first "learning" phase of social change, to be followed by "action" phases. The grant-making process is usually more organic. Research and action grants get funded at the same time in part because of an impatience to begin the process of change. Often, by the time research findings are emerging, the Foundation's priorities have shifted and the research never gets used--at least by the Foundation. This is not always the case, however. The Foundation is awarding grants to improve care at the end of life following its large, well-planned research investment.³⁰

In order to more consistently follow a two-phase approach to grant making (learning followed by action), the Foundation would have to stick with priorities for longer periods of time and be more patient in getting to the action phase of social change. Given the inherent tendency of The Robert Wood Johnson Foundation to respond to emerging, changing problems, a patient, two-phase approach to social change is not likely to emerge. It is more realistic to base decisions about programs on the best available evidence, wherever it comes from, and to consider research as an investment in learning that can be used in the future by the Foundation and by others.

Strengths of the Foundation's Grant Making

On the positive side, the Foundation's approach and commitment to research has allowed it to accomplish a great deal. For example, the scale of investments has allowed the Foundation to help establish entire new fields of analysis--most notably, public policy research related to tobacco control.

Large-scale research investments also have led to the development of important new data sets and a better understanding of health and health care: for example, market forces in health care (Health Tracking), state public policy initiatives to combat substance abuse (Bridging the Gap), and alternative approaches to caring for people at the end of life (SUPPORT). Although many early research grants at the Foundation were aimed at evaluating specific areas of concern, the data collected during the evaluations often found their way into professional journals, and made contributions beyond narrow assessments of whether Foundation initiatives achieved their intended goals.

The Foundation's willingness to nurture young researchers also represents a long-run investment in society's ability to develop new ideas and new information. Finally, the ability of the Foundation to be flexible and relatively quick in making grants often makes possible applied research that can chart fast-moving change.

Perhaps we will never know whether our research investments make tangible contributions to solving the social problems the Foundation cares about. We may have to trust the more commonsense observations that societies, businesses, and institutions that invest in research and information tend to do better than those that lag in these investments. We can draw upon the logic of our first president, David Rogers, who in the Foundation's 1980 Annual Report noted that "increases in the effectiveness and efficiency in various sectors of American enterprise have closely followed significant investments in research and development in a climate receptive to innovation."³¹

Notes

1. The range of approaches include research, which includes services research that identifies the relationship between outcomes and the way care is organized, policy research that studies how social policies and the policy process affect the health system, behavioral research that explores why people make health or health care choices, and survey research that measures attitudes, behavior, and outcomes experienced by people or organizations; research-based demonstration programs that test researchers' hypotheses through real-world practice; policy analysis that examines options for addressing health care; evaluations that look at how service programs or policy initiatives affect outcomes; and training programs that support the development of future researchers. (return to article)
2. Many Foundation grants involve both research and nonresearch activities. The 1,154 grants involving research, policy analysis, or evaluation include all those where research, evaluation, or policy analysis played some role. The $366 million figure only counts the subset of dollars devoted to research, evaluation, or policy analysis for each of the 1,154 grants. (return to article)
3. L. H. Aiken, R. J. Blendon, D. E. Rogers, and H. E. Freeman. "Evaluating a Private Foundation's Health Program." Evaluation and Program Planning, 1980, 3, 119-129. (return to article)
4. J. J. Salley. "Providing Dental Care to the Handicapped." Journal of Dental Education, 1980, 44(3), 136-140. (return to article)
5. D. L. Olds. "Improving the Life-Course Development of Socially Disadvantaged Mothers: A Randomized Trial of Nurse Home Visitation." American Journal of Public Health, 1988, 78(11), 1,436-1,445. (return to article)
6. The Infant Health and Development Program. "Enhancing the Outcomes of Low-Birth-Weight, Premature Infants: A Multisite, Randomized Trial." Journal of the American Medical Association, 1990, 263(22), 3,035-3,042. See also A. A. Baumeister and V. R. Bacharach. "The Infant Health and Development Program: Results at 8 Years." Journal of the American Medical Association, 1997, 277(16), 1,278-1,279. (return to article)
7. S. P. Klein, H. M. Bohannon, R. M. Bell, J. A. Disney, C. B. Foch, and R. C. Graves. "The Cost and Effectiveness of School-Based Preventive Dental Care." American Journal of Public Health, 1985, 75, 382-391. (return to article)
8. S. Sofaer. "Qualitative Research Methods: What Are They and Why Use Them?" Forthcoming. (return to article)
9. P. B. Ginsburg, R. G. Hughes, and J. R. Knickman. "A Robert Wood Johnson Program to Monitor Health System Change." Health Affairs, 1995, 14(1), 287-289. See also P. Kemper, D. Blumenthal, J. Corrigan, and others. "The Design of the Community Tracking Study: A Longitudinal Study of Health System Change and Its Effects on People." Inquiry, 1996, 33, 195-206. (return to article)
10. A. Kondratas and others. "Assessing the New Federalism: An Introduction." Health Affairs, 1998, 17(3), 17-24. (return to article)
11. M. L. Berk and C. L. Schur. "A Review of the National Access to Care Surveys." In S. L. Isaacs and J. R. Knickman (eds.), To Improve Health and Health Care 1997: The Robert Wood Johnson Foundation Anthology. San Francisco: Jossey-Bass, 1997, pp. 53-77. See also M. L. Berk and C. L. Schur. "Access to Care: How Much Difference Does Medicaid Make?" Health Affairs, 1998, 17(3), 169-180, and M. L. Berk, C. L. Schur, and J. C. Cantor. "Ability to Obtain Health Care: Recent Estimates from The Robert Wood Johnson Foundation Access to Care Survey." Health Affairs, 1995, 14(3), 139-146. (return to article)
12. L. H. Aiken and others. "The Contribution of Specialists to the Delivery of Primary Care." New England Journal of Medicine, 1979, 300(24), 1,363-1,370. (return to article)
13. A. B. Cohen, J. C. Cantor, D. C. Barker, and R. G. Hughes. "Young Physicians and the Future of the Medical Profession." Health Affairs, 1990, 9(4), 138-148. See also J. C. Cantor, L. C. Barker, and R. G. Hughes. "Preparedness for Practice: Young Physicians' Views of Their Professional Education," Journal of the Amercian Medical Association, 1993, 270(9), 1,035-1,040. (return to article)
14. L. H. Aiken, R. J. Blendon, D. E. Rogers, and H.E. Freeman. "Evaluating a Private Foundation's Health Program." Evaluation and Program Planning, 1980, 3, 119-129. (return to article)
15. Information about the data files can be found online at http://www.icpsr. umich.edu (return to article)
16. B. A. Stevens and L. D. Brown. "Expertise Meets Politics: Efforts to Work with States." In S. L. Isaacs and J. R. Knickman (eds.), To Improve Health and Health Care 1997: The Robert Wood Johnson Foundation Anthology. San Francisco: Jossey-Bass, 1997, pp. 78-96. (return to article)
17. M. A. Gutman, D. G. Altman, and R. Rabin. "Tobacco Policy Research." In S. L. Isaacs and J. R. Knickman (eds.), To Improve Health and Health Care 1998-99: The Robert Wood Johnson Foundation Anthology. San Francisco: Jossey-Bass, 1998, pp. 19-42. (return to article)
18. http://www.ahsr.org/rwjf (As of September 2004, this link is no longer active)(return to article)
19. http://www.phs.bgsm.edu/ (return to article)
20. http://www.ac.org/httpdocs/hcfo.html (As of September 2004, this link is no longer active) (return to article)
21. L. Saxe. "Think Globally, Act Locally: Assessing the Impact of Community-based Substance Abuse Prevention." Evaluation and Program Planning, 1997, 20(3), 357-366. (return to article)
22. S. A. Schroeder. "Reflections on the Challenges of Philanthropy." Health Affairs, 1998, 17, 209-216. (return to article)
23. Many of these are described in S. A. Schroeder and J. R. Knickman. "Views from Funding Agencies: The Robert Wood Johnson Foundation." Medical Care, 1998, 36(5), 621-624. See also S. L. Isaacs and L.G. Sandy. "Grants to Shape the Health Care Workforce: The Robert Wood Johnson Foundation Experience." Health Affairs, 1996, 15, 279-295. (return to article)
24. (As of September 2004, this link is no longer active) http://www.ahsr.org/pubs/training/robert.htm(return to article)
25. http://www.rwjf.org/grant/minmed99.htm (return to article)
26. http://www.rwjf.org/library/genph99.htm (return to article)
27. http://management.bu.edu/research/centers/Johnson.html (As of September 2004, this link is no longer active) (return to article)
28. J. S. Feinstein. "The Relationship Between Socioeconomic Status and Health: A Review of the Literature." Milbank Quarterly, 1993, 71(2), 279-322. (return to article)
29. A. F. Connors Jr. and others. "The Effectiveness of Right Heart Catheterization in the Initial Care of Critically Ill Patients." Journal of the American Medical Association, 1996, 276(11), 889-897. (return to article)
30. J. Lynn. "Unexpected Returns: Insights from Support." In S. L. Isaacs and J. R. Knickman (eds.), To Improve Health and Health Care 1997: The Robert Wood Johnson Foundation Anthology. San Francisco: Jossey-Bass, 1997, pp. 161-186. (return to article)
31. D. E. Rogers. "The President's Statement: The 1980s--Adapting the Foundation's Program to Changing Problems." The Robert Wood Johnson Foundation Annual Report. Princeton, N.J.: The Robert Wood Johnson Foundation, 1980, pp. 8-21. (return to article)