LB 2842.4 .C2 S74 IZAU(IIE Do Financial Incentives Draw Promising Teachers to, Low-Performing Schools? Assessing the Impact of the California Governor’s Teaching Fellowship Jennifer L. Steele Richard J. Murnane Jenn B. Willett1 Policy Brief 10—3 Jennifer L. Steele is an Associate Policy Researcher at the RAND Corporation in Washington, DC. Her research addresses public sector labor market incentives, school reform, and data-driven decision , making in educational organizations. She is co-editor‘ of Data Wise in Action and of the Harvard Educational Review’s Special Issue on Adolescent Literacy. Richard Murnane is Thompson Professor of Education and Society at the Harvard University Graduate School of Education. He is an economist who focuses his research on the relationships between education and the economy, teacher labor markets, and strategies for making schools more effective. Murnane is currently co—director (with Greg Duncan) of a large research project examining the sources of social laeqflallty and educational disadvantagein American society. John B. Willett is Charles WilliamEliet Professor at the Harvard University Graduate School of Education. He is a statistician Whose research focuses on quantitative methods for measuring change over time and for analyzing the occurrence, , timing and duration of events. His most recent book (co-authored by Richard Murnane) is entitled Methods Matter: Improving Causal Inference in Educational and Social Science Research (Oxford U. Press, forthcoming August 2010). 2010 V""W IGSL UCB INSTITUTE OF GOVERNMENTAL STUDIFRI IRRARY UNIVERSITY OF CALIFORNIA POLICY BRIEF MAY 2010 The Governor’s Teaching Fellowship: A Natural Experiment in California n the fall of 2000, Califor- nia introduced the Governor’s Teaching Fellowship, a $20,000 conditional scholarship designed to attract academically talented, newly licensed teachers to schools in the bot— tom half of the state’s Academic Perfor- mance Index4 (API) and retain them in low-performing schools for at least four years (Steele, Murnane, 8: Wil- lett, forthcoming). Only prospective teachers who were enrolled full-time in accredited, post—baccalaureate teacher licensure programs were eligible to apply. The fellowship was merit-based (California State University Office of the Chancellor, 2002, p. 5) and competitive. Applicants submitted undergraduate and graduate school transcripts, letters of recommendation, a resume, and a personal essay, and were interviewed by telephone. The state awarded 249 fellowships in the first year (2000—01) and 947 fellowships in the second year (2001—02). The pro- gram was discontinued the following year due to high overhead costs and statewide budget constraints (Califor— nia Legislative Analyst’s Office, 2002). The full amount of the fellowships was paid at the time that the awards were issued, and recipients who did not ful- fill their four-year commitments were Executive Summary During a two—year period from 2000-2002, California awarded a $20,000 Governor’s Teaching Fellowship (GTF) to 1,169 people enrolled in traditional, post-bac- calaureate teacher licensure programs who agreed to teach in low-performing public schools for four years after earning their licenses. Schools designated as low-performing were those that ranked in the bottom half of the state’s Academic Performance Index (API). GTF regulations speci- fied that recipients who did not fulfill their four—year teaching com- mitments would repay $5,000 for each year of service not completed. The GTF was a policy response to longstanding evidence from within and outside California that low- income students and students of color are disproportionately taught by’teachers with weak academic backgrounds and limited prepara— tion.2 The GTF’s objective was to promote a more even distribu- tion of teacher qualifications by helping low-performing schools recruit, and keep, promising new teachers with strong academic backgrounds. Fellowships were awarded by a committee based on a range of criteria, including applicants’transcripts, essays, resumes, recommendation letters, and telephone interviews. POLICY BRIEF PACE Executive Summary continued After its second year, the GTF was discontinued for budgetary reasons (California Legislative Analyst’s Office, 2002). Other than a descriptive report issued after the fellowship’s first year (California State University Office of the Chancellor, 2002), the policy’s impact was not formally analyzed or reported. Assessing the GTF’s impact is important for both state and national reasons. From a state perspective, understand— ing whether the GTF achieved its objectives can inform policy decisions to ensure that California students with the greatest instructional needs have access to skilled teachers. From a national perspective, estimates of the GTF’s impact can contribute to a limited body of evidence about the effective- ness of this type of incentive. Targeted ’ recruitment and retention incentives, including those tied to financial aid, are popular tools that states and the federal government use to induce professionals such as doctors, lawyers, and teachers to work with under—served populations. However, only in rare cases have these incentives been subject to rigorous evaluation.3 As a result, little is known about their influence on employment decisions or the distribution of talent in public-service careers, including the teaching profession. Given the need for rigorous research on incentives like the GTF, we undertook a study to estimate the fellowship’s causal impact on recipients'decisions to teach in low-performing schools. The results are summarized in this policy brief. Using longitudinal employment records for a large subset of California teacher licensure candidates, we establish that, for every seven GTF recipients who began teaching in a low performing school, two would not have done so in the absence of the incentive. Among teachers who began working in low-performing schools, we find that the retention rate of GTF recipients was no different from that of non-recipients, and that 75 percent of teachers in both groups remained in low-performing schools for at least four years. Based on these estimates, we calculate that California spent $9,800 in fellowship dollars for every one—year teaching position staffed by a GTF recipient who would not have otherwise taught in a low-performing schoo|.This amount represents approximately 30 percent of a beginning teacher’s salary in California in the 2000—01 academic year. We see two related lessons from our evaluation. The first is that financial incentives can be an important policy tool in attracting skilled professionals to work with underserved populations. The second is the importance of explor- ing whether an alternative policy design might have been equally or more cost— effective. It is important to keep in mind that our evaluation could not consider two important questions. The first concerns how long beyond the initial four-year window explored in this study the GTF recipients continued to teach in low—performing schools. The second concerns the instructional effectiveness of GTF recipients relative to their peers. We conclude with recommendations about the kind of data that would allow California to answer important policy questions of this nature. required to repay $5,000 per year of service not completed. By effectively granting teachers $5,000 for each year of qualifying service, the fellowship offered a 15.1 percent annual premium over the 2000-01 average starting sal- ary of $33,121 for California teachers (American Federation of Teachers, 2002). The GTF was a policy response to longstanding evidence that low- income students and students of color are disproportionately taught by teachers with weak academic back— grounds and limited preparation. For example, Betts and colleagues (2000) found that in the late 19903, California schools serving students in the bottom socioeconomic quintile were staffed by teachers with markedly lower educa- tion levels, experience levels, and cer- tification rates than their counterparts in more affluent schools. Though the California study did not examine the distribution of academic talent per se, a study in New York state (Lankford et a1, 2002) found that in schools where more than 20 percent of fourth graders had failed the language arts test, the percentage of teachers who had failed their own licensure examinations on the first try was 36 percent, versus 9 percent of teachers in the state’s high-performing schools. Given that the unequal distribution of teacher qualifications has been a longstanding problem in the US. (Becker, 1952), the GTF was intended to help equal- ize this distribution by helping low- performing schools recruit and retain new teachers with strong academic backgrounds. DO FlNANClAL INCENTIVES DRAW PROMlSlNG TEACHERS TO LOW-PERFORMING SCHOOLS? Lg234Q$e CZ $7,341— 2:010 Taking advantage of the sudden arrival and subsequent disappearance of the GTF, we conducted a study to estimate the awards causal impact on the early-career decisions of its recipi— ents. In particular, we asked: to what extent did receiving a GTF increase the probability that an academically talented novice teacher would take a job in a low-performing school? We also asked a second, descriptive question: Conditional on beginning to teach in a low—performing school, how long did GTF recipients continue teaching in low—performingschools relative to demo- graphically similar, academically talented teachers who did not receive a GTF? An Unconventional but Useful Data Source The ideal approach to our research question would have been to estimate the effect of the GTF program by comparing the job placement pat- terns of teacher licensure candidates enrolled in the academic years before, during, and after GTF availability. However, California does not main- tain longitudinal data about teachers’ school-level employment. While the California State Teachers’ Retirement System (CalSTRS) does record teach- ers’ employment histories, it tracks their employment only at the district level and thus does not indicate which teachers worked in low-performing schools in a given year. , As an alternative, we used a longitu- dinal dataset that tracks the school- level employment of a large subset of teacher licensure candidates for up to four years after they earn their licenses. The candidates in the dataset were those who received loan-for- giveness contracts from the Assump- tion Program of Loans for Education (APLE) program, a longstanding and currently large loan-forgiveness program for teacher licensure can- didates in California. Our analytic sample included APLE contract recip- ients enrolled in licensure programs between the 1998-99 and 2002-03 academic years—that is, two years prior to the GTF, two years during the GTF, and one year after the GTF ended. The baseline APLE contract forgives between $11,000 and $19,000 of student loans5 in exchange for four years of service in shortage subject areas or hard-to-staff schools (includ- ing not only low-performing, but also low-income, rural, and poorly staffed schools). However, APLE awards are paid after each year of teaching service is completed, so the contracts are non-binding for recipients. Since the 1998-99 academic year, when the APLE program expanded the number of loan forgiveness contracts offered annually from 400 to 4,500, these contracts have been widely available to California teacher-licensure can- didates, and APLE recipients have constituted a very large subset of teachers pursuing first-time teaching licenses in the state.6 Like other licen— sure candidates, those who held APLE contracts during the years of GTF availability were eligible to apply for and receive the full GTF award, so the GTF would have served as a $20,000 add-on to the APLE loan-forgiveness contract for these individuals. ICsSL, Ucfi The APLE program was important for this study because it provided the only dataset tracking school-level employment for several cohorts of novice California teachers in the four years after they became licensed. In this sense, it approximated the ideal dataset, but because it provided infor- mation on only a subset of California teachers in the relevant years, it also had some limitations. A crucial limi- tation is that we could only examine the employment decisions of licensure candidates who received APLE con- tracts. For instance, because APLE is a loan-forgiveness program, the dataset excluded teacher licensure candidates who had no outstanding undergradu- ate or graduate—level student loans and thus contained 725, or 61 percent, of all GTF recipients.7 A second limitation is that, because all APLE recipients held loan—forgiveness contracts, we estimated the GTF’s impact as an add- on incentive rather than estimating its first-dollar effect. The existence of loan forgiveness provisions in federal Per- kins and Stafford Loans and in numer- ous state policies suggests that it is not uncommon for teacher recruitment and retention incentives to supple- ment one another in this way. Still, it is important to clarify that because the APLE population is a subset of licensure candidates in California, our results are only generalizable to licen- sure candidates who have student loans and who have an existing financial incentive for teaching in disadvantaged schools. Insofar as such teachers are especially sensitive to financial incen- tives (a speculation that we cannot test ASSESSING THE IMPACT OF THE CALIFORNIA GOVERNOR’S TEACHING FELLOWSHIP POLICY BRIEF PACE empirically) our estimates of the GTF effect in this population may represent an upper bound on its effect among all novice teachers. The final dataset tracked individu- als’ school placements longitudinally through the completion of their fourth teaching year, the full payoff of their student loans, or the 2004-05 academic year—whichever occurred first. After removing cases with missing infor- mation on key variables, our analytic sample included 27,106 licensure can- didates, of whom 718 (or 2.65 percent) were GTF recipients.8 To estimate the causal effect of the GTF on the employment choices of its recipi- ents, we treat the sudden introduction and termination of the program as a natural experiment. A general descrip- tion of this technique is presented in “An Analysis to Eliminate Selection Bias” in the appendix. A more detailed techni- cal explanation can be found in Steele, Murnane, and Willett (forthcoming). Who Received the GTF? Figure 1 illustrates the estimated effect of enrollment in a GTF year on the probability of receiving a GTF as a function of CBEST licensure test scores and licensure institution type. The most striking feature of the figure is the difference between the probability of receiving a GTF for licensure candidates who were enrolled in a University of FIGURE 1. Fitted effects of licensure test scores and licensure institution type on the probability that an APLE recipient enrolled during GTF availability received a GTF (n=27,106) .3 l 4__ UC u_ .025 , l- —— U to 2‘ :E .2 a: U a r: ‘5 .15 5‘ .3 2 g .l n. 'U a; L: CSU LL '05 >l\ independent l/ l 0 110 130 150 170 190 210 . Combined CBEST Score >K== median test score in each licensure institution type Note: Fitted lines apply to females in the 2000-01 APLE cohort of teacher licensure candidates, who earned elementary school teaching licenses without bilingual authorization within two years, and who received APLE contracts at the sample mean age of3i .5. California (UC) program during the GTF years and those who were enrolled in a California State University (CSU) or independent program. The displayed trend lines apply to 31-year old female elementary school teachers who received APLE contracts in 2000-01 and were not authorized to teach bilingual educa- tion. At any given licensure test score, the estimated probability of receiving a GTF for a licensure candidate at a UC institution is 20.4 percentage points higher than the probability for a CSU licensure candidate and 22.2 percent— age points higher than for a licensure candidate at an independent college or university. The difference in estimated probabilities between UC licensure candidates and the other two groups is statistically significant (p<.001), while the fitted probabilities for candidates at CSU and independent institutions are not statistically distinguishable from each other. When we asked GTF com- mittee members whether their prefer— ence for UC students was deliberate, they indicated that it was not. However, they speculated that these students may have demonstrated especially strong academic credentials. Though licensure test scores were not available to the GTF selection commit- tee, we also find a positive relationship between composite licensure test scores and the probability of receiving a GTF.9 For example, holding all else constant, a licensure candidate with a CBEST score of 193 (the sample 95th percentile) has a 1 percentage—point higher probability of receiving a GTF than her counterpart scoring at the sample median of 148 (p<.001). DO FINANCIAL INCENTIVES DRAW PROMISING TEACHERS TO LOW-PERFORMING SCHOOLS? How Did the GTF Affect New Teachers’ Entry Rates into Low- Performing Schools? Using the analytic strategy described above, we estimate that the GTF award increased by 28 percentage points the probability that its recipi- ents taught in low-performing school within two years of receiving APLE contracts. Thus, for every seven GTF recipients who began working in a low performing school, two would not have done so in the absence of the incentive. Figure 2 illustrates this estimated effect. The upper trend line, plotted between licensure years 2001 and 2003, repre— sents the fitted entry probabilities of prototypical GTF recipients, and the dotted lines represent the 95 percent confidence interval around the esti- mate, ranging from an effect of 14.5 to 41.6 percentage points. The longer solid line (also bracketed by its 95 percent confidence interval) represents the fitted counterfactual—the expected probability that the prototypical GTF recipients would have taken teaching positions in low-performing schools had they been enrolled in licensure programs in years when the GTF was not offered.10 In addition, juxtaposing these results with results from a simple comparison of GTF recipients and non-recipients suggests that the GTF successfully chose recipients who, in the absence of the fellowship, would have been less likely than other APLE recipients to begin working in low-performing schools. FIGURE 2. Impact of the GTF award (with 95-percent confidence intervals) on recipients’ probability of teaching in a low-performing school within two years after receiving an APLE contract (n=27,106) 3 1 o .C u - o - - - - - W - - - - — - _ 2‘ 0.9 — E ~3 nu." 0.8 ' . \ . 3 Wlth GTF 3 M 2‘ 0.7 < Estimated treatment '5 on-treated effect: __________ _ E 28 percentage ' ' ~05 0.6 . pomts (p<.001) .é‘ _ _ 5 r ““““““““““““ 3 ------- ‘ — - _ - _ _ _ -------------- E 0.5 - --------- " . w ithout GTF B .‘E “L 0.4 . . . . . , l 998 1999 2000 2001 2002 2003 2004 2005 Year of Licensure Note: Fitted probabilities apply to a female elementary teacher from a CSU licensure institution with a sample mean age of31.5 and a CBEST score at the sample mean of 152. How Long Did GTF Recipients Stay In Low-Performing Schools Relative to Non- Recipients? For the group of novice teachers who did begin working in low-performing schools, Panel A of Figure 3 shows the hazard probability of leaving the set of low-performing schools by the end of each school year, conditional on their not having left in a previous year. We do not show the hazard func- tions separately for GTF recipients and non-recipients because there appears to be no difference between them. On average, GTF recipients stayed in low- performing schools as long as, but no longer than, other APLE recipients in the dataset. Panel A also illustrates that the risk of exit was lowest in the first teach- ing year, at 6.4%, and highest in the third teaching year, at 11.6%. This positive trajectory may be due to the fact that the APLE contracts served as a baseline incentive for teachers to persist in their jobs. Panel B of Figure 3 shows the estimated percentage of teachers persisting in low—performing schools beyond each teaching year. It indicates that an estimated 74.9 percent of entrants to low-performing schools stayed in the set of low-performing schools beyond year three and persisted into year four. Because APLE tracks its recipients only through their fourth teaching year, we are unable to say how many left the set ASSESSING THE IMPACT OF THE CALIFORNIA GOVERNOR’S TEACHING FELLOWSHIP POLICY BRIEF PACE FIGURE 3A. Fitted Hazard Function Describing Exit from the Set of Low- Performing Schools by the End of Each of the First Three Teaching Years (n=9,495) 0.2 0.15 ' 0.116 0.1 - 0.092 0.067 0.05 - Fitted Hazard Probability of Leaving the Set of Low-Performing Schools 0 - . 0 1 2 3 Year of Teaching in a Low-Performing School FIGURE 33. Fitted Baseline Probabilities of Continuing to Teach in a Low- Performing School Beyond Each of the First Three Years (n=9,495) 1 0.933 10 0.847 0.749 Fltted Probability of Remaining m a Low-Performing School Beyond the Current Year K: b0 0 1 2 3 4 Year of Teaching in a Low-Performing School of low-performing schools after their agreements ended at the conclusion of year four. Because this part of our analysis is strictly descriptive, it is possible that the GTF recipients who began teach- ing in low-performing schools differed in unobserved ways from their non- recipient counterparts who did so. As a result, we cannot conclude from this retention analysis that the GTF influ- enced retention rates either positively or negatively. Magnitude and Cost of the GTF’s Impact A critical policy question is whether the GTF program’s estimated impact on teachers’ career decisions is large enough to justify the cost to the tax- payers of California. We have esti- mated that for every seven teachers who received the GTF, two decided to teach in a low performing school and would not have done so otherwise. Due to the nature of our sample, we can generalize only to APLE recipients, 725 whom received GTFs. We estimate that roughly 28 percent of these indi— viduals, or 203 novice teachers, entered low-performing schools and would not have done so in the absence of the GTF incentive. And based on the descriptive retention analysis, we further estimate that the GTF staffed 716 one—year, full- time teaching slots in low-performing schools with academically talented teachers who would have not otherwise taught in such schools. But how much did this benefit cost the state of California? Excluding a DO FINANCIAL INCENTIVES DRAW PROMISING TEACHERS TO LOW-PERFORMING SCHOOLS? administrative and overhead costs and restricting our analysis just to the value of the awards themselves, California spent $14.5 million on GTF award payments of $20,000 each to the 725 recipients in our dataset. Of that, we estimate that roughly $7.5 million was repaid by recipients who did not complete their teaching requirements. Ignoring collection costs and foregone interest, this leaves $7 million in net award payments. For that money, the state recruited 203 teachers to low—performing schools, who, within the next four years, staffed 716 one— year teaching positions in those schools. This suggests that California paid roughly $34,500 in recruitment costs for each person whose entry decision it influenced, and about $9,800 in recruitment costs for every one-year teaching position the GTF staffed with an academically talented teacher. Assuming that each of those teachers educated an average of 40 students per year, the per-pupil cost would have stood at about $245—a small fraction of California’s average per-pupil expenditure of $7,055 in 2001-02 (Education Data Partnership, 2008).11 What Additional Information Do Policymakers Need? The estimate of benefits relative to costs depends on two pieces of information that cannot currently be estimated using California data. First, it depends on how long the GTF recipients remained in low- performing schools beyond the four- year window of this study. The longer they stay, the lower the recruitment cost per teaching year, and the more high-need students they will be able to teach. To answer this question, a database is needed that tracks the school-level employment histories of all teachers in California throughout their careers in the state. Not only would such a dataset have enabled a longer-term benefit/ cost analysis, but it would also have allowed us to con- duct an analysis generalizable beyond the large pool of APLE recipients. This type of dataset might be estab- lished with relative ease if the State Teachers Retirement System, which currently tracks teachers’ employ- ment histories at the district level, were to record and make available to researchers teachers’ school-level assignments. Second, the true benefit/cost ratio depends on the instructional effec- tiveness of GTF recipients relative to their peers. In light of the aforemen- tioned teacher distribution research in California and elsewhere, it is entirely plausible that the GTF recipients took teaching slots that would otherwise have been filled by weaker teachers. But to learn whether this was the case would require a dataset that linked longitudinal information about stu— dents’ performance to their teachers each year. While the cost of estab— lishing such a longitudinal student data system would be substantial, it would make possible research that would shed light on the efficacy of many California educational policy initiatives. Considerations for Future Policy Academic talent has long been shown to predict teachers’ ability to increase student achievement. Thus, from a policy perspective, the goal of distrib- uting academically talented teachers more equitably remains worthwhile. On the other hand, academic talent explains only a small percentage of the variation in teachers’ effective— ness in raising student achievement (Boyd, Grossman, Lankford, Loeb, & Wyckoif, 2005; Clotfelter et al., 2007; Goldhaber, 2006; Kane & Staiger, 2005). Consequently, it is worthwhile to explore ways of targeting incentives more precisely. One option would involve a recruitment incentive aimed at attracting academically talented teachers to high-need schools, coupled with a structured retention incentive based on multiple measures of teach- ers’ effectiveness in enhancing their students’ skills and knowledge. Recruitment and retention incentives might also be expanded beyond mon- etary support. Selection committee members suggested to us that if the GTF had included ongoing networking or professional development opportu- nities, recipients’ sense of commitment to the award and its purpose might have been strengthened. Evidence from Teach for America (TFA) sup- ports this proposition. TFA provides not only a fast-track entry route into teaching, but also external recognition, a strong sense of group identity, and ongoing opportunities for networking and professional support. And TFA’s ASSESSING THE IMPACT OF THE CALIFORNIA GOVERNOR'S TEACHING FELLOWSHIP POLICY BRIEF PACE recruitment efforts appear relatively efficient: the organization spent $6,379 per teacher they recruited into their 2007-08 cohort (Teach for America, 2008). However, the TFA model is not designed to emphasize teacher reten- tion. Instead, the organization portrays teaching as a starter career and seeks to incubate future leaders who are sensitive to education issues into busi— ness, law, medicine, and public policy (Teach for America, 2006). Although 61 percent of TFA recruits remain in the teaching profession beyond the end of their two-year commitments (Donaldson, 2008) we estimated that 85 percent of GTF recipients remained in low-performing schools beyond the second year, and that 75 percent ful- filled their four-year commitments in those schools. In fact, designers of the GTF have told us that they required four years of service and restricted eligibility to traditionally licensed teachers partly to distinguish the program from TFA and reinforce the notion of teaching as longer-term career commitment. However, it is possible that access to strong networking and professional support opportunities, like those offered by TFA, would have increased not only the entry and retention rates of GTF recipients, but also their capacity to work effectively with students and colleagues and influence the climates of their schools. Whether such non- pecuniary incentives might have cost- effectively supplanted a portion of the $20,000 awards remains an interesting question for future policy experiments to address. The essential policy question, of course, is whether the GTF was a wise invest- ment for California. In an effort to increase the number of academically skilled teachers in low—performing schools, spending just under $10,000 for every one-year position staffed by a promising teacher who would have oth- erwise gone elsewhere does not seem an extraordinarily high price to pay. Moreover, the longer the GTF recipi- ents remained in underperforming schools beyond their four-year fellow- ship terms, the lower the recruitment cost per teacher-year would ultimately have been. However, viewing the policy as a positive use of resources assumes that, as the literature suggests, these teachers were more effective than their counterparts at raising student achieve- ment-an assumption that we cannot test with available data. As California moves to collect data tracking teachers’ career histories and linking teachers to students, it will become possible to evaluate policies like the GTF in terms of both their long-term costs and their student achievement benefits. Better data will thus allow the state to learn more about the benefits and costs of policy initiatives such as the GTF that are aimed at equalizing educational opportunity in California. Appendix An Analysis to Eliminate Selection Bias Because GTF recipients had to self-select into the applicant pool and then be selected by a committee to receive the award, they may differ in unobserved ways from other licensure candidates. To the extent that they do differ, any simple comparison of their employment outcomes to those of other licensure candidates may incorrectly estimate the true impact of receiving the fellowship. In an effort to eliminate this bias and estimate the fellowship’s causal effect on recipients’ choice to teach in low-performing schools, we take advantage of the sudden introduction and termination of the GTF program, which, according to our searches of newspaper archives and interviews with selection committee members, would have been difficult for prospective licensure candidates to anticipate. We treat the GTF’s abrupt introduction and termination as policy shocks that randomly assigned licensure candidates to GTF eligibility, based on the years they were enrolled in a licensure program. This incidental random assignment creates a naturally occurring experiment, and we can estimate the probability of receiving the GTF as a function of this incidental random assignment. Given that the GTF was an academically competitive award, we allow the effect of enrollment in a GTF- eligible year to vary by the two academic background characteristics we have on record: licensure program type (i.e., University of California, California State University, or a program offered by a private college or university) and performance on the California Basic Educational Skills Test (CBEST) a mandatory licensure examination that measures prospective teachers’ mathematics, reading, and writing skills. Thus, the estimated probabilities of receiving a GTF reflect not only the effect of enrollment in a GTF year, but also the differential effect of academic background on the ineligible (where it doesn’t change the prob- ability from zero) and the eligible (where it affects the probability substantially). Consequently, when we estimate the effect of GTF-receipt probability on the probability of teaching in a low-performing school, we are capturing not only the incidentally assigned effect of enrollment year, but also the difierential effect of enrollment year on individuals with different academic backgrounds. Assuming that labor market trends over time affected higher and lower ability licensure candidates similarly, estimating the prob- ability of teaching in a low-performing school as a function of the probability of receiving a GTF yields an unbiased estimate of the causal impact of the GTF on the first job placement of its recipients.” a DO FINANCIAL INCENTIVES DRAW PROMISING TEACHERS TO LOW-PERFORMING SCHOOLS? References Aaronson, D., Barrow, L., 8r Sander, W. (2007). Teachers and Student Achievement in the Chicago Public High Schools. Journal of Labor Economics, 25, 95-135. American Federation of Teachers. (2002). Table III -1 : Actual Average Beginning Teacher Salaries, 2000- 01 and 2001—02. Retrieved December 14, 2006, from http://www.aft.org/salary/ZOOZ/download/ Table_III-l.pdf Becker, H. S. (1952). The Career of the Chicago Public School Teacher. American Journal of Sociology, 57(5), 470-477. Betts, 1. R., Rueben, K. S., & Danenberg, A. (2000). Equal Resources, Equal Outcomes: The Distribution of School Resources and Student Achievement in California. San Francisco: Public Policy Institute of California. Boyd, D., Grossman, P., Lankford, H., Loeb, S., & Wyckoff, 1. (2005). How Changes in Entry Require- ments Alter the Teacher Workforce and Affect Stu- dent Achievement (No. 11844). Cambridge, MA: National Bureau of Economic Research. Burke, 8., & Errett, M. (2000). Teacher Supply in Cali- fornia: A Report to the Legislature. Second Annual Report, 1998-99. Sacramento, CA: California Com- mission on Teacher Credentialing. Burke, 8., 8: Errett, M. (2001). Teacher Supply in Cali- fornia: A Report to the Legislature. Third Annual Report, 1999-2000. Sacramento, CA: California Commission on Teacher Credentialing. Burke, 8., & Errett, M. (2002). Teacher Supply in Cali- fornia: A Report to the Legislature. Fourth Annual Report, 2000-01. Sacramento, CA: California Com- mission on Teacher Credentialing. Burke, 8., & Errett, M. (2003). Teacher Supply in California: A Report to the Legislature. Fifth Annual Report, 2001 —02. Sacramento, CA: California Com- mission on Teacher Credentialing. California Education Code, Section 69612-696156. (2000). Retrieved Dec. 7, 2006, from http://www. leginfo.ca.gov/cgi-bin/waisgate?WAISdocID=513 042124+0+0+0&WAISaction=retrieve California Legislative Analyst’s Oflice. (2002). Analysis of the 2002-03 Budget Bill. Retrieved September 5, 2005, from http://www.lao.ca.gov/analysis_2002/ education/ed_29_6610_an102.htm California State University Oflice of the Chancellor. (2002). The 2000-2001 Annual Report of the Gov- ernor’s Teaching Fellowship Program. Long Beach, CA: Author. California Student Aid Commission. (2004). APLE 2002-03 Report to the Legislature. Rancho Cordova, CA: Author. California Student Aid Commission. (2005). 2005— 2006 Assumption Program of Loans for Education Coordinator’s Guide and Forms. Rancho Cordova, CA: Author. Clotfelter, C. T., Glennie, E., Ladd, H. F., 8: Vigdor, 1. L. (2008, Ian. 6). Would Higher Salaries Keep Teach- ers in High-poverty Schools? Evidence from a Policy Intervention in North Carolina. Paper presented at the Annual Meeting of the American Economic Association, New Orleans, LA. Clotfelter, C. T., Ladd, H. F., 81 Vigdor, J. L. (2006). Teacher-student Matching and the Assessment of Teacher Effectiveness. Journal of Human Resources, 41(4), 778-820. Clotfelter, C. T., Ladd, H. F., 8r Vigdor, 1. L. (2007). How and Why do Teacher Credentials Matter for StudentAchievement? (Working Paper No. 12828). Washington, DC: National Center for Analysis of Longitudinal Data in Education Research. Donaldson, M. L. (2008). Teach for America Teach— ers’ Careers: Whether, When, and Why they Leave Low-income Schools and the Teaching Profession. Cambridge, MA: Harvard University. Education Commission of the States. (2004). Recent State Policies/Activities: Teaching Qual- ity-Hard-to-Staff Schools. Retrieved Oct. 24, 2006, from http://www.ecs.org/ecs/ecscat.nsf/ WebTopicView?OpenView&count=-1&Restr ictToCategory=Teaching+Quality—Hard-to- Staff+Schools Education Data Partnership. (2008). Comparing Cali - fornia. Retrieved Oct. 3, 2009, from www.cd—data. kl2.ca.us/Articles/artic1e.asp?title=California%20 comparison Ferguson, R. F., & Ladd, H. F. (1996). How and why money matters: An analysis of Alabama schools. In H. F. Ladd (Ed), Holding Schools Accountable: Per- formance-based Education Reform (pp. 265-298). Washington, DC: The Brookings Institution. Field, E. (2009). Educational Debt Burden and Career Choice: Evidence from a Financial Aid Experiment at NYU Law School. American Economic Journal: Applied Economics, 1(1), 1-21. Goldhaber, D. (2006). Everyone’s Doing It, but What Does Teacher Testing Tell Us About Teacher Effectiveness? Journal of Human Resources, 42(4), 765-794. Hanushek, E. A. (1971). Teacher Characteristics and Gains in Student Achievement: Estimation Using Micro Data. American Economic Review, 61 (2), 280-288. Johnson, 1. (2005). State Financial Incentive Policies for Recruiting and Retaining Eflective New Teach- ers in Hard-to-stafl Schools. Retrieved October 20, 2006, from http://www.ecs.org/clearing- house/6l/61/6161.htm - Kane, T. 1., & Staiger, D. O. (2005). Using Imper— fect Information to Identify Eflective Teachers. Cambridge, MA: National Bureau of Economic Research. Kirshtein, R. 1., Berger, A. R., Benatar, E., & Rhodes, D. (2004). Workforce Contingent Financial Aid: How States Link Financial Aid to Employment. India— napolis, IN: Lumina Foundation for Education. Lankford, H., Loeb, S., & Wyckoff, 1. (2002). Teacher Sorting and the Plight of Urban Schools: A Descriptive Analysis. Educational Evaluation and Policy Analysis, 24(1), 37-62. Rivkin, S. G., Hanushek, E. A., 8: Kain, 1. F. (2005). Teachers, Schools, and Academic Achievement. Econometrica, 73(2), 417-458. Rockoff, 1. E. (2004). The Impact of Individual Teachers on Student Achievement: Evidence from Panel Data. American Economic Review, 94(2), 247-252. Steele, 1. L., Murnane, R. 1., & Willett, 1. B. (forthcom- ing). Do Financial Incentives Help Low-performing Schools Attract and Keep Academically Talented Teachers? Evidence from California. Journal of Policy Analysis and Management, 29(3). Summers, A., & Wolfe, B. (1977). Do Schools Make a Difference? American Economic Review, 67(4), 639-652. Teach for America. (2006). Teach for America — Our Alumni. Retrieved Oct. 10, 2008, from http://www. teachforamerica.org/alumni/index.htm Teach for America. (2008). 2007 Annual Report. New York: Author. End notes 1 Jennifer L. Steele is an Associate Policy Researcher at the RAND Corporation in Washington, DC. Richard 1. Murnane, an economist, is the Thomp- son Professor of Education and Society at the Har— . vard Graduate School of Education in Cambridge, MA. 10hn B. Willett, a statistician, is the Eliot Pro- fessor of Education at the Harvard Graduate School of Education. Their respective email addresses are jsteele@rand.org, richard_murnane@harvard.edu, and john_willett@harvard.edu. ASSESSING THE IMPACT OF THE CALIFORNIA GOVERNOR’S TEACHING FELLOWSHIP a POLICY BRIEF PACE For examples, see Becker, 1952; Betts, Rueben, 8: Danenberg, 2000; Clotfelter, Ladd 8r Vigdor, 2006; Lankford, Loeb 8: Wyckoff, 2002. For examples, see Clotfelter, Glennie, Ladd, 8r Vigdor, 2008; Field, 2009. A school’s ranking on the Academic Performance Index is a function of students’ performance on the Statewide Testing and Reporting (STAR) program of annual standardized tests (California Education Code, Section 69612-6961 5. 6, 2000). Teachers of mathematics, science, or special educa- tion qualify for an additional $1,000 per year, and teachers of those three subjects working in schools with API rankings in the bottom 20% can earn yet another $1,000 annually. Thus, the largest possible award over four years is $19,000. Between the 1999-00 and 2002—03 academic years, the number of standard, in-state APLE contracts awarded each year exceeded the number of first- time, in-state teaching licenses by an annual average of 2,612, or 86 percent (Burke 8r Errett, 2000, 2001, 2002, 2003; California Student Aid Commission, 2004). This is possible because, while the standard APLE contracts are targeted (with a few exceptions) at teachers without prior licenses, some may nevertheless have held prior emergency credentials. It is also possible because not all APLE recipients went on to earn their licenses. This pat- tern may be due to the relative ease of contract attainment. The two-page APLE application can be quickly completed by hand. Recipients are chosen by their licensure programs, and as the APLE pro- gram have expanded, these programs have worked aggressively to award all the contracts allocated to them (California Student Aid Commission, 2005) Relative to first-time licensure recipients who did not have APLE contracts, a similar share of APLE recipients attended a University of Cali- fornia licensure program (roughly 6 percent did so in both groups). However, while only about 45 percent of APLE recipients attended a California State University Licensure program, we estimate that 80 to 90 percent of non-APLE recipients did so. The remaining 49 percent of APLE recipients attended independent institutions, versus 20 percent or fewer of first-time licensure recipients who did not receive APLE contracts. However, the average CBEST scores of APLE recipients are similar to those of other CBEST takers. Using averages provided directly by the CCTC, we find that APLE recipients averaged a score of 50.1 in reading, 42.3 in writing, and 49.2 in math, versus other CBEST takers, whose scores averaged 50.0 in reading, 43.3 in writing, and 47.5 in math. This represents a positive difference of 0.01 standard deviation in reading (sd=15) a negative difference of 0.09 standard deviations in writing (sd=10) and a positive difference of 0.11 standard deviations in math (sd=15). These figures are based on the authors’ calculations using statewide averages provided by the California Commission of Teacher Credentialing. To conduct our analysis, we supplemented the anonymous APLE records with teacher licensure test scores provided by the California Commission on Teacher Credentialing (CCTC) which issues teaching licenses, and with indicators provided by the California State University Chancellor’s Oflice of which APLE recipients received the GTF. The slope of the relationship is not noticeably dif- ferent when we use only math, reading, or writing scores instead of composite scores as an indicator of academic skills. In the APLE recipient database, 54.7 percent of GTF recipients begin teaching in low-performing schools within two years after receiving APLE contracts. This figure differs from the estimate in Figure 2 for two reasons. First, to illustrate the fitted effect of the GTF, Figure 2 holds covariates constant at values prototypical for both GTF recipients and the full sample. These prototypical values are associated with higher rates of entry into low-performing schools than we find in the sample as a whole. Second, Figure 2 incorporates our strat- egy for removing selection bias. As described in the technical article, Steele, Murnane, and Willett (forthcoming), we find that in the absence of the incentive, individuals chosen to receive the GTF were systematically less likely than their observably similar counterparts to begin working in low-per- forming schools. This is consistent with a scenario in which the fellowship selection committee, based on numerous criteria including transcripts, essays, interviews, and recommendation letters, selected candidates with the highest opportunity costs and the most attractive labor market prospects. Figure 2 thus expresses the estimated rates of entry into low-performing schools for prototypical GTF recipients in the absence of this negative selection bias—i.e., it presents an unbiased estimate of the impact the “treatment” on those who received it. We use an average of 40 students per year based on our observation that 43% of GTF recipients in the dataset who taught worked in secondary schools. We conservatively assume that each elementary teacher educated an average of 25 students per year, and that each secondary school teacher educated 60 students per year. 12 Additional details about this method can be found in Steele, Murnane, and Willett (forthcoming). DO FINANCIAL INCENTIVES DRAW PROMISING TEACHERS TO LOW-PERFORMING SCHOOLS? U. C. BERKELEY LIBRARIES CllfibH7351 A POLICY BRIEF PACE School of Education, Stanford University 520 Galvez Mall CERAS Rm. 401 Stanford, CA 94305-3084 (650) 724-2832 We would like to thank the James Irvine Foundation and the William and Flora Hewlett Foundation for financial support for the publication of this policy brief. The Views expressed are those of the authors, and do not necessarily reflect the Views of PACE or its funders. Recent PACE Publications Martin Carnoy. California’s Impending College Graduate Crisis and What Needs To Be Done About It. Policy Brief 10-2, April 2010. Charles Taylor Kerchner. Thereis Lots to Learn from LA: Policy Levers for Institutional Change. Policy Brief 10-1, January 2010. Susanna Loeb and Jon Valant. Leaders for California’s Schools. Policy Brief 09-4, September 2009. Sean F. Reardon and Michal Kurlaender. Efiects of the California High School Exit Exam on Student Persistence, Achievement, and Graduation. Policy Brief 09—3, September 2009. Heather J. Hough. The Quality Teacher and Education Act in San Francisco: Lessons Learned. Policy Brief 09-2, May 2009. Heather J. Hough and Susanna Loeb. The Development of a Teacher Salary Parcel Tax: The Quality Teacher and Education Act in San Francisco. May 2009. Julia E. Koppich and Jessica Rigby. Alternative Teacher Compensation: A Primer. March 2009. Katharine Strunk. Collective Bargaining Agreements in California School Districts: Moving Beyond the Stereotype. Policy Brief 09- 1, January 2009. Conditions of Education in California, October 2008. Susanna Loeb and David Plank. Learning What Works: Continuous Improvement in California’s Education System. Policy Brief 08—4, August 2008. Julia Koppich. Reshaping Teacher Policies to Improve Student Achievement. Policy Brief 08-3, March 2008. Susanna Loeb, Tara Beteille, and Maria Perez. Building an Information System to Support Continuous Improvement in California Public Schools. Policy Brief 08-2, February 2008. Jennifer Imazeki. Meeting the Challenge: Performance Trends in California Schools. Policy Brief 08—1, February 2008. Anne K. Driscoll. Beyond Access: How the First Semester Matters for Community College Students'Aspirations and Persistence. Policy Brief 07—2, August 2007. W. Norton Grubb and David Stern. Making the Most of Career— Technical Education: Options for California. Policy Brief 07-1, April, 2007.