MSP Papers on Evidence-Based Design and Outcomes: A dynamically generated bibliography of MSP authored papers
Abstract
The MSP Program supports research that is based on evidence. Below you will find a dynamically generated bibliography of papers relating to evidence-based design and outcomes, drawn from papers authored by the MSPs, which you can view with or without abstracts. This will automatically update as new papers are added to the MSpnet library. We invite you use this list as you contribute to the literature on evidence-based design and outcomes.
hide abstracts
91 documents as of 07/22/2019
Abstract: Research recognizes that the greatest determinate of student achievement is the teacher, yet questions remain as to what characteristics of teachers are the most influential. This paper explores the teacher effect through the evaluation of the interaction among characteristics of four teacher profiles using Surveys of Enacted Curriculum. This Teacher Effect Model was designed to evaluate the future impacts of sustained professional development.
Abstract: Over the past six years, the $9.3 million NSF-funded NebraskaMATH grant has impacted more than 700 teacher leaders across the state. The NebraskaMATH Final Report, published in 2015, showcases the scope of the grants three core programs, Primarily Math, New Teacher Network and Nebraska Algebra, as well as the programs that grew from them. The report tells some of the stories of NebraskaMATH and its impact on mathematics teaching and learning in Nebraska.
Abstract: "The Life Sciences for a Global Community (LSGC) Teacher Institute is a collaboration among Washington University in St. Louis, Missouri Botanical Gardens, the Donald Danforth Plant Science Center, and St. Louis Public Schools to establish an advanced degree program and leadership training for teams of high school biology teachers. The three-year degree program includes a total of nine week-long summer courses focused on disciplinary content. Academic year coursework focused on pedagogical strategies for incorporating new content into teachers' classrooms. As an example, the Evolutionary Ecology summer course addressed major ecological theories through investigations that focused on a research question, data collection and analysis related to the question, and interpretation of the analytic results. The Program Capstone 1 course was taught by science educators as a distance-learning course during the academic year; participating teachers engaged in action research around their implementation of curricular pieces they developed based on content and pedagogical strategies they encountered at the Institute."
Related Case Studies:
- Introduction
- The Indiana Mathematics Initiative Partnership
- The Milwaukee Mathematics Partnership
- The Southwest Pennsylvania Math Science Partnership
- The Arizona Teachers Institute
- The Michigan Teaching Excellence Program
- The Greater Birmingham Mathematics Partnership
- The North Cascades and Olympic Science Partnership
- The Oregon Mathematics Leadership Institute
Abstract: "The problem of how to improve elementary student science achievement in the United States is multi-faceted. To be effective, interventions must consider challenges associated with teaching, learning, and implementing instructional changes at a large scale. In this paper, we present findings from a study of an educative curriculum materials-based intervention that has three central design principles: 1) the materials are aligned with current knowledge about how people learn; 2) the educative components support teacher content and pedagogical content knowledge, and facilitate instructional implementation; and 3) the instructional activities use low-cost, readily available materials amenable to large-scale implementation. Our findings indicate that student learning gains are greater in classes where teachers implement the intervention than in comparison classes. In addition, the extent of materials implementation and fidelity to the pedagogical approach embodied in the materials are positively associated with student achievement gains. Implications of these findings for supporting implementation of the Next Generation Science Standards are discussed."
Abstract: A major current policy question is whether in-service training can improve teacher quality and student outcomes in K-12 schools. This paper argues that the evidence on the effectiveness of professional development thus far is less definitive than it should be, in part, because past evaluations have failed to account for the non-random selection of teachers into the programs. If high quality teachers, a priori, are provided with professional development, evidence may suggest an effective treatment even when the measured effect is actually reflecting the fact that the treatment is applied to already high-performing teachers. The reverse holds if less effective teachers are targeted for professional development.
This paper recognizes this potential bias and provides a method for correcting the non-random selection into the program. The correction offered in this paper is complicated by the fact that teaching effectiveness can be measured only partially with observable factors. By estimating the contribution of unobservable factors on a teacher's own past effectiveness, we find that this particular program, through targeting poor performing districts, succeeded in attracting the weakest teachers into the professional development activities. Controlling for this, the professional development program shows positive effects. This paper provides a road map for future evaluations not only of this program but other teacher professional development training programs in which selection of teachers is likely to be non-random and often based on unobservable teacher effectiveness. Only with evaluation replication for a variety of programs can policy advice about designing future professional development be offered.
NOTE: This resource is published on the Evaluation Review web site. The link above will take you to an abstract page. To view the full-text for this article, you will need to either sign-in to Evaluation Review with the appropriate membership or purchase this article.
This paper recognizes this potential bias and provides a method for correcting the non-random selection into the program. The correction offered in this paper is complicated by the fact that teaching effectiveness can be measured only partially with observable factors. By estimating the contribution of unobservable factors on a teacher's own past effectiveness, we find that this particular program, through targeting poor performing districts, succeeded in attracting the weakest teachers into the professional development activities. Controlling for this, the professional development program shows positive effects. This paper provides a road map for future evaluations not only of this program but other teacher professional development training programs in which selection of teachers is likely to be non-random and often based on unobservable teacher effectiveness. Only with evaluation replication for a variety of programs can policy advice about designing future professional development be offered.
NOTE: This resource is published on the Evaluation Review web site. The link above will take you to an abstract page. To view the full-text for this article, you will need to either sign-in to Evaluation Review with the appropriate membership or purchase this article.
Abstract: "The process of science program evaluation needs more voice; there is as much to be gained from sharing how critical thinking evolves into data collection instruments as there is in describing the results of a program's impact. Knowledge-sharing around instrumentation and data collection builds expertise in the field of program evaluation and treats the process of documenting program and research design as a collective responsibility (Design-Based Research Collective, 2003; National Research Council, 2001). In this paper, we practice what's been preached by articulating the steps we took to develop an observation protocol for observing science content professional development workshops for high school teachers. Understanding and discussing these processes illustrates the iterative process through which teaching practice informs instrument development and vice versa. What follows is a narrative of how we ended up with our final measure and what we learned about instrument development and data collection along the way. The end goal is to stimulate conversations among program evaluators and science educators about the characteristics of research design and instrumentation."
Abstract:
"The research and evaluation theme of this proposal from the Mathematical ACTS MSP project at the University of California-Riverside is evidence-based design for facilitating teacher change. Using the project's logic model this paper will begin with an overview of its activities, intended outcomes, and evaluation design. The remainder of the paper will focus on the development, training and administration, and results from an observation instrument used to assess the classroom practices of teachers who participated in Mathematical ACTS professional development and follow-up support over the previous school year, as well as the practices of comparison group teachers. The role of this assessment in the project's evidence-based design will also be explained.
Specifically, the paper will describe: the project's decision to create an observation instrument rather than use one of several existing instruments, the collaborative development process involving core team members, training of observers, procedures for establishing inter-rater reliability, the use of the instrument, analyses conducted and lessons learned for this pilot year.
Preliminary findings from the observations will compare Mathematical ACTS participants and non-participants. Interpretation of these findings will be discussed, along with anticipated further analyses. The paper will conclude with a discussion of issues and challenges, plans for observations to be conducted in the 2005-06 school year, and commentary on this assessment's role in providing evidence for the effectiveness of Mathematical-ACTS."
Abstract: "The Southwest Pennsylvania MSP's work in deepening teacher content knowledge has scalability and sustainability at its core. The project focuses its efforts on preparing teacher leaders, involving national experts from groups such as the BSCS National Academy of Curriculum Leadership and the Exploratorium's Institute for Inquiry. Over the course of two years, elementary teacher leaders (1) learn about science as a discipline, especially the role of inquiry in generating new knowledge; and (2) deepen their disciplinary content knowledge and pedagogical content knowledge in selected areas of science. Prospective teacher leaders learn forces and motion content and consider classroom applications of what they are learning in the context of commonly-used student instructional materials. They learn about electricity by engaging with case materials developed by WestEd, which include analysis of student thinking instructional strategies for teaching that content. In turn, these teacher leaders engage other elementary teachers in their districts in defined learning experiences, drawn from the professional development materials and strategies they themselves had experienced."
Related Case Studies:
- Introduction
- The Indiana Mathematics Initiative Partnership
- The Milwaukee Mathematics Partnership
- The Arizona Teachers Institute
- The Michigan Teaching Excellence Program
- The Life Sciences for a Global Community Teacher Institute
- The Greater Birmingham Mathematics Partnership
- The North Cascades and Olympic Science Partnership
- The Oregon Mathematics Leadership Institute
Abstract: "The transition into the twenty-first century has led to a greater emphasis placed on student proficiencies in Science, Technology, Engineering, and Mathematics (STEM), with a particular focus on how these skills will help students thrive in the technological world and society. Most jobs require some level of mathematical proficiency, and mathematics skills are crucial for successful integration and independence in the home and community (Patton, Cronin, Bassett, & Koppel, 1997). Yet, mathematical achievement in the United States has been below the level attained by students in other countries, with American students becoming notably behind once they reach late middle school (U.S. Department of Education, 2008). To address these weaknesses, educators have developed various methods to increase student's STEM content knowledge. These include, but are not limited to, furthering teacher's professional development (PD) requirements, varying curriculum, and adding additional STEM classroom time."
Abstract: "Purpose: This study examines the feasibility and utility of a daily log for measuring principal leadership practice.
Setting and Sample: The study was conducted in an urban district with approximately 50 principals.
Approach: The log was assessed against two criteria: (a) Is it feasible to induce strong cooperation and high response rates among principals with a daily instrument? and (b) Can daily logs accurately measure important aspects of principal leadership? The first criterion was assessed through a discussion of data collection procedures and results. The second criterion was assessed through mixed-method analyses comparing daily logs, observations, and an experience-sampling instrument.
Results: The authors found that substantial participant contact time and strategic follow-up achieved strong cooperation and yielded high response rates. The accuracy of the log was confirmed through comparisons with an experience-sampling instrument and direct observations. The results also contribute to a broader understanding of how principals allocate their time across leadership domains. Like earlier structured observation studies, the authors found that principals spend more time on management, personnel issues, and student affairs and less time on instructional leadership than advocated by leadership scholars and professional standards.
Implications for Research and Practice: Daily logs appear to be a viable means of measuring important aspects of principal practice and overcoming measurement errors associated with one-time surveys that are common in leadership research. Strategies used to maintain high participation rates are discussed in detail, and an example of a district's adaptation of the daily log methodology is provided."
Abstract: "Elementary mathematics coaches are placed in schools to construct leadership roles and to provide on-site, collaborative, professional development addressing mathematical content, pedagogy, and curriculum in an effort to enhance instruction and to improve student achievement. This 3-year randomized control study found that over time coaches positively impacted student achievement in Grades 3, 4 and 5. In these grades, this significant positive effect on student achievement was not evident at the conclusion of the first year of placement of a coach in a school. It emerged as knowledgeable coaches gained experience and as a school's instructional and administrative staffs learned and worked together. The coaches in this study engaged in a high degree of professional coursework addressing mathematics content, pedagogy, and coaching prior to and during at least their first year of placement. Findings should not be generalized to coaches with less expertise."
Abstract: "SCALE is a five-year project, and thus it provides us with an excellent opportunity to
examine partnerships as they develop. SCALE's proposal writers included a longitudinal
organizational analysis study with formative and summative components as part of the
partnership's research and evaluation plan. Our research asks the following broad
questions about the SCALE partnership:
1. Why do particular structures, functions, and people come to work within the
SCALE partnership?
2. How do leaders organize people and tools to accomplish partnership goals, as
stated in the formal initiating documents?
This paper particularly addresses the second research question by describing how the
partnership, as an organization, has been positioned to meet its goals. In describing the
structures, functions, and people actively engaged in SCALE, we introduce
"organizational mapping" as a method of census-taking, which provides a partial
representation and understanding of the partnership."
Abstract: This paper discusses the design of the case studies of the effects of the SCALE partnership on district policy and organization. (See Appendix A. for an overview of SCALE). Part I discusses the SCALE theory of action -- how the partnership goes about producing change -- which then becomes the focus of evaluation. Part II discusses research methods. But first what are the case studies?
The case studies are intended to study the effects of SCALE on district policy and organization, especially instructional guidance. (See Appendix A. for description of the SCALE research and evaluation). This is an area "downstream" from the focus of the Building a Partnership (BP) team, which examines partnership dynamics and the formation of partnership initiatives. And it is "upstream" from the focus of targeted studies and indicators, which examine outcomes of SCALE interventions on instruction and student achievement. Like BP, the case studies use mainly qualitative methods in tracking organizational change and dynamics, but like targeted studies and indicators, they are mainly concerned with effects inside the partner school districts."
The paper and its accompanying slides are both available in PDF format. Click on the file links above.
Abstract: Many urban districts have adopted interim assessments in recent years as a supplement to annual testing. Possible purposes for such tests include monitoring and assisting student progress, aligning the curriculum, and practicing for state exams, but little is known about how effectively real systems advance these purposes. The Providence Public School District (PPSD) implemented a well-crafted system of quarterly assessments at every grade starting in 2004 and discontinued them in 2007. During the implementation period, we interviewed district officials about what they were trying to accomplish, and we asked teachers and other school personnel about their experiences with the assessments and their use of the results. We found some evidence of effectiveness as well as some limitations. Uncertainty about the magnitude of positive impacts raised questions about the net value of the assessments given their considerable costs to districts, teachers, and students.
Abstract: This guiding framework arose out of a need to:
"(a) provide guidance for evaluation planning and evaluation activities to NSF's MSP projects and other projects, and to groups submitting proposals to NSF programs;
(b) have a consistent framework by which to assess project-level evaluation; and
(c) develop a document about project-level evaluation, grounded in the expertise and experience of the scholarly community having that expertise."
..."This document includes the following major sections:
(a) A statement about high quality evidence of project effectiveness and efficiency.
(b) A description of the DIO [Design-Implementation-Outcomes] Cycle of Evidence as a guiding framework for planning, gathering, and using evidence.
(c) The relationship of the DIO Cycle of Evidence to other frameworks used in evaluating projects.
(d) The role of context in establishing evidence of project effectiveness.
(e) Resources to help projects learn more about planning, gathering, and using evidence.
(f) A glossary of terms and abbreviations used throughout this document. (g) Appendices that contain supplemental resources."
"(a) provide guidance for evaluation planning and evaluation activities to NSF's MSP projects and other projects, and to groups submitting proposals to NSF programs;
(b) have a consistent framework by which to assess project-level evaluation; and
(c) develop a document about project-level evaluation, grounded in the expertise and experience of the scholarly community having that expertise."
..."This document includes the following major sections:
(a) A statement about high quality evidence of project effectiveness and efficiency.
(b) A description of the DIO [Design-Implementation-Outcomes] Cycle of Evidence as a guiding framework for planning, gathering, and using evidence.
(c) The relationship of the DIO Cycle of Evidence to other frameworks used in evaluating projects.
(d) The role of context in establishing evidence of project effectiveness.
(e) Resources to help projects learn more about planning, gathering, and using evidence.
(f) A glossary of terms and abbreviations used throughout this document. (g) Appendices that contain supplemental resources."
Abstract: This study examines the contributions to education research and practice by NSF's Math and Science Partnership (MSP) Program. NSF's overall strategic plan calls for "advancing discovery, innovation and education beyond the frontiers of current knowledge, and empowering future generations in science and engineering" (NSF 06-48). In keeping with the overall strategic plan, the 2008 MSP Program solicitation (NSF 08- 525) specifically states: "The Math and Science Partnership (MSP) program is a major research and development effort...MSP projects contribute to the knowledge base for mathematics and science education and serve as models that have a sufficiently strong evidence base to be replicated in educational practice." The MSP Program therefore falls within the broader context of research and development and current efforts to improve science, technology, engineering, and mathematics (STEM) education.
The MSP Program consists of comprehensive, targeted, and institute awards (MSPs) as well as research, evaluation, and technical assistance (RETA) awards. The study's findings show that the awardees have generated a large number of scholarly and practice- oriented publications. By this measure, the MSP Program appears to be making a solid contribution to education research and practice.
Abstract: Starting in 2002, the National Science Foundation has made 77 grants to implement its Math and Science Partnership (MSP) Program (NSF-MSP).1 Besides these 77 grants, other program activities have included an MSP-Management Information System (MSP-MIS), whose purpose is, in part, to assess the overall implementation of the MSP Program and to monitor the progress of individual MSP grants.2 Such implementation and monitoring are complex affairs because of the complexity of the MSPs grants. Each grant is a partnership, minimally involving a K-12 district and an institution of higher education (IHE). More often, however, multiple districts and multiple IHEs are engaged in a single MSP grant.
The MSP-MIS collects annual data from all grantees, based on four instruments:
- K-12 District Survey;
- Survey for Partnership Projects;
- IHE survey; and IHE participant survey.
Abstract: This study in the evaluation design of the MSP-PE investigates changes in student mathematics and science achievement across three school years, 2002- 03, 2003-04, and 2004-05, for partnership-related schools using MIS data with the Annual K-12 District Survey. First, changes in percent of students (at or above) proficient on state assessments in math and science were investigated by gender, ethnicity, special education, and students with limited English proficiency using schools for which data were available for all three years. The results indicated that partnership schools continued to show improvement in student math and science proficiency over the three-year time period. Second, schools were examined by frequency and effect size of increase, decrease, or no change in student math and science proficiency from the "start" (2002-03) to the "end" (2004-05) of the time period for this study. The schools with positive changes were in much higher numbers and higher mean effect size of change compared to schools with negative (or no) changes in student math and science proficiency. Third, the relationship between the schools' targeted teacher participation in partnership-related activities over the entire time period of three years and the student math and science proficiency at the "end" year of this time period (2004-05) was also investigated. It was found that this relationship was positive and significant for the elementary and high schools, but there was no evidence for its significance at the middle school level.
Abstract: This substudy in the evaluation design of the Math and Science Partnership (MSP) Program Evaluation examines student proficiency in mathematics and science for the MSPs' schools in terms of changes across three years (2003/04, 2004/05, and 2005/06) and relationships with MSP-related variables using Management Information System data with the Annual K-12 District Survey. First, changes in percentages of students at or above proficient on state assessments in math and science were investigated by gender, ethnicity, special education, and students with limited English proficiency across the targeted three-year period (2003/04 - 2005/06). The classification of MSP schools with and without focus on math or science during this time period was also taken into account. The results indicated that the MSP-related schools demonstrate sustained increase in percent of students at or above proficient in both math and science at the elementary and middle school levels, but not quite so at the high school level. Second, schools were examined by frequency and effect size of increase, decrease, or no change in student math and science proficiency. The schools with positive changes were in much higher numbers and higher mean effect size of change compared to schools with negative (or no) changes in student math and science proficiency. Third, the relationship between the schools' targeted teacher participation in MSP-related activities over the entire period of three years and the student math and science proficiency at the "end" year of this period (2005-06) was also investigated. This relationship was positive, yet small, at all school levels for mathematics, and also positive, yet much better pronounced, at the high school level for science. Fourth, longitudinal growth trajectories in math and science proficiency across the three years were also investigated. The results showed that the schools with MSP focus on math (or science) increase at higher rate in math (or science) proficiency compared to those without MSP focus on math (or science) at the middle school level.
Abstract: This study is one in a series of substudies for the National Science Foundation's Math and Science Partnership (MSP) Program Evaluation. The study examines student proficiency in math and science for the MSPs' schools in terms of changes across four years (2003/04, 2004/05, 2005/06, and 2006/07) and relationships with MSP-related variables using MSP-Management Information System data from the Annual K-12 District Survey. First, changes in percentages of students at or above proficient on state assessments in math and science were investigated by gender, ethnicity, special education, and students with limited English proficiency across the targeted four-year period (2003/04 - 2006/07). The classification of MSP schools with and without focus on math or science during this time period also was taken into account. The results indicated that the MSP-related schools demonstrate a sustained increase in the percent of students at or above proficient in both math and science at all school levels. This trend was more clearly pronounced for schools with focus on math or science. Second, schools were examined by frequency and effect size of increase, decrease, or no change in student math and science proficiency. The schools with positive changes were in much higher numbers and higher mean effect size of change compared to schools with negative changes in student math and science proficiency. This trend was better pronounced for schools with focus on math at the elementary and middle school levels and for schools with focus on science at the middle and high school levels. Third, the relationship between the schools' targeted teacher participation in MSP-related activities over the four-year time period (2003/04-2006/07) and the students' math and science proficiency at the "end" year of this period (2006-07) also was investigated. This relationship was positive, yet relatively small, at elementary and high school levels for mathematics, and also positive, yet somewhat better pronounced, at the high school level for science. Fourth, longitudinal growth trajectories in math and science proficiency across the four years also were investigated. The results showed that the schools with MSP focus on math (or science) increase at a higher rate in math (or science) proficiency compared to those without MSP focus on math (or science) for the elementary and middle schools in math and for the elementary schools in science.
Abstract: This substudy in the evaluation design of the Math and Science Partnership (MSP) Program Evaluation examines student proficiency in mathematics and science for the MSPs' schools in terms of changes across five years (2003/04, 2004/05, 2005/06, 2006/07, and 2007/08) and relationships with MSP-related variables using Management Information System data with the Annual K-12 District Survey. First, changes in percentages of students at or above proficient on state assessments in math and science were investigated by gender, ethnicity, special education, and students with limited English using the MIS data available for (a) across the five-year period (2003/04 - 2007/08) and (b) same schools across the last four years (2004/05-2007/08), with the purpose to obtain a sample of schools without missing data for dependable longitudinal analyses. The classification of MSP schools with and without focus on math or science for the longitudinal data over this four-year time period (2004/05-2007/08) was also taken into account. The results indicated that the MSP-related schools demonstrate sustained increase in percent of students at or above proficient in both math and science at all school levels. This trend was more clearly pronounced for schools with focus on math or science. Second, schools were examined by frequency and effect size of increase, decrease, or no change in student math and science proficiency. The schools with positive changes were in much higher numbers and higher mean effect size of change compared to schools with negative changes in student math and science proficiency. This trend was better pronounced for schools with focus on math at the elementary and middle school levels and for schools with focus on science also at the elementary and middle school levels. Third, longitudinal growth trajectories in mathematics and science proficiency across the four years (2004/05-2007/08) were investigated. The results indicated the existence of different latent classes of growth trajectories of school success on state assessments in mathematics and science
Abstract: "The National Science Foundation (NSF) has released its first national impact report assessing the NSF Math and Science Partnership (MSP) program, which was established in 2002 to integrate the work of higher education with K-12 to strengthen and reform mathematics and science education.
The document reports progress on improving teacher quality, quantity and diversity; developing challenging courses and curricula; emphasizing evidence-based design and outcomes; and promoting institutional change. It highlights examples of partnerships at all levels of education in communities across the country, and outlines impacts on student proficiency and benefits of professional development for teachers."
Abstract: Strengthening America by advancing academic achievement in mathematics and science
"The National Science Foundation (NSF) has released its second national impact report for the NSF Math and Science Partnership (MSP) program, which was established in 2002 to integrate the work of higher education with K-12 to strengthen and reform mathematics and science education. The document identifies progress on improving teacher quality, quantity and diversity; developing challenging courses and curricula; emphasizing evidence-based design and outcomes; and promoting institutional change. The report highlights examples of partnerships at all levels of education in communities across the country, and includes examples of positive impacts for students and benefits of professional development for teachers." Related Document: MSP Impact Report, Jan. 2007
"The National Science Foundation (NSF) has released its second national impact report for the NSF Math and Science Partnership (MSP) program, which was established in 2002 to integrate the work of higher education with K-12 to strengthen and reform mathematics and science education. The document identifies progress on improving teacher quality, quantity and diversity; developing challenging courses and curricula; emphasizing evidence-based design and outcomes; and promoting institutional change. The report highlights examples of partnerships at all levels of education in communities across the country, and includes examples of positive impacts for students and benefits of professional development for teachers." Related Document: MSP Impact Report, Jan. 2007
Abstract: The AMSP has funded more than 50 Partnership Enhancement Program grants as a means of initiating and furthering partnerships between K-12 and higher education institutions working to address local mathematics and science education challenges. The purpose of this study was to analyze the effect and activities of 10 PEP awards during 2005-06 in terms of leadership and overall structural characteristics that lead to successful outcomes. The underlying hypothesis of this study is that "distributed" leadership, that is, leadership responsibilities that are shared across institutional levels, provides an ideal structure for promoting and encouraging teacher leadership and successful university-school partnerships.
Abstract: Teachers participating in the College Ready MSP have used the enactment of curriculum resources as a professional development activity. Curriculum materials and/or instructional methods from the College Ready workshops are revised in collaboration with program staff to best serve their students and their classrooms. These projects use research-based methods including backwards design and learning cycles to shift to more student-centered instructional methods.
Abstract: A brief summary of evidence-based strategies for professional development is presented. The objective is to support College Ready teachers in the collection of data that will inform their Curriculum Enactment projects.
Abstract: "College Ready participants have been encouraged to revise curriculum materials in ways that will better serve their students and be consistent with their instructional resources. To help guide decisions about the revision of curriculum, participants each year have been given the opportunity to survey their students readiness to learn. These surveys have addressed student beliefs about the nature of knowledge, the nature of science, and their role in learning. Teachers have also been given the opportunity to complete similar surveys that can inform curriculum development by investigating the accuracy of the teachers perception of student beliefs. Summaries of these comparisons are provided here for all courses for which both College Ready teachers and their students responded to the survey."
Abstract: This document provides a brief description of the Inventory of Teaching and Learning (ITAL) developed and currently being implemented as one component of an external evaluation of the Georgia Partnership for Reform in Science and Mathematics (PRISM). The Georgia PRISM program is a cooperative partnership between four regional higher education institutions and their surrounding school districts. The PRISM external evaluation is a longitudinal (2003- 2008), mixed methods (qualitative and quantitative) effort to document change processes and outcomes in K-16 science and mathematics. The change model includes emersion of K-12 teachers and higher education faculty in professional development and in-service education programs, learning communities, and faculty institutes. The ITAL is a web-based survey of teaching and learning practices designed to reflect reformed (inquiry-based and standards-based) teaching and learning environments/practices and more traditional learning environments/practices. Included in this document are: (a) a brief description of the historical development of the ITAL; (b) a summary of the current (third) form of the ITAL items and their empirical classifications on three measurement dimensions; and (c) some suggested uses of the ITAL. More detailed descriptions of the methodology used to develop the ITAL can be found in Ellett, Monsaas, Payne, & Pevey (2005; 2006) and Ellett & Monsaas (2007).
Abstract: "Through the Indiana University-Indiana Mathematics Initiative (IMI) Partnership, Indiana University collaborated with nine school districts from across Indiana to provide professional development for teachers of mathematics. In the elementary grades, the goal of the project was to support the implementation of a curriculum program chosen by all partner districts, Everyday Mathematics. In the early years of the project, IMI professional development focused on the mathematics content of the curriculum which was closely aligned with the Indiana Academic Standards. In cross-district, grade-level group meetings in the summer and three to four times during the academic year, teachers engaged in key activities from the materials to experience the mathematics content development for their grade level as learners, and then planned together for classroom implementation.
After this first year of support, a subset of teachers from each grade level in each district was invited to join the IMI Select Cadre. Select Cadre teachers continued their professional development in subsequent years with emphases on content development across grade levels, along with leadership training to provide professional development and mentoring support for other teachers. Select Cadre teachers continued to attend annual summer workshops, as well as three to four meetings during each school year. Teachers also engaged in online reflections, through weekly Learning Logs the first year and, for the Select Cadre teachers, through monthly Assessment Logs in subsequent years. In their Learning Log entries, teachers reflected on their progress and challenges implementing the Everyday Mathematics curriculum, evidence of their students' learning, and their own mathematical insights. Assessment Log entries were tied to each unit taught. Teachers administered both designated common assessment items and assessment tasks of their own choosing to document, interpret, and report their students' progress with the mathematics learning goals of each unit and continued to reflect in writing on their own mathematical insights. Professional development providers read and responded to each Learning Log and Assessment Log entry as a means of ongoing support for teachers' learning."
Related Case Studies:
- Introduction
- The Milwaukee Mathematics Partnership
- The Southwest Pennsylvania Math Science Partnership
- The Arizona Teachers Institute
- The Michigan Teaching Excellence Program
- The Life Sciences for a Global Community Teacher Institute
- The Greater Birmingham Mathematics Partnership
- The North Cascades and Olympic Science Partnership
- The Oregon Mathematics Leadership Institute
Abstract: In this paper, the authors outline the financial support and structure that will enable university faculty and school system administrators and teachers to establish Mathematics Specialists in school systems across Virginia. Support from both corporate and federal/state grants will support programs to train and place Math Specialists in schools, and will also fund research to prove that well-prepared, school-based Mathematics Specialists will lead to major gains in student learning.
Abstract: This paper is one in a series of briefs for the Math and Science Partnership Program Evaluation (MSP-PE), conducted for the National Science Foundation's Math and Science Partnership Program (NSF-MSP).
The paper's topic originates from discussions held at the MSP-PE's advisory board meeting in October 2006. Advisory board members suggested that "expectations" about the nature and level of outcomes from programs like the NSF MSP Program often go unaddressed. This suggestion led to the commissioning of the present paper. Although the paper does not set specific expectations, it offers a useful framework for others to do so. In this sense, the paper serves as part of the MSP-PE's evaluation design work.
Abstract: "The growing interest in distributed leadership reflects an effort to reconceptualize
leadership in schools by exploring how leadership is spread across a variety of roles and
to explore the process of leadership. Using case studies of four schools in three districts,
this paper explores how leadership is distributed in school districts and asks about the
role of teacher leaders in particular. It proposes that teacher leaders and districts can
share three leadership tasks: procuring and distributing materials, monitoring
improvement, and developing people. The district and teacher leaders play
complementary roles. Districts tend to be distant forces, and teacher leaders are more
personal. How effective teacher leaders do at people development will depend on the
time they have, the knowledge they have, and their monitoring responsibility. These
conditions depend partly on their administrative support. We suggest that districts may
have more opportunity to influence teaching practice than past research had indicated."
Abstract: "Federally funded K-12 science, technology, engineering, and mathematics (STEM) education projects are generally expected not only to use what is already known in designing and implementing interventions but also to add to the knowledge base. STEM education projects supported by NSF, the Department of Education, and other funders, often involve a mix of people with quite different backgrounds and prior experiences: STEM faculty; STEM education faculty; district supervisors; and master teachers. Involving people with diverse backgrounds helps bring a great deal of expertise to the table, but at the same time it can create communication challenges, as the terminology that some people find extremely useful can seem like just a lot of jargon to others. This document provides a brief introduction to research on K-12 STEM education interventions; it is intended to help people who may be new to social science research understand some of the key issues. We include some terminology commonly used in social science, but the emphasis is on developing concepts that project teams can refer to as they design and implement research."
Abstract: Alternative statistical models have been proposed for evaluating the effects of the Milwaukee Mathematics Partnership (MMP) on teachers and students. These models have been articulated in the form of hierarchical linear models that incorporate various combinations of student, teacher, classroom, and school-level variables. For example, one HLM approach would nest students within teachers, within schools to estimate the effect of MMP activities on student achievement.
Each attempt to estimate the impact of the MMP on teachers and students, however, share a common, underlying hypothesis: MMP activities, e.g., teacher professional development are impacting teacher mathematical knowledge for teaching (MKT), which is in turn impacting classroom practice, leading to improved student achievement. This paper steps away from the prior HLM analyses and explores these relationships using structural equation modeling techniques.
The path model proposed by this paper hypothesizes relationships between teacher education, teacher experience, professional development hours, mathematical knowledge for teaching (MKT), classroom practice, and student achievement (see Figure 1). We hypothesize that education, experience, and professional development hours are predictors of MKT and classroom practice. We further hypothesize that MKT and classroom practice are predictors of student achievement.
Given the exploratory nature of this work, the purpose of this paper is to present the results from an initial attempt to apply path analysis techniques to the problem of linking MMP activities to teacher performance and student achievement. We expect this work will be refined in the remaining years of the MMP evaluation. As such, in addition to presenting the results from this work, recommendations for improving the path model, as well as the implications of this work for evaluation are offered.
Each attempt to estimate the impact of the MMP on teachers and students, however, share a common, underlying hypothesis: MMP activities, e.g., teacher professional development are impacting teacher mathematical knowledge for teaching (MKT), which is in turn impacting classroom practice, leading to improved student achievement. This paper steps away from the prior HLM analyses and explores these relationships using structural equation modeling techniques.
The path model proposed by this paper hypothesizes relationships between teacher education, teacher experience, professional development hours, mathematical knowledge for teaching (MKT), classroom practice, and student achievement (see Figure 1). We hypothesize that education, experience, and professional development hours are predictors of MKT and classroom practice. We further hypothesize that MKT and classroom practice are predictors of student achievement.
Given the exploratory nature of this work, the purpose of this paper is to present the results from an initial attempt to apply path analysis techniques to the problem of linking MMP activities to teacher performance and student achievement. We expect this work will be refined in the remaining years of the MMP evaluation. As such, in addition to presenting the results from this work, recommendations for improving the path model, as well as the implications of this work for evaluation are offered.
Abstract: "The ultimate goal of any professional development opportunities for teachers ought to be the improvement of student learning. It is a goal of the Math in the Middle Institute Partnership (M2) to examine the link between increasing teachers' capacity through professional development and raising the level of student achievement. There are a number of other researchers who have pursued similar lines of inquiry (Cohen & Hill, 2000; Desimone, Porter, Garet Yoon, & Beirman, 2002; Swanson & Stevenson, 2002). This project faces the unique challenge of trying to study change in student learning as a result of professional development offered through the Math in the Middle Institute Partnership to 96 Nebraska teachers spread across more than 77,000 square miles, teaching in 47 different school districts and 67 different schools in a context where there exists no common statewide assessment system. This study presents preliminary findings from one part of the Math in the Middle Institute Partnership's research agenda, the Math in the Middle Institute Partnership's initial efforts to create and administer a common assessment to Math in the Middle teachers' middle school students. The study includes both quantitative and qualitative analyses of the variation of students' responses to the assessment given in Fall 2005 as well as a critique of the assessment problems themselves."
Abstract: This article describes three challenging yet accessible mathematics courses designed especially for middle school teachers and offered by the Department of Mathematics at the University of Nebraska-Lincoln (UNL). The descriptions found in this article are based on the courses as we have taught them as part of the Math in the Middle Institute Partnership (M2), a National Science Foundation Math Science Partnership (MSP) program that works with practicing teachers. As a grant funded MSP, we take seriously the responsibility to share information about our program and the courses we have created. Readers are directed to our web site for a wealth of information about our program and specifically to our course materials for a link to additional information about the courses described in this article as well as other courses that we offer.
Abstract:
"The purpose of this paper is to illustrate the use of evidence collected though an online instructional log system in the evaluation of a Math and Science Partnership (MSP). The Design, Implementation, Outcome (DIO) cycle for the use of evidence that has been identified by the Building Evaluation Capacity project (Callow-Heusser, 2005) provides a helpful framework for the paper. The online log system was developed to support teachers involved in the MSP's elementary grades program. The log system has been used for several purposes:
- To support the design of the program;
- As a tool for teacher reflection;
- For communication with the project; and
- To collect evidence regarding the implementation and outcomes of the program.
That evidence, as the DIO cycle suggests, has in turn been used to reconsider and modify the design of the program as the partnership has moved forward during its first three years. The paper begins with background information on the Indiana University-Indiana Mathematics Initiative Partnership, followed by a description of the online instructional log system. The next two sections illustrate: (1) how the partnership has used evidence from the logs about implementation to make modifications and decisions about its design, and (2) preliminary evidence of outcomes and impacts that can be gleaned from the logs. The conclusion summarizes lessons learned about using the online log system in the evaluation of the MSP."
Abstract: Classroom observation is an important component of mathematics and science teacher professional development programs and other educational evaluation activities. In this paper, the authors extend their earlier qualitative examination and comparison of two popular classroom observation tools (Horizon Research, Inc.'s Inside the Classroom Observation and Analytic Protocol (ITC) and Sawada et al.'s Reform Teacher Observation Protocol (RTOP)) with a quantitative analysis. The instruments substantially are based on comparable assumptions, foundational philosophies and domains of interest, and appear to be used with an expectation that they yield similar results, not to mention that their domains adhere internally. A single well-trained educator observed 21 teachers from a Mathematics and Science Partnership over the course of two years. Pearson Correlation Coefficient analysis was applied to items across and within instruments. Because of the relatively small number of teachers and the use of one rater, the authors applied a relatively strict interpretation (.75) of high correlations. While some items correlated as expected within and between instruments based on domain and item construction, many items lacked matches, including those with an apparently similar focus. RTOP items showed greater alignment than did ITC items. Among summative "synthesis" domain ratings in the ITC, none of the four aligned with any of the specific ITC item indicators within their own domain categories. Subtle differences in wording and implicit differences in overall focus especially appear to restrict both internal matches and matches between superficially comparable items across the two instruments. These findings complicate the interpretation of observational results and challenge assumptions that the instruments are interchangeable or necessarily internally consistent. Additional research and development of observation instruments is needed, and users of existing instruments must carefully assess their own needs and understandings before attempting to draw conclusions about classroom practice based on them.
Abstract: Observation of classroom lessons is an important if problematic component of educational evaluation. Where programs aim at teacher professional development, student performance enhancement, curricular change, or implementation of reform teaching methods, direct observation is crucial to validating and explaining results. However, the challenges of funding and maintaining uniformly trained observers and optimally timing observations have led some evaluators to avoid this evaluative technique. Of greatest importance is the selection of an observational instrument that captures information across domains of interest with sufficient validity and reliability. This presentation offers an analysis and comparison of the two most often used observational instruments in math and science K-12 classrooms. These instruments " Inside the Classroom Observation and Analytic Protocol and the Reform Teaching Observation Protocol " evidence not only similarities but marked differences in design, focus, documentation and observer skill requirements. The authors offer recommendations for potential users to enhance usefulness and avoid pitfalls.
Abstract: The Math and Science Partnership Management Information System (MSP MIS) presents annual findings from the MSP MIS for Comprehensive, Targeted, and Institute MSP projects for 2002-03 through the 2010-11 school years.
Key Findings include:
- What organizations were involved in the MSP program?
- What were the contributions of the individuals involved in the design and delivery of MSP activities?
- What MSP activities were targeted to IHE recipients?
- What MSP activities were targeted to K-12 recipients?
- What challenges did MSP projects face?
Abstract: "Acting on the assumption that improved teacher knowledge will yield gains in student achievement, scholars and policy-makers have focused increasing attention and resources on improving teachers' mathematical knowledge for teaching. Content-focused professional development, mathematically-supportive curriculum materials, and redesigned pre-service preparation programs are all examples of this effort. However, few studies have empirically demonstrated that teachers' mathematical knowledge is related to student achievement, especially at the elementary level. Further, existing studies have neglected to explore key questions about how this relationship is constituted.
Using data from students, teachers, and schools participating in a large study of comprehensive school reform, and using novel measures that capture both common and specialized mathematical knowledge for teaching, we explore the degree to which teachers' mathematical knowledge contributes to gains in student achievement. We find a positive effect of teacher mathematical knowledge on first and third graders' gain scores. We investigate the linearity of this relationship, discuss other findings from our models, and suggest implications for policy, professional development, and further research."
Abstract: "This paper describes the analysis of the National Science Foundation
(NSF) Math and Science Partnership (MSP) program's January 2008
Learning Network Conference (LNC). This study focuses on the
methodologies used by the MSP community to generate evidence and
seeks to understand topics of interest to the 320 LNC participants.
The data set for this study included the 47 abstracts accepted for
presentation, 68 interviews conducted during the conference,
observations of all 26 breakout sessions, and 98 "think pieces"
written by conference attendees. The analytic procedures included a
holistic scoring rubric for the abstracts and inductive analyses of
the interview, observation, and think-piece data using a structured
approach to grounded theory. Findings included enthusiasm for the
conference theme, respondent focus on realistic and field-tested ways
to generate evidence instead of theory and implementation reports,
and a strong assumption that student learning outcomes are the type
of outcome data of primary interest to the NSF. The study also
identified factors that influence the MSP community's approach to
evaluation. Overall, the study is framed by observed patterns in how
principal investigators (PIs) and their teams responded to project
evaluation requirements. Some PIs experienced a dilemma as to whether
their dominant operational approach should be discovery--as for
science, technology, engineering, and mathematics (STEM) research
projects--or delivery of pre-specified outcomes. Other PIs and
project leaders were slow to start the evaluation and were impressed
by the complexity of producing sound evaluation findings."
Abstract: "The Milwaukee Mathematics Partnership (MMP) is a collaboration among the University of Wisconsin-Milwaukee, Milwaukee Public Schools, and Milwaukee Area Technical College. A major component of MMP has been the development of a cadre of approximately 130 elementary and middle grades teacher leaders committed to leading improvements of their schools' mathematics programs while maintaining full-time teaching responsibilities. For each of five years, MMP provided a series of monthly professional development sessions targeting a selected mathematics content strand (e.g., algebra and algebraic reasoning, measurement and geometric reasoning). Through activities designed and facilitated by teams of university mathematics faculty, mathematics educators, and teachers-in-residence, the teacher leaders considered mathematics content in terms of "big ideas." For example, the algebra strand included five big ideas: equivalence, variable, patterns, linearity, and properties. Mathematics content knowledge was addressed from four perspectives to situate it within various aspects of the work of teacher leaders: (1) mathematical knowledge held by oneself, (2) mathematical knowledge held by other teachers, (3) mathematical knowledge held by students, and (4) the mathematical knowledge given emphasis in curriculum and assessments, specifically the Wisconsin Model Academic Standards and the Wisconsin Assessment Framework."
Related Case Studies:
- Introduction
- The Indiana Mathematics Initiative Partnership
- The Southwest Pennsylvania Math Science Partnership
- The Arizona Teachers Institute
- The Michigan Teaching Excellence Program
- The Life Sciences for a Global Community Teacher Institute
- The Greater Birmingham Mathematics Partnership
- The North Cascades and Olympic Science Partnership
- The Oregon Mathematics Leadership Institute
Abstract: The Milwaukee Mathematics Partnership (MMP), a comprehensive MSP project, resulted in significant improvement in mathematics achievement for students in the Milwaukee Public Schools, with a substantial achievement gap reduction between the school district and the state. The University of Wisconsin-Milwaukee, Milwaukee Public Schools, and Milwaukee Area Technical College shared in the leadership for this student success as core partners to this unique collaboration among a large urban district, a four-year urban university, and a two-year technical college. The MMP final report (NSF Grant No. 0314898) includes a summary of project goals, activities, and findings. Also included are appendices on the MMP Continuum of Professional Work for Mathematics and an overview of the MMP Toolkit.
Abstract: "Recent policy recommendations involving the putative primacy of randomized clinical trials in educational settings have reignited research paradigm debates. The authors of this article use the vehicle of strengthening structured journal abstracts to point out the argumentative character of all education research claims. They offer suggestions to authors to help inform academic and policy-oriented consumers of research articles about the various logical and evidentiary limitations that temper research-based claims, whether they emanate from quantitative or qualitative methods."
Visit Related Discussion: Creating Claim-Based Structured Abstracts for Education Research
Visit Related Discussion: Creating Claim-Based Structured Abstracts for Education Research
Abstract: "The Michigan Teaching Excellence Program (MiTEP) is a partnership between Michigan Technological University, Grand Valley State University, Grand Rapids Area Pre-College Engineering Program, and the Grand Rapids Public School District, targeting 15 teachers of middle grades earth sciences in its first year. Teachers engage in two weeks of summer field studies with supporting lectures; the primary goals are to deepen teacher disciplinary content knowledge beyond the level expected of their students, and to enhance teacher understanding of the nature of scientific inquiry. To facilitate teacher application of their enhanced content knowledge in their classrooms, the project selected experiences that relate to both the state's content expectations for earth science education and to geology features encountered locally by the district's students. Project team members are expected to model appropriate pedagogical practices throughout the professional development, including requiring learners to provide evidence to support their claims, and encouraging them to reflect on their emerging understanding of earth science concepts in daily journals. In addition, one day during the first week and two days during the second week are designated as pedagogy days, where teachers adapt the learned content to district adopted pedagogical norms and models, including the Disciplinary Literacy framework developed by the Institute for Learning at the University of Pittsburgh. Finally, in response to participants' requests, project staff are developing classroom activities similar to those the teachers experienced, but at a level appropriate for middle grades students."
Related Case Studies:
- Introduction
- The Indiana Mathematics Initiative Partnership
- The Milwaukee Mathematics Partnership
- The Southwest Pennsylvania Math Science Partnership
- The Arizona Teachers Institute
- The Life Sciences for a Global Community Teacher Institute
- The Greater Birmingham Mathematics Partnership
- The North Cascades and Olympic Science Partnership
- The Oregon Mathematics Leadership Institute
Abstract: "In hopes of reducing the percent of students receiving non-passing grades in elementary statistics classes, interactive hands-on projects were implemented in the classes. These projects gave students the opportunity to use personal data, discuss statistical concepts with each other, and strengthen the students' understanding, mastery and appreciation of the material covered in an elementary statistics class. Although no decrease in the percent of non-passing grades was observed and no noticeable shift in the overall grades occurred, there was a statistically significant increase in the students' perception of the teacher's ability to teach the course and the students' perception of the teacher's interest in the success of the students. Through further investigation, partial justification for the lack of significant change in non-passing grades was attributed to circumstances in students' personal lives that are beyond the control of the teacher and would not be affected by any change in teaching style."
Abstract: "The Oregon Mathematics Leadership Institute (OMLI) project served 180 Oregon teachers, and 90 administrators, across the K-12 grades from ten partner districts. OMLI offered a residential, three-week summer institute. Over the course of three consecutive summers, teachers were immersed in a total of six mathematics content classes--Algebra, Data & Chance, Discrete Mathematics, Geometry, Measurement & Change, and Number & Operations--along with an annual collegial leadership course. Each content class was designed and taught by a team of expert faculty from universities, community colleges, and K-12 districts. Each team chose a few "big ideas" on which to focus the course. For example, the Algebra team focused on algebraic structure and properties of the concept of a group, while the Data & Chance team centered their activities on the exploration of ideas of central tendency and variation using statistics and data analysis software packages. The content in all of the courses was addressed through deep investigation of the mathematics of tasks that had been selected and adapted from resources for K-12 mathematics classrooms. In addition to mathematics content, the courses were designed with specific attention to socio-mathematical norms, issues of status differences among learners, and the selection and implementation of group-worthy tasks for group work. The faculty attended sessions grounded in the work of Elizabeth Cohen on strategies for working with heterogeneous groups of learners (Cohen, 1994; Cohen et al, 1999) which was central to the OMLI design and implementation. Institute faculty modeled these strategies in the Institute classrooms and made their moves as transparent as possible, so that the teachers would be able to grapple with these strategies during the Institute and plan for implementation in their own classrooms. The Data & Chance course also modeled uses of technology in instruction using Tinkerplots. Generalization and justification were emphasized as mathematical ways of learning and knowing, and institute faculty conducted classroom discussions that intentionally modeled pushing for generalization and justification."
Related Case Studies:
- Introduction
- The Indiana Mathematics Initiative Partnership
- The Milwaukee Mathematics Partnership
- The Southwest Pennsylvania Math Science Partnership
- The Arizona Teachers Institute
- The Michigan Teaching Excellence Program
- The Life Sciences for a Global Community Teacher Institute
- The Greater Birmingham Mathematics Partnership
- The North Cascades and Olympic Science Partnership
Abstract: "The National Research Council and the National Science Resources Center have been awarded a three-year MSP/RETA grant for Facilitating Mathematics and Science Partnerships. The primary objective of this project is to provide a series of workshops that will assist ... in improving K-16 STEM education programs through the MSP initiative.
...
As part of the NRC's commitment to develop a summative evaluation, in addition to a formative evaluation for this project, our RETA project is collaborating with a University System of Maryland's (USM) MSP project that is looking broadly at how the knowledge and understandings gleaned from the MSP projects become embedded in the culture of higher education. The USM's CASHE (Change and Sustainability in Higher Education) project is examining institutional change in higher education that has come about as a result of the MSP national effort (see description below). Current plans for conducting this summative evaluation are detailed below."
...
As part of the NRC's commitment to develop a summative evaluation, in addition to a formative evaluation for this project, our RETA project is collaborating with a University System of Maryland's (USM) MSP project that is looking broadly at how the knowledge and understandings gleaned from the MSP projects become embedded in the culture of higher education. The USM's CASHE (Change and Sustainability in Higher Education) project is examining institutional change in higher education that has come about as a result of the MSP national effort (see description below). Current plans for conducting this summative evaluation are detailed below."
Abstract: The Alliance for the Improvement of Mathematics Skills -- PreK-16 (AIMS), funded fall 2002 by NSF, has four goals, one of which focuses on teacher professional development and its affects on student achievement. The partnership consists of two institutions of higher education and nine independent school districts (ISDs).
Results from the first four years of focused professional development offered to the mathematics teachers of the partners have been examined. Offerings were based on the Texas Teachers Empowered for Achievement in Mathematics and Science (TEXTEAMS) and the Texas Essential Knowledge and Skills (TEKS). In addition to the workshops, the AIMS Mathematics Specialists have provided coaching, mentoring, lesson modeling, curriculum alignment, lesson planning, teaching strategies, and learning styles. Grade specific sessions have been offered to improve workshop application and adaptations.
Preliminary data indicates increases in the Texas Assessment of Knowledge and Skills (TAKS) percentage passing as well as in actual numbers of students passing at the met standard percentage. Passing percentages from area districts and the region have been examined to determine the effects of the focus professional development. The panel recommendation passing percentages were collected to provide a common passing score basis for all years.
Comparisons of Year One and Year Four data indicates all partner districts increased in passing percentages ranging from 8% to 24% for the sum of all grades tested. Only two districts had decreases when examining individual grade level percentages. Individual details will be provided in tables. Each district's data allows the cohorts of students to be followed as they move from grade to grade. The overall trend appears to be increases in the numbers passing, the percent passing, or in both. Enrollment in Algebra I, Geometry, Algebra II, and all pre-calculus courses (pre-calculus, statistics, etc.) has increased. Because of the amount of data, details and data tables will be provided in the full paper.
- The focus:
- Did our research design produce student achievement results above those expected from no additional activity?
- Did a focus on TEXTEAMS with teacher follow-up produce higher mathematics achievement?
- Does the data support these assumptions?
Results from the first four years of focused professional development offered to the mathematics teachers of the partners have been examined. Offerings were based on the Texas Teachers Empowered for Achievement in Mathematics and Science (TEXTEAMS) and the Texas Essential Knowledge and Skills (TEKS). In addition to the workshops, the AIMS Mathematics Specialists have provided coaching, mentoring, lesson modeling, curriculum alignment, lesson planning, teaching strategies, and learning styles. Grade specific sessions have been offered to improve workshop application and adaptations.
Preliminary data indicates increases in the Texas Assessment of Knowledge and Skills (TAKS) percentage passing as well as in actual numbers of students passing at the met standard percentage. Passing percentages from area districts and the region have been examined to determine the effects of the focus professional development. The panel recommendation passing percentages were collected to provide a common passing score basis for all years.
Comparisons of Year One and Year Four data indicates all partner districts increased in passing percentages ranging from 8% to 24% for the sum of all grades tested. Only two districts had decreases when examining individual grade level percentages. Individual details will be provided in tables. Each district's data allows the cohorts of students to be followed as they move from grade to grade. The overall trend appears to be increases in the numbers passing, the percent passing, or in both. Enrollment in Algebra I, Geometry, Algebra II, and all pre-calculus courses (pre-calculus, statistics, etc.) has increased. Because of the amount of data, details and data tables will be provided in the full paper.
Abstract: "The North Cascades and Olympic Science Partnership (NCOSP) project served teacher leaders of grades 3-10 and disciplinary science faculty. NCOSP created and conducted a sequential series of three 80-hour residential Summer Academies. A central feature of the academies is the use of content immersions to help participants develop a more scientifically-accurate understanding of relevant big ideas in science. For example, the Life Sciences Academy included a focus on the function of food for animals, and the Earth Science Academy contained several chapters about plate tectonics. In addition to deepening teacher understanding of fundamental scientific content ideas, NCOSP devoted substantial time to pedagogical knowledge goals based on the application of principles from How People Learn (National Research Council, 1999) to science education. Facilitators modeled instructional strategies as they addressed scientific content goals, and some Academy assignments directly addressed the pedagogical knowledge goals. For example, participants revised selected lessons using resource materials that provided information about scientific content, instructional strategies, and common student misconceptions. Attention to scientific content, learning theory, and instructional strategies was cyclic, and work on instructional strategies incorporated scientific knowledge acquisition. The roles of evidence and questioning as scientific ways of building knowledge were modeled through the professional development activities."
Related Case Studies:
- Introduction
- The Indiana Mathematics Initiative Partnership
- The Milwaukee Mathematics Partnership
- The Southwest Pennsylvania Math Science Partnership
- The Arizona Teachers Institute
- The Michigan Teaching Excellence Program
- The Life Sciences for a Global Community Teacher Institute
- The Greater Birmingham Mathematics Partnership
- The Oregon Mathematics Leadership Institute
Abstract: The purpose of this article is to present the findings of a quantitative study focused on primary mathematics teachers who participated in an intensive professional development program and then had leadership responsibility for the implementation of a new primary mathematics curriculum in their district. The study examines the effect of the professional development on teachers' and students' mathematical knowledge. Results indicate that effective implementation of a new curriculum can improve students' mathematical knowledge, but that the effect of the curriculum implementation on student mathematical knowledge is enhanced when teachers improve their mathematical knowledge for teaching and are simultaneously involved in the leadership activities of the curriculum implementation.
Abstract: "There's a ton of data being collected. The trick is to know how to use it effectively."
"Schools are gathering more and more data, but having data available does not mean the data are used to guide instructional improvement. Many schools lack the process to connect the data they have with the results they must produce. The Using Data Project focuses on developing professional developers, administrators, and teachers who can lead a collaborative inquiry process and strengthen the collaborative culture of their schools or departments. The aim is to influence school culture to be one in which educators use data continuously, collaboratively, and effectively to improve teaching and learning mathematics and science."
Abstract: "The Arizona Teaching Institute (ATI) is a partnership between the University of Arizona and Tucson Unified School District. ATI has developed and implemented a Master's degree in Middle School Mathematics Leadership for middle school teachers, a Certificate in Mathematics Teacher Mentoring for secondary-certified teachers, and a postdoctoral fellowship in Teacher Preparation for recent mathematics Ph.D. graduates. The Master's degree in Middle School Mathematics Leadership is designed as a three-year, part-time degree consisting of two years of mathematics and education coursework, followed by a year of fieldwork. Courses are offered during the summer and part-time during the academic year. For example, the Numbers and Number Sense course described here addressed algorithms for operations; properties of arithmetic operations; and meanings for integers, rational numbers, and real numbers with a special emphasis on fractions. Instructors deliberately incorporated review of elementary and middle-school level mathematics concepts into explorations of more advanced topics. The course attended to pedagogical content knowledge through discussions about common misconceptions and strategies for presenting difficult material, interwoven with discussions of the mathematics disciplinary content. Although anticipated and encouraged by facilitators, these discussions were often participant-initiated, rather than planned. The sequence of mathematics courses addresses logic and proof as mathematical ways of knowing, and those learning goals are primarily dealt with in the Geometry and Algebra courses. In addition, the Numbers and Number Sense course modeled the process of making distinctions among related mathematical constructs as a way of knowing and understanding mathematics."
Related Case Studies:
- Introduction
- The Indiana Mathematics Initiative Partnership
- The Milwaukee Mathematics Partnership
- The Southwest Pennsylvania Math Science Partnership
- The Michigan Teaching Excellence Program
- The Life Sciences for a Global Community Teacher Institute
- The Greater Birmingham Mathematics Partnership
- The North Cascades and Olympic Science Partnership
- The Oregon Mathematics Leadership Institute
Abstract: "Drawing on case studies of schools participating in a school-university partnership to improve math and science teaching, this paper explores the configurations of distributed leadership in four schools in three districts. To examine how leadership is distributed, we examine how a set of change-related leadership tasks are accomplished, and in particular what contributions are made by district leaders, principals, formally identified teacher leaders and other teachers. This analysis suggested that principals and district office staff were more active in performing leadership tasks than teachers and teacher leaders. However, the vision or approach to improve mathematics was not always share among leaders. We describe patterns of leadership distribution looking into these two dimensions: visions and tasks. The configuration of leadership showed a typology of alignment."
Abstract: In this article we describe how an underperforming school district used research and theory on curriculum, assessment, implementation, and school and classroom organization to develop and implement district standards and improve the achievement of elementary school students. Key reforms included teachers developing essential curriculum standards, standards-based criterion-referenced tests, and standards-based extended learning opportunities. Teachers rated the reform efforts positively and reported a high likelihood of implementation. Using California Department of Education data, we employed econometric analyses to estimate program effects for the district's elementary schools from 1999 to 2002. A difference-of-differences model estimated 10 of 16 positive and significant effects on grades 2 to 5 SAT9 scaled score mathematics achievement, ranging from .2σ to .7σ. The same model applied to reading scores showed 12 of 16 insignificant or negative grade-level effects. As predicted, however, second and third graders in 2002--those students who had 3 and 4 years of exposure to the program exclusively--experienced only positive (4 of 4) and mostly significant (3 of 4) reading and mathematics effects. Results may guide district administrators implementing standards or comprehensive school reforms.
Abstract: "The Greater Birmingham Mathematics Partnership (GBMP) is a collaborative effort involving two institutions of higher education, University of Alabama at Birmingham and Birmingham- Southern College; one private non-profit organization, the Mathematics Education Collaborative (MEC); and nine Greater Birmingham, Alabama school districts. The project's work included professional development for in-service teachers; teacher leader development; revised coursework for pre-service teachers, including a new track for mathematics majors and accompanying new certification for middle school mathematics teaching; and community outreach. GBMP conducted seven mathematics content courses, offered as two-week summer immersion experiences to teachers in the partner districts. Additional professional development for teacher-leaders was provided during the school year. Patterns: The Foundations for Algebraic Reasoning was required as a prerequisite to all other courses. In it, mathematics teachers explored patterns in an inquiry-based environment using a variety of representations including graphs, tables, geometric models, algebraic expressions, and verbal contexts. Course participants were K-20 teachers of mathematics who learned mathematics together, which allowed them to see how to meet a range of learning needs and to see how the foundation of algebraic reasoning is laid throughout the grades. They also discussed articles chosen to help participants better understand the theoretical underpinnings of constructivist theory and the implications for classroom practice. Throughout all of the courses, participants engaged in generalization and justification as mathematical ways of learning and knowing. Over the course of the five-year project several local teacher-leaders and higher education faculty associated with the GBMP completed an internship process to become MEC instructors, and they now teach the courses. In the final year of the project, school-based professional learning communities were initiated to support teachers in implementing inquiry-based instruction and performance assessments."
Related Case Studies:
- Introduction
- The Indiana Mathematics Initiative Partnership
- The Milwaukee Mathematics Partnership
- The Southwest Pennsylvania Math Science Partnership
- The Arizona Teachers Institute
- The Michigan Teaching Excellence Program
- The Life Sciences for a Global Community Teacher Institute
- The North Cascades and Olympic Science Partnership
- The Oregon Mathematics Leadership Institute
Abstract: "This paper contrasts performances overall and by gender, ethnicity, and socioeconomic status (SES) for middle school students learning science through traditional scripted inquiry versus a design-based, systems approach. Students designed and built electrical alarm systems to learn electricity concepts over a fourweek period using authentic engineering design methods. The contrast study took place in the eighth grade of an urban, public school district, with the systems approach implemented in 26 science classes (10 teachers and 587 students) and the scripted inquiry approach implemented in inquiry groups of 20 science classes (five teachers and 466 students). The results suggest that a systems design approach for teaching science concepts has superior performance in terms of knowledge gain achievements in core science concepts, engagement, and retention when compared to a scripted inquiry approach. The systems design approach was most helpful to low-achieving African American students."
Abstract: "The Math in the Middle Institute Partnership (M2) is designed to improve K-12 student mathematics achievement by creating sustainable partnerships among individuals at each of the following entities: the University of Nebraska-Lincoln, regional Educational Service Units (ESUs), and local school districts. These partnerships are designed to educate and support teams of outstanding middle-level mathematics teachers (Grades 5-8) who will become intellectual leaders in their schools, districts, and ESUs. M2 is particularly committed to improving the capacity of rural teachers, schools, and districts. The design of M2 is grounded in evidencebased research on learning, teaching, and teacher professional development. M2 involves multiple cohorts of teachers, each of which consists of approximately 30 lead teachers.
...preliminary analyses suggest positive effects of participation in M2 on a range of teacher outcomes, including ratings of preparedness and confidence, philosophy of mathematics teaching and learning, instructional emphasis, use of assessment, and professional interaction among mathematics teachers. A handful of differences in outcomes were found for teachers from Lincoln Public Schools and those with different levels of undergraduate mathematics coursework and teaching experience. Analysis of student mathematics achievement data indicated both positive and negative effects of teacher participation in M2, which were small after controlling for a range of student characteristics. Examination of teacher-level factors that contributed to student achievement outcomes showed mixed effects."
...preliminary analyses suggest positive effects of participation in M2 on a range of teacher outcomes, including ratings of preparedness and confidence, philosophy of mathematics teaching and learning, instructional emphasis, use of assessment, and professional interaction among mathematics teachers. A handful of differences in outcomes were found for teachers from Lincoln Public Schools and those with different levels of undergraduate mathematics coursework and teaching experience. Analysis of student mathematics achievement data indicated both positive and negative effects of teacher participation in M2, which were small after controlling for a range of student characteristics. Examination of teacher-level factors that contributed to student achievement outcomes showed mixed effects."
Abstract: "Over the past five years I have sought to better understand student thinking and learning in the context of topics central to trigonometry, including angle measure, the unit circle, trigonometric functions, periodicity, and the polar coordinate system. While each study has provided unique insights into students' learning of trigonometry, a common theme connects the studies' findings: quantitative reasoning plays a central role in students' trigonometric understandings. In this chapter, I first describe a coherent system of understandings for trigonometry that is grounded in quantitative reasoning. Against this backdrop, I compare students' quantitative reasoning in the context of trigonometry in order to illustrate the role of quantitative reasoning in the learning of a particular mathematical topic."
Abstract: "The maker movements, a general term for the rise of inventing, designing, and tinkering, and the addition of engineering standards to the Next Generation Science Standards (NGSS) have spawned a major evolution in technology classes throughout the country. At Georgia Institute of Technology, a new curriculum attempts to bring the maker movement to high school audiences through both curricular and extra-curricular channels. The curriculum is structured around engineering standards and learning goals that reflect design and advanced manufacturing content, along with employability skills, while borrowing best practices from 'wood shop' and 'technology education' classes. The hope is that this course will bolster many of the 'Attributes of Engineers in 2020' described by the National Academy of Engineering and 21st Century Skills--these skills and attributes can be beneficial to any college or career path, not just one in engineering. The course incorporates design-build activities into entrepreneurial and business contexts, providing relevance to foundational math skills and science practices while integrating problem solving and cutting-edge technology. The course requires that students draw and render design concepts, communicate design concepts to their peers and clients, fabricate design artifacts, and document their requirements and decisions while engaging in the engineering design process.
The purpose of this paper is to explore the results from the first and second year implementation of a maker-infused Advanced Manufacturing (AM) course for high school students in a low income, rural-fringe school system. Results from a portfolio assessment and 21st Century Skills surveys will be discussed in terms of course effectiveness and challenges to implementation. Similarities and differences between learning goals for this new AM course and the more traditional wood shop and technology education classes will be highlighted. Implications for engineering education, theory, and practice are discussed."
Abstract: In 2003, the Allegheny Intermediate Unit received a grant under the National Science Foundation's Math and Science Partnership program to establish the Math Science Partnership of Southwest Pennsylvania (MSP). The MSP brings together regional K-12 school districts, institutions of higher education, and intermediate units with the goals of increasing K-12 students' knowledge of mathematics and science, improving the quality of the K-16 educator workforce, and creating sustainable partnerships among participants. To achieve these goals, the MSP has instituted a variety of activities, including leadership academies, fellowship programs, and networking opportunities, to enact three key intervention strategies: professional development for content and leadership, curriculum alignment and pedagogical and course refinement, and support for and dissemination of research-based resources and tools. As RAND's involvement with the MSP comes to an end (the MSP will continue through 2010), the findings presented here provide preliminary evidence regarding the MSP's progress. The analyses draw on five years of qualitative and quantitative MSP data, including interviews, surveys, participation records, student achievement assessments, and other sources. The findings suggest that MSP partners are making progress toward the three MSP goals, although statistical analysis did not find evidence that the MSP was directly responsible for this progress.
Abstract: "Designers of professional development for mathematics and science teachers are faced with many decisions related to the content they choose to address, the selection of strategies to reach professional development goals, and how they facilitate transfer of teachers' new knowledge to their classroom practice. Many of the Math Science Partnership (MSP) projects funded by the National Science Foundation (NSF) include professional development activities to deepen mathematics/science teacher knowledge. The MSP Knowledge Management and Dissemination project, in collaboration with staff from nine MSP projects, created this volume of "stories" of professional development, along with an introduction describing different facets of teacher knowledge, to illustrate some of the choices MSP projects have made and how their decisions played out over time. It is our hope that designers of future professional development endeavors will find these stories, and the authors' lessons learned, useful when considering their design options."
Related Case Studies:
- The Indiana Mathematics Initiative Partnership
- The Milwaukee Mathematics Partnership
- The Southwest Pennsylvania Math Science Partnership
- The Arizona Teachers Institute
- The Michigan Teaching Excellence Program
- The Life Sciences for a Global Community Teacher Institute
- The Greater Birmingham Mathematics Partnership
- The North Cascades and Olympic Science Partnership
- The Oregon Mathematics Leadership Institute
Abstract: "This brief provides guidance for how to design a research plan using DBIR. It can also serve as a resource for preparing a research proposal to a federal agency or foundation that employs a DBIR approach. This guidance is informed by the development of the approach in a range of settings, including projects of the Research+Practice Collaboratory. The ideas expressed in this guidance document are those of the Research and Practice Collaboratory project and are not necessarily those of any agency that funds research."
Abstract: "Primary school students participating in the HOS program (N=72) and their parents (N=43) were asked to complete questionnaires prior to and at the end of the program indicating attitudes about science. Parents responded about their child who participated in the program. Ten of 11 questions (5 of 6 for students, 5 or 5 for parents) showed shifts in a positive direction, and 5 of these shifts, all in the positive direction, were significant at the .10 level by one-tailed test. In addition to the results for specific items, these results reveal a statistically significant pattern of positive findings in the overall trends. These findings demonstrate that hands-on-science programs can have positive effects on young students' attitudes toward science."
Abstract: "[T]his paper reports on findings from the first three years of implementation within Montgomery County Public Schools (MCPS), specifically changes in teachers' instructional practices and attitudes, as well factors that have facilitated or hindered teacher change. Additionally, VIP has strongly promoted science as inquiry, the ability of students to use "scientific reasoning and critical thinking to develop their understanding of science," which is a reflection of national standards (National Science Teachers Association, 2004; National Research Council, 1996; American Association for the Advancement of Science, 1993). An emphasis on inquiry-based instruction has been infused throughout VIP-supported activities, and given its importance within VIP, as well as in standards-based educational reforms more generally, particular attention is paid to its implementation and differing conceptions of inquiry-based teaching within the VIP community.
Data are drawn primarily from two sources: 1) pre- and post-program surveys of VIP's Biology Cohort and 2) classroom observations of a sample of VIP's Matter and Energy/Earth Space Systems Cohort. Data collected through other means, such as teacher evaluations of VIP-sponsored conferences and summer institutes, a focus group of Biology Cohort teachers, and a survey of Master Science Teachers (MSTs) reinforce findings from the survey and observations. The discussion of inquiry and factors influencing teacher change has broad applicability to other projects embarking on educational reform."
Data are drawn primarily from two sources: 1) pre- and post-program surveys of VIP's Biology Cohort and 2) classroom observations of a sample of VIP's Matter and Energy/Earth Space Systems Cohort. Data collected through other means, such as teacher evaluations of VIP-sponsored conferences and summer institutes, a focus group of Biology Cohort teachers, and a survey of Master Science Teachers (MSTs) reinforce findings from the survey and observations. The discussion of inquiry and factors influencing teacher change has broad applicability to other projects embarking on educational reform."
Abstract: The Next Generation Science Standards now integrate science and engineering through their Core Ideas and Practices dimensions. Portions of the engineering design process included in these standards emphasize: 1) defining problems through identifying criteria and constraints, 2) developing solutions to those problems, and 3) optimizing those solutions to best fit the criteria and constraints. The Science Learning Integrating Design, Engineering, and Robotics (SLIDER) project, funded through the NSF DRK-12 program for five years, set out to investigate this integration through the use of robotics and design to develop conceptual understanding among 8th grade physical science students. Through three years of curriculum development and iteration, the SLIDER curriculum faced several challenges in making this integrated approach both effective and practical in a diverse array of schools. This paper presents that story and makes suggestions critical to others designing for the NGSS and developing theory around integrated STEM learning.
Abstract: "This paper introduces a variation on the post-test only cohort control design and addresses questions concerning both the methodological credibility and the practical utility of employing this design variation in evaluations of large-scale complex professional development programmes in mathematics education. The original design and design variation, which adds a pre-test measure and gain scores for both pre-treatment and post-treatment conditions, are compared theoretically for their respective controls on threats to internal validity, and practically for their comparative ease of implementation in field settings. We conclude that the design variation adds important controls to selection bias that the original design cannot. From a utility perspective, however, there are trade- offs, because this design variation requires more data collection observations, more complex data management and certain psychometric characteristics for the outcome measure."
Abstract: The Math and Science Partnership (MSP) Management Information System (MIS) is designed to obtain annual information from each MSP-funded project. This information can be used to describe the implementation and impact of the overall MSP program and to monitor the progress of individual MSP awards. This report summarizes the initial findings for 34 Cohort 1 and 2 MSP projects for the 2002-03 and 2003-04 school years and provides evidence that projects are laying the groundwork for significant changes in their participating educational institutions. Most notably, projects are making progress in establishing the kinds of partnerships envisioned by NSF. The unique feature of these partnerships, the involvement of disciplinary faculty in the reform efforts, is in place and growing. The number of teachers and students involved in participating K-12 schools is also increasing--and data suggest that projects are, in fact, addressing the needs of urban and rural students with significant needs.
Abstract: "Most work on professional learning opportunities in education focuses on classroom teachers and school principals. In this paper the authors take a broader look at school leaders' opportunities to learn from a distributed perspective. Using one midsized urban school district as their case, they examine the opportunities to learn (OTL) of school principals, administrators and teacher leaders, attending to both formal and on-the-job OTL. The analysis shows that an exclusive focus on the school principal substantially underestimates the school system's investment in school leader learning. The authors argue that efforts to understand school leader learning must move beyond an exclusive focus on the school principal to include the OTL of other formally designated school leaders - both administrators and teacher leaders. Further, they contend that analyzing school leaders' opportunities to learn must also focus on both formal professional development as well as on-the-job learning, and the relations between the two."
Abstract: "Over the last decade, researchers in The Distributed Leadership Studies (DLS) at Northwestern University have been developing a framework for examining school leadership and management with an emphasis on their relations to classroom instruction (http://www.distributedleadership.org). Drawing on theoretical and empirical work in distributed cognition and socio-cultural activity theory, our distributed perspective involves two aspects principal plus and practice (Spillane 2006; Spillane, Halverson, and Diamond 2001, 2004). The principal plus aspect acknowledges that the work of leading and managing schools involves multiple individuals. The practice aspect foregrounds the practice of leading and managing, framing this practice as emerging from the interactions among school leaders and followers, mediated by the situation in which the work occurs. Practice is more about interaction than action. The school subject matter mathematics, science and language arts has figured prominently in our efforts to build knowledge about and for the practice of leading and managing.
In this chapter, I use our hypotheses-generating research and development work as part of the Distributed Leadership Studies (DLS) as an example of connecting research with practice and policy. I begin by briefly describing our research and development work on school practice and give attention to our various goals. I then describe some of the ways in which the DLS have forged connections with policy-makers and practitioners
through three different partnering experiences. I next consider, in more detail, one facet of our work involving the use of our research findings to engage policymakers and practitioners in diagnosing and design work so as to develop practical knowledge how knowledge as distinct from what knowledge. I conclude by reflecting on some of the challenges the DLS has encountered in engaging partners in policy and practice."
Abstract: One issue that will influence how teachers adopt NGSS-aligned standards is teachers'
preexisting beliefs and attitudes about what constitute effective methods for science teaching and learning (Banilower, Trygstad, and Smith, 2015; Trygstad, Smith, Banilower, and Nelson, 2013). The Framework and NGSS describe a vision for science education in which students will primarily learn science concepts by engaging in SEPs. Students are to generate and interpret evidence and develop explanations through sustained investigations, all while increasing their
capacity to direct all aspects of the process over time (National Research Council, 2012). This contrasts with the current state of science education in many classrooms, in which students primarily learn concepts through direct instruction with occasional reinforcement through engagement in SEPs (Banilower et al., 2015). According to results from the 2012 National Survey of Mathematics and Science Education (Banilower et al., 2013), around 60% of teachers believe that hands-on experimentation should reinforce concepts students have already learned, 40-50% of teachers believe that they should explain a concept to students before the students consider evidence related to the concept, and 90% of teachers believe that
vocabulary should come before conceptual understanding. Interventions to support teachers in adopting NGSS-aligned standards will need to take into account that many educators' beliefs may not align with the notion of consistently teaching science content through SEPs, as envisioned by authors of the Framework and NGSS.
preexisting beliefs and attitudes about what constitute effective methods for science teaching and learning (Banilower, Trygstad, and Smith, 2015; Trygstad, Smith, Banilower, and Nelson, 2013). The Framework and NGSS describe a vision for science education in which students will primarily learn science concepts by engaging in SEPs. Students are to generate and interpret evidence and develop explanations through sustained investigations, all while increasing their
capacity to direct all aspects of the process over time (National Research Council, 2012). This contrasts with the current state of science education in many classrooms, in which students primarily learn concepts through direct instruction with occasional reinforcement through engagement in SEPs (Banilower et al., 2015). According to results from the 2012 National Survey of Mathematics and Science Education (Banilower et al., 2013), around 60% of teachers believe that hands-on experimentation should reinforce concepts students have already learned, 40-50% of teachers believe that they should explain a concept to students before the students consider evidence related to the concept, and 90% of teachers believe that
vocabulary should come before conceptual understanding. Interventions to support teachers in adopting NGSS-aligned standards will need to take into account that many educators' beliefs may not align with the notion of consistently teaching science content through SEPs, as envisioned by authors of the Framework and NGSS.
Abstract: "There is, then, a very important paradox to be understood here: The goal of investments in educational capacity building is always to increase student learning, but the effectiveness of the investment can not be assessed by measures of student achievement. The failure to understand this reality is not only a technical flaw in the system but actually creates a major obstacle to all existing efforts to improve the functioning of the educational system....we are seeking a rigorous way to evaluate appropriate outcomes. We are looking for a scientific way to assess the degree to which and the ways in which educational investments are, in fact, leading to increased system capacities for continued program improvement."
Abstract: This document contains four brief reports that illuminate key strategies employed in the Appalachian Math Science Partnership (AMSP). They are intended to be read along with the main report, The Appalachian Math Science Partnership: A Multi-State Umbrella Partnership Promoting Local Mathematics And Science Reform.
The reports included in this document are:
I. The Regional Program Coordinators: Making Connections and Developing Local Leadership
II. Baseline Improvement Sites and the Program Improvement Review: Promoting School-wide Involvement in Math and Science Reform
III. The Partnership Enhancement Program: A Strategy for Supporting Locally Designed Partnerships
IV. Motivating Change in Institutions of Higher Education Through Collaboration with K-12 Partners
Abstract: The Appalachian Math Science Partnership (AMSP) is a project within NSF's Math Science Partnerships (MSP) initiative. Funded at $23 million over five years, the AMSP involved 51 school districts and nine higher education institutions in Kentucky, Tennessee and Virginia.
The AMSP faced two significant challenges in its effort to live up to the vision of the MSP initiative and to its own hopes and goals. First, its service area, Appalachia, comprises some of the most isolated and stressed communities and school systems in the United States. Second, the mere scale of the partnership--60 organizations in several states--defied easy implementation. Yet despite these challenges (indeed, working to address them directly), the AMSP built upon existing leadership capacity in the region and created not only effective partnerships, but ultimately formed what we view as the foundation for a sustainable regional "improvement infrastructure" for science and mathematics.
In this set of five papers, we portray the evolution, design and strategies of the AMSP. We believe that the design principles the AMSP adhered to, the responsiveness of AMSP leaders to local needs and issues, and the strategies they devised to make good on their promise have relevance for others who invest in the improvement of science and math education, particularly in rural regions.
This paper, entitled "The Appalachian Math Science Partnership: A Multi-State Umbrella Partnership Promoting Local Mathematics And Science Reform," is the core document of the set of five and stands alone. It provides background on the initiative and on the regional landscape, gives an account of how the AMSP's overall design as an "umbrella partnership" formed, describes the major strategies and components of the partnership (including the benefits they produced and lessons learned from their implementation), documents the core values and design principles underlying the umbrella partnership, and offers our conceptualization of the AMSP as a regional improvement infrastructure.
The four papers identified as AMSP Close-ups (a single document) are companions to this core document. They illuminate in more detail key strategies of the larger partnership.
Abstract: "In this article, we report on work thus far (Years One and Two) of the Southwest
Pennsylvania Math Science Partnership in using evaluation as a "bridge" to support
and encourage the development of collaborative partnerships. We begin with a
description of the SW PA MSP followed by its "theory of action" for intervention
among and with stakeholders. We explore the overall evaluation plan that is built from this logic model (theory of action) and discuss examples of the role evaluation has
served for the project in partnership development. We then consider a number of
factors we have found to be of issue in the key school-level (K-12) / higher education
(IHE) partnerships the Math Science Partnership is designed to foster."
Abstract: Members of Mathematical ACTS MSP project presented on project activities and findings at the MSP Evaluation Summit in October 2006.
Abstract: "...Part of the evaluation of the Milwaukee Mathematics Partnership (MMP) focused on trying to capture some of the school level variability in variables that are not under the control of the MMP, with the thought that any increase in student achievement in mathematics is ultimately affected by differences in how the model of professional development is being implemented in each of the respective schools. The primary purpose of this paper is to determine whether reported differences in these variables are related to differences in student achievement gains in mathematics. For example, is there a greater increase in student achievement in mathematics in schools that report discussing mathematics more often or for schools with a collaborative and supportive LT? A secondary purpose of this paper is to compare and contrast the two aforementioned accountability systems, status based and value-added, in terms of assessing the effectiveness of a large-scale educational reform such as the Milwaukee Mathematics Partnership (MMP). These two methods will be compared in terms of outcomes and viability of application. Finally issues specific to the MMP that have been encountered in attempting to implement a system of accountability will be addressed. These issues are vitally important because they threaten the validity of any conclusions that may be drawn in terms of the impact of the MMP on increasing student achievement in mathematics."
Abstract: "Most quantitative evaluation endeavors undertaken to assess the effectiveness of large
scale educational reform efforts must rely on self-report data to some extent. For example, the
analytical framework used to assess the impact of the Milwaukee Mathematics Partnership
(MMP) on increasing student achievement in mathematics is largely dependent on data obtained
from an online survey administered to all Milwaukee Public School District (MPS) employees at
the K-8 level who have the potential to positively affect student achievement in mathematics.
This survey was designed to measure various aspects of educators’ daily responsibilities that
would likely be affected by MMP activities and in turn, affect student achievement in
mathematics. However, the validity of self-report data is always suspect due to the possibility of
response bias, which occurs if participants respond to items in a more socially appealing manner.
In the current survey, because respondents are primarily reporting on the quality of their own
work and that of their colleagues it is quite possible that the validity of our data is compromised
by response bias. Moreover, if this is true then the results of any statistical tests conducted on
these data may be biased because these results are wholly dependent on how well the variables
have been measured.
Therefore, this year the evaluation team of the MMP made a concerted effort to assess the validity of our self-report data. Two approaches were taken to attain this goal: (1) A global approach that utilized all schools in the district that participated in the quantitative evaluation efforts; and (2) A case-study approach that only made use of schools that participated in the qualitative evaluation efforts. For the global approach, overall indicators of the quality and quantity of a school’s participation in MMP related activities obtained from the self-report survey data were compared to ratings obtained from the district level Mathematics Teaching Specialists (MTS) assigned to that school. For the case-study approach, an attempt was made to triangulate the self-report survey data obtained with other external criteria in a subset of schools targeted to participate in more in-depth intensive evaluation efforts. The primary purpose of this paper is to describe the methodology utilized and report the findings for both of these approaches to assess the validity of our data."
Therefore, this year the evaluation team of the MMP made a concerted effort to assess the validity of our self-report data. Two approaches were taken to attain this goal: (1) A global approach that utilized all schools in the district that participated in the quantitative evaluation efforts; and (2) A case-study approach that only made use of schools that participated in the qualitative evaluation efforts. For the global approach, overall indicators of the quality and quantity of a school’s participation in MMP related activities obtained from the self-report survey data were compared to ratings obtained from the district level Mathematics Teaching Specialists (MTS) assigned to that school. For the case-study approach, an attempt was made to triangulate the self-report survey data obtained with other external criteria in a subset of schools targeted to participate in more in-depth intensive evaluation efforts. The primary purpose of this paper is to describe the methodology utilized and report the findings for both of these approaches to assess the validity of our data."
Abstract: "A strong base of research is needed to guide investments in teacher professional development (PD). This article considers the status of research on PD and articulates a particular direction for future work. Little is known about whether PD can have a positive impact on achievement when a program is delivered across a range of typical settings and when its delivery depends on multiple trainers. Despite a consensus in the lit- erature on the features of effective PD, there is limited evidence on the specific features that make a difference for achievement. This article explains the benefits offered by experiments in addressing current research needs andfor those conducting and interpreting such studiesdiscusses the unique methodological issues encountered when experimental methods are applied to the study of PD."
Abstract: "The Oregon Mathematics Leadership Institute (OMLI) National Science Foundation Mathematics and Science Partnership project partners are Oregon State University, Portland State University, Teachers Development Group, and ten Oregon school districts. The primary activities of the project were a sequence of three intensive three-week residential institutes emphasizing mathematics content knowledge for teaching, collegial leadership, and the building of Professional Learning Communities. Teachers at all levels of grades K-12 participated together in the mathematics content courses. By the conclusion of the third Summer Institute, teachers had shown significant improvements in mathematical content knowledge for teaching. Analysis of student achievement data in participating schools was initially inconclusive. However, once implementation fidelity traits were taken into account, a positive relationship between project participation and student achievement emerged. The degree to which schools implement the practices promoted by the OMLI project is a significant positive predictor of student performance above and beyond what can be explained by the socioeconomic factor as indicated by the percentage of students who qualify for the free and reduced lunch program. This relationship is particularly acute at secondary levels, but additional factors appear to be at play at elementary grade levels."
Abstract: "The SCALE Quality Indicator System is a work in progress. The purpose of this paper is to describe in some detail the design of the indicator system and to provide samples of some statistics that will likely evolve into indicators. The paper concludes with the acknowledgement of major challenges that exist."
The paper and its accompanying slides are both available in PDF format. Click on the file links above.
Abstract:
"Periodically the field summarizes what is known in particular areas, in some cases supplementing the findings from empirical research with the insights of experts in the area. Typically involving people with a variety of backgrounds and perspectives in the development process, these summaries not only document the state of knowledge at a given juncture, but also provide guidance to practitioners and policymakers based on what is currently known.1
But comprehensive efforts of this sort are time consuming and expensive, and there are many areas of interest to STEM educators where the available knowledge has not been compiled in a practitioner-friendly form. Rather, a mathematics/science supervisor, teacher, or other educator may find out about studies in presentations at professional association meetings, newsletters, or journals and want to learn more. This guide is intended to help consumers of research assess the quality and implications of both individual studies of interventions and research syntheses.2
This paper addresses two key questions that should guide practitioners in reviewing research:
1. How much should I trust the findings?
2. What are the implications, if any, for my context?"
Abstract: In our latest continuation study, we matched MSP schools with non-MSP schools on seven demographic variables for comparing school-level achievement from 2001-02 to 2007-08. The study only found statistically signficant results at the high school level in mathematics. This general absence of statistically significant findings may be due to a number of factors, such as the use of only a subset of data from the entire Cohort I MSPs. In this exploratory, descriptive study, we focus on one specific condition that may affect our matched results, namely, the extent to which non-MSP matched schools in our sample carried out MSP-like activities during 2001-02 to 2007-08.
Abstract: This study is a follow-up to our previous pilot study comparing schools that participate in the National Science Foundation's Math and Science Partnership (MSP) Program with their intrastate non-participating peers in four states (Wong, Boben, Kim, & Socha, in press). The study is part of a larger effort to evaluate the MSP Program's role in student achievement, along with two companion studies. Although the pilot and this follow-up study use a comparative approach, the study by Dimiter Dimitrov (in press) follows a within-group design and the third study by Robert K. Yin (in press) covers the varied results reported by the MSPs themselves in their own "local" evaluations.
At the end of the previous pilot study, the study team suggested items for future work, including adding more categories of what it means to be a participating school. There is only one criterion available that categorizes the level of participation of each school for a given year. The study team used this criterion to categorize the schools into four levels of participation: high, moderate, low, and non-participating. For each of three levels of MSP participating schools, each school has been carefully matched with the non-participating schools on seven demographic variables to form a comparison group. Research conducted by Wong, Boben, and Socha (Wong & Socha 2008; Wong, Boben, Kim, & Socha, in press) offered detailed documentation on how the team operationalized matching methods for comparative purposes. The present study summarizes the matching methodology and discusses the results from the three levels of participation. The analysis leads us to believe that carefully executed matching methods are promising for large- scale comparative analysis on the effects of the MSP Program across states.
The present study draws on publicly accessible school-level standardized test data for 2001-02 and 2007-08 from four states and from data available from the National Center for Education Statistics' Common Core of Data (NCES CCD). In addition, the study uses documents available via MSPnet, and Web site information reported by the individual MSPs accessible through the school year 2007-08.
Abstract:
This pilot study proposes a set of analytical steps for comparing schools that participate in the National Science Foundation's Math and Science Partnership (MSP) Program and their intrastate non-participating peers. This pilot is part of a larger effort to evaluate the MSP Program's role in student achievement, along with two companion analyses. While our pilot study uses a comparative approach, the paper by Dimiter Dimitrov (this volume) follows a within-group design. The third analysis by Robert K. Yin (this volume) covers the varied designs used by the MSPs themselves in their own evaluations. As this pilot study has progressed, there has emerged three distinct phases of analysis. Phase I focused on the participating schools within four Math and Science Partnerships (MSPs) located in three states. Phase II expanded Phase I to focus on three more MSPs in three additional states. The Phase III study was conducted using six cohort I MSPs in four states, and will eventually include the nine cohort I MSPs in all six states from Phase I and Phase II. For each phase, the MSP participating schools were carefully matched with the non-participating schools on eight demographic variables to form a comparison group. This paper offers detailed documentation on how we operationalize matching methods for comparative purposes. We conclude that carefully executed matching methods are promising for large-scale comparative analysis on the effects of the MSP Program across all involved states. The study draws on publicly accessible school-level standardized test data from six states and from data available at the National Center for Education Statistics' Common Core of Data (NCES CCD). In addition, the study uses documents available via MSPnet, and Web site information reported by the individual MSPs in the MSP Program accessible through the school year 2005-06.
Abstract: This pilot study proposes a set of analytical steps for comparing schools that participate in the National Science Foundation's (NSF's) Math and Science Partnership (MSP) Program and their non-participating peers in the same state. This pilot is part of a larger effort to evaluate the MSP Program's role in student achievement, with two companion analyses. In this pilot, we focus on a sample of participating schools in one partnership in one state. The non-participating schools were carefully matched with the program participating schools on eight demographic variables to form a comparison group. This paper offers detailed documentation on how we operationalize two matching methods for comparative purpose. We conclude that carefully executed matching methods are promising for large scale comparative analysis on the effects of MSP Program across different states. This illustrative study draws on publicly accessible school-level data files from one state and from data available at the National Center for Education Statistics' Common Core of Data. In addition, the study consults secondary materials only: available literature together with all of the annual reports, evaluation reports, MSPnet documents, and Web site information reported by the individual Math and Science Partnerships (MSPs) in the MSP Program accessible through the school year 2004-05.
Abstract: This study is one in a series of briefs for the Math and Science Partnership Program Evaluation (MSP-PE) conducted for the National Science Foundations Math and Science Partnership Program (NSF MSP). The study focuses on student achievement in the MSP program as illuminated by data collected in the program's formal management information system (MSP-MIS).
Abstract: A systematic review of the evidence on the effects of teacher professional development on student achievement; a study supported by U.S. Department of Education, Institute of Education Science, National Center for Education Evaluation and Regional Assistance, Regional Laboratory Southwest
Abstract: "Deep engagement of science, technology, engineering, and mathematics (STEM)
disciplinary faculty is a hallmark of the MSP program, which posits that disciplinary
faculty hold the knowledge that K – 12 teachers need, and that if faculty are substantially
involved, the chain of professional knowledge will be strengthened and result in
improved student achievement. Westat’s research, evaluation, and technical assistance
(RETA) grant aims to examine this assumption empirically. Specifically, we are asking
how are STEM faculty engaged in MSP? Does the involvement make any difference in
enhancing teacher quality and increasing student achievement? And are there particular
circumstances in which certain types of involvement contribute more or less than others
on these dimensions? The paper presents preliminary findings from the first 2 years of
this 4-year project. "
Abstract: In order to learn more about the use of the RITES Investigations a number of research and evaluation activities have been conducted, including visits to selected classrooms by staff of the Education Alliance, and a meeting at which RITES teachers presented posters about their use of an Investigation. This document is a report on a survey of the cohort 1 RITES teachers (whose participation began in the summer of 2009) that was conducted during the spring of 2010. These science teachers had agreed to use a RITES Investigation during the 2009-2010 school year.