This question and answer section references questions posed during the Carnegie Foundation’s January 2022 webinar on Technology and Creating a Theory of Change in Networks.

Q: Will this powerpoint be available after the presentation/discussion?

At this time, Carnegie is unable to provide the webinar slides separately from the display in the recording video.

Q: What about quantitative causal modeling? System dynamics, Bayesian belief networks, other simulation models?

If the NIC leadership team is so fortunate as to have a member with strong analytical skills who can conduct these types of more sophisticated analyses, the data that is gathered would lend an even deeper understanding of the problem and the system causing it.  These types of analyses would yield valuable information about system processes (how work gets done) and the interdependencies or lack of dependencies of system elements.  They would lend more accuracy and complexity to the causal system analysis as a result. Another consideration, when it comes to testing a theory of improvement, is that it’s worth thinking about analyzing mediation in nested data to examine the causal relationships that we hypothesize. Methods like hierarchical linear modeling (HLM) and multilevel structural equation modeling (MSEM) might be of use here.

Q: Is NILS currently available for purchase?

Yes, NILS is now available for purchase. The platform’s pricing structure is based upon an initial set up cost and an annual license cost per user, dependent upon the size of a network. Additional support services are available for an additional fee. Please contact support@carnegienetworks.org for more information.

Q: Have any journalism-focused organizations worked with NILS?

NILS currently supports organizations working in the education and health sectors. However, the platform is content-agnostic, and can support the enactment of system improvement in a variety of sectors.

Q:

Is the NILS driver diagram 3D, and how do viewers expand and collapse it?

The NILS driver diagram can be viewed in two forms: an “accordian”-style list format, in which elements can be expanded and collapsed, and a “branch”-style diagram view, in which each element (drivers, change ideas) can be expanded to show additional information. The diagram can also be downloaded as a PNG. It is not a 3D model. 

Q: Do teachers improve their pedagogical practice by using NILS?

Yes, many educators have found NILS useful for their work. Join Carnegie and improvement champion Sofi Frankowski, Chief Learning Officer for Schools That Lead, on May 11, 2022 for the fourth webinar in the series on Technology and Collective Learning in Networks and a discussion of how the NILS platform facilitated her improvement and NIC-management efforts.

Q: Which network is adding the science of learning to their teacher prep programs?

Deans for Impact

Q: Do you have any active university educational leadership groups working on a common problem?

Yes. NILS supports a variety of networked improvement communities working on a variety of problems, including leadership groups in universities.

Q: What books would you recommend for beginning an improvement science bibliography?

• Bryk, Anthony S., Louis M. Gomez, Alicia Grunow, and Paul G. LeMahieu. 2015. Learning to improve: How America’s schools can get better at getting better. Cambridge, MA: Harvard Education Press.
• Bryk, Anthony S., 2020. Improvement in Action: Advancing Quality in America’s Schools. Cambridge, MA: Harvard Education Press.
• Crowe, Robert, Brandi Hinnant-Crawford, and Dean T. Spaulding. 2025. Improvement Science Across the Disciplines. Gorham, ME: Myers Education Press.
• Deming, W. Edwards. 1986. Out of the crisis. Cambridge, MA: Massachusetts Institute of Technology.
• Hinnant-Crawford, Brandi. 2020. Improvement Science in Education: A Primer. Gorham, ME: Myers Education Press.
• Langley, Gerald J., Ronald D. Moen, Kevin M. Nolan, Thomas W. Nolan, Clifford L. Norman, and Lloyd P. Provost. 2009. The improvement guide: A practical approach to enhancing organizational performance. 2d ed. San Francisco: Jossey-Bass.

Q: When building an action plan, how do you collect the evidence to make future analyses?

Before collecting evidence, we have to first specify for ourselves what we want to learn. What are our learning questions and how might the questions evolve over time? We should start with brainstorming the kind of evidence we need to answer the learning questions instead of the kind of evidence we already have. Also, when thinking about “future analyses”, we have to reflect on the nature of the analyses. Are they analyses for improvement/research/accountability? Are we prioritizing collecting data for improvement that would provide actionable information to practitioners? Disciplined inquiry is both a science and an art, and it takes iterations to optimize data collection and analysis processes. As for what this process looks like in a networked setting, evidence is collected through a knowledgement management system that is, in most cases, developed and coordinated by the leadership team. A solid knowledge management system in the context of a networked improvement community should enable the sharing, sensemaking, and consolidation of knowledge in the form of both quantitative and qualitative data that is collected across all levels of the network.  This is especially important at the level of the improvement sites where inquiry cycles on change ideas are being applied and valuable learning is occurring.  This learning represents evidence, whether it is measurement data at the PDSA or driver level or qualitative data such as user perspectives, observations, narratives, etc.