By Timothy D. Harfield
Our spring 2015 series of learning analytics brown bag seminars opened today with a stimulating discussion led by Dr. Ben Sayeski, Managing Partner of a Virginia-based educational analytics firm called Education Strategy Consulting. In addition to demonstrating the visualization tool that ESC has successfully deployed in k-12 school districts all over the nation, he also demonstrated the value that can be gained in higher education through the visualization of even openly available data sets (like from the Integrated Postsecondary Education Data System (IPEDS), for example).
Ben touched on a lot of things in his presentation, but four major themes stood out in my mind that are of relevance to folks working with educational data (either as producers, analysts, or consumers) in general:
1. Moving from the knowable to the known
All too often we are inclined to argue in defense of positions about things that are knowable, but on the basis of opinion and anecdotal evidence alone. The problem with this is that it leads to disagreement, and decisions made on the basis of consensus. Truths, however, are not the result of agreement. The power of educational data, when effectively visualized, lies in their ability to mitigate conflict and ensure sound decision-making.
2. Importance of Transparency
Too many commercial analytics tools function as black boxes. Data is thrown in, and analytics come out, but the transformations that take place in the conversion of data into information are left (intentionally) shrouded in mystery. A decision that Ben and his business partner, Dr. Matthias Hild, made early on was to provide their customers to clear information about every stage in the process, from preparation, to analysis, to visualization. This approach extends even to the models that are employed in the analytics phase. The power of transparency lies in its ability to generate confidence in the data (which is incredibly important where that data is used as the basis for decisions about school closures, budgetary considerations, etc). In analytics, the acceptance of a tool is the result of three factors (not two, as Ali et al have argued) : (1) usefulness, (2) useability, and (3) trust. In order to achieve true return on investment, what what Ben calls ‘Impact,’ all three must be in place.
3. Raising Questions vs. Making Recommendations
The goal of analytics is not to predict outcomes or make recommendations. The goal, rather, is to put information before the eyes of decision-makes in such a way as to encourage them to ask questions and dig deeper. There are major statistical problems associated with presenting data in too granular a fashion, which may give users a false sense of objectivity and lead them to make ‘data-driven’ decisions. Decisions should never be data-driven, but should rather be data-informed. Where analytics show interesting patterns in the data, it is an opportunity for users to go to the places that are represented and see for themselves the qualitative factors that contribute to a particular effect in the data.
4. Platform Agnosticism
There are several inherent difficulties of data warehousing as a practice that creates problems when working with large amounts of data coming in from various sources, in various formats, and in various states of cleanliness. Data warehouseing technologies and data models change over time, as do the systems that are generating data for analysis. Data warehousing practices all too often assume static information flows, and then are caught off guard and slow to adapt to change. The workflow and technology that ESC has developed is designed to accommodate a dynamic data ecosystem. “Give me whatever you have, in whatever form you have it in, and let us make sense of it.” This approach not only avoids the need for institutions to implement strict policies about data preparation, and to train personnel (or hire personnel) to prepare data for import, but it also frees up institutions from lock-in. The technologies that we select should be the ones that best suit our needs, and not those that play best with our existing warehousing infrastructure.
On February 11, the ALE community looks forward to a presentation from Nancy Bliwise (Assistant Vice Provost for Academic Planning, Emory University), who will detail the university’s new data governance policy and discuss several questions that it raises for the use of educational data. For more information about this and other upcoming events, please visit our schedule page here: https://scholarblogs.emory.edu/ale/events/seminars.