(This is the first of four articles to discuss why it is so difficult to agree how many of something we have, a proven approach to overcome this, a worked example and an idea of what to do next).
A question often asked and rarely answered the same way twice! There are many reasons for this, and most of those can be seen on that quick sketch above which led to a long LINKEDIN discussion about the work we’d done at a recent workshop.
Essentially there are two issues to address:
1) What question are you trying to answer. If we define a student as, for example “a person engaged on a learning experience at our university’ (and yes this is not all encompassing, I’m looking at you PGR/T but just bear with me) , then already our question must be extended to cover what type, where from, how far through their studies, who pays, who awards, etc, etc. There are so many permutations, getting two to agree even if the question is reasonably well formed is difficult.
2) Why can’t we agree how to count them. The obvious remedy to 1) is to create ‘populations’ that answer the most oft asked questions; how many students are paying fees? How many do we need to timetable on campus, which ones do we need to report to regulators etc. Seems sensible but it is an absolute nightmare of nuance and often dies a death by a thousand edge case cuts!
Populations though are at least some of the answer to the counting students problem. There are three steps you need to follow to ensure the time invested has real value to the entire university.
Defining your populations
Start where you mean to end is good advice here. Most universities will already have a number of populations, generally with a varying degree of trust. So the choice is to increase the trust (and thereby usage) of existing populations or creating new ones that satisfy a requirement that ideally is university wide.
In the next article we’ll show what kind of populations to start with, what data is needed to create them, what the use cases are, and how they can be modified to meet many other use cases/types of questions without having to create another population.
Verifying your populations
Once defined, we need to gain agreement these are both the right ones to be focussed on, and the methodology to create them is sound. It is vital to establish wide spread support and known limitations by engaging across the whole institution – both professional and academic services.
While this may seem an onerous process, populations are used to support or enable important and far reaching decisions. They are often contentious because of how many different ways they are used. This is why creating ‘simple’ populations and slicing them by modifiers moves us away from creating many different reports with only a few different attributes.
An existing Data Governance framework really helps here. Both in terms of understanding who uses these types of populations, and accountability for the quality and the terms that assure the dataset. This aids both the reviewing and agreeing the technical specification (what goes in, what is counted, etc) and the qualitative one (description of the population, how it is used, etc). Both can be co-ordinated through Subject Matter Experts (“Stewards in our DG framework).
To speed up verification, don’t attempt to answer each use case. Ask yourself if the cost to modify / extend the population is commensurate to the value of doing so.
Even after extensive collaboration and engagement, there will be uses that have not been tested, or new ones with different requirements. Creating a robust and visible challenge process (far easier if you already have a network of data owners and stewards) is vital to ensure the populations can adapt to how the university wishes to use them.
It is worth keeping a history of challenges to help understand what new populations may be needed, or what limitations have to be accepted with the current ones
Using your populations
The “publish and be damned” has some currency here. We recommend publishing a ‘defensible draft’ and use the challenge process to improve the population. This, of course, comes with risks but in our experience it is a far more effective pathway to acceptance than trying to create the perfect population.
When using these populations, it is good practice to include how they have been created / derived. This definitely cuts down on confusion and frustration especially when staff are first introduced to them.
Finally these populations need to be considered ‘for the university’ regardless of who created them. They cannot be ‘plannings data’ or ‘faculty view’. This is why the verification step is so important.
Starting with the modifiers. If you ask everyone how they would like the populations cut, it will be exceeding difficult to synthesise those into a few well understood populations. Instead agree what would benefit the university the most (often funding, support for internal KPIs etc) and define these first.
Not being transparent about the process. Populations can be contentious. They will never satisfy everyone. So being clear on how they are created and – more importantly – how they can be challenged (with a rider than it is a very high bar for change) can mitigate some of the mistrust those who haven’t been involved in their creation sometimes have.
Having too wide a scope. While we’ll stick to our ‘publish and be damned, attempting to launch a raft of new populations at once should be avoided. Instead a clear and prioritised roadmap of which ones and when will make landing them a little easier.
Attempting the hardest ones first. We see this a lot! There is understandable excitement that ‘this problem can be solved’ and the group then dives into the most difficult/contentious/widest uses first. We would recommend starting somewhere ‘in the middle’ so processes can be fine tuned and engagement approaches refined,
We hope you share our view that we can count students if we apply data governance and change management best practice. Every institution will be different though, so please take the above as guidance not gospel. We’ve focussed on students here, but of course this approach can work for Staff (“how many employees do we have”) and lots of other areas of the university.
We’ll be back next week with the second article showing a worked example of creating and using a population.