Jodie and I had some work on holistic assessment published a few days ago—you can read it here.
This post isn’t a recap of the paper; it’s a set of ideas inspired by having ruminated on what we wrote. The points below are an attempt to capture some of the questions and propositions that have come out of this reflection.
1. What are theories good for?
I spend a lot of time defending theory.
A common complaint about this is that this isn’t practical; this is because theories are theories. Talking about theory as a homogenous category doesn’t help the issue. My coffee preference is a theory. My idea about what is ‘sky blue’ is a theory.
Why should a theory be practical?
To my mind, what is implied in this idea is that we want something of theory. But is this wanting really an expectation that theory will do something for us? Are we comfortable with the idea that we might need to do something with a theory? And that in the process we will probably come up with an intuitive, tacit, twist on the original stimuli?
It makes sense to think about theory as stimuli. We tend to encounter them in moments that are separate those in which they hope they will help us (e.g. reading something about productivity on the internet, ostensibly to help us be productive later).
Some theories provide recipes for us to test out, others are more like firm rules w’ve tested and established (my coffee preference), and some of them are about zooming out to think about the way things are—getting these differences mixed up seems likely to lead to dissatisfaction with theory.
Is theory the same thing as philosophy (or as philosophising)? I’m not uncomfortable with that idea.
2. ‘Explanations run out somewhere’
This is Wittgenstein’s idea, and I feel like I’m quoting it on a weekly basis.
The gist, to my mind, is this: what is meant is not equal to a given set of words.
We use words to signify things all the time. Some of these usages are more ‘official’ than others—words lend a firmity to things, especially when we title a bunch of words with something like ‘rules’ or ‘guidelines’ or ‘law’.
The tension I find interesting is in the point at which we realise that words can’t usually say all of what is meant.
At the same time, claiming that a set of words does do this opens up a claim; sometimes a claim to power. The individual aligned to the words has some sort of authority.
3. Human works are composed along a spectrum of divergence
When people do things (what I am calling ‘works’), they generally do them one of a larger set of possible ways they could be done.
In a factory, things happen in a very consistent way.
In the ways we talk about what people do (especially when we set 300 students the same task, like write an essay on topic ‘x’) we sometimes give the impression that this applies to other things.
The point is that there is not really such a defined recipe for success for many normal (and abnormal) tasks. To ‘cook dinner’ is flexible (that is, divergent) work; more so than the directive to ‘bake me an apple pie following this specific recipe’. How do we account for the fact that said recipe perhaps includes ingredients not available in the time and place w find ourselves?
When ‘official’ signage is applied to divergent works, there is a risk that remarkable outliers are lost. The cost might be what William James called ‘unselected geniuses’.
For more about James and his genius theory, see the introduction and first chapter of this book.
For more about the concept of divergence, read our paper, or read this one by Royce Sadler.
4. Statements and criteria offer different impressions of knowledge
This is an idea I came to via Stanley Cavell’s Claim to Reason (1979). Here’s the gist of the idea, in Cavell’s own words:
When epistemology raises the question of knowledge, what is asks for are the grounds of our certainty. But we are reminded that what we call knowledge is also related to what we call getting to know, or learning, e.g., to our ability to identify or classify or discriminate different objects with and from one another. Criteria are criteria of judgment; the underlying idea is one of discriminating cases, of identifying by means of differences…
In the modern history of epistemology, the idea of judgment is not generally distinguished from the idea of statement generally, or perhaps they are too completely distinguished. The problem is not that in focusing upon those forms of utterance which are characterized by their exclusive possession of truth or falsity a philosopher fails to attend to other “uses” of language… The problem is to see whether the study of human knowledge may as a whole be distorted, anyway dictated, by this focus.
The focus upon statements takes knowledge to be the sum (or product) of true statements, and hence construes the limits of human knowledge as coinciding with the extent to which it has amassed true statements of the world… The focus upon judgment takes human knowledge to be the human capacity for applying the concepts of a language to the things of a world, for characterizing (categorizing) the world when and as it is humanly done, and hence construes the limits of human knowledge as coinciding with the limits of its concepts (in some historical period).
(pp. 16–17, bolding my own)
What’s the thought here?
Cavell seems to be talking about the difference between claims and queries as bases for knowledge.
Criteria are flexible, comprising a non-exhaustive set of components making up the benchmarks (standards) against which we experience things (not just academically recognise them).
What are statements then?
Perhaps these are claims that are established as ostensible criteria.
5. How can the difference between ‘official’ and ‘formal’ criteria be usefully teased out?
I used to use the word ‘formal’ to talk about fixed aspects of the human world, in the sense of things set in place via policy (or other types of rules and regulations).
I thought about this idea differently after listening to Emily Akuno give a key note at a conference this year on assessment in music. Her point (as I understood it) was that form inheres in most aspects of social practice—for example, children’s play may be a formal activity in this sense (when order is present).
I like ‘official’ as an alternative term, because it points to some designation by an authoritative figure about the importance of something, usually in an explicitly signed way.