As a boy, when I learned the difference between induction and deduction, I was deeply impressed, and went looking for instruction on how to do induction. Everybody knows how to do deduction: Socrates is a man; all men are mortal; therefore, Socrates is mortal. — But where do you get the rules?
You get them, of course, from induction. But all the material I found on induction was really stupid. One explained that, you look at Mercury and determine it’s sphereoidal; and at Venus, and determine it’s sphereoidal; and so on to Pluto; and from this you determine “inductively” that all planets are sphereoidal.
Which is useless, of course.
John Stuart Mill, of intro philosophy course fame for his ethical theory, identified and formalized the rules we intuitively use to work from specific cases to general ones. Get good at them and you can work with fuzzy, non-quantifiable data scientifically.
These are the basic rules that Jared Diamond used to organize his historical observations in Guns, Germs, and Steel. I’m writing them up to encourage you to use them for cross-comparison of IF Comp reviews this year.
Key. We’ll have A, B, C, D, E, F, G refer to properties of the game being reviewed, and t, u, v, w, x, y, z refer to opinions of the reviewer. The question is, what game properties reliably elicit what reviewer opinions.
Agreement. If two or more reviewer opinions have only one game property in common, and games with that property are invariably given that opinion, then that game property causes that opinion.
A, B, C, D -> w, t, u, v; and
A, E, F, G -> w, x, y, z; then
A -> w
For example, if we look at a number of games that several reviewers have considered “highly immersive,” and notice that they all have an introduction with a strong narrative hook, and there are no games with a strong narrative hook that reviewers have said were not immersive, then we’ll conclude that the narrative hook causes reviewers to consider the game immersive.
Difference. If two games are identical except one has and the other lacks a certain property, and the game that has the property is given a particular opinion, which the other game is not, then the game property causes reviewers to have that opinion.
A, B, C, D -> w, x, y, z, and
B, C, D -> x, y, z, then
A -> w
For example, let’s say we have two remarkably similar games. One has a well-characterized PC and the other does not. Reviewers consistently say that the first game gives a good sense of player agency while the second does not. From this we will conclude that PC characterization causes reviewers to have a strong sense of agency.
Joint Agreement and Difference. This is just the two above methods applied together.
Covariation. This is an analog version of Agreement and Difference. If one game has a little of some property, and reviewers have a certain mild opinion about it, and another game has a lot of the same property, and reviewers have a much stronger form of the same opinion about this game, then we will conclude that this property causes reviewers to have that opinion.
A, B, C -> x, y, z, and
AA, B, C -> xx, y, z, then
A -> x
For example, let’s say one game has no puzzles, and is considered “okay.” Another game has easy puzzles, and is considered “good.” A third game has tricky puzzles, and is considered “bad.” A fourth game has killer difficult puzzles, which are well-hinted, and is considered “very good.” If we order these games by how soluble the puzzles were, we find we’ve also ordered them by how good reviewers said they were. Therefore, we will conclude that reviewers consider good games to have soluble puzzles.
Residues. This is incredibly useful. If a game has a complex set of properties, and a reviewer has a complex set of opinions, we can cancel out the known patterns of cause and effect, looking only at those game properties and opinions we don’t have maps for.
A, B, C, D, E -> v, w, x, y, z
B -> w is known
C -> x is known
D -> y is known, and
E -> z is known, then
A -> v
For example, let’s say we have a game with well-hinted puzzles of moderate difficulty, a generic PC, several chatty NPCs, and a strong narrative hook; and we want to compare this to another game with no narrative hook, a strongly characterized PC, and extremely difficult puzzles with no hints. Reviewers say the first game is pretty good because it’s immersive and realistic, while the second is not too good, because, although it has a good sense of agency, it lacks immersiveness.
Looking at the patterns of cause and effect we’re pretending we found in prior examples, we say: The first game is “pretty good” because it had well hinted puzzles; it’s immersive because it had a strong narrative hook; and we don’t know why it’s “realistic.” The second game is “not too good” because the puzzles weren’t well hinted enough; it lacked immersiveness because it lacked a narrative hook; it has a good sense of agency because it had a well developed PC.
Ignoring all these knowns, we see that the first game featured conversation with several NPCs, and was considered “realistic,” while the second did not and was not. Now we apply the method of difference, to conclude that conversation with NPCs causes reviewers to call a game “realistic.”
There’s one other thing to look at that J. S. Mill doesn’t talk about. This is not too useful in comparing reviews, but it’s very useful when reading a commented transcript, or a review written real-time.
Timing. If the game does something new, and the player’s attitude changes, then the thing the game did caused the change in attitude.
(The other one that’s useful is covariation: if the game does a little of something, and the player has a small change in attitude, and later the game does it a lot more, and the player has a big change in attitude, then that thing the game is doing is driving the attitudinal change.)
These may seem obvious, but they can take a lot of mystery out of player responses. When a player says or does something remarkable — decides to stop playing the game, for example — look at what the game just did. Did it just symbolically abuse an NPC the player might have been in sympathy with? –and so on.
The problem with timing is that it’s an over-simplification. We very often build up a response over time, until suddenly something happens to trigger it. What’s happening here is that the prior game events are building up a particular response potential — they’re drops gradually filling the player’s glass, which the last game event tips over.
I may be away-from-computer during the IF Comp this year, in which case I won’t be able to write up my reviews of reviews. That’s all right; if you’re interested in unearthing patterns of cause and effect, game design-wise, you can do it yourself. You’re qualified — this is all stuff we know how to do automatically; we just don’t always know when to do it.