A Puzzle With Broader Implications

[SnerpGoodWord]

A couple of forumites have suggested to me that I should do an economics and/or trading blog or website. I'm still on the fence about the idea, but I thought I'd give it a try and see if people find what I have to say interesting.

Here's a post about an economics/mathematics puzze I like: Sages and Fools.

Eventually if I stick with this I'll put up a discussion forum on the site, but for now I figure discussing it here would be easier.

Edit: BlueHost was having problems, but they now seem to be resolved. If the link doesn't work for you, please let me know what you see.

[Fred Tracy]

Link is broken buddy.

[SnerpGoodWord]

Originally Posted by Fred Tracy

Link is broken buddy.

Database error?

BlueHost appears to be jacked up.

[SnerpGoodWord]

I think it's back up now.

[aelle]

Interesting stuff.

I'm just out of bed, but my thoughts so far are:
- first, you want to weed out the decision makers with a limited history M. As note 1 points out, the more limited the history, the more likely it is that a high success rate is luck rather than skill. I guess how much you want to weed out depends on 1. population size N, 2. available history sizes, 3. acceptable risk (it's easy enough to calculate the chances that a particular sage relied on pure luck if M is known, then it's just a matter of how much of a bet you are willing to take that this wasn't the case).

- then, you want to weed out the fools entirely. Anyone with A=0.5 on a long history is useless.

- intuitively, I would then give more weight to the people with more extreme values of A (and giving a negative vote to those with A<0.5, as suggested).

Putting the problem into mathematical form is an interesting choice. I would have started with completely different moves if the problem had been described as a committee of experts, or a national voting system.

I'll have the boyfriend read it and see if we come up with a finer method together.

[aelle]

[aelle's boyfriend]
With regards to the observations.

I was initially confused how a master could get odds or evens right and have to guess the others. It was put to me in the form of a politician who gets the ministry in his field one term and then a non-related one the other, making him an informed master making decisions on "odds" and then having to guess his way in "evens".

To put that in a mathematical way you'd have to form subsets of M along expertise lines, preferably in such a way you can then line up expertise of different sages (so M_A, M_B, M_C representing some area of expertise rather than M_E and M_N representing expertise and non-expertise). Idea to include vote weights based on area of expertise?

To weed out the students sages you raise M as already described by the girlfriend. (Am I really referred to here as such?)

Form the committee with at least 2 master sages for every area of expertise, where master sage is determined by a large enough M_[relative letter] and A_[relative number].



On a side note, how would you deal with motivation issues on the negative sage as described by aelle? Being on a committee and having people really interested in your opinion so they can swing the other way has to have some impact on the long run.

Another important human trait are opinion leaders. The problem touched on it with the professor student relationship, but it can be larger. You could identify them from the data by identifying clusters of decision makers who always vote the same way, but are more random when a particular person is absent from the vote. If the opinion leader has an A that lowers the group's A, you want to kick him out... unless you can pair him with an expert.

[SnerpGoodWord]

One aspect of the problem as intended is that M is fixed global constant - you only have as much history as you have, and every person in the room has opined on all M decisions (and only those M). I think this is consistent with many real-world applications and makes the problem conceptually economic - a (meta)decision made under uncertainty.

This raises the question: under what circumstances is M too low and/or N too high to behave rationally?

Weeding out the fools seems conceptually attractive to me, and my solution initially included it. But I can't prove it's actually necessary, which is intriguing or frustrating depending on perspective.

I agree that the correct solution must, in the general case, involve finding sages who are good at different things. I think the methodology is the trick though - the internal structure of which decisions are of which type is hidden, and thus must somehow be deduced.

Here's an interesting and very strange case: the decisions are secretly of 2 classes, D1 and D2, equally likely. The room consists solely of sages of two classes S1 and S2, both in abundant supply.
A(S1, D1) = 1.0
A(S1, D2) = 0.5
A(S2, D1) = 0.1
A(S2, D2) = 0.6

Note that sages of class S2 are "anti-sages" with a global A = 0.35. However, an optimal committee is to have an equally large number of S1s and S2s, all with equally positive voting weights. This gives a committee accuracy approaching 1.0. What makes this case interesting is that it's correct to include an anti-sage with a POSITIVE voting weight. He's useful specifically because he's knowledgeable about the topic which the globally useful S1 is ignorant about.

[SnerpGoodWord]

Originally Posted by aelle

Putting the problem into mathematical form is an interesting choice. I would have started with completely different moves if the problem had been described as a committee of experts, or a national voting system.

This is a very interesting philosophical point: once put into mathematical terms, the space of solutions worth considering is essentially disjoint from those actually used in the real world for problems that map closely (or even perfectly?) onto this. Homo economicus seems to be missing in action.

[SnerpGoodWord]

Originally Posted by aelle

[aelle's boyfriend]
You could identify them from the data by identifying clusters of decision makers...

This idea might be of value even if you weren't concerned about opinion leaders. If you computed the correlation matrix of the decision-makers past decisions, the number of Eigenvectors of that matrix with magnitude > 1.0 (or really, with magnitude greater than you would expect from randomness) should tell you a lot about the types of decision makers available to you and/or decisions to be made. In essence, this would be the start of factor analysis on the problem.

My quasi-solution doesn't include this technique, but it does something that achieves the same effect.

[Brutha]

I agree that the correct solution must, in the general case, involve finding sages who are good at different things.

The way the problem is stated I think that's false.
Why? You lack a way to distinguish the different expertises when you are faced with an actual problem.

I think the correct solution to the problem as it's stated is to do Principal component analysis and take the first principal component as solution.

How do you solve the issue in the real world? Every decision maker who makes his prediction should also rate the confidence that he has in the truth of his prediction.

[SnerpGoodWord]

Originally Posted by Brutha

The way the problem is stated I think that's false.

While not perfectly applicable to this problem, you might consider the proof of the training set optimality of AdaBoost. It would appear to be, in more general terms, a strong argument for adding specialists to a committee and simply coping with their flaws by outvoting them elsewhere.

It also explains how the S2 sage above can be included in a committee with a positive weight despite his poor accuracy. Because, once the Di coefficients of AdaBoost are applied, he has an A > 0.5 on the questions that really matter - those S1 gets wrong.