Wisdom or madness of crowds?

Here are my present conclusions concerning my readings of the "Wisdom of Crowds" by James Surowiecki and some other articles.

First and foremost -in order to kill any controversy-, accepting the wisdom of crowds is not accepting an unproven concept coming from above like the idea of God. Accepting the wisdom of crowds is opening the mind to some cases where, when specific conditions are fulfilled, the crowd will be wiser or as wise as the wisest individual in the crowd. Accepting the wisdom of crowds is rather asking the old concept that only experts can resolve problems to step down its throne, to accept its failures once in a while and that there are other ways to find a solution.

If we do accept that an expert or a group of experts can perform the job in some circumstances and that a group of laymen can perform the job in other circumstances, then there is no controversy. By accepting this, not only the experts still appear useful for a society, and more than before, but new ways to tackle an issue are found. These new ways to resolve a problem are due to our recent technological prowess such as the Internet; bear in mind that these are just the tools, and that they have invaded our society before we even formulated their purpose and function. That is why we, as a society, have so much difficulty in accepting their capabilities.

So, yes an expert can be the perfect solution for a job. And yes, a crowd can also be the perfect solution for another job. So, what types of job each is capable of performing? In the former case, a job which needs very specific knowledge, so specific that only a few people are familiar with. This happens in research for instance -and I am not writing this because I am a researcher myself. We are not going tomorrow to make a group of laymen to work on the Equatorial Deep Jets (a very obscure set of currents found around the equatorial belt of some oceans -my subject) or on the latest string theory. On the other hand, crowds will perform well if the job consists in repeated predictions on a complex problem, the issues and outcomes of which are not too far from the general cultural baggage of the members of the crowds. In this case, the experts not only might be quickly dry of new ideas to resolve the new problems but they may actually be wrong, basing their prediction on out-of-date or too simplistic models.

We do know many examples when crowds are wrong (or mad). This is such a cliché that it does not need a list of examples. On the other hand, although we do complain once in a while how experts can be wrong, we always have the feeling that we are unfair and speak more with our heart than with reason. But the facts are, and there are many, so-called experts do not perform well and can actually over a long period of time perform very badly. This is the case, to take one example, of the CEOs and other financial gourous who on average performed poorly during the speculative financial bubble of the 90's. The share of their companies have actually decreased over this time.

Now, we know, or guess, the necessary conditions to be an expert: many years of study, scrutinized work by peers, etc. But what about the crowds? How can we assure that a group a priori ready to resolve a problem will succeed? Three main points: 1) diversity, 2) independence and 3) a dose of centralization. Diversity will assure you a wide range of solutions, one of them being maybe the best one. Independence will avoid one member to influence the thinking of the other members and avoid the group to actually act as an individual. The last point is that you need information to flow between all the members and in all directions: too much centralization and the system is dead, too tied, not enough and the system is anarchist, too dispersive, incoherent.

Good and bad experts exist. Mad and wise crowds exist also. It is the latter that we just discovered and goes so much against our individual-oriented society, where each brain is important and has its value, that it is hard to accept.

Now, I have to admit that there is a trick. How can you define the "best" solution as I have vaguely used? In some cases, such as the market, the "best" solution varies from one person to another and it is then a question of values rather than of performance. If you ask me if the market performs well, I will say yes if what you want is the shares of each company to increase with time in some orderly way. I will say no if what you want is a better society where its actual living members are doing well. So, next time you discuss the madness or wisdom of crowds, be sure first that everybody agrees on what is the best solution. Depending on the solution, either experts, or crowds or a mix of the two will be necessary.