Jul 162013

In the last few days I finished reading the “Black Swan” by Nassim Nicholas Taleb. Around last January I saw Günther Palfinger mentioning it in my G+ stream, looked it up and bought it.

At first, the book seemed to present some interesting ideas on error statistics and the first 20 or 30 pages are giving good examples for conscious knowledge we posses but don’t apply in every day actions. Not having a trading history like the author, I found reading further until around page 100 to be a bit of a drag. Luckily I kept on, because after that Taleb started to finally get interesting for me.

Once upon a time…
One of the lectures I attended at university touched on black box analysis (in the context of modelling and implementation for computer programs). At first of course the usual and expected or known input/output behavior is noted, e.g. calculus it may perform or pattern recognition or any other domain specific function. But in order to find out hints about how it’s implemented, short of inspecting the guts which a black box won’t allow for, one needs to look at error behavior. I.e. examine the outputs in response to invalid/undefined/distorted/erroneous/unusual inputs and assorted response times. For a simple example, read a sheet of text and start rotating it while you continue reading. For untrained people, reading speed slows down as the rotation angle increases, indicating that the brain engages in counter rotation transformations which are linear in complexity with increasing angles.

At that point I started to develop an interest in error analysis and research around that field, e.g. leading to discoveries like the research around “error-friendliness” in technological or biological systems or discoveries of studies on human behavior which implies corollaries like:

  • To enable speedy and efficient decision making, humans generally rely on heuristics.
  • Displaying heuristic behavior, people must make errors by design. So trying to eliminate or punish all human error is futile, aiming for robustness and learning from errors instead is much better.
  • Perfectionism is anti-evolutionary, it is a dead end not worth striving for. For something “perfect” lacks flexibility, creativity, robustness and cannot be improved upon.

A Black Swan?
Now “Black Swan” defines the notion of a high-impact, low-probability event, e.g. occurring in financial trading, people’s wealth or popularity – events from an extreme realm. That’s in contrast to normally distributed encounters like outcomes of a dice game, people’s body size or the number of someone’s relatives – encounters from a mediocre realm.

From Mediocre…
Here’s a short explanation for the mediocre realms. Rolling a regular dice will never give a number higher than 6 no matter how often it’s thrown. In fact, the more it’s thrown, the more even it’s numbers are distributed and the clearer its average emerges. Measuring people’s weight or number of relatives shows a similar pattern to throwing a dice, the more measurements are encountered the more certain the average becomes. Any new encounter is going to have lesser and lesser impact on the average of the total as the number of measurements increases.

To Extreme…
On the other hand there are the extreme realms. In trading or wealth or popularity, a single encounter can outweigh the rest of the distribution by several orders of magnitude. Most people have an annual income of less than $100k, but the tiny fraction of society that earns more in annual income possesses more than 50% of the entire distribution of wealth. A similar pattern exists with popularity, only very few people are so popular that they’re known by hundreds of thousands or maybe millions of people. But only very very few people are super popular so they’re known by billions. Averaging over a given set only works for so long, until a high-impact “outlier” is encountered that dominates the entire distribution. Averaging the popularity of hundreds of thousands of farmers, industrial workers or local mayors cannot account for the impact on the total popularity distribution by the encounter of a single Mahatma Gandhi.

On Errors
Taleb is spending a lot of time in the book on condemning the application of the Gauss distribution in fields that are prone to extreme encounters especially economics. Rightfully so, but I would have enjoyed learning more about examples of fields that are from the extreme realms and not widely recognized as such. The crux of the inapplicability of the Gauss distribution in the extreme realms lies in two things:

  1. Small probabilities are not accurately computable from sample data, at least not accurately enough to allow for precise decision making. The reason is simple, since the probabilities of rare events are very small, there simply cannot be enough data present to match any distribution model with high confidence.
  2. Rare events that have huge impact, enough impact to outweigh the cumulative effect of all other distribution data, are fundamentally non-Gaussian. Fractal distributions may be useful to retrofit a model to such data, but don’t allow for accurate predictability. We simply need to integrate the randomness and uncertainty of these events into our decision making process.

Aggravation in the Modern Age
Now Taleb very forcefully articulates what he thinks about economists applying mathematical tools from the mediocre realms (Gauss distribution, averaging, disguising uncertain forecasts as “risk measurements”, etc) to extreme realm encounters like trade results and if you look for that, you’ll find plenty of well pointed criticism in that book. But what struck me as very interesting and a new excavation in an analytical sense is that our trends towards globalisation and high interconnectedness which yield ever growing and increasingly bigger entities (bigger corporations, bigger banks, quicker and greater popularity, etc) are building up the potential for rare events to have higher and higher impacts. E.g. an eccentric pop song can make you much more popular these days on the Internet than TV could do for you 20 years ago. A small number of highly interconnected banks these days have become so big that they “cannot be allowed to fail”.

We are all Human
Considering how humans are essentially functioning as heuristic and not precise systems (and for good reasons), every human inevitably will commit mistakes and errors at some point and to some lesser or larger degree. Now admitting we all error once in a while, exercising a small miscalculation during grocery shopping, buying a family house, budgeting a 100 people company, leading a multi-million people country or operating a multi-trillion currency reserve bank has of course vastly different consequences.

What really got me
So the increasing centralisation and increasing growth of giant entities ensures that todays and future miscalculations are disproportionally exponentiated. In addition, use of the wrong mathematical tools ensures miscalculations won’t be small, won’t be rare, their frequency is likely to increase.

Notably, global connectedness alerts the conditions for Black Swan creation, both in increasing frequency and increasing impact whether positive or negative. That’s like our modern society is trying to balance a growing upside down pyramid of large, ever increasing items on top of its head. At some point it must collapse and that’s going to hurt, a lot!

Take Away
The third edition of the book closes with essays and commentary that Taleb wrote after the the first edition and in response to critics and curios questions. I’m always looking for relating things to practical applications, so I’m glad I got the third edition and can provide my personal highlights to take away from Taleb’s insights:

  1. Avoid predicting rare events
    The frequency of rare events cannot be estimated from empirical observation because of their very rareness (i.e. calculation error margin becomes too big). Thus the probability of high impact rare events cannot be computed with certainty, but because of the high impact it’s not affordable to ignore them.
  2. Limit Gauss distribution modeling
    Application of the Gauss distribution needs to be limited to modelling mediocre realms (where significant events have a high enough frequency and rare events have insignificant impact); it’s unfortunately too broadly abused, especially in economics.
  3. Focus on impact but not probability
    It’s not useful to focus on the probability of rare events since that’s uncertain. It’s useful to focus on the potential impact instead. That can mean to identify hidden risks or to invest small efforts to enable potentially big gains. I.e. always consider the return-on-investment ratio of activities.
  4. Rare events are not alike (atypical)
    Since probability and accurate impact of remote events are not computable, reliance on rare impacts of specific size or around specific times is doomed to fail you. Consequently, beware of others making related predictions and/or others relying them.
  5. Strive for variety in your endeavors
    Avoiding overspecialization, learning to love redundancy as well as broadening one’s stakes reduces the effect any single “bad” Black Swan event can have (increases robustness) and variety might enable some positive Black Swan events as well.

What’s next?
The Black Swan idea sets the stage for further investigations, especially investigation of new fields for applicability of the idea. Fortunately, Nassim Taleb continues his research work and has meanwhile published a new book “Antifragile – Things that Gain from Disorder”. It’s already lying next to me while I’m typing and I’m happily looking forward to reading it. 😉

The notion of incomputable rare but consequential events or “errors” is so ubiquitous that many other fields should benefit from applying “Black Swan”- or Antifragile-classifications and corresponding insights. Nassim’s idea to increase decentralization on the state level to combat escalation of error potentials at centralized institutions has very concrete applications at the software project management level as well. In fact the Open Source Software community has long benefited from decentralized development models and through natural organization avoided giant pitfall creation that occur with top-down waterfall development processes.

Algorithms may be another field where the classifications could be very useful. Most computer algorithm implementations are fragile due to high optimization for efficiency. Identifying these can help in making implementations more robust, e.g. by adding checks for inputs and defining sensible fallback behavior in error scenarios. Identifying and developing new algorithms with antifragility in mind should be most interesting however, good examples are all sorts of caches (they adapt according to request rates and serve cached bits faster), or training of pattern recognition components where the usefulness rises and falls with the variety and size of the input data sets.

The book “Black Swan” is definitely a highly recommended read. However make sure you get the third edition that has lots of very valuable treatment added on at the end, and don’t hesitate to skip a chapter or two if you find the text too involved or side tracking every once in a while. Taleb himself gives advice in several places in the third edition about sections readers might want to skip over.

Have you read the “Black Swan” also or heard of it? I’d love to hear if you’ve learned from this or think it’s all nonsense. And make sure to let me know if you’ve encountered Black Swans in contexts that Nassim Taleb has not covered!

  3 Responses to “5 Insights from the Black Swan”

  1. I haven’t read this book, but tell me if I’m wrong (or my english, or math is) :

    You say that “Now “Black Swan” defines the notion of a high-impact, low-probability event”. I don’t know how old is this reference to Black swans. K. Popper use it to show the problem of induction reasoning :

    It doesn’t matter how many white swans you have seen, it can’t be a proof that black swans don’t exist. The view of a single black swan prove that “all swans are white” is wrong.

    But for Popper, as far as I now, it’s not about probabilities. You can’t really make probability to answer a question like ‘all swans are white’, because it’s always possible that there are plenty of blacks swans you don’t see yet (they are in Australia and you don’t know Australia).

    Now I understand that you can evaluate the probability of an event that never occured happens. Like an earthquake bigger that the ones ever seens, thanks to a distribution analyse. But we define how big is an earthquake a a continuous variable. How to do it with swans being black or white ? Especially if you don’t even know if black swans exists ?

    Being not a mathematician at all, I really unjoined learning a few things about probability, but I do think that some events can’t be seen as more or less likely, because they are qualitatively different or because we just can’t have any idea about them (we never saw them happening). For swans, you can’t even imagine a process making swans black. If you want to evaluate the probability of an event you have never seen to occur, you need to have at least a few idea on how it can occur

    So my question is : Why talking about black swans for low probability ? Who is it a reference to ? Does the book tell ? Does what I say is wrong so that even this kind of events can be considered as having a low probability ? Thanks.

    • Thanks for your thoughtful comments. First, I’m not trying to summarize the full book or even just the Black Swan event definition here. So you are indeed missing some critical bits that the book covers and that I’m not perfectly representing in my insights list. One is, we can *not* compute the probability of rare events reliably, i.e. the rarer the event, the higher the uncertainty (disproportionally). The problem with modeling the probability through a distribution is, is the model the right one? You’d need lots of data to be confident about picking the right distribution model, but we’re dealing with *rare* events, so there’s little data to start with. A small, miniscule error in the model or its parameterization will yield vastly different probability outcomes in the extremes (due to non-linearity in the models), which means the computed probabilities have a high degree of uncertainty, so in effect, they’re not precisely computable.

      Nassim Taleb goes beyond Popper’s notion of falsification, he uses the Black Swan as a synonym for an encounter that was unpredictable, rare and consequential. That notion is subjective (i.e. on thanksgiving, a Turkey will experience a Black Swan event, but not the butcher who’s planned for it in years) and some rare events that are consequential can be predicted (e.g. if you have some deterministic knowledge about their creation) which he refers to as “Grey Swan”.

      His main point is that we shouldn’t rely on false models that can be proven to yield false predictions about rare event probabilities and he suggests avoidance measures to avoid blowing up from them. So in “Black Swan”, he focuses on “via negativa”, accept that certain events can *not* be accurately predicted and as such we should avoid depending on their occurrence or depending on them not occurring.

      Last but not least, yes, the book will tell how exactly he defines and uses “Black Swan”, tell known unknowns from unknown unknowns, describe meta probabilities for rare events (which really is the level of uncertainty about your probability calculations) and he positions his idea relative to pretty much all relevant philosophers out there. It’s new, it has not been addressed before and that’s why he should be read. 😉

  2. Thanks for the clarification. I should read it 🙂

 Leave a Reply

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>