Views > Business modeling > The danger of using the wrong metrics for policy making

“War is ninety percent information.”

Napoléon Bonaparte
French military and political leader

Adopted public policies affect everyone—They're called public for a reason after all. Nowhere is this truer than in the realm of defense policy. On the one hand, money spent on the military means money not spent on healthcare, education, infrastructure, or other important domestic spending priorities. On the other, underspending on defense may jeopardize the security of a nation and its people.

No doubt, data plays a major role in policy formulation—that’s a good thing. However, the way data is manipulated and displayed can lead to contradictory policy proposals. Public opinion is swayed. Political leaders are affected. Wrong policies are adopted. My aim in the following paragraphs is to show how using the wrong metrics—either intentionally or for lack of competence—leads to promoting political agendas rather than inform readers and decision makers about what's going on in the data. To achieve this end, I’ll consider an example taken from Alberto Cairo’s book The Functional Art.

In Chapter 2 of his book, Alberto looks at the infographic shown below and prepared by Folha de São Paulo—Brazil’s main newspaper—as part of a story highlighting Brazil’s new strategic defense plan. The infographic gives an overview of the size of the armed forces, defense budgets and population of countries around Brazil. Alberto—rightly—argues that the infographic fails to show comparisons, rankings and correlations in an efficient way. In Alberto’s own words, he goes on asking: “… Imagine you’re a concerned Brazilian patriot. Your first impulse will probably be to compare your country with Venezuela and Argentina, Brazil’s main commercial and strategic rivals in the region. See how difficult this operation is? If you want to compare, say, population, you will have to read all the numbers, memorize them, and then organize them in your head. The same thing happens when you try to compare the countries’ defense budgets.”

Source: “The Functional Art”, Alberto Cairo.

Then Alberto proposes a solution. He uses a series of bar charts to rank countries by the size of their armed forces, defense budget and population—as shown in the graph below. An elegant solution indeed. Clear, easy to read and efficient. It took me less than 4 seconds to learn that Brazil tops the ranking in all three measures.

THE DANGER OF USING THE WRONG METRICS FOR POLICY MAKING - THE CASE FOR OR AGAINST DEFENSE SPENDING

At this point, Alberto faces a challenge. He claims that based on the above charts “…It would seem Brazil will be the winner in all of the rankings, because the bigger the population a country has, the bigger its military forces and heftier its defense budget will be. But is this relationship perfectly proportional? In other words, if population is n times bigger in one country than in another, will armed forces also be n times larger?”

Alberto Cairo is concerned that “… Proportionally, some of Brazil’s neighbors, such as Venezuela and Chile, invest much more each year in keeping their armies, naval fleets, and air forces up to date. And that’s an issue: Brazilians are proud of their newly earned status as an almost-developed nation, and they want the world to know about it.”

He goes on proposing that if the data is presented on a relative basis—as per the graph below—the picture changes completely and Brazilian patriots should be alarmed. As a result of these derived measures, Alberto claims that “…The biggest armies in absolute terms may not be the biggest ones in relative terms. In relative terms, our Brazilian patriot would surely be alarmed, because her country ranks second from the bottom. Countries like Venezuela and Bolivia are more militarized, and Colombia is far ahead because of its war on narco-guerrillas.”

And finally Alberto ties everything together with the following graph. But then, what are decision makers and the general public supposed to do with this graphic? What’s the take away? Given the size of population, armies and military budgets of its neighbors, should Brazil spend more on defense than it is spending today? Should Brazil increase the size of its army? Or should Brazil just do the opposite—cut defense budgets and army sizes—and allocate the money to other sectors of the economy? According to Cairo, in absolute terms Brazil is a winner, but in relative terms, Brazilian patriots should be alarmed.

Let’s see what happens if Brazil were to top the rankings based on the relative measures. The size of the Brazilian army has to grow to 1.1 million from 368,000. That’s roughly 1.5 times the size of the armies of the next seven neighboring countries combined. Is it really necessary? Similarly, to match the per capita spending of Chile, Brazil has to increase its defense budget to 53.6 billion USD from 21.6 billion USD. That’s roughly 2.9 times the defense budget of all neighbors combined.

So how relevant are relative—derived—measures to defense policy making? Should defense spending and size of armed forces be tied to population size? Does it really matter that a country like Bolivia—with a tiny army of 46,000 compared to Brazil’s of 368,000—has 5.1 armed forces employees per 1,000 people versus 1.9 for Brazil? Brazil’s army is 8 times bigger than that of Bolivia. That’s a large difference. Which country wants to mess with its neighbor given this disadvantage? Obviously, using relative measures is not an appropriate way to measure necessary military budget allocation.

Here is my take on the numbers. Simply put, our Brazilian patriot should not be alarmed at all. Actually, the ones who should be really worried are Brazil's neighbors.

Source: “The Functional Art”, Alberto Cairo.

ALBERTO CAIRO'S REPLY

First, design-wise I really like your graphic better than mine. It’s clean and elegant. Contrary to what you wrote, though, my point in that part of the book was not to suggest changes in policy. If that’s what it seems, I’ll need to word things better if I ever do a second edition. My goal was to warn people about showing just counts without considering anything else or controlling for other variables to get a clearer picture.

Adjusting for population is just one way to do this. Additional ones —pointed out to me by Brazilian military personnel in 2009— would be to also think of country area, and border and coastal length, as the Brazilian military doesn’t care about invading other countries, fortunately; they worry about protecting their own citizens and territory from both external and internal threats (therefore the employees per 1,000 people). If we don’t control for anything, we’ll miss the fact that both Venezuela and Colombia are highly militarized countries for different reasons. Or that Brazil has a hard time keeping and eye on its borders and on the huge Amazon region, something I was also told about by Brazilian military, and that is constantly in the news.

All this is important.All this considering again that a deep discussion about these matters doesn’t fit into the goals of ‘The Functional Art’. Again, my purpose wasn’t analysis, but presentation.