Methodology

The Bertelsmann Stiftung’s Transformation Index (BTI) analyzes and evaluates whether and how developing countries and countries in transition are steering social change toward democracy and a market economy. Guided by a standardized codebook, country experts assess the extent to which a total of 17 criteria have been met for each of the 137 countries. These experts ground the scores they provide in assessments that comprise the country reports, all of which are available online. A second country expert then reviews these assessments and scores. In a final step, consistency is ensured by subjecting each of the 49 individual scores given per country to regional and interregional calibration processes. Standardizing the analytical process in this way makes targeted comparisons of reform policies possible.

The BTI aggregates the results of this comprehensive study of transformation processes and political management into two indices: The Status Index and the Governance Index. The Status Index, with its two analytic dimensions of political and economic transformation, identifies where each of the 137 countries stand on its path toward democracy under the rule of law and a social market economy. The Governance Index assesses the quality of political leadership with which transformation processes are steered.

The BTI is published every two years. This biennial evaluation of transformation and development allows us to assess observed trends and identify the outcomes of transformation strategies. The BTI expands the available body of knowledge about how political processes are managed and decision-making is conducted, and makes this knowledge available to policymakers and other advocates of reform. Overall, the BTI offers a comprehensive body of data, allowing a broad spectrum of actors to assess and compare the factors driving success in developing and transformation countries.

Analytical framework

The state of political transformation is measured in terms of five criteria, which in turn are derived from assessments made in response to 18 questions. The BTI’s concept of democracy goes well beyond other definitions of democracy, which are limited primarily to basic civil rights and the conduct of free elections. Stateness, which is seen as a precondition to democracy, is included in the BTI’s definition of political transformation and examined through questions specifically dealing with the state’s monopoly on the use of force and basic administrative structures. It also entails an evaluation of the rule of law, including the separation of powers and the prosecution of office abuse. The BTI puts a special emphasis on the evaluation of democratic consolidation. It assesses the quality of representation with regard to the party system and interest groups, and also measures social capital and the approval of democratic norms and procedures.

The state of economic transformation is measured in terms of seven criteria, which are based on a total of 14 indicators. The BTI’s concept of a market economy includes not only aspects such as economic performance, regulatory or competition policy, and property rights; it also contains elements of social inclusion, such as social safety nets, equality of opportunity and sustainability. In BTI terms, comprehensive development not only aims at economic growth, but also requires successful poverty alleviation and the freedom of action and choice for as many citizens as possible.

The Governance Index is comprised of five criteria, which are based on a total of 20 indicators. It focuses on how effectively policymakers facilitate and steer development and transformation processes. By examining and evaluating decision-makers’ reform policies, the BTI sheds light on those factors determining success and failure on the way to democracy and a market economy. Successful governance implies that governments are consistent in pursuing their goals and use their resources wisely and effectively. It further implies that decision-makers cultivate the broadest possible consensus for their transformation goals and work reliably with external supporters and neighboring states.

Governance performance is weighted with the level of difficulty, which is derived from three qualitative and three quantitative indicators. It reflects the observation that each country’s quality of transformation is influenced by structural constraints. In this way, difficult conditions and the scarcity of resources in a given country are factored in. With its focus on political actors’ steering capacity, the BTI is the only index to analyze and compare governance performance with self-collected data.

Country selection

Because the BTI focuses in its analysis on transformation toward democracy under the rule of law and a market economy anchored in principles of social justice, it excludes countries that might be considered long-consolidated democratic systems and in which economic development can be regarded as well-advanced. In the absence of a clearly defined “threshold of consolidation,” the Transformation Index therefore excludes all countries that were members of the Organisation for Economic Co-operation and Development (OECD) by the year 1989. This is not to suggest that these countries have achieved a static end-state. Rather, it reflects the observation that the reform agenda and the political priorities in a consolidated democracy with a highly developed market economy differ markedly from those that emerge during transformation.

Small states with fewer than 1 million residents are also not examined in the BTI. Exceptions to this rule have been made, however, to allow for the inclusion of particularly interesting examples of development and transformation: Bhutan, Djibouti and Montenegro.

Since 2003, the number of countries surveyed has increased from 116 to 137. They are divided into seven regional groups: Latin America and the Caribbean (22 countries), West and Central Africa (22), Southern and Eastern Africa (22), Middle East and North Africa (19), East-Central and Southeast Europe (17), Post-Soviet Eurasia (13) and Asia and Oceania (22).

Measurement and review process

The Transformation Index is based on a qualitative expert survey in which written assessments are translated into numerical ratings and examined in a multistage review process so as to make them comparable both within and across regions. This method enables those factors of political and economic development that elude purely quantitative assessments to be captured in the experts’ qualitative appraisals. This method presents significant advantages as it allows, for example, a distinction to be made between rights granted de jure and their de facto implementation. In addition, statements can be made about the magnitude of social capital and the extent to which civil society is integrated into political decision-making processes. Furthermore, the quality of governance can be assessed and compared. Facts such as constitutional provisions or official economic data can be interpreted and weighed in context. The resulting country assessments render fully transparent and verifiable the reasons behind each of the BTI’s 7,124 individual scores.

Nevertheless, this type of qualitative expert survey will always contain a degree of subjectivity. The BTI survey process takes this into account during the preparation of reports and evaluations, as well as during the review of the data. It is designed to minimize subjective factors as far as possible throughout the process. The process of country assessment has both a qualitative and quantitative component, in each case performed by two country experts. As a rule, one foreign and one local expert are involved in the evaluation process; this ensures that both external and internal perspectives are taken into account in the course of assessment, and helps counteract subjective influence. In total, 269 experts from leading research institutions around the world contributed to the production of the country reports.

A standardized codebook serves as the foundation of the survey process, providing a single reference framework for the experts when answering the questions. The first expert drafts a detailed report on the basis of the criteria outlined in the codebook, referencing the qualitative indicators associated with each criterion. The second expert reviews, comments on and adds to this country report. In addition, in the course of answering 11 of the 49 questions (indicators), the country experts are required to draw upon a set of quantitative indicators (ranging from inflation rates to education spending). Independently of one another, the two country experts translate the assessment into a numerical rating on a scale of one (the lowest value) to 10 (highest value), structured by four levels of score-based categories contained in the codebook. In this way, countries are evaluated on the basis of whether and to what extent they comply with the specified rating levels and fulfill the BTI criteria.

In order to ensure the validity, reliability and comparability of the assessment, each individual score undergoes a multistep process of review by the country experts, the regional coordinators, the project team and the BTI board. The scores and responses provided by the experts for each of the 49 indicators are initially reviewed by regional coordinators, who examine the content to ensure it is both complete and consistent. The regional coordinators, all political scientists with expertise in comparative studies, participate in each step of the report-creation process and apply their regional expertise to ensure the high quality of the country reports. They subsequently perform an intraregional calibration of their countries’ scores, and then they join with the project team to carry out an interregional score calibration for all 137 countries, this time checking for across-the-board comparability and viability. Finally, all scores are discussed once again by the BTI board before being adopted. The BTI board, a panel of scholars and practitioners with long-term experience in the field of development and transformation, provides the project with ongoing support and advice.

Index aggregation

The Status Index is aggregated by calculating the average of the total scores given for the dimensions of political (democracy status) and economic (economy status) transformation. The state of transformation in each analytic dimension is equivalent to the average of the scores of the associated criteria. Criterion scores are, in turn, based on the average scores of the equally weighted indicators that comprise the criterion.

Combining the two analytical dimensions into a Status Index follows the normative premise of the BTI, under which transformation is always conceived of as a comprehensive transition toward democracy and a market-economic system.

The Governance Index is aggregated by calculating the average of scores given for the governance performance criteria, which is then offset against the assigned level of difficulty.

Democracies and autocracies

The indicators on the state of political transformation are also used in determining whether a country is classified as a democracy or autocracy. This analysis comprises more than just whether sufficiently free and fair elections are held. In accordance with the Transformation Index’s comprehensive concept of democracy, seven threshold values marking minimum requirements are considered. The country is classified as an autocracy if even one score falls short of the relevant threshold. Thus, the classification of a country as an autocracy is not determined by the aggregate political transformation score, but rather by the thresholds listed below. A moderate autocracy, such as Singapore, which fails to meet all minimum requirements to be classified as a democracy, can and does score higher in the BTI’s Democracy Index than a highly defective democracy, such as Lebanon.

Failing states are considered autocracies. They are defined as countries in which the state’s monopoly on the use of force and basic administrative structures are lacking to such an extent that the government is severely limited in its capacity to act.