What Does Benchmarking Infrastructure Development 2020 Measure?

Benchmarking Infrastructure Development 2020 assesses the quality of regulatory frameworks for preparation, procurement, and management of large infrastructure projects. To do this, it relies on standardized questionnaires designed to collect data for further comparison of each country’s regulatory framework with internationally recognized good practices. As announced earlier in the report, Benchmarking Infrastructure Development 2020 expands its thematic coverage to include the assessment of TPI regulatory frameworks in a subset of 40 pilot economies. Consequently, the team underwent a similar survey development process as was used for the PPP survey. Identification of internationally recognized good practices for the development of large infrastructure projects, both through PPP and TPI modalities, relied on research of the relevant literature (see Bibliography). The Expert Consultative Group (ECG), which includes seasoned PPP and TPI professionals, academics, and individuals from the private sector, was consulted at various stages. Initial consultations with key ECG members allowed the team to produce an initial meaningful version of the new TPI survey. A wider virtual review with all ECG members was performed to confirm the contents of the new TPI survey and to refine the PPP survey.

---

Thematic Coverage

Benchmarking Infrastructure Development 2020 serves as an indicator of the regulatory quality to develop infrastructure projects in those sectors where intervention of public entities is still necessary to achieve the optimal service level. These sectors vary by country, level of economic development, and cultural and political context. However, even within those sectors the intensity of the government’s involvement can vary. In a PPP, the government contracts out a whole infrastructure project throughout its lifecycle, including design, construction, and operation, to the private sector, thereby transferring a significant portion of the associated risks. In a TPI, the government retains the responsibility for the design, construction, and operation of projects with the possibility of sub-contracting certain aspects of technical execution at each stage to the private sector without transferring the core risks.

To capture these two main alternatives in the current edition of the report, two separate surveys were administered: (a) a PPP survey and (b) a TPI survey. While both surveys have a common structure for assessing the preparation, procurement, and contract management phases of the project cycle, they differ in the special modules. For the PPPs, management of unsolicited proposals (USPs) is assessed in a special module, while for the TPIs, operation and maintenance of infrastructure assets after construction is covered. In addition, the TPI survey expands the preparation stage by including questions about the public investment management system and the overall planning and budgeting process for infrastructure investments. Both surveys use common case study assumptions (a national highway transport project) to ensure cross-country comparability and aggregability. The differences between the two include project size in the case study assumptions and the number of economies covered: the PPP survey covers 140 economies while the TPI survey is only a pilot for 40 economies.

---

The Pilot De Facto Questions

New to the 2020 edition, pilot de facto questions were included in both the PPP and TPI surveys. Both questionnaires include the de jure and de facto questions, which separately capture regulatory aspects and implementation of those regulations in practice. These two types of data points allow for a comprehensive assessment of the legal rules in place and their actual enforcement:

• The de jure data points are strictly regulatory-based and don’t capture practice, unless reflected in the legal framework. This type of data assesses compliance of regulatory frameworks with internationally recognized good practices in terms of efficiency, transparency, and accountability;
• The de facto data points are subjective in nature and are based on opinion. They try to assess the actual implementation of the legal requirements identifying those that are de facto respected in practice.

Analysis and interpretation of the de facto data must be made cautiously because responses to this type of question are difficult to measure precisely other than through some proxy measures. While effort was made to administer practice-based questions to as many and as broad a spectrum of stakeholders as possible, a sample of contributors who provided their feedback is not necessarily representative of the targeted population in each economy. Moreover, the answers involve an element of a judgment and are subjective in nature. The reported answers show the median value of all the responses received.

---

Geographic Coverage

The Benchmarking Infrastructure Development 2020 PPP survey includes 140 economies, consisting of the 135 economies covered in the Procuring Infrastructure PPPs 2018 report and five additional economies: the Islamic Republic of Iran, Israel, Samoa, Uzbekistan, and Vanuatu. The geographical distribution of the 140 economies is as follows: 30 Organisation for Economic Co-operation and Development (OECD) high-income economies, 34 countries in the Sub-Saharan Africa (SSA) region, 17 economies in the East Asia and Pacific (EAP) region, 22 economies in the Europe and Central Asia (ECA) region, 18 economies in the Latin America and the Caribbean (LAC) region, 13 economies in the Middle East and North Africa (MENA) region, and six economies in the South Asia (SAR) region.

The Benchmarking Infrastructure Development 2020 TPI survey includes a pilot of 40 economies, which were selected based on the following criteria: (i) the sample is representative of the global regional distribution; (ii) within each region, all income groups must be present; (iii) within each regional and income group, the economies with the largest population size are selected. Certain corrections to this decision-making process were made to avoid including economies that proved to be rather challenging for data collection during the Procuring Infrastructure PPPs 2018 report. The geographical distribution of the 40 economies is as follows: six OECD high-income economies, 10 economies in the SSA region, seven economies in the ECA and LAC regions each, five economies in the EAP region, four economies in the MENA region, and one economy in the SAR.

Figure. Economies covered in the Benchmarking Infrastructure Development 2020 Report

---

How the Data are Collected?

Survey Contributors

The standardized questionnaires for the PPP and TPI surveys were distributed to approximately 20,000 contributors in the 140 and 40 economies included in the PPP and TPI survey, respectively. Data collection, analysis, and validation spanned nine months and ended in March 2020. Once the initial data collection was completed, a series of follow-up Q&A sessions via conference calls and emails was performed to address and resolve any contradictions or discrepancies in the data provided by various contributors. The preliminary data were finalized and then shared with the World Bank Group’s Country Management Units (CMUs) for final validation with each economy’s respective governments.

The standardized questionnaires were distributed to practitioners who have knowledge and expertise related to PPPs and TPIs. Respondents were selected based on their experience and availability to contribute meaningfully to each questionnaire. The report’s main contributors were law firms that have experience advising clients on PPP and TPI transactions, public officials involved in establishing and implementing PPP and TPI policy, chambers of commerce, consultants and academics knowledgeable in the topics of PPPs and TPIs, and others.

The following sources were utilized to identify the appropriate pool of contributors:

• International guides, such as Chambers and Partners guides, the International Financial Law Review (IFLR), The Legal 500, Martindale-Hubbell, HG Lawyers’ Global Directory, Who’s Who Legal directory, Lexadin, and country-specific legal directories. The guides permitted the identification of the leading providers of legal services, including their specializations, in each economy;
• Major international law, accounting, and consulting firms that have large and well-connected global networks through their partner groups or foreign offices;
• Members of the American Bar Association, country bar associations, chambers of commerce, and other legal membership organizations;
• Government organizations that formulate PPP and TPI policy in each economy and undertake individual projects, including ministries of finance, ministries of transport, PPP procuring authorities, and PPP units; and
• Secondary resources and professional service providers recommended by the World Bank staff as well as those found through embassy websites and business chambers.

---

Scoring and Methodological Changes

The scoring methodology for the Benchmarking Infrastructure Development 2020 PPP data was mostly inherited from the Procuring Infrastructure PPPs 2018 report. As in the previous editions, scores for the PPP survey are aggregated for each thematic area: preparation, procurement, contract management, and USPs. Only areas recognized as international good practices were scored. The current scoring methodology allocates the same weight to all benchmarks. For the TPI survey, a scoring methodology has been developed following the same principles. TPI scores are likewise aggregated for each thematic area: preparation, procurement, contract management, and asset management.

The possible scores range from 0 to 100. Economies with the highest scores, nearing 100, are considered to have PPP or TPI frameworks that are closely aligned with international good practices in each thematic area. On the contrary, economies with the scores at the bottom (nearing 0) have considerable room for improvement. The country tables at the end of the report contain only the scores at the thematic area level. However, all information collected during the implementation of the PPP and the TPI surveys is publicly available on this website.

It is important to note that scores cannot be compared across the different editions of this initiative. There were significant changes in the scoring methodology in Benchmarking Infrastructure Development 2020 compared with Benchmarking Infrastructure PPPs 2018. First, the survey instrument for PPPs has changed between the 2018 and 2020 editions. During the refinement of the PPP survey, 63 new questions were added, 12 questions were dropped and nine were reformulated. This was mainly done to ensure consistency with the new TPI questionnaire. Furthermore, Benchmarking Infrastructure Development 2020 included a pilot of a de facto question for each legal question. This had significant implications in the scoring methodology because the 2020 edition no longer considers as valid answers to regulatory questions that are based on experience or established traditions and practice. To receive a full point for the scored legal question there must be a valid provision in the regulatory framework that addresses the issue. The scoring methodologies for both PPP and TPI surveys, as well as the details of the methodological changes with respect to the 2018 edition are available on this website.

---

The Scope and Limitations of the Assessment

Understanding the scope of the data utilized in this report is important to interpreting the results. The data have both strong and weak sides that readers should bear in mind.

Firstly, procurement of both PPPs and TPIs can be carried out at the different levels of government and sometimes along sectoral lines. While the report recognizes associated complexities, because of the limited resources, it examines only procuring authorities at either the national or federal level. However, certain exceptions were made for some economies due to their specific constitutional and/or administrative configuration. For both PPP and TPI surveys, Australia is represented by the regulatory framework of the State of the New South Wales. Additionally, in the PPP survey the assessment of Bosnia and Herzegovina is based on the Sarajevo Canton; of the United Arab Emirates, on the Emirate of Dubai; and of the United States, on the Commonwealth of Virginia. This approach was adopted to address the fact that the federal governments in these economies have a limited authority regarding PPPs in infrastructure. This limitation, along with their particular constitutional arrangements, makes it unfeasible to evaluate the development of PPPs at the national or federal level. Specifics of each of these four economies are discussed in the description of corresponding regulatory frameworks available on this website.

Secondly, the regulatory framework to procure both PPPs and TPIs may differ across sectors. However, it is not feasible to design a survey that covers different regulations for all possible sectors and types of PPP and TPI projects. While most of the answers to both surveys may apply to many or all sectors, contributors were referred to a specific case study for the transportation sector (a highway project) to ensure cross-sectoral and cross-country comparability.

Thirdly, Benchmarking Infrastructure Development 2020 follows the World Bank definition of PPPs and applies this definition irrespective of the specific terminology used in a country or jurisdiction. It includes such modalities as concessions, build (rehabilitate)-own-operate, build (rehabilitate)-own-transfer, build (rehabilitate)-own-operate-transfer, and similar contract modalities under which an infrastructure asset is built (expanded, reconstructed, or upgraded), owned, and operated by a private partner, and transferred or leased back to a public partner upon expiration of a contract term. In the following economies the authors detected two clearly separate regulatory regimes for concessions (sometimes defined as user-pay arrangements) and PPPs (sometimes defined as government-pays arrangements): Argentina, Brazil, Costa Rica, France, Mauritius, Niger, the Russian Federation, Senegal, and Togo. For these economies, the concession regime was evaluated and scored separately, but the findings in this report refer exclusively to the PPP regime for consistency. Information regarding concession regimes in these economies is available on this website.

Fourthly, the Benchmarking Infrastructure 2020 survey (both PPPs and TPI) uses a broad definition for the regulatory framework and includes any applicable legal texts and other binding documents (such as policies, standardized transaction documents, and contracts), as well as judicial decisions and administrative precedents regarding procurement of large infrastructure projects. This broad understanding of the regulatory framework helps prevent, to the extent possible, any bias towards a particular legal system (civil law countries versus common law countries) or formal configuration of the regulatory framework.

Fifthly, the pilot de facto or practice-based questions aim to capture the extent to which regulatory frameworks in each economy are respected in practice. These types of questions represent the contributors’ perceptions, judgments, and opinions, based on their exposure to and experience with PPP and TPI projects in their home economies. However, for most economies the pool of contributors that provided feedback may not always constitute a representative sample. It is important to recognize that de facto questions are at a pilot stage and results must be interpreted with caution, bearing in mind limitations of this type of assessment.

It is also important to note that Benchmarking Infrastructure Development 2020 does not assess individual PPP or TPI projects and contracts on a regular basis or treat them as a source of information. For regulatory-based questions, the actual laws and regulations in place as of the cut-off date are used as the main information source. For practice-based questions, opinions of the contributors are the main information source.

Moreover, regulatory-based questions do not cover all regulatory challenges related to PPP and TPI project cycles. In particular, they do not consider the capacity of implementing agencies as demonstrated by staffing numbers, staff competence levels, professionalism, and experience, and macroeconomic stability or the prevalence of the corruption in each economy. Additionally, Benchmarking Infrastructure Development 2020 does not capture all the related elements and cannot be considered a complete and full assessment for a straightforward classification of economies based on their capacity to manage the PPP or TPI processes.

Furthermore, the relevant legal and regulatory provisions noted in the report reflect a moment in time. Thus, readers should note that the legal situations may have changed since then. Specifically, the cut-off date for the Benchmarking Infrastructure Development 2020 report was June 30, 2019. Hence, any regulatory reforms or any changes in practice that occurred after that date are not taken into consideration in this edition of the report.

Finally, the report and the data points are meant to be actionable by lawmakers and governments. Thus, the report highlights the relevant regulatory aspects of the PPP and TPI legal frameworks in the hope of giving the governments and parliaments of the respective economies an opportunity to have a critical look at possible areas for improvement within their PPP and TPI procurement frameworks and help them in formulating the direction of change that might be needed going forward. However, given the limitations discussed, the report is not meant to be prescriptive and does not attempt to rank economies by their capability to procure PPP or TPI projects.