Open Budget Survey methodology

Implementing the Open Budget Survey 2017 and Calculating the Open Budget Index and Other Scores
The Open Budget Survey (OBS) assesses the three components of a budget accountability system: public availability of budget information; opportunities for the public to participate in the budget process; and the role and effectiveness of formal oversight institutions, including the legislature and the national audit office (referred to here as the “supreme audit institution”). The majority of the survey questions assess what occurs in practice, rather than what is required by law.

The questions included in the OBS are based on generally accepted good practice for public financial management. For example, the survey assesses the public availability of budget information by considering the timely release and contents of eight key budget documents that all countries should issue at different points in the budget process. Many of these criteria are drawn from those developed by multilateral organizations, such as the International Monetary Fund’s (IMF) Code of Good Practices on Fiscal Transparency, the Public Expenditure and Finance Accountability initiative (whose secretariat is hosted by the World Bank), the Organisation for Economic Co-operation and Development’s (OECD) Best Practices for Fiscal Transparency, and the International Organisation of Supreme Audit Institutions’ Lima Declaration of Guidelines on Auditing Precepts. Similarly, the criteria used to assess opportunities for the public to participate in the budget process are based on the Global Initiative for Fiscal Transparency’s Principles for Public Participation in Fiscal Policy. The strength of such guidelines lies in their universal applicability to different budget systems around the world, including those of countries at different income levels.

The OBS 2017 is the culmination of a collaborative research process in which the International Budget Partnership (IBP) worked with civil society partners in 115 countries – encompassing all regions of the world and all income levels – over the past 18 months. This is the sixth round of the OBS, which is typically conducted biennially. Earlier rounds were completed in 2006, 2008, 2010, 2012, and 2015.

The Open Budget Questionnaire
The results for each country in the OBS 2017 are based on a questionnaire, comprising 145 scored questions, that is completed by researchers typically based in the surveyed country.[1] Almost all of the researchers responsible for completing the questionnaire are from civil society organizations or academic institutions. Although the mandates and areas of interest of the research groups vary widely, all have a common interest in promoting transparent and responsive budgeting practices in their countries. Most of the researchers belong to organizations with a significant focus on budget issues.

Most of the survey questions require researchers to choose from five responses. Responses “a” or “b” describe best or good practice; with “a” indicating that the standard is fully met or exceeded, and “b” indicating the basic elements of the standard have been met or largely met. Response “c” corresponds to minimal efforts to attain the relevant standard, while “d” indicates that the standard is not met at all. An “e” response indicates that the standard is not applicable, for example, when an OECD country is asked about the foreign aid it receives. Certain questions, however, have only three possible responses: “a” (standard met), “b” (standard not met), or “c” (not applicable).

Once completed, the questionnaire responses are quantified. For the questions with five response options: an “a” receives a numeric score of 100, a “b” receives 67, “c” receives 33, and “d” receives 0. Questions receiving an “e” are not included in the country’s aggregated scores. For the questions with three response options: “a” receives 100, “b” receives 0, and “c” responses are not included in the aggregated score.

The Research Process
For the OBS 2017 researchers collected data on their country between September 2016 and the end of December 2016. Thus, the OBS 2017 assesses only those events, activities, or developments that should have occurred up to 31 December 2016; any actions occurring after this date are not accounted for in the 2017 survey results.

All responses to the OBS questions are supported by evidence. This includes citations from budget documents; the country’s laws; or interviews with government officials, legislators, or experts on the country’s budget process. Throughout the research process, IBP staff assisted the researchers in following the survey methodology, particularly the guidelines for answering survey questions.[2]

Upon completion, IBP staff analyzed and discussed each questionnaire with the individual researchers over a three- to six-month period. IBP sought to ensure that all questions were answered in a manner that was internally consistent within each country, and consistent across all survey countries. The answers were also cross-checked against published budget documents and reports on fiscal transparency issued by international institutions, such as the IMF, the World Bank, and the OECD.

Each questionnaire was then reviewed by an anonymous peer reviewer who has substantial working knowledge of the budget systems in the relevant country. The peer reviewers, who were not associated with the government of the country they reviewed, were identified through professional contacts and various other channels.

IBP also invited the governments of nearly all survey countries to comment on the draft OBS results. The decision to invite a government to comment on the draft results was made after consulting with the relevant research organization responsible for the survey. IBP made a major effort to encourage governments to comment on the draft results; many governments that did not initially respond to IBP letters were contacted on multiple occasions. Of the 112 governments that IBP contacted, 76 commented on the OBS 2017 results for their country. These comments can be seen in their entirety in the relevant questionnaires at www.openbudgetsurvey.org.

IBP reviewed peer reviewer comments to ensure that they were consistent with the study’s methodology. Any peer reviewer comments that were inconsistent were removed, and the remaining comments then were shared with researchers. Researchers responded to comments from peer reviewers and their government, if applicable, and IBP refereed any conflicting answers in order to ensure the consistency across countries in selecting answers.

The Open Budget Index
The Open Budget Index (OBI) assigns each country a score from 0 to 100 based on the simple average of the numerical value of each of the responses to the 109 questions in the questionnaire that assess the public availability of budget information. A country’s OBI score measures the extent to which it makes the eight key budget documents available to the public on the relevant government website in a timely manner and the comprehensiveness of publicly available budget information.

Measures for Oversight Institutions and Public Participation
Forty of the survey questions that are not used to calculate the OBI assess the opportunities for public engagement during the budget process and the oversight capacity of independent fiscal institutions, legislatures, and supreme audit institutions. To measure the extent to which governments include the public in budget decision making and monitoring, as well as the role of the legislature and supreme audit institution in the budget process, the responses to the questions pertaining to each are averaged. Thus each area is given a separate score. IBP also collects information on the role of independent fiscal institutions (IFIs) – independent, nonpartisan institutions, normally attached to the executive or legislature that make fiscal forecasts and estimate the cost of policies. However, IBP does not calculate a separate score for the role of IFIs.

Weighting the Relative Importance of Key Budget Documents and Implications on Scores
As mentioned above, each country’s OBI 2017 score is calculated from a subset of 109 survey questions. Though each of the eight key budget documents assessed may have a different number of questions related to it, the OBI score is a simple average of the responses to all 109 questions. In calculating the OBI scores, no method of explicit weighting is used.

Though using a simple average is clear, it implicitly gives more weight to certain budget documents than others. In particular, 54 of the 109 OBI questions assess the public availability and comprehensiveness of the Executive’s Budget Proposal, and thus are key determinants of a country’s overall OBI score. In contrast, the Citizens Budget and the Enacted Budget are the focus of only four and six questions, respectively.

This implicit weighting is justified. From a civil society perspective, the Executive’s Budget Proposal is the most important budget document, as it lays out the government’s budget policy objectives and plans for the upcoming year. It typically provides details on government fiscal policies not available in any other document. Access to this information is critical for civil society to understand and influence the budget prior its approval, and to have as a resource throughout the year.

''[1] The 145 scored questions included in the questionnaire include 109 questions that assess the public availability of budget information, 18 questions that assess opportunities for the public to participate in the budget process, and 18 questions that assess the role and effectiveness of the legislature and the supreme audit institution. The questionnaire also includes an additional 83 questions that are not used to calculate individual scores but help to complete the OBS research by collecting background information on key budget documents and exploring different characteristics of a country’s public finance management.''

[2] See the Guide to the 2017 Open Budget Questionnaire