Introduction and Overview of the Open Budget Survey

The first step for researchers in all countries included in the Open Budget Survey is to complete the Open Budget Questionnaire. The data collected through the Open Budget Questionnaire will form the basis of the Open Budget Index and Open Budget Survey results, which will be released worldwide at the end of the research period. The Open Budget Questionnaire has two broad purposes. First, it is designed to assist independent civil society researchers in understanding select international good practice benchmarks for budget transparency and accountability and applying these standards to the practices the researchers observe in their countries. Second, by providing a common methodology for investigating budget transparency and accountability, the questionnaire is intended to link civil society research efforts on these issues across countries. By applying this standard methodology to research in all countries, comparisons across countries are possible. By examining each country’s performance against internationally accepted standards within a cross-national context, the Open Budget Questionnaire draws global attention to the importance of open and accountable government budgets.

Civil society budget groups have a unique interest in promoting public access to budget information during all phases of the budget cycle. The availability of comprehensive, accessible, useful, and timely budget information is essential for civil society to participate effectively in the budget process. The questionnaire is specifically designed to help researchers identify what and how much information is publicly available during each of the four stages of the budget process (i.e., formulation, approval, implementation, and audit), whether it is timely and accessible, and whether there are any information gaps.

This focus on public access to information, as well as opportunities to participate in budget processes, is what makes the questionnaire unique among assessments of government transparency and accountability. However, the Open Budget Questionnaire addresses only a limited set of topics related to the larger issues of effective public financial management, public participation, and accountability. Thus, the Open Budget Questionnaire is intended to complement rather than replace other research related to these topics. The IBP strongly encourages researchers to develop their own country-specific research as a way to highlight and draw attention to important issues in their country.

It is important to keep in mind that there are some areas the questionnaire does not examine at all, including the legal and administrative framework for public expenditure management within countries, or the transparency and accountability of subnational levels of government. These are both highly important matters, but are difficult to capture across different types of budget systems in a way that allows for comparisons between countries.

Important Principles Related to Budget Transparency and Accountability
Notwithstanding its limitations, the questionnaire does attempt to capture and make operational some important basic principles and standards, or norms, related to transparent and accountable budgeting systems and practices. These include:

Prior authorization: The legislature should authorize measures involving expenditure, revenue collection, and debt prior to the executive taking action.

Unity: All proposed annual expenditures, revenues, and other government financial activities should be presented to the legislature in one consolidated presentation, the Executive’s Budget Proposal. (Although in practice in many of the countries studied, this consolidated presentation takes the form of a primary document, with several supporting documents.)

Comprehensiveness: Not only should the government disclose its financial activities in its annual budget proposal but it also should issue documents at other times during the budget year to provide a comprehensive, updated picture of the government’s actions.

Periodicity: The executive should adhere to regular deadlines in an annual calendar for drafting the budget, presenting it to the legislature for debate and enactment, executing the budget, and presenting its final accounts to the public and the Supreme Audit Institution (SAI). Timely and regular reporting on each of these four phases of the budget process is essential. It is also important for the legislature and SAI to perform their critical roles in a timely manner throughout the budget year, especially when it comes to debating and approving the annual budget and reviewing the final accounts.

Specificity: The description of every budget item should result in a clear overview of the government’s expenditure plans. The descriptions and figures provided for budget items should not be so highly aggregated (presented as grand totals) that they do not give a clear picture of the government’s intentions, and this information should be presented according to internationally accepted classification systems to avoid obfuscation.

Legality: All expenditures and activities should be in keeping with the law. Where the law does not adhere to or promote basic principles of transparency and accountability, civil society should make recommendations to amend it.

User-friendly structure: A country’s budget documents are its government’s key accountability documents: they should be presented in a manner that is manageable and understandable by the full spectrum of audiences, and their varying levels of technical capacity, including the legislature, civil servants, and public.

Publicity: All of a country’s citizens should have the right, as individuals or in association in the form of civil society organizations (CSOs), to make and express judgments on the country’s budget. This requires that budget documents be widely available. This is especially important for the Executive’s Budget Proposal, which should be available to citizens before it is adopted by the legislature.

Means and ends: Clear links should be made between the government’s policies, plans, budget inputs and outputs, and desired outcomes. The explanation accompanying the budget should communicate clearly its aims in terms of inputs, objectives to be achieved, and measurable results.

Other Useful Tools for Budget Transparency Research
We suggest that researchers visit the Open Budget Initiative on the [http://www.internationalbudget.org/opening-budgets/open-budget-initiative/ IBP website) for other useful links and tools that can help focus additional budget transparency research they might be considering.

In addition, the International Monetary Fund (IMF), the World Bank, and the Organization for Economic Cooperation and Development (OECD) have developed comprehensive questionnaires and diagnostic tools for examining fiscal transparency, public expenditure management, and budget practices and procedures. (See, for example, www.imf.org, pefa.org, and www.oecd.org.

Taken together, these various instruments highlight the breadth and complexity of budget transparency. The Open Budget Questionnaire is intended to supplement these other efforts, primarily emphasizing matters related to public access to information.

General Information about the Questionnaire
The Open Budget Questionnaire is designed to be completed by an independent civil society researcher, or a team of such researchers, in order to produce a single set of responses for each participating country. The questionnaire seeks to determine whether a country’s government provides certain types of information to the public, and whether certain activities occur during the budget process. Answers to the questions must be based on empirical research, and researchers are required to provide evidence as to why they have selected a particular answer. This evidence can include references to specific documents, in their entirety or sections/chapters/pages, or interviews with public officials. Each question will provide a “Comment” box in which researchers can summarize supporting evidence and explain why a particular answer option has been selected over another. Please see the discussion that follows on citations and references.

Similar to earlier rounds of the survey, all researchers will use an online survey management tool to complete the 2019 Open Budget Questionnaire. The survey tool will help integrate and expedite the collection, discussion, and transfer of raw data among researchers working in 117 countries worldwide. The online survey tool will not only help expedite the Open Budget Questionnaire data collection process, but will also make it easier for researchers, activists, and the IBP to highlight the Survey results.

Once the questionnaire is completed, it will be reviewed by an anonymous peer reviewer who has substantial knowledge of the particular country’s budget system to help ensure the validity and objectivity of the results produced by the researcher or research team. With the researcher’s permission, a second set of comments provided by the government will be solicited.

It is important for researchers to note the following:


 * The questionnaire focuses on the activities of central government and does not address the role of state and local governments. Because the questionnaire was designed to measure budget transparency and accountability in a way that would allow for cross-country comparisons, it is not well suited to investigate intergovernmental relationships, the nature of which can vary significantly from country to country. Researchers interested in examining intergovernmental fiscal relations in their country may want to consider developing supplementary questions to capture these activities.


 * For the most part, the questionnaire seeks to discover what occurs in practice, rather than what the law requires. The small number of questions that do refer to legal provisions state this clearly in the question. Researchers should answer the vast majority of questions based on the actual practice in their country. This focus on what occurs in practice recognizes that, in some cases, well-intentioned budget laws are not actually implemented and thus have little impact on the budget process. At the same time, this focus should not be interpreted as minimizing the importance of codifying certain aspects of the budget process; legal requirements for openness and transparency are an essential part of guaranteeing sustained transparent, responsive, accountable, efficient, and effective budgeting.


 * Unless stated otherwise, the questionnaire is concerned with those budget documents that are made available to the public. Please answer the 142 questions in Sections Two through Five of the questionnaire based on publicly available budget documents.


 * Unless stated otherwise, the questions are intended to apply to the most recently completed part of the budget process covered by the question. As a result, the questionnaire will likely apply to budgets from several years. Please note that for countries included in the 2017 Open Budget Survey, documents that were used to respond to the 2017 Open Budget Survey cannot be used to respond to the 2019 Open Budget Questionnaire.

Structure of the Questionnaire
The Open Budget Questionnaire is composed of five sections. The first section (Section 1: Public Availability of Key Budget Documents) contains a series of questions that allow researchers to examine and map the availability and dissemination of a country’s eight key budget documents. This first section requires the researcher to identify each of the key budget documents that a country issues (or fails to issue) during each of the four phases of the budget process, provide details on document availability and timely publication, and note any laws governing the country’s budget process. The researcher will then use this information to answer the majority of the 142 questions in the remaining sections of the questionnaire (sections 2-5). The 142 questions are grouped into four sections: Section 2: Comprehensiveness of the Executive’s Budget Proposal; Section 3: Comprehensiveness of Other Key Budget Documents; Section 4: Role and Effectiveness of Oversight Institutions in the Budget Process; and Section 5: Public Engagement in the Budget Process.

The second section of the questionnaire (Comprehensiveness of the Executive’s Budget Proposal) seeks to assess the information provided in the Executive’s Budget Proposal and its supporting documentation. The Executive’s Budget Proposal merits significant attention because it is the blueprint for how the government will raise and spend funds to realize its economic and social policy goals. As such, it is frequently subject to the most scrutiny and debate during the course of the budget year.

How closely the Executive’s Budget Proposal adheres to good practices for presenting budget information is generally carried through and reflected in the other budget documents that are issued throughout the process. For example, if a government adopts the good practice of providing in the Executive’s Budget Proposal a highly detailed classification of its expenditures that include functional, economic, and administrative breakdowns, it is much more likely to be in a position to carry this practice through to its reporting in other phases of the budget cycle, such as that in its In-Year and Year-End Reports.

The third section of the questionnaire (Comprehensiveness of Other Key Budget Documents) assesses the information provided in the other seven key budget documents that should be published throughout the budget process. The questions in this section parallel many of those asked in Section 2, but rather than evaluate the Executive’s Budget Proposal, the questions focus on the Pre-Budget Statement, the Enacted Budget, the Citizens Budget, In-Year Reports, the Mid-Year Review, the Year-End Report, and the Audit Report.

The fourth section of the questionnaire (Role and Effectiveness of the Oversight Institutions in the Budget Process) assesses the effectiveness of the legislature and Supreme Audit Institution (SAI) in overseeing the budget process. In particular, the indicators assess whether the broader architecture of the budget system includes adequate checks and balances for ensuring integrity and accountability in the use of public resources.

The fifth section (Public Engagement in the Budget Process) focuses on opportunities for public engagement during the budget process—a necessary complement to budget transparency. For access to budget information to contribute to improvements in budget policies, execution, and outcomes, it must be coupled with opportunities for the public to use this information to participate meaningfully in budget decisions and oversight. The final section of the questionnaire looks at the processes, mechanisms, and practices the executive, legislature, and Supreme Audit Institution have in place to ensure that the public is included in the formulation, discussion/approval, execution, and auditing of their country’s national budget. The indicators are aligned with the Global Initiative for Fiscal Transparency’s new principles on public participation, which now serve as the basis for widely accepted norms on public participation in national budget processes.

Questionnaire Breakdown by Key Budget Document
As noted above, the majority of the questionnaire is broken down by budget document type. Section 2 focuses on the Executive’s Budget Proposal and Section 3 covers the remaining seven budget documents. A detailed description of each of the eight key budget documents can be found in the questionnaire guidelines, preceding the series of questions related to that particular document. The box below shows the range of questions that correspond to each budget document.

Please note that researchers should not use information contained in a document that is issued during one phase of the budget cycle to answer questions related to another budget document. This is because the questionnaire is specifically intended to evaluate the information available to the public during each of the various phases of the budget year. For example, when answering questions on the Executive’s Budget Proposal, researchers should not use information from documents that are issued during other stages of the budget cycle (e.g., In-Year, Year-End, or Audit Reports).

Structure of Responses to the Questions
The majority of questions have four possible responses, as well as “not applicable/other.” There are a handful of yes/no questions, with option “c” being “not applicable/other.” In general, the first response (letter “a”) is the most positive answer, reflecting best practice for the subject matter of that question. The second response (letter “b”) is intended to reflect good practice. A response of “c” reflects poor or weak practice, while the fourth response (letter “d”) is the most negative. Detailed explanations of appropriate answer choices are provided individually in the guidelines following each of the Survey’s 142 questions.

For the purposes of numeric scoring of the responses, an “a” response is scored as 100 percent, “b” is scored as 67 percent, “c” is scored as 33 percent, and “d” is scored as 0 percent. Responses of “e” (not applicable/other) are not considered in the scoring.

As mentioned above, the “a” responses to the questions attempt to capture “best practice” when it comes to transparent and accountable budgeting. For the most part, these best practices regarding the timeliness and content of specific budget documents are drawn from guidelines found in the OECD’s Best Practices for Budget Transparency; the IMF's Manual on Fiscal Transparency (2007) and Fiscal Transparency Handbook (2018); the IMF’s 2001 Government Finance Statistics Manual; and the Public Expenditure and Financial Accountability (PEFA) Framework. However, the questionnaire covers topics in addition to those related to budget documents, and for many of these issues standardized “best practices” do not exist. Therefore, the questionnaire sets reasonable goals in these areas with an eye to establishing standards or norms.

When Does the “Not Applicable” Response Apply?
Although the Open Budget Questionnaire questions and responses were drafted to apply to virtually all countries, there are occasions when they do not correspond to the particular circumstances in a country. In these situations, the “e” response (not applicable/other) should be selected. Researchers should use this response sparingly and choose it only after carefully reviewing the applicability of all other responses. For the purposes of numerically scoring the questionnaire, an “e” response (not applicable/other) results in the corresponding question being dropped from consideration and omitted from the numerical score. Note that during the vetting process IBP staff members will contact researchers to discuss all questions marked not applicable.

For all questions answered with “not applicable/other,” researchers should provide a brief explanation in the “Comment” section as to why they believe the answer choices provided are not applicable, or why another response would be more appropriate.

Answers Involving Documents that Are Not Available to the Public
The Open Budget Questionnaire is intended to evaluate information on the central government’s budget that is available to the public. Therefore, researchers should not answer questions based on information that they may have unique access to through, for example, contacts in the executive, or from a document that might have been provided by a friendly legislator. If no information is available to the public, as a general rule, the answer should be marked “d.” (For questions with only three answer options, the answer should be marked “b” if no information is available to the public.)

Moreover, if a document is not available to the public, then all the questions regarding that document should be marked “d.” For example, if the Executive’s Budget Proposal is not made available to the public after it is tabled in the legislature and is only available after the legislature enacts the budget, then all of questions 1-52 should be marked “d.” Similarly, if a Mid-Year Review is not made available to the public, or not within the specified timeframe, then all of questions 76-83 must be marked “d.”

Citations and References
The releases of past Open Budget Survey results have been high-profile events that have attracted significant scrutiny and spurred a global public debate on the issue of budget transparency. We expect the release of the cross-country results of the 2019 Open Budget Survey to generate a similar response. As such, we ask researchers to cite a source or reference when answering questions 1-142 of the questionnaire. This supporting evidence for answers is intended to enhance the confidence of the media and other users in the Open Budget Survey results.

Briefly cite a source or provide a short comment (which may be as short as one sentence or one paragraph, as appropriate) for answers to each question. The citation or comment should reference the title and page numbers or subsection of the budget document from which you have drawn your answer, or it should describe the conditions that led you to select a particular answer.

Please keep all citations and comments within the word limit set by the online survey platform.

Researchers also should keep in mind that this is an international initiative and many of those who might read a country’s questionnaire may not be familiar with the details of the budget process in that country, or aware of important persons or actors on the budget scene. Therefore, researchers should spell out the titles of documents, avoid or explain any specialized terminology that might be used in their country, and explain to the reader any special circumstances that might have influenced their answer, but which may not be apparent to an international audience.


 * A short citation for a reference to information drawn from a budget document could be one or two sentences long and read as follows (researchers should include the Internet address/URL for the document cited, if it is posted on the web):

Information identifying different sources of tax revenues by type can be found on page 36 of Appendix 5 of the draft of the federal budget for the year 2012, titled "Amount of Federal Budget Revenues in the Year 2012."

OR The information appears in Section 5 of the transmittal message that accompanies the budget, which is titled “The Consolidated Public Sector Budget for 2010.” This transmittal message can be seen on the Internet at [www.mecon.gov.ar/onp/html/proy2010/mensaje/cap5.htm www.mecon.gov.ar/onp/html/proy2010/mensaje/cap5.htm].


 * In cases in which the budget documents present only partial information, researchers may choose to provide a citation that gives some insight into why they chose a “b” or “c” answer. A comment as follows could be provided:

''Detailed program information accounting for more than two-thirds of expenditures is provided, beginning on pg. 34 of the FY 2012 Budget of the Republic. However, some agencies and entities, such as the Ministry of Agriculture, do not provide program-level detail for their budgets. Instead, expenditures for these entities are simply listed as an aggregated total for the ministry, or as a transfer to a public entity.''

OR

''Multiyear expenditure information is presented for two of three classifications: economic and functional. Information can be found in Paragraphs 2-5 of the Executive’s Budget Proposal, titled “The Consolidated Budget of the Public Sector for 2013, and the Multi-Year Budget Projections.” Detailed information by administrative classification is generally provided for the budget year in question, or in specific reports to international donors.''


 * If the document in question or information is not available to the public, researchers may decide to provide a citation as follows:

This document is not available to the public.

OR

This document is produced for internal purposes but it is not made available to the public.

OR

A review of the Executive’s Budget Proposal for the year 2012, "Consolidated Budget of the Public Sector in the Year 2012," and the supporting documents that accompany it (“The Investment Budget for State Enterprises” and “Terms of Loans”) indicates that this information does not appear in the documents.


 * In some instances, researchers may wish to provide an example to strengthen the credibility of their answers. An example also can give insight to readers about why a particular answer was selected. Providing concrete details (dates, names, actions) about examples provided can strengthen their usefulness. For example:

''The deadline for the executive to present its budget proposal to the legislature is established in the Law of Financial Administration (Law No. 24.156). However, the executive rarely complies with this deadline. For example, during the 2013 budget year, the executive was required by the law to submit its proposal by April 30, but it did not make the submission until May 26.''

OR

''There is no legal requirement regarding the deadline for the release of the budget to parliament prior to the beginning of the budget year. This limits the opportunity for advocacy and participation by civil society and the public. For example, the government budget year ends on June 30, but during the 2013 budget year the budget was not tabled until the first week of July. The legislature was under pressure to adopt the document, and the debate before it was adopted lasted only from July 3-8.''

IMPORTANT NOTE FOR THOSE RESEARCHERS IN COUNTRIES WHERE THE OPEN BUDGET QUESTIONNAIRE WAS COMPLETED IN 2017: Please provide in the “Comment” section an explanation for a change in the responses from the 2017 questionnaire. This explanation might be very brief, such as:

The answer selected demonstrates an improvement [deterioration] in performance from the 2017 round of research.

OR

The presentation [or practice] is similar to that in the previous period, but researcher reassessment has led to a change in response.

Providing such a brief explanation for changes in responses from one period to the next will greatly speed the review time required for the questionnaire. (IBP staff members intend to contact researchers about any changes in response from one research round to the next that are not accompanied by an explanation.)

Citations for Information from Interviews with Government Officials
Interviews with officials in the executive, legislature, and the Supreme Audit Institution will be necessary to complete some portions of the questionnaire. For example, an interview with an official from the appropriate ministry will be necessary to determine whether a document is not produced or produced for internal purposes only.

Some questions explore the internal capacity of the legislature and the information that the executive and the Supreme Audit Institution provide to members of the legislature. These questions will likely require an interview with a legislator or a legislative staff member. Researchers should ensure that they interview legislators who are not members of the ruling party or coalition in their country, both to obtain answers to these questions and to understand their perspective. Failure to provide information to opposition parties should be noted in the “Comment” section for these questions.

Researchers also may choose to use the “Comment” section to note any significant political parties in their countries that have been excluded from access to budget information or the budget debate. This might be because a lack of free and fair legislative elections denies these parties access to the legislature itself, or due to such other conditions as arbitrary denial or withdrawal of political party registration.

Most of the questions in Section 5 on public engagement are likely to require the researcher to both examine publicly available documents and interview officials in order to capture the actual practice of the various institutions in terms of engaging with the public throughout the budget process.

Researchers should try to identify all of the officials that they interview for the purposes of completing the questionnaire in order to strengthen the credibility of their citations. This may be especially important when the Survey results are released in countries that do not perform well. However, researchers are not required to identify the officials interviewed by name in the questionnaire. (In some countries, it will not be possible to obtain an on-the-record interview with an official, and many officials will only grant the interview on the condition that their names not be published.)

Nevertheless, if possible, researchers should try to identify officials interviewed with as much specificity as possible. For example, they should include as much of the following as possible: the name of the official, his or her title, the ministry or agency, and the date of the interview, in addition to the pertinent information that came out of the interview.

Peer Review
After the researcher or research team has completed and submitted the questionnaire, IBP staff members will review it and discuss with researchers any issues that indicate that answers were chosen using assumptions that were not consistent with the Open Budget Survey methodology. The questionnaire will then be submitted to an anonymous peer reviewer for review. The use of an anonymous peer review system is intended to increase the confidence of the media and other users that the data is accurate and free of potential bias. Unless the researcher objects, the IBP also will seek comments from the country’s government, which will be included as a third set of review comments.

Once the questionnaire has been reviewed, the IBP will contact the researcher or research team to further discuss whether differences between the researcher’s answers and those of the peer reviewers need to be reconciled. As part of this process, we may ask researchers to write a further elaboration of their answer choice. We expect this elaboration to be brief and not to exceed four paragraphs in length per question.