Working to ensure the transparency of government finances has a long history. By 1850, many countries in Europe had already enacted constitutional requirements that government budgets or accounts be published, leading to what Irwin1refers to as an “avalanche of data” that was sparked, in part, by “rulers’ need to persuade creditors to lend and taxpayers’ representatives to approve new taxes”. However, this avalanche of annual accounts, published in printed paper reports, seems miniscule when compared to the data on government finances that could be made available today. When the East Asian financial crisis hit in 1997, fiscal transparency was firmly placed on the global agenda, and principles were put forward calling for disclosure of information across government operations, not just budgets.2And as the open data movement has developed over the last decade, it has brought a particular focus on transparency in government finances, adding a particular digital spin to advocacy and calling not only for data but for machine-readable data that is ready for public analysis.
Public finances are ultimately at the heart of government activity, constituting one of the main levers of public action through which governments shape society. The study of public finance may historically have been regarded as a question of simply determining the income and expenditures of governments. However, since the middle of the 20th century, this has expanded to recognise the role that taxes and spending play in shaping the wider economy (e.g. taxing activities that may have negative consequences and spending that may stimulate economic development and trade, including research grants or development aid). As such, citizen scrutiny and a clear understanding of all aspects of public finances is crucial. Debt, taxation, contracting, grants, and subsidies are all topics to be covered within the context of fiscal transparency, alongside more obvious themes of budgets and expenditures. With the right mechanisms in place, improved citizen understanding of the state’s fiscal behaviour can encourage greater civic participation and oversight, can promote public accountability, and, most importantly, can potentially enhance the effectiveness and efficiency of public budgets and spending.34
From the start, the open data movement has placed an emphasis on government finances with projects such as the 2007 “Where Does My Money Go” prototype (see box below) that demonstrated the potential of open data in this sector. Over the last decade, civil society and government-led projects around the world have sought to make public finance data more accessible with initiatives on almost every continent. However, the latest findings from the Open Data Index5and Open Data Barometer illustrate that just 10% of surveyed governments publish fully open budget data (12 countries in total) and only 3% publish disaggregated open spending data (just 4 countries).6In some countries, such as the United Kingdom (UK), an early publisher of spend data, reliable data availability has not been sustained, and it is not clear how far citizens have engaged with the data that has been made available.7
A decade into the new wave of open data-driven financial transparency, it is important to take stock of progress and to ask whether efforts to open up financial data have delivered results or whether activity is beginning to stall. This chapter takes a look at the arc of activity since 2005, taking stock of the state of initiatives, issues, and communities related to open government finance data.
The new wave of fiscal transparency: From documents to datasets
“Fiscal transparency – the comprehensiveness, clarity, reliability, timeliness, and relevance of public reporting on the past, present, and future state of public finances – is critical for effective fiscal management and accountability. It helps ensure that governments have an accurate picture of their finances when making economic decisions, including of the costs and benefits of policy changes and potential risks to public finances. It also provides legislatures, markets, and citizens with the information they need to hold governments accountable.”8
What counts as “public reporting” depends on your perspective. For much of the history of fiscal transparency, the focus has been on access to information being provided through the publication of government reports on budget formation and execution (including spending). These reports are generally static documents prepared by selecting, analysing, and summarising data from one or more “live” financial information systems. Governments may, in some cases, provide interactive tools to support the user’s ability to “drill-down” into the contents of those reports. However, with documents, there is a limit on how far users can dig into the data or remix the information to present it in different ways.
This is where calls for “raw data” come in: asking not just for reports and documents about budgets, taxes, and spending, but also for the underlying granular data. Where a row in a published document might represent hundreds of individual budget allocations, an open dataset could include a row for every allocation, along with detailed classification information. Where a spending report might contain an aggregated figure on payments by a particular agency, spending data could contain a row for each payment with details on the suppliers paid in each case and information on the timing of those payments. The move from documents to data provides for both increased granularity (or disaggregation) of information and increased flexibility in how users can work with it (see Figure 1). With access to data, rather than documents alone, it becomes possible for a wider range of users to create a wider range of visualisations, interfaces, and analysis, although such applications are very dependent on the quality of the raw data and on the metadata to provide context.
The pioneers: Early steps to open financial data
In 2005, a trio of data journalists launched FarmSubsidy.org with the goal of facilitating access to information on the subsidy payments under the European Union (EU) Common Agricultural Policy (CAP). The platform, constructed with data accessed via freedom of information (FOI) requests to governments across Europe, made structured data accessible to search and explore, providing detail not only on subsidy payments, but also the details of the companies who receive subsidies. Danish journalists were able to use this information to challenge the dominant political narrative that the CAP supported primarily the poorest farmers by showing that it was actually large landowners and agri-businesses that received the most funds.9By 2009, EU member states were mandated to publish their subsidy data, removing the need for FOI requests, although, even now, the data is not always available in machine-readable formats. The growth of the project played a key role in demonstrating the value of data-driven public finance journalism and attracted interest from a range of funders, including the Hewlett Foundation.10
In 2007, the Open Knowledge Foundation’s Jonathan Gray developed the idea for “Where Does My Money Go”11as a visual breakdown of the UK budget, tapping into a growing appetite for both data visualisation and open data ideas (see Figure 2). In 2008, the project was a winner of the UK Government’s “Show Us A Better Way” competition and had soon secured grant funding from government to develop a working prototype.12Further funding from a UK state broadcaster (4IP), the Open Society Foundation, the Knight Foundation, the Hewlett Foundation, and the Omidyar Network enabled the evolution of the project into the global Open Spending platform,13which now hosts elements of fiscal data from at least 70 countries. The nascent community related to the project was not comprised of accountants or public finance experts but rather civic hackers and citizens interested in making complex government finance more accessible and supporting wider citizen engagement.
By 2010, more governments were starting to explore the direct publication of machine-readable budget data, reducing the need for citizens, organisations, and projects to manually scrape data out of documents and PDFs. The United States (US) government’s USASpending.gov, originally created in response to legislation passed in 2006 requiring all “federal contract, grant, loan, and other financial assistance awards of more than $25,000 to be displayed on a publicly accessible and searchable website to give the American public access to information on how their tax dollars are being spent”,14went through a number of relaunches in 2009 and 2010 with increasing emphasis placed on the availability of downloadable open data and enhanced granularity. Although the site had provided an application programming interface (API) since 2007, it was the addition of downloadable open data in subsequent versions that gained it an increased profile.
Intense policy competition between the UK and US during this period may be behind the UK government’s 2010 publication of the COINS (Combined Online INformation System) dataset,15providing detailed “fact tables” that presented disaggregated spending data from across the public sector. The Guardian newspaper was one of the early users of this data, creating a public data explorer interface to help citizens search the large dataset and working with Open Knowledge Foundation to use citizen research and FOI requests to fill gaps in the data, particularly around individual supplier names.16The Guardian went on to write a number of stories based on their analysis of the COINS data and used its release to explore gaps in the quality of public financial management in the UK.17In parallel, government departments and local authorities were asked in a letter from Prime Minister David Cameron on 31 May 2010 to publish details on all expenditures over GBP 25 000. The letter also committed to the online publication of information on all new central government contracts and all international development project spending over GBP 500 from January 2011 onward.18
Latin American governments also took a lead during this early wave of government activity. In Mexico, the first budget dataset was published in 2011 by the Ministry of Finance as part of a project under Mexico’s Open Government Partnership (OGP) action plan.19The portal that was created published basic information about federal programmes with quarterly updates on the money spent, information on external evaluations, and a matrix illustrating progress toward planned and achieved goals. The intent of this project was to provide a place where both citizens and decision-makers could find government finance data in a unified format. An OGP case study credits the portal with generating “commitments from the Federal Public Administration to make progress on public projects and initiatives which [had] fallen behind”.20Although such portals could theoretically be created without using open data, taking an open data approach helped to provide Mexico with a common format for aligning data from different departments and agencies, supporting the integration of information that originated from many different IT systems.
Although Brazil launched a National Transparency Portal in 2004, Beghin and Zigoni (2014) documented21that it was not until the passage of an Access to Information Act in 2011, establishing procedures for federated entities to follow in the disclosure of information, that access to government finance data increased. However, they note that, in 2014, there was still a long way to go before all budget and spending data would be accessible in machine-readable form.
It is no surprise then that a World Bank study in 2013 cited the UK, Mexico, and Brazil as members of a small pioneering group of countries working to provide good access to reliable open budget data from financial management information systems.22The full list of countries noted included Brazil, Germany, South Korea, Mexico, New Zealand, Spain, Sweden, the UK, and the US. They all have a high Open Budget Index score (above 60)23and OGP commitments to promote fiscal transparency in common.24
In this pioneer phase, we can see how the interaction between select journalists, civil society, and governments spurred action to make more granular and machine-readable data available on government finance. But whether or not this data can be used to answer questions like “where does the money go?” and whether these early publication projects are sustainable depends on a much wider network of actors and activities.
A growing community: Creating tools and capacity
While many of the most prominent actors working in the area of open budgets are intergovernmental organisations, international NGOs, and multi-stakeholder initiatives, according to Gray’s analysis of the open budget data landscape, the active grassroot community is composed of a myriad of international and local CSOs involved in open government, government transparency, aid transparency, open data, and related topics.25The way these groups are making use of public finance data is innovative and ingenious and reflects a limited but dedicated citizen interest in understanding how public money is spent. For example, in Nigeria, BudgIT26has worked since 2011 on creating infographics that explain elements of the budget and, since 2014, has used Tracka to crowdsource information on the progress of development projects in local communities. Communicating through social media, mainstream media, and community outreach, BudgIT reports reach over 4 million Nigerians with their information.27
In an effort to help scale up innovations, equip organisations with open source tools, and improve data literacy around spending data, the Open Knowledge Foundation launched the OpenSpending project in 2011.28Its vision was to provide a central database of budget and spending data, as well as to build a community of groups and individuals who could work together to acquire, use, and add their contributions to the platform. From its launch, the resources made available increased substantially as the project grew, including a spending data handbook,29an open-source CKAN data portal with extensions,30a visualisation library based on Where Does My Money Go,31and a data specification called the Fiscal Data Package.32As of November 2018, OpenSpending contains government finance datasets from over 80 countries, although at varying levels of granularity and timeliness.
The tools provided by OpenSpending have been used by different civil society projects and platforms to provide citizens with accessible and user-friendly budget information (e.g. the German project Offener Haushalt, the budget explorer tool in Kosovo, and the Open Budget platform in Ukraine).33Many other organisations have developed their own technology and visualisations. This is the case for the Open Key project in Israel, the Open Spending portal in the Netherlands, the Vuleka Mali project in South Africa, and the Dónde van mis impuestos platform in Spain.34A key driver for the editorial and technological choices of these projects has been the goal of building visualisations that reflect the needs of citizens and a desire to embed data within a pedagogical context that provides education on government finance.
Between 2013 and 2017, as the community grew, many more projects and platforms emerged from civil society organisations (CSOs), some of them with the specific objectives of using public finance data for investigation in journalism or to enhance civic participation. One notable data journalism project using public finance data is Spending Stories, a project by the former data journalism agency J++ that was developed in 2013 to allow comparisons between big and small amounts of money to give users a context to understand how money is being spent while referencing original news stories.35The Farmsubsidy.org network has also continued to play an important part in building data journalism capacity related to open financial data, giving rise to the annual European investigative journalism Dataharvest Conference that now brings together as many as 400 journalists, coders, and scholars from all over Europe each year.36
As Gray’s map of the linkages between open budget data-related websites from 2015 suggests (Figure 3), it is also important to recognise different sub-communities working in the open finance data domain. As well as local groups, there are a number of overlapping global communities of practice, some with specific thematic areas of focus. Examples are the International Aid Transparency Initiative (IATI), the Extractives Industry Transparency Initiative (EITI), and others looking at particular sources of data, such as the Open Contracting Partnership which has, since 2015, developed a global network of governments, civil society organisations, and companies working with data on public procurement to enable a different way to “follow the money” that complements budget and spending data. Gray’s 2015 mapping does not, however, capture groups working in the area of tax justice. Since 2017, the Open Data for Tax Justice network has sought to put more focus on companies reporting the tax payments they make to government,37which, once again, fills in another part of the complex government finance picture.
Although they do not feature heavily in Gray’s mapping, we should also not ignore private sector actors. Firms like SpendNetwork38clean and re-package government spending data for firms interested in securing government contracts, and there is some evidence to suggest government spending data feeds into a range of other private sector products. This said, more could be done to understand the role of the private sector in this field.
It should be clear from the examples above that there is widespread interest in, and engagement with, open data on government finances. Networks like the FollowTheMoney network39host regular community calls to connect organisations working on different parts of the governance finance puzzle, and groups like the Global Initiative for Fiscal Transparency (GIFT)40place an emphasis on open data as part of wider fiscal transparency reforms. Yet there remain many shared practical challenges that mean the vision of timely, accessible, and accurate open data on government finances is far from fully realised.
Ongoing challenges and developments
Even though much progress has been made in opening public finance data, some gaps remain, including those related to policy and high-level commitments, technical platforms for data, linking data to decision-making, and challenges in encouraging the use of data.
In 2009, the Sunlight Foundation in the US started the Clearspending41project to generate an annual report on the consistency, completeness, and timeliness of federal data published on USASpending.gov. The project discovered over USD 1.3 trillion worth of missing or inaccurate data.42The Guardian similarly reported problems with the accuracy and coverage of the early UK COINS datasets,43and monitoring of UK government departments’ compliance with requirements to publish expenditures over GBP 25 000 indicates that many are failing to publish the required data on time.44When data quality is low, it becomes hard for citizens to use and interpret data or to draw conclusions from it. This can be addressed by providing documentation that explains how the data was created and its limitations. In other cases, independent monitoring of data quality can provide an impetus for governments to improve their data. However, it is difficult for civil society (and even governments themselves) to sustain a quality control over published data. For example, the Clearspending project in the US only ran until 2012, and a number of other projects that have sought to monitor the quality of data in specific countries or localities are now defunct.
One of the key barriers to improving data quality has been the lack of a legislative basis for open data publication. In the US, the Digital Accountability and Transparency Act of 2014 (DATA Act)45has addressed this in part, setting out standards for data publication and leading to the creation of detailed standards and procedures that apply quality assurance in stages as data is collated. Yet, in many countries, legislation or regulations supporting the transparency of government finance, even where they exist, have stopped short of providing enough detail to allow quality requirements to be enforced.
Standards and interoperability
The OpenBudgets.eu project looked at the standardisation of budget and spending datasets across the EU in 2016 and concluded that there were a “plethora of budget and spending data models which reflect … fine-tuned differences in the legislative design of political entities”,46although they also recognised the need for common approaches to data publication. One standard that has been put forward to address this gap, developed by a consortium of global organisations, including GIFT, the World Bank, and Open Knowledge International,47is the Open Fiscal Data Package (OFDP). Rather than impose a particular structure on source data, the latest iteration of the OFDP allows datasets coming from countries with different fiscal and accountability structures to be published in any tabular form and then subsequently annotated to explain how data should be interpreted and visualised.
Adoption of the OFDP remains limited at present; however, the way in which data standards can facilitate global collaboration around government finance data has already been demonstrated through the adoption of more mature standards for aid flows (IATI) and contracting data (the Open Contracting Data Standard (OCDS)), and with the right backing, there are opportunities for the OFDP to support a step-change in the accessibility and re-use of budget and spend data.
As noted in the introduction to this chapter, to construct a full picture of government finances, more than budget and spend data is needed. This calls for interoperability between standards. There has been some recent progress on this with extensions to the OCDS (Figure 4) being designed to provide interoperability with the OFDP, although this work is currently untested.
The Ministry of Finance in Mexico has been working on a pilot to link federal budget and spending data with investment projects through the use of two standards: OCDS and OFDP. They have successfully linked budget data from the planning phase of procurement processes with amounts spent per project in the implementation phase and have made this available.48 Through work with the Open Contracting Partnership, a proposed extension to OCDS has been developed to describe how other governments could make similar linkages.49
The greatest challenges (and opportunities) to increased adoption and impact from open data activities related to government financial transparency ultimately relate to policy. In 2017, the International Budget Partnership’s Open Budget Survey (OBS) of 115 countries suggested that progress on opening up budgets had stalled for the first time in a decade.50Although the OBS does not look specifically at open data publication, its findings suggest that the global political will to increase financial transparency may be at a low ebb. There have also been long-standing challenges to securing public attention on open government finance data, as noted by Carter in 2013 that “budget transparency has still not captured global attention in the way that other related movements have”.51
Regardless, in some noteworthy countries, open data regulations and legislative frameworks are being used successfully to enforce either the publication of public finance data52or to make finance data a priority within wider programmes of open data release across government.53The OGP has also provided a key forum for increasing the disclosure of contracting data in recent years with many commitments secured to adopt the OCDS.54This suggests that the current wave of interest in open data and data standards could still be used to help advance the financial transparency agenda. Crucially, getting to joined-up data that presents a full picture of government finances means overcoming silos in government and securing data across agencies. For this, the importance of political leadership cannot be underestimated.
User-engagement and capacity
Government finances are undoubtedly complex. Increasing the use of available data requires accessible technical platforms, skilled intermediaries, and capacity building for citizen-users of data. As a whole, the last decade has seen an increase in resources to support the development of data literacy skills which enable users to work with public finance data through digital tools, and many resources are still improving based on cases studies, user involvement, stakeholder feedback, and innovations in technology. However, continued capacity building will be needed for increased data availability to drive new models of citizen engagement around government finances.
Conclusion: Looking to the future
In the years ahead, the key challenge will be to better connect the current wave of the open data-driven transparency movement with other grassroots advocacy networks and government decision-makers. When it comes to securing impactful results from open government finance data, the evidence suggests that projects will require unique partnerships between technologists, CSOs, and government. This is the model followed in South Africa with the Vuleka Mali project, a partnership between the National Treasury and a coalition of CSOs called Imali Yethu to make government budget data and processes accessible to all citizens and interested parties. Their motto, “We aren’t interested in transparency for transparency’s sake”,55should be one that more organisations place at the heart of their thinking. Technical work on government financial data also needs to connect with wider social agendas. For example, Carter notes that the potential exists to apply a gender lens to budget analysis;56however, we have not yet found open data projects that directly apply a gender lens to open budget data creation and sharing.
Given the long history of work on opening up government finance, we should not expect a complete transformation in less than 15 years of open data activity. The vision of current advocates for open government finances is an ambitious one – to provide more granular data than ever before. There are signs, however, that when data is used, and governments are willing to open up, substantial progress can be made. In their brief history with the DATA Act, Sunlight Foundation has described how “reporting bad data drove reform” to secure new legislation, better processes, and ultimately improved data.57Rather than waiting for perfect data, it is possible to publish data and then improve it with subsequent iterations.
The foundations laid in the last decade in terms of technology platforms and data standards, and in terms of networks and communities, is impressive. Long-term, opening up and securing the use of government finance data will require significant resources in terms of technology, financial and human capacities, as well as time and strong political support. Not all the organisations explored in this chapter will have the resources they need for sustainability, but all have demonstrated what could be possible in their local contexts, and they have collectively re-imagined ways to engage citizens on governance finances.
Government finance data has played a key role in shaping the early development of the open data movement. The challenge for the decade ahead is to see how far, and to what end, open data advocates and practitioners can shape a sustainable ecosystem of open government finance data.