Blockchain for governments: ideas to move from hype to reality

One informal way to describe the value of blockchain is that it can manage trustworthy electronic transactions between strangers without a trusted third party. It is, definitely, a significant breakthrough that will create a vast variety of new and exciting business applications.

Nowadays, our economy is strongly based on intermediaries who guarantee or certify a transaction or some information. The intermediaries are usually expensive, slow and, often, offer little value. Some of them are banks, insurances, public registers (land, commerce, etc.), notaries, certification organizations and, of course, governments. One of the main functions of the public authorities is managing information and provide official documents or certificates about licenses, permits, registration in public databases, etc.

Internet was a revolution of communication and information. Since 1993, we can connect and share any information with anybody at any time, at the cost of zero. The blockchain is argued to be the revolution of trust between foreigners: the promise of disintermediation. But, let’s be fair, there are many areas where the intermediaries still provide value (not just keeping a database of transactions) that cannot be replaced by a distributed ledger or by smart contracts. In some situations (not all), notaries, official registers, etc., provide worthy legal advice, very hard to automate.

One of the areas where blockchain has already created popular business cases is in cryptocurrency (bitcoin, etherium, and a hundred more). It seems likely that the future of currency will be based somehow on the blockchain. We could expand it to the tokenization of assets, either digital or physical. For instance, Singapore (a financial center in East Asia) has announced the development of a blockchain platform for stock exchange securities’ settlements.

What are the limitations of blockchain?

The blockchain is a disruptive technology, but to take real advantage of it, we should have a clear understanding of its limitations. Let’s explain some of them:

  • It does not record information: It just saves logs (date, time, quantity, origin, destination, hash, etc.) and, at most, small pockets of data used to execute and guide smart contracts. But it is not designed for data storage. For example, it records the metadata that you have a property, but not the official document and detailed information that describes your property.
  • It is not cheap. Each transaction saved in a public blockchain network cost a few cents to fund the members of the network and the huge energy consumption. If we have a business case that generates millions of transactions, it will be a costly project, much more than a centralized database. We can create a private or a consortium blockchain, but then there are, again, relevant costs to build the network and maintain it.
  • It is not fast. Transactions are grouped in blocks, but by security reasons, a difficult puzzle has to be solved, and it has to be validated by most of the members of the network, before being added to the distributed ledger. So, that means a block is not saved immediately. Depending on the network and the complexity of the algorithm, it may take seconds 10 seconds (Etherium), 10 minutes (Bitcoin) or even hours (some services aggregate many blocks to reduce costs).
  • Anonymity but no confidentiality. The DNA of the blockchain is to manage anonymous transactions: it is not known who is behind each one. But all of them are published in all the nodes of the blockchain network and, therefore, there is no confidentiality (unless some encryption techniques are used). By design, Governments are designed (regulated) on the other way around: the users are very well identified, and the data is confidential.
  • It is not easy to scale, as each transaction (and block) is saved in all the members of the network.

There are complementary technologies to overcome some of these limitations, but it requires additional developments and costs. It is essential to understand what blockchain was designed for and what is good at. The blockchain can be used in almost any current use case, but because of its limitations, it is not always the best option.

Because of all these constraints, some experts are now talking about distributed ledger technologies (DLT) instead of the blockchain. The blockchain is just a type of DLT, a democratic one where all are equal, and all record the same information. We are in a learning phase, and some experts are already pointing out that the best solutions would be a sort of DLT, which may not be exactly the blockchain we know now.

Where are we now in the hype cycle?

There is a lot of fuss, but the reality is that we are just in the early stages and there are very few real (in production) blockchain projects in governments. Gartner estimates that blockchain will reach ‘maturity’ in the next five to ten years. It has already passed the “peak of inflated expectations,” and now it is going “through the disillusionment phase.

The Economist magazine stated in a recent article “Blockchain has been hyped to the skies’ (August 30th, 2018), “progress has been slower than hoped, and some apparent successes turn out to have been exaggerated.” Cryptocurrencies are far from being in our daily lives yet. For instance, it is very difficult to find a shop where to pay online or offline with bitcoins, at least in Barcelona where I work.

To learn a bit more about the limitations and the hype, I strongly recommend to have a look at the following videos:

Approaches about using the blockchain

Although it is not mature yet, governments have the responsibility to learn, experiment, and evaluate government use cases to be well-prepared when the revolution comes and make sure that the public authorities take advantage of it. That means promoting prototypes and pilots in use cases that make sense from the public value perspective to provide better and innovative services to citizens.

When planning blockchain projects, there are two radically different approaches:

First approach: the blockchain can be used in almost every use case. Projects are usually launched on the criteria to have the most significant media impact. It is a fair approach, but we have to be honest: we will basically learn about how to use the blockchain technology, not about the added public value it provides in comparison with existing technologies.

Moreover, there is the risk to create problems where we already had good solutions because we change applications that have been performing fine for years without any relevant issue, just because we now have a trendy new technology. Unfortunately, most of the initiatives fall in this approach. There are several guides to help you decide where the blockchain is the suitable technology.

Second approach: let’s identify the public challenges that we have not been able to solve so far with existing technologies and then analyze if blockchain can help us out. It is a disruptive technology, so let’s use it in disruptive business cases. This approach is much harder because we have to understand first, what are the public challenges, and then the restrictions of the blockchain.

In Jim Collins’ brilliant book “Good to Great”, there is extensive research about successful long-term companies. One of the conclusions is that “they do not adopt a new technology just because it is the latest trend or state of the art: if it does align with your mission and strategy, then you need to take it by the horns and become a pioneer in it. But, if it isn’t, then you should ignore it.

So, the key question is what are the government business cases unsolved or with a poor performance where disintermediation could create better public services. In this approach, it is essential to foster experimentation and prototyping blockchain labs to learn about their potentialities, where any government with a good idea can prove the concept with a small effort and cost. We should encourage the start-up’s methodology of learning by doing, “fail fast, fail cheap” and user validation of the value proposition.

Let me point out a couple of interesting use cases (at least in Spain) that I am working right now:

  • Global identity for both, public and private sector, based on evidence provided by governments (ID card, passports, driver license, etc.), private companies (banks’ documents, insurance contracts, telecom operator registers, etc.) and social reputation (your social network activity that has been validated by many other users). Your identity is not based on a piece of plastic issued by the government. Your identity is the collection of evidence about your life.
  • Empowering citizens with the control of their personal records: citizens own their data, and they can share it with governments and with private organizations, with their explicit consent. The blockchain can support this initiative by having the logs about the authenticity of the data/documents and recording the authorized data exchanges.

Real use cases of blockchain in government

There are hundreds of potential use cases that are in the research or the development phase. The Illinois Blockchain Initiative has done excellent work to create a database tracker to compile the blockchain projects in government worldwide.

In this database, we can find (at them moment of writing this article) 17 in-production/live projects, with a description and a project link for additional information. But, sadly, only 11 of them (4 from Estonia) have a project link to a government website: some of them have a link to media articles and some even don’t have a link. Furthermore, some of the 11 initiatives were temporary projects and, in others, the described project is live but, related about using blockchain, they are at the development phase and they are not using it in production yet: that is why in some of the government project links they do not mention blockchain at all.

One of the live projects is the Official Gazette of the Republic of Argentina. They deserve a lot of credit to be innovative but the fact is that they have followed the 1st approach and the way they are using blockchain in this use case is not providing any added value to the existing technologies and, at the end, it is less usable for a citizen to understand and to validate the published date.

Conclusions

I definitely go for the second approach to focus on disruptive business cases and learning by doing, fostering experimentation, pilots and evaluating the impact in government. We are living exciting times, and we have great opportunities to create better transparent, open, efficient, and trustworthy governments. The blockchain may help the process as long as we are smart enough to focus its application to provide public value and not just create great marketing campaigns.

Whoever wants to learn more about blockchain in Government I highly recommend the OECD guide “Blockchains Unchained: blockchain technology and its use in the public sector”

P.S.

  • I am not a technology expert in the blockchain, but I have extensive experience in digital government transformation. This article is based on the learning of the blockchain technology application in the public sector during the last year.
  • Thanks to Daniel Martínez for his contributions
  • This article was originally published on October 28, 2018. It was reviewed in Novembre 11, 2018.

Articles mentioned:

  • Berryhill, J., T. Bourgery and A. Hanson (2018), “Blockchains Unchained: Blockchain Technology and its Use in the Public Sector”, OECD Working Papers on Public Governance, No. 28, OECD Publishing, Paris, https://doi.org/10.1787/3c32c429-en.
Anuncis

Proprietary software vs open source in the public sector

First of all, as a civil servant, I understand that one of the essential government priorities is to provide efficient public services. When we have to decide the best software for a specific need, we must take into account, at least, the criteria about cost, functionalities, implementation time, internal effort, access to the source code, and risk of failure.

Based on these, we have the following alternatives:

1) A custom software development. It fulfills the functionalities and access to the code, but it has high stakes at implementation time, internal effort, cost, and risk of failure.

2) Using a vendor solution. It fulfills most of the criteria, but it is not highly customizable, and there is no access to the code.

3) Using a vendor solution with license access to the source code. Same advantages of the former option plus having access to the source code, but it does not mean knowing how to evolve it. Some software vendors will not tender with this condition.

4) Creating collaborative development of communities of open software. It raises excellent opportunities regarding savings and evolving in a more agile and innovative way. However, governments do not have the skills yet to manage them effectively.

The decision is not unique. It depends on each specific problem, but too often Government takes the easy choice: customized software or using vendors’ solutions. The great challenge of the public sector, despite its enormous complexity, is to promote collaborative communities of open software.

Three trends:

– Digital sovereignty: the public sector will demand much more control of the source code, data, algorithms, and technologies to ensure the common good.

– Open innovation by promoting public collaborative communities of open software.

– Software license agreements will include escrow arrangement that allows using source code if specific events occur.

El Consorci AOC un decálogo de innovación y transformación digital

Hace unos dias tuve la ocasión de departir en Barcelona con Miquel Estapé (@MiquelEstapeV), subdirector del ConsorciAOC. Un consorcio público de las Administraciones catalanas que es un ejemplo de gestión en la prestación de servicios digitales a entidades locales y autonómicas en Cataluña. De nuestra conversación extraigo los siguiente titulares, que espero que os resulten de interés.

  1. El modelo. El ConsorciAOC nace hace 15 años porque existe una visión sobre el gobierno electrónico. La singularidad del proyecto resulta de que es compartido por diferentes Administraciones Públicas y de distintas sensibilidades políticas en un afán por desarrollar una estrategia de servicios singular hasta el momento en España. Transformar la administracion es una maratón y la ventaja de este “consorci” es que el pacto inicial definió una estrategia a largo plazo que se ha mantenido. La neutralidad política es una baza muy potente.
  2. La prioridad es la transformación digital con un perspectiva de liderazgo colaborativo fundamental para afrontar los retos de unas Administraciones en pleno cambio constante. Entienden la misión del ConsorciAOC para promover gobiernos ágiles, lógicos y colaborativos.
  3. La tecnología como gran aliado del ciudadano. El ciudadano debe obtener servicios digitales de las Administraciones que le sean usables, accesibles… que pueda palpar que sus Administraciones son cercanas. Su exigencia nos ayuda a mejorar la implementación de servicios. Se trata, en resumen,  que los ciudadanos disfruten de servicios públicos digitales de calidad y vivan en una sociedad abierta.
  4. Las lineas estratégicas prioritarias del ConsorciAOC son los servicios en la nube, la interoperabilidad, la identificación y firma electronica, sin olvidar la gestión del cambio. Y es que el Consorci no solo trabaja en el nivel tecnológico ofreciendo servicios digitales de calidad sino que apuesta por la transformación de las organizaciones para que puedan y este en mejores en condiciones de utilizar las TICs.
  5. El servicio del ConsorciAOC es a coste “0” y el soporte que dan a sus clientes públicos es integral: implantación, formacion, divulgación y asistencia tecnológica. Atienden mas de 4.000 peticiones al mes. Esto da a entender la organización milimetrica que está detrás. Trayectoria y profesionalidad.
  6. El soporte de proximidad es un puntal y la comarca es un ámbito territorial. El valor añadido de la implantación del Consorci en el mapa administrativo catalán es crucial para conseguir optimizar recursos y aprovechar oportunidades. Además, el estar en el territorio supone tener un conocimiento directo mas real de los servicios prestados y de la satisfacción de los clientes públicos.
  7. Una de las consecuencias de este modelo es la generación de redes. La red colaborativa por todo el territorio catalán es crucial para la consecución de la implantación de servicios, desde la Generalitat hasta los municipios. La creacion de sinergias, relaciones, espacios de seguimiento de los servicios, reuniones, foros… son una constante liquida en el funcionamiento del consorci.
  8. Un proyecto innovador y vertebrador del conocimiento de los servicios digitales ofrecidos por el Consorci es el mapa de administracion electronica que analiza unos 30 indicadores básicos, orientadores del estado de esta materia en Cataluña (aquí).
  9. En transparencia, uno de los logros conseguidos es la automatización de la carga de datos en áreas tan relevantes para la transparencia activa como personal, convenios, presupuestos, datos económicos, cargos electos… Las herramientas de opendata juegan un papel clave en esta mecanización de procesos y datos. Entiende el equipo del ConsorciAOC que una plataforma de datos abiertos a nivel individual no aporta valor. Integrar plataformas y entidades administrativas es fundamental para los tecnólogos, organizaciones y explotadores de datos abiertos.
  10. Actualización e investigación en nuevas tecnologías es uno de los objetivos del equipo del ConsorciAOC. Detectar que potenciales servicios, iniciativas… consiguen que se enfoquen los futuros servicios desde una Administración pro-activa, centrada en el ciudadano y en su bienestar social, son el acicate para estar atentos a la impresión 3D, blockchain, drones, bigdata… como nuevas formas de ser mas eficaces y eficientes en facilitar la vida a todos.

Moving forward to custom & proactive services to citizens. MWC 2018

Introduction

Open Government of Catalonia (AOC) is a public digital agency with the mission to foster the digital transformation of the 2.200 public administrations of Catalonia (Spain).

The challenges

  • Data is the new oil of the 21st Governments have a huge amount of citizen information but they are not taking advantage of it to provide better services.
  • New generations expectations are much higher than in the past. Some of the demands are to improve customization, proactivity, usability, trust and cross-government approach.
  • Public services are usually developed to meet the administration interests. Governments must use co-design methodologies putting citizens at the center when designing the services so that they fulfill the citizen needs and expectancies

The goal

We applied all the previous ideas to one specific project which tried to respond to the following question: How can Governments provide a personalized, proactive and trustworthy relationship to citizens by using their personal data while respecting privacy regulations?

A citizen usually has three questions about their information: what is the status of my applications? which of my personal data do governments have? and what are governments doing with my personal data? These questions are currently being answered from a cross-government point of view by some of the most advanced governments in the world .

In this sense, it is paramount to respond to the following question “what can governments do for me? In other words, how can governments move from a reactive and standard service practice to a user centric, proactive and customized approach?”

Taking all these issues into account, the ultimate aim of this project was to develop a solution, called MyGov, that accomplishes all the previous requirements.

Methodology and process

The project included the following phases:

  • Legal evaluation: we validated that it is possible to conduct data science analysis according to the European Union legal framework, if we anonymize the personal data, making sure that there are no direct or indirect ways to identify a specific person.
  • Data science analysis: we arrived to the following preliminary conclusions:
    • Little data: AOC holds a large amount of public records but only a small fraction of them is relevant to define citizen profiles
    • One of the areas for which AOC holds larger and most relevant data is “citizens with social needs”. It was decided that this group would be our target early adopters.
    • Mobile-first: citizen with social demands do not have a good internet connection and a PC at home, but most of them have a smartphone. So, we decided to make MyGov a mobile-first solution.
  • Research: we conducted an extensive research in the public and the private sector, inspired by some solutions in the banking industry.
  • Co-design: we carried out several co-design activities:
    • An online open innovation challenge with public employees to manage collective creativity.
    • A design thinking workshop with public employees.
    • Several co-design workshops with citizens to define archetypes and identify their expectations by using design thinking techniques. In these workshops we also identified the citizen journey using ethnographic methodologies; and co-created the main functionalities of the solution.
  • Prototype: from the previous results we developed a minimum viable product to be tested by the users.
  • Validation with end users: we made new iterative versions of the prototype according to the feedback and validation of the end users.

The final prototype

The final prototype done using Invision is the following:

https://projects.invisionapp.com/share/97D54EXP4#/screens/249746824_MyGov_Tens_4_Nous_Avisos_01

Validation by end users

We have successfully validated the final prototype with end users (citizens and social public employees) and we have compiled the expected benefits.

  • For citizens: empowerment through increased transparency related to citizen’s data, increased confidence, increased satisfaction, time saving and reduced anxiety and stress as citizens know all their entitled benefits.
  • For governments: better valuation of government services by citizens, making sure social benefits are provided to the citizens in need, savings in time and administrative tasks and increased satisfaction of public employees.

Next steps

After several prototypes developed using a mock-up tool, now we are currently developing the solution that will go into production.

Authors:

  • Miquel Estapé – Deputy CEO of the Open Government of Catalonia
  • Anna Forment – Manager of Everis
  • Francisco García-Moran – Former Director General of IT at European Commission and EU Fellow at UC Berkely

A collaborative approach to digital transformation across the public sector in Catalonia

Interview with Miquel Estapé by Priyankar Bhunia. Malaysian Government Public Sector CIO Congress.  OpenGov Asia (original source)

“What makes our organization possibly different from others, is that we provide digital government services to all the public administrations in the area, currently more than 2000 government organizations in Catalonia. This includes regional governments, provincial governments, local governments, universities and others,” he continued.

This is done in a very collaborative way. AOC also provides training and consultancy and works together with the public administrations on improving its services.

The government administrations in Catalonia are AOC’s clients but at the same time, they are also AOC’s stakeholders and are represented in AOC’s executive committee.

AOC is funded by the regional government and all public administrations in the region. The administrations pay for some services provided by AOC but that represents only 10% of the AOC’s budget.

Mr. Estapé has been working for 15 years with the government of Catalonia and for the last 3 years, he has occupied the position of the deputy director, managing operations in various areas of the organization.

Following a mandate from a new General Manager last year to increase investment in innovative services, Mr. Estapé has been working on innovative projects. He leads a team to foster innovation initiatives with potentially high impact on society.

Services provided by AOC to government administrations

The first area is related to digital identity. There are two main systems right now. One based on the use of digital certificates and the other using 2FA (two-factor authentication) via mobile SMS. The former is meant for use by the public sector and companies/ businesses.

Digital certificates are based on the logic of private and public keys. It comprises a piece of software which can be stored in a card and the user has a password to access the private key. Every time a user submits a form or accesses a public website, there is a process of encrypting and making sure that everything is safe. The user uses their private key to authenticate or sign documents. The governments have access to the public key of the user to verify the identity.

For ordinary users, who are not technical experts and do not have access to technical support, AOC focuses on using 2FA-based on mobile. It works well but Mr. Estapé said that the issue with the 2FA is that the codes are sent using SMS. There are some concerns about the security of the SMS technology. AOC is looking for solutions to this. A new technology is being explored, called mobile connect, which uses the SIM card of the phone as a token.

The second area of government-wide services AOC is responsible for ensuring interoperability. Different administrations need to exchange a lot of information. It was done on paper in the past, now it is mostly electronic.

Another area is the deployment of digital services in the cloud. AOC provides some of the services required to manage productivity, such as Software-as-a-Service (SaaS) solutions used by the government administrations.

The final area is change management. Mr. Estapé explained, “It’s important to have very good digital services but if you don’t have a smart change management strategy, it’s difficult to achieve real digital transformation. We have many training courses, so that we can make sure people learn how to use these new services.”

Disruptive technologies

AOC is conducting doing proof of concepts and pilots in several emerging technology areas.

For instance, AOC is using chatbots to improve user support. This involves natural language processing to understand the users’ queries and provide the right answers.

AOC is also providing blockchain-as-a-service so that any public administration in Catalonia can do trials, tests and proof of concepts using the technology. Mr. Estapé explained that at the moment there is a lot of hype around the technology, but ideas need to be tested and business cases found which will provide real value.

Citizen-centric services

Another area being explored is forecasting citizens’ needs and providing anticipatory services. This is This is done using big data and data science techniques to analyse the needs of users, detect patterns and provide personalised services to citizens, especially to the socially and economically vulnerable.

This is part of AOC’s attempts to adopt citizen-centric approach to delivering government services. “The issue is that I don’t think governments are used to working with a citizen-centric approach. We are trying to change the culture now,” Mr. Estapé said.

AOC is developing a systematic way to collect consumer feedback and get new ideas from citizens. Every time a citizen conducts a transaction, at the end there is a survey, which asks the user if they have any idea to improve the service. Users can submit ideas, see all the ideas, vote. AOC also launches challenges to solve specific issues that the public administrations do not know how to solve. Ideas are sought from the community, including citizens, public employees, government officials and so.

AOC is also analysing user behaviour when browsing public services. Mr. Estapé said that sometimes it’s better to see what citizens really do, rather than asking them what they want. There are tools to analyse and record all the browsing and so on.

Sometimes for these citizen-centric services, cross-agency or cross-government collaboration is required between governments and government agencies at different levels. AOC was created to facilitate and enable regional co-operation.

“Cooperation is one of our core values and we have a committee made up of representatives of several public administrations. So, this is not a project managed by one government. Having said that, there are quite a few collaboration challenges within Catalonia and also with the Spanish Government because there are different agendas, different objectives and it’s not always easy to align to work together along the same path,” Mr. Estapé said.

That’s why AOC spends a lot of time and effort trying to get everybody in the same boat within Catalonia and with the Spanish Government. That’s one area always taken into consideration every time a new initiative is launched, or new services deployed.

What does Open Government mean?

When asked the question above, Mr. Estapé responded, “For me, it’s what creates trust, accountability and collaboration with citizens. The outcome of a good Open Government policy is that citizens trust the government, they have accountability about government actions and they are really keen on collaboration with the government, because they are engaged with the government. So basically, Open Government is fundamental for a democratic society.”

There is a new regulation in Spain requiring opening up a lot of public information. Mr. Estapé said that it was a big challenge about how to open up so much information to the public. AOC has a shared digital service where all the public administrations in Catalonia can open all the information, proceedings, processes and so on. All this information is opened in a very structured way standard way for ease of access of the citizens.

Through an open data platform, information from all the local public entities in Catalonia is consolidated. All this information is available and compliant with all the international standards about interoperability and so on. AOC is also providing some visualisations of big data and easy-to-understand reports so citizens can understand how institutions are spending their money etc.

Instead of the town halls publishing the information on their own website, they are using AOC’s shared service and 50% of the information is published automatically. An estimated 1.5 million euros are being saved every year.

AOC also applies the ‘once only’ principle. It means that all Catalonian public administrations have the duty to submit information to the regional and federal governments. This includes information on personnel, budgets, debts, subsidies and so on and it ensures that the same information is not collected multiple times at multiple points.

One of the big challenges has been managing all the negotiations with the central government and local government to open up all their databases. “We have much more power because we represent all the public administrations in Catalonia. Basically, the challenge is a change of culture. We call this the “Gollum barrier” – after the character from the Lord of the Rings, who just doesn’t want to let go of his precious ring.”

“We are the owners of government data – we have been spending all our lives preserving and protecting the government data, and now there is a new regulation saying all this data should be open. But then they suffer from some kind of paranoia when someone wants to have access to this data. The challenge is how to change the mindsets of public employees and managers to embrace the values of sharing information, open data and accountability.”

Open data syndromes: from Gollum to Despacito

 

During the last two years, I have participated in the project of implementing open government policies in local authorities in Catalonia, within the framework of the “Network of Transparent Local Governments of Catalonia” and we have promoted the opening of public information ensuring open formats, interoperability, standardization, reusability, and good data quality.

We have done it by applying the “once only” principle of efficiency: that is, we do not ask local administrations to publish data that has already been reported previously to province, regional, or federal administrations: we have contacted them, and we have requested the opening of their consolidated data for all the local administrations about budgets, personnel, debt, subsidies, contracts, etc. This information has been made available on a shared open data platform, and more than 1,000 local public authorities that use the transparency service of the Open Government of Catalonia, have this information automatically published and regularly updated on its website, without the need to perform any additional task.

From this experience, I would like to present a list of the main barriers, attitudes, or “syndromes” that have shown up during the process of opening consolidated public data. According to the Collins dictionary, a “syndrome” is a group of symptoms that, together, are signs of a certain specific disorder or disease. Here is the list of “syndromes”:

Raiders of the lost law. Nobody in Spain mentions, remembers, comprehends what it says, or knows where to find the Directive on the re-use of public sector information (Directive 2003/98/EC entered into force in 2003 and revised by Directive 2013/37/EU, which came into effect on 2013). Their obligations are almost unheard of. Apparently, after its approval, this Directive was lost in the secret valley of the Well of the Souls in Egypt, next to the lost ark.

Gollum’s syndrome. In the movie, Lord of the Rings, the ring is “my treasure” and transforms the owner in a powerful but selfish person, obsessed with not sharing it and paranoid about “losing” it. It happens the same way with the government data. There are roles in the authorities that have spent their lifetime preserving and protecting the government data, and now suffer from “paranoia” when someone wants to make it open. The famous sentence, the “data belongs to citizens” is a chimera for them.

Kilian Jornet syndrome. The list of difficulties that show up when trying to open the public-sector data seems longer and more complicated than climbing Mount Everest, twice in a week, without artificial oxygen, as the famous Catalan climber, Kilian Jornet, did in May 2017. The data “guardians” claim privacy problems, poor data quality (with the risk of giving a very bad image), technological difficulties, high costs, that no one needs it, and so on. An extensive list of excuses: as Kilian Jornet says, “I do not look for excuses to train, do you?”

Twin Peaks syndrome. As in the Twin Peaks series, nothing is as it seems and at every step, there is a surprise. Data controllers agree to share the information in an open data format, but our initial satisfaction is transformed when we realize that it is only made available in PDF format. They are “semi-open” data, but quite “dark” data because they are not easy to exploit and analyze.

Forest Gump syndrome. This fictional character unintentionally became a world champion of ping-pong with a very simple strategy: to return all the balls and to defeat to the contrary by exhaustion. In the public authorities, we have many “Forest Gump” experts in the art of returning ping-pong requests. They are asked to share information in an open data format and respond, but with a different thing that obviously is not what we have asked for. We return the ping-pong ball with more energy and precision, and return it to us on the other side of “the table”, but continue to be of little use. And so, on and on, until we give up in exhaustion.

“Home alone” syndrome. We, sometimes, get this answer: of course, you can access “my public” information, but at my “home” alone (website). To access the data, we should create a link to “their house” and thus, it is evident who the “owner” of the data is and, moreover, the visitor counter of “their website” goes up.

Syndrome of the Despacito. You can access the data, and you can even download it in open and structured data files (CSV or spreadsheet) using a search engine but, as the song says, “despacito” (slowly). A restriction has been implemented in the search engine that only allows downloading the information in “little bits” of a few hundred records at a time, due to some strange technological limitations and so that no one gets filled up with too much data. If someone wants to do a global analysis of the data, they can do it, but “pasito a pasito” (step by step).

The Da Vinci Code syndrome. The data are all available in an open format, structured and standardized. It seems that we have finally succeeded, but we are surprised that some of the key codes to solve some puzzles are missing: for example, in a contractor’s database, the company code is missing, and we only have access to the name. As each public authority may have written the name of the company differently, it would be very complicated to do data crossings and analytical reports. Not even a Da Vinci’s genius would get it: for example, to analyze all contracts awarded to a company in various Administrations.

“Others have it bigger” syndrome. Excuse the foul language, but it is a very graphical expression. We eventually get to publish a data set of, for example, all the information about the budget of all the local administrations of Catalonia in open, interoperable, standardized, with all the key code data, etc. But then, it shows up an alleged expert who elaborates open data rankings that disregards it because it is only a single dataset, and puts as an example of good practice the authorities that have published hundreds of datasets. Apparently, the more the better. Analyzing the supposed good practices, we see that there are open data portals that have a dataset for each entity, for each year and for a very specific concept (for example, the budget of expenses by one of the three concepts of the Spanish accountability). Well, if we used this criterion, a single dataset of the budget of all local governments of Catalonia would become at least 30,000 datasets: (+1000 local entities) x (10 years of budget history) * (3 concepts). But what is the most useful if you want to do a comparative analysis per year or between local authorities?. I think, the simplest is the best.

Despite the difficulties, I believe that an excellent job has been done, as we have managed to publish 35 sets of open data, with consolidated information for all the local authorities of Catalonia, although certainly much remains to be done. As Confucius said, “the man who moves a mountain begins by carrying away small stones”. There we are.

Note: Thanks to Josep Matas

Síndromes de la apertura de datos públicos: de Gollum al Despazito

Durante los dos últimos años he participado en el proyecto de implantación de la transparencia en las entidades locales de Cataluña, en el marco de la “Red de Gobiernos Locales Transparentes de Cataluña” y hemos impulsado la apertura de la información pública en formatos abiertos, fácilmente reutilizables, interoperables, estandarizados y de calidad.

Lo hemos hecho aplicando el principio de eficiencia “once only” (solo una vez): es decir, no volver a solicitar a las administraciones locales que publiquen los datos que ya han comunicado, en algún momento, a una administración supramunicipal o superior: no hemos dirigido a éstas y hemos solicitado la apertura de sus ficheros que contienen los datos consolidados de todas las administraciones locales: presupuestos, personal, deuda, subvenciones, contratación, etc. Esta información la hemos puesto a disposición en una plataforma compartida de datos abiertos y, las más de 1.000 entidades locales usuarias del servicio de Transparencia del Consorci AOC, disponen de esta información publicada automáticamente y actualizada periódicamente en su portal, sin necesidad de realizar ninguna tarea adicional.

Os presento a continuación un listado de las principales barreras, actitudes o “síndromes” que han dificultado este proceso de apertura de los datos públicos en poder de una administración supramunicipal o superior. Según la Real Academia Española, un “síndrome” es un conjunto de signos o fenómenos reveladores de una situación negativa.

Síndrome de “En búsqueda de la Ley perdida”. Nadie menciona, ni sabe lo que dice, ni recuerda, ni tiene en cuenta la Ley 37/2007, sobre reutilización de la información del sector público. Hace diez años de su aprobación, pero sus obligaciones son unas perfectas desconocidas. Parece ser que, tras su aprobación, esta Ley se perdió en el valle incógnito del pozo de Almas en Egipto, junto al arca perdida.

Síndrome de Gollum. Como en la película de “El señor de los anillos”, el anillo es “mi tesoro” y se convierte en una persona poderosa y, al mismo tiempo, egoísta, obsesionada en no compartirlo y paranoica con que nadie se lo quite. Con los datos pasa lo mismo. Hay roles en la administración cuya función ha sido toda la vida conservar y proteger los datos, y sufren de “paranoias” cuando alguien los quiere “tocar”. Aquello de que “los datos son de los ciudadanos” es una quimera.

Síndrome de Kilian Jornet. La lista de dificultades que se exponen para no implantar la apertura de los datos del sector público parece más larga y compleja que escalar el Everest. Los responsables de los ficheros aducen problemas de privacidad, mala calidad de los datos (con el riesgo de dar muy mala imagen), de dificultades tecnológicas, que tendrá un coste económico grande, que nadie lo necesita, etc. Una larga lista de excusas. Como dice Kilian Jornet “yo no busco excusas para entrenar ¿y tú?”

Síndrome de Twin Peaks. Como en la serie de Twin Peaks, nada es lo que parece y a cada paso hay una sorpresa. Los responsables de los datos aceptan compartir la información en formato de datos abiertos, pero nuestra satisfacción inicial se transforma cuando se comprueba que únicamente se ponen a disposición en formato PDF. Son datos “semiabiertos”, tirando a “oscuros” porque no son nada fáciles de explotar y analizar.

Síndrome de Forest Gump. Este personaje de ficción se convirtió por casualidad en un campeón mundial de ping-pong con una estrategia muy simple: devolver todas las pelotas y vencer al contrario por agotamiento. En la Administración tenemos muchos “Forest Gump”, expertos en el arte del ping-pong de las solicitudes. Se les pide que compartan la información en formato de datos abiertos y responden, pero con una cosa diferente que, evidentemente, no es lo que hemos pedido. Devolvemos la pelota de ping-pong con más energía y precisión, y nos la devuelven por otro lado de “la mesa” pero continúa siendo poco útil. Y así repetidamente, hasta que desistimos por agotamiento.

Síndrome de “Solo en (mi) casa”. Por supuesto que puedes consultar “mi” información pública, pero “solo en mi casa”. Para consultar los datos debes crear enlace de tu web a “mi casa”, dónde hay un buscador para acceder a ella, y así queda muy claro quién es el “amo” de los datos y así, además, el contador de visitas a “mi web” sube.

Síndrome del “Despazito”. Puedes acceder a los datos e incluso te los puedes descargar en ficheros de datos abiertos y estructurados (CSV u hoja de cálculo) desde un buscador, pero “despazito”. Se ha implantado una restricción en el buscador que solo permite descargar la información en “trozitos” de pocos centenares de registros, por supuestas limitaciones tecnológicas y para que nadie se empache de tantos datos. Si alguien quiere hacer un análisis global de los datos, lo podrá hacer, pero muy “despazito”.

Síndrome de “El Código Da Vinci”. Los datos están todos disponibles en formato abierto, estructurado y estandarizado. Parece que finalmente lo hemos conseguido, pero nos encontramos con la sorpresa que faltan los códigos clave para resolver algunos enigmas: por ejemplo, en un fichero datos de contratación con las licitaciones de todas las entidades locales no se publica el código del NIF/CIF de la empresa adjudicataria, solo el nombre. Como cada administración ha escrito el nombre de la empresa de forma diferente, será tan complicado hacer análisis y cruces de datos que ni el genio de Da Vinci lo conseguiría. Por ejemplo, analizar todos los contratos adjudicados a una determinada empresa en diversas Administraciones.

Síndrome de “Los otros la tienen más grande”. Disculpar el lenguaje soez, pero es una expresión muy gráfica. Cuando finalmente se consigue publicar un conjunto de datos de, por ejemplo, toda la información de la liquidación del presupuesto de todas las administraciones locales de Cataluña en formato abierto, interoperable, estandarizado, bien codificado, etc, aparece algún “experto” que elabora ránquines de datos abiertos que lo desprecia porque es solo un único dataset, y pone como ejemplo de buena práctica a otras administraciones que tienen centenares de datasets. Analizadas esas supuestas buenas prácticas, vemos que hay portales de datos abiertos que publican un dataset por cada entidad, para cada año y por concepto detallado (por ejemplo, el presupuesto de gastos por capítulo económico). Bien, si utilizáramos este criterio, un único dataset de la liquidación del presupuesto de todas las administraciones locales de Cataluña se convertiría en, como mínimo 30.000 datasets: (+1000 entidades locales) x (10 años de historia de presupuestos) * (3 conceptos: ingresos, gastos por clasificación económica, gastos por programa). Pero, ¿qué es lo más útil si se quiere hacer un análisis comparativo por año o entre ayuntamientos?

A pesar de las dificultades, creo que se ha hecho un excelente trabajo, se ha conseguido publicar 35 conjuntos de datos abiertos, con información consolidada de todas las administraciones locales de Cataluña, aunque ciertamente queda mucho por hacer. Como decía Confucio “el hombre que mueve montañas empieza apartando piedrecitas”. Ahí estamos.

Nota: Agradecimientos a Josep Matas