Murmurations is an open source decentralised data sharing project, which provides a flexible approach to ontology to accelerate the use of interoperable, composable data.
The ultimate aim of Murmurations is to accelerate the use of interoperable, composable data – to put end users in charge of their own data and to increase the reliability, integrity, and portability of shared data.
Building on existing semantic web standards, Murmurations provides an alternative to centralised data silos, which makes it easy for people, projects and organisations to host and share data in standardised formats, making it interoperable across platforms and networks.
Every website, network or platform stores data about ‘nodes’ of some kind (e.g. Members, Organisations, Products, Pages etc) but sites rarely allow users to ‘own’ the data they run on, or to share this data openly, which stifles innovation and requires complex APIs to enable data exchange, combination, aggregation, integration or computation on separate data sets. Data sharing is often hard to achieve because every site has its own ontology or vocabulary and even minor differences in ontology make data seemingly incompatible.
Murmurations provides a mechanism for end users, website owners, platforms and networks to share data, regardless of their specific ontology. Instead of trying to enforce any particular ontology, Murmurations applies the principles of composability to enable data holders to share data using existing ontologies, or to build on other’s work, to liberate the potential of open data sharing across platforms and networks.
The flexible approach to ontology is built around an open-source field library, to which anyone can contribute new fields. These fields are assembled into schemas, which anyone can use, or build on, to create new schemas. Whenever fields are re-used, data which is shared using these fields becomes interoperable, increasing the total value of the data in the network – and the ability for deeper integration and advanced uses of the data.
Although Murmurations has been designed to enable data owners (end users) to host their own data, a ‘bridging mechanism’, known as the ‘data proxy’ has been designed to make it easy for networks (and other holders of large data sets) to open up their data to make it interoperable, without needing to wait for every node (person) to create a self-hosted profile.
By sharing data with the Murmurations network via a data-proxy, networks can avoid the need to build complex APIs and guarantee that their data will be compatible, and interoperable, with other data in Murmurations. As nodes start to host their own Profiles (which are more likely to be accurate and kept up to date than data maintained by a third party) aggregators can pull details from self-hosted nodes to supersede the details provided by third party data-proxies. For example, the Murmurations WordPress Aggregator plugin already has a working mechanism built in, which ranks Profile data which is stored at its own primary_url as more authoritative than data hosted at a third party URL – see this demo.
Murmurations and personal data
The Murmurations project has initially focussed on Organisational and Project level data, rather than Individual’s data – avoiding the complications of GDPR. However, we have already drafted a ‘Person’ schema based on the Schema.org Person taxonomy to explore the sharing of personal data. This, however, adds further complexity in order to safe-guard any private data which should not be openly accessible.
In order to enable Murmurations for personal profiles optional encryption will be required. There are several existing projects which are looking to solve this challenge and Murmurations does not aim to replicate their work.
One of the tasks on our roadmap is to research the current solutions and to integrate with the most robust and technically compatible technology in order to provide end users with a simple way to share their data openly, whilst retaining privacy of any element of their profile which they prefer to be private, or only accessible to authorised entities. We are involved with https://coopcreds.com/ and have also spoken to https://www.jlinc.com/ who are already developing solutions to this challenge. Please get in touch if you are interested in developing this aspect of Murmurations.
Components we have built so far include:
- An open source library of fields and schemas (collections of fields) and the specifications for creating new fields and schemas
- The Library API, which serves up all of the Schemas in the Library repository.
- An Index, which keeps track of all the Nodes (Profiles) in the network. Whenever someone (or something) updates its Profile it tells the Index. Aggregators can query the Index to find Profiles using their Schemas along with changes to existing Profiles, enabling them to provide up to date information in their maps, directories and content aggregators. The Index is built from several micro services and can be easily replicated.
- The Index API, which enables Nodes, using predefined Schemas, to add, update and delete their data in the Index so that Aggregators can discover them.
- The Murmurations Index Explorer – a GUI for exploring, and performing detailed searches on, the data in the Murmurations Index
- The Murmurations Profile Generator or MPG for short – a web application for creating Profiles. MPG automatically converts Schemas in the Library into forms to enable people to create the JSON documents for their Profiles. MPG can also host these JSON documents, which may seem an odd way to manage decentralised data, but is an important feature which makes the protocol accessible to people who can’t host their own Profile data.
- The Murmurations WordPress Node Plugin which makes it easy for WordPress users to publish Profiles for People, Projects and Offers & Wants.
- The Murmurations WordPress Aggregator plugin provides a quick and powerful way to set up a site that fetches and displays data from the Murmurations network. Out of the box, this data can be displayed on a geographic map, or in a directory. There are also endpoints compatible with Kumu to enable systems mapping, and client-side interfaces built in React that can be embedded in the WordPress site.
- The WordPress Node Plugin is currently being rewritten to be compatible with the latest version (v2.0.0) of the Murmurations Protocol.
- A live Status reporting tool to show the performance of the network
- Several demonstration maps and directories using the components above
- Docs explaining how all the components can be used
Murmurations has been under part-time development (slowed by budget constraints) for roughly three years and is now in its third iteration. Previous versions (v0.1 and v1.0) enabled us to discover some of the more complex technical challenges, which we have now addressed in version 2.0 – which includes a completely re-built API.
Expected outcomes
The expected outcomes of the project are a growing volume of interoperable data – and open source building blocks which simplify decentralised data sharing – to enable more advanced uses of related data. Composable data – the idea that the information that powers our online experiences can be shared, used, and ‘composed on’ by applications across the web – is a core piece of the Web3 vision. Murmurations aims to enable this vision by providing a mechanism to share data in interoperable formats, which has the potential to change not only how applications are built but what an application is.