In order to accommodate more features and improve various aspects of how Murmurations works we have developed an improved version (V2) of the Index and its API to allow aggregators to perform more sophisticated searches.
We are also making significant changes to the schemas and fields in the Murmurations Library, which will enable better forward compatibility of schemas and reuse of fields.
All of this is still a work in progress, but we are currently testing V2 in our staging environment and encourage aggregators to try it out. Once we’ve incorporated everyone’s feedback and are happy that V2 is working as intended we will make it live, thus phasing out V1.
Ultimately the V2 updates will mean aggregators can do more things to deliver better usability for searchers. For example, ‘tag input’ will enable aggregators to create maps of projects within specific areas which match a certain topic.
The main changes from V1 include:
Index
- Enabled the storing and searching of tags/keywords in the index (issue 227) – The Library now defines a tags field that holds an array of strings to describe a node profile. These keywords enable aggregators to search the Index for relevant nodes by type or activity, for easier filtering of the data.
- Added a `primary_url` field to the index to uniquely identify each node in the network based on their website (issue 238) – The purpose of the primary_url field is to uniquely identify each node in the network and enable aggregators to query the index by `primary_url`.
- Added `deleted` status to the index (issue 217) – In v1 of the index API, when a node was deleted it was simply removed from the database, which meant there was no way to find out that it was deleted other than to try to retrieve it again. Now a deleted node is labelled as such and the `last_updated` timestamp set when the deletion occurs, so aggregators can find deleted nodes more easily and remove them from their own databases.
- Added a profile validation endpoint so node operators can test their profile is valid before adding it to the index (issue 245) – this helps nodes ensure their profile(s) validates to the schema(s) they have chosen before they post it to their website and submit the profile to the Index.
- Enabled support for two types of country inputs (issue 211) – so now the index can accept either `country_iso_3166-1` and/or `country_name`.
- Enabled flattened geolocation fields to better accommodate SQL database structures when importing data (issue 208) – so now it’s possible to provide geopoints in a flattened structure, i.e., two separate fields: latitude (number) and longitude (number).
- Enabled flattened location fields (issue 210) – so now location data (country, region, locality) can be submitted, and returned, in a flattened format rather than being embedded in a location object.
- Added the ability to search against multiple versions of the same schema in the index (issue 252) – so now, even when schemas are updated it is possible for aggregators to search for profiles which match multiple versions of the same schema using semver wildcards.
- Changed `last_validated` parameter to `last_updated` to better reflect that profile changes are tracked by the index (issue 184) – ‘last updated’ is a more accurate definition since a profile is always validated when it is posted to the index, so we wanted to make it clearer that the timestamp is about a change to the profile itself (i.e., an update).
- Improved developer experience by enabling the registration of a profile and receiving the outcome in one step (issue 152) – previously, in V1, to register a profile, and ensure that it worked, you needed to Register by posting the profile’s URL to the Index and then check whether it succeeded by fetching the node’s status from the Index. Now the Index API V2 returns the node’s status immediately in the response after a node is submitted.
- Improved testing infrastructure by adding end-to-end tests for all code pushes and deployments. This means we can introduce new changes with the confidence that they have not broken existing functionality.
Library/Schemas
- We’ve updated the schemas in the Murmurations Library to use JSON Schema version 7 (v1 used version 4) which will improve future compatibility and enable the use of v7 features.
- We’ve added semantic versioning to all schemas to make schema management easier and predictable.
- We’ve removed semantic versioning from fields to avoid field version conflicts when creating profiles with multiple schemas.
We’re now working on a new version of the Murmurations Profile Generator, the Murmurations WordPress Aggregator and a new, V2 compatible, version of the Murmurations WordPress Node Plugin.
Watch this space.