v1.8 Release Notes
This release marks a turn in the Nexus Ecosystem with different important changes:
The support for Cassandra as a primary store is removed and PostgreSQL 15+ is now the only supported primary store.
The sourcing and indexing engine have been completely reworked to improve flexibility, scalability and performance.
For the detailed list of updates in this release, see the list of addressed issues since v1.7.
If you are currently using Nexus Delta from 1.7.x, please visit the migration instructions for the necessary steps to successfully upgrade.
Nexus Delta
The goal of this release is to improve performance, flexibility, observability and developer productivity. This is an incremental software release that also brings several improvements.
The API is mostly backwards compatible to v1.7 with two breaking changes due to changes in the sourcing and indexing engine:
- The offset is not a time-based uuid anymore but a sequence which consists of a bigint,
- As the indexing engine do not rely on events on anymore, the statistics for the different views returns the number of records that have been processed/remains to be processed.
New primary store
Until Nexus 1.7, Nexus Delta was relying on Apache Cassandra as a primary store that is to say the source of truth on which the Nexus ecosystem depends.
However, Cassandra became a growing problem for Nexus both in development and production contexts:
- The learning curve is steep both for using it in a developer context and running it in production,
- The tooling that allow to run it in a daily basis can be insufficient, time-consuming to properly set it up,
- The data modeling it imposes did not allow to accommodate the growing variety of present and future usages of Nexus,
- The eventual consistency was another hurdle for the Nexus usages
-
The akka-persistence library we relied on which is part of the akka stack changed its license in September 2022, so we could not get new updates from it,
-
This library also did not fit anymore our needs in terms of indexing to external systems as performance is degrading over time but also for data deletion.
So this pushed us to look for alternatives that would better address our needs.
PostgreSQL is a database system with a strong reputation for performance, reliability and flexibility. It allows to solve the pain points we were facing with Cassandra:
- Easier to learn, to operate in a daily basis,
- Can be run in different contexts from integration to large-scale production deployments,
- Widely used, a lot of resources and tools to use/deploy/monitor it,
- Flexible about data modeling,
- Transaction support, data integrity,
- Json support.
These reasons make PostgreSQL a better solution to be the primary store from Nexus 1.8 and to create a new sourcing and indexing library on top of it.
New sourcing and streaming library
This change of primary store in Nexus also comes with a new sourcing and streaming library. It relies on a custom schema and is designed to:
- Embed directly the key concepts of project and entity type in the data model
- Be extensible by allowing to introduce new entities both global and related to a project
- Keep the history of changes by storing the log of events of the different entities
- Keep snapshots of entity states to allow:
- Faster responses from the API as latest and tagged states don’t need to be recomputed read and write actions
- Faster indexing by allowing to fetch directly states before processing them
- Allow to delete undesirable projects in an efficient way
New clustering deployment
If you were running Nexus in a cluster configuration, Nexus 1.8 doesn’t rely anymore on Akka Cluster and moved to a new model described in the Architecture page.
We moved away from akka-cluster for the same reasons as for akka-persistence (the license change and it did not fit anymore Nexus needs).
Name and description for resolvers, storages and views
It could be difficult for users to identify these different entities (for example in Fusion). These two additional fields fills this gap by allowing users to add extra information.
Views
Better reporting of indexing errors
Indexing errors for the different views are now available as Server Sent Events via the API making it easier to identify and fix them by either updating the resource or updating the view.
Errors related to indexing into Elasticsearch have also been improved to give more precise and relevant feedback.
Only trigger reindexing when indexing is impacted
Indexing views will now only restart if its new configuration changes the way, data is impacted. For example, updating the name, the description or the permission of the view won’t trigger reindexing anymore.
Indexing @id for custom Elasticsearch views
When a resource does not include an id in its original payload, the id of the resource (for example in the case, the @id is generated) will be injected in the Elasticsearch payload.
Projects
Project deletion
Before Nexus 1.8, akka-persistence and Apache Cassandra made it very hard to delete entities in a performant and reliable way.
PostgreSQL and the new sourcing library are now much more adapted to this special use-case and the feature has been completely refactored.
Resources and schemas
Search by @id and _self
A new parameter locate has been introduced in the listing API that enables searching a resource within a project, an org or within all projects by providing its @id or its _self
Refresh resources and schemas
A new refresh operation is available for resources and schemas. It allows to update a resource/schema when one of its upstream dependencies has changed without having to provide again its original payload. More information for resources More information for schemas
Validate
This new validate operation allows to check a resource against a provided schema. This operation is read-only and allows to test a schema against different resources
Annotated original payload
It is now possible to fetch in a single the original payload of a resource alongside its metadata: More information
Supervision API
A supervision API has been added to get information about the running projections on the current node:
Archives
Archives can now be downloaded in a zip format.
Deprecations
In the next version, the support of tarballs as a format for archives is going to be removed because its limitations make it difficult to work properly with Nexus as resource and project identifiers tend to be long and the tar format is very restrictive on that. Only the zip format will remain as an option to download archives.
Nexus Fusion
The goal of this release was to improve the user experience and functionality of the product. Specifically, redesigning and enhancing key pages, including Login, Home, , Organization listing, Projects listing, Studios listing, Project, Studio, Data Panel, and About Modal. Additionally, it introduced improvements to plugins and app configuration. These enhancements aim to provide users with better navigation, easier access to resources, improved data management capabilities, and an overall more user-friendly interface.
- The Home page was redesigned and includes :
- An improved top header navigation menu
- A global search page types navigation cards.
- A top level entity navigation cards.
- A “my data” section to facilitate users to find recently created resources.
- Rebuilt the Login page and shortcuts to documentation and public studios.
- Improved Organization listing page:
- List all the organisations with count of projects
- Added a “Create New organization” action button
- Improved Projects listing page:
- Ability to show the overall projects in Nexus deployment or a specific organization’s projects,
- Show each project’s creation/update dates and count of datasets,
- Create New Project action button.
- Improved Studios listing page:
- Ability to show the overall studios in Nexus deployment or a specific project’s studios,
- Added “Create New Studio” action button.
- Improved Project page:
- Enhanced Browse tab (filtering and searching)
- Enhanced Setting tab
- Enable the user to create new View, Storage, Resolver,
- Run ID resolution by Resolvers.
- Enhanced Query tab
- Present errors to the user when running a
SPARQLQuery against a View - Danger zone
- ability to deprecate a project with security checks.
- Improved Studio page:
- Enhance the Error Display in Studio.
- Enable sorting and filtering for columns.
- Show total results count.
- Allow users to select rows for downloading.
- Improved Resource Panel:
- Allow the user to select resource data from different places in the application (My data page, Global Search Types Pages, Studio page).
- Improved ability to download all the data, or some specific data based on file type extension and bundle the data in a .tar file.
- Plugins:
- Improved image plugin:
- The user can display the images of a resources as a gallery mode, with the ability to download them.
- Improved Preview plugin
- Preview
csvfiles.
- Improved About Modal:
- Show the Nexus ecosystem versions (delta and fusion).
- Show the current deployment location.
- Allow user to copy environment information.
- App Configuration:
- The Application manager has the ability to select his favourites images for different top-level pages and the video in the Login page.
Nexus forge
TODO