ElasticSearchView

This view creates an ElasticSearch index and stores the targeted Json resources into an ElasticSearch Document.

The documents created on each view are isolated from documents created on other views by using different ElasticSearch indices.

A default view gets automatically created when the project is created but other views can be created.

Processing pipeline

An asynchronous process gets trigger for every view. This process can be visualized as a pipeline with different stages.

The first stage is the input of the pipeline: a stream of events scoped for the project where the view was created.

The last stage takes the JSON document, generated through the pipeline steps, and stores it as a Document in an ElasticSearch index

ElasticSearchView pipeline

Payload

{
  "@id": "{someid}",
  "@type": "ElasticSearchView",
  "resourceSchemas": [ "{resourceSchema}", ...],
  "resourceTypes": [ "{resourceType}", ...],
  "resourceTag": "{tag}",
  "sourceAsText": {sourceAsText},
  "includeMetadata": {includeMetadata},
  "includeDeprecated": {includeDeprecated},
  "mapping": _elasticsearch mapping_
}

where…

  • {resourceSchema}: Iri - Selects only resources that are validated against the provided schema Iri. This field is optional.
  • {resourceType}: Iri - Select only resources of the provided type Iri. This field is optional.
  • {tag}: String - Selects only resources with the provided tag. This field is optional.
  • _elasticsearch mapping_: Json object - Defines the value types for the Json keys, as stated at the ElasticSearch mapping documentation.
  • {sourceAsText}: Boolean - If true, the resource’s payload will be stored in the ElasticSearch document as a single escaped string value of the key _original_source. If false, the resource’s payload will be stored normally in the ElasticSearch document. The default value is false.
  • {includeMetadata}: Boolean - If true, the resource’s nexus metadata (_constrainedBy, _deprecated, …) will be stored in the ElasticSearch document. Otherwise it won’t. The default value is false.
  • {includeDeprecated}: Boolean - If true, deprecated resources are also indexed. The default value is false.
  • {someid}: Iri - The @id value for this view.

Example

The following example creates an ElasticSearch view that will index resources validated against the schema with id https://bluebrain.github.io/nexus/schemas/myschema. If a resource is deprecated, it won’t be selected for indexing.

The resulting ElasticSearch Documents fields will be indexed according to the provided mapping rules and they won’t include the resource metadata fields.

{
  "@id": "https://bluebrain.github.io/nexus/vocabulary/myview",
  "@type": [
    "ElasticSearchView"
  ],
  "mapping": {
    "dynamic": false,
    "properties": {
      "@id": {
        "type": "keyword"
      },
      "@type": {
        "type": "keyword"
      },
      "name": {
        "type": "keyword"
      },
      "number": {
        "type": "long"
      },
      "bool": {
        "type": "boolean"
      }
    }
  },
  "includeMetadata": false,
  "includeDeprecated": false,
  "sourceAsText": false,
  "resourceSchemas": [
    "https://bluebrain.github.io/nexus/schemas/myschema"
  ],
  "resourceTypes": []
}

Endpoints

The following sections describe the endpoints that are specific to an ElasticSearchView.

The general view endpoints are described on the parent page.

Search Documents

POST /v1/views/{org_label}/{project_label}/{view_id}/_search
  {...}

The supported payload is defined on the ElasticSearch documentation

The string documents is used as a prefix of the default ElasticSearch view_id

Example

Request
curl -XPOST -H "Content-Type: application/json" "https://nexus.example.com/v1/views/myorg/myproj/nxv:myview/_search" -d \
'{
  "query": {
    "term": {
      "_deprecated": true
    }
  }
}'
Payload
{
  "query": {
    "term": {
      "_deprecated": true
    }
  }
}
Response
{
  "_shards": {
    "failed": 0,
    "skipped": 0,
    "successful": 0,
    "total": 0
  },
  "hits": {
    "hits": [
      {
        "_score": 1.2039728,
        "_id": "http://example.com/myview",
        "_index": "kg_v1_803da242-d510-4e1a-a12d-d0e0b7018a41_684bd815-9273-46f4-ac1c-0383d4a98254_1",
        "_source": {
          "@id": "http://example.com/myview",
          "@type": [
            "https://bluebrain.github.io/nexus/vocabulary/View",
            "https://bluebrain.github.io/nexus/vocabulary/ElasticSearchView",
            "https://bluebrain.github.io/nexus/vocabulary/Alpha"
          ],
          "_original_source": "{\"mapping\":\"{\\\"dynamic\\\":false,\\\"properties\\\":{\\\"@id\\\":{\\\"type\\\":\\\"keyword\\\"},\\\"@type\\\":{\\\"type\\\":\\\"keyword\\\"},\\\"name\\\":{\\\"type\\\":\\\"keyword\\\"},\\\"number\\\":{\\\"type\\\":\\\"long\\\"},\\\"bool\\\":{\\\"type\\\":\\\"boolean\\\"}}}\",\"_uuid\":\"89dd9858-cf87-4c46-94c2-24da0fcae0aa\",\"@context\":\"https://bluebrain.github.io/nexus/contexts/view\",\"@type\":[\"View\",\"ElasticSearchView\"],\"includeMetadata\":true,\"sourceAsText\":false}",
          "_incoming": "https://nexus.example.com/v1/schemas/myorg/myproj/nxv:myview/incoming",
          "_outgoing": "https://nexus.example.com/v1/schemas/myorg/myproj/nxv:myview/outgoing",
          "_self": "https://nexus.example.com/v1/schemas/myorg/myproj/nxv:myview",
          "_constrainedBy": "https://bluebrain.github.io/nexus/schemas/view",
          "_project": "https://nexus.example.com/v1/projects/myorg/myproj",
          "_rev": 6,
          "_deprecated": true,
          "_createdAt": "2018-09-18T16:39:22.748Z",
          "_createdBy": "https://nexus.example.com/v1/realms/myrealm/users/john",
          "_updatedAt": "2018-09-18T17:10:22.748Z",
          "_updatedBy": "https://nexus.example.com/v1/realms/myrealm/users/john"
        },
        "_type": "doc"
      }
    ],
    "max_score": 1.2039728,
    "total": 1
  },
  "timed_out": false,
  "took": 11
}

Fetch statistics

GET /v1/views/{org_label}/{project_label}/{view_id}/statistics

Example

Request
curl "https://nexus.example.com/v1/views/myorg/myproj/nxv:myview/statistics"
Response
{
  "@context": "https://bluebrain.github.io/nexus/contexts/statistics.json",
  "@type": "ViewStatistics",
  "totalEvents": 3754,
  "processedEvents": 3754,
  "evaluatedEvents": 3754,
  "remainingEvents": 0,
  "discardedEvents": 0,
  "failedEvents": 0,
  "delayInSeconds": 0,
  "lastEventDateTime": "2020-01-31T15:04:44.021Z",
  "lastProcessedEventDateTime": "2020-01-31T15:04:44.021Z"
}

where…

  • totalEvents - total number of events in the project
  • processedEvents - number of events that have been considered by the view
  • remainingEvents - number of events that remain to be considered by the view
  • discardedEvents - number of events that have been discarded (were not evaluated due to filters, e.g. did not match schema, tag or type defined in the view)
  • evaluatedEvents - number of events that have been used to update an index
  • lastEventDateTime - timestamp of the last event in the project
  • lastProcessedEventDateTime - timestamp of the last event processed by the view
  • delayInSeconds - number of seconds between the last processed event timestamp and the last known event timestamp

Restart view

This endpoint restarts the view indexing process. It does not delete the created indices but it overrides the resource Document when going through the event log.

DELETE /v1/views/{org_label}/{project_label}/{view_id}/offset

Example

Request
curl -XDELETE "https://nexus.example.com/v1/views/myorg/myproj/nxv:myview/offset"
Response
{
  "@context": "https://bluebrain.github.io/nexus/contexts/offset.json",
  "@type": "NoOffset"
}
0cf8a360