v1.3 To v1.4 Migration

As part of the v1.4 release, cassandra schema has changed due to a dependency update. Additionally, the 3 existing keyspaces (iam, admin and kg) must be merged into a single keyspace (delta).

Steps

The migration steps are as follows:

1. Stop the 3 services: iam, admin and kg.

2. Backup the cassandra store.

3. Delete all ElasticSearch indices:

   curl -XDELETE 'http://{elasticsearch_host}/kg_*'
   

4. Delete all BlazeGraph namespaces:

   for i in `curl -s 'http://{blazegraph_host}/blazegraph/namespace?describe-each-named-graph=false' | grep sparqlEndpoint | grep -o --color "rdf:resource=\"[^\"]*" | sed 's/rdf:resource="//' | sed 's#/sparql$##' | grep -v kb | grep -v LBS`
      do curl -X DELETE "$i"
   done
   

5. Make sure to allocate enough space to Cassandra, at least twice the space used by the keyspaces iam, admin and kg. The data migration involves copying the messages tables.

6. Create the delta keyspace and tables:

   CREATE KEYSPACE IF NOT EXISTS delta WITH replication = {'class': 'NetworkTopologyStrategy', '<your_dc_name>' : 3 };
   
   CREATE KEYSPACE IF NOT EXISTS delta_snapshot WITH replication = {'class': 'NetworkTopologyStrategy', '<your_dc_name>' : 3 }; 
   
   CREATE TABLE IF NOT EXISTS delta.messages (
     persistence_id text,
     partition_nr bigint,
     sequence_nr bigint,
     timestamp timeuuid,
     timebucket text,
     writer_uuid text,
     ser_id int,
     ser_manifest text,
     event_manifest text,
     event blob,
     meta_ser_id int,
     meta_ser_manifest text,
     meta blob,
     tags set<text>,
     PRIMARY KEY ((persistence_id, partition_nr), sequence_nr, timestamp))
     WITH gc_grace_seconds =864000
     AND compaction = {
       'class' : 'SizeTieredCompactionStrategy',
       'enabled' : true,
       'tombstone_compaction_interval' : 86400,
       'tombstone_threshold' : 0.2,
       'unchecked_tombstone_compaction' : false,
       'bucket_high' : 1.5,
       'bucket_low' : 0.5,
       'max_threshold' : 32,
       'min_threshold' : 4,
       'min_sstable_size' : 50
       };
   
   CREATE TABLE IF NOT EXISTS delta.tag_views (
     tag_name text,
     persistence_id text,
     sequence_nr bigint,
     timebucket bigint,
     timestamp timeuuid,
     tag_pid_sequence_nr bigint,
     writer_uuid text,
     ser_id int,
     ser_manifest text,
     event_manifest text,
     event blob,
     meta_ser_id int,
     meta_ser_manifest text,
     meta blob,
     PRIMARY KEY ((tag_name, timebucket), timestamp, persistence_id, tag_pid_sequence_nr))
     WITH gc_grace_seconds =864000
     AND compaction = {
       'class' : 'SizeTieredCompactionStrategy',
       'enabled' : true,
       'tombstone_compaction_interval' : 86400,
       'tombstone_threshold' : 0.2,
       'unchecked_tombstone_compaction' : false,
       'bucket_high' : 1.5,
       'bucket_low' : 0.5,
       'max_threshold' : 32,
       'min_threshold' : 4,
       'min_sstable_size' : 50
       };
   
   CREATE TABLE IF NOT EXISTS delta.tag_write_progress(
     persistence_id text,
     tag text,
     sequence_nr bigint,
     tag_pid_sequence_nr bigint,
     offset timeuuid,
     PRIMARY KEY (persistence_id, tag));
   
   CREATE TABLE IF NOT EXISTS delta.tag_scanning(
     persistence_id text,
     sequence_nr bigint,
     PRIMARY KEY (persistence_id));
   
   CREATE TABLE IF NOT EXISTS delta.metadata(
     persistence_id text PRIMARY KEY,
     deleted_to bigint,
     properties map<text,text>);
   
   CREATE TABLE IF NOT EXISTS delta.all_persistence_ids(
     persistence_id text PRIMARY KEY);
   
   CREATE TABLE IF NOT EXISTS delta.projections_progress (
       projection_id text PRIMARY KEY,
       progress text
   ) WITH bloom_filter_fp_chance = 0.01
       AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
       AND comment = ''
       AND compaction = {'class': 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy', 'max_threshold': '32', 'min_threshold': '4'}
       AND compression = {'chunk_length_in_kb': '64', 'class': 'org.apache.cassandra.io.compress.LZ4Compressor'}
       AND crc_check_chance = 1.0
       AND dclocal_read_repair_chance = 0.1
       AND default_time_to_live = 0
       AND gc_grace_seconds = 864000
       AND max_index_interval = 2048
       AND memtable_flush_period_in_ms = 0
       AND min_index_interval = 128
       AND read_repair_chance = 0.0
       AND speculative_retry = '99PERCENTILE';
   
   CREATE TABLE IF NOT EXISTS delta.projections_failures (
       projection_id text,
       offset text,
       persistence_id text,
       sequence_nr bigint,
       value text,
       PRIMARY KEY (projection_id, offset, persistence_id, sequence_nr)
   ) WITH CLUSTERING ORDER BY (offset ASC, persistence_id ASC, sequence_nr ASC)
       AND bloom_filter_fp_chance = 0.01
       AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
       AND comment = ''
       AND compaction = {'class': 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy', 'max_threshold': '32', 'min_threshold': '4'}
       AND compression = {'chunk_length_in_kb': '64', 'class': 'org.apache.cassandra.io.compress.LZ4Compressor'}
       AND crc_check_chance = 1.0
       AND dclocal_read_repair_chance = 0.1
       AND default_time_to_live = 0
       AND gc_grace_seconds = 864000
       AND max_index_interval = 2048
       AND memtable_flush_period_in_ms = 0
       AND min_index_interval = 128
       AND read_repair_chance = 0.0
       AND speculative_retry = '99PERCENTILE';
   
   CREATE TABLE IF NOT EXISTS delta_snapshot.snapshots (
     persistence_id text,
     sequence_nr bigint,
     timestamp bigint,
     ser_id int,
     ser_manifest text,
     snapshot_data blob,
     snapshot blob,
     meta_ser_id int,
     meta_ser_manifest text,
     meta blob,
     PRIMARY KEY (persistence_id, sequence_nr))
     WITH CLUSTERING ORDER BY (sequence_nr DESC) AND gc_grace_seconds =864000
     AND compaction = {
       'class' : 'SizeTieredCompactionStrategy',
       'enabled' : true,
       'tombstone_compaction_interval' : 86400,
       'tombstone_threshold' : 0.2,
       'unchecked_tombstone_compaction' : false,
       'bucket_high' : 1.5,
       'bucket_low' : 0.5,
       'max_threshold' : 32,
       'min_threshold' : 4,
       'min_sstable_size' : 50
       };
   

7. Check and update the necessary configuration environment variables/JVM properties according to configuration migration guide

8. Deploy the new service images (tag 1.4) for delta and run it with the MIGRATE_V13_TO_V14=true and REPAIR_FROM_MESSAGES=true environment variables. This will instruct the service to copy all the messages table from iam, admin and kg into the delta keyspace. Afterwards the tag_views table and their related tables will be properly initialized. When populating the tag_views table consider increasing the batch_size_fail_threshold_in_kb in your cassandra.yaml configuration file if you see TagWriter error messages. Decreasing the value of the JVM property akka.persistence.cassandra.events-by-tag.max-message-batch-size and akka.persistence.cassandra.journal.max-message-batch-size to 1 can also help mitigate the issue.

9. Once the service is up and running, and you have verified the resources are accessible, you can proceed to delete the previous keyspaces:

   DROP KEYSPACE iam;
   DROP KEYSPACE admin;
   DROP KEYSPACE kg;
   

Configuration

This section provides a table with the equivalences in terms of configuration environment variables from 1.3 (and older versions) to 1.4. Only the most relevant environment configuration properties are described

There are 2 ways to modify the default configuration values:

• Using JVM properties as arguments when running the service: -D{property}. For example: -Dapp.instance.interface="127.0.0.1".
• Setting FORCE_CONFIG_ style environment variables. In order to enable this style of configuration the JVM property -Dconfig.override_with_env_vars=true needs to be set. For example: CONFIG_FORCE_app_instance_interface="127.0.0.1".

Note that there has been one important change which cannot be mapped using the following table: the service account token.

In 1.4 release, the IAM_SA_TOKEN is not used. Instead, the JVM properties app.service-account-caller.realm, app.service-account-caller.subject and app.service-account-caller.groups should be set matching the identity of the service account. If these properties are not present, the service account identity will be set to anonymous.