Migrating to Unified Data Classification

The Unified Data Classification method is the default data classification method.
With release 2024.07, a migration process is available. The migration process:

  • Copies classification information from older, deprecated classification methods, old Edge classification method and Cloud Data Classification Platform, into the Unified Data Classification method.

    Note When you use the API to manage classifications and Unified Data Classification is not activated, then the data you manipulate is considered to be part of the Cloud Data Classification Platform or old Edge data classification method, and needs to be migrated.

  • Creates data classes in the Unified Data Classification method for existing Advanced Data Types (ADTs). ADTs are supported only for Jobserver, which will be end of life on September 30, 2024.

What happened during the 2024.07 upgrade and what are the possible next steps?

During the 2024.07 upgrade, we didn't update the data classification information in the environment.
After the 2024.07 upgrade, you can Start the migration process manually if you want to migrate old classification information and ADTs.

What happened during the 2024.07 upgrade?

  • We activated the Unified Data Classification method.
  • We added the Catalog Data Classification capability to all Edge connections that have the JDBC Profiling capability.
  • We started the migration process. For more information go to What happens during the migration process?.

What are the possible next steps?

  • Start using the Unified Data Classification method.

    1. On the main toolbar, click Products icon, and then click Stewardship.
    2. Click the Data Classification tab.
    3. Check the migrated data classes and classifications.
      In the Data Classification page, you can filter the data classes based on their name. The names of migrated data classes end with (migrated) or (ADT).

      Tip If the renaming data classes leads to duplicate data class names, merge your data classes instead.

    4. Delete any unneeded data classes or add a classification rule if none was added by the migration process.
    5. If you are using Protect, check the standards and rules that are based on data classifications and make sure the correct data classes are added.
    6. Once the migration is completed, disable the old Edge classification feature in your Edge site.
      You no longer need to enable classification on your Edge site because Unified Data Classification uses an Edge capability instead.
      • K3S
      • Managed Kubernetes

      To disable classification on an existing Edge site, deployed on K3S, run this command:

      Copy
      sudo ./edgecli update --set collibra.classification.enabled=false

      Tip The only difference between disabling classification and enabling classification is that the last argument is false instead of true.

      Note Edge supports the following managed Kubernetes clusters:
      • AKS
      • AWS Fargate using EKS
      • EKS
      • GKE
      • OpenShift

      Follow the steps below if you use a supported Kubernetes cluster.

      To disable classification on an existing Edge site, deployed on your dedicated cluster, run this command:

      Copy
      ./edgecli update --set collibra.classification.enabled=false

      Tip The only difference between disabling classification and enabling classification is that the last argument is false instead of true.

  • Disable Unified Data Classification and migrate manually at a later date.

    1. On the main toolbar, click Products icon, and then click Stewardship.
    2. Click the Data Classification tab.
    3. Delete all data classes with a name that ends with (migrated) or (ADT) from the page.
      If you delete a data class from Unified Data Classification, it’s not removed from the old classification method.
    4. Disable Unified Data Classification.
      The old Edge classification method and data is shown again. Note that this method will be end of life in September 2024.
      1. In Collibra Console, go to set the Unified Classification enabled setting to false.
      2. Click Save and restart Collibra.

    5. Once you’re ready for the migration, Start the migration process manually.

What happened during the 2024.07 upgrade?

What are the possible next steps?

  • Start using the Unified Data Classification method.

    1. Migrate to Edge.
      Make sure to add at least the Catalog JDBC ingestion capability and the Catalog Data Classification capability to use Unified Data Classification.
    2. Once you have migrated to Edge, start using Unified Data Classification.
      1. On the main toolbar, click Products icon, and then click Stewardship.
      2. Click the Data Classification tab.
      3. Check the migrated data classes and classifications.
        In the Data Classification page, you can filter the data classes based on their name. The names of migrated data classes end with (migrated) or (ADT).

        Tip If the renaming data classes leads to duplicate data class names, merge your data classes instead.

      4. Delete any unneeded data classes or add a classification rule if none was added by the migration process.

  • Disable Unified Data Classification and migrate at a later date.

    1. On the main toolbar, click Products icon, and then click Stewardship.
    2. Click the Data Classification tab.
    3. Delete all data classes with a name that ends with (migrated) or (ADT) from the page.
      If you delete a data class from Unified Data Classification, it’s not removed from the old classification method.

      Tip If the renaming data classes leads to duplicate data class names, use merge your data classes instead.

    4. Disable Unified Data Classification.
      The Cloud Data Classification Platform method and data is shown again. Note that this method will be end of life in September 2024.
      1. In Collibra Console, go to set the Unified Classification enabled setting to false.
      2. Click Save and restart Collibra.

    5. Once you’re ready for the migration:

      1. Migrate to Edge.
        Make sure to add at least the Catalog JDBC ingestion capability and the Catalog Data Classification capability to use Unified Data Classification.
      2. Start the migration process manually.

What happened during the 2024.07 upgrade?

What are the possible next steps?

  • Start using Unified Data Classification.

    • If you created the classifications based on data class name:
      1. On the main toolbar, click Products icon, and then click Stewardship.
      2. Click the Data Classification tab.
      3. For all data classes that have a name that ends with (migrated) or (ADT), remove the (migrated) or (ADT) suffix.
        In the Data Classification page, you can filter the data classes based on their name. The names of migrated data classes end with (migrated) or (ADT).

        Tip If the renaming data classes leads to duplicate data class names, use merge your data classes instead.

      4. Delete any unneeded data classes.

    • If you created the classifications based on data class ID, update the API calls to refer to the new data class IDs.

      1. Get a list of the old data class IDs in your environment.
      2. Get a list of the migrated data class IDs.
      3. Update the API calls to refer to the new data class IDs.

  • Disable Unified Data Classification and migrate at a later date.
    Note that using the Classification APIs with Unified Data Classification disabled will be end of life in September 2024.

    1. On the main toolbar, click Products icon, and then click Stewardship.
    2. Click the Data Classification tab.
    3. Delete all data classes with a name that ends with (migrated) or (ADT) from the page.
      If you delete a data class from Unified Data Classification, it’s not removed from the old classification method.
    4. Disable Unified Data Classification.
      The previous data is shown again.
      1. In Collibra Console, go to set the Unified Classification enabled setting to false.
      2. Click Save and restart Collibra.

    5. Once you’re ready for the migration:

      1. Get a list of all the data classes in your environment.
      2. Enable the Unified Data Classification and the Unified Classification migration tool enabled settings.
      3. Start the migration process manually.
      4. Do one of the following:
        • If you created the classifications based on data class name:
          1. On the main toolbar, click Products icon, and then click Stewardship.
          2. Click the Data Classification tab.
          3. For all data classes that have a name that ends with (migrated) or (ADT), remove the (migrated) or (ADT) suffix.
            In the Data Classification page, you can filter the data classes based on their name. The names of migrated data classes end with (migrated) or (ADT).

            Tip If the renaming data classes leads to duplicate data class names, merge your data classes instead.

          4. Delete any unneeded data classes.

        • If you created the classifications based on data class ID, update the API calls to refer to the new data class IDs.

          1. Get a list of the old data class IDs in your environment.
          2. Get a list of the migrated data class IDs.
          3. Update the API calls to refer to the new data class IDs.

What happens during the migration process?

Important 
  • The migration process enriches the Unified Data Classification method. It doesn’t remove anything from your old classification method.
  • Migrated data classes receive a new name to avoid conflicts with possible newly created data classes in Unified Data Classification. This allows you to quickly identify the migrated classification information.

The migration process does the following:

  • Migrated data classes receive a new ID and a name with suffix (migrated). For example, Email (migrated).

    • The process adds all custom data classes from the old data classification method to the Unified Data Classification method, without a classification rule.
      Example 

      You have created a custom data class called "email". Even though there's an out-of-the-box data class with the same name in Unified Data Classification, the migration process will create a new data class in Unified Data Classification called "email (migrated)" without any classification rule.

    • The process adds all used out-of-the-box data classes from the old classification method to the Unified Data Classification method. This means only out-of-the-box data classes that appear in classifications are migrated.

      If an old out-of-the-box data class matches an out-of-the-box data class in the Unified Data Classification method, the migration process copies the classification rules from the Unified Data Classification out-of-the-box data class to the migrated data class. For the matching, a mapping tool is used.
      Example 

      You have a data class called "Emails", which is a renamed, old, out-of-the-box data class with id email_address. The migration process:

      • Detects that "Emails" is an out-of-the-box data class, based on the id email_address.
      • Looks up the mapping for this out-of-the-box data class. In this case, it finds that the data class matches the Unified Data Classification data class "email".
      • Creates a new data class called "Emails (migrated)", which has the classification rule from the Unified Data Classification data class "email".
  • The process adds all classifications that were manually or automatically assigned to assets to the Unified Data Classification method.
  • The process adds all linked data categories, data concepts, and so on to the data classes in Unified Data Classification.

The process adds some existing ADT as a new data class to the Unified Data Classification method. Such new data classes receive a name with suffix (ADT). The following conversion is applied:

ADT Type Data Class in Unified Data Classification
True/False Data class with list of values. All values, both true and all false together, are converted to lists of values.
Text Data class with regular expression rules. Each ADT regular expression is copied to a regular expression rule.
Geographical Data class with regular expression rules.
Decimal number Data class with a classification rule.
Whole number Data class with a classification rule.
Date None. We don't create a data class for this ADT.
Time None. We don't create a data class for this ADT.
Date and time None. We don't create a data class for this ADT.
Array None. We don't create a data class for this ADT.
N/A None. We don't create a data class for this ADT.

Start the migration process manually

You can start the migration process manually:

Steps

Important 

In Collibra 2024.05, we launched a new user interface (UI) for Collibra Data Intelligence Platform! You can learn more about this latest UI in the UI overview.

Use the following options to see the documentation in the latest UI or in the previous, classic UI:

  1. If you are using Protect, make sure to capture the data classes used in data protection standards and data access rules so that you can add them back after the migration.
  2. If you were not yet using Unified Data Classification, activate and set up Unified Data Classification.
  3. Enable the Unified Classification migration tool enabled setting.
  4. On the main toolbar, click Products icon, and then click Stewardship.
  5. Click the Data Classification tab.
  6. Click Migrate Data.
    The migration process starts. For larger classification sets, this process takes a few minutes.
    You can follow up on the job and the results from the Activities page.

    For more information go to What happens during the migration process?.
  7. Check the migrated data classes and classifications.
    In the Data Classification page, you can filter the data classes based on their name. The names of migrated data classes end with (migrated) or (ADT).
  8. You can now delete any unneeded data classes, merge data classes if needed, or add a classification rule to data classes if none was added by the migration process.
  9. If you are using Protect, add the data classes you captured in step 1 to the affected standards and rules.
  10. Once the migration is completed:
    1. Disable the Unified Classification migration tool enabled setting.
    2. If you were using classification on Edge before, disable the old Edge classification feature on your Edge site.
      You no longer need to enable classification on your Edge site because Unified Data Classification uses an Edge capability instead.
      • K3S
      • Managed Kubernetes

      To disable classification on an existing Edge site, deployed on K3S, run this command:

      Copy
      sudo ./edgecli update --set collibra.classification.enabled=false

      Tip The only difference between disabling classification and enabling classification is that the last argument is false instead of true.

      Note Edge supports the following managed Kubernetes clusters:
      • AKS
      • AWS Fargate using EKS
      • EKS
      • GKE
      • OpenShift

      Follow the steps below if you use a supported Kubernetes cluster.

      To disable classification on an existing Edge site, deployed on your dedicated cluster, run this command:

      Copy
      ./edgecli update --set collibra.classification.enabled=false

      Tip The only difference between disabling classification and enabling classification is that the last argument is false instead of true.

FAQ on the migration process

  • Does the migration process migrate the trained Machine Learning (ML) models from the Cloud Data Classification Platform?
    No, the migration process does not take the ML into account.
  • Can I still deactivate Unified Data Classification after 2024.07 and use my old classification method?
    Yes, that's possible. However, take into account that both, the old Edge classification and the Cloud Data Classification Platform are end of life on September, 30 2024.
  • What happens if I run the migration process multiple times?
    Data classes from the old classification methods and ADTs are migrated by the migration process only if they were not migrated before. To make sure a data class or ADT is migrated again, delete the created related data class from Unified Data Classification. Migrated data classes have a name that ends with (migrated) or (ADT).
  • Can I run the migration process after the end of life date for the old Edge classification and Cloud Data Classification Platform?
    Yes, you can. The migration process works only on the data stored in the Collibra database and doesn't connect to the old Edge or Cloud Data Classification Platform systems. However, note that the migration process will be removed at a later date.
  • I only use API calls to classify data. Am I impacted by this?
    Yes, all environments have to move the Unified Data Classification method APIs.