System requirements of an Edge site

To use Edge, you must ensure that the following system requirements are met.

  • K3S
  • EKS cluster

Software requirements

  • You must be able to install the Edge software on the latest version of RedHat Enterprise Linux 8.
  • The sudo package is installed on the Linux host.
  • The user who installs Edge has full sudo access (ALL=(ALL) ALL).
  • Optionally, if you want SE Linux enabled, install the following policy packages before installing Edge:
    Packages

    These packages are not hosted by Collibra. If you have any questions, contact your internal teams.


Tip If you are an early adopter or you use Edge for beta testing purposes, we highly recommend that you disable SELinux.

Hardware requirements

You need the following minimum hardware requirements:

  • 64 GB memory
  • 16-core CPU with x86_64 architecture
  • At least 60 GB of free storage for Edge application storage requirements:
    • You have at least 50 GB of free storage on the partition that contains /var/lib/rancher/k3s. The partition mountpoint should not have the noexec option.
      mkdir -p /var/lib/rancher/k3s
mkfs.xfs /dev/<block-device-name>
mount /dev/<block-device-name> /var/lib/rancher/k3s
echo '/dev/<block-device-name> /var/lib/rancher/k3s xfs defaults 0 0' >> /etc/fstab

      Note This is the default install path. If it is not created as a separate mount point after following the steps above, the install will use 50 GB of disk space from either /var, or if not present, the root level of the drive.

      Warning Any data in this location is fully managed by the Edge site. Do not save any other data in this location as the data can be removed by Edge without notification.

    • You have at least an additional 5 GB of space in /var/log for Edge components. Edge uses hardcoded /var/log to write logs:
      • Up to 1.1 GB of space for writing K3S audit logs.
      • Maximum of 60 MB per container for pod logs. The number of containers depends on the workload.
    • You have at least an additional 5 GB of space on the partition that holds /var/lib/kubelet. Edge uses hardcoded /var/lib/kubelet/pods/*/volumes/kubernetes.io~empty-dir/* to write ephemeral data related to kubernetes.
  •  At least 500 GB of dedicated storage for Edge data storage requirements:
    • You have mounted at least 500 GB of dedicated storage for the Edge site data on a freely chosen mountpoint, for example, /var/edge/storage.

      mkdir -p /var/edge/storage
mkfs.xfs /dev/<block-device-name>
mount /dev/<block-device-name> /var/edge/storage
echo '/dev/<block-device-name> /var/edge/storage xfs defaults 0 0' >> /etc/fstab

      Note  Change <block-device-name> to the name of the device that contains the storage.

      Warning This dedicated storage must not be shared with other services because Edge can delete and overwrite files on this location without notice. Therefore, do not use /home/<username> or /var.

  • If you run the Linux server on AWS or GCP, disable the services nm-cloud-setup.service and nm-cloud-setup.timer. 

    systemctl disable nm-cloud-setup.service nm-cloud-setup.timer 
reboot

Warning When new capabilities are added in the future, the hardware requirements may change.

Network requirements

  • An Edge site needs outbound connections to all of the following:
    • The URL of your Collibra Data Intelligence Cloud environment.
    • https://*.datadoghq.com
      Note  If the allowlist does not accept wildcards:
      • https://http-intake.logs.datadoghq.com
      • https://7-40-1-app.agent.datadoghq.com
      • https://agent-http-intake.logs.datadoghq.com
      • https://api.datadoghq.com
    • https://*.repository.collibra.io
      Note If the allowlist does not accept wildcards:
      • https://repository.collibra.io
      • https://edge-docker-delivery.repository.collibra.io
      • https://mirror-docker.repository.collibra.io
    • https://otlp-http.observability.collibra.dev/
  • Access to all data sources you need to connect to your Edge sites.
  • Your Edge site has to be able to connect to port 443.
  • The resolve configuration file of your Linux host has maximum of 3 search domains and 2 name servers.

Note If a firewall is enabled, run the following commands to add the cni0 and loopback interfaces to a trusted zone so that Kubernetes can use it between its services: 

firewall-cmd --zone=trusted --change-interface=cni0 --permanent
firewall-cmd --zone=trusted --change-interface=lo --permanent
firewall-cmd --reload

EKS requirements

You can install the Edge software on managed Kubernetes clusters.

Important A managed Kubernetes cluster must be fully dedicated for Edge, do not use the cluster for other purposes.

  • AWS EKS 1.21, 1.22, 1.23 (all only with --container-runtime containerd) and 1.24 are supported.
    • We support EBS-CSI driver for 1.23.
  • AWS EKS worker nodes use the EKS optimized Amazon Linux 2 AMI
  • EKS cluster has IRSA enabled
  • AWS EKS worker nodes need to be in the same (one) Availability Zone!
    This can be implemented by creating just one node group for the EKS cluster, which limits the subnets to just one subnet, that is one of the subnets of the EKS cluster.
  • Set up security groups to ensure that worker nodes can communicate with each other on non-privileged ports.
    module "eks" {
	source            = "terraform-aws-modules/eks/aws"
	version           = "17.24.0"
	cluster_name      = "${var.vpc_name}-${var.cluster_name}-eks"
    	cluster_version   = "1.21"
	vpc_id            = var.vpc_id
	subnets           = data.aws_subnet_ids.public_subnet_ids.ids  # Subnets specified must be in at least two different AZs
	worker_additional_security_group_ids = [aws_security_group.worker_sg.id]
	enable_irsa       = true  # enable iam role for service account, for later use
	worker_groups = [
		{
    			name                = "${var.vpc_name}-${var.cluster_name}-eks-workers"
			instance_type        = var.worker_type
			asg_desired_capacity = var.instance_count_workers
			key_name             = aws_key_pair.cluster-ssh-keypair.key_name
			bootstrap_extra_args = "--container-runtime containerd"  # mandatory to run with containerd if on 1.21
			subnets              = [subnet1]                             # restriction for now to use only 1 subnet due to EBS tied to AZ
    		},
		]
	map_accounts = [
		data.aws_caller_identity.current.account_id
		]

	tags = {
		Name            = "${var.vpc_name}-${var.cluster_name}-eks"
    		}
}

Software requirements

  • A Linux server with bash available. This is the server from which you install the Edge software on EKS.

    Tip This server will also contain the Edge tools.

  • Plain cluster_admin kubectl access to the EKS cluster using its kubeconfig. With this kubeconfig, you must be able to use the kubectl command to communicate with the Kubernetes API server with full cluster access.
  • Ensure your Kubectl client is compatible with the relevant EKS version.
  • EBS volumes should be provisioned by the storage provider of the default storageclass.
  • The default EBS-based storageclass should have the delete reclaim policy.

Hardware requirements

You need an operational EKS cluster with at least 1 worker node. The cluster must meet the following requirements:

  • The total cluster capacity has at least 16 core CPU and 64 GB memory, for example 4 worker nodes with each 4 core CPU and 16 GB.
  • Each worker node needs at least 100 GB free disk space to store Docker images.
  • Ability to create EBS-based persistent volumes as a default storage class, at least 500 GB in total.

Network requirements

  • An Edge site needs outbound connections to all of the following:
    • The URL of your Collibra Data Intelligence Cloud environment.
    • https://*.datadoghq.com
      Note If the allowlist does not accept wildcards:
      • https://http-intake.logs.datadoghq.com
      • https://7-40-1-app.agent.datadoghq.com
      • https://agent-http-intake.logs.datadoghq.com
      • https://api.datadoghq.com
    • https://*.repository.collibra.io
      Note If the allowlist does not accept wildcards:
      • https://repository.collibra.io
      • https://edge-docker-delivery.repository.collibra.io
      • https://mirror-docker.repository.collibra.io
    • https://otlp-http.observability.collibra.dev/
  • Access to all data sources you need to connect to your Edge sites.
  • Your Edge site has to be able to connect to port 443.
  • The resolve configuration file of your Linux host has maximum three search domains and two name servers.

Note If a firewall is enabled, run the following commands to add the cni0 and loopback interfaces to a trusted zone, so that Kubernetes can use it between its services: 

firewall-cmd --zone=trusted --change-interface=cni0 --permanent
firewall-cmd --zone=trusted --change-interface=lo --permanent
firewall-cmd --reload