What Happened?
Create custom resource (CR)/customer resource definition (CRD) to enable Seldon (a platform to deploy AI/ML models) and Ambassador's Emissary ingress on F5 App Stack sites.
Environment
- F5 Distributed Cloud Services
Prerequisites
The minimum instance types required for CustomResourceDefinition (CRD) are the following:
- AWS: minimum size: t3.2xlarge
- GCP: minimum size: n1-standard-8
- Azure: minimum size: Standard_D4_v2
Resolution/Answer
1. In F5 Distributed Cloud Console or API, create two cloud App Stack sites with local K8s API access enabled (also known as Managed K8s). For each site:
-
- Enable the option for VoltConsole Access.
- Allow the option for Cluster Scoped RBAC and Webhook Resource Access to enable creation of cluster role and cluster role bindings for emissary CRDs.
- Allow the option for Vk8s Namespace Access.
-
- Enter the following command: export KUBECONFIG=$HOME/Downloads/ves_system_person-seldon_kubeconfig_global.yaml
2. For each App Stack site, download Helm and get repositories:
-
- curl https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash
- helm repo add datawire https://app.getambassador.io
- helm repo update
3. For each App Stack site, use managed K8s API install Ambassador and Seldon Helm charts:
Note: This is the step where the CRDs are installed on the App Stack sites. Their installation status will be reflected in the visual k8s instance created later.
-
- kubectl create namespace emissary
- kubectl apply -f https://app.getambassador.io/yaml/emissary/3.8.1/emissary-crds.yaml
- kubectl wait --timeout=90s --for=condition=available deployment emissary-apiext -n emissary-system
- helm install emissary-ingress --namespace emissary datawire/emissary-ingress
- kubectl -n emissary wait --for condition=available --timeout=90s deploy -lapp.kubernetes.io/instance=emissary-ingress
- kubectl create namespace seldon-system
- helm install seldon-core seldon-core-operator --repo https://storage.googleapis.com/seldon-charts --set usageMetrics.enabled=false --namespace seldon-system --set ambassador.enabled=true --set storageInitializer.image=docker.io/seldonio/rclone-storage-initializer:1.18.0-dev --set predictor_servers.SKLEARN_SERVER.protocols.seldon.image=docker.io/seldonio/sklearnserver
4. Install an Ambassador listener withkubectl apply -f default-listener.yaml
.
apiVersion: getambassador.io/v3alpha1
kind: Listener
metadata:
name: emissary-ingress-listener-8080
namespace: emissary
spec:
port: 8080
protocol: HTTP
securityModel: XFP
hostBinding:
namespace:
from: ALL
5. Create a virtual site seldon-cluster, with label key and value.
6. Attach the label to the two App Stack sites to form a fleet.
7. In Distributed Cloud Console or with API, create a namespace f5xc-seldon and virtual K8s instance, which selects the virtual site seldon-cluster.
8. Install sample custom resource by creating a file with iris-model.yaml. Apply the CR in the SLI with kubectl apply -f iris-model.yaml, or paste the content of the CR in the Console. To apply the above CR via the Console, under the Custom Resources tab, click on the Add Custom Resource and paste the CR, with the appropriate vk8s namespace and click Save. Afterwards, the panel should be updated with the deployment status on each site, with a description of what version is available on each site and whether the deployment is successful.
apiVersion: machinelearning.seldon.io/v1
kind: SeldonDeployment
metadata:
name: iris-model
namespace: f5xc-seldon
spec:
name: iris
predictors:
- graph:
implementation: SKLEARN_SERVER
modelUri: gs://seldon-models/v1.18.0-dev/sklearn/iris
name: classifier
name: default
replicas: 1
9. Verify that all resources are deployed to all sites in the virtual site. Also, verify that any corresponding services are available and serving.
10. Create a Distributed Cloud Services L4 TCP load balancer advertised on virtual site seldon-cluster, with origin, and select emissary-ingress service in emissary namespace. Advertise on the outside network of the site.
11. In the load balancer config, enable the health check option. Add an ambassador Mappings CR. Add the Path as /live.
12. Make a request to the prediction model using curl command. You should see a 200 response with {"data":{"names":["t:0","t:1","t:2"],"ndarray":[[0.0006985194531162835,0.00366803903943666,0.995633441507447]]},"meta":{"requestPath":{"classifier":"docker.io/seldonio/sklearnserver:1.17.1”}}}.
Deploying emissary-ingress on Baremetal/VMware/KVM Kubernates Clusters
In an on-premise data center, emissary-ingress is deployed on the Kubernetes cluster. Instead of exposing it via the LoadBalancer service type, emissary-ingress needs to be exposed as a NodePort. Traffic is sent to a specific port on any of the nodes in the cluster, which route the traffic to emissary-ingress, which then routes the traffic to seldon services running in Kubernetes. This is because Kubernetes does not offer an implementation of network load balancers (Services of type LoadBalancer) for bare-metal clusters. The implementations of network load balancers that Kubernetes does ship with are all glue code that calls out to various IaaS platforms (GCP, AWS, or Azure).
To install NodePort, apply the following:
---
apiVersion: v1
kind: Service
metadata:
name: ambassador
spec:
type: NodePort
ports:
- name: http
port: 8088
targetPort: 8080
nodePort: 30036 # Optional: Define the port you would like exposed
protocol: TCP
selector:
service: ambassador
Then follow the instructions at https://metallb.universe.tf/installation/.
a. Follow the prerequisite steps - https://metallb.universe.tf/installation/#preparation
b. You can install the CRDs using helm by following the steps available at https://metallb.universe.tf/installation/#installation-with-helm and the steps following to set up MetalLB.
Commands to run from the above site:
- helm repo add metallb https://metallb.github.io/metallb
- helm install metallb metallb/metallb OR kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.14.5/config/manifests/metallb-native.yaml.
Known Issues
Issue 1: When the iris-model CR is applied and it fails on some sites. The likely cause for this is the site does not have enough resources - CPU and/or memory and therefore failing to apply the CR. Check the Kubernetes logs or the events for this CR to verify resource issues.
Issue 2: While applying the emissary-ingress helm chart: helm install emissary-ingress --namespace emissary datawire/emissary-ingress. This may sometimes fail with the error Error: INSTALLATION FAILED. If this error occurs, reapply the helm chart to fix the issue.