Kubecon Europe 2020 - Day 3
Mis à jour : avr. 8
Third day of the KubeCon CloudNativeCon Europe 2020 virtual event.
Again lot of interesting sessions !
Check out other days:
Keynotes
Vicky Cheung presented the release notes of Kubernetes 1.18.
38 enhancements
Release team: 34 persons
40 000 individual contributors to date !
Key features:
Storage Enhancement:
Raw block device support graduates to Stable
alpha version of CSI Proxy for Windows (to perform privileged storage operations in Windows)
Scheduling Enhancements:
Run multiple scheduling profiles
Taint based Eviction graduation to Stable
PodTopologySpred graduation to Beta
HPA (Horizontal Pod Autoscaler)
Feature in Alpha
Finer grain control on autoscaling rates
Avoid flapping of replicas
Adjust scaling behavior based on application profile
Kubectl alpha debug
Ephemeral containers were added in 1.16
When “kubectl exec” isn’t enough
1 kubectl alpha debug -it demo --image=debian --target=demo
Priority and Fairness for API Server Requests
Protect API servers from being overloaded while ensuring critical requests go through
Prevent loss of user access should anything run amok
Node Topology Manager graduates to Beta:
Useful for high performance computing and machine learning workloads
Other Notable features:
IPv6 Beta
Certificate API
APIServer DryRun and “kubectl diff”
In the next Kubernetes 1.19:
Generic ephemeral volumes
Kube-proxy IPV6DualStack support on Windows
Initial support for cgroup v2 (YES !!!)
Briana Franck explained how to transform your IT projects into the Cloud Native world.
New projects allow teams to re-evaluate the current path of transformation using Design thinking workshop. Take advantage of the start of new projects to catalyze innovation and enforce the culture of automation to gain efficiency. Human and communication is key. An exemple for operate and improve communication is ChatOps.
Automate the deployment of thousands of clusters with ease: exemple with Razee.io
This talk could have been renamed “Zombie land” !
Excellent keynote of Holly Cummins explaining the impacts of bad resources usage on the Planet.
The consumption of resources between data centers and aviation is not that far! (1 to 2% vs. 2.5%)
The Kubesprawls trend (pattern of “many clusters” rather than “one big shared” cluster) is not uncommon for multiple teams in the same organization. Cluster are less elastic than applications and have overhead.
There are several reasons to get your own cluster: isolation, security, perf, name collisions, …
Consolidation with multi-tenancy could be a good approach.
In all cases don’t forget your resources : Zombie workloads ! You know the things you forget in your cluster and that stay “alive” but not really and that consume resources for a while !
manual solutions exists: meeting, tags on manifest objects, …
The solution is to do the right thing
GitOps helps a lot with the infra-as-code: “disposable infrastructure” you can deploy, redeploy and delete in a single operation.
To avoid Zombie workload, See the talk “Sharing Clusters: Learnings From Building a Namespace On-Demand Platform“ especially the part “Monitor cost and identify idle namespaces”
In this presentation, Hannah Foxwell explains what is the path of least resistance to a successful Cloud Native project:
Start by identify Early Adopters
Build a Minimum Viable Product
Don’t think like building the platform as a silo and more as service
Scaling the success
Communication and relation is key
Small steps are easier
Adopt the KISS principle: Keep It Short and Simple
And remember we are human not super human !
Containerd Deep Dive
By Akihiro Suda (Software Engineer, NTT) and Wei Fu (Software Engineer, Alibaba)
This talk was about Containerd, a container runtime implemented the CRI runtime specification.
Containerd is not only used by Docker but also by many distributions: K3s, kubespray, micro8s, Charmed Kubernetes, Kind, minikube, Alibaba ACK, Amazon EKS, Azure AKS, Google GKE, … And also by libs/frameworks like BuildKit, LinuxKit, Fassd, Vmware Fusion Nautilus
A lots of nice features were introduced with version 1.4:
Lazy pulling of images: run containers before completion of downloading the images -> improve start speed: it is based on Stargz & eStargz. With a tar format you cannot navigate to a specific offset until all the file is downloaded
Support for cgroup v2 and improved support for rootless mode (resource limitation support)
Windows CRI support
and much more (see release notes: https://github.com/containerd/containerd/releases/tag/v1.4.0)
Containerd is highly Customizable throught its V2 Runtime:
Runtime v2 with a shim API that allows for instance integration with low level runtime like gVisor, KataContainer, Firecracker.
Binary: specific naming convention
Support pluggable logging via STDIO URIs
Containerd is definitely a good CRI runtime with lots of nice features in v1.4. We especially appreciate the lazy pulling of images and expect a lot from cgroup V2.
What You Didn’t Know About Ingress Controllers’ Performance
By Mikko Ylinen (Senior Software Engineer, Intel) and Ismo Puustinen (Cloud Software Engineer, Intel)
Basically Ingress exposes HTTP(s) routes to services within the cluster. The Ingress controller follows the Kubernetes objects and creates the route configuration. The Ingress Proxy reads the configuration and handles the actual traffic routing
There are several implementations like Nginx, HAProxy, Envoy…
Performance bottleneck are mostly bandwidth and latency.
Tuning areas concern:
A focus on TLS Handshake because most of the time of a request is spent on the TLS handshake.
How to improve it:
TLS 1.3 or a faster cypher…
Sync vs async TLS
Async TLS is currently being added in Ingress
Async TLS Offloading
When to offloading depends on CPU-bound of the controller, if lot of new HTTPs connections, …
Exemple with HAProxy Hardware Acceleration, HAProxy RSA Multibuffer
Call to action:
Check where are your bottlenecks
Ingress Controllers: check config of crypto offload, non native resources, node affinity labels
Ingress Proxy devs: switch to async TLS and allow custom TLS handshake
A good and very technical presentation on how to tune your Ingress if you need to get the best performances.
Managing Multi-Cluster/Multi-Tenant Kubernetes with GitOps
By Chris Carty (Customer Engineer, Independent)
The author presented what is GitOps (You will also find a presentation of GitOps on the Sokube Blog)
2 main GitOps projects:
Project Structure contains your deployment yaml files.
Some useful tools in the GitOps context :
OPA (Open Policy Agent)
Conftest (helps you write tests against structured configuration data)
Kubeval (tool for validating a Kubernetes YAML or JSON configuration file)
Kind (tool for running local Kubernetes clusters using Docker container "nodes" like k3d/k3s):
Git Single vs Multi Repo in case of multi-tenancy
Single Git repo:
Containing the CI, Cluster-admins (monitoring, networking, security), teams-1 (app resources), teams-2 (App resources), … :
each cluster (Dev, QA, Prod, …) Sync on a specific branch or tag
Multi-repo:
Several Git repos can target the same cluster but in different namespaces
Branch/Tag = Grouping/Environnement of cluster
see more info here: fluxcd/multi-tenancy
A nice presentation for this very important topic. The speaker did not have time to go into details and to present the pros and cons of each approach, but it gives important pointers and a way to achieve multi-tenancy using Flux in Kubernetes .
Sharing Clusters: Learnings From Building a Namespace On-Demand Platform
By Lukas Gentele (CEO, DevSpace Technologies Inc.)
2 Approaches
Single-Tenant k8s: 1 Team/App per Cluster => too expensive
Multi-tenant k8s: Sharing large Cluster => less expensive but more complex
Several learning:
Centralize user management and authentification
SSO for k8s via Dex for instance
Restrict user but use smart defaults (UX matters)
Pod Security Policy
Resource Quotas: Set default via LimitRange (Mutating Admission Controller)
Network Policies; Default to deny all (allow traffic from inside the namespace)
Automate as much as possible
template for RBAC, Quotas, Net Pol
OPA for dynamic admission control (ex: Hostname validation for ingress resource or for certificate resource or block certain storage and network conf…)
Store everything in kubernetes + git
Use annotation, labels, secrets config map to store info about owner, tenants, …
GitOps for History, audit, rollback and Approval process vial PR and Code Owner
CRDs for even more control & automation
Do not hide kubernetes but make it easier to use
Engineers need direct access to Kubernetes (to verify, debug, ….)
Kubectl is an API not a dev tool
Simplifying Local Development in Kubernetes with Telepresence
File-Sync-based Dev Experience: Skaffold, DevSpace, Tilt
Monitor cost and identify idle namespaces
automate the shutdown of idle namespaces with:
Cluster-turndown by Kubecost
Sometimes users need more than just namespaces
Namespaces based multi tenancy has limitations: CRD needed by users or specific versions of k8s
Virtual Cluster can solve this problem: vCluster
A very interesting talk on how to address multi-tenancy providing best practices and tools like Telepresence, Cluster-turndown, Sleep-mode, VCluster and Kiosk…
Multi-Tenant Clusters with Hierarchical Namespaces
By Adrian Ludwin (Senior Software Engineer, GKE, Google)
Concept of hierarchical namespaces is new in Kubernetes.
Multitenancy to care about cost and velocity.
One tenant per cluster was for small team
Kubesprawl : it does scale very well to get 1 cluster per tenant or team
A Kubernetes multi-tenancy group exists to adresse those issues.
Namespaces are the primary unit of tenancy in kubernetes but some security features require namespaces:
RBAC works best at the namespace level
Also applies to most other policies : resource Quota, NetworkPolicy, …
Policies across namespaces
need a tool and source of truth outside k8s: Flux, ArgoCD, ….
Alternatively, some in-cluster solution add accounts or tenants: Kiosk or Tenant CRD
Hierarchical Namespace Controller (HNC):
Hierarchical namespaces make it easier to share your cluster by making namespaces more powerful. For example, you can create additional namespaces under your team's namespace, even if you don't have cluster-level permission to create namespaces, and easily apply policies like RBAC and Network Policies across all namespaces in your team (e.g. a set of related microservices).
Entirely Kube native
Builds on regular kube namespaces
Delegate subnamespace creation without cluster privileges !
policy propagation
subnamespace hierarchy
It cannot be moved
Trusted label with tree suffix in the child namespace
Easy to extend
Other features:
Authorization checks before modifying the hierarchy
Cascading deletion of subnamespaces
Monitoring options (Metrics via OpenCensus)
Uninstallation support (to avoid data deletion)
Hierarchical namespaces is in alpha but can simply be added like an addon on Kubernetes 1.15+ (Well done !)
It is a really cool feature that will simplify the multi-tenancy approach in a K8S cluster (especially with a GitOps approach) and give teams control over their own namespace hierarchy.
Automating Load Balancing and Fault Tolerance via Predictive Analysis - Steven Rosenberg, Red Hat
By Steven Rosenberg (Software Engineer, Red Hat)
Load Balancing → Type of solution Fault Tolerance → Live Migration Scheduling → Predictive Analysis
Load Balancing
Priority based upon urgency
Even Distribution within categories :
Urgent priority - Mission Critical - Real Time Processing
High Priority - High Importance - near Real Time Processing
Neutral Priority - Medium importance - Normal Processing
Low Priority - Low importance - Not Time Critical Processing
No Priority - Unimportant - Unimportant Proecesses
Fault Tolerance Redundancy Example
Scheduling
Ability to launch processes based upon needed resources
Monitor the amount of resource each process utilizes
Type of Launching/Migration Scenarios :
Initial Launch
Migration for maintenance
Re-balancing - Migration to another host
Fault recovery - Migrating to mitigate system/process failure
Policy units - Attributes of scheduling Migrations
Filters
Weights/Scoring
Balancers :
Even distribution
Power saving
Prioritizing
Affinity
CPU/Non-Unimform Memory Access Pinning for optimal performance
Live migration process to migrate from source host toward destination host :
Network connectivity
Remote disk availability
Migration data on local disk(s)
Copying memory state in phases
All of the Curren memory contents
Current differences before VM pausing
Minimal difference during VM pausing
Copy CPU State
The goal is to limit pausing of the VM
Restarting the VM on the destination host
Clean up on the source host
Predictive Analysis Topics
Predicting future occurrences via analysis of past performance
Techniques for predictive analysis
Process for developing a prediction model
Predictive analytics methodology :
Get historical data → Create a training Set → Create an algorithm and a model → Get result → Restart process
Process fo developing a prediction model :
Applying Predictive Analytics to Schedulers
Criteria for Data
Processing time / Iteration - Adjusted for resource capacity and priority
Percentage of resources used - Adjusted for capacity and priority
Adjust for anomalies when calculating averages
Ideas - DSelective techniques applied for other scheduling applications :
Comining regression-like modeling and functional approximation using the sum of exponential functions to produce probability estimates
Machine Learning and advanced mathematical models
Predictive analysis architecture :
Tracking historical data :
The time each process starts and terminates
The resources used by each process
The time each process uses to migrate
The time/iteration that memory/disk transfer occurs per size
Consideration based upon analysis
If early can proceed
When early migration shall start
Error correction/anomaly detection for accurate results
This topic was interesting but is a complex subject. It demonstrates how to be proactive about load-balancing and fault-tolerance, based on collected historical data and mathematical compute to prevent future faults. There are interesting models to improve infrastructure reliability.
Simplify Your Cloud Native Application Packaging and Deployments - Chris Crone, Docker
By Chris Crone (Engineering Manager, Docker)
What is a cloud native application ?
“A program or piece of software designed to fulfill a particular purpose” - Oxford English Dictionary
A cloud native application is :
Compute :
Containers
Function (AWS Lambda, Azure Functions…)
Virtual Machines
Storage :
Databases
Object storage
Volumes
Networking
The CNCF Cloud Native Landscape map represents a view of many applications, tools , runtime and more recommended by a CNCF. On this interactive map you can directly visit the product website by clicking on the product logo.
Deploying application
Are you encountering the following errors ? Probably yes !
Often need more than one tool to deploy an application
Is the ReadMe up to date ?
Which version of the tools ?
What if I'm using Windows or Mac and not Linux ?
Difficult coordination problem between team members, CI, users …
Ideal deployment tooling ?
Defined as code : tools, versions options → what the solution ?
Same deployment environment everywhere → what the solution ?
Packaging application
Different parts, different places :
Ideal application packaging ?
Immutable application artefact → what's the solution ?
Store the whole application in a registry → what's the solution ?
Ability to store application artefact online → what's the solution ?
There are many questions without answers!
Cloud Native Application Bundles
→ Here the Cloud Native Application Bundles (CNAB) enter in the picture.
CNAB is a package format specification that describes a technology for building, installing and managing distributed applications, that are by design, cloud agnostic.
CNAB specification
Target is tolling developers
Packaging specification (bundle)
Bundle runtime (action)
Install, upgrade, uninstall
Optionally
Lifecycle tracking
Registry storage
Security
Dependency
Bundle structure :
CNAB runtime
Standard actions : install, upgrade, uninstall
Custom actions :
status, logs …
Stateful, stateless
Application lifecycle tracked by claims
Keep track of state of installation
keep record of parameters, outputs…
Only data structure defined in specification
Finally CNAB can respond to previous questions :
→ Ideal deployment tooling ?
Defined as code : tools, versions options → solution :
Porter tool : Package everything you need to do a deployment (command-line tools, configuration files, secrets and bash scripts to glue it together). Package that into a versions bundle distributed over standard Docker registries or tgz files.
porter.yaml
Stored in CNAB invocation image
Same deployment environment everywhere → solution :
Containers
→ Ideal application packaging ?
Immutable application artifact → solution :
Hashes for components
Leverage OCI image specification
Store the whole application in a registry → solution :
Any OCI compliant container registry
Ability to store application artifact online → solution :
OCI image layout
CNAB in registries :