jimscard / devopd

2024-05-26: Updated to use dolphin-mistral:v2.8

Helpful assistant for your DevOps needs. Based on dolphin-mistral, with a system prompt generated by mistral for me. Seems to work fast, and can answer questions like:

Our application is experiencing intermittent downtime due to resource exhaustion on our Kubernetes cluster. How can we prevent this from happening in the future?

Which it answers with explanations and examples:

To prevent resource exhaustion in your Kubernetes (K8s) cluster, you should follow these best practices:

1. Resource requests and limits: Define request and limit for every Pod to ensure that containers have at least the requested amount of resources and can never consume more than the limit. This prevents a single rogue or poorly-configured container from consuming all cluster resources.

Example (in YAML):

spec:
  containers:
  - name: my_app
    image: my_registry.example.com/my_image:latest
    resources:
      requests:
        cpu: "100m"  # Minimum CPU allocation
        memory: "512Mi"   # Minimum Memory allocation
      limits:
        cpu: "750m"     # Maximum CPU allocation
        memory: "2Gi"    # Maximum Memory allocation

1. Horizontal Pod Autoscaler (HPA): Use K8s’ built-in HPA to scale your application based on actual resource consumption or external metrics like request rate. This ensures that you have enough resources available during peak traffic and can save costs during off-peak hours.

Example (YAML snippet for CPU-based scaling):

apiVersion: autoscaling/v2beta2
kind: HorizontalPodAutoscaler
metadata:
  name: my-app-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: my-app-deployment
  minReplicas: 2
  maxReplicas: 20
  targetCPUUtilizationPercentage: 50

1. Resource Quality of Service (QoS): Use QoS classes like BestEffort, Burstable, Guaranteed to prioritize Pods based on their resource requirements and usage. This way, mission-critical applications will always get the necessary resources first before other less important applications.

kubectl label nodes --all node-role.kubernetes.io/node-role.system:infra
kubectl label pods my-app-pod-1 -l app=my-app,tier=frontend,dept=dev --overwrite node-role.kubernetes.io/prd-ready=true
kubectl cmp group/storage.k8s.io/v1 beta.kubernetes.io/qos -o name | column -t

1. Container optimization: Optimize your container images by reducing the base image size, using multi-stage builds, and removing unnecessary dependencies. This will help reduce the memory footprint of containers, leading to better resource utilization.

2. Cluster capacity planning: Regularly review and update your cluster’s capacity based on actual usage patterns, growth projections, and new feature rollouts. Adjusting the number and types of nodes (worker, control plane) in your cluster can help balance resource allocation efficiently.

By following these best practices, you should be able to prevent resource exhaustion issues in your Kubernetes cluster and ensure its reliability and scalability.