KB: Fixing Kubernetes Cluster after expired certs

Fixing Kubernetes API Server and Kubelet Issues After Expired Certificates

  1. Backup Existing Kubernetes Configuration

    • Create a backup of Kubernetes certificates and configuration files
    • Copy /etc/kubernetes/pki/ and /etc/kubernetes/*.conf to a backup directory

  2. Renew Expired Certificates

    • Check expired certificates using kubeadm certs check-expiration
    • Renew all certificates with kubeadm certs renew all
    • Restart kubelet and container runtime using systemctl restart kubelet and systemctl restart containerd or systemctl restart docker

  3. Regenerate bootstrap-kubelet.conf if Missing

    • Run kubeadm init phase kubelet-start to regenerate it
    • If still missing, manually copy admin.conf to kubelet.conf
    • Restart kubelet using systemctl restart kubelet

  4. Fix Unauthorized Errors

    • Ensure kubectl is using the correct kubeconfig
    • Copy /etc/kubernetes/admin.conf to ~/.kube/config and set proper ownership and permissions
    • Check kubelet logs for authentication errors with journalctl -u kubelet --no-pager | tail -50

  5. Use crictl to Check and Troubleshoot Running Containers

    • Check running containers: crictl ps
    • Check all containers including stopped ones: crictl ps -a
    • View logs of a specific container: crictl logs <container-id>
    • Inspect a container: crictl inspect <container-id>
    • If the API server container is missing, restart kubelet and check logs

  6. Rejoin the Master Node if Needed

    • Create a new bootstrap token using kubeadm token create --print-join-command
    • Run the generated kubeadm join command to reconnect the node
    • Restart kubelet using systemctl restart kubelet

  7. Final Checks

    • Verify Kubernetes nodes are running using kubectl get nodes
    • Check API server status with kubectl get pods -n kube-system
    • Review kubelet logs with journalctl -u kubelet --no-pager | tail -50
    • Ensure all control plane components are running using kubectl get componentstatuses

This process ensures Kubernetes master nodes recover from expired certificates, missing kubelet configuration, and API server failures.

Comments

Post a Comment

Popular posts from this blog

KB: Azure ACA Container fails to start (no User Assigned or Delegated Managed Identity found for specified ClientId)

Image
When deploying secure workloads using Azure Container Apps (ACA) , teams often face confusion between User Assigned Managed Identities (UAMI) and App Registrations . While both entities are visible in Azure Active Directory and have similar identifiers (Application ID, Object ID), they serve very different purposes . This confusion can lead to authentication failures when accessing services like Azure App Configuration or Key Vault . A common issue occurs when a container app is configured with a managed identity, but the environment variables or role assignments mistakenly reference an App Registration instead, causing errors like: "No User Assigned or Delegated Managed Identity found for specified ClientId" This article breaks down the difference between UAMI and App Registrations, explains why this issue happens, and outlines the correct approach to resolve it. Symptoms Azure Container App fails to authenticate to Azure App Configuration or other services. Logs...
Read more

Electron Process Execution Failure with FSLogix

  Technical Note: Electron Process Execution Failure with FSLogix 1. Overview When running Electron-based applications in environments using FSLogix Profile or Office Containers , users may encounter issues where the Electron process fails to launch or execute properly . This behavior has been observed in Azure Virtual Desktop (AVD) , Windows Virtual Desktop (WVD) , and other environments where FSLogix filter drivers are active. 2. Symptoms Electron-based applications (e.g., desktop apps built on Electron, CLI wrappers) do not start , remain unresponsive, or terminate silently. No visible logs or error messages are generated by the application. Standard executables run correctly when placed outside of the FSLogix-controlled profile path (e.g., copying to C:\Temp allows execution). The issue is reproducible across all Electron apps in the FSLogix-managed profile. 3. Root Cause The issue is linked to FSLogix filter drivers ( frxdrv , frxdrvvt , frxccd ) interfe...
Read more

KB:RMM VS DEX (Remote Monitoring Management vs Digital Employee Experience)

Digital Employee Experience (DEX) and Remote Monitoring and Management (RMM) tools serve different purposes and cater to distinct aspects of IT and employee management. Digital Employee Experience (DEX) Focus: Enhancing the overall experience of employees with their digital workplace tools and environment. Key Features: User Experience Monitoring: Tracks how employees interact with software and hardware, identifying issues impacting productivity. Performance Analytics: Provides insights into the performance of applications from the end-user's perspective. Feedback Mechanisms: Allows employees to report issues and give feedback on their digital tools and environment. Employee Well-being: Monitors factors that affect employee satisfaction and well-being, such as system performance and usability. Proactive Support: Identifies potential issues before they become significant problems, enabling proactive IT support. Goals: Improve employee satisfaction and productivity. Ensure a sm...
Read more