# Access and manage embedded clusters (Beta)

This topic describes managing nodes in clusters created with Replicated Embedded Cluster.

## Access the cluster

You can use the CLI to access the cluster. This is useful for development or troubleshooting.

To access the cluster and use other included binaries:

1. SSH into a controller node.

     :::note
     You cannot run the `shell` command on worker nodes.
     :::

1. Use the Embedded Cluster shell command to start a shell with access to the cluster:

     ```
     sudo ./APP_SLUG shell
     ```
     Where `APP_SLUG` is the unique slug for the application.

     The output looks similar to the following:
     ```
        __4___
     _  \ \ \ \   Welcome to APP_SLUG debug shell.
    <'\ /_/_/_/   This terminal is now configured to access your cluster.
     ((____!___/) Type 'exit' (or CTRL+d) to exit.
      \0\0\0\0\/  Happy hacking.
     ~~~~~~~~~~~
    root@alex-ec-1:/home/alex# export KUBECONFIG="/var/lib/embedded-cluster/k0s/pki/admin.conf"
    root@alex-ec-1:/home/alex# export PATH="$PATH:/var/lib/embedded-cluster/bin"
    root@alex-ec-1:/home/alex# source <(k0s completion bash)
    root@alex-ec-1:/home/alex# source <(cat /var/lib/embedded-cluster/bin/kubectl_completion_bash.sh)
    root@alex-ec-1:/home/alex# source /etc/bash_completion
    ```

     The appropriate kubeconfig is exported, and the location of useful binaries like kubectl and Replicated’s preflight and support-bundle plugins is added to PATH.

1. Use the available binaries as needed.

     **Example**:

     ```bash
     kubectl version
     ```
     ```
     Client Version: v1.29.1
     Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
     Server Version: v1.29.1+k0s
     ```

1. Type `exit` or **Ctrl + D** to exit the shell.

## Configure multi-node clusters

This section describes how to join nodes to a cluster with Embedded Cluster.

### Limitations

Multi-node clusters with Embedded Cluster have the following limitations:

* All nodes joined to the cluster use the same Embedded Cluster data directory as the installation node. You cannot choose a different data directory for Embedded Cluster when joining nodes.

* You should not join more than one controller node at the same time. When joining a controller node, Embedded Cluster prints a warning explaining that you should not attempt to join another node until the controller node joins successfully.

* You cannot change a node's role assignment after you join the node. To change a node's roles, reset the node and add it again with the new role selection. For more information about customizing node roles, see [roles](embedded-config#roles) in _Embedded Cluster Config_.

### Requirement

To deploy multi-node clusters with Embedded Cluster, you must enable the **Multi-node Cluster (Embedded Cluster only)** license field for the customer. For more information about managing customer licenses, see [Create and Manage Customers](/vendor/releases-creating-customer).

### Join nodes {#add-nodes}

To join a node:

1. SSH into a controller node.

1. Run the following command to generate the `.tar.gz` bundle for joining a node:

   ```bash
   sudo ./APP_SLUG create-join-bundle --roles ROLE[,ROLE...]
   ```
   Where:
   * `APP_SLUG` is the unique slug for the application.
   * `--roles` is one or more role names to assign the node, comma-separated. Valid role names are the default controller role and any custom roles declared in the Embedded Cluster Config under [roles](embedded-config#roles).

     :::note
     You cannot change role assignments after a node is joined. To change them, reset the node and add it again with the new role selection.
     :::

1. Use `scp` to copy the `.tar.gz` bundle to the node that you want to join.

1. Extract the `.tar.gz`.

1. Run the join command to add the node to the cluster:

   ```bash
   sudo ./APP_SLUG node join
   ```

1. Repeat these steps for each node you want to add.

### Headless joins {#headless-joins}

For automation and orchestrated deployments, you can join nodes without going through the web UI by calling the Embedded Cluster [external API](embedded-cluster-external-api) from any host in the cluster network. The flow is:

1. Fetch a JWT token from a controller using the installer password:

   ```bash
   TOKEN=$(curl -X POST https://<controller-ip>:30081/v1/auth/token \
     -H 'Content-Type: application/json' \
     -d '{"password":"<installer-password>"}' | jq -r .token)
   ```

2. Poll [`GET /v1/nodes`](embedded-cluster-external-api#get-v1nodes) until at least one controller reports `ready: true`. This is the signal that the cluster is ready to issue join commands:

   ```bash
   until curl -s https://<controller-ip>:30081/v1/nodes \
     -H "Authorization: Bearer $TOKEN" \
     | jq -e '.nodes | any(.ready)' >/dev/null; do
     sleep 5
   done
   ```

3. Request the join commands. Pass the desired role names from the [`roles`](embedded-config#roles) key in the Embedded Cluster Config:

   ```bash
   RESPONSE=$(curl -X POST https://<controller-ip>:30081/v1/create-join-command \
     -H "Authorization: Bearer $TOKEN" \
     -H 'Content-Type: application/json' \
     -d '{"roles":["controller"]}')
   ```

4. Execute each command in the returned `joinSteps` array on the joining node:

   ```bash
   echo "$RESPONSE" | jq -r '.joinSteps[].command' | while IFS= read -r cmd; do
     eval "$cmd"
   done
   ```

For the full set of endpoints and response schemas, see [External API](embedded-cluster-external-api).

## High availability for multi-node clusters {#ha}

Embedded Cluster automatically enables high availability (HA) when at least three `controller` nodes are present in the cluster.

In HA installations, Embedded Cluster deploys multiple replicas of the OpenEBS and image registry built-in extensions. Also, any Helm [extensions](embedded-config#extensions) that you include in the Embedded Cluster Config are installed in the cluster depending on the given chart and whether or not it is configured to be deployed with high availability.

### Best practices for high availability

Consider the following best practices and recommendations for HA clusters:

* HA requires at least three _controller_ nodes that run the Kubernetes control plane. This is because clusters use a quorum system, in which more than half the nodes must be up and reachable. In clusters with three controller nodes, the Kubernetes control plane can continue to operate if one node fails because the remaining two nodes can still form a quorum.

* Always use an odd number of controller nodes in HA clusters. Using an odd number of controller nodes ensures that the cluster can make decisions efficiently with quorum calculations. Clusters with an odd number of controller nodes also avoid split-brain scenarios where the cluster runs as two, independent groups of nodes, resulting in inconsistencies and conflicts.

* You can have any number of _worker_ nodes in HA clusters. Worker nodes do not run the Kubernetes control plane, but can run workloads such as application workloads.

### Create a multi-node cluster with HA

To create a multi-node cluster with HA:

* During installation with Embedded Cluster, follow the steps in the Embedded Cluster UI to join a total of three `controller` nodes to the cluster. For more information about joining nodes, see [Join nodes](#add-nodes) on this page.

   Embedded Cluster automatically converts the installation to HA when three or more controller nodes are present.

### Enable HA for an existing cluster {#enable-ha-existing}

To enable HA for an existing Embedded Cluster installation with three or more controller nodes:

* On one of the controller nodes, run this command:

   ```bash
   sudo ./APP_SLUG enable-ha
   ```

   Where `APP_SLUG` is the unique slug for the application.

## Reboot nodes

Embedded Cluster runs Kubernetes (k0s) as a systemd service, so a node automatically rejoins the cluster after a reboot. No special preparation is required — you do not need to cordon or drain the node before rebooting.

**Single-node clusters:** The application is unavailable while the node is rebooting. Schedule reboots during a maintenance window, and expect a short outage (typically a few minutes) while the node restarts and pods are rescheduled.

**Multi-node clusters:** Reboot one node at a time. Wait for the rebooted node to fully rejoin the cluster and report `Ready` before rebooting the next node. Do not reboot two or more controller nodes at the same time — losing multiple controllers simultaneously can break etcd quorum and take the entire cluster offline.

To check whether a node has rejoined after a reboot, run the following command from a controller node:

```bash
sudo ./APP_SLUG shell -c "kubectl get nodes"
```

Where `APP_SLUG` is the unique slug for the application. All nodes should show `Ready` status before proceeding with the next reboot.

## Remove nodes and reset machines

This section describes how to remove individual nodes from a cluster and how to delete an entire multi-node cluster using the Embedded Cluster [remove-node](embedded-cluster-remove-node) and [reset](embedded-cluster-reset) commands.

### About the `remove-node` and `reset` commands

To remove a node from a multi-node cluster, run the `remove-node` command from a controller node. Then, if you want to reuse the machine that you removed, run the `reset` command on it to remove Embedded Cluster from that machine.

Run the `remove-node` command from a controller node other than the node that you want to remove. The command performs the following steps:

1. Drain workloads from the node so that they reschedule onto other nodes in the cluster
1. For controller nodes, remove the node from the etcd cluster
1. Delete the node from Kubernetes

The `reset` command removes Embedded Cluster and your application from the machine where it is run. This is useful for iteration, development, and when you make mistakes because you can reuse the machine instead of having to procure a new one.

The `reset` command performs the following steps:

1. Run safety checks. For example, `reset` does not proceed on a controller node that is still a member of a multi-node cluster's etcd
1. For worker nodes that are still in the cluster, drain the node and evict all the Pods gracefully
1. Stop and reset k0s
1. Remove all Embedded Cluster files

The `reset` command does not remove the node from the cluster. To reset a controller node in a multi-node cluster, first run `remove-node` from another controller node. If you run `reset` on a worker node without first running `remove-node` from a controller node, the node's workloads are drained, but the node remains in the cluster in a `NotReady` state until you remove it with `remove-node`.

For more information about these commands, see [remove-node](embedded-cluster-remove-node) and [reset](embedded-cluster-reset).

### Remove a node from a multi-node cluster {#remove-a-node}

To remove a node from a multi-node cluster:

1. SSH onto a controller node other than the node that you want to remove.

1. Remove the target node from the cluster:

    ```bash
    sudo ./APP_SLUG remove-node NODE_NAME
    ```
    Where:
    * `APP_SLUG` is the unique slug for the application.
    * `NODE_NAME` is the Kubernetes node name of the node to remove. To list the node names in the cluster, run `sudo ./APP_SLUG shell -c "kubectl get nodes"`.

    :::note
    If the node is unreachable, include the `--skip-drain` flag to skip draining workloads from the node. If the node's Kubernetes object is already gone, `remove-node` skips directly to the etcd cleanup.
    :::

1. (Optional) Reset the machine that you removed so that you can reuse it. See [Reset a machine](#reset-a-node) in this document.

### Reset a machine {#reset-a-node}

For single-node installations, or for the last remaining controller node when deleting a multi-node cluster, run the `reset` command directly. For all other nodes in a multi-node cluster, first remove the node from the cluster by running `remove-node` from a controller node. See [Remove a node from a multi-node cluster](#remove-a-node) in this document.

To reset a machine:

1. SSH onto the node. Ensure that the Embedded Cluster binary is still available on the machine.

1. Run the following command to reset the machine:

    ```bash
    sudo ./APP_SLUG reset
    ```
    Where `APP_SLUG` is the unique slug for the application.

1. Reboot the machine.

### Remove a multi-node cluster

To remove an entire multi-node cluster, remove each node with `remove-node`, starting with the worker nodes and ending with the controller nodes. On the last remaining controller node, run `reset` directly, because there is no other controller left to run `remove-node` from. Optionally, reset each machine that you remove so that you can reuse it. See [Reset a machine](#reset-a-node) in this document.

For more information, see [Remove a node from a multi-node cluster](#remove-a-node) in this document.

### Limitations and best practices

Before you remove a node or delete a cluster, consider the following limitations and best practices:

* When you reset a node, Embedded Cluster deletes OpenEBS PVCs on that node. Kubernetes automatically recreates only PVCs created as part of a StatefulSet on another node in the cluster. To recreate other PVCs, redeploy the application in the cluster.

* If you need to remove one controller node in a three-node cluster, first join a fourth controller node to the cluster before removing the target node. This ensures that you maintain a minimum of three nodes for the Kubernetes control plane. You can add and remove worker nodes as needed because they do not have any control plane components.

* When resetting a single node or deleting a test environment, you can include the `--force` flag with the `reset` command to skip the confirmation prompt and continue past any errors.

* When removing a multi-node cluster, remove the worker nodes before the controller nodes. For more information, see [Remove a multi-node cluster](#remove-a-multi-node-cluster) in this document.

* In air gap installations with high availability, SeaweedFS stores object storage data across nodes in the cluster. After removing a node, verify that all pods in the `seaweedfs` namespace are in a `Running` state before removing additional nodes.