I got the decoded values "mega_secret_key" and "really_secret_value1" from from your encoded data. Seems they are not encoded in right way. So, encode your data in right way:

$ echo "mega_secret_key" | base64
bWVnYV9zZWNyZXRfa2V5Cg==


$ echo "really_secret_value1" | base64
cmVhbGx5X3NlY3JldF92YWx1ZTEK

Then check whether they are encoded properly:

$ echo "bWVnYV9zZWNyZXRfa2V5Cg==" | base64 -d
mega_secret_key


$ echo "cmVhbGx5X3NlY3JldF92YWx1ZTEK" | base64 -d
really_secret_value1

So they are ok. Now use them in your

dummy-secret.yaml

apiVersion: v1
kind: Secret
metadata:
  name: dummy-secret
type: Opaque
data:
  API_KEY: bWVnYV9zZWNyZXRfa2V5Cg==
  API_SECRET: cmVhbGx5X3NlY3JldF92YWx1ZTEK

And run

$ kubectl create -f dummy-secret.yaml

UPDATE on 11-02-2022:

The newer versions of Kubernetes support the optional

stringData

All key-value pairs in the

stringData

field are internally merged into the

data

field. If a key appears in both the

data

and the

stringData

field, the value specified in the

stringData

field takes precedence.

apiVersion: v1
kind: Secret
metadata:
  name: dummy-secret
type: Opaque
stringData:
  API_KEY: mega_secret_key
  API_SECRET: really_secret_value1

UPDATE:

If you use

-n

$ echo "some_text"

\n

$ echo "some_text"
some_text
$ echo -n "some_text"
some_text⏎

Just try it,

# first encode
$ echo -n "mega_secret_key" | base64
bWVnYV9zZWNyZXRfa2V5
$ echo -n "really_secret_value1" | base64
cmVhbGx5X3NlY3JldF92YWx1ZTE=
# then decode and check whether newline is stripped
$ echo "bWVnYV9zZWNyZXRfa2V5" | base64 -d
mega_secret_key⏎
$ echo "cmVhbGx5X3NlY3JldF92YWx1ZTE=" | base64 -d
really_secret_value1⏎

You can use these newly (without newline) decoded data in your secret instead. That also should fine.

$ cat - <<-EOF | kubectl apply -f -
apiVersion: v1
kind: Secret
metadata:
  name: dummy-secret
type: Opaque
data:
  API_KEY: bWVnYV9zZWNyZXRfa2V5
  API_SECRET: cmVhbGx5X3NlY3JldF92YWx1ZTE=
EOF
secret/dummy-secret created

At the time of update, my kubernetes version is,

Minor:"17", GitVersion:"v1.17.3",
GitCommit:"06ad960bfd03b39c8310aaf92d1e7c1 2ce618213",
GitTreeState:"clean", BuildDate:"2020-02-11T18:14:22Z",
GoVersion:"go1.13.6", Compiler:"gc", Platform:"l inux/amd64"} Server
Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.3",
GitCommit:"06ad960bfd03b39c8310aaf92d1e7c1 2ce618213",
GitTreeState:"clean", BuildDate:"2020-02-11T18:07:13Z",
GoVersion:"go1.13.6", Compiler:"gc", Platform:"l inux/amd64"} ```

This was already answered but for future reference, there is no need to encode the strings by using

stringData

data

#secrets.yaml

apiVersion: v1

kind: Secret

metadata:

  name: mysecret

type: Opaque

stringData:

  API_KEY: "STRING_IN_CLEAR_TEXT"

  API_SECRET: "STRING_IN_CLEAR_TEXT"

Looks like your error message happens with a different

dummy-secret.yaml

apiVersion: v1

kind: Secret

metadata:

  name: dummy-secret

type: Opaque

data:

  API_KEY: af76fsdK_cQ89_Hj1Aq

  API_SECRET: bsdfmkwegwegwe

Then:

$ kubectl create -f s.yaml
Error from server (BadRequest): error when creating "dummy-secret.yaml": Secret in version "v1" cannot be handled as a Secret: v1.Secret.Data: decode base64: illegal base64 data at input byte 8, error found in #10 byte of ...|Q89_Hj1Aq","API_SECR|..., bigger context ...|sion":"v1","data":{"API_KEY":"af76fsdK_cQ89_Hj1Aq","API_SECRET":"bsdfmkwegwegwe"},"kind":"Secret","m|...

If I use your original it works fine:

apiVersion: v1
kind: Secret
metadata:
  name: dummy-secret
type: Opaque
data:
  API_KEY: bWVnYV9zZWNyZXRfa2V5
  API_SECRET: cmVhbGx5X3NlY3JldF92YWx1ZTE=

Then:

$ kubectl create -f dummy-secret.yaml
secret/dummy-secret created

I'm using the following version:

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.2", GitCommit:"17c77c7898218073f14c8d573582e8d2313dc740", GitTreeState:"clean", BuildDate:"2018-10-30T21:39:38Z", GoVersion:"go1.11.1", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.1", GitCommit:"4ed3216f3ec431b140b1d899130a69fc671678f4", GitTreeState:"clean", BuildDate:"2018-10-05T16:36:14Z", GoVersion:"go1.10.4", Compiler:"gc", Platform:"linux/amd64"}

The reason I was getting the error was a redundant space trailing my base64 encoded value:

Please check the difference:

kubectl patch secret mycred --type=json \
  --patch="[{\"op\": \"replace\", \"path\": \
  \"/data/.dockerconfigjson\", \
  \"value\": \"$(echo -n ${CRED_VALUE} | base64 -w0) \" }]"

and the correct one

kubectl patch secret mycred --type=json \
  --patch="[{\"op\": \"replace\", \"path\": \
  \"/data/.dockerconfigjson\", \
  \"value\": \"$(echo -n ${CRED_VALUE} | base64 -w0)\" }]"

(no space after the last base64 and before the quote)

Also please note that there can be multiple levels of enconding. E.g. the K8s YAML value is a base64 encoded value of another JSON, in which one of the values is again base64 encoded. The error can occur at any of those levels.

This may also happen when trying to remove the new line characters in a wrong way (the correct way is to remove the suffix "Cg==").

I used base64 from cli although there are workarounds to avoid the NL, like

https://superuser.com/questions/1225134/why-does-the-base64-of-a-string-contain-n/1225334

they don't work in MacOS, I found it simpler to use python like this

import base64

data = "abc123!?$*&()'-=@~"

# Standard Base64 Encoding
encodedBytes = base64.b64encode(data.encode("utf-8"))
encodedStr = str(encodedBytes, "utf-8")

Or still using a pure bash-based solution:

echo 'secret_to_encode' | tr -d \\n | base64

In my case, I forgot to convert a value for a key. It can be a thing like this :). Check your values

make backup

    sudo mkdir backup 
    sudo cp -R /etc/kubernetes backup/
    sudo tar -cvzf backup/pki_backup_`hostname`-`date +%Y%m%d`.tar.gz backup/kubernetes/

look to dir /etc/kubernetes/

ls -l

total 80
-rw------- 1 root root 5440 Mar  3 13:21 admin.conf
drwxr-xr-x 2 root root 4096 Aug 17  2020 audit-policy
-rw-r--r-- 1 root root  368 Mar  4  2020 calico-config.yml
-rw-r--r-- 1 root root  270 Mar  4  2020 calico-crb.yml
-rw-r--r-- 1 root root  341 Mar  4  2020 calico-cr.yml
-rw-r--r-- 1 root root  147 Mar  4  2020 calico-node-sa.yml
-rw-r--r-- 1 root root 6363 Mar  4  2020 calico-node.yml
-rw------- 1 root root 5472 Mar  3 13:21 controller-manager.conf
-rw-r--r-- 1 root root 3041 Aug 14  2020 kubeadm-config.v1alpha3.yaml
-rw------- 1 root root 5548 Mar  3 13:21 kubelet.conf
-rw-r--r-- 1 root root 1751 Mar  4  2020 kubelet.env
drwxr-xr-x 2 kube root 4096 Aug 14  2020 manifests
lrwxrwxrwx 1 root root   28 Mar  4  2020 node-kubeconfig.yaml -> /etc/kubernetes/kubelet.conf
-rw------- 1 root root 5420 Mar  3 13:21 scheduler.conf
drwxr-xr-x 3 kube root 4096 Mar  3 10:20 ssl

try to find k8s cluster config, in my case this is kubeadm-config.v1alpha3.yaml if same file absent you can generate

kubectl get cm kubeadm-config -n kube-system -o yaml > /etc/kubernetes/kubeadm-config.yaml

also in my case absent folder /etc/kubernetes/pki and exist ~/ssl I create symlink /etc/kubernetes/pki from /etc/kubernetes/ssl

ln -s /etc/kubernetes/ssl /etc/kubernetes/pki

let regenerate certs

kubeadm alpha phase certs apiserver  --config /etc/kubernetes/kubeadm-config.v1alpha3.yaml

[certificates] Using the existing apiserver certificate and key.

kubeadm alpha phase certs apiserver-kubelet-client

I0303 13:12:24.543254   40613 version.go:236] remote version is much newer: v1.20.4; falling back to: stable-1.12
[certificates] Using the existing apiserver-kubelet-client certificate and key.

kubeadm alpha phase certs front-proxy-client

I0303 13:12:35.660672   40989 version.go:236] remote version is much newer: v1.20.4; falling back to: stable-1.12
[certificates] Using the existing front-proxy-client certificate and key.

kubeadm alpha phase certs  etcd-server --config /etc/kubernetes/kubeadm-config.v1alpha3.yaml

[certificates] Generated etcd/server certificate and key.
[certificates] etcd/server serving cert is signed for DNS names [prod-uct1-mvp-k8s-0001 localhost] and IPs [127.0.0.1 ::1]

kubeadm alpha phase certs  etcd-server --config /etc/kubernetes/kubeadm-config.v1alpha3.yaml

[certificates] Using the existing etcd/server certificate and key.

kubeadm alpha phase certs  etcd-healthcheck-client --config /etc/kubernetes/kubeadm-config.v1alpha3.yaml

[certificates] Generated etcd/healthcheck-client certificate and key.

kubeadm alpha phase certs  etcd-peer --config /etc/kubernetes/kubeadm-config.v1alpha3.yaml

[certificates] Generated etcd/peer certificate and key.
[certificates] etcd/peer serving cert is signed for DNS names [product1-mvp-k8s-0001 localhost] and IPs [192.168.4.201 127.0.0.1 ::1]

check actuality certs

find /etc/kubernetes/pki/ -name '*.crt' -exec openssl x509 -text -noout -in {} \; | grep -A2 Validity

        Validity
            Not Before: Mar  4 10:29:44 2020 GMT
            Not After : Mar  2 10:29:44 2030 GMT
--
        Validity
            Not Before: Mar  4 10:29:44 2020 GMT
            Not After : Mar  3 10:07:29 2022 GMT
--
        Validity
            Not Before: Mar  4 10:29:44 2020 GMT
            Not After : Mar  3 10:07:52 2022 GMT
--
        Validity
            Not Before: Mar  4 10:29:44 2020 GMT
            Not After : Mar  3 10:06:48 2022 GMT
--
        Validity
            Not Before: Mar  4 10:29:44 2020 GMT
            Not After : Mar  2 10:29:44 2030 GMT
--
        Validity
            Not Before: Mar  4 10:29:44 2020 GMT
            Not After : Mar  2 19:39:56 2022 GMT
--
        Validity
            Not Before: Mar  4 10:29:43 2020 GMT
            Not After : Mar  2 10:29:43 2030 GMT
--
        Validity
            Not Before: Mar  4 10:29:43 2020 GMT
            Not After : Mar  2 19:40:13 2022 GMT
--
        Validity
            Not Before: Mar  4 10:29:44 2020 GMT
            Not After : Mar  2 19:36:38 2022 GMT

next step to generate new config files: admin.conf, controller-manager.conf, kubelet.conf, scheduler.conf, in first step move old files to /tmp

cd /etc/kubernetes/
mv {admin.conf,controller-manager.conf,kubelet.conf,scheduler.conf} /tmp/

kubeadm alpha phase kubeconfig all  --config /etc/kubernetes/kubeadm-config.v1alpha3.yaml 

[kubeconfig] Using existing up-to-date KubeConfig file: "/etc/kubernetes/admin.conf"
[kubeconfig] Using existing up-to-date KubeConfig file: "/etc/kubernetes/kubelet.conf"
[kubeconfig] Using existing up-to-date KubeConfig file: "/etc/kubernetes/controller-manager.conf"
[kubeconfig] Using existing up-to-date KubeConfig file: "/etc/kubernetes/scheduler.conf"

after restart master kubelet and check status after restart

sudo systemctl stop kubelet; sudo docker stop $(docker ps -aq); sudo docker rm $(docker ps -aq); sudo systemctl start kubelet

systemctl status kubelet -l

● kubelet.service - Kubernetes Kubelet Server
   Loaded: loaded (/etc/systemd/system/kubelet.service; enabled; vendor preset: disabled)
   Active: active (running) since Wed 2021-03-03 14:00:22 MSK; 10s ago
     Docs: https://github.com/GoogleCloudPlatform/kubernetes
  Process: 52998 ExecStartPre=/bin/mkdir -p /var/lib/kubelet/volume-plugins (code=exited, status=0/SUCCESS)
 Main PID: 53001 (kubelet)
   Memory: 51.2M
   CGroup: /system.slice/kubelet.service

check access master node

kubectl get nodes

kubectl get ns

NAME                  STATUS   AGE
default               Active   464d
product1-mvp          Active   318d
infra-logging         Active   315d
infra-nginx-ingress   Active   386d
kube-public           Active   464d
kube-system           Active   464d
pg                    Active   318d

check cert

notAfter=Mar  3 07:40:43 2022 GMT

repeat procedure for all master nodes.

Now we ready to renew certs for work nodes. In first step we must delete or move kubelet.conf

cd /etc/kubernetes/

mv kubelet.conf kubelet.conf_old

after change bootstrap-kubelet.conf

**apiVersion: v1 clusters:

• cluster: certificate-authority-data: | LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUN5RENDQWJDZ0F3SUJBZ0lCQURBTkJna3Foa2lHOXcwQkFRc0ZBREFWTVJNd0VRWURWUVFERXdwcmRXSmwKY201bGRHVnpNQjRYRFRJd01ETX server: https://192.168.4.201:6443 name: product1 contexts:
• context: cluster: product1 user: tls-bootstrap-token-user name: tls-bootstrap-token-user@product1 current-context: tls-bootstrap-token-user@product1 kind: Config preferences: {} users:
• name: tls-bootstrap-token-user user: token: fgz9qz.lujw0bwsdfhdsfjhgds**

where: certificate-authority-data – root certificate PKI CA master, may take from /etc/kubernetes/kubelet.conf on master nodes

• server: https://192.168.4.201:6443 - ip api server master node virtual balance ip

token: fgz9qz.lujw0bwsdfhdsfjhgds - token, generate on master node

kubeadm token create

after restart and check kubelet work node and work node come back to cluster:

systemctl restart kubelet

systemctl status kubelet -l

● kubelet.service - Kubernetes Kubelet Server
   Loaded: loaded (/etc/systemd/system/kubelet.service; enabled; vendor preset: disabled)
   Active: active (running) since Wed 2021-03-03 14:06:33 MSK; 11s ago
     Docs: https://github.com/GoogleCloudPlatform/kubernetes
  Process: 54615 ExecStartPre=/bin/mkdir -p /var/lib/kubelet/volume-plugins (code=exited, status=0/SUCCESS)
 Main PID: 54621 (kubelet)
   Memory: 52.1M
   CGroup: /system.slice/kubelet.service

check update certs may be ls in dir

ls -las /var/lib/kubelet/pki/

total 24
4 -rw-------. 1 root root 1135 Mar  3 14:06 kubelet-client-2021-03-03-14-06-34.pem
0 lrwxrwxrwx. 1 root root   59 Mar  3 14:06 kubelet-client-current.pem -> /var/lib/kubelet/pki/kubelet-client-2021-03-03-14-06-34.pem
4 -rw-r--r--. 1 root root 2267 Mar  2 10:40 kubelet.crt
4 -rw-------. 1 root root 1679 Mar  2 10:40 kubelet.key

after repeat this procedure by all work nodes.

I encountered the same issue.<br> In my case the encoding was fine - I just copied the output without the last character of

"="

Additional scenarios that might lead to the same error:

1. Misplaced whitespaces or broken lines in the base64 copied output.

2. Not specifying the

   type

   field in the secret.