Common errors in OCP on bare metal
Failed to apply spec due Forbidden: may not be used when type is ClusterIP
If you had an older version of Portworx manifests installed, and you try to apply the latest manifests, you might see the following error during kubectl apply
.
Service "portworx-service" is invalid: [spec.ports[0].nodePort: Forbidden: may not be used when `type` is 'ClusterIP', spec.ports[1].nodePort: Forbidden: may not be used when `type` is 'ClusterIP', spec.ports[2].nodePort: Forbidden: may not be used when `type` is 'ClusterIP', spec.ports[3].nodePort: Forbidden: may not be used when `type` is 'ClusterIP']
Error from server (Invalid): error when applying patch:
To fix this:
-
Change the type of the
portworx-service
service to type ClusterIP. If the type was NodePort, you will also have to remove the nodePort entries from the spec.kubectl edit service portworx-service -n <px-namespace>
-
Change the type of the
portworx-api
service to type ClusterIP. If the type was NodePort, you will also have to remove the nodePort entries from the spec.kubectl edit service portworx-api -n <px-namespace>
-
Reapply your specs.
Failed DNS resolution
If you encounter the following error:
Jan 18 12:48:03 node1 portworx[872]: level=error msg="error in obtaining etcd version: \
Get http://_some_host:2379/version: dial tcp: lookup _some_host on [::1]:53: dial udp \
[::1]:53: connect: no route to host"
Please ensure that the NetworkManager
service has been stopped and disabled on your Linux host system.
Cause
The Portworx processes running inside the OCI container must be able to perform the DNS hostname resolution, especially if using hostnames for KVDB configuration, or the CloudSnap feature. However, host's NetworkManager
service can update the DNS configuration (the /etc/resolv.conf
file) after the Portworx container has started, and such changes will not propagate from host to container.
Failure to install Portworx on SELinux
You may have experienced the following issue installing Portworx (e.g. Fedora 28 host)
# sudo docker run --entrypoint /runc-entry-point.sh --rm -i --name px-installer --privileged=true \
-v /etc/pwx:/etc/pwx -v /opt/pwx:/opt/pwx portworx/px-base-enterprise:2.1.2
docker: Error response from daemon: OCI runtime create failed: container_linux.go:345: starting \
container process caused "process_linux.go:430: container init caused \
\"write /proc/self/attr/keycreate: permission denied\"": unknown.
Cause
This error is caused by a Docker issue (see moby#39109), which prevents Docker from running even the simplest containers:
# sudo docker run --rm -it hello-world
docker: Error response from daemon: OCI runtime create failed: container_linux.go:345: starting \
container process caused "process_linux.go:430: container init caused \
\"write /proc/self/attr/keycreate: permission denied\"": unknown.
To work around this issue, either turn off SELinux support, or make sure to use docker-package provided by the host's platform.
Failure to install Portworx on Kubernetes
You may experience the following issue deploying Portworx into Kubernetes:
...level=info msg="Locating my container handler"
...level=info msg="> Attempt to use Docker as container handler failed" \
error="/var/run/docker.sock not a socket-file"
...level=info msg="> Attempt to use ContainerD as container handler failed" \
error="Could not load container 134*: container \"134*\" in namespace \"k8s.io\": not found"
...level=info msg="> Attempt to use k8s-CRI as container handler failed" \
error="stat /var/run/crio/crio.sock: no such file or directory"
...level=error msg="Could not instantiate container client" \
error="Could not initialize container handler"
...level=error msg="Could not talk to Docker/Containerd/CRI - please ensure \
'/var/run/docker.sock', '/run/containerd/containerd.sock' or \
'/var/run/crio/crio.sock' are mounted"
Cause
In Kubernetes environments, Portworx installation starts by deploying OCI-Monitor Daemonset, which monitors and manages the Portworx service. In order to download, install and/or validate the Portworx service, the OCI-Monitor connects to the appropriate Kubernetes container runtime via socket-files that need to be mounted into the OCI-Monitor's POD.
Inspect the Portworx spec, and ensure the appropriate socket-files/directories are mounted as volumes from the host-system into the Portworx POD. Alternatively, you can reinstall Portworx, or at minimum generate a new YAML-spec via the Portworx spec generator page in Portworx Central, and copy the volume-mounts into your Portworx spec.
Failed to attach volume, device exists
When using Dynatrace on your cluster, you may encounter the following volume attach error:
failed to attach volume: rpc error: code = Internal desc = failed to attach volume: Cannot attach volume <volume-id>, device exists
Cause
Dynatrace OneAgent, running on all worker nodes, opens a file handle on a Portworx internal path (/var/lib/osd/pxns
). This prevents Portworx from detaching its volumes, causing issues when you try to reattach them.
You can correct this issue by instructing Dynatrace to skip monitoring Portworx internal mountpaths. Portworx can then cleanly detach and remove its block devices from the host.