As a system administrator, chances are that there are some tasks that you do repetitively every single day that takes up a lot of your time that could have been invested elsewhere. What if you could solve problems once and then automate your solutions going forward? That is what Ansible is good at and once you get to see its benefits, you will never look back again. Ansible is a simple, yet powerful IT automation engine that thousands of companies are using to drive complexity out of their environments and accelerate DevOps initiatives. In this article we will perform an installation of Ansible AWX on CentOS 8 / Rocky Linux 8 Server.

Be it the deployment of applications, routine maintenance of your servers, Configuration Management, Continuous Delivery, Orchestration or any repetitive work that you can describe, Ansible can handle it for you. To add beauty to this beast, AWX (Ansible Web eXecutable) provides a web-based user interface, REST API, and task engine built on top of Ansible. The AWX Project is an open source community project, sponsored by Red Hat, that enables users to better control their use of Ansible project in IT environments. AWX is the upstream project from which the Red Hat Ansible Tower offering is ultimately derived.

“Don’t wait. The time will never be just right.”
–Napoleon Hill

Setup minimum requirements

Before you can run a deployment, you’ll need the following installed in your local environment

  • Kubernetes Cluster / Docker
  • User with sudo access
  • CentOS 8 / Rocky Linux 8 server
  • At least 8GB of RAM – More is better if available
  • 4vcpus – Minimum CPU but add more if available
  • 25GB minimum disk space

We should be now ready to roll up our sleeves, put on our boots and get to work.

Step 1: Update your system

The first step is performing a system update.

sudo dnf -y update

Disable Firewalld. This is recommended by K3s.

sudo systemctl disable firewalld --now

Once the update is successful perform a system reboot

sudo reboot

Step 2: Install K3s Kubernetes Distribution

AWX is supported and can only be run as a containerized application using Docker images deployed to either an OpenShift cluster, a Kubernetes cluster, or docker-compose. We shall use K3s Kubernetes setup to run AWX on CentOS 8 / Rocky Linux 8.

Put SELinux in permissive mode:

sudo setenforce 0
sudo sed -i 's/^SELINUX=.*/SELINUX=permissive/g' /etc/selinux/config
cat /etc/selinux/config | grep SELINUX=

Install k3s by running the commands below:

curl -sfL https://get.k3s.io | sudo bash -
sudo chmod 644 /etc/rancher/k3s/k3s.yaml

If running as non root user configure kubeconfig.

mkdir ~/.kube
sudo cp /etc/rancher/k3s/k3s.yaml ~/.kube/config
sudo chown $USER ~/.kube/config

Check k3s service to confirm it is running and working:

$ systemctl status k3s.service
systemctl status k3s.service
 k3s.service - Lightweight Kubernetes
   Loaded: loaded (/etc/systemd/system/k3s.service; enabled; vendor preset: disabled)
   Active: active (running) since Wed 2024-01-24 20:47:10 UTC; 13min ago
     Docs: https://k3s.io
  Process: 3209 ExecStartPre=/sbin/modprobe overlay (code=exited, status=0/SUCCESS)
  Process: 3203 ExecStartPre=/sbin/modprobe br_netfilter (code=exited, status=0/SUCCESS)
  Process: 3200 ExecStartPre=/bin/sh -xc ! /usr/bin/systemctl is-enabled --quiet nm-cloud-setup.service 2>/dev/null (code=exited, status=0/SUCCESS)
 Main PID: 3212 (k3s-server)
    Tasks: 85
   Memory: 1.5G
   CGroup: /system.slice/k3s.service
           ├─3212 /usr/local/bin/k3s server
           ├─3234 containerd
           ├─3758 /var/lib/rancher/k3s/data/28f7e87eba734b7f7731dc900e2c84e0e98ce869f3dcf57f65dc7bbb80e12e56/bin/containerd-shim-runc-v2 -namespace k8s.io -id 7137f7799637ca3b3486327bb4bb78f9bfda27>
           ├─3892 /var/lib/rancher/k3s/data/28f7e87eba734b7f7731dc900e2c84e0e98ce869f3dcf57f65dc7bbb80e12e56/bin/containerd-shim-runc-v2 -namespace k8s.io -id 3ee13bd8b699fd2313a76ad01bc4cab473d4bc>
           ├─3975 /var/lib/rancher/k3s/data/28f7e87eba734b7f7731dc900e2c84e0e98ce869f3dcf57f65dc7bbb80e12e56/bin/containerd-shim-runc-v2 -namespace k8s.io -id 46623440fab9606bb2f47f1e05d97d3f877cdf>
           ├─4758 /var/lib/rancher/k3s/data/28f7e87eba734b7f7731dc900e2c84e0e98ce869f3dcf57f65dc7bbb80e12e56/bin/containerd-shim-runc-v2 -namespace k8s.io -id cf023b329ddd5579585c9451b485266861ffdf>
           └─4808 /var/lib/rancher/k3s/data/28f7e87eba734b7f7731dc900e2c84e0e98ce869f3dcf57f65dc7bbb80e12e56/bin/containerd-shim-runc-v2 -namespace k8
....

As root user do a validation on use of kubectl Kubernetes management tool:

$ kubectl get nodes
NAME              STATUS   ROLES                  AGE   VERSION
rocky8.mylab.io   Ready    control-plane,master   14m   v1.28.5+k3s1

You can also confirm Kubernetes version deployed using the following command:

$ kubectl version
Client Version: v1.28.5+k3s1
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.28.5+k3s1

The K3s service will be configured to automatically restart after node reboots or if the process crashes or is killed.

Step 3: Deploy AWX Operator on Kubernetes

This Kubernetes Operator has to be deployed for the management of one or more AWX instances in any namespace.

Install git and make tools:

sudo yum -y install tar git make

Clone operator deployment code:

# git clone https://github.com/ansible/awx-operator.git
Cloning into 'awx-operator'...
remote: Enumerating objects: 5626, done.
remote: Counting objects: 100% (2840/2840), done.
remote: Compressing objects: 100% (1025/1025), done.
remote: Total 5626 (delta 1910), reused 2434 (delta 1716), pack-reused 2786
Receiving objects: 100% (5626/5626), 1.38 MiB | 12.01 MiB/s, done.
Resolving deltas: 100% (3191/3191), done.

Create namespace where operator will be deployed. I’ll name mine awx:

export NAMESPACE=awx
kubectl create ns ${NAMESPACE}

Set current context to value set in NAMESPACE variable:

$ kubectl config set-context --current --namespace=$NAMESPACE 
Context "default" modified.

Switch to awx-operator directory:

cd awx-operator/

Save the latest version from AWX Operator releases as RELEASE_TAG variable then checkout to the branch using git.

sudo yum -y install jq
RELEASE_TAG=`curl -s https://api.github.com/repos/ansible/awx-operator/releases/latest | grep tag_name | cut -d '"' -f 4`
echo $RELEASE_TAG

Deploy AWX Operator into your cluster:

git checkout $RELEASE_TAG
export NAMESPACE=awx
make deploy

Command execution terminal output:

namespace/awx configured
customresourcedefinition.apiextensions.k8s.io/awxbackups.awx.ansible.com created
customresourcedefinition.apiextensions.k8s.io/awxrestores.awx.ansible.com created
customresourcedefinition.apiextensions.k8s.io/awxs.awx.ansible.com created
serviceaccount/awx-operator-controller-manager created
role.rbac.authorization.k8s.io/awx-operator-awx-manager-role created
role.rbac.authorization.k8s.io/awx-operator-leader-election-role created
clusterrole.rbac.authorization.k8s.io/awx-operator-metrics-reader created
clusterrole.rbac.authorization.k8s.io/awx-operator-proxy-role created
rolebinding.rbac.authorization.k8s.io/awx-operator-awx-manager-rolebinding created
rolebinding.rbac.authorization.k8s.io/awx-operator-leader-election-rolebinding created
clusterrolebinding.rbac.authorization.k8s.io/awx-operator-proxy-rolebinding created
configmap/awx-operator-awx-manager-config created
service/awx-operator-controller-manager-metrics-service created
deployment.apps/awx-operator-controller-manager created

Wait a few minutes and awx-operator should be running:

# kubectl get pods -n awx
NAME                                               READY   STATUS    RESTARTS   AGE
awx-operator-controller-manager-68d787cfbd-z75n4   2/2     Running   0          48s

Uninstalling AWX Operator – Don’t run, only for reference🙂

You can always remove the operator and all associated CRDs by running the command below:

# export NAMESPACE=awx
# make undeploy
/root/awx-operator/bin/kustomize build config/default | kubectl delete -f -
namespace "awx" deleted
customresourcedefinition.apiextensions.k8s.io "awxbackups.awx.ansible.com" deleted
customresourcedefinition.apiextensions.k8s.io "awxrestores.awx.ansible.com" deleted
customresourcedefinition.apiextensions.k8s.io "awxs.awx.ansible.com" deleted
serviceaccount "awx-operator-controller-manager" deleted
role.rbac.authorization.k8s.io "awx-operator-leader-election-role" deleted
role.rbac.authorization.k8s.io "awx-operator-manager-role" deleted
clusterrole.rbac.authorization.k8s.io "awx-operator-metrics-reader" deleted
clusterrole.rbac.authorization.k8s.io "awx-operator-proxy-role" deleted
rolebinding.rbac.authorization.k8s.io "awx-operator-leader-election-rolebinding" deleted
rolebinding.rbac.authorization.k8s.io "awx-operator-manager-rolebinding" deleted
clusterrolebinding.rbac.authorization.k8s.io "awx-operator-proxy-rolebinding" deleted
configmap "awx-operator-manager-config" deleted
service "awx-operator-controller-manager-metrics-service" deleted
deployment.apps "awx-operator-controller-manager" deleted

Step 4: Install Ansible AWX

Now that we have the operator pod running we are ready to initiate installation of Ansible AWX on CentOS 8 / Rocky Linux 8. But first we’ll need to create a PVC for public and static web data.

Create a file named public-static-pvc.yaml:

vi public-static-pvc.yaml

Input below contents in the file:

---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: public-static-data-pvc
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: local-path
  resources:
    requests:
      storage: 5Gi

Apply configuration manifest:

# kubectl apply -f public-static-pvc.yaml -n awx
persistentvolumeclaim/public-static-data-pvc created

PVC won’t be bound until the pod that uses it is created.

# kubectl get pvc -n awx
NAME                     STATUS    VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS   AGE
public-static-data-pvc   Pending                                      local-path     43s

Create AWX deployment file:

vi awx-instance-deployment.yml

Paste below contents to the file created.

---
apiVersion: awx.ansible.com/v1beta1
kind: AWX
metadata:
  name: awx
spec:
  service_type: nodeport
  projects_persistence: true
  projects_storage_access_mode: ReadWriteOnce
  web_extra_volume_mounts: |
    - name: static-data
      mountPath: /var/lib/projects
  extra_volumes: |
    - name: static-data
      persistentVolumeClaim:
        claimName: public-static-data-pvc

Install AWX on CentOS 8 / Rocky Linux 8:

# kubectl apply -f awx-instance-deployment.yml -n awx
awx.awx.ansible.com/awx created

After few minutes check pods creation status:

# watch kubectl get pods -l "app.kubernetes.io/managed-by=awx-operator" -n awx
NAME                   READY   STATUS    RESTARTS   AGE
awx-postgres-0         1/1     Running   0          2m58s
awx-75698588d6-qz2gf   4/4     Running   0          2m42s

 You can track the installation process at the operator pod logs:

kubectl logs -f deployments/awx-operator-controller-manager -c awx-manager

Extra PVCs are created automatically:

# kubectl  get pvc
NAME                      STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
postgres-awx-postgres-0   Bound    pvc-34a25045-486c-42a8-9763-d14a7bb3e9e8   8Gi        RWO            local-path     72s
public-static-data-pvc    Bound    pvc-3484e513-8d00-482c-bdce-6e77820f237e   1Gi        RWO            local-path     5m13s
awx-projects-claim        Bound    pvc-e56ab471-97f1-455b-9d51-ba05b7d9982b   8Gi        RWO            local-path     60s

Fixing the error “mkdir: cannot create directory ‘/var/lib/postgresql/data’: Permission denied”

If you see the error message from postgres pod logs:

# kubectl get pods
# kubectl logs awx-postgres-13-0
mkdir: cannot create directory ‘/var/lib/postgresql/data’: Permission denied

It means the Postgres pod cannot write to the persistent volume directory inside /var/lib/rancher/k3s/storage/:

# ls -lh /var/lib/rancher/k3s/storage/ | grep awx-postgres-0
total 0
drwx------. 3 root root 18 Aug  3 14:04 pvc-8110b494-d9ed-450a-94c0-b9dfd2bd73f7_default_postgres-awx-postgres-0

Try setting the directory mode to 777

# chmod -R 777  /var/lib/rancher/k3s/storage/*
# kubectl delete pods -l "app.kubernetes.io/managed-by=awx-operator" -n awx
pod "awx-75698588d6-x79g2" deleted
pod "awx-postgres-0" deleted

The Postgres container pod should come up in few seconds:

# kubectl get pods -n awx
NAME                            READY   STATUS    RESTARTS   AGE
awx-operator-545497f7d5-bqlcs   1/1     Running   0          65m
awx-postgres-0                  1/1     Running   4          8m22s
awx-75698588d6-7kg9j            4/4     Running   0          8m10s

Checking AWX Container’s logs

The awx-xxx-yyy pod will have four containers, namely:

  • redis
  • awx-web
  • awx-task
  • awx-ee

As can be seen from below command output:

# kubectl get deploy
NAME                              READY   UP-TO-DATE   AVAILABLE   AGE
awx-operator-controller-manager   1/1     1            1           11m
awx-task                          1/1     1            1           7m14s
awx-web                           1/1     1            1           5m44s

# kubectl -n awx  logs deploy/awx-web 
...
2024-01-24 21:29:49,683 INFO     Listening on TCP address 127.0.0.1:8051
2024-01-24 21:34:15,259 INFO     [-] awx.main.consumers client 'specific.8c10297088744213b264f42ae113861c!1d5294ef84e34870aa1e889fd8569537' joined the broadcast group.
2024-01-24 21:34:15,259 INFO     [-] awx.main.consumers client 'specific.8c10297088744213b264f42ae113861c!1d5294ef84e34870aa1e889fd8569537' joined the broadcast group.
2024-01-24 21:34:15,259 INFO     client 'specific.8c10297088744213b264f42ae113861c!1d5294ef84e34870aa1e889fd8569537' joined the broadcast group

You’ll need to provide container name after the pod:

kubectl -n awx  logs deploy/awx-web -c redis
kubectl -n awx  logs deploy/awx-web -c awx-web
kubectl -n awx  logs deploy/awx-web -c awx-rsyslog

Access AWX Container’s Shell

Here is how to access each container’s shell:

kubectl exec -ti deploy/awx-web -c redis -- /bin/bash
kubectl exec -ti deploy/awx-web  -c  awx-web -- /bin/bash
kubectl exec -ti awx-postgres-13-0  -c  postgres -- /bin/bash

Upgrading AWX Instance

For upgrade process refer to our guide in the link below:

Step 5: Access AWX Web Interface

Get the AWX Web service port:

# kubectl get service -n awx
NAME           TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)        AGE
awx-postgres   ClusterIP   None           <none>        5432/TCP       5m17s
awx-service    NodePort    10.43.54.203   <none>        80:30080/TCP   5m7s

From the output we can confirm service node port is 30080.

To have access to AWX web console, point your browser to your Ansible’s AWX server IP:

http://your-server-ip-address:30080

You should be welcomed to a Login page well illustrated below.

install ansible awx ubuntu using operator 01

The login username is admin

Obtain admin user password by decoding the secret with the password value:

kubectl -n awx get secret awx-admin-password -o jsonpath="{.data.password}" | base64 --decode

Better output format:

kubectl -n awx get secret awx-admin-password -o go-template='{{range $k,$v := .data}}{{printf "%s: " $k}}{{if not $v}}{{$v}}{{else}}{{$v | base64decode}}{{end}}{{"\n"}}{{end}}'

Login with the admin username and decoded password from above commands:

install ansible awx ubuntu using operator 02

Configure Ingress for AWX

If you would like to access your AWX using domain names and SSL, check out our ingress articles:

Once the authentication is successful, you’ll get to AWX administration dashboard. Therein, there is a lot of stuff to do and we shall cover just a few. Along the left side of the Dashboard is the navigation menu, where you can quickly navigate to your Projects, Inventories, Job Templates, and Jobs

install ansible awx ubuntu using operator 03

Step 6: Create User and Team

AWX provides a default user called admin that you can use to do your tests. But if you are need a different user, you can create another one.

To add a new user apart from the admin one you find by default, click on Users tab found on the left menu. A new page will be displayed where you can add a new user. Click on the green + icon then fill in the details of the new user to be added.

AWX 1 Create a new user to add to organization 1

After you are done, click “Save“.

AWX Create a new user to add to organization

The same applies for creating a new Team. Click on “Teams” tab found on the left menu. A new page will be displayed where you can add a new team. Click on the green + icon then fill in the details of the new team to be added.

AWX 1 Create a new Team to add to organization

After you are done, click “Save“.

AWX 2 Create a new Team to add to organization

Step 7: Create an Organization

An organization is a logical collection of users, teams, projects, and inventories. It is the highest level object in the AWX object hierarchy. From the left navigation bar, click the Organizations icon. You will find that a default organization has been automatically created and is available to all users of Ansible AWX.

Awx ansible dashboard organization 3

It can be used as is or edited as needed. Let us edit it by changing its name and adding users, permissions and more.

Awx ansible dashboard organization edit 4

Click on the “Users” tab and add a new user. I had not created a new user before, so admin default user appears. But you can create other users as shared in Step 5.

Awx ansible dashboard organization add user 5

If you had other users, they would be populated therein

Awx ansible dashboard organization add user 6 1

Clicking on Users displays all the Users associated with this Organization. A User is someone with access to Ansible AWX with associated roles and Credentials. Adding a user to an organization adds them as a member only, specifying a role for the user can be done in the the Permissions tab, as shown in the example below:

Awx ansible dashboard organization add permissions 7

Step 8: Create credentials

In order for Ansible to log in and execute tasks, it will need credentials to access the remote hosts. AWX provides this feature for us. Click on “Credentials” on the left menu and then click the green + to add a new server credential.

AWX 1 Create a new Credential to add to organization

Fill in the name, description, organization, username, password and the type of credential which is a long list. We shall choose machine since we intend to configure a single host in this example. You can add your ssh keys in case you prefer key-based authentication mechanism. Also add privilege escalation below to sudo.

AWX 2 Create a new Credential to add to organization

Click “Save” once done.

AWX 3 Create a new Credential to add to organization done

Step 9: Create a new Inventory and add it to the Organization

An inventory is a collection of hosts managed by Ansible. Inventories are assigned to organizations, while permissions to launch playbooks against inventories are controlled at the user and/or team level.

To create and review existing inventories click the Inventories icon from the left navigation bar. To create a new inventory, click the add (+) button and select Inventory from the drop-down menu list.

AWX 1 Create a new Inventory to add to organization

Enter the name and Organization that this Inventory will belong to. Click “Save” so that the other tabs can be activated.

AWX 2 Create a new Inventory to add to organization 1

For this example, we are going to add one host but know that you can create a group that contains one or more hosts you would wish to execute something on them simultaneously. Click on “Hosts” tab and click on + to add a new host.

AWX 3 Create a new Inventory host to add to organization

On the new page loaded, add an IP or resolvable hostname and a description. We are going to install nginx on the given host in this example thus the intuitive name.

AWX 4 Create a new Inventory host to add to organization

Click “Save” once done and the new Inventory should be created.

AWX 5 Click on the new Inventory host

The concept of Groups and Hosts

Note that inventories are divided into groups and hosts. A group might represent a particular environment (e.g. “Datacenter 1” or “Testing Bed”), a server type (e.g. “Web Servers” or “DB Servers”), or any other representation of your environment.

Step 10: Setting up a Project

A Project is a logical collection of Ansible playbooks, represented in Ansible AWX. You can manage playbooks and playbook directories by either placing them manually under the Project Base Path on your Ansible AWX server, or by placing your playbooks into a source code management (SCM) system supported by Ansible AWX, including Git, Subversion, and Mercurial.

To create a new project, follow the same procedure as the rest we have seen thus far. Click on the “Projects” tab on the left menu and then click on add(+) to create a new project.

AWX 1 Create a new project for our organization

Once there, fill the details to suit your needs. On the Source Control Manager (SCM) type, you can create a git repo that has all of your playbooks or create a local folder on your server as advised when you choose manual under SCM type. I preferred adding a git repo in this example. Note that we are still referring to the Organization of your choice in case you have several.

AWX 2 Create a new project for our organization

Enter all the details then click “Save“. The file created on the git repo (nginx.yml) has the following:

---
- hosts: all
  gather_facts: true
  become: true
  become_user: root
  tasks:
    - apt:
        name: nginx
      when: ansible_os_family == "Debian"

Attached screenshot below:

Nginx Git yaml

Step 11: Setting up a template and launching it

Thus far, we have done a lot and what remains is putting it all together into a template. A job template combines an Ansible playbook from a project and the settings required to launch it. Create a new job template by clicking on “Templates” tab on the left navigation menu. Click on he green add (+) to add a new template just like the others. When the window opens, it is just a matter of picking the Inventory, Project, Credential and Playbook we had already configured in the previous steps. Give your template a name and an optional description as well.

AWX 7 Add a new template details

When done scroll down and hit “Save“. Once you save, you can “Launch” it as a job immediately.

AWX 8 Add a new template details save and launch

Once you hit Launch you will be redirected to the “Jobs” page where you will see it executing live with output being displayed as it happens.

Nginx Playbook running

If everything goes well, the playbook should be run successfully by Ansible behind the scenes. All errors encountered along the way will be displayed on the right side.

Nginx Playbook successful

Login to the server being tested on and check if nginx was installed successfully

Nginx in debian10 running

Amazing stuff, Ansible AWX installation is officially setup and running properly.

More about Ansible Tower | AWX can be found on RedHat’s Ansible Tower Quick Start Guide.

To Conclude

Ansible can change the way you handle your day to day administration tasks and it can offload a lot of manual work from your hands and gift you with ample time for other tasks. Leverage this technology as your innovation engine, and deliver your applications faster and win big.

Other electrifying guides you might enjoy include:

105 COMMENTS

  1. The line to install powertools has changed:
    sudo dnf config-manager –set-enabled PowerTools
    Error: No matching repo to modify: PowerTools.

    This works:
    sudo dnf config-manager –set-enabled powertools

  2. observations with my effort:
    neither ‘dnf config-manager –set-enabled …’ worked, no repo of either name found
    no ~/awx/installer/ folder found
    invetory file is: ~/awx/tools/docker-compose/inventory, is this correct?
    no install.yml found under ~/awx/, different location?

    so, did I miss a step?

  3. I can build everything out following your steps – pods are created, everything bound – but when I try to get to the AWX login screen I get the following page in the browser:

    AWX is currently upgrading.
    This page will refresh when complete.

    I’ve attempted on both Rocky 8 and CentOS 8 with the same result. Tried rebooting, disabling SELinux, and opening http/80 with firewall-cmd. No change.

    Any suggestions?

  4. awx-75698588d6 Remains Pending
    # kubectl get po
    NAME READY STATUS RESTARTS AGE
    awx-75698588d6-8c87s 0/4 Pending 0 33m
    awx-operator-545497f7d5-svmdj 1/1 Running 1 37m
    awx-postgres-0 1/1 Running 1 33m

  5. Thanks for the detailed post.
    I am stuck at this point after applied the deployment and the PVCs. Pods are pending due to “Normal FailedBinding 7s (x24 over 5m28s) persistentvolume-controller no persistent volumes available for this claim and no storage class is set”

    kubectl get pods -l “app.kubernetes.io/managed-by=awx-operator” -w
    NAME READY STATUS RESTARTS AGE
    awx-75698588d6-jsmg6 0/4 Pending 0 9m11s
    awx-postgres-0 0/1 Pending 0 9m22s

    Any suggestions?

  6. Postgres keeps crashing. I keep getting this:
    # kubectl get pod
    NAME READY STATUS RESTARTS AGE
    awx-operator-545497f7d5-s4s7m 1/1 Running 0 2m45s
    awx-75698588d6-26dx8 0/4 ContainerCreating 0 11s
    awx-postgres-0 0/1 CrashLoopBackOff 1 20s

  7. # kubectl logs -f pods/awx-postgres-0
    mkdir: cannot create directory ‘/var/lib/postgresql/data’: Permission denied

  8. Events:
    Type Reason Age From Message
    —- —— —- —- ——-
    Normal Scheduled 2m54s default-scheduler Successfully assigned default/awx-postgres-0 to phltans0001
    Normal Pulling 2m54s kubelet Pulling image “postgres:12”
    Normal Pulled 2m50s kubelet Successfully pulled image “postgres:12” in 4.247522726s
    Normal Pulled 87s (x4 over 2m49s) kubelet Container image “postgres:12” already present on machine
    Normal Created 87s (x5 over 2m49s) kubelet Created container postgres
    Normal Started 87s (x5 over 2m49s) kubelet Started container postgres
    Warning BackOff 72s (x9 over 2m48s) kubelet Back-off restarting failed container

  9. I follow your steps, but i get this at the end:

    [root@localhost ~]# kubectl get pods
    NAME READY STATUS RESTARTS AGE
    awx-operator-545497f7d5-fdwmz 1/1 Running 0 4m2s
    awx-75698588d6-d9xw6 0/4 ContainerCreating 0 82s
    awx-postgres-0 0/1 CrashLoopBackOff 3 89s

  10. Sorry, I’m Kubernet agnostic. I followed the instructions within a virtual machine and everything seemed to go well. But now I cannot open the webgui. All i see are internal IP’s and I have no idea how to make it available to the world.
    Ay hints, what am I overlooking

    Paul

  11. Hello John

    First of all good paper with usefull details

    When I finished the install and try to login via https://myip.example.com I got :
    “404 page not found”

    When I try the NodePort I got blank page :
    [root@vmtowerlux ~]# kubectl get service -n awx
    NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
    awx-postgres ClusterIP None 5432/TCP 23h
    awx-service NodePort 10.43.242.21 80:30154/TCP 23h

    http://myip.example.com:30154 -> blank page

    Thanks for your help

  12. Hi

    Installation completed successfully but I am not able to login to web interface

    When I try : https://myip.example.com -> “404 page not found”

    When I try : http://myip.example.com:30154 -> blank page

    [root@vmtowerlux ~]# kubectl get svc -n awx
    NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
    awx-postgres ClusterIP None 5432/TCP 23h
    awx-service NodePort 10.43.242.21 80:30154/TCP 23h

    Thanks for your help

  13. @birb I had the same thing, I fixed it by manually creating the webdir (mkdir -p /var/lib/awx/public/static/) and a reboot!

  14. root@rocky8 ~]# kubectl get pods -l “app.kubernetes.io/managed-by=awx-operator” -n awx
    NAME READY STATUS RESTARTS AGE
    awx-548ccb647c-cdd69 0/4 Pending 0 7m47s
    awx-postgres-0 1/1 Running 0 7m39s
    there are no logs for each of the awx parts but I see this event in /var/log/messages
    Aug 15 12:29:39 rocky8 k3s[5916]: I0815 12:29:39.790467 5916 event.go:291] “Event occurred” object=”awx/public-static-data-pvc” kind=”PersistentVolumeClaim” apiVersion=”v1″ type=”Normal” reason=”WaitForPodScheduled” message=”waiting for pod awx-548ccb647c-cdd69 to be scheduled”

  15. Thanks for this great write up.
    I found out that all the issues I had are related to resources; ie cpu, memory and storage.
    I ended up allocating 8 GB ram, 5 vcpus and 25 GB storage.

    Did everyone have the same issue ? maybe these should be included in the write up.

    How can I reduce cpu and memory resources. I have run AWX 17 on docker with a lot less than these numbers, exactly 4 GB, 2 vcpu.

  16. AWX on RockyLinux 8.4 does not work. Cannot access WebGUI
    Seems postgres database is unreachable

    [root@awx02 ~]# ip a show ens192
    2: ens192: mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 00:50:56:a7:0a:b0 brd ff:ff:ff:ff:ff:ff
    inet 172.20.152.5/24 brd 172.20.152.255 scope global noprefixroute ens192
    valid_lft forever preferred_lft forever
    inet6 fe80::250:56ff:fea7:ab0/64 scope link
    valid_lft forever preferred_lft forever

    [root@awx02 ~]# kubectl get pods -n awx
    NAME READY STATUS RESTARTS AGE
    awx-postgres-0 1/1 Running 0 76m
    awx-548ccb647c-qv4bx 4/4 Running 10 77m

    [root@awx02 ~]# kubectl get service -n awx
    NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
    awx-postgres ClusterIP None 5432/TCP 76m
    awx-service NodePort 10.43.233.188 80:31364/TCP 77m

    [root@ansi ~]# ping -c 3 172.20.152.5
    PING 172.20.152.5 (172.20.152.5) 56(84) bytes of data.
    64 bytes from 172.20.152.5: icmp_seq=1 ttl=64 time=0.459 ms
    64 bytes from 172.20.152.5: icmp_seq=2 ttl=64 time=1.39 ms
    64 bytes from 172.20.152.5: icmp_seq=3 ttl=64 time=0.709 ms

    — 172.20.152.5 ping statistics —
    3 packets transmitted, 3 received, 0% packet loss, time 22ms
    rtt min/avg/max/mdev = 0.459/0.852/1.390/0.394 ms
    [root@ansi ~]# curl http://172.20.152.5:31364
    curl: (7) Failed to connect to 172.20.152.5 port 31364: Connection refused
    [root@ansi ~]# curl http://172.20.152.5
    404 page not found

    [root@awx02 ~]# kubectl -n awx logs awx-548ccb647c-qv4bx -c awx-web
    [wait-for-migrations] Waiting for database migrations…
    [wait-for-migrations] Attempt 1 of 30
    [wait-for-migrations] Waiting 0.5 seconds before next attempt
    [wait-for-migrations] Attempt 2 of 30
    [wait-for-migrations] Waiting 1 seconds before next attempt
    [wait-for-migrations] Attempt 3 of 30
    [wait-for-migrations] Waiting 2 seconds before next attempt
    [wait-for-migrations] Attempt 4 of 30
    [wait-for-migrations] Waiting 4 seconds before next attempt
    [wait-for-migrations] Attempt 5 of 30
    [wait-for-migrations] Waiting 8 seconds before next attempt
    [wait-for-migrations] Attempt 6 of 30
    [wait-for-migrations] Waiting 16 seconds before next attempt
    [wait-for-migrations] Attempt 7 of 30
    [wait-for-migrations] Waiting 30 seconds before next attempt
    [wait-for-migrations] Attempt 8 of 30
    [wait-for-migrations] Waiting 30 seconds before next attempt
    [wait-for-migrations] Attempt 9 of 30
    [wait-for-migrations] Waiting 30 seconds before next attempt
    [wait-for-migrations] Attempt 10 of 30
    [wait-for-migrations] Waiting 30 seconds before next attempt
    [wait-for-migrations] Attempt 11 of 30
    [wait-for-migrations] Waiting 30 seconds before next attempt
    [wait-for-migrations] Attempt 12 of 30
    [wait-for-migrations] Waiting 30 seconds before next attempt
    [wait-for-migrations] Attempt 13 of 30
    [wait-for-migrations] Waiting 30 seconds before next attempt
    [wait-for-migrations] Attempt 14 of 30
    [wait-for-migrations] Waiting 30 seconds before next attempt
    [wait-for-migrations] Attempt 15 of 30
    [wait-for-migrations] Waiting 30 seconds before next attempt
    [wait-for-migrations] Attempt 16 of 30
    [wait-for-migrations] Waiting 30 seconds before next attempt
    [wait-for-migrations] Attempt 17 of 30
    [wait-for-migrations] Waiting 30 seconds before next attempt
    [wait-for-migrations] Attempt 18 of 30
    [wait-for-migrations] Waiting 30 seconds before next attempt
    [wait-for-migrations] Attempt 19 of 30
    [wait-for-migrations] Waiting 30 seconds before next attempt
    [wait-for-migrations] Attempt 20 of 30
    [wait-for-migrations] Waiting 30 seconds before next attempt
    [wait-for-migrations] Attempt 21 of 30
    [wait-for-migrations] Waiting 30 seconds before next attempt
    [wait-for-migrations] Attempt 22 of 30
    [wait-for-migrations] Waiting 30 seconds before next attempt
    [wait-for-migrations] Attempt 23 of 30
    [wait-for-migrations] Waiting 30 seconds before next attempt
    [wait-for-migrations] Attempt 24 of 30
    [wait-for-migrations] Waiting 30 seconds before next attempt
    [wait-for-migrations] Attempt 25 of 30
    [wait-for-migrations] Waiting 30 seconds before next attempt
    [wait-for-migrations] Attempt 26 of 30
    [wait-for-migrations] Waiting 30 seconds before next attempt
    [wait-for-migrations] Attempt 27 of 30
    [wait-for-migrations] Waiting 30 seconds before next attempt
    [wait-for-migrations] Attempt 28 of 30
    [wait-for-migrations] Waiting 30 seconds before next attempt
    [wait-for-migrations] Attempt 29 of 30
    [wait-for-migrations] Waiting 30 seconds before next attempt
    [wait-for-migrations] Attempt 30 of 30
    [root@awx02 ~]# kubectl -n awx logs awx-548ccb647c-qv4bx -c awx-web
    [wait-for-migrations] Waiting for database migrations…
    [wait-for-migrations] Attempt 1 of 30

    [root@awx02 ~]# kubectl -n awx -c awx-web exec –stdin –tty awx-548ccb647c-qv4bx — /bin/bash
    bash-4.4$ awx-manage
    Traceback (most recent call last):
    File “/var/lib/awx/venv/awx/lib64/python3.8/site-packages/django/db/backends/base/base.py”, line 217, in ensure_connection
    self.connect()
    File “/var/lib/awx/venv/awx/lib64/python3.8/site-packages/django/db/backends/base/base.py”, line 195, in connect
    self.connection = self.get_new_connection(conn_params)
    File “/var/lib/awx/venv/awx/lib64/python3.8/site-packages/django/db/backends/postgresql/base.py”, line 178, in get_new_connection
    connection = Database.connect(**conn_params)
    File “/var/lib/awx/venv/awx/lib64/python3.8/site-packages/psycopg2/__init__.py”, line 126, in connect
    conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
    psycopg2.OperationalError: could not connect to server: No route to host
    Is the server running on host “awx-postgres” (10.42.0.16) and accepting
    TCP/IP connections on port 5432?

    The above exception was the direct cause of the following exception:

    Traceback (most recent call last):
    File “/usr/bin/awx-manage”, line 8, in
    sys.exit(manage())
    File “/var/lib/awx/venv/awx/lib64/python3.8/site-packages/awx/__init__.py”, line 155, in manage
    if (connection.pg_version // 10000) < 12:
    File "/var/lib/awx/venv/awx/lib64/python3.8/site-packages/django/db/__init__.py", line 28, in __getattr__
    return getattr(connections[DEFAULT_DB_ALIAS], item)
    File "/var/lib/awx/venv/awx/lib64/python3.8/site-packages/django/utils/functional.py", line 80, in __get__
    res = instance.__dict__[self.name] = self.func(instance)
    File "/var/lib/awx/venv/awx/lib64/python3.8/site-packages/django/db/backends/postgresql/base.py", line 282, in pg_version
    with self.temporary_connection():
    File "/usr/lib64/python3.8/contextlib.py", line 113, in __enter__
    return next(self.gen)
    File "/var/lib/awx/venv/awx/lib64/python3.8/site-packages/django/db/backends/base/base.py", line 593, in temporary_connection
    with self.cursor() as cursor:
    File "/var/lib/awx/venv/awx/lib64/python3.8/site-packages/django/db/backends/base/base.py", line 256, in cursor
    return self._cursor()
    File "/var/lib/awx/venv/awx/lib64/python3.8/site-packages/django/db/backends/base/base.py", line 233, in _cursor
    self.ensure_connection()
    File "/var/lib/awx/venv/awx/lib64/python3.8/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
    self.connect()
    File "/var/lib/awx/venv/awx/lib64/python3.8/site-packages/django/db/utils.py", line 89, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
    File "/var/lib/awx/venv/awx/lib64/python3.8/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
    self.connect()
    File "/var/lib/awx/venv/awx/lib64/python3.8/site-packages/django/db/backends/base/base.py", line 195, in connect
    self.connection = self.get_new_connection(conn_params)
    File "/var/lib/awx/venv/awx/lib64/python3.8/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
    connection = Database.connect(**conn_params)
    File "/var/lib/awx/venv/awx/lib64/python3.8/site-packages/psycopg2/__init__.py", line 126, in connect
    conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
    django.db.utils.OperationalError: could not connect to server: No route to host
    Is the server running on host "awx-postgres" (10.42.0.16) and accepting
    TCP/IP connections on port 5432?

    bash-4.4$ ping 10.42.0.16 -c3
    PING 10.42.0.16 (10.42.0.16) 56(84) bytes of data.
    64 bytes from 10.42.0.16: icmp_seq=1 ttl=64 time=0.263 ms
    64 bytes from 10.42.0.16: icmp_seq=2 ttl=64 time=0.206 ms
    64 bytes from 10.42.0.16: icmp_seq=3 ttl=64 time=0.141 ms

    — 10.42.0.16 ping statistics —
    3 packets transmitted, 3 received, 0% packet loss, time 2032ms
    rtt min/avg/max/mdev = 0.141/0.203/0.263/0.051 ms
    bash-4.4$ cat < /dev/tcp/10.42.0.16/5432
    bash: connect: No route to host
    bash: /dev/tcp/10.42.0.16/5432: No route to host
    bash-4.4$

    Any ideas how to fix it?

    • And postgres logs:
      [root@awx02 ~]# kubectl -n awx logs awx-postgres-0
      The files belonging to this database system will be owned by user “postgres”.
      This user must also own the server process.

      The database cluster will be initialized with locale “en_US.utf8”.
      The default database encoding has accordingly been set to “UTF8”.
      The default text search configuration will be set to “english”.

      Data page checksums are disabled.

      fixing permissions on existing directory /var/lib/postgresql/data/pgdata … ok
      creating subdirectories … ok
      selecting dynamic shared memory implementation … posix
      selecting default max_connections … 100
      selecting default shared_buffers … 128MB
      selecting default time zone … Etc/UTC
      creating configuration files … ok
      running bootstrap script … ok
      performing post-bootstrap initialization … ok
      syncing data to disk … ok

      Success. You can now start the database server using:

      pg_ctl -D /var/lib/postgresql/data/pgdata -l logfile start

      initdb: warning: enabling “trust” authentication for local connections
      You can change this by editing pg_hba.conf or using the option -A, or
      –auth-local and –auth-host, the next time you run initdb.
      waiting for server to start….2021-08-17 09:31:28.012 UTC [48] LOG: starting PostgreSQL 12.8 (Debian 12.8-1.pgdg100+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 8.3.0-6) 8.3.0, 64-bit
      2021-08-17 09:31:28.035 UTC [48] LOG: listening on Unix socket “/var/run/postgresql/.s.PGSQL.5432”
      2021-08-17 09:31:28.118 UTC [49] LOG: database system was shut down at 2021-08-17 09:31:26 UTC
      2021-08-17 09:31:28.138 UTC [48] LOG: database system is ready to accept connections
      done
      server started
      CREATE DATABASE

      /usr/local/bin/docker-entrypoint.sh: ignoring /docker-entrypoint-initdb.d/*

      2021-08-17 09:31:29.049 UTC [48] LOG: received fast shutdown request
      waiting for server to shut down….2021-08-17 09:31:29.141 UTC [48] LOG: aborting any active transactions
      2021-08-17 09:31:29.143 UTC [48] LOG: background worker “logical replication launcher” (PID 55) exited with exit code 1
      2021-08-17 09:31:29.144 UTC [50] LOG: shutting down
      2021-08-17 09:31:29.297 UTC [48] LOG: database system is shut down
      done
      server stopped

      PostgreSQL init process complete; ready for start up.

      2021-08-17 09:31:29.422 UTC [1] LOG: starting PostgreSQL 12.8 (Debian 12.8-1.pgdg100+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 8.3.0-6) 8.3.0, 64-bit
      2021-08-17 09:31:29.422 UTC [1] LOG: listening on IPv4 address “0.0.0.0”, port 5432
      2021-08-17 09:31:29.423 UTC [1] LOG: listening on IPv6 address “::”, port 5432
      2021-08-17 09:31:29.515 UTC [1] LOG: listening on Unix socket “/var/run/postgresql/.s.PGSQL.5432”
      2021-08-17 09:31:29.722 UTC [76] LOG: database system was shut down at 2021-08-17 09:31:29 UTC
      2021-08-17 09:31:29.833 UTC [1] LOG: database system is ready to accept connections

      • I installed it on a VM running Rocky Linux 8.4 and it is working.
        Make sure all resources are adequate: my installation would not work until I had 5vcpus, 8 GB ram, and 25GB disk space.
        Make sure you follow the instructions and you will get there.

  17. Hey @Josphat Mutai
    Its very nice writeup i have ever read for awx with k3s.
    Can you pls confirm whether these same steps will work on centos7 ?

      • Thank you its working..meanwhile i have noticed one big difference between the UI what you have shared in this blog and what is installed in my local.

        In my local i am seeing a UI with very less features like inventory scripts. scheduling job template feature is missing and its in dark mode.
        And the UI which you have shown is in light mode and it has inventory scripts and even the logo is different with wings.

        can you please tell me what i am doing wrong and how can i get the same UI which you have shown?

  18. I have noticed one big difference between the UI what you have shared in this blog and what is installed in my local.

    In my local i am seeing a UI with very less features like inventory scripts. scheduling job template feature is missing and its in dark mode.
    And the UI which you have shown is in light mode and it has inventory scripts and even the logo is different with wings.

    can you please tell me what i am doing wrong and how can i get the same UI which you have shown?

  19. need help please

    [root@awx ~]# kubectl apply -f awx-instance-deployment.yml -n awx
    awx.awx.ansible.com/awx created
    [root@awx ~]# kubectl get pods -l “app.kubernetes.io/managed-by=awx-operator” -n awx
    No resources found in awx namespace.
    [root@awx ~]# kubectl get pods -l “app.kubernetes.io/managed-by=awx-operator” -n awx
    NAME READY STATUS RESTARTS AGE
    awx-postgres-0 0/1 ContainerCreating 0 9s
    awx-548ccb647c-4r42n 0/4 Pending 0 2s
    [root@awx ~]# kubectl get pods -l “app.kubernetes.io/managed-by=awx-operator” -n awx
    NAME READY STATUS RESTARTS AGE
    awx-548ccb647c-4r42n 0/4 Init:0/1 0 9s
    awx-postgres-0 0/1 CrashLoopBackOff 1 16s
    [root@awx ~]# kubectl get pods -l “app.kubernetes.io/managed-by=awx-operator” -n awx
    NAME READY STATUS RESTARTS AGE
    awx-postgres-0 0/1 CrashLoopBackOff 1 22s
    awx-548ccb647c-4r42n 0/4 PodInitializing 0 15s
    [root@awx ~]# kubectl get pods -l “app.kubernetes.io/managed-by=awx-operator” -n awx
    NAME READY STATUS RESTARTS AGE
    awx-postgres-0 0/1 CrashLoopBackOff 1 25s
    awx-548ccb647c-4r42n 0/4 PodInitializing 0 18s
    [root@awx ~]# kubectl get pods -l “app.kubernetes.io/managed-by=awx-operator” -n awx
    NAME READY STATUS RESTARTS AGE
    awx-postgres-0 0/1 CrashLoopBackOff 1 27s
    awx-548ccb647c-4r42n 0/4 PodInitializing 0 20s
    [root@awx ~]# kubectl get pvc
    No resources found in default namespace.

  20. Hi There,
    Need some help please –> followed the above steps
    I am very new to linux
    1) Disabled firewall / SELINUX=permissive
    2) Installed: k3s-selinux-0.3-0.el8.noarch
    3) K3S service running – k3s.service – Lightweight Kubernetes
    4)kubectl get nodes
    NAME STATUS ROLES AGE VERSION
    rh1 Ready control-plane,master 2m11s v1.21.4+k3s1
    5)kubectl version –short
    Client Version: v1.21.4+k3s1
    Server Version: v1.21.4+k3s1
    6)kubectl apply -f (7 things created)
    7)kubectl get pods
    NAME READY STATUS RESTARTS AGE
    awx-operator-69c646c48f-2t7qq 1/1 Running 0 34s
    8)kubectl create ns awx
    namespace/awx created
    9)kubectl get pvc -n awx
    NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
    public-static-data-pvc Pending local-path 20s
    10)kubectl apply -f awx-instance-deployment.yml -n awx
    awx.awx.ansible.com/awx created
    11)kubectl get pods -l “app.kubernetes.io/managed-by=awx-operator” -n awx
    NAME READY STATUS RESTARTS AGE
    awx-postgres-0 1/1 Running 0 4m24s
    awx-548ccb647c-9szwr 4/4 Running 0 4m16s

    Now when i run # kubectl get pvc
    ######################
    No resources found in default namespace.
    ######################

    Really appreciate your help . Thanks

  21. kubectl get pvc — returns No resources found in default namespace.. Seems like the pvc’s aren’t being setup. Any advice how to proceed? (The web interface is there, but I can’t seem to get a shell to setup a local source for playbooks).

    • Hello,

      Check the updated version of the article. Operator deployment process had been updated in the upstream project documentation. We modified our article to match official doc.

      Let us know if you encounter any issues.

  22. When I try do ‘make deploy’ I have error:
    Warning Failed 4m15s (x4 over 5m37s) kubelet Failed to pull image ” quay.io/ansible/awx-operator:latest”: rpc error: code = NotFound desc = failed to pull and unpack image “quay.io/ansible/awx-operator:latest”: failed to resolve reference ” quay.io/ansible/awx-operator:latest”: quay.io/ansible/awx-operator:latest: not found
    Normal BackOff 34s (x21 over 5m37s) kubelet Back-off pulling image “quay.io/ansible/awx-operator:latest”

  23. I have a message

    There are no available playbook directories in /var/lib/awx/projects. Either that directory is empty, or all of the contents are already assigned to other projects. Create a new directory there and make sure the playbook files can be read by the “awx” system user, or have AWX directly retrieve your playbooks from source control using the Source Control Type option above.

    What should I do?

  24. how do I verify if ansible is installed and the version on the container?
    I was trying to launch the archive module through AWX GUI but I was getting an error, not sure if that error has something to do with the ansible version.

    ERROR! couldn’t resolve module/action ‘archive’. This often indicates a misspelling, missing collection, or incorrect module path.

  25. # kubectl get pvc -n awx
    NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
    postgres-awx-postgres-0 Bound pvc-26b1a363-3948-449c-b66c-a48d9ee63acc 8Gi RWO local-path 23m
    public-static-data-pvc Bound pvc-34e576a6-fb29-463b-a284-514a803a0dbe 5Gi RWO local-path 25m
    awx-projects-claim Bound pvc-6984b3c1-3212-4d3c-9f13-453c5a12dba9 8Gi RWO local-path 23m

    # kubectl get pods -l “app.kubernetes.io/managed-by=awx-operator” -n awx
    NAME READY STATUS RESTARTS AGE
    awx-postgres-0 1/1 Running 1 25m
    awx-6846fd65b5-m86m6 4/4 Running 4 25m

    # kubectl get service -n awx
    NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
    awx-operator-controller-manager-metrics-service ClusterIP 10.43.30.166 8443/TCP 29m
    awx-postgres ClusterIP None 5432/TCP 25m
    awx-service NodePort 10.43.110.125 80:32493/TCP 25m

    Everything appears to be working on the back end but the page still comes up as a blank white page. I’ve got 4vcpu and 8GB RAM (4668.4 free)

  26. Hello everyone
    I have a question, when I launch the command kubectl get pods the awx pod stays in pending status. What could be the problem?

    NAME READY STATUS RESTARTS AGE
    awx-6846fd65b5-zdbgr 0/4 Pending 0 23m

  27. I have installed AWX in my Oracle Linux 8 but I can´t access with http://IP:30604, all my browsers fail to connect.
    Can you deploy AWX using HTTPS ?

    ====
    [sysadmin@awx ~]$ kubectl get service -n awx
    NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
    awx-operator-controller-.. ClusterIP 10.43.136.88 8443/TCP 3h34m
    awx-postgres ClusterIP None 5432/TCP 3h26m
    awx-service NodePort 10.43.131.255 80:30604/TCP 3h26m

    ====
    [sysadmin@awx ~]$ kubectl get pods
    NAME READY STATUS RESTARTS AGE
    awx-operator-controller-man… 2/2 Running 0 4h57m
    awx-postgres-0 1/1 Running 0 4h49m
    awx-6846fd65b5-c568d 4/4 Running 0 4h48m

  28. Resolved blank webpage issue post installation (installed using this doc)

    My system conf:
    OS: CentOS 8, CPU: 4, Mem: 8GB, Hypervisor: VMware workstation 16
    I installed following the steps provided in this page, but webpage was blank. In the awx-web logs I could see below error:

    “/var/lib/awx/public/static/js/2.53c634ac.chunk.js” failed (2: No such file or directory), client: 127.0.0.1, server: _, request: “GET /static/js/2.53c634ac.chunk.js HTTP/1.1”, host: “10.0.40.62”, referrer: “http://10.0.40.62/”

    I checked and could see /var/lib/awx folder is not there rather below is there:
    /var/lib/rancher/k3s/agent/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/100/fs/var/lib/awx/public
    and it had all the files.

    I followed below simple steps to get the path corrected:
    #export NAMESPACE=awx
    #make undeploy
    #kubectl create ns ${NAMESPACE}
    #kubectl config set-context –current –namespace=$NAMESPACE
    #make deploy
    #kubectl apply -f public-static-pvc.yaml -n awx
    #sed -i ‘s#/var/lib/awx/public#/var/lib/rancher/k3s/agent/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/100/fs/var/lib/awx/public#’ awx-instance-deployment.yml
    #kubectl apply -f awx-instance-deployment.yml -n awx

    Instance will come up again. Check for new port and that’s it!!

    Hope it helps.

  29. Very nice guide, but I also ran into the black gui page issue.

    In my case I checked logs: kubectl -n awx logs awx-6dcb9cb747-jxcgb -c awx-web

    2022/01/08 20:27:14 [error] 53#0: *5 open() “/var/lib/awx/public/static/js/main.7b1f208d.chunk.js” failed (2: No such file or directory), client: 10.42.0.1, server: _, request: “GET /static/js/main.7b1f208d.chunk.js HTTP/1.1”, host: “192.168.9.137:30038”, referrer: “http://192.168.9.137:30038/”
    10.42.0.1 – – [08/Jan/2022:20:27:14 +0000] “GET /static/js/main.7b1f208d.chunk.js HTTP/1.1” 404 564 “http://192.168.9.137:30038/” “Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/97.0.4692.71 Safari/537.36 Edg/97.0.1072.55” “-”

    So I did a find on “main.7b1f208d.chunk.js” and found under /var/lib/rancher/k3s/agent/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/99/fs/var/lib/awx/public

    I then followed Manash Sharma’s undeploy and deploy after making the path changes.

    This worked for Me !!!

  30. As I still fully don’t understand AWX and especially not Kubernettes, so would appreciate if the author could shed some light on why this is happening.

    Thanks.

  31. The install is stuck in:

    NAME READY STATUS RESTARTS AGE
    awx-operator-controller-manager-6795f9f5f5-d5d2m 0/2 ContainerCreating 0 15h

  32. Hello, I installed in Rocky Linux 8.5 – Vmware – 8G RAM – 4 vcpu – 30G HDD

    I have the first problem when I check:

    kubectl get pods -l “app.kubernetes.io/managed-by=awx-operator” -n awx
    NAME READY STATUS RESTARTS AGE
    awx-postgres-0 1/1 Running 1 (3m8s ago) 2d14h

    -> Don’t see: awx-75698588d6-qz2gf

    And the second problem:

    kubectl get pvc -n awx
    NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
    NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
    public-static-data-pvc Pending local-path 2d14h
    postgres-awx-postgres-0 Bound pvc-606b0020-8693-4fce-8463-de30ff541b1a 8Gi RWO local-path 2d14h
    awx-projects-claim Pending local-path 2d14h

    -> public-static-data-pvc and awx-projects-claim IS PENDING

    Please help me, thank you so much!

  33. I am getting below error.

    # kubectl get pods
    NAME READY STATUS RESTARTS AGE
    awx-5979c454c4-ncdfr 3/4 CrashLoopBackOff 6 (3m3s ago) 17m
    awx-operator-controller-manager-7bf76b4c4c-jdjxs 2/2 Running 0 21m
    awx-postgres-0 1/1 Running 0 19m

    This is failed at awx-ee

    Logs are as below.

    kubectl logs awx-5979c454c4-ncdfr -c awx-ee
    panic: qtls.ClientHelloInfo doesn’t match

    goroutine 1 [running]:
    github.com/marten-seemann/qtls-go1-15.init.0()
    /root/go/pkg/mod/github.com/marten-seemann/[email protected]/unsafe.go:20 +0x132

    I have tried with multiple version but getting same error

  34. mine is a bit different to that . I amanged to get it all working the uRL is up etc tec.
    I am completely new to this
    I am unable to create manual scm project on new project on ansible awx

    Error:
    Manual SCM (Source Control Credential Type) project on new Project on Ansible AWX:
    There are no available playbook directories in /var/lib/awx/projects. Either that directory is empty, or all of the contents are already assigned to other projects. Create a new directory there and make sure the playbook files can be read by the “awx” system user, or have AWX directly retrieve your playbooks from source control using the Source Control Type option above.

    Where do I locate this path: I am unable to locate /var/lib/awx/projects
    Just cant see it anywhere

  35. Hi all
    I have the following error while creating a project and selecting “Manual Type”. i receive this error message and i cannot access to the local directory “/var/lib/awx/project”

    Warning alert:WARNING:
    There are no available playbook directories in /var/lib/awx/projects. Either that directory is empty, or all of the contents are already assigned to other projects. Create a new directory there and make sure the playbook files can be read by the “awx” system user, or have AWX directly retrieve your playbooks from source control using the Source Control Type option above.

    have any on solved this problem ??

  36. Using both Rocky Linux 8.5 and CentOS 8.5, I was running into a problem after installing the AWX Operator. When trying to install AWX, it would spin up the postgres container but never spin up the 4 main AWX containers/pods.

    I found this issue posted in the awx-operator Github repo:

    https://github.com/ansible/awx-operator/issues/814

    I changed nodeport to clusterip and it did continue and spin up the AWX bits …. but I don’t know how to connect to it as no mapped port is listed in the service output. For now, I can use an SSH port forward to the K3S host to get to the K3S cluster IP.

  37. Hi David,

    I had same issue, only awx-postgres-0 pod was in running stage, without 4x AWX main pods.

    1. Regarding Github issue and changed value into “loadbalancer”:
    service_type: loadbalancer

    2. Also used pined k3s version with this command:
    curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION=v1.21.3+k3s1 sh –

    And now everything is working as expected (on Rocky Linux 8.5)
    Doesnt work on Centos 7.9

  38. Hi all,

    how to update a running awx instance to the latest awx version?
    The running instance was installed a few months ago with this article.

    Thanks!

  39. Hi!
    I have two questions:
    1. In what directory should I run git clone and which user should run this command (root?)
    2. After kubectl apply -f awx-instance-deployment.yml -n awx
    awx.awx.ansible.com/awx created
    it says:
    awx-postgres-0 error image pull
    I run # kubectl describe pod/awx-postgres-0
    and see the following:
    Normal Scheduled 117s default-scheduler Successfully assigned awx/awx-postgres-0 to ansible-awx001
    Normal Pulling 117s kubelet Pulling image “quay.io/centos/centos:stream8”
    Normal Pulled 106s kubelet Successfully pulled image “quay.io/centos/centos:stream8” in 11.079940612s
    Normal Created 105s kubelet Created container database-check
    Normal Started 105s kubelet Started container database-check
    Normal BackOff 24s (x4 over 100s) kubelet Back-off pulling image “postgres:12”
    Warning Failed 24s (x4 over 100s) kubelet Error: ImagePullBackOff
    Normal Pulling 10s (x4 over 105s) kubelet Pulling image “postgres:12”
    Warning Failed 6s (x4 over 100s) kubelet Failed to pull image “postgres:12”: rpc error: code = Unknown desc = failed to pull and unpack image “docker.io/library/postgres:12”: failed to copy: httpReadSeeker: failed open: unexpected status code https://registry-1.docker.io/v2/library/postgres/manifests/sha256:f2765d6a2a6459578274645b4d801345060322da2ba855af3d84878be28fe923: 429 Too Many Requests – Server message: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit
    Warning Failed 6s (x4 over 100s) kubelet Error: ErrImagePull

  40. Very nice guide for a noob like me.
    I having an issue logging into AWX Web GUI

    user: admin
    password: output of the below command

    kubectl -n awx get secret awx-admin-password -o go-template='{{range $k,$v := .data}}{{printf “%s: ” $k}}{{if not $v}}{{$v}}{{else}}{{$v | base64decode}}{{end}}{{“\n”}}{{end}}’

    not sure what went wrong?

  41. Please help me i am running AWX on the VM on ESX/VMware and got the following error.
    Fatal glibc error: CPU does not support x86-64-v2

    i just try more than 100 on different version of Linux but all the same

  42. Hello,

    i could install AWX with your tutorial, thank you!
    how can we configure SSL Certificates and is there a possibility to reach trough https instead of the specific port?

    Thank you in advance for any help.

  43. How can I install python libraries in node controller? I’m trying to start a playbook in awx and I get this message:

    {
    “msg”: “The ipv4 filter requires python’s netaddr be installed on the ansible controller”,
    “_ansible_no_log”: false
    }

    Even running a pip install netaddr it installs but doesn’t recognize

  44. Hello,
    first congrats for this tutotial !
    I have succed running the AWX frontend but i faced a new issue, each time i try to run a kubectl command, i have this kind of message :
    Error from server: Get “https://10.128.x.x:10250/containerLogs/awx/awx-6b8f4b758-5w7jj/redis”: net/http: TLS handshake timeout

    Does someone got an idea ?

    BR

  45. Hello,
    Thank you for the detailed installation guide. When you do your install do you
    – do all work as root?
    – do all work as your user?
    – do all work on a service account that you have created?

    Thank you!

  46. Hello,

    I am having issue with some pods ,

    my configuration is : OS: Rocky Linux 8.6, CPU: 4, Mem: 8GB, Hypervisor:KVM (host is Centos Stream 8, 8 CPU, 32GB)

    root@AWX awx-operator]# kubectl get pods -n awx
    NAME READY STATUS RESTARTS AGE
    awx-postgres-13-0 1/1 Running 2 (4m38s ago) 30m
    awx-operator-controller-manager-5c87bcdb4b-7m44b 2/2 Running 4 (4m38s ago) 33m
    awx-554fbd6db9-hzv4p 0/4 Init:CrashLoopBackOff 18 (82s ago) 29m

    The awx pod keeps crashing …

    Please advise, thank you!

  47. Greate write-up!

    For those who are having issues running it on a VM, make sure your CPU has virtualization capability, else it will fail and you will experience CrashInitErrorLoopback or something.

    I successfully run the steps, however, I don’t have listening 30080 and the IP address is different. My IP address is 10.11.x.x but `kubectl get service -n awx` I am getting 10.43.x.x.

    Is there a way that I can have this listening to my eth0? (10.11.x.x)

  48. Hello All,

    I have built my cluster with k3s using your guide and that’s a great guide honestly very great for beginning on Kubernetes as a first project.

    my awx-deploy.yaml
    >>>
    apiVersion: awx.ansible.com/v1beta1
    kind: AWX
    metadata:
    name: awx
    spec:
    service_type: nodeport
    projects_persistence: true
    hostname: awxsbox.example.com
    projects_storage_access_mode: ReadWriteOnce
    web_extra_volume_mounts: |
    – name: static-data
    mountPath: /var/lib/projects
    extra_volumes: |
    – name: static-data
    persistentVolumeClaim:
    claimName: static-data-pvc
    <<>>
    apiVersion: networking.k8s.io/v1
    kind: Ingress
    metadata:
    namespace: awx
    name: awx-ingress
    annotations:
    traefik.ingress.kubernetes.io/router.entrypoints: web
    spec:
    rules:
    – host: awxsbox.example.com
    http:
    paths:
    – path: /
    pathType: Prefix
    backend:
    service:
    name: awx-service
    port:
    number: 80
    << Not worked for me.

    Created a new USER still the same can’t login.

    This is what I see in the logs.

    >>>>>>>>>>
    10.42.0.8 – – [25/May/2023:15:50:01 +0000] “GET /api/login/ HTTP/1.1” 200 5710 “https://awxsbox.example.com/” “Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:109.0) Gecko/20100101 Firefox/113.0” “10.42.0.1”

    2023-05-25 15:50:01,483 WARNING [900725a931ed464a9415b6f5520f9169] django.security.csrf Forbidden (Origin checking failed – https://awxsbox.example.com does not match any trusted origins.): /api/login/

    10.42.0.8 – – [25/May/2023:15:50:01 +0000] “POST /api/login/ HTTP/1.1” 403 1019 “https://awxsbox.example.com/” “Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:109.0) Gecko/20100101 Firefox/113.0” “10.42.0.1”
    [pid: 36|app: 0|req: 5/23] 10.42.0.8 () {74 vars in 1856 bytes} [Thu May 25 15:50:01 2023] POST /api/login/ => generated 1019 bytes in 46 msecs (HTTP/1.1 403) 7 headers in 276 bytes (1 switches on core 0)
    <<<<<<<<<<

    How can I solve this issue please I need some help here.

    Thank you

  49. Hi all,
    have any of you tried to automate configuration of a Cisco routers ?
    for me it is not working because of what i think is a missing ansible galaxy for cisco devices

    have anyone installed these packages or automated cisoc routers with this version of AWX

    thanks in adavance

    • Do you mean setting a port for the AWX service? You can use an Ingress controller and use static ports. Loadbalancer can also be an option if you wnat to use static ports decalred in the container service.

  50. Hi All,
    Can some one please let me know how to install winrm package inside the container awx-task or where can we find awx password?

    error:
    ERROR: Could not install packages due to an OSError: [Errno 13] Permission denied: ‘/var/lib/awx/.local/lib’
    Check the permissions.

  51. I’ve received a lot of help from you. Thank you.

    One problem is left, the company proxy environment I set up during the installation process will not be removed from the container.

    Which file should I modify?

LEAVE A REPLY

Please enter your comment!
Please enter your name here