Automation

How To Install Ansible AWX on Ubuntu 22.04|20.04|18.04

Josphat Mutai Updated Aug 24, 2023 · 10 min read

Welcome to today’s guide on how to install Ansible AWX on Ubuntu 22.04|20.04|18.04 with Nginx Reverse Proxy and optional Let’s Encrypt SSL Certificate. Ansible AWX is an open source tool which provides a web-based user interface, REST API, and task engine for easy and collaborative management of Ansible Playbooks and Inventories.

Original content from computingforgeeks.com - post 35629

AWX allows you to centrally manage Ansible playbooks, inventories, Secrets, and scheduled jobs from a web interface. It is easy to install AWX on Ubuntu Linux system. Use the steps shared below to install and configure Ansible AWX on Ubuntu Linux server.

Starting in version 18.0 of AWX, the recommended installation method is via AWX Operator. As the operator installation method requires a Kubernetes Cluster, we will perform a single node Kubernetes installation on Ubuntu Linux using k3s.

Setup minimum requirements

  • Ubuntu 22.04|20.04|18.04 LTS Server
  • At least 16GB of RAM – More is better
  • 4vcpus – Pump more if you have
  • 20GB free disk storage
  • root or user with sudo for ssh

Step 1: Update Ubuntu system

Update and upgrade your system

sudo apt update && sudo apt -y upgrade
[ -f /var/run/reboot-required ] && sudo reboot -f

Step 2: Install Single Node k3s Kubernetes

We will deploy a single node kubernetes using k3s lightweight tool. K3s is a certified Kubernetes distribution designed for production workloads in unattended, resource-constrained environments. The good thing with k3s is that you can add more Worker nodes at later stage if need arises.

K3s provides an installation script that is a convenient way to install it as a service on systemd or openrc based systems

Let’s run the following command to install K3s on our Ubuntu system:

curl -sfL https://get.k3s.io | sudo bash -

Installation process output:

[INFO]  Finding release for channel stable
[INFO]  Using v1.27.4+k3s1 as release
[INFO]  Downloading hash https://github.com/k3s-io/k3s/releases/download/v1.27.4+k3s1/sha256sum-amd64.txt
[INFO]  Downloading binary https://github.com/k3s-io/k3s/releases/download/v1.27.4+k3s1/k3s
[INFO]  Verifying binary download
[INFO]  Installing k3s to /usr/local/bin/k3s
[INFO]  Skipping installation of SELinux RPM
[INFO]  Creating /usr/local/bin/kubectl symlink to k3s
[INFO]  Creating /usr/local/bin/crictl symlink to k3s
[INFO]  Creating /usr/local/bin/ctr symlink to k3s
[INFO]  Creating killall script /usr/local/bin/k3s-killall.sh
[INFO]  Creating uninstall script /usr/local/bin/k3s-uninstall.sh
[INFO]  env: Creating environment file /etc/systemd/system/k3s.service.env
[INFO]  systemd: Creating service file /etc/systemd/system/k3s.service
[INFO]  systemd: Enabling k3s unit
Created symlink /etc/systemd/system/multi-user.target.wants/k3s.service → /etc/systemd/system/k3s.service.
[INFO]  systemd: Starting k3s

The kubeconfig file stored at /etc/rancher/k3s/k3s.yaml is used to configure access to the Kubernetes cluster. You can set permissions and set KUBECONFIG for global access.

sudo chmod 644 /etc/rancher/k3s/k3s.yaml
export KUBECONFIG=/etc/rancher/k3s/k3s.yaml
kubectl get pods --all-namespaces

You can as well copy /etc/rancher/k3s/k3s.yaml on to a machine located outside the cluster as ~/.kube/config.

mkdir ~/.kube
sudo cp /etc/rancher/k3s/k3s.yaml ~/.kube/config

Validate K3s installation:

The next step is to validate our installation of K3s using kubectl command which was installed and configured by installer script.

$ kubectl get nodes
NAME    STATUS   ROLES                  AGE   VERSION
jammy   Ready    control-plane,master   81s   v1.27.4+k3s1

You can also confirm Kubernetes version deployed using the following command:

$ kubectl version --short
Client Version: v1.27.4+k3s1
Kustomize Version: v5.0.1
Server Version: v1.27.4+k3s1

The K3s service will be configured to automatically restart after node reboots or if the process crashes or is killed.

Step 3: Deploy AWX Operator on Kubernetes

This Kubernetes Operator has to be deployed in your Kubernetes cluster, which in our case is powered by K3s. The operator we’ll deploy can manage one or more AWX instances in any namespace.

Install git and make tools:

sudo apt update
sudo apt install git build-essential -y

Clone operator deployment code:

$ git clone https://github.com/ansible/awx-operator.git
Cloning into 'awx-operator'...
remote: Enumerating objects: 8773, done.
remote: Counting objects: 100% (1388/1388), done.
remote: Compressing objects: 100% (201/201), done.
remote: Total 8773 (delta 1223), reused 1275 (delta 1184), pack-reused 7385
Receiving objects: 100% (8773/8773), 2.35 MiB | 17.69 MiB/s, done.
Resolving deltas: 100% (5033/5033), done.

Next we create a namespace where operator will be deployed. We’ll name this awx:

export NAMESPACE=awx
kubectl create ns ${NAMESPACE}

Set current context to value set in NAMESPACE variable:

# kubectl config set-context --current --namespace=$NAMESPACE 
Context "default" modified.

Switch to awx-operator directory:

cd awx-operator/

Save the latest version from AWX Operator releases as RELEASE_TAG variable then checkout to the branch using git.

sudo apt install curl jq -y
RELEASE_TAG=`curl -s https://api.github.com/repos/ansible/awx-operator/releases/latest | grep tag_name | cut -d '"' -f 4`
echo $RELEASE_TAG
git checkout $RELEASE_TAG

Deploy AWX Operator into your cluster:

export NAMESPACE=awx
make deploy

Command output:

namespace/awx configured
customresourcedefinition.apiextensions.k8s.io/awxbackups.awx.ansible.com created
customresourcedefinition.apiextensions.k8s.io/awxrestores.awx.ansible.com created
customresourcedefinition.apiextensions.k8s.io/awxs.awx.ansible.com created
serviceaccount/awx-operator-controller-manager created
role.rbac.authorization.k8s.io/awx-operator-awx-manager-role created
role.rbac.authorization.k8s.io/awx-operator-leader-election-role created
clusterrole.rbac.authorization.k8s.io/awx-operator-metrics-reader created
clusterrole.rbac.authorization.k8s.io/awx-operator-proxy-role created
rolebinding.rbac.authorization.k8s.io/awx-operator-awx-manager-rolebinding created
rolebinding.rbac.authorization.k8s.io/awx-operator-leader-election-rolebinding created
clusterrolebinding.rbac.authorization.k8s.io/awx-operator-proxy-rolebinding created
configmap/awx-operator-awx-manager-config created
service/awx-operator-controller-manager-metrics-service created
deployment.apps/awx-operator-controller-manager created

Wait a few minutes and awx-operator should be running:

# kubectl get pods
NAME                                               READY   STATUS    RESTARTS   AGE
awx-operator-controller-manager-68d787cfbd-z75n4   2/2     Running   0          40s

How To Uninstall AWX Operator (Don’t run this unless you’re sure it uninstalls!“

You can always remove the operator and all associated CRDs by running the command below:

# export NAMESPACE=awx
# make undeploy
/root/awx-operator/bin/kustomize build config/default | kubectl delete -f -
namespace "awx" deleted
customresourcedefinition.apiextensions.k8s.io "awxbackups.awx.ansible.com" deleted
customresourcedefinition.apiextensions.k8s.io "awxrestores.awx.ansible.com" deleted
customresourcedefinition.apiextensions.k8s.io "awxs.awx.ansible.com" deleted
serviceaccount "awx-operator-controller-manager" deleted
role.rbac.authorization.k8s.io "awx-operator-leader-election-role" deleted
role.rbac.authorization.k8s.io "awx-operator-manager-role" deleted
clusterrole.rbac.authorization.k8s.io "awx-operator-metrics-reader" deleted
clusterrole.rbac.authorization.k8s.io "awx-operator-proxy-role" deleted
rolebinding.rbac.authorization.k8s.io "awx-operator-leader-election-rolebinding" deleted
rolebinding.rbac.authorization.k8s.io "awx-operator-manager-rolebinding" deleted
clusterrolebinding.rbac.authorization.k8s.io "awx-operator-proxy-rolebinding" deleted
configmap "awx-operator-manager-config" deleted
service "awx-operator-controller-manager-metrics-service" deleted
deployment.apps "awx-operator-controller-manager" deleted

Step 4: Install Ansible AWX on Ubuntu using Operator

Create Static data PVC – Ref AWX data persistence:

cat <<EOF | kubectl create -f -
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: static-data-pvc
  namespace: awx
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: local-path
  resources:
    requests:
      storage: 5Gi
EOF

PVC won’t be bound until the pod that uses it is created.

# kubectl get pvc -n awx
NAME                     STATUS    VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS   AGE
static-data-pvc          Pending                                      local-path     43s

Let’s now create AWX deployment file with basic information about what is installed:

vim awx-deploy.yml

Paste below contents into the file:

---
apiVersion: awx.ansible.com/v1beta1
kind: AWX
metadata:
  name: awx
spec:
  service_type: nodeport
  projects_persistence: true
  projects_storage_access_mode: ReadWriteOnce
  web_extra_volume_mounts: |
    - name: static-data
      mountPath: /var/lib/projects
  extra_volumes: |
    - name: static-data
      persistentVolumeClaim:
        claimName: static-data-pvc

We have defined resource name as awx and service type as nodeport to enable us access AWX from the Node IP address and given port. We also added extra PV mount on the web server pod.

Apply configuration manifest file:

$ kubectl apply -f awx-deploy.yml
awx.awx.ansible.com/awx created

Wait a few minutes then check AWX instance deployed:

watch kubectl get pods -l "app.kubernetes.io/managed-by=awx-operator"

Once running you can confirm with the following commands: 

$ kubectl get pods -l "app.kubernetes.io/managed-by=awx-operator"
NAME                        READY   STATUS    RESTARTS   AGE
awx-postgres-13-0           1/1     Running   0          7m58s
awx-task-6874d8656b-v75s4   4/4     Running   0          7m10s
awx-web-6c797b8657-sllzt    3/3     Running   0          5m52s

You can track the installation process at the operator pod logs:

kubectl logs -f deployments/awx-operator-controller-manager -c awx-manager

Fixing Postgres Pod in CrashLoopBackOff state (Skip if you don’t have this error message)

Check status of PostgreSQL Pod:

$ kubectl get pods -l "app.kubernetes.io/managed-by=awx-operator"
NAME                   READY   STATUS             RESTARTS   AGE
awx-postgres-0         0/1     CrashLoopBackOff   4          2m38s
awx-66c64f8d67-qvfk9   3/4     ImagePullBackOff   0          2m27s

Check logs of the Pod:

$ kubectl logs awx-postgres-xxx
mkdir: cannot create directory ‘/var/lib/postgresql/data’: Permission denied

It means the Postgres pod cannot write to the persistent volume directory inside /var/lib/rancher/k3s/storage/:

# ls -lh /var/lib/rancher/k3s/storage/ | grep awx-postgres-0
total 0
drwx------ 3 root root 4.0K Aug 24 09:18 pvc-b2acb2d0-6d2f-457b-a40c-390cff5ec6d2_default_postgres-awx-postgres-0

Try setting the directory mode to 777

# chmod -R 777  /var/lib/rancher/k3s/storage/*
# kubectl delete pods -l "app.kubernetes.io/managed-by=awx-operator"
pod "awx-66c64f8d67-qvfk9" deleted
pod "awx-postgres-0" deleted

The Postgres container pod should come up in few seconds:

# kubectl get pods -l "app.kubernetes.io/managed-by=awx-operator"
NAME                   READY   STATUS             RESTARTS   AGE
awx-postgres-0         1/1     Running            0          42s
awx-66c64f8d67-4qmh5   4/4     Running            0          42s

If you experience any issues with the Pods starting check deployment logs:

kubectl logs -f deployments/awx-operator-controller-manager -c awx-manager

Data Persistence

The database data will be persistent as they are stored in a persistent volume:

$ kubectl get pvc
NAME                            STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
postgres-13-awx-postgres-13-0   Bound    pvc-1ff27783-a9d7-46c0-8160-803615784267   8Gi        RWO            local-path     20m
static-data-pvc                 Bound    pvc-cac663a8-0f9d-4d8e-96a2-4821576fab97   5Gi        RWO            local-path     21m
awx-projects-claim              Bound    pvc-e49c3bd8-79d4-4640-acd1-990578ca27c1   8Gi        RWO            local-path     19m

Volumes are created using local-path-provisioner and host path 

$ ls -1 /var/lib/rancher/k3s/storage/
pvc-1ff27783-a9d7-46c0-8160-803615784267_awx_postgres-13-awx-postgres-13-0
pvc-cac663a8-0f9d-4d8e-96a2-4821576fab97_awx_static-data-pvc
pvc-e49c3bd8-79d4-4640-acd1-990578ca27c1_awx_awx-projects-claim

Checking AWX Container’s logs

The awx-xxx-yyy pod will have four containers, namely:

  • redis
  • awx-web
  • awx-task
  • awx-ee

Get deployments:

$ kubectl get deploy
NAME                              READY   UP-TO-DATE   AVAILABLE   AGE
awx-operator-controller-manager   1/1     1            1           22m
awx-task                          1/1     1            1           20m
awx-web                           1/1     1            1           19m

List Pods:

$ kubectl get pod
NAME                                               READY   STATUS    RESTARTS   AGE
awx-operator-controller-manager-74889d49c8-kch8t   2/2     Running   0          36m
awx-postgres-13-0                                  1/1     Running   0          34m
awx-task-6874d8656b-v75s4                          4/4     Running   0          33m
awx-web-6c797b8657-sllzt                           3/3     Running   0          32m

List containers in a Pod

$ kubectl get pods  awx-web-6c797b8657-sllzt -o jsonpath='{range .spec.containers[*]}{.name}{"\n"}{end}'
redis
awx-web
awx-rsyslog

$ kubectl get pods  awx-postgres-13-0 -o jsonpath='{range .spec.containers[*]}{.name}{"\n"}{end}'
postgres

$ kubectl get pods  awx-task-6874d8656b-v75s4  -o jsonpath='{range .spec.containers[*]}{.name}{"\n"}{end}'
redis
awx-task
awx-ee
awx-rsyslog

$ kubectl get pods  awx-operator-controller-manager-74889d49c8-kch8t  -o jsonpath='{range .spec.containers[*]}{.name}{"\n"}{end}'
kube-rbac-proxy
awx-manager

To check logs in deployment.

# kubectl -n awx  logs deploy/awx-web
Defaulted container "redis" out of: redis, awx-web, awx-rsyslog, init (init), init-projects (init)
1:C 16 Jun 2023 10:26:05.653 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
1:C 16 Jun 2023 10:26:05.653 # Redis version=7.0.11, bits=64, commit=00000000, modified=0, pid=1, just started
1:C 16 Jun 2023 10:26:05.653 # Configuration loaded
1:M 16 Jun 2023 10:26:05.654 * monotonic clock: POSIX clock_gettime
1:M 16 Jun 2023 10:26:05.655 * Running mode=standalone, port=0.
1:M 16 Jun 2023 10:26:05.655 # Server initialized
1:M 16 Jun 2023 10:26:05.655 * The server is now ready to accept connections at /var/run/redis/redis.sock
1:M 16 Jun 2023 10:31:06.022 * 100 changes in 300 seconds. Saving...
1:M 16 Jun 2023 10:31:06.023 * Background saving started by pid 21
21:C 16 Jun 2023 10:31:06.030 * DB saved on disk
21:C 16 Jun 2023 10:31:06.031 * Fork CoW for RDB: current 0 MB, peak 0 MB, average 0 MB
1:M 16 Jun 2023 10:31:06.123 * Background saving terminated with success
1:M 16 Jun 2023 10:36:07.077 * 100 changes in 300 seconds. Saving...
1:M 16 Jun 2023 10:36:07.077 * Background saving started by pid 22
22:C 16 Jun 2023 10:36:07.088 * DB saved on disk
22:C 16 Jun 2023 10:36:07.089 * Fork CoW for RDB: current 0 MB, peak 0 MB, average 0 MB
1:M 16 Jun 2023 10:36:07.178 * Background saving terminated with success
1:M 16 Jun 2023 10:41:08.027 * 100 changes in 300 seconds. Saving...
1:M 16 Jun 2023 10:41:08.028 * Background saving started by pid 23
23:C 16 Jun 2023 10:41:08.038 * DB saved on disk
23:C 16 Jun 2023 10:41:08.039 * Fork CoW for RDB: current 0 MB, peak 0 MB, average 0 MB
1:M 16 Jun 2023 10:41:08.129 * Background saving terminated with success

You’ll need to provide container name after the pod:

kubectl -n awx  logs deploy/awx-web -c redis
kubectl -n awx  logs deploy/awx-web -c awx-web
kubectl -n awx  logs deploy/awx-web -c awx-rsyslog

Access AWX Container’s Shell

Here is how to access each container’s shell:

kubectl exec -ti deploy/awx-web -c redis -- /bin/bash
kubectl exec -ti deploy/awx-web  -c  awx-web -- /bin/bash
kubectl exec -ti awx-postgres-13-0  -c  postgres -- /bin/bash

Upgrading AWX Operator and instance

We have created a dedicated guide for upgrading the Operator and AWX instance:

Step 5: Access Ansible AWX Dashboard

List all available services and check awx-service Nodeport

$ kubectl get svc -l "app.kubernetes.io/managed-by=awx-operator"
NAME              TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)        AGE
awx-postgres-13   ClusterIP   None            <none>        5432/TCP       153m
awx-service       NodePort    10.43.179.217   <none>        80:30059/TCP   152m

We can see the service type is set to NodePort. To access our service from outside Kubernetes cluster using k3s node IP address and give node port

http://hostip_or_hostname:30080

You can edit the Node Port and set to figure of your preference, but it has to be in the range of 30000-32768

On accessing AWX web portal you are presented with the welcome dashboard similar to one below.

install ansible awx ubuntu using operator 01

The login username is admin

Obtain admin user password by decoding the secret with the password value:

kubectl get secret awx-admin-password -o jsonpath="{.data.password}" | base64 --decode

Better output format:

kubectl get secret awx-admin-password -o go-template='{{range $k,$v := .data}}{{printf "%s: " $k}}{{if not $v}}{{$v}}{{else}}{{$v | base64decode}}{{end}}{{"\n"}}{{end}}'

Login with admin username and decoded password:

install ansible awx ubuntu using operator 02

The new AWX interface is beautifully designed with Love. It is a serious redesign with good polishing.

install ansible awx ubuntu using operator 03

In the articles to follow we will explore Ansible AWX administration and usage in automation Infrastructure repetitive tasks.

Review Kubernetes Node resources to ensure they are enough to run AWX:

$ kubectl top nodes --use-protocol-buffers
NAME     CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%
ubuntu   102m         1%     2534Mi          15%

Step 6: Configure Ingress for AWX

If you would like to access your AWX using domain names and SSL, check out our ingress articles:

Similar articles:

Josphat Mutai

Founder of Computingforgeeks. Expertise in Virtualization, Cloud, Linux/UNIX Administration, Automation,Storage Systems, Containers, Server Clustering e.t.c.

Related Articles

Databases Install and Configure PostgreSQL 17 on Ubuntu 24.04 / Debian 13 Ansible How To Manage PostgreSQL Database with Ansible Virtualization How To Install oVirt guest agent on Linux Ansible Automated Logical Volume Manager (LVM) Management on Linux using Ansible

147 thoughts on “How To Install Ansible AWX on Ubuntu 22.04|20.04|18.04”

  1. If I run through the manual. All tasks of the playbook are skipping with “Conditional result was False”.

    Before the playbook everything else was fine.

    Does anyone know what I have to do regarding this?

    Reply

  4. 1. Everything is fine and works until reboot
    2. After reboot web interface returns 400 for all requests / page parts
    — But all pods / service are up and running
    Could you please help with this?

    Reply

      • – Yes, service node port (32000) is open and accepts connections
        – Browser opens it
        – But every file request (*.js) leads to 404 error
        – It was ok before reboot
        – Browser gets main page with 200 (there are some links to AWX *.js files) and all next requests fails with 404 🙁

        awx-operator services are up:

        NAME READY STATUS RESTARTS AGE
        awx-7c5d846c88-mw98c 4/4 Running 8 21h
        awx-postgres-0 1/1 Running 2 21h

        NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
        awx-postgres ClusterIP None 5432/TCP 21h
        awx-service NodePort 10.43.42.47 80:32000/TCP 21h

        Reply

      • – I`ve analyzed it with two installations (cool guide, thanks!): one fresh rollout and the other one has been rebooted
        – Seems like after reboot there is no folder /var/lib/awx/public with web content in container “awx-web”
        – I`m still analyzing. If you have any ideas – please help. 🙂 Details go next:

        – After reboot there is no folder and system can not find files /var/lib/awx/public
        – Here are “awx-web” nginx logs with 404 errors:
        $ sudo kubectl logs awx-7c5d846c88-mw98c -c awx-web
        ……
        10.42.0.1 – – [02/Jul/2021:11:29:34 +0000] “GET / HTTP/1.1” 200 1190 “-” “Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36” “-”
        10.42.0.1 – – [02/Jul/2021:11:29:34 +0000] “GET /static/css/2.687a9035.chunk.css HTTP/1.1” 404 564 “http://192.168.88.36:32000/” “Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36” “-”
        2021/07/02 11:29:34 [error] 25#0: *44 open() “/var/lib/awx/public/static/css/2.687a9035.chunk.css” failed (2: No such file or directory), client: 10.42.0.1, server: _, request: “GET /static/css/2.687a9035.chunk.css HTTP/1.1”, host: “192.168.88.36:32000”, referrer: “http://192.168.88.36:32000/”

        Jusr after rollut everything is okay with these files:
        10.42.0.1 – – [02/Jul/2021:12:02:01 +0000] “GET /static/js/main.6ab990a9.chunk.js HTTP/1.1” 200 1328503 “http://192.168.88.33:31429/” “Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36” “-”
        10.42.0.1 – – [02/Jul/2021:12:02:01 +0000] “GET /static/js/2.53c634ac.chunk.js HTTP/1.1” 200 2298882 “http://192.168.88.33:31429/” “Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36” “-”
        10.42.0.1 – – [02/Jul/2021:12:02:01 +0000] “GET /static/js/3.14a7b215.chunk.js HTTP/1.1” 200 169799 “http://192.168.88.33:31429/” “Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36” “-”

        /var/lib/awx content after rollut:
        bash-4.4$ ls -l /var/lib/awx/
        total 16
        prw——- 1 awx root 0 Jul 2 11:24 awxfifo
        drwxrwxrwx 2 root root 4096 Jul 2 11:23 projects
        drwxr-xr-x 3 awx root 4096 Jul 2 11:24 public
        drwxrwxrwx 2 root root 4096 Jul 2 11:23 rsyslog
        drwxr-xr-x 3 root root 4096 Jun 28 21:49 venv
        bash-4.4$

        /var/lib/awx content after reboot (no “public” folder):
        bash-4.4$ ls -l /var/lib/awx/
        total 12
        prw——- 1 awx root 0 Jul 1 17:08 awxfifo
        drwxrwxrwx 2 root root 4096 Jun 30 07:27 projects
        drwxrwxrwx 2 root root 4096 Jul 1 17:08 rsyslog
        drwxr-xr-x 3 root root 4096 Jun 28 21:49 venv
        bash-4.4$

        Reply

      • – Solved it with running /usr/bin/launch_awx.sh
        — It copied me missing files
        – After reboot all files gone again
        — Will need to run it again (or make it autorun)
        – Still investgating, if anybody can hep – write your solution pls

        bash-4.4$ /usr/bin/launch_awx.sh
        186 static files copied to ‘/var/lib/awx/public/static’.
        Error: Another program is already listening on a port that one of our HTTP servers is configured to use. Shut this program down first before starting supervisord.
        For help, use /usr/local/bin/supervisord -h
        bash-4.4$

        Reply

  5. – Googled it and solved with running /usr/bin/launch_awx.sh inside awx-web container
    – But after every reboot these files are missing: still investigating, help if you know what to do 🙂

    bash-4.4$ /usr/bin/launch_awx.sh

    186 static files copied to ‘/var/lib/awx/public/static’.
    Error: Another program is already listening on a port that one of our HTTP servers is configured to use. Shut this program down first before starting supervisord.
    For help, use /usr/local/bin/supervisord -h
    bash-4.4$

    Reply

    • Hi,

      I found a way to persist /var/lib/awx/public/static data in a PVC. Please check updated article.

      Let me know if it works for you. I tested on my end and AWX login worked after reboot.

      Reply

  7. Thanks for this. Got it installed and accessible but it doesn’t seem to use DNS unless there’s 5 dots in the url. CoreDNS seems to default to ndots 5. You can force it to use DNS if you add a dot at the end (e.g. example.com wouldn’t resolve but example.com. would). I would just change every url to have that dot at the end but you can’t seem to do that in the requirements.yml for collections. Have you solved this issue on your install? Could you share how if so please?

    Reply

  8. I’m having problem with postgres, keep restarting

    root@Ansible-Server:~# kubectl get pods
    NAME READY STATUS RESTARTS AGE
    awx-operator-545497f7d5-v8mkh 1/1 Running 0 20m
    awx-ffb48ff7d-2xngt 4/4 Running 0 14m
    awx-postgres-0 0/1 CrashLoopBackOff 7 15m

    root@Ansible-Server:~# kubectl get events -w
    15m Normal Provisioning persistentvolumeclaim/static-data-pvc External provisioner is provisioning volume for claim “default/static-data-pvc”
    15m Normal ProvisioningSucceeded persistentvolumeclaim/awx-projects-claim Successfully provisioned volume pvc-7bc74fc1-1418-45f9-b2de-235682fde4a2
    15m Normal ProvisioningSucceeded persistentvolumeclaim/static-data-pvc Successfully provisioned volume pvc-9b87b0ce-edf4-4568-8bf7-10efefb52973
    15m Normal Scheduled pod/awx-ffb48ff7d-2xngt Successfully assigned default/awx-ffb48ff7d-2xngt to ansible-server
    15m Normal Pulling pod/awx-ffb48ff7d-2xngt Pulling image “docker.io/redis:latest”
    15m Normal Pulled pod/awx-ffb48ff7d-2xngt Successfully pulled image “docker.io/redis:latest” in 960.880704ms
    15m Normal Created pod/awx-ffb48ff7d-2xngt Created container redis
    15m Normal Started pod/awx-ffb48ff7d-2xngt Started container redis
    15m Normal Pulling pod/awx-ffb48ff7d-2xngt Pulling image “quay.io/ansible/awx:19.2.2”
    15m Normal Pulled pod/awx-ffb48ff7d-2xngt Successfully pulled image “quay.io/ansible/awx:19.2.2” in 18.554564271s
    15m Normal Created pod/awx-ffb48ff7d-2xngt Created container awx-web
    15m Normal Started pod/awx-ffb48ff7d-2xngt Started container awx-web
    15m Normal Pulled pod/awx-ffb48ff7d-2xngt Container image “quay.io/ansible/awx:19.2.2” already present on machine
    15m Normal Created pod/awx-ffb48ff7d-2xngt Created container awx-task
    15m Normal Started pod/awx-ffb48ff7d-2xngt Started container awx-task
    15m Normal Pulling pod/awx-ffb48ff7d-2xngt Pulling image “quay.io/ansible/awx-ee:0.5.0”
    15m Normal Pulled pod/awx-ffb48ff7d-2xngt Successfully pulled image “quay.io/ansible/awx-ee:0.5.0” in 20.711193131s
    15m Normal Created pod/awx-ffb48ff7d-2xngt Created container awx-ee
    15m Normal Started pod/awx-ffb48ff7d-2xngt Started container awx-ee
    14m Normal Pulled pod/awx-postgres-0 Container image “postgres:12” already present on machine
    14m Normal Created pod/awx-postgres-0 Created container postgres
    14m Normal Started pod/awx-postgres-0 Started container postgres
    61s Warning BackOff pod/awx-postgres-0 Back-off restarting failed container

    Reply

  9. Have a strange message when I try to load a module .

    ERROR! couldn’t resolve module/action ‘win_chocolatey’.

    I suspect it’s DNS related; however, I’m able to run the playbook just fine from the command-line on the host. Does something need to be setup in AWX so that modules can resolve?

    Reply

  10. Hello!

    Thank you very much, this tutorial was very helpful!
    With the new AWX release (19.3.0 and operator 0.13.0), there seems to be a problem with the postgres pod.
    When i follow this tutorial, which worked fine before the new release, the postgres pod always enters a “CrashLoopBackOff” state.
    I already tried downgrading to operator 0.12.0/0.11.0 but the problem persists.
    Is there any solution for this?

    best regards,
    Fabian

    Reply

  12. Hello. Thanks for the guide. All goes fine, until this point
    NAME READY STATUS RESTARTS AGE
    awx-postgres-0 1/1 Running 2 9h
    awx-66c64f8d67-m9pdz 3/4 CrashLoopBackOff 7 17m
    …
    18m Normal Created pod/awx-66c64f8d67-m9pdz Created container awx-web
    18m Normal Started pod/awx-66c64f8d67-m9pdz Started container awx-web
    17m Normal Pulled pod/awx-66c64f8d67-m9pdz Container image “quay.io/ansible/awx:19.3.0” already present on machine
    3m47s Warning BackOff pod/awx-66c64f8d67-m9pdz Back-off restarting failed container
    I´m very new on this….thanks.

    Reply

  13. Hi Guys, I just installed it and worked fine. Just one question, I am running a template that calls a python script that uses pynetbox module and AWX is sending an errror “ModuleErrorNotFound pynetbox at line … ” The pynetbox module is already installed on ubuntu but AWX seems not to detect it, please help will be very appreciated.

    Reply

  15. FYI – Just passing this on:
    After much digging and hair pulling found this to make my playbooks show up in /var/lib/awx/projects/
    This seems to be persistent across reboots. Just testing now. Good luck.

    kubectl get pods
    my pod was -> awx-66c64f8d67-525c9

    kubectl exec –stdin –tty awx-66c64f8d67-525c9 –container awx-task — /bin/bash

    Reply

  16. should read kubectl exec – -stdin – -tty awx-66c64f8d67-525c9 – -container awx-task – – /bin/bash
    Note: startup scirpt on each reboot to make know to OS

    Reply

  20. Hello!

    I have error ImagePullBackOff

    ~/awx-operator# kubectl get pods
    NAME READY STATUS RESTARTS AGE
    awx-operator-controller-manager-68bdfbbc96-nqslr 1/2 ImagePullBackOff 0 2m48s

    Reply

  22. Great article, it could do with a line adding to set permissions on /etc/rancher/k3s/k3s.yaml so k3s newbies like me don’t hit the same permissions error I did…

    # sudo chmod 644 /etc/rancher/k3s/k3s.yaml

    or modify the initial install command to something like this:

    # curl -sfL https://get.k3s.io | sudo bash -s – –write-kubeconfig-mode 644

    Reply

  25. I have error ErrImagePull

    ~/awx-operator# kubectl get pods -l “app.kubernetes.io/managed-by=awx-operator”
    NAME READY STATUS RESTARTS AGE
    awx-postgres-0 1/1 Running 0 58s
    awx-64bc58f8d6-k88sc 3/4 ErrImagePull 0 49s

    ~/awx-operator# kubectl describe pod awx-64bc58f8d6-k88sc

    Warning Failed 51s kubelet Failed to pull image “quay.io/ansible/awx-ee:latest”: rpc error: code = Unknown desc = failed to pull and unpack image “quay.io/ansible/awx-ee:latest”: failed to copy: httpReaderSeeker: failed open: failed to do request: Get “https://quay.io/v2/ansible/awx-ee/blobs/sha256:e0413392ed8b63d513b016f28fa25128151455a95bca856216603e27921385f6”: dial tcp: lookup quay.io: Try again
    Normal Pulling 36s (x2 over 71s) kubelet Pulling image “quay.io/ansible/awx-ee:latest”
    Warning Failed 22s kubelet Failed to pull image “quay.io/ansible/awx-ee:latest”: rpc error: code = Unknown desc = failed to pull and unpack image “quay.io/ansible/awx-ee:latest”: failed to copy: httpReaderSeeker: failed open: failed to do request: Get “https://quay.io/v2/ansible/awx-ee/blobs/sha256:b13b1c06994ef8145c55d9871ceec46c63fe29c92ecf43a2d4d30f12c0d0d8a8”: dial tcp: lookup quay.io: Try again
    Warning Failed 22s (x2 over 51s) kubelet Error: ErrImagePull
    Normal BackOff 9s (x2 over 50s) kubelet Back-off pulling image “quay.io/ansible/awx-ee:latest”
    Warning Failed 9s (x2 over 50s) kubelet Error: ImagePullBackOff

    Reply

  26. 4 CPU and 8Gb RAM

    Events:
    Type Reason Age From Message
    —- —— —- —- ——-
    Warning FailedScheduling 8m59s default-scheduler 0/1 nodes are available: 1 Insufficient cpu, 1 Insufficient memory.
    Warning FailedScheduling 8m58s default-scheduler 0/1 nodes are available: 1 Insufficient cpu, 1 Insufficient memory.

    I do not understand why this is not work!!

    Reply

  28. Thank you for the article, I followed the steps, but at the end I’m getting a blank login page and not the AWX login page. What could cause that?

    Reply

      • By checking, it appears that there is no content available on awx-web for the web site. Most of the content returns 404, but I couldn’t figure out how to upload the content to the site. This became apparent to me immediately after installation. All PODs are working and the repository seems to be functional (checking permissions is OK).

        Reply

    • Hi @Josphat, thanks for the guide!

      @AO
      I had the same issue, log into the container using:
      kubectl exec –stdin –tty awx-7d9cc5698c-bw2rl -c awx-web — /bin/bash

      Then remove the current static file as it’s empty:
      rm -rf /var/lib/awx/public/static

      Then create a symlink:
      ln -s /var/lib/awx/venv/awx/lib/python3.8/site-packages/awx/ui/build/static/ /var/lib/awx/public/static

      This should fix your problem 🙂

      Reply

  29. Hi,
    I am struggling with infoblox module installation inside awx-web/task containers via
    #pip3 install infoblox-client
    i got a permission denied error; do you know how to get root (or cleared) in order to make this kind of module installation?
    Thank you

    Reply

  30. Hello, thanks for the article, everything worked out great the first time. But there is a problem in the browser only a blank page is displayed, no errors.

    Reply

  32. Everything was good, but finally i ended up with a white screen when i tried the url. Anyone else faced the same issue, anyway to fix it?

    Also when i tried URL\api – I was able to see the api screen with broken images.

    Reply

      • Ha Josphat,
        Is there already an update I can use to redirect my awx as tutored here onto https? It does not work directly configuring nginx. Have to use the embedded ingress/traefik? Would be very much appreciated to retrieve the guide for https redirection.
        Regards
        Netty

        Reply

  34. Is the steps above still working? New to AWX and all steps are ok on my Ubuntu 20.04.3 untill I’ve open the browser – just white page no login page.

    awx@awx:~$ kubectl get svc -l “app.kubernetes.io/managed-by=awx-operator”
    NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
    awx-postgres ClusterIP None 5432/TCP 59m
    awx-service NodePort 10.43.90.201 80:32000/TCP 58m

    can anyone help?

    Reply

  35. Does anyone know how to add private CA to the AWX instance? When adding a project and trying to sync I get the following issue:

    fatal: [localhost]: FAILED! => {“changed”: false, “cmd”: “/usr/bin/git ls-remote https://github-01.local/Group/Ansible.git -h refs/heads/HEAD”, “msg”: “fatal: unable to access ‘https://github-01.local/Group/Ansible.git/’: SSL certificate problem: unable to get local issuer certificate”, “rc”: 128, “stderr”: “fatal: unable to access ‘https://github-01.local/Group/Ansible.git/’: SSL certificate problem: unable to get local issuer certificate\n”, “stderr_lines”: [“fatal: unable to access ‘https://github-01.local/Group/Ansible.git/’: SSL certificate problem: unable to get local issuer certificate”], “stdout”: “”, “stdout_lines”: []}

    Thanks in advance

    Reply

  36. Hi, I’ve been secussful install by all steps. But when I create project to Gitlab/Girhub, they both return failed. Then I check the pods my “awx-operator-controller-manager-647f4c5c7d-5gss2” is turn to error. And here is the log of it:

    {“level”:”error”,”ts”:1642065249.175949,”logger”:”controller-runtime.manager”,”msg”:”Failed to get API Group-Resources”,”error”:”Get \”https://10.43.0.1:443/api?timeout=32s\”: dial tcp 10.43.0.1:443: i/o timeout”,”stacktrace”:”sigs.k8s.io/controller-runtime/pkg/manager.New\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/manager/manager.go:312\ngithub.com/operator-framework/operator-sdk/internal/cmd/ansible-operator/run.run\n\t/workspace/internal/cmd/ansible-operator/run/cmd.go:191\ngithub.com/operator-framework/operator-sdk/internal/cmd/ansible-operator/run.NewCmd.func1\n\t/workspace/internal/cmd/ansible-operator/run/cmd.go:79\ngithub.com/spf13/cobra.(*Command).execute\n\t/go/pkg/mod/github.com/spf13/[email protected]/command.go:856\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\t/go/pkg/mod/github.com/spf13/[email protected]/command.go:960\ngithub.com/spf13/cobra.(*Command).Execute\n\t/go/pkg/mod/github.com/spf13/[email protected]/command.go:897\nmain.main\n\t/workspace/cmd/ansible-operator/main.go:40\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:225″}
    {“level”:”error”,”ts”:1642065249.1761334,”logger”:”cmd”,”msg”:”Failed to create a new manager.”,”Namespace”:”awx”,”error”:”Get \”https://10.43.0.1:443/api?timeout=32s\”: dial tcp 10.43.0.1:443: i/o timeout”,”stacktrace”:”github.com/operator-framework/operator-sdk/internal/cmd/ansible-operator/run.NewCmd.func1\n\t/workspace/internal/cmd/ansible-operator/run/cmd.go:79\ngithub.com/spf13/cobra.(*Command).execute\n\t/go/pkg/mod/github.com/spf13/[email protected]/command.go:856\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\t/go/pkg/mod/github.com/spf13/[email protected]/command.go:960\ngithub.com/spf13/cobra.(*Command).Execute\n\t/go/pkg/mod/github.com/spf13/[email protected]/command.go:897\nmain.main\n\t/workspace/cmd/ansible-operator/main.go:40\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:225″}

    Does anyone have idea?

    Reply

  37. I used a fresh AWS t3a.large (8GB) instance with Ubuntu 20.04 and when I got to

    watch kubectl get pods -l “app.kubernetes.io/managed-by=awx-operator”

    it stayed at 0/4 and 1/1 for >13 minutes

    tried
    kubectl describe pods awx
    and got “Warning FailedScheduling 13m default-scheduler 0/1 nodes are available: 1 Insufficient cpu.”

    CPU never went above 25% in AWS console.

    Tried reboot but then I don’t know how to get things running again… when I run
    kubectl describe pods awx
    I now get
    WARN[0000] Unable to read /etc/rancher/k3s/k3s.yaml, please start server with –write-kubeconfig-mode to modify kube config permissions
    error: error loading config file “/etc/rancher/k3s/k3s.yaml”: open /etc/rancher/k3s/k3s.yaml: permission denied

    Reply

  38. Thank you for you easy to follow installation procedure. I tried it a few times on a newly installed ubuntu server and every time i get stuck at the following place (it could be a lack of experience, but I have problems troubleshooting this). Any suggestions or help will be appresciated! :

    # kubectl get pods -l “app.kubernetes.io/managed-by=awx-operator”

    NAME READY STATUS RESTARTS AGE
    —> awx-75bd7d77d5-dddbc 0/4 Pending 0 12m
    awx-postgres-0 1/1 Running 0 12m

    My one pod stay on pending status no matter how long I wait. I delete them as follows :

    # kubectl delete pods -l “app.kubernetes.io/managed-by=awx-operator”
    pod “awx-75bd7d77d5-dddbc” deleted
    pod “awx-postgres-0” deleted

    They regenerate and again the pod stay on pending, no matter how long I wait. Looking at the pod logfiles i get the following :

    # kubectl logs -f deployments/awx-operator
    Error from server (NotFound): deployments.apps “awx-operator” not found

    Reply

  39. Hello I follow the step and the $ watch kubectl get pods -l “app.kubernetes.io/managed-by=awx-operator” is giving me just awx-postgres-0 Pods is it normal ? I use the latest awx 0.16.0

    NAME READY STATUS RESTARTS AGE
    awx-postgres-0 1/1 Running 0 75s

    Reply

  40. Man, I don’t know what I am doing wrong. I have gone through this up and down but I am only getting the postgres pod.

    kubectl get pods -l “app.kubernetes.io/managed-by=awx-operator”
    NAME READY STATUS RESTARTS AGE
    awx-postgres-0 1/1 Running 0 29m

    I’ve never used kubernetes.

    Reply

  41. Hi, everything is working fine. But when I try to create a projet using manual, it told me no playbook available.

    I do have folder inside /var/lib/awx/projects like : /var/lib/awx/projects/test/test.yml (and syntax of test.yml is good too).

    I check awx-task and awx-web container, and both have files. Why ?

    Reply

  42. This was an awesome walkthrough, thank you. I had googled many sources before stumbling on this page, and it was the most useful and it all worked.

    Reply

  43. Do I need to make sure all the services under this command are running?

    kubectl get pods -l “app.kubernetes.io/managed-by=awx-operator”

    Reply

  44. Thank you for the great write up. I have everything up and running now however I am a bit stumped on how to proceed. How do you add projects to the “/var/lib/projects” directory? It does not exist on the host system that I could find. This is my first time using AWX and K3s.

    Reply

  45. Thank you for this excellent Tutorial!
    When i wait for the pods starting up, i only get the postgres pod starting.

    The second doesnt show up
    NAME READY STATUS RESTARTS AGE
    awx-postgres-0 1/1 Running 0 42s
    -> X awx-66c64f8d67-4qmh5 4/4 Running 0 42s

    What am i doing wrong?

    Reply

  46. Had to manually set the Version 0.15.0 so everything worked.
    RELEASE_TAG=0.15.0 # Latest didn’t work for deploying awx. used 0.15.0
    echo $RELEASE_TAG
    git checkout $RELEASE_TAG

    Reply

  47. Hi, For me awx-service newer comes up, it even do not appear to start doing that. Only awx-postgress-0 starts without any issues. Has someone faced the same issue?

    Reply

      • Hello,
        Also new to k3s. Just ran into the same issue as @Uvis. I can deploy awx-postgres no problem. But for the life of me cannot figure out why awx-service won’t deploy.

        $ kubectl logs awx.awx.ansible.com/awx
        error: no kind “AWX” is registered for version “awx.ansible.com/v1beta1” in scheme “k8s.io/kubectl/pkg/scheme/scheme.go:28”

        Reply

      • Hi, I have a problem same with Uvis

        Here is the result:

        $kubectl get pods -l “app.kubernetes.io/managed-by=awx-operator”
        NAME READY STATUS RESTARTS AGE
        awx-postgres-0 1/1 Running 0 75s

        $ kubectl get pvc
        NAME STATUS VOLUME…….
        postgres-awx-postgres-0 Bound pvc-c0149545-8631-4aa1-a03f-2134a7f42aa6
        static-data-pvc Pending
        awx-projects-claim Pending

        Reply

      • I’ve got the same issue with Uvis

        Here is the result:

        $kubectl get pods -l “app.kubernetes.io/managed-by=awx-operator”
        NAME READY STATUS RESTARTS AGE
        awx-postgres-0 1/1 Running 0 75s

        Reply

    • Hi, I have a problem same with Uvis

      Here is the result:

      $kubectl get pods -l “app.kubernetes.io/managed-by=awx-operator”
      NAME READY STATUS RESTARTS AGE
      awx-postgres-0 1/1 Running 0 75s

      $ kubectl get pvc
      NAME STATUS VOLUME…….
      postgres-awx-postgres-0 Bound pvc-c0149545-8631-4aa1-a03f-2134a7f42aa6
      static-data-pvc Pending
      awx-projects-claim Pending

      Reply

  50. hello i’ve got this issue, no errors but process is pending forever and no way to see whats going on unless there is a logfile somewhere?
    NAME READY STATUS RESTARTS AGE
    awx-postgres-0 1/1 Running 0 9m48s
    awx-5b5c856b94-zrf9c 0/4 Pending 0 8m18s

    Reply

  52. After restart:

    1) I have to re-establish awx as the default namespace:

    $ export NAMESPACE=awx
    $ kubectl config set-context –current –namespace=$NAMESPACE \

    2) awx-ee evaporates collections I’ve installed.

    Not sure if this is expected or not.

    Reply

      • Part 1) on awx-ee container, run $ ansible-galaxy collection install chocolatey.chocolatey

        Part 2) verify chocolatey.chocolatey installed:

        $ ansible-galaxy collection verify chocolatey.chocolatey

        collection found at ‘/home/runner/.ansible/collections/ansible_collections/chocolatey/chocolatey’

        $ ansible-doc chocolatey.chocolatey.win_chocolatey

        [returned documents]

        Part 3) Restart Ubuntu, discover chocolatey.chocolatey is no longer installed / data evoporated:

        ~/.ansible/collections/**ansible_collections**/ doesn’t exist.

        $ ansible-galaxy collection verify chocolatey.chocolatey

        fail

        $ ansible-doc chocolatey.chocolatey.win_chocolatey

        fail

        Reply

  53. Hey Josphat, thanks for the great guide. Could you kindly provide some guidance on how to perform a backup and restore? Thanks lots!

    Reply

  55. After a reboot there are no ressources in the container…???

    also in the deployment:
    Every 2.0s: kubectl get pods -l app.kube… ubuntults: Mon May 16 13:39:26 2022

    NAME READY STATUS RESTARTS AGE
    awx-544f7bf9c4-xkt45 0/4 ContainerStatusUnknown 4 32m
    awx-544f7bf9c4-m29h9 0/4 ContainerStatusUnknown 3 29m
    awx-544f7bf9c4-kz5j8 0/4 PodInitializing 0 19m
    awx-postgres-0 1/1 Running 0 20m

    Reply

  56. During the deployment:
    $ kubectl get pods -l “app.kubernetes.io/managed-by=awx-operator”
    NAME READY STATUS RESTARTS AGE
    awx-fd64bc5b8-jfd6q 0/4 ContainerStatusUnknown 2 24m
    awx-fd64bc5b8-xt477 0/4 Error 0 21m
    awx-fd64bc5b8-jws4k 0/4 ContainerStatusUnknown 2 14m
    awx-postgres-0 0/1 Pending 0 112s
    awx-fd64bc5b8-7nn4b 0/4 Error 1 (109s ago) 7m1s
    awx-fd64bc5b8-jkp9p 0/4 Pending 0 81s

    Reply

  58. Hello

    Thanks for the good tutorial.

    But, the pods after the postgres pod dosn’t start.

    I get an error:
    awx-6b57998ddc-67b9n 0/4 Init:CrashLoopBackOff 6 (2m33s ago) 9m28s

    And if i check with kubectl describe pod awx-6b57998ddc-67b9n
    it shows me on the Events, that it stucks on init:

    Events:
    Type Reason Age From Message
    —- —— —- —- ——-
    Normal Scheduled 10m default-scheduler Successfully assigned awx/awx-6b57998ddc-67b9n to ansible-test
    Normal Pulling 10m kubelet Pulling image “quay.io/ansible/awx-ee:latest”
    Normal Pulled 9m16s kubelet Successfully pulled image “quay.io/ansible/awx-ee:latest” in 44.496250886s
    Normal Pulled 7m28s (x4 over 9m2s) kubelet Container image “quay.io/ansible/awx-ee:latest” already present on machine
    Normal Created 7m28s (x5 over 9m16s) kubelet Created container init
    Normal Started 7m28s (x5 over 9m16s) kubelet Started container init
    Warning BackOff 5m (x19 over 9m1s) kubelet Back-off restarting failed container

    How can i fix this?

    Thanks and best regards
    AD

    Reply

  59. I was able to setup AWX using this post for the first time without any error, but now when I am trying to do it again on fresh AWS VM its failing.
    My Init Containers are failing with below error, can you please suggest what changes should I need to make.

    chmod: changing permissions of ‘/var/lib/awx/projects’: Operation not permitted
    chgrp: changing group of ‘/var/lib/awx/projects’: Operation not permitted

    Reply

  60. Thank you for the guide,
    I am facing a problem with the second pod (not postgres)
    NAME READY STATUS RESTARTS AGE
    awx-postgres-13-0 1/1 Running 0 5m12s
    awx-6ccc9fd4fb-rqstx 0/4 Init:CrashLoopBackOff 5 (2m24s ago) 5m13s

    i tried the fix in the guide and didn’t fix it, i tried to check the logs but there are too much entries and i’m a noob.
    anyway to fix it ?

    Reply

  62. After carefully retrying this on a clean install of Ubuntu 22.04 following these instructions exactly i’m still running into the Init:CrashLoopBackOff error but on the awx-6xxxxxxxx pod. I’ve tried the above fix but verification on the filesystem shows permissions are already set correctly.

    the outbut of kubectl logs awx-65db48d759-cm5qt:

    Defaulted container “redis” out of: redis, awx-web, awx-task, awx-ee, init (init), init-projects (init)
    Error from server (BadRequest): container “redis” in pod “awx-65db48d759-cm5qt” is waiting to start: PodInitializing

    Any help or hint for where to look?

    Reply

    • For anyone struggeling to figure out the same issue. I finally got to the bottom of this one.

      If you’re running your server as a virtual (in my case proxmox) and the CPU type is not set to HOST, this issue will come up. shut down the server, change the CPU type to HOST in the vm config and reboot. everything should come up after that.

      Good luck!

      PS. instead of a NGINX reverse proxy with letsencrypt why not use traefik? I would be interested in seeing an article for that.

      Reply

  63. Hi,

    I implemented AWX and would need to run some automation routines on the postgres database.
    The Ansible command line ran normally, but when implementing AWX via K3s, it informs me that the module is not installed, I have already tried to install it in the AWX directory and in the home directory but without effect, would it be possible to install it directly in the AWX cluster?

    Att.

    Reply

      • Josphat,

        Playbook:
        —
        – name: UpdateDatabase
        hosts: all

        tasks:
        – name: Update value field send
        community.postgresql.postgresql_query:
        db: database
        login_user: username
        login_password: P@$$w0rd!
        query: update api set value = ‘300’ where property = ‘send’;

        The message I get when running the playbook is below:

        “ERROR! couldn’t resolve module/action ‘community.postgresql.postgresql_query’. This often indicates a misspelling, missing collection, or incorrect module path.”

        I also tried with: postgresql_query.

        But the same error

        Thank you in advance for your help

        Reply

  67. I have been trying to spin this up for two weeks and no luck. The actual awx operator will build, then postgress builds, then when it gets to the awx container it crashes and breaks every container.

    awx-operator-controller-manager-65897dbf99-v8ddv 0/2 Pending 0 2m13s
    awx-operator-controller-manager-65897dbf99-h4srr 0/2 Error 1 8m6s
    awx-demo-d9b84d689-xgbvk 0/4 Error 0 4m32s
    awx-demo-d9b84d689-lhxcr 0/4 Pending 0 2m11s
    awx-demo-postgres-13-0 0/1 Pending 0 2m10s

    Reply

  68. Hi Jos,
    I got this running flawless running under Vmware 7.x cluster in a RHEL 9.1 VM. The k3s configs as outlined are flawless.

    When I get to the GUI and try to build my vcenter inventory I am getting error, the steps I follow are the following:

    -Define vcenter credentails
    -Define vcenter invnetory

    The error on the job on source sync is just giving me a single line “finished” but actually failed. Any clues on how to get the Dynamic inventory from vcenter working on this. Thanks in advance.

    Reply

  69. AWX works as expected, but these commands don’t work for me:
    kubectl -n awx logs deploy/awx -c awx-web
    kubectl -n awx logs deploy/awx -c redis
    …

    I got this error message:
    “Error from server (NotFound): deployments.apps “awx” not found”

    Reply

  70. i’m deploying this in Proxmox ve and I’m having chrashLoopBackIff on awx-task and awx-web. What can it be?

    Reply

  71. hello. I’m having troubles debloying awx-task and awx-web it stay in Init:CrashLoopBackkOff. It occuers when I deplay aw-deploy.yml I already tested with ubuntu server, debian 11 on Proxmox Ve and does not work. I can not know what can be the problem. Can someone help me please?

    Reply

  72. @josphat Mutai : Hello , can we use these instructions to install the AWX Tower with operator version 2.2.1 .
    Thanks in advance .

    Reply

  73. Does anyone have any idea why’ll after installation, within the GUI I can’t see any playbooks? I have been beating my head on this one for a while on version 22.3.0 I have playbooks in /var/lib/awx/projects/ but I just get the error – “There are no available playbook directories in /var/lib/awx/projects. Either that directory is empty, or all of the contents are already assigned to other projects. Create a new directory there and make sure the playbook files can be read by the “awx” system user”.

    I’m lost at this point if I can’t get these playbooks to show up in the GUI.

    Reply

  74. There are no available playbook directories in /var/lib/awx/projects. Either that directory is empty, or all of the contents are already assigned to other projects. Create a new directory there and make sure the playbook files can be read by the “awx” system user.

    I can’t any manual playbooks to show up in /var/lib/awx/projects/. I have followed these steps at least 3 times and still no luck.

    What in the heck am I missing?

    Reply

    • You should make them on the hosts on the PVC path.

      For me i had to create at:

      /var/lib/rancher/k3s/storage/pvc (projects is identified here)/{subdirectory}/{playbook}
      make sure that you set the permissions.. I did 777 for testing only.

      And for me it works.

      Reply

  77. Hi, firstly thank you for this great guide it’s been extremely helpfull

    Quick question, what is the awx user password when trying to run sudo within the shell -> kubectl exec -ti deploy/awx-web -c awx-web — /bin/bash

    Reply

  78. Hello,

    I was able to install AWX, and the install appeared to work. I had the issue with Proxmox where I needed to set the cpu to “Host” mode instead of the default mode.

    After AWX was installed, I tried to pull down a repository using the projects tab and I was getting an error, “Github.com could not be resolved.” I even tried to resync the default repo that was on the project tab from the AWX website, but I got the same error “github.com could not be resolved.”

    I went into the VM that i am using to run AWX and I was able to clone repositories and resolve websites just fine. I think there may be an issue with the DNS in one of the pods in the Kubernetes cluster. Do you know how i can test out DNS in the pods, or if you have seen any issues with DNS?

    Reply

Leave a Comment

Press ESC to close