Installing AWX on AlmaLinux 9

I ran into some issues installing AWX on AlmaLinux 9 on Proxmox (I had the same issues with Alma 8.7). This also applies to RockyLinux 9.

I was installing AWX via Rancher following https://github.com/ansible/awx-operator#basic-install. I made it all the way to the section where you create the awx-demo.yaml, add it to your kustomization.yaml and build via kustomize build . | kubectl apply -f -. From there I was receiving errors such as “unable to determine if virtual resource”,”gvk”:”apps/v1″ and the build would ultimately fail out.

In order to make it past that error I found a found a few posts which suggested changing the CPU type from “Default (kvm64)” to Host. This sets the VM to match the CPU of the host.

***If you are running HyperV, there is a similar option, see the final post in this Google Group conversation: https://groups.google.com/g/awx-project/c/4tmP0TlRODU.***

After resetting the CPU type, rebooting the vm and re-running the kustomize build, I was able to make it quite a bit further. The logs looked like there were no issues, then towards the end the script once again failed. This time I was seeing the following error: “awx unable to retrieve the complete list of server APIs: metrics.k8s.io/v1beta1:”. The Pod itself was also down with a CrashLoopBackOff error. From there I found the following link which was able to get me past all of my installation issues: https://stackoverflow.com/questions/62442679/could-not-get-apiversions-from-kubernetes-unable-to-retrieve-the-complete-list

I ran: kubectl api-resources which listed the resources and metrics.k8s.io/v1beta1 was in fact down.

Next I ran: kubectl delete apiservice/v1beta1.metrics.k8s.io

From there I re-ran the kustomize build command and awx installation completed successfully after the installation. I did have to open the firewall ports in Alma to allow my browser to access AWX.

Steps to Install AWX:

#Install Rancher
curl -sfL https://get.k3s.io | sh -

#Install Kustomize
curl -s "https://raw.githubusercontent.com/kubernetes-sigs/kustomize/master/hack/install_kustomize.sh"  | bash

#Move Kustomize binary
mv kustomize /usr/local/bin/

#Goto AWX Readme and follow along from there:
# https://github.com/ansible/awx-operator#basic-install

Feel free to contact me if you have any comments or questions

Disabling Inactive Domain User and Computer Accounts in Active Directory with Ansible

In my last article I wrote about having Ansible run several audit requests including: “We need a list of all inactive user accounts” as well as “We need a list of inactive computer accounts”. Now that we have those listed, we can let Ansible clean those up. I preferred to create a new playbook for these tasks. First it will list the Users and Computers it will be handling first, next it will disable the account, followed by moving it to either the Inactive_Users or Inactive_Computers OU. I never delete the accounts as we prefer to disable, then move them.

Below is my ansible playbook “fix_AD_Inactive-Users-AND-Computers-90days.yml”

---
- hosts: pdc
  gather_facts: no
  tasks:
     - name: copy file to windows
       win_copy:
          src: files/fix_inactive_usr.ps1
          dest: c:\it\fix_inactive_usr.ps1

     - name: copy file to windows
       win_copy:
          src: files/fix_inactive_pc.ps1
          dest: c:\it\fix_inactive_pc.ps1

     - name: Fix inactive users - 90 days
       win_shell: c:\it\fix_inactive_usr.ps1
       register: inactive_usr

     - debug: var=inactive_usr.stdout_lines

     - name: Fix inactive computers - 90 days
       win_shell: c:\it\fix_inactive_pc.ps1
       register: inactive_computer

     - debug: var=inactive_computer.stdout_lines

Below is the code for “fix_inactive_usr.ps1”

$date = (get-date).AddDays(-90)

$USR = (Get-ADUser -Filter {LastLogonDate -lt $date} -Property Enabled | Where-Object {$_.Enabled -like "true"} | Select DistinguishedName).DistinguishedName
echo $USR
ForEach ($Item in $USR){
   Disable-ADAccount $Item
   Move-ADObject -Identity $Item -TargetPath "OU=Disabled_Accounts,DC=contoso,DC=com"
   }

Please note in the PowerShell scripts above and below, you will need to change “DC=contoso,DC=com” to reflect your actual domain

Below is the code for “fix_inactive_pc.ps1”

# Specify inactivity range value below
$DaysInactive = 90
# $time variable converts $DaysInactive to LastLogonTimeStamp property format for the -Filter switch to work

$time = (Get-Date).Adddays(-($DaysInactive))

# Identify inactive computer accounts

$PC = (Get-ADComputer -Filter {LastLogonTimeStamp -lt $time} -Property Enabled | Where-Object {$_.Enabled -like "true"} | Select DistinguishedName).DistinguishedName
echo $PC
ForEach ($Item in $PC){
   Disable-ADAccount $Item
   Move-ADObject -Identity $Item -TargetPath "OU=Disabled_Computers,DC=contoso,DC=com"
   }

Audit Active Directory with Ansible

Everyone loves an audit right? We have to deal with audits quite a bit and that requires remedial tasks like “We need a list of AD user accounts that have been locked out”, “We need a list of all inactive user accounts”, “We need a list of inactive computer accounts”, “We need a list of all members of Domain Admins group” as well as “We need a list of all AD accounts”. All of these requirements can easily be scripted with PowerShell. Since I love to automate things and I would rather not run these commands separately, I figured I would just create an Ansible script to run all request at the same time. that way I could logon once, select my Ansible playbook and let it run and I don’t even need to logon to the DC to run theses tasks. I can sit back and let Ansible deal with this.

This simple Ansible playbook uses 3 PowerShell commands and 2 PowerShell scripts that I’m sure most Windows Administrators are familiar with.

---
- hosts: pdc
  gather_facts: no
  tasks:
     - name: copy audit_AD_inactive_users.ps1 to Windows
       win_copy:
          src: files/audit_AD_inactive_users.ps1
          dest: c:\cit\audit_AD_inactive_users.ps1

     - name: copy audit_AD_inactive_computers.ps1 to Windows
       win_copy:
          src: files/audit_AD_inactive_computers.ps1
          dest: c:\cit\audit_AD_inactive_computers.ps1

     - name: Run Audit for Locked-Out Accounts
       win_shell: Search-AdAccount -LockedOut | select Name, LockedOut,LastLogonDate,distinguishedName
       register: lockedoutaccounts

     - debug: var=lockedoutaccounts.stdout_lines

     - name: Run Audit of inactive users - 90 days
       win_shell: c:\cit\audit_AD_inactive_users.ps1
       register: inactive_users

     - debug: var=inactive_users.stdout_lines

     - name: Run Audit of inactive computers - 90 days
       win_shell: c:\cit\audit_AD_inactive_computers.ps1
       register: inactive_computers

     - debug: var=inactive_computers.stdout_lines

     - name: Run Audit for members of Domain Admins group
       win_shell: Get-ADGroupMember -Identity 'Domain Admins' | Select-Object name, objectClass,distinguishedName
       register: dom_admin_users

     - debug: var=dom_admin_users.stdout_lines

     - name: Run Audit for all domain users
       win_shell: Get-ADUser -Filter * -SearchBase "dc=contoso,dc=com" | select Name, objectClass,distinguishedName
       register: all_dom_users

     - debug: var=all_dom_users.stdout_lines

Not bad right? Ansible Rocks! The only complaint I may see is I’m not outputting the results to a CSV file, but if you run this script often, you shouldn’t need the fancy format.

Below is the first PowerShell script “audit_AD_inactive_users.ps1”

$date = (get-date).AddDays(-90)

Get-ADUser -Filter {LastLogonDate -lt $date} -Property Enabled | Where-Object {$_.Enabled -like “true”} | Select Name, SamAccountName, DistinguishedName

Below is the second PowerShell script “audit_AD_inactive_computers.ps1”

# Specify inactivity range value below
$DaysInactive = 90
# $time variable converts $DaysInactive to LastLogonTimeStamp property format for the -Filter switch to work

$time = (Get-Date).Adddays(-($DaysInactive))

# Identify inactive computer accounts

Get-ADComputer -Filter {LastLogonTimeStamp -lt $time} -ResultPageSize 2000 -resultSetSize $null -Properties Name, OperatingSystem, SamAccountName, DistinguishedName, LastLogonDate | Select DNSHostName, LastLogonDate, DistinguishedName

HAProxy to the rescue

I have a client that did not want some of their employees having internet access due to loss of productivity. The employee workstations were on their own network that was firewalled off the regular network. The firewall allowed very limited access to the internal office network and no access to the internet.

They ran into an issue where some of the employees required access to a certain website to do their job. I could easily open a hole in the firewall to that site, but this site was hosted by AWS and the IP’s changed daily. I could continue adding in new IP’s, or I could go the proxy route. In the past I setup and configured a Squid proxy server to handle this, but I really wanted to see if I could get HAProxy to handle this. I knew HAProxy could forward web traffic, but that was to a specific site with static IP’s. I tested using HAProxy in http mode as well as tcp pointing to known ips and it would work until the ip changed.

After some searching, I found HAProxy was able to use DNS service discovery to detect server changes on the fly and then apply them to your system automatically. All I needed to do was add a DNS Resolvers configuration to my HAProxy config along with load balancing. I will post my configuration below with an explanation following. In the code below, I’m changing the name of the website the client is using to a more generic name like “fedex.com”

global
   stats socket :9000 mode 660 level admin

resolvers dns1
   nameserver dns1 192.168.3.53:53
   accepted_payload_size 8192 # allow larger DNS payloads

frontend https
   bind *:443
   option tcplog
   mode tcp
   default_backend fedex-https


backend fedex-https
   mode tcp
   balance source
   server-template fedex1 3 www.fedex.com:443 check resolvers dns1 init-addr none check inter 2000 rise 2 fall 5 verify none

The frontend listens on port 443 (Clients are directed to this in their proxy configuration via AD GPO). The backend server template will add (3) entries from DNS lookups to the backend. You would determine the number you want by first running a manual nslookup against the host you are looking to connect to and see how many results you get back, in my case I got 6, so I added 3 (you never want to go above the amount of servers your manual nslookup resolves). I could have easily set this number at 2 and the backend would swap between the first 2 host it gets when it checks DNS. In my actual configuration that I’m not showing, I set the number to (2). The “init-addr none” allows HAProxy to run if it is unable to resolve the hostname on startup.

Now I have a hole in my firewall allowing access from the firewalled employees to my HAProxy server only via port 443. I have an AD GPO that sets their computers to use my HAProxy server for internet access. They can try to go any other site and they get nothing. It only allows them to fedex.com.

A more detailed explanation can be found here:

https://www.haproxy.com/blog/dns-service-discovery-haproxy/

and:

https://www.haproxy.com/blog/client-ip-persistence-or-source-ip-hash-load-balancing/

Ansible – List all powered on VM’s to CSV

Sometimes we need to audit our VMWare environment and it is nice to have ansible gather this information in seconds into a format that is easy to import. This can take an hours long job down to seconds. I initially had an ansible script that would list all vm’s including powered templates, powered down or paused vms. That was nice, but I only really needed the powered on vms.

This grew from running the ansible script and manually scraping the output for what I needed which still took some time. The second iteration had me run the ansible script and “tee” the output to a file which I would then run a series of 12 sed statements against the file to gather the information I needed. That was great and it took less time, but I wanted to get it down to a one liner.

My third iteration is where I am at today. This is a one liner that isn’t pretty. I was able to join several sed statements into one, but the last 4 sed statements I still had to run separately due to the fact if they were joined to the first, the output wasn’t what I was expecting.

This is what I am using now (I will break it down after):

ansible-playbook VMWARE_list_all_powered-on_vms.yml --ask-vault-pass | sed -e '/"msg"/,$!d; /"msg"/d; / ____________/,$d; s/        {//g; s/        },//g; s/            "guest_name": "//g; s/            "ip_address": "//g; s/"//g'  > list.csv && sed -i '1d' list.csv && sed -i '/        }/,$d' list.csv && sed -i 'N;s/\n//' list.csv && sed -i '/^[[:space:]]*$/d' list.csv && cat list.csv

I know the above code is not too appealing to the eye (Feel free to message me if you have any suggestions). The first statement is running the ansible playbook “VMWARE_list_all_powered-on_vms.yml” Since I don’t want to store passwords in plain text, in this example I’m using ansible vault (there are better options out there). I am piping the output of ansible into 5 sed statements. The first sed statement is where I take the ansible output (which contains Ansible cowsay… which makes Ansible output fun) and do the following:

  1. Strip the first several lines of Ansible output down to the “msg”: [ line
  2. Remove the msg line
  3. Remove the trailing Ansible output (PLAY RECAP)
  4. Remove all lines starting with “{“
  5. Remove all lines starting with “},”
  6. Remove everything before and including “guest_name”: “
  7. Remove everything before and including “ip_address”: “
  8. remove all quotes and output to list.csv (I’m not finished yet)

Now I start separate sed statements because if I included them into one statement, the format wasn’t what I expected:

  1. Remove the first line of output
  2. Remove everything after and including “}”
  3. Join the VM name and IP address lines together
  4. Remove all lines with blank spaces

Below is an example of my output (Some vm’s did not include their ip. This has to do with VMware tools not being installed or running on the vm. I will follow that with the actual Ansible playbook):

COUNT DOOKU, 192.168.77.4
COUNT CHOCULA, 192.168.192.
COUNT VON COUNT - 123 AHHH HA HA, 192.168.3.192
COUNT DRACULA, 192.168.1.81
GREEDO, 192.168.3.3
POE, 192.168.3.8
TARKIN, 192.168.4.4
GENERAL GRIEVOUS, 192.168.3.7
GROGU - ITS BABY YODA FOOL, 192.168.3.159
DEATH STAR, 192.168.144.14
DEATH STAR 2, 192.168.192.15
BB-8, 192.168.1.199
JABBA, 192.168.192.86
FINN, 192.168.144.3.
MANDO,
TRAWN, 192.168.3.8
KIRK, 192.168.3.3
SPOCK, 192.168.3.176
DATA, 192.168.1.86
WORF, 192.168.5.7
PICARD, 192.168.1.178
RIKER, 192.168.19.84
McCOY, 192.168.9.81
La FORGE,
SCOTTY, 192.168.3.55
ARCHER, 192.168.19.99
RON BURGANDY, 192.168.144.3
SULU,
PIKE, 192.168.1.78
T'POL, 192.168.192.24
T-PAIN -LOL, 192.168.1.192
ENTERPRISE, 192.168.1.194
INTREPID, 192.168.3.49
USS VIRGINIA CGN-38, 192.168.8.38
USS LaSALLE AGF-3, 192.168.8.3
MISS PIGGY,
KERMIT THE FROG, 192.168.1.174
GONZO, 192.168.192.5
FOZZIE, 192.168.7.21
ANIMAL, 192.168.3.92
BEAKER, 192.168.192.26
ROWLF, 192.168.3.80
SCOOTER, 192.168.3.36
SAM EAGLE, 192.168.192.1
DR BUNSEN HONEYDEW,
STALER, 192.168.1.155
WALDORF, 192.168.3.58
SWEDISH CHEF - BORK BORK BORK, 192.168.1.3
PIGS IN SPACE, 192.168.1.48
RIZZO THE RAT, 192.168.144.16
FRANK RIZZO - LOL,
OSCAR, 192.168.4.9
BIG BIRD, 192.168.3.30
BERT, 192.168.5.45
ERNIE, 192.168.1.118
GROVER, 192.168.3.55
SNAKE EYES,
COBRA COMANDER, 192.168.3.3.
STORM SHADOW, 192.168.3.36
LADY JAYE, 192.168.3.5
BARONESS, 192.168.3.8
DUKE, 192.168.1.177
DESTRO, 192.168.3.1
SCARLETT, 192.25.160.1
FLINT, 192.168.3.17
HAWK, 192.168.192.1
BIG CHUCK,
LITTLE JOHN, 192.168.144.13
COOL GHOUL, 192.168.3.6
ZARTAN, 192.168.192.7
MEGATRON, 192.168.14.3
STARSCREAM, 192.168.1.185
ICE CREAM, 192.168.33.3
ME GRIMLOCK, 192.168.5.101
JAZ, 192.168.3.54
OPTIMUS PRIME, 192.168.3.53
IRONHIDE, 192.168.1.116
SOUNDWAVE - THE BEST, 192.168.1.136
KUP, 192.168.192.8
SLUDGE, 192.168.1.184
LASERBEAK, 192.168.192.14
BUMBLEBEE, 192.168.144.3
GRAPPLE, 192.168.3.1
SMOKESCREEN, 192.168.3.45
RUMBLE, 192.168.14.7
RAVAGE, 192.168.5.99
MAGNUM PI, 192.168.3.30
A-TEAM, 192.168.144.17
MR T - I PITTY THE FOOL, 192.168.6.3.
TRAP - ITS A TRAP, 192.168.1.135
MACGYVER,
THE DUKES OF HAZZARD, 192.168.1.136
BOSS HOG,
TOUR OF DUTY, 192.168.3.56
VOLTRON, 192.168.1.19
TIMMY, 192.168.1.15
JIMMY, 192.168.3.4
MR-HANKEY, 192.168.3.5
CARTMAN, 192.168.5.44
KENNY, 192.168.3.3
STAN, 192.168.19.168
KYLE, 192.168.5.60
TOLKIEN,
CHEF, 192.168.3.99
LIAN-CARTMAN, 192.168.1.3
THE-SCARY-MONSTER, 192.168.1.100
BEBE, 192.168.1.156
SHARON-MARSH, 192.168.1.101
TOWELIE,
LINDA-STOTCH, 192.168.192.3
GARY, 192.168.1.3
MR GARRISON, 192.168.12.3
BONO, 192.168.1.16
WENDY TESTABURGER, 192.168.192.8
ANAKIN, 192.168.3.37
DARTH VADER, 192.168.1.115
LUKE, 192.168.3.38
OBI-WAN, 192.168.1.49
HAN SOLO, 192.168.35.22
SHEEV, 192.168.3.33
LEA, 192.168.1.117
YODA, 192.168.192.168
CHEWBACA, 192.168.5.41
BOBA FETT, 192.168.192.11
JENGO-FETT, 192.168.3.58
R2-D2, 192.168.144.11
C-3PO, 192.168.22.45
STORM TROOPER, 192.168.4.3
SNOW TROOPER, 192.168.69.101
CLONE TROOPER,
SUPER TROOPERS - LOL, 192.168.99.100
REY, 192.168.192.4
LANDO,
PADME, 192.168.88.88
KYLO REN, 192.168.1.194
MACE WINDU - LIVES, 192.168.192.5
QUI-GON JIN,
GIN AND JUICE, 192.168.1.189
ADMIRAL ACKBAR,
DARTH MAUL, 192.168.3.41
AHSOKA TANO, 192.168.77.78

Here is the actual Ansible Playbook “VMWARE_list_all_powered-on_vms.yml”


---
- hosts: localhost
  vars:
    vcenter_hostname: vcenter.domain.local
    vcenter_user: ansibleuser@DOMAIN.LOCAL
    vcenter_pass: !vault |
          $ANSIBLE_VAULT;1.1;AES256
    
    esxhost: 192.168.1.101
    name: "{{ vm_name }}"
    notes: Ansible Test
    dumpfacts: False

  tasks:
  - name: Gather all VMs information
    vmware_vm_info:
      hostname: '{{ vcenter_hostname }}'
      username: '{{ vcenter_user }}'
      password: '{{ vcenter_pass }}'
      validate_certs: no
    register: all_vm_info
    delegate_to: localhost


  - name: Gather a list of all powered on VMs
    set_fact:
      on_vm: "{{ all_vm_info.virtual_machines | json_query(query) }}"
    vars:
      query: "[?power_state=='poweredOn']"
    register: jsoncontent

  - name: Gather a list of all powered on VM names
    debug: msg="{{ on_vm | json_query(jmesquery) }}"
    vars:
      jmesquery: "[*].{guest_name: guest_name, ip_address: ip_address}"


Ansible Automation: Gather list of all services on windows servers and clients

I had another audit request to gather all services on windows servers in an environment of about 70+ servers. I knew doing this through Ansible would be allot faster than going to each server individually. In the end it took less than 5 minutes to gather the services on 70+ servers.

When running the script I usually tee the output to text:

IE: ansible-playbook Audit_win_list_all_services.yml | tee /tmp/audit/Windows_services.txt

Here is my playbook:

Audit_win_list_all_services.yml

Ansible Automation: Gather list of all software installed on windows servers and clients

I had a request to gather all software installed on windows servers in an environment of about 70+ servers. I knew doing this through Ansible would be allot faster than going to each server individually. In the end it took less than 5 minutes to gather the installed software on 70+ servers.

I had seen a few playbooks online from other Ansible Admins doing this via Win32_Product, but I have seen warnings about using Win_32Product causing problems.

So after reading this article, I created the following playbook (I initially used a normal debug statement, but the output had allot of unnecessary info, so I split the output by newline and printed that list):

Below is my playbook:

win_list_all_programs.yml

Automating with Ansible: Adding new windows server clients to Prometheus/Grafana

I needed a way to install the Windows_Exporter on our Window systems as well as automating the configuration of the client in Prometheus. I came up with this Ansible playbook to handle this task. I’m sure there may be other ways of doing this and I’m always open to any suggestions. Here is what I have:

Playbooks (Can be downloaded):

win_install_prometheus.yml which calls install_prometheus_part2.yml

I imported a dashboard from Grafana.com, but at the time it only exported the older wmi_exporter. I was able to edit the dashboard and update it to work with the new exporter. Here is my dashboard (in JSON format for importing):

Move KVM VM vm to HyperV

Found the following online on Novell’s site and it worked perfectly (link below)

Copy the disk to a pc with virtualbox installed the convert with:

c:\Program Files\Oracle\VirtualBox>VBoxManage convertfromraw c:\be\disk0-test.ra

w test.vhd –format VHD

Then copy the vhd to hyperv and do it

1. Boot the target machine into rescue mode, using appropriate boot media

2. Run the following command to determine which devices are /, /boot and swap:

fdisk -l

3. Using that information mount the appropriate devices:

mount /dev/%root device% /mnt

mount /dev/%boot device% /mnt/boot

mount –rbind /proc /mnt/proc

mount –rbind /sys /mnt/sys

mount –rbind /dev /mnt/dev

4. Change the system root to the newly selected location:

chroot /mnt

5. Modify the /etc/fstab file to make sure the correct devices are being used.  For example the root, boot, and swap devices might be listed with these device names:

/dev/sda1

/dev/sda2

/dev/sda3

According to the fdisk -l output, however, the devices should be listed as follows:

/dev/cciss/c0d0p1

/dev/cciss/c0d0p2

/dev/cciss/c0d0p3

6. Modify the /boot/grub/menu.lst file and replace the boot partition information with the correct device id.  For example, the boot partition may be listed as this device:

/dev/sda2

Where it should be this device:

/dev/cciss/c0d0p2

7. Make sure that the /var/tmp directory exists and then run the following command (note: the /var/tmp directory may need to be manually created first):

mkinitrd

8. Reboot the target machine

From <https://www.novell.com/support/kb/doc.php?id=7009643>

Linux: Output your microphone to other computer’s speaker & vise-versa over SSH

$ dd if=/dev/dsp | ssh username@host dd of=/dev/dsp

The default sound device on Linux is /dev/dsp. It can be both written to and read from. If it’s read from then the audio subsystem will read the data from the microphone. If it’s written to, it will send audio to your speaker.

This one-liner reads audio from your microphone via the dd if=/dev/dsp command (if stands for input file) and pipes it as standard input to ssh. Ssh, in turn, opens a connection to a computer at host and runs the dd of=/dev/dsp (of stands for output file) on it. Dd of=/dev/dsp receives the standard input that ssh received from dd if=/dev/dsp. The result is that your microphone gets output on host computer’s speaker.

Want to scare your colleague? Dump /dev/urandom to his speaker by dd if=/dev/urandom.

It works the other way as well:
ssh -C user@ipaddress arecord -f dat -D plughw:1,0|aplay -f dat