Recently I had to setup a deployment system from scratch. In the world of road side units and other DOT roadside devices firmware updates and patch deployments can be rough. Security is usually taken very seriously and getting access to the network segment for the devices you care for can be difficult to outright impossible.

To make matters more difficult for the maintainer many times there is no mass package deployment in place. Such was the case I ran into.

Disclaimer

I’m a strong proponent of testing before implementation. Therefore, I do not take any responsibility for your use of this information on your systems and the consequences that will ensue (good or bad). Remember the golden rules of IT:

1) Have a backup plan
2) Have a data backup

Follow these rules, and you will successfully recover most of the time.

Tools

This script specifically targets road side units however you can utilize these same principles for a variety of other projects.

1. Shell, as I work out of the terminal 98% of the time I use native commands in shell, preferably BASH when I can. This is not the best way (python would be better here actually).
2. Windows Subsystem Linux (you do not have to use this but I did and my scripts reflect this). I used Debian but there are other flavors that will work as well. Alpine, Busybox, etc will not be ideal choices for this exercise.
3. Install Python3
4. Install PSSH (uses python3), PSCP, etc
5. Install Curl, WGET, gzip

Picking up from Deployment Script I, this is where we get to use the cool PSSH, PSCP, and PNUKE tools.

PSSH

Let’s start with PSSH. With this you can connect to multiple devices via ssh at one time. Better than that you can use a key setup that will avoid having to type the password each time you run the command. The first step you will need for any of these tools is a simple text file filled with IP’s and the correct ssh port.

1.1.1.1:22
2.2.2.2:2222
3.3.3.3:22

You can name this file what you like but keep it short because we’ll use it later. Let’s define a function that will allow me to call a Rest API that will start a software function for connected vehicles.

startEtrans () {
if [[ $location -eq 1 ]]; then
ip_loc="/usr/local/bin/flagstaff_connects.txt"
elif [[ $location -eq 2 ]]; then
ip_loc="/usr/local/bin/rochester_connects.txt"
elif [[ $location -eq 3 ]]; then
ip_loc="/usr/local/bin/salem_connects.txt"
fi
echo "Start Etrans"
pssh -h $ip_loc -l root -i "-o StrictHostKeyChecking=no" "curl -s -o /dev/null -u 1Xmly02xivjsre:1Xmly02xivjsre http://localhost/apps/si?start=Start"
}

Let’s dissect this. First I start out with a series of “if” statements. If you remember part one we setup some case logic to determine what place we were working on. This simply checks the response of that function using numbers. Now, this is not the best way to do this. If the script gets really big figuring out what number goes where will get complicated. For small, quick, and dirty scripts this will work fine though.

At this point I set a variable for the text file filled with IP’s and ports that we set up earlier. Then the fun part. We call the pssh command. The “-h” switch takes the list of IP’s. Keep in mind this uses multi-threading so it is advised to keep the amount of IP’s limited. A specific number is not given in general likely as it depends on your network and computing equipment.

The next switch “-l” sets the user name. If you have keys for root already installed this is an easy way to keep things clean. it’s also the reason we are not use the “-A” switch. You need that switch if you’re running keyless and intend on putting in the password for the command.

The next part takes into account if the key has not been stored into your system before. If you don’t take this into account then the commands will fail.

Finally we run our command on multiple devices, at the same time. The neat thing is we can run chained commands or scripts. How to get the scripts on the device? Well, with PSCP of course.

PSCP

PSCP is known for being included with the Putty software. It is also included as part of the PSSH python package. This works in the same way as PSSH by allowing you to copy packages to multiple devices in much the same way. Let’s take a look at another function.

copySNMPScript() {
clear;
echo "########################################"
echo "Beginning SNMP Script Copy"
ip_loc="/usr/local/bin/rochester_connects.txt"
cd /mnt/c/Users/RMath/connects/snmp_scripts/;
echo "Copy over script"
pscp -A -h $ip_loc -l root -x "-o StrictHostKeyChecking=no" snmp_relaunch.sh /usr/bin/
echo "Fix Script Permissions and set in background"
pssh -A -h $ip_loc -l root -i -x "-o StrictHostKeyChecking=no" "cd /usr/bin/; chmod 755 snmp_relaunch.sh;"
echo "Reboot Device"
pssh -Av -h $ip_loc -p 1 -l root -x "-o StrictHostKeyChecking=no" "killall PT_Proxy"
echo "Tasks completed. Check for errors."
echo "########################################"
}

This function has a lot going on in it. We call PSSH and PSCP to copy over and fix permissions on the snmp script. Specifically though we’ll focus on PSCP. This time since we don’t have a key on the device we have to tell PSCP that it must ask us for the password. For each command we run with a “-A” switch we will be forced to input the password. The rest of it we just ran through. At the end of the day it basically works like SCP, just on a larger scale.

PNUKE

The final command we will run is PNUKE. This is useful for killing services. Not much is said about this command online but I found it works a lot like the “kill -9 <pid>” command. Below is another function with an example of PNUKE usage. Basically it searches the services for the item you’re looking for and applies a “kill -9” command.

connectEtrans() {
clear;
echo "########################################"
echo "Beginning Connect:ITS Etrans Upgrade Deployment Process"
if [[ $location -eq 1 ]]; then
ip_loc="/usr/local/bin/flagstaff_connects.txt"
elif [[ $location -eq 2 ]]; then
ip_loc="/usr/local/bin/rochester_connects.txt"
elif [[ $location -eq 3 ]]; then
ip_loc="/usr/local/bin/salem_connects.txt"
fi
cd /mnt/c/Users/RMath/OneDrive\ /Etrans/$version;
echo "Copy over Etrans"
pscp -h $ip_loc -l root -x "-o StrictHostKeyChecking=no" kapschrcu-connectits-$version.gz /tmp/
echo "Unzip"
pssh -h $ip_loc -l root -i -x "-o StrictHostKeyChecking=no" "sed -i 's/1/0/g' /etc/apt/apt.conf.d/20auto-upgrades;cat /etc/apt/apt.conf.d/20auto-upgrades;"
pssh -h $ip_loc -l root -i -x "-o StrictHostKeyChecking=no" "gunzip /tmp/etrans-connectits-$version.gz"
echo "Kill etrans process"
pnuke -h $ip_loc -l root -x "-o StrictHostKeyChecking=no" "etransrsu"
echo "Install new etrans"
pssh -h $ip_loc -l root -i -x "-o StrictHostKeyChecking=no" "rm -rf /opt/etrans/etransrsu; mv /tmp/etrans-connectits-$version /opt/etrans/etransrsu; chmod 755 /opt/etrans/etransrsu;"
echo "Clean up"
pssh -h $ip_loc -l root -i -x "-o StrictHostKeyChecking=no" "rm -rf /tmp/*"
echo "Restart Etrans"
pssh -h $ip_loc -l root -i "-o StrictHostKeyChecking=no" "curl -s -o /dev/null -u 1Xmly02xivjsre:1Xmly02xivjsre http://localhost/apps/si?start=Start"
echo "Tasks completed. Check for errors."
echo "########################################"
}

That’s it for our walk through on setting up a deployment script. Using PSSH and PSCP you can make a rudimentary deployment service for immature environments that don’t support agents or places you cannot place keys (embedded systems, really poorly run IT environments with broken deployment systems requiring manual installs, or small business applications). This is better built directly in python but for a quick and dirty setup it’s hard to beat a Windows Subsystem Linux setup, OneDrive, and a nice deployment bash script.

Recently I had to setup a deployment system from scratch. In the world of road side units and other DOT roadside devices firmware updates and patch deployments can be rough. Security is usually taken very seriously and getting access to the network segment for the devices you care for can be difficult to outright impossible.

To make matters more difficult for the maintainer many times there is no mass package deployment in place. Such was the case I ran into.

Disclaimer

I’m a strong proponent of testing before implementation. Therefore, I do not take any responsibility for your use of this information on your systems and the consequences that will ensue (good or bad). Remember the golden rules of IT:

1) Have a backup plan
2) Have a data backup

Follow these rules, and you will successfully recover most of the time.

Tools

This script specifically targets road side units however you can utilize these same principles for a variety of other projects.

1. Shell, as I work out of the terminal 98% of the time I use native commands in shell, preferably BASH when I can. This is not the best way (python would be better here actually).
2. Windows Subsystem Linux (you do not have to use this but I did and my scripts reflect this). I used Debian but there are other flavors that will work as well. Alpine, Busybox, etc will not be ideal choices for this exercise.
3. Install Python3
4. Install PSSH (uses python3), PSCP, etc
5. Install Curl, WGET, gzip

Beginning the Script

I always start my scripts with variables

#!/bin/bash
#########################################
# Script Name: Deployment System
# Date:        1/3/2021
# Author:      Robert Mathis
#########################################

#########################################
# Variables
#########################################

version='1.2.3'
container_image='https://microsoft_one_drive&download=1'
answer=1;

If you’ve not worked with scripting before, don’t fear, variables are fun! You can stick useful bits into them, often things that repeat throughout your script that would be a pain to change by hand. Of course there are other uses for variables but for now just think of them as boxes or containers.

Case Logic

Next we go right for the jugular with some basic questions. To do this we’re going to create some functions.

#########################################
# Functions
#########################################

locationsetup() {
while true; do
clear
echo "Upgrade System for Somewhere"
echo "This upgrade provided by Something"
echo "########################################"
echo ""
echo "Location Selection"
echo "########################################"
echo "1 Flagstaff"
echo "2 Rochester"
echo "3 Salem"
echo "########################################"
read -p "Where are we upgrading? Enter a number: " location
echo ""
  read -r -p "Is location $location correct?? [y/n]" answer
  case "$answer" in
        [Yy][Ee][Ss]|[Yy]) # Yes or Y (case-insensitive).
        return 0
        ;;
      *) # Anything else (including a blank) is invalid.
        ;;
  esac
done
}

deploymentsetup() {
while true; do
clear
echo ""
echo "Deployment Type"
echo "########################################"
echo "1 Connect:ITS Something"
if [[ $location -eq 2 ]];
then
echo "2 CVCP Something"
echo "3 VCCU Something"
fi
echo "########################################"
read -p "Enter the number of the deployment you would like to complete: " deployType
echo ""
  read -r -p "Is deployment type $deployType correct? [y/n]" answer
  case "$answer" in
        [Yy][Ee][Ss]|[Yy]) # Yes or Y (case-insensitive).
        return 0
        ;;
      *) # Anything else (including a blank) is invalid.
        ;;
  esac
done
}

The first thing you might notice is that we start with a function. Something like this:

function () {}

We can put arguments in the function if we want but what we’re after is some simple answers to some questions. The idea being to automate this process as much as possible.

We use a “while” loop to kick off both of our functions. The while loop has one purpose. It’s to ensure that if an answer is not typed in correctly the user of the script can retype their new answer in before proceeding. To make the while loop work we set a variable at the beginning called “answer”. If “yes” is not specified a 1 is returned. The loop will start over again until a 0 is returned which would be a successful function exit.

One thing to remember is that when checking against integers as opposed to strings (numbers verses words) double brackets need to be used for if statements. Also the “-eq” operator as opposed to the “==” operator needs to be used. The rest is fairly self explanatory and fairly reusable. To call the function simply invoke it like so:

#########################################
#Execution
#########################################

locationsetup; deploymentsetup;

Because we did not have arguments for the function there is no need for anything further. But if we did have arguments they would look like the following:

snmp_array_walker() {
  arr=("$@");
  for x in "${arr[@]}";
    do
       echo "Working on OID $x";
       snmpget -v 2c -c public $ip $x;
       echo " ";
       sleep 1;
    done;
}

In this script the function is expecting an array to be passed to it. In the world of shell you pass the argument in the following way:

snmp_array_walker "${array1[@]}"

You may not realize this but many times in Alpine or older Debian (9 and prior) versions calling something like the following:

service mysql status

Is the equivalent to calling a function with an argument. In fact if you were to go about it this way it would look far more familiar perhaps:

/etc/init.d/mysql status

In this case we’ve simply passed one of the function arguments to the service.

Going back to the earlier example with the function and the array. What happened here was we called the function and then passed one of the arrays to it. The argument is placed beside the function. There can be as many arguments as needed. In this case this is a special way to pass an array to the function. Basically I’ve requested the array1 variable and have called every item of the array to be passed to the function.

Stay tuned for part two when we actually get to walk through some other useful functions and if statements.

Enter WinApps by Fmstrat (https://github.com/Fmstrat/winapps). The idea behind WinApps is to create the reverse of the Windows Subsystem for Linux. With a valid Windows license (or by perhaps using the developers ISO which will expire every 60 days), you can set up your own Linux Subsystem for Windows. Designed for Ubuntu and Fedora the WinApps project will sweep the VM you setup looking for officially supported applications like an installed version of Office 365 for instance. With community powered logos available, WinApps will do the heavy lifting to “install” the logos and pathing on your system so that they are available to click on and use right from your Linux menu! So when you click on “cmd” you will actually see a Windows Command Prompt appear on your Linux desktop.

I liked this idea but I’m using Alpine. Ruh Roh Raggy, that’s a problem. Based on musl libc and busybox I wasn’t sure that this project would work. Turns out, you can make the project work just fine.

The instructions that are provided are really quite excellent. Scary things like KVM and QEMU are handled easily using Virt-Manager for initial VM setup and there is even a walk through for that. Wonderful. Really great project documentation.

There are some pitfalls though, especially since we are using Alpine which means we’re not of the systemd persuasion. So first things first, ensure you are running a desktop manager like XFCE. Otherwise the exercise is moot to start out with (ok maybe you could do this with Xserver only but why suffer that pain?). Then make sure your repositories file has edge repo listings in it like the example below. If not you can copy the ones from my file and place into yours.

cat /etc/apk/repositories
....outputs _>

#/media/usb/apks
http://dl-cdn.alpinelinux.org/alpine/v3.12/main
http://dl-cdn.alpinelinux.org/alpine/v3.12/community
http://dl-cdn.alpinelinux.org/alpine/edge/main
http://dl-cdn.alpinelinux.org/alpine/edge/community
http://dl-cdn.alpinelinux.org/alpine/edge/testing

<_ends_outputs....

Next make sure that at least the following are installed first and ensure that libvirtd starts at boot.

apk --no-cache add git freerdp virt-manager xf86-video-qxl libvirt qemu dbus;
rc-service libvirtd start;
rc-update add libvirtd;

Next we need to ensure that your user is added to the correct groups.

adduser <yourusername> libvirt;
adduser <yourusername> kvm;

The biggest pitfall is to come though. After going through all of that setup you find that you still can’t start your VM using virsh. This was a major stumbling block for me. Luckily the answer is simple.

First edit the libvirtd file:

nano /etc/libvirt/libvirt.conf

Uncomment the following line: “uri default – ‘qemu:///system'” and save the file.

The other part of this pitfall is that you must now copy this same file to the following place:

cp /etc/libvirt/libvirt.conf ~/.config/libvirt/

What this does is it allows your user to start the VM from the terminal successfully without using “sudo su”. This in turn means you can click on the icons from the desktop manager menu.

And now they will display as you expect.

The other issue you may run into is getting “tun” to load automatically for the bridged network setup. To fix this issue on Alpine you’ll want to do the following:

nano /etc/modules;

Add “tun” to the file and save. Reboot your system.

At this point the system should come up automatically when you log in and you should be able to click as needed on the commands.

Finally, here is what you can expect by clicking on those shiny new icons. Enjoy!