Over the past few days I’ve been working to get a new CI/CD pipeline set up for the small business I work for. We decided on using Jenkins as it’s open source and fairly intuitive. I tested TeamCity for some time and while it was a very good tool, the simplicity of Jenkins was better for our implementation.

I started by setting up a VM that uses Ubuntu 18.04. This being my base VM and the company being small I determined that allocating the following was sufficient for our uses (keep in mind this will also house SonarQube so I made the specs a bit more robust):

CPU : 4 cores
RAM: 8GB
Storage: 100GB

After the initial install and setup I installed a few items that would be needed or nice to have.

apt-get update;
apt-get -yf upgrade; 
apt-get -y autoremove;
apt-get -y install nano wget curl openjdk-11-jdk ant

Then I simply used the deb installer for Jenkins. You’ll need to have “wget” installed and then use the following below to get setup.

wget -q -O - http://pkg.jenkins-ci.org/debian/jenkins-ci.org.key | sudo apt-key add -
;

sudo sh -c 'echo deb https://pkg.jenkins.io/debian-stable binary/ > /etc/apt/sources.list.d/jenkins.list';

apt-get update;apt-get install -y jenkins;

systemctl start jenkins; systemctl enable jenkins;

If everything has gone well we can check to see if Jenkins is up.

systemctl status jenkins;

Assuming that it is we can reach the web interface, but before we do we need to grab the administrator key first.

cat /var/lib/jenkins/secrets/initialAdminPassword
;

Armed with this we can go to the web browser and put in the following:

https://<your_machines_ip>:8080
example: https://192.168.23.7

You’ll land on a screen that will ask you to login using the password we just pulled. Use that password to login and install plugins you want to use. Install as needed (this will take some time) and once finished you will be asked to setup a new password for the system. Once done, I was able to get started on a freestyle project using Jenkins.

Start by clicking on “New Item”

Next you need to input a name for your freestyle project. I’m using Whiskey Mellon as an example but you likely have a name that you use internally. Later on I’ll use Cylindra as the project name.

Once you input the project name, click on Freestyle Project. Now we can setup the actual build steps.

General Tab

We start in the general tab. You can input a description if you want. Pretty self explanatory. Next is the logs. I decided that 30 days was sufficient for me. We use Github now after having used Apache Subversion and the project I’m going to pull in will be Cylindra.

Source Code Management (SCM)

The next section is our SCM. Setting up a pull to the SCM isn’t hard but it does take an extra step.

First go to Github where your project is located. Create a token for this project. I won’t cover this but here is a link to the section you need to be in and a screenshot of the permissions I assigned. (https://github.com/settings/tokens)

Now to set up our pull with our shiny new token. Pattern your pull after this:

https://<token>@github.com/<yourorgname>/cylindra.git

The default for many projects is Master or Main. You can setup as many branches as you want. I’m going to leave mine as Master for demonstration purposes.

Build Triggers

There are quite a few options available here but we ran into an issue. Since we didn’t set this up in the cloud and we run Github in the cloud getting the data to the on premise setup was going to be hard. We couldn’t setup the pull for each commit which normally would be setup via a webhook but we were able to set up the following under “Build Periodically”

TZ=America/New_York
#create a nightly build schedule
H 17 * * *
#create a multitime build schedule
#H H(9-16)/2 * * 1-5

Being a small business we don’t have many engineers committing code. If we did, then this would have to be done differently. Instead this will pull once a day and run a build. Treat this section like a Cronjob essentially.

Build Environment

This section is straight forward for us as we’re building with Apache Ant.

Build

Finally we have gotten to the build section. Keep in mind this build section is handled by a separate build server or node. We cover the setup for this in “How to Create a Distributed Build System to use with Jenkins”. The next steps will make sense from that context.

I assume that you either know how to use Apache Ant or you have had Eclipse (for instance) create a build.xml file. Our setup is a bit more challenging in that we have two directories we must build from. Making things worse by default Jenkins (unlike TeamCity) pulls down the entire repo. This is a one time deal (unless you blow away the downloaded repo each time) and does take into account changes made to the Master or Main as well as any branches that are listed.

So our steps will likely be different from someone with a ‘mono repo’ or a repo with submodules. If however you’ve stepped into a setup like the one I’m demonstrating you will likely find this very illuminating.

I’d like to note that while we use Apache Ant and Ant can do many many things, we’re not purists. We use Ant when it makes sense and we use Shell scripting to handle the rest. This is primarily for speed and because, again, we’re not purists.

Shell

rm -rf /var/lib/jenkins/workspace/Cylindra/CAM1/bin /var/lib/jenkins/workspace/Cylindra/build /var/lib/jenkins/workspace/Cylindra/CAM1/build;

Ant

cleanall

Shell

mkdir /var/lib/jenkins/workspace/Cylindra/CAM1/bin;
mkdir /var/lib/jenkins/workspace/Cylindra/WX/bin;

Ant

jar

Shell

mv /var/lib/jenkins/workspace/Cylindra/CAM1/cylindra.jar /var/lib/jenkins/workspace/Cylindra/CAM1/build/cylindra.jar;

Finish

That’s pretty much it. Now you can save the project and force a manual run. Hopefully this helps make sense of the basics for setting up a Java project using Github as the SCM and Apache Ant as the compiler.

Introduction

Recently I had to setup a deployment system from scratch. In the world of road side units and other DOT roadside devices firmware updates and patch deployments can be rough. Security is usually taken very seriously and getting access to the network segment for the devices you care for can be difficult to outright impossible.

To make matters more difficult for the maintainer many times there is no mass package deployment in place. Such was the case I ran into.

Disclaimer

I’m a strong proponent of testing before implementation. Therefore, I do not take any responsibility for your use of this information on your systems and the consequences that will ensue (good or bad). Remember the golden rules of IT:

1) Have a backup plan
2) Have a data backup

Follow these rules, and you will successfully recover most of the time.

Tools

This script specifically targets road side units however you can utilize these same principles for a variety of other projects.

1. Shell, as I work out of the terminal 98% of the time I use native commands in shell, preferably BASH when I can. This is not the best way (python would be better here actually).
2. Windows Subsystem Linux (you do not have to use this but I did and my scripts reflect this). I used Debian but there are other flavors that will work as well. Alpine, Busybox, etc will not be ideal choices for this exercise.
3. Install Python3
4. Install PSSH (uses python3), PSCP, etc
5. Install Curl, WGET, gzip

Picking up from Deployment Script I, this is where we get to use the cool PSSH, PSCP, and PNUKE tools.

PSSH

Let’s start with PSSH. With this you can connect to multiple devices via ssh at one time. Better than that you can use a key setup that will avoid having to type the password each time you run the command. The first step you will need for any of these tools is a simple text file filled with IP’s and the correct ssh port.

1.1.1.1:22
2.2.2.2:2222
3.3.3.3:22

You can name this file what you like but keep it short because we’ll use it later. Let’s define a function that will allow me to call a Rest API that will start a software function for connected vehicles.

startEtrans () {
if [[ $location -eq 1 ]]; then
ip_loc="/usr/local/bin/flagstaff_connects.txt"
elif [[ $location -eq 2 ]]; then
ip_loc="/usr/local/bin/rochester_connects.txt"
elif [[ $location -eq 3 ]]; then
ip_loc="/usr/local/bin/salem_connects.txt"
fi
echo "Start Etrans"
pssh -h $ip_loc -l root -i "-o StrictHostKeyChecking=no" "curl -s -o /dev/null -u 1Xmly02xivjsre:1Xmly02xivjsre http://localhost/apps/si?start=Start"
}

Let’s dissect this. First I start out with a series of “if” statements. If you remember part one we setup some case logic to determine what place we were working on. This simply checks the response of that function using numbers. Now, this is not the best way to do this. If the script gets really big figuring out what number goes where will get complicated. For small, quick, and dirty scripts this will work fine though.

At this point I set a variable for the text file filled with IP’s and ports that we set up earlier. Then the fun part. We call the pssh command. The “-h” switch takes the list of IP’s. Keep in mind this uses multi-threading so it is advised to keep the amount of IP’s limited. A specific number is not given in general likely as it depends on your network and computing equipment.

The next switch “-l” sets the user name. If you have keys for root already installed this is an easy way to keep things clean. it’s also the reason we are not use the “-A” switch. You need that switch if you’re running keyless and intend on putting in the password for the command.

The next part takes into account if the key has not been stored into your system before. If you don’t take this into account then the commands will fail.

Finally we run our command on multiple devices, at the same time. The neat thing is we can run chained commands or scripts. How to get the scripts on the device? Well, with PSCP of course.

PSCP

PSCP is known for being included with the Putty software. It is also included as part of the PSSH python package. This works in the same way as PSSH by allowing you to copy packages to multiple devices in much the same way. Let’s take a look at another function.

copySNMPScript() {
clear;
echo "########################################"
echo "Beginning SNMP Script Copy"
ip_loc="/usr/local/bin/rochester_connects.txt"
cd /mnt/c/Users/RMath/connects/snmp_scripts/;
echo "Copy over script"
pscp -A -h $ip_loc -l root -x "-o StrictHostKeyChecking=no" snmp_relaunch.sh /usr/bin/
echo "Fix Script Permissions and set in background"
pssh -A -h $ip_loc -l root -i -x "-o StrictHostKeyChecking=no" "cd /usr/bin/; chmod 755 snmp_relaunch.sh;"
echo "Reboot Device"
pssh -Av -h $ip_loc -p 1 -l root -x "-o StrictHostKeyChecking=no" "killall PT_Proxy"
echo "Tasks completed. Check for errors."
echo "########################################"
}

This function has a lot going on in it. We call PSSH and PSCP to copy over and fix permissions on the snmp script. Specifically though we’ll focus on PSCP. This time since we don’t have a key on the device we have to tell PSCP that it must ask us for the password. For each command we run with a “-A” switch we will be forced to input the password. The rest of it we just ran through. At the end of the day it basically works like SCP, just on a larger scale.

PNUKE

The final command we will run is PNUKE. This is useful for killing services. Not much is said about this command online but I found it works a lot like the “kill -9 <pid>” command. Below is another function with an example of PNUKE usage. Basically it searches the services for the item you’re looking for and applies a “kill -9” command.

connectEtrans() {
clear;
echo "########################################"
echo "Beginning Connect:ITS Etrans Upgrade Deployment Process"
if [[ $location -eq 1 ]]; then
ip_loc="/usr/local/bin/flagstaff_connects.txt"
elif [[ $location -eq 2 ]]; then
ip_loc="/usr/local/bin/rochester_connects.txt"
elif [[ $location -eq 3 ]]; then
ip_loc="/usr/local/bin/salem_connects.txt"
fi
cd /mnt/c/Users/RMath/OneDrive\ /Etrans/$version;
echo "Copy over Etrans"
pscp -h $ip_loc -l root -x "-o StrictHostKeyChecking=no" kapschrcu-connectits-$version.gz /tmp/
echo "Unzip"
pssh -h $ip_loc -l root -i -x "-o StrictHostKeyChecking=no" "sed -i 's/1/0/g' /etc/apt/apt.conf.d/20auto-upgrades;cat /etc/apt/apt.conf.d/20auto-upgrades;"
pssh -h $ip_loc -l root -i -x "-o StrictHostKeyChecking=no" "gunzip /tmp/etrans-connectits-$version.gz"
echo "Kill etrans process"
pnuke -h $ip_loc -l root -x "-o StrictHostKeyChecking=no" "etransrsu"
echo "Install new etrans"
pssh -h $ip_loc -l root -i -x "-o StrictHostKeyChecking=no" "rm -rf /opt/etrans/etransrsu; mv /tmp/etrans-connectits-$version /opt/etrans/etransrsu; chmod 755 /opt/etrans/etransrsu;"
echo "Clean up"
pssh -h $ip_loc -l root -i -x "-o StrictHostKeyChecking=no" "rm -rf /tmp/*"
echo "Restart Etrans"
pssh -h $ip_loc -l root -i "-o StrictHostKeyChecking=no" "curl -s -o /dev/null -u 1Xmly02xivjsre:1Xmly02xivjsre http://localhost/apps/si?start=Start"
echo "Tasks completed. Check for errors."
echo "########################################"
}

That’s it for our walk through on setting up a deployment script. Using PSSH and PSCP you can make a rudimentary deployment service for immature environments that don’t support agents or places you cannot place keys (embedded systems, really poorly run IT environments with broken deployment systems requiring manual installs, or small business applications). This is better built directly in python but for a quick and dirty setup it’s hard to beat a Windows Subsystem Linux setup, OneDrive, and a nice deployment bash script.

Introduction

Recently I had to setup a deployment system from scratch. In the world of road side units and other DOT roadside devices firmware updates and patch deployments can be rough. Security is usually taken very seriously and getting access to the network segment for the devices you care for can be difficult to outright impossible.

To make matters more difficult for the maintainer many times there is no mass package deployment in place. Such was the case I ran into.

Disclaimer

I’m a strong proponent of testing before implementation. Therefore, I do not take any responsibility for your use of this information on your systems and the consequences that will ensue (good or bad). Remember the golden rules of IT:

1) Have a backup plan
2) Have a data backup

Follow these rules, and you will successfully recover most of the time.

Tools

This script specifically targets road side units however you can utilize these same principles for a variety of other projects.

1. Shell, as I work out of the terminal 98% of the time I use native commands in shell, preferably BASH when I can. This is not the best way (python would be better here actually).
2. Windows Subsystem Linux (you do not have to use this but I did and my scripts reflect this). I used Debian but there are other flavors that will work as well. Alpine, Busybox, etc will not be ideal choices for this exercise.
3. Install Python3
4. Install PSSH (uses python3), PSCP, etc
5. Install Curl, WGET, gzip

Beginning the Script

I always start my scripts with variables

#!/bin/bash
#########################################
# Script Name: Deployment System
# Date:        1/3/2021
# Author:      Robert Mathis
#########################################

#########################################
# Variables
#########################################

version='1.2.3'
container_image='https://microsoft_one_drive&download=1'
answer=1;

If you’ve not worked with scripting before, don’t fear, variables are fun! You can stick useful bits into them, often things that repeat throughout your script that would be a pain to change by hand. Of course there are other uses for variables but for now just think of them as boxes or containers.

Case Logic

Next we go right for the jugular with some basic questions. To do this we’re going to create some functions.

#########################################
# Functions
#########################################

locationsetup() {
while true; do
clear
echo "Upgrade System for Somewhere"
echo "This upgrade provided by Something"
echo "########################################"
echo ""
echo "Location Selection"
echo "########################################"
echo "1 Flagstaff"
echo "2 Rochester"
echo "3 Salem"
echo "########################################"
read -p "Where are we upgrading? Enter a number: " location
echo ""
  read -r -p "Is location $location correct?? [y/n]" answer
  case "$answer" in
        [Yy][Ee][Ss]|[Yy]) # Yes or Y (case-insensitive).
        return 0
        ;;
      *) # Anything else (including a blank) is invalid.
        ;;
  esac
done
}

deploymentsetup() {
while true; do
clear
echo ""
echo "Deployment Type"
echo "########################################"
echo "1 Connect:ITS Something"
if [[ $location -eq 2 ]];
then
echo "2 CVCP Something"
echo "3 VCCU Something"
fi
echo "########################################"
read -p "Enter the number of the deployment you would like to complete: " deployType
echo ""
  read -r -p "Is deployment type $deployType correct? [y/n]" answer
  case "$answer" in
        [Yy][Ee][Ss]|[Yy]) # Yes or Y (case-insensitive).
        return 0
        ;;
      *) # Anything else (including a blank) is invalid.
        ;;
  esac
done
}

The first thing you might notice is that we start with a function. Something like this:

function () {}

We can put arguments in the function if we want but what we’re after is some simple answers to some questions. The idea being to automate this process as much as possible.

We use a “while” loop to kick off both of our functions. The while loop has one purpose. It’s to ensure that if an answer is not typed in correctly the user of the script can retype their new answer in before proceeding. To make the while loop work we set a variable at the beginning called “answer”. If “yes” is not specified a 1 is returned. The loop will start over again until a 0 is returned which would be a successful function exit.

One thing to remember is that when checking against integers as opposed to strings (numbers verses words) double brackets need to be used for if statements. Also the “-eq” operator as opposed to the “==” operator needs to be used. The rest is fairly self explanatory and fairly reusable. To call the function simply invoke it like so:

#########################################
#Execution
#########################################

locationsetup; deploymentsetup;

Because we did not have arguments for the function there is no need for anything further. But if we did have arguments they would look like the following:

snmp_array_walker() {
  arr=("$@");
  for x in "${arr[@]}";
    do
       echo "Working on OID $x";
       snmpget -v 2c -c public $ip $x;
       echo " ";
       sleep 1;
    done;
}

In this script the function is expecting an array to be passed to it. In the world of shell you pass the argument in the following way:

snmp_array_walker "${array1[@]}"

You may not realize this but many times in Alpine or older Debian (9 and prior) versions calling something like the following:

service mysql status

Is the equivalent to calling a function with an argument. In fact if you were to go about it this way it would look far more familiar perhaps:

/etc/init.d/mysql status

In this case we’ve simply passed one of the function arguments to the service.

Going back to the earlier example with the function and the array. What happened here was we called the function and then passed one of the arrays to it. The argument is placed beside the function. There can be as many arguments as needed. In this case this is a special way to pass an array to the function. Basically I’ve requested the array1 variable and have called every item of the array to be passed to the function.

Stay tuned for part two when we actually get to walk through some other useful functions and if statements.

I write technical documentation for my place of employ and I am accustomed to being able to utilize the full features of Microsoft Word for instance. Anyone who has ever had to do formatting or adding a table with the Office 365 online web application can attest that the lacking features make the online app at best, a lite version.

Enter WinApps by Fmstrat (https://github.com/Fmstrat/winapps). The idea behind WinApps is to create the reverse of the Windows Subsystem for Linux. With a valid Windows license (or by perhaps using the developers ISO which will expire every 60 days), you can set up your own Linux Subsystem for Windows. Designed for Ubuntu and Fedora the WinApps project will sweep the VM you setup looking for officially supported applications like an installed version of Office 365 for instance. With community powered logos available, WinApps will do the heavy lifting to “install” the logos and pathing on your system so that they are available to click on and use right from your Linux menu! So when you click on “cmd” you will actually see a Windows Command Prompt appear on your Linux desktop.

I liked this idea but I’m using Alpine. Ruh Roh Raggy, that’s a problem. Based on musl libc and busybox I wasn’t sure that this project would work. Turns out, you can make the project work just fine.

The instructions that are provided are really quite excellent. Scary things like KVM and QEMU are handled easily using Virt-Manager for initial VM setup and there is even a walk through for that. Wonderful. Really great project documentation.

There are some pitfalls though, especially since we are using Alpine which means we’re not of the systemd persuasion. So first things first, ensure you are running a desktop manager like XFCE. Otherwise the exercise is moot to start out with (ok maybe you could do this with Xserver only but why suffer that pain?). Then make sure your repositories file has edge repo listings in it like the example below. If not you can copy the ones from my file and place into yours.

cat /etc/apk/repositories
....outputs _>

#/media/usb/apks
http://dl-cdn.alpinelinux.org/alpine/v3.12/main
http://dl-cdn.alpinelinux.org/alpine/v3.12/community
http://dl-cdn.alpinelinux.org/alpine/edge/main
http://dl-cdn.alpinelinux.org/alpine/edge/community
http://dl-cdn.alpinelinux.org/alpine/edge/testing

<_ends_outputs....

Next make sure that at least the following are installed first and ensure that libvirtd starts at boot.

apk --no-cache add git freerdp virt-manager xf86-video-qxl libvirt qemu dbus;
rc-service libvirtd start;
rc-update add libvirtd;

Next we need to ensure that your user is added to the correct groups.

adduser <yourusername> libvirt;
adduser <yourusername> kvm;

The biggest pitfall is to come though. After going through all of that setup you find that you still can’t start your VM using virsh. This was a major stumbling block for me. Luckily the answer is simple.

First edit the libvirtd file:

nano /etc/libvirt/libvirt.conf

Uncomment the following line: “uri default – ‘qemu:///system'” and save the file.

The other part of this pitfall is that you must now copy this same file to the following place:

cp /etc/libvirt/libvirt.conf ~/.config/libvirt/

What this does is it allows your user to start the VM from the terminal successfully without using “sudo su”. This in turn means you can click on the icons from the desktop manager menu.

And now they will display as you expect.

The other issue you may run into is getting “tun” to load automatically for the bridged network setup. To fix this issue on Alpine you’ll want to do the following:

nano /etc/modules;

Add “tun” to the file and save. Reboot your system.

At this point the system should come up automatically when you log in and you should be able to click as needed on the commands.

Finally, here is what you can expect by clicking on those shiny new icons. Enjoy!