Building with Docker 1 of 2

Introduction

If you’ve used Docker for a little while (and have not had the need to use Kubernetes yet) and you’ve wondered what you need to do to make your own image, you’re in luck. I recently had a project where we explored that very thing. This post will give you a basic overview of some of the things you need to do as well as my personal docker build command set up, and in my next post in the series, we’ll look at an actual example of a build. For bonus points, we’ll look at a situation where if someone does connect to the container, they are not root, and to make it more interesting, cannot easily become root.

This post is targeted to those who might use microservices in their development environment or have a special need to do just a little bit more with Docker and not work with Kubernetes. Mostly that covers special projects, startups, small businesses, or personal growth projects. I’m fully and highly aware that there are different, better ways to utilize containers and microservices. If you’re a Kubernetes superhero, great, but this post isn’t likely to give you the twingy tingles. Just a word of warning. 🙂

Disclaimer

I’m a strong proponent of testing before implementation. Therefore, I do not take any responsibility for your use of this script on your systems and the consequences that will ensue (good or bad). Remember the golden rules of IT:

1) Have a backup plan
2) Have a data backup

Follow these rules, and you will successfully recover most of the time.

Needed Tools and Assumptions

Before we get started there are some tools we’ll need:
1) Docker for Windows or Docker on your Linux distro of choice
2) An understanding of Docker. I won’t be covering commands deeply in this post
3) Create a build location
4) Have a build script in that location
5) Be in that location before running the following commands

#ASSumption: I am working off of Docker for Windows, so I am running all of this in Powershell.

Rolling Docker Commands like a Boss

## Hi, I'm powershell based. Adjust accordingly.
## Normally this is a chained single line of code
## I'm breaking this out to make it read easier
## This script assumes I've done some builds already

## I've done some builds so this is valid
docker stop connects;
docker container rm connects;

## Clean up time. 'docker images -a' is a mess!
docker rmi connects_v1;
docker rmi –f $(docker images –f “dangling=true” -q);

## If I've done NO builds, this is where I start
## Magic line!
docker build -f .\docker_build_v1 . --tag connects_v1;

docker run -d --network bridge -p 8081:80/tcp -p 9000/tcp -h connects --name connects connects_v1;

## Gives me 4 seconds to ensure the container didn't crap out
ping localhost; 
docker ps;
docker exec -it connects htop;
docker exec -it connects /bin/sh

Break it down

Ok, so this is fairly standard with a dash of build to it. I start with an assumption that I have built this before (which I have very likely done 100 times before), so for you, I would skip those first two lines. The real potatoes of this is the:

docker build -f .\docker_build_v1 . --tag connects_v1;

Simply put, I want docker to build from the file docker_build_v1 that I have in “.” (this local) directory using a tag name of connects_v1 for the image. It’s key that I’ve changed into this directory wherever that may be and have my build script in there. Otherwise, you’re not going to get the results you were looking for.

At this point, you could say you’re done. Run a build command like this:

docker run -d --network bridge -p 8081:80/tcp -p 9000/tcp -h connects --name connects connect_v1;

And away you go. But perhaps you’re wondering why I have additional commands here. Perhaps you’re wondering (once you read my next post in this series) why I need to set my ports when I’ve already “exposed” them (no it’s not dirty!) in my build script. Or why I have these commands:

## Gives me 4 seconds to ensure the container didn't crap out
ping localhost; 
docker ps;
docker exec -it connects htop;
docker exec -it connects /bin/sh

We’ll attack the ports publishing question first. Basically, in a build script (next post), you can expose the ports you will need, but this does not translate to actually publishing them. From what I’ve read, this is more of a nice guy thing to let the person running the build script know, “hey, you might wanna go ahead and publish these ports when you run the container.”

https://imgflip.com/memegenerator/That-Would-Be-Great

Well good for you, right? What’s the rest of this mean? Well, I’ve found that containers will die on their own if they don’t have a process inside of them running. And by default, no services start when docker publishes a container. You have to call those in advance. Usually with magic, voodoo, or a Chucky doll sacrifice to IT. If you are banging your head on this problem, no worries, I’ll have a post for you soon covering that exact issue. For now, just know that I give myself four seconds (ping localhost), then I check to see if the container survived (docker ps). If it did, I want to make sure that my services are running properly in the container (my app in other words). And if it is or, more likely, it isn’t, I want to be able to drop into the container to tweak it and see what I need to do to change my build script.

Since this was a repetitive process and I’m really, really lazy, I chained these commands together so that I didn’t have to work as hard. My wrist is destined to be used for great things; I can’t be wearing it out this early in the evening you know.

No really, I want to know what those clean up commands are about

You crazy nut! More punishment? Ok, well, you asked for it. First the commands and then the explanation.

## I've done some builds so this is valid
docker stop connects;
docker container rm connects;

## Clean up time. 'docker images -a' is a mess!
docker rmi connects_v1;
docker rmi –f $(docker images –f “dangling=true” -q);

The first part is fairly generic. Basically, I’m stopping my test container (assuming it comes up at all); then I remove it from the container stack. The next part is the neat part. I just stumbled upon this a few days ago. You can short hand a bit of the code by calling “rmi“. This is docker short command for “remove image”. But that doesn’t explain the $() stuff going on. Fortunately, all that is doing is allowing us to run a command in a command. Remember order of operations from math class? Same deal here. The first commands run will be the commands in the $(). Hold your breath, I’m going to tell you. What the commands do in the $() is forcefully query all of the “dangling” or untagged images (which if your build is long like mine are now cluttering up your command output):

docker images -a

No worries though. Because with the command sequence above it will proceed to forcefully clean all that excess out. Pretty neat, huh? Because that sequence is very linux-similar and linux is the root (no pun intended and not the root of all evil, or is it? hmmmm) of containers (lxc, lxd, docker, etc), it’s very likely this command will run in linux as well. I won’t say it certainly will because I haven’t tried it. Give it a go and let me know in the comments. 🙂

Wrap Up

Stay tuned for the next post where I break every rule when creating a container or a build for a container. Like installing more than one service, making the image repository dirty, and not fully locking down root services. I have a pretty good reason for it though, so in the next post, I’ll show you how to create a build file and tie all of this together. Get ready to level up “in the bad way” (to rip-off what a good friend says to me on occasion). I can hear so many smacks right now as devs facepalm. No worries, hang tight, and we’ll get into the meat of this and see if we can’t shake out the issues in the next post.

Gentoo PkgConfCleaner Script

Introduction

I’m a messy guy when it comes to my /etc/portage/package.accept_keywords or my /etc/portage/package.use conf files. To that end, I created a short script using sed to get things cleaned up. I will admit that I could have avoided using a script at all if I asserted more control over what is put in those files. But if you’re lazy like me and were checking out the deployment script I cobbled together, you might find this useful. Hopefully, my laziness is of benefit to you.

Disclaimer

I’m a strong proponent of testing before implementing. Therefore, I do not take any responsibility for your use of this script on your systems and the consequences that will ensue (good or bad). Please do not just run this on a Gentoo system without first backing up your files. Remember the golden rules of IT:

1) Have a backup plan
2) Have a data backup

Follow these rules, and you will successfully recover most of the time.

Code Me

#!/bin/bash
##Name: pkgconfcleaner
##Author: Cephas0
##License? No Brah!

sed -i 's+#.*++g' /etc/portage/package.use
sed -i '/^\s*$/d' /etc/portage/package.use

sed -i 's+#.*++g' /etc/portage/package.accept_keywords
sed -i '/^\s*$/d' /etc/portage/package.accept_keywords

What does it do?

This is a rather simple thing. The first string finds all of the commented lines in your conf file of choice (hard coded here) and removes them. Then the second line removes all of the leftover space. It looks gnarly but it’s pretty simple in its objective.

I don’t know anything about scripts. How do I get started?

Using a text editor like “nano”, create a file named “pkgconfcleaner” in the /usr/local/bin folder so that the script will be in the path. Next copy and paste the code into the editor. Ok, so here’s a testing hint. If you remove the “-i” from the sed commands, it will not implement the code, but it will print out what the file would look like if it did. Neat, huh? The last thing you need to do to make this work is to run the “chmod” command to make the file executable.

chmod a+x /usr/local/bin/pkgconfcleaner

Something like the code above should be sufficient. (It might not work for you if you are not root or using “sudo”. If you got an error use “sudo” and try again. If it just failed, don’t type it out again, use “sudo !!”)

Wrap up

That’s a wrap on this post. While scripting this short script to fix my laziness may not have been the best allocation of my time, it does have other applicable uses you may find helpful.

Debian Package and Dependency Downloader

#!/bin/bash

read -p "What pkg are you building?: " pkg

##Code attribution for the code below
##https://www.ostechnix.com/download-packages-dependencies-locally-ubuntu/

for i in $(apt-cache depends $pkg | grep -E 'Depends|Recommends|Suggests' | cut -d ':' -f 2,3 | sed -e s/'<'/''/ -e s/'>'/''/); do sudo apt-get download $i 2>>errors.txt; done

This post is about something I tried when I was working on an offline Debian upgrade project. While it didn’t ultimately provide the solution to that project, it did open up a wonderful possibility. To kick this post off, we must have a talk about dependencies, and since that can become mind numbing quickly, I’m only going to gloss over that topic. We’ll talk about what this script does, how to use it, and then turn you loose.

Dependencies _> The Underworld

Dependencies are what the majority of packages or projects rely on to work. Think of it like a base foundation that many people contribute to. This is usually in the form of “lib” or library packages. Other developers will use this pre-written code in their projects, and that’s the end of it right? Not really. Actually, a single project can use dozens to hundreds of dependencies all stacked upon one another like a pyramid of code. This can quickly become a large security issue as the more a system has installed the more dependencies it relies upon. It is at that point that the system’s security becomes more and more dependent (no pun intended) upon every dependency. In other words, the weakest link in any program is the amount of dependencies it uses as much as a chain is only as strong as its weakest link. 

So there’s some of the ugly; let’s talk about the bad for a second. Let’s say you’ve gotten entangled in a project that needs some offline packages installed. Where do you start?

The Journey

For me I started at the online Debian package repository. I needed to download Java for another project. Needless to say you quickly find that you need at least four packages right off the bat.

openjdk-8-jre.deb openjdk-8-jre-headless.deb openjdk-8-jdk-headless.deb openjdk-8-jdk.deb

Yikes! Each package has even more dependencies. And those have even more dependencies. Wouldn’t it be nice if you could just get all the packages and the dependencies without the downloads?

The Solution

I was getting desperate for a solution. Downloading package after package after package is the worst. I have a life and better things to do. Enter salvation in the form of ingenius scripting from OSTechNix. Simply make a folder of the package you wish to download and get cracking.

Here’s the code again below for reference. We’ll step through it.

#!/bin/bash

read -p "What pkg are you building?: " pkg

##Code attribution for the code below
##https://www.ostechnix.com/download-packages-dependencies-locally-ubuntu/

for i in $(apt-cache depends $pkg | grep -E 'Depends|Recommends|Suggests' | cut -d ':' -f 2,3 | sed -e s/'<'/''/ -e s/'>'/''/); do sudo apt-get download $i 2>>errors.txt; done

The Code

I’m going to assume you have made your directory and you are ready to proceed to the next step. If you want you can copy the script above and put it in your /usr/local/bin which will make your script available in your system paths. Make sure it’s executable. I usually run my scripts as root on test systems, so for your system you may wish to use “sudo” in front of whatever you named this script.

read -p "What pkg are you building?: " pkg

This is the first line I added, and it offers some bonuses. You can put as many different packages as you want, spaced out of course. It’s a simple input line for bash with the variable at the end. As you can see, we use that later.

for i in $(apt-cache depends $pkg | grep -E 'Depends|Recommends|Suggests' | cut -d ':' -f 2,3 | sed -e s/'<'/''/ -e s/'>'/''/) 
     do 
        sudo apt-get download $i 2>>errors.txt
     done

I’m going to skip over the code attribution because I think that’s rather self-documenting. The rest of this code starts with a standard for loop. What follows next is a calling of the apt-cache command and the depends command for the package ($pkg, told you we’d use it later) you want to download. Then we pipe to grep, doing a little cutting, run sed (which does some awesome clean up), and then we finally get to downloading the packages.

Wrapping it up

Before you start running this script, make sure you’re in the actual folder you created. Otherwise you could end up with a lot of deb packages everywhere. Not to worry if you did though. Here’s some shortcode to get things cleaned up. We’ll assume you’re in the /tmp/ folder, and you ran, for example, the java packages I listed out earlier. What a mess!

cd /tmp/
##gotta get in the tmp directory first right?
##remember the java folder (package folder) I made?
mv *.deb /tmp/java

And boom. You’re all good. Hope it helps.

Gentoo Package Deployment Script

Introduction

So you wanna make your life easier when emerging a gentoo package? Good news, so did I. I’m going to start this off with, as with many things I do, it’s not always the best way. I would say the backtracking option is very likely something you can and should take out. Read up on that and see if it makes sense for you. For me, this is something I cobbled together. It has some weak points and places it can be improved. It gets the job done though, and that’s enough for me. That said, I’ve felt like if I continue to just keep this knowledge until it’s perfect, no one can benefit from any of it. Therefore, it is incumbent on you to take responsibility for testing, improving, and revising anything I release. Furthermore, if you want to make recommendations with code examples in the comments, I’m all for it. Help each other out, and together we can make things better. Alright, with that, let’s delve into the code and start making our lives a little easier.

Gentoo? Dear Lord Man. For serial.

If you detected a Southpark vibe there, good on you. Gentoo is not for the faint of heart. It will, however, force you to be better than what you are. You will learn a lot about compiling, systems, kernels, and multiple other things. I started with Calculate Linux (a gentoo based Russian distro). If you’re completely new to linux, don’t let the steep climb throw you. Start with Calculate and learn some of the basics as well as the pitfalls. For my master project, I built out my first Gentoo system. I won’t cover that here, but in an upcoming post, I will go over what and how I did my first build. It took a month because I started with a Dell laptop and had issues with the wifi broadcom drivers. Fun times. Let’s get back to the deployment script though.

Nailed it. Code me.

#!/bin/bash
##Author: Cephas0
##Distro: Gentoo
##Knowledge level needed: Copy and Paste?
##License Brah? No...?

array=($(ls | sed 's/\(Manifest\|metadata.*\|.ebuild\|files\)//g'));

for index in ${!array[@]};do echo "$index ${array[index]}";done

read -p "Enter the pkg to be deployed: " num
pkg=${array[$num]}
current=($(pwd | sed 's+/usr/portage/++g'| sed 's|\(.*\)/.*|\1|'))
emerge -av --autounmask-continue --backtrack=200 =$pkg; dispatch-conf; 
exists=($(find /var/db/pkg | grep $pkg))
if [ -d "$exists" ]; then
   echo "Package has been deployed."
   exit
else 
   emerge -v =$pkg
   echo "Package has been deployed."
fi

Put the alcoholic beverage down and step away from the mouse

What’s all of that Greek?!?! No worries – we’ll go line-by-line through it. For now, if you want to kick it off, put it in your /usr/local/bin which will put it in the system path. Small disclosure: on my personal systems, I run everything as root. It just makes it easier for me. If you’re a sudo person, no problems, just run things as sudo. Make sure you make the script executable.

Code Breakdown

The first line of code is below.

array=($(ls | sed 's/\(Manifest\|metadata.*\|.ebuild\|files\)//g'));

I’ve set up a variable called array. The key to this script is that it assumes you have already found the package you want to install and you are in the correct portage directory.

🔥 Spark Note From the Forge 🔥

But wait. How can I get there? Well let’s use an example. Let’s say I want to install “xclock”. How would I find it?

Well there are two ways I do it; although, full admission, there are more, shall we say, proper ways to get the job done. Regrettably, you poor soul, I’m not very proper.

You could change directory into /usr/portage and run:

find | grep -i xclock

Or you could run something like this:

find /usr/portage | grep -i xclock

And your output should look something like this:



Just make sure you change into the xclock directory before running the deployment script.

Ok, back to it. The first line runs the “ls” command and pipes it through “sed”. This proceeds to filter out unneeded extra things you typically find in the directory that won’t be needed for this process. To that end, we extract the manifest file, any metadata file, and cut off the .ebuild and files from the input that will be shuttled to the array variable.

for index in ${!array[@]};do echo "$index ${array[index]}";done

Ok, so the next line of code is basically a one-liner which is why it has the semicolons. You can put this in a more readable format, but I chose not to because for me it was fairly readable. Just something to keep in mind. We start with a for statement and begin to sort through the array. The nitty-gritty of the “!array[@]” here means that instead of just looping through the array itself and using only the elements in the array (which this code would do shown below)…

array[@]

…we want to go through the slices of the array and use the indices like (0,1,2,3,4).

🔥 Spark Note From the Forge 🔥

Indices? I’m not a programmer!

Don’t sweat it. There’s only one thing to remember: indices start at 0 always. That’s how the system begins its counts. It’s true for all programming, and it’s true here. Here’s one for you LOTR fans: if you think of the “0” as the One Ring, you can think of the rest of the numbers as rings bound to the One Ring which starts at 0.

Geez, what a nerd. Hey, get a room, nerd, with your weird numbers and…stuff!

This is going great. We’re over one of the biggest humps believe it or not. The rest of this line goes through each index number. Then we use “echo” to print out the index number and the array element for each package or element in the array. Simple.

read -p "Enter the pkg to be deployed: " num

If you’ve been reading my other blogs, this will be a familiar one. Skip over it if you have or you know what this is. Otherwise, stay tuned. All this one line does is pauses the script (-p) and takes user input (yours) and then puts that input into a variable (num). Why did I use the variable “num” though? Because the script expects you to put in the index number that was printed.

The script in action

So if we were picking a new virtualbox to install and I wanted the latest package, I would enter the number “4” into the response prompt and press enter.

pkg=${array[$num]}

If you’ve done programming with arrays before, you’ll recognize this move. If not, no worries; I’ll cover it now. All we’re doing is using the array index to select the correct package from the array and putting only that package name into a new variable called “pkg”.

So if we were speaking about the example above and writing pseudo code, it would look somewhat like this:

pkg=${virtualbox-6.1.4[index number 4]}

Ok, so I feel pretty good about the progress we’ve made, and hopefully, if you didn’t know much about bash arrays or indices, you’re picking up some useful ideas for your next script.

current=($(pwd | sed 's+/usr/portage/++g'| sed 's|\(.*\)/.*|\1|'))

The next line is looking for what directory we’re in by basically using the pwd command, which is great, but dumps a lot of extra junk with it. Thankfully we have sed which allows us to clean things up. So if we’re using the virtualbox example above, we’re going to have a string looking like this from pwd:

/usr/portage/app-emulation/virtualbox

But what we really need is only the:

app-emulation

Enter the sed sphere. We pipe the output of pwd to sed and strip the front two tags. Great. That just leaves:

app-emulation/virtualbox

So how to get that last pesky section off? And that, my friends, is why we pipe in the next sed statement of glop. I’ll be completely honest here: I don’t fully remember what it does other than it will pick off the slash and what is after the slash. The how’s and why’s are on the internet and were in my head when I was working on this diligently.

emerge -av --autounmask-continue --backtrack=200 =$pkg; dispatch-conf;

This should look familiar, right? If not, what’s going on is a very questionable use of the emerge command in a script. Basically I’m running emerge, telling it to be verbose, ask, automatically unmask any package requirements (a.k.a. /etc/portage/package.accept_keywords), and backtrack. Why backtrack? The emerge man page puts it best:

–backtrack=COUNT
Specifies an integer number of times to backtrack if dependency calculation fails due to a conflict or an unsatisfied dependency
(default: ´10´).

I’ll leave it there for that. Next we run the “dispatch-conf” command so that we can automatically accept all changes. Side bar here: this will dump an enormous amount of junk into your package files like /etc/portage/package.accept_keywords. If you are strenuous about keeping the amount of comments down and your files clean then, by all means, you can add some code to the script and keep your package.accept_keywords cleaner than mine. I have another script that cleans up that mess for me, but that’s for another post.

exists=($(find /var/db/pkg | grep $pkg))

This line sets up a scenario for the remainder of the script. Namely, we check to see if the package was installed and we set the variable “exists”.

if [ -d "$exists" ]; then
   echo "Package has been deployed."
   exit
else 
   emerge -v =$pkg
   echo "Package has been deployed."
fi

We are in the home stretch now. We start out with an “if else” statement. We check to see if the directory exists (meaning a successful install), and if it does, we end the script and print a success message. If not, we strip all of the options out and automatically try to emerge the package. This is there because of the dispatch-conf option I called earlier.

Wrap up

That’s it for this post. Feels like a lot of ground got covered in a short amount of time. It’s not the most perfect script, but it does get the job done. I won’t promise you it will work all the time. You will have to contend with certain upgrades in a different way such as python and perl. Additionally, dependency conflicts are generally not going to be solved by this. I’ll cover some of these in other posts down the line. For now, enjoy the deployment script and have fun.