This post is about something I tried when I was working on an offline Debian upgrade project. While it didn’t ultimately provide the solution to that project, it did open up a wonderful possibility. To kick this post off, we must have a talk about dependencies, and since that can become mind numbing quickly, I’m only going to gloss over that topic. We’ll talk about what this script does, how to use it, and then turn you loose.

Dependencies _> The Underworld

Dependencies are what the majority of packages or projects rely on to work. Think of it like a base foundation that many people contribute to. This is usually in the form of “lib” or library packages. Other developers will use this pre-written code in their projects, and that’s the end of it right? Not really. Actually, a single project can use dozens to hundreds of dependencies all stacked upon one another like a pyramid of code. This can quickly become a large security issue as the more a system has installed the more dependencies it relies upon. It is at that point that the system’s security becomes more and more dependent (no pun intended) upon every dependency. In other words, the weakest link in any program is the amount of dependencies it uses as much as a chain is only as strong as its weakest link. 

So there’s some of the ugly; let’s talk about the bad for a second. Let’s say you’ve gotten entangled in a project that needs some offline packages installed. Where do you start?

The Journey

For me I started at the online Debian package repository. I needed to download Java for another project. Needless to say you quickly find that you need at least four packages right off the bat.

openjdk-8-jre.deb openjdk-8-jre-headless.deb openjdk-8-jdk-headless.deb openjdk-8-jdk.deb

Yikes! Each package has even more dependencies. And those have even more dependencies. Wouldn’t it be nice if you could just get all the packages and the dependencies without the downloads?

The Solution

I was getting desperate for a solution. Downloading package after package after package is the worst. I have a life and better things to do. Enter salvation in the form of ingenius scripting from OSTechNix. Simply make a folder of the package you wish to download and get cracking.

Here’s the code again below for reference. We’ll step through it.

#!/bin/bash

read -p "What pkg are you building?: " pkg

##Code attribution for the code below
##https://www.ostechnix.com/download-packages-dependencies-locally-ubuntu/

for i in $(apt-cache depends $pkg | grep -E 'Depends|Recommends|Suggests' | cut -d ':' -f 2,3 | sed -e s/'<'/''/ -e s/'>'/''/); do sudo apt-get download $i 2>>errors.txt; done

The Code

I’m going to assume you have made your directory and you are ready to proceed to the next step. If you want you can copy the script above and put it in your /usr/local/bin which will make your script available in your system paths. Make sure it’s executable. I usually run my scripts as root on test systems, so for your system you may wish to use “sudo” in front of whatever you named this script.

read -p "What pkg are you building?: " pkg

This is the first line I added, and it offers some bonuses. You can put as many different packages as you want, spaced out of course. It’s a simple input line for bash with the variable at the end. As you can see, we use that later.

for i in $(apt-cache depends $pkg | grep -E 'Depends|Recommends|Suggests' | cut -d ':' -f 2,3 | sed -e s/'<'/''/ -e s/'>'/''/) 
     do 
        sudo apt-get download $i 2>>errors.txt
     done

I’m going to skip over the code attribution because I think that’s rather self-documenting. The rest of this code starts with a standard for loop. What follows next is a calling of the apt-cache command and the depends command for the package ($pkg, told you we’d use it later) you want to download. Then we pipe to grep, doing a little cutting, run sed (which does some awesome clean up), and then we finally get to downloading the packages.

Wrapping it up

Before you start running this script, make sure you’re in the actual folder you created. Otherwise you could end up with a lot of deb packages everywhere. Not to worry if you did though. Here’s some shortcode to get things cleaned up. We’ll assume you’re in the /tmp/ folder, and you ran, for example, the java packages I listed out earlier. What a mess!

cd /tmp/
##gotta get in the tmp directory first right?
##remember the java folder (package folder) I made?
mv *.deb /tmp/java

And boom. You’re all good. Hope it helps.

If you’ve been following along, you’ve gotten all of your offline files ready for deployment. If you missed that section you can go there now. In the sections below, we’ll discuss: the offline deployment, suggestions for running your offline deployment, and finally some fascinating ideas and projects I tried which have interesting potential but, unfortunately, did not work for me in this instance.

Disclaimer

This guide strictly deals with upgrading your system. It will not cover dist-upgrade although that is certainly something you can try and test. This information is provided as-is and, therefore, I take no responsibility for incidents with your equipment. I am a huge proponent of testing. Please ensure you know what you are doing before you attempt this.

Tools you need

1. WinSCP (If you’re using Windows and, for this, it’s almost, almost worth using Windows just to use this awesome, free tool)
2. Two systems. One should be online and the other is, of course, the offline one. They both should be very close build-wise. NOTE: If you want to test this out, I recommend changing the /etc/resolv.conf file on one of the systems. Comment: Remove everything in there and save it. This ensures apt will break without using the correct options and your test is as clean as it’s going to get without introducing USB flash drives.
3. Putty, or if you’re on Linux, SSH
4. Internet connection with both systems on the same network if you’re testing. Otherwise, you’ll just need internet for the online portion.

Copying over the Needed Files to the Offline System

Are you ready to get this done?

We’ll use WinSCP and transfer our files over to the “offline” system in its /tmp/ folder assuming it’s on your network and the only edits you made were to the /etc/resolv.conf file for testing. Otherwise, if you cannot reach your offline system with a network connection, you’ll have to use a flash drive. Mounting a flash drive is out of scope for this post, but the rest of the commands are relevant to your endeavor.

Assuming you’ve connected to your remote system with WinSCP, it’s time to copy some files over. This is where WinSCP shines because it saves so much time. We’re going to specifically copy over the:

1. Archives folder
2. Lists folder
3. Any additional packages (if its a .deb just put it in the archives folder once you’ve noted the full package name including its .deb extension) or scripts you may need.

We’re going to place these folders/files into the /tmp/ directory. Once everything is in the directory, assuming you kept the file names, we can get the actual update process started.

Spark Note From the Forge

Be a boss. Tar your files or zip them to make the transfer faster. Why? Because an archive, tar, or zip presents as a single file. The network won’t speed up and slow down as it finishes a file and starts a new one. Instead, as an archive or zip shows up as a single file the network keeps the pedal to the floor the entire way through. Want more mileage? Go full bore and use “xz” compression.

Moving files to the correct directories and cleanup

I kept the file names the same, so starting the in the /tmp/ directory I will run the following:

## Time to make the money. Clean out the archives section first. Make room on the tiny system.
## Move in the new archives and lists and clean up.
cd /tmp/;
apt-get clean;
mv archives/* /var/cache/apt/archives;
rm -rf archives;
rm -rf /var/lib/apt/lists/*;
mv lists/* /var/lib/apt/lists/;
rm -rf lists;

Can this be written more elegantly? Yes. But my intent is to fully show what I’m doing. Do I need semicolons at the end of these? No, but I tend to like to chain my commands together for situations where I can only paste one line into the terminal as it doesn’t make sense to write a script if I’m going to be on and off the system quickly. Believe me, I have found in the world of embedded systems that that happens. Like with Road Side Units (RSU) where you may be doing the same thing with very little variation dozens of times. You may love vi (one of the oldest unix/linux text editors), but me, not so much.

If everything has completed properly then we start pulling the triggers on things.

Running the Offline Update and Upgrades – Finally!

sed -i 's/jessie/stretch/g' /etc/apt/sources.list;
apt-get update --no-download;
apt-get upgrade -yf --no-download --ignore-missing;

For the upgrade, you can try this instead to keep the default options, but I have not had much success with it:

DEBIAN_FRONTEND=noninteractive apt-get upgrade -yf --no-download --ignore-missing -o "Dpkg::Options::=--force-confdef" -o "Dpkg::Options::=--force-confold";

So what does all of this do? If you remember from the previous blog, the sed command changed over “jessie” to “stretch” in one line. You can change the words to be what you need. For instance, you could make it look like this:

sed -i 's/stretch/buster/g' /etc/apt/sources.list;

But for this case, whatever you used on the online system must be used for the offline system. Is it really necessary? For documentation purposes at a minimum, yes. It’s a cheap, short, lazy way to let folks know what the offline system has. Don’t be that guy that leaves other folks wondering.

The next line simply updates the package lists in the system. This is important as we want the package database to be updated with the latest packages for our distributions archives we just moved from /tmp/ to the archives folder.

Once we get to the upgrade line, you’re in the home stretch. The system should be able to begin checking the archive folder for the necessary packages and will begin the upgrade process. I found there was no automation on my side as sometimes I needed glibc to be upgraded, for instance, which brought up a blue screen (updating grub will also bring up a blue screen requesting input). With that in mind, I’d plan to stick around and see things through.

But wait! What about my other offline packages that are not part of the main repository? Like that influxdb package, you pulled down. What happens there? Does apt-get upgrade or apt-get install work for that?

Not in my experience. What I did was throw those packages in the archives folder and then used dpkg to install it like so:

pushd /var/cache/apt/archives;dpkg -i influxdb_1.7.9-1_amd64.deb;popd

This is, again, not the slickest way you can write this, but, hopefully, it gives you an idea for a one-liner install.

Things I Tried that Failed

My main, initial challenge was that I had no internet on the remote boxes I needed to reach. I had a vpn connection and ssh. So naturally I tried to do a reverse ssh proxy which failed for me. I tried using port forwarding with ssh as well, and again, it didn’t come through. Trying to provide internet to a remote box over vpn was making me bang my head!

Time for plan b, and that turned out to be a project called sshuttle. It’s a neat project written in python that performs as a sublevel vpn, dns tunnel and more. It didn’t work out for me, but it might work out for you. Here’s what the project creator had to say on their github:

“Transparent proxy server that works as a poor man’s VPN. Forwards over ssh. Doesn’t require admin. Works with Linux and MacOS. Supports DNS tunneling.”

https://github.com/sshuttle/sshuttle

Spark Note From the Forge

What a bummer! I couldn’t get this python project installed right away. I’m so accustomed to working with linux scripts that it didn’t cross my mind that you might need to run a python project with a python installer! This project does need a separate project called “setuptools“.

Inside of that project, you will need to run an initial setup script as well. Another “gotchya” is you will need to match the version of setuptools that the creator of sshuttle is using, but here is a link you can use to get an idea and get started (https://pypi.org/project/setuptools/).

Well how do I install this project offline then? You will need to get all the project files, and then do what you’ve done with the archive files and copy them to /tmp/. Change directory into that file. From there, you can run the following code to install it on the user account you are logged into the remote system with:

python install setup.py –user

Well, so much for sshuttle then. I also gave a project called apt-offline a try. This was promising, but in the end, it also didn’t fulfill my needs. It did, however, set me down the path to understanding the Debian system better. While this project simplifies things for the average user, it still didn’t do everything I wanted. You can check out that project here if you want to give it a try: (https://github.com/rickysarraf/apt-offline).

That wraps it up for this post. Hopefully, you found the droids you were looking for and this has been useful.

As part of an R&D project, I had to find a way to update a few old systems we have in the field that have been completely locked down (i.e. you can use a VPN to visit but that’s it). There is no DNS and no other external internet connection. To add to the challenge, I needed to work with a system that has 3.4GB of storage with 2.5GB of it used, and I needed to update Java and install some other test .deb files. In this series, I’ll tell you what I did to make it work, what I tried that didn’t work, and how you can be your own offline master.

Disclaimer

This guide strictly deals with upgrading your system. It will not cover dist-upgrade although that is certainly something you can try and test. This information is provided as-is, and, therefore, I take no responsibility for incidents with your equipment. I am a huge proponent of testing. Please ensure you know what you are doing before you attempt this.

Tools you need

1. WinSCP (If you’re using Windows and, for this, it’s almost, almost worth using Windows just to use this awesome, free tool)
2. Two systems. One should be online and the other is, of course, the offline one. They both should be very close build-wise. NOTE: If you want to test this out, I recommend changing the /etc/resolv.conf file on one of the systems. Comment out everything in there and save it. This ensures apt will break without using the correct options and your test is as clean as it’s going to get without removing an ethernet cable and introducing USB flash drives.
3. Putty, or if you’re on Linux, SSH
4. Internet connection with both systems on the same network if you’re testing. Otherwise, you’ll just need internet for the online portion.

Online System Update, Upgrade, and Captures

Ok now let’s get to the interesting part. We start with the online system. In my case, I had an original image and a test box to play with. I won’t cover the uses of the “dd” command in this post, but that’s what I used to image my test drive. Once you get the system booted and ssh’d into, the first thing to do is to clear the archives.

Use the following to do so:

 apt-get clean

This cleans out all of the downloaded packages in the archives. This is a critical part as you’ll see later.

Next we’re going to edit the /etc/apt/sources.list. For this experiment, we were moving from Debian 8 (Jessie) to Stretch (9) as Buster (10) the current version is still too new. Now to edit, you can do this by hand using the editor of your choice. Mine is nano like so:

nano /etc/apt/sources.list

And you can replace each of the “jessie” words with “stretch”. Or if you want to save a bit of time you can try this sed code. If you want to change from stretch to buster for example just change the words as needed below.

sed -i 's/jessie/stretch/g' /etc/apt/sources.list;

All good? Good deal. Next, we’re going to do the normal song and dance we do to upgrade a Debian-based system. I like chaining commands and automating things a little bit, so I’m going to tell the apt system to update itself and its files (yes, this is important for something later on), and then I’m going to ask the system to proceed to upgrade itself.

apt-get update; apt-get upgrade -y;

You may wish to install what I call “generic system packages”. What I mean by that is these packages are part of the default repo’s found in the /etc/apt/sources.list. I didn’t need to add a key or another repository to get them. This is important because to run what I call “special packages”, as you’ll see, I need to do something else. We’ll get to that in a little further down. For now, we’ll run something generic like Java11.

apt-get install -y openjdk-11-jre openjdk-11-jre-headless openjdk-11-jdk openjdk-11-jdk-headless;

Now that we have that installed, it’s time to put our WinSCP into play. (As a Linux guy, I’m going to utter a curse…. Due to the simplicity of this, you might get Windows envy if you don’t have one to work with.)

I’m not going to go over WinSCP in this post, but it’s a fairly intuitive tool to use especially if you’ve ever used Putty before. Once you’ve created your scp or sftp connection through WinSCP, you should create a “dump” folder. I named mine “apt-offline” after a tool that was not useful to me at all.

Now for the difficult part. I’m joking. On the left is my Windows computer drive and on the right is the remote computer drive. So I’m going to do some clicking around where you see /tmp/ listed out. Specifically, I’m going to click on “/” because that takes me to the root directory. Where we need to go is:

/var/cache/apt

Once there, click on the “archives” folder. That’s where the meat is. Let’s drag the archives folder to the folder we set up and (hopefully) navigated to on our Windows computer. You will get errors on the “lock” file and the “partial” folder. That’s perfectly fine as we don’t need them.

Now we run:

apt-get clean

The archives directory will be empty now with the exception of the “lock” file and “partial” folder. Ok so remember when I said we’d cover what to do if we had a special case that required installing something additional that is not in the main repo’s? Well, I’m going to deliver on that promise here. To make this simple, I’m going to install “influxdb” which requires a key to be added and a repo and is NOT part of the main repo’s.

##Things I found I needed, we run slim systems
##Below could have been installed at time of the
##Java install
#apt-get install -y gnupg2 apt-transport-https ca-certificates curl software-properties-common;

##Now lets get installing. This is from the 
##InfluxDB install page for your reference :-)

wget -qO- https://repos.influxdata.com/influxdb.key | sudo apt-key add - ;
source /etc/os-release
test $VERSION_ID = "7" && echo "deb https://repos.influxdata.com/debian wheezy stable" | sudo tee /etc/apt/sources.list.d/influxdb.list
test $VERSION_ID = "8" && echo "deb https://repos.influxdata.com/debian jessie stable" | sudo tee /etc/apt/sources.list.d/influxdb.list
test $VERSION_ID = "9" && echo "deb https://repos.influxdata.com/debian stretch stable" | sudo tee /etc/apt/sources.list.d/influxdb.list

apt-get update; 
apt-get install -y influxdb;

Imagine everything installed. Now it’s time to check that treasure trove in archives and see what we got. This should have given us the .deb file and any other dependencies we didn’t know about.

In this case, I found all that had been downloaded was the “influxdb_1.7.9-1_amd64.deb” file. I’ll copy this over with WinSCP and place it with the other packages in the archives file, but NOT before I note the full name. I will use this when I run the dpkg script later on in the archives folder.

dpkg -i influxdb_1.7.9-1_amd64.deb;

Well, that wraps up this section of prep. In the next post, I’ll show you how you can put this into action.