Bash shell syntax goodies and gotchas

In my time bash scripting, I've come across a number of CLI goodies and gotchas, so I've decided to compile a post with a list of neat examples. These may be about individual packages or specific to the shell.

pdflatex -- the order of arguments matters :( For instance, I've written a bash script that's cron-scheduled to re-compile my resumé each morning. But, you must specify flags and options before the file name you're aiming to compile.pdflatex -var-value="$out" -output-directory="$misc" "$location" Otherwise, it fails. Watch out for rm'ing against filesystems as such -- read -r -p "Give me a file to delete: " var rm -r "$var/" If $var does not contain anything, then this could destroy your machine :P Instead, use rm -r "${var:?}/" For example, on my own machine -- root@ideapad:~# read -r -p "This: " that; echo "${that:?}/" This: -bash: that: parameter null or not set root@ideap…

Setting up a virtual router at home

This post is a bit of a mind-dump, but I thought I'd write down things I learned this past weekend while switching my home router from a physical NetGear R6260 to a virtual pfSense instance. I've also been recommended to use OPNSense by my colleagues for a nicer interface, but I'll stick with pfSense for the moment.

The reason for making the switch includes the fact that VLAN configuration on the R6260 requires a reboot on every modification, and moreover, you can't have more than one VLAN on each physical port.
 I'm not exactly sure why that would be the case, either.
Hardware To start, I purchased an i340-T4 card on eBay for ~$25. Brand new, these cards are still running at $100 or more, but mine works just fine for the price. (The only thing I wasn't happy about was that it arrived sitting in bubble wrap instead of an ESD bag.)

After shutting down my host and installing the card, I didn't see the interfaces come up automatically (as expected). Instead, …

Setting up and connecting to a SQL database

This blog post may be a bit redundant, considering how ubiquitous SQL databases are today, but I thought I'd write a summary of what I've accomplished so far with configuring a simple MySQL database on my home LAN.

I haven't messed around with SQL too much, having never come across so much data that a database was required to organize it in a more manageable way. However, at some point in the future I know I'll come across this problem, so it's probably best to start learning the toolkit now.

For starters, I have a VM running a Ubuntu 18 server instance that's hosting a mysql-server install. Next, I've configured the SQL server to listen on, as opposed to just localhost, which is the bridged IP of the VM. EDIT 12/10: I've since edited the configuration file to allow incoming connections from any IP by setting this value to That way I can connect to it on port 3306 from any other machine. You may find these configuration options in…

Working with an AMD WX 3100 Pro and (Py)?OpenCL

This is a bit of a recap on getting something useful out of my AMD Radeon Pro WX 3100 after toiling with drivers for a couple of days.

After following the installation process for ROCm, I discovered this GPU isn't supported as of yet.

However, I was able to get something out of clinfo (which you also have to install) after following laanwj's blog post about OpenCL. For Python there's a wrapper module, pyopencl, that provides a wrapper via pybind11. $ clinfo Number of platforms 1 Platform Name Clover Platform Vendor Mesa Platform Version OpenCL 1.1 Mesa 18.0.5 Platform Profile FULL_PROFILE Platform Extensions cl_khr_icd Platform Extensions function suffix MESA Platform Name Clover Number of devices 1…

Another attempt at the RX algorithm in Python

At the request of a commentor on a former post, I decided to write an updated Python implementation of the RX algorithm with NumPy (which actually out-performs my naive C implementation). You can download and install the module from GitHub.

At the heart of it all is the Mahalanobis distance metric,
\begin{align}d(\,\tilde{X},X_{i}\,)&=\sqrt{(\,X_{i}-\tilde{X}\,)^{\text{T}}\,\Omega\,(\,X_{i}-\tilde{X}\,)},\end{align}where \(X_{i}\) is some vector in the image's space (many times RGB), \(\tilde{X}\) is the average (or approximately average) vector along each channel, and \(\Omega=\Sigma^{-1}\) is the precision matrix.

I've found it's not really necessary to have the exact precision matrix in practice, so we may reduce the overhead of computing an average vector and covariance if the image is largely one color.  This computation can be neatly expressed with numpy.einsum.

Scaling an image to an appropriate number of pixels for your project

Today I'm writing a Python module that works with image data (check back in a few days for my post about it) and I wanted to arrive at the optimal dimensions an image must be to have a certain number of pixels.

For example, if I want to see how fast my algorithm is for, say, 1 million RGB vectors, then I want to scale my example image to 1 megapixel.

In this post I'm working with an image that has dimensions \((3840,2160)\), or a size of about 8.3 million pixels. But what should the \(X\) and \(Y\) dimensions of my image be then, if I want it to be as close to 1 million pixels as possible?
First Attempt Let's write a Python function that preserves the aspect ratio of an image's dimensions and allows us to vary its size. from typing import Tuple def scale(X: int, Y: int, factor: float =1.0) -> Tuple[float, Tuple[float, float]]: """ Scale our image's dimensions by some factor, preserving aspect ratio. """ X_, Y_ = fa…

A few takeaways from PyCon, 2018

There were a lot of interesting talks this year, and although I didn't attend nearly as many talks as I would have liked, I got a lot out of those that I did.
Performance Python First and foremost has to be Jake Vanderplas' talk, Performance Python: Seven Strategies for Optimizing Your Numerical Code. My first takeaway from his talk is line_profiler, which is a command line tool that's intended to be used to profile your Python functions' execution time, line-by-line.

For example, let's use code from my last post and apply the convert function to a single image. We use line_profiler in the terminal by adding the @profile decorator to the function head (no imports necessary).
@profile def convert(im: np.array, transform: np.array) -> np.array: """ Convert an image array to another colorspace """ dimensions = len(im.shape) axes = im.shape[:dimensions-1] # Create a new array (respecting mutability) new_ = np.empty(…

Color space conversion

I've been intrigued by Steven Pigeon's series on color spaces lately and wanted to give color space conversion a try in Python. In the following script I've reproduced several of the color spaces Pigeon mentions in the first five parts of the series.

Using NumPy this operation is almost trivial, but I added a few fun tidbits as well. Also note that, as I've mentioned before, PEP 465 has added the infix matrix multiplication operator @ to Python 3.5+.
#! /usr/bin/env python3.6 # -*- coding: utf-8 -*- # vim:fenc=utf-8 """ Convert colorspaces. """ from typing import List from tqdm import tqdm from functools import reduce from operator import mul import numpy as np import imageio import os ## Colorspaces Kodak_1 = np.array([[ 1, 1, 1], [-1, -1, 1], [ 1, -1, -1]]) Kodak_YCC = np.array([[ 0.299, 0.587, 0.114], [-0.299, -0.587, 0.886], [ 0.701, -0.58…

Fold a String into an Int type

While working my way through (Sullivan, Goerzen & Stewart, 2009), I came across the problem of converting a String to an Int using folds. This isn't that difficult a problem, but because I'm still a beginner to Haskell, I had to think about it for a few days.

Finally, I came up with the following (unsafe) solution using foldl.
import Data.Char (digitToInt) asInt :: String -> Int asInt [] = 0 asInt (x:xs) | x == '-' = (-1) * asInt xs | otherwise = sum $ foldl helper [] $ zip (reverse [0..(length (x:xs) - 1)]) (x:xs) where helper :: [Int] -> (Int,Char) -> [Int] helper totalList (order,digit) = digitToInt digit * 10 ^ order : totalList I separated the state of summing from the rest of the problem, using the fold to  'loop' over the list and raise each digit to the appropriate magnitude based on its position, just as one would in an imperative language. Thus, I believe this is a hybrid solution that uses both recursion and a f…

Binary Search in Haskell and Python 3.6

I've been learning a lot about Haskell lately, so I decided to write my own implementation of binary search after coming across Jacob Sheehy's implementation. In the following I wanted a nicer functional interface to locating a value in an entire list. I also wanted it to be safe, using the Maybe monad in the event a value doesn't exist or the list is empty.
-- A humble implementation of binary search in Haskell binarySearch :: Int -> [Int] -> Maybe Int binarySearch _ [] = Nothing binarySearch value list = search 0 ((length list) - 1) where search :: Int -> Int -> Maybe Int search i j sublist = do if i > j then Nothing else do let midPoint = ((i + j) `quot` 2) currentValue = list !! midPoint -- compare outputs one of LT, GT, EQ case compare value currentValue of LT -> search i (midPoint - 1) GT -> search (midPoint + 1) j EQ -> Just midPoint main :: IO() …