When developing software, it’s good practice to put anything you don’t wish to be public, as well as anything that’s “production-environment-dependent” into environment variables. These stay with the local machine. This is especially true if you are ever publishing your code to public repositories like Github or Dockerhub.

Good candidates for environment variables are things like database connections, paths to files, etc. Hosting platforms like Azure and AWS also let you easily set the value of variables on production and testing instances.

I switch back and forth between Windows, OSX and even Linux during development. So I wanted a quick cheatsheet on how to do this.

Writing Variables

Mac OSX (zsh)

The default shell for OSX is now zshell, not bash. If you’re still using bash, consider upgrading, and consider using the great utility “Oh My Zsh.”

printenv

will print out environment variables

To save new environment variables:

export BASEBALL_TEAM=Seattle Mariners

To save these permanently, you’ll want to save these values to:

~/.zshenv

So, in terminal you’d bring up the editor:

sudo nano ~/.zshenv

and add export BASEBALL_TEAM=Seattle Mariners to this file. Be sure to restart a new terminal instance for this to take effect, because ~./zshenv is only read when a new shell instance is created.

bash shell (Linux, older Macs, and even Windows for some users)

export BASEBALL_TEAM=Seattle Mariners
​
echo $BASEBALL_TEAM
Seattle Mariners 
​
printenv 
< prints all environment variables > 
​
// permanent setting
sudo nano ~/.bashrc
// place the export BASEBALL_TEAM=Seattle Mariners in this file
// restart a new bash shell 

Windows

• Right click the Windows icon and select System
• In the settings window, under related settings, click Advanced System Settings
• On the Advanced tab, click Environment variables
• Click New to create a new variable and click OK

Dockerfile

ENV [variable-name]=[default-value]
​ 
ENV BASEBALL_TEAM=Seattle Mariners

Reading Variables

Python

import os
​
print(os.environ.get("BASEBALL_TEAM"))

Typescript / Javascript / Node

const db: string = process.env.BASEBALL_TEAM ?? ''

C#

var bestBaseballTeam = Environment.GetEnvironmentVariable("BASEBALL_TEAM");

Steve Murch

Steve’s a Seattle-based entrepreneur and software leader, husband and father of three. He’s American-Canadian, and east-coast born and raised. Steve has made the Pacific Northwest his home since 1991, when he moved here to work for Microsoft. He’s started and sold multiple Internet companies. Politically independent, he writes on occasion about city politics and national issues, and created voter-candidate matchmaker Alignvote in the 2019 election cycle. He holds a BS in Applied Math (Computer Science) and Business from Carnegie Mellon University, a Masters in Computer Science from Stanford University in Symbolic and Heuristic Computation, and an MBA from the Harvard Business School, where he graduated a George F. Baker Scholar. Steve volunteers when time allows with Habitat for Humanity, University District Food Bank, Technology Access Foundation (TAF) and other organizations in Seattle.

read more

stevemurch.com

This is a technote for myself for the future and perhaps a note that’ll help people searching for this obscure but important and confusing tech roadblock.

The Problem

I’m working on a web app. It’s hosted at Azure, a platform that grows more and more impressive every day.

As we all know by now, any respectable web app needs a full https trusted connection, which is best for customer privacy, and now appropriately mandated by Google, Facebook and the like.

Now, with plain-vanilla web apps, Azure has a very handy SSL provisioning process which does the grunt-work for you, and gets basic SSL certificates up and running and even custom domain names assigned. I recommend it.

But with my app, I’ve got some specific reasons beyond the scope of this article as to why I needed to provision this certificate manually, which have to do with the fact that it’s an Angular Universal app, running on Node/Express on the back end, with an Azure Application Gateway on the front-end, which doesn’t yet appear to robustly support automatic certificate provisioning.

The point is, to get a fully trusted https (SSL/TLS) connection, you need to install both a main certificate and any intermediate certificates at the server. Or more accurately, you need a single certificate with both the “main” certificate and any “intermediate” certificates.

At this writing, Azure’s Application Gateway service insists that this be a single certificate in PFX format.

Complicating matters further is that vendors like GoDaddy (which is the one I use) issue certificates in .crt and .p7b formats. You don’t get a PFX certificate from GoDaddy. You get a .crt (main) certificate and a .p7b (intermediate) certificate when GoDaddy thinks their work is “done.”

So there you are, needing a .PFX and only having (and having paid hundreds of dollars for) a .crt and a .p7b.

So, off to Google you go. You quickly find that you can convert the .crt to a .pfx, and that does appear to work.

But… and with Certificate Hell, there’s always a gotcha!… if you solely focus on the main certificate that GoDaddy (or others) issue, you’re given the illusion of it working but it really isn’t right, end-to-end.

It will appear to work on, say, Google Chrome, presenting you with the nice shiny lock icon. But then, you head on over to Facebook’s “Linter” (their social media debugger), and it will bark at you that the SSL of your site is invalid. So will more complete tools like SSL Labs Validity Checker.

That’s because it’s missing the Intermediate Certificate. This certificate is essentially telling the Internet that GoDaddy has the rights to issue certificates for domains, and that this “main” certificate has been issued by the real GoDaddy itself.

And with so many steps in this process, the error messages you get don’t always point in the right direction. Because why would you want to do that if your goal is to maximize complexity?

The Goal

In general, you need a certificate which contains the full “chain” of trust, and in my case, it included both the final certificate as well as GoDaddy’s intermediate certificate. This all needs to be bundled into a single PFX file in the end.

That’s what most Azure services want — a final PFX that includes any intermediate certificates in the chain.

Too Much Confusing Jargon

The maddening bit about all this is that the documentation seems written by people from NASA, and uses overly complex lingo and jargon. Even the many helpful people on StackOverflow and the like make all kinds of assumptions about what you know and what you don’t.

So how do you get it all to work?

I spent much of this weekend trying various approaches. So many dead-ends. Such a pain!

But I finally solved it. Here are the steps.

First, buy an “SSL Certificate” from a vendor, such as GoDaddy. At GoDaddy at least, buying an SSL certificate gives you a “slot” to begin the process. You tell it the domain name you want. Better yet, go with a wildcard domain; you should always think about buying a wildcard like *.domainname.com, because in many cases, you’ll very likely want to secure subdomains.

This gives you a spare “SSL Certificate” process to initiate in the “My Products” area. Click on that to get started.

Go to a Windows machine and open up a terminal. Make sure you have openssl downloaded and installed somewhere on your system PATH.

Note to self: DO NOT try this process using the IIS initiation process, documented in a lot of places on the Web. That for me was one of many dead-ends and confusing aspects to the whole thing. You can complete the entire process with openssl at a terminal, and uploading to GoDaddy, downloading the certificates, and some conversion. Doing so, you will not need any of the “Add/Remote Snap In” MMC gobbledygook.

You do not need to touch the remote server until you have the final .PFX file. It is perfectly OK to initiate an SSL certificate signing request on a development machine and uploading it to the final (say, Azure-hosted) server later. A certificate, once issued, can be uploaded to different machines.

Note also that I only had luck on Windows after trying both OSX and Windows, but I do know from elsewhere on Stack Overflow that it works on OSX too. OSX should work fine, but in a couple of attempts I had difficulties — very likely user error.

First, generate yourself a private key:

openssl genrsa -out myserver.key 2048

That’ll create a file called “myserver.key” in your current directory. This is your private key, hashed to your machine that you generated it from, known only to you. It says “I am the one creating this request.”

Use openssl to generate a certificate signing request:

openssl req -new -key myserver.key -out myserver.csr

Clarifying the Prompts at CSR Time

The prompts during the Certificate Signing Request generation aren’t clear at all. But it’s VERY important what you enter when generating the Certificate Signing Request.

The most important thing is what you enter in the “Common Name” field. This should be your domain name (e.g., domainname.com) or your wildcard (*.domainname.com).

For good measure, I put it as Organization Name as well.

Once it saves a .csr (certificate signing request) file, upload that CSR, including the —- BEGIN CERTIFICATE REQUEST —– and —– END CERTIFICATE REQUEST —— markers to initiate the request.

After five-ten minutes of verification, you’ll be issued a download payload. Grab it in IIS format.

Included in that file will be a main certificate (marked .crt) and an Intermediate certificate (maddeningly in a different format, .p7b.)

Ignore that intermediate certificate issued in the download and INSTEAD (if you’re using GoDaddy) visit GoDaddy’s repository of primary and intermediate certificates to grab their Intermediate Certificate gdig2.crt.pem:

Don’t be scared of .pem and .crt files — they’re just text files. You’re going to open your main .crt file in Notepad++ and APPEND the intermediate certificate from GoDaddy from their repository, saving them both as a file called, say, both.crt.

The Crucial Step: Use Notepad++ to Combine the Certificate Files — Your Main One FOLLOWED by the Intermediate One

You are literally pasting a text file of the two certificates, including the begin/end markers, one after the other. Save that file as something like “both.crt” (meaning, both your main certificate and your intermediate certificate.)

Note that you need this file in pure UTF-8 format. Windows gives you files downloaded in UTF-8-BOM format which is NOT recognized by the openssl tool and you will get an error if you try to use it. In my experience, this can also hork your requested files, so double-check your work before you continue. You should have a file called “both.crt” which includes the full contents of your issued main “.crt” file, followed by the godaddy intermediate certificate above. (Again, that’s only if your SSL cert is issued by GoDaddy and only if it relies upon an intermediate certificate.)

Be sure to use Notepad++ to check the encoding of the text file, and if necessary, switch the encoding type to just UTF-8. It matters. openssl is very finicky, and hates input files in other formats, and your process may fail, requiring you to start all over again.

Once you have everything in a single place, you can get the needed PFX file with this command:

openssl pkcs12 -export -in both.crt -inkey my.key -out www-example-com.pfx

Finally, you should have a completed .pfx file, which includes not just the final certificate but the full chain of authority. (Facebook and Firefox both require full chains of authority, so if you only include the final certificate, it may work on some browsers but others will find it “untrustworthy”.)

This final .PFX file can then be uploaded to Azure. And voila! Use the Facebook Linter and force a refresh of the page and it’ll be happy again.

How was your weekend? 🙂

Useful articles in this journey:

https://www.ssl.com/how-to/manually-generate-a-certificate-signing-request-csr-using-openssl/

and this one in particular:

https://stackoverflow.com/questions/22618108/adding-an-intermediate-certificates-to-a-pkcs12-file

Steve Murch

Steve’s a Seattle-based entrepreneur and software leader, husband and father of three. He’s American-Canadian, and east-coast born and raised. Steve has made the Pacific Northwest his home since 1991, when he moved here to work for Microsoft. He’s started and sold multiple Internet companies. Politically independent, he writes on occasion about city politics and national issues, and created voter-candidate matchmaker Alignvote in the 2019 election cycle. He holds a BS in Applied Math (Computer Science) and Business from Carnegie Mellon University, a Masters in Computer Science from Stanford University in Symbolic and Heuristic Computation, and an MBA from the Harvard Business School, where he graduated a George F. Baker Scholar. Steve volunteers when time allows with Habitat for Humanity, University District Food Bank, Technology Access Foundation (TAF) and other organizations in Seattle.

read more

stevemurch.com

I have an old private blog from more than a decade ago that I’m shutting down. It was basically a semi-private journal, related to the construction of our home. It has a lot of useful photographs for me on it — i.e., in-progress construction shots, design inspiration shots, sketches and renderings, plumbing and electrical rough-ins, etc. I wanted to archive them.

I tried a few free tools to download the images. I found various freeware programs as well as free Chrome extensions, but for some reason they all seem to stop at the “thumbnail” image, and did not crawl through to get the full, highest-resolution image.

Enter Scrapy

Nowadays, when there’s a quick “power tool” automated task you’d like to perform, there’s very likely a python library that’ll help with it. So I was happy to discover the excellent Scrapy library, which is a spider/crawling framework.

There’s also BeautifulSoup, and, in the .NET world, HTMLAgilityPack, which are very good at scraping pages… but Scrapy comes with a full spidering framework, letting you crawl the website to fetch what you want with minimal code.

Using Python 3.7+, first, create a virtual environment, since python library conflicts are a pain in the neck:

virtualenv venv

Activate the virtual environment; to do so, on Windows it’s:

.\venv\scripts\activate.bat 

Then, it’s as simple as:

pip install scrapy

Then, simply create a python file in that directory with the code. Here’s mine; I called this file “blogimages.py”. Note that I took some quick-and-dirty shortcuts here because I just wanted this process to work for this specific one-time task. Obviously it can be further generalized:

import scrapy
import urllib.request 

class ImageSpider(scrapy.Spider):
    name = 'images'
    start_urls = [
        'http://sample-blog.blogspot.com/',
        'http://sample-blog.blogspot.com/2005/01/',
        'http://sample-blog.blogspot.com/2005/02/'
    ]

    def parse(self, response):
        # find matching image links -- grab all images
      

        for imgurl in response.xpath("//a[contains(@href,'jpg')]"):
            the_url = imgurl.css("a::attr(href)").extract()[0]
            print(the_url)
            filename = the_url[the_url.rfind("/")+1:]
            print(filename)
            filename = filename.replace("%","") #get rid of bad characters for filenames
            filename = filename.replace("+","")
            print("======")
            urllib.request.urlretrieve(the_url, "images\\"+filename)

            yield {
                'url': imgurl.css("a::attr(href)").extract()[0]
            }


        #next_page = response.css('ul.archive-list li a::attr("href")').get()
        #print("NEXT PAGE ==== ")
        #if next_page is not None:
        #    yield response.follow(next_page, self.parse)

Make a subfolder called “images”, because, as you can see in the hacky code above, it’s going to try to save to that folder.

To run it, all you do is:

scrapy runspider blogimages.py -o images.json

That’s it! The images should now be in your images folder. You can certainly enhance this spider easily to automatically find and crawl the pagination links; I chose not to do that, because, well, I never put pagination links in the old blog — only an “archive” section with a table of contents. I simply fed a list of “start_urls” in the upper portion to traverse.

Steve Murch

Steve’s a Seattle-based entrepreneur and software leader, husband and father of three. He’s American-Canadian, and east-coast born and raised. Steve has made the Pacific Northwest his home since 1991, when he moved here to work for Microsoft. He’s started and sold multiple Internet companies. Politically independent, he writes on occasion about city politics and national issues, and created voter-candidate matchmaker Alignvote in the 2019 election cycle. He holds a BS in Applied Math (Computer Science) and Business from Carnegie Mellon University, a Masters in Computer Science from Stanford University in Symbolic and Heuristic Computation, and an MBA from the Harvard Business School, where he graduated a George F. Baker Scholar. Steve volunteers when time allows with Habitat for Humanity, University District Food Bank, Technology Access Foundation (TAF) and other organizations in Seattle.

read more

stevemurch.com