Swim #5

ruferp · 2021-11-17T07:25:30Z

This creates version 1.2 of the swi tools.
It will support EOS images starting with release 4.27.2 (Uppsala).
SWI images in 4.27.2 contain multiple images, and thus also need multiple signatures.
A published 4.27.2 SWI is a universal image ("single image" for all platforms) that contains baby images.
When such an image is downloaded to a switch, it might get downsized (bits not relevant to local hardware platform will be dumped even before hitting the flash, which might otherwise be too small).

Because of the need for multiple signatures, the current prepare/sign mechanism is not practical and was modified.
If a signing key is provided on the command line, no difference, we just use it to sign all images, and will add a null-signature into each of them firsts (and replace any already present signature by force).
But if no such key is provided, instead of printing a litany of digests and asking for them all to be signed before proceeding with the next step, we assume there is a binary called 'swi-signing-server' that can be given the digest and will return the signature. An example signing server script is provided (nothing glorious, uses a local key).
That is, for 4.27.2+ images, there is no "swi prepare", only a "swi sign".
If the image is a pre 4.27.2 image, the "swi prepare" stage is still required as before.

Here are example usages:

Swim Image
==========

With a local signing key provided on command-line
-------------------------------------------------

> swi-signature sign EOS.swi /etc/swi-signing/signing.crt /etc/swi-signing/root.crt --key /etc/swi-signing/signing.key
Optimizations in EOS.swi: Default Sand-4GB Strata-4GB
Default sha256: a3276b9976bb2471838dc95fbd2a38dcf1e7e5510bcfa8dfe0f0eff8b935a709
Sand-4GB sha256: 4d0d23293eaecc7f55e7ee9776b0c250bef1a335e1b4ea28ef4eea989eb917f9
Strata-4GB sha256: 520cbde6d60d1089bfbc95d9086c47ce4a6fc0399a5b8a29fc6c1bd95c7d029a
Adding signature files to EOS.swi: Default.signature Sand-4GB.signature Strata-4GB.signature
EOS.swi sha256: 913eb842e408ceddea914543230c9e13ff63e360fc6a6735ef9c0c1571209fb3
SWI/X file EOS.swi successfully signed and verified.

> verify-swi EOS.swi --CAfile /etc/swi-signing/root.crt
Optimizations in EOS.swi: Default Sand-4GB Strata-4GB
Default: SWI/X verification successful.
Sand-4GB: SWI/X verification successful.
Strata-4GB: SWI/X verification successful.
SWI/X verification successful.

Using the signing server (no key provided on command-line)
----------------------------------------------------------

> which swi-signing-service
/usr/local/bin/swi-signing-service

> swi-signature sign EOS.swi /etc/swi-signing/signing.crt /etc/swi-signing/root.crt
Optimizations in EOS.swi: Default Sand-4GB Strata-4GB
Default.swi sha256: 780bb2154f5ae75b4f43d72c5e3859bf73ed1c68c578f4a526aac681f94fe016
Sand-4GB.swi sha256: ac8da1ecc9058b90cb66a0192e7ac878efaa9ad2bc7a52bd1c584b16ca68bd7d
Strata-4GB.swi sha256: cdbecdce8450df37f6dba48c631a84ef1cab116a219c27ff432b484566d796b8
Adding signature files to EOS.swi: Default.signature Sand-4GB.signature Strata-4GB.signature
EOS.swi sha256: 24c96413165d13c7b702aa75ff23d7264526be44f00d3bd4aee403c66ab905aa
SWI/X file EOS.swi successfully signed and verified.

> verify-swi EOS.swi --CAfile /etc/swi-signing/root.crt
Optimizations in EOS.swi: Default Sand-4GB Strata-4GB
Default: SWI/X verification successful.
Sand-4GB: SWI/X verification successful.
Strata-4GB: SWI/X verification successful.
SWI/X verification successful.

Legacy image (non modular)
===========================

With local signing key provided
-------------------------------

> swi-signature sign EOS.swi /etc/swi-signing/signing.crt /etc/swi-signing/root.crt --key /etc/swi-signing/signing.key
EOS.swi sha256: edf05da38ca66e0eb95f7fc910b46cb00feba9765ccf460587881a6f40e609a1
SWI/X file EOS.swi successfully signed and verified.

> verify-swi EOS.swi --CAfile /etc/swi-signing/root.crt
SWI/X verification successful.

Using the signing server (no key provided)
------------------------------------------

> swi-signature sign EOS.swi /etc/swi-signing/signing.crt /etc/swi-signing/root.crt
EOS.swi sha256: abb8ca2bdadc39b7cd26648eafcf827a0af0674fcf37f80861959bd6b6e989af
SWI/X file EOS.swi successfully signed and verified.

> verify-swi EOS.swi --CAfile /etc/swi-signing/root.crt
SWI/X verification successful.

Using the provided signature file (computed by other means)
------------------------------------------

> unzip EOS.swi swi-signature
> cat swi-signature | grep Signature: | sed 's/Signature://' > sig

> swi-signature sign EOS.swi /etc/swi-signing/signing.crt /etc/swi-signing/root.crt --signature=./sig
SWI/X file EOS.swi successfully signed and verified.

> verify-swi EOS.swi --CAfile /etc/swi-signing/root.crt
SWI/X verification successful.

…otfs (swim)

…e when a signing key is provided

README.md

switools/signaturelib.py

johnrclarke

You'll see that most of my suggestions in this review involve removing the use of shell scripts run with os.system. They're ugly, inefficient and prone to failure due to the shell's parsing of the constructed command line e.g. simply adding a space into the SWI file path will break some of them. Also, although this is probably not a concern in these tools, as a general rule I prefer to avoid them because they open up the possibility of a shell injection vulnerability. Rather than concatenating strings to form a command line which is then interpretted and split by the shell, I think it's better either use python modules where we can, or to use the subprocess module to run any external executables directly.

Also, apologies if I've made any typos or mistakes in my suggested changes. I did run pylint and flake8 over it to make sure I didn't do anything obviously stupid, but I haven't actually run the modified code and made sure it works.

johnrclarke · 2021-11-24T04:46:44Z

switools/swisignature.py

+def extractSignature( swi, destFile, sigFileName=None ):
+   sigFn = sigFileName
+   if not sigFn:
+      sigFn = signaturelib.SWI_SIG_FILE_NAME
+   ret = os.system( "set -e;"
+                    "destDir=$(readlink -f $(dirname %s));"
+                    "destFile=$(basename %s);"
+                    "swi=$(readlink -f %s);"
+                    "sigFile=%s;"
+                    "cd $destDir;"
+                    "unzip -o -q $swi $sigFile;"
+                    "mv $sigFile $destFile" % (
+                      destFile, destFile, swi, sigFn )
+                  )
+   if ret != 0:


Suggested change

def extractSignature( swi, destFile, sigFileName=None ):

sigFn = sigFileName

if not sigFn:

sigFn = signaturelib.SWI_SIG_FILE_NAME

ret = os.system( "set -e;"

"destDir=$(readlink -f $(dirname %s));"

"destFile=$(basename %s);"

"swi=$(readlink -f %s);"

"sigFile=%s;"

"cd $destDir;"

"unzip -o -q $swi $sigFile;"

"mv $sigFile $destFile" % (

destFile, destFile, swi, sigFn )

)

if ret != 0:

def extractSignature( swi, destFile, sigFileName=signaturelib.SWI_SIG_FILE_NAME ):

try:

destDir = os.path.dirname( destFile )

with zipfile.ZipFile( swi ) as zf:

zf.extract( sigFileName, destDir )

os.rename( '{}/{}'.format( destDir, sigFileName ), destFile )

except Exception:

Latest code uses subprocess everywhere.
Note that when somebody runs "swi-signature " at a bash prompt he does not need to craft a malicious to cause mayhem, he can just run it directly...

zipfile.py is very nasty: it requires zip version 2.0 to extract things it inserted...

Which is why I only suggested using zipfile when extracting files or checking the contents of an existing zip 😃

My dislike of os.system isn't just its potential for allowing shell injection attacks, it's also that concatenating strings then splitting them again is prone to failure if one of the substrings contains a space e.g. in a pathname, and as there's a perfectly good alternative available (i.e. subprocess) then why not use it?

Well, we can quote the path and the swi in the os.system call if some people ask for troubles and put spaces in filenames...

Which is why I think it's better to use subprocess.check_call and pass it a list so it doesn't matter if any arguments contain spaces (or any other special characters).

The weird part is that only one of the inner signature fails with your code

I can't see any obvious reason why that change would cause this. I'll try to reproduce it.

I can't reproduce this: it verifies ok every time I run it. What I do notice though is that all of the sha256 sums are different each time I run it on a SWI. I've tested both your branch as it is now, and the previous commit with all of my suggested changes applied, and I get similar results from each.

(venv) [johnc@us200 ~/work]$ swi-signature prepare --outfile ./EOS.swi --force-sign ../EOS.swi 27210b1056d406e09da97b4244a7c4443226fb0d04bceb0bc76d36a4edbade5b (venv) [johnc@us200 ~/work]$ swi-signature sign EOS.swi /etc/swi-signing-devCA/signing.crt /etc/swi-signing-devCA/root.crt --key /etc/swi-signing-devCA/signing.key Optimizations in EOS.swi: Default Sand-4GB Strata-4GB Default sha256: cf9f3e15ca740a54578a6031325e5cdb88122f912565270b86a74f58813afb06 Sand-4GB sha256: 93fba52a4f5761dcc2752c7ee6929ec0925643ace9a30a819f5b77a331897ecb Strata-4GB sha256: ea8adf67fbc7a6c679475ba7b99bc7a578b18a1748cdf6dec72a05b5685b2bb2 Adding signature files to EOS.swi: Default.signature Sand-4GB.signature Strata-4GB.signature EOS.swi sha256: 731ab15ef7eb784def9162efc6418dba881e02175aa25d5d7a90d38ab95debd7 SWI/X file EOS.swi successfully signed and verified. (venv) [johnc@us200 ~/work]$ verify-swi EOS.swi --CAfile /etc/swi-signing-devCA/root.crt Optimizations in EOS.swi: Default Sand-4GB Strata-4GB Default: SWI/X verification successful. Sand-4GB: SWI/X verification successful. Strata-4GB: SWI/X verification successful. SWI/X verification successful. (venv) [johnc@us200 ~/work]$ (venv) [johnc@us200 ~/work]$ (venv) [johnc@us200 ~/work]$ rm EOS.swi (venv) [johnc@us200 ~/work]$ swi-signature prepare --outfile ./EOS.swi --force-sign ../EOS.swi c4f84e794a5d7404f13099f9ce9b25cdfe76c264f47066ee3beb64cc172decab (venv) [johnc@us200 ~/work]$ swi-signature sign EOS.swi /etc/swi-signing-devCA/signing.crt /etc/swi-signing-devCA/root.crt --key /etc/swi-signing-devCA/signing.key Optimizations in EOS.swi: Default Sand-4GB Strata-4GB Default sha256: 219a21b01e0a4b074ea0c7098d3566ad89276c39e7aa879e8f6efc185cacefb7 Sand-4GB sha256: aa438f6d7b96c682d0bcca21195d4805bdc926c5da6992599ed0ada363323001 Strata-4GB sha256: 6d1f64b584e5abb599300c296c0bbeb9fbfd5c7d304f23c990d3d707b2d1858c Adding signature files to EOS.swi: Default.signature Sand-4GB.signature Strata-4GB.signature EOS.swi sha256: 5408ae5c0501f10dbf22c39bc5b2cdae705faf9fd46847f7d0eec02d0671b0d2 SWI/X file EOS.swi successfully signed and verified. (venv) [johnc@us200 ~/work]$ verify-swi EOS.swi --CAfile /etc/swi-signing-devCA/root.crt Optimizations in EOS.swi: Default Sand-4GB Strata-4GB Default: SWI/X verification successful. Sand-4GB: SWI/X verification successful. Strata-4GB: SWI/X verification successful. SWI/X verification successful.

I've pushed a branch with all of my suggested fixes applied to my fork of this repo: https://github.com/johnrclarke/swi-tools.git, branch swim-fixes

The new null-signature we insert each time is probably what changes the sha256 each time, because of the timestamp.

johnrclarke · 2021-11-24T04:49:30Z

switools/signaturelib.py

+def getOptimizations( swi, workDir ):
+   ret = os.system( "set -e; swi=$(readlink -f %s); cd %s; "
+                    "unzip -q -o $swi swimSqshMap" % ( swi, workDir ) )
+   if ret:
+      return None # legacy image
+   optims = []
+   with open( "%s/swimSqshMap" % workDir ) as f:
+      for line in f:
+         optim, _ = line.split( "=", 1 )
+         optims.append( optim )
+   os.system( "rm %s/swimSqshMap" % workDir )
+   return optims


If the swi parameter is changed to be a zipfile.ZipFile object (as suggested later in this review), then this can be simplified to:

Suggested change

def getOptimizations( swi, workDir ):

ret = os.system( "set -e; swi=$(readlink -f %s); cd %s; "

"unzip -q -o $swi swimSqshMap" % ( swi, workDir ) )

if ret:

return None # legacy image

optims = []

with open( "%s/swimSqshMap" % workDir ) as f:

for line in f:

optim, _ = line.split( "=", 1 )

optims.append( optim )

os.system( "rm %s/swimSqshMap" % workDir )

return optims

def getOptimizations( swi ):

optims = []

if 'swimSqshMap' in swi.namelist():

for line in swi.read( 'swimSqshMap' ):

optim, _ = line.split( "=", 1 )

optims.append( optim )

return optims

So I made a couple of mistakes in the above code. It should be:

for line in swi.read( 'swimSqshMap' ).splitlines(): optim, _ = line.decode().split( "=", 1 )

yeah, I noticed...

johnrclarke · 2021-11-24T04:51:12Z

switools/signaturelib.py

+def extractSwadapt( swi, workDir ):
+   ret = os.system( "set -e; image=$(readlink -f %s); cd %s;"
+                    "unzip -o -q $image swadapt" % ( swi, workDir ) )
+   if ret:
+      print( "Error: '%s' does not contain the 'swadapt' utility" % swi )
+      shutil.rmtree( workDir )


As above, changing the swi parameter and accepting @mbwang 's suggestion to move the exit out of this function, this can become:

Suggested change

def extractSwadapt( swi, workDir ):

ret = os.system( "set -e; image=$(readlink -f %s); cd %s;"

"unzip -o -q $image swadapt" % ( swi, workDir ) )

if ret:

print( "Error: '%s' does not contain the 'swadapt' utility" % swi )

shutil.rmtree( workDir )

def extractSwadapt( swi, workDir ):

if 'swadapt' not in swi.namelist():

return False

swi.extract( 'swadapt', workDir )

return True

uhm, indeed, that's nicer, especially given unzip's nasty willingness to abide by its own -qq argument (very quiet) and printing warnings when the file to extract does not exist....

zipfile is so much nicer to use than unzip. One cool thing I was able to do with it was to mount squashfs files contained in a swix without extracting them. You can't do that with unzip 😃

zipfile doesn't preserve permissions when extracting so this needs to be added after swi.extract:

os.chmod('{}/swadapt'.format(workDir), 0o755)

So, while zipfile is nice to use and makes some things very easy, it has a couple of bugs which make it unsuitable for some of the things we need to do in these scripts, so I'll try to mark those suggestions as resolved. It might still be useful in some places (e.g. it's easier to check if a file is in the archive with zipfile), but not in others.

switools/swisignature.py

switools/verifyswi.py

johnrclarke · 2021-11-24T05:29:23Z

switools/signaturelib.py

 # that can be found in the LICENSE file.

 from __future__ import absolute_import, division, print_function
+import os


Can be removed if you make the changes I've suggested below.

switools/swisignature.py

Don't use os.system, let people have images called $(reboot). Do use tempfile to create temp directories. Don't call sys.exit(), let it unravel (and cleanup) where it may via exception. Not a review comment, but print errors to stderr

switools/signaturelib.py

johnrclarke · 2021-11-25T06:11:51Z

switools/signaturelib.py

+   return True

+def getOptimizations( swi, workDir ):
+   # unzip spits warnings when extracting non-existant files, so check first :-(


Please use zipfile here to read the map instead of running unzip twice.

johnrclarke · 2021-11-25T06:12:08Z

switools/signaturelib.py

+   return optims
+
+def extractSwadapt( swi, workDir ):
+   cmd = "unzip -o -qq %s swadapt" % os.path.abspath( swi )


Please use zipfile.

Forget that, zipfile doesn't restore the execute permissions, and although you can workaround that with os.chmod it's a bit ugly (although whether it's more ugly than running unzip is debatable 😛 )

johnrclarke · 2021-11-25T06:12:20Z

switools/signaturelib.py

+   if not os.path.isfile( swi ):
+      return False
+   # unzip spits warnings when extracting non-existant files, so check first :-(
+   cmd = "unzip -Z1 %s" % swi


zipfile again.

I was going to drop this comment, but then I can't see where you use version after extracting it, so maybe zipfile would be simpler here?

switools/swisignature.py

switools/swisigningservice.py

Added swim_test.sh: this is a bash script that will run several test cases and record the output to a log file. Added a test mini swi in new modular format (3.0) as well as one in legacy format (1.0). Added swim_test.ref, the reference output. At the very end, the test will compare the logfile to the reference. If there is an error, and the displayed diff context is too small, just vim -d swim_test.log swim_test.ref. The reference file can also be seen as further examples of usage, or a quick way to see what was tested and how it all looked like.

Fixup some comments in the test

johnrclarke · 2021-12-03T04:56:30Z

switools/swisignature.py

 def insertSignature( swi, sigFileName, workDir ):
-   cmd = "zip -q -0 -X %s %s" % ( os.path.abspath( swi ), sigFileName )
+   cmd = [ "zip", "-q", "-0", "-X", os.path.abspath( swi ) ]
+   cmd = cmd + sigFileName.split( " " )


It would be better to pass sigFileName as a list rather than joining then splitting the names.

Peter Rufer added 5 commits November 12, 2021 02:09

Update signing tools for working with images that contain multiple ro…

50aa46f

…otfs (swim)

Add the signing-service demo script

5f7a102

Add errors for signing server case, correctly handle legacy image cas…

16cee66

…e when a signing key is provided

Handle pre swadapt legacy swi (swimSqshMap not trustable)

7816bde

Update the documentation

8ae6d90

ruferp requested a review from mbwang November 18, 2021 18:04

mbwang reviewed Nov 19, 2021

View reviewed changes

README.md Outdated Show resolved Hide resolved

mbwang reviewed Nov 22, 2021

View reviewed changes

switools/signaturelib.py Outdated Show resolved Hide resolved

mbwang reviewed Nov 22, 2021

View reviewed changes

switools/signaturelib.py Outdated Show resolved Hide resolved

review comments

c8c96f3

johnrclarke suggested changes Nov 24, 2021

View reviewed changes

Review comments from mbwang and johnrclarke

3043446

Don't use os.system, let people have images called $(reboot). Do use tempfile to create temp directories. Don't call sys.exit(), let it unravel (and cleanup) where it may via exception. Not a review comment, but print errors to stderr

johnrclarke suggested changes Nov 25, 2021

View reviewed changes

Peter Rufer added 4 commits November 27, 2021 23:06

Use zipfile where possible/simpler

7000c4b

Fixup some comments in the test

Allow directory paths to have spaces in them

5a2971e

apply leftover review comments

a89ab60

johnrclarke reviewed Dec 3, 2021

View reviewed changes

Left over review comments, update readme.

0b507b8

ruferp merged commit b5f087e into master Apr 7, 2022

Swim #5

Swim #5

Uh oh!

Conversation

ruferp commented Nov 17, 2021

Uh oh!

Uh oh!

Uh oh!

Uh oh!

johnrclarke left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

johnrclarke Nov 25, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

johnrclarke Nov 25, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

johnrclarke left a comment •

edited

Loading

johnrclarke Nov 25, 2021 •

edited

Loading

johnrclarke Nov 25, 2021 •

edited

Loading