...someplace, where there isn't any trouble? Do you suppose there is such a place, Toto?

Downloading all msdn magazine issues in one fell swoop

Unfortunately, in late 2019 the venerable msdn magazine's last issue appeared. In the mid nineties I had a subscription of its predecessor, the MSJ, which was later converted into an msdn magazine subscription. For archiving purposes I was looking for a way to download all past issues of msdn magazine in some easy way and decided to massage the download document on the web (which might disappear without any notice at any time) a bit in order to extract the raw download URLs. The result is this batch file that requires curl to be found via the PATH environment variable. It downloads a whopping 3.3GB of pdf and chm files that fit neatly on a writable 4.7 GB DVD. Download your copy of all msdn magazine articles before the download links go the way of all flesh...

The original and official page on the web that I parsed for getting all download links is this.

Building OpenSSL for Android on Windows with the Visual Studio 2017 native Toolchain and Cygwin

Building OpenSSL is an arcane thing to do sometimes. I have nowhere on the web found a real simple solution to build OpenSSL for Android that met the number one requirement for me: Being able to build on a build host (or development host) that runs Windows. I simply do not want to maintain another build machine running some Linux flavor only in order to be able to build an OpenSSL static library and a shared library that uses the former. I would rather want to use Visual Studio 2017 with its support for native C/C++ development which also suggests using the same toolchain for building the static OpenSSL library and the shared libraries that are going to consume it. Since I am planning to use Xamarin for my mobile apps, having the entire development and build process on a Windows box would be a big win for me.

 

DISCLAIMER: I have not yet tried, whether the OpenSSL libraries produced with my set of scripts and configuration files actually work in a real world Android app. But the build procedures themselves succeed and create the familiar libcrypto.a, libcrypto.so.1.0.0, libssl.a and libssl.so.1.0.0 files which are also all in the correct binary format as you can easily verify running readelf on them (the build script actually does this).

 

Challenges

Since OpenSSL for Android is usually built on a Linux host using Google's NDK, its entire build procedure also heavily relies on infrastructure that is usually found on a Linux box: Shell scripts, PERL, symbolic links and build tools like make and friends. None of these infrastructure elements can be found on a standard Windows box and things like symbolic links have been unfeasible until the arrival of Windows 10 Build 10.0.15063, at least for non-administrative accounts (you do develop and build as a LUA user, don't ya?). So we need a Unix-like environment for this purpose running on Windows plus some other arcane preconditions set which we will see in a moment. WSL cannot be used for this purpose because it only allows 64-bit Linux applications to be executed. However, the NDK that Google ships for Linux comes with a number of 32-bit applications which are not a problem when executed on a "real" installation of Linux but call for trouble on WSL. So the only option left over for this purpose is Cygwin.

 

Using the scripts

In order to use the scripts I have written, follow theses steps:

  • Make sure all preconditions are fulfilled, as described in the paragraph below
  • Unpack the zip File (Download) into a writable directory, make sure that you have the "Change permissions" right for the directory and all its files and subdirectories granted, otherwise the calls to chmod in the build scripts will fail (cygwin places some strange NULL SID ACEs into the DACL of files that are granted execution right using chmod which requires the "Change permissions" access right)
  • Download openssl-1.0.2q.tar.gz from openssl.org (Download) and copy it into the directory created in the previous step.
  • Open a windows command prompt and change the current directory to the directory from the above steps
  • Execute "build.bat" and wait (on my Ryzen Vega this takes something like 22 minutes)
  • Find the header files and libraries for the build in a platform specific subdirectory like arch-x86, arch-arm in the subdirectory build\<android-api-version>

 

Preconditions

 In order to use the build scripts, you need Microsoft Visual Studio 2017 installed with the "Mobile development with C++" workload enabled. Unless disabled during Visual Studio Setup, this will create subdirectories like these in c:\Microsoft\AndroidNDK64: android-ndk-r12b, android-ndk-r13b, android-ndk-r15c. You might have them also in the directory c:\Microsoft\AndroidNDK. These directories contain the NDK in different versions and I assume that later versions of Visual Studio will ship with even newer Versions of the NDK than the ones mentioned here. The scripts of mine will assume that the r15c-Version of the NDK is installed in c:\Microsoft\AndroidNDK64\android-ndk-r15c but you can set your installation directory of the NDK prior to execution of the build scripts by setting the OPENSSL_ANDROID_NDK_ROOT environment variable in the command prompt to something like c:\Microsoft\AndroidNDK64\android-ndk-r12b like this:

 

set OPENSSL_ANDROID_NDK_ROOT=c:\Microsoft\AndroidNDK64\android-ndk-r12b

You also have to install cygwin, I have not tried with the 32-bit version of cygwin but I assume it works as well. Cygwin is a huge beast and we have to use only a tiny fraction of it for our purposes. Unfortunately is installer is not even digitally signed with an Authenticode signature, so be sure to check it with its signature file and a tool like Kleopatra in order to verify authenticity. When installing cygwin you can use the standard installation options (which will create a bash environment with PERL and other good stuff) but be sure to also install the following programs: dos2unix, makedepend, automake. We need dos2unix in order to convert a dynamically created shell script from CRLF-notation to LF-notation consumable by the cygwin bash and we need makedepend and automake for building via the OpenSSL shell scripts. The build scripts of mine assume that cygwin is installed in the  c:\cygwin64 directory but you can override this much like with the above OPENSSL_ANDROID_NDK_ROOT environment variable. So, e.g. if you have installed cygwin into d:\cygwin64, type the following in the command prompt prior to execution of build.bat:

set CYGWIN_PATH=d:\cygwin64

You might want to create a permanent CYGWIN_PATH environment variable for you development or build machine unless it is installed at the default location c:\cygwin64.

 

Since the build scripts for OpenSSL on Linux make heavy use of symbolic links for header files, you should also turn on "Developer Mode" on your build and development host (which should be Windows 10 10.0.15063 or later), otherwise the build scripts cannot be executed successfully by a standard user. From Windows 10 10.0.15063 on, symbolic links can be created on Windows machines without administrative permissions if the box is running in "Developer Mode".

 

How it works

build.bat will invoke creatcfg.bat and buildall.bat in succession. The purpose of creatcfg.bat is to create the Setenv-build.sh build script and the buildall.bat batch file with the configuration set via the CYGWIN_PATH and the OPENSSL_ANDROID_NDK_ROOT environment variables. While we are at it: You can also set the Android API version to be used for the build with the ANDROID_API environment variable (default value: android-21) like this prior to execution of build.bat:

set ANDROID_API=android-26

buildall.bat, which is created by creatcfg.bat is then executed by build.bat and will invoke the cygwin bash three times with arguments to execute. The first one will convert the newly created Setenv-build.sh from CRLF notation to LF notation with dos2unix, so the cygwin bash can consume it properly. The second will execute "chmod a+x" for all shell script files (this is the place where you will get an Access Denied error if the user executing this script does not have the "Change permissions" access right). And the last one will execute the buildall.sh shell script.

buildall.sh will first clean up the build subdirectory, if it exists and will recreate it and grant everyone full control to the build subdirectory (ok, this is cargo cult, I don't really know if we really need this with developer mode turned on and the "Change permissions" access right set as documented before). After that it will invoke buildplatform.sh or each Android hardware platform and pipe stdout and stderr into a log file (arm64_build.log, arm_build.log, mips64_build.log, mips_build.log, x86_64_build.log and x86_build.log).

buildplatform.sh takes the build architecture as an argument and will first invoke the dynamically created shell script Setenv-build.sh, which will set a couple of environment variables that depend on your cygwin installation path, the NDK path being set and the Android API version chosen, as explained above. After that, buildplatform.sh will invoke an architecture specific configuraton file like  Setenv-mips.sh which will set architecture specific environment variables. It then creates a subdirectory for the architecture value passed as an argument and will then unpack the openssl tarball into this directory. It is the buildplatform.sh shell script where you also would want to change the name of this tarball once a newer version as 1.0.2q has been released, simply by changing the value of the OPENSSLNAME variable.

After the tarball has been unpacked, the buildplatform.sh script will copy the Configure file from my set of script files over the one that comes with the tarball. The reason is that the Configure file in the tarball is missing entries for the x86_64 build, the arm64 build and the mips64 build. You might want to check, once a newer version than 1.0.2q is used, if this copy step is still necessary and if it doesn't break other builds, e.g. if the compiler options for GCC are changed in the standard Configure file from the tarball.

After that, the buildplatform.sh will perform the usual OpenSSL build sequence of

 

perl -pi -e 's/install: all install_docs install_sw/install: install_docs install_sw/g' Makefile.org
perl configure $OPENSSL_CONFIG_ARCH shared no-asm no-ssl2 no-ssl3 no-comp no-hw no-engine make depend
make all

in order to build the platform (OPENSSL_CONFIG_ARCH is a variable denoting the hardware platform name).

Once a hardware platform is built with make all, the buildplatform.sh script invokes readelf for the generated library files so you can verify in the platform build log, that the created binaries have the proper format. The subsequent make install_sw step will then copy the build output into the build subdirectory. For some reason, this step fails, as you can see in the log files, but library files and header files are copied successfully, so I assume this step is complete enough to be usable.

Fun fact: Try to observe a running build with Process Explorer and the image path being shown as a column. You will then notice that the actual make.exe being invoked in the build process is not the one from cygwin but instead a binary that is part of the Visual Studio NDK installation.

 

 

Leftover stuff

I would guess that the same result with my scripts could be achieved with the Windows version of Google's NDK. Drop me a note if you would like to share your results. Initially, some three years ago, I had started creating a build recipe for OpenSSL on Windows with Cygwin which finally emerged into today's version of the build scripts, so simply pointing the OPENSSL_ANDROID_NDK_ROOT environment variable to the location where the Google NDK is installed might already work. At some point I decided to give the Microsoft Compilers a chance and this worked out pretty flawlessly. This also relieved me from having to install yet another SDK on my development and build boxes, this time Google's NDK.

 

License

All the files I have written use the OpenSSL license, so nothing should change for you if you are already using OpenSSL.

Witzle Punkt IM, "Fuckup as a Service"

Nexus vom CCC Hannover nimmt  whistle.im auseinander und nimmt dabei keine Gefangenen. Nee, nee, Jungs. Krypto ist nicht so einfach wie Ihr denkt. Dass whistle.im broken by design ist, war mir schon, ohne mich naeher damit zu beschaeftigen, schon klar, als da deren Crypto als JavaScript im Browser angekuendigt wurde. Wegwerfen, das Ganze, ein paar gute Buecher ueber Crypto lesen und dann einen Messenger schaffen, der mit PGP-Keys funktioniert. Und den auditen lassen ohne Ende. So koennte das vielleicht funktionieren.

Certified for Windows 8.1

...der IDERI Note Client. Seit heute. Bin stolz auf mein Baby. Und das von Joe, Alex und Sebastian. Leider kann ich noch keine Logografik dazu abbilden, weil es die anscheinend noch nicht gibt. Wir haben auf der Preview von Ende Juni getestet, einen Waiver gut begründet und scheinen einer der ersten zu sein, die überhaupt das Logo bekommen, denn Windows 8.1 wird offiziell ja erst im Oktober releast. Wir wollen früher am Start sein und im September releasen :-)

 

Surprise, surprise! The windows installer lies!

Link: https://blogs.msdn.com/b/cjacks/archive/2009/05/06/why-custom-actions-get-a-windows-vista-version-lie-on-windows-7.aspx

Yesterday I lost sort of like two hours of work because of a behaviour of the Windows Installer that I haven't come across before. I was testing an installer of mine on Windows 7 and I was really surprised from the behaviour of a Custom Action (CA) in my installer. Inside the CA my code determined if it is running on Windows 7 or later and in that case, the CA was supposed to execute a certain piece of code. But that code never seemed to execute at all. I inspected the code, but there was no apparent bug in it. I wrote a small executable that did the same thing but that binary worked as expected. I added log messages to the code, but those log messages never appeared in the log file. Finally I attached the debugger and was baffled when I stepped over the code that executes the GetVersionEx API, our beloved old friend from kernel32. It returned major version 6 and minor version 0, which is the version number tuple for Windows Vista, not Windows 7. "Oh, stupid", I thought, "you thought you were testing on Windows 7, but it was actually a Window Vista VM". I fired up winver, but it showed that this was clearly Windows 7. So I started scratching my head and googling. Here is, where I found an answer. It turns out that this is on purpose. Beginning with Windows 7, the GetVersionEx API is shimmed with a VersionLie shim in msiexec.exe. Purportedly this is, because so many developers were not competent enough to properly determine the operating system version inside their CAs, so the Windows Installer Team (or the app compat team) decided to always lie to CAs from Windows 7 onwards. CAs will always think they are running on Windows Vista, no matter what the real operating system version is. This is quite unusual because up to now I was under the impression that VersionLies are only applied to third party processes, not MS's own programs, such as msiexec.exe.

The moral of the story is: Never call GetVersionEx from a CA. Do any conditioning by conditioning (yeah, I like to verb words!) your CA itself with the VersionNT property, not by conditioning inside your CA, even if you then end up with twice the number of CAs. I wonder if the fine folks from the WINE project got that right...

1 2 3 4 5 6 7 8 9 10 11 ... 46 >>