Building Python 3.3.4 with Visual Studio 2013

4th revision, April 12, 2014.
A document history can be found at the very end of this page

Downloads

For the impatient:

Note: these builds use updated versions of OpenSSL (1.0.1g) - fix for the heartbleed issue - and SQLite (3.8.4.3), plus tons of other new goodies. Plus document revision 4 adds a new naming scheme, so as to keep the different build versions separate.

Why bother?

Python 3 on Windows is built with Visual Studio 2010 Professional. If you have an application built with Visual Studio 2013 (or any other Visual Studio version), and you have Python embedded in it, you have two problems:

  1. It is a very bad idea to mix different Visual C++ runtimes. See http://siomsystems.com/mixing-visual-studio-versions/ for an explanation. This means if you use the official libraries to link in Python, you are in for some nasty surprises.
  2. The official distribution does not contain debug libraries. For some reason I don’t fully understand the python libraries have different names for debug and release builds (rather than just be located in a different directory, as I would normally approach this). This means that you cannot easily build a working debug build of your project.

So, you need to rebuild Python with the Visual Studio version you are using. In the past (read: in Python 2.x) this process was worse, but it has definitely improved in Python 3.3. In the following post I am going to explain what you need to do.

Build Instructions

Get the sources from http://python.org/download/. At the time of this writing, Python 3.3.4 is the official release, so I will use Python-3.3.4.tar.bz2 in the following.

Extract the files. You’ll get a folder Python-3.3.4 with several subfolders for sources, headers and so on.

Go to the PCBuild folder and open readme.txt in an editor capable of showing unix-style newlines, i.e. ANYTHING BUT NOTEPAD. I personally am a hardcore fan of scite, which is available at http://www.scintilla.org/SciTE.html.

Read readme.txt. No, I mean seriously: read it. In particular, let’s recap the section named Python-controlled subprojects that wrap external projects and follow slowly.

We’ll start at the top, with SQLite:

_sqlite3 Wraps SQLite 3.7.4, which is currently built by sqlite3.vcproj (see below).

Well, 3.7.4 is old – it dates back from 2010. So since we’re going to rebuild everything anyway, why not upgrade sqlite as we go along? (You might not care so much, but me being a heavy SQLite user do). If you open the project sqlite3.vcxproj in an editor (SciTE), you’ll see that the sources are supposed to be in a directory with the variable $(sqlite3Dir), which in turn is kept in pyproject.props. Once you open that in an editor, you’ll notice it defines all the dependent directories as well. Take a look at this:

<PropertyGroup Label="UserMacros">
<PyDllName>python33$(PyDebugExt)</PyDllName>
<PythonExe>$(OutDir)python$(PyDebugExt).exe</PythonExe>
<KillPythonExe>$(OutDir)kill_python$(PyDebugExt).exe</KillPythonExe>
<externalsDir>..\..</externalsDir>
<font color="#FF0000"><sqlite3Dir>$(externalsDir)\sqlite-3.7.12</sqlite3Dir></font>
<bz2Dir>$(externalsDir)\bzip2-1.0.6</bz2Dir>
<lzmaDir>$(externalsDir)\xz-5.0.3</lzmaDir>
<opensslDir>$(externalsDir)\openssl-1.0.1e</opensslDir>
<tcltkDir>$(externalsDir)\tcltk</tcltkDir>
<tcltk64Dir>$(externalsDir)\tcltk64</tcltk64Dir>
<tcltkLib>$(tcltkDir)\lib\tcl85.lib;$(tcltkDir)\lib\tk85.lib</tcltkLib>
<tcltkLibDebug>$(tcltkDir)\lib\tcl85g.lib;$(tcltkDir)\lib\tk85g.lib</tcltkLibDebug>
<tcltk64Lib>$(tcltk64Dir)\lib\tcl85.lib;$(tcltk64Dir)\lib\tk85.lib</tcltk64Lib>
<tcltk64LibDebug>$(tcltk64Dir)\lib\tcl85g.lib;$(tcltk64Dir)\lib\tk85g.lib</tcltk64LibDebug>
</PropertyGroup>

So first we’re going to update sqlite. No risk, no fun! Head over to sqlite.org and download the latest amalgamation sources; at the time of the writing these were 3.8.4.3. Extract the archive, you should have 4 files in there. Now, where to move the files: notice that ..\.. is the externals directory, which is on the same include level as Python-3.3.4. Add a folder sqlite-3.8.4.3 and copy the amalgamation files there.

Next, you’ll also need bzip2-1.0.6. Download the source tarball at http://www.bzip.org/downloads.html and install it in the correct directory.

Next, you’ll also need lzma, which here comes as xz 5.0.3. Google it, ist over at http://tukaani.org/xz/. I didn’t know the site before either, I’ll admit! The latest version seems to be 5.0.5 rather than 5.0.3: take a plunge, download it and copy it to the correct folder, but don’t forget to update $(lzmaDir) variable as well.

Next, openssl. There is a version 1.0.1e in subversion, but 1.0.1f is current, and we want to have a current SSL, don't we? At this point, I should probably point out that I have written together specific instructions on building version 1.0.1g.

We Interrupt Your Regularly Scheduled Program to Bring You This Important Message:

The case of the missing Visual Studio 2013 command prompt

If you are using Windows 8.1 (like I do), then you have probably a bad case of missing Start Menu. Chances are you use something like "Classic Start Menu". An when you do, and you install Visual Studio 2013 - there is no command prompt. AAAAAAAAAAAAAAAAA MICROSOFT! PLEASE! I have lots of love for Microsoft, but ever so often they do something so completely brainless I am starting to pray:

DEAR LORD, GRANT ME THE ABILITY TO PUNCH PEOPLE IN THE FACE OVER STANDARD TCP/IP

At any rate, this article on MSDN explains what you need to do to get your prompt.

Carrying on:

Time to open Pcbuild.sln. I am doing this with Visual Studio 2013, the IDE asks me whether I want to upgrade the projects: yes, I do

Open the batch build options dialog and select all 32-bit debug and release binaries. Batch build ahead!

You’ll see a ton of warnings like this:

blablabla\python-3.3.2\include\pymath.h(22): warning C4273: 'round' : inconsistent dll linkage

This is so because HAVE_ROUND is not defined on Windows; your most simple option is to modify pymath.h to define HAVE_ROUND right before the #ifdef.

After the build completes, for me I get two projects that don’t build: _msi and _lzma

Let’s look at the first one first: _msi fails because fci.lib cannot be found. A bit of google comes up with this: http://support.microsoft.com/kb/310618. BUT THEN YOU CANNOT DOWNLOAD IT. Duh!. More google to the rescue: http://www.pixelsplasher.com/_downloads/software/Microsoft-Cabinet-SDK/. And true to form, it does include a fci.lib. The easiest option is to include a link to the fci.lib to the _msi project settings. Build it, and lo and behold: IT STILL FAILS..

You get this error code: http://stackoverflow.com/questions/14710577/error-lnk2026-module-unsafe-for-safeseh-image. The option is in _msi project properties, Linker, Advanced, “Image Has Safe Exception Handlers: NO”. Rebuild – works. Do it for both the debug and release builds.

Note: the 64-bit build fails for FCI.LIB as well for me. I have no idea why, but it does. I haven't invested the time and energy to fix this: if you really need it, drop me a line.

One down, one left to go: lzma.h. It turns out I should have read the website and just downloaded the pre-built windows binaries. Duh! Once you have these, you can do the build just fine – except for the SAVESH exception, but you already know how to fix that one by now.

That’s it: by now a batch-build for debug and release should work just fine.

Building an updated install

Understanding the mysteries surrounding the Python Debug builds

Why Batch Building both Debug and Release targets fails

The Visual Studio solution contains both Debug and Release targets. These have separate names following the standard conventions: so for example there is a _ctypes.pyd in Release Build, and a _ctypes_d.pyd in Debug Build.

This holds true for allmost all projects in the solution: except for four:

  1. make_buildinfo.exe. This is the one that is causing the problem, and the problem is that this project is built first to create a configuration file used in subsequent builds. And if you do batch-build, then this file is created only once, so its settings are overwritting the settings for the other projects. Solution: do a separate debug and release build, and you'll be fine
  2. w9xpopen.exe. Let me quote the documentation: " Serves as an intermediate stub Win32 console application to avoid a hanging pipe when redirecting 16-bit console based programs (including MS-DOS console based programs and batch files) on Window 95 and Windows 98.". OK, this is for Windows 95, a version that will be 20 years old soon. If you are still use the Win9x series, you don't deserve an upgraded Python build :) so I choose not to include it in the binary distribution.
  3. kill_python.exe. This is a helper program, quote, " for killing lingering python[_d].exe processes before building, thus attempting to avoid build failures due to files being locked.". OK, this is basically the same in both debug and release builds.
  4. make_versioninfo.exe. This file generates the resource file version number, and OK this is the same in debug and release builds, so no problems here.

Modules

The download includes the following modules - sorted alphabetically - that were more or less easy to build. Some of them are included because I use them in my professional life, some are included because they are fashionable, and some are included because people asked for them ;)

beautifulsoup 4.3.2

boto 2.27.0

this module is not compatible with Python 3

Cheetah 2.4.4

this module is not compatible with Python 3

CherryPy 3.2.4

construct 2.5.1

cssselect 0.9.1

cython 0.20.1

dateutil 2.2

Django 1.5.1

dnspython 1.11.1

flask 0.10

httplib2 0.8

itsdangerous 0.23

jinja 2.7.2

lxml 0.20.1

Markdown 2.4

MarkupSafe 0.19

pillow 2.3.0

pip 1.5.4

psycopg2 2.5.2

py2exe 0.6.9

this module is not compatible with Python 3

PyChart 1.39

pycrypto 2.6.1

pyftpdlib 1.3.0

PyGreSQL 4.1.1

this module is not compatible with Python 3

pyodbc 3.0.7

pyOpenSSL-0.13.1

***

pyserial 2.7

pywin32 218

queuelib 1.1.1

reportlab 3.0

requests 2.3.0

Scrapy 0.22.2

setuptools 2.2

simplejson 3.3.3

six 1.5.2

SQLAlchemy 0.9.3

twisted 13.2.0

this module is not compatible with Python 3

virtualenv 1.11.4

werkzeug 0.9.4

wxPython 3.0.0

this module is not compatible with Python 3

w3lib 1.5

zope.interface 4.1.0

Verifying you are using the updated SSL versions:

Python 3.3.4 (default, Apr 12 2014, 18:42:19) [MSC v.1800 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import ssl
>>> ssl.OPENSSL_VERSION
'OpenSSL 1.0.1g 7 Apr 2014'
>>> ...
>>> import OpenSSL
>>> import OpenSSL.SSL
>>> OpenSSL.SSL.SSLeay_version(OpenSSL.SSL.SSLEAY_VERSION)
b'OpenSSL 1.0.1g 7 Apr 2014'

Document history

Back to the build overview

GK, Apr 12, 2014.