Virtualization and Cloud executives share their predictions for 2016.  Read them in this 8th Annual VMblog.com series exclusive.

Contributed by Tom Radcliffe, Director of Engineering, ActiveState

Feather-Weight Full Virtualization for Python

Virtualization is getting simpler. It used to be that virtual machines were heavy-weight application layers on top of the native OS that had to go through all kinds of clever negotiations with the kernel to give them something like native hardware access.

That changed with Linux Containers and the technologies built on top of it, like Docker. Linux Containers give kernel-level support for independent process groups and add kernel-level namespaces that provide many of the benefits of virtualization without the high resource cost.

There are still issues in talking to hardware, though. There's only one physical screen, for example, no matter how cleverly you remap ptty's. So for UI-heavy applications, Docker-ization can prove an exciting experience.

At the bottom end of the spectrum there is lowly virtualenv, a Python-specific tool for isolating various instances of Python in a project-specific way. virtualenv is considerably less sophisticated than Docker: it just copies files to a sub-directory and sets some environment variables appropriately. Furthermore, it's language specific: it will use Python's setup tools to install modules into the virtual environment, but if you're working on a project that requires changes to or installation of non-Pythonic components they will get stuck into /usr/ or /usr/local/ as usual, potentially contaminating the environment for other projects.

Technological progress is driven by a process of teasing out new optima in the complex landscape of features and capabilities in any given problem area. Discovering the sweet spot--the optimal mix of speed, power and flexibility for any given problem--is just that: a process of discovery via time and experimentation with different options.

The move from VMs to containers has illustrated that heavy-weight virtualization is not necessarily the sweet spot for running multiple isolated software environments on the same hardware. Containers, which are a middle-weight solution, are definitely closer to the sweet spot.

So the question is: can we get any closer to the ultimate optimum by going to a featherweight solution? virtualenv as it stands is not a serious challenger to Docker. But is there something in between? A Python-only virtual environment that's almost as complete and portable as those provided by Docker, capable of doing copy-on-write updates for any part of the userland ecosystem, and ideally portable across operating systems without recourse to the kind of additional VM layer that Docker necessarily relies upon?

My prediction for 2016 is that we will see experiments in Python-based featherweight virtualization via enhancements to virtualenv so it can cover non-Pythonic components and thereby provide options for Python developers who want something closer to the full benefits of virtual environments, without the OS-specific and hardware-problematic features of Docker.

##

About the Author

Tom Radcliffe has over 20 years experience in software development and management in both academia and industry. He is a professional engineer (PEO and APEGBC) and holds a PhD in physics from Queen's University at Kingston. Tom brings a passion for quantitative, data-driven processes to ActiveState. He is deeply committed to the ideas of Bayesian probability theory, and assigns a high Bayesian plausibility to the idea that putting the best software tools in the hands of the most creative and capable people will make the world a better place.