Monthly Archives: June 2008

Blocks and Contexts

My background has been in engineering and scientific circles mainly in academia, so programming has been something I have primarily used to solve problems. At Enthought we solve problems, but most of the time we are creating solutions to help others solve problems. We are trying to create the tools that someone like me would want to use.

There are several compelling ideas and tools that are buried in the Enthought Tool Suite that we have used to build commercial scientific applications over the past several years. Two of these ideas are the Block and the Context. These concepts have wiggled into the foundation of almost all of our most recent applications because they seem to be solving fundamental problems with how to integrate scientists and engineers typically procedural workflows into a larger GUI application in flexible and re-usable ways.

Eric developed the ideas behind the Block and the Context over 18 months ago and presented them at the SciPy 2007 conference. His slides for that talk are here. Our implementation of these ideas has been slowly but steadily improving as we have learned more about how to make use of them. They are becoming a common abstraction that we use all the time in thinking about new problems. In this way they are a bit like the reactive programming and automatic GUI building of Traits, the pluggable, interactive plotting of Chaco, and the plugin-based application building of Envisage in their importance.

In order to bring to the surface the Block and Context abstractions that have been buried in the BlockCanvas project, two days ago I created a new project in the Enthought Tools Suite called CodeTools. This project contains the two packages enthought.blocks and enthought.contexts. While I’m thinking about these ideas, I wanted to demonstrate a little bit how you can use these tools as well as showcase a few things that I’ve added to enthought.blocks which makes working with them even easier. I think there are some good ideas here, but I’m not sure our implementation is the best one yet. I’m anxious to engage more people in the discussion about how we should be doing this sort of thing.

To get CodeTools you need to get the latest version of ETS. Instructions for how to do this are here. You don’t actually need to make sure all of ETS is working, Traits is enough except for the units package that is still in BlockCanvas, but which I will moving to SciMath soon.

As explained by Eric in his slides, a Block is a set of executable instructions — an intelligent version of the string you might pass to the exec statement. The intelligence of the blocks come from the code analysis they perform when they are created. With these smarts, you get three very nice things:

  1. Information about the “inputs” required and “outputs” delivered by the code block.
  2. The ability to execute the same code over and over again using different namespaces (thus taking inputs from different sources and delivering outputs to a dictionary-like object of your choice).
  3. The ability to restrict execution to the portion of the code block that depends on a specified set of inputs and/or delivers a specified set of outputs.

Over the past few days, besides moving the blocks into their own project I’ve added a few enhancements to the blocks which makes them easier (in my mind) to use. These enhancements are:

  1. Decorators to turn the body of a function into a Block (or a string that could be later turned into a block).
  2. Separation from the outputs of the variables imported from somewhere. These are now stored as “fromimports” attributes
  3. Cataloging of any variables which are assigned constant expressions in the Block.
  4. A method to return a function from the block given a list of inputs to vary and the list of outputs to return. This function will only execute that portion of the block needed to produce the requested outputs.

With the decorators, you can now create a block very easily:

from enthought.blocks.api import func2block
def block():
    c = a + b
    d = a - b
    g = f + e
    h = f - e

This little decorator means no more little code snippets running
around in strings messing up the syntax highlighting and the tab
indentation capabilities of your favorite editor. There is also a
func2str decorator which will take the body of a function and produce
a string (with the first level of indentation removed).

The other little improvement I want to advertise is the ability of the
block to now return a function that evaluates the (restricted) block
based on the list of inputs and outputs provided. Thus,

func1 = block.get_function(inputs=['a','b'], outputs='c')
func2 = block.get_function(inputs=['f','e'], outputs=['g','h'])

returns two functions equivalent to

def func1(a,b):
    c = a + b
    return c
def func2(f,e):
    g = f + e
    h = f - e
    return g, h

The big deal about blocks is really the code restriction. You can
take a scientists script and re-evaluate (restricted) versions of it
for “what-if” analysis, optimization, or even stochastic processing.

It’s a cool idea, but it will have to be left to another blog to make
this more clear.

My Scientific Tool Stack

Numenta is distributing some of my favorite tools with their NuPIC toolkit including Python itself (on windows), wxPython, IPython, Traits and matplotlib. They’re also building some of their prototype and example applications using Traits. Enthought’s typical tool stack for building scientific apps looks like this (with the stuff Numenta currently distributes in white letters):

Yesterday, I gave a brief talk [.pdf] about the stack of tools that Enthought uses for building scientific applications at Numenta’s HTM Workshop (a gathering of over 200 folks interested in their library which implements Jeff Hawkins’ Hierarchical Temporal Memory ideas). My talk included a very gentle introduction to Traits/TraitsGUI.

I also briefly demonstrated some of the capabilities of the Chaco library (2D visualization) by showing some of the examples, and Mayavi (3D visualization) as an example of a Traits and Envisage (plugin framework) application.

Of particular interest to me today: there was a presentation by Subutai Ahmad showing a nice TraitsGUI-based node inspector for Numenta’s HTM library. It reminded me a lot of the “GUI for free” that happened with the TConfig project.

Compile decorator

The PyPy project has some very nice things in it. Among those things is the translator module. The purpose of this module is to compile a restricted subset of Python (RPython) into C and other lower level languages. Within the PyPy project, the translator is used to compile the PyPy interpreter. However, it is also possible to compile individual Python functions at runtime and dynamically execute the compiled function. The translator module makes this task very easy. I have written a class which allows using the compiled version of a Python functions simply by adding a decorator to the function:

def foo(x, y):
    return x + y

The nice thing about doing things this way is that all the code is pure Python code, and switching between the compiled and uncompiled version of the function is as simple as possible. You can find the definition of this decorator on my personal enthought site.

My Kind of People: Summer Python Gatherings

You used to have to identify fellow “technical” folks by the velcro on their shoes (hey, velcro is very efficient), but today there are a myriad of ways to spot and mingle with like-minded people. So if you agree that an evening sharing tales of code vectorization is an evening well spent, there are a few special gatherings you may be interested to attend this year. I’ve listed the meetings that I’m going to (or helping with) and some important dates:

EuroSciPy 2008

EuroSciPy 2008: Scientific Computing with Python, European Style

This is the first annual Europe-hosted SciPy Conference. It will match the stye and content of the popular US version (see below) which has been around for 7 years now. It will be in Leipzig, Germany. Travis Oliphant is delivering the keynote talk.

  • June 20: Early-bird registration deadline
  • July 26-27: Conference

SciPy 2008

SciPy 2008: Scientific Computing with Python

This will be the 7th annual gathering. Proceedings will be published for the first time this year. Alex Martelli is delivering the keynote talk.

  • June 27: Abstracts Due
  • July 11: Early-bird registration deadline
  • August 19-20: Tutorials
  • August 21-22: Conference
  • August 23-24: Sprints


Texas Python Unconference (link to last year’s site)

An informal, self-organizing gathering. It will be hosted in Austin this year.

  • October 4-5 (tentative)

Feel free to list any other meetings in the comments.