Blocks and Contexts

My background has been in engineering and scientific circles mainly in academia, so programming has been something I have primarily used to solve problems. At Enthought we solve problems, but most of the time we are creating solutions to help others solve problems. We are trying to create the tools that someone like me would want to use.

There are several compelling ideas and tools that are buried in the Enthought Tool Suite that we have used to build commercial scientific applications over the past several years. Two of these ideas are the Block and the Context. These concepts have wiggled into the foundation of almost all of our most recent applications because they seem to be solving fundamental problems with how to integrate scientists and engineers typically procedural workflows into a larger GUI application in flexible and re-usable ways.

Eric developed the ideas behind the Block and the Context over 18 months ago and presented them at the SciPy 2007 conference. His slides for that talk are here. Our implementation of these ideas has been slowly but steadily improving as we have learned more about how to make use of them. They are becoming a common abstraction that we use all the time in thinking about new problems. In this way they are a bit like the reactive programming and automatic GUI building of Traits, the pluggable, interactive plotting of Chaco, and the plugin-based application building of Envisage in their importance.

In order to bring to the surface the Block and Context abstractions that have been buried in the BlockCanvas project, two days ago I created a new project in the Enthought Tools Suite called CodeTools. This project contains the two packages enthought.blocks and enthought.contexts. While I’m thinking about these ideas, I wanted to demonstrate a little bit how you can use these tools as well as showcase a few things that I’ve added to enthought.blocks which makes working with them even easier. I think there are some good ideas here, but I’m not sure our implementation is the best one yet. I’m anxious to engage more people in the discussion about how we should be doing this sort of thing.

To get CodeTools you need to get the latest version of ETS. Instructions for how to do this are here. You don’t actually need to make sure all of ETS is working, Traits is enough except for the units package that is still in BlockCanvas, but which I will moving to SciMath soon.

As explained by Eric in his slides, a Block is a set of executable instructions — an intelligent version of the string you might pass to the exec statement. The intelligence of the blocks come from the code analysis they perform when they are created. With these smarts, you get three very nice things:

  1. Information about the “inputs” required and “outputs” delivered by the code block.
  2. The ability to execute the same code over and over again using different namespaces (thus taking inputs from different sources and delivering outputs to a dictionary-like object of your choice).
  3. The ability to restrict execution to the portion of the code block that depends on a specified set of inputs and/or delivers a specified set of outputs.

Over the past few days, besides moving the blocks into their own project I’ve added a few enhancements to the blocks which makes them easier (in my mind) to use. These enhancements are:

  1. Decorators to turn the body of a function into a Block (or a string that could be later turned into a block).
  2. Separation from the outputs of the variables imported from somewhere. These are now stored as “fromimports” attributes
  3. Cataloging of any variables which are assigned constant expressions in the Block.
  4. A method to return a function from the block given a list of inputs to vary and the list of outputs to return. This function will only execute that portion of the block needed to produce the requested outputs.

With the decorators, you can now create a block very easily:

from enthought.blocks.api import func2block
@func2block
def block():
    c = a + b
    d = a - b
    g = f + e
    h = f - e

This little decorator means no more little code snippets running
around in strings messing up the syntax highlighting and the tab
indentation capabilities of your favorite editor. There is also a
func2str decorator which will take the body of a function and produce
a string (with the first level of indentation removed).

The other little improvement I want to advertise is the ability of the
block to now return a function that evaluates the (restricted) block
based on the list of inputs and outputs provided. Thus,

func1 = block.get_function(inputs=['a','b'], outputs='c')
func2 = block.get_function(inputs=['f','e'], outputs=['g','h'])

returns two functions equivalent to

def func1(a,b):
    c = a + b
    return c
def func2(f,e):
    g = f + e
    h = f - e
    return g, h

The big deal about blocks is really the code restriction. You can
take a scientists script and re-evaluate (restricted) versions of it
for “what-if” analysis, optimization, or even stochastic processing.

It’s a cool idea, but it will have to be left to another blog to make
this more clear.

3 thoughts on “Blocks and Contexts

  1. avatarPhillip J. Eby

    So, is this based on source code analysis or what? My “Trellis” library does dependency analysis via the reads themselves in a somewhat different way, but it seems to me that you could pull out the dependencies in the blocks you’re showing by exec’ing a block with a dictionary-like object that tracked loads and stores. (After changing all the LOAD_FAST and STORE_FAST codes to LOAD_NAME and STORE_NAME, of course.) But that wouldn’t handle conditionals or loops, of course. You’d have to dynamically update your dependencies (like the Trellis does) for that to work.

    Reply
    1. avatarBryce Hendrix

      @Stan- I haven’t tried, though it is all pure python. The part that is most likely to cause problems is the static analysis, but its worth a shot.

      Reply

Leave a Reply

Your email address will not be published. Required fields are marked *

Please leave these two fields as-is:

Protected by Invisible Defender. Showed 403 to 102,429 bad guys.