Monthly Archives: December 2007

Musings on SciPy and ETS

It’s been about 4 months since I picked up my personal family (Amy + 5 kids with one due any day now) and moved to Austin to join with Travis Vaught, Eric Jones and the rest of the Enthought family. I’ve been a long time admirer of Enthought and have known Travis and Eric since the fall of 2000 when Eric called me to propose creating SciPy which was one of Enthought’s first market splashes. It has been a thrill to finally take the leap from academia into industry and join them full time in their work of improving the software engineering of scientific and engineering applications.

Enthought spent a good deal of time getting SciPy up and running (as well as paying for several developers to help). They also co-sponsor the annual SciPy conference. However, their specific contributions to SciPy have dwindled substantially since the early days of its founding (with the marked exception of Robert Kern’s contributions). This reality was disappointing to me a bit, because I was originally so thrilled to be collaborating with Enthought in getting SciPy up and going. I’ve also had big dreams for SciPy and its potential ever since then. In fact, the primary reason I took what ultimately became a huge step in leaving my tenure-track position at BYU by writing NumPy was because I perceived the Numarray / Numeric split was hindering SciPy’s progress.

I think I generally understood that the reason Enthought did not contribute more to SciPy’s direct development was that consulting contracts were not well aligned with SciPy’s specific needs. However, what I did not really grasp until recently is that Enthought’s vision and promise goes far beyond what I had been thinking about for SciPy.

While scientists and engineers enjoy having access to the basic algorithms that SciPy has been bringing to Python, most of them can pull together enough know-how to link to a favorite routine when it is needed. This is even more true now with tools like ctypes, f2py, weave, and pyrex (or Cython). What scientists and engineers have typically struggled with, however, is putting together applications that provide others with an easy-to-use interface to their ideas.

I remember as a graduate student at the Mayo Clinic, pulling old PowerMac boxes together to create a computing cluster I called the orchard (I never knew how many different varieties of apples existed which enabled me to give a name to each box). I wrote an MRI simulation engine in C++ to run on that cluster (using raw socket communication — this is before I learned about MPI). The parameters for the simulation were entered using a configuration file.

After everything was working, I decided to try and put a friendly face on my simulation so that I could show it off to my other colleagues who cared about more about getting science done than playing with technology (and looked at me with some amusement as I recycled their old computers). I quickly discovered how difficult “putting a friendly face” on something could be. It probably took me almost as long to write a simple UI (to allow semi-intelligent entry of all the parameters) for the simulation as it did to write the parallel simulation engine itself.

I’ve talked with several people over the years who have similar stories to tell. In fact, As I’ve been learning more about scientific software in corporate settings, I’ve seen that it can sometimes take a team of computer-scientists a few months to take advantage of the tools that their internal scientists and engineers prototype in a few weeks. Most of this is the UI. Now, by UI, I don’t just mean the Graphical User Interface (GUI). UI includes everything that the user of an algorithm (or workflow) sees or expects: persistence, undo/redo, facilities for safe-exploration, pluggable modularity, scripting, and data-flow. All of this is part of the UI. When you hear UI, you can just as easily think IPython as wxPython because both kinds of elements are involved. Anything that the user interacts with is part of the UI.

Making good UI’s is hard. Enthought strives to make excellent UI’s. This is at the heart of everything we do. Mostly, making good UI’s is hard, because the tools for making them are not very good. While computer technology has come reasonably far in providing a mechanism for translating human-readable algorithms into machine code, it is fair to say that the ability to translate human-usable interactions into machine code needs more work. This is especially true for scientific applications.

Helping the scientific computing world (and beyond) write better UI’s for applications is the whole point of the Enthought Tool Suite (ETS). There have been some pretty impressive applications built with ETS so far, and that is just the tip of the iceberg of what is possible (chaco2 and tvtk are pretty-darn impressive). There are also several big-picture concepts that still need to be worked out. However, while I’m sure Eric would be uncomfortable with my bubbling, I’m convinced that ETS has the potential to rival any other “application-development” paradigm in existence (and there are some big names out there).

The reason for my optimism is both the power of writing in the Python language, and the community that has already made NumPy and SciPy a growing success. There are some members of that community that would benefit greatly from more intimate involvement with ETS, and ETS would benefit greatly as well. Currently, the biggest weakness of ETS is that it is still too hard for a newbie to get up and running with it. This is a symptom of it’s rapid development and heavy internal usage. Taking a break from developing the code and delivering customer applications to make easy-to-install releases is expensive.

Fortunately, Enthought has been paying $$ specifically to get ETS out to a wider audience over the last few years. In particular, much work has gone in to creating a system that will allow us to release binary packages (of ETS and many other tools) — look for Enthought Python Edition Release 1 at the first of the year. In addition, Dave Peterson has made heroic efforts to make releases of ETS, and to re-organize the growing cornucopia of tools that Enthought team members have created into more manageable projects that can be released on their own time-table.

All of ETS is open source, and there are some amazing tools there. I’m still learning about all of what is available. But, if you haven’t learned about Traits, TVTK, Enable, Chaco2, Kiva, and Envisage you owe it to yourself to learn about how these tools can help you write better UI’s. In the process, Enthought would love feedback and help on how these tools can be improved and made more accessible. ETS is a really big idea. It is one that should be getting a lot more attention and support from the wider Python community. I would love to understand what has been standing in the way of that.

In summary, while for some time I felt sad that Enthought could not help more directly with SciPy, I’ve since learned that Enthought has been doing fantastic things for SciPy by contributing tools that help connect the algorithms of SciPy with real users who need so much more in the way of UI to make valuable use of those algorithms.

New Install/Build instructions on the Enthought wiki

At the urging of various parties through direct e-mail, I've updated the
Enthought wiki pages (https://svn.enthought.com/enthought) so that we have a new set of installation and build
pages.   The new framework is not fleshed out for every platform, but
the structure is in place to guide people to the appropriate
instructions for their chosen product, version, platform, and
architecture.   We still need help writing the actual instructions
though, so please volunteer by stepping forward and claiming a platform.  Some of you already have but haven't yet
written anything. Hint. Hint.

Big thanks to Kunal Puri for writing Ubuntu
Gutsy instructions right out of the gate!