Monthly Archives: July 2008

ETS 3.0.0b1 Released!

I am very pleased to announce that ETS 3.0.0b1 has just been tagged and released!

All ETS sub-projects, including Traits, Chaco, Mayavi, and Envisage, have been tagged and released in one form or another (alpha, beta, final) as part of this effort. All ETS projects have now been registered with PyPi and source tarballs have been uploaded! As a result, for the first time ever, you can now easy_install all of ETS via a simple command like:

    easy_install ETS

Or, alternatively, you can get only specific projects with commands like:

    easy_install Mayavi

Or

    easy_install Chaco

We anticipate the final release of ETS 3.0, and all of its sub-projects at their current version number, in a matter of weeks. Please join us in helping make this final release the most robust release it can be by participating in this beta shakeout period. Discussions about the status of ETS and its sub-projects happens on the enthought-dev mailing list, whose home page is at:
https://mail.enthought.com/mailman/listinfo/enthought-dev

Thank you to all who have worked so hard on making this release possible!

— Dave

Enthought Tool Suite

The Enthought Tool Suite (ETS) is a collection of components developed by Enthought and open source participants, which we use every day to construct custom scientific applications. It includes a wide variety of components, including:

  • an extensible application framework
  • application building blocks
  • 2-D and 3-D graphics libraries
  • scientific and math libraries
  • developer tools

The cornerstone on which these tools rest is the Traits package, which provides explicit type declarations in Python; its features include initialization, validation, delegation, notification, and visualization of typed attributes. More information is available for all the packages within ETS from the Enthought Tool Suite development home page at http://code.enthought.com/projects/tool-suite.php.

Testimonials

“I set out to rebuild an application in one week that had been developed over the last seven years (in C by generations of post-docs). Pyface and Traits were my cornerstones and I knew nothing about Pyface or Wx. It has been a hectic week. But here … sits in front of me a nice application that does most of what it should. I think this has been a huge success. … Thanks to the tools Enthought built, and thanks to the friendly support from people on the [enthought-dev] list, I have been able to build what I think is the best application so far. I have built similar applications (controlling cameras for imaging Bose-Einstein condensate) in C+MFC, Matlab, and C+labWindows, each time it has taken me at least four times longer to get to a result I regard as inferior. So I just wanted to say a big “thank you”. Thank you to Enthought for providing this great software open-source. Thank you for everybody on the list for your replies.”
Gal Varoquaux, Laboratoire Charles Fabry, Institut dOptique, Palaiseau, France

“I’m currently writing a realtime data acquisition/display application I’m using Enthought Tool Suite and Traits, and Chaco for display. IMHO, I think that in five years ETS/Traits will be the most comonly used framework for scientific applications.”
Gary Pajer, Department of Chemistry, Biochemistry and Physics, Rider University, Lawrenceville NJ

EuroSciPy

On Monday night I returned from my first ever trip to Germany where I had the pleasure of attending the first EuroSciPy conference which was held in Leipzig. Mike Mueller did an excellent job of organizing and hosting the conference (the venue was great and the catering was superb). About 45 people attended the conference resulting in very interesting talks and several productive discussions. I had a chance to give a talk on the history of NumPy/SciPy. (I hope the fact that I did not sleep the night before did not detract too much from the talk.) The slides from the talk are here, although it may be hard to glean the entire talk from just the slides. Travis Vaught took video of the entire conference and left it in the hands of Ondrej Certik. This video should find it’s way to the Net someday.

The talks were very interesting. I learned more about SimPy which is a mature discrete event simulation engine written in Python (which makes heavy use of generators). The author of the package, Dr. Klaus Mller, has had much experience in the area and was very interesting to listen to. I learned that you can use the sound card plus Python to measure the coefficient of restitution (which measures how elastic a collision is), and also to tune an instrument. I also learned that Python is being used to help search for neutrinos in the Ice of Antarctica.

There are several symbolic packages (Computer Algebra Systems) emerging for Python. Pearu Peterson is working on a very fast one: sympycore, while Ondrej Certik is trying to implement a lot more functionality more quickly in SymPy (not to be confused with the discrete simulation package mentioned earlier). Hopefully, the two projects will merge sometime soon.

Unfortunately, I missed the lightning talk session because I was absolutely exhausted after not sleeping for more than 36 hours (twice) on the trip. At that point I’d been driving on the German autobahn for more hours than I’d slept (going 210km/h on the autobahn is very exciting by the way — especially when you are tired). I heard there were some very good talks, however.

The next day, I missed the first two talks because I went to church in the morning (there was a very charming congregation in Leipzig there and a kind American missionary translated most of it for me).

Later, Mike Mueller presented on his abstraction called PyModelData for handling data for scientific simulations (it uses YAML). The talk by Klaus Zimmerman later in the day suggested using yet another markup language called FMF as part of his presentation on making sure meta-data accompanies all data for rapid information processing (he also showed us Pyphant at that time). Pyphant is a data-analysis framework from the University of Freiburg (that should be using more of the Enthought Tool Suite IMHO and hopefully will be in future versions 🙂 ). The University of Freiburg also gave a talk earlier in the day on machine learning.

I really enjoyed the talks by David Albanese on his machine learning toolbox (called mlpy) and by Robert Cimrman on his finite-element toolbox called sfepy because I love to see these kinds of libraries becoming available. It shows that the Scientific and Engineering community surrounding Python is really growing. Travis Vaught ended the day with a description/demonstration of the Enthought Tool Suite and how it can be used to build rich scientific applications. These are the kind of applications we build everyday, but rather than keep our special sauce a secret we are giving it away to grow the community and to get help in building these foundational tools in the best way possible.

The Enthought demo went on until about 7:30pm after which about 15 of the attendees including me went to dinner in Leipzig at a Greek (I think) restaurant. The food was great, but the company was even better. I didn’t want to leave, but I had to drive to Frankfurt to get at least a couple of hours of sleep before flying out the next morning.

There are a few photos from the event that you should check out here. Overall, EuroSciPy was a great event. If you are in Europe and can’t come to the SciPy conference in CalTech, make sure and attend EuroSciPy next year. You can even help to plan and organize it in a city of your choice by making contact with Mike Mueller and/or Travis Vaught early enough.

Everybody

Back of the room

Traits 3.0 released!

I am very pleased to announce that Traits 3.0 has just been released!

All Traits projects have been registered with PyPi (aka The Cheeseshop)
and each project’s listing on PyPi currently includes a source
tarball. In the near future, we will also upload binary eggs for
Windows and Mac OS X platforms. Installation of Traits 3.0 is now as
simple as:

easy_install Traits

The Traits projects include:

The Traits project is at the center of all Enthought Tool Suite
development and has changed the mental model used at Enthought for
programming in the already extremely efficient Python programming
language. We encourage everyone to join us in enjoying the productivity
gains from using such a powerful approach.

The Traits project allows Python programmers to use a special kind of
type definition called a trait, which gives object attributes some
additional characteristics:

  • Initialization: A trait has a default value, which is
    automatically set as the initial value of an attribute before its first
    use in a program.
  • Validation: A trait attribute’s type is explicitly declared. The
    type is evident in the code, and only values that meet a
    programmer-specified set of criteria (i.e., the trait definition) can be
    assigned to that attribute.
  • Delegation: The value of a trait attribute can be contained either
    in the defining object or in another object delegated to by the trait.
  • Notification: Setting the value of a trait attribute can notify
    other parts of the program that the value has changed.
  • Visualization: User interfaces that allow a user to interactively
    modify the value of a trait attribute can be automatically constructed
    using the trait’s definition. (This feature requires that a supported
    GUI toolkit be installed. If this feature is not used, the Traits
    project does not otherwise require GUI support.)

A class can freely mix trait-based attributes with normal Python
attributes, or can opt to allow the use of only a fixed or open set of
trait attributes within the class. Trait attributes defined by a classs
are automatically inherited by any subclass derived from the class.

Scientific Computing with Python Course: October 6-10

Open Course

Introduction to Scientific Computing with Python

Date: October 6-10, 2008
Location: Austin, Texas, at Enthought’s offices
Cost: $2500.
Registration: Contact Leah Jones at 1-512-536-1057 to register via credit card, or email info@enthought.com with any questions.

Overview: This class is intended for the scientist or engineer interested in using Python for their day-to-day computational tasks. It begins with a day long introduction to the Python language focusing on the standard data structures, control constructs, and text/file processing. Object oriented development is briefly discussed. Day two introduces numeric data processing using NumPy arrays as well as displaying data as 2D and 3D plots. Day three surveys the scientific algorithms available in SciPy including interpolation, integration, linear algebra, signal/image processing, optimization, and others. The fourth day is split between a primer on integration with other languages (C/C++/Fortran) and an introduction to building simple graphical user interfaces with Traits, a package from the Enthought Tool Suite. The last morning covers interactive 2D visualization using the Chaco plotting library.

The class culminates with an opportunity for students to work with experts on a coding project of their choosing. Several members of the Enthought team will join the instructor on for the last several hours to offer design ideas, suggestions, and even pitch in on development.

Teaching Format: Enthought classes are hands-on affairs with 15-30 minute lectures interleaved with programming exercises typically done within IPython. The exercises provide practical experience with the tools on real-world problems. Upon completion, students will be comfortable building computational models and analyzing data at the python command prompt as well as writing scripts to automate their computational tasks.

Target Audience: The class is geared toward scientist and engineers that would like to learn how to use python in their day-to-day work. Practical data processing topics form the meat of the course, but some more involved development topics (UIs and interactive plots) are also covered. Programming experience in some language (C, VB, Fortran, Matlab) is expected. Experience with C, C++, and/or Fortran is useful for some topics. Object oriented programming skills are not necessary but will be helpful on the last two days. Knowledge of calculus, statistics, signal and image processing, optimization, are all valuable but not absolutely required.

Print Materials: Enthought will provide a hard copy of the class materials as well as soft copies of all exercises used in the class.

Software/Hardware Requirements: Students should bring a laptop pre-loaded with Python and the tools to the class. Any number of operating systems including Windows, Linux, and OS X will work fine. A large number of libraries are used including Python, IPython, NumPy, SciPy, Matplotlib, SWIG, Cython, f2py, and the Enthought Tool Suite. EPD (Enthought Python Distribution) provides a convenient way to grab all of these for a variety of different platforms. It is highly recommended to ensure compatibility with class exercises. If necessary, Enthought can provide a computer for classroom use, but it is helpful if the student can work in their “standard” work environment.

Food/Travel: Breakfast and classroom snacks are included with the course. Students are responsible for setting up their own travel and accommodations.

ETS 3.0 release plans – beta 1, beta 2, and full release

Hello all,

If you’ve been watching our enthought-dev mailing list, ‘m sure you’ve noticed that a large number of people have been working on making sure that all the unit tests and examples in the ETS projects “just work”. For tests, that means they either pass or get skipped by the test suite collector, and that you can run them by doing ‘python setup.py test’ for the project. For examples, that means invoking the example module using the python interpreter. This effort is part of our push to release ETS 3.0 ASAP — we’re trying very hard to have it out before SciPy (mid-August). I apologize for the late public notice about this — I’m not sure how I skipped announcing it as I’ve been talking about it internally here for more than a month!

* We are aiming for a FULL ETS 3.0b1 BETA RELEASE BY EARLY NEXT WEEK, Tuesday, July 29th.

* We are also aiming for a TRAITS 3.0 *RELEASE* BY THE END OF THIS WEEK, Friday, July 25th.

* Following the ETS 3.0b1 release, we will take a week to do more testing and fix outstanding bugs, and if need be release a 3.0b2 release on Tuesday, August 5th. If that happens, the official release of ETS 3.0 will then be Tuesday, August 12th. If not, it will be August 5th.

BTW: I’m setting up milestones in the Enthought Trac (not done yet at the time of this post) to document the efforts needed to hit these dates. If you want to help us out by testing and/or fixing issues (feel free to submit patches in tickets!), please make sure to enter any issues you discover in the next earliest milestone.

Please provide feedback ASAP if you are involved in something that will make it difficult to meet this schedule!

grin 1.1

I am pleased to announce the first release of grin. Colleagues and a few other interested parties have been pleasantly using the tool for a while now, but this is the first time it has been announced properly.

I wrote grin to help me search directories full of source code. The venerable
GNU grep and find are great tools, but they fall just a little short for my
normal use cases.

grin does exactly what I want 99% of the time with the least amount of thinking: recursive grep that skips crud I’m almost never interested in. For example, many of the projects I work on use Subversion as source control. Copies of the unmodified files are stored in the directory .svn/text-base/. When doing a typical recursive grep, say for a particular import Im trying to remove from my code, grep will search these files too and find all kinds of false positives.

Now, one could construct a find command that would do the equivalent and store it away as little oneliner shell script, but that’s boring. Im a coder, and this is an excuse to code up something fun and useful. If Im going to use a tool 50 times a day, I might as well make it do exactly what I want. This also gives me an opportunity to make a library module of a featureful grepalike that can be repurposed to make any number of small, specific tools. But before we get to that, lets take a look at some typical examples of using grin at the command line.

To recursively search the current directory for a regex:

$ grin some_regex

To search an explicit set of files:

$ grin some_regex file1.txt path/to/file2.txt

To search data piped to stdin:

$ cat somefile | grin some_regex -

To only search Python .py files:

$ grin -I "*.py" some_regex

To just show the names of the files that contain matches rather than the matches
themselves:

$ grin -l some_regex

Match highlighting was a feature added by my colleague Peter Wang. He took care to only use ANSI color escapes when they would be interpreted correctly. If you are piping the output to a file or a pager, you typically don’t want the ANSI escape sequences in there. But if you need to explicitly suppress the use of color highlighting:

$ grin --no-color some_regex

To force the use of color highlighting when piping the output to something that
is capable of understanding ANSI color escapes:

$ grin --force-color some_regex | less -R

To avoid recursing into directories named either CVS or RCS:

$ grin -d CVS,RCS some_regex

By default grin skips a large number of files. To suppress all of this behavior
and search everything:

$ grin -sbSDE some_regex

To use another program to determine the list of files to search, you have two options. The most common is to pass it on the command line using your shells backtick mechanism:

$ grin some_regex `find . -newer some_file.txt`

But this fails, sometimes. My shells typically break up arguments on whitespace, so if you have directories with spaces in them, grin will get busted paths. xargs helps, but I can never remember how to use it without reading the man page. Instead, grin can read the list of paths to search from a file, one path per line.

$ find . -newer some_file.txt | grin -f - some_regex

If you have embedded newlines in your directory names…you have my sympathy. Fortunately, you also have an option to use embedded NULs as the separator. But mostly sympathy.

$ find . -newer some_file.txt -print0 | grin -0 -f - some_regex

Now lets talk about libraries. I tried to write grin with a clean design such that people could reuse pieces of it and shim in custom behavior in useful places. Want to search text files inside ZIP files? Just replace the generator that recurses through the paths with one that recurses into ZIP files and provide a function that will make an a filelike object for the ZIPped files. We can grep through texty-but-not-actual-plaintext files with the appropriate converters, too. For example, pyPdf is a nice library for reading and munging PDF files. It can extract the plain text from many PDF files. If you write a function that will do this and return a StringIO with the extracted text, grin will search through the plain text of PDFs.

I am also considering more exotic uses, too. For example, lets say that I want to find all instances of a particular import in my Python code. There are a number of variations on the syntax for importing a package.

import foo.bar
from foo import bar
from foo import baz, bar
from foo import \
    bar
from foo import (baz,
    bar)
# Plus comments that tell you to do "from foo import bar"

As an example included in the source tarball, I wrote up a little script that will use the compiler module to parse Python files, extract just the import statements, and normalize them so they are easily searchable. Lets say I have the following Python file:

$ cat example.py
import foo
import foo.baz as blah
from foo import bar, baz as bat
def somefunction():
    "Do something to foo.baz"
    import from_inside.function

We can look at just the normalization by searching for “import” since that will show up on every line:

$ grinimports.py import example.py
example.py:
    1 : import foo
    2 : import foo.baz as blah
    3 : from foo import bar
    4 : from foo import baz as bat
    5 : import from_inside.function

To look for just the imports of foo.baz, a fairly simple regex can find all of them in the normalized output:

$ grinimports.py "import foo\.baz|from foo import baz" example.py
example.py:
    2 : import foo.baz as blah
    4 : from foo import baz as bat

The script that implements this is small. Outside of the code that implements the normalization and help text, it just recapitulates grin’s main() function with small modifications. There are more possibilities along these lines, of course. A grin that just searches docstrings and comments? Or reformats paragraphs to be on a single line so you can search for phrases that happen to be broken by a newline? Or search through the wiki pages in Tracs relational database? Id like to hear about what you think you can do with grin. Leave a comment here or send me email.

grin can be downloaded from its PyPI page or checked out from Subversion:

$ svn co https://svn.enthought.com/svn/sandbox/grin/trunk grin

If you have setuptools, you may also easy_install it:

$ easy_install grin

The Enthought Python Distribution currently has version 1.0 of grin, which is only missing one or two features compared to 1.1.

SciPy 2008 Conference: Looking Back and Looking Ahead

The upcoming SciPy 2008 Conference should turn out to be one of the most interesting gatherings we’ve ever had. The most exciting part is how the community has mobilized in organizing this year’s meeting. Many thanks to the organizersfor a stellar job so far.

In reflecting on how far we’ve come, I dug out the results of a survey [PDF] that we conducted the first year we held the SciPy meeting (called the “SciPy02 Workshop” at that time). (Note: the PDF is crudely formatted to imply that it’s quite old.) This can be compared to the results of the 2007 Survey [PDF] published here for the first time. The surveys don’t have the same questons, but it’s interesting to see how we’ve changed over the years).

If you haven’t registered already, the early-bird deadline is this Friday, July 11, 2008. So sign up and take advantage of the lower registration fee.

Also, there’s still time to register for the EuroSciPy meeting which will be held in a few weeks in Leipzig, Germany. See you there…feedback appreciated.

EPD for OS X: public beta available

I am pleased to announce that we have just made available our first public beta of the Enthought Python Distribution for Mac OS X! This is a universal binary (x86, PPC) for OS X 10.4 and 10.5. It contains all the same libraries and components as our other EPD platform releases, but does not yet contain the EPD documentation nor ETS examples. Those will be added soon.

You can download a trial, academic version, or make a purchase here.

Download and mount the .dmg, read the “EPD README.txt” in the root of the image, then run the “EPD.mpkg” to install. This will not overwrite your OS X system Python, nor a previous install of MacPython. It does install itself as the default Python on your system, but a few symlinks changes will allow you to control that. Uninstall instructions are documented in the “EPD README.txt”.

Please report any problems, comments, or suggestions on our EPD Trac site at https://svn.enthought.com/epd

About EPD
The Enthought Python Distribution (EPD) is a “kitchen-sink-included” distribution of the Python Programming Language, including over 60 additional tools and libraries. The EPD bundle includes the following major packages:

Python – Core Python
NumPy – Multidimensional arrays and fast numerics for Python
SciPy – Scientific Library for Python
Enthought Tool Suite (ETS) – A suite of tools including:
Traits – Manifest typing, validation, visualization, delegation, etc.
Mayavi – 3D interactive data exploration environment.
Chaco – Advanced 2D plotting toolkit for interactive 2D visualization.
Kiva – 2D drawing library in the spirit of DisplayPDF.
Enable – Object-based canvas for interacting with 2D components and widgets.
Matplotlib – 2D plotting library
wxPython – Cross-platform windowing and widget library.
Visualization Toolkit (VTK) – 3D visualization framework

There are many more included packages as well. Please see the complete list of the packages.

License
EPD is a bundle of software–every piece of which is available for free under various open-source licenses. The bundle itself is offered as a free download to academic and individual hobbyist use. Commercial and non-degree granting institutions and agencies may purchase individual subscriptions for the bundle or contact Enthought to discuss an Enterprise license. Please see the FAQ for further explanation about how the software came together.