Category Archives: NumPy

Canopy v1.1 – Linux, Command Line Interface and More

Final-version-canopy-logo (1)

With version 1.1, Enthought Canopy now:

1) addresses, much more completely, the command line use cases that EPD users and IT managers expect from their Python distributions,

2) makes Linux support generally available,

3) streamlines installation for users without internet access with full, single-click installers,

4) supports multiple virtual environments for advanced users via “venv” backported to Python 2.7, and

5) provides updates like numpy 1.7.1, matplotlib 1.3.0 and more.

It’s been just over 4 months since Canopy v1.0 shipped with the new desktop analysis environment and our updated Python distribution for scientific computing. Canopy’s analysis environment seems to be well-received by users looking for a simpler GUI environment, but the Canopy graphical installation process left something to be desired by our EPD users.

Along with the Canopy desktop for users that don’t want to work directly from a command line, Canopy version 1.1 now provides command-line utilities that streamline the installation of a complete Python scientific stack for current EPD users who want to work from the shell or command line. In addition, IT groups or tools specialists that need to manage a central install of Python for a workgroup or department now have the tools they need to install and maintain Canopy. Version 1.1’s command-line installation and setup (and the 1-click, full installers detailed below) are much better for supporting Canopy installations on clusters as well.

Canopy for Linux is now fully released. We have full, tested support for RedHat5, CentOS5, and Ubuntu 12.04. Linux distros and versions beyond those work as well (anecdotally and based on some in-house use), but those are our tested versions.

With Canopy v1.0 we implemented a 2-step installation process. The installer includes the Canopy desktop, the Python packages needed by Canopy itself, and other core scientific Python stack packages for a minimal install (the libraries in Canopy Express). For those with a subscription, the second step requires downloading any additional packages using the Package Manager. This 2-step process is problematic for users that don’t have easy internet access or need to install centrally for a group. To help, we now provide full installers with all the Python packages we support included. This provides a streamlined 1-step install process for those who need it or want it.

To ensure users can install any package updates they wish without messing up package dependencies for Canopy itself, we use virtual environments under the hood. With v1.1 we now provide command-line access to our backport of “venv”. The new CLI provides utilities to create, upgrade, activate and deactivate your own virtual environments. Now its much easier to try out new Python environments or set up multiple configurations for a workgroup.

Canopy v1.1 ships many updates to packages and many new ones: OpenCV, LLVM, Bottleneck, gevent, msgpack, py, pytest, six, NLTK, Numba, Mock, patsy and more. You can see the full details on the Canopy Package Index page.

We hope you find version 1.1 useful!

SciPy 2013 Conference Recap

Author: Eric Jones

Another year, another great conference.  Man, this thing grew a ton this year.  At final count, we had something like 340 participants which is way up from last year’s 200 or so attendees.  In fact, we had to close registration a couple of weeks before the program because that is all our venue could hold.  We’ll solve that next year.  Invite your friends.  We’d love to see 600 or even more.

Many thanks to the organizing team.  Andy Terrell and Jonathan Rocher did an amazing job as conference chairs this year both managing that growth and keeping the trains on time.  We expanded to 3 parallel sessions this year, which often made me want to be in 3 places at once.  Didn’t work.  Thankfully, the videos for all the talks and sessions are available online.  The video team really did a great job — thanks a ton.

I’ve wondered whether the size would change the feel of the conference, but I’m happy to report it still feels like an gathering of friends, new and old.  Aric Hagberg mentioned he thinks this is because it’s such a varied (motley?) crowd from disparate fields gathered to teach, learn, and share software tools and ideas.  This fosters a different atmosphere than some academic conferences where sparring about details of a talk is a common sport.  Hmh.  Re-watching the videos, I see Fernando Perez mentions this as well.

Thanks again to all who organized and all who attended.  I’m already looking forward to seeing you again next year.  Below are my personal musings on various topics at the conference:

  • The tutorials were, as usual, extremely well attended.  I spent the majority of my time there in the scikits learn track by Gael VaroquauxOlivier Grisel, and Jake VanderPlas.  Jeez, has this project gone far.  It is stunning to see the breath and quality of the algorithms that they have.  It’s obviously a hot topic these days; it is great to have such an important tool set at our disposal.
  • Fernando Perez gave a keynote this year about IPython.  We can safely say that 2013 is the year of the IPython notebook.  It was *everywhere*.  I’d guess 80+% of the talks and tutorials for the conference used it in their presentations.  Fernando went one step further, and his slide deck was actually live IPython notebooks.  Quite cool.  I do believe it’ll change the way people teach Python…  But, the most impressive thing is that Fernando still has and can execute the original 250 line script that was IPython 0.00001.  Scratch that.  The most impressive thing is to hear how Fernando has managed to build a community and a project that is now supported by a $1.1M grant from the Sloan foundation.  Well done sir.  The IPython project really does set the standard on so many levels.
  • Olivier Grisel, of scikits learn fame, gave a keynote on trends in machine learning.  It was really nice because he talked about the history of neural networks and the advances that have been made in “deep learning” in recent years.  I began grad school in NN research, and was embarrassed to realize how recent (1986) the back propagation learning algorithm was when I first coded it for research (1993).  It seemed old to me then — but I guess 7 years to a 23 year is, well, pretty old.  Over the years, I became a bit disenchanted with neural nets because they didn’t reveal the underlying physical process within the data.  I still have this bias, but Olivier’s discussion of the “deep learning” advances convinced me that I should get re-educated.  And, perhaps I’m getting more pragmatic as the gray hairs fill in (and the bald spot grows).  It does look like it’s effective for multiple problems in the detection and classification world.
  • William Schroeder, CEO of Kitware, gave a keynote on the importance of reproducible research which was one of the conference themes.  It was a privilege to have him because of the many ways Kitware illuminated the path for high quality scientific software in the open source world with VTK.  I’ve used it both in C++ and, of course, from Python for many, many years.  In his talk, Will talked about the existing scientific publication model doesn’t work so well anymore, and that, in fact, with the web and tools that are now available, direct publishing of results is the future together with publishing our data sets and code that generated them.  This actually dovetailed really well with Fernando’s talk, and I can’t help but think that we are on this track.
  • David Li has been working with the SymPy team, and his talk showed off the SymPy Live site that they have built to interactively try out symbolic calculations on the web.  I believe David is the 2nd high school student to present in the history of SciPy, yes? (Evan Patterson was the other that I remember)  Heh.  Aaand, what were you doing your senior year?  Both were composed, confident, and dang good — bodes well for our future.
  • There are always a few talks of the “what I have learned” flavor at Python.  This year, Brian Granger of IPython fame gave one about the dangers of features and the benefits of bugs.  Brian’s talks are almost always one of my favorites (sorta like I always make sure to see what crazy stuff David Beazley presents at PyCon).  Part of it is that he often talks about parallel computing for the masses which is dear to my heart, but it is also because he organizes his topics so well.
  • Nicholas Kridler also unexpectedly hooked me with another one of these talks.  I was walking out of conference hall after the keynote to go see what silly things the ever smiling Jake Vanderplas might be up to in his astronomy talk.  But derned if Nicholas didn’t start walking through how he approaches new machine learning problems in interesting ways.  My steps slowed, and I finally sat down, happy to know that I could watch Jake’s talk later.  Nicholas used his wits and scikits learn to win(!) the Kaggle whale detection competition earlier this year, and he gave us a great overview of how he did it.  Well worth a listen.
  • Both Brian and Nicholas’ talks started me thinking how much I like to see how experts approach problems.  The pros writing all the cool libraries often give talks on the features of their tools or the results of their research, but we rarely get a glimpse into their day-to-day process.  Sorta like pair programming with Martin Chilvers is a life changing experience (heh.  for better or worse… :-)), could we have a series of talks where we get to ride shotgun with a number of different people and see how they work?  How does Ondrej Certik work through a debugging session on SymPy development?  Does his shiny new cowboy hat from Allen Boots help or not?  When approaching a new simulation or analysis, how does Aric Hagberg use graph theory (and Networkx) to set the problem up?  When Serge Rey gets a new set of geospatial data, what are the common things he does to clean and organize the data for analysis with PySAL?  How does Wes McKinney think through API design trade-offs as he builds Pandas?  And, most importantly, how does Stefan Van Der Walt get the front of his hair to stand up like that? (comb or brush? hair dryer on low or high?)  Ok, maybe not Stefan, but you get the idea.  We always see a polished 25 minute presentation that sums up months or years of work that we all know had many false starts and painful points.  If we could learn about where people stubbed their toe and how to avoid it in our work, it would be pretty cool.  Just an idea, but I will run it by the committee for next year and see if there is any interest.
  • The sprints were nothing short of awesome.  Something like 130+ people were there on the first day sprinting on 10-20 different libraries including SymPy, NumPy, IPython, Matplotlib as well as more specific tools like scikits image and PySal.  Amazing to see.  Perhaps the bigger surprise was that at least half also stayed for Saturday’s sprints.  scikits learn had a team of about 10 people that worked two full days together (Friday and Saturday activity visible on the commit graph), and I think multiple other groups did as well.  While we’ve held sprints for a while, we had 2 top 3 times as many people as 2012, and this year’s can only be described as wildly successful.

  • While I was there, I spent most of my time checking in on the PySide sprint where John Erhsman of Wingware got a new release ready for the 4.8 series of Qt (bless him), and Robin Dunn, Corran Webster, Stefan Landgovt, and John Wiggins investigated paths forward toward 5.x compatibility.  No one was too excited about Shiboken, but the alternatives are also not a walk in the park.  I think the feeling is, long term, we’ll need to bite the bullet and go a different direction than Shiboken.

PyGotham Sneak Peek: GPU Computing

This week, we are highlighting GPUs. Sean Ross Ross has put together a brief sneak peek screencast (above) of his GPU computing PyGotham talk that covers some basic terminology and a short example. The example ports a python/numpy algorithm to the GPU via CLyther. CLyther offers a higher level of abstraction that allows programmers to more easily leverage OpenCL and GPUs from Python. Again, don’t worry if some of the terminology is not defined, that’s what the talk is for! And in case you missed it, go ahead and check out last week’s Portable Parallel Python sneak peek.

Although new advances out of Intel, AMD, and Nvidia promise more seamless interaction between CPUs and GPUs, for the time being CUDA and OpenCL dominate the landscape. The learning curve associated with these technologies can make it a challenge to make effective use of your GPU assets. Sean will introduce you to heterogeneous computing, particularly with OpenCL, and show you how CLyther can help simplify your GPU life.

SciPy for .NET & IronPython

This week Enthought and Microsoft are announcing the availability of NumPy and SciPy for IronPython and the .NET Framework, released in conjunction with the release of Microsoft’s Python Tool for Visual Studio release. These packages implement a fast and flexible multi-dimensional array package (NumPy) and a large collection of scientific and numerical algorithms built on top of the array (SciPy) andare two fundamental building blocking for technical computing in Python. CPython users are well familiar with both but up until now they have not been available on other Python environments. This is because both packages include many heavily optimized native-code (typically C or FORTRAN) implementation of the algorithms, thus requiringcustom wrappers to integrate them into each Python environment.

The result of the project isIronPython ports of NumPy and SciPy, which are full .NET ports and include custom C#/C interfaces to a common native C core. This means that the full functionality is available not only to IronPython but to all .NET languages such as C# or F# by directly accessing the C# interface objects.

Completing the port required considerably more work than just writing new interfaces IronPython. To start with large parts of the NumPy C code were written specifically for CPython and made free use of the CPython APIs. To support additional Python environments the codebase had to be refactored into a Python-independent core library plus a CPython-specific interface layer. A big advantage to this architectural separation is that the core NumPy library can now be used from applications written in C or C++ much more easily and ports to other Python environments will require less work.

The second supporting project was an extension of Cython to support IronPython. Cython, not to be confused with CPython, is a tool that allows C extensions to be written for Python using a language based on Python with extensions for interacting with native code libraries. Up until now Cython has only supported the CPython environment. We have extended Cython to be able to generate C++/CLI code interfaces for .NET, and thus IronPython. Using a single Cython input file it is now possible to target both CPython and IronPython environments.

The main driver for the Cython project was the nature of the SciPy library. SciPy consists of dozens of different packages, many of which have a hand-written C interface for CPython. One option was to write a matching C# interface for each of these to support IronPython. While this would have taken less time, the result would have been a significant duplication of code and more maintenance work in the future. Instead we chose to port Cython to .NET and then rewrite each C module in Cython so one a single source file is needed.

The first release of SciPy and NumPy for .NET are available now as binary distributions from SciPy.org or directly from Enthought. All of the code for these and the supporting projects are open source and available at the links below.

Fast Numpy/Protobuf Deserialization Example

In my last post I wrote about how I was able to improve protobuf deserialization using a C++ extension. I’ve boiled down the code to its essence to show how I did it. Rather than zip everything up in a file, the code is short enough to show in its entirety.

Here’s the simplified protobuf message which is used to represent a time series as 2 arrays:

[sourcecode language=”python”]
package timeseries;

message TimeSeries {
repeated double times = 2;
repeated double values = 3;
}
[/sourcecode]

I then wrote a test app in Python and C++ to provide a benchmark. Here is the Python version:

[sourcecode language=”python”]
import numpy
import time_series_pb2

def write_test():
ts = time_series_pb2.TimeSeries()
for i in range(10000000):
ts.times.append(i)
ts.values.append(i*10.0)

import time
start = time.time();

f = open("ts.bin", "wb")
f.write(ts.SerializeToString())
f.close()

print time.time() – start

def read_test():
ts = time_series_pb2.TimeSeries()

import time
start = time.time();

f = open("ts.bin", "rb")
ts.ParseFromString(f.read())
f.close()

t = numpy.array(ts.times._values)
v = numpy.array(ts.values._values)

print ‘Read time:’, time.time() – start
print "Read time series of length %d" % len(ts.times)

if __name__ == "__main__":
import sys
if len(sys.argv) < 2:
print "usage: %s <–read> | <–write>" % sys.argv[0]

if sys.argv[1] == "–read":
read_test()
else:
write_test()
[/sourcecode]

I will spare you the C++ standalone code, since it was only a stepping stone. Instead here is the C++ extension, with 2 exposed methods, one which deserializes a string and the other which operates on a file.

[sourcecode language=”cpp”]
#include <fcntl.h>

#include <Python.h>
#include <numpy/arrayobject.h>
#include <google/protobuf/io/coded_stream.h>
#include <google/protobuf/io/zero_copy_stream_impl_lite.h>
#include <google/protobuf/io/zero_copy_stream_impl.h>

#include "time_series.pb.h"

static PyObject* construct_numpy_arrays(timeseries::TimeSeries* ts)
{
// returns a tuple (t,v) where t and v are double arrays of the same length

PyObject* data_tuple = PyTuple_New(2);

long array_size = ts->times_size();
double* times = new double[array_size];
double* values = new double[array_size];

// the data must be copied because the tsid will go away and its mutable data
// will too
memcpy(times, ts->times().data(), ts->times_size()*sizeof(double));
memcpy(values, ts->values().data(), ts->values_size()*sizeof(double));

// put the arrays into numpy array objects
npy_intp dims[1] = {array_size};
PyObject* time_array = PyArray_SimpleNewFromData(1, dims, PyArray_DOUBLE, times);
PyObject* value_array = PyArray_SimpleNewFromData(1, dims, PyArray_DOUBLE, values);

PyTuple_SetItem(data_tuple, 0, time_array);
PyTuple_SetItem(data_tuple, 1, value_array);

return data_tuple;
}

static PyObject* TimeSeries_load(PyObject* self, PyObject* args)
{
char* filename = NULL;

if (! PyArg_ParseTuple(args, "s", &filename))
{
return NULL;
}

timeseries::TimeSeries ts;

int fd = open(filename, O_RDONLY);
google::protobuf::io::FileInputStream fs(fd);
google::protobuf::io::CodedInputStream coded_fs(&fs);
coded_fs.SetTotalBytesLimit(500*1024*1024, -1);
ts.ParseFromCodedStream(&coded_fs);
fs.Close();
close(fd);

return construct_numpy_arrays(&ts);
}

static PyObject* TimeSeries_deserialize(PyObject* self, PyObject* args)
{
int buffer_length;
char* serialization = NULL;

if (! PyArg_ParseTuple(args, "t#", &serialization, &buffer_length))
{
return NULL;
}
google::protobuf::io::ArrayInputStream input(serialization, buffer_length);

google::protobuf::io::CodedInputStream coded_fs(&input);
coded_fs.SetTotalBytesLimit(500*1024*1024, -1);

timeseries::TimeSeries ts;
ts.ParseFromCodedStream(&coded_fs);
return construct_numpy_arrays(&ts);
}

static PyMethodDef TSMethods[] = {
{"load", TimeSeries_load, METH_VARARGS, "loads a TimeSeries from a file"},
{"deserialize", TimeSeries_deserialize, METH_VARARGS, "loads a TimeSeries from a string"}
};

#ifndef PyMODINIT_FUNC /* declarations for DLL import/export */
#define PyMODINIT_FUNC void
#endif
PyMODINIT_FUNC inittimeseries(void)
{
import_array();
(void) Py_InitModule("timeseries", TSMethods);
}
[/sourcecode]

Calling the exension from python is trivial:
[sourcecode language=”python”]
import time
import timeseries
start = time.time()
t, v = timeseries.load(‘ts.bin’)
print "read and converted to numpy array in %f" % (time.time()-start)
print "timeseries contained %d values" % len(v)
[/sourcecode]

Fast Protocol Buffers in Python

A couple of years ago I worked on a project which needed to transport a large dataset over the wire. I looked at a number of technologies, and Google Protocol Buffers looked very interesting. Over the past week, I’ve been asked about my experience a couple of times, so I hope this provides a little bit of insight into how to use Protocol Buffers in Python when performance matters.

I wrote a little test case to model the serialization of the data I wanted to send, a list of 100 pairs of arrays, where each array contained 250,000 elements. The raw data size was 381 MB.

First, I ran the pure python test: the write took 83 seconds, the read took 202 seconds. Not good.

Next I tested the same data in C++: the write took 4.4 seconds and the read took 2.8 seconds. Impressive.

The obvious path then was to write the serialization code in C++ and expose it through an extension point. The read function, including putting all of the data into numpy arrays now takes 7.5 seconds. I only needed the read function from Python, but the write function should take about the same time.

Travis Oliphant announces…

Travis announces project to extend NumPy/SciPy to .Net

Travis announces project to extend NumPy/SciPy to .Net

Travis Oliphant kicked off today’s SciPy 2010 Day 2 with a great keynote talk. He told the story of his own path to Python, filling his slides with the faces and work of other developers, scientists, and mathematicians inspiration, teachers, and collaborators. He explained how his academic trajectory, from electrical engineering, through a brief affair with neuroscience, to a biomedical engineering PhD, both drove and competed with his work creating NumPy.
Last, but not least, Travis closed his talk with rather large announcement: Enthought has undertaken the extension of NumPy and SciPy to the .NET framework. For details on the project refer to the official release.

Simplified creation of NumPy arrays from pre-allocated memory.

About 18 months ago, I wrote a blog-post that illustrates how to create a NumPy array that refers to another block of pre-allocated memory by creating a small low-level Python object that wraps around the memory and correctly deallocates the memory once NumPy is done with it.

The basic idea was to create a new Python object in C and point the base member of the NumPy array structure to this newly created object after using PyArray_SimpleNewFromData to create the array from a block of pre-allocated memory. NumPy will decref the object pointed to by its base member when it is destroyed. The example I provided created a new Python object in C and pointed the base member to it. That way, when the NumPy array is destroyed, the tp_dealloc function will be called on the newly created object.

The solution I provided is very flexible and also illustrates how to create a new Python object in C. However, as Lisandro Dalcin pointed out in the comments to that post, there is a simpler way to do it if you just need to call a particular deallocation function to free the underlying memory (or decrease a reference) after the NumPy array is destroyed.

The simple solution is to use a low-level Python object that holds a pointer to anything as well as a destructor. This low-level object is called a CObject in Python 2.x. The equivalent in Python 3.x (backported to Python 2.7) is the Capsule object.

Creating a CObject in C code is very simple. You just call PyCObject_FromVoidPtr with an arbitrary pointer as the first argument and a destructor function as the second argument. The signature of this destructor function should take the arbitrary pointer as its only argument and return nothing. The Python Object returned can be assigned to the base member of the ndarray directly.

Using the CObject removes the need to create your own “dummy” Python Object just to handle calling some needed code after NumPy is done with the memory. Thus, the code example can be updated to just:

[sourcecode language=”c++”]
int nd=2;
npy_intp dims[2]={10,20};
size_t size;
PyObject arr=NULL;
void *mymem;

size = PyArray_MultiplyList(dims, nd);
mymem = _aligned_malloc(size, 16);
arr = PyArray_SimpleNewFromData(nd, dims, NPY_DOUBLE, mymem);
if (arr == NULL) {
_aligned_free(mymem);
Py_XDECREF(arr);
}
PyArray_BASE(arr) = PyCObject_FromVoidPtr(mymem, _aligned_free);
[/sourcecode]

For completeness, the original code containing the _aligned_malloc and
_aligned_free function calls is reproduced here:

[sourcecode language=”c++”]
#include
#define uintptr_t size_t

#define _NOT_POWER_OF_TWO(n) (((n) & ((n) – 1)))
#define _UI(p) ((uintptr_t) (p))
#define _CP(p) ((char *) p)

#define _PTR_ALIGN(p0, alignment) \
((void *) (((_UI(p0) + (alignment + sizeof(void*))) \
& (~_UI(alignment – 1)))))

/* pointer must sometimes be aligned; assume sizeof(void*) is a power of two */
#define _ORIG_PTR(p) (*(((void **) (_UI(p) & (~_UI(sizeof(void*) – 1)))) – 1))

static void *_aligned_malloc(size_t size, size_t alignment) {
void *p0, *p;

if (_NOT_POWER_OF_TWO(alignment)) {
errno = EINVAL;
return ((void *) 0);
}

if (size == 0) {
return ((void *) 0);
}
if (alignment < sizeof(void *)) { alignment = sizeof(void *); } /* including the extra sizeof(void*) is overkill on a 32-bit machine, since malloc is already 8-byte aligned, as long as we enforce alignment >= 8 …but oh well */
p0 = malloc(size + (alignment + sizeof(void *)));
if (!p0) {
return ((void *) 0);
}
p = _PTR_ALIGN(p0, alignment);
_ORIG_PTR(p) = p0;
return p;
}

static void _aligned_free(void *memblock) {
if (memblock) {
free(_ORIG_PTR(memblock));
}
}
[/sourcecode]

Thanks to Lisandro for pointing out this simplified approach.