SciPy for .NET & IronPython

This week Enthought and Microsoft are announcing the availability of NumPy and SciPy for IronPython and the .NET Framework, released in conjunction with the release of Microsoft’s Python Tool for Visual Studio release. These packages implement a fast and flexible multi-dimensional array package (NumPy) and a large collection of scientific and numerical algorithms built on top of the array (SciPy) andare two fundamental building blocking for technical computing in Python. CPython users are well familiar with both but up until now they have not been available on other Python environments. This is because both packages include many heavily optimized native-code (typically C or FORTRAN) implementation of the algorithms, thus requiringcustom wrappers to integrate them into each Python environment.

The result of the project isIronPython ports of NumPy and SciPy, which are full .NET ports and include custom C#/C interfaces to a common native C core. This means that the full functionality is available not only to IronPython but to all .NET languages such as C# or F# by directly accessing the C# interface objects.

Completing the port required considerably more work than just writing new interfaces IronPython. To start with large parts of the NumPy C code were written specifically for CPython and made free use of the CPython APIs. To support additional Python environments the codebase had to be refactored into a Python-independent core library plus a CPython-specific interface layer. A big advantage to this architectural separation is that the core NumPy library can now be used from applications written in C or C++ much more easily and ports to other Python environments will require less work.

The second supporting project was an extension of Cython to support IronPython. Cython, not to be confused with CPython, is a tool that allows C extensions to be written for Python using a language based on Python with extensions for interacting with native code libraries. Up until now Cython has only supported the CPython environment. We have extended Cython to be able to generate C++/CLI code interfaces for .NET, and thus IronPython. Using a single Cython input file it is now possible to target both CPython and IronPython environments.

The main driver for the Cython project was the nature of the SciPy library. SciPy consists of dozens of different packages, many of which have a hand-written C interface for CPython. One option was to write a matching C# interface for each of these to support IronPython. While this would have taken less time, the result would have been a significant duplication of code and more maintenance work in the future. Instead we chose to port Cython to .NET and then rewrite each C module in Cython so one a single source file is needed.

The first release of SciPy and NumPy for .NET are available now as binary distributions from SciPy.org or directly from Enthought. All of the code for these and the supporting projects are open source and available at the links below.

23 thoughts on “SciPy for .NET & IronPython

    1. avatarjmccampbell

      Thank you. I like IronPython a lot and I hope the packages turn out to be valuable and add to the technical computing environment.

      Reply
      1. avatarTom Kennedy

        Hi JM,

        I have tried to get these libs working but it doesnt seem to install all of the Scipy stuff.

        I have asked the question on the enthought list but nobody has replied.

        Could I private mail you?

        Regards,
        Tom

        Reply
  1. avatarCraig

    Are the changes that were required to Cython, SciPy, and NumPy merged into their repositories so that a user of SciPy for .NET will benefit from continued work on these other tools / libraries?

    Reply
    1. avatarjmccampbell

      For the most part, no not yet. It will require involvement from the respective communities and whether they see value in the changes. I suspect some of this will depend on how the community develops around the .NET ports. My hope is that by adding the .NET platform it grows the overall community to the benefit of users on both platforms.

      Reply
    1. avatarjmccampbell

      Part of it perhaps. The core NumPy work is easy to port and I have run parts of it under Mono during development. All of the code is C# and cross-platform C/C++.

      Anything dependent on Cython (a few of the smaller NumPy support packages and most of SciPy) is a much bigger issue. The problem is Cython generates C++/CLI instead of C# and Mono doesn’t support C++/CLI. We tried to avoid it but the nature of Cython requires the generation of managed code that can access native data structures. Getting the field offsets correct is handled automatically in C++/CLI because it can parse the header files. Reproducing this and instead generating C# is a fairly large project.

      Reply
  2. avatararman

    This is wonderful. Thanks for your efforts.

    I wonder if your work will also be useful in getting Cython to work with pypy. I really hope the Cython people will merge your efforts.

    Reply
  3. avatarManuel

    Which fwrap branch on github is the one used by scipy? Is it the ‘scipy’ branch? And has it waf already integrated? Thanks for the great work!

    Reply
  4. avatarnaveen

    Very exciting news.
    Thanks for putting the source code on github.
    I am trying to build from source for ipy27 for .net3.5.
    doing c:/> /ipy27/ipy iron_install.py doesn’t work, neither does building NumpyDotNet.sln in visual studio 2010.
    Could you please provide some build instructions ?
    are there any prerequisites besides the 4 repositories you have put on github ?
    do we need to change visual studio configuration (include libraries / headers between repos) ?

    Reply
  5. avatararman

    Are there any instructions for building Cython on Windows? A very quick attempt I building it from the bitbucket link above fails in converting Pyrex files to C. I can provide additional details if necessary.

    Reply
  6. avatarDirk

    Impressive! A commendable effort and a dream come true for our research department…
    I do have a practical question though. I installed everything and can work with numpy in IronPython etc. but we usually work in F# and I was hoping to be able to simply reference some DLL and get access to the well-known numpy functions.
    Is there a way to do this without running an embedded interpreter? Where do I find the C# interface that you talk about?
    For instance if I would want to use the poly1d function in C# or F#, how would I go about? Do I first need to compile .py scripts to a dll? Is there documentation about what is available in the precompiled dlls I can find in the C:\Program Files (x86)\IronPython 2.7\DLLs folder?
    Apologies for the many questions, but I am very enthusiastic about the possibilities and I have the impression I’m missing something…

    Reply
  7. avatarBrian

    Unfortunately, despite what the documentation says, integrate is missing from scipy in the 1.0 release. Hope it will be in a future release.

    Reply
  8. Pingback: What is the typical time overhead in using an open source package vs. an equivalent commercial package? - Quora

  9. avatarLuis

    Hi Guys!

    It seems that NumPy and SciPy were working along with PyTools very well until PyTools 1.1 RC

    http://pytools.codeplex.com/workitem/656

    are you aware of this issue?

    Is there a fix for this or a work arround?

    I really appreciate your help! I’m a Python n00b and really need this working for my MS thesis.

    Reply
  10. avatarTim Hochberg

    I’m very happy to see this project, but what is the correct forum for posting bug reports about NumPy for IronPython?

    There is a bad interaction between copy.copy and the numpy scalar types:

    IronPython 2.7.2.1 (2.7.0.40) on .NET 4.0.30319.239 (32-bit)
    Type “help”, “copyright”, “credits” or “license” for more information.
    >>> import numpy, copy
    >>> x = numpy.float32(1)
    >>> print x, copy.copy(x), copy.copy(float(x))
    1.0 6.72623e-44 1.0

    Note that copy.copy works fine on float, but fails on float32 (and other numpy scalar types).

    Reply
    1. avatarJason McCampbell

      Hi Matthew, no new work is going into SciPy for .NET from our end, though all of the work is open source and I know some people are using it. I’m not sure if anyone else is contributing right now. When Microsoft switched from IronPython to more of a implementation-agnostic strategy funding dried up. The cost of porting a critical mass of packages from CPython is substantial so in our projects involving .NET we have preferred to make CPython work with .NET as opposed to using IronPython.

      We never released the 64-bit binaries though it should generally be 64-bit clean. I’m not sure what work is needed to make it run.

      Reply
    2. avatarMark DeArman

      I just built a slim version of NumPy for IronPython 2.7.4 64-bit from the source and the only top level issue I ran into was some improper type definitions in ndarray. If you are still interested in getting this working you can email me. I haven’t fully tested all the functionality I wanted to include in my application, so there might be other issues lurking.

      Reply
      1. avatarPatrick

        n a 64bit machine I am getting a BadImageFormat for NpyAccessLib.dll. Does you version solve that?

        If so please post somewhere.

        Reply

Leave a Reply

Your email address will not be published. Required fields are marked *

Please leave these two fields as-is:

Protected by Invisible Defender. Showed 403 to 100,964 bad guys.