Tag Archives: EPD

New Year, New Enthought Products!

We’ve had a number of major product development efforts underway over the last year, and we’re pleased to share a lot of new announcements for 2017:

A New Chapter for the Enthought Python Distribution (EPD):
Python 3 and Intel MKL 2017

In 2004, Enthought released the first “Python: Enthought Edition,” a Python package distribution tailored for a scientific and analytic audience. In 2008 this became the Enthought Python Distribution (EPD), a self-contained installer with the "enpkg" command-line tool to update and manage packages. Since then, over a million users have benefited from Enthought’s tested, pre-compiled set of Python packages, allowing them to focus on their science by eliminating the hassle of setting up tools.

Enthought Python Distribution logo

Fast forward to 2017, and we now offer over 450 Python packages and a new era for the Enthought Python Distributionaccess to all of the packages in the new EPD is completely free to all users and includes packages and runtimes for both Python 2 and Python 3 with some exciting new additions. Our ever-growing list of packages includes, for example, the 2017 release of the MKL (Math Kernel Library), the fruit of an ongoing collaboration with Intel.

The New Enthought Deployment Server:
Secure, Onsite Access to EPD and Private Packages

enthought-deployment-server-centralized-management-illustration-v2

For those who are interested in having a private copy of the Enthought Python Distribution behind their firewall, as well as the ability to upload and manage internal private packages alongside it, we now offer the Enthought Deployment Server, an onsite version of the server we have been using for years to serve millions of Python packages to our users.

enthought-deployment-server-logoWith a local Enthought Deployment Server, your private copy will periodically synchronize with our master repository, on a schedule of your choosing, to keep you up to date with the latest releases. You can also set up private package repositories and control access to them using your existing LDAP or Active Directory service in a way that suits your organization.  We can even give you access to the packages (and their historical versions) inside of air-gapped networks! See our webinar introducing the Enthought Deployment Server.

Command Line Access to the New EPD and Flat Environments
via the Enthought Deployment Manager (EDM)

In 2013, we expanded the original EPD to introduce Enthought Canopy, coupling an integrated analysis environment with additional features such as a graphical package manager, documentation browser, and other user-friendly tools together with the Enthought Python Distribution to provide even more features to help “make science and analysis easy.”

With its MATLAB-like experience, Canopy has enabled countless engineers, scientists and analysts to perform sophisticated analysis, build models, and create cutting-edge data science algorithms. The all-in-one analysis platform for Python has also been widely adopted in organizations who want to provide a single, unified platform that can be used by everyone from data analysts to software engineers.

But we heard from a number of you that you also still wanted the capability to have flat, standalone environments not coupled to any editor or graphical tool. And we listened!  

enthought-deployment-manager-cli-screenshot2So last year, we finished building out our next-generation command-line tool that makes producing flat, standalone Python environments super easy.  We call it the Enthought Deployment Manager (or EDM for short), because it’s a tool to quickly deploy one or multiple Python environments with the full control over package versions and runtime environments.

EDM is also a valuable tool for use cases such as command line deployment on local machines or servers, web application deployment on AWS using Ansible and Amazon CloudFormation, rapid environment setup on continuous integration systems such as Travis-CI, Appveyor, or Jenkins/TeamCity, and more.

Finally, a new state-of-the-art package dependency solver included in the tool guarantees the consistency of your environment, and if your workflow requires switching between different environments, its sandboxed architecture makes it a snap to switch contexts.  All of this has also been designed with a focus on providing robust backward compatibility to our customers over time.  Find out more about EDM here.

Enthought Canopy 2.0:
Python 3 packages and New EDM Back End Infrastructure

Enthought Canopy LogoThe new Enthought Python Distribution (EPD) and Enthought Deployment Manager (EDM) will also provide additional benefits for Canopy.  Canopy 2.0 is just around the corner, which will be the first version to include Python 3 packages from EPD.

In addition, we have re-worked Canopy’s graphical package manager to use EDM as its back end, to take advantage of both the consistency and stability of the environments EDM provides, as well as its new package dependency solver.  By itself, this will provide a big boost in stability for users (ever found yourself wrapped up in a tangle of inconsistent package versions?).  Alongside the conversion of Canopy’s back end infrastructure to EDM, we have also included a substantial number of stability improvements and bug fixes.

Canopy’s Graphical Debugger adds external IPython kernel debugging support

On the integrated analysis environment side of Canopy, the graphical debugger and variable browser, first introduced in 2015, has gotten some nifty new features, including the ability to connect to and debug an external IPython kernel, in addition to a number of stability improvements.  (Weren’t aware you could connect to an external process?  Look for the context menu in the IPython console, use it to connect to the IPython kernel running, say, a Jupyter notebook, and debug away!)

Canopy Data Import Tool adds CSV exports and input file templates

Enthought Canopy Data Import ToolAlso, we’ve continued to add new features to the Canopy Data Import Tool since its initial release in May of 2016. The Data Import Tool allows users to quickly and easily import CSVs and other structured text files into Pandas DataFrames through a graphical interface, manipulate the data, and create reusable Python scripts to speed future data wrangling.

The latest version of the tool (v. 1.0.9, shipping with Canopy 2.0) has some nice new features like CSV exporting, input file templates, and more. See Enthought’s blog for some great examples of how the Data Import Tool speeds data loading, wrangling and analysis.

What to Look Forward to in 2017

So where are we headed in 2017?  We have put a lot of effort into building a strong foundation with our core suite of products, and now we’re focused on continuing to deliver new value (our enterprise users in particular have a number of new features to look forward to).  First up, for example, you can look for expanded capabilities around Python environments, making it easy to manage multiple environments, or even standardize and distribute them in your organization.  With the tremendous advancements in our core products that took place in 2016, there are a lot of follow-on features we can deliver. Stay tuned for updates!

Have a specific feature you’d like to see in one of Enthought’s products? E-mail our product team at canopy.support@enthought.com and tell us about it!

For all you EPD Users: Canopy v1.1

EPD (Enthought Python Distribution) provided a simple install of Python for scientific computing on the major platforms: Windows, Linux and Mac-OS. Those looking for a clean, straightforward Python stack to unpack into a particular directory found EPD to be pretty ideal.

With the introduction of Enthought Canopy, we began addressing users who are more engineer or scientist than programmer and were much less familiar with command-line interfaces. The Canopy desktop (in the vein of MATLAB or Spyder) aims at these technical users who want to use Python, but more as an application or IDE. To implement the desktop in Python and to allow both it and a user-defined Python environment to co-exist and be separately updated, we used virtual environments. As a consequence Canopy can feel a bit foreign to EPD users. With 1.1 we have added a new command line interface (CLI) that will hopefully make EPD users feel more at home in Canopy while retaining many of the Canopy advantages such as in-place update and virtual environment support.

Now, EPD users who just want to use Canopy as a plain Python environment with their own tools or IDE can easily create one or more Python environments. For example, from the command line on Windows:

        Canopy_cli.exe setup C:\Python27

or on Linux:

        canopy_cli setup ~/canopy

The target directory can be any you choose. If you want to make this Python environment the default on your system, you can specify the –default switch, and Canopy will add the appropriate bin directory (Scripts directory on Windows) to your PATH environment variable. On Mac OS and Linux systems, Canopy does this by appending a line to your ~/.bash_profile file which activates the correct virtual environment. On Windows, this Python environment is also added to the system registry so third-party tools can correctly find it.

Since we use virtual environments, the installation layout for Canopy is different. With Canopy we install what is referred to as “Canopy Core”: the core Python environment and a minimum set of packages needed to bootstrap Canopy itself. With it we can lock down the Canopy environment, facilitate the automatic update mechanism, and provide reliable startup and fail-safe recovery. For the user, there is a different environment. This means when a Python update comes out, it is no longer necessary to install a whole new environment plus all of your packages and get everything working again. Instead, simply update Canopy and go back to working — all of your packages are still installed but Python has been upgraded.

To complete an install, Canopy creates two virtual environments named ‘System’ and ‘User’. System is where the Canopy GUI runs; no user code runs in this environment. Updates to this virtual environment are done via the Canopy update mechanisms. The User environment is where the kernel and all user code runs. This virtual environment is managed by Package Manager from the desktop or by enpkg from the command line; any packages can be updated and installed without fear of disrupting the GUI. Similarly, updates to the Canopy GUI will not affect packages installed in the User environment and break your code.

So why stick with virtual environments for an “EPD-like” install? One of the big challenges with the old, “flat” EPD installation method was updating an install, or trying out different package configurations. With virtual environments, you can create a new environment which inherits packages from another virtual environment, and try out a few package changes. When you are satisfied, it’s straightforward to throw away the experimentation area and make the changes to the original, stable virtual environment.

For more details, check out Creating an EPD-like Python environment in our online docs. And you can download Canopy v1.1 now.

Canopy v1.1 – Linux, Command Line Interface and More

Final-version-canopy-logo (1)

With version 1.1, Enthought Canopy now:

1) addresses, much more completely, the command line use cases that EPD users and IT managers expect from their Python distributions,

2) makes Linux support generally available,

3) streamlines installation for users without internet access with full, single-click installers,

4) supports multiple virtual environments for advanced users via “venv” backported to Python 2.7, and

5) provides updates like numpy 1.7.1, matplotlib 1.3.0 and more.

It’s been just over 4 months since Canopy v1.0 shipped with the new desktop analysis environment and our updated Python distribution for scientific computing. Canopy’s analysis environment seems to be well-received by users looking for a simpler GUI environment, but the Canopy graphical installation process left something to be desired by our EPD users.

Along with the Canopy desktop for users that don’t want to work directly from a command line, Canopy version 1.1 now provides command-line utilities that streamline the installation of a complete Python scientific stack for current EPD users who want to work from the shell or command line. In addition, IT groups or tools specialists that need to manage a central install of Python for a workgroup or department now have the tools they need to install and maintain Canopy. Version 1.1’s command-line installation and setup (and the 1-click, full installers detailed below) are much better for supporting Canopy installations on clusters as well.

Canopy for Linux is now fully released. We have full, tested support for RedHat5, CentOS5, and Ubuntu 12.04. Linux distros and versions beyond those work as well (anecdotally and based on some in-house use), but those are our tested versions.

With Canopy v1.0 we implemented a 2-step installation process. The installer includes the Canopy desktop, the Python packages needed by Canopy itself, and other core scientific Python stack packages for a minimal install (the libraries in Canopy Express). For those with a subscription, the second step requires downloading any additional packages using the Package Manager. This 2-step process is problematic for users that don’t have easy internet access or need to install centrally for a group. To help, we now provide full installers with all the Python packages we support included. This provides a streamlined 1-step install process for those who need it or want it.

To ensure users can install any package updates they wish without messing up package dependencies for Canopy itself, we use virtual environments under the hood. With v1.1 we now provide command-line access to our backport of “venv”. The new CLI provides utilities to create, upgrade, activate and deactivate your own virtual environments. Now its much easier to try out new Python environments or set up multiple configurations for a workgroup.

Canopy v1.1 ships many updates to packages and many new ones: OpenCV, LLVM, Bottleneck, gevent, msgpack, py, pytest, six, NLTK, Numba, Mock, patsy and more. You can see the full details on the Canopy Package Index page.

We hope you find version 1.1 useful!

“Why We Built Enthought Canopy, An Inside Look” Recorded Webinar

We posted a recording of a 30 minute webinar that we did on the 20th that covers what Canopy is and why we developed it. There’s a few minutes of Brett Murphy(Product Manager at Enthought) discussing the “why” with some slides, and then Jason McCampbell (Development Manager for Canopy) gets into the interesting part with a 15+ minute demo of some of the key capabilities and workflows in Canopy. If you would like to watch the recorded webinar, you can find it here (the different formats will play directly in different browsers so check them and you won’t have to download the whole recording first):

Summed up in one line: Canopy provides the minimal set of tools for non-programmers to access, analyze and visualize data in an open-source Python environment.

The challenge in the past for scientists, engineers and analysts who wanted to use Python had been pulling together a working, integrated Python environment for scientific computing. Finding compatible versions of the dozens of Python packages, compiling them and integrating it all was very time consuming. That’s why we released the Enthought Python Distribution (EPD) many years back. It provided a single install of all the major packages you needed to do scientific and analytic computing with Python.

But the primary interface for a user of EPD was the command line. For a scientist or analyst used to an environment like MATLAB or one of the R IDEs, the command line is a little unapproachable and makes Python challenging to adopt. This is why we developed Canopy.

Enthought Canopy is both a Python distribution (like EPD) and an analysis environment. The analysis environment includes an integrated editor and IPython prompt to faciliate script development & testing and data analysis & plotting. The graphical package manager becomes the main interface to the Python ecosystem with its package search, install and update capabilities. And the documentation browser makes online documentation for Canopy, Python and the popular Python packages available on the desktop.

Check out the Canopy demo in the recorded webinar (link above). We hope it’s helpful.

Introducing Enthought Canopy

-Eric Jones, Enthought CEO

Yesterday we launched Enthought Canopy, our next-generation, Python-based analysis environment and our follow-on to EPD.

Since 2003, the Enthought Python Distribution (EPD) has helped hundreds of thousands of scientists, engineers and analysts develop and deploy with Python. 2013 is its 10th anniversary! It’s hard to believe it’s been that long. Time flies when you are having fun.

Over the years, we’ve watched customers use Python for a variety of scientific and analytic computing applications. Their tasks fell into three areas: exploration, development, and visualization. We developed Canopy to specifically help with these 3 key analysis tasks.

Exploration:

Python makes it straightforward to acquire and manage data from a variety of sources like the web, databases and files on the computer. Using Canopy’s IPython interactive prompt, you can interactively access, combine and explore data from all these sources. With the IPython Notebook, you can capture your sessions, re-use them later or share them with others.

Development:

Analysts develop algorithms and scripts to work with their data and to deliver results. Canopy’s editor includes syntax highlighting to help identify elements in the Python code, auto completion to speed programming, and error checking to reduce time spent hunting bugs. You can execute entire scripts or selected lines from the editor at the IPython prompt to test and debug your code.

Python’s scientific and analytic computing ecosystem is quite large, with hundreds, if not thousands, of packages. Canopy itself ships with over 100 math, science, and analytic packages. And the ecosystem continues to grow and evolve. The Canopy Package Manager makes it simple to find, install and update these Python packages.

Visualization:

Beyond the numbers and the scripts, one of the best ways to understand and explain data is with visualization. Python provides many tools and packages to help visualize data and results. Within Canopy, Matplotlib works seamlessly with IPython and provides an extensive toolset for data plotting and presentation.

For more extensive graphical user interfaces, Canopy includes 2-D and 3-D visualization packages like Chaco and Mayavi.

We’re really excited about Canopy’s design. It brings together the full extent and power of Python in an easy-to-use analysis environment today and provides an analysis platform on which to build for the next 10 years. For more details, check the Canopy web-page.

 

Just released! EPD 7.3 plus beta version of EPD 8.0

Today we have for you not just one, but two exciting EPD releasesan update of EPD to 7.3 and a beta release previewing new features coming in EPD 8.0.

The EPD 7.3 update adds several new packages including Shapely, openpyxl, and a new package from Enthought named Enaml.  Enaml is a new package for rapidly building UIs using a very “Pythonic” declarative language heavily inspired by Qt’s QML langage.

EPD 7.3 also includes a host of package updates, including adding OpeNDAP support to the NetCDF4 package. Overall over 30 packages have been updated, including Cython, IPython, pandas, basemap, scikit-learn, Twisted and SQLAlchemy.

EPD 8.0 beta takes all that we know and love in EPD 7.x and adds an all-new graphical development and analysis environment.  The new GUI is focused on providing a fast, lightweight interface designed for scientists and engineers.  Some of the key features are:

  • A Python-centric text editor including tab-completion plus on-the-fly code analysis.
  • An interactive Python (IPython) prompt integrated with the code editor to enable rapid prototyping and exploration.
  • A Python package manager to make is easier to discover, install, and update packages in the Enthought Python Distribution.
  • Integrated documentation, both on the GUI itself and standard online documentation.

Which version should you install?  EPD 7.3 is the current production release and has the most testing behind it.  If you want to try out EPD 8.0 beta we would love to hear what you think.  EPD 8 is designed to work with earlier versions so once you install the GUI you can either setup a new EPD environment or tell it to use your existing one.  And don’t worry, it’s safe — it won’t modify your existing EPD.

As always, if you have an existing EPD 7.x install that you want to keep you can update all of the Python packages to EPD 7.3 by running the command “enpkg epd” from the command line.  Or, if you have installed the EPD 8 GUI, just open the package manager, select the ‘EPD’ package and click ‘Install’.  It will update you EPD 7.x environment for you.

EPD 8 can be downloaded from https://beta.enthought.com/EPD_8/download/.

Any comments, questions, complaints, or compliments on the beta release can be sent to beta-feedback@enthought.com.

EPD 7.2 released

EPD 7.2I am pleased to announce the release of Enthought Python Distribution, EPD version 7.2, along with its “EPD Free” counterpart. The highlights of this release are: the addition of GDAL and updates to over 30 packages, including SciPy, matplotlib and IPython. The new IPython 0.12 includes the HTML notebook, which is why the Tornado web-server was also added to EPD.

Import registry hooks in EPD

At the end of April (2011), Travis and I were playing with Python importhooks. The goal was to create an import registry mechanism which wouldallow multiple versions of a package to be installed and easily switchingbetween those versions. The idea is not new, setuptools provides such amechanism by installing packages into egg directories and using theeasy-install.pth file. However, as EPD packages are (as of EPD 5.0) notbeing installed using setuptools anymore (because of all the problems wehad with it), we were looking for a better alternative.

Starting with EPD 7.1 (including EPD Free), we have added an importhook registry mechanism to the core Python interpreter. This mechanismis useful for setting up different environments, i.e. Python runtime
environments in which a specific set of packages are importable.
For example, it is possible to set up one environment in which numpy 1.5.1
and pyzmq 2.1.7 are present, and another environment in which numpy 1.6.0
and no pyzmq is present. This can be useful is may different scenarios,
e.g. testing ipython with or without different versions of optional
dependencies installed. The alternative would be to set up multiple
Python interpreters and install the desired packages into each one.
virtualenv can help here, but adds it’s own set of problems.

This mechanism is based on the not very well known sys.meta_path listof finder objects, see PEP 302 for the original specification. By default sys.meta_path isan empty list, in any regular Python and EPD. However, when adding a file <sys.prefix>/registry.txt (or setting the environment variable EPDREGISTRY to a special registry file), the new registry mechanism is activated, and sys.meta_path now contains an instance of our new PackageRegistry object. As sys.meta_path is searched before any implicit default finders or sys.path, the mechanism allows importing different modules (and packages) than the ones installed into site-packages. The PackageRegistry class is quite simple. Basically, the find_module method checks whether the module name is contained in the registry, and the load_module method uses imp.load_module for the actual importing. The code can be found in <sys.prefix>/lib/python2.7/custom_tools/hooks.py, which also defines a main function for setting up the hook, which is called from Python’s site.py.

Let’s return to the above example, suppose you want to have two environments:

  • env1: numpy 1.5.1, pyzmq 2.1.7
  • env2: numpy 1.6.0, no pyzmq

First, you need to download (or create) the corresponding eggs. Now theseeggs need to be installed in a special location (not into site-packages,or any other directory on the PYTHONPATH):

$ egginst --hook numpy-1.5.1-2.egg
$ egginst --hook numpy-1.6.0-5.egg
$ egginst --hook pyzmq-2.1.7-1.egg

This will install the eggs into versioned directories in <sys.prefix>/pkgs. During the install process egginst also creates a registry file for each package, e.g. <sys.prefix>/pkgs/numpy-1.5.1-2/EGG-INFO/registry.txt. Now simply concatenate the desired registry files to create the registry file EPDREGISTRY is going to point to:

$ cd <sys.prefix>
$ cat pkgs/numpy-1.5.1-2/EGG-INFO/registry.txt > registry1.txt
$ cat pkgs/pyzmq-2.1.7-1/EGG-INFO/registry.txt >> registry1.txt
$ cp pkgs/numpy-1.6.0-5/EGG-INFO/registry.txt registry2.txt
$ EPDREGISTRY=registry1.txt python
>>> import numpy
>>> numpy.version
'1.5.1'
>>> import zmq
>>> exit()
$ EPDREGISTRY=registry2.txt python
>>> import numpy
>>> numpy.version
'1.6.0'
>>> import zmq # this is supposed to fail
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: No module names zmq

The files registry1.txt and registry2.txt do not have to reside in sys.prefix, they can be anywhere on your system. All that is important is that EPDREGISTRY points to their path.

We have tested the new import registry mechanism with all packages in EPD and initially considered installing all EPD packages in this manner. However Robert Kern was against it, and we came to the realization that he was right. The multiversion directory layout for packages is not good for development, as one is never really sure what versions are active. This is the same reason I created egginst in the first place, with the ability to install packages “splatted out” into site-packages and remove them or upgrade them cleanly.