Category Archives: PyXLL

5 Simple Steps to Create a Real-Time Twitter Feed in Excel using Python and PyXLL

PyXLL 3.0 introduced a new, simpler, way of streaming real time data to Excel from Python.

Excel has had support for real time data (RTD) for a long time, but it requires a certain knowledge of COM to get it to work. With the new RTD features in PyXLL 3.0 it is now a lot simpler to get streaming data into Excel without having to write any COM code.

This blog will show how to build a simple real time data feed from Twitter in Python using the tweepy package, and then show how to stream that data into Excel using PyXLL.

(Note: The code from this blog is available on github https://github.com/pyxll/pyxll-examples/tree/master/twitter). 

Create a real-time Twitter data feed using Python and PyXLL.

Create a real-time Twitter feed in Excel using Python and PyXLL.

Step 1: Install tweepy and PyXLL
————————

As we are interested in real time data we will use tweepy’s streaming API to use Python to connect to Twitter. Details on this are available in the tweepy documentation. You can install tweepy and PyXLL from the Canopy package manager. You may also download PyXLL here.

Easily install Tweepy from Canopy's Package Manager.

Easily install Tweepy and PyXLL from Canopy’s Package Manager.

Step 2: Get Twitter API keys
————————

In order to access Twitter Streaming API you will need a Twitter API key, API secret, Access token and Access token secret. Follow the steps below to get your own access tokens.

1. Create a twitter account if you do not already have one.
2. Go to https://apps.twitter.com/ and log in with your twitter credentials.
3. Click “Create New App”.
4. Fill out the form, agree to the terms, and click “Create your Twitter application”
5. In the next page, click on “API keys” tab, and copy your “API key” and “API secret”.
6. Scroll down and click “Create my access token”, and copy your “Access token” and “Access token secret”.

Step 3: Create a Stream Listener Class to Print Tweets in Python
————————–

To start with we can create a simple listener class that simply prints tweets as they arrive

[sourcecode language=”python”]
from tweepy.streaming import StreamListener
from tweepy import OAuthHandler
from tweepy import Stream
import logging

_log = logging.getLogger(__name__)

# User credentials to access Twitter API
access_token = “YOUR ACCESS TOKEN”
access_token_secret = “YOUR ACCESS TOKEN SECRET”
consumer_key = “YOUR CONSUMER KEY”
consumer_secret = “YOUR CONSUMER KEY SECRET”

class TwitterListener(StreamListener):

def __init__(self, phrases):
auth = OAuthHandler(consumer_key, consumer_secret)
auth.set_access_token(access_token, access_token_secret)
self.__stream = Stream(auth, listener=self)
self.__stream.filter(track=phrases, async=True)

def disconnect(self):
self.__stream.disconnect()

def on_data(self, data):
print(data)

def on_error(self, status):
print(status)

if __name__ == ‘__main__’:
import time
logging.basicConfig(level=logging.INFO)

phrases = [“python”, “excel”, “pyxll”]
listener = TwitterListener(phrases)

# listen for 60 seconds then stop
time.sleep(60)
listener.disconnect()
[/sourcecode]

If we run this code any tweets mentioning Python, Excel or PyXLL get printed:

~~~
python twitterxl.py

INFO:requests.packages.urllib3.connectionpool:Starting new HTTPS connection (1): stream.twitter.com
{“text”: “Excel keyboard shortcut – CTRL+1 to bring up Cell Formatting https://t.co/wvx634EpUy”, “is…
{“text”: “Excel Tips – What If Analysis #DMZWorld #Feature #Bond #UMI https://t.co/lxzgZnIItu #UMI”,…
{“text”: “How good are you at using #Excel? We’re looking for South Africa’s #ExcelChamp Ts & Cs…
{“text”: “The Best Data Scientists Run R and Python – insideBIGDATA https://t.co/rwty058dL2 #python …
{“text”: “How to Create a Pivot Table in Excel: A Step-by-Step Tutorial (With Video) \u2013 https://…
{“text”: “Python eats Alligator 02, Time Lapse Speed x6 https://t.co/3km8I92zJo”, “is_quote_status”:…

Process finished with exit code 0
~~~

In order to make this more suitable for getting these tweets into Excel we will now extend this TwitterListener class in the following ways:

– Broadcast updates to other *subscribers* instead of just printing tweets.
– Keep a buffer of the last few received tweets.
– One ever create one listener for each unique set of phrases.
– Automatically disconnect listeners with no subscribers.

The updated TwitterListener class is as follows:

[sourcecode language=”python”]
class TwitterListener(StreamListener):
“””tweepy.StreamListener that notifies multiple subscribers when
new tweets are received and keeps a buffer of the last 100 tweets
received.
“””
__listeners = {} # class level cache of listeners, keyed by phrases
__lock = threading.RLock()
__max_size = 100

@classmethod
def get_listener(cls, phrases, subscriber):
“””Fetch an ExcelListener listening to a set of phrases and subscribe to it”””
with cls.__lock:
# get the listener from the cache or create a new one
phrases = frozenset(map(str, phrases))
listener = cls.__listeners.get(phrases, None)
if listener is None:
listener = cls(phrases)
cls.__listeners[phrases] = listener

# add the subscription and return
listener.subscribe(subscriber)
return listener

def __init__(self, phrases):
“””Use static method ‘get_listener’ instead of constructing directly”””
_log.info(“Creating listener for [%s]” % “, “.join(phrases))
self.__phrases = phrases
self.__subscriptions = set()
self.__tweets = [None] * self.__max_size

# listen for tweets in a background thread using the ‘async’ keyword
auth = OAuthHandler(consumer_key, consumer_secret)
auth.set_access_token(access_token, access_token_secret)
self.__stream = Stream(auth, listener=self)
self.__stream.filter(track=phrases, async=True)
self.__connected = True

@property
def tweets(self):
return list(self.__tweets)

def subscribe(self, subscriber):
“””Add a subscriber that will be notified when new tweets are received”””
with self.__lock:
self.__subscriptions.add(subscriber)

def unsubscribe(self, subscriber):
“””Remove subscriber added previously.
When there are no more subscribers the listener is stopped.
“””
with self.__lock:
self.__subscriptions.remove(subscriber)
if not self.__subscriptions:
self.disconnect()

def disconnect(self):
“””Disconnect from the twitter stream and remove from the cache of listeners.”””
with self.__lock:
if self.__connected:
_log.info(“Disconnecting twitter stream for [%s]” % “, “.join(self.__phrases))
self.__listeners.pop(self.__phrases)
self.__stream.disconnect()
self.__connected = False

@classmethod
def disconnect_all(cls):
“””Disconnect all listeners.”””
with cls.__lock:
for listener in list(cls.__listeners.values()):
listener.disconnect()

def on_data(self, data):
data = json.loads(data)
with self.__lock:
self.__tweets.insert(0, data)
self.__tweets = self.__tweets[:self.__max_size]
for subscriber in self.__subscriptions:
try:
subscriber.on_data(data)
except:
_log.error(“Error calling subscriber”, exc_info=True)
return True

def on_error(self, status):
with self.__lock:
for subscriber in self.__subscriptions:
try:
subscriber.on_error(status)
except:
_log.error(“Error calling subscriber”, exc_info=True)

if __name__ == ‘__main__’:
import time
logging.basicConfig(level=logging.INFO)

class TestSubscriber(object):
“””simple subscriber that just prints tweets as they arrive”””

def on_error(self, status):
print(“Error: %s” % status)

def on_data(self, data):
print(data.get(“text”))

subscriber = TestSubscriber()
listener = TwitterListener.get_listener([“python”, “excel”, “pyxll”], subscriber)

# listen for 60 seconds then stop
time.sleep(60)
listener.unsubscribe(subscriber)
[/sourcecode]

When this is run it’s very similar to the last case, except that now only the text part of the tweets are printed. Also note that the listener is not explicitly disconnected, that happens automatically when the last subscriber unsubscribes.

“`
python twitterxl.py

INFO:__main__:Creating listener for python, excel, pyxll
INFO:requests.packages.urllib3.connectionpool:Starting new HTTPS connection (1): stream.twitter.com
Linuxtoday Make a visual novel with Python: Linux User & Developer: Bridge the gap between books…
How to create drop down list in excel https://t.co/Ii2hKRlRBe…
RT @papisdotio: Flying dron with Python @theglamp #PAPIsConnect https://t.co/zzPNSFb66e…
RT @saaid230: The reason I work hard and try to excel at everything I do so one day I can take care …
RT @javacodegeeks: I’m reading 10 Awesome #Python Tutorials to Kick-Start my Web #Programming https:…
INFO:__main__:Disconnecting twitter stream for

[python 1=”excel,” 2=”pyxll” language=”,”][/python]

Process finished with exit code 0
“`

Step 4: Feed the real time Twitter data into Excel using PyXLL
————————–

Now the hard part of getting the streaming Twitter data into Python is taken care of, creating a real time data source in Excel using PyXLL is pretty straightforward.

PyXLL 3.0 has a new class, `RTD`. When a function decorated with the `xl_func` decorator returns an RTD instance, the value of the calling cell will be the `value` property of the RTD instance. If the value property of the returned RTD instance later changes, the cell value changes.

We will write a new class inheriting from RTD that acts as a subscriber to our twitter stream (in the same way as TestSubscriber in the code above). Whenever a new tweet is received it will update its value, and so the cell in Excel will update.

[sourcecode language=”python”]
from pyxll import RTD

class TwitterRTD(RTD):
“””Twitter RTD class that notifies Excel whenever a new tweet is received.”””

def __init__(self, phrases):
# call super class __init__ with an initial value
super(TwitterRTD, self).__init__(value=”Waiting for tweets…”)

# get the TwitterListener and subscribe to it
self.__listener = TwitterListener.get_listener(phrases, self)

def disconnect(self):
# overridden from RTD base class. Called when Excel no longer
# needs the RTD object (for example, when the cell formula
# is changed.
self.__listener.unsubscribe(self)

def on_error(self, status):
self.value = “#ERROR %s” % status

def on_data(self, data):
self.value = data.get(“text”)
[/sourcecode]

To expose that to Excel all that’s needed is a function that returns an instance of our new TwitterRTD class

[sourcecode language=”python”]
from pyxll import xl_func

@xl_func(“string[] phrases: rtd”)
def twitter_listen(phrases):
“””Listen for tweets containing certain phrases”””
# flatten the 2d list of lists into a single list of phrases
phrases = [str(x) for x in itertools.chain(*phrases) if x]
assert len(phrases) > 0, “At least one phrase is required”

# return our TwitterRTD object that will update when a tweet is received
return TwitterRTD(phrases)
[/sourcecode]

All that’s required now is to add the module to the pyxll.cfg file, and then the new function ‘twitter_listen’ will appear in Excel, and calling it will return a live stream of tweets.

screengrab1

Step 5: Enrich the feed information with Tweet metadata
————————–

So far we’ve got live tweets streaming into Excel, which is pretty cool, but only one tweet is visible at a time and we can only see the tweet text. It would be even better to see a grid of data showing the most recent tweets with some metadata as well as the tweet itself.

RTD functions always return just a single cell of data, so what we need to do is write a slightly different function that takes a couple more arguments: A key for the part of the tweet we want (e.g. ‘text’ or ‘created_at’) and an index (e.g. 0 as the latest tweet, 1 the second most recent tweet etc).

As some interesting bits of metadata are in nested dictionaries in the twitter data, the ‘key’ used to select the item from the data dictionary is a ‘/’ delimited list of keys used to drill into tweet data (for example, the name of the user is in the sub-dictionary ‘user’, so to retrieve it the key ‘user/name’ would be used).

The TwitterListener class we’ve written already keeps a limited history of the tweets it’s received so this isn’t too much more than we’ve already done.

[sourcecode language=”python”]
class TwitterRTD(RTD):
“””Twitter RTD class that notifies Excel whenever a new tweet is received.”””

def __init__(self, phrases, row=0, key=”text”):
super(TwitterRTD, self).__init__(value=”Waiting for tweets…”)
self.__listener = TwitterListener.get_listener(phrases, self)
self.__row = row
self.__key = key

def disconnect(self):
self.__listener.unsubscribe(self)

def on_error(self, status):
self.value = “#ERROR %s” % status

def on_data(self, data):
# if there are no tweets for this row return an empty string
tweets = self.__listener.tweets
if len(tweets) < self.__row or not tweets[self.__row]: self.value = “” return # get the value from the tweets value = tweets[self.__row] for key in self.__key.split(“/”): if not isinstance(value, dict): value = “” break value = value.get(key, {}) # set the value back in Excel self.value = str(value) [/sourcecode] The worksheet function also has to be updated to take these extra arguments [sourcecode language=”python”] @xl_func(“string[] phrases, int row, string key: rtd”) def twitter_listen(phrases, row=0, key=”text”): “””Listen for tweets containing certain phrases””” # flatten the 2d list of lists into a single list of phrases phrases = [str(x) for x in itertools.chain(*phrases) if x] assert len(phrases) > 0, “At least one phrase is required”

# return our TwitterRTD object that will update when a tweet is received
return TwitterRTD(phrases, row, key)
[/sourcecode]

After reloading the PyXLL addin, or restarting Excel, we can now call this modified function with different values for row and key to build an updating grid of live tweets.

screengrab3

One final step is to make sure that any active streams are disconnected when Excel closes. This will prevent the tweepy background thread from preventing Excel from exiting cleanly.

[sourcecode language=”python”]
from pyxll import xl_on_close

@xl_on_close
def disconnect_all_listeners():
TwitterListener.disconnect_all()
[/sourcecode]

The code from this blog is available on github https://github.com/pyxll/pyxll-examples/tree/master/twitter.

Just Released: PyXLL v 3.0 (Python in Excel). New Real Time Data Stream Capabilities, Excel Ribbon Integration, and More.

Download a free 30 day trial of PyXLL and try it with your own data.

Since PyXLL was first released back in 2010 it has grown hugely in popularity and is used by businesses in many different sectors.

The original motivation for PyXLL was to be able to use all the best bits of Excel combined with a modern programming language for scientific computing, in a way that fits naturally and works seamlessly.

Since the beginning, PyXLL development focused on the things that really matter for creating useful real-world spreadsheets; worksheet functions and macro functions. Without these all you can do is just drive Excel by poking numbers in and reading numbers out. At the time the first version of PyXLL was released, that was already possibly using COM, and so providing yet another API to do the same was seen as little value add. On the other hand, being able to write functions and macros in Python opens up possibilities that previously were only available in VBA or writing complicated Excel Addins in C++ or C#.

With the release of PyXLL 3, integrating your Python code into Excel has become more enjoyable than ever. Many things have been simplified to get you up and running faster, and there are some major new features to explore.

  • If you are new to PyXLL have a look at the Getting Started section of the documentation.
  • All the features of PyXLL, including these new ones, can be found in the Documentation

NEW FEATURES IN PYXLL V. 3.0

1. Ribbon Customization

Screen Shot 2016-02-29 at 15.57.12

Ever wanted to write an add-in that uses the Excel ribbon interface? Previously the only way to do this was to write a COM add-in, which requires a lot of knowledge, skill and perseverance! Now you can do it with PyXLL by defining your ribbon as an XML document and adding it to your PyXLL config. All the callbacks between Excel and your Python code are handled for you.

See the Customizing the Ribbon for more detailed information or try the example included in the download.

2. RTD (Real Time Data) Functions

rtd

PyXLL can stream live data into your spreadsheet without you having to write any extra services or register any COM controls. Any Python function exposed to Excel through PyXLL can return a new RTD type that acts as a ticking data source; Excel updates whenever the returned RTD publishes new data.

See Real Time Data for more detailed information or try the example included in the download.

3. Function Signatures and Type Annotation

xl_func and xl_macro need to know the argument and return types to be
able to tell Excel how they should be called. In previous versions that was always done by passing a ‘signature’ string to these decorators.

Now in PyXLL 3 the signature is entirely optional. If a signature is not supplied PyXLL will inspect the function and determine the signature for you.

If you use Python type annotations when declaring the function, PyXLL will use those when determining the function signature. Otherwise all arguments and the return type will be assumed to be `var`.

4. Default Keyword Arguments

Python functions with default keyword arguments now preserve their default value when called from Excel with missing arguments. This means that a function like the one below
when called from Excel with b or c missing will be invoked with the correct default values for b and c.

@xl_func
 def func_with_kwargs(a, b=1, c=2):
 return a + b + c

 5. Deep Reloading

If you’ve used PyXLL for a while you will have noticed that when you reload PyXLL only the modules listed in your pyxll.cfg file get reloaded. If you are working on a project that has multiple modules and not all of them are added to the config those won’t get reloaded, even if modules that are listed in the config file import them.

PyXLL can now track all the imports made by each module listed in the config file, and when you reload PyXLL all of those modules will be reloaded in the right order.

This feature is enabled in the config file by setting

[PYXLL]
deep_reload = 1

6. Error Caching

Sometimes it’s not convenient to have to pick through the log file to determine why a particular cell is failing to calculate.

The new function get_last_error takes an XLCell or a COM Range and returns the last exception (and traceback) to have occurred in that cell.

This can be used in menu functions or other worksheet functions to give end users better feedback about any errors in the worksheet.

7. Python Functions for Reload and Rebind

PyXLL can now be reloaded or it can rebind its Excel functions using the new Python functions reload and rebind.

8. Better win32com and comtypes Support

PyXLL has always had some integration with the pythoncom module, but it required some user code to make it really useful. It didn’t have any direct integration with the higher level win32com package or the
comtypes package.

The new function xl_app returns the current Excel Application instance either as a pythoncom PyIDispatch instance, a win32com.client.Dispatch instance or a wrapped comtypes POINTER(IUnknown) instance.

You may specify which COM library you want to use with PyXLL in the pyxll.cfg file

[PYXLL]
com_package = <win32com, comtypes or pythoncom>

Download a free 30 day trial of PyXLL and see how PyXLL can help you use the power of Python to make Excel an even more powerful data analysis tool.

Plotting in Excel with PyXLL and Matplotlib

Author: Tony Roberts, creator of PyXLL, a Python library that makes it possible to write add-ins for Microsoft Excel in Python. Download a FREE 30 day trial of PyXLL here.


Plotting in Excel with PyXLL and MatplotlibPython has a broad range of tools for data analysis and visualization. While Excel is able to produce various types of plots, sometimes it’s either not quite good enough or it’s just preferable to use matplotlib.

Users already familiar with matplotlib will be aware that when showing a plot as part of a Python script the script stops while a plot is shown and continues once the user has closed it. When doing the same in an IPython console when a plot is shown control returns to the IPython prompt immediately, which is useful for interactive development.

Something that has been asked a couple of times is how to use matplotlib within Excel using PyXLL. As matplotlib is just a Python package like any other it can be imported and used in the same way as from any Python script. The difficulty is that when showing a plot the call to matplotlib blocks and so control isn’t returned to Excel until the user closes the window.

This blog shows how to plot data from Excel using matplotlib and PyXLL so that Excel can continue to be used while a plot window is active, and so that same window can be updated whenever the data in Excel is updated. Continue reading

PyXLL: Deploy Python to Excel Easily

PyXLL Solution Home | Buy PyXLL | Press Release

Today Enthought announced that it is now the worldwide distributor for PyXLL, and we’re excited to offer this key product for deploying Python models, algorithms and code to Excel. Technical teams can use the full power of Enthought Canopy, or another Python distro, and end-users can access the results in their familiar Excel environment. And it’s straightforward to set up and use.

Installing PyXLL from Enthought Canopy

PyXLL is available as a package subscription (with significant discounts for multiple users). Once you’ve purchased a subscription you can easily install it via Canopy’s Package Manager as shown in the screenshots below (note that at this time PyXLL is only available for Windows users). The rest of the configuration instructions are in the Quick Start portion of the documentation. PyXLL itself is a plug-in to Excel. When you start Excel, PyXLL loads into Excel and reads in Python modules that you have created for PyXLL. This makes PyXLL especially useful for organizations that want to manage their code centrally and deploy to multiple Excel users.

Enthought Canopy Package Manager   Install PyXLL from Enthought Canopy's Package Manager

Creating Excel Functions with PyXLL

To create a PyXLL Python Excel function, you use the @xl_func decorator to tell PyXLL the following function should be registered with Excel, what its argument types are, and optionally what its return type is. PyXLL also reads the function’s docstring and provides that in the Excel function description. As an example, I created a module my_pyxll_module.py and registered it with PyXLL via the Continue reading

Avoiding “Excel Hell!” using a Python-based Toolchain

Update (Feb 6, 2014):  Enthought is now the exclusive distributor of PyXLL, a solution that helps users avoid “Excel Hell” by making it easy to develop add-ins for Excel in Python. Learn more here.

Didrik Pinte gave an informative, provocatively-titled presentation at the second, in-person New York Quantitative Python User’s Group (NY QPUG) meeting earlier this month.

There are a lot of examples in the press of Excel workflow mess-ups and spreadsheet errors contributing to some eye-popping mishaps in the finance world (e.g. JP Morgan’s spreadsheet issues may have led to the 2012 massive loss at “the London Whale”). Most of these can be traced to similar fundamental issues:

  • Data referencing/traceability

  • Numerical errors

  • Error-prone manual operations (cut & paste, …)

  • Tracing IO’s in libraries/API’s

  • Missing version control

  • Toolchain that doesn’t meet the needs of researchers, analysts, IT, etc.

Python, the coding language and its tool ecosystem, can provide a nice solution to these challenges, and many organizations are already turning to Python-based workflows in response. And with integration tools like PyXLL (to execute Python functions within Excel) and others, organizations can adopt Python-based workflows incrementally and start improving their current “Excel Hell” situation quickly.

For the details check out the video of Didrik’s NY QPUG presentation.  He demonstrates a an example solution using PyXLL and Enthought Canopy.

[vimeo 67327735 http://vimeo.com/67327735]

And grab the PDF of his slides here.

QPUG_20130514_ExcelHell_Slides

It would be great to hear your stories about “Excel Hell”. Let us know below.

–Brett Murphy