Notify with Python

Notify with Python

  • 89

Make life easier with Python built email notifications .For this purpose, I put together a set of Python scripts built for this exact problem. I use these scripts to send process updates, visualizations and completion notifications to my phone

Working in Python, I often run data processing, transfer, and model training scripts. Now, with any reasonable degree of complexity and/or big data, this can take some time.

Although often enough we all have some other work to be done whilst waiting for these processing to complete, occasionally, we don’t.

For this purpose, I put together a set of Python scripts built for this exact problem. I use these scripts to send process updates, visualizations and completion notifications to my phone.

So, when we do occasionally have those moments of freedom. We can enjoy them without being worried about model progress.

What we need

Okay so the first thing we need to ask is — what do we need to know?

Now of-course, this really depends on the work you are doing. For me I have three main processing tasks that have the potential to take up time:

  • Model training
  • Data processing and/or transfer
  • Financial modelling

For each of these, there are of-course different pieces of information that we need to stay informed about. Let’s take a look at an example of each.

Model Training

Update every n epochs, must include key metrics. For example, loss and accuracy for training and validation sets.

Notification of completion (of-course). For this I like to include:

  • prediction outputs, for text generation, the generated text (or a sample of it) — for image generation, a (hopefully) cool visualization.
  • visualization of key metrics during training (again, loss and accuracy for both training and validation sets)
  • other, less essential but still useful information such as local model directories, training time, model architecture etc

Let’s take the example of training a neural network to reproduce a given artistic style.

For this, we want to see; generated images from the model, loss and accuracy plots, and current training time, and a model name.

import notify

START = datetime.now()  # this line would be placed before model training begins
MODELNAME = "Synthwave GAN"  # giving us our model name
NOTIFY = 100  # so we send an update notification every 100 epochs

# for each epoch e, we would include the following code
if e % notify_epoch == 0 and e != 0:
    # here we create the email body message
    txt = (f"{MODELNAME} update as of "
           f"{datetime.now().strftime('%H:%M:%S')}.")

    # we build the MIME message object with notify.message
    msg = notify.message(
        subject='Synthwave GAN',
        text=txt,
        img=[
            f'../visuals/{MODELNAME}/epoch_{e}_loss.png',
            f'../visuals/{MODELNAME}/epoch_{e}_iter_{i}.png'
        ]
    )  # note that we attach two images here, the loss plot and
    #    ...a generated image output from our model
           
    notify.send(msg)  # we then send the message

notify_ml.py
In this scenario, every 100 epochs, an email containing all of the above will be sent. Here is one of those emails:

This is image title

Data Processing and Transfer

This one is slightly less glamorous, but in terms of time consumed, is number one by a long-shot.

We will use the example of bulk data upload to SQL Server using Python (for those of us without BULK INSERT).

At the end of the upload script, we include a simple message notifying us of upload completion.

import os
import notify
from data import Sql  # see https://jamescalam.github.io/pysqlplus/lib/data/sql.html

dt = Sql('database123', 'server001')  # setup the connection to SQL Server

for i, file in enumerate(os.listdir('../data/new')):
    dt.push_raw(f'../data/new/{file}')  # push a file to SQL Server

# once the upload is complete, send a notification
# first we create the message
msg = notify.message(
    subject='SQL Data Upload',
    text=f'Data upload complete, {i} files uploaded.',
)

# send the message
notify.send(msg)

notify_sql.py

If errors are occasionally thrown, we could also add a try-except clause to catch the error, and add it to a list to include in our update and/or completion email.

Financial Modelling

In the case of financial modelling, everything I run is actually pretty quick, so I can only provide you with an ‘example’ use-case here.

We will use the example of a cash-flow modelling tool. In-reality, this process take no more than a 10–20 seconds, but for now let’s assume we’re hot-shot Wall Street quants processing a few million (rather than hundred) loans.

With this email, we may want to include a high-level summary of the analysed portfolio. We can randomly select a few loans and visualize key values over the given time period — giving us a small sample to cross-check model performance is as expected.

end = datetime.datetime.now()  # get the ending datetime

# get the total runtime in hours:minutes:seconds
hours, rem = divmod((end - start).seconds, 3600)
mins, secs = divmod(rem, 60)
runtime = '{:02d}:{:02d}:{:02d}'.format(hours, mins, secs)

# now built our message
notify.msg(
    subject="Cashflow Model Completion",
    text=(f'{len(model.output)} loans processed.\n'
          f'Total runtime: {runtime}'),
    img=[
        '../vis/loan01_amortisation.png',
        '../vis/loan07_amortisation.png',
        '../vis/loan01_profit_and_loss.png',
        '../vis/loan07_profit_and_loss.png'
    ]
)

notify.send(msg)  # and send it

notify_model.py

The Code

All of the functionality above filters from a single script called notify.py.

We will use Outlook in our example code. However, translating this to other providers is incredibly easy, which we will also cover quickly at the end.

There are two Python libraries we need here, email and smtplib.

  • [**email**](https://docs.python.org/3/library/email.html) — For managing email messages. With this we will setup the email message itself, including subject, body, and attachments.
  • [**smtplib**](https://docs.python.org/3/library/smtplib.html)— Handles the SMTP connection. The simple mail transfer protocol (SMTP) is the protocol used by the majority of email systems, allowing mail to be sent over the internet.’

MIME

The message itself is built using a MIMEMultipart object from the email module. We also use three MIME sub-classes, which we attach to the MIMEMultipart object:

  • MIMEText — This will contain the email ‘payload’, meaning the text within the email body.
  • MIMEImage — Reasonably easy to guess, this is used to contain images within our email.
  • MIMEApplication — Used for MIME message application objects. For us, these are file attachments.

In addition to these sub-classes, there are also other parameters, such as the Subject value in MimeMultipart. All of these together gives us the following structure.

This is image title

Let’s take a look at putting these all together.

import os
from email.mime.text import MIMEText
from email.mime.image import MIMEImage
from email.mime.application import MIMEApplication
from email.mime.multipart import MIMEMultipart

def message(subject="Python Notification", text="", img=None, attachment=None):
    # build message contents
    msg = MIMEMultipart()
    msg['Subject'] = subject  # add in the subject
    msg.attach(MIMEText(text))  # add text contents

    # check if we have anything given in the img parameter
    if img is not None:
        # if we do, we want to iterate through the images, so let's check that
        # what we have is actually a list
        if type(img) is not list:
            img = [img]  # if it isn't a list, make it one
        # now iterate through our list
        for one_img in img:
            img_data = open(one_img, 'rb').read()  # read the image binary data
            # attach the image data to MIMEMultipart using MIMEImage, we add
            # the given filename use os.basename
            msg.attach(MIMEImage(img_data, name=os.path.basename(one_img)))

    # we do the same for attachments as we did for images
    if attachment is not None:
        if type(attachment) is not list:
            attachment = [attachment]  # if it isn't a list, make it one
            
        for one_attachment in attachment:
            with open(one_attachment, 'rb') as f:
                # read in the attachment using MIMEApplication
                file = MIMEApplication(
                    f.read(),
                    name=os.path.basename(one_attachment)
                )
            # here we edit the attached file metadata
            file['Content-Disposition'] = f'attachment; filename="{os.path.basename(one_attachment)}"'
            msg.attach(file)  # finally, add the attachment to our message object
    return msg

notify_message.py

This script, for the most part, is reasonably straightforward. At the top, we have our imports — which are the MIME parts we covered before, and Python’s os library.

Following this define a function called message. This allows us to call the function with different parameters and build an email message object with ease. For example, we can write an email with multiple images and attachments like so:

email_msg = message(
    text="Model processing complete, please see attached data.",
    img=['accuracy.png', 'loss.png'],
    attachments=['data_in.csv', 'data_out.csv']
)

First we initialize the MIMEMultipart object, assigning it to msg.

We then set the email subject using the 'Subject' key.

The attach method allows us to add different MIME sub-classes to our MIMEMultipart object. With this we can add the email body, using the MIMEText sub-class.

For both images img and attachments attachment, we can pass either nothing, a single file-path, or a list of file-paths.

This is handled by first checking if the parameters are None, if they are, we pass. Otherwise, we check the data-type given, is it is not a list, we make it one — this allows us to use the following for loop to iterate through our items.

At this point we use the MIMEImage and MIMEApplication sub-classes to attach our images and files respectively. For both we use os.basename to retrieve the filename from the given file-path, which we include as the attachment name.

SMTP

Now that we have built our email message object, we need to send it.

This is where the smtplib module comes in. The code is again, pretty straight-forward, with one exception.

As we are dealing directly with different email provider’s, and their respective servers, we need different SMTP addresses for each. Fortunately, this is really easy to find.

Type “outlook smtp” into Google. Without even clicking on a page, we are given the server address smtp-mail.outlook.com, and port number 587.

We use both of these when initalizing the SMTP object with smtplib.SMTP — near the beginning of the send function.

import smtplib
import socket

def send(server='smtp-mail.outlook.com', port='587', msg):
    # contain following in try-except in case of momentary network errors
    try:
        # initialise connection to email server, the default is Outlook
        smtp = smtplib.SMTP(server, port)
        # this is the 'Extended Hello' command, essentially greeting our SMTP or ESMTP server
        smtp.ehlo()
        # this is the 'Start Transport Layer Security' command, tells the server we will
        # be communicating with TLS encryption
        smtp.starttls()
        
        # read email and password from file
        with open('../data/email.txt', 'r') as fp:
            email = fp.read()
        with open('../data/password.txt', 'r') as fp:
            pwd = fp.read()
            
        # login to outlook server
        smtp.login(email, pwd)
        # send notification to self
        smtp.sendmail(email, email, msg.as_string())
        # disconnect from the server
        smtp.quit()
    except socket.gaierror:
        print("Network connection error, email not sent.")

notify_smtp.py

smtp.ehlo() and smtp.starttls() are both SMTP commands. ehlo (Extended Hello) essentially greets the server. starttls informs the server we will be communicating using an encrypted transport level security (TLS) connection. You can learn more about SMTP commands here.

After this we simply read in our email and password from file, storing both in email and pwd respectively.

We then login to the SMTP server with smtp.login, and send the email with smtp.sendmail.

I always send the notifications to myself, but in the case of automated reporting (or for any other reason), you may want to send the email elsewhere. To do this, change the destination_address: smtp.sendmail(email, **_destination_address_**, msg.as_string).

Finally, we terminate the session and close the connection with smtp.quit.

All of this is placed within a try-except statement. In the case of momentary network connection loss, we will be unable to connect to the server. Resulting in a socket.gaierror.

Implementing this try-except statement prevents the program from breaking in the case of a lapse in network connection. How you deal with this may differ, depending on how important it is for the email to be sent.

For me, I use this for ML model training updates and data transfer completion. If an email does not get sent, it doesn’t really matter. So this simple, passive handling of connection loss is suitable.

Putting it Together

Now we have written both parts of our code, we can send emails with just:

# build a message object
msg = message(text="See attached!", img='important.png',
              attachment='data.csv')
send(msg)  # send the email (defaults to Outlook)

That is all for email notification and/or automation using Python. Thanks to the email and smptlib libraries this is an incredibly easy process to setup.

Common email provider server address and TLS ports. If you have any to add, let me know!
For any processing or training tasks that consume a lot of time, progress updates and notification on completion, is often truly liberating.

I hope this article has been useful to a few of you out there, as always, let me know if you have any questions or suggestions!

Thanks for reading!