I have my Eastern Bank account configured to email me notifications about checking deposits. There are three problems with these notifications:
- They include only the amount of the deposit, not whom it’s from.
- They only put the deposit amount in the body of the email, not in the subject.
- They email me about every deposit, including the ones I already know about since I’m the one who made them. I want to be notified about direct deposits, but I really don’t want to be notified about check deposits. Eastern Bank’s online banking platform doesn’t support this distinction.
(Century Bank’s online banking platform had the same problem. I was hoping things would approve in this area after the acquisition by Eastern Bank, but alas no.)
This finally annoyed me enough today that I decided to do something about it. After implementing the hack described below, each deposit notification email is handled in one of three ways:
- If the deposit is already in my bookkeeping system (GnuCash), the email is simply discarded; I don’t need to see it!
- Otherwise, an attempt is made to log into Eastern Bank online banking automatically to fetch a description of the deposit from the list of recent transactions in my checking account. If successful, then the deposit amount and description are appended to the Subject of the email before it is put in my inbox.
- If step 2 fails for whatever reason, the email is put in my inbox as is.
All that sounds pretty straightforward, but there are actually a lot of moving parts under the surface.
Logging into Eastern Bank online banking automatically requires storing my username and password somewhere I feel comfortable with and enabling automatic capturing of the multi-factor authentication (MFA) code sent by Eastern Bank when it decides that MFA is required to log in.
I am certainly not going to do all this on my mail server in the cloud, because wow holy cow that is not a good idea! So instead I do the actual processing on my home server in my basement, which is protected by enough layers of paranoid security that I’m OK with it. (The risk calculus I’ve done to conclude this is interesting, but that’s a topic for another blog posting.)
So let’s talk about the nuts and bolts of how all this works. This rule in my procmailrc file on my mail server forwards the incoming deposit notification email to my home server and waits for it to process the email and return either the modified email or nothing:
| :0 | |
| * ^From: .*Eastern Bank | |
| * ^Subject: Eastern Bank Deposit Alert | |
| { | |
| :0 fw : $MAILDIR/deposit-notification$LOCKEXT | |
| |tf=/tmp/notif.$$; curl –silent –data-binary '@-' http://fillmein/deposit_notification.cgi > $tf; if [ -s $tf ]; then cat $tf; e=0; else e=1; fi; exit $e | |
| :0 | |
| * ^X-In-GnuCash-File: Yes | |
| /dev/null | |
| } |
(Notice that in this gist and others I have replaced sensitive or site-specific strings with “fillmein”. You will need to do the needful if you want t use any of this stuff!)
If the script on my home server returns a non-empty response, the procmail rule assumes that it is a filtered version of the message and uses it; otherwise the filter fails and the message is preserved as-is.
After filtering the message the procmail rule above checks if the script has added the “X-In-GnuCash-File” header to it. If so, it discards that message because that means I already know about the deposit and don’t need to see the notification about it.
On my home server, the CGI script called above looks like this:
| #!/bin/bash -e | |
| TF=/fillmein/deposit_notification.$$ | |
| trap "rm -f $TF" EXIT | |
| echo "Content-type: message/rfc822" | |
| echo | |
| if /fillmein/eastern-deposit-notification.py >| $TF; then | |
| cat $TF | |
| fi |
The directory this script lives in is protected by a .htaccess file which restricts access to it so only my mail server and a few other local machines and networks can talk to it:
| <FilesMatch "^deposit_notification\.cgi$"> | |
| Order deny,allow | |
| Deny from all | |
| Allow from fillmein | |
| Allow from fillmein | |
| Allow from fillmein | |
| Allow from fillmein | |
| </FilesMatch> |
This brings us to the “meat” of this whole endeavor, the script that parses the notification email, checks my GnuCash file, and logs into Eastern Bank to fetch the deposit description if necessary:
| #!/usr/bin/env python3 | |
| # Objective: decide whether to notify about a deposit notification email | |
| # from Eastern Bank, and if so, try to fetch a description of the deposit | |
| # from online banking. | |
| # | |
| # 1. Read notification email from Eastern Bank on stdin. | |
| # 2. Extract deposit amount from email. | |
| # 3. Check if deposit is in GnuCash file. If so, then: | |
| # a. Add a header to the message indicating that for procmail to read. | |
| # b. Spit the modified message to stdout. | |
| # c. Exit. | |
| # 4. Try to log into online banking, requesting MFA by email. | |
| # 5. Wait for MFA code to be saved to the filesystem by a different script. | |
| # 6. Finish logging in using saved MFA code and then delete it. | |
| # 7. Go to checking account page. | |
| # 8. Find recent transaction with matching amount. | |
| # 9. Add description of transaction to email. | |
| # 10. Spit the modified message to stdout. | |
| # 11. Exit. | |
| import datetime | |
| import os | |
| from pandas import Timestamp | |
| import pprint | |
| import re | |
| from selenium import webdriver | |
| from selenium.webdriver.common.by import By | |
| from selenium.common.exceptions import ( | |
| NoSuchElementException, | |
| StaleElementReferenceException, | |
| ) | |
| from selenium.webdriver.common.action_chains import ActionChains # noqa | |
| from selenium.webdriver.common.keys import Keys | |
| import sys | |
| import time | |
| import xml.etree.ElementTree as ET | |
| gnucash_file = os.path.expanduser('/fillmein') | |
| mfacode_file = os.path.expanduser('/fillmein/mfacode_eastern.txt') | |
| password_file = os.path.expanduser('/fillmein') | |
| ns = { | |
| 'gnc': 'http://www.gnucash.org/XML/gnc', | |
| 'act': 'http://www.gnucash.org/XML/act', | |
| 'trn': 'http://www.gnucash.org/XML/trn', | |
| 'ts': 'http://www.gnucash.org/XML/ts', | |
| 'split': 'http://www.gnucash.org/XML/split', | |
| } | |
| crlf = False | |
| wait_time = 10 | |
| username_xpath = '//*[@id="AuthenticationFG.USER_PRINCIPAL"]' | |
| password_xpath = '//*[@id="AuthenticationFG.ACCESS_CODE_1"]' | |
| login_button_xpath = '//*[@id="VALIDATE_CREDENTIALS1"]' | |
| account_xpath = '//span[contains(text(), "PREMIER CHECKING")]' | |
| mfa_button_xpath = '//p[contains(text(), "Text Message")]' | |
| mfa_code_xpath = '//*[@id="AuthenticationFG.RSA_GENERATED_OTP"]' | |
| username = 'fillmein' | |
| password = open(password_file).read().strip() | |
| def get_email(): | |
| global crlf | |
| if len(sys.argv) == 2: | |
| email = open(sys.argv[1]).read() | |
| else: | |
| email = sys.stdin.read() | |
| if '\r\n' in email: | |
| email = re.sub(r'\r\n', r'\n', email) | |
| crlf = True | |
| return email | |
| def print_email(email): | |
| if crlf: | |
| email = re.sub(r'\n', r'\r\n', email) | |
| print(email) | |
| def get_deposit_amount(email): | |
| r = r'A deposit in the amount of \$\s*([\d.]+)' | |
| match = re.search(r, email) | |
| if not match: | |
| raise Exception(f'No match for {r} in email') | |
| return float(match.group(1)) | |
| def is_in_gnucash_file(amount): | |
| tree = ET.parse(gnucash_file) | |
| root = tree.getroot() | |
| accounts = root.findall('*/gnc:account', ns) | |
| # Assumes accounts are sorted by depth | |
| assets_guid = None | |
| for account in accounts: | |
| name = account.find('act:name', ns).text | |
| guid = account.find('act:id', ns).text | |
| if not assets_guid and name == 'Assets': | |
| assets_guid = guid | |
| continue | |
| parent = account.find('act:parent', ns) | |
| if parent is not None: | |
| parent = parent.text | |
| if assets_guid and name == 'Checking' and parent == assets_guid: | |
| checking_guid = guid | |
| break | |
| else: | |
| raise Exception('Failed to find checking account in GnuCash file') | |
| all_transactions = (t for t in root.findall('*/gnc:transaction', ns)) | |
| before = (datetime.date.today() – datetime.timedelta(days=7)).\ | |
| strftime('%F') | |
| recent_transactions = ( | |
| t for t in all_transactions | |
| if t.find('trn:date-posted/ts:date', ns).text > before | |
| ) | |
| checking_splits = ( | |
| s | |
| for t in recent_transactions | |
| for s in t.findall('trn:splits/trn:split', ns) | |
| if s.find('split:account', ns).text == checking_guid | |
| ) | |
| deposit_amounts = ( | |
| int(match.group(1)) / 100 | |
| for s in checking_splits | |
| for value in s.findall('split:value', ns) | |
| for match in (re.match(r'([0-9]+)/100', value.text),) | |
| if match is not None | |
| ) | |
| return (amount in deposit_amounts) | |
| def reset_mfa_code(): | |
| try: | |
| os.remove(os.path.realpath(mfacode_file)) | |
| except FileNotFoundError: | |
| pass | |
| def get_mfa_code(wait=30): | |
| when = time.time() + wait | |
| while time.time() < when: | |
| try: | |
| code = open(mfacode_file).read().strip() | |
| break | |
| except FileNotFoundError: | |
| time.sleep(1) | |
| else: | |
| raise Exception('Failed to read MFA code') | |
| os.remove(os.path.realpath(mfacode_file)) | |
| return code | |
| def wait_for_stale(elt): | |
| end = time.time() + wait_time | |
| while time.time() < end: | |
| try: | |
| elt.find_elements(By.ID, 'nosuchid') | |
| time.sleep(1) | |
| except StaleElementReferenceException: | |
| break | |
| else: | |
| raise Exception("New page didn't load") | |
| def wait_for_xpath(driver, xpath): | |
| end = time.time() + wait_time | |
| while time.time() < end: | |
| try: | |
| return driver.find_element(By.XPATH, xpath) | |
| except NoSuchElementException: | |
| time.sleep(1) | |
| print(driver.page_source, file=sys.stderr) | |
| raise Exception(f'No match for xpath {xpath}') | |
| def find_one_of_several(driver, xpaths, timeout=wait_time): | |
| end_at = time.time() + timeout | |
| while time.time() < end_at: | |
| for xpath in xpaths: | |
| try: | |
| driver.find_element(By.XPATH, xpath) | |
| return xpath | |
| except Exception: | |
| continue | |
| time.sleep(1) | |
| else: | |
| print(driver.page_source, file=sys.stderr) | |
| raise TimeoutError(f'Could not find any of {xpaths}') | |
| def add_description_from_web(amount, email): | |
| options = webdriver.ChromeOptions() | |
| options.add_argument('–disable-gpu') | |
| options.add_argument('headless') | |
| driver = webdriver.Chrome(options=options) | |
| try: | |
| driver.get('https://online.easternbank.com') | |
| driver.find_element(By.XPATH, username_xpath).send_keys(username) | |
| password_field = driver.find_element(By.XPATH, password_xpath) | |
| password_field.send_keys(password) | |
| password_field.send_keys(Keys.ENTER) | |
| wait_for_stale(password_field) | |
| xpath = find_one_of_several(driver, (account_xpath, mfa_button_xpath)) | |
| if xpath == mfa_button_xpath: | |
| button = driver.find_element(By.XPATH, xpath) | |
| reset_mfa_code() | |
| # This should work but doesn't. I have no idea why. It clicks in | |
| # the wrong place. :shrug: | |
| # ActionChains(driver).move_to_element(button).click().perform() | |
| # So instead I fetch the anchor link and get it directly rather | |
| # than clicking. | |
| anchor = button.find_element(By.XPATH, '..') | |
| driver.get(anchor.get_attribute('href')) | |
| code_input = wait_for_xpath(driver, mfa_code_xpath) | |
| mfa_code = get_mfa_code() | |
| code_input.send_keys(mfa_code) | |
| code_input.send_keys(Keys.ENTER) | |
| wait_for_stale(code_input) | |
| account_tile = wait_for_xpath(driver, account_xpath) | |
| account_tile.click() | |
| wait_for_stale(account_tile) | |
| wait_for_xpath(driver, '//td[contains(@ng-if,"row.colorCode")]') | |
| then = datetime.date.today() – datetime.timedelta(days=7) | |
| rows = [] | |
| for row in driver.find_elements(By.TAG_NAME, 'tr'): | |
| columns = [c.get_attribute('innerHTML') | |
| for c in row.find_elements(By.TAG_NAME, 'td')] | |
| rows.append(columns) | |
| if not any(c for c in columns | |
| if c.startswith('+') and c.endswith(f'{amount:,.2f}')): | |
| continue | |
| if Timestamp(columns[0]).date() < then: | |
| continue | |
| # So gross | |
| description = re.search(r'</span>(.*?)\n', columns[1]).group(1) | |
| break | |
| else: | |
| pprint.pprint(rows, stream=sys.stderr) | |
| print(driver.page_source, file=sys.stderr) | |
| print(f'Did not find {amount:,.2f} deposit in online banking', | |
| file=sys.stderr) | |
| return | |
| description = re.sub(r'^(?:Preauthorized|Transfer) Credit ', '', | |
| description) | |
| email = re.sub(r'^(Subject: .*)', | |
| f'\\1 (${amount:,.2f}, {description})', | |
| email, 1, re.M) | |
| print(email) | |
| sys.exit() | |
| finally: | |
| driver.quit() | |
| def main(): | |
| email = get_email() | |
| deposit_amount = get_deposit_amount(email) | |
| print(f'Deposit amount is {deposit_amount}', file=sys.stderr) | |
| if is_in_gnucash_file(deposit_amount): | |
| email = re.sub(r'\n\n', '\nX-In-GnuCash-File: Yes\n\n', email, 1) | |
| print_email(email) | |
| sys.exit() | |
| add_description_from_web(deposit_amount, email) | |
| sys.exit(1) | |
| if __name__ == '__main__': | |
| main() |
It’s worth noting that this script assumes that the GnuCash data file is uncompressed, since that’s how I have GnuCash configured. You’d have to modify the script to uncompress the file when reading it if you keep yours compressed.
If you don’t use GnuCash you could modify script so that instead of parsing a GnuCash file, it fetches the deposit description from online banking and add “X-In-GnuCash-File” to the header if it’s “Mobile Check Deposit” or a similar description that you don’t care about.
The function add_description_from_web is where the logging-into-online-banking magic happens, using Selenium. The tricky part there is handling MFA. That requires more infrastructure! If the site asks for MFA, the script asks it to send a text message, which is not terribly secure (I wish Eastern Bank MFA supported TOTP!) but more secure than email. This causes a text message to be sent to my Android phone. There, Tasker + AutoNotification intercepts the message, extracts the MFA code from it, and sends it to my home server. Here’s the Tasker description (the XML is here; it’s too ungainly and ugly to be worth embedding, so you can just download it):
| Profile: Forward Eastern MFA Codes | |
| Settings: Restore: no | |
| Event: AutoNotification Intercept [ Configuration:Event Behaviour: true | |
| Notification Type: Only Created Notifications | |
| Persistency Type: Non-Persistent Only | |
| Notification Apps: Messages | |
| Notification Text: Eastern Bank will not call you for this code. ] | |
| Enter Task: Send Eastern MFA Code | |
| Task: Send Eastern MFA Code | |
| A1: Variable Search Replace [ | |
| Variable: %antext | |
| Search: Your security code is \d+ | |
| Multi-Line: On | |
| One Match Only: On | |
| Store Matches In Array: %mfamatch ] | |
| A2: Variable Search Replace [ | |
| Variable: %mfamatch1 | |
| Search: \d+ | |
| One Match Only: On | |
| Store Matches In Array: %mfacode ] | |
| A3: HTTP Request [ | |
| Method: GET | |
| URL: http://fillmein/mfacode.cgi?source=eastern&c=%mfacode1 | |
| Timeout (Seconds): 30 ] | |
| A4: AutoNotification Cancel [ | |
| Configuration: Id: %anid | |
| Timeout (Seconds): 20 ] |
This is the mfacode.cgi script that Tasker is posting the MFA code to on my home server:
| #!/bin/bash -e | |
| declare -A params | |
| while read kv; do | |
| key="${kv%%=*}" | |
| if [ ! "$key" ]; then | |
| continue | |
| fi | |
| if [[ "$kv" =~ = ]]; then | |
| value="${kv#*=}" | |
| else | |
| value="" | |
| fi | |
| params[$key]="$value" | |
| done < <(echo "${QUERY_STRING}" | sed -E "s/&/\n/g") | |
| # For debugging | |
| #for key in "${!params[@]}"; do | |
| # echo "params[$key]=${params[$key]}" | |
| #done | |
| #exit | |
| safe() { | |
| echo "$1" | tr -c -d — -_A-Za-z0-9 | |
| } | |
| source=$(safe "${params}") | |
| code=$(safe "${params}") | |
| D=/fillmein | |
| if [ -n "$source" ]; then | |
| F="$D/mfacode_$source.txt" | |
| else | |
| F="$D/mfacode.txt" | |
| fi | |
| if [ -n "$code" ]; then | |
| echo "$code" >| $F.tmp | |
| mv -f $F.tmp $F | |
| fi | |
| echo "Content-Type: text/plain" | |
| echo | |
| echo ok |
