Bin Day Notifications

Posted on 23/03/25 in project, bindicator

Tags: python, mqtt, web scraping, sms, node-red

We’ve all been there—dealing with the mundane task of handling the bins. It’s my responsibility at home to take them out each week, and I can’t say I enjoy it. Week after week, I find myself trying to figure out which bin needs to be taken out, as the collection schedule constantly alternates between landfill and recycling. It may appear simple on the surface, but if you’re able to effortlessly remember and keep track of it, you’re certainly a more organized individual than I am.

The current Process

Let me set the scene for you: Bin collection day is typically on a Friday, which means the anticipation starts on Thursday evening. I find myself constantly glancing out the window, observing my neighbors’ bin movements. However, there are times when their bins remain untouched, and that’s when the mental games begin. Throughout the evening, I repeatedly check, hoping for some sign of activity, but nothing happens. I wonder, do they not know it’s bin day? Or have they simply chosen not to bother? This situation troubles me because we’re usually up and about or away from home before they and the bins are collected, leaving me with a limited window to figure out what’s going on.

On Friday morning, as I’m about to embark on the school run and head to work, I notice that my neighbors still haven’t put out their bins. Now I have no choice but to resort to using the local authority website, something I was hoping to avoid. Unfortunately, the website designed to provide information on which bin to put out is the epitome of a terrible user experience.

Local Authority

First of all, accessing the bin collection information on the local authority website is a hassle. You can’t find it directly from the homepage and instead rely on Google’s index to locate it. Once you finally reach the bin collection portal, you’re required to enter your postcode. However, as you enter the postcode, the website continuously loses focus on the input field due to an ongoing address lookup process. Only when you manage to input the complete postcode does a selection box appear, displaying all the addresses associated with that postcode. After submitting this form, the real frustration begins. The website’s performance is agonizingly slow, and the experience is even worse on mobile devices. Eventually, a table is presented, showing two entries for each collection: one for landfill or recycling and another for the food caddy. It’s perplexing why they bother listing the food caddy since it’s collected every time, rendering its inclusion somewhat pointless.

It gets worse

Bin collection day is typically on a Friday, but things get complicated when there’s a Bank Holiday in the UK. On those occasions, the collection day shifts added an extra layer of confusion to the process. The reliability of neighbors in putting out their bins diminishes even further. Although this doesn’t occur frequently, it’s enough to be bothersome.

However, what pushed me over the edge was the series of shutdowns and national holidays during Queen Elizabeth II’s funeral. The bin collections became a chaotic mess. They were rescheduled, brought forward, delayed by a week, and ended up on random weekdays. It was a complete disruption. Even though the local authority website provided accurate bin collection dates, as I mentioned earlier, the user experience was dreadful.

Frustrated with the situation, I decided it was time for a change. I began searching for inspiration and alternatives.

Inspiration

In search of inspiration, I turned to Google. Most of the results led me to specific local authorities offering notification services for bin collection dates, either through email or SMS. Excited about the possibility of saving time, I swiftly checked the Harlow local authority website to see if such a service was available. To no surprise, there was no such service offered. Otherwise, this post wouldn’t exist, as there would be no need for it. It left me wondering what the point was if such a convenient service wasn’t provided.

During my search, I stumbled upon the most remarkable bin collection notification service called the Bindicator, created by Darren Tarbard. This innovative device illuminates with different colors on the designated collection day, indicating which bin should be put out. It’s awesome!

Although the Bindicator created by Darren Tarbard was truly impressive and something worth aspiring to, I made the decision not to pursue that path due to resource limitations. During the time I considered this, obtaining single-board computers and microcontrollers was quite challenging unless one was willing to spend a significant amount of money. Therefore, the idea of using a microcontroller to light up a bin or acquiring a Raspberry Pi Zero 2W felt a bit excessive and impractical given the circumstances.

Therefore, I settled for the simplicity of receiving an SMS notification on my mobile device on bin collection day. The message would inform me that it’s bin day and specify which bin should be put out. However, I couldn’t resist borrowing the name “Bindicator” from Darren for this purpose.

Harlow Bindicator

Since I’ve borrowed the name, I should make a slight modification to avoid any misunderstandings. Therefore, I’ll call it the Harlow Bindicator, a name that reflects its purpose. As for the notification method, I’ve decided that SMS is the most convenient way to receive updates. To achieve this, I’ll utilize Twilio, which simplifies the process of sending SMS notifications. Now, I need to focus on the remaining aspects of the implementation.

To fetch the bin collection data from the Harlow Local Authority website, the Harlow Bindicator will utilize Selenium to request and render the web page. Once the page is obtained, I’ll employ BeautifulSoup to parse the data and extract the relevant information about the bin day and which bin needs to be put out. With the data retrieval and parsing steps established, the next task is to seamlessly integrate them.

Based on my previous decision, I’ve opted to leverage MQTT and Node-RED for the implementation. There are two main reasons behind this choice. Firstly, I already have an MQTT broker running on my home network, and Node-RED is installed and ready to use. Additionally, Node-RED provides a convenient Twilio Node that simplifies the process of sending SMS notifications. With just an API key, message, and phone number, the Twilio Node handles the SMS functionality effortlessly. By taking this approach, the Python application becomes much simpler to develop. Moreover, if someone else wishes to utilize it, they can utilize the message from Node-RED and customize the output according to their preferences.

Let’s break down the application section of the flowchart:

Harlow Bindicator Application: This refers to the Python application responsible for fetching bin collection data, processing it, and publishing the relevant information via MQTT.
MQTT Broker: This is the MQTT broker running on your home network. It acts as the intermediary for communication between different devices and services using the MQTT protocol.
MQTT Publish: In the Harlow Bindicator Application, after retrieving and processing the bin collection data, it publishes the information (such as the bin day and type) as an MQTT message to a specific topic on the MQTT broker.
Node-RED: This is an open-source flow-based programming tool that serves as a visual programming interface. It allows you to create flows and automate tasks by connecting various nodes.
MQTT Subscribe: Within Node-RED, you configure an MQTT node to subscribe to the topic where the Harlow Bindicator Application publishes the bin collection information.
Twilio Node: Node-RED provides a Twilio Node, which simplifies the integration with Twilio’s SMS functionality. You set up the Twilio Node with your API key, message content, and the phone number to send the SMS notification to.
SMS Notification: Once the MQTT message containing the bin collection details is received by the MQTT Subscribe node in Node-RED, it triggers the Twilio Node to send an SMS notification to the specified phone number, providing the bin day and type.

By breaking down the application section, we can see how the Harlow Bindicator Application interacts with the MQTT broker and how Node-RED facilitates the communication and SMS notification process.

Bindicator Python Code

Certainly! Here are the states of the Python application and what it needs to accomplish:

Fetch Bin Collection Data: The application needs to access the Harlow Local Authority website using Selenium to retrieve the bin collection data. This involves requesting and rendering the web page.
Parse Bin Collection Data: Once the web page is obtained, the application utilizes BeautifulSoup to parse the data and extract the relevant information, such as the bin day and the type of bin to be put out.
Publish Bin Collection Information: After parsing the data, the application publishes the bin collection details (bin day and type) as an MQTT message. This message is published on a specific topic on the MQTT broker.

That’s it! The Python application’s primary responsibility is to fetch the bin collection data, parse it, and then publish the extracted information as an MQTT message. The communication and notifications are handled by external components, such as the MQTT broker and Node-RED with the Twilio Node.

Configuration

The Harlow Bindicator Python application starts by loading a configuration file. This file includes MQTT connection information and a UPRN (Unique Property Reference Number). The UPRN is used to retrieve bin collection data for a specific address. You can find your UPRN using this site by providing your street name and door number.

The MQTT settings in the configuration file determine how Harlow Bindicator connects and communicates with the MQTT broker. For simplicity, the assumption is made that there are no usernames, passwords, or certificates required on the broker. The settings include the broker’s address (hostname), the topic for publishing messages, and a client ID for identification purposes.

Lastly, there is a configuration property that specifies the interval at which the Harlow Bindicator checks the bin collection data. By default, it is set to 60 minutes.

Load Bin Collection Data

Once the necessary information is available, the Harlow Bindicator needs to request to load the bin collection data from the local authority website. To achieve this, Selenium, a popular automation tool, will be used. The process involves creating a WebDriver, providing it with the URL of the local authority site along with the UPRN, and then retrieving the page source using the page_source method of the WebDriver object. However, setting up the WebDriver requires some preliminary steps. In this case, a Chromium browser is used, and the setup process is as follows.

In a terminal run the following:

1
2
3
sudo apt-get update
sudo apt-get install chromium-browser
sudo apt-get install chromium-chromedriver

Running the above will update the distribution lists, then install the Chromium browser and then the correct Chrome driver. To ensure that the above steps have been completed enter the following and check that both chromediver and chromium_browser are present.

1
2
cd /usr/bin
ls -l chrom*

Parsing the Source Code

If the WebDriver and Selenium have done their job correctly, we should now have the source code for the rendered Harlow Local Authority bin collection website, for the given UPRN.

The data we want is held in something that looks like a table. So we need to do some inspecting and work out what we need to tell BeautifulSoup to parse to get the data we want.

Turns out it’s rather straightforward as each entry is a “collectionsrow” class. Then we parse each of those to find the bin type and the collection date.

MQTT Message

Having obtained a series of data, our focus is solely on the first entry, which corresponds to the latest date. To send a message, we invoke the publish function from our MQTT library, which was configured during the setup process. However, we need to provide the function with a message and specify the topic to which the message should be published.

During the parsing step of the data, a collection of data objects was generated to represent the bin collection dates. To facilitate message creation, a method was added to the CollectionDate object, defined as follows:

1
2
3
4
5
6
7
payload = json.dumps({
            "date": self.date.strftime("%d/%m/%Y"),
            "bin_day": self.is_bin_day(),
            "bin_type": self.wheelie.bin_type
        })

return payload

Finally

Now that the Harlow Bindicator has rendered the website, parsed the data, created the object representation, and then published the message of the latest bin collection to MQTT it can now sleep for the configured amount, which by default is 60 minutes.

Once the 60 minutes have passed, it will start again with the rendering of the website.

Node-Red Process

Above is the flow that’s in use to complete the requirement from the Harlow Bindicator which is to send me a notification, in this case an SMS text message. The nodes are as follows.

MQTT Listener

This node is configured to be connected to my network MQTT broker. This node is going to trigger whenever there is a message published to the queue named “bindicator”. It will pass the message from the topic onto the next node which is a switch.

Switch Node

This is a simple node that behaves like an if statement. You can add multiple clauses to it to then execute different paths. In this case, it’s got a single path exiting it. This single path is only executed if the JSON field “bin_day” in the message from “bindicator” is true. This means that the Harlow Bindicator believes that today is bin day and this means that we should send a notification. So if it’s not bin day, the flow terminates here. If the JSON states that today is bin day, then this node hands off to a limiter.

Limiter

Because sending a text message costs money, and no one likes spending money. I’ve placed a limiter between the bin day check and sending a message. This node has been configured to only allow a single message to pass through it a day. We only need a single notification, and the bindicator runs every hour. Therefore this saves us multiple messages and the cost of sending them messages. If this is not the first message of the day, then the flow terminates here, otherwise it hands over to a template node.

Template Node

This node takes a message and just outputs a different message. This message is configured to take the JSON object that’s been passed along the nodes and generate a simple friendly string that just says “Today is bin day for recycling!” for example. Because if the message has got this far, we know it’s bin day so we just populate the message with the bin that needs putting out. Now that we have a message we just need to send the notification. We now pass execution to the final node in the flow.

Twilio Node

This node has a single job, which is to take the message that we have created in the Template Node and send it to my mobile number. I had to create a Twillo account for this to work, as the node needs API keys and authorisation to do so.

What Have I Learned?

So there are a few things that I’ve learned doing this project. Firstly it continues to help me learn Python which I believe is going well. Also to complete this project I had to scrape a website, so I’ve now used Selenium Web driver in Python code, and also some hands-on use of Beautiful Soup to parse the data.

However, the biggest skill I’ve learned from this project is unit testing in Python, so I’ve got the coverage to 100% on this project and had to get to grips with using @patch and mocking in general. It’s very different coming from a Java background using Mockito, but it makes so much sense once the penny drops. But that’s like all languages and frameworks.

The source code is available on Codeberg