Creating an Automated Hash Analyzer
Creating an Automated Hash Analyzer
Embedded Files
I recently came across the need to check a website rather periodically for updates. Not wanting to manually check the site every few hours or a couple of times a day I decided to pursue a script to do the work for me. I will walk you through the process of setting up a script that will check a website for updates and send you an email when something changes.
Not having a great deal of python experience I decided to see what simple scripts I could work with to try and build this script. I found a simple script on Geeks for Geeks providing the basics.
Code from Geeks for Geeks on Monitoring site changes
# Importing libraries
import time
import hashlib
from urllib.request import urlopen, Request
# setting the URL you want to monitor
url = Request('https://leetcode.com/',
headers={'User-Agent': 'Mozilla/5.0'})
# to perform a GET request and load the
# content of the website and store it in a var
response = urlopen(url).read()
# to create the initial hash
currentHash = hashlib.sha224(response).hexdigest()
print("running")
time.sleep(10)
while True:
try:
# perform the get request and store it in a var
response = urlopen(url).read()
# create a hash
currentHash = hashlib.sha224(response).hexdigest()
# wait for 30 seconds
time.sleep(30)
# perform the get request
response = urlopen(url).read()
# create a new hash
newHash = hashlib.sha224(response).hexdigest()
# check if new hash is same as the previous hash
if newHash == currentHash:
continue
# if something changed in the hashes
else:
# notify
print("something changed")
# again read the website
response = urlopen(url).read()
# create a hash
currentHash = hashlib.sha224(response).hexdigest()
# wait for 30 seconds
time.sleep(30)
continue
# To handle exceptions
except Exception as e:
print("error")
What is the Script's Process?
What is the Script's Process?
The script analyzes the site that you want to track, it then generates a sha224 hash of the site's current state. It stores that hash, waits 30 seconds then checks the hash again. It proceeds to do this infinitely or until the script is stopped. When information on the site changes, the hash value changes. When the script sees that the stored hash does not match the hash of the site it then prints "something changed".
NOTE: If the site you plan to monitor isn't updating regularly you may not get a good sense if the script is working right. You can use a busy site such as a news site or e-commerce to test as they are always being updated.
Configuring the Script
Configuring the Script
To configure the script you simply need to change the URL within the script (highlighted in bold in the script above). You can certainly customize the script more if you desire. Further down in this article I go over the process of adding another tool to the script.
After getting the script configured and running through testing I realized that I did not want to have to go back and check the terminal to see when something may have been updated. I decided to try and find a way to get the script to send an email to me when it did detect a change. I ended up coming across yagmail. Yagmail is a tool that can interact with a Gmail account in order to send mail.
What should I run this on?
What should I run this on?
When combining and testing the scripts I was using a Linux Ubuntu Virtual Machine. In the end I decided to run this on GCP (Google Cloud Platform) so that I don't have to worry about keeping a computer on all the time.
Image of the script running and relaying that the hash value changed.
Configuring Yagmail
Configuring Yagmail
There will be a couple of things needed when setting up Yagmail.
A Gmail account to send from
An app password for that account
NOTE: As of May 30th, 2022 Google removed the ability to allow "less secure apps" for Gmail account access. The easiest way around this is to enable 2FA and then generate an "app password" This password is a randomly generated password to be used for an application that can't prompt for a 2FA code. I tested this in my environment and had no issues.
Install Yagmail using the following command: pip install yagmail
Once you have obtained all those items you can then use the script provided below to send an email. You simply need to edit the sender's address and receiver's address, then edit the message and subject.
After getting those details in place you can test this in its own python script. This way you can make sure it authenticates properly and is able to send.
Once you have tested both scripts and have them successfully working, you then can work on combining the two scripts to work together.
Yagmail Script
import yagmail
receiver = "receiver@gmail.com"
body = "Looks like something changed on the site. Go check it out!"
yag = yagmail.SMTP("sender@gmail.com")
yag.send(
to=receiver,
subject="Looks like Something Changed...",
contents=body,
)
Combining the Scripts
Combining the Scripts
The next step is to put the script and tool together to get it all working. I copied the yagmail code (shown above) and inserted it into the script right below the print function. This way from the terminal interface I can still monitor and see that it is outputting "something changed". If all is done right, there shouldn't be any errors and you will see emails being sent/received when something on the site changes (again, use an active/busy site to test consistency).
Full Script
# Importing libraries
import time
import hashlib
from urllib.request import urlopen, Request
# setting the URL you want to monitor
url = Request('https://website.com',
headers={'User-Agent': 'Mozilla/5.0'})
# to perform a GET request and load the
# content of the website and store it in a var
response = urlopen(url).read()
# to create the initial hash
currentHash = hashlib.sha224(response).hexdigest()
print("running")
time.sleep(10)
while True:
try:
# perform the get request and store it in a var
response = urlopen(url).read()
# create a hash
currentHash = hashlib.sha224(response).hexdigest()
# wait for 30 seconds
time.sleep(30)
# perform the get request
response = urlopen(url).read()
# create a new hash
newHash = hashlib.sha224(response).hexdigest()
# check if new hash is same as the previous hash
if newHash == currentHash:
continue
# if something changed in the hashes
else:
# notify
print("something changed")
import yagmail
receiver = "receiver@gmail.com"
body = "Looks like something changed on the site. Go check it out!"
yag = yagmail.SMTP("sender@gmail.com")
yag.send(
to=receiver,
subject="Looks like Something Changed...",
contents=body,
)
# again read the website
response = urlopen(url).read()
# create a hash
currentHash = hashlib.sha224(response).hexdigest()
# wait for 30 seconds
time.sleep(30)
continue
# To handle exceptions
except Exception as e:
print("error")
Finalizing the Script
Finalizing the Script
After combining and testing you should have a completed script that will monitor your desired website for changes and then notify you when something does change. As previously mentioned, I did not want to worry about running this script locally and wanted to run it in the cloud. If you want to run it locally you can just use your computer, a VM, or even a raspberry pi. Below I will go over running this in the GCP.
Configuring Google Cloud Platform to host my script
Configuring Google Cloud Platform to host my script
Using GCP to host a script
Using GCP to host a script
I decided to take advantage of what Google has to offer with GCP and spin up a small VM to host this script. Using one of Google's smallest VM instances I was able to set up a VM using Ubuntu.
NOTE: For those that do not know or are interested in learning more about using the cloud/GCP, Google is offering a $300 credit when signing up for the GCP. See more here. Even after using the credit, they do have some free-tier products to use.
Setting up the VM
Setting up the VM
You can start by proceeding to "Marketplace" and searching for Ubuntu.
Marketplace
Ubuntu Options
For my instance I chose Ubuntu 20.04 LTS (Focal)
Once you select your version of Ubuntu you can select "Launch" and customize the VM Settings.
These options should be adequate to run the VM
Once that is done processing you can use the "SSH" option to gain access to the machine.
Preparing for the Script
Preparing for the Script
From there you should be all set to run your script. You will need to install a few items such as...
Python 3 - Sudo apt-get install Python3
Install Pip by running - Sudo apt install python3-pip
You may need to update your version of Linux for Pip to install - Sudo apt-get update & Sudo apt-get upgrade
Yagmail Module - pip3 install yagmail
Also to note, you may have issues with keyring when trying to store your Gmail password. I ended up needing to use the command pip3 install keyrings.alt This allowed the keyring tool to function as normal. I didn't have any issues after installing this.
Side note: I noticed when using this that the script doesn't want to stay running all the time when you leave the SSH session. Taking advantage of a tool like tmux was able to keep it running despite leaving the session. You can read more on tmux here.
After all of that, you should have everything you need to run a successful automated hash analyzer and alert script. Have something that I can improve on in this? I am always open to learning more and improving. Visit my "Contact Page" and let me know. Thanks!
Google Sites
Report abuse