Last edited by greenbreakfast, 2016-06-13 21:52:47
The main goal of this project is to demonstrate the Omega's capabilities in a practical webscraping application. We will be making a stock ticker using our Omega to retrieve prices and the OLED Expansion to display them.
We will be using python to do our webscraping as it provides us with a convenient set of tools to retrieve and parse web data as well as a shell script to print our stock information onto the OLED display.
The Expansion Dock
Using the OLED Expansion
Using Shell Scripts
Scheduling with Cron
Stock Tickers are used display the most recent selling price for a stock on the market. Historically, they have been the most fundamental tool used by "wall streeters" for tracking performance. There are plethora of free stock ticker services for your computer, phone, tablet ...etc. However, we will provide you with the back bone to make a highly customizable stock ticker on a dedicated mini-computer (Omega) and display(OLED Exp) in under 15 min!
Pre-requisite: If you haven't already follow the get started guide to setup your Omega.
Firstly, connect your OMEGA and your OLED Expansion to your Expansion Dock. Refer to the picture below.
You will need to setup wifi so that we can install python and fetch stock data from google. Go ahead and enter the wifisetup command into your omega.
Follow the instructions and give the Omega access to your wifi network.
We will need to install the full version of Python so that we have access to some libraries necessary for webscraping. If you have already installed python skip to the next step. To do this simply enter,
opkg update opkg install python
You may or may not need to clear some space on your Omega for the installation. If you are doing this right out of the box, you should be fine.
Navigate to your "/" directory. Create your shell script file by executing the following command.
cat > stock.sh
You will be prompted to enter the contents of the file. Copy the following code into the file.
#!/bin/sh oled-exp -c echo $1 > /root/ticker.txt VAR=$(python ./stock_script.py) oled-exp -i oled-exp write $VAR
You can save and exit the command by entering CTRL + D. Firstly, we add the the shell to our path so that we can run the file as an executeable. We then clear the display using oled-exp -c. We write the ticker of interest to the following file
/root/ticker.txt. Next we run our python script and store the output to our string variable, VAR. Then we initialize the display and write the value of VAR, which is our ticker and price.
Now lets do the same with our python script.
cat > stock_script.py
#!/usr/bin/env python import urllib import json myfile = open('/root/ticker.txt', 'r') rg=myfile.read() site="http://www.google.com/finance/info?q="+rg jfile=urllib.urlopen(site) jsfile=jfile.read() jsfile=jsfile.replace("\n","") jsfile=jsfile.replace("/","") jsfile=jsfile.replace("]","") jsfile=jsfile.replace("[","") a=json.loads(jsfile) ticker=a['t'] price=a['l_fix'] info=ticker+":"+price print info
To start we added python to our environment so that it is executable from the command line or a shell script. Next we imported urllib and json libraries, these will allows us to webscrape information and parse the data for our desired information respectively. Next we open the file that stores our desired ticker and store it as a string variable. Then we append that to our URL. Then we will generate the URL and open the file from the web, urllib.urllopen(site). We will store it as a string using jfile.read(). Next we will clean it up using the replace functions so that it can be parsed by the json library. We use the json.load(jsfile) to create a dictionary. Then we retrieve the values of interest using the keys for ticker and price. We append the two and spit the output to our shell script.
Make both files we just created executable by entering this:
chmod +x stock.sh stock_script.py
Now let's see how Apple is doing.
Parts of this project can and are encouraged to be used in other projects.
To take the project one step further, let's create a stream by having the Omega run the same script once every minute. To do this we will use the cron tool. cron allows us to schedule jobs(commands/programs/script) for execution at a fixed time. First lets schedule our sript to run every minute. To do this, enter the command:
You will be taken to the vi-editor of the scheduling file which can be found in /etc/crontabs/root. Add the following line to your file and make sure to end the file with an empty line or #.
*/1 * * * * /stock.sh AAPL
You can save and quit the editor by entering ZZ.
Using this project as a template, we can go even further with webscraping projects. Maybe build a new headline streamer or an upto date weather tracker.
If you enjoyed this project, you should check this out OLED-QR-Code-Generator.