• Home
• help4407
• Schedule
• Information resources
• Frequent commands
• Extra reading

Exercise sheet 4 - for the week commencing 20th April 2020

These exercises should be undertaken using bash and the Terminal window application (under macOS, Linux, or Windows WSL).

In the lecture of Week-7 we introduced two new shell programming concepts - shell functions and hereis documents, that enabled us to transform simple CSV textual data into an HTML file suitable for viewing in a web-browser.

---

1. [files required: cssubmit.html, and subtimes.csv]
   Clicking on the cssubmit.html link, above, should display a histogram plot in your web-browser.
   In the lecture of Week-7 we used Google Charts to plot the submission times of students' assignments for our first Assignment. We employed a Column Chart to plot our data (while the knowledge on Google Charts website is very accessible, it does have a steep learning curve, and you'll not need to understand how the charts are rendered by Google's Javascript libraries!)
   For this first exercise you're asked to reproduce the results developed in the lecture material.
   An example of the final HTML file, cssubmit.html appears above, but you'll need (want) to save it to your computer's local disk to view it. It includes the static set of commands/text to request our column chart. Remember that there is static material at both the top and bottom of the cssubmit.html file.
   The middle of the cssubmit.html file presents the data to Google Charts, but it is in a slightly different format to the original data provided in subtimes.csv (in fact, they are slightly different datasets).
   Our task, here, is to reproduce the cssubmit.html file (into a different HTML) using the new shell features of shell functions and hereis documents.
   The suggested approach is to follow these steps:
   1. Examine the data in the file subtimes.csv, noting that it's the familiar CSV format. For each line in the file (each single assignment submission), use a for loop in your shellscript to break the data into its four fields of year, month, day, and hour, and simply print those values out again to ensure that your loop is working.
   2. Next, examine the format required for the data rows of the cssubmit.html file (the middle data rows). Each field appears in the same order, but with some additional square brackets and commas to separate them. Note also, that there's an extra data item at the end of each row - that's the count of how many submissions were made at each specific hour. Determine how to produce this count - it won't always be 1 - and extend your shellscript to correctly report the count.
   3. Now extend your shellscript from one that produces, not just the data rows, but also the 'top' and 'bottom' content of the HTML file. The content doesn't change (unlike the data), so storing it in a fixed text file is a good solution.
   4. If your shellscript is not produce the 'top' material, the data rows, and finally the bottom material, redirect the output of your shellscript to a file, such as new.html, and see if you can open that file in your web-browser.
      Note that the HTML being produced must be in exactly the correct format (other than spaces and tab characters). If your HTML format is not correct your web-browser will offer very little help, and will just display a white webpage. Check your HTML carefully, particularly your use of commas, brackets, and quotation characters. Just one missing item will result in that white webpage!
   5. Next, as seen in the lecture video, repalce your use of static text files, for the 'top' and 'bottom' material, with shell hereis documents. Choose any token (not in the text!), such as THE_END to indicate where your static text finishes. Re-run your shellscript, view the resulting HTML in your browser.
   6. And, finally, add shell functions to your shellscript, to produce the 'top', 'middle', and 'bottom' HTML segments. Perhaps, as in the video, modify your shellscript to accept a different data file on its command-line, or even a different colour for the column chart. Done.

2. Fuel prices in Western Australia are described as following a 'weekly fuel cycle', in which its maximum price is seen on one day of the week, and it then drops, daily, to reach its minimum 6 days later. Then, magically, is jumps to a new maximum, sometimes by as much as 30 cents overnight. Of course fuel companies deny that there is such a regulated cycle, and several Government inquiries have failed to determine why the price fluctuates weekly(!)

   Historically, the maximum fuel price at most petrol stations has been seen on Tuesdays, it steadily drops until the following Monday, and then jumps up again the following day. Of course, wise customers anticipate this cycle, and try to purchase their fuel on Mondays.

   Recently, it has been observed that the lowest price point is now on Tuesdays, not Mondays. Does the historical data support this observation?

   1. Revisit the FuelWatch Historical Data website (used in Exercise Sheet 2) and download their files for all of 2020 (January→April).
   2. Once you have all 4 files, combine them into a single file with the command:
      shell> cat  file1.csv  file2.csv  file3.csv  file4.csv  >  alldata.csv

   3. Now, again for the 'Caltex StarMart Nedlands' petrol station, extract all unleaded petrol (ULP) prices, maintaining the dates of each price.

   4. Now, using a Google Line Chart, plot the ULP price for each of the 110+ days of this year. The day of the year (as just a number) should appear on the X-axis of your plot, and the fuel price on the Y-axis. A single line (from the data rows) should show the variation in price across the year. Note that this is a Line Chart, whereas the first task used a Column Chart, so the 'static top and bottom text' will be different for this task.

3. 🌶 Revisit the use of a Line Chart to now produce two lines on the same plot - such as one for March, and one for April.

   You will find the join command helpful for this exercise. For example, if one file contains:

   1  129
   2  126
   3  125
   

   and a second file contains:

   1  117
   2  115
   3  114
   

   then the command:

   shell> join  file1.csv  file2.csv  >  joineddata.csv

will create a 3-column file, using each file's first column as a common key.

Does your Line Chart show that the lowest price, each week, has shifted from a Monday to a Tuesday?

---

Chris McDonald
April 2020.

This Page

Written by: [email protected]