~~TOC 1-2~~ |<100% 25% - >| ^ \\ DATA ANALYTICS REFERENCE DOCUMENT\\ \\ ^^ ^ Document Title:|Document Title| ^ Document No.:|1569148014| ^ Author(s):|Gerhard van der Linde, Rita Raher| ^ Contributor(s):| | **REVISION HISTORY** |< 100% 10% - - 10% 17% 10% >| ^ \\ Revision\\ \\ ^\\ Details of Modification(s)^\\ Reason for modification^ \\ Date ^ \\ By ^ | [[:doku.php?id=modules:52957&do=revisions|0]] |Draft release|Document description here| 2019/09/22 10:26 | Gerhard van der Linde | ---- ====== 52957 - Data Representation ====== ====== Week 1 - Introduction ====== The syllabus has 4 learning outcomes - Compare data models and architecture used in applications - Write software applications that adhere to common standards and protocols. - Explain the basic mechanisms by which application data is transmitted across the internet. - Design and Utilise application programming interfaces for interacting with data sources. Which in practices means: * XLM and JSON * How the data is transferred ie HTTP and Restful APIs * Being able to create a web-application ==== Indicative Schedule ==== {{:modules:screenshot_2019-09-26_at_07.51.12.png?400|}} * 40% - ongoing assessments * 60% - Project ====== Week 2 - XML and DOM ====== xml -> eXtensible Markup Language * **Extensible** - Designed to accommodate change * **Markup** - Annotates text * **Language** - Set of rules for communication Python ProgrammingPrentice Hall Peter van der Linden * XML looks like HTML, but it is different. * XML was designed to carry/represent data - with focus on what data is * XML tags are not predefined like HTML tags are * HTML was designed for browser to use to display - It has predefined tags * XML's purpose is to represent information in text form * XML does not do anything (either does HTML) * There are no pre-defined tag names - you make them up yourself * XML has a tree-like syntax. * The document Object Model (DOM) can be applied to XML. Text Text < is an example of encoding {{:modules:screenshot_2019-09-26_at_08.32.16.png?600|}} | Declaration | XML documents should have a single line at the start stating that it's xml, the versionof xml it is and an encoding| | Root Element | XML must have a single root element that wraps all others | |Elements | XML is structured as elements, which are enclosed in angle brackets. Elements must have a closing tag.| | Tag| indicate the start and the end of each element(end tag indicated with )| | Attributes | Elements can have attributes, which are name-value pairs within the angle brackets. A given attribute name can only be specified once per element. | | Entity References | Certain characters must be escaped with entity references, e.g < for<. | | case sensitive | Everything in XML is case sensitive. (HTML is not)| | Plain text | Or Data, can be inside the tags | ==== Exercise ==== * Create an xml file that stores information for a breakfast menu. * It should contain food items, and the price, description and calories for each item. Belgian waffles $5.95 Two of our famous Belgian Waffles with syrup 650 Strawberry Belgian waffles $7.95 Light Belgian waffles covered with strawberries and whipped cream 900 ===== DOM ===== * The Document Object Model (DOM) is how programs store xml internally * It provides a model of the XML as a structured group of nodes that have three properties - Type (element, tree, comment, atrribute) - Name(tag name or attribute name) - Value (value of the text node, comment or attribute value) * Programs can use the DOM to manipulate the XML(or HTML) * E.g in JavaScript we can use commands like: * document.createElement * document.createTextNode and * document.element.appendChild to add to the DOM * document.getElementByID to access elements to the DOM {{:modules:screenshot_2019-09-26_at_08.50.38.png?600|}} ==== HTML ==== * Using W3Schools to learn html * The page tags ''%%, ,%%'' * The attribute id and class (for later) * The ''%%
%%'' tag * The table tags ''%%, , %%'' is the ''%% %%'' by: var rowElement = buttonElement.parentNode.parentNode //or Var rowElement = buttonElement.closest('tr') {{:modules:screenshot_2019-10-04_at_20.04.32.png?600|}} Exercise 2.9 Write the code so that when the user clicks on the button in a row, the contents of the cells in that row are stored in an object and outputted to the console.         
, %%'' * The form tags ''%%
, , * Document getElementById
document.getElementById("messageOut").innerText = message **Exercise 2.3** Make a webpage with an input and a button, when the user clicks the button then the contents of the input will display in another div. e2.2

output will go here
===== JavaScript 2===== ==== Variables ==== * Define a variable with the var keyword, eg var age =21 * Variable types are defined by the value, they can be * Integre * Float * String * Boolean * Null * Undefined * Object * Array * function var i = 12 var fl = 3.3 var firstName = "andrew" var lastName = "O'Neill" //or 'O\'Neill' var admin = true var foo = null var dunno //= undefined var car = {} var books = [] var fun = function(){console.log('in fun')} ==== Objects ==== * Properties can be added to objects car.reg = '12 mo 1234' car.make = "ford" * Or defined using JSON var employee = { name:'joe', role:"Manager"} * Use JSON.stringfy() to output console.log("car" +JSON.stringify(car)) **Exercise 2.4** Create a book object as a variable with Title, author and ISBN and display it in the console 2.4 ==== Manipulate Display ==== * We can make am element visible and hidden document.getElementById("div1").style.display = "block" document.getElementById("div2").style.display = "none" * or we can disable a button document.getElementById("button1").disabled = true **Exercise 2.5:** Make a webpage that has two div, only one should be visible at a time, each ''%%
%%'' will have a button that will make the other ''%%
%%'' visible (and hide itself) 2.4

some text
==== Control ==== if statement if(condition){ do stuff }else { do other stuff } if(value < 10){ outputElement.innerText = "too low" }else if(value >20){ }else{ outputElement.innerText ="goo enough" } **Exercise 2.6:** Make a web page that has a text box, when the content of the text area changes the page will display whether the value entered is greater or less than 10 or equal to 10                  
                     output here         
              ==== For loop ==== * Simple count for(var i =0;i<10;i++){ console.log(i); } * Iterate an array var names=['Joe', 'Mary', 'Fred'] * Iterate an object var book ={title:'harry Potter and something',author:'JKR', isbn:"12345"} for(propName in book){ bookoutput += propName+'='+book[propName]+'
' }
**Exercise 2.7:** * Make a webpage that outputs 1 to 10(or N) e2.7 * Make a webpage that outputs all the values of an array e2.8
output here
==== DOM ==== * Get Element By id, we have seen this already * QuerySelector: The querySelector function searches all the child nodes of a particular element for nodes that match the query, and returns the first one (it is like CSS selectors and JQuery selectors, see later). * Name name of tag * #id a node with id=“id” * .classname A node with class=“classname” * [atName=“atVal”] A node with the attribute atName=“atValue” * More data at https://www.w3schools.com/cssref/css_selectors.asp **Exercise 2.8:** Create a webpage with multiple inputs and a button, when the user hits the button read all the inputs, store the values into a class and output the class to the console                          First name: 
            Last name: 
            age: 
            go         
              ==== Child Nodes ==== * Consider
data1 data2
To get the content of the first cell, you need to get the child of that cell and its text value rowElement.cells[0].firstChild.textContent {{:modules:screenshot_2019-10-04_at_19.58.43.png?600|}} ==== Parent Nodes ==== * Similarly we can look up the DOM Tree * Consider the html in the previous slide * What is the relationship between the row element and the button? * ''%%
%%'' is the parent and ''%%
%%'' so we can access the ''%%
                                                                                     
how nowbrown cowoutput
                   ==== Add to DOM Tree ==== * Add element var para = document.createElement("p"); var node = document.createTextNode("This is new"); para.appendChild(node); * Add row var rowElement =table.Element.insertRow(-1) * Add cell var cell = rowElement.insertCell(0); * Add attribute rowElement.setAttribute("id", car.reg) **Exercise 2.9.2** Create a button that when it is clicked adds a row to the table(above) with data function createRow(){             var tableElement = document.getElementById('mainTable')             var data = {cell1:'hi',cell2:'bye',cell3:'thanks for the fish'}             var rowElement = tableElement.insertRow(-1)             var cellElement = rowElement.insertCell(0)             cellElement.innerHTML = data.cell1;             cellElement = rowElement.insertCell(1)             cellElement.innerHTML = data.cell2;             cellElement = rowElement.insertCell(2)             cellElement.innerHTML = data.cell3;                      } ====== Week 4 - HTTP and Web Scraping ====== What is HTTP? * HTTP: HyperText Transfer Protocol * HyperText Text with Links * Transfer communication of data * Protocol set of rules for communication * HTTP Versions * HTTP/1.0 first version - * HTTP/1.1 - * HTTP/2.0 use is growing (current version) * Browsers use HTTP {{:modules:screenshot_2019-10-10_at_19.35.17.png?400|}} ===== HTTP Request ===== {{:modules:screenshot_2019-10-10_at_19.36.39.png?400|}} ===== HTTP Response ===== {{:modules:screenshot_2019-10-10_at_19.38.01.png?600|}} ===== URL: Universal Resource Locator ===== * Resources can be anything: text, images, pages, files or videos. * You see URLs at the top of browsers * E,g http://gmit.ie * ftp://ftp.domain.com ===== Parts of URL ===== http://gmit.ie {{:modules:screenshot_2019-10-10_at_19.42.09.png?600|}} http://learnonline.gmit.ie/course/view.php?id=1318&blah=8 {{:modules:screenshot_2019-10-10_at_19.43.23.png?600|}} ===== Encoding ===== * Special Characters need to be encoded (eg space ?:/& etc) * Incorrect http://ps.com/martin Kenirons * Correct http://ps.com/martin%20Kenirons {{:modules:screenshot_2019-10-10_at_19.50.29.png?600|}} ===== Further Reading ===== * A more detailed look at http http://www.jmarshall.com/easy/http/ * Interview with Tim Berners Less (inventor of the Web) http://www.youtube.com/watch?v=TkOpzbTsDJE ====== Web Scraping Part 1 ====== ===== This lecture ===== - Reading web pages with Request - Reading last weeks html file - Using BeautifulSoup4 to go through the html - Searching down (and up) the DOM file - Saving Content in a csv file - The dark art of web scraping ===== Request Module ===== Request module import requests page = requests.get("http://dataquestio.github.io/web-scraping-pages/simple.html") print(page) print("------------") print(page.content) ===== Use BeautifulSoup to prettify ===== import requests from bs4 import BeautifulSoup page = requests.get("http://dataquestio.github.io/web-scraping-pages/simple.html") print(page) print("------------") print(page.content) soup1 = BeautifulSoup(page.content, 'html.parser') print(soup1.prettify()) ===== Parsers ===== {{:modules:screenshot_2019-10-10_at_20.04.44.png?400|}} ===== Read in from a file ===== * I assume that you have covered reading from files before. * This just opens a file and sends its contents to BeautifulSoup * BeautifulSoup parses the file and stores a DOM tree in memory from bs4 import BeautifulSoup with open("../week02/carview2.html") as fp: soup = BeautifulSoup(fp, 'html.parser') print(soup.prettify()) ===== To search the DOM Tree ===== * For any element .tagname will find the first occurance of it * You can use find() this will allow you to find the first element by id and/or class aElement = someElement.find(id="someID") aElement = someElement.find(class="aClass") aElement = someElement.find(div, class="aClass") * You can use findAll to return a list of elements that match the search criteria listOfElement = someElement.findAll(tr, class="aClass") ===== Navigate the DOM tree ===== * .children * .parent * .parents(a list of parents) from bs4 import BeautifulSoup with open("lab02-JS/carviewer2.html") as fp: soup = BeautifulSoup(fp, 'html.parser') print(soup.tr) from bs4 import BeautifulSoup with open("lab02-JS/carviewer2.html") as fp: soup = BeautifulSoup(fp, 'html.parser') rows = soup.findAll("tr") for row in rows: print(row) from bs4 import BeautifulSoup with open("lab02-JS/carviewer2.html") as fp: soup = BeautifulSoup(fp, 'html.parser') #print(soup.prettify()) #print(soup.tr) rows = soup.findAll("tr") for row in rows: #print(row) datalist = [] cols = row.findAll("td") for col in cols: datalist.append(col.text) print(datalist) ===== Using CSV Package ===== * Write to a csv(comma delimiter file) * You can read a csv with Excel * We could create an excel file in python, maybe later import csv employee_file = open('employee_file.csv', mode='w') employee_writer = csv.writer(employee_file, delimiter=',', quotechar='"', quoting=csv.QUOTE_MINIMAL) employee_writer.writerow(['John Smith', 'Accounting', 'November']) employee_writer.writerow(['Erica Meyers, what', 'IT', 'March']) employee_file.close() ====== Web Sraping part 2 ====== ===== Finding what you want from a web page ===== * This is a dark art * Web pages are not designed to be read like this * If a webapplication wants you to be able to extract data they use JSON (they used to use XML in RSS feeds). * Case Study: Get the price and address from myhome.ie - We will need the URL - Don’t use inspect because a lot of webpages use javascript to modify themselves, we need to use viewsource or curl - This is a dark art (yep I am saying it again) - ID and class are what you are looking for, possibly tag ===== HTTP and URL's ===== https://learnonline.gmit.ie/course/view.php?id=1318&blah=8 https://andrewbeattycourseware:pass@github.com/andrewbeattycourseware/dataRepresentation.git You can also have a username:password pair before a @ if you need to log into the host There are other authentication methods (EG Oauth) import requests import csv from bs4 import BeautifulSoup url = "https://www.myhome.ie/residential/mayo/property-for-sale?page=1" # load the page from the above url in the object page. page = requests.get(url) #parse only the html from page.content soup = BeautifulSoup(page.content, 'html.parser') #open a file and csv writer objects home_file = open('week03MyHome.csv', mode='w', newline='\n', encoding='UTF-8') home_writer = csv.writer(home_file, delimiter=',', quotechar='"', quoting=csv.QUOTE_MINIMAL) #parse the listings from soup using the class label listings = soup.findAll("div", class_="PropertyListingCard" ) #loop trhough the listings for listing in listings: #create an empty list for writing out to csv entryList = [] #parse and append price to listing price = listing.find(class_="PropertyListingCard__Price").text.strip() entryList.append(price) #parse and append address to listing address = listing.find(class_="PropertyListingCard__Address").text.strip() entryList.append(address) #write parsed data to the csv file home_writer.writerow(entryList) #close the file home_file.close() **Note:** Note the specification for '''UTF-8''' encoding and ''strip()'' commands to clean the parsed data and specify the newline characters "€625,000","Curraghmore, Belcarra, Castlebar, Co Mayo" "Reserve Not To Exceed €80,000","April Cottage (Folio MY51001F), Balloughadalla, Killala, Co. Mayo" "AMV €30,000","C. 4.09 Acres of Land, Dooagh, Achill, Mayo" "€140,000","15 Riverside Drive, Ballina, Mayo" "Reserve Not To Exceed €110,000","Riverbank House, Bachelors Walk, Abbeyhalfquarter, Ballina, Co. Mayo" "€115,000","Croghan, Killala, Mayo" "€195,000","Carrowbaun, Westport, Co Mayo, F28K298" "€235,000","3 Rosmore, Newport, Co Mayo, F28 VF38" "AMV €245,000 -€30,000 on 5th Oct 19","ONLINE AUCTION Camcloon More, Newport, Mayo" "€250,000","Roonith, Louisburgh, Mayo" "€125,000","8 Rathkelly , Ballinrobe, Mayo" "€179,500","5 Ballyhaunis Road, Knock, Co. Mayo" "€175,000","1 Cinnamon Wharf, The Quay, Westport, Co Mayo, F28 K6W8" "Reserve Not To Exceed €15,000","Ballymunnelly (Folio MY14396F), Bellacorick, Co. Mayo" "Reserve Not To Exceed €130,000","Loughrusheen, Castlebar, Co. Mayo" "€198,000","13 Church Manor, Ballina, Mayo" "€110,000","Cappaghduff East, Tourmakeady, Mayo" "€215,000","12 Rushbrook, Claremorris, Mayo" "€120,000","Knockfin, Westport, Mayo" "Reserve Not To Exceed €40,000","5A and 5B, Main Street, Ballyhaunis, Co. Mayo" "€45,000","Main Street, Ballycastle, Mayo" ===== References ===== * A very good tutorial, takes it from html and builds to making a python program to read it. Be warned he is using python2 and we are using python3, the main difference is with the print statements. https://www.dataquest.io/blog/web-scraping-tutorial-python/ * Documentation on BeautifulSoup4 https://buildmedia.readthedocs.org/media/pdf/beautiful-soup-4/latest/beautiful-soup-4.pdf ====== Week5 - JSON, AJAX and REST ====== ===== HTTP 2: methods and Response Codes ===== ==== Request Methods ==== ^ Method ^ Description ^ In RESTful API ^ | GET | Retrieves a Resource(s) | Read | | POST | Updates/creates/changes a Resource | Create | | PUT | Used more to update a Resourse | Update | | DELETE | Remove a resourse | Delete | | HEAD | Retrieves header info | | | TRACE | Used for Debugging | | | OPTIONS | Retrieve allowable options | | | PATCH | Partial resource modification | | | CONNECT | Set up a tunnel for other traffic to pass through HTTP| | ==== HTTP Methods GET vs POST ==== ^ ^GET ^POST^ | Security |Parameters are visible to the user and browser History. |POST is a little safer than GET because the parameters are not stored in browser history or in web server logs | | |Never use GET when sending passwords or other sensitive information! | | | Back Button |harmless |Data will be resubmitted | | Bookmarked |Can be bookmarked |Parameters are not bookmarked | | Cached |Can be cached |Not cached | | Encoding type |application/x-www-form-urlencoded |application/x-www-form-urlencoded or multipart/form-data. Use multipart encoding for binary data | | History |Parameters remain in browser history |Parameters are not saved in browser history | | Visibility |Data is visible to everyone in the URL |Data is not displayed in the URL | | Restrictions on data length |data length |No Restrictions | | |Yes, when sending data, the GET method adds the data to the URL; and the length of a URL is limited (maximum URL length is 2048 characters) | | | Restrictions on data type |Only ASCII characters allowed |No restrictions. Binary data is also allowed | ==== HTTP Response ==== Requests and responses both have this format: • Intial line. * Zero or more header lines. * A blank line. * Optional message body (e.g. a HTML file) HTTP/1.1 200 OK Date: Mon, 27 Jul 2009 12:28:53 GMT Server: Apache/2.2.14 (Win32) Last-Modified: Wed, 22 Jul 2009 19:15:56 GMT Content-Length: 88 Content-Type: text/html Connection: Closed

Hello, World!

==== Status codes ==== ^ Code Range ^ Status ^ | 100 – 199 | Informational| | 200 – 299 | Successful | | 300 – 399 | Redirection | | 400 – 499 | Client Error | | 500 – 599 | Server Error | ^ Code ^ text ^ description ^ | 200 | OK | Successful | | 201 | Created | Resource created | | 204 | No Content | The requested resource is empty | | 400 | Bad Request | Bad Syntax | | 401 | Unauthorised | Client might need to be authenticated | | 403 | Forbidden | Operation not allowed | | 404 | Not Found | Resource does not exists | | 500 | Internal Server Error| Something went wrong during processing| More at [[https://www.restapitutorial.com/httpstatuscodes.html]] ==== Summary ==== * HTTP request have methods * HTTP responses have status codes \\ (These are used in restful APIs) ===== JSON ===== * Javascript Object Notation * JavaScript - A scripting/programming language. * Object - Groups of name-value pairs. * Notation - Set of rules for representing objects. * Human readable. * Is just text. * Open standard. * Data interchange format. { "employees": [ {"firstName":"John", "lastName":"Doe"}, {"firstName":"Anna", "lastName":"Smith"}, {"firstName":"Peter", "lastName":"Jones"} ] } // Turning text into a JavaScript object. var obj = JSON.parse(text); // obj is an object. // Turning a JavaScript object into text. var text = JSON.stringify(obj); // text is a string. ==== Syntax ==== * Name/Value pairs separated by a colon. ''"name": "Martin"'' * Objects identified by curly braces. ''{ }'' * Lists identified by square brackets. ''[]'' * All strings (names if space in it) use double quotes (not single). ''"Martin"'' === Examples === Numbers - 123.456 Boolean - true Objects - {"name": "Ian"\} Functions - function(){//commands} Strings - "Hello, world!" Arrays - [1,2,3] null - null ==== Consider ==== "employees":[ {"firstName":"John", "lastName":"Doe"}, {"firstName":"Anna", "lastName":"Smith"}, {"firstName":"Peter", "lastName":"Jones"} ] Exercise (not an assignment!): - Add two New employees to the above JSON? - Extend the JSON to include address? - Extend the JSON object to include age? ==== JSON in the wild ==== {     "time": {         "updated": "Oct 16, 2019 15:03:00 UTC",      },     "disclaimer": "This data was produced from the CoinDesk Bitcoin Price Index (USD). ",     "chartName": "Bitcoin",     "bpi": {         "USD": {             "code": "USD",             "symbol": "$",             "rate": "7,968.2517",             "description": "United States Dollar",             "rate_float": 7968.2517         },              "EUR": {             "code": "EUR",             "symbol": "€",             "rate": "7,209.3474",             "description": "Euro",             "rate_float": 7209.3474         }     } } **Look at:** https://api.coindesk.com/v1/bpi/currentprice.json ===== AJAX (with Jquery) ===== ^ Asynchronous: |Does not block code, the return function gets called when complete| ^ Javascript: |The language| ^ And XML: |Well not much anymore, usually JSON or html is used| More info at https://www.w3schools.com/xml/ajax_intro.asp We will use AJAX with Jquery not the plain JavaScript version: https://www.w3schools.com/jquery/jquery_ajax_intro.asp ===== What does AJAX do? ===== * Allows us to call up a resource with HTTP from inside a web page {{:modules:picture1.gif?nolink|}} ===== JQuery ===== * We will be using JQuery for AJAX, * JQuery is a javascript library that make finding and manipulating elements in the HTML DOM tree easier. * I am not going into detail about JQuery now * You will need this magic line of code in the header of any page that uses Jquery ===== $.AJAX function ===== * We will use the %%$%%.AJAX function, it takes on parameter, a JSON Object that describes everything. * ''%%$%%.ajax({the object that describes everything});'' $.ajax({ "url": "https://api.coindesk.com/v1/bpi/currentprice.json ",      "method":"GET",      "data":"",      "dataType": "JSON",      "success":function(result){       console.log(result);       },       "error":function(xhr,status,error){           console.log("error: "+status+" msg:"+error);       } } ) ^ url: |the url to call| ^ method: |the HTTP method| ^ data: |parameters (none in this case)| ^ datatype: |that is returned JSON or HTML or XML| ^ success: |the function to call if this is a success (with the result from server)| ^ error: |The function to call if we don’t get a status code in the 200s| ===== Putting the code together =====                                    read a simple json                            go           ===== Read the euro price from the JSON ===== Looking at the JSON you can see the value we are looking for is in the object: ''bpi.EUR.rate'' {     "time": {         "updated": "Oct 16, 2019 15:43:00 UTC",         "updatedISO": "2019-10-16T15:43:00+00:00",         "updateduk": "Oct 16, 2019 at 16:43 BST"     },     "disclaimer": "This data was produced from the CoinDesk Bitcoin Price Index (USD). Non-USD currency data converted using hourly conversion rate from openexchangerates.org",     "chartName": "Bitcoin",     "bpi": {         "USD": {             "code": "USD",             "symbol": "$",             "rate": "7,979.8633",             "description": "United States Dollar",             "rate_float": 7979.8633         },         "EUR": {             "code": "EUR",             "symbol": "€",             "rate": "7,220.6990",             "description": "Euro",             "rate_float": 7220.699         }     } } So the code to extract it is: var rate = result.bpi.EUR.rate console.log(rate); //document.getElementById("output").innerText = rate; ===== RESTful API ===== ==== API ==== * Application * Programming * Interface I.e. An interface that a program that you write can use to transfer data to another program ==== SOAP ====        Apples    ==== RESTful ==== * **REST** stands for **Re**presentational **S**tate **T**ransfer. * REST is an architecture describing how we might use HTTP. * RESTful APIs make use of more HTTP methods than just GET and POST. * Most HTTP APIs are not RESTful. * RESTful APIs adhere to a few loosely defined constraints. * Two of those constraints are that the API is stateless and cacheable. * Idempotency  * More at https://www.restapitutorial.com/ ==== REST Example ==== ^ Method ^URL ^Description ^Sample return^ | GET |/emails |Get all the Emails |[\\ {id:32,from:”joe@joe.ie”,message:”hi”},\\ {id:33,from:”me@gmail.ie”,message:”i”},\\ {id:35,from:”a@b.com”,message:”blah”}\\ ] | | POST |/emails |Create an Email |{created:true} | | GET |/emails/32 |Retrieve the Email with id 32 |{id:32,from:”joe@joe.ie”,message:”hi”} | | PUT |/emails/32 |Update the Email with id 32 |{id:32,from:”joe@joe.ie”,message:”bye”} | | DELETE |/emails/32 |Delete the Email with id 32 |{deleted:true} | ==== Stateless ==== * Statelessness is a REST constraint. * HTTP uses the client-server model. * The server should treat each request as a single, independent transaction. * No client state should be stored on the server. * Each request must contain all of the information to perform the request. ==== Cachable ==== * REST APIs should provide responses that are cacheable. * Intermediaries between the client and server should be able to cache responses. * This should be transparent to the client. * Cacheability increases response time. * Browsers usually cache resources, in case they are requested again. * There is usually a time limit on cached resources. ==== Site that use RESTful APIs ==== * Facebook, google, stripe, CSO etc * More at https://any-api.com/ * Python has a lot of packages that handle these APIs (more on week06) ==== Summary ==== * REST is a loose set of constraints/guidelines used for making an API * Stateless * Cacheable * Use HTTP methods * I will be going through this in more detail next week * We will create a web page that consumes a RESTful API I have created already for a database (CRUD) ===== Labs ===== ==== Lab01-simpleJSON.html ==== simple json ==== Lab02-json-array.html ==== lab02 ==== Lab03-ajax-readsimple.html ==== read a simple json
==== LAB04-extra.html ==== read etherium
====== Week 6 - AJAX and REST ====== ===== What is CURL ===== * Client-side URL * Used to call a URL and retrieve its response * Handy for testing and debugging * (I have used it to webscrape, “back in the day”) * curl http://www.google.com ==== Installation ==== * Come built in to Windows 10 and mac * Other wise download the zip from https://curl.haxx.se/download.html * Unzip it and save the curl.exe and any .dll file to a directory in your PATH variable ==== Some options ==== |<100% 14% 21% - >| ^Option^Description^Example^ | -i |see the response header|curl -i %%http://dummy.restapiexample.com/api/v1/employee/2201%%| | -X |set the method|curl -i -X DELETE %%http://dummy.restapiexample.com/api/v1/delete/2201%%| | -d |set the data to be uploaded|curl -i -H "Content-Type:application/json" -X PUT -d '{"name":"test1","salary":"1123","age":"23"}' %%http://dummy.restapiexample.com/api/v1/update/2201%%| | -H
|set the header| :::| [[https://www.google.ie/search?source=hp&ei=es-yXcrDF6uDjLsPqPO_2Ac&q=sample+rest+api+for+testing&oq=sample+rest+api+for+&gs_l=psy-ab.3.0.0l10.1984.8859..11491...2.0..0.290.2008.19j2j1......0....1..gws-wiz.......0i131j0i10j0i13i30j0i8i13i30j0i8i13i10i30j33i160.Ktpr0YJ33NU|Sample Rest API for testing]] ==== Conclusion ==== * CURL is handy for testing * We will use in in the next lecture to test the server we are running ===== FlasK - introduction to app server ===== * What is an app-server? * And what is a web-server? {{:modules:52957:flask_1.jpg?nolink|}} ==== Simple FLASK ==== {{:modules:52957:flask_2.jpg?nolink|}} #!flask/bin/python from flask import Flask   @app.route('/') def index():     return "Hello, World!"   if __name__ == '__main__' :     app.run(debug= True)   ==== Server static content ==== {{:modules:52957:flask_3.jpg?nolink|}} #!flask/bin/python from flask import Flask   app = Flask(__name__,             static_url_path='',              static_folder='../') @app.route('/') def index():     return "Hello, World!"   if __name__ == '__main__' :     app.run(debug= True)   ==== Get REST server working ==== * Copy the code and run it * https://github.com/andrewbeattycourseware/dataRepresentation/blob/master/code/week05-REST/server/b_restserver.py ==== Looking at the code ==== cars = [     {         "reg":"181 G 1234",         "make":"Ford",         "model":"Modeo",         "price":18000     }    ] * Make an array (list) for storing the cars, * These are stored in memory, so as soon at the program ends the data is lost @app.route('/cars', methods=['GET']) def get_cars():     return jsonify( {'cars':cars}) * url map for /cars for method GET (only) * Returns the list converted in JSON {{:modules:52957:flask_4.jpg?nolink|}} @app.route('/cars', methods=['POST']) def create_car():     if not request.json:         abort(400)     if not 'reg' in request.json:         abort(400)     car={         "reg":  request.json['reg'],         "make": request.json['make'],         "model":request.json['model'],         "price":request.json['price']     }     cars.append(car)     return jsonify( {'car':car }),201 * Method is POST * Check that the request has JSON data (if not return a 400 error) * Read the request object and create a new car * Append it to the list of cars * Return what we just added @app.route('/cars/', methods =['PUT']) def update_car(reg):     foundCars=list(filter(lambda t : t['reg'] ==reg, cars))     if len(foundCars) == 0:         abort(404)     if not request.json:         abort(400)     if 'make' in request.json and type(request.json['make']) != str:         abort(400)     if 'model' in request.json and type(request.json['model']) is not str:         abort(400)     if 'price' in request.json and type(request.json['price']) is not int:         abort(400)     foundCars[0]['make']  = request.json.get('make', foundCars[0]['make'])     foundCars[0]['model'] =request.json.get('model', foundCars[0]['model'])     foundCars[0]['price'] =request.json.get('price', foundCars[0]['price'])     return jsonify( {'car':foundCars[0]}) * This is a put and it takes in the reg from the URL. * Use the the filter to find the car. * Check that the JSON in the request is properly formatted. * Update the car using the data from the JSON, if no matching attribute make the attribute equal to what it was before. * Return the updated car @app.route('/cars/', methods =['DELETE']) def delete_car(reg):     foundCars = list(filter (lambda t : t['reg'] == reg, cars))     if len(foundCars) == 0:         abort(404)     cars.remove(foundCars[0])     return  jsonify( { 'result':True }) * Similar to the previous methods * Remove the found car from the list of cars * Return JSON { “result” : ”true”} @app.errorhandler(404) def not_found404(error):     return make_response( jsonify( {'error':'Not found' }), 404) @app.errorhandler(400) def not_found400(error):     return make_response( jsonify( {'error':'Bad Request' }), 400) * Two functions that will handle the errors 404 and 400 * They return simple JSON ==== Conclusion ==== * I will get you to write your own flask server in week08. * For now I just want you to have an idea what is in it. ===== Calling server with AJAX ===== {{:modules:52957:ajax_1.jpg?nolink|}} ==== Test get all ====                                    test get all                            getAll         
                                      regmakemodelprice                                 ==== Test Create==== {{:modules:52957:ajax_2.jpg?nolink|}} function createCar(){             var car = {"reg":"12 D 1234","make":"Fiat","model":"Punto","price":3000}             console.log(JSON.stringify(car));             $.ajax({                 "url": "http://127.0.0.1:5000/cars",                 "method":"POST",                 "data":JSON.stringify(car),                 "dataType": "JSON",                 contentType: "application/json; charset=utf-8",                 "success":function(result){                     console.log(result);                     document.getElementById("output").innerText = JSON.stringify(result);                     },                 "error":function(xhr,status,error){                     console.log("error: "+status+" msg:"+error);                 }             });           } ==== Test Update==== {{:modules:52957:ajax_3.jpg?nolink|}}  function updateCar(){             var car = {"reg":"181 G 1234","make":"Ford","model":"Modeo","price":"00"}             console.log(JSON.stringify(car));             $.ajax({                 "url": "http://127.0.0.1:5000/cars/"+encodeURI(car.reg),                 "method":"PUT",                 "data":JSON.stringify(car),                 "dataType": "JSON",                 contentType: "application/json; charset=utf-8",                 "success":function(result){                     console.log(result);                     document.getElementById("output").innerText = JSON.stringify(result);                     },                 "error":function(xhr,status,error){                     console.log("error: "+status+" msg:"+error);                 }             });         } ==== Test Delete ==== function deleteCar(){             var car = {"reg":"12 D 1234"}             console.log(JSON.stringify(car));             $.ajax({                 "url": "http://127.0.0.1:5000/cars/"+encodeURI(car.reg),                 "method":"DELETE",                 "data":"",                 "dataType": "JSON",                 contentType: "application/json; charset=utf-8",                 "success":function(result){                     console.log(result);                     document.getElementById("output").innerText = JSON.stringify(result);                     },                 "error":function(xhr,status,error){                     console.log("error: "+status+" msg:"+error);                 }             });           } ====== Week 7 - Python and API ===== ===== Python consuming APIs ===== * We can use python to do anything that we can do in CURL. * There are many web services that have APIs, that we can consume. * Github * Gmail * The server we made for the carviewer ===== What is needed ===== * Read and write JSON (usually to DICT) import json json.dump() json.load() * Send HTTP requests (with requests package) import requests requests.get(url) * Read and write to files f = open("../../week02/carviewer2.html", "r") html = f.read() * Write to excel file from xlwt import * w = Workbook() w.save('cars.xls') ===== JSON package ===== * https://realpython.com/python-json/ * In simple terms * ''dump'': converts Python object to JSON string and outputs to a file (''dumps'' returns a string) * ''load'': read a JSON string from a file and converts to Python object (''loads'': read string directly ===== mapping ===== ^ JSON ^Python^ | object |dict | | array |list | | string |str | | number (int) |int | | number (real) |float | | TRUE |TRUE | | FALSE |FALSE | | null |None | ==== Python Code Sample using Json ==== import json data = { 'name':'joe', 'age':21, 'student':'True'} print(data) ==== Code output ==== {'name': 'joe', 'age': 21, 'student': 'True'} ===== Sending HTTP requests ===== ==== Use the requests module ==== * https://requests.kennethreitz.org/en/master/user/quickstart/ * response = requests.get(url) * response = requests.post(url, json=data) * Response = requests.post(url, data=data) The ''requests'' matches HTTP methods: * GET, * POST, * PUT, * DELETE, * PATCH ==== Response has the http response ==== * response.json() * response.status_code() * response.content() * response.data() * response.headers() ==== Python Code Sample #1 ==== import requests url = 'https://gmit.ie' response = requests.get(url) print(response.status_code) #print(response.status_text) print(response.status_headers) === code output === 200 === headers ouput === {'Server': 'nginx', 'Date': 'Tue, 05 Nov 2019 08:23:04 GMT', 'Content-Type': 'text/html; charset=utf-8', 'Transfer-Encoding': 'chunked', 'Connection': 'keep-alive', 'X-Powered-By': 'PHP/7.1.29, PleskLin', 'X-Drupal-Cache': 'HIT', 'Content-Language': 'en', 'X-Frame-Options': 'SAMEORIGIN', 'X-Generator': 'Drupal 7 (http://drupal.org)', 'Cache-Control': 'public, max-age=1800', 'Expires': 'Sun, 19 Nov 1978 05:00:00 GMT', 'Vary': 'Cookie,Accept-Encoding', 'Content-Encoding': 'gzip', 'X-Content-Type-Options': 'nosniff', 'Etag': '"1572939088-1"', 'Last-Modified': 'Tue, 05 Nov 2019 07:31:28 GMT'} ==== Python code example #2 ==== === Python code === import requests url = 'http://127.0.0.1:5000/cars' data = {'reg':'123','make':'blah', 'model':'blah','price':1234} response=requests.post print(response.status_code) print(response.json()) === Code output === 201 {'car':{'make': 'blah', 'model': 'blah', 'price': 1234, 'reg': '123'}} ===== Files ===== * It is assumed that you have covered reading and writing to files previously * https://www.w3schools.com/python/python_file_handling.asp ==== Method One ==== with open(filename, 'w') as f:         json.dump(data, f, indent=4) ==== Method Two ==== f = open("demofile.txt", "rt") Data= json.load(f) ===== Excel ===== * There are a lot of packages in Python for Excel * http://www.python-excel.org/ * We are using xlwt (simple built in package) from xlwt import * w = Workbook() ws = w.add_sheet('cars’) ws.write(0,0, "data1") row = 1 col = 1 ws.write(row,col, "data") w.save('cars.xls') ===== Conclusion ===== * We can use python to do anything we can do in CURL. * This opens up all the webservices that have an API. * We will look at security next ===== API keys and OAUTH ===== ==== Restricting access ==== * A resource owner may want to restrict access to the resource. * May not want anyone to: * change data * View sensitive data (your bank details) * May want to limit the amount of times someone can make a request, especially if the request will take a lot of processing. E.g. searching twitter feeds. ==== API Keys ==== * API keys exist, and can be used to provide authorisation to access resources * The resource owner can generate a key can give it to a third party, to allow the third party access to the resource. * Hotel key * In practice: * Get a key: * Apply for a key from the resource owner * Log in and generate a key * Include it in the request to access the resource * Put it in the URL (GET) as a ''parameter'' * Put it in the header as a attribute ''parameter'' - Just like any other bit of data import requests import json ===== Demo 1 ===== #html = '

hello world

This is html' f = open("../../week02/carviewer2.html", "r") html = f.read() #print (html) apiKey = '46ceed910c24ff7cce8240e89ec7b71912f6f40f2ec55fd217ce150ad6d4f1c4' url = 'https://api.html2pdf.app/v1/generate' data = {'html': html,'apiKey': apiKey} response = requests.post(url, json=data) print (response.status_code) newFile = open("lab06.02.01.htmlaspdf.pdf", "wb") newFile.write(response.content)
===== Demonstration 2 Github ===== * We saw in the last lecture that github has an API, and used it to get public information * We can use the api get private information and make commits to the repository * Setting => generation a API key * Use the key in requests requests.get(url, auth=('token',apiKey)) * I have made a github account for this user: datarepresentationstudent * Key: b4ddb9e5603da11cd857b83bad6ea6eb1819b92d * More info * https://developer.github.com/v3/ * https://developer.github.com/v3/guides/ import requests import json apiKey = 'f59b1cdb743ce0ab9b21a94ac246b02f06ee2d85' url = 'https://api.github.com/repos/datarepresentationstudent/aPrivateOne' filename ="repo.json" response = requests.get(url, auth=('token',apiKey)) repoJSON = response.json() #print (response.json()) file = open(filename, 'w') json.dump(repoJSON, file, indent=4) ===== A bit more on OAuth ===== * OAuth1 and OAuth2 * Lots of different use-cases * 2 legged * Resource owner of a techy and can get the key from the authorisation server * 3 legged * Resource owner is not a techy, so we need to write the code to get the key from the authorisation server. * This is beyond the scope of this course….. But worth a quick explanation * https://auth0.com/docs/api-auth/which-oauth-flow-to-use {{:modules:oath_api_1.png?nolink|}} {{:modules:oath_api_2.png?nolink|}} ===== Conclusion ===== * You may need keys to access the a resource * Log on the resource server * Get the key * Include it in the requests ==== References ==== * JSON package: * https://docs.python.org/3/library/json.html * https://realpython.com/python-json/ * Requests * https://requests.kennethreitz.org/en/master/user/quickstart/ * Files * https://www.w3schools.com/python/python_file_handling.asp * Excel * https://xlwt.readthedocs.io/en/latest/api.html * http://www.blog.pythonlibrary.org/2014/03/24/creating-microsoft-excel-spreadsheets-with-python-and-xlwt/ ====== Week 8 - Server side ====== ====== Flask and virtual environments ====== ===== What is in this weeks Lectures ===== * Re-visit of Flask and virtual environments * Walkthrough of creating an App Server (and designed API) - A basic one - for a voting application * Hosting your app server - Azure - Python anywhere ===== What is an App-server?===== and what is a web-server? {{:modules:screenshot_2019-11-21_at_18.40.07.png?600|}} ===== virtual environment ===== * What problem does a virtual environment solve? * The more Python projects you have, the more likely it is that you need to work with different versions of Python libraries, or even Python itself. Newer versions of libraries for one project can break compatibility in another project.(from documentaiton) * on linux/mac python -m venv venv .\venv\scripts\activate.psl source venv/bin/activate * On windows ( I have not tested this, yet) virtualenv env \path\to\env\scripts\activate To save all the packages pip freeze pip install flask pip freeze > requirements.txt To install them again pip install -r path/to/requirements.txt ===== Simple Flask ===== #!flask/bin/python from flask import Flask app = Flask(__name__) @app.route('/') def index(): return "Hello, World!" if __name__ == '__main__' : app.run(debug= True) ===== Starting Application ===== * Two ways, what we know python b_restserver.py * And using Flask command * This has the advantage or being able to set environmental variables, so your server can run differently depending on whether is is in the cloud or on your own machine. * Eg debug mode: export FLASK_DEBUG=1 * Linux mac export FLASK_APP=b_restserver flask run * Windows (not tested yet) set FLASK_APP=b_restserver flask run * We will see more on this when uploading to Azure pip install flask_cors from flask_cors import CORS ===== Flask ===== * Routing (URL mapping) @app.route('/user’, methods=['GET', 'POST']) * Variables in route (URL) @app.route('/user/') @app.route('/user/’) * Use the url_for() function to generate a url to a particular function @app.route('/') def index(): return 'index’ #somewhere else print(url_for(index) # will print out / * Rendering Templates vs JSON * Flask can be used to create html pages on the fly As oppose to * Just returning the data and allowing the static web pages use AJAX to get that data. * We use the latter method ===== Flask ===== To access request data import request object from flask import request See the code we did in week05 "reg": request.json['reg'], Other data you can get * request.path => /user * request.method => GET * request.form => form data * request.args => arguments from the URL * request.files[‘the_file’] => an uploaded file ===== Flask ===== * aborts and redirects from flask import abort, redirect, url_for @app.route('/') def index(): return redirect(url_for('login')) @app.route('/login') def login(): abort(401) this_is_never_executed() ===== Flask ===== * Sessions * Allow data to be stored between requests (e.g is a user is logged in) * I will go through this in week 10 when I go through authorisation (login) * Flashing * Logging * More data (link in references) https://buildmedia.readthedocs.org/media/pdf/flask/latest/flask.pdf ====== Creating web-applications and designing example APIs ====== ===== REST - a server for the project ===== ===== Requirements ===== * An app server that has a RESTful interface to provide CRUD operations for one database table * I am going to pick something at random like a book, you should choose your own entity * A Book will have * An id (Integer, auto increment) KEY i.e. this will be the unique identifier * A title * An Author * A price (integer, the price in cent) * There are other attributes it could have ISBN (this could have been the unique identifier, I choose to use an ID instead to make this as general as possible. ===== Step 1: design the API ===== * This is just a CRUD interface so this will be similar to app server we created in week05 * It will need to allow us * Get all books * Get a book by id (might not actually be needed, but I am putting it in anyway) * Create a book * Update a book * Delete a book * Other applications may require other functionality in the interface. And we can always add functionality later. * NOTE: no code yet ===== Proposed interface ===== {{:modules:52957:81b.jpg?nolink|}} ===== Step2 : make app-server with skeleton functions ===== * OK Now we start to code * Make a very basic appserver, test it. * Add a function and URL map for each of the functions we require in our interface. Each function should just return text saying what they are. Test them using CURL ===== STEP3: write the code for each of the functions ===== * For this stage we will not link to a database, we will just store the books in a list like we did in week 05. * Do the get all first that should be the easiest, TEST if with CURL * Do find by id, TEST IT * Do create, TEST IT * Do update, TEST IT * Do delete, TEST IT ===== REST - a server for a voting app ===== ===== Requirements ===== * We are having a get lippy competition in the college on the 5 December in the IVY tower in Castlebar (you are welcome to come). * One of the organisers has asked me to create an application that will allow the people on the desk register votes for each act. * We are going to make this a very very basic application, no authentication. * The users will go to a website, pick from a list of acts and add a number of votes to that acts tally (not just one vote) * I am not looking at the web interface for problem, just the server ===== Step 1: design the API ===== * There are two types of users we think about * The person (admin) who sets up all the act names (probably me, so I could just enter them directly into the data base table….. But it should be through a web interface) * The person on the desk (User) they need to: * View all the acts (just their names) * And register a number of votes for each act * View the sum of all the votes for each act * I could just have one entity for this called act, but I am going to have two act and vote ===== Proposed Solution ===== {{:modules:52957:8.2b.jpg?nolink|}} ===== Step2 : make app-server with skeleton functions ===== * OK Now we start to code * Make a very basic app-server, test it. * Add a function and URL map for each of the functions we require in our interface. Each function should just return text saying what they are. Test them using CURL ===== STEP3: write the code for each of the functions ===== * For this stage we will not link to a database, we will just store the acts (and votes ) in a list(s) like we did in week 05. * I suggest that we do the CRUD on the acts first. * Then the two votes function. * NOTE: the interface allows us to implement this in a number of ways, * Either two entities, * Or just store the running total in each act. * I am choosing the former. ====== Hosting web-applications on the cloud ====== ===== References ===== ==== Virtual environment ==== * https://docs.python-guide.org/dev/virtualenvs/ ==== Using Flask ==== * https://buildmedia.readthedocs.org/media/pdf/flask/latest/flask.pdf ==== hosting on Azure, tutorial: ==== * https://medium.com/@nikovrdoljak/deploy-your-flask-app-on-azure-in-3-easy-steps-b2fe388a589e * https://github.com/smartninja/example-azure-flask ==== Hosting on PythonAnywhere ==== * http://help.pythonanywhere.com/pages/Flask ===== Database ===== * We will be using MySQL(a relational database) * other databases are * MongoDB * SQLLite * PostGres * Install it * Windows I recommend WAMP * MAC/Linux install MySQL and MySQLWorkbench ==== sql ==== * The language to interact with is SQL * Structured Query Language * Tutorial on W3schools * https://www.w3schools.com/sql/ === some commands === show databases; create database test; use test create table test{ id int NOT NULL AUTO_INCREMENT, name varchar(250), PRIMARY KEY(id) }; insert into test (name) values ('jow'); select * from test; update test set name='blah' where id = 1; delete from test where value =1; ===== MySQL Connector ===== * We will use mysql's mysql-connector * Can use SQLAlchemy, which does ORM(so same code will connect to any database) * Install pip install mysql-connector ==== Use ==== import mysql.connector mydb = mysql.connector.connect( host="localhost", user="root", password="" ) mycursor = mydb.cursor() mycursor.execute("CREATE DATABASE datarepresentation ") ==== Prevent SQL Injection ==== sql = "insert into student (name, address) values (%s, %s)" values=("Mary", "Galway") import mysql.connector db = mysql.connector.connect( host="localhost", user="root", password="", database="datarepresentation" ) cursor = db.cursor() sql="insert into student (name, address) values (%s,%s)" values = ("Mary", "Galway") cursor.execute(sql, values) db.commit() print("1 record inserted, ID:", cursor.lastrowid) ==== Get Data out ==== # get connection code here cursor = db.cursor() sql = "Select * from student where id = %s" values = (1, ) cursor.execute(sql, values) result = cursor.fetchall() for x in result: print(x) ==== Put into a file that can be reused ==== * Make a class put all the functions into it * Make an instance of that class * Import into another filer (will be flask later) ===== Linking server to Database ===== ==== in this video ==== * Convert the server we did, so that it uses a database to store the data and not a list in memory * I will use the sample DAO that we did in the last lecture * In this walkthrough I convert the getAll and findById. * I realise while trying to do the create that I am working on the student table and not a book table, but as you can see it makes no difference. ==== Converting a Tuple to Dictionary Object ==== * I think that I should do this in the DAO functions * Create a dictionary object and populate it with values taken from the databse * I can create an array of attribute names and iterate through it atking the matching values * that will work for one item being returned, it is going to be a but clunky for the getAll ==== Find by ID ==== def findById(id): foundBook = studentDAO.findByID(id) return jsonify(foundBook) ==== Create ==== * The student DAO takes in a tuple of values in order that they appear on the database. * This is not the most robust way of doing this, but will do for the moment ===== Configuration files ===== * Different configurations depending on where your applications is run. * Eg. * Are you developing on different machines, i have a PC in my office and a mac at home * Production machine (needs a more secure database) * Testing machines * Colleagues machines * We have already seen requirements.txt * We need away for storing environment specific variable ==== One way ==== * A python file that has the variable in an object * Import that file into yours and use it * Make sure you do not put it into github ie use .gitignore * A copy of this file will have to be made for each environment. mysql ={ 'host':'localhost', 'user':'root', 'password':'my secret password', 'db':'write-math' } import databaseconfig as cig connect(cfg.mysql['host'], cfg.mysql['user'], cfg.mysql['password']) ===== Connect to DB on PythonAnywhere ===== ==== Setup ==== * I assume that: * you have a server that connects to your local database. * You are using a configuraion file for the credentials for the database ToDo - Create a password for the remote DB. - Create the table on mysql - Upload the full server - Make dbconfigfile - Test it ==== Remote DB ==== * Create password * Note credentials * Create table ==== Step 2 ==== * Copy server code to repository on local machine, test it * Upload the server and pull to host * Create a dbconfig.py file for DB * Make sure the web-app configuration is good * TEST IT