DATA ANALYTICS REFERENCE DOCUMENT |
|
---|---|
Document Title: | Document Title |
Document No.: | 1569148014 |
Author(s): | Gerhard van der Linde, Rita Raher |
Contributor(s): |
REVISION HISTORY
Revision | Details of Modification(s) | Reason for modification | Date | By |
---|---|---|---|---|
0 | Draft release | Document description here | 2019/09/22 10:26 | Gerhard van der Linde |
The syllabus has 4 learning outcomes
Which in practices means:
xml → eXtensible Markup Language
<?xml version="1.0" encoding="UTF-8"?> <book isbn-13="978-0131774292" isbn-10="0131774298"> <title>Python Programming</title <publisher>Prentice Hall</publisher> <author>Peter van der Linden</author> </book>
<?xml version="1.0" encoding="UTF-8"?> <root-element attribute-name="attribute-value"> <child name="value">Text</child> <child name="value">Text</child> <child>< is an example of encoding</child> <lone-warrior/> </root-element>
Declaration | XML documents should have a single line at the start stating that it's xml, the versionof xml it is and an encoding |
Root Element | XML must have a single root element that wraps all others |
Elements | XML is structured as elements, which are enclosed in angle brackets. Elements must have a closing tag. |
Tag | <Tags> indicate the start and the end of each element(end tag indicated with </tagname>) |
Attributes | Elements can have attributes, which are name-value pairs within the angle brackets. A given attribute name can only be specified once per element. |
Entity References | Certain characters must be escaped with entity references, e.g < for<. |
case sensitive | Everything in XML is case sensitive. (HTML is not) |
Plain text | Or Data, can be inside the tags |
<?xml version="1.0" encoding="UTF-8"?> <breakfast_menu> <food> <name>Belgian waffles</name> <price>$5.95</price> <description>Two of our famous Belgian Waffles with syrup</description> <calories>650</calories> </food> <food> <name>Strawberry Belgian waffles</name> <price>$7.95</price> <description>Light Belgian waffles covered with strawberries and whipped cream</description> <calories>900</calories> </food> </breakfast_menu>
<html>, <head>,<body>
<div>
tag<table>, <tr>, <th>, <td>
<form>, <input>, <button>, <select>, <option>
I am not going through how to make pages look nice, that is a whole course in itself. Know that CSS exists and is used to format webpages. The script tag I will go through next week.
<div style="display:block"> text </div>
<input type="text" value="blah" disable/>
Exercise 2.1
Write some html that has a hidden element
<html> <head> <title>2.1</title> </head> <body> <div style="display:none">Hi Mom</div> <div style="display:block">div1</div> <div style="display:block">div2</div> <div style="display:inline">div3</div> </body> </html>
<script></script>
tags<script> console.log("hello world") </script>
<script> function sayHelloAgain(){ console.log("helloagain") } sayHelloAgain() </script>
Exercise: Print out Hello World
<html> <head> <title>e2.2</title> </head> <body> hello <script> console.log("Hello world") function sayhelloagain(message){ console.log("Hello" + message); } sayhelloagain("Andrew") </script> </body> </html>
<button onclick="myFunction('hello')">click me</button>
<div id="messageOut"></div>
document.getElementById("messageOut").innerText = message
Exercise 2.3
Make a webpage with an input and a button, when the user clicks the button then the contents of the input will display in another div.
<html> <head> <title>e2.2</title> </head> <body> <input id="name" type="text" /> <br/> <button onclick="buttonClicked()">Click me</button> <br/> <div id="output">output will go here</div> <script> function buttonClicked(){ var output =document.getElementById('name').value document.getElementById('output').innerText = output } </script> </body> </html>
var i = 12 var fl = 3.3 var firstName = "andrew" var lastName = "O'Neill" //or 'O\'Neill' var admin = true var foo = null var dunno //= undefined var car = {} var books = [] var fun = function(){console.log('in fun')}
car.reg = '12 mo 1234' car.make = "ford"
var employee = { name:'joe', role:"Manager"}
console.log("car" +JSON.stringify(car))
Exercise 2.4
Create a book object as a variable with Title, author and ISBN and display it in the console
<html> <head> <title>2.4</title> </head> <body> <script> var book = {} book.title= "Harry Potter" book.author = "JK" book.isbn="1234" console.log("book is "+ JSON.stringify(book)) </script> </body> </html>
document.getElementById("div1").style.display = "block" document.getElementById("div2").style.display = "none"
document.getElementById("button1").disabled = true
Exercise 2.5:
Make a webpage that has two div, only one should be visible at a time, each <div>
will have a button that will make the other <div>
visible (and hide itself)
<html> <head> <title>2.4</title> </head> <body> <div id="div1" style="background-color:royalblue"> <br/> some text <br/> <button onclick="showDiv2()">Show Div2</button> </div> <div id="div2" style="background-color:red;display:none"> <br/> some text <br/> <button onclick="showDiv1()">Show Div2</button> </div> <script> function showDiv2(){ document.getElementById("div1").style.display = "none"; document.getElementById("div2").style.display = "block"; } function showDiv1(){ document.getElementById("div2").style.display = "none"; document.getElementById("div1").style.display = "block"; } </script> </body> </html>
if statement
if(condition){ do stuff }else { do other stuff }
if(value < 10){ outputElement.innerText = "too low" }else if(value >20){ }else{ outputElement.innerText ="goo enough" }
Exercise 2.6:
Make a web page that has a text box, when the content of the text area changes the page will display whether the value entered is greater or less than 10 or equal to 10
<body> <input type="number" onchange="checkNumber(this)"/> <br/> <div id="output"> output here </div> <script> function checkNumber(inputElement){ var value = inputElement.value if( value <10){ document.getElementById("output").innerText = "too low" }else if (value > 10){ document.getElementById("output").innerText = "too high" }else{ document.getElementById("output").innerText = "just right" } } </script> </body>
for(var i =0;i<10;i++){ console.log(i); }
var names=['Joe', 'Mary', 'Fred']
var book ={title:'harry Potter and something',author:'JKR', isbn:"12345"} for(propName in book){ bookoutput += propName+'='+book[propName]+'<br/>' }
Exercise 2.7:
<!DOCTYPE HTML> <html> <head> <title>e2.7</title> </head> <body> <script> for(var i=0;i<10;i++){ console.log(i) } </script> </body> </html>
<!DOCTYPE HTML> <html> <head> <title>e2.8</title> </head> <body> <div id="output"> output here </div> <script> for(var i=0;i<10;i++){ console.log(i) } var names=['Joe', 'Mary', 'Fred'] var output = "" for(name of names){ output += name + '<br//>' } document.getElementById("output").innerHTML = output </script> </body> </html>
Exercise 2.8:
Create a webpage with multiple inputs and a button, when the user hits the button read all the inputs, store the values into a class and output the class to the console
<body> <div id="myForm"> First name: <input type="text" name='firstName'/><br/> Last name: <input type="text" name='lastName'/><br/> age: <input type="text" name='age'/><br/> <button onclick="doForm()">go</button> </div> <script> function doForm(){ var employee ={} var form = document.getElementById("myForm") employee.firstName = form.querySelector('input[name="firstName"').value employee.lastName = form.querySelector('input[name="lastName"').value employee.age = form.querySelector('input[name="age"').value console.log("form data "+JSON.stringify(employee)) } </script> </body>
<table> <tr> <td>data1</td> <td>data2</td> <td><button onclick="doRow(this)">output</button></td> </tr> </table>
To get the content of the first cell, you need to get the child of that cell and its text value
rowElement.cells[0].firstChild.textContent
<td>
is the parent and <tr>
is the <td>
so we can access the <tr>
by:
var rowElement = buttonElement.parentNode.parentNode //or Var rowElement = buttonElement.closest('tr')
Exercise 2.9
Write the code so that when the user clicks on the button in a row, the contents of the cells in that row are stored in an object and outputted to the console.
<body> <table> <tr> <td>how now</td> <td>brown cow</td> <td><button onclick="doRow(this)">output</button></td> </tr> </table> <script> function doRow(buttonElement){ var rowElement = buttonElement.parentNode.parentNode var data={} data.cell1 = rowElement.cells[0].firstChild.textContent; data.cell2 = rowElement.cells[1].firstChild.textContent; console.log(JSON.stringify(data)) } </script> </body>
var para = document.createElement("p"); var node = document.createTextNode("This is new"); para.appendChild(node);
var rowElement =table.Element.insertRow(-1)
var cell = rowElement.insertCell(0);
rowElement.setAttribute("id", car.reg)
Exercise 2.9.2
Create a button that when it is clicked adds a row to the table(above) with data
function createRow(){ var tableElement = document.getElementById('mainTable') var data = {cell1:'hi',cell2:'bye',cell3:'thanks for the fish'} var rowElement = tableElement.insertRow(-1) var cellElement = rowElement.insertCell(0) cellElement.innerHTML = data.cell1; cellElement = rowElement.insertCell(1) cellElement.innerHTML = data.cell2; cellElement = rowElement.insertCell(2) cellElement.innerHTML = data.cell3; }
What is HTTP?
http://ps.com/martin Kenirons
http://www.jmarshall.com/easy/http/
Request module
import requests page = requests.get("http://dataquestio.github.io/web-scraping-pages/simple.html") print(page) print("------------") print(page.content)
import requests from bs4 import BeautifulSoup page = requests.get("http://dataquestio.github.io/web-scraping-pages/simple.html") print(page) print("------------") print(page.content) soup1 = BeautifulSoup(page.content, 'html.parser') print(soup1.prettify())
from bs4 import BeautifulSoup with open("../week02/carview2.html") as fp: soup = BeautifulSoup(fp, 'html.parser') print(soup.prettify())
aElement = someElement.find(id="someID")
aElement = someElement.find(class="aClass")
aElement = someElement.find(div, class="aClass")
listOfElement = someElement.findAll(tr, class="aClass")
from bs4 import BeautifulSoup with open("lab02-JS/carviewer2.html") as fp: soup = BeautifulSoup(fp, 'html.parser') print(soup.tr)
from bs4 import BeautifulSoup with open("lab02-JS/carviewer2.html") as fp: soup = BeautifulSoup(fp, 'html.parser') rows = soup.findAll("tr") for row in rows: print(row)
from bs4 import BeautifulSoup with open("lab02-JS/carviewer2.html") as fp: soup = BeautifulSoup(fp, 'html.parser') #print(soup.prettify()) #print(soup.tr) rows = soup.findAll("tr") for row in rows: #print(row) datalist = [] cols = row.findAll("td") for col in cols: datalist.append(col.text) print(datalist)
import csv employee_file = open('employee_file.csv', mode='w') employee_writer = csv.writer(employee_file, delimiter=',', quotechar='"', quoting=csv.QUOTE_MINIMAL) employee_writer.writerow(['John Smith', 'Accounting', 'November']) employee_writer.writerow(['Erica Meyers, what', 'IT', 'March']) employee_file.close()
https://learnonline.gmit.ie/course/view.php?id=1318&blah=8
https://andrewbeattycourseware:pass@github.com/andrewbeattycourseware/dataRepresentation.git
You can also have a username:password pair before a @ if you need to log into the host There are other authentication methods (EG Oauth)
import requests import csv from bs4 import BeautifulSoup url = "https://www.myhome.ie/residential/mayo/property-for-sale?page=1" # load the page from the above url in the object page. page = requests.get(url) #parse only the html from page.content soup = BeautifulSoup(page.content, 'html.parser') #open a file and csv writer objects home_file = open('week03MyHome.csv', mode='w', newline='\n', encoding='UTF-8') home_writer = csv.writer(home_file, delimiter=',', quotechar='"', quoting=csv.QUOTE_MINIMAL) #parse the listings from soup using the class label listings = soup.findAll("div", class_="PropertyListingCard" ) #loop trhough the listings for listing in listings: #create an empty list for writing out to csv entryList = [] #parse and append price to listing price = listing.find(class_="PropertyListingCard__Price").text.strip() entryList.append(price) #parse and append address to listing address = listing.find(class_="PropertyListingCard__Address").text.strip() entryList.append(address) #write parsed data to the csv file home_writer.writerow(entryList) #close the file home_file.close()
Note: Note the specification for 'UTF-8
' encoding and strip()
commands to clean the parsed data and specify the newline characters
"€625,000","Curraghmore, Belcarra, Castlebar, Co Mayo" "Reserve Not To Exceed €80,000","April Cottage (Folio MY51001F), Balloughadalla, Killala, Co. Mayo" "AMV €30,000","C. 4.09 Acres of Land, Dooagh, Achill, Mayo" "€140,000","15 Riverside Drive, Ballina, Mayo" "Reserve Not To Exceed €110,000","Riverbank House, Bachelors Walk, Abbeyhalfquarter, Ballina, Co. Mayo" "€115,000","Croghan, Killala, Mayo" "€195,000","Carrowbaun, Westport, Co Mayo, F28K298" "€235,000","3 Rosmore, Newport, Co Mayo, F28 VF38" "AMV €245,000 -€30,000 on 5th Oct 19","ONLINE AUCTION Camcloon More, Newport, Mayo" "€250,000","Roonith, Louisburgh, Mayo" "€125,000","8 Rathkelly , Ballinrobe, Mayo" "€179,500","5 Ballyhaunis Road, Knock, Co. Mayo" "€175,000","1 Cinnamon Wharf, The Quay, Westport, Co Mayo, F28 K6W8" "Reserve Not To Exceed €15,000","Ballymunnelly (Folio MY14396F), Bellacorick, Co. Mayo" "Reserve Not To Exceed €130,000","Loughrusheen, Castlebar, Co. Mayo" "€198,000","13 Church Manor, Ballina, Mayo" "€110,000","Cappaghduff East, Tourmakeady, Mayo" "€215,000","12 Rushbrook, Claremorris, Mayo" "€120,000","Knockfin, Westport, Mayo" "Reserve Not To Exceed €40,000","5A and 5B, Main Street, Ballyhaunis, Co. Mayo" "€45,000","Main Street, Ballycastle, Mayo"
https://www.dataquest.io/blog/web-scraping-tutorial-python/
https://buildmedia.readthedocs.org/media/pdf/beautiful-soup-4/latest/beautiful-soup-4.pdf
Method | Description | In RESTful API |
---|---|---|
GET | Retrieves a Resource(s) | Read |
POST | Updates/creates/changes a Resource | Create |
PUT | Used more to update a Resourse | Update |
DELETE | Remove a resourse | Delete |
HEAD | Retrieves header info | |
TRACE | Used for Debugging | |
OPTIONS | Retrieve allowable options | |
PATCH | Partial resource modification | |
CONNECT | Set up a tunnel for other traffic to pass through HTTP |
GET | POST | |
---|---|---|
Security | Parameters are visible to the user and browser History. | POST is a little safer than GET because the parameters are not stored in browser history or in web server logs |
Never use GET when sending passwords or other sensitive information! | ||
Back Button | harmless | Data will be resubmitted |
Bookmarked | Can be bookmarked | Parameters are not bookmarked |
Cached | Can be cached | Not cached |
Encoding type | application/x-www-form-urlencoded | application/x-www-form-urlencoded or multipart/form-data. Use multipart encoding for binary data |
History | Parameters remain in browser history | Parameters are not saved in browser history |
Visibility | Data is visible to everyone in the URL | Data is not displayed in the URL |
Restrictions on data length | data length | No Restrictions |
Yes, when sending data, the GET method adds the data to the URL; and the length of a URL is limited (maximum URL length is 2048 characters) | ||
Restrictions on data type | Only ASCII characters allowed | No restrictions. Binary data is also allowed |
Requests and responses both have this format: • Intial line.
HTTP/1.1 200 OK Date: Mon, 27 Jul 2009 12:28:53 GMT Server: Apache/2.2.14 (Win32) Last-Modified: Wed, 22 Jul 2009 19:15:56 GMT Content-Length: 88 Content-Type: text/html Connection: Closed
<html> <body> <h1>Hello, World!</h1> </body> </html>
Code Range | Status |
---|---|
100 – 199 | Informational |
200 – 299 | Successful |
300 – 399 | Redirection |
400 – 499 | Client Error |
500 – 599 | Server Error |
Code | text | description |
---|---|---|
200 | OK | Successful |
201 | Created | Resource created |
204 | No Content | The requested resource is empty |
400 | Bad Request | Bad Syntax |
401 | Unauthorised | Client might need to be authenticated |
403 | Forbidden | Operation not allowed |
404 | Not Found | Resource does not exists |
500 | Internal Server Error | Something went wrong during processing |
More at https://www.restapitutorial.com/httpstatuscodes.html
{ "employees": [ {"firstName":"John", "lastName":"Doe"}, {"firstName":"Anna", "lastName":"Smith"}, {"firstName":"Peter", "lastName":"Jones"} ] }
// Turning text into a JavaScript object. var obj = JSON.parse(text); // obj is an object. // Turning a JavaScript object into text. var text = JSON.stringify(obj); // text is a string.
“name”: “Martin”
{ }
[]
“Martin”
Numbers - 123.456 Boolean - true Objects - {"name": "Ian"\} Functions - function(){//commands} Strings - "Hello, world!" Arrays - [1,2,3] null - null
"employees":[ {"firstName":"John", "lastName":"Doe"}, {"firstName":"Anna", "lastName":"Smith"}, {"firstName":"Peter", "lastName":"Jones"} ]
Exercise (not an assignment!):
{ "time": { "updated": "Oct 16, 2019 15:03:00 UTC", }, "disclaimer": "This data was produced from the CoinDesk Bitcoin Price Index (USD). ", "chartName": "Bitcoin", "bpi": { "USD": { "code": "USD", "symbol": "$", "rate": "7,968.2517", "description": "United States Dollar", "rate_float": 7968.2517 }, "EUR": { "code": "EUR", "symbol": "€", "rate": "7,209.3474", "description": "Euro", "rate_float": 7209.3474 } } }
Asynchronous: | Does not block code, the return function gets called when complete |
---|---|
Javascript: | The language |
And XML: | Well not much anymore, usually JSON or html is used |
More info at https://www.w3schools.com/xml/ajax_intro.asp
We will use AJAX with Jquery not the plain JavaScript version: https://www.w3schools.com/jquery/jquery_ajax_intro.asp
<head> <script src="https://ajax.googleapis.com/ajax/libs/jquery/3.4.1/jquery.min.js"></script> </head>
$.ajax({the object that describes everything});
$.ajax({ "url": "https://api.coindesk.com/v1/bpi/currentprice.json ", "method":"GET", "data":"", "dataType": "JSON", "success":function(result){ console.log(result); }, "error":function(xhr,status,error){ console.log("error: "+status+" msg:"+error); } } )
url: | the url to call |
---|---|
method: | the HTTP method |
data: | parameters (none in this case) |
datatype: | that is returned JSON or HTML or XML |
success: | the function to call if this is a success (with the result from server) |
error: | The function to call if we don’t get a status code in the 200s |
<html> <head> <script src="https://ajax.googleapis.com/ajax/libs/jquery/3.4.1/jquery.min.js"></script> <title> read a simple json </title> </head> <body> <button onclick="readJSON()">go</button> </body> <script> function readJSON(){ $.ajax({ "url": "https://api.coindesk.com/v1/bpi/currentprice.json ", "method":"GET", "data":"", "dataType": "JSON", "success":function(result){ console.log(result); }, "error":function(xhr,status,error){ console.log("error: "+status+" msg:"+error); } }); } </script> </html>
Looking at the JSON you can see the value we are looking for is in the object: bpi.EUR.rate
{
"time": {
"updated": "Oct 16, 2019 15:43:00 UTC",
"updatedISO": "2019-10-16T15:43:00+00:00",
"updateduk": "Oct 16, 2019 at 16:43 BST"
},
"disclaimer": "This data was produced from the CoinDesk Bitcoin Price Index (USD). Non-USD currency data converted using hourly conversion rate from openexchangerates.org",
"chartName": "Bitcoin",
"bpi": {
"USD": {
"code": "USD",
"symbol": "$",
"rate": "7,979.8633",
"description": "United States Dollar",
"rate_float": 7979.8633
},
"EUR": {
"code": "EUR",
"symbol": "€",
"rate": "7,220.6990",
"description": "Euro",
"rate_float": 7220.699
}
}
}
So the code to extract it is:
var rate = result.bpi.EUR.rate console.log(rate); //document.getElementById("output").innerText = rate;
I.e. An interface that a program that you write can use to transfer data to another program
<?xml version="1.0"?> <soap:Envelope xmlns:soap="http://www.w3.org/2003/05/soap-envelope/" soap:encodingStyle="http://www.w3.org/2003/05/soap-encoding"> <soap:Body> <m:GetPrice xmlns:m="https://www.w3schools.com/prices"> <m:Item>Apples</m:Item> </m:GetPrice> </soap:Body> </soap:Envelope>
Method | URL | Description | Sample return |
---|---|---|---|
GET | /emails | Get all the Emails | [ {id:32,from:”joe@joe.ie”,message:”hi”}, {id:33,from:”me@gmail.ie”,message:”i”}, {id:35,from:”a@b.com”,message:”blah”} ] |
POST | /emails | Create an Email | {created:true} |
GET | /emails/32 | Retrieve the Email with id 32 | {id:32,from:”joe@joe.ie”,message:”hi”} |
PUT | /emails/32 | Update the Email with id 32 | {id:32,from:”joe@joe.ie”,message:”bye”} |
DELETE | /emails/32 | Delete the Email with id 32 | {deleted:true} |
<html> <head> <title>simple json</title> </head> <script> var obj = JSON.parse('{ "isbn": 1234567, "Author": "Gerhard van der Linde","Title" : "the rain in spane falls mailey on the plane"}') console.log(obj.Title) </script> </html>
<html> <head> <title>lab02</title> </head> <body> <button onclick="displayJSON()">go</button> </body> <script> function displayJSON(){ var obj = JSON.parse('{"employees": [{"firstName": "John", "lastName": "Doe"},{ "firstName": "Anna","lastName": "Smith"},{"firstName": "Peter","lastName": "Jones"}]}'); //for (employee of obj.employees){ // console.log(employee.firstName) //} //var employees=obj.employees; obj.employees.forEach(employee => { console.log(employee.lastName); }); } </script> </html>
<html> <head> <script src="https://ajax.googleapis.com/ajax/libs/jquery/3.4.1/jquery.min.js"></script> <title> read a simple json </title> </head> <body> <button onclick="readJSON()">go</button> <div id="output"></div> </body> <script> function readJSON(){ $.ajax({ "url": "https://api.coindesk.com/v1/bpi/currentprice.json ", "method":"GET", "data":"", "dataType": "JSON", "success":function(result){ console.log(result); var rate = result.bpi.EUR.rate //console.log(rate); document.getElementById("output").innerText = rate; }, "error":function(xhr,status,error){ console.log("error: "+status+" msg:"+error); } }); } </script> </html>
<html> <head> <script src="https://ajax.googleapis.com/ajax/libs/jquery/3.4.1/jquery.min.js"></script> <title> read etherium </title> </head> <body> <button onclick="readJSON()">go</button> <div id="output"></div> </body> <script> function readJSON() { $.ajax({ "url": "https://min-api.cryptocompare.com/data/price?fsym=ETH&tsyms=BTC,USD,EUR", "method": "GET", "data": "", "dataType": "JSON", "success": function (result) { console.log(result); var rate = result.USD //console.log(rate); document.getElementById("output").innerText = rate; }, "error": function (xhr, status, error) { console.log("error: " + status + " msg:" + error); } }); } </script> </html>
Option | Description | Example |
---|---|---|
-i | see the response header | curl -i http://dummy.restapiexample.com/api/v1/employee/2201 |
-X <method> | set the method | curl -i -X DELETE http://dummy.restapiexample.com/api/v1/delete/2201 |
-d <data> | set the data to be uploaded | curl -i -H “Content-Type:application/json” -X PUT -d '{“name”:“test1”,“salary”:“1123”,“age”:“23”}' http://dummy.restapiexample.com/api/v1/update/2201 |
-H <header> | set the header |
#!flask/bin/python from flask import Flask @app.route('/') def index(): return "Hello, World!" if __name__ == '__main__' : app.run(debug= True)
#!flask/bin/python from flask import Flask app = Flask(__name__, static_url_path='', static_folder='../') @app.route('/') def index(): return "Hello, World!" if __name__ == '__main__' : app.run(debug= True)
cars = [ { "reg":"181 G 1234", "make":"Ford", "model":"Modeo", "price":18000 } ]
@app.route('/cars', methods=['GET']) def get_cars(): return jsonify( {'cars':cars})
@app.route('/cars', methods=['POST']) def create_car(): if not request.json: abort(400) if not 'reg' in request.json: abort(400) car={ "reg": request.json['reg'], "make": request.json['make'], "model":request.json['model'], "price":request.json['price'] } cars.append(car) return jsonify( {'car':car }),201
@app.route('/cars/<string:reg>', methods =['PUT']) def update_car(reg): foundCars=list(filter(lambda t : t['reg'] ==reg, cars)) if len(foundCars) == 0: abort(404) if not request.json: abort(400) if 'make' in request.json and type(request.json['make']) != str: abort(400) if 'model' in request.json and type(request.json['model']) is not str: abort(400) if 'price' in request.json and type(request.json['price']) is not int: abort(400) foundCars[0]['make'] = request.json.get('make', foundCars[0]['make']) foundCars[0]['model'] =request.json.get('model', foundCars[0]['model']) foundCars[0]['price'] =request.json.get('price', foundCars[0]['price']) return jsonify( {'car':foundCars[0]})
@app.route('/cars/<string:reg>', methods =['DELETE']) def delete_car(reg): foundCars = list(filter (lambda t : t['reg'] == reg, cars)) if len(foundCars) == 0: abort(404) cars.remove(foundCars[0]) return jsonify( { 'result':True })
@app.errorhandler(404) def not_found404(error): return make_response( jsonify( {'error':'Not found' }), 404) @app.errorhandler(400) def not_found400(error): return make_response( jsonify( {'error':'Bad Request' }), 400)
<html> <head> <script src="https://ajax.googleapis.com/ajax/libs/jquery/3.4.1/jquery.min.js"></script> <title> test get all </title> </head> <body> <button onclick="getAll()">getAll</button> <div id="output"></div> <table id="carsTable"> <tr> <th>reg</th><th>make</th><th>model</th><th>price</th><th></th><th></th> </tr> </table> </body> <script> function getAll(){ $.ajax({ "url": "http://127.0.0.1:5000/cars", "method":"GET", "data":"", "dataType": "JSON", "success":function(result){ console.log(result); }, "error":function(xhr,status,error){ console.log("error: "+status+" msg:"+error); } }); } </script> </html>
function createCar(){ var car = {"reg":"12 D 1234","make":"Fiat","model":"Punto","price":3000} console.log(JSON.stringify(car)); $.ajax({ "url": "http://127.0.0.1:5000/cars", "method":"POST", "data":JSON.stringify(car), "dataType": "JSON", contentType: "application/json; charset=utf-8", "success":function(result){ console.log(result); document.getElementById("output").innerText = JSON.stringify(result); }, "error":function(xhr,status,error){ console.log("error: "+status+" msg:"+error); } }); }
function updateCar(){ var car = {"reg":"181 G 1234","make":"Ford","model":"Modeo","price":"00"} console.log(JSON.stringify(car)); $.ajax({ "url": "http://127.0.0.1:5000/cars/"+encodeURI(car.reg), "method":"PUT", "data":JSON.stringify(car), "dataType": "JSON", contentType: "application/json; charset=utf-8", "success":function(result){ console.log(result); document.getElementById("output").innerText = JSON.stringify(result); }, "error":function(xhr,status,error){ console.log("error: "+status+" msg:"+error); } }); }
function deleteCar(){ var car = {"reg":"12 D 1234"} console.log(JSON.stringify(car)); $.ajax({ "url": "http://127.0.0.1:5000/cars/"+encodeURI(car.reg), "method":"DELETE", "data":"", "dataType": "JSON", contentType: "application/json; charset=utf-8", "success":function(result){ console.log(result); document.getElementById("output").innerText = JSON.stringify(result); }, "error":function(xhr,status,error){ console.log("error: "+status+" msg:"+error); } }); }
import json json.dump() json.load()
import requests requests.get(url)
f = open("../../week02/carviewer2.html", "r") html = f.read()
from xlwt import * w = Workbook() w.save('cars.xls')
dump
: converts Python object to JSON string and outputs to a file (dumps
returns a string)load
: read a JSON string from a file and converts to Python object (loads
: read string directlyJSON | Python |
---|---|
object | dict |
array | list |
string | str |
number (int) | int |
number (real) | float |
TRUE | TRUE |
FALSE | FALSE |
null | None |
import json data = { 'name':'joe', 'age':21, 'student':'True'} print(data)
{'name': 'joe', 'age': 21, 'student': 'True'}
The requests
matches HTTP methods:
import requests url = 'https://gmit.ie' response = requests.get(url) print(response.status_code) #print(response.status_text) print(response.status_headers)
200
{'Server': 'nginx', 'Date': 'Tue, 05 Nov 2019 08:23:04 GMT', 'Content-Type': 'text/html; charset=utf-8', 'Transfer-Encoding': 'chunked', 'Connection': 'keep-alive', 'X-Powered-By': 'PHP/7.1.29, PleskLin', 'X-Drupal-Cache': 'HIT', 'Content-Language': 'en', 'X-Frame-Options': 'SAMEORIGIN', 'X-Generator': 'Drupal 7 (http://drupal.org)', 'Cache-Control': 'public, max-age=1800', 'Expires': 'Sun, 19 Nov 1978 05:00:00 GMT', 'Vary': 'Cookie,Accept-Encoding', 'Content-Encoding': 'gzip', 'X-Content-Type-Options': 'nosniff', 'Etag': '"1572939088-1"', 'Last-Modified': 'Tue, 05 Nov 2019 07:31:28 GMT'}
import requests url = 'http://127.0.0.1:5000/cars' data = {'reg':'123','make':'blah', 'model':'blah','price':1234} response=requests.post print(response.status_code) print(response.json())
201 {'car':{'make': 'blah', 'model': 'blah', 'price': 1234, 'reg': '123'}}
with open(filename, 'w') as f: json.dump(data, f, indent=4)
f = open("demofile.txt", "rt") Data= json.load(f)
from xlwt import * w = Workbook() ws = w.add_sheet('cars’) ws.write(0,0, "data1") row = 1 col = 1 ws.write(row,col, "data") w.save('cars.xls')
parameter
parameter
- Just like any other bit of data
import requests import json
#html = '<h1>hello world</h1>This is html' f = open("../../week02/carviewer2.html", "r") html = f.read() #print (html) apiKey = '46ceed910c24ff7cce8240e89ec7b71912f6f40f2ec55fd217ce150ad6d4f1c4' url = 'https://api.html2pdf.app/v1/generate' data = {'html': html,'apiKey': apiKey} response = requests.post(url, json=data) print (response.status_code) newFile = open("lab06.02.01.htmlaspdf.pdf", "wb") newFile.write(response.content)
requests.get(url, auth=('token',apiKey))
import requests import json apiKey = 'f59b1cdb743ce0ab9b21a94ac246b02f06ee2d85' url = 'https://api.github.com/repos/datarepresentationstudent/aPrivateOne' filename ="repo.json" response = requests.get(url, auth=('token',apiKey)) repoJSON = response.json() #print (response.json()) file = open(filename, 'w') json.dump(repoJSON, file, indent=4)
python -m venv venv .\venv\scripts\activate.psl source venv/bin/activate
virtualenv env \path\to\env\scripts\activate
To save all the packages
pip freeze pip install flask pip freeze > requirements.txt
To install them again
pip install -r path/to/requirements.txt
#!flask/bin/python from flask import Flask app = Flask(__name__) @app.route('/') def index(): return "Hello, World!" if __name__ == '__main__' : app.run(debug= True)
python b_restserver.py
export FLASK_APP=b_restserver flask run
set FLASK_APP=b_restserver flask run
pip install flask_cors
from flask_cors import CORS
@app.route('/user’, methods=['GET', 'POST'])
@app.route('/user/<username>') @app.route('/user/<int:post_id>’)
@app.route('/') def index(): return 'index’ #somewhere else print(url_for(index) # will print out /
As oppose to
To access request data import request object
from flask import request
See the code we did in week05
"reg": request.json['reg'],
Other data you can get
from flask import abort, redirect, url_for @app.route('/') def index(): return redirect(url_for('login')) @app.route('/login') def login(): abort(401) this_is_never_executed()
https://buildmedia.readthedocs.org/media/pdf/flask/latest/flask.pdf
SHOW DATABASES; CREATE DATABASE test; USE test CREATE TABLE test{ id INT NOT NULL AUTO_INCREMENT, name VARCHAR(250), PRIMARY KEY(id) }; INSERT INTO test (name) VALUES ('jow'); SELECT * FROM test; UPDATE test SET name='blah' WHERE id = 1; DELETE FROM test WHERE VALUE =1;
pip install mysql-connector
import mysql.connector mydb = mysql.connector.connect( host="localhost", user="root", password="" ) mycursor = mydb.cursor() mycursor.execute("CREATE DATABASE datarepresentation ")
sql = “insert into student (name, address) values (%s, %s)” values=(“Mary”, “Galway”)
import mysql.connector db = mysql.connector.connect( host="localhost", user="root", password="", database="datarepresentation" ) cursor = db.cursor() sql="insert into student (name, address) values (%s,%s)" values = ("Mary", "Galway") cursor.execute(sql, values) db.commit() print("1 record inserted, ID:", cursor.lastrowid)
# get connection code here cursor = db.cursor() sql = "Select * from student where id = %s" values = (1, ) cursor.execute(sql, values) result = cursor.fetchall() for x in result: print(x)
def findById(id): foundBook = studentDAO.findByID(id) return jsonify(foundBook)
mysql ={ 'host':'localhost', 'user':'root', 'password':'my secret password', 'db':'write-math' }
import databaseconfig as cig connect(cfg.mysql['host'], cfg.mysql['user'], cfg.mysql['password'])
ToDo