scheme:[//[user:password@]host[:port]][/]path[?query][#fragment]
- Describe content and structure of a webpage.
- Made up by elements marked by tags
<element>[content]</element>
- required tags
thirdLevel
<!DOCTYPE html>
<html>
thirdLevel<body>
- Designed to support modern multimedia
- Describe appearance, layout and presentation of info
- Describe how info is to be displayed
<style> or separate .css file
<link href="styles.css" rel="stylesheet" />
- syntax
css selector { property: value; property: value; }
- Bowser's default styles -> external css (in a
<link>
tag) -> embedded<style>
tag -> inline style (style="name:value;"
) ->!important
- id provides a unique identifier for an element
- class provides a general way of accessing certain elements
- pseudo-classes: a:hover
- Each element: content, padding, border margin
width
= content width + L/R padding + L/R border + L/R marginheight
= content height + T/B padding + T/B border + T/B marginwidth
: percentage or pixels (can apply max-width, min-width)float
: remove element from normal document flow- Need to
clear
element that follow a floating element
- Protocols:
- Application layer: FTP, HTTP, SSH, IMAP
- Transport Layer: TCP, UDP
- Internet Layer: IP
- Link Layer: Ethernet, Wi-Fi
- IP:
- Connectionless protocol
- Transfer packets from source address to dest
- TCP/IP:
- TCP: Transmission Control Protocol
- Connection Oriented
- Reliable
- TCP: Transmission Control Protocol
- Application Layer Protocols
- One piece of an app
- Defines messages exchanged by apps
- Uses services provided by lower layer protocols
- Application:
- Running on network hosts in user space
- Exchange messages to implement app
- API:
- Application programming interface
- Define interface between application and transport layer
- Socket:
- Send, receive
- HTTP:
- Hypertext Transfer Protocol
- Sits on top of TCP
- Transfer objects between client and server
- Process:
- http client initiates TCP connection on http server
- http server accepts the connection, notify client
- client sends http request message into TCP socket
- http server receives request message, form response message and sends into socket
- HTTP is stateless:
- Does not maintain status info across client reqs
- No way to correlate multiple req from some user
- Protocols that maintain state are complex
- History must be maintained
- If server of client crashes, their view of state may be inconsistent and must be reconciled.
- Simple server
- Not related to java
- Always
"use strict"
- Immediately invoked function expression
(function defining)();
- no function name to display in stack trees
- can't refer to itself for recursion or unbinding and event handler
return
var point = {x: 10, y: 20};
var point = {"x": 10, "y": 20};
- is a container of properties (
name: value
pairs) - Methods in Objects:
- (
methodName: functionName
) -Object.methodName()
- Adding properties:
Object.newProperty = newProperty
- Different from other programming languages
- Refers to the containing object of the call-site of a function, not where the function is defined
- Not eligible for this binding under
"use strict"
- [IMAGE]
- Lost binding:
- [IMAGE]
- Explicit binding:
- [IMAGE]
- Document Object Model
- [IMAGE]
- Browser Object Model
- [IMAGE]
window
- Top level object in BOM
- Methods:
thirdLevel
alert
,confirm
,prompt
setInterval
,setTimeout
,clearInterval
,clearTimeout
open
,close
thirdLevelblur
,focus
,moveBy
,moveTo
,print
,resizeBy
,resizeTo
,scrollBy
,scrollTo
document
:- Current page and its elements
- Properties:
- anchors, body, cookie, domain, forms, images, links, referrer, title, URL
- Methods:
getElementById
getElementsByName
getElementsByTagName
close
,open
,write
,writeln
location
:- URL of the current page
- Properties:
- host, hostname, href, pathname, port, protocol, search
- Methods:
thirdLevel
assign
,reload
,replace
navigator
- Information about the browser
- Properties:
- appName, appVersion, browserLanguage, cookieEnabled, platform, userAgent
screen
- Info about display screen
- Properties:
availHeight
,availWidth
,colorDepth
,height
,pixelDepth
,width
- [IMAGE]
- a library to make it easier to manipulate DOM
- traversal
- event handling
- animation
- Ajax
- Syntax
$(selector).action()
$("#id")
,$(".class")
,$("tag")
$("[attr]")
to select all elements with a certainattr
$("[attr=value]")
to select all elements which attr = value$('this')
to set current element- like CSS
- actions
- getter
- read the selection
- operate on the first element in the selection
- setter
- change the selection
- operate on all elements in the selection (implicit iteration)
- return the selected element so they can be chained
- getter
- Event delegation - [IMAGE]
- Asynchronized Javascript and XML
- Name of a great person
- An Ajax call:
- User interaction invoke an event handler
- Event handler create XMLHttpRequest obecjt
- XMLGrrpRequest object requests a page from server
- Server retrieves appropriate data based on the page requested and sent it back
- XMLHttpRequest object fires a callback when data arrives, often a callback function
- Callback event handler processes the data and updates the DOM accordingly
- [IMAGE]
- usage
- Webstore store data as XML files
- Databases that return query results as XML
- Web services that use XML to communicate
- a universal format for exchange of data
- can use schema or DTD to describe structure/markup
- standardize integrity and interoperability
- language independent convention for formatting data as a set of JS objects
- value can include an ordered list of values (array) Ajax and JSON
- JSON.parse(string)
- JSON.stringify(object)
$.ajax({
url: "url",
data: {},
datatype: type,
success: function() {
JQuery
}
});
- allow submission of data to the server
- GET and POST
- GET: best for small amounts of insecure data.
- Asks a server for a page or data
- Parameters are sent in URL query (
?name=value&name=value
)
- POST: large chunks or sensitive data
- Retrieves the server's response
- Parameters are embedded in request's HTTP packet
- GET: best for small amounts of insecure data.
- Special characters will be encoded for URL query
- Automatically encoded by browser
<input type="file" name="avatar">
- Ensuring the necessary values when the field is required
- Preventing blank
- Ensuring type of values
- Format and range
- Ensuring confirmation fields fit together (retype password/email)
- Approaches
- HTML 5 validation
- Client-side
- Server-side
- Provide appropriates and useful feedback on front-end to end-users
- On-submit or on-the-fly
- Static hints beside fields
- Provide tooltips (on-hover tips)
- Provide dynamic tips (shown as user interacts with a field)
- CAPTCHA
- Completely Automated Public Turning test to tell computers and Humans Apart
- Problems easy for humans to solve, but hard/impossible for computers to solve
- Let to spam-sponsored CAPTCHA farms
- reCAPTCHA
- Put all that free labor to good use
- Translate real words in images and archival texts
- NoCAPTCHA
- Ask to click a box (I am not a robot)
- HTTP simply allows request a single document from a web server, doesn't remember or keep track of anything between invocations (short-lived)
- Every resource that is accessed via HTTP is a single, one-off request with no threaded connection between them
- Need:
- A way to remember/track states
- Uniquely identify each client from the server
- A way of providing custom content for each client
- Options:
- Client mechanisms:
- Cookies
- Hidden Variables
- URL rewriting
- HTML5 local storage (HTTP 1.1)
- Server mechanisms:
- Session
- Client mechanisms:
- Cookies
- Small piece of data sent by server to a browser, and sent back by browser on future page requests.
- Authentication
- User tracking
- Personalization: Maintain user preferences, shopping carts, etc.
- Get/set cookie manually by:
document.cookie=""
Cookies.set("", "");
- Keep track of state on the client
- Mostly only visible to application
- HttpOnly: don't allow Javascript to manipulate cookies (but still may be able to read them or learn of their existence)
- Secure: Only send over HTTPS
- Doesn't seem to be best practices yet
- Lifetime
- Session cookies by default are temporary cookie stored only in browser's memory. Erased when browser closed.
- Can't be used for long-term tracking
- Safer, because only the browser has access
- Persistent cookies are stored in a file on the browser's computer and can track long-term info
- Potentially less secure because users/programs can open cookie file, see/change cookie values
- Problems and Limitations
- Browsers can disable cookies.
- Users can delete and disable cookies.
- Size limit / expiration policy
- Security issue: stored in plaintext
- Privacy issue: ID of the users
- Not handling cookie expiry well
- Small piece of data sent by server to a browser, and sent back by browser on future page requests.
- Hidden Variables
- Store state info in webpages
- Cross-browser support
- For form-based apps only - Change of URL losses the state - Irrespective of what was submitted previously
- URLS
- Store state in the URL such that URL becomes a GET request
- Supported by all browsers
- Requires all URLs contain all state info
- Current submitted page represents current state
- Independent of what was done previously
- Example - [IMAGE]
- Sessions:
- A server-side option
- Store the current states on the server
- Each request includes a token identifying browser's session - Tokens can be passed via cookies, hidden variables, URL rewriting - At each request, executing script uses the token to fetch the session state. - ##Session Hijacking
- HTML5 local storage
- Store data with no expiration date
localStorage.setItem("lastname", "Smith");
localStorage,getItem("lastname");
- session storage
- store data for one session (lost when browser tab is closed)
- REpresentational State Transfer
- Architectural style for using HTTP to provide resources over the web
- Focus on roles and actions
- Uniquely access resources through URLs
- Use GET, POST, PUT, DELETE
- User standard data format: HTML, XML, JSON
- Stateless protocol
- Benefits
- Performance (light weight)
- Scalability due to client-server separation
- Simplicity (nouns and verbs)
- Visibility of communication
- Portability (platform independent)
- Reliability at the system level
- REST verbs
GET
: Read a specific resource (by identifier)PUT
: Update or create a specific resource (by identifier)DELETE
: Remove a specific resourcePOST
: Create or Update a resource- Need two basic URLs per resource:
- Collection of items in the resource
- Select a resource
- RESTful API design
- Use logical URLs that are human-understandable and don't point to a file
- If dealing with a lot of data provide a paging mechanism
- Document everything and provide instructions
- Use POST (not GET) to make a change
- Provide multiple output data formats (JSON, XML, CSV, RSS. HTML)
- Use authentication if your API allows change/deletion/adding
- Server-side
- Process HTTP requests, and output a combination of HTML, JSON, JS to client
- event-driven, I/O model-based runtime environment and library for developing server-side web apps using JS.
- Threads
- Event
- Event Queue
- Inner loop
while (true) { if (!eventQueue.notEmpty()) { eventQueue.pop().call(); } }
- Event Queue
- Get the same JS on browser and server
- Don't need DOM on the server
- Add events and an event queue, everything runs as a call from event loop. (See above)
- Make event interface to all OS operations (???)
- Wrap all OS blocking calls
- Add some data handling support
- Add a proper module system
- Each module gets its own scope
- Module
- Import with:
- System module:
require(“module name”); // looks in nodes_module
- From a file:
require(“./^^^.js”); // reads specified file
- From a directory:
require(“./myModule”); // reads myModule/index.js
- System module:
- Modules have private scope
- Require returns what is assigned to module.exports
- Many standard Node modules
- Huge library of module (
npm
) - We use:
- Express (framework)
- Mongoose (MongoDB object modeling)
- Import with:
npm
npm init
- creates package.json
npm install module –save
- for any new modules you wanna install
.gitignore
- ignore files for git
- Programming with events/callbacks
- Threads: Blocking/waiting is transparent
r1 = step1(); console.log('step1 done', r1); r2 = step2(r1); console.log('step2 done', r2); r3 = step3(r2); console.log('step3 done', r3); console.log('All Done!');
- Events: Blocking/waiting requires callback
step1(function(r1) { console.log('s1 done', r1); step2(r1, function (r2) { console.log('s2 done', r2); step3(r2, function (r3) { console.log('s3done',r3); console.log('All Done!'); }); }); });
- Listener/emitter pattern
- When programming with events, a listener/emitter is used.
- Listener - Function to be called when the event is signaled
- Emitter - Signal that an event has occurred
var events = require('events'); var myEmitter = new events.eventEmitter(); myEmitter.on('myEvent', function(param1, param2) { console.log('myEvent occurred with ' + param1 + 'and' + param2 + '!'); // like wait }); myEmitter.emit('myEvent', 'arg1', 'arg2'); // like signal
-
relatively thin layer on top of the base
node.js
functionality.var express = require('express'); var expressApp = express(); expressApp.get('/', function (httpRequest, httpResponse) { httpResponse.send('hello world'); }); expressApp.listen(3000);
-
http methods:
expressApp.get(urlPath, requestProcessFunction); expressApp.post(urlPath, requestProcessFunction); expressApp.put(urlPath, requestProcessFunction); expressApp.delete(urlPath, requestProcessFunction); expressApp.all(urlPath, requestProcessFunction);
- Can contain parameters (e.g.
'/user/:user_id'
)
- Can contain parameters (e.g.
-
http
req
objectreq
ofexpressApp.get('/user/:user_id', function (req, res)
request.params
- Object containing URL rout params (e.g.user_id
)request.query
- Object containing query params (e.g&foo=9
=>{foo: 9}
)request.body
- Object containing the parsed bodyrequest.get(field)
- Returns the value of the specified HTTP header field
-
http
res
objectres
ofexpressApp.get('/user/:user_id', function (req, res)
res.write(content)
- Build up the response body with contentres.status(code)
- Set the HTTP status code of the replyres.set(prop, value)
- set the response header property to valueres.end()
- End the request by responding itres.send(content)
- Do write and End
- collection of tables with rows and columns
- Each row represents a record
- Each column represents an attribute of the records contained in the table
- Why
- Data safety
- Concurrent access
- Fault tolerance
- Data integrity
- Scalability
- Reporting
- Atomic
- All operations in a transaction complete, or none do
- Consistent
- On completion of transaction, database is sound (?)
- Constraints of the database are applied and database is in a valid state
- Isolated
- transactions don't interfere with each other
- Durability
- results of a transaction are permanent even in the case of failures
-
Structured Query Language
-
A special purpose programming language
-
Data definition, manipulation, and controls.
- Insert, Query, Update, delete
-
Schema creation and modification
-
Examples:
- SQLite
- MySQL
- PostgreSQL
- Access
- PointBase
-
Keys
- Primary Key: organize data around accesses
- Uniquely identifies a row in the table
- Secondary Key: Other columns that are having unique values
- Foreign Key: The Primary Key defined in another table
- Primary Key: organize data around accesses
-
Schemas
- Define the structure of the database
- tables, column indices
- Needs to be defined before adding data
- Not great match with agile development approaches (Each new feature may require a schema change)
- Database migration allow incremental changes
- NoSQL databases are built to allow the insertion of data without a predefined schema
- Define the structure of the database
- Do not use relational model or SQLite
- Flexible schema
- Quicker/cheaper to setup
- Scalability
- No declarative query language -> more programming
- Relaxed consistency
- -> Higher performance and availability
- -> fewer guarantees
- Document types apply a key to a "document" with a complex data structure
- Contain many different
key: value
pairs - Use standard data format (e.g. JSON, HTML)
- Can be organized by collection, tags, etc.
- Contain many different
- Relational databases usually scale vertically
- NoSQL databases usually support
- auto-sharding: natively and automatically store data across an arbitrary number of Servers
- automatic replication: High availability and disaster recovery without involving separate applications to manage these tasks
- caching: keep frequently used data in system memory and removing the need for a separate caching layer
- Basic Availability
- Most of the database is available most of the time
- Soft-state
- Stores don't have to be write consistent
- Eventual consistency
- At some later point, data stores will be Consistent
- NoSQL database uses a JSON-like (BSON) document oriented model
- Data is stored in collections (rather than tables)
- Dynamic schema
- Works with many programming languages
- Caching: Most recent kept in RAM.
- No transactions, but allows atomic operations.
- BSON
- Binary JSON
- MongoDB encodes JSON documents in binary format behind the scenes
- Extends JSON to provide data types
- Document validation provided within the database
- Connect to MongoDB instance
mongoose.connect('mongodb://localhost/myproject');
- wait for connection to Complete
mongoose.connection.on('open', function () { // Can start processing model fetch requests }); mongoose.connection.on('error', function (err) { });
- Can also listen for connecting, connected, disconnecting, disconnected
-
Defines collections
-
Types:
- String, Number, Data, Buffer, Boolean
- Array - e.g comments
[ObjectId]
- ObjectID - reference to another object
- Mixed - anything
-
Defining:
var userSchema = new mongoose.Schema({ first_name: String, last_name: String, emailAddresses: [String], location: String });
-
Secondary indices
- faster query
- slower mutating operations: Add, delete, update must update indexes
- Uses more space
- When:
- Common queries spend a lot of time scanning
- Need to enforce uniqueness
-
Making model from schema
var User = mongoose.model('User', userSchema);
-
Making objects from model
User.create({ first_name: 'Amy', last_name: 'Pond'}, doneCallback); function doneCallback(err, newUser) { assert (!err); console.log('Created object with ID', newUser._id); }
-
Queries
User.find(function(err, users) { /* users is an array of objects */ }); User.findOne({key: value}, function(err, user) { /* user is an object */ }) User.findOne({key: value}, function(err, user) { // Update user object user.save(); })
var query = User.find({}); // Projection query.select("first_name last_name") .exec(doneCallback); // Sorting query.sort("first_name").exec(doneCallback); //Limits query.limit(50).exec(doneCallback); query.sort("-location");
// Deleting one User.remove({_id: user_id}, function (err) { } ); // Deleting all User.remove({}, function (err) { } );
- Create, Read, Update, delete
- Simpler access model than SQL
- Object Definition Language (ODL)
- Higher level structures underlying a web app
- Made up of elements, their properties and relationships among elements
- separate the components of a web app into different tiers of layers
- Components:
- presentation: the static or dynamically generated content rendered by the browser (front-end)
- processing logic: dynamic content processing and generation, e.g. using node.js, PHP, Java EE, ASP.NET
- data management: database, comprising data sets and data management system (back-end)
- 1-tier
- All three layers are on the same machine and presentation, logic and data are tightly connected
- Scalability: single processor, no separation, hard to scale
- Portability: Moving to a new machine, or changing anything may mean major re-writes.
- Maintenance: Changing one layer requires changing other layers
- 2-tier
- Database runs on a separated server
- Presentation and logic layers are tightly coupled
- Can easily switch to a different Database
- Coupling of presentation and logic can lead to heavy server load and network congestion
- 3-tier
- Each tier is separate with a clear interface between them
- Each tier can potentially run on a different machine
- client-server
- unconnected tiers should not communicate (in our project, client never communicates with database server directly)
- change in platform affects only that tier
- Presentation Layer
- user interface and Interactions
- client-view or front-end
- should not contain business logic or data access code
- Logic Layer
- Set of rules for processing the data and accommodates many individual users
- No Presentation or access code
- Data Layer
- Physical storage for data Persistence
- Manages access to the DB or file system
- Also called back-end
- Benifites
- Easier Maintenance
- Reusable Components
- Faster development
- Model View Controller
- User uses Controller
- Controller manipulates Model
- Model updates View
- View seen by User
- Model manages application state by responding to reqs for info about its state (from view) or instructions to change state (from controller)
- Database
- Session info
- rules governing transaction
- View renders the model into a form suitable for interaction (UI). Multiple views can exist for a single model give different contexts
- front-end website (HTML, CSS, and server-side templates)
- Controller receives user input and initiates a response by making updates to the state of objects in the model and selecting a view for the response
- Client-side scripting that manages user interactions, HTTP request processing and business logic preprocessing
- Benefits:
- Clarity of design
- Separation of concerns
- Parallel development
- Distributable
- communication
- 3-tier:
- Presentation Layer never communicates directly with data layer (only through logic layer)
- MVC
- All layers communicate directly
- 3-tier:
- Usage
- 3-tier
- Client, middleware and data tiers are on physically separate platforms
- MVC
- model, view and controller are together
- 3-tier
- Mustache, Pug (Jade), EJS, Handlebars, (Our project used nunjucks)...
- Move Javascript out of HTML
- Separate logic and display
- Create a readable, maintainable language to write views
- Modularize views to reuse pieces
- User variables in HTML
- Goals
- Confidentiality
- integrity
- availability
- Risks
- Financial
- Reputation
- physical
- Threats
- Insiders
- Criminals
- Commercial competitors
- Nation-states
- Law Enforcement
- Vandals
- Directly on system:
- Stealing data, passwords, credit card Number
- defacing, DOS (denial of service), link spam
- On users
- cross site scripting, request forgery (CSRF), Man-In-The-Middle (MITM), profiling
- Both
- Hosting bad content, "drive-by download"
- Neither
- Ad-based malware
- Security mechanism underlying HTTPS
- Often called SSL
- Client and server use public-key encryption on shared per-session secret, then use secret to encrypt session data
- Always store passwords salted & hashed, use trusted algorithms (we used passport in our project)
SQL Injection:
-
Example 1
String query = "SELECT * FROM accounts WHERE custID='" + request.getParameter("id") + "'";
With a input of
foo' or 'x'='x
will formString query = "SELECT * FROM accounts WHERE custID=' foo' or 'x'='x '";
Which returns all customer information.
-
Example 2
SELECT email, passwd, login_id, full_name FROM members WHERE email = '?';
When replacing
?
withx';UPDATE membersSET email = 'steve@unixwiz.net' WHERE email = 'bob@example.com
will resultSELECT email, passwd, login_id, full_name FROM members WHERE email = 'x'; UPDATE members SET email = 'steve@unixwiz.net' WHERE email = 'bob@example.com';
Which replaces original email information with
'steve@unixwiz.net'
.
Prevention:
- User parameterized Queries
- Escape everything
- Validate input
- Never trust raw input
Vulnerabilities
- User credentials not stored using hashing or encryption
- Credentials can be guessed or overwritten
- Session IDs exposed in URL
- Session IDs don't timeout
- Auth tokens not invalidated on log out
- Passwords, session IDs sent over unencrypted Connections
- Auth framework that delegates user authentication to a service that hosts the user account
- Must register app with the auth service, including the URL to redirect to
- Auth service redirects to a registered URL with an authorization code
- Data enters web application and is included in dynamic content sent to a web user without being validated for malicious content.
- Usually JS but may also include HTML and other Data
- Types (3):
- stored
- injected script is permanently stored on server
- reflected
- DOM-based
- stored
Prevention:
- Escape all untrusted data
- Attacker is an authorized user
- Attacker can gain access to objects by guessing parameters
Prevention:
- Ensure user is authorized for access
- Software out of date
- Unnecessary features enabled
- default accounts and passwords still enabled / unchanged
- error handling reveal stack traces or overly informative error messages sent to users
- Security settings in development frameworks not set to secure values
Sensitive data:
- passwords
- credit cards
- personal info
Prevention:
- Encrypt all sensitive data
- Modify URL to gain unauthorized Access
- Internal ids appear in URLs
Prevention:
- Check access authorization on every object
- Not sufficient to simply not show privileged operations to unprivileged users in the UI
- Some Content on unrelated site includes a POST to app
- If a user of the app navigates to compromised site while logged into the app, malicious POST request can pretend to be the user and steal the user's info from app
Prevention:
- Include unpredictable token with each HTTP request (usually in a hidden field)
- Dev team / Dev Ops people must be vigilant
- Be careful what you include in your app
- Update software always
Apps user redirects or internal forwards where unvalidated parameters are used
- Callback function have been the main mechanism for managing asynchronous programming
- Callbacks can be hard to trance and reason about
- Promises are a different type of abstraction for managing asynchronous programming
- Promise is an object or function with a
then
method whose behavior confirms its specification - thenable is an object or function that defines a then method
- value is any legal JS value
- exception is a value that is thrown using the throw statement
- reason is a value that indicates why a promise was rejected
- States: pending, fulfilled, rejected
- settled: fulfilled or rejected
-
Registers a callback to receive either a promise's eventual value, or the reason it cannot be fulfilled
-
returns a promise
myPromise.then(handleResolve, handleReject); function handleResolve(data) { //handle success } function handleReject(error) { //handle failure }