juglouvain http revisited

HTTP revisited& some Java networking

Java User Group Louvain-La-Neuve @ EPHEC

20/11/2014

Marc Tritschler

24/10/2014 Copyrigth (c) Marc Tritschler 1

Program

1.Introduction

2.Internet Stack (reminder ?)

3.Java and the Internet stack

4.Coding time


PLEASE PLEASE INTERRUPT ME (IRQ-0 or any other )

1. Introduction

Already heard of Gopher ?

Internet = HTTP


Internet = HTTP

• Google• Facebook• Gmail• Yahoo• Youtube• Twitter• Amazon• …24/10/2014 Copyrigth (c) Marc Tritschler 4

HTTP

Almost EVERYTHING runs over HTTP

• HTTP ~ 75 % of traffic (http://www.caida.org/publications/papers/1998/Inet98/Inet98.html MUST

READ)

– WebServices (SOAP & REST)– HTML– AJAX– Email (webmail)

• Exceptions• Email (smtp/imap/pop3)• DNS• FTP• WebSocket which 'upgrades' from HTPP (previous JUG)


http://www.caida.org/publications/papers/1998/Inet98/Inet98.html



HTML, JS, GIF, MP4 … over HTTP


2. The Internet Stack

Forget about

the

7 layers OSI model


The Internet Stack (4 layers)

TCP/IP familly

HTTP

Physical Layer

SSL

80 443

Part of OS. C/C++

In the JRE. Java

Number of Job & Products Opportunities

ElectronicsAssembly


My App

Where's HTML in this Stack ???

DO NOT MIX DATA, API and PROTOCOL•Data (= contents = payload = BYTES)– Binary vs Text– HTML, CSS, XML, JavaScript, JPEG, MP4, …– Text Data Encodings (UTF-8)

•API vertical links (no bytes on the wire)•Protocol horizontal links•AJAX = JavaScript performing HTTP requests


TCP portshttp://fr.wikipedia.org/wiki/Liste_de_ports_logiciels

Well Known (0 – 1024)20, 21 FTP

22 SSH

23 Telnet

25, 110 SMTP/POP3

80 HTTP

53 DNS

137 … 139 NETBIOS

389 LDAP

443 HTTPS

Others (1025-65535)1521 Oracle DB

8080 http proxies, Tomcat


http://fr.wikipedia.org/wiki/Liste_de_ports_logiciels

HTTP versions

• HTTP 1.0 @DEPRECATED– each request/response new TCP connection (= exchange of 3 TCP packets

(SYN, SYN/ACK, ACK))

• HTTP 1.1 CURRENT– Keep TCP session

• HTTP 2.0 FUTURE (around DEC 2014)– Negotiation (1.1, 2.0, other protocols)– Close to 1.1 (methods, status codes, …)– Server Push– Fix HOL problem– Loads page elements in parallel over single TCP connection

http://en.wikipedia.org/wiki/HTTP/2 for more info24/10/2014 Copyrigth (c) Marc Tritschler 11

http://en.wikipedia.org/wiki/HTTP/2

HTTP Refresher • RFC/IETF Standards (read this only if …)• Simple request/response• Header + [Body]• Stateless• Bytes and Chars (use UTF-8 encoding)• Synchronous HALF-DUPLEX (request ALWAYS

initiated by the client remeber the problems for interactive games

• Can be verbose (http headers) (~600 bytes for simple Hello World)


HTTP Overview


REQUEST (GET, POST, …)

RESPONSE (CODE + [DATA])1xx : Informational - Request received, continuing process2xx : Success - The action was successfully received, understood, and accepted3xx : Redirection - Further action must be taken in order to complete the request4xx : Client Error - The request contains bad syntax or cannot be fulfilled5xx : Server Error - The server failed to fulfill an apparently valid request

Client Server

HTTP Request : methodshttp://www.w3.org/Protocols/rfc2616/rfc2616-sec9.html

• Safe (GET/HEAD) & Idempotents methods• GET, HEAD• OPTIONS• POST, PUT• DELETE• TRACE• CONNECT FREEDOM


http://www.w3.org/Protocols/rfc2616/rfc2616-sec9.html

HTTP Responses : Status Codes


• 200 OK• 400 Bad Request• 401 Unauthorized (WWW-Authenticate header)• 403 Forbidden• 404 Not Found• 407 Proxy Authentication Required (Proxy-Authenticate header)• 500 Internal Server Error

• Complete List

http://www.w3.org/Protocols/rfc2616/rfc2616-sec6.html#sec6.1.1

HTTP Headershttp://en.wikipedia.org/wiki/List_of_HTTP_header_fields

• A lot of "standards" and "non standards" defined … a little bit messy

• Firefox Dev console


http://en.wikipedia.org/wiki/List_of_HTTP_header_fields

HTTP Request ExamplePOST http://sghrsot.cc.cec.eu.int:1045/hermes/Proxy/1.17/DocumentWebServicePS HTTP/1.1Accept-Encoding: gzip,deflateContent-Type: text/xml;charset=UTF-8SOAPAction: ""User-Agent: Jakarta Commons-HttpClient/3.1Host: host1.domain1.company :1045Content-Length: 585

<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:typ="http://xx.xxxxxx.eu/sg/hrs/types"> <soapenv:Header/> <soapenv:Body> <typ:getDocument> <typ:header> <typ:userName>xyz</typ:userName> <typ:ticket>onetimeticket</typ:ticket> <typ:applicationId>myapp</typ:applicationId> </typ:header> <typ:documentId>080166e48102103b</typ:documentId> </typ:getDocument> </soapenv:Body></soapenv:Envelope>


HTTP Response ExampleHTTP/1.1 200 OKDate: Mon, 20 Oct 2014 16:12:22 GMTContent-Length: 9159Content-Type: text/xml; charset=utf-8

<?xml version="1.0" encoding="UTF-8"?><S:Envelope xmlns:S="http://schemas.xmlsoap.org/soap/envelope/"><S:Body><typ:getDocumentResponse xmlns:typ="http://xx.xxxxxx.eu/sg/hrs/types"><typ:document><typ:documentId>080166e48102103b</typ:documentId> … (stripped)</typ:getDocumentResponse></S:Body></S:Envelope>


Quizz Time

Guess number of HTTP requests per web site …


Example 1 : www.lesoir.be


HOW Many HTTP requests ?

Example 2: mon-programmetv.be



Example 3: www.google.be



Number of HTTP requests per single web site visited

1. 200 requests/responses for www.lesoir.be OMG !!!

2. It's full of advertisements (visible) and invisible personal tracking systems (cookies, javascript, re-directs, …)

3. js is evill

Conclusion : YOU ARE NOT ANONYMOUS


http://www.lesoir.be/

How your browser gets its proxy ?

• Web Proxy Autodiscovery Protocol


HTTP Advanced

• Authentication• HTTP Proxies• HTTP Tunnelling• HTTP Pipelining• HTTPS


HTTP Authentication(RFCs 2616, 2617, 7235)

BasicThe client sends the user name and password as unencrypted base64 encoded text. It should only be used with HTTPS, as the password can be easily captured and reused over HTTP.

DigestThe client sends a hashed form of the password to the server. Although, the password cannot be captured over HTTP, it may be possible to replay requests using the hashed password.

NTLM (Windows)This uses a secure challenge/response mechanism that prevents password capture or replay attacks over HTTP.


HTTP Authentication401 – Access Denied


GET /securefiles/ HTTP/1.1

HTTP/1.1 401 Access DeniedWWW-Authenticate: Basic realm="My Server"Content-Length: 0

GET /securefiles/ HTTP/1.1Host: www.httpwatch.comAuthorization: Basic aHR0cHdhdGNoOmY=

Client(browser)

Server

User types his/her password

HTTP Authentication407 – Proxy Authentication Required

• Same as 401 excepted proxy MUST return a Proxy-Authenticate header

• Browser asks user to type his/her password


HTTP Proxy/Reverse Proxy

• Proxy : local net internet• Reverse Proxy: internet local net


Client Client

Direct Connection

HTTP

HTTP

Proxyied Connection

Client Proxy ServerHTTP

HTTP Tunnelling


HTTP

CONNECT

Client Proxy ServerTCP

Port forwarding

HTTP Pipelininghttp://en.wikipedia.org/wiki/HTTP_pipelining


http://en.wikipedia.org/wiki/HTTP_pipelining

HTTPS

• HTTP over SSL• Secure Browsing ?– HeartBleed – SSL3.0 recently found weak– TLS 1.0 min– Root certificate


3. Java & The Internet Stack


?

Java and Internet

• Java is (my favorite) language to work @ application layer, up to TCP/IP … (wait next slide )

• Java has no access to protocols below IP (needs call to native libs, not in the HTTP scope)

• Don't underestimate the complexity of SSL interactions, even in Java !!!


Java and the Internet Stack


TCP/UDP

HTTP

Physical Layer

Socket API (java.net) or JSSE (javax.ssl)

IPv4 and IPv6

ICMP, ARP, DHCP, …

WebSocke

t

SMTP/POP3FTP

DNS

Web Services

53

80/443

25, 110

JavaMailjavax.mail

Web Browser

Implemented in the OS. Java has limited access via API

Implemented in OS or hardware. No 'direct' access

Available in Java SE

Open Source or future

My Application My AppONLY FOCUS ON YOUR BUSINESS

JRE

Linux

API vs Protocol

• API vertical links (no bytes on the wire)• Protocol horizontal links


Socket API(java.net)

• Most important (access • Server Sockets• Client sockets• Base for YOUR protocol !• Base for HTTP, SMTP, …


Socket API - Main Classes

• Socket & ServerSockethttps://docs.oracle.com/javase/7/docs/api/java/net/Socket.html (Java 7)

https://docs.oracle.com/javase/8/docs/api/java/net/Socket.html (Java 8 :-))

• URL• URLConnection• HttpURLConnection• …• java.net package http://docs.oracle.com/javase/8/docs/api/java/net/package-summary.html • Stack properties http://docs.oracle.com/javase/8/docs/technotes/guides/net/properties.html


https://docs.oracle.com/javase/7/docs/api/java/net/Socket.html

https://docs.oracle.com/javase/7/docs/api/java/net/Socket.html

http://docs.oracle.com/javase/8/docs/api/java/net/package-summary.html

SMTP/POP3(java mail)

https://javamail.java.net/nonav/docs/api/com/sun/mail/smtp/package-summary.html


SSL/TLS(java.net javax.ssl)

• Socket API(java.net) JSSE (javax.ssl)

• Sockets– (Client) Socket SSLSocket– ServerSocket SSLServerSocket

• HttpUrlConnection HttpsUrlConnection


Others

• WebSocket– http://www.websocket.org/

• Java Specifics– RMI– JMX

• Web Services– SOAP JAX-WS– REST JAX-RS


Part 3: Code Time

WARNING

Several packages and many classes

challenge is to use the right classes


Setup - Toolbox

• Developer– Java JDK (of course)– Editor (Eclipse, NetBeans, …)

• Client Side– Putty– Web Browser + DEV console !

(Chrome, IE, FireFox, …)– soapUI (Web Services)

• Server Side– Apache HTTP server (min)– Apache Tomcat

(recommended)– Full JEE (GlassFish, WildFly, …)

• Cloud– Red Hat OpenShift– …


https://github.com/tritschler/LLN_JUG/tree/master/2014_11_20

Example 1 – Echo protocol(ClientSocket & ServerSocket)

• No HTTP, directly over TCP

https://docs.oracle.com/javase/tutorial/displayCode.html?code=https://docs.oracle.com/javase/tutorial/networking/sockets/examples/EchoServer.java

DON'T DO THIS IN REAL LIFE


Example 1 - Echo


TCP

Physical Layer

Socket API (java.net)

IPv4

Echo (Client)JVM

TCP

Physical Layer

Socket API (java.net)

IPv4

Echo (Server)JVM

Hello

Hello IP

real data flow

logical flow

Example 2 – Basic Web Crawler(URL, HttpUrlConnection)

•Example 1 : no proxy•Example 2 : proxy + basic http authentication



Java HTTP Client App


TCP/UDP

HTTP

Physical Layer

Socket API

IP

My Application…

(JVM)

ANY HTTP Server(Apache, Nginex,

Tomcat, Jboss, Microsoft IIS, …) implemented in

any programming language (Java, PHP, C,

…)

ANY OS (Linux, Windows, Mac OS, …)

Example 3 – ServletNo networking code on the Server Side

• Servlet = java spec for writing the HTTP server side• No networking code ! (thanks to your AS)• Web.xml + class extends HttpServlet

1. Browser – Servlet

2. Browser – HttpTrace – Servlet

3. HttpUrlConnection (no proxy) – Servlet

4. HttpUrlConnection – HttpTrace – Servlet



Java HTTP Client App – Java Servlet


ANY HTTP Server+ Servlet Container

Apache Tomcat


ANY HTTP Client(Web Browser, …)


Example 4 – HTTP proxy

• Start local Tomcat• Start HttpTrace• Start Browser and point to localhost• Launch httpclient


Resources(on the web of course, over HTTP )


juglouvain http revisited

Internet

copyrigth c marc tritschler2

copyrigth c marc tritschler1

informational request

traffic http

server error

internet stackcoding

single tcp connectionhttp

currentkeep tcp sessionhttp