This article is inspired by another post described the disapproving on current common misunderstanding among concepts of RESTful, CRUD, Safety, Security and Idempotency. [RESTful Img][https://i.imgur.com/tS7bZ9im.png] This post targets to rephase RESTful concept and describes common check points against RESTful API architecture test during Software Design Review.As practical samples, Jenkins API and CircleCI/TravisCI API will be analyzed, tested and evaluated.
1 RESTful Style Design
1.1 RESTful Maturity Model and the Background
\(\mathsf{\color{olive}{✐\ RESTful}}\): REpresentational State Transfer, by Roy Thomas Fielding Phd Dessertation, 2000.
RESTful is a style of architecture. It is also called Resource Oriented Architecture (ROA) Which defines a group of restrictions for common architecturing practices as a meta architecture. RESTful compliant deliveries take the conventions of restrictions to build self-describable (semantic) network all together.
Approaching with RESTful is like a ceremony to take a breakfast coffee ☕ in the morning ☀ to start the day with greeting and smile to people in community while exchaning news, sports and weather. The network is community which Apps and Services live within and people in ceremony are the nodes compliant to RESTful. Those common restrictions release the stress of life and offer people dimensions of freedom to move up (evolve, or, live with happiness) and exchange information.
Traditionally a common style of API service architecture focused on operations. For example, the client creates the form and fulfills it with operation code, data and adjacent properties (session, auth) then sends it to server and expect a state transition on server. It creates a coupling between client and server, hardly to keep URL consistent in system evolution, and, operation focused design brings side effects as stateful constrain, cache application issue.
While RESTful service is designed against resources. In short, URI is only designed for resources, Aka. design only servers data models. Which leaves the freedom to system evolution in three dimensions: service side, client side and resource linkages(resource descriptions). In concept, RESTful is a series of constrains towards an open architecture for next 10 years. Therefore the key points are:
- Stateless: Server side openness ☞ A dimension for Server side to evolute within the distributed system.
- Hyper-media driven: Business logic openness ☞ The dimension of freedom to support business evolution. Clients start from limited entry points and the requests are driven by hyperlinks in between.
- Uniform Interface: Client side openness ☞ A dimension for Client side to evolute and separate implementation details.
HTTP is the only implementation base so far. In common practices, the verbs and constrains on verbs are mixed from HTTP or RESTful. In other words, in context of concrete implementations, some conventions are borrowed from HTTP constrains.
With HTTP and the using of HTTP Methods, RESTful also offers a kind of transparency to the network. Cache support is a widely mentioned feature comparing to traditional SOAP. With SOAP, the intermediate nodes can hardly know whether the operation is cache friendly unless it decode the message and find the Action Code within XML body. The existing HTTP headers as "modified-after-*" is not reusable with SOAP. But RESTful architecture can reuse existing infrastructures of Web Services and Network. It is friendly to existing Load Balance, CDN/Cache, and, Health-check facilities.
From a practice point of view, Infosys published a handbook describing three wheels of RESTful: the Resources(URI)1, Verbs(Methods) and Representation (Format). Before starting practicing, my recommendation is still to think more on what is resource and how the resources are represented in such a semi-transparent and open architecture sytle.
\(\mathsf{\color{olive}{✐\ RESTful\ Maturity\ Model}}\): 4 Levels of RESTful Maturity definition to show the way to improve and where we are, by Leonard Richardson, 2008.
[RESTful Maturity Model][https://i.imgur.com/3f6qnOEm.png]
It is still debating whether the first three levels are within RESTful scope. Because the maturity model might mislead people to understand the objectives of RESTful to build an eco of HATEOAS. At least, a common understanding is the maturity model provides a way to evaluate on a growth view towards semantic network.
It is said that a large part of current online so-called RESTful services are quick works moved to HTTP verb driven, not hyper-media driven. And they are designed against operations, not resources.
Level 0: Just one URI, as XML-RPC
The HTTP headers have no relation with transaction data. It is just a sharing point. All business and model logic is encoded in payload with a tight coupling between clients and servers. The client must have internal knowledge on how server works.
Level 1: Multiple URI but one method
The RESTful degrades from Application Protocol to a Transportation Layer. As in SOAP or XML-RPC, an action code is in payload like "Do.Action" and "Operation=**".
Level 2: Multiple URI, Multiple method
Methods and Status Code in Headers are representing transaction logic.
Notes: > CRUD: Post, Get, Put/PATCH, Delete > PUT: Update the whole object, while, PATCH: Update the properties of object. > HEAD: Fetch meta data. > OPTIONS: Fetch which properties can be altered
Level 3: Hypermedia as Application Engine -- HATEOAS
Leave the server side a freedom of service evolution. The application just enters from the entry point and will triger State Transfer on server objects via a hyperlinks.
GET: can use filter, paging, not idempotent, POST: seems better and it return links to results.
Brief History
When RESTful was published 2000, it was ahead of time. With more and more services were delivered in the name of RESTful, the situation was not getting better towards self-describable network.
When the first internet bubble was over, more studies were spent on this topic. The popular Rails makes CRUD a widely accepted concept in a narrowed context. Together with ROR is the dominancy trend of various application clients than browser. Mobile traffics exceeded desktop from 2016 and Apps dominant mobile internet More people turn to review this topic (from 2014) and publish articles and posts towards core concept of RESTful. 
In short, CRUD is a selected implementation offered by most of modern Apps and Frameworks. It can be roughly regarded as a craft way between level-2 and HATEOAS. While RESTful is on a higher abstract layer to define the architecture style.
1.2 Applications
Two kinds: Browser and Other Application. The Clients details are protected from coupling up with distributed systems. However, it can be checked through the User-Agent header.
1.3 Verbs in HTTP
1.3.1 Safety and Idempotence
\(\mathsf{\color{olive}{✐\ Safety}}\): The operation won't change resource state on server side. Therefore if a safe operation is performed once or multiple times, the server state does not change as client never make the operation at all.
Tips: Client does not know whether the server really does not change anything. Here safety is on service layer of perspective. The client does not couple up with whether server keeps audit, logs or runs an iterator as internal implementation. From a service layer view, if the resource remains, the uniformed interface with curren method is safe. So does Idempotence.
Notes: > Safety vs Security: In Chinese language, the two concepts are not distinguished by default. The two terms defined different scope in general English context. Safety refers to the infrastructural problems as climate disaster, earthquake, or, severes threats without indistinctive targets. Security is a bit narrow concept on individual level protection from explosing, targeted attacks.
Examples: GET and HEAD are safe methods. POST is not safe.
\(\mathsf{\color{olive}{✐ Idempotence}}\): The operation on resource(s) by making one request is the same as making a series of identical requests. This concept is from Maths. When an operation has same effect whether it is applied once or more than once. For example,\(4\times0\times0 == 4\times0\).
Examples: GET, PUT and DELETE are idempotent. POST is not idempotent, neither.
HTTP v1.1 defines idempotence as: >Methods can also have the property of "idempotence" in that (aside from error or expiration issues) the side-effects of N > 0 identical requests is the same as for a single request.
In distributed systems, idempotence is a key feature to decouple systems to eliminate strong constrains. For example, withdraw money from a deposit
Safety and idempotency allow clients to make reliable request over unreliable network environment. If the operation is safe, it can be cached and mutiplied without worry about side effects. If it is idempotent, the request can be replayed in unreliable environment.
\(\color{brown}{✓\ Misusing\ GET\ for\ Unsafe\ Operations}\) There is a common checkpoints for testing team to pay attention if safe metod as "GET" is misused to expose unsafe interfaces.
1.3.2 Summary of HTTP Methods
| Method | Safe | Idempotent |
|---|---|---|
| POST | no | no |
| GET | yes | yes |
| HEAD | yes | yes |
| PUT | no | yes |
| PATCH | no | no |
| DELETE | no | yes |
| LINK | no | yes |
| UNLINK | no | yes |
Notes: > PATCH: Unlike PUT, PATCH is not idempotent, both are not safe. Which is to say, PATCH is not required to be idempotent as I updates selected properties. Client shall not assume PATCH as idempotent operation. However, in purticular context, PATCH can be idempotent with headers "If-Match" or "If-Unmodified-Since".
Another point, PATCH is not natively defined by HTTP specification. Which is introduced by RFC 5789, 2010 as an extension for Web APIs.
GET: Safe and Idempotent. It is a hint to test cache via proxy or packet analyzer.
POST: Two cases for POST method: Post-to-append (appending POST) and Post-to-overloaded (overloading POST). Post-to-append is the HTTP request to create a new resource beneath another resource.
POST_and_PUT: POST regards the URI as resource "acceptor", not the resource itself to create.
POST https://api.forum.com/articlesis to create an article "for" articles resource, or, owned by articles. In the return result links to the new resource shall be indicated. The 2nd POST request will create another URI for articles. Therefore POST is not safe, non-idempotent.PUT is idempotent.
PUT https://api.forum.com/articles/101is to update or create the resource "in" 101 URI. Multiple POST actions make no difference here. Hence PUT is idempotent method by definition.
HEAD: Similar to GET but only has HTTP headers.
OPTIONS: It is an HTTP native exploring verb. The return result holds a header of
which defines all methods current resource supports.
LINK/UNLINK: It is not recommended to adapt LINK/UNLINK so far since it is not popular. The LINK/UNLINK are proposed in an IETF draft.
1.4 HTTP Status Code
| Range | Transition Information | Notes |
|---|---|---|
| 1XX | Information | 100 - Continue |
| 2XX | OK | 201 - Created 204 - No Content |
| 3XX | Redirection | 301 - Redirect Permanent 302 - Redirect Temporary |
| 4XX | Client Error | 400 - Bad Request (Malformated) 401 - Unauthorized 403 - Forbidden 404 - Not Found 405 - Now Allowed (wrong method) 409 - Resource Conflict |
| 5XX | Server Error | 501 - Not Implemented 505 - HTTP Version Not Supported |
It is widely seen in practices the return status code is only checked whether it is 2XX as in many front end applications. On syntax level, this is effective. However, it is recommended to apply semantic checks as well. For example, it the request is to Create an entity, then the status code in the response header is expected to be "201". On the other hand, when the request is to Delete an entity, the status code is expected to be "204".
From the architecture view, if a Front-End App triggers a state change via request to a service or micro-service, and the payload is transparently carried over to downstream micro-services. When the downstream service reports an payload error as 4XX (client side error), what would be returned to the Front-End App? It will depend on the information disclosure from overall strategy. For many of the applications, the Front-End runs in Browser or Mobile are not permitted to disclose more information for security sake. Therefore, it is possible to see 5XX errors on Front-End instead.
Notes: Here is a complete decision diagram on HTTP status code. For a quick reference on HTTP status code, readers are recommended to visit httpstatuses.com
2 Testing RESTful APIs
In current section, the check points list is shared. It will be maintained lively. The list shall be able to apply in architecture review and testing works directly. I would try to add more concrete examples in following days.
✓ HTTPs protocol always
As common practice, CircleCI, travis-ci and Github are on HTTPs. However there are still couple of issues to pay special attention on security (refer to previous tips section in this post on safety vs security).
- Certificate verification between peers, which can reuse HTTPs infrastructure;
- API keys to offer an RBAC or other access control;
- Allow one client to work with multiple API keys;
- Adapt API-key + Secret-Key, sign key value pairs before URL encoding;
It is recommended to take a further step from API-key authentication to API-Key + Secret-Key authentication with a convention of Request Signature. It is hard to say which signature procedure is "right". There are several good practices to refer as Amazon AWS API parameter signature (v2.0, v1.0 is insecure), Baidu API and other famous providers. Here is a sample from developer.baidu.com. The URL parameters are signed with timestamp in predefined format and key value sorted in ascending order with session_secret appended. A kind reminder is to add timestamp as mentioned common practice to ignore expired requests. This protects services from Replay and Man-In-The-Middle attacks.
GET /rest/2.0/passport/users/getInfo?session_key=9XNNXe66zOlSassjSKD5gry9BiN61IUEi8IpJmjBwvU07RXP0J3c4GnhZR3GKhMHa1A%3D×tamp=2011-06-21+17%3A18%3A09&format=json&uid=67411167&sign=d24dd357a95a2579c410b3a92495f009 HTTP/1.1 |
Here is code sample from AWS on how to sign requests to AWS API. http://docs.aws.amazon.com/general/latest/gr/sigv4-signed-request-examples.html?shortFooter=true
- CircleCI API runs on HTTPs, but not with signed request.
- Travis-CI takes token from HTTP auth header also.
✓ Prefer to locate API to subdomain as api.dome.com instead of domain.com/api
This piece of tips makes APIs friendly to the network including Routers, Firewalls and Domain security infrastructures.
- CircleCI API service is not on subdomain,
https://circleci.com/api/v1.1/ - Travis-CI runs service on
https://api.travis-ci.com/
✓ Combine version to URI, e.g. api.domain.com/v1
- CircleCI URI combines up version information.
- Travis-CI accepts API version from HTTP Header.
✓ Use nouns but not verbs.
- Plura nouns in most time
- Nouns in plura reflect table names in DB
Take below example from CircleCI API doc, cancel here is regarded as a resource like command queue. Travis-CI has similar interface /job/{job.id}/cancel. Here job.cancel will accept a resouce of command with POST.
POST: /project/:vcs-type/:username/:project/:build_num/cancel |
✓ Verbs (Methods) correctly utilized
An example from Travis-CI API doc about 'PATCH' method to update properties:
PATCH |
- CRUD: Post, Get, Put/Patch, Delete
- Head/Options
- Get is idempotent (with all parameters)
✓ Filtering, Sorting, Selecting, Limit/Pagination Supports
An example of RUI with pagination api.domain.com/v1/zoos?id=2&limit=10
- Travis-CI offers pagination supports
✓ Status Code
As mentioned in section 1.4, status codes are expected to be compliant to HTTP convention.
✓ Error Handling
RESTful service returns verbose error information in Payloads, e.g. {error: "XXX"}
- Travis-CI error message example:
{
"@type": "home",
"@href": "/",
"errors": {
"login_required": {
"status": 403,
"default_message": "login required",
"additional_attributes": [ ]
},
"method_not_allowed": {
"status": 405,
"default_message": "method not allowed",
"additional_attributes": [ ]
},
"not_found":{
"status": 404,
"default_message": "resource not found (or insufficient access)",
"additional_attributes": [
"resource_type"
]
}
}
}
✓ Return Result compliancy
- A set or an obj.
- Sub resources for relations:
GET /cars/711/drivers/ Returns a list of drivers for car 711 |
✓ Hypermedia context
Provides Hyperlink driven APIs as manual exploring with the services.
✓ Allowing Overriding Methods
Use X-HTTP-Method-Override in cases some proxies only supports POST and GET.
3 Test the RESTful Service
(TODO)
4 DevOps on RESTful Service
Dashboard on Delivery Pipeline
Dashboard on E2E RESTful Status
(TODO)
5 References
Roy Thomas Fielding Phd Dessertation
Leonard Richardson Personal Site 10 Best Practices for better RESTful
REST Cookbook mobile web vs app
HTTP Status Code Decision Diagram in High Resolution
Principles for Standardized RESTful Authentication
6 Change Logs
Jun 02, 2019: Fix image links and update HTTP status code section. May 29, 2017: Add security check point examples. May 27, 2017: Minor change, updated pictures. May 23, 2017: Updated HTTP methods and Status Code overview. May 20, 2017: Initial the draft.
7 Terms and Notes
URI - The Uniform Resource Identifier has two forms: URN and URL. URN maps resources by unique names and URL locates resources with unique locations.↩︎