...
Streams vs. Resources
This page is meant to start as a conceptual design for a 2nd generation REST API for ESME. The idea is to learn from the discussions around and use of the original REST API.
REST API Design
...
In the design below, all parts of ESME are modeled as resources, in keeping with a RESTful approach. For things like message streams, this is not an optimal way to model these entities. I'd like to maintain the ability to interface with the full ESME API in a RESTful manner, while also encouraging the use of more optimal interface approaches when this makes sense.
The existing entities called out as resources below that should be available in some manner as streams are:
- Messages (user-specific) api/users/USERID/messages
- Messages api/messages
- Tags (api/tags)
- Conversations (api/conversations)
- Pools (api/pools)
- Searches ??
Our streaming resources (also thought of as delta-queue resources) are collections that are only additive. We have defined a general streaming interface that strives to be RESTful while prioritizing the performance benefits of streams.
References - Dev mailing list thread - http://www.mail-archive.com/esme-dev@incubator.apache.org/msg00976.html
Resource/Object/Stream Hierarchy
The above is based on a rough object hierarchy as follows:
- ESME API instance (api/)
- Session (api2/session)
- Users (api2/users)
- Specific User (api2/users/USERID)
- User's tokens (api2/users/USERID/tokens)
- Specific User (api2/users/USERID)
- Messages posted by logged in user (api2/user/messages) (1)
- Users followed by logged in user (api2/user/followees)
- Users following logged in user (api2/user/followers)
- Trackers belonging to logged in user (api2/user/tracks)
- Messages from a track (api2/user/tracks/TRACKID/messages) (1)
- Actions belonging to logged in user (api2/user/actions)
- Messages (api2/messages) (1)
- Tags (api2/tags)
- Messages posted to a tag (api2/tags/TAG/messages) (1)
- Conversations (api2/conversations)
- Messages posted to a conversation (api2/conversations/CONVERSATIONID/messages) (1)
- Pools (api2/pools)
- Users associated with a pool (api2/pools/POOLID/users)
- Messages posted to a pool (api2/pools/POOLID/messages) (1)
- Searches ?? (1)
- Trends ??
(1) Stream interface
Each of these bullets represents a set of objects. The resource representing an individual object lives at api/objects/OBJECTID. For example, api/conversations/1. As much as is reasonable, one would expect to be able to GET (read), POST (create), PUT (update/amend), or DELETE (delete) any individual member of each of these object sets. Going through each of these objects to ask what it would mean to create, read, update, or delete that object may reveal holes in the existing API, some of which I have filled in above.References -
Dev mailing list thread - http://www.mail-archive.com/esme-dev@incubator.apache.org/msg00976.html
Methods, Resources, and Descriptions
Bold means the resource and method is implemented in the current /api2/ endpoint.
Resource | Method | Description/Payload schema/Response schema | Streaming? | Admin Only? | |
---|---|---|---|---|---|
api2/session api/sessions | GET,POST | ||||
api/sessions/SESSIONID | DELETE | ||||
api/users | GET | ||||
,DELETE | Post parameter: token |
| |||
api2/users | GET, POST | Post parameters: nickname, password - returns: user created |
| POST | |
api2/users/USERID | GET |
|
| ||
api2api/users/USERID/messagestokens | GET | should support/encourage a stream-based interface | |||
api/users/USERID/followees | GET | ||||
api/users/USERID/followers | GET | ||||
api/users/USERID/followees/USERID | POST,DELETE | or POST api/users/USERID/followees?user=USERID2 | |||
api/users/USERID/tracks | GET,POST | ||||
api/users/USERID/tracks/TRACKID | DELETE | ||||
, POST | Post parameters: description - returns: token created |
| GET, POST | ||
api2/user/messages | GET,POST | Post parameters: message, via (opt), pool (opt), realm (opt), metadata (opt), tags (opt), replyto (opt) | Yes | ||
api2/user/tags/TAG | GET |
|
| ||
api2/user/tags/TAG/messages | GET |
| Yes | ||
api2/user/followers | GET |
|
| ||
api2/user/followees | GET,POST | Post parameter: userId |
| ||
api2/user/followees/USERID | DELETE |
|
| ||
api2/user/tracks | GET,POST | Post parameter: track (regex) |
| ||
api2/user/tracks/TRACKID | GET,DELETE |
|
| ||
api2/user/tracks/TRACKID/messages | GET |
| Yes | ||
api2/userapi/users/USERID/actions | GET,POST | Post parameter: name, test, action |
| ||
api2/userapi/users/USERID/actions/ACTIONID | GET,PUT,DELETE | Put parameter: enabled (boolean) |
| ||
api2api/messages/MESSAGEID | GET ,PUT?,DELETE? |
|
| ||
api2api/messages | GET,POST | ||||
api/tags | GET | Return all of the tags, or user-specific tags (GET api/tags/USERID?) and let the front-end decide what to do with it. | |||
api/tags/TAGID | GET | Gets the information about a particular tag | |||
| Yes | ||||
api2/conversations/CONVERSATIONID | GET |
|
| ||
api2/conversations/CONVERSATIONID/messages | GET,POST |
| Yes | ||
api2/pools | GET,POST |
|
| ||
api2/pools/POOLID | GET,DELETE |
|
| ||
api2/pools/POOLID/users | GET,POST | Post parameters: realm, userId, permission |
| ||
api2/pools/POOLID/users/USERID | DELETE |
|
| ||
api2/pools/POOLID/messages | GET,POST |
| Yes | api/conversations/CONVERSATIONID | GET |
One point to note is that many some HTTP clients do not currently support the "PUT" or "DELETE" methods, so these may have to be simulated through POST methods with an extra parameter. I think that because of the close mapping to resource verbs, is worth using these methods in the specification and defining the simulation method for the entire API separately.
Resource/Object Hierarchy
The above is based on a rough object hierarchy as follows:
- ESME API instance (api/)
- Sessions (api/sessions)
- Users (api/users)
- Messages posted by a user (api/users/USERID/messages) (1)
- Users followed by a user (api/users/USERID/followees)
- Users following a user (api/users/USERID/followers)
- Trackers belonging to a user (api/users/USERID/tracks)
- Actions belonging to a user (api/users/USERID/actions)
- Messages (api/messages) (1)
- Tags (api/tags) (1)
- Conversations (api/conversations) (1)
- Pools (api/pools) (1)
- ?
- Searches ?? (1)
- Trends ??
(1) Stream interface should be available and use should be encouraged
...
Note on the call: api2/user/messages (http://esmecloudserverapache.dickhirsch.staxapps.net/api2/user/messages?via=SAPRiver&tag=bob&message=User+')
That call uses "tag", not "tags". Looking at the API2 code, the parameter name needs to be "tags".
When it does work, I think the behavior may be a bit misleading, as tags added using the tags parameter do not necessarily show up in the message at all. So make sure to keep an eye on the tag cloud or go to the page for the specific tag you are trying to add and see if it shows up there.
...
Streams
There are a lot of ways we can model streams and I'm very interested in input here. Options for interfacing to streams that I have seen:
- XMPP - http://xmpp.org/
- AMQP - http://jira.amqp.org/confluence/display/AMQP/Advanced+Message+Queuing+Protocol
- HTTP
- Polling (bad)
- Comet/long-polling - Bayeux - http://svn.cometd.com/trunk/bayeux/bayeux.html
- Reverse HTTP - http://www.reversehttp.net/
- PubSubHubBub? - possibly via Reverse HTTP - http://www.reversehttp.net/demos/endpoint.html
Authorization and roles
Certain API methods are only available to certain roles, as defined in property files. If an API method has a notation in the "Admin only?" column, this means that the user attempting to interact with that API method must have the proper role assigned in the relevant property file.
For example, take a look at the test.default.props file in the /server/resources/props directory.
Or, for example, create a prod.default.props file in the "server/resources/props" directory. In this file, add the line "role.myuser=integration-admin", where
"myuser" is the username that you want to have super-powers.
("integration-admin" is what we're calling the role.)
ESME HTTP Comet/long-polling implementation for streams
In general, for any resource URL that indicates streaming is implemented above, we will implement an HTTP-based streaming interface. This interface might better be referred to as a "delta" interface. The URL will behave as follows (we use api2/user/messages as an example):
URL | Request | Query Parameters | Behavior |
---|---|---|---|
api2/user/messages | GET |
| Returns any messages not already read by the client |
api2/user/messages | GET | timeout=20 | If new messages not already read by the client exist, these are returned. If no new messages exist, the request will stall for a number of seconds as defined in the timeout= parameter. If a new message arrives during this time, the request will immediately resolve, returning the new message. |
api2/user/messages | GET | history=10 | Returns the last 10 messages in the timeline |
Response codes follow normal HTTP meanings:
Response Code | Meaning |
---|---|
200 | Success - Response body should include messages |
204 | No Content - No new messages in the stream (only returned for base URL and ?timeout= requests, not for ?history= requests). Should be 304 Not Modified, but not currently supported in Lift. |
403 | Forbidden - Either the session is invalid or non-existent, or the user is not authorized to access the resource. Eventually we'll move to 401s for invalid/non-existent sessions, but currently we can't return a WWW-Authenticate header field that will result in the right behavior, so we're just stopping the request dead with a 403. |
404 | Not found - for example, a request for the stream of a tag that doesn't exist |
Once this interface is implemented where planned for a URL, the "Yes" above in the "Streaming?" column will be bolded.
...
Streams/Message-queues in ESME and in general
...
Formats
Request formats
Format specification
If there is a request body, format should be specified using the Content-Type HTTP header.
...
- XML ?
- JSON ?
- Multi-part Form-encoded ?
- Form-encoded
Response formats
Format specification
Format could be specified using the HTTP Accept header - http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html
...
Formats to be supported
- XML ?
- JSON ?
- ...
Authorization
Currently the ESME REST API uses tokens as the authorization mechanism. A token is used to establish a session and then the session is used to persist the authorization of the API client across the length of the session.
...
- Sessions are not natively supported in a lot of API programming environments, especially environments that do not have a persistent data-store available to the application.
- The current API design appears encourage that the token sent to establish the session be sent in the clear over an unencrypted connection.
Points for discussion, resolution, further work
Clients
Daniel Koller has created an initial client for the Streaming api. The client is currently attached to a Jira item
...
- ESME has webhooks as part of its actions framework, but we may want to document their existence as part of the API, and possibly improve the functionality if there are use cases (http://incubator.apache.org/esme/actions.html)
...