Summary

This is a basic proposal for refactoring the twitter provider, which currently relies on twitter4j.  Twitter4j has dependencies that are now category X.

Objectives

The refactor should:

1. Remove twitter4j as a dependency
2. Replace twitter4j solely with ASF foundation code if possible
3. Retain all existing tests, with minimal change to the tests
4. Maintain or improve performance as measured by total execution time of the integration tests
5. Reduce number of source files in the module (currently > 20) 
6. Reduce module lines of code by at least 50%
7. Provide a pattern that we can follow in other components that rely on non-ASF third party modules for integration with APIs

Proposal

People Pattern has seen good results adopting apache juneau (incubating) for our JVM-based microservices.  Juneau exposes most of the capability of apache httpclient with simple interfaces and native json parsing, making it simple to GET with arguments, POST with json, and access the json entity returned as a POJO.

As it happens, streams-provider-twitter already contains POJOs for the data it pulls from twitter.  It does not have POJOs matching the requests that would be sent to the twitter REST interface, or the precise json payloads returned, but those would be trivial to add.

So my proposal is to replace twitter4j classes with streams classes built primarily with juneau. This will require adding beans for the exact request and response payloads of each supported endpoint.

Considerations

• Authentication - twitter4j handled the oauth handshaking and tokens with no involvement by streams.  we will need to replicate that.  basic auth alone won't cut it.
• DateTimes - twitter4j gives streams strings, which streams converts to pojos with jackson.  might get better performance embedding the Joda handling into a parser.
• Interfaces - twitter4j gave us sensible java methods to call to talk to twitter.  we should try to replicate that by defining interfaces for each endpoint and implementing each in a remotable proxy
• Additional Properties - we've historically built all pojos in streams with additionalProperties = true. the juneau serializer doesn't treat this map specially the way jackson does.  perhaps this module should disable it.
• Retries - we have a nasty mess of error handling code, that we could potentially replace by using retryable and in necessary a custom interceptor with delays in onRetry

References

http://streams.incubator.apache.org/site/0.3-incubating-SNAPSHOT/streams-project/streams-contrib/streams-provider-twitter/index.html

http://streams.incubator.apache.org/site/0.5-incubating-SNAPSHOT/streams-project/streams-contrib/streams-provider-twitter/apidocs/index.html

http://juneau.incubator.apache.org/site/apidocs/org/apache/juneau/rest/client/package-summary.html#RestClient

https://twittercommunity.com/t/solved-java-oauth-request-token-flow-example-without-libraries/1440