...
- Packet or connection marks (on the same machine only)
- Type of Service (ToS) or Differentiated Services (DiffServ) Field (IP packet)
- Priority Code Point (PCP) Field (Ethernet frame)
There's Traffic Server offers more than one way to set some of these marks and some ways don't work in all scenarios:
...
These marks work only when Traffic Server and iproute2 etc. are on the same machine, they aren't communicated to a separate device, however a separate device can might want to copy marks from the ToS/DiffServ or PCP fields to the connection mark to mark traffic sent *from* the origin. iproute2, iptables/ip6tables, and ebtables all support the packet mark:
...
The ToS Field was originally specified in RFC 791. Both it and the IPv6 Traffic Class Field were superseded by the DiffServ Field specified in RFC 2474. The IPv4 socket option is setsockopt(sockfd, IPPROTO_IP, IP_TOS, &optval, optlen)
. The and the IPv6 socket option is setsockopt(sockfd, IPPROTO_IPV6, IPV6_TCLASS, &optval, optlen)
...
Use the tsxs utility to compile it:
Code Block | ||||
---|---|---|---|---|
|
$ tsxs -o priority.so priority.cc
||||
Add lines like the following to remap.config to configure it:
Code Block | ||||
---|---|---|---|---|
| ||||
map http://example.com http://example.com @plugin=priority.so @pparam=4
regex_map http://.*\.example.com http://$0 @plugin=priority.so @pparam=4
|
Traffic Sent *From* the Origin
Traffic Sent *From* the Origin
You can copy marks from the ToS/DiffServ or PCP fields to the connection mark with the CONNMARK iptables/ip6tables target, to mark traffic sent *from* the origin, even if Traffic Server is on a separate device. In the following example, if you set the ToS/DiffServ Field (0x0c, 0x1c, etc.) on traffic sent to the origin, it will You can copy marks from the ToS/DiffServ or PCP fields to the connection mark with the CONNMARK iptables/ip6tables target, to mark traffic sent *from* the origin, even if Traffic Server is on a separate device. In the following example, if you set the ToS/DiffServ Field (0x0c, 0x1c, etc.) on traffic sent to the origin, it will add both the origin request and response to the same iproute2 class (2:1, 2:3, etc.):
Code Block | ||
---|---|---|
| ||
iptables -t mangle -A POSTROUTING -m tos --tos 0x0c -j CONNMARK --set-mark 1
iptables -t mangle -A POSTROUTING -m connmark --mark 1 -j CLASSIFY --set-class 2:1
iptables -t mangle -A POSTROUTING -m tos --tos 0x1c -j CONNMARK --set-mark 2
iptables -t mangle -A POSTROUTING -m connmark --mark 2 -j CLASSIFY --set-class 2:3
|
Zero Penalty Hit
Zero Penalty Hit is a feature of Squid to mark traffic sent to the client based on the cache lookup status (hit, miss, etc.) The use case for this is the management of upstream bandwidth without limiting access to content that's already cached. Here's an example of a Traffic Server plugin that does the same thing but I'm skeptical that it isn't better to directly manage the bandwidth between the proxy and origin? What effect does constricting traffic between the proxy and client have on the upstream traffic? See background fill and Read While Writer. The CONNMARK iptables/ip6tables target and transparency can help directly manage the bandwidth between the proxy and origin.
Use the tsxs utility to compile the plugin:
Code Block | ||
---|---|---|
| ||
$ tsxs -o tos.so tos.cc
|
If the content was already cached (cache hit) then the plugin sets the ToS/DiffServ Field on the client response to 0x0c. Edit the source code to change the value or implement a configuration variable to set it. You could adapt it to mark other cache lookup statuses (see TSHttpTxnCacheLookupStatusGet()
) or instead set the packet mark, connection mark, or PCP Field.
There's a feature request (TS-2597) to implement this in Traffic Server.
Transparency
When shaping traffic between the origin and proxy, iproute2 etc. normally can't tell which client a request originated from, they know only that it was sent by the proxy. You can however configure Traffic Server to make the client or origin connections transparent. If the client connection is transparent then the source of client responses and destination of client requests is the address of the origin. If the origin connection is transparent then the source of origin requests and destination of origin responses is the address of the client. You can enable the two features independently. See Transparent Proxying.
This is handy for communicating more detail to iproute2 etc. For example here's how to divide upstream bandwidth equally among all clients with iproute2 and SFQ:
Code Block | ||||
---|---|---|---|---|
| ||||
# The source of origin requests and destination of origin responses is
# the address of the client
CONFIG proxy.config.http.server_ports STRING 8080:tr-out
|
Code Block | ||
---|---|---|
| ||
# Remember if traffic originated from our Internet connection
iptables -t mangle -A PREROUTING -i eth0.2 -j MARK --set-mark 1/1
ifconfig ifb0 up
# A qdisc is required before we can add a filter
insmod sch_prio
tc qdisc add dev br-lan root handle 1 prio
# Shape only traffic originating from our Internet connection
# (packet mark 1/1)
insmod cls_u32
insmod act_mirred
tc filter add dev br-lan parent 1: protocol ip pref 1 u32 match mark 1 1 flowid 1:1 action mirred egress redirect dev ifb0
# Don't shape traffic (reorder/delay/drop) while there's available
# capacity. Unfortunately available capacity must be manually
# configured and fine-tuned. The following assumes isolated
# up/downstream capacity (full-duplex).
insmod sch_tbf
tc qdisc add dev eth0.2 root handle 1 tbf rate .5mbit burst 5k latency 70ms
tc qdisc add dev ifb0 root handle 1 tbf rate 2.5mbit burst 5k latency 70ms
# Schedule an equal amount of traffic for each client
insmod sch_sfq
tc qdisc add dev eth0.2 parent 1: handle 2 sfq
tc qdisc add dev ifb0 parent 1: handle 2 sfq
# Divide downstream traffic into clients by destination IP address.
# Divide upstream traffic into clients by *Netfilter connection
# tracking* source IP address (after NAT all upstream traffic shares the
# same source IP address).
insmod cls_flow
tc filter add dev eth0.2 parent 2: pref 1 handle 1 flow hash keys nfct-src divisor 1024
tc filter add dev ifb0 parent 2: protocol ip pref 1 handle 1 flow hash keys dst divisor 1024
|
Background
Doing the actual traffic shaping with standard tools means you can combine all of their existing features with Traffic Server, you don't need to know and maintain another system specifically for Traffic Server, and you can shape the aggregate of proxy and non-proxy traffic. For example you can limit the sum of all traffic except access to Wikipedia and Khan Academy.
Something you can't yet do with iproute2 is specify a bandwidth for each connection or each client (without enumerating all of the clients in advance). This feature is available in MikroTik RouterOS, they call it PCQ.
If you need to distinguish one transaction from another you're required to use configuration variables for origin traffic but header_rewrite operators or API functions for client traffic. I think this asymmetry unnecessarily burdens the administrator with implementation details. It would be more consistent and more user friendly to just implement configuration variables for each socket option and make them work in all scenarios (origin and client traffic, records.config, the conf_remap plugin, the set-config header_rewrite operator, and TSHttpTxnConfigIntSet()
). It might be possible to update an existing socket when you change the configuration variables by implementing a callback.
I wonder how the current implementation works with persistent connections? For example I suspect if you set a socket option per transaction and then reuse the connection for another origin request or if the client reuses it to make another request, the option doesn't get reset?
TS-1090 and commit b77838991531d6cb402618c3d690b83e95b92d63 originally added the packet mark and ToS/DiffServ Field configuration variables and API functions. TS-3002 added the set-conn-dscp rewrite_header operator.
Example
Here's a full example of how to shape traffic between the proxy and origin based on the request target. It assigns websites one of three priorities, after the upstream bandwidth is scheduled by priority, each client gets an equal portion of each priority.
Traffic Server and iproute2 can be on different devices. It communicates the priority to iproute2 in the ToS/DiffServ Field which it overrides with the tos_out configuration variable and the conf_remap plugin. It shapes traffic sent *from* the origin by copying the priority from the ToS/DiffServ Field to the connection mark. The origin connection is transparent so iproute2 can tell which client the traffic belongs to. The client connection is transparent so Traffic Server can get the address of the origin in the rare case that a client neglects to send a Host header (HTTP/1.0 doesn't require it). You could alternatively configure each client to use the proxy explicitly.
Zero Penalty Hit
Zero Penalty Hit is a feature of Squid to mark traffic sent to the client based on the cache lookup status (hit, miss, etc.) The use case for this is the management of upstream bandwidth without limiting access to content that's already cached. Here's an example of a Traffic Server plugin that does the same thing but I'm skeptical that it isn't better to directly manage the bandwidth between the proxy and origin? What effect does constricting traffic between the proxy and client have on the upstream traffic? See background fill and Read While Writer. The CONNMARK iptables/ip6tables target and transparency can help directly manage the bandwidth between the proxy and origin.
Use the tsxs utility to compile the plugin:
Code Block | ||
---|---|---|
| ||
If the content was already cached (cache hit) then the plugin sets the ToS/DiffServ Field on the client response to 0x0c. Edit the source code to change the value or implement a configuration variable to set it. You could adapt it to mark other cache lookup statuses (see TSHttpTxnCacheLookupStatusGet()
) or instead set the packet mark, connection mark, or PCP Field.
There's a feature request (TS-2597) to implement this in Traffic Server.
Transparency
When shaping traffic between the origin and proxy, iproute2 etc. normally can't tell which client a request originated from, they know only that it was sent by the proxy. You can however configure Traffic Server to make the client or origin connections transparent. If the client connection is transparent then the source of client responses and destination of client requests is the address of the origin. If the origin connection is transparent then the source of origin requests and destination of origin responses is the address of the client. You can enable the two features independently. See Transparent Proxying.
This is handy for communicating more detail to iproute2 etc. For example here's how to divide upstream bandwidth equally among all clients with iproute2 and SFQ:
Code Block | ||||
---|---|---|---|---|
| ||||
Code Block | ||
---|---|---|
| ||
Background
Doing the actual traffic shaping with standard tools means you can combine all of their existing features with Traffic Server, you don't need to know and maintain another system specifically for Traffic Server, and you can shape the aggregate of proxy and non-proxy traffic. For example you can limit the sum of all traffic except access to Wikipedia and Khan Academy.
Something you can't yet do with iproute2 is specify a bandwidth for each connection or each client (without enumerating all of the clients in advance). This feature is available in MikroTik RouterOS, they call it PCQ.
If you need to distinguish one transaction from another you're required to use configuration variables for origin traffic but header_rewrite operators or API functions for client traffic. I think this asymmetry unnecessarily burdens the administrator with implementation details. It would be more consistent and more user friendly to just implement configuration variables for each socket option and make them work in all scenarios (origin and client traffic, records.config, the conf_remap plugin, the set-config header_rewrite operator, and TSHttpTxnConfigIntSet()
). It might be possible to update an existing socket when you change the configuration variables by implementing a callback.
I wonder how the current implementation works with persistent connections? For example I suspect if you set a socket option per transaction and then reuse the connection for another origin request or if the client reuses it to make another request, the option doesn't get reset?
TS-1090 and commit b77838991531d6cb402618c3d690b83e95b92d63 originally added the packet mark and ToS/DiffServ Field configuration variables and API functions. TS-3002 added the set-conn-dscp rewrite_header operator.
Example
Here's a full example of how to shape traffic between the proxy and origin based on the request target. It assigns websites one of three priorities, after the upstream bandwidth is scheduled by priority, each client gets an equal portion of each priority.
Traffic Server and iproute2 can be on different devices. It communicates the priority to iproute2 in the ToS/DiffServ Field which it overrides with the tos_out configuration variable and the conf_remap plugin. It shapes traffic sent *from* the origin by copying the priority from the ToS/DiffServ Field to the connection mark. The origin connection is transparent so iproute2 can tell which client the traffic belongs to. The client connection is transparent so Traffic Server can get the address of the origin in the rare case that a client neglects to send a Host header (HTTP/1.0 doesn't require it). You could alternatively configure each client to use the proxy explicitly.
Be careful of ICMP redirects, they can sometimes cause clients to route non-web traffic to the proxy!
Code Block | ||||
---|---|---|---|---|
| ||||
Code Block | ||||
---|---|---|---|---|
| ||||
Code Block | ||
---|---|---|
|
Code Block | ||||
---|---|---|---|---|
| ||||
# The source of client responses and destination of client requests is
# the address of the origin. The source of origin requests and
# destination of origin responses is the address of the client.
CONFIG proxy.config.http.server_ports STRING 8080:tr-full
|
Code Block | ||||
---|---|---|---|---|
| ||||
# Give high priority to Wikipedia, low priority to YouTube
map http://wikipedia.org http://wikipedia.org @plugin=conf_remap.so @pparam=proxy.config.net.sock_packet_tos_out=0x0c
regex_map http://.*\.wikipedia\.org http://$0 @plugin=conf_remap.so @pparam=proxy.config.net.sock_packet_tos_out=0x0c
map http://youtube.org http://youtube.org @plugin=conf_remap.so @pparam=proxy.config.net.sock_packet_tos_out=0x1c
regex_map http://.*\.youtube\.org http://$0 @plugin=conf_remap.so @pparam=proxy.config.net.sock_packet_tos_out=0x1c
|
Code Block | ||
---|---|---|
| ||
Resources
The best source of iproute2 documentation is the Linux Advanced Routing and Traffic Control project.
...