Page History

...

Here are some ways Traffic Server can communicate with iproute2 etc.

• Packet or connection mark marks (on the same machine only)
• Type of Service (ToS) or Differentiated Services (DiffServ) Field (IP packet)
• Priority Code Point (PCP) Field (Ethernet frame)

...

mark_in and tos_in mark traffic destined for the client (the packets that make up a client response) and mark_out and tos_out mark traffic destined for the origin (the packets that make up an origin request). Sometimes you can mark traffic sent *from* the origin with the Netfilter CONNMARK iptables/ip6tables target (the packets that make up an origin response). mark_in and mark_out set the packet mark and tos_in and tos_out set the ToS/DiffServ Field. Configuration variables for the connection mark and PCP Field haven't been implemented.

...

ClientPacketMarkSet() and ClientPacketTosSet() mark traffic destined for the client and ServerPacketMarkSet() and ServerPacketTosSet() mark traffic destined for the origin. Sometimes you can mark traffic sent *from* the origin with the Netfilter CONNMARK iptables/ip6tables target. ClientPacketMarkSet() and ServerPacketMarkSet() set the packet mark and ClientPacketTosSet() and ServerPacketTosSet() set the ToS/DiffServ Field. API functions for the connection mark and PCP Field haven't been implemented but you can call TSHttpSsnClientFdGet() or TSHttpTxnClientFdGet() to get the client socket and call setsockopt() to set the option yourself.

Unlike the configuration variables the API functions work whether the socket has already been created or not. They immediately update an existing socket , and the options are also applied when a new socket is eventually created.

...

The set-conn-dscp operator immediately updates an existing socket. (The client socket will already have been created because a transaction doesn't exist without one.)

Packet or Connection

...

Marks

These marks work only when Traffic Server and iproute2 etc. are on the same machine, they aren't communicated to a separate device, however a separate device can copy marks from the ToS/DiffServ or PCP fields to the connection mark to mark traffic sent *from* the origin. The socket option is setsockopt(sockfd, SOL_SOCKET, SO_MARK, &optval, optlen)

ToS/DiffServ Field

iproute2, iptables/ip6tables, and ebtables all support the packet mark:

• iproute2 can match marked traffic directly, the mark u32 filter selector applies to the packet mark.
• The -m mark --mark iptables/ip6tables match extension matches traffic based on the packet mark. You can apply an iproute2 class with the CLASSIFY target or set the connection mark with the CONNMARK target.
• The -m m_mark --mark ebtables match extension matches traffic based on the packet mark.

You can match traffic based on the connection mark with the -m connmark --mark iptables/ip6tables match extension. The packet mark socket option is setsockopt(sockfd, SOL_SOCKET, SO_MARK, &optval, optlen)

ToS/DiffServ Field

This is an 8-bit field in the IPv4 and IPv6 packets but the 2 least significant bits are now reserved for Explicit Congestion Notification (ECN). You can't set them! The values you can set are 0x00, 0x04, 0x08, ... 0xfc. If you set the field to 0xff the This is an 8-bit field in the IPv4 and IPv6 packets but the 2 least significant bits are now reserved for Explicit Congestion Notification (ECN). You can't set them! The values you can set are 0x00, 0x04, 0x08, ... 0xfc. If you set the field to 0xff the effective value will be 0xfc. Furthermore

• the values XXX000XX have special meaning for backwards compatibility with the IP Precedence Field (see RFC 2474 section 4.2)
• and the values XXXXX0XX are reserved for standards action (see RFC 2474 section 6).
• The values XXXX10XX are initially available for experimental or local use but future standards should preferentially claim them if other values are exhausted.

Only the values XXXX11XX or 0x0c, 0x1c, 0x2c, ... 0xfc are reserved for experimental or local use.

• 6).
• The values XXXX01XX are initially available for experimental or local use but future standards should preferentially claim them if other values are exhausted.

The standardized values are listed in the IANA registry along with their applicable specifications and the differentiated services Wikipedia article discusses them in greater detail. Only the values XXXX11XX or 0x0c, 0x1c, 0x2c, ... 0xfc are reserved for experimental or local use.

These marks work even if a separate device does the actual traffic shaping. iproute2, iptables/ip6tables, and ebtables all support the ToS/DiffServ Field:

• iproute2 can match marked traffic directly with the ip tos and ip6 priority u32 filter selectors.
• iptables/ip6tables can match marked traffic with any of the following match extensions: -m dscp --dscp, -m dscp --dscp-class, and -m tos --tos. You can apply an iproute2 class with the CLASSIFY target or set the connection mark with the CONNMARK target.
• ebtables can match marked traffic with the -m ip --ip-tos and -m ip6 --ip6-tclass match extensions.

These marks work even if a separate device does the actual traffic shaping. The ToS Field was originally specified in RFC 791. Both it and the IPv6 Traffic Class Field were superseded by the DiffServ Field specified in RFC 2474. The IPv4 socket option is setsockopt(sockfd, IPPROTO_IP, IP_TOS, &optval, optlen). The IPv6 socket option is setsockopt(sockfd, IPPROTO_IPV6, IPV6_TCLASS, &optval, optlen)

...

These marks work even if a separate device in the same broadcast domain does the actual traffic shaping. You can match marked traffic with the ebtables -m vlan --vlan-prio match extension. The socket option is setsockopt(sockfd, SOL_SOCKET, SO_PRIORITY, &optval, optlen)

...

You can copy marks from the ToS/DiffServ or PCP fields to the connection mark with the Netfilter CONNMARK iptables/ip6tables target, to mark traffic sent *from* the origin, even if Traffic Server is on a separate device. In the following example, if you set the ToS/DiffServ Field (0x0c, 0x1c, etc.) on traffic sent to the origin, it will add both the origin request and response to the same iproute2 class (2:1, 2:3, etc.):

  iptables -t mangle -A POSTROUTING -m tos --tos 0x0c -j CONNMARK --set-mark 1
  iptables -t mangle -A POSTROUTING -m connmark --mark 1 -j CLASSIFY --set-class 2:1
  iptables -t mangle -A POSTROUTING -m tos --tos 0x1c -j CONNMARK --set-mark 2
  iptables -t mangle -A POSTROUTING -m connmark --mark 2 -j CLASSIFY --set-class 2:3

...

Zero Penalty Hit is a feature of Squid to mark traffic sent to the client based on the cache lookup status (hit, miss, etc.) The use case for this is the management of upstream bandwidth without limiting access to content that's already cached. Here's an example of a Traffic Server plugin that does the same thing but I'm skeptical that it isn't better to directly manage the bandwidth between the proxy and origin? What effect does constricting traffic between the proxy and client have on the upstream traffic? See background fill and Read While Writer. The Netfilter CONNMARK iptables/ip6tables target and transparency can help directly manage the bandwidth between the proxy and origin.

...

Here's a full example of how to shape traffic between the proxy and origin based on the request target. It assigns websites one of three priorities. After , after the upstream bandwidth is scheduled by priority, each client gets an equal portion of each priority.

...

Be careful of ICMP redirects, they can sometimes cause clients to route non-web traffic to the proxy. !

  # The source of client responses and destination of client requests is
  # the address of the origin.  The source of origin requests and
  # destination of origin responses is the address of the client.
  CONFIG proxy.config.http.server_ports STRING 8080:tr-full

  # Give high priority to Wikipedia, low priority to YouTube
  map http://wikipedia.org http://wikipedia.org @plugin=conf_remap.so @pparam=proxy.config.net.sock_packet_tos_out=0x0c
  regex_map http://.*\.wikipedia\.org http://$0 @plugin=conf_remap.so @pparam=proxy.config.net.sock_packet_tos_out=0x0c
  map http://youtube.org http://youtube.org @plugin=conf_remap.so @pparam=proxy.config.net.sock_packet_tos_out=0x1c
  regex_map http://.*\.youtube\.org http://$0 @plugin=conf_remap.so @pparam=proxy.config.net.sock_packet_tos_out=0x1c

  # Remember if traffic originated from our Internet connection
  iptables -t mangle -A PREROUTING -i eth0.2 -j MARK --set-mark 1/1

  # Route web traffic to the proxy server except traffic already
  # originating from it.  Matching web traffic by port number isn't
  # perfect but it's good enough.  This is the MAC address of the proxy
  # server.  Because it's configured to make origin connections
  # transparent this is the only way to match traffic already originating
  # from it:
  # http://thread.gmane.org/gmane.comp.security.firewalls.netfilter.general/45405
  iptables -t mangle -A PREROUTING -m mac --mac-source 00:22:15:d2:1e:61 -j RETURN
  iptables -t mangle -A PREROUTING -p tcp --dport 80 -j MARK --set-mark 2/2
  iptables -t mangle -A PREROUTING -i eth0.2 -p tcp --sport 80 -j MARK --set-mark 2/2

  # Web traffic is medium priority by default but the proxy server further
  # breaks down some high/low priority traffic.  It communicates this by
  # setting the ToS/DiffServ Field (it uses the pool of codepoints reserved
  # for experimental or local use, 0x0c/0x0c).  Mark the connection to
  # remember the priority and apply the same classification to response
  # traffic (on which the ToS/DiffServ Field is not set).
  iptables -t mangle -A POSTROUTING -m tos --tos 0x0c -j CONNMARK --set-mark 1
  iptables -t mangle -A POSTROUTING -m connmark --mark 1 -j CLASSIFY --set-class 2:1
  iptables -t mangle -A POSTROUTING -m tos --tos 0x1c -j CONNMARK --set-mark 2
  iptables -t mangle -A POSTROUTING -m connmark --mark 2 -j CLASSIFY --set-class 2:3

  # Route web traffic to the proxy server
  ip route add table 1 via 192.168.1.2
  ip rule add fwmark 2/2 table 1

  ifconfig ifb0 up

  # A qdisc is required before we can add a filter
  insmod sch_prio
  tc qdisc add dev br-lan root handle 1 prio

  # Shape only traffic originating from our Internet connection
  # (packet mark 1/1)
  insmod cls_u32
  insmod act_mirred
  tc filter add dev br-lan parent 1: protocol ip pref 1 u32 match mark 1 1 flowid 1:1 action mirred egress redirect dev ifb0

  # Don't shape traffic (reorder/delay/drop) while there's available
  # capacity.  Unfortunately available capacity must be manually
  # configured and fine-tuned.  The following assumes isolated
  # up/downstream capacity (full-duplex).
  insmod sch_tbf
  tc qdisc add dev eth0.2 root handle 1 tbf rate .5mbit burst 5k latency 70ms
  tc qdisc add dev ifb0 root handle 1 tbf rate 2.5mbit burst 5k latency 70ms

  # Schedule traffic according to three priorities
  tc qdisc add dev eth0.2 parent 1: handle 2 prio
  tc qdisc add dev ifb0 parent 1: handle 2 prio

  # For each priority schedule an equal amount of traffic for each client
  insmod sch_sfq
  tc qdisc add dev eth0.2 parent 2:1 handle 3 sfq
  tc qdisc add dev ifb0 parent 2:1 handle 3 sfq
  tc qdisc add dev eth0.2 parent 2:2 handle 4 sfq
  tc qdisc add dev ifb0 parent 2:2 handle 4 sfq
  tc qdisc add dev eth0.2 parent 2:3 handle 5 sfq
  tc qdisc add dev ifb0 parent 2:3 handle 5 sfq

  # Divide downstream traffic into clients by destination IP address.
  # Divide upstream traffic into clients by *Netfilter connection
  # tracking* source IP address (after NAT all upstream traffic shares the
  # same source IP address).
  insmod cls_flow
  tc filter add dev eth0.2 parent 3: pref 1 handle 1 flow hash keys nfct-src divisor 1024
  tc filter add dev ifb0 parent 3: protocol ip pref 1 handle 1 flow hash keys dst divisor 1024
  tc filter add dev eth0.2 parent 4: pref 1 handle 1 flow hash keys nfct-src divisor 1024
  tc filter add dev ifb0 parent 4: protocol ip pref 1 handle 1 flow hash keys dst divisor 1024
  tc filter add dev eth0.2 parent 5: pref 1 handle 1 flow hash keys nfct-src divisor 1024
  tc filter add dev ifb0 parent 5: protocol ip pref 1 handle 1 flow hash keys dst divisor 1024

...