Versions Compared

Old Version 22

changes.mady.by.user Bryan Call

Saved on

compared with

New Version Current

changes.mady.by.user Bryan Call

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.
Comment: Migrated to Confluence 5.3

oprofile

This is a sampling profiler, so it has minimal impact on Traffic Servers performance.

Running the profiler and checking the status. Make sure the call depth is set to where you want it.

Panel

// start the deamon
bcall@snowball trafficserver$ sudo opcontrol --start-daemon
WARNING: Deprecated config file /etc/modprobe.conf, all config files belong into /etc/modprobe.d/.
Detected stale lock file. Removing.
Using default event: CPU_CLK_UNHALTED:100000:0:1:1
Using 2.6+ OProfile kernel interface.
Using log file /var/lib/oprofile/samples/oprofiled.log
Daemon started.

// check the status
bcall@snowball trafficserver$ sudo opcontrol --status
Daemon running: pid 29524
Separate options: library
vmlinux file: none
Image filter: none
Call-graph depth: 6

After making sure the daemon is running aand your settings are set to what you want. Start traffic server and start running the benchmark. I normally wait a few seconds for things to warm up before I start to profile the code.

Panel

// profile the code for 60 seconds
bcall@snowball trafficserver$ sudo opcontrol -s; sleep 60; sudo opcontrol -t
Profiler running.
Stopping profiling.

Running a basic report will tell you what functions are showing up in the sampled profile the most.

Panel

bcall@snowball trafficserver$ sudo opreport --symbols /usr/local/bin/traffic_server | head
CPU: Core 2, speed 2003 MHz (estimated)
Counted CPU_CLK_UNHALTED events (Clock cycles when not halted) with a unit mask of 0x00 (Unhalted core cycles) count 100000
samples % image name app name symbol name
5158 4.6037 traffic_server traffic_server ink_freelist_new
3223 2.8767 traffic_server traffic_server HttpSM::_instantiate_func(HttpSM*, HttpSM*)
3060 2.7312 traffic_server traffic_server mime_hdr_field_find(MIMEHdrImpl*, char const*, int)
3044 2.7169 traffic_server traffic_server ink_freelist_free
2586 2.3081 libc-2.11.1.so traffic_server memcpy
2487 2.2197 traffic_server traffic_server HttpCompat::parse_tok_list(StrList*, int, char const*, int, char)
2309 2.0609 libc-2.11.1.so traffic_server vfprintf

Running a callgraph report.

Panel

bcall@snowball trafficserver$ sudo opreport -cg --symbols /usr/local/bin/traffic_server | head -50
samples % linenr info image name app name symbol name
-------------------------------------------------------------------------------
18 26.4706 HttpSM.cc:2640 traffic_server traffic_server HttpSM::main_handler(int, void*)
20 29.4118 HttpAccept.cc:33 traffic_server traffic_server HttpAccept::mainEvent(int, void*)
30 44.1176 HttpClientSession.cc:428 traffic_server traffic_server HttpClientSession::state_api_callout(int, void*)
5158 4.6037 ink_queue.cc:169 traffic_server traffic_server ink_freelist_new
5158 100.000 ink_queue.cc:169 traffic_server traffic_server ink_freelist_new self
-------------------------------------------------------------------------------
26 100.000 HttpSM.cc:2640 traffic_server traffic_server HttpSM::main_handler(int, void*)
3223 2.8767 HttpSM.cc:154 traffic_server traffic_server HttpSM::_instantiate_func(HttpSM*, HttpSM*)
3223 100.000 HttpSM.cc:154 traffic_server traffic_server HttpSM::_instantiate_func(HttpSM*, HttpSM*) self
-------------------------------------------------------------------------------
3060 2.7312 MIME.cc:1380 traffic_server traffic_server mime_hdr_field_find(MIMEHdrImpl*, char const*, int)
3060 100.000 MIME.cc:1380 traffic_server traffic_server mime_hdr_field_find(MIMEHdrImpl*, char const*, int) self

Google profiler

This too is a sampling profiler, so it has minimal impact on Traffic Server's performance. Support for the profiler needs to be compiled into Traffic Server for it to work. There is an option to add profiling support when running configure and is described in the steps below.

Install the Google's perftools development package, the command may vary on the OS, below is on Fedora 12:

Panel

bcall@snowball trafficserver$ sudo yum install google-perftools-devel
...
Installed:
google-perftools-devel.x86_64 0:1.4-1.fc12

Dependency Installed:
google-perftools.x86_64 0:1.4-1.fc12

Complete!

Run configure with the option to add profiling support. It will check to see if the header and library is installed. You will have to recompile traffic server to add calling the profiler's start and stop functions.

Panel

bcall@snowball trafficserver$ ./configure --with-profiler
bcall@snowball trafficserver$ gmake -j >& /dev/null && echo good
good

You can either install the package with "sudo gmake install" or run the program for the current directory if you already have an installation with all the logs and configuration directories setup.

Panel

bcall@snowball trafficserver$ sudo proxy/traffic_server
// or
bcall@snowball trafficserver$ sudo gmake install
bcall@snowball trafficserver$ sudo /usr/local/bin/trafficserver start
Starting Apache Traffic Server: [ OK ]

Next you will need to run your benchmark. After the benchmark is complete, kill the traffic_server process or stop traffic server with the startup script. The traffic server process will need to be shutdown before a proper profile file is written.

Panel

bcall@snowball trafficserver$ sudo killall traffic_server
// or
bcall@snowball trafficserver$ sudo /usr/local/bin/trafficserver stop
Stopping traffic_cop: [ OK ]
Stopping traffic_manager: [ OK ]
Stopping traffic_server: [ OK ]

Now we can look at the profiling information.

Panel

bcall@snowball trafficserver$ pprof --text proxy/traffic_server /tmp/ts.prof | head
Total: 14 samples
5 35.7% 35.7% 5 35.7% memset
3 21.4% 57.1% 3 21.4% __strncpy_ssse3
1 7.1% 64.3% 1 7.1% __pread_nocancel
1 7.1% 71.4% 1 7.1% build_part_hash_table
1 7.1% 78.6% 1 7.1% initialize_all_global_stats
1 7.1% 85.7% 1 7.1% ink_freelist_new
1 7.1% 92.9% 1 7.1% pthread_sigmask
1 7.1% 100.0% 1 7.1% time
0 0.0% 100.0% 2 14.3% AIOCallbackInternal::io_complet

To make a callgraph of the program in PDF form.

Panel

bcall@snowball trafficserver$ pprof --pdf proxy/traffic_server /tmp/ts.prof > ts.pdf

If you want to investigate a certain function you can the focus option.

Panel

bcall@snowball trafficserver$ pprof --pdf --focus=HttpSM::main_handler proxy/traffic_server /tmp/ts.prof > ts-main_handler.pdf

Callgrind

Callgrind is will slow down the performance of traffic server significantly.