oprofile
This is a sampling profiler, so it has minimal impact on Traffic Servers performance.
Running the profiler and checking the status. Make sure the call depth is set to where you want it.
Panel |
---|
// start the deamon bcall@snowball trafficserver$ sudo opcontrol --start-daemon WARNING: Deprecated config file /etc/modprobe.conf, all config files belong into /etc/modprobe.d/. Detected stale lock file. Removing. Using default event: CPU_CLK_UNHALTED:100000:0:1:1 Using 2.6+ OProfile kernel interface. Using log file /var/lib/oprofile/samples/oprofiled.log Daemon started. // check the status bcall@snowball trafficserver$ sudo opcontrol --status Daemon running: pid 29524 Separate options: library vmlinux file: none Image filter: none Call-graph depth: 6 |
After making sure the daemon is running aand your settings are set to what you want. Start traffic server and start running the benchmark. I normally wait a few seconds for things to warm up before I start to profile the code.
Panel |
---|
run some benchmark // profile the code for 60 seconds bcall@snowball trafficserver$ sudo opcontrol -s; sleep 60; sudo opcontrol -t Profiler running. Stopping profiling.
|
Running a basic report will tell you what functions are showing up in the sampled profile the most.
Panel |
---|
bcall@snowball trafficserver$ sudo opreport --symbols /usr/local/bin/traffic_server | head CPU: Core 2, speed 2003 MHz (estimated) Counted CPU_CLK_UNHALTED events (Clock cycles when not halted) with a unit mask of 0x00 (Unhalted core cycles) count 100000 samples % image name app name symbol name 5158 4.6037 traffic_server traffic_server ink_freelist_new 3223 2.8767 traffic_server traffic_server HttpSM::_instantiate_func(HttpSM*, HttpSM*) 3060 2.7312 traffic_server traffic_server mime_hdr_field_find(MIMEHdrImpl*, char const*, int) 3044 2.7169 traffic_server traffic_server ink_freelist_free 2586 2.3081 libc-2.11.1.so traffic_server memcpy 2487 2.2197 traffic_server traffic_server HttpCompat::parse_tok_list(StrList*, int, char const*, int, char) 2309 2.0609 libc-2.11.1.so traffic_server vfprintf |
Running a callgraph report.
Panel |
---|
bcall@snowball trafficserver$ sudo opreport -cg --symbols /usr/local/bin/traffic_server | head -50 samples % linenr info image name app name symbol name ------------------------------------------------------------------------------- 18 26.4706 HttpSM.cc:2640 traffic_server traffic_server HttpSM::main_handler(int, void*) 20 29.4118 HttpAccept.cc:33 traffic_server traffic_server HttpAccept::mainEvent(int, void*) 30 44.1176 HttpClientSession.cc:428 traffic_server traffic_server HttpClientSession::state_api_callout(int, void*) 5158 4.6037 ink_queue.cc:169 traffic_server traffic_server ink_freelist_new 5158 100.000 ink_queue.cc:169 traffic_server traffic_server ink_freelist_new self ------------------------------------------------------------------------------- 26 100.000 HttpSM.cc:2640 traffic_server traffic_server HttpSM::main_handler(int, void*) 3223 2.8767 HttpSM.cc:154 traffic_server traffic_server HttpSM::_instantiate_func(HttpSM*, HttpSM*) 3223 100.000 HttpSM.cc:154 traffic_server traffic_server HttpSM::_instantiate_func(HttpSM*, HttpSM*) self ------------------------------------------------------------------------------- 3060 2.7312 MIME.cc:1380 traffic_server traffic_server mime_hdr_field_find(MIMEHdrImpl*, char const*, int) 3060 100.000 MIME.cc:1380 traffic_server traffic_server mime_hdr_field_find(MIMEHdrImpl*, char const*, int) self |
Google profiler
This too is a sampling profiler, so it has minimal impact on Traffic Servers performance.Server's performance. Support for the profiler needs to be compiled into Traffic Server for it to work. There is an option to add profiling support when running configure and is described in the steps below.
Install the Google's perftools development package, the command may vary on the OS, below is on Fedora 12:
Panel |
---|
bcall@snowball trafficserver$ sudo yum install google-perftools-devel ... Installed: google-perftools-devel.x86_64 0:1.4-1.fc12 Dependency Installed: google-perftools.x86_64 0:1.4-1.fc12 Complete! |
Run configure with the option to add profiling support. It will check to see if the header and library is installed. You will have to recompile traffic server to add calling the profiler's start and stop functions.
Panel |
---|
bcall@snowball trafficserver$ ./configure --with-profiler bcall@snowball trafficserver$ gmake -j >& /dev/null && echo good good |
You can either install the package with "sudo gmake install" or run the program for the current directory if you already have an installation with all the logs and configuration directories setup.
Panel |
---|
bcall@snowball trafficserver$ sudo proxy/traffic_server // or bcall@snowball trafficserver$ sudo gmake install bcall@snowball trafficserver$ sudo /usr/local/bin/trafficserver start Starting Apache Traffic Server: [ OK ] |
Next you will need to run your benchmark. After the benchmark is complete, kill the traffic_server process or stop traffic server with the startup script. The traffic server process will need to be shutdown before a proper profile file is written.
Panel |
---|
bcall@snowball trafficserver$ sudo killall traffic_server // or bcall@snowball trafficserver$ sudo /usr/local/bin/trafficserver stop Stopping traffic_cop: [ OK ] Stopping traffic_manager: [ OK ] Stopping traffic_server: [ OK ] |
Now we can look at the profiling information.
Panel |
---|
bcall@snowball trafficserver$ pprof --text proxy/traffic_server /tmp/ts.prof | head Total: 14 samples 5 35.7% 35.7% 5 35.7% memset 3 21.4% 57.1% 3 21.4% __strncpy_ssse3 1 7.1% 64.3% 1 7.1% __pread_nocancel 1 7.1% 71.4% 1 7.1% build_part_hash_table 1 7.1% 78.6% 1 7.1% initialize_all_global_stats 1 7.1% 85.7% 1 7.1% ink_freelist_new 1 7.1% 92.9% 1 7.1% pthread_sigmask 1 7.1% 100.0% 1 7.1% time 0 0.0% 100.0% 2 14.3% AIOCallbackInternal::io_complet |
To make a callgraph of the program in PDF form.
Panel |
---|
bcall@snowball trafficserver$ pprof --pdf proxy/traffic_server /tmp/ts.prof > ts.pdf |
If you want to investigate a certain function you can the focus option.
Panel |
---|
bcall@snowball trafficserver$ pprof --pdf --focus=HttpSM::main_handler proxy/traffic_server /tmp/ts.prof > ts-main_handler.pdf |
Callgrind
Callgrind is will slow down the performance of traffic server significantly.