Extracting Scalability and Performance Metrics from TCP Traffic

Many systems that speak over TCP have simple call-and-response protocols. These lend themselves well to extracting reasonably accurate request and response start and end times. This is a fabulously rich source of data that can be mined for many types of purposes. All that is needed is the first 384 bytes of each packet, containing IP and TCP headers but no payload. This is simple and non-intrusive to capture.

What can we do with this data? Here are some examples:

• Apply Little's Law to determine metrics such as utilization and concurrency.
• Extract the data needed for queueing theory analysis.
• Apply the Universal Scalability Law to determine the system's scalability limits.
• Compute aggregate metrics such as response-time quantiles for SLA management.
• Apply algorithms to detect spikes and stalls at various levels of granularity.
• Produce plots and graphs such as time-series and scatter-plot to reveal important performance and scalability characteristics of the system.
• Collect historical data for later analysis, trending, and capacity planning.

This analysis technique works for any system, at any layer in the stack, as long as the TCP-based protocol follows call-and-response semantics. Examples of systems that conform to this are most databases, including relational and NoSQL databases; key-value caches such as memcached; and many types of HTTP interactions.
This talk will introduce tools and techniques for capturing and analyzing the network traffic to give deeper insight into system behavior. This knowledge can reveal patterns that can be predictive of emerging problems, such as impending saturation or lock contention in an architectural layer. This overall system performance analysis is a valuable technique for finding problems without needing to analyze each layer independently.