Visualizing System Latency 68
ChelleChelle writes "Latency has a direct impact on performance — thus, in order to identify performance issues it is absolutely essential to understand latency. With the introduction of DTrace it is now possible to measure latency at arbitrary points; the problem, however, is how to visually present this data in an effective manner. Toward this end, heat maps can be a powerful tool. When I/O latency is presented as a visual heat map, some intriguing and beautiful patterns can emerge. These patterns provide insight into how a system is actually performing and what kinds of latency end-user applications experience."
Re: (Score:2)
Or female. What's your point?
another solution to an already solved problem... (Score:1, Insightful)
Re: (Score:1, Informative)
How bout you RTFA before you make you're smartass comments, since yours is almost a direct fucking quote from it. However, this isn't about measuring network latency, it's about disk latency, something that until recently was extraordinarily hard to measure.
Re: (Score:1)
Re: (Score:1)
Re: (Score:1)
then you sweep in like a donkey with claims that that didn't just happen...
you are the worst kind of stupid.
if i respond to an AC, and an AC responds back, i will assume it was the original AC until told otherwise. the burden of proof is not on my shoulders... i've already proven i'm the same person. you can't prove you aren't the same AC.
you
Re: (Score:1)
you are NOTHING.
Re: (Score:2)
Shame, I'll have to find another source of amusement now.
Re: (Score:2)
Re: (Score:2)
That wouldn't leave me with very many people to talk to ;)
Re: (Score:2)
something that until recently was extraordinarily hard to measure.
Really?
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await svctm %util
hda 0.00 0.00 114.85 0.00 0.45 0.00 8.00 0.73 6.28 6.34 72.87
Where await and svctm are average wait (milliseconds) for the disk & queue and service time for the disk.
Or do you mean something else?
Re: (Score:3, Insightful)
The data presented in the article are actually quite a bit more subtle and interesting than the summary data you've got there. It's probably be impossible to notice the effects of the "icy lake" phenomenon they describe with av
Re: (Score:1)
the summary data you've got there
Funny. I recall the command syntax for that one lets you setup intervals per second. That would be the "black foot" that gets you out of the "icy lake" phenomenon you describe.
Relevance to Latency (Score:1, Offtopic)
So shouting is bad (Score:2)
Re: (Score:2)
Correct - see here [youtube.com]
The sky is falling... (Score:5, Funny)
Informative article, all on one page, not chock full of ads. Now excuse me while I stock my bunker.
Re: (Score:3, Informative)
At Queue, the sky is always falling... (Score:2)
I really like ACM Queue, which regularly prints articles for practitioners about things which both we and our more academic colleagues care.
I recommend it, and on rare occasions, contribute [slashdot.org].
--dave
Re:pretty graphs (Score:5, Insightful)
These visualizations are used to condense the information gathered on one second intervals from running systems. Any graph of substantially advanced material is going to require explanation until you understand what is being measured, how it is being graphed, and how this information translates in real world performance.
Of course a casual reader from the net needs to read text to understand what is going on. These aren't sales figure pie-charts and shouldn't necessarily be accessible for uninformed parties.
On another note.. Do you think casual readers would have any more success interpreting the raw data files? Anyhow, I am interested in the technique as it is not one I am currently using. With a little practice this may be a good at a glance technique.
Re: (Score:2)
Do you actually think the concept of a heat map is new?
A great graph has, (1) a title, (2) labeled x- and y-axes, (3) a 3D figure should also have labels for intensity and the z-axis. All text should be readable or removed. Generally, difficult to interpret figures should have a paragraph below them explaining (a) what they show and--ideally--(b) what the researcher concludes form the graph when this is not obvious.
Re: (Score:2)
Yes -- I do things like these all the time, and I frequently have reason to go: "Oh, our systems behave like *this* in reality?" I wrote an article on the subject (with gnuplot as the visualization tool): http://snipabacken.se/~grahn/gnuplot_kicks_ass/ [snipabacken.se]
Re: (Score:1, Insightful)
The article presented plenty of information related to it's topic. The topic was that using a heat map to describe latency is more useful than simple averages and maximums displayed as line graphs. The article then analyzed certain interesting cases were a heat map had information that would not have existed in a line graph. What you are griping about is that the topic itself is simple and that the article is full of individual analyses that provide support for the topic.
Re: (Score:2)
But almost all there concluding remarks on a figure are, "we don't understand this graph"
Re: (Score:2, Insightful)
That's the point, a good engineer's (or scientist's) response to new data that they can't fully explain is generally unmitigated glee, it means they've found something new. My takeaway from the article is, "try this new technique/tool, you'll see new data".
On another note, I've done some very basic analysis of disk performance at work, and this approach would have allowed me to be much more confident in my results. As it was, basically all I could do when comparing disks and filesystems was use iozone to
Re: (Score:2)
uh, "a good engineer's (or scientist's) response to new data that they can't fully explain is generally unmitigated glee, it means they've found something new." perhaps, but generally you don't ask others to read about it until you understand something about the phenomenon OR it has withstood several attempts to understand it.
You also wrote, "new technique" but what is new? Do you think they invented the heat map, or exploratory data analysis?
Re: (Score:1)
ACM is a scholarly, research-oriented group. If you're looking for spoon-fed, PCMagazine types of charts and graphs, look elsewhere. Lots of text generally means that someone with brains has to interpret the data, because the interpretation is non-trivial.
old school visualization (Score:5, Interesting)
Re: (Score:2)
This must have been a long time ago, back when you had easy access to the address lines.
Now, that same job would be VERY difficult! Most data accesses occur to data in the cache, which is not brought out to pins outside of the processor. And when memory accesses do happen, they happen over dedicated DDR address lines, which are very high speed (hard to probe), and the address lines are used to access both rows and columns, so some external circuitry is needed in order to determine what the real address is
Re: (Score:1)
As a compromise, consider capturing the return address in a timer interrupt and stuffing into a DAC or two. Of course that requires that you have an unused DAC (or two) on board, a timer, and processing time available in the interrupt, and the result probably won't be as smooth as the DAC directly on the bus. Still, if you can do it it's better than nothing.
If you can't, perhaps you can use a different peripheral and some external logic. SPI to a shift register, perhaps? Or I can see having a second process
Re: (Score:1, Troll)
Re: (Score:2)
Yep, probably very long time ago, not that it was easy even in mainframes but very useful, no overhead to measure, as you maybe know - a mainframe is happy when 101% busy - the measurement overhead is very often a bad thing! It was fun, really, but reading the results wasn't always easy - is anything? Later on 80's / 90's simulating, estimating, measuring, etc file / disk / network systems the heat maps created with our hardware people on channels, controllers, disks, caches, DMA, etc timings / sizes / rate
Game Theory and other modeling (Score:2)
After reading the article, this idea of a "heat map" or frequency distribution mapping (of sorts) can (sort of) be summed in:
A particular advantage of heat-map visualization is the ability to see outliers.
I find this particularly interesting as this graphically now allows a way to "filter" the real outlier out from a sea of data. Also,
Instead of a random distribution, latency is grouped together at various levels that rise and fall over time, producing lines in a pattern that became known as the icy lake. This was unexpected, especially considering the simplicity of the workload.
And concluding the section on what they dub as the "icy lake"...
To summarize what we know about the icy lake: lines come from single disks, and disk pairs cause increasing and decreasing latency. The actual reason for the latency difference over time that seeds this pattern has not been pinpointed; what causes the rate of increase/decrease to change (change in slope seen in figure 5) is also unknown; and, the higher latency line seen in the single-disk pool (figure 4) is also not yet understood. Visualizing latency in this way clearly poses more questions than it provides answers.
Without actually seeing the data or knowing the specifics of latency, from a pure mathematical standpoint I wonder what would result if one treated the set of numbers (from each disk) as a r
DTrace is amazing, but... (Score:1)
...it's a shame that instrumentation of things such as EMC's PowerPath are a little painful. I guess there will always be gaps where vendor meets vendor and closed source meets open source, but it remains rather complex to analyse what's happening in Solaris with PowerPath and some Storage Foundation stirred in for good measure. Impossible? No...but maybe we'd all benefit from a little more interoperabilty?
It's a great article though - Brendan's a DTrace authority is impressive.
easy. (Score:3, Funny)
Current: Snail
Expected, after customers leave in droves over data plan changes: Snail on meth (see yesterday
Expected, once AT&T upgrades equipment: Sloth on vallium
Heat? (Score:1, Funny)