Ask the Expert

The relationship between latency and buffer credits

I am currently testing the distance impact of an ISL connection between two Connectrix (from McData). My reference job is a DSS copy of a local source disk to a target disk behind the SAN. First, I ran the job while the Connectrixes are back-to-back connected (no distance). Then, I connected the ISL (E-ports) of the Connectrixes over a DWDM pair, still not introducing distance. The elapsed time of the job did not significantly increase, proving DWDM does not introduce latency. Next, I put 20 km into my DWDM circuit. Now, my reference job took 10% longer. I expected this, given the extra time round-trip (albeit at light speed) of 20 km.

I hoped to overcome this latency by leveraging on the extra buffer credits of the Connectrix. Activating those extra BCs is merely a matter of flagging one field (the "10-100 km" field) on the Connectrix. That's what I did, but the elapsed time of the job does not improve at all. I tried with flagging the field either on the E-port only, or on both E-port and F-port. Any ideas?

    Requires Free Membership to View

Congratulations! It sounds like you have performed a fairly methodical test process, which is important in order to understand the cause and effect of the different configurations. Now I have a few questions before addressing your problem.

First, did you benchmark the target disk on the local switch first (e.g. no ISL at all) to establish a local baseline and if so, how did that differ if at all from the baseline ISL test? Second, what speed (1 Gb or 2 Gb) are the Fibre Channel ports from the server to the switch, in the ISLs between the switches, and from the switch to storage? What model of Connectix are you using? Were you able to note how many buffer credits were actually assigned to each port? How did you simulate the 20 km distance? Was it using a spool of 20 km single mode fiber (SMF) or was it over an actual circuit?

The Connectrix model number would have an impact on the number of available buffer credits dedicated or available to each port. Buffer credits will have an impact on throughput performance particularly as you go further, and faster. For example, a general rule of thumb is about 1 buffer credit per 2 km at 1 Gb, 1 buffer credit per km at 2 Gb, 1 buffer credit per 1/2 km at 4 Gb and so forth. As you found, DWDM should not introduce any bandwidth degradation until you encounter droop (loss of flow control and loss of protocol efficiency performance due to lack of buffers) at longer distances. Buffer credits deal with keeping the network pipe full and thus a measure of bandwidth. Latency will increase with distance over any network interface or protocol. The speed of light and associated propagation delay is about 5 microseconds per km. So a round trip latency would be about ( ( 5usec * 20 [distance] ) * 2 [round trip] = 200usec ). You should be able to measure any delay introduced by the switches, DWDM, or other components based upon your baseline tests.

You may also hear about something called db loss, which does not impact performance as much -- unless you have a lot of disruptions, breaks and connections in your network -- as it impacts the effective distance for transmission. Put simply, db loss is the amount of light bandwidth, or, I should say, the strength and accuracy of the light beam in the fiber optic circuit, due to cable connectors, cable quality, devices in the circuit and so forth. Buffer credits do not help with latency; rather, they help with bandwidth. Given your environment and application I'm not sure that a distance compression device like those from Adva, Ciena, Cisco, CNT, McData, Nortel, etc. would benefit or not.

This was first published in September 2004

There are Comments. Add yours.

TIP: Want to include a code block in your comment? Use <pre> or <code> tags around the desired text. Ex: <code>insert code</code>

REGISTER or login:

Forgot Password?
By submitting you agree to receive email from TechTarget and its partners. If you reside outside of the United States, you consent to having your personal data transferred to and processed in the United States. Privacy
Sort by: OldestNewest

Forgot Password?

No problem! Submit your e-mail address below. We'll send you an email containing your password.

Your password has been sent to: