RedisTimeoutException with "bw: SpinningDown"

[wenqibu]

We encounter below RedisTimeoutException with "[bw SpinningDown](bw: SpinningDown)“ using HLEN. does it retry work for this scenario? What the best practice retry for this kind of error? Do we need reset connection for this kind of error?

Failed to operate data with Redis Cache while error is: Microsoft.DataTransfer.Common.AzureRedisDataManagement.RedisDataManagementException: HashLengthAsyncInternal failed with key xxxx and exception StackExchange.Redis.RedisTimeoutException: Timeout awaiting response (outbound=0KiB, inbound=0KiB, 8360ms elapsed, timeout is 8000ms), command=HLEN, next: INFO, inst: 0, qu: 0, qs: 0, aw: False, bw: SpinningDown, rs: ReadAsync, ws: Idle, in: 0, last-in: 1, cur-in: 0, sync-ops: 25, async-ops: 23552, serverEndpoint: 20.228.32.214:8500, conn-sec: 85343.29, aoc: 0, mc: 1/1/0, mgr: 10 of 10 available, clientName: xxx(SE.Redis-v2.10.1.65101), PerfCounterHelperkeyHashSlot: 11464, IOCP: (Busy=0,Free=1000,Min=200,Max=1000), WORKER: (Busy=4,Free=32763,Min=200,Max=32767), POOL: (Threads=7,QueuedItems=0,CompletedItems=33319324,Timers=45), v: 2.10.1.65101 (Please take a look at this article for some common client-side issues that can cause timeouts: https://stackexchange.github.io/StackExchange.Redis/Timeouts)

Another error we found is Reading is not allowed after reader was completed. Is this also retriable with resetting connection?

Failed to operate data with Redis Cache while error is: Microsoft.DataTransfer.Common.AzureRedisDataManagement.RedisDataManagementException: DeleteValueAsyncInternal failed with key xxx, field xxx and exception System.InvalidOperationException: Reading is not allowed after reader was completed.
at System.IO.Pipelines.ThrowHelper.ThrowInvalidOperationException_NoReadingAllowed()
at StackExchange.Redis.PhysicalConnection.ReadFromPipe()

[mgravell]

from the client's perspective, HLEN is a trivial operation; SpinningDown just tells me that the backlog processor doesn't think it has anything to do. I wonder if what really happened here is more server-side; have you looked at SLOWLOG GET to see whether there are any server-side ops that have taken a long time?

On the NoReadingAllowed; sounds like a connection was fatally injured already; while the error message isn't ideal, the real problem here: something bad happened at the connection level. Note that this code has been overhauled in 3.0, which should deal with some of the problems with the Pipelines intermediary, which should avoid any confusing messages.

[wenqibu]

No, I don't see any typical SLOWLOG items. We're using the Azure Redis server, and the server load is minimal since it's just a canary region: server load is under 5%, CPU less than 10%, and used memory is almost zero. Latency is below 5 ms, and all server-side telemetry appears to be fine.

Regarding NoReadingAllowed, do you mean it will be fixed in StackExchange.Redis client version 3.0? Is this a known issue? Would retrying help now? What the best precise if retry?

[mgravell]

I mean it is a symptom / ungraceful side-effect of a connection failure, not the direct cause. It should be transient as a connection is dying, and a replacement should already be in the process of coming up. To clarify: is the system remaining unusable, i.e. is this non-transient?