breiza
Guest
|
Posted:
Thu Feb 03, 2005 2:19 am Post subject:
stress testing results triggers some questions! |
|
|
Hi guys,
we completed stress/load testing on our new e2k3 servers and I have some
counters above recommended limits.
Maybe you can help answering some of the questions below, but first here is
the config:
Servers:
2A2P windows 2003/Exchange 2003 EE clusters
fully loaded (4SG, 5DB, 60GB/DB) 1650 users/cluster or 830 users/EVS
storage:
HP EVA5000, 146GB, 10K RPM disks (around 58 spindles only)
vraid5 for DB LUNs and vraid1 for logs LUNs
mixed-Disk Group configuration for exchange (i.e DG can contain DB or Logs
LUNs) I know not good for exchange!
We have all usual reg tweaks for improving perf on clusters/eva storage
When running loadsim2003 for that configuration, we see high Database\Log
Record Stalls/sec MAX (up to 305/s, limit being 100/s). (no log buffer tweaks
implemented, default of e2k3 as "not set" which means 500)
Database\Log Record Stalls/sec AVE is around 1.5/s, well within the 10/s
limit.
Similarly, we see high MSExchangeIS\RPC Averaged Latency Max (up to 149,
limit being 50)
MSExchangeIS\RPC Averaged Latency AVE is averaging 10-15, well within the
limit of 50.
All other counters check out well within the very high performance limits.
Latency is excellent for log write, DB read & write.
Note that the other factor that didn't check out well was anything to do
with the DC we were using...it was an old machine and a bit weak for the test.
Would you have any ideas what the high Database\Log Record Stalls/sec MAX
and high MSExchangeIS\RPC Averaged Latency Max could be caused by?
how can we bring this down?
Also, looking at the 95th percentile results, we see
- "Open Msg Store" averaging around 2sec instead of 1sec.
- "Logon" averaging 10s (this could be due to our slow DC....)
- "Apply view/sort" up to 5sec (HP's performance paper all show high counter
here too)
The overall weighted average results are not above .558sec so that's pretty
good but I'd like to know wether these 3 tasks are meant to be higher than
1sec.
Thanks for taking time to read this long post....
Breiza
|
|
Al Mulnick
Guest
|
Posted:
Thu Feb 03, 2005 7:54 am Post subject:
Re: stress testing results triggers some questions! |
|
|
Before you try to bring down the log record stalls, wouldn't it make sense
to verify the DC/GC wasn't a part of the bottleneck?
Also, when you say
"All other counters check out well within the very high performance limits.
Latency is excellent for log write, DB read & write.
Note that the other factor that didn't check out well was anything to do
with the DC we were using...it was an old machine and a bit weak for the
test."
What was CPU, Memory and Network and disk counters (you mention the
latencies were great, but did you look at spikes and correlate?) at the time
of the test and the time of the spikes? Was your sampling interval enough
to capture the spikes?
One thing you don't give us either is the profile of the user which is
important to such a conversation. You also didn't mention the loadsim
topology for this.
Logon and open message store could both be related to the domain or even to
the workstations that ran this.
Al
"breiza" <breiza@discussions.microsoft.com> wrote in message
news:9E5FFC17-3467-47E9-AAE8-0193A29E3A5B@microsoft.com...
| Quote: | Hi guys,
we completed stress/load testing on our new e2k3 servers and I have some
counters above recommended limits.
Maybe you can help answering some of the questions below, but first here
is
the config:
Servers:
2A2P windows 2003/Exchange 2003 EE clusters
fully loaded (4SG, 5DB, 60GB/DB) 1650 users/cluster or 830 users/EVS
storage:
HP EVA5000, 146GB, 10K RPM disks (around 58 spindles only)
vraid5 for DB LUNs and vraid1 for logs LUNs
mixed-Disk Group configuration for exchange (i.e DG can contain DB or Logs
LUNs) I know not good for exchange!
We have all usual reg tweaks for improving perf on clusters/eva storage
When running loadsim2003 for that configuration, we see high Database\Log
Record Stalls/sec MAX (up to 305/s, limit being 100/s). (no log buffer
tweaks
implemented, default of e2k3 as "not set" which means 500)
Database\Log Record Stalls/sec AVE is around 1.5/s, well within the 10/s
limit.
Similarly, we see high MSExchangeIS\RPC Averaged Latency Max (up to 149,
limit being 50)
MSExchangeIS\RPC Averaged Latency AVE is averaging 10-15, well within the
limit of 50.
All other counters check out well within the very high performance limits.
Latency is excellent for log write, DB read & write.
Note that the other factor that didn't check out well was anything to do
with the DC we were using...it was an old machine and a bit weak for the
test.
Would you have any ideas what the high Database\Log Record Stalls/sec MAX
and high MSExchangeIS\RPC Averaged Latency Max could be caused by?
how can we bring this down?
Also, looking at the 95th percentile results, we see
- "Open Msg Store" averaging around 2sec instead of 1sec.
- "Logon" averaging 10s (this could be due to our slow DC....)
- "Apply view/sort" up to 5sec (HP's performance paper all show high
counter
here too)
The overall weighted average results are not above .558sec so that's
pretty
good but I'd like to know wether these 3 tasks are meant to be higher than
1sec.
Thanks for taking time to read this long post....
Breiza
|
|
|