JetStress Database Population pre-Exchange 2003 Rollout

[kingkong]

I am using the latest version of JetStress (with GUI. Version 6.5.7408.2) to
test the disk subsystem of my clustered SAN environment (W2K3 / E2K3).
However the reported database creation times in the JetStress documentation
relates to about 1.4Gb/min. My findings though are nothing like this. I
managed to build a 4 x Storage Group JetStress environment in 50 hours, which
is way longer than expected. Admittedly the DB's were 60Gb each, but I still
think the base build time shouldnt take this long and certainly took longer
than the prescribed time.

Anyone else finding JetStress takes forever to build these DB's? I'm sure
it's not a disk bottleneck, in case anyone asks!

[Al Mulnick]

How come you're so sure it's not a disk bottleneck? What have you done to
prevent it from being a disk bottleneck and to verify that it't not?

And don't tell me because it's EMC and the sales engineer told you so ;)

Keep in mind this is a write heavy function. You may have a problem
already, right? What was the most taxed resource in the chain?

Last time I used it, I didn't have that time to build; it was much faster.
I didn't build them all at once either, but more for concern of server
resources than for disk issues. The disk was slow but that's a different
issue where EMC doesn't seem to know their products or laws of physics.
Like I said, different story though.

Al


"kingkong" <kingkong@discussions.microsoft.com> wrote in message
news:1C1C9E98-C31C-4CE5-9138-90963B89ADB4@microsoft.com...

I am using the latest version of JetStress (with GUI. Version 6.5.7408.2)
to
test the disk subsystem of my clustered SAN environment (W2K3 / E2K3).
However the reported database creation times in the JetStress
documentation
relates to about 1.4Gb/min. My findings though are nothing like this. I
managed to build a 4 x Storage Group JetStress environment in 50 hours,
which
is way longer than expected. Admittedly the DB's were 60Gb each, but I
still
think the base build time shouldnt take this long and certainly took
longer
than the prescribed time.

Anyone else finding JetStress takes forever to build these DB's? I'm sure
it's not a disk bottleneck, in case anyone asks!

[kingkong]

Hi Al. I know it's not a disk bottleneck because:
a) It's a brand new EVA3000 HP SAN storage unit 10k disk. 146Gb each.
b) I've subsequently run JetStress tests against the disk and it has
performed brilliantly, esp write operations.
c) I'll take my SAN engineers word for it!

Were you using the latest version of JetStress with the GUI ? I found the
documentation to be a bit misleading (or lacking clarity), probably due to my
inexperience with this version. I'm just very surprised at the DB creation
times which surely can't be right...

Perhaps 15K disk would be better but not significantly.
What kind of topology had you used, I wonder if this was a contributing
factor:
1.5 IOPS
1.2Gb Mailbox Size (true reflection of users!!)
1000 mailboxes per EVS
Hardware storage cache size 512Mb
4 x Storage Groups / fully loaded.

"Al Mulnick" wrote:

How come you're so sure it's not a disk bottleneck? What have you done to
prevent it from being a disk bottleneck and to verify that it't not?

And don't tell me because it's EMC and the sales engineer told you so ;)

Keep in mind this is a write heavy function. You may have a problem
already, right? What was the most taxed resource in the chain?

Last time I used it, I didn't have that time to build; it was much faster.
I didn't build them all at once either, but more for concern of server
resources than for disk issues. The disk was slow but that's a different
issue where EMC doesn't seem to know their products or laws of physics.
Like I said, different story though.

Al


"kingkong" <kingkong@discussions.microsoft.com> wrote in message
news:1C1C9E98-C31C-4CE5-9138-90963B89ADB4@microsoft.com...
I am using the latest version of JetStress (with GUI. Version 6.5.7408.2)
to
test the disk subsystem of my clustered SAN environment (W2K3 / E2K3).
However the reported database creation times in the JetStress
documentation
relates to about 1.4Gb/min. My findings though are nothing like this. I
managed to build a 4 x Storage Group JetStress environment in 50 hours,
which
is way longer than expected. Admittedly the DB's were 60Gb each, but I
still
think the base build time shouldnt take this long and certainly took
longer
than the prescribed time.

Anyone else finding JetStress takes forever to build these DB's? I'm sure
it's not a disk bottleneck, in case anyone asks!

[Brian Henderson]

Hi King Kong.

JetStress takes awhile to build - no doubt about it. The new JetStressUI
tool will generate the database a lot faster than in the past, but the speed
to build a new DB with JetStress can be painfully slow. But, for a free
lightweight tool, once the DB is built, it is an effective way to measure
disk IOPS performance.

Thanks,
Brian
EMC

[Al Mulnick]

I'll agree it was slow, but the claim that there's nothing in the disk
layout that could be at fault is likely premature. If it wasn't I doubt
there'd be a real need to run the jetstress tool right? I mean, that is the
reason jetstress exists right? And if all was well, shouldn't you be able
to get the full 1.something GB per hour rate??

There are a lot of factors that need to be dealt with when it comes to disk
performance. While the EVA is a great system, it has physical limitations
just like any other, e.g. EMC systems. A platter can only rotate just so
fast right?
Taking your calculations, I'd have to say a few things about it:
1) we really don't know much from that description. For example, do you
have 2 disks? Fully populated single array with 8 spindles?
2) we also don't know how you have it laid out. Do you have it laid out for
optimal performance? Is it enough spindles to provide the IOPS you're
after? Are you checking the perfmon counters to find out if your
disk/backplane are bottlenecks?
3) we further don't know if you've checked the server itself to ensure it's
not the bottleneck. Building a database is an intensive application.
You're trying to do in several hours what takes many mail servers /years/ to
accomplish: build a large mail database.

So doing the convential storage math:
8 x 10K drives in a RAID0 configuration /should/ yield approximately 800 -
1040 total (using a per-disk IOPS rate of 100-130/sec for 10K drives) IOPS.
I know that's short of your goal based on your user density and profile,
but..

There are some things in a SAN that can be helpful if taken advantage of,
especially during a write transaction. Cache is one of them. If you have
some cache available, that may off-set some of the impact. You have 512MB.
The problem with cache is that it's like a glass of water intead of some
panacea equilizer. When it's full, you can't put any more water (writes)
into it. When the application has nowhere to write it's transaction, it
gets grumpy and let's the consumers of the server know it. Sometimes just
in delayed delivery, and sometimes in other ways. In your case, 1.5 IOPS *
1000 users is not something you want glued up because you won't unglue from
it easily. It'll get snarled up and stay that way if your numbers are what
the users end up doing depending on how they do it of course.

Let's reset for a second: you're running this tool as a way to ensure that
your disk subsystem (the EVA and all the parts from the HBA driver back) are
properly setup and tuned. There's something else we never asked: Which
drivers did you pick for the HBA? The Storport? Or other? Is this
important? I think it can be useful information in guaging whether or not
you setup your subsystem correctly.

Right, so where were we? Oh yeah...
Assuming you get a high cache hit rate, you may be able to take your write
performance up to higher levels than listed for the planning purposes.
Let's say you did. Let's say you could get that EVA up to >1100 IOPS.
Would you still see a bottle neck with jetstress? (most likely yes).

What do those numbers look like with 15K drives?
Same formula: 8 x 15K in RAID0 configuration /should/ yield approximately
1200 - 1440 total (using a per-disk IOPS rate of 150 - 180/sec for 15k
drives).

These are all planning numbers. Your mileage may vary of course. But keep
in mind this is a tool that is useful for guaging where you stand, it is not
an absolute. The long build time may or may not be indicative of what
you'll see on a production deployment. What's more important is the
response times you see in production under your production load. Jetstress
is one indicator, but using other tools such as iometer might give you some
reference points to work with. Loadsim is a very helpful tool when used
with perfmon to capture and allow you to test the setup under load.

What happens if you don't lay the disk out in an optimal configuration I
hear you ask? Why, your calculations have to be revised downward. That's a
seek time penalty while the disk heads seek back and forth in a sub-optimal
fashion looking for and writing data to the disk. One example of this is if
you were to put the log files on the same spindles as the DB's. That would
especially be evident after the disk cache was full while the controller
waits to commit to physicals. The semi-structured Exchange db is not "cache
friendly" so you may not get a lot of benefit from the cache there. The log
files on the other hand, can be quite write cache friendly since they tend
to write in a sequential stream. The DB is randomized r/w (read heavy
usually) so if you combine the four components: disk, cache, DB's and the
log files, when the cache fills you have to rely on the disk speeds (closer
to really) while the cache empties. What happens when the cache is full?
It will typically reject write transactions until it has a place to put the
data: either in cache or on platter.

One last thing: your user profile is similar to what OTG deployed for
Microsoft. You may want to consider seeing the whitepapers on that to see
what you can gain from their work.


Your config:
=======================================
Perhaps 15K disk would be better but not significantly.
What kind of topology had you used, I wonder if this was a contributing
factor:
1.5 IOPS
1.2Gb Mailbox Size (true reflection of users!!)
1000 mailboxes per EVS
Hardware storage cache size 512Mb
4 x Storage Groups / fully loaded.
========================================

Mine? It's a 4000 user density machine with an EMC backed but the user
profile is much smaller; total IOPS for the server shouldn't be beyond 3000
on any given day if calculations are accurate (per user on this gig are
estimated around .6 and total size of the implementation is roughly 600GB
per server for on SAN storage including log files, queue, SMTP, TEMP and AV
expansion).

Which EMC platform? I don't yet know (currently they're 10K drives in a
Symmetrix 8830 could change, but who knows?) Like I said before, if you
deal with EMC sales engineers, anything can happen. For me, anything has
happened, but it has NOT been pleasant believe me. Pretty much would never
recommend that experience in this realm to anyone I wanted to call a friend
until that portion of their product gets better (the sales-engineering
portion). But that's another story right?

To be fair: having implemented on EMC in the past, I can tell you it's
possible to make it work. Having implemented on HP in the past, I can tell
you it's possible to make it work. I have not implemented on a Shark or
Hitachi, although I've seen some that have on Hitachi ;)

Keep in mind I say all of this to get you to think about the results you're
seeing. If you suspect you have a slowdown, you might. It may be worth it
to investigate it further and make as sure as you're going to get.

My $0.02 worth anyway (all figures in USD)
-ajm



"Brian Henderson" <BrianHenderson@discussions.microsoft.com> wrote in
message news:203A317F-43D8-4F6A-BEE5-A983CD641CAE@microsoft.com...

Hi King Kong.

JetStress takes awhile to build - no doubt about it. The new JetStressUI
tool will generate the database a lot faster than in the past, but the
speed
to build a new DB with JetStress can be painfully slow. But, for a free
lightweight tool, once the DB is built, it is an effective way to measure
disk IOPS performance.

Thanks,
Brian
EMC