21 Mai 2015
von Adam Robertson
Find out how the FastStats team work hard to deliver at an
Olympic speed across the board.
The phrase “Faster, Higher, Stronger” is, in addition to being
the Olympic motto, something that you should be demanding of your
data analysis software.
Deep down, the whole development team working on FastStats knows
that application performance is one of the most important things
we have to care about. “Fast” has been in the name of the
product since the beginning and so we have nowhere to hide if
things start to slow down. We even have a special high
performance group who work on eking more out of the engine inside
FastStats. To try and avoid surprises when people show us
the things that they ask our software to do we also develop new
ways of monitoring how FastStats performs in the real world to
help us get the edge on the challenges ahead of us.
Part of this development process is talking to our users. We saw
at the FastStats User Group last November that most people were
already dealing with large amounts of data and, what is more, for
a lot of these people their data volumes were going to increase.
Of the roughly 250 attendees that knew about their data
volumes, about a quarter reported having 100s of millions of
transactions in their FastStats systems and over 15% were dealing
with billions of transactions. Then, of those that had
knowledge of data growth, over 20% thought their data volumes
would grow by half again over this year and even more thought
their data would at least double in size.
So how do we make sure we can cope with these data volumes?
One way is to develop monitoring tools to look more
intelligently at real life systems and see where the bottlenecks
start to appear. The last couple of FastStats releases have
included an unheralded new utility called the Job Visualiser:
This shows when jobs run within FastStats as well as providing
some simple stats on how long they are taking on average.
We have used this tool on real-world and test systems to
see how FastStats behaves under different loads. If you are
interested in how your system is performing and have access to
your FastStats Service, try it out and let us know how you get
As well as monitoring tools, those of us working on PeopleStage
have also developed performance and load testing tools to
simulate running campaigns in different situations and with
different numbers of users:
These tools have helped us reproduce the demanding scenarios that
people expect FastStats to cope with and improve the software’s
To list a couple of examples of the improvements we have made:
- The Q1 2015 release of PeopleStage can process a simple
campaign with 1 million people roughly 25% faster than the August
2014 release. This has been due to some streamlined
handling of campaign constraints and database access.
- We are working on ways to make it quicker to interact with
PeopleStage and the Q2 2015 release should have a much reduced
login time and quicker library refresh. This is something the
guys here have always nagged me about and so I’m pleased to make
an improvement in this area.
- All of this comes on top of general improvements in the
FastStats engine resulting in improved query times and system
load times. On one of our benchmark systems running on a
reasonably fast desktop machine the load time for a system with
about 750 million rows of input data was reduced by over a
quarter from 3 hours 24 minutes to 2 hours 29.
But there is always more to do, and we are now looking at how we
can further integrate these tools in to our day to day
development processes. The Olympic motto of “Faster,
Higher, Stronger” seems apt here as whilst data volumes keep
increasing, FastStats performance will have to keep up - but it’s
a challenge that will never end. This constant improvement
enables you and your business to create data driven marketing
campaigns that target exactly who you need in a manner they want