This post is inspired by Abdel’s comment here in which he asks for some specifics about how we actually apply Scrum in a BI/DW environment. I’ll answer the questions directly and add a few bits and pieces as they come to me. Of course this is just one perspective and as with most things in life, we’re learning as we go but I am more convinced than ever that Agile is the way to go for BI/DW development.
How do you specify teams?
We have little option here. There are 3 of us in our BI/DW team so we are always involved. We’re doing a lot of work with the Planning department at the moment so they are also in the team most of the time. There are of course times when we need to pull in a boutique developer - someone who has a special skill or expertise we need for a specific purpose - so they become part of the team for perhaps just one or two sprints.
The biggest hurdle we’ve had to overcome is identifying and getting a product owner, someone who not only has the vision but can also make decisions. These people are in high demand and are few and far between in most organisations. If they do their job properly they also need to do a lot of running around and consulting in order to make sure what we’re working on is indeed what we should be working on. We’ve found one now, I just hope we can hang onto them.
How do you choose the sprint process or sub-process?
This is where the product owner comes in although generally in practice we just know what it is we are supposed to be working on. We still have some kind of governance structure in place and I imagine you do too. There will be someone saying ‘you need to get x, y and z delivered’. On this basis you can say we’ll work on x for 2 months and then review what has been achieved. In that 2 months you might do 7 sprints and each one will chip away at the backlog of requirements until either your product owner says they have enough or you have some other higher priority to focus on.
When is it ok to stop your datamart development and move on?
I definitely think it is better and easier to focus on a delivery area for a period of time rather than saying you will deliver a product in its entirety - you’re getting into dangerous territory if you do the latter as you can’t accurately estimate when that will be and as we know, the scope and priority of the product delivery components will change constantly from day one. You specifically asked about delivering a datamart - you might for example, choose to deliver a single fact table with maybe 6 key dimensions. You might have other stories on your backlog that you might not get to. These could be things like:
- deliver an aggregated fact table or some kind of high-performance summary table
- migrate historical data into the core fact table
- add new attributes to some of the core dimensions
- add new dimensions to the model
This is where priorities and time come into your equation. If you can, you may keep delivering all of the above, or you may stop and put your energies into something entirely different. The approach manages whatever is needed for your organisation. The main thing is to ensure you deliver something of value in each sprint. Dimensional models are by design very extensible so this again lends itself very well to the above.
Sprint Duration
We’ve moved to 10 business day sprints, that works much better for us. A month is just too long and we run out of steam after the second week. With a demo day, we’re now not starting sprints on the same day of the week each time but that seems to be working out ok. Keep the flexibility to try new things, I think that is the most important point here.
Daily Stand-Up Location
In an attempt to improve collaboration we are having our daily stand-ups in the business area that we are doing the development for. Although this is a little bit inconvenient for us, it seems to help, we are working with them on their terms and often it is the best way for them to see and hear what we are doing. It also means they can easily get involved in our daily meetings and I think sends the right signals.
Thanks a lot Rob,
Very interesting, now i got the process very well.
I still have some questions about :
- Budgeting a BI/DW project (when scrum is used).
- Specifying the project number of releases.
- spécifying the ressource needed for each release.
could we use this kind of approach, when we are a consultant ? How could one convainc that it’s the best approach (fast, better, cheaper), taking into consideration that we could not give any plan to do that ?
During the two years ago, we worked on a big BI/DW project. At first we started with the traditionnal approach, Business requirement (2 months), Dimensional modeling (1 month)… Since relationship with our customer is a good one, i proposed to fragment releases into small pieces so we can deliver something quickly. In order to do that i suggested to install our team in the business unit we are working for (finance in our case), so we can communicate easily, and more than that, we can “absorb” changes quickly and with less impact on dealine (We agree that changes made in the fly are more simple to do than ones after 3 month of developpement)… I found that very usefull, and the whole project was very successfull… As you can see, we applied a lot of Agile data warehousing concepts without any knowledge about scrum and data warehousing… I think that it make a lot of sense working with this approach, i found it just “Logical”…
The point i try the introduce here, is the importance of the relationship established with the customer… a lot off customers don’t agree with that, because they don’t have any idea about Budget, deadlines… evenmore, they don’t (IT departement) want us to work directly with the BU in charge…
Thanks again
“I am more convinced than ever that Agile is the way to go for BI/DW development.”
I totally agree with you. Unless you want to over-spend, it is indeed the only way to go.
You may be interested in a white paper I recently published on this topic. It is not a commercial white paper but it does try to sell the idea of using Scrum (and other Agile principles) to BI/DW projects.
http://danossia.wordpress.com/files/2009/06/improving-roi-and-success-rate-of-your-business-intelligence-project.pdf
If you wish to collaborate in writing a book on this topic, you may be interested in the following blog post: http://analytical-mind.com/2009/06/01/agile-business-intelligence-collaborative-book/
Hi there
I have started to follow your blog recently. I am doing my masters degree on the use of scrum to promote agile development in datawarehousing, and i was hoping to get in contact with you?
We started to use scrum for our datawarehouse development last year august, and so far we seem to have had our ups and downs, with major improvements through out. would you care to compare notes?
Susan and Abdel, I appreciate your follow-ups and we’ve had some off-line email conversations on this topic. For the benefit of everyone else, I’ll be posting a further update soon.