Letztes Wochenende war die Froscon 2011 in St. Augustin. Wie angekündigt habe ich dort die Vorträge "Eight rollouts a day" und "NoSQL, NewSQL, MySQL" gehalten, und dabei eine ganze Menge wertvolles Feedback bekommen und eine Menge guter Kontakte gemacht. Allen, die mir zugehört haben oder mit denen ich gesprochen habe: Vielen Dank, und wenn Ihr Fragen habt, meldet Euch bitte!
Die Folien zu dem Talk "Eight Rollouts a day keeping downtime away" habe ich unten angehängt.
This is a description of how we are working at Booking.com with respect to code generation and deployment.
It is not a technical talk, there is not a single line of code. Instead I am describing the business, the culture and the organizational and personal framework that control development and operations.
The short summary of what we are doing is shown an this slide.
Essentially, we generate code and roll it out as quickly as is safe. There are reasons for that: Booking is a rather unique environment, and it is less insane than it sounds, because there is a technical, organizational and personal framework around it. It is also rather stable, we are successfully doing it for several years now.
But let me start at the beginning.
Hello, my name is Kristian Köhntopp, and I am working for Booking.com.
We are doing Online Hotel Reservations, as you can see on the website. It is all that we are doing:
- We selling online, there are no brick and mortar stores that we have. We do in fact not even advertise offline.
- We are doing hotel reservations, we are not doing flights, rental cars or other stuff. Other parts of the Priceline group of companies are doing that, but we don't.
- And we are doing reservations, which means that we facilitate a contract between the hotel and a guest and cash in on comission later with the hotel. Doing it that way simplifies the dealings between hotel and guest and hotel and us a lot. There is no contract between the guest and us, it cannot be any simpler than that.
Speaking of simple: This is what the site looks like.
There is a search bar, date selectors and a search button, and then there is a bit of information to allow for serendipity and browsing. We do have a simple and clearly stated business mission, and we do have a simple site.
Nonetheless the hotel business is complicated, and very labor intensive.
Contrary to what you may think, most hotels are not part of a hotel chain, they are independently owned and operated. We need to maintain contracts with all of them individually, and we need to contact them several times a year for various reasons. We need to remind them to put rooms online to top up supply, we need to talk to them about the services they advertise and how they are doing that or we need to talk to them because of issues with bookings that have been made. We also need to send them faxes, for example many hotels do still receive bookings from us via fax.
We also need to talk to guests. Nobody wants to sleep under a bridge, so when there are problems with a booking, we are ready to help. We do have a call center, and we are doing service in your language, in the target countries language and in english. Like many of our customers, we do prefer to do self-service via the website, but we do answer our phones, around the clock, every day of the year. And we will help if there are any problems with your booking.
Any day of the year we are doing business in more than 110 jurisdictions and in more than 40 languages. We are one of the largest translation offices in Europe, and we have large customer support and hotel support departments. Of our more than 3000 employees, most are not in IT.
The problem that we have to solve looks like this.
Booking is a high-growth environment. As you can see, this is the number of hotels online with booking, over time.
A nice straight line. On a logarithmic scale.
This here is the Cynefin model. It can be used to describe market maturity, among other things.
The lower left 'chaotic' quadrant describes the Internet as a market at around 1996 to 1998. It is a market that does not even have successful business model, companies were measured in eyeballs captured, and everybody was frantically trying to find a way to extract money from people on the internet.
A 'complex' market is one where you have a business model that is proven to work, and you stick to it. The objective is now to build a practice and then improve it. You no longer try out new things, but modify existing things, and you are trying to build process. That is the internet past 2000, when everybody was building their business.
These markets eventually grow into a 'complicated' market, with the advent of quantitative data. You are still trying out new stuff, but you are now metering that as numbers, build models and improve on that. You do have a best practice, but it is not yet complete and certainly not optimal.
Mature markets finally are 'simple', best practice exist, and so does a canned catalog of responses to events, which need to be implemented efficiently and quickly. Companies in the 'complex' and 'complicated' markets are often growing in a growing market. Expenses matter, but cost is not what drives these companies, bringing in more revenue is. Best practice may exist, but also may be invalidated due to changes in the market or due to growth. Flexibility and velocity (development speed = adjustment speed) matters most for these companies.
Not many companies are in these markets, or in these states, though. Most companies are in the 'simple' phase most of the time - they are operating in a stagnating market and are growing at the expense of the competition. Efficiency and cost matter most for such companies, and they seek stability, which they need to optimize their processes in order to make them as lean and cheap as possible.
Booking is a phase 2 company transitioning into phase 3.
This is due to growth. Here you can see a hypothetial growth pattern, which reaches its maximum some time in August (green line).
Comparing the current year to the previous year, we see that previous years maximum is being reached in the current year some time in March/April. The rest of the year the company basically operates in a realm where it has never been before.
So the objective is to manage growth of a rapidly growing business.
The time to duplication is seriously below two years. What makes it manageable is that the year-on-year pattern is somewhat predictable, and the growth factor is somewhat stable if you smooth out day-to-day outliers.
But: In such an environment what worked last year will no longer work this year. This is true for technology used, but also for organizational growth or for approaches to training or supervision of people.
Due to growth, the database that last year still fit into memory is now disk-bound, write rates of the current year cannot longer be handled by last years disk configurations, and approaches to search or reporting do no longer scale to this years data volumes.
The organization you build last year does not work for this years headcount.
And the training that you have set up for last years newcomers does not scale to this years personnel growth, and the content is outdates, also.
New bottlenecks that have never been experienced before in the history of the organization spring up all of the time. Doing nothing is not an option, growth will kill you within the current year, and last years knowledge is obsolete and wrong.
In such an environment the management objective is to deliver the required structure in time, in order to match the expected and predicted size of the organisation. And that is not just tech. Tech is in fact the least important factor.
Companies are social engines. When you have a business and a business model, and you work alone, you are running a process in your brain - a set of rules that describe how to handle the business if this or that happens.
Scaling a business means externalizing this process into a set of written rules that can be taught and run on more than one brain. That is what companies and growing a company is about:
Externalizing the program that is running in the head of the founders, in order to run it in a distributed wetware processing environment, to run it on many brains.
Writing it down is also good, because it forces people to make unclear ideas explicit and face all details as they are being pulled out of the squishy and vague brainspace and put into writing. What is written is also accessible and open for discussion, and conscious review and change, so writing down process is good to make it accessible and enabling reference to it.
So the shortest possible way to state the problem we are facing is: We are changing, because we are growing.
If we are not addressing that, we will die from overload. There is no way around that, there is no rest or stability for us, ever. We need to embrace that, we need to embrace change and live it. At a scale.
That is what we do. That is Booking.
We came to this conclusion consciously some time in 2006.
Back then we had less than 40 people in IT. These were very good people, most of them had been around a long time, even back then, and they wrote the code of the site, maybe even rewrote it multiple times. In face, we still have people around today, who did write 'Booking.com' multiple times and the CPAN modules underneath that power it, too.
Even back then, we already had conditional code in production that could be enabled and disabled at will, and we did collect statistics about that. We also had not a proper ops department, but devs were also doing ops work. Some more than others.
We had a lot of severe problems back then.
One of them was an extremely inefficient onboarding process for newbies. They would be not producing useful code for a very long time, some for 6 months or more.
We had unstable and slow rollouts, and stability problems due to that and insufficient monitoring. Code was hard to test, because it had too many variants (conditional code!), and it was also hard to test because it was changing too fast.
We were also drowning in ops work, and everything was manual.
We sat down as a company at that time, and made a few decisions, then acted on these. In fact, for a short time we even stopped hiring in order to be able to do that.
Management decided that it wanted a startup. Even if we were to become a four-digit sized company in personnel, we wanted to be able to act like a startup: What we value most is velocity, development speed, time to adjust to change. Performance, Latency and Stability are important, but we will do them only as needed. We will focus on Velocity.
We will not put this velocity on the road without providing a sense of direction. Instead we will make only changes that have a positive impact on business, proven with hard quantitative metrics. That is, we need to put code out and let people use it, fast, in order to be able to meter how that change affects business with real customers. Only code that influences our business for the better is allowed to live.
And we need to scale our method up, in order to be able to do that, in order to create more of these great old ones that know every subsystem in our system so well.
That is when we needed to stop and write down even more stuff.
"Wait! We do have a method? What is our method of development?"
This is a street scene in Brighton in the UK.
What you are looking at is a Shared Space. That is a local region of public space with traffic on it, but pedestrian traffic, bicycle traffic and cars are not separated, and there are no zebra crossings, traffic signs or lights at all.
Traffic in Shared Space is supposedly way safer than in conventional areas, and the throughput is also higher.
The theory behind a Shared Space is the observation that people will always drive as fast as possible, as long as they feel safe.
If you are creating an environment where you are not providing any hints that can make people feel safe, if you are not making decisions to stop or go with any signage, traffic lights and so on, then they will realize that they, and only they themselves, can make safe decisions about their driving. And they will evaluate risks properly and act correctly.
Well, in fact it is not quite so simple, we need to look at that in more detail.
In order to enable a responsible individual to make safe decisions, three things are needed.
The first is education. Like Germany, the Netherlands and the UK do have a very intense traffic education which starts in Kindergarten, continues in school and is complemented by a formal drivers education and a test in order to get a license. People in these countries know how to behave in traffic safely, in any role, be it as a pedestrian, a cyclist or as the driver of a car.
Next is visibility. In a Shared Space, traffic signs are not taken down. The streed is instead reengineered in such a way that participants in traffic can see each other easily, and that only a single decision needs to be made at any time. That is for example why dangerous crossings in such areas are rebuilt as roundabouts. So we do have people that know how to behave in traffic, and we enable them to notice each other and to engage with each other and negotiate a way to deal with the situation as it arises.
And finally, and that is only possible because of the two other things, we take away everything that can bias or short circuit these negotiations in any way. That is, we empower them to make their own decisions and live with the consequences - that is what being an adult is about, after all. And that works.
Compare this street scene from Brighton with an average street scene. This one is from Hamburg, I believe.
Shared Spaces work as development environments as well.
We are hiring accomplished and experienced developers.
We expect them to know their trade. We are teaching them about our trade. We are painting them blue, Booking blue.
We are then enabling these people to notice their surroundings by providing proper monitoring.
The effects of changes to our systems should be obvious, the actions of others on the systems should be obvious as well. People should be able to engage and negotiate around impediments on the spot.
System behavior should be linear and predictable. Systems shall not fail without warning - we are not building for efficiency, we are building systems that are as simple as possible and that provide feedback about unstable or dangerous zones before failure, when they enter these load levels.
"Obvious" is a system property we value.
Such people and such an environment enable us to empower our people.
We are taking away all mechanism that makes decisions, we are giving back responsibility to our developers. They decide, as individuals and as a group, how to work, how to implement and the if and when of deployments.
So there are rules.
Or actually, questions that anybody doing anything must be able to answer with a resounding and confident 'Yes'.
The first: "If you break it, will you even notice?"
If you cannot answer positively here, you have no place in production. You do need training, and a test system that you can safely break. That is not a problem, we have that and we will provide it to you.
We hired you, and it is our responsibility to teach you this.
The second: "If you break it, can you fix it?"
If not, get review. There are plenty of great old ones around in all their tentacular glory and we taught you their names. Go to them, let them review your change, and ask them about their thoughts.
They will not decide, because it is your code, and you are responsible, but they will offer their advice and opinion, and will try to predict how your change will behave in our systems as they are.
Still, it is your responsibility to act or not act on that.
We are calling all this 'Craftsmanship'. A craftsman is an expert in his field - and your field is your change, your project.
A craftsman takes the piece he makes from the idea stage, discussions about possible changes with product owners and managers, through design, implementation and test down into rollout, all the way. Our devs, all of them, are rolling out code, all of the time. They are responsible for every single aspect of their change.
They will of course fail. That is how we learn, after all: If we succeed, we are only confirming what we already know. From failure, from every single failure, there is a lesson to take away. On failure, it is important to notice to fail, quickly. One then needs to abort to a safe position, and then decide how to proceed.
A bad fail is one that takes long, because failure is not noticed. Or one that has no plan B, C, D and E, providing safe positions. This kind of fail is full of extended suffering and pain.
This is the result of all of that - the amount of failure in minutes per month.
As you can see, it works pretty well most of the time. We have had a problem in April, where we failed to fail properly, and we adressed that in the subsequent month.
So what exactly are we doing?
We are hiring experienced people, with a proven track record in Open Source Software, or a comparable closed source background. We are looking for people who do know the value of real world code, code that is earning actual money, over software engineering esthetics. We do have quite a bit of code that is butt ugly, and is making us very, very rich.
Anyway, the type of person we are looking for in hiring has experience with the tools we are using, and with the style of cooperation that is familiar from the great Open Source projects. They have a self-organizing, self-motivating personality that fits nicely into our culture of getting things done.
We are not doing software development in academia, and so most of our code is not build from scratch out there on a green lawn in the middle of nowhere.
This is the Teatro di Marcello in Rome.
It was completed about 10 years before christ, and used as a theatre. It fell in ruins, was converted into into a fortress in the 11th and 13th centuries, and then into a residence in the 16th century, remodeled into modern apartments today, and the traffic in front of it is also 21st century.
It is a good model for the realities of architecture and the realities of software engineering alike.
Most people do not write new code most of the time. They are instead remodeling existing code, upgrading it to be able to perform tasks that have not been anticipated by the original engineers that built these structures.
Unfortunately, this is not what education at university focuses on, neither in architecture nor in software engineering.
Anyway, we are not building our software on a green lawn in the middle of nowhere, but we have a codebase that is older than 12 years, and are changing that, a lot.
Yet we do acknowledge that code has in fact a very finite lifetime. In face, more than 90% of all code in frontend is thrown away after testing for business impact.
We are focusing on measuring quantitatively the performance of site changes in terms of business metrics.
That is, for each and every change, no exceptions, we want to know how it affects conversion, customer care load, loyalty and so on. We also want to know the technical cost of a change, in terms of memory, execution time, and so on.
For that purpose we created the experiment-framework, which allows us to roll out conditional code that is inert at first, and which we can put life very gradually.
Enabled experiments are exposing a certain percentage of the userbase to either a base variant or a changed variant, and are keeping track of the effect that either variant of an experiment has.
This is the enemy, which is killed by experiments.
It is the HIPPO. The Highest Paids Persons Opinion.
We have found through experimentation that there is no way to evaluate the success of an experiment other than actually trying it out. The experiments suggest by HIPPOS fare no better than experiments suggest by experienced developers, newbies or random people on the street.
That leaves exactly one way to validate changes: Implement them, and roll them out. As fast as possible.
Then put users into one or two experiments, and record the outcome of users exposed to either variant. And turn off experiments that are unsuccessful.
Successful experiments are turned full on, even if they are painfully inefficient or otherwise lacking: They are already earning money, and we are not going to miss out on that.
But we do turn the performance team on them, in order to optimize them. That means that such an experiment, which is no longer experimental at all once it is full on, is being either refactored or rewritten in a different way with identical functionality and look.
It also means that the performance team and any performance optimization always happens on business-positive code. We never optimize code that is detrimental to the business.
In order to be able to do all that, safely, each and every change needs to be small, trivial even.
Only then searching for problems on fail is fast, because the diff to the last known good version of the code is small. If a change is small, errors are spotted easily once they are known, and in at least half of all cases we do not in fact roll back, but fix the problem on the spot and roll forward instead.
Experimental changes are being encapsulated inside the experiment framework, that is what it is for.
That means, we can turn code on and off at will, and we can in fact turn code on very gradually and selectively. And we can see if it works, quickly.
For that we have an UDP based syslog-like system called Eventlogging, that collects not machine data, but application-level business events such as sales made, money earned, faxes sent, customers notified, and so on.
We are collecting this data, and are running life aggregations on the tails of these event streams, called Monitors. A Monitor can be queried on the command line, or visualized on a graph that is shown, well, on a physical monitor.
Here is one type of example Monitor, a very simple one that shows last weeks and yesterdays pageviews, and todays pageviews superimposed on that. Yellow vertical bars indicate configuration changes or other events, such as experiments turned on, rollouts and the like. If the red line dives down after a yellow bar you are responsible for, you know you just fucked up.
Experiments and monitors are giving us feedback from the real world. This is better than testing and simulation: The best simulation of a real world user is, well, a real world user.
We are locked in to our model of development. We are utterly and inescapably dependent on small changes, since we have nothing that enables us to debug or test large changes. So we must roll out often in order to survive.
Sometimes we don't, and that hurts a lot. We do have "app" servers which present the catalog of hotels, and "book" servers which make the actual reservation. "book" changes slower than app, and if we indeed do not rollout "book" for some months, as we did once, it will take many weeks to fix the code so that it does not break bookings.
Similar things happen in the PCI zone, where the "crypt" servers are. Changes here do not work well with our method at all.
We are also locked into fast and functional monitoring: If Eventlogging is down, or is lagging because monitors are not running, we cannot roll out. We need to fix Eventlogging first instead.
We do have physical monitors all over the place in our workplace, and we are running the most crucial business statistics on them for all to see.
This is also reflected in our corporate culture: We are obsessed with data-based reasoning in meetings and sessions.
Some tools have features that are not working well with us: GIT branches for example are a tool to accumulate patches into a larger patch which can then be merged with a main branch.
We do not like large change sets, and consequently we are not using GIT branches a lot, if at all. They create problems in the way we are working, as we try to have a unified codebase everywhere, if possible (with one exception). But that means we need to roll out systems such as "book" and "crypt" even if they do not change on the surface, in order to be able to notice if change in underlying libraries is breaking them.
Our code changes fast, and often. That does have performance implications, and makes capacity planning hard. Because we do focus on velocity, not efficiency or latency, a lot of bad code is put life, and that has potential to limit capacity, if we are not monitoring this closely.
We do monitor capacity in production, of course, by playing with the weights on load balancer front ends. By laying a probe into the tested cluster, we are not generating load, but measuring response time. When response time goes up, because the tested system is saturated and a queue builds, we watch for errors and finally abort the test.
This gives us a lot of insight: We learn the capacity of that part of our system and can interpolate to the whole system. But we are also observing system behavior under overload (remember the slide about linear systems and being able to get warning?), and will know how critically loaded the system is before it fails. Simulations cannot do that.
So we are doing this more or less since 2006. Does it work? Well, it does, for us.
To make it work, it needs support at the C-level, CEO, COO, CFO and so on. Remember that companies are social engines that run the processes that existed inside their founders minds on a larger scale, on many minds.
Here is how the mind of my boss works:
He is sitting next to a new face over lunch, and asks that person when they started and how they fare. The new developer answers that they are getting around nicely, and he asks further if they broke the site already. The answer is of course a shy 'no'. With a smile the CEO asks 'What? How can that be? Are you not working? What am I paying you for?'
In earnest, we are all trying very hard to not break the site. But of course it is inevitable that somebody making a change will eventually fail, and break it.
So it is pretty much understood that everybody has broken the site already in the past, and will do so again in the future. It is a feature of how we work, and people are instead taking pride in not breaking the site twice in the same way, ever, or noticing a problem and rolling back fast.
In order to help our people to not break the site, we are championing craftsmanship, the combination of Fachwissen - possible choices to make, and the costs associated with them - and Sachwissen - the choices Booking made in the past and how they came to be.
We are also teaching business to devs and development methods and thinking to business people, so that they have common goals, vocabulary and understand each others problems.
This is in fact working frighteningly well: At the IT developers meeting this summer, the first session was about this years business and how it is developing. In the Q&A that followed, the developers did ask actually smart business questions.
And finally, at the core of what we do there is giving back responsibility to the people doing things. Sometimes we have to force that and make people understand that they can get counseling and advice, but they cannot get somebody making decisions for them.
If you write code, you are the only expert on that code that we have. Nobody else, anywhere in the world, exists who would know better about that code. So it is your call - is it ready to roll out? This avoids risk compensation, and it forces people to actually care about what they are doing and how things work.
The 'surprising' result is: If you create an environment for adults - give them the knowledge and the tools necessary to navigate, and treat them like adults, they will actually act like adults. They do care about the site, about each others work and about future implications of what they are doing.
Doing that will affect your entire (IT) organization.
For us it means:
Small projects, quick feedback. If it needs a project plan, it is probably too large. Split it up, create measureable deliverables as small as possible, and act the entire thing out as a sequence of dependent small projects.
Also, build teams of interdisciplinary experts. Let business people sit with developers in 'turn laptop screen and have a look'-distance. Let operations people sit there as well. Enable people to see and hear each other, and work next to each other.
Announce the goal. Our goal is velocity. Development speed. We are doing other things, but they are always second to that.
That means we are deploying a lot of badly written, horribly performing, or otherwise broken code. Code that hurts when you look at it. Schemas that should not be. For suprisingly many people it is hard to get their heads around that. Or if they do, they still are unable to accept that, and have to quit because they cannot stand imperfection to this degree.
Audits have been a problem. We are of course partitioning off our PCI zone, and are keeping it as small as possible.
We are also doing another thing, and that is apparently rather innovative: We are implementing the necessary controls not as choke points that can slow down developers, but as trailing controls that generate an audit trail after the fact and then crossreference that with the necessary change requests, mails or other documentation in order to satisfy the auditors.
We are certified, and the auditor actually were impressed with our solution. The developers like it as well, because PCI mostly does not get into their way at all.
So is this for you? - the summary
Keep in mind that we are a very special, very quickly growing environment.
We are operating in unknown territory for the larger part of the year. In our environment, slack is good. We do not follow the cult of efficiency, in fact, if we did, we'd die. That is because growth is not completely stable, so you need buffers and healthy safety margins in order to manage sudden demand or changes.
If you are in a stage 4 environment (Cynefin 'simple'), and most companies are, then this is likely not for you. You have no need to the speed of adjustment that we are creating here.
There are some things that will work in any environment, though. You should look at them, and then decide what they will look like in your company.
And at the core of it all, you need to live by the rules:
1. If you break it, will you even notice?
2. If you break it, can you fix it?
LaHaine sez, "According to the first prognosis, the German Pirate Party has entered the state parliament of the city-state of Berlin with 8,5% of the votes, counting for 14 of a total of 149 seats."
This cop attempted to be sympathetic. He proceeded to tell us that he was trying to be a gentleman by just giving us summonses instead of taking us in for questioning, because that was what "they" wanted him to do. If he just gave us warnings and told us to leave, he would get in trouble for "doing nothing all day." He went on to say that all he did when he was growing up was "do Tae Kwon Do and go to school." "Are you trying to say that we are bad people for sitting on a bench in a park and eating doughnuts?" I asked him, just trying to figure out where he was going with this. "No, no, I'm just saying that I never got in trouble. Sometimes I play basketball," he said, pointing at the courts behind him. Not in that park, he doesn't. Not unless he has a kid strapped to his back at the time.