- Wednesday, May 14, 2008
A DVCS Story
Seattle, 1990
Bob leaned back, stretched his arms and took in the view from his window. It was a lazy Sunday, and he'd just finished reading his messages on his favorite BBS. An aging Amiga decorated with pink stickers courtesy of Bob's daughter sat at his feet. His sleek new home phone was perched prominently on the corner of the desk, the cord snaking its way to the wall by way of a splitter shared with his modem.
As Bob surveyed the scene, his phone started to ring. Pausing a moment, he smiled, enjoying the harmonic ring of the new phone that he lovingly customized earlier that morning before answering.
"Hey Bob, it's Alice!" Alice, a co-worker, was one of Bob's closest friends. They had both graduated from the computer science program at the same school, managing to stay in touch over the years, usually by trading "war stories". Recently, Alice had joined Bob's group from another company. To the chagrin of Bob and his co-workers, she was brought in directly as a lead programmer. "Bah, it's just a title", Bob was fond of saying. He hid his resentment well when interacting with Alice socially.
"Bob, my life just changed forever." Bob's eyebrow slowly began to raise. His mind raced through the possibilities. Had she been promoted again? Was she moving on to another opportunity? I wonder if I'll make lead if she does!. On the verge of Bob's moment of imagination turning into an uncomfortable pause, Bob snapped out of it.
"Alice, my goodness, I've never heard you so excited."
"Okay, you know I'm not normally ahead of the technology curve," Alice continued. "But today, I just got a cellular telephone!".
Bob's eyes narrowed. A cellular phone? He'd recently seen a story about them on the evening news, and thought they were the most ridiculous thing he'd ever seen. Who on earth needs to make phone calls in the middle of the street?, Bob had thought to himself when he saw the story. Bob responded forcefully. "A cellular phone? Those crazy looking things with the huge antennas? Alice, you've got to be kidding me." Bob admiringly moved his hand across the smooth plastic shell of his phone's handset.
"I'm serious! I know they're a bit different from what you're used to, but to be honest I kind of like the styling. I didn't realize how much fun it could be to be on the cutting edge of technology. You should have seen the looks I got in the park down the street this morning!"
Bob rolled his eyes and interrupted. "Look, I'm sure it's fun to be mistaken for an FBI agent, but you can't possibly think this is rational. When the heck would you need to make a phone call in the middle of the street? I have two phones at home, and another at work. What else could I possible need?"
"I can see where you're coming from. I'm sure that to you it doesn't sound that different from your home phone. I mean, you pick up the phone, dial the numbers, and it connects to the other side. Nothing groundbreaking there. But it's so freeing to be able to talk anywhere. I can completely understand how this might seem like a novelty. To be honest, listening to myself explain it to you, it doesn't really sound that compelling. But imagine the possibilities! I'm always connected. I can have meetings in my car, and I can leave the office for a long lunch on a sunny day without worrying about missing out on anything important."
Bob chuckled, making sure he drew it out long enough to suggest that he'd heard enough. As the final note of his chuckle diminished, Bob tried to finish the conversation.
"Okay, Alice, I guess I can sort of see how that might be nice. But learning a whole new speed dial system, and working with a phone I don't understand just doesn't seem worth it for those few situations where it might be useful. I'm glad your happy with your purchase, in any case."
Alice shook her head slowly, and put in a last word before hanging up. "You'll see, Bob, just wait."
Seattle, Present Day
Bob was crouched beneath his desk, reaching for the mini-USB cable attached to the back of his computer. He plugged the other end into his battery-drained Blackberry, and its bright welcome screen came to life. Come on, come on he urged. After a brief delay, the phone's LCD dimmed and the network connected. A high pitched bell ring alerted him to a new message, and Bob quickly pressed the "Read" button. She remembered!, Bob thought to himself excitedly. His daughter had sent the text message announcing her SAT results, just as promised. I knew it!, thought Bob, proudly inspecting the score. She's smarter than her old man after all.
As Bob turned to his computer, he heard the familiar sound of his favorite classical melody. He had only last week taken an MP3 of the song and made a custom ring tone, something that impressed his daughter far more than his last promotion. Who's calling me during lunch?, he thought. Leaning back, Bob peered under the lip of his desk, eyeing the blazing white LCD screen of his phone, now perched atop his computer chassis. Recognizing the name on caller ID, he immediately grabbed the phone and answered.
It had been several months since Bob last spoke to Alice. The time between their phone conversations had grown progressively longer of late, and Bob was happy to hear her voice. It had been five years since Alice left for Silicon Valley to run the software group at a small search startup. Having moved his way up to middle management, he was always excited to hear from someone who still had their finger on the pulse of technology.
Alice got right to the point. "Bob, I am about to change your life." Bob sensed excitement in her voice. In the two decades they had known each other, Alice had embraced the role of technology evangelist, and Bob that of technology skeptic. They each enjoyed their roles, and Bob could tell immediately that Alice couldn't wait to tell him about the next big thing. He gathered up as much feigned skepticism as he could in a feeble attempt to mask his genuine curiousity, and offered a response.
"Alright, what is it this time?"
"DVCS. Distributed Version Control Systems", Alice responded. "We just migrated our entire source control system to Mercurial. I think in the first week using it we've already gained 100 hours of productivity. The developers love it. I've made it my mission to tell everyone I know."
Bob pursed his lips. She's got to be kidding, he thought, this is what she's so excited about?
Bob announced his skepticism. "Alice, Alice, Alice. I've been around a long time, and it's not quite so easy these days to put one past me. What are you really so excited about?". Alice laughed long and hard.
"Oh you old curmudgeon, here we go again. Look, we've gone through this before, and I am going to convince you that distributed version control is serious stuff. Why are you so skeptical already? What have you heard about it?"
"To be honest", he started, "I first started hearing about distributed version control when we hired on a new developer who'd been working on an open source project. He was almost as excited as you about it. He had some trouble explaining what was so great about it, so he sent the entire team a link to a video where the presenter bashed our current version control system for an hour. The guy who manages our VCS server was really offended. So as far as I can tell, distributed version control is only relevant in open source projects run by opinionated bullies who see diff & patch as a perfectly acceptable source control system. I'm an old corporate soul -- I haven't used patch since college, Alice, and I don't miss it."
"Ah, I know the video you're talking about," Alice said. "Forget about that. I'm going to make the DVCS case for you, right here, right now. Pretend you've never heard of it."
"You've got your work cut out for you. To be honest with you, nobody has ever really bothered to tell me what problem this thing solves for me."
"That's always the first question!", Alice said. "I've actually practiced this speech on a few other ex-colleagues, and everybody asks that question. Unfortunately, that's the wrong question to be asking. Instead of asking what problem it solves, you should be asking what new possibilities it offers. That's been the real win for us."
Bob leaned as far back as his chair would allow, and propped his feet up on his desk. He had been through this many times before, and he knew he was in for a long ride. He gave Alice her opening: "Alright, I'll admit to being slightly intrigued. But we're really happy with our current VCS, everybody here knows how to use it, and it handles everything we need without any issues."
Alice smiled widely. She normally had to work much harder than this to get Bob on the hook. She knew that beneath the curmudgeon, he still had a passion for technology. It was her job to make sure that he didn't lose that, and she embraced it proudly.
"Okay, I'm going to ask you a very important question, and I expect an honest answer. Has your source control server ever had any downtime?"
Bob thought for a moment. "I suppose, sure, but nothing more than the usual. I mean, our server is up all the time, really. We plan all our maintenance for the weekends, so other than the occassional hiccup ..."
Before he could finish his sentence, Alice interrupted. "Yes! The hiccups! Those 15 minutes of downtime because of the urgent security patch, the 10 minutes of slowness when two machines are pulling down copies of the repository! They happen, and you shrug them off because it's just the way it is. But do you know how much productivity you lose when somebody loses their train of thought because the server isn't available for their 'annotate' command?"
"Look, I get what you're saying. But really, these hiccups aren't very common", Bob retorted. "I mean, even if they were, you have to compare them against the cost of having everybody switch over to a whole new system. The last time we did that was a total nightmare. One of our developers even quit over it! And that didn't involve explaining to everyone this crazy new distributed source control model. I'd have a revolution on my hands."
Alice responded reassuringly. "I completely understand. I had my own reservations until very recently. One of our developers had been trying to get us to adopt a distributed source control system for a few months. She liked DVCS so much that she had found some way to use it on her local system and still interoperate with our centralized system. I was skeptical for the same reasons as you, though, Bob. I convinced myself that the server was more reliable than it was, and I tried to forget that our VPN would sometimes be down all weekend, forcing developers to come into work to make fixes. Like you, I didn't really think it was worth it."
Bob interjected impatiently. "OK, so what changed your mind?"
"Last week our system administrator told us they were going to be rebooting the server during lunch time. This wasn't a big deal at the time. Everybody had advance warning, and they made all of their checkins before lunch, just in case something went wrong. When we returned, we found our source control system in pieces, its parts splayed all over the desk. The admin had installed a few security patches, and the server wasn't booting. He said he thought it might be bad memory, but he wasn't really sure."
Alice's horror story had reminded Bob of a similar situation a year ago when something remarkably similar happened at his own company. They expected the server to be repaired in 30 minutes, but it dragged to 60, 90, then 120 minutes. By 3:30 PM most of the office had cleared out, leaving a few frustrated developers emailing files to each other as they raced towards the important deadline which was now in jeopardy.
"Bob, are you still there?" Bob was growing increasingly nervous as he enumerated in his mind the many things that can go wrong with a server.
"Yes, yes, sorry, you just reminded me of something. Please, go on."
"Okay, so the admin couldn't give us an ETA on the repair. We were feeling really helpless, and called a team meeting to decide on a protocol for getting work done while the server was down. Fortunately, one of our team members had a plan. You remember that developer I told you about earlier who was using the DVCS on her own system?" Alice was talking excitedly now, and didn't bother to wait for Bob's answer. "So, she immediately took control of the meeting, and laid out the plan. She had created a complete copy of the source system just before it went down, you know, just in case of emergency, and it was checked into the DVCS repository on her system. She gave the group some brief instruction on how to copy the repository, and in less than thirty minutes our entire team was back to work with their own copies of the source."
"Wait a second," Bob interrupted. "You're telling me that you temporarily switched your entire source control system over in a half hour in the middle of an emergency? Forgive me for being skeptical, but I'm not buying it. For starters, what happens to your revision history?"
"When she took a backup of the centralized system, she had done it via a script that preserved the entire revision history. She said that migrating from centralized systems was common, and writing the script was a breeze since most of the 'heavy lifting' was built into Mercurial."
Bob made no attempt to mask his skepticism. "Okay, I get that, and it's all very impressive, really. But what about all the work you did while the server was down? How did you get it back to the main repository once it was revived?"
"That's the best part. We didn't." Alice heard a squeak on the other end of the phone, and knowing that Bob was about to ask her if she was crazy, continued without hesitation. "The admin kept working on the server throughout the day. During this time, after everybody had their servers up ..."
Bob cut her off. "Wait, what? Servers? I thought they just checked out the source code from their co-worker's machine? Where did all of these servers come from?"
"That's exactly the difference between DVCS and a centralized system. You don't check out the source code from the server, you clone the repository from the server to your local machine. Once you've made the copy, you're a server too."
Bob was finding this DVCS concept more and more ridiculous as Alice went on. "Wait, wait, wait. Last I checked you had a team of 20 developers."
"We're up to 25 now", Alice corrected.
"And you're telling me that in response to a temporary server outage, you created 25 separate source control servers, and that's somehow a good thing? How on earth does anybody know what state the source code is in? Please tell me I'm missing something. I'm starting to believe you've gone mad."
"I know, it sounds like chaos. And frankly, it could be if we let it. That's where process comes in. In this case, everybody initially cloned the repository from the same server. So, we designated that as the official server where everybody shared their changes during the downtime."
"Aha!" Bob was sure that by the end of this conversation, he'd would bring Alice back down to earth. "So you're taking this confusing and complicated distributed system, and making it act just like a centralized system! What happens when that server goes down? You're no better off!"
Alice let out a slightly annoyed chuckle. "Not exactly. If the server we've designated as our central server goes down, which it surely will at some point, so what? That's yet another wonderful feature of a distributed system. We've got backups of the code all over the place, without even trying. Every developer's server contains a backup copy of most or all of the code from the shared repository. So, if the main server goes down, we can designate another server as the central server temporarily, or we can not worry about it, since developers can make checkins to their own servers, only sharing changes when needed."
"Wait a second!", Bob interrupted. "If everybody is making checkins to separate servers, how do you get those changes back together in the so-called central server once it's back up?"
"That, my skeptical friend, is called a merge."
"Oh, great," Bob murmered, "you mean like merging branches?"
"That's exactly what I mean. In fact, repositories and branches are pretty much the same thing in Mercurial. Whether or not there is a central server, everybody commits changes to their own servers. Period. That's the only place you can commit to. When you want to combine your changesets with those from your repository, you need to merge them."
Bob groaned. "Yuck! Branches are such a huge pain. And you're seriously suggesting that making people merge branches every single time they want to share changes is a good thing? I'm serious, Alice, if you need some help, I know some really good people."
"You actually have a point there. We used to think branches were a pain too. In fact, we only kept a single branch aside from our main repository because merging branches was such a pain. This is another huge difference between distributed and centralized systems. Because you merge branches all the time, distributed version control systems make it incredibly easy. Easy to create branches, easy to share them. Flexibility, Bob, that's what it's all about."
"I don't see what's so flexible about chaos. All this complexity, just to compensate for a few minor blips of downtime? I'm not buying it."
"Yes, initially it was just a response to the downtime." Alice sensed that she was losing Bob, a phenomenon she was quite familiar with. She softened her tone. "On the day we started using Mercurial, it was going to be a temporary thing. My thought was the same as yours, that it was handy as a temporary crutch, but too complex to keep around. However, our temporary outage ended up being not-so-temporary."
"What a nightmare." Bob said. As Alice spoke, he had been working hard trying to convince himself that this was an incredibly uncommon scenario. Geez, I guess I'd better confirm that we're backing our server up daily, Bob thought. And I wonder if I can get budget for a backup source control server?
Bob had neglected to mention that he had only a year ago spent $25,000 on licenses for his company's source control software. While he was proud to now be using the same source control system as big companies like Google and Microsoft, it left him with very little budget for hardware. As a result, their source server was running on an underpowered machine, and it was not uncommon to hear complaints about its sluggishness. Bob had also recently received the bill to renew his yearly contract for support and upgrades. That was going to take another $5,000 out of his budget, at the cost of some much needed upgrades to his developers' machines.
Alice continued. "It turned out that our source server's RAID card had died, and we were going to be down for three more days before the part could be delivered. It was in these three days that we discovered the great value of DVCS, not before. One group of our developers, for example, had previously been trying to do 'buddy builds', where they share their changes with each other before committing to the main repository. They had initially tried to coordinate this work on a separate branch, but the pain of merging so frequently was killing them, not to mention the fact that they often forgot to make their changes on the branch. Then they started emailing each other patches, but this, too was prohibitively cumbersome. To get around this, they ended up sharing their source directories to each other directly, and emailing the names of files that needed to be copied over. Changes got lost all the time, it was a total mess."
Alice was talking quickly now, hoping to stop Bob from interrupting. "The afternoon we started using Mercurial, however, all this stopped. Each developer would finish their work, commit the relevant changes to their own servers, and then make the changesets available for their colleagues to pull and merge into their repositories."
Alice had said the magic word. "Ack! More merging!"
"Yes! More merging! I'm making it sound too complicated though. You only have to actually merge files if the changes you're pulling overlap with your own changes. When you do inevitably have to merge, though, you do so after you've checked in your own changes. Thus, even if the merge goes awry, you never lose your work because of it, something that is extremely important to us. Regardless, merging is so simple and fast that it doesn't even matter."
"Well, my source control system supports branches," Bob replied, "why couldn't we do this?"
"Many of the new and interesting workflows enabled by DVCS are possible in centralized systems, but are simply too much of a pain in practice to have any chance of adoption. Plus, they can grow organically. You don't need to explicitly decide to start working on a branch for buddy builds, you can just make some checkins and choose where to send them. You're always working on a branch, in effect, since your local repository is a branch. Not to mention the fact that you don't need to ask a server administrator for permission to create a branch."
"Okay, so you can do buddy builds", protested Bob, "and if the server hiccups it's not that big of a deal. But so what? Those things aren't very useful for us. People around here just come in and do their work. Even if these benefits are as profound as you suggest, I really don't think it's worth training my entire team on some complicated new source control system."
Alice hadn't expected to sway Bob in only a single conversation, but she was nonetheless growing a bit frustrated with her progress. She had used the same sales pitch on others to great effect, but there was something missing. It just wasn't having the same effect on Bob. She continued with her defense. "The things I've mentioned so far are 'big deals'. Having a server down even for only a minute or two is a huge deal, in fact. Developer time is expensive, and knocking a developer out of the zone because of a server hiccup is, in my mind, totally and completely unacceptable. Heck, even a bit of server latency can ruin a developer's ability to stay in the zone."
"One thing I think you're forgetting is that we're using the best centralized system money can buy," Bob said. "I'm sure our server is significantly more speedy and reliable than your free, open source centralized server was. Everybody here is really happy with it. I've even heard them bragging about it to friends at other companies."
"Okay," said Alice. "I'll play by your rules, even though it's unfair. Let's assume that your server is up 100% of the time, never any downtime, never a hiccup, never a bit of slowness. Oh, and everybody has constant access to it. DVCS still offers advantages beyond simply not needing live server access to get work done."
"Oh? Like what?", queried Bob, now feeling proud of his progress towards the goal of dragging Alice back into reality.
"Like no more 'check-in races', for starters. It's a common complaint every place I've worked. The first person to commit their changes to the central server avoids having to be the one who merges changes. Thus, the next committer is forced to perform the merge, whethero r not they are the best person to merge the changes. I've actually seen people switch workstations for ten minutes so the changes could be merged by the proper resource. How ridiculous is that? With DVCS, because changes are committed locally, nobody is ever denied the ability to do so because of someone else's changes. And once they are committed, anyone can pull the changes together and merge them. If you had two developers racing towards a deadline, another less busy developer could volunteer to do all of the change merging."
"Yeah, well," Bob smirked, "I guess our needs aren't quite as sophisticated as yours. Don't get me wrong, some of the stuff you're talking about sounds interesting, and hey, if we weren't using the best centralized system that money can buy, I might be more interested. Really, whatever flaws there are in our current system, we've made the necessary adjustments. People here are happy. We're doing great work, the team gets along well. I just don't see the case for DVCS."
Alice realized that there was no more she could do. She bit her lip, resisting the urge to respond defensively. Planting the seed, she realized, was the best she could hope for with Bob. She leaned back in her chair, shook her head softly, and ended the conversation. "You'll see, Bob, just wait."
Comments
- Wednesday, May 14, 2008 5:37:08 PM by Tom FakesThe cool Rails kids are using Git
- Wednesday, May 14, 2008 5:37:47 PM by Tom FakesThe cool Rails kids are using Git
- Wednesday, May 14, 2008 5:38:44 PM by Tom FakesOoh, these comments with captcha suck
- Thursday, May 15, 2008 7:30:33 PM by Willtl;dr, tell me when the cliff notes are ready.
I actually use a DVCS for my smaller personal projects; its called fileshare. Branch and merge are a bitch, and forget about history... - Monday, May 19, 2008 5:18:26 AM by Chris HahnGreat write-up, I was laughing almost the whole way through!
- Tuesday, May 20, 2008 8:24:46 PM by Matt Doar"What?", said Bob, "three days! where was the standby server, ready for recovering from the backup? You did take backups, right?"
"Oh, kind of, and we always meant to get around to a standby server" confessed Alice with a toss of her head, "but, anyway, now our build machines each have a complete repository on them, so we use them for releases."
"How do you choose which one?" asked Bob.
"The one that we used last time" smiled Alice, impishly.
"Wait a minute, hold on now." said Bob, "I've just remembered. What about access control? How do I stop just anyone changing the code?"
"Well, whichever machine we declare to be the server for holy. blessed builds that we release, we're really careful about who is allowed to merge changesets to it."
"Really careful? I want a bit more than that" humphed Bob.
"Ok, we have the usual authentication using SSH against our LDAP server and then we do authorization based on file paths in the changesets."
"Sounds better; I guess you do know what you're doing. But does the authorization stay with each cloned repository?"
"Er, we're working on that." admitted Alice, "but once it does, we'll be able to move our official release repository to any machine we like."
"Groovy!" said Bob, glad he had been able to add to the story of DVCS. - Tuesday, May 20, 2008 11:13:19 PM by JohnAlice, stunned by the turn of events, thanked Bob for his time. She knew the transition to DVCS was inevitable and was happy to see that Bob and his team would get a productive head start. Always a student of humanity, the level of resistance she encountered was immense yet the conversations were illuminating.
- Wednesday, May 21, 2008 2:26:48 PM by AI think more use cases for the offline access might help -
You are demoing some middleware to a customer and need to make many fixes / changes at his place. You still can have your version control with you! Not all companies will allow offsite access to the version control server. Further even within a company many times some locations cannot access the version control or its extremely slow. - Wednesday, May 21, 2008 7:09:30 PM by Jack9The biggest problem with Version control is merging time, not downtime. More DVCS nonsense.
- Thursday, June 12, 2008 6:49:42 PM by John StrackeBob had only once heard of cellphones in 1990? What hole was he living in? By 1990, they were showing up on TV (I remember a pocketable one on LA Law), and car phones were going for one penny plus contract.
- Saturday, June 14, 2008 11:56:40 PM by AaronJack9, have you used a DVCS? If not, you're as bad as "Bob"; blind and ignorant and unaware. Go try hg. It's not as slick and fast as bitkeeper, but it sure beats svn, cvs, p4, etc. Every time I have to go back to a centralized system, I'm depressed.
- Saturday, June 28, 2008 8:38:39 PM by Jakub NarebskiWhile "one branch per repository" (or "repository / clone is branch") workflow is default Mercurial workflow, distributed version control systems of now (for example Git, but this includes Mercurial with localbranches extension) can use "multiple branches in repository" workflow. This workflow has grown at demand of users (as you can read for example in blog of Junio C Hamano, current git maintainer).
One issue of note is that while push based workflows are not that different from first glance from centralized SCM workflow (but for the backups, commit then merge, developing feature in multiple commits on topic branch, off-line commits, etc.), pull [request] based workflow require having project maintainer (like Linus Torvalds for Linux kernel, or Junio C Hamano for git itself; he/she does need to do all the work if he/she sets up "network of trust" Linus Torvalds was talking about in his Git Talk at Google Tech Talk). But if you read old but (in some) good "The Mythical Man-Month" having maintainer for a project is one of recommendations, I think one that survived passage of time. - Saturday, June 28, 2008 8:41:54 PM by Jakub Narebski> "The biggest problem with version control is merging time, not downtime. More DVCS nonsense."
@Jack9: with distributed version control systems you can commit, _then_ merge when you have time to resolve it; heck, you can ask somebody else to resolve merge in his/her repository if you are not able to do it for some reason (Linus does it sometimes, by the way). - Saturday, June 28, 2008 11:06:40 PM by Jakub Narebski> "The biggest problem with version control is merging time, not downtime. More DVCS nonsense."
@Jack9: An alternative to merge (well, not exactly aternative, more like alternate workflow) is something that is called "rebase" in Git, "transplant" in Mercurial, and "graft" in Bazaar. See for example http://www.kernel.org/pub/software/scm/git/docs/git-rebase.html
Git has also additional feature making repeated merging easier, namely git-rerere (reuse recorded resolution of conflicted merges). This tool records how did you resolved merge so you don't have to repeatedly do the same resolving. As far as I know this feature is unique to Git. Lately there was talk on git mailing list about extending feature list of git-rerere, among others deleting resolutions and enabling sharing recorded resolutions...